I understand why they built their own editor and code completion tool based on Codex, since they wanted to be able to collect detailed data about the editing sessions, but I think doing so raises serious questions about the applicability of their results. They’re ignoring all of the UX design of a real code assistant and focusing only on the underlying model.

For example, including temperature control in the UI is just stacking the cards against the AI group. No sanely designed AI assistant would draw attention to that parameter, and there’s not much reason for it to be user configurable at all. It would be like if you were testing how well drivers performed in cars with the radio on, and you put a big dial in front of the radio group for controlling antenna position. You’re just encouraging them to waste effort on something they don’t know how to adjust properly.