When AI Is Allowed in the Coding Round, Judgment Is the Test

What Changes When AI Is Allowed?

Nothing about the problem gets easier; what gets scored moves. In a pooled study across three companies and 4,867 developers, those with an AI coding assistant completed 26.08% more tasks on average, a noisy estimate with a standard error of 10.3 points, and the biggest gains went to less experienced developers ⁴. With that lift available to everyone in the room, finishing stops being the differentiator, so the round has to measure something upstream of the finished answer.

The risk moves with the tool rather than disappearing. In one field experiment, consultants using GPT-4 on tasks within the tool's strong suit finished tasks 25.1% more quickly and produced work rated more than 40% higher in quality ¹. On a separate task chosen to sit outside that strength, though, consultants using AI were on average 19 percentage points less likely to reach the correct answer than a control group ². The consultants given a prompt-engineering overview beforehand did worse on that task than the ones given no coaching at all, so more fluency with the tool was not protection ². That is the failure mode an AI-open interview is built to catch.

A few things follow from that shift:

• Producing a solution is closer to table stakes than it used to be
• The graded axis moves to what you tested, refused, and kept for yourself
• A flawless-looking answer with no visible checking is not automatically a strong one

The two findings sit together on purpose. AI is genuinely good at the parts of a task it was trained on and genuinely bad at spotting when a task has quietly moved outside that range, and neither the assistant nor a nervous candidate reliably notices the shift while it is happening. An interviewer who has read the research is not looking for you to avoid the assistant. They are looking for you to behave like someone who knows the tool has a blind spot and is checking for it, which is a different skill from writing correct code unaided.

Don't Optimize for Speed

Speed is the wrong thing to chase, and self-reported speed is worse than useless. In a randomized trial, experienced developers using AI tools on code they already knew well finished 19% slower, despite forecasting a 24% speedup beforehand and still believing afterward that they had gained about 20% ³. If developers who were tracking the clock got their own speed wrong both before and after the fact, an interviewer timing you silently is not the test worth preparing for.

What replaces speed as a target is legible reasoning: stating the constraint you are solving for before you prompt, so the interviewer can see the shape of your thinking instead of reverse-engineering it from a finished function. Employers running these rounds well are told to score four things instead of a stopwatch: what got tested against something outside the chat, what got refused and on what grounds, what you kept for yourself, and what changed in the final call.

What Reads Well, and What Doesn't

A few concrete moves read as judgment rather than luck. Testing the generated function against a case you chose yourself, one the interviewer did not hand you, is one of the clearest signals available in the room. Saying out loud that you don't trust a line and then checking it is another. Both are visible acts an interviewer can point back to, which a silent, correct-looking paste cannot offer.

What reads badly is different from what you might expect: pasting an answer without reading it first, being unable to explain a specific line when asked, and treating a generated solution as finished the moment it runs without an error. None of these are about how much AI you used. A candidate who leaned on the assistant heavily and can explain every choice reads better than one who used it once and cannot explain the result.

For a longer answer on demonstrating this without narrating every keystroke, showing that you catch AI's mistakes is worth reading before the interview.

One habit sits between the two categories and deserves its own mention: asking the assistant a clarifying question before you accept its first answer. Interviewers who have run enough of these rounds can tell the difference between a candidate who treats the first output as a draft and one who treats it as a verdict, and that difference shows up well before anyone has tested anything.

How to Prepare Without Gaming the Round

Prepare the habit, not a script. Pick a problem you already know cold, generate a solution with an assistant, and force yourself to find one thing wrong with it before you accept it. Do this two or three times and skepticism gets easier to perform live, because it is genuinely there rather than rehearsed for the interviewer's benefit.

A short list worth rehearsing out loud, not just thinking through silently:

• Name the constraint before you prompt, in one sentence
• Pick your own test case, not the one already in the prompt
• State what you would check next if you had another five minutes

None of this requires memorizing a framework. It requires practicing the moment where you stop and check something, until doing that on camera feels like your own habit instead of a performance.

If you only have time for one rehearsal, make it this: solve a familiar problem with the assistant, then stop the moment it finishes and ask yourself what you would need to see before you would trust the answer with your own name on it. Whatever you find yourself checking is very likely the same thing an interviewer is quietly watching for.

See what gets scored >