Why Do Candidates Who Interviewed Brilliantly Struggle in Their First Quarter?

Why does a brilliant interview predict so little about the first quarter?

Because the interview samples the one state the job almost never occupies: supervised, time-boxed, on a problem someone else framed, with an audience and nothing riding on whether the answer is true. The first quarter is unsupervised, open-ended, and full of confident drafts that have to be checked by the person who ordered them. A round can watch someone reason. It cannot watch them verify.

Start with the ceiling, because it reframes the surprise. In the revised validity estimates that replaced decades of overcorrected meta-analysis, structured interviews are the best-ranked widely used predictor at a mean operational validity of .42, with an 80% credibility interval running from .18 to .66, and unstructured interviews at .29 ¹. That is the best case: a proper structured round, scored against a rubric. A correlation at that level says something real about a group of hires and very little that is certain about any one of them. A brilliant interview followed by a hard quarter is not an anomaly in that data. It is the ordinary width of the interval.

The second mechanism is that people cannot reliably report their own process. In a randomized trial, sixteen experienced open-source developers took 19% longer to complete issues when AI tools were allowed, and still believed afterwards that the tools had sped them up by about 20% ². They were working on their own repositories, on real issues, with nobody interviewing them. If self-assessment fails under those conditions, a candidate narrating how they work with AI in a 45-minute round is not a witness you can cross-examine into accuracy. They are describing a process they cannot see.

The third runs the wrong way from what a round rewards. Across 319 knowledge workers describing 936 first-hand examples of AI-assisted work, higher confidence in the AI predicted less critical thinking, while higher confidence in one's own ability at the task predicted more ⁵. Interviews reward the first kind, because it sounds like fluency. The quarter charges for the absence of the second. The polished answer everyone now gives is exactly the surface on which those two are indistinguishable.

Which part of the job is unsupervised generation?

The part where a model produces the first version and nobody else reads it before the work leaves your hands. Count it as a fraction: of the deliverables a person ships in a normal week, how many now begin as a generated draft, and how many of those get a second reader? That fraction is the share of the job your interview did not sample, and it is not the same number in two roles.

Adoption is broad but uneven. As of late 2024, 23% of employed US respondents had used generative AI for work in the previous week and 9% used it every workday ³. Every workday is the figure that matters here: it marks the roles where the first version of most artifacts is now machine-produced, so a weak framing habit compounds daily rather than monthly.

The shape of the use differs too, not just the amount. A classification of 200,000 anonymized Copilot conversations against O*NET work activities found AI applicability cutting across sectors, because most occupations contain information work. It also separated occupations likely to delegate a task to AI outright from those using it to assist an existing workflow ⁶. Delegation and assistance fail differently. A delegated task fails when nobody framed it; an assisted task fails when nobody checked it.

Two roles make that concrete. A marketing brief starts as generated copy, gets read by three people who are not its author, and its weakest point is usually the most on-message statistic, the one least likely to have been opened. That fails in weeks, in public, as a positioning line nobody can source. A diligence memo also starts as a draft, but the check that decides it, recomputing a figure against the filing, requires a practitioner's judgment about which figure matters at all. That fails later and more quietly, in a recommendation that reconciles to nothing. Same tools, different interval, different tell. Work out what AI actually does in the role before deciding which round in your loop is underweight.

What actually breaks between week two and week ten?

Four things, in roughly that order: the brief never gets framed, the confident claim never gets checked, the judgment that should have been kept gets handed over, and nothing the model produced is ever refused. None of the four is visible in an hour with an interviewer. All four are expensive at forty hours a week with a deadline attached.

• The brief was never framed. In the round you supplied the problem. On the job the request arrives vague, and the model resolves that vagueness confidently and in the wrong direction. The tell: a finished, well-made deliverable that answers a question nobody asked, arriving early.
• The claim was never checked. Almost-right is the dominant failure mode, not obvious error. In the 2025 Stack Overflow developer survey, the most common frustration with AI tools, at 66%, was output that is almost right but not quite, and 46% of developers said they distrust the accuracy of AI output against 33% who trust it ⁴. The tell: a number, a citation or a claimed behavior in the work that nobody can say where they got.
• The judgment was delegated. Total delegation reads as speed in week one and as a person who cannot answer a question about their own work in week six. The tell: they go quiet when the assistant is wrong or unavailable, and their rework arrives with no account of what changed.
• Nothing was ever refused. Accepting everything is the fastest route to a finished artifact, which is exactly what a timed round selects for. The tell: fluent editing and zero rejection (prose gets tidied, framings never get thrown out).

What those share is that each is an act, with a time and a place, that only exists once the work is long enough to be boring and real enough to cost something. An interview can capture a candidate describing all four with total conviction. The follow-up question is where the description thins out, and it is the cheapest correction available inside a loop you are not allowed to lengthen.

Make the round predict work, not performance

Change what the round asks for. An interview that asks a candidate to perform reasoning measures performance; an interview that hands them finished, flawed work and fifteen unstructured minutes measures whether they check. Producing more output cannot score anything once the artifact already looks done, so checking becomes the only move available. That one change converts an audition into a sample of the behavior the quarter bills for.

Build the exercise out of your own material. Take a real deliverable from the role, have a model produce it, then plant one error only a practitioner would catch: a figure on the wrong basis, a source that does not say what the paragraph claims, a dependency that does not do what its name implies. No preparation reaches your error, which is the point, because coaching reliably produces fluency about AI work, not the instinct to open the filing.

Write the rubric before the round, in the words of someone who has done the job: what strong, adequate and weak look like on each of the four behaviors above. Without that sheet, panels fall back on polish and confidence, which is what the interview already measured. Score acts rather than impressions (what was framed, what was opened, what was kept, what was refused) and record the reason beside each rating, because the reason is the only part another interviewer can audit.

Two smaller changes cost nothing at all. Say in the invitation that AI is expected rather than tolerated, so you stop measuring how well a candidate guesses your policy. And ask about one specific task with a date attached instead of a general account of how someone works; generalities are rehearsed, a particular Tuesday is not. If the loop cannot absorb another stage, add the judgment signal without lengthening the loop by replacing a round rather than appending one.

What can an interview never tell you?

Whether the person does any of it when nobody is watching. Every answer in a round is a self-report from someone with an obvious stake in it, and people misread their own process even while doing real work on their own repositories ². Closing that gap takes a longer sample with a genuine cost of being wrong, or a first-quarter measurement you actually collect and feed back into the loop.

So instrument the other end. Take four deliverables from a new hire's first eight weeks and record how many rounds of correction each needed and of what kind: wrong question, unsourced claim, no account of what the person did themselves. Three or four of those files per role, read side by side, tell you which stage of your loop is not doing its job. The answer is usually specific, along the lines of nothing here tests verification, rather than a general verdict that the interview was too easy.

Be careful what you conclude from that record. First-quarter output is confounded by manager, onboarding, project difficulty and luck, and a handful of hires will never separate them; what an assessment score predicts about performance is a claim that needs its own evidence rather than one you can borrow. Use the first-quarter record to find the behavior your loop never sampled, not to grade individual hires.

The last honest thing to say is that part of this gap does not close inside a hiring process. An interview buys 45 minutes; the quarter buys 500 hours, and no sample of the first predicts the second cleanly. The realistic ambition is smaller and worth having anyway: make one of those 45 minutes contain a real act (one check, actually performed, on material that arrived finished and wrong), because that is the smallest thing a round can hold that the quarter will also ask for.

See a sample report >