How Do You Test Whether a Candidate Notices AI Errors?

What does a seeded-error exercise actually test?

It tests verification under a confident assistant, and nothing else. Not AI knowledge, not prompt technique, not skepticism as a personality trait. You are looking for one narrow behavior: when a fluent answer contains something wrong, does this person go outside the conversation to settle it, and does the result change what they ship? That behavior is invisible in a conversation about AI and visible in ten minutes of work.

The failure it probes is ordinary rather than exotic. In Stack Overflow's 2025 developer survey, the most-cited frustration with AI tools was "AI solutions that are almost right, but not quite" at 66%, and more developers actively distrusted the accuracy of AI output (46%) than trusted it (33%) ⁴. Domain-specific tooling does not close the gap: Stanford researchers benchmarking purpose-built legal research systems found Lexis+ AI and Ask Practical Law AI produced incorrect information more than 17% of the time, and Westlaw's AI-Assisted Research hallucinated more than 34% of the time ¹.

What makes it a hiring question rather than a tooling question is that the person inside the work cannot feel it happening. METR ran a randomized trial with 16 experienced open-source developers on 246 real issues in repositories they knew well; with AI tools allowed the work took 19% longer, and afterwards the developers still believed the tools had sped them up by 20% ². In a controlled security study, participants with an assistant wrote significantly less secure code than the control group and were more likely to believe their code was secure ³. Confidence moves the wrong way under assistance, which is exactly why self-report is worthless here.

So the interview question "tell me about a time AI got something wrong" measures recall and storytelling. Every candidate has an answer and the answers are indistinguishable. The problem of judging AI-assisted work at all starts here, and the fix is to stop asking about the behavior and stage it.

How do you plant an error your candidate's field would catch?

The source material carries the error, never the model, and the mistake should be one a competent practitioner in that occupation would question while a fluent generalist would not. Model output is not repeatable (ask the same question twice and you get two different errors, or none), so the seeded error goes in the packet, the repo or the dataset the assistant reads from. Then it lands the same way for every candidate.

Six worked versions, one per field:

• Management consulting. The summary page of the market-sizing packet attributes a 2019 segment figure to the whole category in 2025. The underlying source is in the folder and says otherwise. The assistant builds the sizing off the summary because the summary is what it was handed.
• Financial analysis. The revenue bridge is off by one period: the opening balance is last quarter's closing figure, so price and volume reconcile to a total that is internally consistent and wrong. The assistant will narrate the bridge fluently in the wrong direction.
• Software engineering. The docstring says `retry(attempts, backoff)`; the shipped function takes them the other way round. The assistant writes to the docstring, the tests pass because the fixture uses equal values, and production would sleep for three attempts.
• Legal operations. A memo in the packet cites a real case with a pin cite to a paragraph that does not say what the memo claims it says. The citation resolves. Only reading the paragraph settles it.
• Marketing. The most quotable statistic in the research folder comes from a vendor's own 42-person customer survey and is written up as market-wide. It is the number every draft will lead with.
• Healthcare revenue cycle. A denial is coded as a coverage dispute and is actually a timely-filing miss. The assistant will draft a persuasive medical-necessity appeal for a claim that is simply late.

Nothing about this works uncalibrated. Hand the task to two people already doing the job: one should find the error inside ten minutes, and neither should call it a trick. If both miss it, the error is undiscoverable. If both find it in thirty seconds, it is decoration. Non-technical roles are not exempt from this: every occupation has material that reads settled until someone opens it, and naming yours is the whole authoring cost.

Write the rubric before you write the error

The rubric is four columns, filled in from the record rather than from impressions: named (did the error appear as a problem anywhere), timing (before or after the deliverable was drafted), instrument (what was opened, run or recomputed), and consequence (what in the final answer is different because of it). Two people should be able to fill those in independently and agree.

Independent agreement is the part most rubrics skip and every debrief needs. "Showed good judgment" cannot be scored twice the same way; "re-added the bridge column against the raw export" is a yes or a no. Write the observable, not the adjective. If you cannot state the observable, you have not finished designing the exercise. Whether two interviewers can agree on an AI-collaboration rubric at all is a measurable property of the rubric, not a matter of interviewer quality.

Compliance points the same direction as quality here. A step that decides who advances is a selection procedure, and work samples and simulations are named examples ⁵. The Uniform Guidelines are sharper still on the design choice: content validity holds to the extent the procedure is a representative sample of the content of the job, and a procedure resting on inferences about mental processes cannot be supported by content validity alone ⁶. Asking a candidate how they would check a confident claim is an inference about a mental process. Handing them a claim, a source and thirty minutes is a work sample.

Run the columns past whoever signs off on your interview loop before the first candidate, not after the third. A worked set of rows, what a rubric for an AI-assisted answer actually says, is an hour of writing that survives every task you ever swap in behind it.

What counts as catching it?

Four grades, not a yes or a no. Zero: the error is in the deliverable, unremarked. One: flagged as doubtful and left there. Two: settled against something outside the conversation, with what it showed written down. Three: settled, and the check changed a number, a recommendation or a stated limit. Most candidates who impress in conversation land on one; most hires you want land on three.

The distinction between one and two is the one that pays. A candidate who says the number looks off has an instinct. A candidate who opens the source, re-derives the figure and writes down the variance has an instrument, and an instrument transfers to the errors you did not plant. Grade the act in the record, never the assertion about it: "I double-checked that" with nothing opened is a zero, and it is the most common thing a nervous candidate says.

Grade selection too, because checking everything is not available in real work. Which claim did they pick, and was it the one the recommendation rested on? A candidate who verified three peripheral facts and shipped the load-bearing one unchecked has done worse than one who checked only the load-bearing claim. The opposite failure is real and less discussed: someone who challenges every line and finishes nothing is not careful, they are slow, and a seeded-error exercise flatters that behavior unless the rubric counts consequence rather than challenges.

The scale asks what was settled and what the check showed, and it asks that identically of a step the assistant touched and one it never did. A person who judged the assistant was the wrong instrument for a step and did that step by hand has demonstrated the exact thing being tested. Hiring for verification rather than production is the shift this rubric encodes, and volume of tool use points at neither.

What breaks a seeded-error exercise?

Four things, in the order you will hit them. The error is too generic, so everyone catches it or nobody does. It is undiscoverable inside the time limit, which makes it a trick rather than a test. It leaks after the twentieth candidate. And two interviewers turn out to mean different things by "caught it," which you find out during the debrief argument.

Sample size is the harder limit. One planted error is one observation, and a competent person can miss one thing on a bad Tuesday while a lucky one stumbles into it. Two errors of different kinds in the same task (one arithmetic, one attribution) cost nothing extra to author and roughly halve the coin-flip. Two separate tasks are better and cost a second afternoon.

Nothing here transfers between fields. The docstring says nothing about the market-sizing packet and the pin cite says nothing about the denial queue; the rubric transfers, the task never does. That is the real budget line if you are adding an AI-fluency round to an existing loop: one authored task per role, plus a rotation you will write yourself when the first one leaks, plus an interviewer who can read the work rather than the deliverable.

And the exercise tests the one error you thought to plant, in the one field you know well enough to plant it in. That is a narrower claim than it feels like while you are writing the rubric. A candidate can catch your seeded arithmetic error and still hand every framing decision to the model on the job. If you want the broader picture, the seeded error is one probe inside a larger question: what an AI-fluency test should measure covers more ground than any single planted mistake can.

See how it works >