Interviewing
The Easiest AI Mistake Story to Tell Is a Free Catch
A candidate's AI mistake story is worth grading on the difficulty of the catch, not the fact of one. A fabricated citation or a dead link is a free catch: the error announced itself and a single click settled it. The expensive catch is an answer that survived a careful read and was wrong in a detail the work rested on. Ask what the candidate checked it against, when they noticed, and what it would have cost.
The takeThe story has become a credential, and credentials get rehearsed. A candidate who has fielded this question three times will arrive with the best catch of their career, sanded smooth. So grade the second answer instead, the one that is harder to prepare: how they knew to look there. A habit names a class of trigger, something like every number that moves a recommendation goes back to the filing. An account of one lucky afternoon names nothing that would repeat it. Both stories sound the same until you ask.
Where Olive fits
Open a role and see what the work shows
An interview can collect a candidate's account of a catch they made once; it cannot collect a catch. Olive puts one in front of them as work: a role-grounded assignment, an assistant that will overreach if nobody stops it, and a human reviewer who writes or edits the findings on what was checked, against what, and when.
Assess your shortlistWhat does the fabricated-citation story actually prove?
A fabricated citation story proves the candidate opened the link. A model that invents a court case or a paper leaves an error that announces itself: the name resolves to nothing, the address returns a 404, the volume number belongs to a different journal. Checking costs one click.
Any candidate who opened a link can tell the story about making that click, which is the problem with treating it as evidence.
The free errors share one property. The output carried the flaw on its face: a source with no referent, a date that contradicts the sentence before it, arithmetic that does not close, a quoted clause that is in no version of the policy.
Anyone who slows down finds them.
A second story arrives phrased as instinct: the passage read like a machine wrote it, so the candidate went looking.
Unaided reading is weak at that judgment. In a 2021 study presented at the Association for Computational Linguistics meeting, non-expert evaluators asked to separate GPT-3 text from human writing across stories, news articles and recipes performed at random chance 2.
Three quick training methods lifted accuracy only to about 55%, and not consistently across the three domains.
The evaluators were crowdworkers reading short passages of 2021-era output, not specialists reading work in their own field. Whatever a specialist could do, a catch resting on how the prose felt has nothing checkable behind it.
Both stories are worth about a minute. A candidate who has fielded the question before brings whichever version landed best last time, and rehearsal does nothing to the difficulty.
Which AI errors never announce themselves?
The expensive errors are right in form and wrong in one load-bearing detail. A growth figure taken from the wrong base year. Code that passes the fixture it was handed. A summary that drops the qualifying clause and keeps the finding. A citation that resolves to a real paper making a different claim.
Read straight through, every one of those errors looks correct all the way to the decision it supports.
A 2023 field experiment at Boston Consulting Group measured the cost of that failure among its own consultants. On one task deliberately chosen to sit outside the tool's range, 84.5% of the control group reached the correct answer, against 60% and 70% in the two conditions given GPT-4, an average drop of 19 percentage points 1.
The coaching arm ran the wrong way. Consultants who got a prompt-engineering overview alongside GPT-4 finished 24 points below the control, and consultants given GPT-4 with no overview finished 13 points below it 1. More instruction in the tool, worse result, on this one task.
As a group, they could not tell which side of the line the task sat on.
Your candidate can be in that position on any task, and so can everyone already on the team. The catches carrying information are the ones where the output gave nothing away and the person went outside it anyway.
Every release moves the frontier, and the percentages here came from one task, one sitting and a mid-2023 model, so they are not a standing fact. The shape of the failure outlasts the percentages: a wrong answer that looked like every other answer, on a task that looked like every other one.
Why does a candidate's catch count say so little?
Because a count of catches fills up with whichever errors are easiest to see, and those are the cheap ones. The same arithmetic sits under the headline accuracy figure for the federal employment eligibility verification system. The system checks work authorization rather than model output, and the arithmetic carries over anyway.
In a synopsis published on 28 January 2010, US Citizenship and Immigration Services reported Westat's finding that E-Verify queries returned an accurate response 96 percent of the time 3.
Only 6.2 percent of those queries concerned unauthorized workers.
Of that group, roughly 54 percent, with a plausible range of 37 to 64 percent, were inaccurately found to be work authorized, primarily because of identity fraud, where the details presented match government records 3.
A system can be right nearly all the time and still miss half of what it was built to find, because nearly all the time is the easy caseload.
The fieldwork ran to 2008 and the synopsis is dated January 2010, so none of it describes E-Verify today or any commercial checking tool. The measurement problem transfers. A candidate's three stories tend to describe the errors easiest to see, and the ones that cost money are the hardest to see.
Ask what settled it, when they noticed, and what missing it would cost
Three follow-ups separate a habit from an anecdote, and each one asks about the checking around the error. What did you check it against. How long after the output did you notice. What would have happened if you had not. Ask the same three of every candidate who tells one of these stories, in the same order, so the answers can be set side by side later.
Checked against what does the most work of the three. An outside source is a check. Re-reading the output is not, and neither is asking the model whether it was sure.
The filing, the schema, the signed contract, the colleague who owns the number: a strong answer names whichever one settled it.
Timing says where the habit lives. Noticing at the point of use, before the number went into anything, is a routine. Noticing six weeks later, because a reviewer downstream pushed back, is a colleague's routine.
Cost says whether the candidate knows what the check was for. Someone who can say what the error would have done, and to whom, holds a picture of which outputs earn the checking budget. Someone who cannot name a consequence has described the catch without that picture, at least so far.
None of the three asks the candidate to grade themselves.
A fourth question is harder to rehearse: how did you know to look there. Going one question past the prepared answer is how follow-up questions expose real understanding. A habit names a class of trigger. An anecdote names a moment and stops.
Keep the difficulty question apart from the general claim, which anyone can give in the same words. Testing the answer that says they always double-check is the same problem one level up, with the same fix: ask for the instance, then for what it was checked against.
Write down what free, real and habitual sound like
Decide what a strong answer sounds like before you hear one, or the story you happened to enjoy becomes the bar. The Office of Personnel Management's 2008 guide defines a structured interview by three properties: the same questions in the same order for every candidate, a common rating scale, and interviewers who agreed in advance what an acceptable answer looks like 4.
Federal HR guidance from 2008 binds no private employer. The third property is the one a story question needs most.
Three anchors, from weakest to strongest:
- Free. The error announced itself, and the candidate names no trigger, no outside source and no consequence.
- Real. The error was subtle, the check went to something outside the model, and the catch came late.
- Habitual. The check happened at the point of use, on a trigger the candidate states as a rule covering a class of work.
Anchor all three levels, not only the top one. A scorecard that names the best answer and grades the rest by feel hands the round to the better storyteller.
Score apart, then combine by the rule you wrote down.
Kuncel, Klieger, Connelly and Ones classify a group consensus meeting as a holistic method in their 2013 meta-analysis, alongside individual expert judgment 5. A method counts as holistic there when evidence is combined by judgment rather than by a rule applied the same way each time.
A debrief ending in a shared feeling about the catch is that method.
Nothing in the paper argues against holding a debrief, which is where evidence gets surfaced and corrected. The classification lands on the last step, where separate scores get combined into one verdict.
Difficulty is partly a fact about the reader, and that is one case against grading it. A candidate two years into your domain catches a base-year error that a stronger checker from another field reads past. Grading the catch grades exposure alongside judgment.
The same exposure sits under a miss, where reading past an error in an unfamiliar field can be a knowledge gap as easily as a lapse in checking.
The invented court case is still worth hearing. Ask why the candidate opened that citation and not the three beside it. Sometimes the answer is that they open all of them, every time, on anything that will be quoted back to somebody. The story was never the evidence. The answer to that question is.
Common questions
Is a rehearsed answer a red flag?
No. Candidates rehearse the stories they expect to be asked for, and a polished AI mistake story says nothing about the catch underneath it. Grade the content instead: what the error was, what settled it, when it was noticed, and what missing it would have cost. A fluent first answer followed by a blank on how they knew to look there is the informative combination. Treat the polish as a fact about interview preparation, which is a different thing from the skill on the scorecard.
What if a candidate has never caught an AI error?
Ask what they use it for before drawing anything from the silence. Someone who drafts with a model and rewrites every line by hand may genuinely have nothing to report, and someone whose model output is low-stakes text has little worth catching. The answer worth pausing on is the one that treats never catching anything as a fact about the tool's reliability. Follow it with a case: hand over a short output with a plausible error in it and ask what they would confirm before sending it on.
Does a subtle error always beat an obvious one?
No, not always. A candidate who caught a subtle error by accident, six weeks late, after somebody downstream complained, is describing luck with a long tail. A candidate who caught an obvious error at the point of use, because anything carrying a number goes back to the source, is describing a routine that will also catch the subtle ones. Difficulty is the first question rather than the only one. When difficulty and trigger disagree, the trigger and the timing carry more.
How do you keep AI mistake stories from turning into a trick question?
Say what is being graded. Tell candidates the story matters less than the checking behind it, then ask the three follow-ups in the same order for everyone. Nothing here depends on surprise, and a candidate who prepares better answers because the criterion was stated is demonstrating exactly what was asked for. The version that turns into a trick is the unstated one, where the bar lives in one interviewer's head and nobody else in the debrief can name it.
What goes in the interview note afterwards?
Write what the candidate said about the AI error, not what you concluded. Five short fields hold it: the error, the thing it was checked against, the gap between the output and the catch, the consequence they named, and the trigger in their own words. Five fields fit in an applicant tracking note and can be compared across candidates weeks later. A rating with no record behind it cannot be revisited, defended, or explained to the next interviewer who asks why this one advanced.
References
- 1. Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality (Working Paper 24-013) ✓ mitsloan.mit.edu Supports the claim that the expensive AI errors sit on tasks that look like the ones the tool handles well: on one task chosen to sit outside AI capability, 84.5% of the control group answered correctly against 60% and 70% in the two GPT-4 conditions, an average drop of 19 percentage points, with the prompt-engineering condition down 24 points and the condition given no overview down 13.
- 2. All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text ✓ aclanthology.org Supports the claim that a catch resting on how the prose felt has nothing checkable behind it: non-expert evaluators separating GPT-3 text from human writing across stories, news articles and recipes performed at random chance, and three quick training methods lifted accuracy only to about 55%, inconsistently across the three domains.
- 3. Westat Evaluation of the E-Verify Program: USCIS Synopsis of Key Findings and Program Implications ✓ uscis.gov Supports the claim that a headline catch rate is dominated by the easy caseload: E-Verify queries returned an accurate response 96 percent of the time, only 6.2 percent of queries concerned unauthorized workers, and roughly 54 percent of that group (plausible range 37 to 64 percent) were inaccurately found work authorized, primarily through identity fraud.
- 4. Structured Interviews: A Practical Guide ✓ opm.gov Supports the instruction to ask the three follow-ups in the same order and settle the acceptable answer in advance: the guide defines a structured interview by the same questions in the same order for every candidate, a common rating scale, and interviewers agreeing beforehand what an acceptable answer looks like.
- 5. Mechanical Versus Clinical Data Combination in Selection and Admissions Decisions: A Meta-Analysis ✓ gwern.net Supports the claim that a debrief ending in a shared feeling is a holistic combination method: the meta-analysis classifies group consensus meetings alongside individual expert judgment, on the defining characteristic that data are combined using judgment, insight or intuition rather than a rule applied the same way for each decision.
5 sources, numbered by first appearance. Every one was opened and checked against the claim it carries. How Olive sources claims
General guidance for hiring teams. What works at one company and one volume may not transfer to yours.
Olive assesses how a person works with AI. It does not detect AI-written documents, and it never produces a score, a ranking, or a match percentage for a person. Candidates read the same report the employer reads.