Interviewing
One Miss Buys You a Case Review, Not a New Round
One hire whose AI judgment turned out weaker than the interview suggested is not enough evidence to rebuild a hiring loop. It is enough to run a case review. Open the scorecards, name the round that claimed to cover that judgment, and read what evidence it actually recorded. Three findings are common: no round tested it, a round tested it and wrote nothing down, or the evidence was there and the debrief moved past it. Only the first one is about the loop.
The takeA vivid miss produces an extra round because adding one feels like accountability and costs nothing anybody in the room can see. The cost is real and it lands on people who are not in the room. Every future candidate pays an hour, and the round that already had the best shot at this signal stays exactly as vague as it was. A team that adds a stage without fixing the one that missed has bought a story to tell the next time, not a better decision.
Where Olive fits
Open a role and see what the work shows
Olive is priced per attempt rather than per seat, and an attempt returns six evidenced findings on one candidate, each written or edited by a person and anchored to a timestamped excerpt from the session. Ten attempts a month are free and results are kept until you delete them, so a case review a year from now has something specific to reread.
Assess your shortlistWhy can't one miss tell you the loop is broken?
Because one hire is one observation and a loop's accuracy is a rate. A process that works produces misses. A process that is broken produces misses. From inside a single case the two look identical. What one case can tell you is narrower and more useful: which round was supposed to see this, and what it wrote down at the time.
The best-evidenced interview is still a noisy instrument. The 2022 re-analysis of the personnel selection literature by Sackett, Zhang, Berry and Lievens estimates the validity of structured interviews at .42 and unstructured ones at .19, as corrected correlations with supervisor ratings of performance, not accuracy rates 1.
Those numbers move a great deal from setting to setting. The same authors' 2023 follow-up puts an 80 percent credibility interval of .18 to .66 around the structured interview 2. Nothing in a single hire tells you which end of that range your loop sits at.
Misses are the expected output of a working loop.
One failure of AI judgment is a wrong answer on a task that looks like the ones a model handles well.
Harvard Business School's 2023 field experiment covered 758 Boston Consulting Group consultants. On one task chosen to sit outside the model's capability, the groups using GPT-4 reached the correct answer 19 percentage points less often than the control group 3.
One study is one observation too. A single task, a single sample, a model from 2023.
What the experiment shows is narrow: the people using the tool could not tell which side of the line their task was on.
An interview rarely arranges a task on the far side of that line. That omission is part of why a candidate who demos brilliantly with AI can still fall apart in month one.
Start with the scorecards from that candidate's loop
Open the file for the hire, not the process document. Read every scorecard, every interviewer note and the debrief record, then answer one question in writing: which round claimed to cover the judgment that failed, and what did it record as evidence?
The answer is usually shorter than the loop's design implies, and often it is blank.
Expect the record to be thin. A content analysis covered 104 interviews reported across 103 articles published between 1997 and 2010. Studies describing a structured interview used an average of 5.74 of the 15 structure components in Campion, Palmer and Campion's framework 4.
Several of the rarest components sat on the evaluation side. Detailed notes were described in 19 percent of those interviews, and no discussion between interviews in 4 percent 4.
Those figures count what researchers described in studies they published, and part of the shortfall is studies that left components undescribed. Even so, read them as a ceiling. Practice is unlikely to carry more structure than the literature describing it.
Write the finding as a sentence with a round in it. "The technical screen owned verification and its note says strong communicator" is a finding. "The loop missed it" points at nothing anyone can change.
Which of three findings does the record support?
One of three, usually, and each one has a different fix. Either no round ever tested the competency, or a round tested it and recorded nothing about what it saw, or the evidence was recorded and the debrief moved past it.
Only the first is a fact about the loop's design. The other two are facts about how the loop was run.
The first has a recognizable shape. The loop asked candidates to describe how they work with a model. Nobody watched anyone do it.
The second finding leaves a specific trace: a rating with no behavior under it.
The trace worth having is a named capability. Text-mining the post-interview notes on 7,650 candidates hired at a large Chinese internet technology company found that the more job-relevant capabilities a note named, the better that hire's later performance ratings, promotions and retention 5.
The measured size is small: a one standard deviation rise in how closely a note matched the job analysis went with about a 2 percent rise in performance 5.
The finding is correlational, at one firm, and every person in the sample had already passed the interview, so rejected candidates are invisible by construction.
A scorecard that forces a rating on a line the round never reached manufactures that trace. Before the next loop runs, it is worth putting not observed on the scale and counting how often it fires.
The third finding is the debrief, and it shows only if the debrief record says how the evidence was weighed. Kuncel, Klieger, Connelly and Ones classify a group consensus meeting as holistic combination, alongside one expert's individual judgment 6. What makes a method holistic is that evidence is combined by judgment or intuition rather than by a rule applied the same way each time.
Holding the debrief is still right.
The meeting is where evidence gets surfaced and challenged, and where two interviewers reading one behavior in opposite directions can be settled by naming the behavior rather than the verdict.
The trouble sits in the last step, when the conversation ends in a shared feeling with no stated rule. Across nine job-performance studies, the same meta-analysis found holistic combination predicting less well than a mechanical rule, though it tested no debrief on its own 6.
Change what a round asks for before you add one
Adding a round is the expensive fix, and the easiest one to propose. Changing what an existing round asks for is cheaper, faster to run, and aimed at the finding the case review actually produced. A loop that never watched anyone work with a model does not need a sixth conversation. It needs one of the five it already runs to become an exercise.
Sackett, Zhang, Berry and Lievens argue for combining different kinds of evidence. Their 2023 follow-up estimates that a mechanically combined composite of predictors reaches a validity of about .61, above any single method in their table 2.
Four unscored conversations are not that composite. They largely measure the same thing four times, and the estimate assumes weights and intercorrelations a stacked panel does not have.
The practical version is one line inside a round you already run.
Hand the candidate a short piece of model output with a real error in it, and ask what they would check first and against what.
Then look at what an extra hour would displace. Cutting the round that has never changed a decision is the same audit run in the other direction, and a loop that adds without cutting grows every time somebody is disappointed.
Log this miss, and wait for the second
A second case and a third, with the same round named in each. Log this one now: the role, the round that owned the competency, what the record held, and what the hire actually did in month two. Three entries in that log make a pattern with an address.
One entry is a story, and a story is what an extra interview usually gets built on.
The honest case against waiting is that a small team is slow to reach three. A group hiring four people a year may take years, and every hire made in the meantime goes through the loop as it stands.
Two changes can go in now, because neither costs a future candidate an hour: a written record naming the behavior each rating rests on, and a debrief that states its combining rule before anybody speaks.
If the bar does move, decide in advance what would show that it worked.
Proving a change to the hiring process actually improved anything is harder than making the change, and much harder to start after the fact.
The file you opened at the start is where this lands.
Under the round that owned this judgment there is now a sentence saying what was asked and what was seen. The next case review has something to read.
Common questions
Is one bad hire enough evidence to change the interview loop?
Enough for a case review, not for a new round. One hire is a single observation and a loop's accuracy is a rate, so a working process and a broken one both produce misses that look identical from inside one case. What a single case does support is a review: name the round that claimed to cover the competency that failed, and read what it recorded. If no round ever tested it, that is a design gap visible from one observation. If a round tested it and wrote nothing down, the fix belongs in the record.
What goes in a hiring case review?
Five things, written down: the competency that failed after the hire, the round that owned it in the loop's design, the evidence that round actually recorded, the debrief note showing how that evidence was combined, and what the hire did in month two stated as behavior rather than as a rating. It takes about half an hour with the scorecards open. Keep the entries in one place, because the value arrives at the third one, when the same round gets named three times.
How many misses before the bar should move?
Enough that the same round is named more than once. A single case cannot separate a weak stage from the false positives any selection procedure produces, so the log is what turns an anecdote into a pattern with an address. A team hiring four people a year may wait years for a second case naming the same round, and a finding that no round ever tested the competency needs no second case at all. Both are reasons to change what an existing round asks for rather than to add a stage.
Should you add an AI exercise after a hire's AI judgment disappoints?
Change an existing round first. An added stage costs every future candidate an hour and leaves the round that already had the best shot at the signal exactly as vague as it was. The cheaper move sits inside a round you already run: hand the candidate a short piece of model output with a real error in it, and ask what they would check first and against what. If the case review finds that no round ever put a candidate in front of a model at all, that is the finding that justifies changing the loop, starting with what one existing round asks for.
What if the interviewers saw the problem and the debrief overrode them?
Then the finding is about how evidence was combined, not about the questions asked. One meta-analysis of selection studies classes a consensus meeting as holistic combination, the same family as one expert's intuition, and that family predicted job performance less well than a rule applied the same way each time, though the meta-analysis did not test debriefs on their own. Fix the combining step: agree in advance what a low rating on that competency does to the decision, and hold the meeting to it. Cancelling the debrief is the wrong reading, because the meeting is where evidence gets surfaced and challenged.
Does a miss mean the interviewer was wrong about the candidate?
Not necessarily, and the scorecard settles it. An interviewer who recorded a specific behavior and rated it honestly did the job, whatever happened in month two. What is worth checking is whether the rating names a behavior at all. One study of interviewer notes at a single Chinese technology firm, covering only people who were hired, found that the more job-relevant capabilities a note named, the better the hire's later performance, which is a reason to grade the record rather than the person who wrote it. A rating with no behavior under it cannot be reread, and a case review is the moment somebody tries.
References
- 1. Revisiting Meta-Analytic Estimates of Validity in Personnel Selection: Addressing Systematic Overcorrection for Restriction of Range ✓ gwern.net Supports the claim that the best-evidenced interview is still a noisy instrument: structured interviews estimate at .42 and unstructured interviews at .19 in the 2022 re-analysis by Sackett, Zhang, Berry and Lievens.
- 2. Revisiting the design of selection systems in light of new findings regarding the validity of widely used predictors ✓ cambridge.org Supports the 80 percent credibility interval of .18 to .66 around the structured interview, and the composite of about .61 used to argue for a different kind of evidence rather than another conversation of the same kind.
- 3. Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of Artificial Intelligence on Knowledge Worker Productivity and Quality (Working Paper 24-013) ✓ mitsloan.mit.edu Supports the claim that one failure of AI judgment is a wrong answer on a task that looks like the ones a model handles well: on one task chosen to sit outside AI capability, consultants using GPT-4 were 19 percentage points less likely than the control group to reach the correct answer.
- 4. The Structured Employment Interview: Narrative and Quantitative Review of the Research Literature ✓ doi.org Supports the expectation that the recorded evidence will be thin: 104 interviews from 103 articles published 1997 to 2010 averaged 5.74 of 15 structure components, with detailed notes present in 19 percent and no discussion between interviews in 4 percent.
- 5. Predictive Validity of Interviewer Post-interview Notes on Candidates' Job Outcomes: Evidence Using Text Data From a Leading Chinese IT Company ✓ frontiersin.org Supports the claim that a note naming job-relevant capabilities carries signal about later outcomes: across 7,650 hired candidates, the number of job-related capabilities named tracked later performance, promotions and retention, at roughly 2 percent of a performance measure per standard deviation of the matching score.
- 6. Mechanical Versus Clinical Data Combination in Selection and Admissions Decisions: A Meta-Analysis ✓ gwern.net Supports the classification of a group consensus meeting as holistic combination alongside individual expert judgment, and the finding that, across nine job-performance studies, holistic combination predicted less well than a mechanical rule, with no debrief tested on its own.
6 sources, numbered by first appearance. Every one was opened and checked against the claim it carries. How Olive sources claims
General guidance for hiring teams. What works at one company and one volume may not transfer to yours.
Olive assesses how a person works with AI. It does not detect AI-written documents, and it never produces a score, a ranking, or a match percentage for a person. Candidates read the same report the employer reads.