Interviewing
Two Follow-Ups Separate AI Skepticism From Refusal
Criticism of AI tools from a senior finalist suggests discernment when it names conditions and a problem when it names only the category. Two questions tell them apart. Where does the tool help in the work you actually do, and what would have to be true for you to change that view. Tasks, failure modes and exceptions in the answers are evidence of the judgment you are hiring a senior for. An answer that never leaves the technology is a risk where the job means directing other people's AI work.
The takePanels downgrade the vocal critic and almost never write down why. The note names a feeling, never the sentence that caused it, so there is nothing on the page to check. The reflex selects for enthusiasm at the level where enthusiasm is worth least. A senior who cannot say where a tool fails is not the safer hire; that person is the one who signs off on the confident wrong thing. Score the criticism in writing, or admit the mood got scored instead.
Where Olive fits
Open a role and see what the work shows
An interview can capture a candidate saying where a tool fails; it cannot capture them meeting the failure. Olive puts that in front of them as work: a role-grounded assignment, an assistant that will overreach if nobody stops it, and a human reviewer who reads what happened and writes or edits every finding, with the candidate given the same report.
Assess your shortlistWhat separates informed AI criticism from refusal?
Conditions. Informed criticism names the task where the tool fails, the failure it produces, and what the person does instead. It comes with at least one case where they still use it. Refusal names the category and stops there. The first is a boundary drawn on a map. The second is a wall, and no example anybody offers moves it.
Tone is not the tell.
A senior can be flat, tired and unpleasant about the whole subject and still be describing a real boundary.
An enthusiast can be warm for forty minutes and describe nothing.
In 2023, a Harvard Business School field experiment with 758 Boston Consulting Group consultants included one task chosen deliberately to sit outside what GPT-4 could do.
Correct answers came from 84.5% of the control group. The two groups working with AI scored 60% and 70%, an average of 19 percentage points worse 1.
The drop is evidence that people working with the tool could not reliably tell which side of the line the task sat on.
The 19 points are not a rate to expect: one sample, one task, a 2023 model. The shape of the failure travels even when the size does not. Not a refusal to answer, but an answer on the wrong side of a boundary nobody had drawn.
A candidate who can draw that boundary inside their own field is describing the line the experiment was built around. Noticing that a fluent answer is wrong is the competency that stays scarce.
Ask for an exception to the criticism, then a condition that would end it
Two questions, in this order, of every finalist rather than only the one who bothered the room. Where does the tool help in the work you actually do? And what would have to be true for you to change that view? The first asks for an exception to their own position. The second asks for the condition under which the position ends.
An informed critic answers the first with a task attached. Summarizing a deposition they were going to read anyway. Generating the test cases they then run.
The exception is usually small, which is what makes it credible. Someone in refusal answers the first question by restating the criticism at greater length.
The second question is where a position separates from a posture.
A person holding a view about conditions can name one: a retrieval step that cites the paragraph it used, an error rate low enough that checking costs less than doing, a review that catches the near misses before they ship.
A person holding a category cannot, and tends to say the technology would have to be fundamentally different, which names nothing observable, so nothing could ever show it had happened.
Both answers take the same three moves after them. The follow-ups that expose whether someone understood the answer they gave work here too: specify the case, invert the condition, then ask what would falsify it.
Ask both questions of every finalist, score them on one scale, and agree beforehand what an acceptable answer contains.
A structured interview is defined by those three properties and no others in the US Office of Personnel Management's practical guide 2. That guide is federal hiring guidance from 2008 rather than a rule that binds a private employer, and it names what makes an interview comparable across the people who sat it.
Why does a senior's criticism land as culture risk?
Because the debrief turns an unscored impression into a shared verdict without anyone writing the reason down. Somebody says the finalist seemed resistant. Somebody else agrees. What reaches the scorecard is culture, or energy, or not forward-looking, and the sentence the candidate said has been rounded off by then.
The meeting that ends in a shared feeling has a name in the selection literature.
A 2013 meta-analysis in the Journal of Applied Psychology by Kuncel, Klieger, Connelly and Ones classifies the group consensus meeting as a holistic combination method, the same category as one expert's intuition.
What makes a method holistic is that the data get combined by judgment rather than by a rule applied the same way each time 3.
The authors give that a number: holistic combination can cost a 25% reduction in correct hiring decisions at a selection ratio of .30.
Nobody measured that inside a company.
The 25% is a calculation, built from how well these methods predicted performance across the studies the meta-analysis pooled. The paper reports it at that one hiring rate, across a range of assumptions about how often a hire works out.
The meta-analysis never tested consensus meetings on their own, so it cannot say how much of that loss belongs to the meeting. A debrief is where evidence gets surfaced and challenged. The holistic step is the last one, when the combination is a conversation instead of a stated rule.
Seniority raises the price of the criticism without changing its content.
A junior who dislikes the tool reads as green and gets a ramp. A director who dislikes it reads as an obstacle to a program the company has already announced.
The measured gains, meanwhile, sit with the people the panel is not worried about. A generative AI assistant went to 5,179 customer support agents at one software firm in a staggered rollout. Issues resolved per hour rose 14% on average: 34% for novice and low-skilled agents, and almost nothing for the experienced, highly skilled ones 4.
Customer support at one software firm, on a pre-ChatGPT-era assistant trained on that firm's own transcripts, is the whole of that measurement. The study says nothing about what a senior gets from a current model. At most, it gives a reason to question the premise running the debrief, which is that the enthusiastic answer is the capable one.
When does a critic become the wrong hire for the level?
When the job means directing other people's AI work and the candidate holds no view about where it helps. A leader who cannot name one task worth delegating cannot set a boundary for a team, cannot review what comes back across it, and will either ban the tool or ignore what it produces. Both are the decisions the level exists to prevent.
Write that in the debrief in those words: could not name a task where the tool earns its place, asked twice.
Before calling it a problem, check whether the claim is simply true. In a randomized trial run between February and June 2025, 16 experienced open-source developers completed 246 real tasks on repositories they had worked on for about five years, and allowing early-2025 AI tools made them 19% slower 5.
Sixteen is a small enough sample that nothing general follows, and the result does not transfer to juniors, unfamiliar code or greenfield work.
Still, a senior saying the tool costs them time on a codebase they know intimately is making a claim of exactly the kind that has been measured, and measured in their favor once.
A candidate who abstains from the tools is answering a different question from one who argues against them in the room. Whether a candidate who doesn't use AI at all is a dealbreaker turns on access and ramp. The criticism turns on something else: whether the position survives a condition.
Anyone who has read one article about jagged frontiers can produce a task, a failure mode and a falsifying condition while holding none of it. Both questions are coachable, which is the same problem as telling a real catch from an easy one.
The follow-ups test articulacy, and articulacy is not the habit.
Put the two answers in the debrief line
One line in the candidate's own terms, entered before anybody reaches for the word culture. What they said the tool is bad at, which task, what they do instead, the exception they named, and the condition they gave. Five short entries the next reader can disagree with. A rating without them is a mood with a number attached.
A note that names what the job requires can carry signal.
Shanshi Liu and colleagues text-mined post-interview notes on 7,650 candidates hired at a large Chinese internet technology company. Notes naming more of the job's required capabilities went with better later performance, more promotions and less turnover 6.
The size of that link is roughly 2 percent of a performance rating per standard deviation of match between the note and the job analysis.
Only people who passed the interview could be observed, so nothing there tests what a note is worth on a rejection, and the effect is small and correlational at a single firm.
The direction is the usable part.
Rewritten from the two answers, the debrief line reads: named three tasks where the tool costs time on a codebase this old, named one where it saves time, and said a model that cites its own retrieval would change the view. Nobody on the panel has to agree with any of that to score it.
The 16 developers forecast a 24% speedup before they started and still believed afterwards they had gained 20%, against a measurement of 19% slower 5. Confidence about a tool is not evidence about a tool, in either direction. The finalist's criticism is a claim. So is the panel's discomfort with it.
Common questions
Is criticizing AI in an interview a red flag?
Only when the criticism names a category and nothing else. A candidate who says the tool fails on a specific task, produces a specific failure, and gets replaced by a specific alternative is describing a boundary, and boundaries are what seniority is supposed to supply. A candidate whose objection stays at the level of the technology and survives every counterexample is holding a position no evidence can reach. Ask where the tool helps in their own work, then ask what would change their view, before anything goes on the scorecard.
What two follow-up questions separate skepticism from refusal?
Where does the tool help in the work you actually do, and what would have to be true for you to change that view. The first asks for an exception to their own position, and a real one is small and specific, like generating test cases they then run. The second asks for a condition, and a real one is observable, like a retrieval step that cites the paragraph it used. Restating the criticism at greater length answers neither. Ask both of every finalist and score them on the same scale, so the round stays comparable.
Should a senior candidate be held to the same AI bar as a junior?
To a different one. In one large study of customer support agents, the gains from an AI assistant concentrated among novice and low-skilled agents, and the most experienced gained almost nothing. What the senior level is paid for is the boundary: which tasks go to the tool, what gets checked before anything ships, and what a team is not allowed to delegate. Test for that rather than for enthusiasm, and expect the strongest senior answers to contain more criticism than a junior's, not less.
How do you write this up without saying culture fit?
Write what was said. Which task the candidate named as a failure, the failure they described, what they do instead, the exception they gave, and the condition that would change their view. If they gave no exception and no condition after two attempts, write that sentence and nothing about attitude. A debrief line naming behavior can be argued with by the next reader. A line naming culture cannot, and it carries whatever the room felt that afternoon into a record somebody may read back to you later.
What if the candidate's criticism is simply correct?
Then the exercise still worked. A finalist who says the tool slows them down on a mature codebase is making a checkable claim, and one randomized trial of 16 experienced developers, on repositories they had worked on for years, measured that outcome with early-2025 tools. Correct criticism with an exception attached is the strongest version of the answer rather than an exemption from the question. What is being tested is whether the view has a shape: tasks on one side, tasks on the other, and something that would move the line.
References
- 1. Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality (Working Paper 24-013) ✓ mitsloan.mit.edu Supports the claim that the failure informed criticism names is a wrong answer outside the tool's range: on one task chosen to sit outside GPT-4's capability, 84.5% of the control group was correct against 60% and 70% in the two AI conditions, an average of 19 percentage points.
- 2. Structured Interviews: A Practical Guide ✓ opm.gov Supports asking both follow-ups of every finalist on a common rating scale with an acceptable answer agreed in advance, which are the three properties this guide uses to define a structured interview.
- 3. Mechanical Versus Clinical Data Combination in Selection and Admissions Decisions: A Meta-Analysis ✓ gwern.net Supports the claim that a debrief ending in a shared feeling is holistic combination, and the 25% reduction figure quoted as the authors' Taylor-Russell derivation at a selection ratio of .30 rather than a measured employer outcome.
- 4. Generative AI at Work (NBER Working Paper 31161) ✓ nber.org Supports the claim that measured AI gains concentrate in less experienced workers: 14% more issues resolved per hour on average across 5,179 support agents, 34% for novice and low-skilled agents, and minimal impact on experienced, highly skilled ones.
- 5. Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (arXiv:2507.09089) ✓ arxiv.org Supports both the claim that a senior's slowdown claim can be true and the closing point about belief against measurement: 16 developers on 246 tasks forecast a 24% speedup, estimated 20% afterwards, and were measured 19% slower.
- 6. Predictive Validity of Interviewer Post-interview Notes on Candidates' Job Outcomes: Evidence Using Text Data From a Leading Chinese IT Company ✓ frontiersin.org Supports the claim that a note naming more of the job's required capabilities went with better later performance, more promotions and less turnover, across 7,650 candidates who passed the interview and joined the company, at roughly 2 percent of a performance rating per standard deviation of match.
6 sources, numbered by first appearance. Every one was opened and checked against the claim it carries. How Olive sources claims
General guidance for hiring teams. What works at one company and one volume may not transfer to yours.
Olive assesses how a person works with AI. It does not detect AI-written documents, and it never produces a score, a ranking, or a match percentage for a person. Candidates read the same report the employer reads.