Interviewing
Ask Who Wrote the Evaluation Set, Not Which Model It Called
An engineer's claim to have shipped an AI feature is checkable in one place: the parts of the system only a participant would know. The stack is not one of them, since architecture write-ups are public and anyone can recite one. Ask who wrote the evaluation set and how many cases it holds. Ask what the first bad output that reached a user looked like, what changed the week after, what got cut before launch, and what the feature costs to run.
The takeOwnership of the evaluation set is the best proxy available for ownership of the feature, and it is a strange thing to be the best proxy. Prompts move between people. Architecture gets inherited. But somebody sat down and decided which cases the thing had to pass before it went out, and that person remembers where the cases came from, since many came from something going wrong. That story is hard to borrow.
Where Olive fits
Open a role and see what the work shows
An interview can record a candidate describing the bad output they once caught; it cannot show them catching one. Olive puts that in front of a person as work: a role-grounded assignment, an AI assistant that will overreach if nobody stops it, and a human reviewer who writes or edits the findings on what happened, in a report the candidate receives too.
Assess your shortlistWhy is the stack answer free to anyone who read the write-up?
Because the architecture most AI features share is published. Retrieval over a vector store, a re-ranker, a system prompt, a small model routing to a larger one on hard cases: the pattern sits in public engineering write-ups and conference talks. A candidate who read a few of them can describe your own stack back to you inside two minutes. Reciting it is evidence of reading, and nothing else.
What is not published is the record of one system misbehaving in front of one user, and what the team did in the week afterwards. The record lives inside the people who were there, and it transfers badly to whoever only heard it at standup.
An engineer's own estimate of what AI tools did to their speed is weak evidence.
From February to June 2025, METR ran a randomized trial: 16 experienced open-source developers, 246 real tasks, repositories they had maintained for about five years 1.
Before starting, the developers forecast that AI tools would cut completion time by 24 percent. Afterwards they estimated a 20 percent saving. Allowing the tools made them 19 percent slower 1.
Sixteen developers working on code they already knew is one narrow setting, and the result does not transfer to juniors or to unfamiliar work. What travels is the gap between belief and measurement. People inside the work were wrong about it before and after doing it.
So the round cannot run on self-assessment.
A ten-minute screen can verify 'uses AI daily' on a resume with two or three concrete follow-ups. A shipped-system claim needs its own round, because the checkable facts sit further down: the evaluation set, the production failure, and the thing that got rolled back.
Ask who wrote the evaluation set, and how it grew
Start there, because an evaluation set is the one artifact on an AI feature that somebody had to build case by case. A prompt gets pasted from a teammate and a retrieval pipeline gets inherited from whoever left, but a set of test cases with expected answers is assembled by a person who decided what counted as wrong. The person who built it can name the case they added last and what it caught.
The follow-ups are dull on purpose.
How many cases does the set hold, where did they come from, and what is in the newest one? Sets that ran in anger grow out of complaints, support tickets and one demo that went badly in front of a customer. A set assembled entirely from the specification usually means the feature never met anybody.
Listen for the passive voice, then ask who. Things were tested, cases were added, coverage was improved. Some people who wrote the cases talk that way from habit or to credit a team, so the grammar only tells you where to press.
What did the first bad output that reached a user look like?
Asked plainly, the question should draw one specific incident. Anyone who kept a model-backed feature alive for six months watched it do something wrong in front of a real person. They remember the moment in detail: the query, the wrong answer, who noticed, how long the noticing took. A candidate who cannot produce one may not have run the thing, or may not have been near it while it ran.
The interesting failures are the ones nobody planned for.
Harvard Business School ran a field experiment in 2023 with 758 Boston Consulting Group consultants, and included one task deliberately chosen to sit outside what the model could do 2.
The two groups working with GPT-4 scored 60 and 70 percent on that task. The control group got it right 84.5 percent of the time 2.
Nothing about a single task, a single sample and a 2023 model that is no longer current settles the size of that gap. What the result names is the failure mode: the people using the tool could not tell which side of its capability the task fell on.
The first bad output on a real feature can have exactly that shape. Nothing about it read as out of range until somebody checked.
So the follow-up earns more than the question. What shipped differently in the week afterwards?
A team that learned something added a case to the evaluation set, tightened a retrieval filter, or pulled the feature off a surface where being wrong cost too much. A team that learned nothing wrote a postmortem and changed nothing.
In a study published in 2026, engineering students sat open-book take-home exams with ChatGPT allowed, and handed in their interaction transcripts alongside their solutions 3.
Some of the strongest evidence of reasoning showed up where a student judged an incorrect or incomplete model answer. A correct final answer, the author suggests, may no longer be enough evidence of comprehension on its own 3.
The study is one course, qualitative, with no control group, and a take-home exam is not an outage.
Even so, some of the best evidence sat in what a person did with a wrong answer.
Specify, invert, falsify. The three rungs behind follow-up questions expose whether someone understands the answer they gave transfer to this round unchanged.
Ask which log line told them. Ask what the numbers would have had to read for the rollback to have been a mistake.
Ask what it cost to run, and what got cut before launch
The cost question and the cut question look for the same thing: whether the candidate held a constraint. A monthly bill, a latency budget and a rate limit are facts an owner tends to carry around and a bystander seldom learns. So is the feature that was scoped, built halfway and then pulled, along with who made that call and the reason they gave for it.
Cost answers are specific or they are secondhand.
An owner knows roughly what a thousand requests cost, which model they moved off and why, and whether the caching went in before or after the finance conversation. A bystander gives an order of magnitude and a shrug.
The cut is the better half of the pair.
Every AI feature that reached production has a graveyard beside it. The agent that was allowed to write to the database until somebody stopped it. The summary that came off the clinical note. The auto-send that became a draft for a person to approve.
Ask what is in theirs, and ask who decided.
A good answer here has the shape of an answer about what a candidate decided not to use AI for. A boundary, the damage crossing it would do, and who would absorb that damage.
Watch for the version where the cut was somebody else's call and the candidate is reporting it secondhand. Ownership shows up in the argument, not in the outcome.
Write down the acceptable answer to all six before the round starts
Decide in advance what a good answer to each of the six sounds like, or the round measures fluency instead. The US Office of Personnel Management defines a structured interview by three properties: the same questions in the same order for every candidate, a common rating scale, and interviewers who agree beforehand on what an acceptable answer looks like 5. The third property is the easiest one to skip.
The guide is 2008 federal hiring practice, not law, and it binds no private employer 5.
The six, in the order they run best:
1. Who wrote the evaluation set, and how many cases does it hold? 2. Where did the cases come from? 3. What did the first bad output that reached a user look like? 4. What shipped differently in the week afterwards? 5. What got cut before launch, and who decided? 6. What does a month of running it cost?
Structure is what makes those answers comparable.
Structured interviews sit at .42 and unstructured ones at .19 on the corrected scale Sackett, Zhang, Berry and Lievens produced when they re-analysed the personnel selection literature in 2022 4.
Neither number came from an interview about AI work, and none of the underlying studies involved one 4. The mechanism is the part worth borrowing: in those studies, interviews predicted rated job performance better when every candidate met the same questions and the ratings ran on one scale.
Published studies describe less structure than the label promises.
A content analysis of 104 interviews reported in 103 articles, covering studies conducted from 1997 to 2010, found an average of 5.74 of the 15 structure components in Campion, Palmer and Campion's framework 6.
The review reports roughly equal use of the content and evaluation halves. Detailed notes appeared in 19 percent of the 104, and no discussion between interviews in 4 percent, the rarest of the 15 6. Those counts describe what researchers wrote into published studies, so read them as a ceiling rather than a floor.
Writing six questions down takes ten minutes.
Writing down what a good answer to each one sounds like, before anybody has given one, is where the work is.
Running a structured interview about AI use so candidates are actually comparable is a separate build.
The resume line was never the evidence.
The line reports something true about the feature and nothing about the person, and no rewrite of the bullet will change that.
One layer down sits a file of test cases somebody maintained, and one bad afternoon somebody still remembers.
Start the round there.
Common questions
What if the feature never had an evaluation set?
Take the answer at face value and follow it. Plenty of shipped AI features run on spot checks and a shared document of prompts that broke. Ask what stood in for a test set: who read the outputs, how often, against what, and what triggered a change. A candidate who says there was no formal set and then describes a weekly review of flagged conversations is telling you about ownership. A candidate who says testing was handled by another team, and cannot say what that team checked, is telling you about proximity. The absence of a set is a fact about the company. The inability to describe what replaced it is a fact about the person.
How is this different from asking about the AI tools on a resume?
A tool list is a habit claim and a shipped feature is a system claim, and they fail different checks. Whether somebody really uses the six tools they listed can be settled in a short screen with two or three specific follow-ups about last week's work. A shipped-system claim needs the artifacts underneath it: the evaluation set, the production failure, the rollback, the bill. Those take twenty to thirty minutes and an interviewer who can follow a technical answer. Running both is fine. Running only the screen version on a staff engineer wastes the round.
Does this work when the candidate is bound by a confidentiality agreement?
Often, because most of the six ask about shape rather than content, though the agreement's terms decide what is covered. Within them, a candidate under an NDA might say the evaluation set held a few hundred cases, most from support tickets. They might say the first bad output was a confident wrong figure in a customer-facing summary, and that the fix was a retrieval filter plus a human approval step. None of that names the client, the data or the revenue. If they will not give even the shape, ask them to describe a comparable system they can talk about. A refusal to characterise anything at all is worth noting, and it is not proof of an empty claim.
What does a strong answer about the first bad output sound like?
Specific and unflattering. It names the input, the wrong output, who caught it and how long the catch took, and it usually admits the failure looks obvious in hindsight. Strong answers carry the kind of detail that is hard to invent: a stale document that sat in the index for a month, a currency symbol, a customer who sent a screenshot. Weak answers stay at the level of the category, describing hallucinations in general, and get vaguer under a follow-up rather than more concrete. Ask which log line or message told them. The log-line question separates a memory from a summary of somebody else's memory.
Do the same six questions work for a product manager or a designer?
Four of the six carry over unchanged. Anybody who owned a model-backed surface knows what the first bad output was, what changed after it, what got cut, and roughly what it costs to run. The evaluation-set questions shift. A product manager or designer may not have written the cases but should know what the launch bar was, who set it, and what happened the first time the feature missed it. Keep the questions identical inside a role and vary them across roles, so the answers stay comparable where the comparison is actually being made.
References
- 1. Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (arXiv:2507.09089) ✓ arxiv.org Supports the claim that an engineer's own estimate of what AI tools did to their speed is weak evidence: 16 experienced open-source developers, 246 real tasks on repositories they had worked on for about five years, forecast a 24% speedup, estimated 20% afterwards, and were measured 19% slower.
- 2. Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality (Working Paper 24-013) ✓ mitsloan.mit.edu Supports the claim that the dangerous failure looks like a task the model handles well: in a field experiment with 758 Boston Consulting Group consultants, on one task chosen to sit outside model capability, the control group was correct 84.5% of the time against 60% and 70% in the two AI conditions.
- 3. Reimagining Assessment in the Age of Generative AI: Lessons from Open-Book Exams with ChatGPT ✓ arxiv.org Supports the claim that some of the best evidence of AI work sat in what a person did with a wrong answer: students submitted interaction transcripts with their solutions, some of the strongest evidence of reasoning appeared where they evaluated incorrect or incomplete AI responses, and the findings suggest that in AI-mediated assessment environments correctness of final answers alone may no longer provide sufficient evidence of comprehension.
- 4. Revisiting Meta-Analytic Estimates of Validity in Personnel Selection: Addressing Systematic Overcorrection for Restriction of Range ✓ gwern.net Supports the claim that structure is what makes interview answers comparable: mean validity estimates of .42 for structured interviews against .19 for unstructured ones, on the paper's own corrected scale. None of the underlying studies involved AI-conducted interviews or interviews about AI use.
- 5. Structured Interviews: A Practical Guide ✓ opm.gov Supports the three-property definition of a structured interview used here: the same questions in the same order for every candidate, a common rating scale, and agreement in advance on what an acceptable answer looks like. It is 2008 federal HR practice guidance rather than statute.
- 6. The Structured Employment Interview: Narrative and Quantitative Review of the Research Literature ✓ doi.org Supports the counts for how much interview structure published research actually used: 104 interviews from 103 articles, covering studies conducted 1997-2010, used an average of 5.74 of 15 structure components, in roughly equal numbers on the content and evaluation sides, with detailed notes present in 19% and no discussion between interviews in 4%, the rarest of the 15.
6 sources, numbered by first appearance. Every one was opened and checked against the claim it carries. How Olive sources claims
General guidance for hiring teams. What works at one company and one volume may not transfer to yours.
Olive assesses how a person works with AI. It does not detect AI-written documents, and it never produces a score, a ranking, or a match percentage for a person. Candidates read the same report the employer reads.