How Can You Tell If a Candidate Really Uses the AI Tools They List?

What does a six-tool list actually tell you?

That the candidate reads the same feeds as everyone else in their field. Adoption is close to universal in software: 84% of respondents to Stack Overflow's 2025 developer survey use or plan to use AI tools, and 51% of professional developers use them daily ¹. A list of six separates nobody. What a product name can't tell you is whether the person noticed when the tool was wrong.

Self-report doesn't close that gap either. In a randomized trial run by METR, 16 experienced open-source developers worked 246 real issues from their own repositories, and allowing AI tools made them 19% slower. They had forecast a 24% speedup beforehand, and after the fact they still believed the tools had sped them up by 20% ². Sincere people were wrong about their own AI use by roughly forty points. Asking "how good are you with Cursor" samples the same faulty instrument.

So ask about events instead of ability. Events are checkable, they have a date and a consequence, and a candidate who worked the way the resume implies has dozens to choose from. Outside software the list is also a stronger claim than it looks: in late 2024, 23% of employed US respondents had used generative AI for work in the previous week and 9% used it every workday ³. Six tools on a marketing or finance resume is a minority claim, which makes it worth ten minutes. The shorter version of this check is ten minutes with no assessment budget.

Ask one question per tool category, not per tool

Six names usually collapse into three or four categories, and the probe belongs to the category. Ask what it got wrong, how they found out, and what they changed. Run it once per category rather than six times, or the interview becomes a product quiz, and tool trivia is not the thing you're hiring for. Pick the two categories the job actually runs on.

The list is field-coded, so read it that way. Copilot and Cursor on an engineering resume are duty tools and their absence would be the odd thing; a research assistant on a consulting or marketing resume sits closer to the analysis itself, and the honest probe there is about sources rather than about diffs. In finance and revenue operations the tool usually touches a number somebody will act on, which changes the question again, and screening for AI judgment outside engineering runs on the same three-part probe with different artifacts behind it.

Why does the disqualifying answer change by tool?

Because each tool invites a different shortcut. A coding agent invites shipping a diff nobody read; a research assistant invites citing a source nobody opened; a notetaker invites forwarding a summary nobody corrected. The wrong-answer question stays the same, but what counts as a failing answer follows the tool's own failure mode. Decide before the call which answer ends the line of questioning.

• Coding assistants fail plausibly. The top frustration in Stack Overflow's 2025 survey was "AI solutions that are almost right, but not quite," reported by 66% of respondents, with 45.2% naming debugging AI-generated code as more time-consuming than writing it ¹. A candidate who uses one daily has been burned by a near-miss and can name it. "It saves me hours" with no near-miss attached is the thin answer, because near-misses are the dominant experience of the tool.
• Research assistants fail invisibly. The output reads finished and the citations look real. So the only useful probe is whether a source was opened: which claim did you check, what did the page actually say, what changed in the deliverable. A candidate who has never opened a cited source has been handing on someone else's confidence.
• Data copilots fail arithmetically. A formula that runs is not a formula that's right. Ask which figure was re-derived by hand and whether it matched. "I sanity-check the outputs" is a posture; "the growth rate was compounding when it should have been average, and I caught it re-adding the column" is an event.
• Notetakers and workflow agents fail silently. Nobody reads the summary against the meeting. Ask who caught the last error and what it cost.

One pattern runs under all four. The useful answer names something outside the conversation that settled the question: a page opened, a number recomputed, a colleague called. "It seemed off" is taste, and taste is not verification. That distinction is the whole content of testing whether a candidate catches AI errors, and it survives being asked of a senior and a new grad in the same words.

What if they can't name a time it was wrong?

Press once, then write down what you got. Ask for the most recent thing they threw away rather than the most interesting one, because recency is easier to retrieve and harder to invent. If the second attempt still produces nothing, the finding is that the list is thinner than it reads, which is what the question was for. None of this is evidence of dishonesty, and it shouldn't be recorded as if it were.

Three honest reasons a good candidate blanks. Their last employer banned the tools, or ran on air-gapped systems where nothing was allowed near a model. Their work is under NDA and they've been trained not to describe it. Or they narrate their own process badly under time pressure, which is a separate skill from doing the work and is not evenly distributed across candidates. Offer the same fallback to everyone: describe the shape of the correction without the client, the file or the number.

A candidate who says they barely use these tools is answering a different question, not failing this one. What to do when your strongest candidate doesn't use AI is worth settling before the interview rather than during it.

Ask every candidate the same probes, in the same order, and record the answers the same way. A step that decides who advances is a selection procedure, and consistency is what makes it explainable later ⁴. It's also the only way six answers become comparable, and running a structured interview about AI use is the version of this you can put in front of a panel.

Where does this probe stop?

At a story about work you didn't see. The probe gets a candidate describing a correction made weeks ago, reconstructed under time pressure and filtered through how well they talk about themselves. That's better evidence than a list of product names, and it's still description. Nobody in the room watches the candidate catch a confident wrong answer while it's happening.

The ceiling matters because the behavior you're buying is a live one. On a task the candidate has never seen, with a deadline and an assistant that answers everything fluently, what gets framed first, what gets demanded a source, what gets kept by hand, what gets refused. None of that is visible from an account of it. Interview answers are also the most coached surface in the process now, which is why the next step is usually a work sample rather than another conversation.

If you go that way, write the answer key before you write the task, and let the assistant be open rather than banned, because a banned tool just moves the work off-screen. Take-home assignments where candidates use AI still carry signal when the grading is about what was refused and checked rather than about polish. Olive is one instrument in that category and says so; multiple-choice AI literacy tests, code-collaboration graders, unwatched take-homes and in-house rounds are all real approaches with different trade-offs. See how Olive measures this.

See how it works >