How Do You Tell Which Finalist Did the Thinking?

Why the Output Stopped Separating Them

Because the deliverable is now the part a model produces, and both finalists used a model. What used to differ (structure, coverage, polish, tone) converges when two people prompt the same system on the same brief. What still differs is upstream: what each of them decided to ask for, what they refused to accept, and what they went and checked against the world.

The instinct is to ask each finalist how much of it was theirs. That returns an opinion, and the opinion is unreliable in a way that has been measured. METR randomized 246 real issues across 16 experienced open-source developers' own repositories, allowing or forbidding AI tools issue by issue. With the tools allowed the developers took 19% longer, and after finishing they still believed the tools had sped them up by 20% ¹. Self-report about AI-assisted work describes an experience rather than an outcome.

The defect you are hunting is not visible in the artifact either, because it does not look like a defect. In Stack Overflow's 2025 developer survey the most common frustration with AI tools, at 66%, was output that is almost right but not quite; 45% said debugging AI-generated code is more time-consuming ². Almost-right compiles, reads well, and renders. It fails on the one number, the one edge case, or the one claim the recommendation rests on, and it fails after you have hired.

So two finalists can hand in work that is almost-right in different places, or one of them can hand in work whose thin spots were found and closed before submission. Nothing on the page tells you which. The same problem in a different format is worked through in How to Tell Who Did the Work in an AI-Built Portfolio.

Ask for Three Process Traces, in Writing

Three, sent to both finalists as the identical message, answered in under 300 words each, within a day. The framing: what were you deciding, and what would have made an answer wrong? The evidence: which single claim did the work rest on, and what did you open to check it? The outside check: what did you run, recompute or read that the assistant could not have supplied?

• The framing. A candidate who framed the problem first can name the decision and the failure condition in two sentences, because they wrote them down before generating anything. A candidate who asked the model for the deliverable will describe the deliverable back to you. The tell is a constraint or a criterion named specifically enough to be wrong: "the recommendation flips if the churn figure includes trials, and the definition note says it does not."
• The claim they demanded evidence for. Not the reference list: the one claim the whole thing rests on, and what happened when they went to check it. A generated draft arrives already carrying plausible citations, so "I cited five sources" separates nobody. "I opened the one that mattered and it did not say what the draft claimed" separates immediately. Ask which source, what it actually said, and whether it changed anything.
• The outside check. This is the trace hardest to invent, because it has a result, and the result could have come back wrong. A survey of 319 knowledge workers describing 936 real tasks found the effort does not disappear when a tool is used; it moves toward verifying information, integrating the response and stewarding the task. Higher confidence in the tool tracked with less critical thinking, higher confidence in one's own ability with more ³.

Rate the three separately and never blend them into an impression. A strong framing answer should not carry a missing check across the line. The anchoring problem is the same one covered in How to Score an Interview Answer Produced With AI.

What Counts as a Real Outside Check in Your Field

It varies, and that is the point: a check that means something for an engineer means nothing for a marketer. The common shape is an act with a result the assistant did not supply, that could have come back wrong. A test that fails. A source that says something else. A number that does not reconcile. Ask for the result, including the case where the check found nothing.

• Software. A test that could fail, run against the actual repository rather than the snippet. Ask which case they wrote a test for that the assistant had not covered, and what happened the first time it ran. Happy-path tests written alongside the code only prove the code agrees with itself.
• Financial analysis or accounting. One number recomputed from the source document rather than from the summary. Ask for the figure, the page it came from, and whether the recomputation matched. A model reproduces a number fluently and never opens the workbook.
• Consulting or market sizing. A second method that should land in the same order of magnitude. Ask for the back-of-envelope run beside the main estimate, and what they did when the two came out a factor of three apart.
• Anything citing authority. The cited source opened and read. Purpose-built legal research systems still return incorrect information. Stanford researchers measured Lexis+ AI and Ask Practical Law AI at more than 17% of benchmark queries and Westlaw's AI-Assisted Research at more than 34% ⁴. A citation that exists is not a citation that says what the draft claims.
• Marketing or research. The most on-message statistic in the brief, traced to its source, sample and year. It is the one least likely to have been opened, precisely because it fits.
• Product or design. One assumption tested against something that answers back (a real ticket queue, a real analytics number, a real user) rather than against a persona the assistant wrote.

The question underneath is identical everywhere and only the instrument moves: what did this person do that could have proved them wrong? How to Hire for Verification When AI Does the Production works through the same shift at the level of the role itself.

How to Run the Tiebreak Without Adding a Round

Send both finalists the identical message on the same day, cap the reply at 300 words, and write your answer key before either reply lands. Rate the three traces item by item, across both candidates, before you re-open either deliverable. That is fifteen minutes of your time and twenty of theirs, and it is one documented, consistent step rather than a fourth interview nobody has calendar space for.

Consistency is not housekeeping here. A step that decides who gets hired is a selection procedure, and federal guidance on employment tests is explicit that a procedure must be job-related, consistent with business necessity, and applied the same way to everyone it touches ⁵. Two finalists asked two different questions produce two answers you cannot compare and one decision you cannot defend.

Structure also buys accuracy. The most recent re-estimation of selection-method validity puts structured interviews at .42 and work samples at .33 against job performance, with job-specific measures outperforming general ones ⁶. A side-by-side read of two finished artifacts is neither structured nor job-specific; it is a preference, formed in the first two minutes and defended afterwards.

• Write the key first. Two or three lines per trace describing what a weak, an adequate and a strong answer contains for this specific brief. A key written after the first reply is a description of that reply.
• Rate across, not down. Both framing answers, then both evidence answers, then both checks. Reading one reply end to end is how a strong opening carries a missing check through the whole rubric.
• Follow up once, on a specific. One question each, about a detail inside their own answer. Specificity that survives a follow-up is not borrowable, and the mechanics are in Follow-Up Questions That Expose Real Understanding.

Tell both finalists the request is rated and roughly how long it should take. Undisclosed criteria produce noise: half of the replies get treated as paperwork, and you end up rating that guess instead of their judgment.

What a Written Trace Can't Tell You

Whether it happened. A written trace is an account of the work, produced after the fact, by a candidate who knows it is being rated, with the same assistant still open. It is better evidence than the deliverable (it can be checked against the deliverable, which can be checked against nothing), but it is a report of a process rather than a record of one.

Three things narrow the gap. Ask within a day, while the specifics are still recoverable rather than reconstructed. Cross-check every claim against the artifact: an assumption the deliverable contradicts, or a source check the numbers do not reflect, is a finding in itself. And ask that one follow-up, because a borrowed answer holds at the first level of detail and comes apart at the second.

If that still leaves you guessing between two people you are about to bet a year of payroll on, the honest conclusion is that the exercise has to happen somewhere the decisions are visible while they are being made. Take-Home or Live Working Session? covers that trade, and Should You Run a Paid Trial Instead of an Assessment? covers the version where you pay for a week of real work and watch it happen.

Speed will offer itself as the tiebreak. Whoever produced the deliverable faster with an assistant has told you about their throughput, not their judgment, and throughput is the part that got cheap. Measure How Fast They Work With AI, or How Well They Judge It? has the argument in full.

See what gets scored >