What Should You Look for in a Candidate's AI Chat Log?

What does a chat log show that the deliverable can't?

Acts, in order. The deliverable is a result; the log is the sequence of decisions that produced it. Four of those decisions carry almost all the signal: where the first move aimed, which claim got a source demanded for it, what the assistant produced that got thrown out, and what got checked against something outside the chat. The rest is texture.

Volume is the trap, because it is the one thing a log makes easy to count. Forty turns can be one person asking the same question forty ways and pasting the last answer into the document. Six turns can be a plan, a sourced correction, and half an hour of work done outside the window. Counting prompts ranks the first candidate higher, and so does reading for eloquence, since fluent phrasing is what the assistant supplies for free.

The research on knowledge work points at the same handful of moments. A survey of 319 knowledge workers who supplied 936 first-hand examples of using generative AI at work reported that it shifts the nature of critical thinking toward information verification, response integration and task stewardship, and that higher confidence in the tool went with less critical thinking while higher confidence in themselves went with more ¹. Those are acts a transcript records and a finished document hides.

Both working shapes turn up in candidate logs. Across the Claude.ai conversations sampled for Anthropic's Economic Index, 57% of tasks were augmented and 43% automated ². One candidate's log usually contains both on different subtasks, and the useful question is never which one they reached for. It is whether the split was decided or defaulted into. That is also the reason to ask for the log at all: a take-home that comes back polished carries none of this on its face.

Read the first three turns: framing or fetching?

Start there, because nothing has been prompted into the candidate yet. Look at whether the opening message states the problem, a constraint or a criterion in the candidate's own words, or whether it pastes the brief and asks for the finished thing. One of those is a person deciding what the job is. The other is an order being placed.

The same diligence take-home, opened two ways:

• Fetching. "Here's the data-room summary and the brief. Write a two-page memo recommending whether to proceed."
• Framing. "I'm deciding whether to recommend proceeding. Before drafting anything, list the claims in this packet that would flip the recommendation if they turned out to be wrong, and tell me which of them the packet doesn't settle."

Neither message is better written, and the second one is not more polite or more detailed in any way that matters. It names what would make an answer wrong before an answer exists. Mark the turn and move on.

What counts as framing, in a line each:

• A criterion arrives before output is requested. A cost ceiling, the audience, what the decision turns on, what is out of scope.
• The candidate restates the task. Not the brief pasted back, their own shorter version of it, sometimes wrong in a way that tells you something.
• The unknowns get named. "I don't know the renewal rate and it matters" is worth more than a confident opening.

Two caveats worth holding. Framing can arrive at turn nine, after a first draft came back thin, and a candidate who course-corrects has demonstrated the behavior rather than failed it. And an opening that reads like a rubric entry is not automatically the stronger one: if three criteria get named and then ignored for the rest of the log, you have read a performance, and the later turns will show it.

Which claim did they demand a source for?

One claim, not all of them. Asking for sources in general produces a list nobody opens. What carries signal is a specific load-bearing claim being stopped on, a source demanded for that claim, the source actually opened, and what it did or did not support turning up in the deliverable afterwards. Three turns, usually, and they are easy to find.

The tell is the round trip. The assistant asserts something the recommendation rests on. The candidate names that specific thing and asks where it came from. A later turn reports what the source actually said, often correcting the assistant, and something in the deliverable moves as a result. "Cite your sources," answered with a list of links and no follow-up, is the appearance of sourcing with no evidence that anyone opened anything.

The rarer act is the one worth finding. In the same knowledge-worker survey, 114 of the 319 participants described cross-referencing AI output against external sources, while 23 described checking the sources the output itself had cited ¹. Chasing a citation the assistant produced is the less common habit, and a transcript is one of the few places it becomes visible.

Which claim is load-bearing depends on the work, so read for the one the deliverable would collapse without:

• Financial analysis. The growth rate the recommendation turns on, traced to the filing itself rather than to the assistant's summary of the filing.
• Healthcare revenue cycle. The payer policy quoted back with its effective date, because a correct rule from last year is a denial.
• Legal operations. The authority opened and read down to the paragraph, since retrieval is not verification and a real case can be pinned to a proposition it never held.
• Data and analytics. The column definition checked in the schema, because a fluent explanation of the wrong field reads exactly like the right one.
• Supply chain. The freight terms behind a landed-cost figure, since three quotes written on three different terms do not compare.
• Marketing. The market-size number followed past the vendor's blog to the survey underneath it, with the sample size named in the brief.

One asked-and-opened claim beats a bibliography. Write down which claim it was, so a second reader has something specific to disagree with.

What got refused, and what got tested outside the chat?

The two hardest things to fake. A refusal is a turn where the candidate rejects something the assistant produced and says why: an assumption named, a framing dropped, a section cut for doing no work. An outside test is a turn reporting a result the assistant could not have produced, which then changes something in the deliverable. Both leave a mark in the record.

Count refusals as a rate against the answers actually taken up, never as a total. Otherwise thirty prompts with three pushbacks outrank six prompts with three, and you are back to rewarding volume through a side door. Style edits do not qualify. "Make that friendlier" is a preference. "Drop the second recommendation, the packet doesn't support it" is a judgment, and it earns the column. See how Olive measures this.

The outside test has two halves and most logs contain only the first. Someone writes "I checked the numbers and they hold," which is a claim about an act. The evidence is a result the conversation could not have produced, followed by a change: a figure recomputed and different, a page that contradicted the draft, a script that failed. If nothing downstream moves, the check either did not happen or did not matter.

This is where self-report stops being usable, the candidate's and yours alike. In METR's randomized trial, 16 experienced open-source developers working on repositories they had spent an average of five years on completed 246 tasks; afterwards they estimated the AI had cut their completion time by 20%, while the measured effect was a 19% increase ³. People read their own AI-assisted work in the flattering direction. Weigh the turn that shows a result above the turn that reports one, in the log and in the conversation after it.

Absence still proves nothing. A candidate who rewrote three paragraphs by hand in the document made a refusal that never touched the chat. If the artifact carries a version history, ask for it alongside the log and read the two together.

How do you read twenty logs the same way?

Fix the four reads as columns before you open the first log, and fill each with what the log says rather than with an impression. Mark each one demonstrated, partly demonstrated or not demonstrated, and put the turn number beside it, so every mark points at something you could show the candidate. Two readers filling the columns separately should land in the same place.

The wording of the column decides whether it can be filled twice the same way. "Used AI well" collects opinions and nothing else. Write the column as the act instead: named a specific claim and reported what the source said about it. That version is answerable with a yes, a no and a turn number, which is also what makes it reviewable later. Getting two reviewers to score an AI-use rubric the same way is a calibration problem before it is a wording problem, so mark two logs together before anyone marks alone.

Keeping the columns on observable turns has an old and specific reason behind it. The US federal Uniform Guidelines on Employee Selection Procedures, in force since 1978, hold that a selection procedure is supported by content validity "to the extent that it is a representative sample of the content of the job," and that a procedure "based upon inferences about mental processes cannot be supported solely or primarily on the basis of content validity" ⁴. "Seemed to think carefully" is an inference about a mental process. "Opened the filing and changed the number at turn 7" is a work behavior with a work product attached. Check with counsel on your own program.

Ask for the log in one format, in the brief, before anyone starts: the whole exchange in order with nothing removed, plus a short note of anything done outside the chat and what it changed. Say that the note and the log are graded and the polish is not. Then read every candidate's log against the same four columns, including the one whose prose reads normal to you. Whether the tool is permitted at all sits upstream of every one of these columns, so settle that first: the call on allowing AI on the take-home belongs to the job, not to the assessment.

What a pasted log can't tell you

Whether it is complete. A pasted transcript is a document the candidate assembled, so a missing turn is not evidence of a missing act, and a present turn can be written after the fact. Read it as the strongest claim a candidate is willing to make about their own process, then check the load-bearing parts in twenty minutes of conversation about their own submission.

It is also not a detection instrument, and reading it as one wastes the exercise and the candidate's trust in the same move. The question a log answers is what someone did with the assistant; the question of who typed which sentence is a different one, and whether AI detectors work in hiring is settled enough that no decision should rest there. A transcript read for authorship is a worse instrument than the one you already had.

Three more limits, stated out loud:

• Missing turns are usually mundane. A second window, a phone, a colleague, a search that never entered the chat. Ask for the note; do not infer.
• A clean log can be a curated log. Whoever pastes everything, dead ends included, looks messier than whoever pasted the good half. Say in the brief that dead ends are graded as work, or you have built a filter for tidiness.
• One afternoon is one observation. The log shows what happened on this task with this assistant, not what happens when the model is confidently wrong about something the candidate has no way to check.

Tell candidates what the four reads are before they start. Someone who then aims at them has to frame the problem, demand the source, refuse something on substance and test it outside the chat, which is the behavior you were trying to measure in the first place. A read you would not be willing to describe to the person being read is the wrong read.

See what gets scored >