Assessment design
An Agent Run Records the Model's Decisions, Not the Candidate's
An agent run that worked alone for twenty minutes records what the model did, and only a little of what the candidate decided. The evidence about the person sits at the boundaries: the brief and the limits set before it started, every point where they stopped or redirected it, what they checked after it finished and against what, and what they threw away. Read those five places. The middle is the model's transcript.
The takeMost agent exercises hand back a record that was never designed to hold evidence, and the reflex is to read it harder. Reading it harder does not work, because a trace is long, fluent and mostly not the candidate's writing. The exercises worth running build the checkpoints in: a gate the agent cannot pass without a written approval, an input that is wrong in a way only checking catches, and a required note on what got thrown away. Design the boundaries, or spend an hour reading a transcript the candidate did not write.
Where Olive fits
Open a role and see what the work shows
If you build this exercise in house, the expensive parts are the authored case and the evidence trail behind each judgment. Olive ships twelve authored cases per occupation and returns six separately-evidenced findings, each anchored to a timestamped moment in the session, with the candidate granted the same report.
Assess your shortlistWhat does twenty minutes of autonomous work actually record?
Mostly the model's own steps. A twenty-minute run can leave a few hundred entries: prompts it wrote to itself, tool calls, file edits, retries, test runs, timestamps. Most of the small decisions in the record were made by the model while the candidate watched, or while they did something else in another window. The person shows up only a handful of times.
Those few entries are not spread evenly. They sit at the front, where the task was specified and the agent's permissions were set, and at the back, where somebody decided the output was good enough to hand over.
A chat log is a different object. In a conversation the candidate wrote every other turn. The record is a sequence of their choices with the model's replies in between, which is why the advice on what to look for in a candidate's AI chat log reads turn by turn. An agent trace has no such alternation.
Length works against you as well. METR timed people with domain expertise on software tasks, then asked how long a task has to be before a frontier model completes it only half the time. For models available in March 2025, the answer was around 50 minutes of expert human work 1.
The measurement covers software tasks, and the authors name external validity as a limitation. Take the direction rather than the number.
The longer the task behind a run, the less often the model completes it on its own.
Read the brief, the interruptions and the ten minutes after
Five places in the record hold the candidate. The brief they wrote before the run started, the permissions they gave and withheld, each point where they stopped or redirected the agent, what they checked once it finished, and what they discarded. Print the trace, mark those five, and ignore the rest on the first pass.
- The brief. Look for a done condition and at least one prohibition: a file not to touch, an interface not to change, a claim that has to carry a source. A brief with nothing forbidden in it is a wish, and the run will show the agent choosing the boundaries itself.
- The permissions. What the agent could reach, and what was kept out of reach on purpose. Handing it the whole repository is one decision; handing it a branch, a fixture and no credentials is a different one.
- The interruptions. Each stop is a timestamped judgment. Read the three or four entries directly above it and ask what the candidate saw that made them reach for the keyboard.
- The finish. What was checked, against what, and whether the check came from outside the run. A test the agent wrote and then passed is the agent grading its own work.
- The discard. What came back and did not survive, on what stated ground. When it is there, it is the most informative entry in the record.
The finish deserves most of the review, because it is the only boundary that leaves an artifact you can check yourself. A candidate who reconciled the output against the source data left something behind: a second file, a query, a note with a number in it that either matches or does not.
An empty finish is not damning on its own. Some people check in their heads. Ask them to redo the check in front of you, on the same output, and watch what they check it against.
The record is also data about a person, which brings the usual limits on what you may collect and keep from a candidate's AI session.
Ask for the trace, not for access to the machine that produced it.
Why does the middle of an agent trace read so well?
Because the model narrates. A trace carries status lines the model wrote about its own progress, announcing what it is about to check and then pronouncing the result sound, and those lines read as evidence of care. They are output.
The plan the agent wrote at minute two is output as well, and a plan that survives twenty minutes of contact with a real repository is rarer than the trace makes it look.
Fluency travels furthest exactly where it should not. In Harvard Business School's 2023 field experiment with 758 Boston Consulting Group consultants, one task was chosen to sit outside what the model could do. Consultants in the control group were correct about 84.5% of the time, against 60% and 70% in the two groups given GPT-4 2.
The people using the tool could not tell which side of the line the task was on. One task, one sample, a 2023 model. The group given a prompt-engineering overview did worse than the group given none, on this task.
The candidate's own account of the run does not correct the misreading either. METR ran a randomized trial in which 16 experienced open-source developers completed 246 real tasks on repositories they had worked in for about five years.
Allowing early-2025 AI tools made them 19% slower. They had forecast a 24% speedup, and after finishing they still believed they had gained 20% 3.
Magnitude stays inside the study that produced it, sixteen developers in one setting. The gap between what people believed about their own work and what the clock said is the part that travels.
So ask what they checked, not how it went.
Fill five columns from the trace, not from the impression it leaves
Five boundaries, five columns, and one line quoted out of the record in each. Not a summary of the boundary: the line itself, with its timestamp. Two reviewers filling those columns off the same twenty-minute run should land in the same place. If they do not, a column is asking for an impression rather than for something the record either holds or lacks.
A column asking whether the output was checked against a source outside the run takes a yes or a no. A column asking for strong judgment takes neither.
The trace pushes hard toward the second kind.
It is long and it is fluent, and a few hundred entries later what a reviewer is combining is a feeling about the run rather than five facts out of it.
Formulas beat that feeling, and by a measured amount. Across selection and admissions studies, Kuncel, Klieger, Connelly and Ones found that a formula combining applicant data predicted job performance at an average correlation of .44, against .28 for expert judgment combining the same kinds of data 4.
Nine studies sit behind that comparison, and the authors are clear that the formula can be as plain as adding up a written checklist. Nothing in it argues for a number standing for a person, and a five-column grid is not one. What it argues for is that the last run you read this week gets the same five questions as the first.
What goes in a column matters too. Researchers text-mined the post-interview notes on 7,650 people hired at a large Chinese internet company. Notes naming more of the capabilities the job analysis called for went with slightly better later performance, roughly 2% per standard deviation of that match 5.
Only hired candidates could be observed there, the design is correlational, and the effect is small.
A usable entry names a time and an act. The run stopped at 11:04, after the agent edited a migration file the brief had put out of scope. An unusable one says the candidate showed good instincts around scope.
Both come out of the same twenty minutes. Only one of them can be checked by whoever reads the grid next.
Make the next run stop itself at least once
Design the exercise so the five boundaries have to exist, instead of hoping they turn up. Give the agent a task with a gate in the middle that it cannot pass without a written approval from the candidate. Give it source material with something wrong in it that only a check catches. Ask for the brief and the discards as deliverables, in writing, with the run.
The seeded input is the part most exercises skip. What to plant in the starter repo has known answers: a stale constant, a fixture that agrees with the bug, a source that contradicts the summary sitting on top of it. Each one tests checking rather than typing.
The gate is what manufactures the third checkpoint. A run with no stopping point produces no interruptions, and their absence then tells you nothing about whether the candidate would have stopped it.
Interrupting is not a virtue, and a reviewer who counts stops will reward the wrong person. Plenty of work pays for letting an agent run to the end, and a candidate who stops it every ninety seconds is doing the job by hand with extra ceremony. What separates the two is what each stop was for.
If the job is mostly supervision rather than occasional use, the exercise changes shape again: assessing a role that mostly supervises AI agents starts from a finished run with defects planted in it rather than from a blank task.
The twenty-minute record already on your desk is still worth one pass. Read the first message, the stops, and the ten minutes after the agent finished, then write down what each of the three holds. The next brief you send can make sure they are not empty.
Common questions
What is actually in an agent run record?
A timestamped list of what the agent did: the prompts it wrote to itself, tool calls, file reads and edits, commands, test runs, retries, and often a plan it drafted at the start. A twenty-minute run can hold several hundred entries. The candidate's own contributions are a small fraction of that, concentrated in the opening instruction, any point where they intervened, and whatever they did once the run stopped. Some tools also log the permissions the run was given, which is worth having, because what a candidate refused to hand over is a decision the trace will not otherwise show.
Is a run with no interruptions a bad sign?
Not by itself. A tightly specified brief with a hard stop built into it can run clean, and stopping an agent constantly is not evidence of judgment. What settles it is what happened at the end. A candidate who let it run and then checked the output against something outside the run made a defensible call; a candidate who let it run and handed the result over unread made a different one. Read the last ten minutes before drawing anything from the middle of the run.
How do you tell whether a candidate understood the code the agent wrote?
Not from the trace. Ask for a change in front of you: pick one function the agent produced, name a requirement it does not meet, and ask them to fix it while talking through what else that touches. Ten minutes is enough. Someone who followed the run answers in specifics about the file they are in, and someone who did not starts by rereading the whole thing. The same question works on prose, a query or a spreadsheet, with the artifact swapped.
Should candidates submit an agent trace from their real work?
Usually no. A trace from a previous employer carries that employer's code, customer data and internal decisions, and asking for it puts a candidate in the position of leaking something to get a job. Run the exercise on material you own instead, or ask them to describe one run in conversation: what the brief said, where they stopped it, what they checked. If a candidate offers a trace unprompted, check that it has been stripped before it is stored anywhere.
How long should reviewing one agent run take?
About twenty minutes, if the checkpoints were designed in. Read the opening brief, jump to each interruption and the entries just above it, then read everything after the agent stopped. Fill five columns as you go, with one quoted line each. Reading a few hundred tool calls end to end costs an hour and adds almost nothing, because the middle of the run is the model's work. If the review keeps running long, the exercise is producing an artifact it was never designed to produce.
References
- 1. Measuring AI Ability to Complete Long Software Tasks (arXiv:2503.14499) ✓ arxiv.org Supports the claim that a model's success rate falls as the unattended task lengthens. The abstract states: "This is the time humans typically take to complete tasks that AI models can complete with 50% success rate. We first timed humans with relevant domain expertise on a combination of RE-Bench, HCAST, and 66 novel shorter tasks. On these tasks, current frontier AI models such as Claude 3.7 Sonnet have a 50% time horizon of around 50 minutes." The same abstract names external validity among the limitations. Fetched 2026-09-01.
- 2. Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality (Working Paper 24-013) ✓ mitsloan.mit.edu Supports the 758 consultants, the task chosen to sit outside model capability, the 84.5% control accuracy against 60% and 70% in the two AI conditions, and the finding that the group given a prompt-engineering overview did worse than the group given none.
- 3. Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (arXiv:2507.09089) ✓ arxiv.org Supports the claim that a candidate's own account of a run is not a measurement: 16 developers, 246 tasks, repositories known for about five years, 19% slower with AI tools after forecasting a 24% speedup and still estimating a 20% speedup afterwards.
- 4. Mechanical Versus Clinical Data Combination in Selection and Admissions Decisions: A Meta-Analysis ✓ gwern.net Supports the .44 mechanical against .28 clinical average correlation with job performance, the nine studies behind that comparison, and the authors' point that a mechanical method can be as plain as a unit-weighted written checklist.
- 5. Predictive Validity of Interviewer Post-interview Notes on Candidates' Job Outcomes: Evidence Using Text Data From a Leading Chinese IT Company ✓ frontiersin.org Supports the 7,650 hired candidates, the finding that notes naming more of the job analysis's capabilities went with better later performance, and the roughly 2% performance rise per standard deviation of that matching score, with the hired-only sample noted.
5 sources, numbered by first appearance. Every one was opened and checked against the claim it carries. How Olive sources claims
General guidance for hiring teams. What works at one company and one volume may not transfer to yours.
Olive assesses how a person works with AI. It does not detect AI-written documents, and it never produces a score, a ranking, or a match percentage for a person. Candidates read the same report the employer reads.