Screening
A Demo Has a Recording. Something That Ran Has a Failure History.
A resume line about building agents and automations tells you nothing until you ask two questions. First: how long did it run, roughly how many times, and between which dates. Second: what happened the first time it did something wrong, who noticed, and what changed after. A demo has a recording. An automation that ran has a failure history, an owner, and a way to turn it off.
The takeThe screen keeps asking about architecture, because architecture is what the phrase advertises and it is the cheap half. Anyone can name the framework, the orchestration and the tool calls. Fewer people can describe the morning the thing quietly did the wrong thing at volume, and who had to explain it to whom. Where the job is supervising automations, hire for the second description. The first one is a reading list, and the reading list is free now.
Where Olive fits
Open a role and see what the work shows
A phone screen collects a candidate's account of the automation they ran; it cannot capture them supervising AI work as it happens. Olive puts that in front of the person as work: a 40-to-60-minute occupational assignment with an AI assistant that will overreach if nobody stops it, returned as six findings anchored to timestamped excerpts, and granted to the candidate in the same form the employer reads.
Assess your shortlistAsk How Long It Ran, and Between Which Dates
Start with duration, because it is the cheapest fact in the claim to state. A date range and a rough run count are a short answer to give and cost you nothing to ask for. Someone who operated an automation has both numbers close to hand. Someone who has a recording has to reconstruct them.
In March 2025 the research group METR proposed measuring what models can do in human working time. The metric is a 50% task-completion time horizon: the length of task, timed on humans with the relevant expertise, that a model completes with a 50% success rate. On a mixed suite of software tasks, a frontier model of that moment, Claude 3.7 Sonnet, sat at around 50 minutes 1.
A fifty-minute task done right half the time is a demo.
Six months of Tuesdays is a different engineering object, and everything between the two is the work you are trying to hire. The retry. The narrowed job. The alert that fires at 3 a.m. and the person who answers it.
METR's figure describes a model working alone on benchmark tasks, and the same paper has that horizon doubling roughly every seven months, so it dates fast.
The number a resume carries instead is hours saved, and it is the weakest one on the page.
A study run between February and June 2025 followed 16 experienced open-source developers through 246 real tasks on repositories they had worked on for about five years. Afterwards they estimated that AI tooling had made them 20% faster. Measured, it had made them 19% slower 2.
Nobody should read that as a verdict on tooling. It is one small study, 16 developers on code they knew intimately. What it shows is that a person's estimate of their own gain can be wrong in direction and not only in size.
So treat hours saved as a belief, and spend the question on the run count instead, the same ten-minute discipline that works for verifying "uses AI daily" on a resume.
What Happened the First Time It Did Something Wrong?
The failure question separates the operator from the demo builder, and it should be asked in exactly those words. Not whether it ever failed, which invites a policy answer, but what happened the first time it did something wrong: what it did, who noticed, how they noticed, and what changed afterwards. An operator answers with the incident. Often with irritation.
The preference for an incident over a design has a parallel in assessment research. Researchers built an AI literacy assessment for a US Navy robotics training programme. As a measure of applied AI literacy, a scenario task simulating AI use on the job outperformed the knowledge tests they had adopted from earlier research or written themselves.
Prevailing AI literacy assessments, they argue, "emphasize foundational technical knowledge, such as programming, mathematics, and statistics, over practical knowledge such as interpreting model outputs, selecting tools, or identifying ethical concerns" 3.
The paper is a 2025 preprint from one programme, and the cohorts are small: 13 trainees in the first training, and 13 plus five later competitors in the second. It supports a design argument and nothing stronger.
What makes a first failure worth so much is that it is hard to see coming.
In 2023, 758 Boston Consulting Group consultants worked through realistic tasks with and without GPT-4. On one task chosen to sit outside the model's capability, consultants using it were on average 19 percentage points less likely to reach the right answer. In the control group, 84.5% reached it, against 60% and 70% in the two AI conditions 4.
Caveats bound the finding: one task, one sitting, a 2023 model. What survives them is the invisibility. The people using the tool could not tell which side of the line they were standing on.
A first failure is where an operator finds out, once, at a cost somebody absorbed.
Then ask who noticed, because the answer names the monitoring. A teammate noticed, a customer noticed, an alert fired, or nobody noticed for six weeks. Three of those are fine, and the fourth is the interesting one.
Ask how it was turned off. Something that ran has an off switch, a person allowed to use it and, where the work mattered, a name attached to the call. The name on the call is the screen-stage version of who on the team is accountable when AI-assisted work turns out to be wrong.
A demo is turned off by closing the tab.
Why a Demo Is Still Worth Passing Forward
Because building one is a real skill and a screen is not a court. A candidate who wired up a working agent chose a problem, broke it down, picked tools, got them talking to each other and made something run end to end. That is evidence of construction. It is not evidence of supervision, and the two are different bets on different jobs.
Which bet you need is a property of the requisition.
A role that is mostly supervising agents rather than doing the work directly has more use for the failure history. A role where somebody has to build the first automation has more use for the person who has already built three of them.
The run-count question will also fail some strong engineers. People who built internal tooling behind a platform team often never held the pager for it, and the operating history belongs to whoever ran it.
So ask whose pager it was. Someone who names the on-call team, the alert that fired and what got changed in response has answered the question, even though the failure was never theirs to absorb.
Write Down the Checkable Part, Not the Impression
Write down the dates, the run count, the failure and the fix, in the candidate's own numbers, and keep your read of them out of the note. A line saying the candidate seemed to have really operated it cannot be compared with the next person's. A line saying it ran daily from March to November and caught a bad merge in week three can.
Asking for a fact somebody can look up may change what gets written down.
One orthopaedic surgery residency program checked every publication its applicants listed and found 13 misrepresented citations among 1,100 in the 2016-2017 cycle, or 1.18%. The same program's earlier studies had reported 18.0% in 1999 and 20.6% in 2007, though the three rates do not share an identical denominator. The authors credit a PubMed identifier field added to the application form 5.
One program, one specialty, and academic authorship is unusually easy to check against an index. The identifier explanation is the authors' own hypothesis, which the study did not test.
Most agent work is not checkable that way. Asking for a repository gets you an apology about a confidentiality agreement more often than a link. What a candidate can sometimes offer instead, where the agreement allows it, is the shape of the work: a redacted run record, a runbook, an incident write-up, a dashboard with the counts still legible on it.
Where there is a record to read, the checkpoints in it are the evidence: where it stopped, who noticed, and what changed afterwards.
Then read the line again. Built agents and automations is a phrase. Two questions turn it into four things you can write down: a date range, a rough count, one failure with the person who caught it attached, and whether the thing is still running today.
None of that decides the hire. All of it survives the walk to the hiring manager's desk.
Common questions
What does "built an agent" actually mean on a resume?
A resume line about building agents covers anything from a weekend of wiring to a production automation that ran for a year. The phrase carries no standard behind it: no certification, no shared definition, no convention about whether it means designed, deployed or operated. Treat it as a claim rather than a credential and settle it with two cheap questions. How long did it run and roughly how many times, and what happened the first time it did something wrong. Someone who operated the automation answers both from memory, and together the answers place the claim somewhere on that range.
How long does this take in a recruiter screen?
A few minutes for both questions, how long it ran and what happened the first time it went wrong, with their follow-ups. That is why they fit alongside everything else in the call. The duration question is one exchange: dates and a rough count. The failure question runs longer, because a real answer arrives with detail attached and the detail is the point. If the second answer is still abstract after one follow-up, stop pressing and write down what was said. A vague answer is a fact about the screen, not proof of anything, and the hiring manager can take it further.
Is a candidate who only built demos worth passing forward?
Often yes, depending on the job. Building a working agent means choosing a problem, breaking it down, picking tools and getting something to run end to end, which is real evidence of construction. What it does not show is supervision: what the person did when the thing was wrong in a way nobody had planned for. If the role's first task is building the first automation, the demo is the closer match. If the role is mostly watching automations that already run, the failure history matters more.
What if a confidentiality agreement blocks the details?
Ask for the shape instead of the content. How long it ran, roughly how often, whether the inputs came from a live system, who was paged when it misfired, and what changed afterwards. None of that names a client, a dataset or a system, so a candidate who has operated something can often answer all five, though what their agreement allows is theirs to judge.
What if the candidate says it never failed?
Ask what it was allowed to do. A narrow automation with a small blast radius genuinely may never have done anything wrong, and a candidate who describes that narrowness is telling you what it was designed for. The other version of the same answer is that nobody was watching, which the same follow-up can surface: no alerting, no review, no owner after the person who built it moved on. Both are informative. Only one is a problem.
Should you ask to see the code or the run logs?
Ask, and expect not to get them. Most agent work sits inside an employer's systems and cannot leave. What travels more often is a redacted run record, an incident write-up, a runbook, or a dashboard with the counts still legible. Any of those is worth ten minutes of reading, and the checkpoints inside a run record carry more than an architecture diagram does. If nothing can be shared, the two screen questions, how long it ran and what happened the first time it went wrong, are still available, and they were always the cheaper test.
References
- 1. Measuring AI Ability to Complete Long Software Tasks (arXiv:2503.14499) ✓ arxiv.org Supports the duration metric and the 50-minute figure. The abstract states: "we propose a new metric: 50%-task-completion time horizon. This is the time humans typically take to complete tasks that AI models can complete with 50% success rate... On these tasks, current frontier AI models such as Claude 3.7 Sonnet have a 50% time horizon of around 50 minutes", and adds that "frontier AI time horizon has been doubling approximately every seven months since 2019", which is the dating caveat the article states.
- 2. Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (arXiv:2507.09089) ✓ arxiv.org Supports the claim that a person's estimate of their own AI gain can be wrong in direction: 16 developers, 246 tasks on repositories they had worked on for about five years, a 20% speedup estimated after the fact against a measured 19% slowdown.
- 3. AI Literacy Assessment Revisited: A Task-Oriented Approach Aligned with Real-world Occupations ✓ arxiv.org Supports the parallel that a scenario task simulating real use outperformed knowledge tests as a measure of applied AI literacy, and the quoted criticism that prevailing AI literacy assessments emphasize foundational technical knowledge over practical knowledge such as interpreting model outputs. A preprint from one programme, 13 trainees in the first training and 13 plus five later competitors in the second, as the article states.
- 4. Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality (Working Paper 24-013) ✓ mitsloan.mit.edu Supports the outside-the-frontier result used for why a first failure is informative: 758 BCG consultants, one task chosen to sit outside GPT-4's capability, 84.5% of the control group correct against 60% and 70% in the two AI conditions, an average drop of 19 percentage points.
- 5. Update on Misrepresentation of Research Publications Among Orthopaedic Surgery Residency Applicants ✓ ebi.ac.uk Supports the claim that a field somebody can look up changes what gets written: 13 misrepresented citations among 1,100 in the 2016-2017 cycle, or 1.18%, against 18.0% in 1999 and 20.6% in 2007, with the PubMed identifier credited by the authors as a hypothesis rather than a tested cause.
5 sources, numbered by first appearance. Every one was opened and checked against the claim it carries. How Olive sources claims
General guidance for hiring teams. What works at one company and one volume may not transfer to yours.
Olive assesses how a person works with AI. It does not detect AI-written documents, and it never produces a score, a ranking, or a match percentage for a person. Candidates read the same report the employer reads.