Assessment design
Ask Who Fixed It When It Broke, Not Who Built It
An operations candidate who built and shipped an internal tool with AI has demonstrated two things: they found a manual process worth ending, and they carried the task to a working end without a specialist. Neither says much about engineering judgment. For an operations role, count that as a strong positive; for a job that owns production systems, treat it as a starting point. Then ask the maintenance questions: what broke, who fixed it, who uses it now, and what they would not build the same way again.
The takeThe argument in the room is about the tool, and the tool is the least interesting object in the story. Somebody in operations decided a manual process was worth ending, and then did not stop when the first version worked. A candidate who kept going is describing a second week and a third that nobody assigned them, and that is the part no assistant supplies and no resume line can show.
Where Olive fits
Open a role and see what the work shows
If you want to see judgment at work rather than hear a story about it, the expensive parts to build are an authored case and the evidence behind each finding. Olive ships an occupational assignment with an assistant that will overreach if nobody stops it, and returns six findings a person wrote or edited, each anchored to a moment in the session and granted to the candidate in the same form.
Assess your shortlistWhat does shipping an internal tool without an engineer prove?
Two things, both real and both narrower than the sentence sounds. The candidate found a manual process worth ending, and they carried a task to a working end without a specialist, which is a completed loop. Neither of those is engineering judgment. Both are habits an operations job runs on, and both are hard to fake in a follow-up conversation.
The sentence itself arrives on a resume, or in the last five minutes of a screen, and it lands in a room holding two reflexes.
One calls it not real engineering and discounts it to zero. The other reads it as proof the person can now do engineering work, and moves them toward a job nobody has tested them for.
Both reflexes are answering a question about software.
The claim is about a person. Weigh it the way you would weigh any other line describing what somebody did somewhere else, which is to say lightly until it has been opened.
A meta-analysis of 81 independent samples put the corrected correlation between prehire work experience and later job performance at .06, and experience in relevant tasks, jobs or occupations at about .07 1. Those crude measures of the amount, duration or type of experience are what a resume screen reads, and on their own they carry close to nothing.
So the value sits in the follow-up, and the follow-up should not be about the tool.
Why a working internal tool is the cheap half
Because the expensive parts do not announce themselves while the thing is running. Access control, what happens on bad input, what the tool does when an upstream system returns nothing, and who gets called when it stops overnight are all invisible on a working demo. A model writes plausible versions of all four. Nobody opens them, because the tool is doing what it was asked to do.
The closest measurement on part of that gap comes from security research.
Neil Perry, Megha Srivastava, Deepak Kumar and Dan Boneh, then at Stanford, ran a user study in which participants wrote security-relevant code with and without an AI assistant. Participants with the assistant wrote significantly less secure code, and were more likely to believe they had written secure code 2.
One result in the same study says what to screen for. Participants who trusted the assistant less, and who worked harder at the wording and format of what they asked it for, produced code with fewer vulnerabilities 2.
The safer code sat with the doubters.
A similar shape shows up outside code. In a 2023 Harvard Business School field experiment, Boston Consulting Group consultants were given one task deliberately chosen to sit outside GPT-4's capability. Averaged across the two groups working with the model, they were 19 percentage points less likely to reach the right answer than the control group 3.
Keep the scope in view: one task, one sample, a 2023 model that is no longer a current one.
What the experiment shows is not that the tool degrades people, but that the people using it could not tell which side of the line their task was on.
An internal tool built by an operations manager is a task nobody classified either. The failure mode is a procedure that keeps running while it is wrong, which is the problem behind a runbook that fails quietly.
Ask what broke after the tool shipped, and who fixed it
Four questions separate the strong version of this claim from the thin one, and none of them is about how the tool was built. What broke after it shipped. Who fixed it. Who uses it now, and how many of them. What they would not build the same way again. Somebody who has been on the other side of a failure can usually answer all four from memory.
Do not accept a candidate's own account of how much the assistant did, in either direction.
Working on repositories they had known for about five years, 16 experienced developers in a randomized trial forecast that AI tools would cut their completion time by 24 percent. Measured, the tools made them 19 percent slower, and afterwards those developers still believed they had been sped up 4.
Sixteen people, one setting, mature code they knew well. What travels is the direction of the error rather than its size: people can misjudge their own AI-assisted work, even on code they know well.
Resist the pull toward authorship. Whether the assistant or the candidate typed a given function is the question of who actually did the work, and it is a slower road than asking what the thing has cost since.
An origin story can be rehearsed. A maintenance answer has dates in it.
Ask who owns the tool now. Median tenure for US wage and salary workers was 3.9 years in January 2024, so at that moment half of them had been with their employer for less 5.
Whoever inherits an internal tool is often not the person who wrote it. The mismatch is a fact about staffing and does not count against the candidate, though one who has thought about it may raise the handover unprompted.
The build took a week. The upkeep is the job.
Weigh the tool against the operations job, not against engineering
For an operations role, a shipped internal tool is a strong positive and should move the decision. The person automated something nobody assigned them. For a job that owns a production system, the same sentence is a starting point rather than a qualification. The failures that job exists to prevent are the ones a working tool hides.
The weight you land on will be applied to the next candidate who makes the same claim.
Adopted in 1978, the US federal Uniform Guidelines on Employee Selection Procedures define a selection procedure as any measure, combination of measures, or procedure used as a basis for any employment decision. They say the term reaches informal or casual interviews and unscored application forms 6.
So a credit given to a resume line is a selection procedure under that definition whether or not anyone wrote it down, and one invented while reading a resume gets applied to one person.
Check the current position with employment counsel; the text above is public regulation and not advice.
Write the credit down before the next resume arrives. A posting that wants this should name the ownership task rather than the build: carrying a workflow after the person who wrote it left, or being the one called when a scheduled job stops.
Every requirement needs a stage that tests it, and the stage for this one is a twenty-minute conversation about upkeep.
Nothing in the maintenance questions separates a tool that never broke because nobody was watching from one that is small, well-scoped and simply works. A round rewarding war stories will prefer whoever shipped the fragile version.
Decide what a thin maintenance answer is worth before you hear one
Decide before the round. Otherwise the reading gets made under time pressure. Thin means the follow-ups produced no incident, no user count and no regret. Score that as neutral rather than as a negative, and put the weight on whatever else the loop measures. Somebody can have shipped a real thing and be poor at describing it.
A thin answer is not evidence of a lie.
If you cannot evaluate the tool yourself, do not try to.
Hand the maintenance answers to somebody who runs systems for a living. Ask them where the bar sits rather than what the verdict is, and keep the decision, which is the standing move for hiring into a job you have never done yourself.
Then come back to the line on the resume. It said the candidate built and shipped an internal tool with AI.
After twenty minutes it can say something narrower and far more useful. In a hypothetical case, the tool is a reconciliation the finance team runs every Monday. It broke twice since March, was restarted both times by the person who built it, and is now known to a second person as well.
The second version has earned the credit. The first one was never a claim about software.
Common questions
Does an internal tool built with AI mean an operations candidate can do engineering work?
No. A tool built with an assistant means they identified a problem worth automating and carried it to a working end without a specialist, which is initiative and follow-through. Access, bad input and who is called when it stops are what separate a working tool from a maintainable one. An assistant produces those plausibly, and nobody opens them. Treat the claim as strong evidence about how the person operates and weak evidence about software judgment, then test the second thing directly if the role needs it.
What should I ask a candidate who says they built an internal tool with AI?
Four things, all about after the launch. What broke once it was in use. Who fixed it, and how long that took. Who uses it now, and how many people. What they would build differently. Someone who lived through a failure answers with dates, the names of systems and a specific bad input; someone who has only seen a demo tends to stay general, though so do some people who are simply poor at describing their own work. Ask the same four of everyone who makes this claim, so the round stays comparable.
How much should a shipped tool weigh against domain experience?
Set the weight by what the role owns. In an operations job the tool is direct evidence of the work, which is spotting a manual process and ending it. Where the role owns a production system, the tool is a reason to run a technical stage rather than a substitute for one. Write the weight into the scorecard before applications arrive, because a credit invented after reading one resume gets applied to one person and is hard to explain to the first candidate who asks how the decision was made.
What if the tool belongs to a former employer and cannot be shown?
Nothing is lost, because the demonstration was never the evidence. Ask about the failure and the repair, which describes the candidate's own conduct rather than the employer's property: what the tool got wrong, how they found out, what they changed, who they had to tell. A candidate under a confidentiality agreement may be able to describe the shape of a system without naming it or showing code, and is the one who knows what their agreement allows.
Is it a bad sign if the tool broke?
It is usually the best sign available. A tool that ran long enough to break was used by somebody, and the repair is where judgment shows: what the candidate checked first, what they learned about their own assumptions, what they added afterwards. The unhelpful answer is that nothing ever went wrong, which may mean the tool was small and well-scoped, or may mean nobody was watching. Ask how they would tell those two apart.
References
- 1. A meta-analysis of the criterion-related validity of prehire work experience ✓ digitalcommons.unf.edu Supports the claim that a resume line describing prior experience carries little: across 81 independent samples the corrected correlation between prehire work experience and later job performance was .06, and experience with relevant tasks, jobs or occupations was about .07.
- 2. Do Users Write More Insecure Code with AI Assistants? ✓ arxiv.org Supports two claims: participants with access to an AI code assistant wrote significantly less secure code and were more likely to believe they wrote secure code, and participants who trusted the AI less and engaged more with the language and format of their prompts produced code with fewer vulnerabilities.
- 3. Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of Artificial Intelligence on Knowledge Worker Productivity and Quality (Working Paper 24-013) ✓ mitsloan.mit.edu Supports the 19 percentage point drop on one task deliberately chosen to sit outside AI capability: 84.5 percent of the control group answered correctly, against 60 percent and 70 percent in the two AI conditions.
- 4. Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (arXiv:2507.09089) ✓ arxiv.org Supports the 16 experienced developers on repositories they had worked on for about five years, the 24 percent speedup they forecast, the measured 19 percent slowdown, and the 20 percent speedup they still estimated after finishing.
- 5. Employee Tenure Summary (Employee Tenure in 2024, USDL-24-1971) ✓ web.archive.org Supports median tenure of 3.9 years for US wage and salary workers in January 2024 and the reading that at that moment half of them had been with their employer for less.
- 6. 29 CFR Part 1607 - Uniform Guidelines on Employee Selection Procedures (1978), sections 1607.16(Q) and 1607.3(A) ✓ govinfo.gov Supports the definition of a selection procedure as any measure, combination of measures, or procedure used as a basis for any employment decision, covering the full range of assessment techniques through informal or casual interviews and unscored application forms.
6 sources, numbered by first appearance. Every one was opened and checked against the claim it carries. How Olive sources claims
General guidance for hiring teams. What works at one company and one volume may not transfer to yours.
Olive assesses how a person works with AI. It does not detect AI-written documents, and it never produces a score, a ranking, or a match percentage for a person. Candidates read the same report the employer reads.