Assessment design
A Missed Error Is a Verification Failure When Nothing Was Checked
A candidate who missed the planted error has not necessarily failed verification. A miss has two causes and they need different verdicts: nothing was checked at all, or the candidate checked and lacked the domain knowledge to see the problem. What the exercise recorded separates them, if it captured whether a source was opened and whether the doubtful claim was flagged. Plant two errors, one any careful reader catches and one that needs the domain, and score them apart.
The takeThe single-error exercise spread faster than anyone decided what a miss meant, and it quietly favors people who already work in your field. Testing for the domain is fair when the posting demands it on day one and the offer funds no ramp. Fairness ends the moment the requisition promises training, because the exercise is then charging candidates for the thing the employer agreed to supply. Two errors and two lines add little to the design. The alternative is a finding about verification that is mostly a finding about tenure in an industry.
Where Olive fits
Open a role and see what the work shows
Building this yourself means authoring every case and keeping the evidence behind each judgment. Olive ships twelve authored cases per occupation and returns six separately-evidenced findings, each anchored to a moment in the session rather than to a number, with the candidate granted the same report.
Assess your shortlistWhat are the two ways a candidate can miss a planted error?
Two, and they need different verdicts. Either nothing in the submission was checked against anything outside the model, or the candidate checked and did not hold the knowledge to see what was wrong. The first is a finding about verification. The second is a finding about your domain, and it belongs in the decision only if the job required that domain on the day the offer starts.
The result on the page looks the same either way.
So the score you were about to write is a mixture.
Verification behavior is what the exercise was built to observe. Domain knowledge is what came along with it, and on a domain-shaped error it does most of the work.
The US Uniform Guidelines on Employee Selection Procedures have carried the relevant sentence since 1978. Content validity, the regulation says, is not an appropriate strategy when the selection procedure involves "knowledges, skills, or abilities which an employee will be expected to learn on the job" 1.
The same section sets the standard for a knowledge test. A knowledge being measured should be operationally defined as the body of learned information that is used in and is "a necessary prerequisite for observable aspects of work behavior of the job" 1.
A prerequisite is what someone brings through the door. If onboarding teaches it in week two, it is not one.
One more line in that section is about validation strategy rather than about findings, but it points the same way. A selection procedure resting on inferences about mental processes cannot be supported primarily by content validity 1.
Careless is an inference. No source opened and nothing flagged is a record.
Read the record before you reread the submission
Go back to what the session captured rather than to the wrong number. Three observable things separate the two misses: whether anything at all was checked, whether a source outside the model was opened, and whether the doubtful claim was flagged even though it never got resolved. A candidate who wrote that they could not confirm the figure has already answered the question.
Nothing checked is the clean case. No source opened, no claim questioned, no note saying what the deliverable rests on. Write the verification finding and leave the domain out of it.
Checked and flagged is the case rounds get wrong. The candidate opened something, could not settle it, said so in writing, and shipped the work with the doubt attached. The behavior you were testing happened. The domain question is the one they failed, and it goes on its own line.
Checked and resolved needs no adjudication at all.
The same split shows up in scoring an unfinished submission against a wrong one, and the reasoning carries across: a stated doubt is evidence, and a confident wrong answer is not.
None of that sorting works if the exercise recorded only the deliverable. A take-home that comes back as a finished document has destroyed the distinction, because the checking happened somewhere nobody can see.
Whatever the format, the record has to hold the checking and not only the answer.
Asking afterwards is a different instrument. In a study fielded in February 2025, 288 Taiwanese teachers took both a self-report and a knowledge-based test of AI literacy built on the same framework. Correlations between the self-reported and objective factors ran from 0.07 to 0.24 2.
They were teachers rather than job candidates, and the profiles the authors found included people who underrated themselves alongside people who overrated themselves. The finding is a gap between the claimed and the demonstrated, in one setting.
The gap is a reason to keep a candidate's account of their own checking beside the record, not in place of it.
Build the exercise around two errors, not one
One planted error returns one bit, and it is the wrong bit. Two errors of different kinds return two. Write one that any careful reader catches with no knowledge of your field: a total that does not add, a date falling after the event it describes, a claim the packet contradicts three paragraphs later. Then write one only the domain catches.
Score the two on separate lines. The first line reads verification. The second reads domain knowledge, which is a fact about fit rather than about habits. The design underneath is how to test whether a candidate notices when AI gets something wrong, with one change: double the plant, so the result comes apart into those two lines.
One error cannot be taken apart afterwards.
In code the plant has to be placed as well as doubled. Seeding a repo so the exercise tests checking, not typing means the two defect classes have to fail in different places.
The generic error is not a courtesy rung. In a 2023 field experiment with Boston Consulting Group consultants, one task was chosen deliberately to sit outside what GPT-4 could do. On that task, 84.5% of the control group reached the correct answer, against 60% and 70% in the two conditions with AI 3.
The drop averages 19 percentage points, on one task in one sample, with a 2023 model rather than a current one.
What carries over is not that assistants make people worse. The consultants holding one could not tell which side of the capability line the task was on.
What a candidate already knows also changes what the assistant does for them. Across a staggered rollout in 2020 and 2021, 5,179 support agents at one software firm got access to a conversational assistant. Issues resolved per hour rose 14% on average: 34% for the novice and low-skilled agents, and almost nothing for the experienced ones 4.
Those gains sit inside one firm and one occupation, with an assistant trained on that firm's own transcripts, on problems that have correct answers. The direction is all the design point needs. An exercise handing everyone the same assistant is not handing everyone the same task.
Which miss should change the hiring decision?
The requisition decides, and it should have decided before the exercise went out. Where the posting demands the domain on day one and nobody is funding a ramp, a domain miss is a real finding and can be written plainly. Where the plan was always to teach the domain in the first month, the second error is context, and only the first belongs in the write-up.
Record both results either way. The verification line is the one that generalizes, because hiring someone who can tell when AI is wrong is a bet about behavior rather than about what a candidate happens to know this quarter.
A domain gap nobody wrote down turns into a story about judgment by the second debrief, and the story outlives the evidence.
Deciding in advance is the same discipline as choosing whether the AI line is a hurdle or a trade. A bar restated after the results arrive is a bar that already moved.
The evidence base for work samples is quieter about applicants than it sounds. Sackett and colleagues put work sample validity at .33 in their 2022 revision, down from the .54 that circulated for decades. The estimate rests on a meta-analysis of 54 studies of which 53 were concurrent 5.
Applicants are the case the estimate barely measured.
Concurrent means the people tested were already doing the job. They held the domain. So the .33 was measured almost entirely on incumbents, and it says less than it looks about how an outsider performs on the same exercise in an afternoon.
Some work never separates the two misses at all, and the objection built on that is a fair one. A coder who cannot see a wrong code, a lawyer who cannot see a bad citation, a controller who cannot see a misapplied rate: in that work the domain is the verification.
An employer hiring for output next quarter is right to treat the domain miss as the finding.
Send the next packet with two errors in it and a scorecard with two lines for them. A candidate who catches the arithmetic, flags the rate they could not settle, and says so in writing has handed over both answers at once. One of them is about your training plan, not about them.
Common questions
Is missing a planted error automatically a verification failure?
No. A miss has two causes: nothing was checked, or the candidate checked and did not have the domain knowledge to see the problem. Only the first is a verification finding. Separate them with the record the session captured, not with a second reading of the deliverable. The record shows whether a source outside the model was opened, whether the doubtful claim was flagged, and whether the candidate said in writing that they could not confirm it. A candidate who flagged the figure and could not resolve it demonstrated the behavior and failed the domain question, which are two results and not one.
What should you plant if the candidate does not know the field yet?
Two errors of different kinds, scored on separate lines. The first needs no domain: a total that does not add, a date after the event it describes, a claim the packet contradicts a few paragraphs later. Any careful reader catches it, so a miss there is about checking. The second needs the domain: a plausible code, a rate that is wrong only under a local rule. A miss there is about knowledge. If the role trains the domain in the first month, the second error is context for the debrief rather than a finding in the decision.
Can you just ask the candidate whether they verified the output?
Ask, but do not score the answer as the measurement. Self-report and demonstrated capability come apart. Among 288 Taiwanese teachers who took both a self-report and a knowledge-based test of AI literacy built on the same framework, correlations between the self-reported and objective factors ran from 0.07 to 0.24. They were teachers rather than candidates, and the study found people who underrated themselves as well as people who overrated themselves. The point is narrower than a claim about honesty: an account of checking and a record of checking are two different instruments, and the exercise can hold the second.
Does one planted error make the exercise a domain test?
Sometimes, and nothing in the result will tell you which time. With one error, a pass and a fail both compress verification behavior and domain knowledge into a single bit, and the bit favors whoever has worked in the field longest. The Uniform Guidelines make the distinction worth taking seriously. Content validity is not an appropriate strategy where the procedure involves knowledge an employee will be expected to learn on the job. A measured knowledge should be defined as a necessary prerequisite for the work. Check the requisition. If it promises to teach the domain, the exercise should not be pricing it.
How should the finding be written so it survives a debrief?
Write what the record shows, not what it implies about the person. Careless is an inference. Opened no source and flagged nothing is an observation. The US Uniform Guidelines, writing about validation rather than about findings, say that a selection procedure resting on inferences about mental processes cannot be supported primarily by content validity. Two lines do the job: what was checked and against what, and what the candidate did not know. Keep both lines even when only one changes the decision, because a domain gap nobody wrote down comes back six weeks later as a story about judgment.
What if the role really does require the domain on day one?
Then the domain miss is a finding, and you can say so plainly. Some work does not separate the two at all: a coder who cannot see a wrong code, a lawyer who cannot see a bad citation, a controller who cannot see a misapplied rate. In that work the domain is the verification, and hiring for output next quarter makes the domain miss the result that matters. Say it in the posting and in the brief, so a candidate can tell before they start whether the exercise assumes knowledge they do not have.
References
- 1. 29 CFR Part 1607 - Uniform Guidelines on Employee Selection Procedures (1978), section 1607.14(C) ✓ govinfo.gov Supports the three regulatory claims in this article: 1607.14(C)(1) states that content validity is not an appropriate strategy when the procedure involves 'knowledges, skills, or abilities which an employee will be expected to learn on the job' and that a procedure based upon inferences about mental processes cannot be supported solely or primarily on the basis of content validity; 1607.14(C)(4) states that a measured knowledge should be operationally defined as that body of learned information used in and 'a necessary prerequisite for observable aspects of work behavior of the job'.
- 2. How to Assess AI Literacy: Misalignment Between Self-Reported and Objective-Based Measures ✓ arxiv.org Supports the claim that a candidate's own account of their checking is a different instrument from a record of it: correlations between self-reported and objective AI-literacy factors ran from 0.07 to 0.24 among 288 Taiwanese teachers, with profiles including both overestimation and underestimation.
- 3. Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality (Working Paper 24-013) ✓ mitsloan.mit.edu Supports the claim that people cannot reliably tell which side of a model's capability line a task sits on: on one task chosen to sit outside AI capability, 84.5% of the control group was correct against 60% and 70% in the two AI conditions, an average decrease of 19 percentage points.
- 4. Generative AI at Work (NBER Working Paper 31161) ✓ nber.org Supports the claim that the same assistant does different work depending on what the person already knows: issues resolved per hour rose 14% on average across 5,179 support agents at one software firm, 34% for novice and low-skilled workers and minimally for experienced and highly skilled ones.
- 5. Revisiting Meta-Analytic Estimates of Validity in Personnel Selection: Addressing Systematic Overcorrection for Restriction of Range ✓ gwern.net Supports the claim that the work sample evidence base was measured mostly on incumbents: Sackett et al. estimate work sample validity at .33, down from the widely repeated .54, resting on a meta-analysis of 54 studies of which 53 were concurrent.
5 sources, numbered by first appearance. Every one was opened and checked against the claim it carries. How Olive sources claims
General guidance for hiring teams. What works at one company and one volume may not transfer to yours.
Olive assesses how a person works with AI. It does not detect AI-written documents, and it never produces a score, a ranking, or a match percentage for a person. Candidates read the same report the employer reads.