Assessment design
Which Failure Does the Job Survive, Missing Work or Wrong Work?
A complete submission carrying two wrong numbers scores lower than a half-finished one that was checked, in any role where a wrong figure reaches a client, a filing, a patient record or production before anyone catches it. Where volume is the constraint and a second reader touches everything anyway, the trade flips and coverage should carry the heavier weight. Decide which failure the role survives before opening either file, then score coverage and correctness on separate lines.
The takeRubrics reward completion because completion is the one thing a reader can see without doing the work again. It is also the cheapest thing a candidate can now buy, so a completion-weighted rubric can quietly turn into a measure of whose assistant was better. The candidate who stopped at half and wrote down what they could not settle did the harder thing: they read their own output the way a stranger would. Most scorecards carry no line for that, so most scorecards read it as missing work.
Where Olive fits
Open a role and see what the work shows
Building the checked-work version of this in house means authoring the case and the evidence trail behind each judgment, which is where the cost sits. Olive returns six findings from one 40-to-60-minute occupational assignment, each written or edited by a person and anchored to a timestamped excerpt, and the candidate is granted the same report.
Assess your shortlistName the failure the role cannot absorb
Many jobs absorb missing work more easily than wrong work, so start there. An unfinished section is visible on Monday and costs a day of somebody's time. A wrong figure inside a finished deliverable stays invisible until it reaches a client, a filing, a patient record or a merged branch, and by then a person has acted on it. Decide which of the two your role absorbs before you open either file.
Write the answer down before you read either submission.
After reading, you are no longer scoring a trade. You are scoring a person who made one, and the two feel identical from inside the room.
The trade has a name everywhere else in hiring. A process tuned to avoid bad hires turns away good ones, and a process tuned to avoid missed hires lets weak ones through. Which mistake your process is built to avoid is the same question one level up, and most teams have never answered it out loud.
Some roles flip the answer, and pretending otherwise helps nobody. In a support queue where a second reader touches everything before it ships, throughput is what the role is paid for, and coverage should carry the heavier weight. Say so in the brief.
The flip is a decision, not a default.
Why does the finished submission read as the stronger one?
Completion is the one property a reader can check without redoing the work. Correctness is not. Verifying two figures inside a finished model means opening the same sources the candidate opened, which is the cost the exercise existed to save. So a rubric drifts toward what is cheap to see, and finishing is now the cheapest thing on the page.
A confident wrong answer looks exactly like a confident right one.
A pre-registered 2023 field study split 758 Boston Consulting Group consultants between two experiments: 18 set-piece tasks inside what the model could do, and one task outside it.
On the outside task, 84.5% of the control group reached the correct answer, against 60% and 70% in the two groups given GPT-4 1.
Which side of the line the task sat on was not something the consultants could tell. Coaching moved the wrong way: the group given a prompt-engineering overview fell further than the group given none 1.
The study covers one task, one sitting and a 2023 model, so nothing in it says an assistant makes people worse in general.
The finding that travels is narrower: the failure arrives finished.
METR ran 16 experienced open-source developers through 246 real tasks between February and June 2025, in repositories they had worked in for about five years. They were 19% slower with early-2025 AI tools allowed.
Beforehand the same developers had forecast a 24% speedup, and afterwards they still estimated a 20% one 2.
Familiar repositories and sixteen developers make a narrow result, and it does not transfer to juniors or to new work. But they were wrong about the direction of their own speed, not only about its size.
A candidate's sense of having finished is evidence about confidence.
Polish carries no information either, which is why grading a take-home when every submission comes back polished no longer works on the artifact alone.
Score coverage and correctness on separate lines
Two columns, filled in separately, then combined by a rule you wrote before the submissions arrived. Coverage asks how much of the brief was attempted. Correctness asks what share of the load-bearing claims survive a check against the source. Collapsing both into one impression is where the argument in the debrief starts, because nobody can say afterwards which column moved the total.
Combining evidence by a rule beats combining it by judgment, and the gap has been measured. Kuncel, Klieger, Connelly and Ones pooled selection and admissions studies and found applicant data correlated .44 with later job performance when combined mechanically by a formula, against .28 when the same kinds of data were combined holistically by expert judgment 3.
The job-performance half of that comparison rests on 9 studies, with about 1,392 people behind the mechanical estimate and 1,156 behind the other 3.
The comparison speaks to the last step rather than to which evidence gets gathered. A mechanical rule can be two written scores added with weights you set in advance, and the authors say so.
Nothing in that rule requires an algorithm to pick anyone.
Correctness stays an empty column without an answer key. Two graders reading the same finished model will disagree about whether a figure is wrong. Somebody has to decide what right looks like first, which is why writing the answer key before the assignment goes out is the step that makes the second column scorable.
Record one more thing beside the two columns: what the candidate said they had not checked. A submission that names its own gaps is a different object from one that stops without comment, and the difference will be gone from your memory in a week.
Ask each candidate what they cut, and why
Twenty minutes, the same two questions, asked of both candidates in the same order. What did you decide not to do, and why. What in here would you check first with another hour. The half-finisher's answer shows whether stopping was a judgment or a stall, and the finisher's shows whether the two wrong numbers were a lapse or a habit.
A strong stop carries a reason and a boundary. Reconciling three segments to the filing, finding that the fourth needed a source nobody could open, and flagging it rather than estimating it is one act. Running out of time is another.
Both look like an empty section from the outside.
The finisher gets the same twenty minutes. Ask what was checked, against what, and what would have been checked next.
A candidate who names the two wrong numbers before you do has shown the checking habit you were hoping to find. A candidate who defends both without naming what they checked them against has told you the check probably never happened.
A missed error is not automatically a verification failure. Someone new to your field may not know that the figure is out of range, which makes it a knowledge gap rather than a habit. The two separate cleanly once you ask what they would have needed to catch it.
Nothing in two checked pages shows how the candidate behaves at hour four, behind schedule, with the last section still empty. The verified half is simply less evidence, and that is the case for scoring it lower rather than higher. Stopping early is also a safe move once candidates learn that the rubric rewards one.
Write the brief so nobody has to guess the trade
Two lines settle most of this. Say that finishing is not scored and checking is, or say the opposite where throughput is what the role buys. Then scope the task so a careful person can finish it in the time you named, because an unfinishable brief grades whoever was willing to skip the check.
A clock changes what the exercise is testing, and what a time limit on a hiring assessment measures is speed. Speed is the part of this work that got cheap. An exercise weighted toward completion is buying it anyway, whether or not anybody wrote that down.
Under the ADA, whether extended time is owed can turn on the same question: what does the test claim to measure.
Extended time appears in the EEOC's 2022 technical assistance, alongside an alternative version of a test compatible with a screen reader, among accommodations that may be effective in an algorithmic assessment 4.
Those are examples rather than a required menu. The document came off eeoc.gov in January 2025 and now reads only in the Internet Archive.
Make the task about the failure your job actually has.
In the job-knowledge meta-analysis behind Sackett and colleagues' 2022 revision, all 164 studies together produced a mean observed validity of .22, while the 59 studies using tests built for the job in question produced .31 5.
Comparing two subsets of one meta-analysis is weaker than an experiment, and job-knowledge tests presuppose people who already hold the knowledge. The direction is still worth the design cost.
What an exercise is about does more work than the format it arrives in.
Send the next brief with the weights printed at the top and one defect seeded in the source material. Then the two files on your desk stop being a fairness puzzle. One candidate found the defect and ran out of road. The other ran straight past it, and the brief already said which you were paying for.
Common questions
Should a partial submission be marked down for being partial?
By the amount your coverage score says, and only where the brief told candidates coverage was scored. A partial submission that names what was left undone and why is not the same thing as one that stops in silence, so record which one you have. Where the expensive failure in the role is a wrong figure reaching somebody who acts on it, a verified half is stronger evidence of the habit being hired for. Where throughput is what the role buys, weight coverage higher and say so before anyone starts.
How do you check correctness without redoing the whole task?
Pick the load-bearing claims before submissions arrive, usually three to five, and check only those. An answer key naming what each one should resolve to turns a judgment call into a yes or a no, and it lets two graders fill the same column separately. Everything else gets read for coverage rather than for accuracy. The preparation is paid once; the rework would be paid on every submission that lands.
Is it fair to compare two candidates who made different tradeoffs?
Yes, where both received the same brief and the same rubric, and the rubric said which line carried more weight. What is unfair is setting the weights after seeing which candidate you liked. Write the weights down, apply them to everyone who got that brief, and keep the sheet. A candidate who asks why they were rejected then gets an answer that survives being repeated to somebody else.
What if a candidate finishes everything and gets all of it right?
Nothing in a two-column rubric penalizes that. A submission clearing both columns has cleared both columns, and the walkthrough is where it earns the rest. But when every submission clears both, the exercise has stopped separating anyone: raise the difficulty, or seed a defect in the source material that only a check catches, and read what happens next. An exercise nobody fails is measuring who had the time.
Does a wrong number in a take-home predict wrong numbers on the job?
One afternoon is one observation, and no work sample carries more than that. What raises the signal is knowing whether the candidate checked at all, which the artifact does not show and a twenty-minute walkthrough does. Ask what was verified and against what. A candidate who cannot name a single source they opened has said something about the habit. A candidate who names two and missed a third has said something about the day.
References
- 1. Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of Artificial Intelligence on Knowledge Worker Productivity and Quality (Working Paper 24-013) ✓ mitsloan.mit.edu Supports the 758 consultants and 18 set-piece tasks, and the outside-frontier task on which 84.5% of the control group reached the correct answer against 60% and 70% in the two AI conditions, with the prompt-engineering group falling further than the group given no overview.
- 2. Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (arXiv:2507.09089) ✓ arxiv.org Supports the 16 experienced open-source developers, 246 real tasks on repositories worked in for about five years, the measured 19% slowdown, and the 24% forecast speedup beforehand against the 20% estimated speedup afterwards.
- 3. Mechanical Versus Clinical Data Combination in Selection and Admissions Decisions: A Meta-Analysis ✓ gwern.net Supports the .44 correlation with job performance for mechanically combined applicant data against .28 for holistic combination, the 9 studies and roughly 1,392 and 1,156 people behind those two estimates, and the point that a mechanical rule can be a plain weighted scorecard.
- 4. The Americans with Disabilities Act and the Use of Software, Algorithms, and Artificial Intelligence to Assess Job Applicants and Employees ✓ web.archive.org Supports extended time and an alternative version of a test, including one compatible with a screen reader, as accommodations that may be effective in an algorithmic assessment, and the fact that the guidance was removed from eeoc.gov in January 2025 and survives in the Internet Archive.
- 5. Revisiting Meta-Analytic Estimates of Validity in Personnel Selection: Addressing Systematic Overcorrection for Restriction of Range ✓ static1.squarespace.com Supports the job-knowledge comparison: 164 studies with a mean observed validity of .22 against .31 for the 59 studies using tests built for the job in question.
5 sources, numbered by first appearance. Every one was opened and checked against the claim it carries. How Olive sources claims
General guidance for hiring teams. What works at one company and one volume may not transfer to yours.
Olive assesses how a person works with AI. It does not detect AI-written documents, and it never produces a score, a ranking, or a match percentage for a person. Candidates read the same report the employer reads.