Assessment design
Make the Repo Wrong, Not the Task Harder
An AI-open exercise starts testing verification the moment the starter repo is wrong in specific, discoverable ways. Leave the task ordinary. Plant a README describing an older version of the API, a test fixture that encodes the bug it should catch, a docstring that contradicts the endpoint below it, and a pinned dependency carrying a public advisory. Give each plant exactly one observable consequence, so two graders agree on whether it was found. Then leave one thing that looks wrong and is fine.
The takeThe reflex when an assistant finishes the exercise is to reach for a harder problem, and the reflex is wrong twice over. A harder problem still lives in a public corpus, so the margin it buys shrinks with every model release. It also changes what gets selected: recall, tolerance for puzzles, appetite for obscurity that the job never asks for. A wrong starter repo selects for something the job does contain: a person who will not believe a confident answer until something outside the conversation agrees with it.
Where Olive fits
Open a role and see what the work shows
If the seeded repo gets built in-house, the two costs that do not shrink are authoring the material and keeping the evidence behind each judgment. Olive runs a 40-to-60-minute occupational assignment with an assistant that will finish the whole thing if nobody stops it, and returns six separately-evidenced findings written or edited by a human reviewer, each anchored to a moment in the session; the candidate is granted the same report.
Assess your shortlistWhy does a harder problem buy less than it used to?
A harder problem buys a shrinking margin, and it selects for something the job never contained. Public exercise banks are the model's home ground, and difficulty inside that ground is a dial with a short throw. Turn it and the pass rate falls for now, while the pool tilts toward people who practice puzzles rather than people who catch a wrong number before it ships.
In 2022, Anjan Karmakar and three co-authors evaluated OpenAI's Codex against 115 Python problem statements taken from HackerRank. It solved 96% of them with no worked examples, and 100% when given a few 1.
The pass rate was not the paper's point.
Karmakar's group reported clear signs that Codex was reproducing memorized code rather than writing it, and built an evaluation framework around re-testing models on mutated problem statements for exactly that reason 1.
A 2024 study submitted solutions written by GPT-3.5-turbo to LeetCode, the site candidates use to practice for technical rounds, and found the gradient anyone would predict: 92% of easy problems solved, 79% of medium, 51% of hard 2.
But the hard half erodes on contact.
In the same study, handing the model its own failed test cases in a second prompt improved results by 38 to 60%, and switching to GPT-4 improved them by 33 to 58% 2.
Both models are superseded, which is the argument rather than a caveat to it.
The general question here is what a work sample should test now that AI can produce the deliverable. The repo-shaped version is narrower: not what to ask for, but what to hand over.
What goes in the repo, and what does each thing test?
Four plants carry most exercises, and each one fails in a different place. Stale documentation tests whether the candidate reads the code instead of the README. A fixture written to agree with the bug tests whether a green suite ends the inquiry. A docstring that contradicts its own function shows which of the two a person trusts. And a dependency with a filed advisory finds out whether anybody looked outside the checkout at all.
Start with the README, because it is the file most likely to be taken at its word. Document a function signature that changed two releases ago. Generated code will then agree with the documentation and fail against the module, quietly, if nobody opens it.
The fixture is the most realistic of the four. Write the expected value in the test to match what the buggy function returns, so the suite goes green and the answer is still wrong.
Nothing in the run output says so.
The docstring contradiction costs one line: a comment above a handler promises a list, and the handler returns a single object.
The advisory is the only plant that cannot be found inside the repo at all. Pin a version carrying a published security advisory, say nothing about it in the brief, and see whether anyone checks the manifest against something external.
Choose defects your own codebase produces, because the exercise is a selection procedure whatever it gets called internally. The US federal Uniform Guidelines (1978) define one as any measure used as a basis for an employment decision, and say the definition covers the full range of techniques, from paper-and-pencil tests through informal interviews and unscored application forms 3.
Under the same Guidelines a validation burden attaches only where adverse impact appears 3, so the practical rule is the narrower one: a defect nobody in the job would ever meet is testing something the job does not contain.
The plants get their names, their consequences and their point values in the answer key. Writing the answer key before you send the assignment is what stops a grader inventing the standard on the second submission.
Give every planted flaw one observable consequence
One flaw, one place it shows. If the stale README was caught, the submission says so somewhere a grader can point at: a commit message, a comment, a line in the notes, a corrected call. Two graders reading the same submission should reach the same verdict without a conversation. A plant with a diffuse consequence produces an argument about whether the candidate seemed aware of it.
Interviewer ratings already come apart at the boundary. Ashby's 2026 benchmark report examined interviews with more than one interviewer in its own customer data. Around 38% of scorecard pairs carried at least one point of difference, and nearly half of those gaps fell between 2 and 3, which crosses the yes-or-no line on a 1 to 4 scale 4.
Ashby's pairs are scorecards compared against each other, not candidates, and the data carries no outcome measure, so nothing in it says which interviewer was right. Instability at the decision boundary is the whole of the claim.
Found or not found has no boundary.
Write the consequence into the key in the words a grader will actually read. "Called the README stale and used the current signature" is a yes or a no. "Showed good judgment about documentation" is a conversation, and conversations drift by the fourth submission on a Friday.
The key also has to say what happens when someone catches three plants and finishes half the work. An unfinished submission and a finished wrong one both turn up in the same week.
Leave one thing that looks wrong and is fine
One decoy per exercise, and it should read as a bug on first pass. The decoy can be an odd constant with a real reason behind it, or a function that looks unused and is called by a deploy script, or a comment written in an old vocabulary that is still accurate. Without a decoy the exercise rewards suspicion, and suspicion is not verification. A candidate who rewrites the correct thing has told you something.
Telling which side of a model's competence a task sits on is hard, and one experiment put a number on the cost.
In a pair of pre-registered field experiments, 758 Boston Consulting Group consultants were split between two task sets with no overlap, and the 373 in the second set got a task chosen to sit outside the model's capability. On that task, consultants using GPT-4 reached the correct answer 19 percentage points less often than the control group, averaged across the two assisted conditions. Unassisted, they were right 84.5% of the time, against 60% and 70% in the two assisted conditions 5.
Everything in that comparison rests on a single task, in one sitting, against a 2023 model. The transferable half is not the size of the drop but its cause: the consultants could not tell which side of the line the task was on, while the 18 inside-the-frontier tasks went the other way for the other 385.
So score the decoy in both directions.
A candidate who leaves it alone and says why has verified something. A candidate who lists it among the defects has pattern-matched to the exercise, and told you that too.
Staging a mistake and watching what happens to it is the general act, worked through in how to test whether a candidate notices AI errors. The repo is where you build it.
Calibrate the repo before a candidate opens it
Run the seeded repo past two people already doing the job, on the clock the candidate will get. If both find every plant in five minutes, the plants are decoration. If neither finds the fixture in ninety, the plant is a trap rather than a test. Adjust the plants and not the clock, and record what each of the two found, so the key has a real distribution behind it.
Across three engineering courses, 94 students were allowed ChatGPT during take-home open-book exams and had to submit their interaction transcripts alongside the solutions.
The researcher observed three patterns of use: answer retrieval, guided collaboration and critical verification. Some of the clearest evidence of reasoning appeared where students evaluated incorrect or incomplete responses, and the paper concludes that correctness of final answers alone may no longer be sufficient evidence of comprehension 6.
The study is qualitative, computing courses only, no control group and no causal claim, so it supports a direction and not a method. A transcript in hiring is also a disclosure a candidate may reasonably refuse.
Seeded repos leak, and that is the hardest thing to answer here. Send it to enough candidates and the plants get posted somewhere, and the exercise starts measuring recall. Plan rotation by the quarter, not the year.
A missed plant is not always a verification failure either. Someone new to the domain can read the docstring, believe it, and be wrong for a reason that has nothing to do with habit. That is the line between a knowledge gap and a verification failure.
Karmakar's group met a benchmark their model showed clear signs of having memorized. Their answer was not a harder benchmark, but a mutated one: the same problems restated so a remembered solution would not fit 1.
The starter repo is that move, aimed at a person. Leave the task ordinary, and let the ground be wrong.
Common questions
How many defects should a starter repo carry?
Three to five for a ninety-minute exercise, one per defect class, plus a single decoy. Fewer than three and one lucky catch decides the whole result. More than five and the exercise becomes a scavenger hunt with a clock on it, which rewards speed again by a different route. Pick one plant that is cheap to find, one that takes real reading, and one that cannot be found inside the repo at all. Then check the count against how long two people already doing the job needed.
Does seeding defects work outside engineering?
Yes, and the four defect classes have direct analogues. Stale documentation becomes a policy memo a version behind the current rule. The fixture that agrees with the bug becomes a spreadsheet whose check total is computed from the same wrong formula. The contradicting docstring becomes a chart label that disagrees with the series under it. The external advisory becomes a cited source that resolves to a real document saying something else. In every case the plant has to be a thing the role's real material actually does, not a puzzle borrowed from another field.
Should the brief say the material contains planted errors?
Say it contains errors. Do not say how many or where. Saying so keeps the exercise from testing whether a candidate guessed the format. It also holds everyone to the same instruction, which matters if anyone later asks why one submission was read differently. Withholding the count keeps the exercise honest, because part of what you are watching is when a person stops looking. One line in the brief covers it: the starting material contains mistakes, and finding them is part of the work.
What if the candidate uses the assistant to find the planted defects?
Using the assistant to find them is a pass, provided the finding was confirmed against something. A model surfacing a suspicious constant and a candidate confirming it in the code are two different acts, and only the second one is verification. Grade what the submission shows: the plant named, the thing it was checked against, and what changed in the work as a result. A candidate who repeats the model's claim without opening the file has reproduced the exact failure the exercise was built to observe.
Does a seeded repo make grading slower?
It makes grading faster and more repeatable, because found or not found is binary and a written consequence sits next to each plant in the key. The slow part moves earlier, into authoring the repo and calibrating it on people who already do the job. That cost is paid once per requisition rather than once per submission. Budget a day for the first build and expect the plants to need rotating within a quarter, since anything sent to enough candidates eventually circulates.
References
- 1. Codex Hacks HackerRank: Memorization Issues and a Framework for Code Synthesis Evaluation ✓ arxiv.org Supports the claim that a public problem bank stopped separating candidates: Codex solved 96% of 115 HackerRank Python problems zero-shot and 100% few-shot, with clear signs of memorized code, and the authors' own framework re-tests models on mutated problem statements for that reason.
- 2. Evaluating ChatGPT-3.5 Efficiency in Solving Coding Problems of Different Complexity Levels: An Empirical Analysis ✓ arxiv.org Supports the claim that raising difficulty buys a shrinking margin: 92% of easy, 79% of medium and 51% of hard LeetCode problems solved, with 38-60% improvement from supplying failed test cases and 33-58% from switching to GPT-4.
- 3. 29 CFR Part 1607 - Uniform Guidelines on Employee Selection Procedures (1978), sections 1607.16(Q) and 1607.3(A) ✓ govinfo.gov Supports the claim that a hand-built exercise is a selection procedure: the Guidelines define one as any measure used as a basis for an employment decision, covering the full range of techniques from paper-and-pencil tests through informal interviews and unscored application forms.
- 4. Recruiting Operations Benchmarks | 2026 Talent Trends Report ✓ ashbyhq.com Supports the claim that ratings are unstable exactly where the decision is made: around 38% of scorecard pairs differ by at least one point, and nearly half of those one-point gaps fall between 2 and 3 on a 1 to 4 scale.
- 5. Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality (Working Paper 24-013) ✓ mitsloan.mit.edu Supports the case for a decoy: in the pre-registered experiments with 758 BCG consultants split with no overlap between two task sets, the 373 given a task chosen to sit outside AI capability saw 84.5% of the control group correct against 60% and 70% in the two AI conditions, a 19 percentage point drop, because people could not tell which side of the line the task was on.
- 6. Reimagining Assessment in the Age of Generative AI: Lessons from Open-Book Exams with ChatGPT ✓ arxiv.org Supports the claim that the checking has to be made visible to be assessed: 94 engineering students across three computing courses submitted ChatGPT transcripts with open-book exam solutions, three patterns of use were observed, some of the strongest reasoning evidence appeared where students evaluated incorrect responses, and the paper concludes correctness alone may no longer evidence comprehension.
6 sources, numbered by first appearance. Every one was opened and checked against the claim it carries. How Olive sources claims
General guidance for hiring teams. What works at one company and one volume may not transfer to yours.
Olive assesses how a person works with AI. It does not detect AI-written documents, and it never produces a score, a ranking, or a match percentage for a person. Candidates read the same report the employer reads.