Assessment design
Hand a Manager Candidate One Flawed Document and Twenty Minutes
The exercise that shows whether a manager candidate can review AI-assisted work is a review, not a conversation about reviewing. Hand them one document in the shape their team ships, fluent and organised and mostly right, with exactly one thing wrong in a place that only checking catches. Give them twenty minutes and ask for three things: what they send back, what they would ask the person who wrote it, and what standard stops the next error.
The takeManager loops run on conversation because conversation is cheap to schedule and pleasant to sit through. Nobody has to build anything, and every candidate arrives with an answer about their standards already assembled. The cost lands in month three, when the panel finds out whether the person who talked best about review does any. A flawed document takes an afternoon to write and works for every candidate on the requisition. There is no good reason left to infer the central act of the job from a description of it.
Where Olive fits
Open a role and see what the work shows
Building this in-house, the expensive parts are the answer key and the record of what each candidate did with the work. Olive ships authored cases per occupation and returns six separately-evidenced findings, each anchored to a moment in the session, and the candidate is granted the same report.
Assess your shortlistWhy do four rounds of manager interviews contain no review?
Because a manager loop is built almost entirely out of conversation, and conversation measures the account a person gives of their own judgment. Four rounds produce four descriptions of how this person reviews work. Not one of the four contains a piece of work being reviewed, and reviewing work is central to the job once a team starts drafting with a model.
Candidates know the question is coming and arrive with the answer built.
The closest measured relative of that interview question is the situational judgment test, which puts a scenario in front of someone and asks what they would do.
Validity for that format sits at .26 in the 2022 re-analysis of the selection literature by Sackett, Zhang, Berry and Lievens. The figure holds whether the test asks what a candidate should do, across 96 studies, or what they would do, across 22 1.
Virtually every study behind that estimate was run on people who already had the job, not on applicants being sorted. Treat .26 as an average across many different tests, not a verdict on any single question.
Good questions still have a place in the loop. Four questions for a manager whose team ships with AI will tell you what this person has noticed about their own team and where they think the risk sits.
But an account of review and an act of review are different observations, and only one of the two can be wrong in front of you.
Build the document your team would send, then break one thing in it
Build one artifact in the shape your team ships: a memo to a client, a pull request, a forecast model, a customer analysis. It should read the way AI-assisted work reads at its best, fluent and organised and right about almost everything.
Then put one thing wrong in it, somewhere only checking reaches: a figure the recommendation rests on that no cited source supports, or a citation resolving to a real document that says something else.
Plant one error rather than five, because a document with five stops being a review and becomes a puzzle.
The reason for planting anything is that a fluent wrong answer does not announce itself.
In a Harvard Business School field experiment, 373 of 758 Boston Consulting Group consultants were assigned one task deliberately chosen to sit outside GPT-4's capability. In the control group, 84.5% reached the right answer, against 60% and 70% in the two groups working with the model 2.
The numbers come from a single task, a single sample, and a 2023 model whose limits have moved since. The finding was never that the tool makes people worse.
Which side of the line the task sat on was invisible to the consultants, and a planted error puts a manager candidate on exactly that line.
Take the document from work you have already finished, so the answer is known before the question goes out: use a real task you already solved, and keep the outcome. Then write the answer key before you send the assignment: what the error is, what a full catch names, what a partial catch looks like.
A job-shaped exercise is worth building and worth not overselling. The familiar line that work samples predict job performance best of all, at .54, was dropped by its own lineage.
The 2022 re-analysis estimates .33, from 54 studies covering 10,469 people, 53 of which tested people already doing the job 3. The number is a reason to build the exercise, not a reason to let it decide alone.
Which three scores matter, and how does a candidate fail each?
Score the three outputs separately: the response the candidate would send back, the question they would put to the author, and the standard they would set so the next draft needs no conversation. A candidate can pass any one of the three and fail the other two, and rolling them into a single impression hides which one went wrong.
The first failure is the candidate who reviews the prose. They tighten the framing, cut a paragraph, admire the structure, and never open the number the whole recommendation rests on.
Nothing in the document was ever hard to read, and reading was not the thing at risk.
The second failure is quieter, and it can pass a panel. The candidate finds the bad figure, corrects it, and hands back a clean document. Nobody on the team learns what to check, and the review has moved permanently onto the manager's own desk.
The third is the candidate who fixes the instance and sets nothing.
Ask what changes in the template, the checklist or the definition of done, and you learn whether this person manages the work or simply does it faster than the team can.
Write the anchors for each of the three before the first candidate sits down. In Levashina, Hartwell, Morgeson and Campion's review of the structured interview literature, Taylor and Small's meta-analysis covered 19 past-behaviour interview studies. Interviews using anchored rating scales reached higher validity, .35 against .26, and higher agreement between raters, .77 against .73, than interviews without them 4.
The 19 studies were of interviews rather than documents, so treat the numbers as an argument for anchors and nothing more. The cheaper test is local: two people score the same twenty minutes separately, then compare, and eight of ten is not agreement until you discount chance.
The same document, the same twenty minutes, everyone on the req
Run the same document, in the same twenty minutes, with the same three questions, for every manager candidate on the requisition. A comparison exists only where the input was held still. Giving one candidate the flawed forecast and another a conversation about their review philosophy produces two impressions and nothing that can be set side by side.
Running everyone through the same document matters more once the exercise starts deciding who advances. A hiring manager who runs the document review on finalists and passes on the person who missed the error is running a selection procedure, whatever the calendar invitation called it.
The definition is broad: the US federal Uniform Guidelines cover any measure, combination of measures, or procedure used as a basis for any employment decision. The 1978 text reaches as far as what it calls "informal or casual interviews and unscored application forms" 5.
Calling the session informal does not move it outside that definition.
The burden is narrower than the definition. Validation attaches only where adverse impact appears, and where it does, the question belongs with employment counsel. Read anything that moves the decision is a selection procedure before the first session.
If the missed error was a domain gap rather than a habit, the exercise scored the wrong thing. A manager who has never read your revenue model may not know which number in it is load-bearing, and twenty minutes on unfamiliar material is a hard test of unfamiliar material.
Three scores per candidate, written the same day, is the whole record this exercise leaves.
By the fourth candidate the document has stopped being a test of them and started being a record of your own bar. The scores say what this team counts as a catch, what it counts as a fair question to the author, and what it accepts as a standard.
Keep the scores beside the requisition. The next time two managers on one role disagree about whether a candidate checks anything, the argument has something to sit on.
Common questions
How long should a manager review exercise take?
Twenty minutes of the candidate's time, live or asynchronous, on one document of about a page. The length is a design constraint rather than a courtesy. An exercise long enough to need its own scheduling gets skipped for the candidate the panel already likes, and only the exercise short enough to run every time produces a comparison. Send the three questions in writing at the start, so the clock goes on reviewing rather than on working out what is wanted.
What should the planted error be?
Something load-bearing and invisible to a careful read: a figure the recommendation rests on that no cited source supports, a citation resolving to a real document that says something else, a change whose tests agree with the bug. Avoid typos, formatting and anything a spell checker would catch, since those measure attention rather than checking. One error per document. Write down beforehand what a full catch names, what a partial catch looks like, and what counts as a miss.
Should the candidate be told the document contains an error?
Tell them the document stands in for AI-assisted work from their team and should be reviewed the way that work would be. Do not say that an error was planted. Announcing it turns a review into a search, and a search is a different skill: what is worth observing is whether checking is a habit or an instruction. If the candidate asks directly, say you will not describe what the document contains, and that the question is what they would send back.
What if the candidate catches the error but sets no standard?
Score it as a partial pass and tell them so in the feedback. Catching a bad number is the individual contributor half of the job, and a manager who does only that half becomes the permanent last line of defence for a team that never learns where to look. The standard question separates the two: what changes in the template, the checklist or the definition of done so that the next draft needs no conversation at all.
Does the same document keep working once candidates share it?
For a while, and less well over time. A document circulating on a forum measures who read the forum. Write a second version in the same afternoon as the first and alternate between them. The error can move without the document changing much, since the same forecast with the growth rate wrong instead of the discount rate is a new exercise and costs an hour. Retire a version once a candidate arrives already knowing where to look.
Can the exercise replace the manager interview?
No. Twenty minutes on one document is one observation from one afternoon, and it says nothing about how this person runs a one-to-one, handles a missed deadline, or decides who gets the interesting work. It answers a single question the interview cannot answer honestly, which is whether the candidate reviews or describes reviewing. Keep the loop and give the exercise one slot inside it.
References
- 1. Revisiting Meta-Analytic Estimates of Validity in Personnel Selection: Addressing Systematic Overcorrection for Restriction of Range ✓ static1.squarespace.com Supports the claim that asking a candidate what they would do sits at .26, the same estimate as asking what they should do (96 knowledge-instruction studies, 22 behavioural), and that virtually all of those studies were concurrent.
- 2. Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of Artificial Intelligence on Knowledge Worker Productivity and Quality (Working Paper 24-013) ✓ mitsloan.mit.edu Supports the claim that a fluent wrong answer does not announce itself: on one task outside GPT-4's capability, 84.5% of the control group was correct against 60% and 70% in the two AI conditions, among the 373 of 758 BCG consultants assigned to that experiment.
- 3. Revisiting Meta-Analytic Estimates of Validity in Personnel Selection: Addressing Systematic Overcorrection for Restriction of Range ✓ gwern.net Supports the correction of the work sample validity figure from .54 to .33, based on Roth et al. (2005): 54 studies, N = 10,469, 53 of them concurrent.
- 4. The Structured Employment Interview: Narrative and Quantitative Review of the Research Literature ✓ doi.org Supports writing anchors for each of the three scores: across 19 past-behaviour interview studies, anchored rating scales showed validity of .35 against .26 and interrater reliability of .77 against .73.
- 5. 29 CFR Part 1607 - Uniform Guidelines on Employee Selection Procedures (1978), sections 1607.16(Q) and 1607.3(A) ✓ govinfo.gov Supports the claim that an informal manager-run exercise is a selection procedure: the Guidelines define one as any measure used as a basis for any employment decision, reaching "informal or casual interviews and unscored application forms".
5 sources, numbered by first appearance. Every one was opened and checked against the claim it carries. How Olive sources claims
General guidance for hiring teams. What works at one company and one volume may not transfer to yours.
Olive assesses how a person works with AI. It does not detect AI-written documents, and it never produces a score, a ranking, or a match percentage for a person. Candidates read the same report the employer reads.