Assessment design
A PRD Is Free to Draft. Its Refusals Are Not.
A model-drafted PRD still separates candidates on what it refuses. Four things do the work: an explicit non-goal, a metric written with its denominator, the alternative that was considered and lost, and one sentence naming what would make the plan the wrong bet. Clarity, structure and completeness stopped telling two submissions apart, because drafting made all three cheap. Grade the four refusals against a key you wrote first, then spend twenty minutes on the two hardest lines in the candidate's own document.
The takeCompleteness used to be read as a proxy for care. It is now a proxy for whichever assistant the candidate opened. A document that says yes to everything costs nothing to produce and nothing to believe, and a rubric that rewards it will keep returning shortlists of confident, agreeable specs. What is still expensive is the sentence somebody had to defend in a room: this is out of scope, this number is measured against that base, this option lost and here is why. Grade the sentences that cost something.
Where Olive fits
Open a role and see what the work shows
If you build this in-house, the expensive parts are the answer key and the evidence behind each judgment, per occupation. Olive ships twelve authored cases per occupation, product management among them, and returns six separately evidenced findings, each anchored to a moment in the session rather than to a number.
Assess your shortlistWhy does a polished PRD stop separating candidates?
Polish is the cheapest thing in the document now. A product requirements doc drafted with a model arrives structured, complete and readable on the first pass, and so does every other one in the pile. A rubric scoring clarity, structure and coverage therefore ranks the assistants that produced them.
What still varies between two specs is what a model has no way to supply.
In 2023, MIT economists Shakked Noy and Whitney Zhang ran a pre-registered experiment in which 444 college-educated professionals wrote occupation-specific press releases, short reports and analysis plans. The half given ChatGPT finished 10 minutes faster than a control group averaging 27 minutes, and graders scored their output 0.45 standard deviations higher 1.
The weakest writers gained the most.
A writing sample exists to spread candidates out, and the tool compresses the distribution it spreads them on.
The tasks were paid and online, done once, with no colleagues and no revision cycle, and these figures are the working paper's own. Read the finding as one about short professional writing, which sits nearer a PRD's first draft than the decisions inside it.
In 2024, researchers at the University of Reading put fully AI-written answers through their own examinations system: 33 fake student accounts, five undergraduate psychology modules, unedited GPT-4 output with no attempt to imitate a student.
Markers raised no concern of any kind about 94% of those submissions, and the AI grades averaged about half a classification boundary above real students 2.
The Reading study is coursework rather than a hiring take-home, and essay and short-answer psychology questions suit a language model unusually well. The part that transfers is narrower: graders marking under ordinary workload, with no instruction to look for AI, found nothing remarkable in a fluent, complete submission.
The validity folklore a writing sample borrows from work samples was overstated too. Sackett, Zhang, Berry and Lievens put work sample validity at .33 in their 2022 re-analysis, against a .54 that traces back to a 1974 narrative review 3.
Of the 54 studies behind the .33, 53 tested people already doing the job, and the paper reports nothing about take-home assignments.
So the useful question is not who typed it. If the scores on your AI exercise line up with writing quality, the rubric is measuring polish, and a better rubric is the fix rather than a more suspicious reader.
What survives when the prose costs nothing?
A generated draft leaves four decisions open for a person to close: an explicit non-goal, a metric written with its denominator, the alternative that was considered and lost, and one sentence naming what would make this the wrong thing to build. Each one is a refusal. Each closes a door, which is why a draft optimizing for helpfulness fills those slots with lines that close nothing.
Start with the non-goal, because it is the easiest to check. Ask what the document says it will not do, then ask whether a reasonable person wanted that thing.
"No bulk editing in v1" is a decision somebody lost. "Not solving every reporting need" is filler the draft supplied.
The same line belongs in the assignment itself, where naming what is explicitly out of scope is one of the four things that make two submissions comparable.
A metric arrives with its denominator or it is decoration. "Activation up 20 percent" is a sentence any draft produces. "Activation, defined as accounts completing a first import within seven days, currently 31 of every 100 new accounts a week" names what moves and what it moves against, and both halves are checkable.
The losing option is the part nobody generates. A model will happily list alternatives considered. It will not tell you which one the team nearly shipped, who argued for it, and what ended the argument.
Then the sentence most specs never contain: what would make this the wrong bet.
A strong version names a number and a threshold. If fewer than one in five trial accounts imports anything at all, the integration was never the bottleneck and this roadmap is wrong.
One column is worth adding beyond the four, and it is the hardest to borrow: which decision was contested, and who had to be convinced. A candidate who wrote the spec alone can still answer that about the reviewer they expected to object.
Grade the four refusals against a key you wrote first
Write the key before you read a single submission. Do the assignment yourself, inside the time budget you set, and record your own answer in each of the four columns. Then mark each column present, partly present or absent, with a written anchor describing what each level looks like.
Two readers should be able to fill the grid separately and land in the same place.
The step has a general form: write the answer key before you send the assignment.
The gain sits in the combining rule, the step where marks become a decision. Kuncel, Klieger, Connelly and Ones found an average correlation with job performance of .44 when applicant data were combined by a formula, against .28 when the same kinds of data were combined by expert judgment 4.
The comparison rests on nine studies, and a formula there can be as plain as adding unit-weighted marks on a scorecard.
Nothing in the comparison argues for a number standing for a person. What it argues for is applying the same rule to the fourth submission you applied to the first.
Anchors are what make the grid repeatable.
In Levashina and colleagues' 2014 review, Taylor and Small's meta-analysis of 19 past-behavior interview studies found that interviews using anchored rating scales reached higher criterion validity, .35 against .26, and slightly higher interrater reliability, .77 against .73 5.
Those studies were interviews rather than take-homes, and the reliability gain is small.
The same review reports something more useful for a scorecard: Reilly and colleagues found in 2006 that anchoring every point of a five-point scale produced ratings more resistant to disability bias than anchoring only the top and the bottom.
Write an anchor for every level of all four columns, the middle level included, since a scale anchored only at its ends is not what was measured.
Grading a take-home when every submission comes back polished is the same problem one level up, and the same move solves it: shift the graded thing from the artifact to the decisions behind it.
Ask about the two lines they would least like to defend
Book twenty minutes and spend all of it inside their document. Pick the two lines that cost the most to write, usually the non-goal and the metric. Ask what the denominator is, who wanted the excluded thing, and what ships first if the timeline halves.
Every candidate who got that brief gets the same three questions, in the same order. An account of a decision is much harder to borrow than a document is.
Then write down what the answer named, rather than how it landed.
Shanshi Liu and colleagues text-mined the post-interview notes on 7,650 people hired at a large Chinese internet technology company. The number of job-related capabilities an interviewer named in the notes tracked later performance and promotions, and tracked turnover in the other direction 6.
Match between the notes and the job analysis corresponds to around 2 percent of a performance rating per standard deviation, which is small. Everybody in that sample had passed the interview, so it says nothing about the people who were turned down.
What the finding supports is writing "named the denominator without being asked" in place of "strong product sense".
The walkthrough is a selection procedure whatever the schedule calls it. The US federal Uniform Guidelines (1978) define one as any measure or procedure used as a basis for an employment decision, covering the full range from paper-and-pencil tests through "informal or casual interviews and unscored application forms" 7.
So keep the notes, keep the order, and run it with every finalist rather than only with the submission that read too well.
Where a refusal-first reading gets it wrong
Refusals are learnable in an afternoon, which is the strongest case against reading for them. A candidate who has studied one good template will write a non-goals section, name a losing alternative and add a falsifier line, and all four columns will fill without any of the judgment they are meant to stand for.
The grid can be beaten by format.
The twenty minutes are what a format cannot survive. But the honest version is that a better exercise raises the cost of faking rather than removing it.
One structural fix costs nothing. Plant the problem in the material instead of waiting for the candidate to volunteer a refusal: hand over a research summary with one number in it that contradicts the brief, and see whether the metrics section notices.
Seeding a repo so the exercise tests checking, not typing is the engineering version of the same move.
Read the PRD in front of you once more, and read only the sentences that close a door. Count them. If the count is zero, you are holding a first draft the model finished and nobody argued with. The twenty minutes will be very short.
Common questions
Should you still ask for a PRD as a writing sample?
Yes, with the brief changed. Ask for a spec on a problem your team actually owns, cap it at an hour, and say plainly that AI is allowed. Then require four short sections: what this will not do, the metric with its base, the option considered and rejected, and what would make the whole thing the wrong bet. Those four are what gets graded. The prose is not. A brief still asking for a complete, polished document is measuring the assistant, and it returns a pile of submissions that are hard to tell apart.
What does a real non-goal look like in a PRD?
Something a reasonable person wanted and did not get. "No bulk editing in the first release, because the three accounts asking for it are on the enterprise plan and can wait a quarter" names the loser and the reason. "Not solving every reporting need" is decoration. The test is whether the excluded thing has an owner somewhere. If nobody would have argued for it, excluding it cost the writer nothing, and the line proves nothing about the writer.
Can you tell whether the candidate or the model wrote the spec?
No, and a round built on that question wastes itself. In a 2024 study run through a UK university's real examinations system, 94% of fully AI-written submissions drew no concern of any kind from markers, and those submissions on average graded above real students. The submitters made no attempt to imitate a student, which makes the 94% a floor rather than a realistic picture. Ask about the decisions inside the document instead. Write down what each answer names, not how quickly it arrives.
How much of the twenty minutes should be about their own document?
All of it. A general question about how somebody uses AI produces a rehearsed answer any candidate can give. A question about line 14 of their own spec asks for the reasoning that put it there. Pick the two hardest lines before the call, ask the same two of every finalist, and write down which job-relevant capability the answer actually named. Keep the record: under the US Uniform Guidelines, a conversation used as a basis for the decision is a selection procedure.
What if the candidate cannot share a PRD from their last job?
Hand them a problem instead of asking for an artifact. Give a short brief on something your own team shipped, plus the research it rested on, and ask for the four columns rather than a finished document. That removes the confidentiality problem and the polish problem at once, since nobody arrives with a prepared spec for a prompt they first saw an hour ago. It also makes two submissions comparable, which a candidate-supplied sample never is.
References
- 1. Experimental Evidence on the Productivity Effects of Generative Artificial Intelligence (working paper, not peer reviewed) ✓ economics.mit.edu Supports the claim that drafting compresses the writing-quality distribution a writing sample exists to spread out: 444 college-educated professionals on occupation-specific writing tasks, the ChatGPT half finishing 10 minutes faster than a control averaging 27 minutes and graded 0.45 standard deviations higher, with the largest gains going to the weakest writers.
- 2. A real-world test of artificial intelligence infiltration of a university examinations system: A "Turing Test" case study ✓ journals.plos.org Supports the claim that a fluent, complete written submission drew no concern from graders marking under ordinary workload: 33 fake student accounts across five undergraduate psychology modules at the University of Reading, 94% of the AI submissions undetected, and grades averaging about half a classification boundary above real students. Detection here means a marker raising any concern at all, and the output was unedited GPT-4 with no attempt to imitate a student.
- 3. Revisiting Meta-Analytic Estimates of Validity in Personnel Selection: Addressing Systematic Overcorrection for Restriction of Range ✓ gwern.net Supports the correction of the .54 work sample figure to .33, and the composition of the evidence behind it: 54 studies from Roth et al. (2005), 53 of them concurrent, with the .54 traceable to a 1974 narrative review. Cited here to size what a work sample predicts, not to rank it against other methods.
- 4. Mechanical Versus Clinical Data Combination in Selection and Admissions Decisions: A Meta-Analysis ✓ gwern.net Supports the claim that a written combining rule beats a holistic impression: average correlation with job performance of .44 for mechanical combination against .28 for clinical combination, on nine studies for the job performance criterion. Used for the combining rule only, never for a composite number standing for a candidate.
- 5. The Structured Employment Interview: Narrative and Quantitative Review of the Research Literature ✓ doi.org Supports the anchored-rating-scale figures quoted from Taylor and Small's meta-analysis of 19 past-behavior interview studies (validity .35 against .26, interrater reliability .77 against .73) and the review's separate report that anchoring every point of a five-point scale produced ratings more resistant to disability bias than anchoring only the endpoints.
- 6. Predictive Validity of Interviewer Post-interview Notes on Candidates' Job Outcomes: Evidence Using Text Data From a Leading Chinese IT Company ✓ frontiersin.org Supports writing down which job-relevant capability an answer named: on 7,650 hired candidates, the number of job-related capabilities interviewers named in their notes related positively to later performance and promotions and negatively to turnover, at roughly 2 percent of a performance rating per standard deviation of match to the job analysis. Everyone observed had passed the interview.
- 7. 29 CFR Part 1607 - Uniform Guidelines on Employee Selection Procedures (1978), sections 1607.16(Q) and 1607.3(A) ✓ govinfo.gov Supports the claim that a twenty-minute walkthrough is a selection procedure: the Guidelines define one as any measure, combination of measures, or procedure used as a basis for any employment decision, covering the full range of techniques through informal or casual interviews and unscored application forms.
7 sources, numbered by first appearance. Every one was opened and checked against the claim it carries. How Olive sources claims
General guidance for hiring teams. What works at one company and one volume may not transfer to yours.
Olive assesses how a person works with AI. It does not detect AI-written documents, and it never produces a score, a ranking, or a match percentage for a person. Candidates read the same report the employer reads.