A Review Round Tests What to Check, Not Whether They Can Build

What does reviewing measure that authoring never did?

Reviewing measures three decisions an authoring round never asked for: what to check first, what to take on faith, and when to stop looking. A candidate writing code from a blank file makes none of those calls, because nothing arrives already finished. Reading a change that already works and deciding which parts of it to distrust is a separate act, with a separate way of going wrong.

The person who produced the output is poorly placed to check it. In a Stanford user study, people wrote security-relevant functions. Those given an AI assistant wrote significantly less secure code than those without one, and were more likely to believe their code was secure ¹.

Confidence moved opposite to quality.

The same blind spot shows up away from code. A Harvard Business School field experiment split 758 consultants across two experiments, one of which used a task chosen because it sat outside what the model could do.

The control group reached the right answer 84.5% of the time, against 60% and 70% in the two groups using GPT-4 ². The experiment ran a 2023 model on a single task, and the line it fell outside of has moved since.

What travels is not the size of the drop: nobody in the room could see which side of the line the task was on.

So the round worth running hands a person work that looks finished and asks what is wrong with it. Your algorithm screen no longer does that, because it measures preparation rather than programming, and a blank-file exercise never did.

What an exercise is about can matter more than what shape it takes, if a finding about knowledge tests carries over to a review round. Sackett, Zhang, Berry and Lievens lean on a job knowledge meta-analysis: all 164 studies together produced a mean observed validity of .22, while the 59 studies using tests built for the job in question produced .31 ⁶.

The .31 comes from a subset of those same 164 studies, not a controlled comparison.

The US federal standard says something close in 1978 language. Under the Uniform Guidelines, content validity rests on showing that the behavior demonstrated in a selection procedure is a representative sample of the behavior of the job. The regulation then goes further: the closer the content and context of the procedure come to the work, the stronger that basis is ³.

If most of the changes your team reads were drafted by a model, a round about reading one sits close to the work.

Why does an unauthored diff grade nothing?

Because two reviewers of a diff nobody planted will find different things and both be right. A pull request lifted out of your repository carries an unknown number of problems, of unknown severity, and no list exists of what a good reviewer should have caught. The candidate's result then depends on which grader read it, which is a fact about the grader.

Interviewer disagreement is already common, before anything is unauthored. Ashby looked at interviews in its own customer data where more than one interviewer had scored the same candidate, and found around 38% of scorecard pairs carried at least one point of difference ⁴.

Between a 2 and a 3 is where nearly half of those one-point gaps fell, which is the line between yes and no on Ashby's four-point scale.

Disagreement is not error. Nothing in that data says which interviewer was right.

But an unauthored diff makes the disagreement structural, because the two graders are not working from the same list.

Write the key before the first candidate sees the diff: name each planted problem, name the class it belongs to, and name what counts as finding it. A reviewer who says the retry loop will hang has found the missing timeout. A reviewer who says the change feels fragile has not.

The same discipline sits behind an answer key written before the assignment goes out, with one thing made easier here. You put the answers in yourself, so there is no argument later about what they were.

Then check that the key works. Two graders should fill it separately and land in the same place. What counts as agreement once chance is discounted is a stricter number than eight of ten sounds like, because some of those eight matches would have landed by luck.

Which problems go in the diff?

Plant two or three problems, of different kinds, in a change that plausibly works. One wrong assumption the code never states out loud. One test that agrees with the bug instead of catching it. One failure path nobody wrote: the network call with no timeout, the empty result treated as a zero, the retry that never gives up.

Different kinds matter more than the count. A reviewer who finds three problems of one kind has shown you one habit, three times.

Take the diff from your own codebase and break it on purpose. Rewind a change your team merged last quarter to what it looked like before review, then plant the problems. It reads as real because it is. Preparing the repository this way is most of the work in an AI-open exercise, and it is done once for a whole hiring season.

Order is the part of the answer worth scoring hardest.

Ask what the candidate checked first and why. A reviewer who opens the test file before reading the diff has told you something no count of bugs found will.

Give them the diff, an hour, and permission to run the code. Reading and pronouncing is cheap. Writing a failing case, watching it pass and saying why passing is the wrong outcome is the work, and it is what a scorable write-up is made of.

The strongest objection is that a miss is ambiguous. Someone who does not know your domain can walk past the wrong-assumption bug for reasons that say nothing about how carefully they check anything. A missed planted error can be a knowledge gap rather than a habit.

Plant one problem that needs no domain knowledge to see.

Keep one round where the candidate writes code

A review round cannot tell you whether someone can build. Reading a change for what is wrong in it and producing a change that works are different acts. Evidence that a candidate does the first well is not evidence about the second. Keep one round somewhere in the loop that asks for something built, even something small, even with the model open.

One selection paper puts a number on combining kinds of evidence. Sackett, Zhang, Berry and Lievens estimate that a composite of several predictors combined by a rule reaches about .61, against .42 for the strongest single method in their revised table ⁵.

That .61 assumes the predictors are scored and then combined mechanically, with sensible weights. Four unscored conversations averaged in a debrief are not that.

The second round can be small. Twenty minutes on a bug fix in the same repository is enough to see whether the person can move code at all. Leave the model open and ask for a short write-up.

So the loop that works is two-sided: one round where the candidate makes something, one round where the candidate takes something apart. Choosing between live coding, a take-home and an AI-allowed work sample is a question about the first half, and it leaves the second half unanswered.

The 1978 paragraph that supports this round also limits it. A content strategy, the Guidelines say, cannot validate a procedure that purports to measure traits or constructs, and the list they give includes judgment ³.

So keep judgment off the scorecard. Write down what the reviewer found, in what order, and what they left alone. Whether a content strategy covers your own round is a question for employment counsel.

See what gets scored >