Run the Take-Home Walkthrough on Their Decisions, Not Yours

Which four questions carry the round?

Ask everyone who got the brief the same four, in the same order: which decision the deliverable rests on, what that decision was checked against, what the second-best option was and why it lost, and what could not be settled in the time available. Every one of them points at the submission on the screen.

Fixing the questions and the rating scale is where the measured gain sits. In a 2022 re-estimate of the selection literature, Sackett, Zhang, Berry and Lievens put structured interviews at .42 against .19 for unstructured ones, as corrected correlations with supervisor ratings of job performance ¹. Structured there is a research coding of interview format: same questions, rated on a common scale.

A walkthrough left to run itself is the unstructured kind. The reviewer opens with whatever struck them about the file, the candidate responds, and twenty minutes later somebody writes a rating in a box.

Interviewers make sense of almost anything put in front of them. In a ranking task, 96 of 169 participants put an interview in which the interviewee answered at random ahead of no interview at all ².

In the same paper's controlled test, undergraduates predicted classmates' grades. Predictions made after an unstructured interview tracked the real grades at .31, against .65 for the academic record the predictors already held.

The sample is undergraduates in a lab, so the size of that gap doesn't travel. But the mechanism does. Weak information crowds out strong information, and a submission you have already read closely is strong information.

The other failure is an interrogation about the assistant: whether it was used, how much, which model. Those minutes buy nothing scorable, and whether to disqualify a candidate for using AI on a take-home is a policy question that belongs in the brief, settled before anybody submits.

Each question does a different job, and dropping one drops that job.

• Which decision is the whole deliverable resting on? Locates the load-bearing choice. An answer that can't name one is thin on this question.
• What did you check that against, and what did the check change? The second half is the question. A check that changed nothing should still be able to say what it confirmed.
• What was the second-best option, and why did it lose? Reaches the framing that happened before any generating did. Discarded options are the cheapest evidence of it.
• What could you not settle in the time? Invites a stated limit. Naming where the work is thin is a different act from believing it is finished.

Write the anchors before the first submission arrives

An anchor is a written description of what a strong, an adequate and a thin answer contains for one question on one brief. Draft it before any submission lands. Without anchors, four interviewers score the same twenty minutes on four private scales. Anchors are also the half of structure a team can skip without noticing: the questions get fixed and the scoring stays in somebody's head.

Published studies leave parts of the scoring side out too. Levashina, Hartwell, Morgeson and Campion content-analyzed 104 interviews reported in studies published between 1997 and 2010. On average, 5.74 of the 15 structure components were in use ³. Content and evaluation components came out roughly equal in number, and the two that record and combine were both rare: detailed notes in 19 percent of them, and a statistical rule for combining ratings in 11 percent.

Those figures are a ceiling. They count what researchers reported building into published studies, and employers are unlikely to run more structure than that.

Anchored scales buy something modest and real. The same review reports Taylor and Small's meta-analysis of 19 past-behavior interview studies. Interviews using anchored rating scales reached .35 against .26 without them, and interrater reliability ran .77 against .73 ³. The validity gain is the larger of the two, and the comparison runs across studies, with no test of one interview run both ways.

Anchor every point of the scale, not only the top and the bottom. Inside the same review, Reilly and colleagues found that fully anchored ratings held up better against disability bias ³.

Take a brief asking for a one-page recommendation on which of two vendors to renew. The anchor for the checking question, written the week the brief goes out:

• Strong. Names the artifact opened (the contract, the incident log, the invoice history), names what it said, and names what changed in the recommendation as a result, or what the check confirmed.
• Adequate. Names where the figure would be checked and says plainly that the check was not run in the time.
• Thin. Asserts that everything was verified, with no object attached to the verb.
• Not observed. The question was never reached.

Four lines like those are written once per brief, and the work overlaps with setting the standard for grading the submission. If you already write the answer key before you send the assignment, the walkthrough anchors are the second column of that document.

How do you score four answers in twenty minutes?

One mark per question, written as soon as that question and its follow-ups close. Four marks, never one impression converted into a rating at the end. Combine them with a rule fixed in advance. Beside each mark, write what the candidate said, in their own words.

Combining by rule has the better evidence behind it. Kuncel, Klieger, Connelly and Ones pooled selection and admissions studies. They found applicant data combined by a formula correlated .44 with job performance, against .28 when the same kinds of data were combined by expert judgment ⁴. A formula here can be as plain as adding the four marks with equal weights, which the paper counts as mechanical.

Nine studies sit behind the job-performance half of that comparison, so take the direction and leave the size of the gap unsettled.

The evidence on notes is about what goes into them. Researchers text-mined the post-interview notes written on 7,650 candidates hired at one large Chinese internet company ⁵. The more job-related capabilities a note named, the better that hire's later performance ratings and promotions and the lower their turnover. The performance effect was small, roughly 2 percent per standard deviation of the note's match to the job analysis.

But everybody in that sample had passed the interview, so nothing in it speaks for the people who were rejected.

Back in the vendor brief, say a note records that the candidate traced the renewal figure to the contract and found a 12-month term where the vendor's deck said 24. That tells a later reader something. A rating of strong verification instinct tells them nothing.

Keep the not observed row on the scale, and use it. A round that never reached the third question produces no mark for it, and a scorecard forcing a rating there manufactures evidence.

Then count how often that row fires. A question unreached in half the rounds is a scheduling problem.

Who runs the walkthrough, and what do they know going in?

Somebody other than the person who graded the submission, holding the brief and the anchors, with the grade withheld. The reviewer's read is the thing this round exists to test. A reader who has already written that verification was weak is likely to hear weak verification in twenty minutes of ordinary conversation. Two marks that disagree are the output worth having.

The closest evidence comes from a study of interviews. In structured mock interviews with 189 accounting students, interviewers formed an impression during the small talk, before any structured question. That impression correlated .42 with the same interviewer's later structured score ⁶. When a different interviewer supplied the structured score, the correlation fell to .25.

Same candidate, same questions, a different distance from the first impression.

But the paper's own conclusion is narrower than the headline. The effect ran through rated competence, with liking and similarity dropping out across interviewers, and it survived when the structured score came from an interviewer who never did the small talk. Read it as a reason to keep two judgments apart, never as evidence that anybody decides in three minutes.

Fix the four questions and let the follow-ups differ. A fixed question can be rehearsed for, and a second or third push on the answer is much harder to prepare, which is why three questions and nine follow-ups beats twelve questions asked once each.

Push for the specific: which page, which run, which number.

When two people scoring the same walkthrough answer split, start from what the candidate actually did. One scoring it as good delegation and the other as failing to check the model can be describing the same forty seconds. Two interviewers who split on one moment should fix the anchor before anybody argues about the rating.

What a narrated decision cannot show, and what the 1978 rule reaches

An account of a decision is a reconstruction, produced after the fact by the person with the most at stake in it. A candidate who narrates well can describe a loose check as a rigorous one. A candidate who works carefully and explains less fluently can score thin on all four questions. Twenty minutes of talk doesn't separate those two, and no anchor written in advance fixes it.

In the United States, the federal rule that reaches this round dates from 1978. Under the Uniform Guidelines, any measure, combination of measures, or procedure used as a basis for any employment decision is a selection procedure ⁷. Assessment techniques of every kind sit inside it, informal or casual interviews and unscored application forms included.

A twenty-minute conversation with no scorecard, used toward a hiring decision, sits inside that definition.

The validation burden attaches where adverse impact appears: a selection procedure showing adverse impact is treated as discriminatory unless it has been validated under the Guidelines, or the provisions of their section 6 are satisfied ⁷. Confirm the current position with employment counsel; nothing here is legal advice.

So the round produces evidence rather than a verdict.

Four marks, four answers in the candidate's own words, one line on what the follow-ups changed, and an honest blank where the third question was never reached.

That record is a second document to set beside the submission.

Read the two together before anybody argues about the hire. If they agree everywhere, the round was cheap insurance. If they disagree once, the twenty minutes paid for themselves.

See how it works >