Send the Questions in Advance and Change What You Score

What does withholding the questions actually measure?

Who guessed right. That is the whole of it once preparation is universal, and guessing is not randomly distributed: it tracks coaching, alumni networks, friends inside the company, and whoever has read the most forum threads about your process. Sending the set does not remove preparation from the round. It removes the part of preparation that rewarded access, and leaves the part that rewards having something to say.

It also gives a candidate something almost nothing else in a hiring process gives them, which is influence over how the encounter goes. A review of applicants' procedural-fairness perceptions of algorithmic recruitment tools, drawing on seven studies with more than 1,300 participants between them, found overall fairness perceptions mixed but perceptions of behavioral control and social presence mostly negative: people felt unable to affect the outcome and felt the human element was missing ¹. That is a narrative review of scenario-based studies, several with student or panel samples rather than real applicants, and its authors are affiliated with the assessment industry, so it supports a direction and not a magnitude. The direction is the relevant part, and sending the questions is one of the few cheap moves that pushes against it.

There is a narrower legal reason to be able to say what a surprise measures. The ADA's implementing regulation in the United States, 29 CFR 1630.11, requires a covered employer to select and administer employment tests so that the results reflect the skill the test purports to measure rather than an impaired sensory, manual or speaking skill, except where those skills are the point of the test ². That rule is narrower than a general accessibility duty and it mandates no particular format. But a round whose difficulty comes from being sprung on people is a round that will be hard to explain in those terms.

What should you send, and what should you keep back?

Send the questions and the rating dimensions. Keep back the material. The invitation carries the three or four questions, which competency each maps to, the format and the length, and whether AI tools are welcome at any stage. What does not travel is the case data, the artifact you will hand them, and anything carrying a planted error. The first list is fairness; the second is what keeps the round live.

A working invitation is about a paragraph. Something like: this round runs 45 minutes and covers three competencies. Here are the three questions. Bring a recent piece of your own work you would be happy to walk a stranger through. There will be follow-up questions built on what you say, and part of the round uses material you will see for the first time in the session. Use whatever tools you like to prepare.

That last clause matters more than it looks. A rule against preparing with a model is unenforceable, so it functions as a penalty on candidates who follow rules, which is close to the opposite of what any hiring team wants to select for. Whether a structured interview still separates candidates once everyone has been AI-coached is the same question asked from the other end.

Change what you score when the questions go out

Three things advance notice cannot manufacture, and they become the scorecard. What the candidate brought, since you did not know what they had. What they would do differently now, which requires a real opinion about work they actually did. And how far the follow-ups go, since the third question in a chain gets built from the second answer and did not exist before the candidate spoke.

The first-answer quality line has to come off the form. Every candidate's first answer improves when they have the question in advance, so a line that rewards it now measures whether the email was read. What replaces it is a line for whether the answer yielded something you could go and verify: a number with a baseline, a named artifact, a person who disagreed, a mistake somebody found later.

Then pair the round with a different kind of evidence rather than a second round of the same kind. In the applied follow-up to the 2022 selection re-analysis, combining predictors reaches a composite of about .61, and removing cognitive ability from that composite costs .05 ³. Composite validity assumes predictors are combined mechanically with sensible weights, so it is not what a team gets from stacking four unscored conversations, which mostly measure the same thing twice. The usable version: one prepared behavioral round plus one work-shaped exercise beats three conversations, and it beats them on the same clock.

Does the round lose anything worth keeping?

One thing, and it matters for a small number of roles. You lose the chance to watch somebody think from cold. If the job genuinely involves reacting in public without preparation, a live support desk, a trading floor, an on-air segment, keep one unseen element and say in the invitation that it is coming. For everything else, the cold read was measuring a trait the role does not use.

It is worth being honest that the format decides how much of the judgment is about presentation. A meta-analysis of 21 articles yielding 221 effects found non-autistic observers formed less favourable first impressions of autistic people overall, and the size of that penalty tracked the medium: 0.63 for audio-video and 0.56 for video-only, against 0.22 for transcript-only with a confidence interval crossing zero ⁴. Those are thin-slice lab ratings of first impressions and not one included study measured a hiring decision, and the transcript subgroup is the smallest, so this is not proof that written work is neutral. It is a reason to notice that the more of the round rides on live performance, the more of the rating rides on how somebody comes across.

The practical shape of the loss is smaller than the worry. A prepared candidate still has to answer a follow-up nobody wrote down, still has to react to material they have not seen, and still has to have done something worth describing. What goes away is the interviewer's private sense that they caught somebody unready, which was never evidence. How to redesign an interview so AI assistance becomes signal instead of a problem is the fuller rebuild, and whether to keep rotating the question bank is the decision that usually comes next.

See how it works >