Collect the Scores Before the Debrief, Not During It

Why the consensus meeting undoes the scorecards

Because a discussion aimed at agreement mixes the ratings back into a single impression, and the impression belongs to whoever argues best. Three rounds of scored interviews produce several independent readings. The standard debrief then puts them in a room and asks for consensus, and what comes out is one reading, usually the one held by the most senior or most fluent person present.

The combination step is where the evidence is either kept or lost. Combining predictors mechanically, with sensible weights, is what produces the composite figures the selection literature reports, reaching about .61 from two or three different kinds of evidence ¹. That figure assumes the predictors combined are genuinely different from one another, so it is not what a team gets from stacking several unscored conversations, which mostly measure the same thing more than once.

That leaves the debrief one job it can do well and one it cannot. It can surface where the independent readings disagree and put the evidence for each side on the table, which is genuinely valuable and happens nowhere else. It cannot manufacture a better reading by averaging conviction, and a meeting that opens with what did everyone think has already started doing the second thing.

The cost is invisible from inside, which is the awkward part. A panel that talks itself into agreement feels more rigorous than one that reports a split, so the process that destroyed the most information is the one that felt best in the room.

Run the debrief in this order

Ratings in and locked before the invite opens, the spread read aloud, discussion confined to the competencies where ratings disagreed, then the rule. That order is the whole intervention. It costs nothing, needs no software beyond whatever holds the scorecards, and it is the part of structured hiring most often skipped by teams who believe they have adopted the method.

1. Locked before the meeting. Every interviewer submits ratings and the evidence behind them by a stated deadline. Nobody sees another sheet before submitting, and nobody edits after. 2. Spread first. Read out where the panel agreed and where it split, competency by competency, before any argument. The split is the agenda. 3. Only the divergences. Skip the competencies everyone rated the same. They are settled, and rehearsing them is how a meeting talks itself into a mood. 4. Evidence, not seniority. A rating changes when someone quotes something from the interview the other person had not weighed. Edits get recorded with the reason. 5. The rule, last. Apply the decision rule written before the loop opened, and note anywhere the rule did not fit. That note is next quarter's fix.

Two logistics matter more than they look. Set the submission deadline hours before the meeting rather than minutes, because a scorecard written in the corridor is a memory of a conversation the panel is about to have. And give the least senior interviewer the first word on each divergence, which costs nothing and removes the anchor everyone else would otherwise be arguing against.

Where the panel is judging something it never agreed on in advance, the debrief will split on definitions rather than on the candidate. That is worth heading off before the loop starts rather than in the room (get the panel judging AI use the same way).

What counts as a reason to change a rating?

Something quoted from the interview that the rater did not have or did not weigh. That is the entire list. Not seniority, not enthusiasm, not the fact that four other people rated it differently, and not a reconstruction of what the candidate probably meant. If the reason cannot be traced to a specific thing the candidate said or did, the rating stands and the disagreement gets recorded as a disagreement.

The rule is that narrow because panels are good at building coherent accounts out of thin material. In a controlled test of unstructured interviewing, students predicting a classmate's semester grades did worse after conducting the interview, correlating .31 with the outcome against .65 from prior grades alone, and in a further study most participants preferred interviewing someone answering questions at random to not interviewing at all ². Undergraduates predicting grades is a laboratory analogue rather than a selection study, and the sample is small. What travels is the mechanism: people make sense of whatever is put in front of them, including nothing.

The practical form in the room is one question, asked every time somebody offers a conclusion: what did they say. It is not a challenge and it stops being awkward after two debriefs. Where the answer is a quote, the panel now has evidence it can weigh. Where the answer is a restatement of the conclusion, everyone hears that at once and nobody has to be the person who says so.

Interviewers who do not do the work themselves have the hardest time here, because the evidence they are being asked to quote is unfamiliar (judging AI-assisted work without using AI). The fix is the same as everywhere else in this process: an anchored scale written before the loop, with a worked example of what each level sounds like.

Write the decision rule before the loop opens

Decide in advance what pattern of ratings produces an offer, and write it down while nobody knows who the candidates will be. Clears the bar on all four competencies, with no dissent on the one where being wrong is expensive, is a rule. The strongest overall impression is not a rule. It is the debrief deciding what it wants and then finding a reason for it.

A workable rule names three things: which competencies are mandatory rather than tradeable, what a single low rating on a mandatory one does regardless of the rest, and who decides when the rule does not fit the case in front of them. That last one is not a loophole while it is written down and used rarely. It becomes one when it goes unnamed, because then it defaults to whoever is most senior in the room.

The standard definition of a structured interview already contains the seed of this: the same questions in the same order, a common rating scale, and interviewers agreeing in advance on what an acceptable answer looks like ³. Agreeing in advance is the habit. The decision rule is that habit applied one level up, to the pattern of ratings rather than to a single answer.

Write the rule at the same time as the scorecard, because a rule invented after the ratings exist is fitted to them. The version to avoid is a numeric average across competencies, which lets a strong showing on something easy compensate for a gap on something the job requires, and which recreates exactly the problem scorecards were there to solve (set a bar rather than an ordering). None of this holds if the interviews were not comparable in the first place, so if only some of the parts of a structured interview are in place, fix that before redesigning the meeting (what actually makes an interview structured).

See how it works >