Calibrate on Excerpts and a Disagreement Log, Not on What a 3 Means

What actually happens in a calibration session?

Four things, in this order: everyone reads or watches the same real answer, everyone writes independently, everyone reveals at once, and the competency that split the room goes into a log. The order carries more weight than the content. Reveal before writing and you have measured deference. Skip the log and next quarter's session starts from nothing, which is how a panel ends up having the same argument twice.

The reason to bother is that panels disagree exactly where it costs something. Across interviews with multiple interviewers in Ashby's benchmark, around 38% of scorecard pairs include at least one point difference between interviewers, and nearly half of those one-point differences fall between 2 and 3, crossing the yes and no threshold on a 1 to 4 scale ¹. A pair is two scorecards compared against each other, so 38% is a rate over pairs, and the report gives no equivalent rate over interviews or candidates. The dataset carries no outcome measure, so nothing in it says which interviewer was right, and Ashby's customers skew to venture-backed technology employers. What it establishes is that instability concentrates at the decision boundary.

Confidence, meanwhile, does not track agreement. A review of why employers keep trusting intuition in selection reports that the interrater reliability of the traditional unstructured interview is so low that, even with a perfectly reliable and valid criterion, interview-based judgments could never account for more than 10% of the variance in job performance ². That is a ceiling implied by how little two interviewers agree, not a measured validity, and the meta-analysis behind it dates from 1995. The same review reports a survey of 201 HR executives who rated the unstructured interview more effective than any paper-and-pencil assessment procedure. That survey was published in 1996, so it records what practitioners believed then.

So the meeting has a specific subject. Two interviewers who split 4 against 3 on a candidate they both liked have cost you nothing. Two who split 3 against 2 have landed on opposite sides of the yes and no line, and neither number carries any record of what put it there.

Make the excerpt the unit of agreement

Ask for the sentence, not the number. Each interviewer writes the specific thing the candidate said or did and what it evidences, before anyone says a rating out loud. Two write-ups that both land on 4 can rest on completely different moments, and a rating on its own hides that. The excerpt is inspectable by a third person; the number is a summary of a judgment nobody else can reopen.

Agreeing in advance belongs to the definition of a structured interview. The US Office of Personnel Management's practical guide defines one by three properties: every candidate is asked the same questions in the same order, every candidate is evaluated on a common rating scale, and the interviewers agree in advance on what an acceptable answer looks like ³. Teams ship the first two in a template and leave the third to a meeting. That guide is US federal hiring practice from 2008 and binds no private employer, and it predates every AI question by more than a decade, but the third property is the one that needs real material to settle. An acceptable answer described in the abstract is a sentence everyone nods at and nobody can apply.

Written anchors on the scale are worth having, and they solve a different problem than the one in front of you. An anchor tells an interviewer what a 3 is supposed to mean. It does not tell a second reader which sentence the 3 rests on, so a session spent entirely on scale definitions can end with everyone using the numbers the same way and still produce two write-ups nobody can reconcile. Settle the scale once in a document and spend the room's hour on evidence. Where the material is AI-assisted work and two reviewers keep landing in different places, writing a rubric two reviewers score the same way is the narrower problem underneath this one.

How often should you run one?

Three triggers, not a calendar: at the start of a req when the bar is first written, after the first four or five candidates when the pool's real shape is visible, and immediately after any change to the role, the assignment, or the tools candidates are allowed to use. A model release can move the pass rate on a work sample without anyone touching the rubric.

That third trigger is the one no incumbent agenda contains. If the take-home suddenly produces a run of strong submissions, the first hypothesis worth testing is that the exercise got easier, and the panel needs half an hour to re-agree what a strong submission now demonstrates. Pass rates jumping after a model release is the situation version of that, and it arrives with no notice. A change in the tools also moves what the panel argues about: when answers arrive fluent and tidy, two interviewers can accept every fact in a response and still split on how much of the polish to credit, which is a disagreement no scale definition settles.

The first two triggers are cheap because they sit on the calendar already. The kickoff is where the competencies get written, so it is the natural place to score one example against them while the definitions are still soft enough to change. Four or five candidates in, the disagreements are about people the panel has actually met, which is the difference between a productive argument and a philosophical one.

Keep a disagreement log between sessions

One row per split: the competency, the two readings, and what the panel decided the evidence had to show. The log is the whole difference between a ritual and a program, because it turns the next session into thirty targeted minutes on the two competencies that actually split the room. It also survives the people. An interviewer who joins in March inherits arguments the panel already settled.

The log also settles how a debrief ends. A meta-analysis of how selection evidence gets combined classifies group consensus meetings as a holistic method, alongside individual expert judgment: what makes a method holistic is that data are combined by judgment or intuition rather than by a rule applied the same way for each decision ⁴. A debrief that ends in a shared feeling is that method, whatever the scorecards said on the way in. The authors derive a 25% reduction in correct hiring decisions at one selection ratio, which is arithmetic on the meta-analytic validities and was measured at no employer, so treat it as the shape of a cost. The analysis never ran consensus debriefs head to head against averaged independent scores, so how much of that gap belongs to the meeting itself is unmeasured ⁴.

None of that argues for cancelling the debrief. A debrief is how evidence gets surfaced, challenged and corrected, and a panel that never meets loses more than it saves. What the finding indicts is the last step, where a conversation stands in for a stated rule about how the pieces combine. Training someone to interview before they run a round alone gets one person to the bar; calibration is where the rule for combining what a panel found gets written, and the log is where it stays written between searches.

See how it works >