Replace the Five-Point Scale With a Call and Its Reason

Which field should be required?

The evidence box and the call, in that order, with the rating left optional. Those two are the fields a debrief can question. A number is not, which is why nobody ever argues with a 4 and can only answer it with a 3. An interviewer who writes a decision plus the two specific things that produced it has done the whole job.

A usable output block fits in four lines and takes about ninety seconds to fill in:

• Call. Hire or no hire, and the round it rests on.
• Because. Two claims, each with the thing that was said or done sitting next to it.
• Against. The strongest reason not to, written by the same person.
• Unseen. What this round could not reach.

The Against line does more work than the rest of the form. It costs thirty seconds, and it turns a one-sided card into something the room can test, because the person who most wants to hire has already written down the best argument on the other side. It also makes a specific and common failure visible: an interviewer whose Against line is empty on every candidate has stopped weighing and started agreeing.

The fields that go above this block, and the order they belong in, are the subject of what actually goes in a hiring scorecard. The output field is the last decision you make about that form.

Does a number help or hurt?

It helps when a rule consumes it and hurts when a conversation does. The evidence on combining candidate data is unusually clear, and people reach for it to defend their scores when what it tested was the combination rule. What beats an impression is applying the same procedure to the same fields for every candidate, and a written checklist does that as well as an algorithm does.

The meta-analysis behind that claim compared studies in which applicant data were combined mechanically by a formula against studies in which the same kinds of data were combined holistically by expert judgment. Average correlations with job performance were .44 for the mechanical route and .28 for the holistic one, which the authors describe as a population-level improvement in prediction of more than 50% ¹. Read the limits before spending it: the job-performance comparison rests on 9 studies, other criteria show much smaller gaps, and the improvement is in a correlation rather than in hires made. The mechanism is what travels, and the authors are explicit that a mechanical method can be as plain as adding up unit-weighted values on a written scorecard.

So nothing here indicts numbers. A number with no rule waiting for it is decoration, and decoration on a hiring form gets read as measurement. If nothing downstream adds that value up, compares it to a threshold, or weights it against another one, delete the column and keep the call. If something does, publish the rule to everyone who fills in the form, before the loop opens.

Keep the scale narrow and anchor every point

If you keep a scale, keep it short and write an anchor for every point on it, including the middle. The payoff from anchors is real and modest, and it is larger for validity than for agreement between raters. Their main work is forcing the team to say in advance what a middle rating would have to look like in this role.

The standard review of structured interviewing reports a meta-analysis of 19 past-behaviour interview studies in which interviews using anchored rating scales showed higher criterion-related validity (.35 against .26) and slightly higher interrater reliability (.77 against .73) than interviews without them ². Past-behaviour questions only, and a comparison across studies rather than the same interview run both ways, so anchored scales travel with whatever else careful teams already do. The review is blunt that anchored scales became popular for their logical appeal ahead of the evidence, which is a fair description of most scorecard design.

The count of points matters less than the anchors, which is why four-against-five is the least productive part of this argument. Removing the middle option forces a lean without making it informative. If nobody on the team can write a sentence describing what a middle rating looks like for this role, that inability is the finding, and no amount of scale design will repair it.

Never average a tool's value with a human rating

A value produced by a tool goes in its own row, labelled with what produced it, and it never enters the same column as a rating a person wrote after watching somebody work. Averaging the two yields a number whose meaning nobody in the room can state. The two are not measuring comparable things, so anyone who wants them in one column should have to state the rule that combines them first.

The evidence on machine-assessed interviews is thin enough to make that the only supportable position. A peer-reviewed study of automated video interview assessments, whose authors say they are unaware of any previous literature examining such scores against organisational criteria, reports an uncorrected correlation of .24 with job performance across five organisational samples totalling 1,124 people, against .32 for human-rated structured interviews stated in the same uncorrected terms ³. Four of the five authors worked for the vendor whose algorithms were evaluated, five samples stand against more than a hundred effect sizes on the human side, and the authors themselves call the research area in its infancy. Nothing in it licenses treating a machine value and a human rating as the same kind of quantity.

Whether any assessment value still tracks performance once candidates work with an assistant is a live question in its own right, taken up in whether assessment scores still predict performance. The same separation applies to submitted work: a polished deliverable and an observed process belong in different rows, which is the practical problem in grading take-homes that all come back polished.

On Monday, change the required field. Make the form ask for a call and the evidence that produced it, demote the rating to an optional column, and give any tool-produced value its own labelled row. Then sit through the next two debriefs and count how often somebody quotes a number instead of quoting the candidate. That count is the measurement, and it moves before the meeting gets any shorter.

See what gets scored >