Every Interviewer Owns One Competency or Comes Off the Loop

Who should be on the interview loop?

Whoever can produce evidence on a competency the rest of the loop cannot reach. That is the whole test, and it is not the usual answer, which lists the roles that ought to be represented: hiring manager, HR, a peer, someone cross-functional, three to five people. Representation is a fairness and candidate-experience question worth answering on its own terms. Answering it leaves the peer seat's ability to judge the work entirely untested.

The usual answer also leaves the loop's actual output unexamined. In one large recruiting dataset, recruiter screens pass roughly 35% of the candidates who reach them, while later stages convert at 95% post-onsite and 81% at offer ¹. Those are passthrough rates among candidates who reached each stage, they do not multiply into an end-to-end rate, and the heaviest filtering happens earlier still at application review, which is not one of these numbers. A 95% passthrough late in the loop is not evidence that an onsite is easy. It means the decision has effectively been made before the onsite ends, and the seats inside it are ratifying.

Ratifying is worth something, though four calendars and six candidate hours is a steep price for it. A seat earns its place when it changes what the debrief knows, and the only way to find out whether yours do is to read the last three searches and ask which write-up moved a verdict.

Give every seat one competency and one piece of evidence

Write two lines per seat before the invites go out: the competency it owns, and the evidence it will bring back. Evidence here means something a second reader can act on. If two seats would return the same evidence, one of them is a duplicate, and duplicates are where a loop's hours go without anyone having decided to spend them.

A worked example for a mid-level analyst role, four seats:

• Hiring manager: judgment under an ambiguous brief. Evidence is the candidate narrating a real decision where the brief was wrong and saying what they changed.
• Peer analyst: technical depth in this material. Evidence is a specific error the candidate caught in their own past work, and how.
• Cross-functional partner: how the candidate handles being contradicted. Evidence is a described disagreement with an outcome, not a stated philosophy.
• Recruiter: constraints and motivation. Evidence is dates, numbers and the reason they are looking, which is the only seat that has to be the same for every candidate.

Stacking more conversations does not substitute for different kinds of evidence. The applied follow-up to the 2022 revision of the selection-validity literature shows a composite of predictors reaching about .61, and removing cognitive ability from that composite costing .05, dropping it to .56 ². Composite validity like that assumes the pieces get combined mechanically with sensible weights and that they measure genuinely different things. Four unscored conversations mostly measure the same thing four times, which is why a fifth seat rarely buys what a different kind of evidence would. Adding a stage is the expensive way to do that; testing AI skills without adding an hour to the loop is the cheaper one.

When do you take someone off?

Three conditions, any one of which is enough: the seat owns no competency, it duplicates another seat's evidence, or it keeps returning a verdict with nothing a second reader can use. The third is the awkward one, because the person is usually senior and usually certain. Certainty is the tell, and a loop that cannot remove a seat grows until the candidate is paying for the org chart.

The nearest measured version of that third case comes from job tests rather than interviews. Across 15 firms hiring low-skilled service workers, introducing a job test raised completed job tenures by just over 25%, and comparing managers at the same location, a one standard deviation higher rate of hiring against the test's recommendation went with 6% to 7% shorter job durations ³. That is not a randomised experiment, the quality measure is tenure in a high-turnover setting rather than a performance rating, and overrides were common rather than pathological in that data. It also says the average override was worse, never that the test was right about any particular person. What it will not support is the idea that a senior person overriding the structured signal is exercising superior private information as a rule.

Removal is rarely the right first move, though. Most seats that return opinions are held by people who could return evidence on something else. A manager who does not use an assistant on their own work cannot fairly assess output quality on AI-assisted material and can absolutely assess how a candidate decides what to check, which is the reassignment getting managers who don't use AI to judge AI-assisted work works through in detail.

Should the panel get bigger to be fairer?

No, and the extra seats carry a cost that is easy to overlook: every seat brings its own informal opening, and the informal opening feeds the formal rating. More observers of the same kind is not more evidence. It is the same measurement taken again, with more small talk attached and two more hours on the candidate's calendar.

There is a measurement of that leak. In a study of 189 accounting students put through structured mock interviews, the interviewer's overall impression formed during the rapport-building small talk, before any structured question, correlated .42 with that same interviewer's structured interview score. When a different interviewer supplied the structured score, the correlation fell to .25 and .24 ⁴. Much of the larger figure is one interviewer's impression predicting their own later rating. The authors' own conclusion is not that first impressions are pure contamination, since the incremental effect ran through rated competence rather than liking or similarity, and none of it establishes that interviewers decide in the first three minutes. What it does show is that a seat is not a neutral observation post, so an uneven panel calls for structure inside each seat.

What a fifth seat is genuinely good for is building the bench. Rotate one shadow through the loop with no vote, writing independent notes before hearing anyone's read, which is the practice half of training someone to interview before they run a round alone. The bench grows and the candidate pays nothing for it, in hours or in extra deciders.

See how it works >