The Role Owns the AI Bar, Not Whichever Manager Is Hiring

Who owns the AI bar when two managers disagree?

The role owns it, and whoever owns the role definition writes it down. A hiring manager sets emphasis inside a standard; a hiring manager does not get a private standard. When two of them run the same title at different heights, the company has posted one job and is selecting on two.

The recruiter sees both loops, which makes holding the line their work.

Neither manager thinks they are running a test. One asks a few AI questions in a screen and forms an impression, and the other hands over a short exercise with an answer key.

Both of them are running a selection procedure. The US federal Uniform Guidelines define the term as any measure or procedure used as a basis for an employment decision. The 1978 text says it reaches everything from paper-and-pencil tests through informal or casual interviews and unscored application forms ¹.

A casual conversation about AI is a selection procedure with no answer key.

The definition does not oblige two managers to read from one script. The Guidelines put a validation burden on a procedure only where adverse impact appears. What the definition settles is the category, and a rejected candidate comparing the two rounds is comparing two selection procedures, both inside that definition.

Inside a single requisition the allocation is clean: the manager owns the bar and the recruiter owns the evidence. Two requisitions carrying one title break it, because no single manager is available to hold the bar. Ownership moves up a level, to the job.

What two different bars cost, twice

A split bar costs once at the rejection and once at the promotion cycle. The first cost is a candidate turned down under the higher standard who learns that a peer cleared the lower one for the same title, and asks which of the two was the job. The second arrives about a year later, when two people hired at different heights sit in one band and level differently.

Start with the smaller version of the problem, which is already measured. Ashby's 2026 benchmark report looked at interviews where more than one interviewer filed a scorecard, and found around 38% of scorecard pairs carried at least one point of difference ².

A 2 against a 3 is where the yes and the no divide on Ashby's four-point scale, and nearly half of those one-point gaps sat on that line ².

That 38% counts pairs, each one two scorecards read against each other, so it is a rate of disagreeing pairs rather than of interviews or of candidates.

The dataset carries no outcome, so nothing in it says which interviewer was right.

Two interviewers reading one candidate disagree that often, and much of it at the line. Two managers in two rooms, reading different candidates against standards neither has written down, have less still holding them to one line, and nothing in the data measures how far apart they land.

The second cost is slower and harder to unwind. Both hires land in one band on one job code, and the one who cleared the thinner bar reaches the first review never having been asked to show the capability the other was screened for. Nobody recorded that the two were measured differently, so any gap reads as performance.

Pay is the tempting instrument at that point, and it is the wrong one. A band exception needs evidence that survives a pay audit, and papering over an inconsistent hiring standard supplies none.

Decide whether you are hiring one job or two

Settle one job or two before touching either loop, because two legitimate bars are possible. Two teams can share a title and not share the work: one owns a model-facing pipeline, the other owns a spreadsheet and a quarterly report. Where the difference is real, split the requisition and say in the posting what differs. Where it is not, one job is being hired twice.

The instrument that settles it is older than the argument. Under the Uniform Guidelines, any validity study should rest on a review of information about the job the selection procedure will be used for, and that review should include a job analysis ³.

A job analysis is a written account of what the work actually requires. Two managers who cannot agree on a bar may have nothing written to disagree against.

Write it as a list of decisions rather than a list of tools.

What does this person decide without a second reader, what has to be checked before it ships, and where does a confident wrong answer cost real money.

A bar built from those three questions either applies to both teams or it visibly does not, and where it does not, the requisition splits. Naming the difference in the posting costs less than discovering it in a debrief six weeks later.

Whether one instrument can serve both is a separate question. One AI-skills assessment across every department, or per-role turns on how much of the judgment is shared, and the job analysis is what tells you.

Why a calibration meeting does not hold a bar

A meeting surfaces the disagreement and does not settle it when what comes out of the debrief is a shared feeling rather than a stated rule. Kuncel, Klieger, Connelly and Ones's meta-analysis of selection and admissions studies classifies a group consensus meeting as a holistic method, alongside individual expert judgment ⁴.

What makes a method holistic is that data are combined by judgment rather than by a rule applied the same way each time.

Skip the debrief and you lose the place where evidence gets surfaced, challenged and corrected. What the taxonomy calls holistic is the final step, the moment two managers stop comparing what a candidate did and start comparing how they feel about it.

At a selection ratio of .30, the authors calculate, the lower validity of holistic combination can mean a 25% reduction in correct hiring decisions ⁴. The figure is calculated from the validities the meta-analysis reports rather than measured at any employer, and the paper gives it as holding across base rates at that ratio. The meta-analysis never tested consensus meetings on their own, so it cannot say how much of that loss belongs to the meeting.

A meeting of that kind can end by deferring to whoever is held to be the better judge of people.

Scott Highhouse's 2008 review in Industrial and Organizational Psychology grants that it is commonly accepted that some employment interviewers are better than others. But research on variance in interviewer validity suggests the differences are due entirely to sampling error, the review states ⁵.

The same review reproduces a 1943 admissions result. High school rank plus a college aptitude test correlated .45 with academic achievement, and the same two predictors plus counselors' intuitive judgment correlated .35 ⁵.

Adding the human judgment on top made the prediction worse.

Neither number comes from hiring, and the page carrying the 1943 table reports no sample size, so read it as the origin of a long-standing finding rather than as a measurement of your panel.

Finding the bar a manager actually uses is cheaper than asking for it. Get the manager's bar from their rejections, not a meeting, because a stated bar and an applied bar are different objects and only one of them leaves a trail. And once both are written down, calibration is a standing session, not an onboarding slide.

Write one bar, then weight inside it

One page settles most of this: the behaviours the role requires, a scale with a written anchor at every point, and one cut that neither manager may move. Emphasis stays negotiable. A manager running the model-facing team can weight verification heavier than framing, and say so in the debrief, without lowering the line either candidate had to clear.

Publish the cut and the room for emphasis to both managers before either loop opens. A standard produced after a rejection reads as a justification, and reads that way to whoever examines it later.

Then record what each manager applied rather than what they concluded.

Liu, Chang, Jiang, Ma and Zhou text-mined the post-interview notes written about 7,650 candidates hired at a large Chinese internet technology company ⁶. The more job-related capabilities an interviewer named in the notes, the better the hire went: higher performance ratings, more promotions, less turnover. A one standard deviation rise in the note-to-job-analysis matching score corresponded to roughly a 2 percent rise in performance.

The limits ride along. Only candidates who passed and joined could be observed, the design is correlational, the setting is one company in technology roles, and the effect is small. What survives is narrow and worth having: among the people hired, notes that named the capabilities the job analysis called for tracked how the hire went.

One bar set at the wrong height is wrong in both loops at once, and consistency is not accuracy.

The manager whose bar was higher may have been reading something in the work that the job analysis missed.

So make the higher bar argue for itself instead of averaging it away. If verification really is load-bearing on one team and not on the other, that belongs in the job analysis first and in two postings second. Setting a defensible bar for good enough at AI in a specific role is that exercise, done once.

The two open requisitions are where the argument ends. By the next opening they should be one requisition with a written bar under it, or two requisitions that say plainly how the work differs. One job with two answers is the arrangement that has to go.

See the benchmarks >