Interviewing
When Two Interviewers Split on One Moment, Fix the Anchor
Two interviewers who agree on what happened in one AI moment, and still call it strong delegation and outsourced thinking, are not disagreeing about the candidate. The split is about what the rubric line means, and no vote, veto or tiebreak reaches it. Write the behavior down in one sentence with no adjectives, ask each interviewer which anchor on the scale it matches, decide this candidate on that sentence, and fix the anchor before the next candidate.
The takeThe word delegation on a scorecard is doing far less work than the team believes. It names a whole family of acts, from picking a tool to giving away the call the job exists to make. A scale that never separates them will produce this argument forever, with a different candidate in the chair each time. Treat the argument as a personality problem and the panel staffs around it. The line is what is broken.
Where Olive fits
Open a role and see what the work shows
An interview nobody recorded leaves two accounts of one moment and nothing to replay. Olive returns six findings written or edited by a human reviewer, each anchored to the timestamped excerpt behind it, so a second reader argues with the excerpt instead of with the first reader's adjective.
Assess your shortlistWhy can't a vote settle two readings of one moment?
A vote can't settle the split because both interviewers already agree on the facts. They watched the same candidate hand the same task to a model, and neither one disputes what happened next. What they disagree about is where the line falls between handing over work and handing over the call. A tiebreak hands that disagreement to whoever holds more votes and leaves the question where it was.
In 2013 Nathan Kuncel, David Klieger, Brian Connelly and Deniz Ones published a meta-analysis of how selection evidence gets combined. It files group consensus meetings in the same class as one person's private impression 1.
What puts a method in that class is the combining: judgment rather than a rule applied the same way each time.
A debrief that ends in a shared feeling is the holistic method with more people in the room.
The numbers describe the combination step, and the meeting was never tested on its own. Across nine studies of job performance, evidence combined by a formula correlated .44 with later performance, and the same kinds of evidence combined by expert judgment correlated .28 1.
A formula there can be as plain as adding up the points on a written scorecard, and the authors say so.
But a combination rule doesn't reach this split either. Adding two ratings assumes both raters measured the same thing, and these two measured different things and gave both of them one name.
Authority rules still earn their place, and the good one comes with a standard attached: a no has to name the specific thing that was said or done. Both interviewers can meet that standard here, in the same words, and still land on opposite verdicts.
Write down the moment with no adjectives in it
Write the moment down before either interviewer says another word about it. One sentence, past tense, naming what the candidate did and in what order: what they asked for, what came back, what they checked, and what they shipped. Strip every adjective.
Strong and outsourced are the two verdicts under dispute, so a description carrying either one has already decided the question it was written to settle.
The two descriptions can come back different, which is already a finding. Suppose one interviewer watched the candidate name the acceptance criteria before opening the model, and the other joined eight minutes in and saw only a finished answer being defended.
A split can be two people describing two different stretches of one hour.
Ashby's 2026 recruiting benchmarks, drawn from interviews in its own dataset, put a number on how often two interviewers' ratings differ. A scorecard pair there is two scorecards compared against each other. Around 38% of those pairs carry at least a one-point difference 2.
The yes and no line takes nearly half of those one-point gaps, which sit between 2 and 3 on a 1 to 4 scale. The dataset holds no outcome measure, so nothing in it says which interviewer was right.
The narrower finding is the useful one: ratings come apart exactly where the decision gets made.
Hand the signed sentence back to both interviewers before anyone argues again. Disagreement that survives a description both people agree to is the real disagreement, and it is now small enough to work on: one behavior, one sentence, two readings.
Which anchor on the scale does the behavior match?
Ask each interviewer to point at the anchor on the rating scale that the written behavior matches, and to read it out loud. Expect one of two answers. Either the scale carries no anchor that separates handing over a task from handing over the judgment, or both interviewers found the same anchor and are holding it at different levels.
The first case is a hole in the scale. A line reading delegates effectively to AI tools has a subject, a verb and no bar.
What an anchor can hold is an act somebody could have watched: what the candidate framed before generating anything, and one claim they checked.
A published vocabulary doesn't supply the bar either. The AI Fluency Framework was written by Rick Dakan of Ringling College of Art and Design and Joseph Feller of University College Cork. It splits Delegation into Goal and Task Awareness, Platform Awareness and Task Delegation 3.
No levels come with those names. The framework publishes no scoring anchors and no proficiency thresholds, and its authors number it version 1.1 and write of what it identifies in its current version 3. Naming a competency and setting a bar for it are two separate acts.
A construct with a name and no level gets read at whatever level each rater already carries.
The second case is a level disagreement, and it costs less to fix. One interviewer is rating against what a strong senior would have done and the other against what the role requires at this band.
Say which of the two the scale is for and the readings collapse to one.
A third answer turns up often enough to check for. One interviewer never saw the behavior and is rating on inference from somewhere else in the round, which calls for a separate no-evidence box rather than an anchor rewrite.
Write the anchor that would have prevented the split
Write the missing anchor in the vocabulary of the moment already on paper. A delegation split needs two lines: one for handing a task to a model and checking what came back against something outside it, and one for handing over the choice the job pays this person to make. Give each line a real example, the way every number on the scale gets an observable behavior.
Anchors are worth the hour, though the gain is uneven. A 2014 review by Julia Levashina, Christopher Hartwell, Frederick Morgeson and Michael Campion reports Taylor and Small's meta-analysis of 19 past-behavior interview studies 4.
With anchored rating scales, criterion validity was estimated at .35 against .26 without them, and interrater reliability moved less, .77 against .73 4.
The bigger gain is in what the interview predicts. All 19 studies asked about past behavior, so the figures do not carry straight over to a live AI exercise. The comparison was also made across studies, so anchored scales travel with whatever else careful teams already do.
Anchor every point, the middle included. The same review reports one study in which anchoring all five points of a scale, rather than only the top and the bottom, produced ratings more resilient to disability bias 4.
The rarest parts of a structured interview sit on the scoring side.
A content analysis in the same review covered 104 interviews from studies published between 1997 and 2010 and found an average of 5.74 of 15 structure components in use 4. Detailed notes appeared in 19% of those interviews.
The count measures what researchers built into published studies, so read it as a ceiling on practice: employers are unlikely to build in more structure than published studies did.
Then calibrate on something nobody is deciding about. Take two answers recorded in a loop that has already closed, have both interviewers rate them against the rewritten anchor, and compare before the next live candidate arrives.
An hour on two recorded answers is enough to find out whether the rewrite took.
What a rewritten anchor still won't settle
A rewritten anchor won't settle the candidate already on the table. Both interviewers scored against the old line on the day, and a standard written this afternoon can't be run backward over one person without handing everyone else in the loop a different test. Decide this candidate on the description both interviewers signed.
And sometimes the split is about the candidate after all. One interviewer may have seen a later exchange the other missed, or pushed harder and got a different answer, in which case the scale is fine and the rounds were not the same round.
No anchor rewrite recovers an hour that ran two ways.
Record which anchor each interviewer reached for, next to the sentence they both signed. The record is what makes the next split cheap.
Count how often the same line splits a panel.
A rubric line that splits three panels in a quarter is a defect in the line rather than a run of hard-to-rate candidates.
The written sentence outlasts the argument that produced it. Filed beside the anchor it was matched against, it becomes the case the next panel reads before scoring anyone, which is what a calibration log is for.
The two verdicts are the part nobody needs again.
Common questions
Who makes the call while the anchor is being rewritten?
The hiring manager, on the description both interviewers agreed to rather than on the two labels they put on it. Take the signed sentence, ask each interviewer which anchor they reached for and why, and decide on the behavior that is not in dispute. Rewriting the anchor is separate work with a different deadline. It changes the standard for the candidates who come next. Running a new line backward over one person means everyone else in that loop sat a different test.
Does averaging the two ratings work?
Averaging is a formula, the kind of combination with the better record, and it still doesn't reach this problem. Kuncel, Klieger, Connelly and Ones found evidence combined by a formula correlated .44 with later job performance against .28 for the same kinds of evidence combined by expert judgment, across nine studies. But a formula assumes the two inputs measured the same thing. When one interviewer rated the handoff of a task and the other rated the handoff of a decision, the average is a number about nothing in particular. Fix what the scale asks for first, then combine.
What does a delegation anchor look like when it is written down?
Two lines in the language of the job. One names handing a task to a model and checking what came back against something outside the chat: the query run, the figure recomputed, the clause opened. The other names handing the model the call the role is paid to make. Each line carries one real example from work this role contains. The middle point gets an anchor too, since a scale anchored only at the ends leaves the middle to be filled in from memory.
Does the AI Fluency Framework give a rating scale for delegation?
No. The framework, written by Rick Dakan and Joseph Feller, names Delegation as one of four competencies and splits it into Goal and Task Awareness, Platform Awareness and Task Delegation. What comes with those names is a taxonomy: no levels, no scoring anchors, no proficiency thresholds, and a framework its authors number version 1.1. Borrowing the vocabulary is useful and cheap. Borrowing it as a rubric leaves every rater to supply the bar privately, which is the condition that produces two readings of one moment.
How do you tell a rubric defect from one hard-to-rate candidate?
Count the line, not the candidate. Log which rubric line split the panel and how often, then read the log at the end of the quarter. One split on one line is a candidate who sat on a boundary. Three splits on the same line across different candidates is a line no two people read the same way. Raw agreement counts flatter the scale: two raters on a four-point scale can land on the same number often enough by chance to look calibrated. Count the splits themselves.
References
- 1. Mechanical Versus Clinical Data Combination in Selection and Admissions Decisions: A Meta-Analysis ✓ gwern.net Supports two claims: that a group consensus meeting is classified as a holistic combination method alongside one person's private judgment, and that across nine studies of job performance evidence combined mechanically correlated .44 with performance against .28 for the same evidence combined by expert judgment.
- 2. Recruiting Operations Benchmarks | 2026 Talent Trends Report ✓ ashbyhq.com Supports the claim that interviewer ratings disagree at the decision boundary: around 38% of scorecard pairs carry at least a one-point difference, and nearly half of those gaps fall between 2 and 3 on Ashby's 1 to 4 scale.
- 3. Framework for AI Fluency ✓ ringling.libguides.com Supports the claim that the framework names Delegation and its three sub-competencies without publishing levels, scoring anchors or proficiency thresholds, and calls its current list a current version.
- 4. The Structured Employment Interview: Narrative and Quantitative Review of the Research Literature ✓ doi.org Supports three claims: the reported Taylor and Small (2002) meta-analysis of 19 past-behavior interview studies giving .35 against .26 validity and .77 against .73 interrater reliability for anchored rating scales; the finding that anchoring all five scale points beat anchoring only the ends; and the content analysis of 104 interviews from 1997 to 2010 finding an average of 5.74 of 15 structure components, with detailed notes in 19%.
4 sources, numbered by first appearance. Every one was opened and checked against the claim it carries. How Olive sources claims
General guidance for hiring teams. What works at one company and one volume may not transfer to yours.
Olive assesses how a person works with AI. It does not detect AI-written documents, and it never produces a score, a ranking, or a match percentage for a person. Candidates read the same report the employer reads.