Assessment design
Eight of Ten Is Not Agreement Until You Subtract Chance
Eighty percent raw agreement between two reviewers is weaker than it sounds, because it counts every match chance would have produced anyway. If each reviewer marked eight of the ten submissions as meets, chance alone would produce 68 percent agreement, and the corrected figure lands near 0.38. Convert before you rely on it: a kappa for categorical calls, an intraclass correlation for numeric scores, reported per rubric line. Ten cases point a direction and settle nothing.
The takeThe point of the computation is not the number it produces. A kappa per rubric line tells you which line two trained readers cannot apply the same way, and that line is a wording problem you can fix in an afternoon. Teams that stop at a single scorecard-level figure get a grade on their rubric and no instruction. So compute it line by line, keep the list of what split, and treat the published bands as other people's conventions.
Where Olive fits
Open a role and see what the work shows
If you are computing this yourself, the expensive parts are the anchored rubric and evidence a second reader can look at directly. Olive returns six separately-evidenced findings per session, each one tied to a timestamped moment in the work rather than to a figure, and the candidate reads the same report.
Assess your shortlistWhy does eight of ten overstate what your reviewers agreed on?
Because most of those eight matches were free. Raw percent agreement counts every case where two reviewers landed on the same label, including the ones they would have landed on without reading anything. On a rubric line where most submissions get the middle category, that share is large.
Mary McHugh's 2012 review of the kappa statistic dates the objection: in 1960 Jacob Cohen critiqued percent agreement for its inability to account for chance agreement 1.
The size of that free share depends on the shape of your scores.
McHugh puts the mechanism in one line: the greater the expected chance agreement, the lower the resulting value of the kappa 1. A line that spread your ten candidates across the scale has little chance agreement to give up. A line where nine of ten got the same label has almost nothing else.
Low agreement also caps what a rating can predict, which is why the figure is worth computing at all. Scott Highhouse's 2008 review puts a number on that cap for the traditional unstructured interview: even against a perfectly reliable and valid criterion, interview-based judgments could never account for more than 10% of the variance in job performance 2.
That 10% is a ceiling implied by the disagreement rather than a measured result, and Highhouse is reporting Conway, Jako and Goodman's 1995 meta-analysis rather than his own work 2.
The shape of your own line is a fact about your scale, and it is checkable before any statistic. Count how many of your twenty scores (ten submissions, two reviewers) fell in each category. If one category holds most of them, the raw fraction is largely describing the scale, and the repair sits in how you write an AI-use rubric two reviewers score the same way.
Check your eight of ten against what chance alone would produce
For each category, multiply the rate at which the first reviewer used it by the rate at which the second did, and add those products across categories: that sum is the agreement chance would have produced. Subtract it from your observed 0.80, divide by the room that was left above chance, and the quotient is kappa. The whole correction takes about as long as writing the ten scores down.
Say both reviewers marked eight of the ten as meets and two as does not meet.
Chance agreement is then 0.8 times 0.8 plus 0.2 times 0.2, which is 0.68. Two people who never opened a submission, using the two labels at those rates, would match on about seven of your ten.
Kappa keeps only what is left. Observed 0.80 less expected 0.68 is 0.12, the room above chance was 0.32, and 0.12 divided by 0.32 is 0.38.
Now run the same eight matches on a line that actually split the group. If each reviewer used meets on five of the ten, chance agreement falls to 0.50, and the same eight come out at 0.60.
Eight of ten, twice, and two different answers.
Both figures are arithmetic on assumed counts rather than a measurement of your team, and the assumption is the load-bearing part. Use your own tallies, and keep the observed fraction beside the corrected one so a reader can see the size of the correction.
McHugh's own worked example shows the gentle version of that gap. On 222 observations the percent agreement was 0.94 and the expected chance agreement 0.57, which leaves a kappa of 0.85 1. A correction costs nine points there and forty-two points in the hypothetical above, and both the spread of the scores and the lower raw agreement drive that difference.
Pick kappa or an intraclass correlation by what the line asks for
Kappa is for a call out of a fixed set of labels, where a miss is a miss. An intraclass correlation is for a line scored as a number, where a 4 against a 5 is a smaller disagreement than a 2 against a 5 and the statistic should count it that way. Running plain, unweighted kappa on a numeric scale discards the distance between the scores, which is most of what a numeric scale is for.
Koo and Li's 2016 guideline for reporting intraclass correlations calls values under 0.5 poor and reserves excellent for anything above 0.90 3.
The same paper adds the instruction readers skip. The value from a single reliability study is only an estimate of the true one. So Koo and Li write that it is more appropriate to judge the level of reliability from the range around that estimate than from the estimate itself 3.
Their rule of thumb for a reliability study is at least 30 varied cases and, where possible, at least three raters 3. Ten submissions and two reviewers clear neither.
Ten is a pilot.
That is not an argument against running the pair. It is an argument against quoting the number the pair produced. Two graders on the first ten, one after that is what repairs a rubric, and the repair is worth more than the statistic.
Set the number of cases and the stopping rule before the first invite goes out, the way a vendor trial has to (decide the sample size before the first invite).
What do the published bands actually license you to say?
Less than a threshold, because even the kappa bands are disputed by the people who wrote about them and none of them was looking at a hiring rubric. Two tables circulate, one for categorical calls and one for numeric scales, both out of reliability research in medicine.
They place your figure against a convention. They do not tell you the figure is strong enough to move a person out of a process.
McHugh's table puts 0.60 to 0.79 at moderate and 0.80 to 0.90 at strong 1.
She also questions the older reading in print. Cohen's suggested interpretation "may be too lenient for health related studies because it implies that a score as low as 0.41 might be acceptable," she writes 1.
Two authors, two lines. Quoting either as a pass mark imports a judgment somebody else made about a different kind of rating.
No table knows what your rubric line decides.
Set the bar by the consequence instead. A line that only sorts written notes into two piles can live with a weaker figure than a line that ends someone's candidacy. Write down where you are setting it before you compute anything, so the number cannot be read backward into a bar that happens to fit it.
Change one of the ten scores and 0.38 moves a long way. The sample is the weak point in everything above: a team reading that figure as a verdict can rewrite a rubric line that was never the problem.
Keep the disagreement log, and report a figure for every rubric line
One figure for the whole scorecard hides the line you need. Compute agreement separately for each rubric line, and beside every figure record the cases where the two reviewers split and the sentence in the rubric each of them was applying.
The figure says how bad it is. The log says what to change, and no statistic supplies that half.
Where the splits land matters more than how many there are. Ashby's 2026 benchmarks report, drawn from its own customers' interviews that had more than one interviewer, found around 38% of scorecard pairs include at least one point difference between interviewers 4.
Reviewers who differ by a point often differ right at the boundary: nearly half of those one-point differences fall between 2 and 3, crossing the yes/no threshold on Ashby's 1 to 4 scale 4.
The unit being counted is a pair, two scorecards set beside each other, not 38% of interviews or of candidates. And nothing in that data says which interviewer was right, because it carries no outcome measure at all.
The usual repair is to the wording, and written anchors are the standard form of it (every number on the scale needs an observable behavior). Levashina, Hartwell, Morgeson and Campion's 2014 review reports Taylor and Small's meta-analysis of 19 past-behavior interview studies, in which anchored rating scales reached interrater reliability of .77 against .73 and validity of .35 against .26 5.
Those figures come from a comparison across studies rather than one interview run twice, on past-behavior questions only, and the reliability gain is the smaller of the two.
One procedural failure can inflate any agreement figure. The same review's content analysis covered 104 interviews reported in 103 articles published between 1997 and 2010, and only 4% of them reported keeping interviewers from discussing candidates between interviews 5.
Two reviewers who compared impressions before scoring have not produced two readings.
That 4% describes published studies rather than hiring teams, and the authors add their own caution: the results hold only where an article fully described the structure it used.
When two readings do disagree, have each reviewer say what they saw before either one defends a call, which is how you settle a split by naming the behavior, not the verdict.
Nothing about the ten has changed. The eight matches are still eight and the two splits still two. What changed is that you can now say how many of the eight chance was going to hand you anyway, and which rubric line the other two came out of.
Rewrite that line, score the next ten against it, and run the arithmetic again. A figure worth quoting is one that moved after you fixed something.
Common questions
Is 80% agreement between two reviewers good enough to rely on?
Only after the correction, because the raw percentage counts every match chance would have produced. If both reviewers used the same label on eight of the ten, chance alone would produce 68% agreement and the corrected figure lands near 0.38. If each used it on five of the ten, chance produces 50% and the same eight matches come out at 0.60. The raw fraction is identical in both cases. Compute the correction from your own tallies, per rubric line, and keep the observed figure beside it so the size of the correction stays visible.
What counts as a good kappa for a hiring rubric?
Neither of the two usual bands was calibrated on hiring. McHugh's table, written for health research, puts 0.60 to 0.79 at moderate and 0.80 to 0.90 at strong, and she argues Cohen's older reading may be too lenient for health research because it implies 0.41 might be acceptable. Koo and Li's separate bands for the intraclass correlation call anything under 0.5 poor. Both are conventions out of clinical measurement. Set your own bar by what the line decides. A line that sorts notes into two piles can live with less than a line that ends a candidacy, and the bar belongs in writing before the number is computed.
Should agreement be computed per rubric line or for the scorecard overall?
Per line. A single scorecard-level figure averages a line two reviewers apply identically with a line they read differently, and the second one is the one you can fix. Compute a separate figure for each line, and keep a log beside it naming the cases where the two split and the sentence in the rubric each reviewer was applying. The figure tells you how bad it is. The log tells you what to rewrite.
How many scored submissions does an agreement number need?
More than ten. Koo and Li's guideline for reliability studies puts the rule of thumb at a minimum of 30 varied cases and, where possible, at least three raters. The same paper says the level of reliability should be judged from the range around the estimate rather than from the estimate itself. Ten submissions and two reviewers clear neither. A figure from ten points a direction and supports a rubric rewrite. It does not support a sentence in a hiring policy, and a team that treats it as one can rewrite a line that was never the problem.
What if the two reviewers reached the same score for different reasons?
Count it as agreement on the outcome and a failure of the rubric line. Two readers landing on meets because one saw a source checked and the other saw a clean write-up are applying different criteria, and the statistic cannot see that at all. It only compares the labels. The fix is the same as for an open disagreement: record what each reviewer pointed at, then rewrite the line so it names the observable thing rather than a quality. Score independently and without discussion first, or the agreement you measure is agreement about each other.
References
- 1. Interrater reliability: the kappa statistic ✓ pmc.ncbi.nlm.nih.gov Supports the article's chance-correction argument and its reading of the bands: Cohen critiqued percent agreement in 1960 for its inability to account for chance agreement; the greater the expected chance agreement, the lower the resulting value of the kappa; the interpretation table puts 0.60 to 0.79 at moderate and 0.80 to 0.90 at strong; Cohen's suggested interpretation 'may be too lenient for health related studies because it implies that a score as low as 0.41 might be acceptable'; and the Figure 3 worked example reports percent agreement 0.94, expected chance agreement 0.57 and kappa 0.85 on N = 222.
- 2. Stubborn Reliance on Intuition and Subjectivity in Employee Selection ✓ edbatista.com Supports the claim that rater agreement caps what a rating can predict: Highhouse states the interrater reliability of the traditional unstructured interview is so low that, even with a perfectly reliable and valid criterion, interview-based judgments could never account for more than 10% of the variance in job performance, citing Conway, Jako and Goodman (1995).
- 3. A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research ✓ pmc.ncbi.nlm.nih.gov Supports the article's intraclass-correlation bands, the range-not-point instruction and the sample floor: 'Values less than 0.5 are indicative of poor reliability ... values greater than 0.90 indicate excellent reliability'; 'it is more appropriate to evaluate the level of reliability based on the 95% confident interval of the ICC estimate, not the ICC estimate itself'; and 'researchers should try to obtain at least 30 heterogeneous samples and involve at least 3 raters whenever possible when conducting a reliability study.'
- 4. Recruiting Operations Benchmarks | 2026 Talent Trends Report ✓ ashbyhq.com Supports the claim that reviewer disagreement concentrates at the decision boundary: around 38% of scorecard pairs include at least one point difference between interviewers, and nearly half of those one-point differences fall between 2 and 3, crossing the yes/no threshold on Ashby's 1 to 4 scoring scale.
- 5. The Structured Employment Interview: Narrative and Quantitative Review of the Research Literature ✓ doi.org Supports two claims: that anchored rating scales showed interrater reliability of .77 against .73 and criterion-related validity of .35 against .26 across Taylor and Small's meta-analysis of 19 past-behaviour interview studies; and that in the review's content analysis of 104 interviews from 103 articles published between 1997 and 2010, only 4% used no discussion between interviews.
5 sources, numbered by first appearance. Every one was opened and checked against the claim it carries. How Olive sources claims
General guidance for hiring teams. What works at one company and one volume may not transfer to yours.
Olive assesses how a person works with AI. It does not detect AI-written documents, and it never produces a score, a ranking, or a match percentage for a person. Candidates read the same report the employer reads.