Interviewing
Add Not Observed to the Scale, Then Count How Often It Fires
When an interview round never reaches the AI line, the interviewer should record not observed, and a scorecard that forces a rating needs that option added. A forced number means the round ran out of time, and it then averages into the decision as though somebody watched the candidate work. Mark the line, keep it out of the average, name the round that will cover it, and count how often each line comes back unobserved. Move a line missed in a third of rounds to another round.
The takeThe unobserved rate is the most useful number a hiring loop can produce, and a form that forces a rating can never produce it. Ratings describe candidates. A count of blank lines describes the round: whether it can generate the evidence its own rubric asks for. Teams keep tuning questions and anchors while the real defect is that the AI line sits in a round with no room left for it. Ask the loop to report on itself before asking it to grade anyone.
Where Olive fits
Open a role and see what the work shows
An interview round can only record what it had time to watch, and nothing guarantees the AI line gets any of it. Olive puts that work in front of the candidate on their own clock: a role-grounded assignment with an AI assistant, and six findings a human reviewer writes or edits, each anchored to a moment in the session.
Assess your shortlistWhat does a 3 on the AI line record when the round never asked?
The time you ran out of, dressed as a judgment about a person. The line reads AI judgment, the scale runs 1 to 4, the round ended without ever reaching it, and the form will not submit empty. Whatever number goes into that field was produced by the interviewer's general impression of the candidate, not by anything the candidate did.
The guess does not stay a guess for long.
Once it enters the scorecard it becomes arithmetic. The debrief averages it, the summary reports it, and the hiring manager reading a 3.2 on AI judgment has no way to see which of those numbers came from watching someone work.
Ratings wobble even between two interviewers in the same interview. Ashby's 2026 recruiting benchmarks report that around 38% of scorecard pairs (two interviewers' scores set against each other) differ by at least a point 1. Nearly half of those one-point gaps fall between 2 and 3, crossing the yes-or-no threshold on Ashby's 1 to 4 scale 1.
The dataset carries no outcome measure, so none of that says which interviewer was right. It does say the scale is unstable exactly where the decision gets made.
A number no observation produced only adds to that noise.
The rule that combines a scorecard is where a fabricated entry does its damage. In a 2013 meta-analysis by Kuncel, Klieger, Connelly and Ones, applicant data combined mechanically by a formula correlated .44 with job performance, against .28 when the same kinds of data were combined by expert judgment 2.
Nine samples carry that job-performance comparison, so read it as a direction and not a coefficient to bank.
A rule applied identically every time is still only as honest as the cells it reads.
Put not observed outside the 1 to 4, never at the bottom of it
A not-observed outcome belongs on the form itself, as its own entry outside the numeric range. Every line on the scorecard offers it, it means no evidence was collected, and the aggregation rule leaves any line marked with it out of the arithmetic entirely.
Say in the interviewer guide that using it is expected. A rater who reads it as an admission will pick a number instead.
Not observed is not a low rating, and the arithmetic has to say so out loud.
Anchors are meant to make a rating point mean the same thing twice. Levashina, Hartwell, Morgeson and Campion's review reports Taylor and Small's 2002 meta-analysis of 19 past-behavior interview studies 3. Interviews using anchored rating scales showed higher criterion validity, .35 against .26, and slightly higher interrater reliability, .77 against .73 3.
The reliability gain is the smaller of the two. Neither figure comes from running one interview with anchors and then without them.
The scale still has to say what each point means, and what its empty state means.
The repair is the same one that makes the rest of the scorecard work: every number on the scale needs an observable behavior, written in the words of the job. The unobserved outcome gets a written meaning too, and its meaning is procedural. The round did not put this candidate in a position to show what the line asks for.
A scorecard is a record before it is a score. If a scorecard holds evidence, a call, and nothing else, then a line with no evidence has exactly one honest entry available to it, and a three is not that entry.
The summary a hiring manager reads names which lines came back unobserved, in the same place it reports the lines that got a rating. Leave that out and the outcome is decoration.
Should the debrief read a blank AI line as a soft no?
No. A blank line means the round did not ask, and a room that reads it as a soft no has manufactured a finding out of a scheduling failure. Open the debrief by naming the unobserved lines, assigning each one to a round that can produce the evidence, and leaving them out of the comparison until somebody has actually seen something.
The debrief is also where the combination rule can go missing. The same 2013 meta-analysis classifies group consensus meetings as a holistic method, alongside individual expert judgment 2, and a method is holistic when evidence gets combined by judgment rather than by a rule applied the same way every time.
The classification is not an argument against holding the meeting.
A debrief is how evidence gets surfaced and challenged. It should not end in a shared feeling standing in for an observation nobody made.
What a debrief runs on is thin even in published research. In a content analysis of 104 interviews reported across 103 articles published between 1997 and 2010, studies describing a structured interview used an average of 5.74 of the 15 structure components in Campion, Palmer and Campion's framework 3.
Detailed notes are one of those components, and only 19% of the studies described them 3.
A room with no notes in front of it cannot tell a line nobody asked about from a line somebody answered badly. Both reach the debrief as an impression, and an impression fills a blank as readily as it reads one.
The alternative to arguing it out is rarer still. Statistical prediction was described in 11% of those studies 3, so the rest either combined by judgment or never said how they combined.
The count comes from published studies, not from employers, so read it as a ceiling on practice and not a floor.
Two interviewers who watched the same moment and read it differently have a different problem, with a different repair: settle it by naming the behavior, not the verdict. An unobserved line has no behavior to name yet. Keep the two apart, and a scheduling gap never gets read as a weakness in a person.
Which lines go blank often enough to move to another round?
Move the lines that come back unobserved in a third of their rounds, and find them by counting the blanks by line and by round once a quarter, reading the column rather than the individual scorecards. A competency missed that often is not an interviewer discipline problem. Either it sits in a round with no time for it, or it asks for evidence that round cannot produce at all.
A third is a management threshold, not a validated cutoff. Pick one, write it down, and act on it the same way each quarter.
A round's ratings only compare across candidates if the round runs the same way every time. The US Office of Personnel Management's practical guide defines a structured interview by three properties. The same questions in the same order for every candidate, a common rating scale, and interviewers who agree in advance what an acceptable answer looks like 4.
A question half the candidates never hear fails the first of the three.
The guide dates to 2008 and it is federal hiring practice, not statute, so it binds no private employer. The definition is still a clear public one to borrow.
The fix is a grid. Competencies down the side, rounds across the top, and exactly one owner for each line: the round whose format can actually show it. A line owned by three rounds is owned by none of them, which is how it goes unasked in all three.
Before any of this starts, the record is thinner than it looks. In the same Ashby data, scorecard completion runs near 49% at organizations under 25 employees and closer to 72% at organizations of 500 or more 1. Completion also runs higher for candidates who get hired, so what exists on rejected candidates is thinner still 1.
A forced rating converts a missing record into a wrong one. A gap announces itself; a number does not.
When does marking not observed erase a real answer?
When the question was asked and the answer was thin. Not observed means no evidence was sought, never that the evidence disappointed. An interviewer who ran the AI question, heard a general answer about always checking the output, and then marked the line unobserved has erased a real observation.
The candidate loses the one place where a weak answer could be written down and argued with.
Write what happened instead: the question asked, the answer given, and the call it supports.
If the easy exit exists, somebody takes it. A loop that can record nothing will record nothing, and a team that never reads the rate collects less evidence every quarter while its scorecards look tidier than they did before.
Reading the rate every quarter is how you see the easy exit becoming the habit. Put the count on the same review as time-to-fill.
The rating also has a life after the decision. The Uniform Guidelines on Employee Selection Procedures, adopted by four federal agencies in 1978, define a selection procedure as any measure, combination of measures, or procedure used as a basis for an employment decision 5. The definition reaches the full range of assessment techniques, down to informal or casual interviews and unscored application forms 5.
Nothing there requires an audit. The validation burden in the Guidelines attaches only where adverse impact appears, and a US employer's own counsel is the place to settle what applies to a given round.
What the definition turns on is use.
A rating produced on a line nobody watched is used as a basis for the decision the moment it averages in.
The line still reads AI judgment and the scale still runs 1 to 4. What changed is that the form now takes an empty one, marked and dated, routed to the round that can fill it.
Common questions
What should go on a scorecard line the round never reached?
Not observed, recorded as its own outcome rather than as a number. A rating on a line nobody watched carries the interviewer's general impression and nothing about the competency it names, and once it enters an average, no reader can tell it apart from a rating somebody earned. Record the blank, note in one line why the round did not reach it, and name the round that will. The record stays honest and the decision keeps only the evidence it actually has.
Does a not-observed line count as a low score?
No, and the aggregation has to enforce that rather than the interviewer's good intentions. A not-observed outcome sits outside the numeric range, drops out of any average instead of entering it as a one or a two, and prints as a count beside the score so a reader sees how much of the scorecard is filled. A scale that puts the empty state at the bottom of the range will be used as a rating, whatever the guide says.
How often is too often for a line to come back unobserved?
Pick a threshold and write it down; a third of rounds is a defensible place to start. A third is not a validated cutoff, so the number is a management decision rather than a finding, and the value comes from acting on it the same way each quarter. When a competency crosses the line, move it to a round with time and a format that can show it, rather than reminding interviewers to cover more ground in the same slot.
What if the interviewer asked the AI question and the answer was thin?
A thin answer is an observation, and it belongs on the scorecard as one. Not observed means the evidence was never sought, never that it disappointed. Write the question asked, the answer given, and the call it supports, so a later reader can disagree with the reading rather than with a bare number. A thin answer filed as unobserved erases the only place where a weak response could have been challenged, including by the candidate.
Who decides which round owns the AI line?
Whoever owns the loop, before the first candidate enters it. Put competencies down one side of a grid and rounds across the top, then give every line exactly one owner: the round whose format can actually produce that evidence. A competency assigned to three rounds is assigned to none of them, which is how it goes unasked in all three and lands on the scorecard as a guess.
References
- 1. Recruiting Operations Benchmarks | 2026 Talent Trends Report ✓ ashbyhq.com Supports the roughly 38% of scorecard pairs differing by at least one point, the nearly half of those one-point differences falling between 2 and 3 on Ashby's 1 to 4 scale, and the scorecard completion rates of about 49% at organizations under 25 employees against about 72% at 500 or more, with completion higher for candidates who get hired.
- 2. Mechanical Versus Clinical Data Combination in Selection and Admissions Decisions: A Meta-Analysis ✓ gwern.net Supports the .44 mechanical against .28 clinical average correlation with job performance, the nine samples behind that job-performance comparison, and the paper's classification of group consensus meetings as a holistic combination method alongside individual expert judgment.
- 3. The Structured Employment Interview: Narrative and Quantitative Review of the Research Literature ✓ doi.org Supports two claims: the review's report of Taylor and Small's 2002 meta-analysis of 19 past-behavior interview studies, where anchored rating scales showed validity .35 against .26 and interrater reliability .77 against .73; and the content analysis of 104 interviews from 103 articles published 1997 to 2010, averaging 5.74 of Campion, Palmer and Campion's 15 structure components, with detailed notes described in 19% and statistical prediction in 11%.
- 4. Structured Interviews: A Practical Guide ✓ opm.gov Supports the three-property definition of a structured interview used in the coverage section: the same questions in the same order for every candidate, a common rating scale, and interviewers agreeing in advance what an acceptable answer looks like. Cited as 2008 federal hiring guidance, not statute.
- 5. 29 CFR Part 1607 - Uniform Guidelines on Employee Selection Procedures (1978), sections 1607.16(Q) and 1607.3(A) ✓ govinfo.gov Supports the definition of a selection procedure as any measure, combination of measures, or procedure used as a basis for an employment decision, its reach to the full range of assessment techniques including informal or casual interviews and unscored application forms, and the point that the validation burden attaches where adverse impact appears.
5 sources, numbered by first appearance. Every one was opened and checked against the claim it carries. How Olive sources claims
General guidance for hiring teams. What works at one company and one volume may not transfer to yours.
Olive assesses how a person works with AI. It does not detect AI-written documents, and it never produces a score, a ranking, or a match percentage for a person. Candidates read the same report the employer reads.