Pipeline
A Vague AI Answer Is an Open Question, Not a Rejection
A candidate with a vague AI answer and strong evidence everywhere else should go forward, with the vagueness written down as a named open question. Reject at the recruiter screen only on facts another reader can check: an unmet hard requirement, a pay or location conflict, a claim walked back when asked about it directly. One thin answer is none of those. Then track whether the next round ever closes the question you sent.
The takeThe screen gets treated as the place to cut because cutting there is cheap, and cheapness is exactly what makes it the wrong stage to spend a rejection at. A candidacy ends on twenty-five minutes of talk about a subject that usually sits outside the screener's own field. Hold the rejection until the evidence is worth it. A screen that stops people only on checkable facts stops fewer of them, and the ones it stops stay stopped.
Where Olive fits
Open a role and see what the work shows
Olive is priced per attempt rather than per seat, and one attempt returns six evidenced findings about a single candidate, each anchored to a timestamped moment in the session and written or edited by a person. Ten attempts a month are free, so a pilot can run beside the current loop on candidates whose screens left a question open.
Assess your shortlistWhy is the recruiter screen the wrong place to spend a rejection?
Because it pairs the least evidence with heavy cutting. A recruiter screen is twenty-five minutes of talk, run by somebody assessing a skill that sits outside their own field, and it ends more candidacies than any round after it. Spending a rejection there puts the heaviest call in the process where the judgment behind it is weakest.
Ashby's 2026 benchmark report puts recruiter-screen passthrough at roughly 35% of the candidates who reach that stage 1. The report is built on more than 54 million applications logged from January 2021 through March 2026.
The two late stages it reports filter far less. In the same data, 95% of the candidates who reach the post-onsite stage pass it and 81% pass through the offer stage, which says the decision was effectively made before the onsite ended 1.
Heavier filtering happens before any of that, at application review, which those numbers do not cover.
The methods that scale to the top of a funnel are also collectively weaker. In a 2024 paper in the Journal of Applied Psychology, Berry and colleagues modelled a battery of volume-friendly tests, weighted as favourably as arithmetic allows, and reached a validity ceiling of .51 2.
Letting a structured interview into the same model raised that ceiling to .61 2. No real screen is weighted that way, and the model contains no resume review at all, so the pair is a comparison and never a target.
The round that cuts hardest after application review knows the least.
None of that argues for abolishing the screen. The argument is about which errors the screen is allowed to make.
What a recruiter screen decides that the application cannot is one named claim tested twice. A soft read on AI is not that claim, unless you made it the claim before you dialed.
Ask one more question before you call the answer vague
Before writing an answer off as vague, ask for one instance of the work, because vagueness has at least three causes and a screen usually cannot tell them apart. The question may have been too broad to answer well. The candidate's best examples may sit behind a confidentiality agreement. Or the habit may not be there at all.
Only the third is evidence about the person, and one follow-up separates it from the other two.
Ask for one instance instead of a description.
What was the last thing handed to a model, what came back, and what changed before it went anywhere.
A person with the habit can usually name a specific piece of work, with any confidential detail left out. A person without it stays general, and gets more general as you push.
A screener working outside the candidate's field has a shape to listen for rather than a technology to assess. Start from judging the shape of an AI answer, not the technology.
A freeform chat can subtract. In a 2013 paper in Judgment and Decision Making, 76 undergraduates predicted classmates' semester grades: predictions made after an unstructured interview correlated .31 with the real result, against .65 for prior grades alone 3.
Students guessing at grades are not managers predicting performance, and the size of that drop does not transfer. The mechanism does.
Low-diagnostic talk crowds out information that was already good, and interviewers build a coherent read out of almost anything. In the same work, 96 of 169 participants chose an interview whose answers were generated at random over conducting no interview at all 3.
The feeling has to become a fact before it becomes a decision.
"Asked for one worked example, got a list of tools" is checkable by anybody who reads it. "Seemed vague on AI" is not, and it will not survive being read back to you in three months.
What can you reject on at this stage?
Three things, all of them checkable by somebody who was not on the call. A hard requirement the role genuinely carries and the candidate does not meet. A pay, location or work-authorization conflict that no later round can resolve. A claim the candidate walked back when asked about it directly.
Each of the three leaves a record another reader can agree or disagree with. A soft read cannot be checked by anyone, only shared.
Adding an impression on top of evidence you already trust can make the prediction worse. Scott Highhouse's 2008 review in Industrial and Organizational Psychology reproduces Sarbin's 1943 admissions study at the University of Minnesota: high school rank plus an aptitude test correlated .45 with later academic achievement.
The same two predictors plus counselors' intuitive judgment correlated .35 4.
Sarbin's numbers came from admissions in 1939, not from hiring, and one study settles nothing by itself.
The division of labour the selection literature draws from results like this is the part worth keeping. Interviewers are good at collecting information and poor at combining it, and the vague-answer moment is pure combination.
Error costs decide the rest. The question underneath is false positives, false negatives, and which one you pay for. A rejection at the screen buys protection against a bad hire the loop would probably have caught anyway, and pays for it with a miss nobody counts.
One case flips the answer. If the posting stated the AI requirement as a hard line, and the same line reached everyone who got a screen, it belongs on the first list and the rejection is clean. A rule invented after the call is not that.
Write the open question so the next round can close it
Name the capability that went unobserved and the exact question that produced the thin answer. Not a rating, not an impression, and not a theory about why the candidate hedged. The note's job is to hand the manager one specific thing to test in a round that has forty minutes and a domain expert sitting in it.
The capabilities a note names carry signal of their own. Shanshi Liu and colleagues text-mined the post-interview notes on 7,650 candidates hired at one large Chinese technology company between 2016 and 2018. The number of job-related capabilities the notes actually named tracked later performance, promotions and retention 5.
Around 2 percent of a performance rating per standard deviation of matching is a small effect, and everyone in that sample had already passed the interview 5. Nobody who was rejected is in it, and the finding is not evidence that taking notes helps by itself.
What was measured is whether a note named the capability the job called for. A narrower instruction than "write good notes," and a more useful one.
Wording carries the risk. "Could not give an instance of checking a model's output" is a fact about the call. "Probably does not really use AI" is a guess about a person, and it is the line that gets read back to the candidate later.
If the scorecard forces a number, the honest entry is that the round did not observe it. Adding not observed to the scale and counting how often it fires is what makes that entry available. A forced 2 puts a guess into a field that later gets averaged.
Check whether the loop ever closes what the screen forwards
Pull the last twenty screens that advanced with an open question and read what happened after each one. In how many did a later round produce an answer? If the number is near zero, the loop is the problem rather than the screen, and passing forward has been turning into a rejection somebody else makes on no evidence either.
Two outcomes look identical from the recruiter's chair. In the first, the manager reads the note and tests the question in round two. In the second nobody reads it, and the candidate is declined for AI depth that no round examined.
The second one is diagnosable from the rejections themselves rather than from a calibration meeting, which is the point of getting the manager's bar from their rejections.
Ratings are unstable exactly where the decision sits. Around 38% of scorecard pairs in Ashby's data differ by at least a point between interviewers, and nearly half of those differences fall between a 2 and a 3, crossing the yes-or-no line on a four-point scale 1.
Disagreement is not error, and that dataset holds no outcome that could say who was right. One point on a four-point scale is a thin thing to end a candidacy on, whoever wrote it.
A manager with fewer hours than you have pays for every ambiguity that gets forwarded, and a screen forwarding all of them has stopped being a screen. That 35% passthrough is the number that would move, and nothing cited here measures what happens to a loop when it does.
Cap it: one open question per candidate, and only one you would change the decision over.
The file leaves your desk with a line on it that was not there this morning. Six weeks from now, somebody either answered that line or nobody did. Both are findings. Only one of them is about the candidate.
Common questions
Is passing a weak AI answer forward just avoiding the decision?
No, as long as the ambiguity is named. Avoiding the decision looks like an advance with a clean note and no mention of the thin answer, which hands the manager a candidate who reads as uniformly strong. Naming it looks like one line: the question asked, what came back, and what a later round should test. The second version makes the decision, and makes it at the stage that can actually carry it. It also gives you something to audit six weeks later.
What if AI skill is the whole point of the role?
Then it stops being a soft signal and becomes a requirement, and the screen can reject on it. Two conditions have to hold first. The posting has to have stated the requirement before anyone applied, and the same question has to have reached every candidate who got a screen. Where both hold, a thin answer is an unmet requirement and the rejection is checkable by anybody who reads the file. Where only one holds, a rule was invented partway through, and it will not survive the first candidate who asks why.
How should the open question be worded on the handoff?
As what was asked, what came back, and what is still unknown. "Asked for one recent case of catching a model error; described tools rather than a case; unresolved whether verification is a habit" gives the next interviewer a specific thing to test in ten minutes. Skip any sentence explaining why the answer was thin. A note about the person rather than the call is a guess that gets quoted back in a rejection conversation, and it is worth nothing to the round that follows.
Does forwarding ambiguity overload the hiring manager?
It can, which is why passing forward comes with a cap. One open question per candidate, chosen because the answer would change the decision, keeps the handoff readable. Four questions per candidate is a screen that has stopped deciding anything, and the manager will start skipping the notes, which costs more than the original rejection would have. If the number of open questions keeps growing, the fix is a better screen question rather than a lower bar for passing forward.
Can a second screen resolve it instead of passing forward?
Rarely worth the calendar. A second conversation with the same screener adds another unscored talk on a subject outside that screener's field, and the candidate now answers a question they have had a week to prepare. Where the doubt is genuinely about a checkable fact, a two-line email closes it faster. Where it is about how the person works, the round that can see that is the one with a domain expert in it, so the handoff is the cheaper path.
Do you tell the candidate the AI answer was thin?
Yes, and in a form that does not script the next round. Tell them the process is continuing, and that their AI answer left a question a later stage will go further into. What you hold back is the coaching, not the record: handing over the exact question and the answer you were listening for turns the next round into a rehearsal of it. Write the note itself as though the candidate will read it, because at some point they should. A line you would rewrite before showing it to them was never a fact about the call in the first place.
References
- 1. Recruiting Operations Benchmarks | 2026 Talent Trends Report ✓ ashbyhq.com Supports two claims: recruiter screens pass roughly 35% of candidates who reach them against 95% post-onsite and 81% at offer, across over 54 million applications and 93,000 jobs from January 2021 through March 2026; and that around 38% of scorecard pairs differ by at least one point, with nearly half of those differences falling between a 2 and a 3 on a 1 to 4 scale.
- 2. Insights from an Updated Personnel Selection Meta-analytic Matrix: Revisiting General Mental Ability Tests' Role in the Validity-Diversity Tradeoff ✓ filiplievens.squarespace.com Supports the claim that a screening stage built only from methods that run at volume tops out lower than one allowed a structured interview: a modelled validity ceiling of .51 against .61, Pareto-optimally weighted, with no resume screening in the model.
- 3. Belief in the unstructured interview: The persistence of an illusion ✓ sjdm.org Supports the dilution claim: 76 undergraduates predicting classmates' semester grades scored .31 after an unstructured interview against .65 from prior grades alone, and 96 of 169 participants preferred an interview with randomly generated answers to no interview at all.
- 4. Stubborn Reliance on Intuition and Subjectivity in Employee Selection ✓ edbatista.com Supports the claim that layering an intuitive judgment on top of evidence can lower accuracy, via the Sarbin (1943) row of Highhouse's Table 1: high school rank plus a college aptitude test correlated .45 with the criterion, and the same two predictors plus counselors' intuitive judgment correlated .35.
- 5. Predictive Validity of Interviewer Post-interview Notes on Candidates' Job Outcomes: Evidence Using Text Data From a Leading Chinese IT Company ✓ frontiersin.org Supports the claim that a note naming the capabilities the job requires carries signal of its own: across 7,650 hired candidates, the count of job-related capabilities named in interviewer notes tracked performance, promotions and turnover, with roughly a 2 percent performance rise per standard deviation of matching, among candidates who had already passed.
5 sources, numbered by first appearance. Every one was opened and checked against the claim it carries. How Olive sources claims
General guidance for hiring teams. What works at one company and one volume may not transfer to yours.
Olive assesses how a person works with AI. It does not detect AI-written documents, and it never produces a score, a ranking, or a match percentage for a person. Candidates read the same report the employer reads.