Screening
On a 25-Minute Screen, Collect What You Cannot Judge
Judge an AI answer on a 25-minute recruiter screen by its shape, not its technical content: whether it carries a date, a decision, a cost, and something that changed as a result. Correctness belongs to the hiring manager, so ask for two sample answers before the first screen, one that clears the bar and one that does not. Where a claim is beyond you, write it down word for word and pass it up unrated.
The takeThe instruction screeners get is to listen for depth, which cannot be executed and quietly becomes rejecting the answers you could not follow. A screen run by someone outside the domain is a collection instrument, and it is a good one. The rating of content you could not follow is the part that should be dropped. A note saying exactly what a candidate claimed, with the sentence you could not evaluate quoted, is worth more to a hiring manager than a confident number invented on a call you were not equipped to score.
Where Olive fits
Open a role and see what the work shows
Olive returns six findings on how one candidate worked through an occupational assignment with an AI assistant, each finding written or edited by a human reviewer and anchored to a timestamped excerpt from the session. A reader can look at the moment behind each finding rather than take a verdict on trust, and the candidate is granted the identical report.
Assess your shortlistTwenty-five minutes decides who gets read twice
Twenty-five minutes decides one thing: whether a candidate reaches the person who can judge the work. So the question in front of a screener is never whether an AI answer is right. The question is whether it deserves a second reading from somebody who can tell.
Across more than 54 million applications in Ashby's data, from January 2021 through March 2026, recruiter screens pass roughly 35% of the candidates who reach them and handle the highest volume of any stage 1. Later stages convert at 95% after an onsite and 81% at offer 1.
The screen cuts more people than any stage after it because it sees the most. The onsite passes a smaller share still, at 24%; the rounds after it mostly ratify a decision already made 1.
Ashby's figures come from its own customer base, which skews to venture-backed technology employers hiring knowledge roles.
Under US federal selection rules, a short call used to decide who moves on counts as a selection procedure. The Uniform Guidelines adopted by four federal agencies in 1978 define a selection procedure as any measure used as a basis for an employment decision, and say the term reaches informal or casual interviews and unscored application forms 2.
Validation is required only where adverse impact appears. The definition holds either way.
What gets called a structured interview varies widely in how much structure it carries. In a content analysis of 104 interviews reported in the research literature between 1997 and 2010, studies describing a structured interview used an average of 5.74 of the 15 components in the standard framework 3.
The evaluation half was the thinner one. Detailed notes appeared in 19% of those interviews, and a stated rule for combining the evidence in 11% 3.
The two shares come from studies researchers designed and published, so read them as a ceiling for ordinary practice.
Weak structure is not a reason to turn the screen into a technical round. The stage still decides only what a recruiter screen is supposed to decide, and an AI question does not widen it.
What marks does a real episode leave?
A real episode leaves four marks: a date or a stretch of time, a decision the candidate made themselves, a cost attached to getting it wrong, and something that changed afterwards. Not one of the four requires knowing the tool. A generic answer carries adjectives, present tense, and a workflow described in general. Both are audible to an ear that has never opened the software.
The questions that draw the marks out are covered in two AI questions worth their minutes in a fifteen-minute screen.
The four, in roughly the order they go missing:
- A date, or a stretch of time. Last spring. Over about three weeks. The quarter before they left. An answer with no when is an answer about a habit.
- A decision that was theirs. Not what the team did and not what the tool does. What they chose, and what they turned down in order to choose it.
- A cost. What the wrong thing would have cost if it had shipped: a client call, a re-run, a number in a board deck, an hour of somebody else's week.
- A change. What is different now. A step added to a checklist, a prompt retired, a task moved back to a person.
Every one of the four survives a follow-up if the episode happened, and the first follow-up is one question: when was that.
Two published words carry further than an instruction to listen for depth. The AI Fluency Framework, written by Rick Dakan of Ringling College of Art and Design and Joseph Feller of University College Cork, defines both. Discernment is accurately assessing the usefulness of AI outputs, and diligence is taking responsibility and vouching for final products created using AI 4.
No validated test sits behind either word. The framework hands a screener vocabulary, with no scale attached 4.
Get the bar from the manager in two recorded answers
Ask for two answers to the same question, one that clears the bar and one that does not, plus a sentence on what separates them. Fifteen minutes of a manager's time, once per role, settles the bar a screener would otherwise have to invent.
A manager who never wrote the bar down still enforces one, so when the two answers never arrive, the reasons they give for turning down candidates you passed can stand in for them.
The bar belongs to whoever owns the work, and agreeing on it is one of the three things that makes an interview structured at all 5.
The Office of Personnel Management's 2008 guide names all three: the same questions in the same order, a common rating scale, and interviewers in agreement on acceptable answers 5.
Anchors put that third property on paper. Across 19 past-behavior interview studies, interviews using anchored rating scales showed higher validity than interviews without them, .35 against .26, with interrater reliability moving less, .77 against .73 3.
Those studies differ in more than their scales, so anchors travel with whatever else careful teams already do, and two sample answers anchor fewer points than the scales those studies measured.
The case against laying a personal read on top of somebody else's bar is old, and it was made about students. In a 1943 study of University of Minnesota undergraduates admitted in 1939, high school rank plus a college aptitude test correlated .45 with later academic achievement, and the same two predictors plus counselors' intuitive judgment correlated .35 6.
That is a 1943 result about academic achievement, and a screener listening to an AI claim is a long way from it. What survives the trip is the direction and not the size: the judgment added on top did not improve on the bar underneath.
Scott Highhouse, the industrial psychologist who reproduced that pair in 2008, adds that research on differences in validity between interviewers suggests they are attributable entirely to sampling error 6.
The naturally gifted interviewer is not an evidenced category.
Write down the sentence you could not evaluate
When a candidate says something beyond your reach, put their words in the note exactly as they said them, mark the line not evaluated, and name the claim instead of your impression of it. A manager reading a verbatim sentence can judge the claim itself. A manager reading that the candidate seemed to know the tooling has to run the screen again.
A note that records what the candidate said, without your guess at the reason behind it, still reads plainly in a file a year later.
What goes in a note has been measured on the hiring side. Shanshi Liu and colleagues text-mined the post-interview notes written about 7,650 candidates hired at a large Chinese internet technology company.
The number of job-related capabilities an interviewer named in the notes tracked later performance and promotions, and ran the other way against turnover 7.
The study could only observe candidates who passed, and the employer rated its own people. The effect runs at roughly 2 percent of a performance measure per standard deviation of how closely the notes matched the job analysis.
What survives is narrow and useful: naming the capability the job calls for is the part of a note that tracked how people did later.
Gathering evidence and combining it are different jobs, and the evidence favors splitting them. In a meta-analysis of selection and admissions studies, applicant data combined by a formula correlated .44 with job performance against .28 for the same kinds of data combined by expert judgment 8.
Nine studies and about 1,392 people sit behind the mechanical side, which is thin. A formula here can be as plain as adding up a written scorecard, and the authors say so 8.
The scorecard will still want a number in the box. Adding a not-observed option to the scale and counting how often it fires is what stops an unevaluated line from being scored low by default.
Handing everything up stops being a screen
A screener who rates nothing passes the whole problem to the hiring manager, and the busiest stage in the loop stops filtering. Recruiter screens pass roughly 35% of the people who reach them 1, and that cut is the stage's job. Collecting an answer still leaves a decision to make about it.
So split the output of the call in two. Reject what still fails the shape test after the follow-ups and can be named: no when anywhere, no decision the candidate owned, no cost, nothing different afterwards.
Send forward what was merely unfamiliar to you, with the quotation attached.
The manager on the other end may be no better equipped. Some hiring managers do not use these tools themselves, and getting them to judge AI-assisted work is its own problem. A quote handed to somebody equally unequipped only moves the guess.
Ask for the two sample answers before the next screen goes on the calendar, and ask what separates them in one sentence. Then spend the call getting the candidate's own sentence down, word for word.
Twenty-five minutes is enough to record that sentence, and never enough to grade it.
Common questions
How do you judge a technical AI answer when you are not technical?
Judge the structure of the account rather than the content of the claim. Ask when it happened, what the candidate decided, what a mistake would have cost, and what changed afterwards. Those four are audible without any knowledge of the tool. For the content itself, work from two sample answers the hiring manager recorded in advance, one that clears the bar and one that does not.
What should a recruiter write in a note about an AI answer?
The candidate's own words, then the capability the words point at, then a flag on anything you could not assess. A verbatim sentence lets a hiring manager judge the claim itself; a summary makes them rerun the screen. Name the capability the role actually calls for, since in one study of hires at a single firm, that was the part of a note that tracked later performance. Mark the unevaluated line explicitly rather than leaving it blank, because a blank reads as an absence and a flag reads as a handoff.
Can a recruiter reject a candidate on a weak AI answer alone?
Reject on what can be named, never on what was merely unfamiliar. An answer that, after a follow-up, still has no date anywhere, no decision the candidate owned, no cost and nothing different afterwards has failed a test the screener is qualified to run, and the reason fits in one line of the note. An answer that was specific and technical and lost you at minute three has failed nothing. Send that one forward with the quotation attached and let the person who owns the bar read it.
What if the hiring manager will not record two sample answers?
Take the bar from their rejections instead. Every candidate the manager turns down after a screen you passed carries a reason, and three or four of those reasons name the bar more precisely than a meeting would. Ask for the reason in writing each time, collect them for a fortnight, and write the pass and fail examples yourself from what the manager already said. Then send the pair back for a yes or a no, which is a smaller ask than drafting them from nothing.
Is 25 minutes too short to ask about AI at all?
No, provided the screen collects rather than grades. One question and two follow-ups fit inside four minutes and produce a quotable episode or an obvious absence, which is all the stage is meant to establish. What does not fit is an evaluation of whether the technical claim is correct, and attempting one inside 25 minutes is how a screener ends up rating candidates on how closely their answer matched something they already understood.
Does a short informal screen count as a selection procedure?
Under the Uniform Guidelines on Employee Selection Procedures, adopted by four US federal agencies in 1978, a selection procedure is any measure used as a basis for an employment decision, and the definition explicitly reaches informal or casual interviews and unscored application forms. Validation obligations attach only where adverse impact appears, so the definition is not itself an audit requirement. What it removes is the assumption that a quick conversation sits outside the rules that cover a scored test. Employment counsel should confirm how this applies to a specific process.
References
- 1. Recruiting Operations Benchmarks | 2026 Talent Trends Report ✓ ashbyhq.com Supports the claim that recruiter screens pass roughly 35% of the candidates who reach them while handling the highest volume of any stage, that the onsite passes 24%, and that later stages convert at 95% post-onsite and 81% at offer, measured across over 54 million applications and 93,000 jobs from January 2021 through March 2026, and the reading that later rounds mostly ratify a decision already made.
- 2. 29 CFR Part 1607 - Uniform Guidelines on Employee Selection Procedures (1978), sections 1607.16(Q) and 1607.3(A) ✓ govinfo.gov Supports the claim that a short recruiter screen is a selection procedure under the 1978 Uniform Guidelines, whose definition reaches informal or casual interviews and unscored application forms, and that a validation burden attaches only where adverse impact appears.
- 3. The Structured Employment Interview: Narrative and Quantitative Review of the Research Literature ✓ doi.org Supports the average of 5.74 of 15 structure components across 104 interviews reported between 1997 and 2010, the evaluation-side shares (detailed notes .19, statistical prediction .11), and the anchored-rating-scale figures the review reports from Taylor and Small (2002) across 19 past-behavior interview studies: validity .35 against .26 and interrater reliability .77 against .73.
- 4. Framework for AI Fluency ✓ ringling.libguides.com Supports the framework's definitions of Discernment as accurately assessing the usefulness of AI outputs and Diligence as taking responsibility and vouching for final products created using AI, and the point that it is a set of definitions with no validated instrument attached.
- 5. Structured Interviews: A Practical Guide ✓ opm.gov Supports the three properties of a structured interview named in the guide: the same questions in the same order, a common rating scale, and interviewers in agreement on acceptable answers.
- 6. Stubborn Reliance on Intuition and Subjectivity in Employee Selection ✓ edbatista.com Supports the Sarbin (1943) pair reproduced in Table 1, high school rank plus a college aptitude test at .45 against the same two predictors plus counselors' intuitive judgment at .35 for University of Minnesota undergraduates admitted in 1939, and Highhouse's statement that variance in interviewer validity is attributable entirely to sampling error.
- 7. Predictive Validity of Interviewer Post-interview Notes on Candidates' Job Outcomes: Evidence Using Text Data From a Leading Chinese IT Company ✓ frontiersin.org Supports the 7,650 hired candidates whose post-interview notes were text-mined, the finding that the number of job-related capabilities named in the notes was positively related to performance and promotions and negatively related to turnover, the roughly 2 percent per standard deviation effect, and the restriction to candidates who passed the interview.
- 8. Mechanical Versus Clinical Data Combination in Selection and Admissions Decisions: A Meta-Analysis ✓ gwern.net Supports the .44 mechanical against .28 clinical correlation with job performance, the nine studies and 1,392 people behind the mechanical estimate, and the authors' point that a mechanical rule can be as plain as adding up a written scorecard.
8 sources, numbered by first appearance. Every one was opened and checked against the claim it carries. How Olive sources claims
General guidance for hiring teams. What works at one company and one volume may not transfer to yours.
Olive assesses how a person works with AI. It does not detect AI-written documents, and it never produces a score, a ranking, or a match percentage for a person. Candidates read the same report the employer reads.