Pipeline
Bring Them Back Only If the Round Can Change a Decision
Candidates who finished your loop before you added the AI exercise come back only if a poor result would actually stop one of them. If you would advance them anyway, the round decides nothing in this requisition, and another hour of their time buys a comparison nobody will use. If a poor result would stop somebody, send the same brief on the same clock and say the step was added late. Running it on the one candidate still in flight and debriefing everyone together fails either way.
The takeThe argument that eats a whole meeting is fairness, and fairness is not where the decision lives. Ask instead for the sentence nobody wants to write: a candidate who does this on the exercise does not advance. If the hiring manager cannot finish it, the round was never a gate, and four people are about to spend an evening producing a column no one reads. Feeling behind on AI as a team is a good reason to change next quarter's loop and a poor reason to reopen this one.
Where Olive fits
Open a role and see what the work shows
Olive is priced per attempt rather than per seat, so a round added mid-process costs one attempt for each candidate who takes it, and ten attempts a month are free. Each attempt returns six evidenced findings about one candidate, written or edited by a person, and the candidate is granted the same report.
Assess your shortlistWhat would a bad result actually do to those four?
One question decides whether the four come back, and the rest is logistics. If a poor result on the new exercise would not stop any of the four, the round is not a gate in this requisition, and bringing them back buys a comparison nobody will act on. If a poor result would stop one of them, the round is a gate, and a gate reaches every candidate it applies to.
Put the test to the hiring manager in writing, before anything reaches a candidate. Ask them to finish one sentence: a candidate who does this on the exercise does not advance.
The answer usually arrives as something else.
A wish to see how they would do is information, and information is a fine reason to build an exercise for the next requisition. It is not a reason to ask four finished candidates for another hour.
The same test cuts the other way elsewhere in the loop, where the round to drop is the one that has never changed an outcome. A round added in the middle of a live process clears a higher bar than that, because the people it lands on have already given you everything the loop asked for.
Why do late rounds so rarely change an outcome?
Because most of the filtering happens before the last rounds run. By the time a candidate has cleared every scheduled stage, the decision is largely made and the remaining stages record it rather than reach it. One applicant-tracking vendor's funnel data points the same way, and it is worth holding in mind before deciding that one more exercise will separate five people you have already met.
Ashby's 2026 benchmarks read over 54 million applications across 93,000 jobs, from January 2021 through March 2026 1. Recruiter screens pass roughly 35% of the candidates who reach them 1.
Post-onsite stages pass 95%, and the offer stage 81% 1.
A 95% passthrough is not evidence that an onsite is easy. It means the round changed almost nobody's outcome.
Read all three as stage-to-stage rates among candidates who got that far, because that is what they are 1. They do not multiply into an end-to-end number, the heaviest cut happens earlier at application review, and every employer names its own stages inside its own system.
Your four are the strongest version of that pattern. They cleared every stage that existed when they applied, so a new round is being asked to reverse a judgment the whole loop already produced.
None of that makes an AI exercise a bad idea. The funnel data moves the design question, whether a separate AI-fluency round earns its place in the loop, back to where it belongs, before candidates are standing in it.
Run it on all five, or leave it out of the debrief
The move to refuse is the hybrid: give the exercise to the one candidate still in flight, skip it for the four who finished, then sit all five in one debrief and compare. That comparison is not real, and everybody in the room sees the hole in it the moment the exercise comes up and four scorecards are blank.
Structure is what makes a comparison mean anything. The US Office of Personnel Management's guide defines a structured interview by three properties.
Every candidate is asked the same questions in the same order. Every candidate is rated on a common scale. The interviewers agree in advance on what an acceptable answer looks like 2.
Take one of the three away and the interview no longer meets that definition.
The guide is federal practice guidance from 2008, written for merit-system hiring, and it binds no private employer 2. Borrow the definition anyway, because it names exactly what an uneven round breaks.
So pick one.
Either the exercise appears on all five scorecards, or it appears on none of them and the fifth candidate's result stays a note about that person rather than a column in the grid.
The looser question of whether every candidate needs exactly the same questions has a more forgiving answer, because follow-ups can differ where the planned questions and the written anchors do not. A whole round is not a follow-up.
What the record shows about a round added mid-process
A step added in the middle of a live process leaves a mark that is easy to read later and hard to explain. The US federal Uniform Guidelines, a 1978 regulation, do not ask whether the exercise was scored, formal, or called an exercise at all. The Uniform Guidelines define a selection procedure as any measure used as a basis for an employment decision, down to informal or casual interviews and unscored application forms 3.
That definition is not an audit mandate, and the Guidelines attach a validation burden only where adverse impact appears 3. What it settles is narrower: an exercise that feeds a decision is a selection procedure from the day it starts feeding one.
The 1978 text names no software, and it reaches an AI exercise by analogy.
A later reader reconstructs the sequence out of your applicant tracking system. The round appeared on a date, it reached some candidates in the requisition and not others, and no written rule says which.
Fix the record today. Write down the date the exercise was added, the reason it was added, and the candidates it applied to.
Settle when an informal exercise becomes a selection procedure you have to treat as a test before the next requisition rather than during this one. Employment counsel is who confirms how any of this lands on a particular process.
How do you ask four finished candidates for another hour?
Plainly, with the same brief and the same clock the fifth candidate is getting. Say the step was added after they finished, say why it was added, and give the date the requisition closes. An invitation explaining that the format is identical for everyone may read as fairer than one that arrives with no account of where it came from.
Basch and Melchers tested how the way a format is introduced changes its reception, in 2019, on asynchronous video interviews: an explanation stressing the format's higher standardization improved fairness perceptions, while an explanation stressing scheduling flexibility improved perceived usability instead 4.
The abstract gives no sample size and no measure of how large either difference was, and the study predates AI scoring of a recording 4. Take the direction and leave the magnitude alone.
The four have already spent the loop's hours, so this lands on a budget that was full before you opened it. Cap the exercise at what a first-round version would take, and count it against the whole loop's total in candidate hours rather than treating it as free because the schedule is already broken.
Decide inside the week. Four people are holding a process open while somebody works out whether a round matters.
Nothing has been graded on this exercise yet, so nobody can say what a bad result would cost. That gets offered as the argument for running it on all four regardless, and it is the same wish for information the gate test already answered: the next requisition is where the exercise learns what it separates.
Expect candidate goodwill to be argued on both sides, and ask what kind of study stands behind each claim.
Hausknecht, Day and Thomas's meta-analysis of applicant reactions found that across a number of the fairness-to-behavioral-intentions relationships it examined, correlations tended to run stronger in hypothetical settings than in authentic ones 5. Of the samples it pooled, 53.5% came from people who were not actual applicants, and 90% of the hypothetical-context studies used college students 5.
Their conclusion is that the role of fairness may be overestimated in hypothetical selection contexts 5. Most of the measured outcomes were intentions rather than anything a candidate did.
Now look again at the four finished loops. Each one records a process that ended before the exercise existed, and the honest version of that record now says so in a line: the round was added on this date, for this reason, and it reached these candidates.
The four either sat it or they did not, and the file says which.
Common questions
Do you have to bring back candidates who finished before the round existed?
Only where a poor result would change what happens to them. If the exercise is a gate, it reaches every candidate still under consideration in that requisition, including the ones who finished before it existed. If a poor result would stop nobody, the round decides nothing here and skipping it for everyone is the consistent choice. What fails in both directions is running it on some and comparing all of them together, because a blank line on a scorecard is not a low score.
Can you compare a candidate who took the AI exercise with one who did not?
Not on that line. A rating scale means something only when every candidate was rated on it, so an exercise four people never sat produces no comparable result for them. Leave it off the grid and treat the one result as a note about that person, or give the exercise to everybody still in the running and score every one of them. In the debrief, say out loud which stages each candidate completed before anyone starts ranking.
What if a finished candidate declines the extra round?
Decide what a decline means before the invitation goes out, and write it down. If the exercise is a gate, declining ends the candidacy the way declining any required stage does, and the invitation should say so plainly. If it is not a gate, a decline costs nothing and the candidate is right to protect the time. A rule invented after one person says no applies to one person, which is the problem you started with.
Does adding a round mid-process create legal exposure?
The US federal Uniform Guidelines attach a validation burden only where a procedure has adverse impact, so the addition alone is not the issue they describe. Applying it unevenly and keeping no record of why is the pattern worth avoiding. Under those Guidelines a selection procedure is any measure used as a basis for an employment decision, informal interviews included, so the exercise counts whether or not anybody scores it. Record the date the round was added, the reason, and which candidates it reached, then hold one rule across the requisition. Employment counsel confirms how this applies to a particular process.
Should the exercise go to candidates already rejected?
No. A round added today cannot reopen a decision already made and communicated, and reopening only the rejections somebody happens to remember is the same inconsistency pointed backwards. If the exercise turns out to matter, it belongs in the next requisition's loop from the start. Where a rejected candidate asks to be reconsidered, answer that on its own terms rather than by inventing a stage for one person.
References
- 1. Recruiting Operations Benchmarks | 2026 Talent Trends Report ✓ ashbyhq.com Supports the claim that late stages ratify rather than filter: over 54 million applications across 93,000 jobs from January 2021 through March 2026, recruiter screens passing about 35% of candidates who reach them against 95% post-onsite and 81% at offer, quoted as stage-to-stage rates that do not multiply.
- 2. Structured Interviews: A Practical Guide ✓ opm.gov Supports the three properties an uneven round breaks: same questions in the same order, a common rating scale, and agreement in advance on what an acceptable answer looks like. Cited as 2008 federal practice guidance that binds no private employer.
- 3. 29 CFR Part 1607 - Uniform Guidelines on Employee Selection Procedures (1978), sections 1607.16(Q) and 1607.3(A) ✓ govinfo.gov Supports the claim that an added exercise counts as a selection procedure whether or not it is scored, since the definition covers any measure used as a basis for an employment decision down to informal or casual interviews and unscored application forms, with validation attaching only where adverse impact appears.
- 4. Fair and Flexible?! Explanations Can Improve Applicant Reactions Toward Asynchronous Video Interviews ✓ scholarworks.bgsu.edu Supports the claim that saying the format is identical for everyone changes how the invitation lands: an explanation stressing higher standardization improved fairness perceptions, while one stressing scheduling flexibility improved perceived usability. Cited as a direction only, since the abstract carries no sample size or effect sizes.
- 5. Applicant Reactions to Selection Procedures: An Updated Model and Meta-Analysis ✓ ecommons.cornell.edu Supports holding the goodwill argument loosely: 53.5% of samples were not actual applicants, 90% of hypothetical-context studies used college students, and the authors conclude the role of fairness may be overestimated in hypothetical selection contexts.
5 sources, numbered by first appearance. Every one was opened and checked against the claim it carries. How Olive sources claims
General guidance for hiring teams. What works at one company and one volume may not transfer to yours.
Olive assesses how a person works with AI. It does not detect AI-written documents, and it never produces a score, a ranking, or a match percentage for a person. Candidates read the same report the employer reads.