When Everyone Passes, Check the Answer Against the Next Stage

Why does a 100% pass rate answer neither question?

Because both explanations produce the same number. A question whose good answer has become public shows up as a column of passes in the screening notes. So does an applicant pool that genuinely got better. Nothing in that column tells them apart.

A tally of forty passes out of forty counts your own decisions, not the people.

Motivated answering can move structured-interview scores. Buehl, Melchers, Macan and Kuehnel had 111 German psychology undergraduates sit the same 20-question structured interview twice. Once they were told to answer honestly, once to answer as a job applicant would. Mean ratings rose from 3.20 to 3.62 under the applicant instruction ¹.

The gain was not spread evenly. Bigger rises went to people who could work out which quality each question was probing, though the link was weak: a correlation of .16, tested one-tailed ¹. A candidate may pick that up from a coaching video, a forum thread, or one earlier screen at a company asking the same thing, though the study did not test where it comes from.

Those students sat a research interview with nothing at stake. Read the direction, not the size.

The other explanation has evidence behind it too. Handshake compared the resumes graduating seniors uploaded to its platform: the class of 2026 against the class of 2022. AI skills were named more than nine times as often, and two-thirds of those mentions now come from students outside computer science ².

But a mention is a count of a word, not of a skill. How many people write AI on a resume and how many can work with one are different measurements. The report measures only the first ².

Nothing inside the screen will settle which of the two you have.

Sort the last fifty screens, then look at the next stage

Take the last fifty screens where the AI question was asked and the candidate went at least one stage further. Sort them into piles by what each answer contained, rather than by whether you marked it a pass. Then look at what happened to each pile afterwards: interview scores, work-sample outcomes, offers.

A question still doing work leaves a difference in what happened later.

Three piles are usually readable straight from notes that already exist:

• An episode. One task, one month, a decision they made and what it cost them.
• A practice. A rule they follow, described with no occasion attached to it.
• A tool. The model, the plug-in, the workflow, and no judgment anywhere.

Finding no difference between the piles is worse news than it sounds. Across more than 54 million applications, Ashby found recruiter screens passing roughly 35% of the candidates who reached them while post-onsite stages converted at 95% ³. In that data, the recruiter screen filters far more than the stages after it.

A screen that separates nobody has not become harmless. The screen's minutes are spent either way, and a row of identical marks is a first round producing no information at all, narrowed to one line of the script.

Run the same read on the reasons rather than the outcomes, because the criterion actually in use hides in the note. Auditing what screeners advanced on is this exercise done at the resume stage, and it is built to show whether an unwritten criterion is doing the work.

How does a screening question turn into a password?

By having one good answer that becomes public. A question filters for as long as the population cannot produce its answer on demand. The answer stops carrying information about the person once the wording circulates on a forum, or once a model can infer the expected shape from the job title alone.

On the page the answer looks exactly the same.

The decay from signal to password has been measured on cover letters. Cui, Dias and Ye studied an AI cover-letter tool introduced on a large online labour platform. After it arrived, letters matched job posts more closely and callbacks rose.

Then the correlation between how closely a letter matched the post and whether it drew a callback fell by 51% ⁴.

Employers on that platform moved weight onto other signals, prior work history among them ⁴.

The study is a preprint, and its comparison is a difference-in-differences on one platform's data. The 51% is a fall in a correlation rather than in anyone's callback rate ⁴. The mechanism is the part that may transfer to a screening question: a signal everybody can produce stops being read.

Rotation restarts the decay without touching the mechanism. A replacement question with one good answer saturates on the schedule the old one did. The quarter it buys costs comparability between the candidates screened in it.

Anchoring the question to their work instead of rotating it every quarter is the repair that does not expire.

Ask for the episode instead of rotating the question

An episode is harder to pre-generate than a policy, because it has to come from a job the candidate actually held. Ask what the candidate did on one named task in one named month: what the model produced, what they changed about it, and what it cost the week they were wrong. A model will write a convincing policy about verification.

No model supplies the Tuesday.

The format has evidence under it. Levashina and Campion found past-behaviour questions more resistant to faking than situational questions about what a person would do, in the sixth study of the paper that built the interview faking behaviour scale, an experiment with undergraduates ⁵.

The same experiment found something less expected: follow-up questioning increased faking ⁵.

Probing is not automatically a hardening move. A follow-up has to ask for a detail that would have to be invented on the spot and could later be checked, rather than a fuller version of the same claim.

The objection to asking for an episode is rehearsal. Publish the format, and candidates arrive with one ready.

Maurer, Solamon and Lippstreu compared coached and uncoached samples across 146 public safety incumbents on a situational panel interview, and found predictive validity and reliability higher among the coached ⁶. The coaching in that study was content-focused, aimed at getting relevant material across accurately, the questions were situational rather than past-behaviour, and the two samples were compared rather than randomly assigned ⁶.

A prepared population is not the same thing as a saturated question, and treating rehearsal as gaming costs you the candidates who did the reading. If the check said the answer still predicts, the repair is the bar, not the wording.

What the fifty-screen check cannot settle

Whether the question works on the population it never sees. The check runs only on candidates who reached a later stage, so the people the screen stopped contribute nothing to it. The answer quality that would separate best may sit entirely outside the sample.

A narrowed range usually weakens the relationships measured inside it.

The published literature carries the same limit, which is worth knowing before somebody dismisses your fifty screens over it. Sackett, Zhang, Berry and Lievens report that 98% of the studies behind one work-sample meta-analysis were concurrent, measured on people already hired and doing the job rather than on applicants ⁷.

Few published coefficients describe an applicant pool either, so none is a safe benchmark your own numbers can be graded against.

Your data can do something narrower and more useful: say whether one column in your own notes moved with anything downstream. Proving a hiring change actually worked needs that record kept before the change, not reconstructed after it.

Maybe the recruiter screen was never the stage that sorts, and checking it is aimed at the wrong place. A question everybody clears inside it would then be exactly what a well-built funnel produces. Which funnel metrics still mean anything now that candidates use AI is the wider form of that objection.

Forty out of forty is the same tally it was at the top of this page. What you can turn it into is different: forty decisions of yours, set beside forty later outcomes nobody has put next to them yet.

The next fifty screens start Monday either way.

See a sample report >