Read Forty Screening Notes to Find the Criterion Nobody Wrote Down

Why Can't an Advance Rate Name a Criterion?

An advance rate tells you how many applications survived and nothing about what any one decision rested on. Criteria live in the reason a screener typed into the field beside the candidate's name, which no funnel report reads. Two teams with the same advance rate can be running two different screens.

Ashby put recruiter-screen pass-through at roughly 35 percent, against 95 percent after the onsite and 81 percent at offer ¹. The measurement runs across more than 54 million applications and 93,000 jobs from January 2021 through March 2026.

Those are stage-to-stage rates among candidates who reached each stage, and the heaviest filtering sits earlier still, at application review, which the report does not measure.

The direction is the usable part. The onsite mostly ratifies, so a criterion that enters at the screen may meet little independent filtering later.

So the screen is the cheapest place in the loop to be wrong at volume.

But an outcome analysis cannot say why. Whether the screen is throwing away the wrong people is a validity question, answered a year later with performance data on the people who got in.

A reason audit answers a nearer one: whether the criteria the team wrote down are the criteria the team is using.

Take Forty Decisions and Code Every Stated Reason

Forty is enough for an afternoon. Take one requisition's last month of decisions, twenty advanced and twenty rejected, and copy each screener's written reason into a sheet beside the criteria list the team agreed before applications opened. Code every reason three ways: which written criterion it names, whether the application holds something that meets that criterion, and whether the note points at it.

The third column is the one that does the work.

• Names a criterion. The reason maps to a line on the list, or it maps to nothing on the list.
• Supported in the application. Something in the file meets that criterion, whether or not the screener mentioned it.
• Pointed at. The note says where: a project, a tool named with the task attached, a shipped thing, a check the candidate described running.

A reason can name a criterion, be supported, and still point at nothing. Rows like that are the ones where a screener was right for reasons nobody can reconstruct next quarter.

A blank reason field is a finding in its own right.

Ashby reports scorecard completion near 49 percent at organizations under 25 employees and closer to 72 percent at organizations of 500 or more, with completion running higher for candidates who get hired ¹. Those are interview scorecards rather than screen notes, but the pattern is the warning: feedback on the people turned away runs thinner, and an audit needs both halves of the record.

If no written criteria list exists, the audit stops at the first column and the finding is already in hand. Nothing can be coded against a document that was never written, which is the argument for writing the screening criteria before the first application opens.

Two people should code the same forty rows separately, then compare. An audit with one coder measures the coder.

Why Does an Unwritten Criterion Hide in the Advances?

Because an advance note is rarely reread afterwards. A rejection note gets reread whenever a candidate asks or a complaint arrives, so it tends to name something defensible. An advance note ends its life the moment the candidate moves forward: the manager is pleased, the pipeline moved, and a line like strong AI background passes without a second look.

Open the application behind that line and look for what it rests on. Sometimes the support is right there and the note was merely terse. Sometimes the resume names an AI tool and says nothing about what was done with it, and the screener credited the noun.

What an interviewer writes down can carry information the pass decision does not.

Shanshi Liu and four co-authors, writing in Frontiers in Psychology, text-mined the post-interview notes recorded on 7,650 candidates who passed the interview and joined a large Chinese internet technology company ².

The more job-related capabilities a note named, the better the person did afterwards. A one standard deviation rise in how well the notes matched the job analysis went with roughly a 2 percent rise in performance ratings, more promotions and less turnover.

Everyone in that sample had been hired, so the study says nothing about notes written on people who were turned away.

Roughly 2 percent is a small movement, and the design was correlational. What survives is narrow and worth having: a note naming the capability the job called for held signal among people who had all cleared the same bar.

The same reading runs in the other direction. A hiring manager who keeps rejecting screener-approved candidates is stating a bar too, and getting the manager's bar from their rejections rather than a meeting is this audit pointed one stage later.

Sort the Reasons Into Artifacts and Impressions

Sort the coded reasons into two piles: reasons that point at something a second person could open, and reasons about how the application reads. The first pile names an artifact, a tool with a task attached, a decision the candidate described making, a check they ran on model output. The second names polish, fluency, structure, or a document that reads as though a model wrote it.

The second pile is the criterion nobody wrote down.

Register is the easiest thing in an AI-era stack of applications to react to, and the hardest to defend. A clean, evenly paragraphed application is what a careful writer produces and what an assistant produces.

Untrained readers cannot make that call.

In a 2021 study presented at the Association for Computational Linguistics, non-expert evaluators were asked to separate GPT-3 text from human writing across stories, news articles and recipes. They performed at random chance, and three quick training methods lifted accuracy only to about 55 percent ³.

That measurement is five years old, on GPT-3-era text, with crowdworkers rather than recruiters and short passages in three domains.

Nothing in that study measures an experienced screener reading a resume in their own field.

So a screen advancing on readability is measuring the one thing every applicant can now buy. Holding that line is its own piece of work, taken up in how to stop hiring managers rejecting candidates for sounding like AI.

The rewrite is mechanical once the two piles exist.

"Strong AI background" becomes "shipped an internal retrieval tool, names the eval set and what it missed." "Clearly AI-fluent, great communicator" becomes "no artifact, no named task," and the criterion goes down as unmet rather than met.

Report the Counts by Criterion, Not the Screener

Write the audit up as counts by criterion, never as a verdict on a person. Forty rows produce a short table: how many advances rested on each written criterion, how many named nothing on the list, how many pointed at something a second reader could open.

A screener named in a finding defends the note. A criterion named in a finding gets rewritten.

The screen is a selection procedure whether or not anybody scored it.

The Uniform Guidelines, adopted by four federal agencies in 1978, define a selection procedure as any measure or procedure used as a basis for any employment decision. The definition reaches the full range of assessment techniques, through informal or casual interviews and unscored application forms ⁴.

That definition is scope, not an audit mandate.

Nothing in the Guidelines requires reading forty notes, and the validation burden attaches only where adverse impact appears. What they do settle is that a screen nobody scored sits inside the definition, so the wording in it matters: write what you heard, never why you think it happened.

Put the Next Forty on the Calendar a Quarter Out

Nothing holds a written criterion in place between calibrations, so a single audit describes the month it sampled and nothing after it. Nothing else re-checks a criteria list once the requisition is open and the applications are moving, which is why the second pass through forty notes belongs on the schedule before the first one is written up. The clearest published measure of how far a stated criterion travels while nobody counts it is not about screening at all.

IBM's share of IT postings specifying a degree was 21 percent in 2017, by the Burning Glass Institute's count. By 2021 it was 29 percent ⁵.

Those are vacancy-weighted shares in one occupation group at one company in two snapshot years, and IBM still had by far the lowest degree share among the large technology firms studied.

IBM had gone furthest publicly in stripping degree requirements from its postings, and the eight-point rise is not a broken promise; some of it moves with which roles IBM happened to be advertising each year. What it does show is that a stated policy does not hold itself in place. Postings are public, which is why a shift that size could be measured at all. A criteria list and a screener's reason field are read by the team that wrote them, or by nobody.

A screener moving through a full queue types shorthand for a judgment made quickly. The reason field was never built to carry a decision, and reading the shorthand as the judgment finds sloppiness where there was only speed.

How much attention each application gets is a separate question, taken up in splitting the screen into a cheap pass and an expensive one.

The field beside the candidate's name is still one line long. What goes into it after an afternoon with forty of them is a criterion with an address in the application. The date in the calendar is what keeps it there.

See a sample report >