How Do You Know If Your Resume Screen Rejects the Wrong People?

Why can't a resume screen be validated on the people you kept?

Because the screen's errors were removed before anyone could observe them. Performance data exists only for people you advanced, interviewed and hired; the 80% you rejected generate no outcome of any kind. Researchers call this the selective labels problem, and a study of resume screening at a Fortune 500 professional-services employer states the constraint flatly: the authors "do not observe hiring outcomes for applicants who are not interviewed" ¹.

So "the screen works" nearly always means nobody who got through has caused a problem yet. That is a statement about the top of a range you truncated yourself. It is equally compatible with an excellent screen and with one that has been discarding the strongest applicants in the stack for two years, because both produce the same clean record of people who did fine.

The standard vendor checklist stops short for the same reason. Precision, recall and false-negative rate all need labels, meaning a known right answer per applicant, and the labels are exactly what the screen destroyed. A tool evaluated against your historical advance-or-reject decisions is being measured on agreement with the judgment under test, not on accuracy about the job.

The material the screen reads is thinner than the confidence placed in it. Job experience in years correlates with job performance at .07, corrected for measurement error in the criterion, and years of education at .10, corrected for that plus range restriction: roughly 0.5% and 1% of the variance in job performance. That makes them "two of the least valid predictors of job performance (among commonly used screening criteria)" ². It is the ground most resume bars are built on, and the same drift sits behind whether GPA still predicts entry-level performance.

Reviewer agreement does not rescue this. Two people applying the same rule tells you the rule got applied twice, and a screen every reviewer agrees on can still be wrong about the same applicants every time. Resumes make even that consistency hard to reach: applicants choose what to put in, so "there is no structured format across applicants" and no guarantee two files were ever read on the same fields ².

Let a sample through under the bar, then track it

Take the band immediately below your bar, draw a random sample from it, and put those applicants through the same loop everyone else gets. It is the only move that manufactures the data the screen destroyed, and at 80% rejection the volume is small: one in twenty of the near-miss band adds a handful of interviews a month, not a second funnel.

Random is the load-bearing word. An "interesting rejections" pile hand-picked by the same reviewer reproduces the judgment under test and will confirm it. Draw mechanically: every nth application below the line, over a fixed window. Write the rule down before you run it, and blind the downstream stages if you can, because an interviewer who knows a candidate came from below the bar will find the reason, and the exercise then measures the interviewer.

The evidence that this pays is specific, and it is worth stating precisely. Li, Raymond and Bergman simulated screening policies against the historical applications of a Fortune 500 professional-services employer. A model built to explore, meaning one that advances candidates it is uncertain about, would more than double the share of selected applicants who are Black or Hispanic, from 10% to 23%, while the two standard supervised models would cut that share to approximately 2% and 5% ¹. Those are counterfactuals computed on the firm's own records, not a program anyone ran.

The authors' own falsification test is the part worth borrowing. If the extra applicants the exploration model selected were truly weaker, it would learn that and stop selecting them; across their test sample it keeps selecting them even as the exploration bonuses fall ¹. On the quality side, their decomposition puts average hiring rates among model-selected applicants at 25% and 30%, against the observed 10% among recruiter decisions, on the authors' stated assumption that there is no selection on unobservables ¹. That is an estimate carrying a named condition, not a lift you can promise a CFO. What it establishes is that the people a screen passes over are not a settled question.

Measure at a point your own screen cannot contaminate: an offer accepted, a scored work sample, a manager rating at six months. Whether the panel liked them is your own judgment again, so it does not count. Keep the record too. Sampling on your screen's own margin is not selecting on a protected characteristic, but it changes selection rates, and selection rates are what the Uniform Guidelines (US federal, adopted 1978) ask you to document ³. Write the rule, the window and the counts before the first invitation, and put the design in front of employment counsel.

Compare old screen calls against work you later saw

Anywhere a candidate produced observable work after the screen, whether a work sample, a paid trial or a structured exercise, you hold a second reading of the same person the resume never gave you. Pull six months of those, hide the result, have two reviewers re-run the screen decision from the resume alone, then unblind. The disagreements are the finding.

Count both directions and keep them apart. A resume the screen would have rejected, attached to strong observed work, is a false negative you can finally see. A resume that sailed through attached to weak work is the cheaper error, but it is the one that names the criterion doing no work, usually the employer name or the years-of-experience band.

The criterion has to be worth trusting, which rules out most of what sits in an applicant tracking system. The Uniform Guidelines say that whatever criteria are used "should represent important or critical work behavior(s) or work outcomes" ⁵. A hiring manager's overall impression at ninety days is neither, and it quietly inherits the halo of the screen decision that produced the candidate. A scored exercise everyone took under the same conditions qualifies. So does retention, slowly.

The legal standard names the same evidence the audit needs. Evidence from a criterion-related validity study "should consist of empirical data demonstrating that the selection procedure is predictive of or significantly correlated with important elements of job performance" ⁴, and the sample subjects "should insofar as feasible be representative of the candidates normally available in the relevant labor market" ⁵. A validity study run on your hires is not that sample. This check narrows the gap; it does not close it.

The ceiling here is low. This only sees people who reached the work stage, so the applicants your screen cut hardest stay invisible, and it runs on data you already have, which is its entire appeal. It also raises the question of whether to drop the resume screen for a work sample, and it inherits the harder one of whether assessment scores still predict performance once AI is in the workflow.

Audit what your pass pool has quietly converged on

Describe the last hundred people who passed without using the criteria you believe you apply. Which employers, which schools, which title ladders, which resume format, which phrasings. A screen drifts toward whatever reviewers found easy to read, and after a year the pass pool is a profile nobody wrote down and nobody validated. This check needs no new data, just an afternoon.

Tabulate it plainly: prior employer, degree field, years-of-experience band, whether the file echoed your job posting's own vocabulary. Then put the pass pool beside the applicant pool on the same columns. Any attribute that separates the two by a wide margin and appears nowhere in your written bar is the screen's real rule, and it is the one to defend in writing or delete.

Run selection rates by group at the same time, because a resume screen is a selection procedure with obligations of its own. Under 29 CFR 1607.4(D) a selection rate below four-fifths of the highest group's rate "will generally be regarded by the Federal enforcement agencies as evidence of adverse impact" ³. The trap is the bottom line: where the total process shows no adverse impact, the agencies say they "will not expect a user to evaluate the individual components" ³. That is prosecutorial discretion, not a finding that your screen is sound, and a clean overall number can sit on top of a screen doing damage that a later stage compensates for. The Guidelines are US federal and were adopted in 1978, jurisdictions add their own rules on top, so treat this as public legal fact and take the design to counsel. The mechanics on a purchased tool are an adverse impact audit.

Convergence can be legitimate. Unexplained convergence is the thing to chase down: if everyone who passes came from four employers, either those employers genuinely produce the work or the screen learned their formatting, and the first of those is a claim the comparison check can test.

Which role families does your screen rot fastest for?

Look for roles where a resume used to evidence the work and no longer does. A screen is a bet that a document predicts an occupation, and that bet decays at very different speeds. It holds where a licence or a proctored exam makes the claim checkable outside the file, and it goes first where the first year has become judging generated output. Run the checks per role family; a company-wide verdict averages processes with different half-lives.

• Accounting, audit, nursing. A licence, a board result or a cleared exam section is verifiable outside the document, and the curriculum overlaps the first year's knowledge. The screen keeps its signal here longest.
• Software engineering. A repository link used to stand for authorship. Side projects now read uniformly finished, and "three years of React" never described the part of the job that is deciding what not to merge.
• Marketing. A portfolio piece is cheap to produce and expensive to attribute. What separates candidates is what got checked before it shipped, which the artifact does not show.
• Financial analysis. "Built DCF models" was a low-information line before any of this. With the deliverable now generatable, the screen falls back on tenure at a named employer, which is job experience in years, among the least valid predictors in common use ².
• Legal operations, revenue cycle. Named systems, named payers, named jurisdictions still carry information, because a claim that specific can be wrong.

One bar cannot be right across that spread. Where the screen still evidences something, tighten the written criteria and leave it alone. Where it does not, the honest options are to move the evidence earlier in the loop or to stop asking the document to carry it, which is the same argument behind what replaces ATS keyword screening when every resume matches. The revised validity estimates put the predictors specific to individual jobs at the top of the list, with structured interviews at .42 and an 80% credibility interval running .18 to .66, wide enough to read as a caution against trusting any single number too far ⁶. See how Olive measures this.

Write the verdict down per family, with a date and the evidence behind it. A screen nobody has checked in three years is not a standard. It is an inheritance, and the only thing it reliably predicts is who was easy to read in the year it was written.

See a sample report >