Write the Screening Criteria Before the First Application Opens

What belongs in a screening rubric, and what does not

A criterion belongs if an application can settle it and two readers would settle it the same way. That admits far less than the standard templates do. Employment dates and continuity, a licence or registration issued by a body you could call, a stated and job-related minimum, a named system or client attached to a named place: those are checkable. Impact, achievement depth, communication and polish are not, because they are scored from prose.

The prose criteria were reasonable proxies when the applicant wrote the document. They measured effort and care, indirectly, and that was better than nothing. Scored today they partly measure which drafting tool the applicant opened, and the ranking templates in circulation still weight impact and communication as though nothing had changed. Call that an argument, because nobody has measured it. The fix it points at is a test you apply to every criterion before it goes in the file.

Write each one in three parts and the test applies itself:

• The claim. "Has closed books for a multi-entity company."
• The evidence that satisfies it. "An employer and a date range where that was the role, or a named system used to do it."
• Where that evidence comes from. "The employment history section, or a link the candidate provided."

If the third part is empty, the criterion cannot be screened and has to move downstream. That is not a loss. A criterion nobody can settle from an application was never being applied consistently; it was being felt, differently, by whoever was reading that afternoon. The situation where every document already looks strong is worked through in what to screen on when every resume looks perfect.

When should the rubric be written?

Before the posting goes live, and certainly before anybody opens an application. The order matters more than most teams expect. A rubric written after the first fifty applications is tuned by those fifty: a criterion gets added because several impressive-looking people had it, a threshold gets softened because the pile was thin, and none of that is recorded as a change to the bar.

The cost of getting the timing wrong lands on the people cut earliest. Applications one through fifty were screened against a different standard from applications fifty-one through four hundred, and nobody can say which. That is invisible in an outcome report, unfixable afterwards, and it is the honest reason to date the file. Nobody has to trust your memory of what the bar was in week one if the file says.

There is a second timing rule that gets skipped. Two people screen the first twenty applications independently, against the written criteria, before the rest of the pile is touched. Then compare. Where you disagree, the criterion is ambiguous and gets rewritten before it has been applied four hundred times. This costs an hour and it is the only cheap opportunity you will get, because after the pile is read the disagreement is buried in decisions nobody will revisit.

Expect the first comparison to be worse than you think. Even at interview stage, where scoring is deliberate and the scale is written down, around 38% of scorecard pairs in one large benchmark set differ by at least a point, and nearly half of those one-point gaps cross the yes-or-no threshold on a four-point scale ¹. That is disagreement rather than error, measured with no outcome variable attached and no demographic variable either, so it supports nothing about accuracy or bias. What it supports is the modest claim that two careful readers routinely diverge at the boundary, which is precisely where a screen operates.

Write anchors, and use the rule you wrote

An anchor is a written description of what each rating means, and it is what stops a scale drifting between readers and between Tuesdays. "3 = has held the responsibility at a named employer for at least a year" is an anchor. "3 = good" is a number wearing a label. Anchoring costs an afternoon and it is the part of a rubric that most teams skip, because it is the boring part.

Anchors buy something real, and the size is worth stating precisely. Levashina and colleagues report Taylor and Small's meta-analysis of 19 past-behaviour interview studies, in which interviews using anchored rating scales showed higher criterion-related validity, .35 against .26, and higher interrater reliability, .77 against .73 ². The reliability gain is small; the validity gain is the larger of the two. The studies cover past-behaviour interview questions, and none of them is a screen. The comparison also runs across studies, so nothing here tests the same interview with and without anchors, and the review itself is blunt that anchored scales became popular for their logical appeal ahead of the evidence. Take it as a reason to write anchors. It does not forecast what yours will buy. The same review reports a separate finding that matters more in practice: anchoring every point of a five-point scale produced ratings more resilient to disability bias than anchoring only the endpoints ², so a half-anchored scorecard is not what was studied.

Then use the rule you wrote instead of forming an impression of the whole person. In a meta-analysis of selection and admissions studies, combining applicant data mechanically by a formula correlated .44 with job performance against .28 when the same kinds of data were combined holistically by expert judgment ³. Mechanical here can be as plain as adding up unit-weighted scores on a written scorecard, and the paper says so. The job-performance comparison rests on nine studies and other criteria show much smaller gaps. The finding is about how evidence gets combined at the end, and it licenses no number standing in for a person.

What that means at a screen is narrow and practical. Decide in advance which criteria are disqualifying and which are additive, write it down, and apply it. The impression is still yours to have. It just stops deciding, and the argument for rating consistency between two people is carried further in writing a rubric two reviewers score the same way.

How do you know the criteria are wrong?

Read what they rejected. Criteria that are too narrow look identical to criteria that are exactly right when you only inspect the survivors, and the difference only shows up in the pile you cut. Pull twenty rejects at random from a closed requisition, hand them to whoever knows the work, and count how many they would have wanted to meet.

Employers have been agreeing about this outcome for years without acting on it. In the Harvard Business School and Accenture survey of 2,275 executives, 88% agreed that qualified high-skills candidates are vetted out because they do not match the exact criteria in the job description, rising to 94% for middle-skills roles ⁴. That is agreement with a statement about their own process, not a measured exclusion rate, and it was fielded in early 2020, so it describes criterion-based filtering that employers configured themselves. It predates any argument about AI, which is why the reject sample is standing practice in a normal quarter.

Four signals that your criteria need rewriting:

1. The reject sample keeps surfacing people the manager wanted. More than a couple out of twenty and something in the file is selecting for the wrong thing. 2. Two readers disagree on the same application. The criterion is ambiguous, and the repair belongs in its wording. 3. The pass rate moves without the pool changing. Usually a criterion being reinterpreted over time. 4. Nobody can say why a specific person was cut. The decision was made on something that is not in the file.

The qualification worth stating out loud, because it is the real trade: fixed criteria will let through people who look ordinary on paper and cut people who would have interviewed well. You are buying a screen that means the same thing on application four hundred as on application four, and you are paying for it in a certain kind of upside. Whether that trade is worth making is the question behind knowing whether your screen is throwing away the wrong people, and the reject sample answers it from your own pile.

See a sample report >