False Positives, False Negatives, and Which One You Pay For

Which Error Is Your Process Tuned Against?

Look at what happens to an ambiguous candidate. If a mixed signal at any stage resolves to a rejection, the process is tuned against the bad hire, whatever the hiring philosophy on the careers page says. That is a defensible setting for some roles and an expensive one for others, and almost no team has ever stated which it chose, because the setting arrived through a hundred small decisions rather than one.

The vocabulary is worth having, because it makes the trade sayable. A false positive is somebody who was hired and should not have been. A false negative is somebody who was rejected and would have done the job well. Any threshold that reduces one raises the other, holding the quality of your evidence fixed. That last clause is the escape route, and the whole final section is about it.

The economics differ by role, and this is where the decision actually gets made. A mistake recoverable inside a quarter, on a role with a short ramp, a defined probation, a large cohort and work that is visible early, tolerates a looser bar, because the process has a cheap second chance. A mistake that is not recoverable, on a small team where the ramp runs nine months and the person holds a relationship nobody else can hold, does not. Write the role into one of those two categories before arguing about the bar, and half the argument resolves itself.

Why the Missed Hire Never Shows Up in a Review

Because there is nothing to review. A rejected candidate leaves a row in an applicant tracking system and no further data, ever. They do not come back in eighteen months with a promotion and a performance rating to prove the decision wrong, so the error is not merely uncounted, it is structurally uncountable by any process that only observes the people it hired.

That asymmetry compounds. A bad hire generates a manager who remembers it, a post-mortem that names a stage, and a change to that stage. Nothing on the other side generates anything, so every feedback loop in the system pushes the bar in one direction, and each push feels like rigor rather than drift. Ten years of that and a process can be rejecting on evidence nobody would defend if it were written on a slide.

Knowing how much signal the underlying methods carry bounds both errors. In the 2022 re-analysis of the selection literature, structured interviews came out top ranked at .42, ahead of job knowledge tests at .40, work samples at .33, cognitive ability tests at .31 and unstructured interviews at .19 ¹. Those are corrected correlations with supervisor ratings rather than accuracy rates, and the .42 carries a wide credibility interval. Read them as a ceiling: the best evidenced single method in the field leaves a great deal of both mistakes on the table, so a process claiming to have eliminated one of them has simply stopped counting.

Whether a resume screen that rejects most applicants is throwing away the wrong people is the same question at the stage where the volume, and therefore the count of missed hires, is largest. What the mistakes cost in money is a separate calculation worth building from your own payroll.

Set the Error Rate Deliberately, Because You Already Did

A threshold is a choice even when nobody made it on purpose. Every stage with a cutoff has one, and moving it moves both error rates in opposite directions. The automated end of hiring shows this most plainly, because there the threshold is a literal number in a settings panel, and the published research on where those numbers land is unusually direct.

RAID, an evaluation of twelve text detectors across eight domains, found that a tool's false-positive rate is mostly a function of how it is tuned rather than a property of the tool: at naive default thresholds, one open-source detector flagged human writing at a 100% rate while closed-source tools stayed below 1.7%. Even after every detector was calibrated to a 5% overall false-positive rate, individual domains stayed far off, with 20.4% of human reviews, 33.4% of human recipes and 13% of Wikipedia text still flagged ². None of the eight domains is a resume: they run from news and Wikipedia to books, reviews and recipes, so no number there is a measured rate on hiring documents. The transferable part is that the rate is a dial, and the buyer of a tool almost never sees where it is set.

The arithmetic of a small rate at volume is the second half. Vanderbilt University switched off Turnitin's AI detector after publishing the multiplication: a claimed 1% false-positive rate against 75,000 papers submitted in a year works out to roughly 750 papers wrongly labelled ³. Whatever your equivalent numbers are, run that multiplication before adopting any stage with a stated error rate.

And the errors are not distributed randomly across people. Seven detectors run over 91 essays written by non-native English speakers produced an average false-positive rate of 61.3% ⁴. One cohort, one document type, 2023 tool versions, and the figure is an average across seven tools rather than any single one. The durable finding is the direction of the error, which is that it lands hardest on a group defined by something other than ability. Which of a detector, a structured interview and a work sample actually defends a decision is the practical version of that choice.

Test the Half You Cannot See

Two moves are available, and the second is much better than the first. The cheap one is to sample your own rejections: take twenty near-misses a quarter, the ones a panel split on, and record where they landed. It is rough, incomplete and biased toward candidates whose careers are publicly visible, and it is still more than the zero information a normal process produces about its own false negatives.

The better move is to stop trading the two errors and improve the evidence instead. Both rates fall together when a stage produces something specific rather than an impression, because the trade between them is fixed only while the quality of the evidence is fixed. In practice that means one stage where the candidate does a piece of the work and somebody records what happened, replacing one stage where three people formed a view and averaged it.

A short checklist for the stage you are least sure about. Does it produce an artifact somebody else could re-read? Could two reviewers reach the same conclusion from it? Does it reject anybody on something the role never requires? Would you be comfortable telling a candidate exactly why they did not pass? A stage failing all four is generating both errors at once and is the cheapest thing in the loop to replace.

The common failure at this stage is a screen on how a document sounds, which is an impression wearing the costume of evidence. Stopping hiring managers rejecting candidates for sounding like AI is worth doing before touching the bar at all, and what a screen still reads for when every resume looks polished is where the replacement usually starts.

See a sample report >