You Cannot Compare the Model to Recruiters You Never Measured

Why can't the comparison be settled?

Because the human side was never measured. A hiring team that replaced twelve screeners with a model has, at most, a record of who was rejected and no record of why, so there is nothing to compute a before-and-after against. The comparison also assumes a single system can stand in for a room of people, and that the room's errors were partly cancelling each other, which is a strong assumption nobody states out loud.

Where the human side has been measured, the ceiling is low. A review of why employers resist selection decision aids reports that the interrater reliability of the traditional unstructured interview is so low that, even with a perfectly reliable and valid criterion, interview-based judgments could never account for more than 10 percent of the variance in job performance ³. The same review reports a 1996 survey of 201 HR executives who rated the unstructured interview more effective than any paper-and-pencil assessment procedure. Belief and measurement pointed in opposite directions, and the belief won for decades.

That is a finding about interviews rather than about resume screening, and the gap matters: no comparable measurement of recruiters reading resumes exists at all. Which is the point. The half of the comparison a vendor is being asked to beat has never been quantified in any employer's own building, so an improvement claim and a harm claim are equally unfalsifiable. A vendor's own bias audit does not fill that gap: under New York City's 2023 final rules, an employer that has never used a tool may rely on an audit computed on other employers' historical data, or on synthetic test data where too little real data exists ⁷. Anyone who answers this question with a number is answering a different question, usually about their own product.

What does the evidence on each side actually say?

Less than either side claims, and none of it is a head-to-head comparison. Across 15 firms hiring low-skilled service workers, introducing a job test raised completed job tenures by just over 25 percent, and comparing managers at the same location, a one standard deviation higher rate of hiring against the test's recommendation went with 6 to 7 percent shorter job durations ¹. Human discretion over a structured signal was not, on average, superior private information.

Hold that result at the strength it was measured. There was no randomization; it compares managers at the same site who override at different rates. Tenure is the quality measure, chosen because turnover is what this employer pays for; no supervisor performance rating was collected. Overrides averaged 22 percent, so they were routine in these firms. The instrument was an online questionnaire scored by a proprietary algorithm into a green-yellow-red band, rather than an interview or a model reading resumes. It says the average override was worse, and says nothing about whether the test was right about any individual person.

What has been measured on the machine side is version-locked: the findings in model-driven resume screening are attached to specific model releases and specific prompts, and a release resets them. The human side has a steadier measurement, and it is not flattering. In the 83,000-application audit at 108 large US employers, distinctively Black names reduced the chance of employer contact by 2.1 percentage points, an effect equal to 9 percent of the Black mean contact rate, while the average gender contact gap was zero with a between-company standard deviation of 2.7 percentage points roughly symmetric about zero: some firms consistently favored men, others consistently favored women, and the two cancelled only across companies ².

That last cancellation is where the argument against the machine usually starts, and it is weaker than it looks. The offsetting happened across 108 separate employers. Inside any one of them the gap held its direction, which is what a single consistent rule looks like from the applicant's side, and a model applied across a whole pipeline has that property at higher volume. So scale raises the cost of being wrong. It does not show that models are more biased than people, and no employer should read an aggregate of other companies as evidence that its own screeners cancel out.

Which of the two can you inspect afterwards?

The one that leaves a written reason behind. A disparity found in a column of scores can be counted and nothing else, because there is no account of how any individual score was reached; a disparity found in a set of written reasons can be read, disagreed with, and traced to the criterion that produced it. The difference comes from what a process stores, and a person reading resumes without notes stores nothing either.

The research supports the narrow version of that claim. Text-mining post-interview notes on 7,650 candidates hired at a large Chinese internet technology company, researchers found the number of job-related capabilities an interviewer named in the notes was positively related to later job performance and promotions and negatively related to turnover, with a one standard deviation rise in the match between notes and job analysis corresponding to roughly a 2 percent rise in performance ⁴. It is correlational, at one firm in one country, and range-restricted by construction, since only candidates who passed could be observed. What it shows is modest and worth having: what a reviewer writes down carries information that a rating on its own does not.

The one large randomized result on machines in this loop points the same way. In a field experiment where 70,000 applicants were randomly assigned to a human or an AI voice-agent interview, with human recruiters evaluating the transcripts and making every hiring decision in both arms, applicants interviewed by the agent were 12 percent more likely to receive an offer, with higher starts and retention and no drop in the productivity of hires ⁵. The mechanism the authors identify is consistency in gathering evidence. The costs are in the paper: 5 percent of applicants ended the interview rather than speak to an AI, and the agent hit technical difficulties in 7 percent of cases. One firm, one country, one job family, and a working paper that has not been peer reviewed. Reviewers who cannot judge the work in front of them are a separate failure with its own fix, which is where getting hiring managers who don't use AI to judge AI-assisted work starts.

Start recording what each screening decision rested on

Start with the record you wish you had two years ago. From today, store the criterion each screening decision was made against and the specific thing in the application that met or missed it, for every rejection as well as every advance. Six months of that is a baseline, and it is the only version of this comparison you will ever be able to run on your own pool.

1. Write the criteria before the applications arrive. A reason recorded after a decision is a rationalization, and it will read as one later. 2. Record rejections in the same detail as advances. The rejected side is where a disparity lives and where nobody keeps notes, which is exactly why the question is unanswerable now. 3. Keep it at the individual level. A rate is an aggregate of things that already happened; only the individual record can be re-read when someone asks what happened to a particular person. 4. Decide in advance what you will do if you find something. California's amended FEHA employment regulations, effective October 1, 2025, make evidence, or the lack of evidence, of anti-bias testing relevant to a discrimination claim and to any defence, including the quality, recency and scope of the effort and the response to the results, and they extend records retention from two years to four with automated-decision system data expressly included ⁶⁸. They impose no duty to test and set no standard for adequate testing; what changes is what a court may weigh. Testing and then ignoring the result is the worst of the available positions. That is California as of 2025, and the register here is check with counsel; none of this is advice.

Six months in, you will not have settled whether models are less biased than people. You will have something better for the decision in front of you: your own numbers, at your own stages, with an account of every decision behind them. The debate will still be running. The record is an asset, and it is the one thing a candidate who challenges an outcome can be shown, which is the standard an AI-skills assessment has to hold up to as well.

Read the evidence >