Callback Studies Measure the Screen, Not the Hire

What does a name-swap study actually prove?

That a change in one signal on an otherwise identical document changes the reply rate. Bertrand and Mullainathan mailed 4,870 resumes to over 1,300 help-wanted ads in Boston and Chicago in 2001 and 2002, randomly assigning each one a White-sounding or an African-American-sounding name, and recorded a callback for 9.65 percent of the first group against 6.45 percent of the second ¹.

Three details go missing every time the study is retold, and each one changes what the number licenses.

• The manipulation was not the first name alone. Race-typical surnames were assigned as well: Walsh, Murphy and O'Brien against Washington, Jones and Jackson ¹. Anyone summarizing it as a first-name swap has the design wrong.
• The gap is 50 percent one way and about a third the other. White names received 50 percent more callbacks; African-American names received about a third fewer. Quoting both as though they were separate results doubles a single finding.
• The outcome is a reply, not a job. The authors write that they are not able to translate their results into gaps in hiring rates or gaps in earnings ¹. That sentence sits in the paper's own weaknesses section, and it is the sentence that never travels with the citation.

The population is narrow too: sales, administrative support, clerical and customer-service ads, answered through newspaper listings, which excludes every hire that arrived through a referral or a network. So the study is strong evidence that racially distinctive names were treated differently by employers reading documents in two cities in the early 2000s. It is not a national rate of anything, and it is not a claim about the average African-American applicant, since not every employer will have read the name as a racial signal.

One substantive methodological objection exists, and it comes with its resolution attached. Heckman and Siegelman showed that groups differing in the variance of unobserved productivity can generate spurious evidence of discrimination in either direction, even when the applications are identical on everything the researchers control. Neumark showed an unbiased estimate can be recovered when a study deliberately varies applicant quality, and applying that correction to this data strengthened the evidence of discrimination ⁶.

Where does the rest of the gap open?

After the invitation, in roughly equal measure. Quillian, Lee and Oliver pooled the twelve field experiments that follow applicants past the callback and found majority applicants received 53 percent more callbacks than comparable minority applicants and 145 percent more job offers ². Their own reading of that split is that about half the discrimination in offers is already present at the application-to-callback stage.

Two cautions belong with the pair of numbers. The 145 percent is a ratio of offer rates, so it magnifies a small absolute difference when offer rates are low: it is never percentage points, and 53 does not subtract from it. And the pool is twelve studies, selected because they were able to run in-person or telephone stages, which skews the sample toward lower-skill and face-to-face roles.

The line that matters for a hiring lead is the correlation rather than the headline. Additional discrimination after the interview correlates with the callback gap at only r = 0.21 ². A firm with clean screening numbers has learned nothing about its own interview stage in either direction, which is why an end-to-end fairness number and a stage-by-stage one answer different questions. If your pass rates have moved for reasons nobody can name, which funnel metrics still mean anything is the prior question, and how you would know if the screen is throwing away the wrong people is the one after it.

Don't take name-blind screening as the fix

Removing the name is a partial control for one signal, and in the one place it was randomized it backfired. When the French public employment service randomly assigned about 600 participating firms to receive anonymized or name-bearing applications, the interview-rate gap widened: the interview rate of minority candidates fell while that of majority candidates rose ⁴. Two mechanisms explain it, and both are about that program rather than about anonymity as an idea.

The first is selection. Firms volunteered, and the volunteers were already interviewing and hiring relatively more minority candidates, so anonymization took away a discretion they had been using in the candidate's favour. The second is context. Stripping identifiers also strips the information a recruiter uses to discount a negative signal, such as an employment gap, for a candidate they were minded to place.

The treatment in these studies is not uniform either. A survey experiment measuring how people actually read the names used in correspondence audits found the congruent perception rate for Black first names was 75.0 percent with no surname attached, 82.5 percent with a Black surname and 66.5 percent with a White one ³. Names more common among highly educated Black mothers were markedly less likely to be read as Black at all. That cuts both ways: a weakly racialized name dilutes the treatment and understates discrimination, while a strongly racialized one may carry a class signal riding alongside. Neither reading rescues the simple version, which is why whether blind screening reduces bias is still an open question.

Measure your own funnel one stage at a time

Measure each handoff separately, because a gap can sit inside one stage while the end-to-end number still looks acceptable. Count applications, contacts, interviews, offers and hires as distinct populations, using whatever demographic data you actually hold, and compare the ratio at each transition.

1. Cut it at every boundary. Application to contact, contact to interview, interview to offer, offer to acceptance. A process that passes overall can still contain one transition that does not, and averaging hides it by construction. 2. Keep the reason, not only the rate. A rate tells you a difference exists. The note a screener wrote about why this application stopped here is what you can read six months later when the rate looks wrong, and it is the only artifact anyone can argue with. 3. Do not put two different quantities in one sentence. The 50 percent gap measured in 2001 and 2002 and the 2.1 percentage point gap Kline, Rose and Walters measured at 108 large employers are not the same measurement: the later experiment ran at unusually large employers whose overall contact rate was nearly three times higher, which shrinks proportional differences ¹⁵. Quoting them side by side as a trend is an error somebody will catch. 4. Say what your volume can carry. With eleven hires a year, no ratio you compute separates a real gap from noise, and publishing one anyway is worse than publishing nothing.

None of that substitutes for the evidence itself. A callback gap counted in a spreadsheet is a fact about outcomes with no account of how any of them were reached, which is exactly where the correspondence literature is stuck: it can prove that treatment differed and it cannot say what any employer was thinking. Your own process does not have to leave you in the same position. Where that gap concentrates at the employer level is the subject of a minority of employers accounting for most lost callbacks.

See a sample report >