Two Structured Reference Calls Beat Five Courtesy Calls

Are reference checks worth running?

Worth running, and worth running differently from how most teams run them. The evidence behind the standard reference check is thinner than its reputation: the .26 validity figure that circulates as settled comes from ten samples pooled before 1984, and the authors who reprinted it in 2016 say in the same paper that it may not at present be fully accurate, because many former employers in the United States now release only dates of employment and job titles ¹.

It got worse from there. When Sackett and colleagues re-estimated the 1998 validity table in 2022, they excluded reference checks entirely for insufficient information, so the method has no revised estimate at all ². That is not a finding that references do not work. It is a finding that nobody can currently check the number, which is a different and more uncomfortable position to argue from.

What has changed since those samples were gathered is everything on either side of the call. The resume, the cover letter, the portfolio write-up and the take-home can all be drafted with an assistant now, which weakens them as a read on the person without making them worthless. A reference call is one of the few remaining channels where somebody who watched the work describes it without a draft in front of them. That is worth twenty minutes of a recruiter's day, as long as the twenty minutes go somewhere better than whether the candidate was a team player.

Which leaves one distinction doing all the work, between two instruments that share a name. A courtesy call is a freeform conversation with whoever picks up, and it produces a character summary. A structured reference asks every reference the same short set of questions about the same behaviours and records what was said. The US Office of Personnel Management puts the same point in agency language: reference checks are typically collected by phone using a structured interview format, adding structure can enhance their effectiveness, and written requests for work histories typically produce low response rates and less useful information ³.

How many references do you actually need?

Two, asked the same questions, unless the role gives you a specific reason for more. OPM's guidance recommends a minimum of three contacts at roughly twenty minutes each by phone ³, and for a federal process that is a defensible floor. In most private hiring the third call adds a third character summary rather than a third data point, because nothing in the way it is run forces the three to be comparable.

Spend the count you save on structure instead. Three fixed questions asked of two people gives you six answers about the same things. Two open conversations with three people gives you five impressions and no way to line them up. OPM presents its three contacts and its twenty minutes as a recommendation, and publishes no validity coefficient on the page at all ³.

Two rules make whatever count you land on defensible:

• Same questions, same requisition. Every reference for a role gets the identical set. This is the only thing that lets one candidate's calls be read next to another's.
• Same number of calls for every finalist. An extra call made only for the candidate somebody already doubts is a search for a reason, and it will find one.

Record who declined and why, in the file, beside the answers. A former manager who will confirm dates and nothing else is usually describing their employer's policy, and that is exactly the behaviour the researchers reprinting the old figure pointed to when they warned it might have decayed ¹. Reading a policy sentence as a red flag is the most common way a reference check produces a wrong decision.

Make the calls before the offer, not after

Put the calls in the loop before the decision, at the point where the answers could still change it. A reference check run after the offer letter goes out is a formality with a rescission attached, which is the worst of both: the stage costs the candidate and the employer real time, and the only way to act on it is to take back something already given.

The practical slot is after the final round and before the debrief. Two calls take forty minutes of one person's day and the answers arrive while the panel still has the candidate in mind, which is when a quote about a specific piece of work is worth something. Tell the candidate the week it will happen, ask for names then, and give them a day or two to warn the people they nominate. A reference caught cold gives you a shorter and blander call.

One stage genuinely does belong after the offer, and it is not this one. California makes it an unlawful employment practice for an employer with five or more employees to inquire into or consider an applicant's conviction history until after a conditional offer has been made, under its Fair Chance Act, whose current text took effect on January 1, 2019 ⁴. Other fair-chance laws set their own point, so where one applies, counsel sets the order. Criminal-history checking is a different instrument with a legally fixed order and it should not drag the reference calls to the end of the process with it.

If references currently happen after the decision is made, the honest options are to move them or to stop running them. A stage that cannot change the outcome is costing four people an hour each to produce a file note.

What a reference call cannot establish

It cannot establish anything the candidate has not already done, and it is not a fraud control. The candidate picks the references, so what you are buying is specifics about work that happened, from somebody who agreed in advance to speak well of them. That is genuinely useful and it is narrow. Anything that has to be established rather than described belongs somewhere else in the process.

Three things get asked of reference calls that they cannot carry:

1. Whether the record is real. Dates, employer and title are a verification question, and what a verification product actually confirms is narrower than most buyers assume. A reference who is a friend of the candidate confirms nothing about the employer's existence. 2. Whether the person can do work they have not done. A reference describes the last job. A candidate stepping up a level, changing function or joining a very different codebase is being asked about something no reference watched. That is a work-sample question. 3. Whether "they were great" means anything. Warmth is the default register of a reference call and it carries almost no information. The fix is question design, and that is the whole of what to do when the reference just says they were great.

Question design is where the AI-era version of this stage lives. Ask how a specific piece of work got made and a reference who sat beside it will tell you what the constraint was, what got tried first and what got thrown away, which is the part no drafted document carries. They cannot answer a question shaped like an accusation, and they will not answer a rating scale.

See a sample report >