How Do You Reference-Check for AI Judgment When the Reference Says 'They Were Great'?

Why does every reference say 'they were great'?

Because the question had no wrong answer, and the person answering had a reason to stay vague. Many employers refuse to give negative information about a former employee for fear of a defamation suit ¹, and where a policy bars disclosure, most will still provide at least start and end dates and position titles ². The generic endorsement is not evasion. It is what an open question about a whole person produces.

Your reference may never have watched the candidate work with an assistant at all. In NBER surveys of US workers fielded in August and November 2024, 23% of employed respondents had used generative AI for work in the previous week and 9% used it every workday ³. A manager two jobs back may have nothing to report because there was nothing to see, and that is worth ruling out before you push harder.

So open with the prior question (did you see them use AI tools on this work, and on what), and take a plain no as a real answer. If it is no, the call is still worth twenty minutes on everything else, and the AI question moves to a round where somebody can watch it happen. If it is yes, you are one narrow question away from something specific. The resume claim itself is a different check with its own method: how to verify 'AI-proficient' on a resume.

What question gets a specific answer?

One that names an event, a time box, and a decision. "Tell me about a time an AI tool gave them something confidently wrong" works where "how are they with AI" fails, because the reference either has that memory or does not, and both replies tell you something. Then ask the question carrying the weight: what did they do in the next hour?

The shape is three parts, and the third is where an answer stops being a character reference.

• The event. Something with an outcome the reference had to deal with: a rollback, a correction sent to a client, a figure pulled out of a board deck. If the reference has to invent one, you asked about a person instead of about a week.
• The act. What the candidate did next, in verbs. Opened the source. Reran the query. Rewrote it by hand. Called the one person who would know. An act performed outside the conversation with the assistant is the signal; a better-sounding explanation is not one.
• The consequence. What changed: a number, a recommendation, a stated limit, a thing that did not ship. An answer that ends without a consequence is usually a reconstruction rather than a memory.

Keep the questions open-ended and about behavior the reference is likely to have observed ², and build the set from what the job actually requires ¹. Two follow-ups do most of the remaining work: how did they know it was wrong, and what did they check. Follow-up questions that expose real understanding behave the same way here as in an interview: the first answer is prepared, the second is not.

Ask a different question in each field

The question that gets a real answer changes with the work. Ask an engineering manager about a rollback, a finance lead about a number that turned out wrong, a consulting principal about a claim that did not survive the client. Each names an event that field genuinely has, which is what stops a reference reaching back for 'they were great.'

Two rules hold across all six. Ask about work the reference personally handled, not work they heard about, because secondhand praise is where the vagueness starts. And do not ask how much AI the candidate used. Volume answers nothing, and someone who decided the model was the wrong instrument for a step and did it by hand has demonstrated exactly the thing being asked about. What good AI use looks like is a short list, and frequency is not on it.

When a reference stalls, narrow again instead of repeating yourself. "Was there a time it was wrong and nobody caught it until later" gets an answer more often than the same question asked louder, because it hands the reference one specific week to look at. The same probe run on the candidate directly is the natural next round.

How do you grade the answer?

A real answer has a shape, and it is worth settling before you dial. Three things: an act outside the chat window, a refusal with a reason attached, and a piece of work the candidate kept rather than handed over. An answer carrying all three is evidence. An answer carrying none is a story about a pleasant colleague, which you already had.

• Strong. "The model told him the vendor contract auto-renewed. He pulled the signed PDF, found a ninety-day window instead, and we cancelled two weeks before it closed." An act, a source, a consequence.
• Thin. "She's careful, she always double-checks things." A disposition with no event under it. Ask once more for the last time it happened; if nothing comes back, record it as unanswered rather than as a negative.
• Nothing. "He used it for everything and shipped fast." Speed with no refusal in it is the answer you were looking for, and it is not a good one.

Ask the same questions of every reference for every finalist, and write the answers into the same form. That is OPM's own recipe for structure, and structure is what makes a reference check worth doing at all: questions built from the job, an identical set for each contact, and standardized recording ¹. Keep the questions job-related too, because a selection procedure that screens out a protected group has to be job-related and consistent with business necessity ⁴.

Budget the time honestly. A structured reference call runs about twenty minutes, and a minimum of three contacts is recommended ¹, so each finalist costs roughly an hour of somebody senior. That is affordable for two finalists and absurd for ten, which is an argument for settling who did the thinking before you start dialing.

One caution. Some managers narrate other people's work badly and some narrate it beautifully, and neither fact is about the candidate. Hold the answer to whether it names an act, never to how well it was told.

Where does a reference check stop?

At secondhand memory. The best answer you will get is a manager recalling a moment they half-watched months ago, filtered through whether they liked the person. Structure raises what a reference check is worth ¹, and it still cannot show you the candidate deciding, only somebody's account of a decision, with no record left behind it.

That ceiling bites harder on AI judgment than on most competencies, because the behavior happens inside a conversation nobody else was in. The manager saw the pull request. They did not see the four prompts before it, what came back, which paragraph got refused, or whether anything was checked against the world before the branch went up. The record that would settle it belonged to the candidate and their assistant, and nobody kept it.

So use the reference check for what it does well (corroborating that an incident happened and that this person was the one who caught it), and put the remaining question in front of work rather than in front of another conversation. For a small team that usually means one sample, graded against something written down beforehand; the smallest defensible process is a screen, a sample, and a decision. It is also the cheapest insurance against the pattern where somebody interviews and references beautifully and then stalls in the first quarter.

See a sample report >