Assessment design
When the Reference and the Work Sample Disagree, Neither One Wins
An AI work sample says the candidate verifies nothing. Two references call them meticulous. Neither document wins on authority. Meticulous is a trait summary from a person the candidate chose; the work sample is one session on one task somebody at your company designed. Ask what each source watched happen. Go back to the reference for a single episode of catching an error, and read the session again for whether the task contained anything that needed checking. One of the two usually gives way.
The takeThe reflex to let the reference break the tie is deference to a person who put their name on something. But the candidate picked that person, and nobody wrote down what the word meticulous was standing for. A disagreement between two sources is the most useful thing in the file, and averaging it into one verdict throws away the signal. Hold the tie open for one more phone call. Twenty minutes is cheaper than the hire.
Where Olive fits
Open a role and see what the work shows
If the exercise is the thing that needs fixing, the expensive parts are the authored case and the evidence trail behind each judgment. Olive ships twelve authored cases per occupation and returns six findings written or edited by a human reviewer, each anchored to a timestamped moment in the session, and the candidate is granted the same report.
Assess your shortlistWhy does the reference feel like the tiebreaker?
A reference feels decisive because a person staked their name on it, and a work sample feels like a test somebody made up last quarter. The order of trust runs the other way once you ask what evidence sits behind each one.
The exercise has a known task, a known window and a record you can reopen. The call has a person the candidate chose, and nothing you can reopen.
In 2022, Paul Sackett, Charlene Zhang, Christopher Berry and Filip Lievens went back through the meta-analyses behind the table hiring has quoted since 1998 and recomputed every estimate the evidence could carry. Reference checks did not come back lower.
They did not come back. The method is one of eight the authors set aside, because the evidence under it carried too little information to evaluate. So the 1998 figure of .26 stands as both the only published number for a reference check and the least checkable one in the table 1.
The rest of that table moved as well, and the validity numbers everyone quotes were corrected in 2022 across almost every method on it.
The .26 has a short lineage. It traces to one 1984 table pooling ten samples, carried forward unchanged for three decades 2.
Frank Schmidt, In-Sue Oh and Jonathan Shaffer reprinted the figure in a 2016 update and wrote, in the same paper, that the information "may not at present be fully accurate" 2. The reason they give is that many previous employers, in the United States at least, now provide only the dates of employment and the job titles a former employee held.
The studies under that number were run before 1984.
The work sample's number has a different weakness. Sackett and colleagues put work samples at .33, down from the .54 that still circulates, and the estimate rests on 54 studies of which 53 measured people already doing the job rather than applicants 1.
No correction for range restriction was applied, because the set held almost no applicants to correct for.
Neither figure was going to settle a tie anyway, because the two come from different meta-analyses run on different people.
The same paper reports the comparison run both ways. Across separate meta-analyses, cognitive ability tests beat assessment centres, .51 to .37. In the studies where the same people took both, scored against the same performance measure, the order inverted: .44 for the centres and .22 for ability 1.
A league table cannot break a tie between two instruments it never ran on one person.
What did each source watch happen?
Meticulous is a judgment about a person, offered by somebody the candidate chose and asked to be helpful. The work sample is a record of one afternoon. Neither is a lie, and they answer different questions: one reports an impression formed over months, the other what happened inside a window you set.
Madera and four colleagues collected 624 real recommendation letters written for 174 candidates for eight assistant professor positions. Trained coders read every one for doubt raisers, the hedges and faint negatives that sit inside an otherwise positive letter 4.
For women, 54 percent of the letters carried at least one doubt raiser and 13 percent carried two or more. For men, 51 percent and 7 percent.
The consequence turned up in a second study, where 305 professors read letters and rated research competence lower when the letter contained negativity or hedging 4.
Two limits travel with that result. Male and female letter writers did not differ, so the finding is about the language of references rather than about biased individuals, and the lower ratings appeared on research competence and not on teaching competence 4.
Academic hiring, one set of disciplines, letters rather than calls.
None of it tells you what two references meant by meticulous. The study does show that free-text reference language varied with the applicant's gender, which is enough to stop an adjective counting as an observation.
The exercise sees a narrower slice: what the candidate did in the time you gave, on the task you wrote, with whatever the assistant handed them. It stops at the edge of that window.
Both sources are narrow in opposite directions, which is how they can disagree without either one being wrong.
Ask the reference for one episode, not a second impression
Call one reference back and ask for a single occasion: a time this person caught something wrong before it left the building, what the error was, how they found it, and what happened next. A reference who watched that happen will have the story in twenty seconds. A reference describing a general impression will move to another adjective and stay there.
Then ask what the catch cost. An error a downstream system would have flagged anyway is worth less than one nobody else was looking for, and the gap between those two is most of what you are trying to measure.
Set out in agency language, the US Office of Personnel Management's reference-checking guidance points the same way. Adding structure can enhance a reference check's effectiveness, the typical format is a structured telephone interview, and written requests for work histories typically produce low response rates and less useful information 3.
The page budgets about 20 minutes per contact and recommends a minimum of three.
Read it as practice guidance rather than as a finding. The page carries no date and the three sources under it are two decades old 3.
The general form of that call, asking a reference for one episode instead of an overall impression, works for any dimension. Here you already hold a specific claim to test it against: not whether the candidate is careful, but what happened the last time they were not.
A reference who answers in adjectives is the ordinary case rather than the evasive one, and getting something usable out of a reference who only says they were great has its own answer. The move that works on both is to stop asking for a rating and start asking for a date.
If no occasion comes back, you have learned what the word was made of.
Read the session again for what there was to check
Open the exercise before you open the argument about the candidate. A finding that somebody verified nothing is only a finding if the task contained something that could have been wrong. Plenty of well-built assignments hold nothing checkable: the brief supplies clean data, the model's output is a rewrite of the brief, and no moment arrives where a claim has to meet the world.
Read the brief for that moment. A figure that had to reconcile to a source document. A citation that either resolves or does not. A number the model produced that the supplied data cannot support.
If none of those is in there, the finding belongs to the task.
An exercise that tests checking has to contain something checkable. Planting something wrong in the starter material is the repair, and what to put in the repo so the exercise tests checking rather than typing is a design question in its own right.
The second reading is domain. A candidate who walked past a planted error in a field they have never worked in has told you less than the session appears to say, and missing a planted error can be a knowledge gap rather than a habit.
Two references who watched the same person work inside a familiar domain are describing a different situation.
Many of these disagreements close here, in material you already hold.
When both readings survive, what does the split tell you?
When the reference produces a real episode and the session genuinely held something to check, the disagreement stops being a measurement problem and becomes a description of the person. Somebody can be careful inside a domain they know and miss what needs checking outside it.
The split is then a scope decision about the role rather than a verdict about character, and it names what the first six months have to look like.
Keeping both sources is the better move in any case.
In an applied follow-up, Sackett, Zhang, Berry and Lievens put a composite of several different predictors at about .61, above any single method in their own table 5.
The figure carries a condition. A composite reaches .61 when the pieces are combined mechanically, with weights set before anyone meets the candidate, which is not what a room full of people arguing about one finalist produces 5.
Against reopening the file stands the clock. An offer that waits two days on a second reference call is an offer a competing employer can close first, and after reading both documents the answer will feel obvious.
Sometimes it is.
Meticulous entered the file as a fact about the candidate. The word leaves as an episode with a date attached, or as an adjective somebody supplied on request. Either result is worth the two days.
And the next brief gets a line this one lacked: put something wrong in the material, and say the catch is what gets read.
Common questions
Which is more reliable, a reference check or a work sample?
Neither carries a number that can break a tie between a reference check and a work sample. In the 2022 re-analysis of selection-method validity, reference checks were one of eight procedures set aside for insufficient information. The only published figure for them stays the 1998 value of .26, which traces to ten samples gathered before 1984. Work samples come out at .33 in the same paper, from 54 studies in which 53 measured people already doing the job rather than applicants. Both numbers describe something other than the two documents on the desk.
What should you ask a reference who calls someone meticulous?
Ask for one occasion. A time this person caught something wrong before it left the building, what the error was, how they found it, and what happened after. Then ask what the catch cost, since an error a downstream system would have flagged anyway is worth less than one nobody else was looking for. If the answer arrives with apparatus attached, a file name, a figure, a date, the word had something under it. If it moves to another adjective, it did not.
Does a candidate choosing their own references make the reference worthless?
No. It makes the reference a witness statement rather than a sample. A person who agreed to take the call will be favourable, and that is priced in already. What survives the selection is the specific: a candidate cannot control which episode the reference remembers, or how well that episode holds up under a follow-up question. Ask for events rather than for assessments and the selection problem shrinks to something workable.
What if the work sample never gave the candidate anything to verify?
Then the finding is about the exercise. A session cannot show verification if the brief supplied clean data and the model's output only restated the brief. Read the assignment for the moment where a claim had to meet something outside the conversation. If no such moment exists, "does not verify anything" is a description of the task, and the repair is to plant something wrong in the starter material for the next candidate. Grade this one on what it could observe.
Should you run a second work sample instead of calling the reference back?
The call is cheaper and it reaches a different thing. A second exercise gives another session under the same conditions, which mostly says whether the first one was a fluke. A reference call reaches months of work in a domain the candidate already knew. If the disagreement is about whether the habit exists at all, the call is the better instrument. If it is about whether they can do the work in front of them, run the exercise again on a different task.
References
- 1. Revisiting Meta-Analytic Estimates of Validity in Personnel Selection: Addressing Systematic Overcorrection for Restriction of Range ✓ static1.squarespace.com Supports three claims: reference checks are one of eight procedures excluded from the 2022 re-analysis for insufficient information, leaving the 1998 value of .26 as the only published figure (Table 3); the work sample estimate of .33 from Roth et al. (2005), 54 studies with 53 concurrent and no range-restriction correction; and the head-to-head reversal in which cognitive ability beat assessment centres .51 to .37 across separate meta-analyses but lost .22 to .44 when the same people took both.
- 2. The Validity and Utility of Selection Methods in Personnel Psychology: Practical and Theoretical Implications of 100 Years of Research Findings (working paper) ✓ home.ubalt.edu Supports the lineage of the .26 reference-check figure to Hunter and Hunter (1984, Table 9, k = 10), and the authors' own written caveat that the table 'may not at present be fully accurate' because many previous employers now provide only dates of employment and job titles.
- 3. Assessment and Selection: Other Assessment Methods - Reference Checking ✓ opm.gov Supports the description of a defensible reference check: adding structure enhances effectiveness, the format is a structured telephone interview, written requests for work histories produce low response rates and less useful information, about 20 minutes per contact with a minimum of three recommended.
- 4. Raising Doubt in Letters of Recommendation for Academia: Gender Differences and Their Impact ✓ s3.wp.wsu.edu Supports the claim that free-text reference language varies systematically: 624 letters for 174 candidates across eight assistant professor positions, 54 percent of letters for women carrying at least one doubt raiser and 13 percent two or more against 51 and 7 percent for men, the companion experiment with 305 professors rating research competence lower after negativity or hedging, and the two limits (no difference by letter-writer gender, no effect on teaching competence).
- 5. Revisiting the design of selection systems in light of new findings regarding the validity of widely used predictors ✓ cambridge.org Supports the claim that a composite of several predictors reaches about .61, higher than any single method in the authors' own revised table, and the condition attached to it: mechanical combination with weights set in advance.
5 sources, numbered by first appearance. Every one was opened and checked against the claim it carries. How Olive sources claims
General guidance for hiring teams. What works at one company and one volume may not transfer to yours.
Olive assesses how a person works with AI. It does not detect AI-written documents, and it never produces a score, a ranking, or a match percentage for a person. Candidates read the same report the employer reads.