Vendor Validity Evidence Transfers Only If the Jobs Match

What does 'validated' actually claim?

That a score from the procedure related to some measure of job performance, in some jobs, at some point. It is a claim about a study rather than a property of the product, which is exactly the reading a certification-shaped word invites. Three variables decide whether the claim reaches your roles at all, and a deck rarely names any of them: the criterion, the jobs, and the year.

The numbers themselves moved recently enough that material built before 2022 is quoting a superseded table. Sackett, Zhang, Berry and Lievens examined the five approaches used to build range-restriction corrections in selection meta-analyses, concluded each one often produces substantial overcorrection, and re-derived the estimates: most procedures keep their rank order, with mean validity reduced by .10 to .20 points, and structured interviews rather than cognitive ability came out top ¹. The mean across the old top five was .49; across the revised top five it is .37 ¹. Schmidt and Hunter's 1998 figures, still the most quoted table in hiring, are the ones being revised ².

A structural caveat sits under every coefficient in either table and is almost never stated. The overwhelming majority of selection validation research is concurrent, measured on people already hired and doing the job rather than on applicants: 98% of the studies in one work sample meta-analysis, 95% in a situational judgment test meta-analysis, 74% in an interview meta-analysis ¹. Those people all cleared somebody's hiring bar before they were measured, so the study is not direct evidence about how the method sorts an applicant pool. The methods still predict. The evidence behind them is more indirect than a column of decimals suggests, and that is why your own pipeline numbers will not reproduce the published ones.

Name the criterion the tool was validated against

The criterion is what the tool actually predicts, and four vendors can all say validated while predicting four different things. Supervisor ratings, first-year tenure, sales numbers and ticket throughput are not interchangeable. A tool validated against turnover predicts who stays, which is a legitimate thing to want and is not the same as who does the work well.

Ask what the assessment is about, too, because content relevance carries more of the validity than format does. In the job knowledge meta-analysis behind the 2022 revision, all 164 studies together produced a mean observed validity of .22, while the 59 studies using knowledge tests built for the job in question produced .31, rising to .40 once corrected for unreliable performance ratings ¹. That is a comparison between subsets of one meta-analysis rather than a controlled experiment, and job knowledge tests presuppose candidates who already have the knowledge, so it is evidence about hiring experienced people. The direction still travels: an assessment about this job beats a general instrument about people.

One further question about the number itself: ask what it is made of. The .31 reported for integrity tests in the 2022 revision is a sample-size-weighted average of two meta-analyses that disagree by more than a factor of two, .44 against .18, and a dedicated attempt to reconcile them failed ¹. A single point estimate can be a compromise between two studies that cannot be reconciled. A vendor quoting one has usually not looked, and that is worth knowing before the number goes into a board paper.

When does someone else's study transfer to your role?

When three conditions hold, and they are written down. The Uniform Guidelines, the 1978 US federal regulation at 29 CFR 1607.7, let another employer's criterion-related study be used when the evidence clearly demonstrates the procedure is valid, when appropriate job analyses on both jobs show the incumbents perform substantially the same major work behaviors, and when the studies include a study of test fairness for each race, sex and ethnic group significant in the borrowing user's relevant labor market ³.

The job analysis is the load-bearing document and the one that rarely arrives. It is what turns a study about somebody else's contact-centre agents into evidence about your hybrid analyst role, and without it a buyer is accepting a claim about jobs nobody in the room has seen. Ask which jobs were in the study and ask for the analysis on that side, then get one done on your own role, because the regulation asks for both and the buyer's half is the half that gets skipped. The third condition has an exit written into it: where a study meets the first two, carries no investigation of test fairness, and it is not technically feasible for the borrowing employer to run one, the study may be used until evidence from elsewhere shows unfairness ³. The same section adds a limit worth quoting back: where there are variables in the other study likely to affect validity significantly, the user may not rely on it ³. That is public legal fact and not advice, and it names one jurisdiction; whether a particular purchase clears it belongs in front of counsel.

Three more questions fit in one email: the sample size, the year, and whether the model has been retrained since. A retrained model is not the model that was validated, and a retrain date is rarely volunteered. This bites harder for machine-scored formats, where the published evidence is thinner. A peer-reviewed study of one vendor's machine-scored video interviews reports an uncorrected .24 with job performance across five organisational samples totalling 1,124 people, four of its five authors employed by that vendor, against an uncorrected .32 for human-rated structured interviews in the same comparison ⁴. Both figures are uncorrected, which makes the comparison internally fair, and neither is the corrected .42 usually quoted for structured interviews. That figure rests on five samples, against the 105 effect sizes behind the structured-interview estimate it is set beside ⁴.

What to do when transport is weak

Run a small concurrent check against people already doing the job rather than commissioning a full study. Give the assessment to a dozen or two current employees whose performance you can already characterise, and look at whether the results order them in a way anyone recognises. It is weak evidence by design and it is local, which is precisely the trade being made.

Two things to hold steady while doing it. A check on a dozen people is not a validation study and should never be described as one, inside the company or outside it. And the range-restriction problem that produced the 2022 revision applies to your own pipeline data as well, so your numbers will sit below the published ones for reasons that have nothing to do with the tool ¹.

Where transport stays weak, the honest conclusion is usually about what job the tool should be given. An instrument that is one input into a human decision needs less evidence behind it than a gate that stands alone and ends applications, because the failure mode is different and recoverable. That is also the cheapest configuration change available, and it is reversible in an afternoon. How to run the tool before it becomes a gate is the pilot design question; the wider purchase checklist, integrations and pricing included, is how to evaluate an AI-skills assessment before buying one.

On Monday, add three lines to the vendor questionnaire: what was the criterion, which jobs were in the study and can the job analysis be shared, and when was the model last retrained. Write down which vendor answered which, because that table is the most useful artifact the evaluation will produce.

See what gets scored >