Run the Exercise on Your Team. The Number It Returns Will Read Low.

Does testing the current team count as a validation study?

Running the exercise on the current team and correlating the scores with job performance is a concurrent validation study, and federal selection law has a section for it. The Uniform Guidelines on Employee Selection Procedures, written in 1978, set out technical standards for criterion-related validity: review the job, pick criterion measures that represent important work behavior, draw a sample, and report the relationship ¹. Nothing about the design is improvised.

Most of the evidence you have read was built the same way. Sackett, Zhang, Berry and Lievens counted the concurrent share inside the meta-analyses the field cites: 98% of the studies in Roth's work sample analysis, 95% in the situational judgment test one, and 74% in the interview one ³.

So the objection that incumbents are the wrong people to test is an objection to most of that literature too.

What the study cannot tell you is how the exercise would sort an applicant pool. A concurrent study relates scores to performance among people who already cleared your bar. The Guidelines ask that the sample "should insofar as feasible be representative of the candidates normally available in the relevant labor market" ¹, and a payroll is not that.

Career changers, returners and applicants from employers that banned the tools may be missing from the sample entirely, and nothing in the output will mention them.

Why does the correlation come back low?

Because both sides of the comparison can be squeezed before you start. The people taking the exercise were hired once already, so the bottom of the score range thins out: some applicants who would have scored worst are not on the payroll. The performance side can be squeezed too, where the weakest performers have left. Narrower measures shrink a correlation even when the exercise works, though among current staff the squeeze is generally milder than corrections assume ³.

The 1978 definitions section names the problem without using the phrase. A study is technically feasible, it says, only where the employer can obtain "a sufficient range of scores on the selection procedure and job performance measures" ².

Restriction of range is not a statistical nicety. It is the second thing the regulation asks about.

There is a published expectation for the size of the result. Roth and colleagues' work sample meta-analysis, the closest analogue to a job-shaped exercise, pooled 54 studies and 10,469 people, and 53 of the 54 tested people already doing the job ⁴. Mean observed correlation, before any correction: .26.

If your exercise is a work sample in that sense, .26 is a fair starting expectation, though nothing in that evidence is about work done with AI.

Don't correct the number for range restriction

The formula that turns a low in-house correlation into a publishable-looking one is a range-restriction correction, the same kind a 2022 re-analysis of this field's published estimates found wanting. Sackett, Zhang, Berry and Lievens examined the five approaches commonly used to build range-restriction corrections in selection meta-analyses and concluded that each often produces substantial overcorrection ³.

Across the procedures that ranked high before, revised mean validity fell .10 to .20 points, and the paper names three causes for that, of which the correction is only one.

The structured interview shows the size of the step. In McDaniel and colleagues' 1994 meta-analysis, its mean observed correlation with rated performance was .28. Correcting for unreliable performance ratings lifted it to .37, and correcting for range restriction lifted it again to .51, a gain of 38% from that step alone ³.

The .51 is the number the industry quoted for two decades.

Report what you measured, and never set an uncorrected in-house correlation beside a corrected published one, because the published one has been moved and yours has not.

If somebody on the panel wants the corrected version, ask which correction and on what assumed distribution.

The same asymmetry makes a supplier's evidence hard to read. Whether a vendor's validity study transfers to your roles turns on the job analysis behind it, not on the size of the coefficient in the brochure.

Write the criterion before anyone takes the exercise

The performance measure decides what the study is about, and an in-house attempt easily defaults to a manager's overall impression of the person. The Guidelines are direct here: "Whatever criteria are used should represent important or critical work behavior(s) or work outcomes" ¹. An overall impression can also represent tenure, visibility, likeability and one remembered bad quarter, none of which your exercise claims to measure.

The definitions section adds a third condition before a study counts as feasible: "unbiased, reliable and relevant measures of job performance or other criteria of employee adequacy" ². Reliable means two people rating the same work land in roughly the same place.

Anchoring the scale in writing is the cheap fix, and the evidence behind it is narrow. Levashina and colleagues' review reports Taylor and Small's 2002 meta-analysis of 19 past-behavior interview studies. Interviews using anchored rating scales showed higher validity (.35 against .26) and higher agreement between raters (.77 against .73) than interviews without them ⁵.

Those are interviews, not exercises, so read the finding as a direction only.

Agreement is worth measuring before it is worth correlating with anything, and a raw share of matching ratings overstates it until you discount chance.

For an AI exercise, pick a criterion the job already produces: work that came back from review clean, decisions the second reader did not reverse, cases closed without rework. Have the managers rate those before anyone sees an exercise score, and never in the same meeting.

How many people do you need before the exercise gates anyone?

More than most teams have. The Guidelines set their own bar at a relationship "statistically significant at the 0.05 level of significance" ¹, and where that bar sits depends on how many people were in the study. At 30 people it takes a correlation of about .36 to clear. At 50, about .28. At 100, about .20.

The closest published figure, for work samples measured on incumbents, is .26. A thirty-person study of an exercise that works will usually come back with nothing to report.

The failure there is arithmetic. Small employers meet the same wall from the other side when they try to compute a selection rate: whether any fairness number is meaningful at that size is settled by the headcount long before the data arrives.

Run it anyway, as one supporting piece of the argument. A small study will still catch a task that rewards typing speed and expose two reviewers disagreeing before a candidate is on the other end of it.

Pair it with the job analysis that carries a content-validity case, and pilot the exercise in shadow mode before it gates anyone.

If the study waits for a headcount that can carry it, it never happens. An employer holding out for a defensible number leaves the exercise sorting people meanwhile, on no evidence at all.

The condition the 1978 drafters put second is the one your study cannot settle on its own. They wrote it as a question about scores rather than about people: whether the range on hand can "produce validity results which can be expected to be representative of the results if the ranges normally expected were utilized" ².

The ranges normally expected are the applicants. Your team is who is left after them.

See what gets scored >