Policy
The AI Productivity Studies Randomized a Tool, Not a Person
No published study shows that people who score higher on an AI-skill measure perform better on the job, so what you hand legal is not a productivity trial. The randomized AI studies varied the tool, never the person. What holds up is a content validity claim under the 1978 US federal Uniform Guidelines: a job analysis showing the role does this work, an exercise that samples it, and a record of how every candidate was administered and scored.
The takeReaching for the Boston Consulting Group percentage is the tell that a bar got set before anybody wrote down what the job requires. The productivity literature is the wrong shelf, and legal will find that faster than the hiring team will. The narrow claim is also the sturdier one: an employer who can show the work sits inside the role, and that the exercise samples it, is defending something the regulation already has a name for. The overclaim trades a defensible position for a percentage.
Where Olive fits
Open a role and see what the work shows
Olive produces no composite, and no decision about a candidate is made by a machine alone: a person writes or edits each of the six findings, every finding names the timestamped excerpt it rests on, and each released report exports with its rubric, scorer and bank versions attached. That export is one part of the administration record a content validity claim depends on; the job analysis stays the employer's to write.
Assess your shortlistWhat did the AI productivity trials actually randomize?
The tool. Each of the three trials below randomized access to an assistant, then compared output. Nobody in those studies was measured on AI skill before the work started, and nobody was sorted by it. So the results describe what happens to a task when software is added, which is a different question from what happens to a team when one candidate is hired over another.
The most quoted of them ran in mid-2023. Researchers working with Boston Consulting Group randomly assigned 758 of the firm's consultants to GPT-4 access or to none, across two experiments. In the first, 385 of them worked 18 realistic consulting tasks chosen to sit inside the model's frontier, and the groups with the assistant completed 12.2% more tasks and produced work graders scored more than 40% higher 1.
The same study carries its own warning.
On a task deliberately placed outside the model's competence, consultants using GPT-4 were 19 percentage points less likely to reach the correct answer. The control group got it right 84.5% of the time, against 60% and 70% in the two AI conditions 1.
Quoting the 12.2% without that task misrepresents the paper.
METR ran the counterexample between February and June 2025. Sixteen experienced open-source developers worked 246 real tasks on repositories they had worked on for about five years. With early-2025 AI tools allowed, they ran 19% slower. They had forecast a 24% speedup, and afterwards still believed the tools had saved them 20% 2.
The gap between what those developers believed and what the clock recorded is the part that travels.
The largest of the three measurements found a real gain with a loose range around it. Three company-run randomized trials at Microsoft, Accenture and one unnamed Fortune 100 firm pooled 4,867 developers. Access to a coding assistant raised completed tasks by 26.08%, loosely enough that the honest range runs from roughly 6% to 46% 3.
Less experienced developers adopted it more and gained more.
All three trials assigned the tool without sorting anyone by AI skill first. There was no score to be right or wrong about.
Which validity strategy is left once the trials are out?
Content validity, if the role genuinely includes the work and the exercise is a sample of it. The Uniform Guidelines on Employee Selection Procedures, adopted by four federal agencies in 1978, accept three kinds of evidence: criterion-related, content, and construct 4. The three are not interchangeable, and naming the one being claimed is most of what legal is asking for.
Criterion-related evidence is what the productivity studies resemble and are not. The Guidelines ask for empirical data showing that the selection procedure predicts or correlates with important elements of job performance, and that sentence sits at 1607.5(B) 4. Randomizing a tool produces no such data, because the score being validated was never taken.
Content validity asks a smaller question and answers it with a job analysis.
A procedure earns content validity to the extent that it is "a representative sample of the content of the job", which is the test written at 1607.14(C)(1) 5. A knowledge, skill or ability can qualify without being a literal sample, provided it is operationally defined and is a necessary prerequisite to successful job performance.
Two limits sit beside that test, and both bite here. A procedure resting on inferences about mental processes cannot be supported solely or primarily by content validity, which rules content validity out as the main support for an instrument claiming to measure judgment as a trait.
Nor is content validity appropriate where the skill is one an employee will be expected to learn on the job 5.
So the exercise that survives is the one that looks like an ordinary Tuesday in the role. If the analysts on the team check model output against a filing before it leaves the building, the exercise hands a candidate model output and a filing.
Scope is wider than a formal test.
The Guidelines define a selection procedure as any measure used as a basis for an employment decision, reaching from paper-and-pencil tests through "informal or casual interviews". A procedure with adverse impact is treated as discriminatory unless it has been validated 7.
When an informal AI exercise becomes a test you have to validate turns on that definition rather than on what the step is called internally.
Why is there no published criterion study on AI skill?
None has been published, and the regulation shows why one is hard to run. A criterion study for an AI-skill measure needs enough candidates scored on the same instrument, a measure of their job performance, and, in a predictive design, time for that performance to show. Before any of that, the regulation asks a prior question at 1607.14(B)(1): is such a study technically feasible in that employment context 5?
Feasible is a defined word, spelled out in the Guidelines' own definitions at 1607.16. The study needs an adequate sample of people available for it, a sufficient range of scores on both the procedure and the performance measure, and "unbiased, reliable and relevant measures of job performance" 6.
A single role filled in a single year can easily fail the first condition.
The absence is worth stating to legal in a sentence, because silence there reads as an implication.
In 2022 Sackett, Zhang, Berry and Lievens re-analysed the meta-analytic selection literature. Their revised ranking of the predictors with evidence behind them runs: structured interviews, job knowledge tests, empirically keyed biodata, work samples, and cognitive ability and integrity tests tied behind those.
No AI-skill or AI-collaboration measure appears on that list.
The same re-analysis moved the numbers down. Mean validity across the top five predictors fell from .49 to .37 once an over-applied range correction was dropped, while all five stayed in the top group 8. The validity table everyone quotes was corrected in 2022, which is worth knowing before any of it reaches a memo.
Published coefficients are also indirect in a way the decimals hide. Much of the research behind them is concurrent, run on people already hired and doing the job: 98% of the studies in the work sample meta-analysis those authors rely on, and 74% in the interview meta-analysis 8. A number built that way describes employees who already cleared a hiring bar.
Asking candidates to rate their own AI skill fills none of that gap. What a one-to-five self-rating actually tracks is a separate question, and the answer does not help the file either.
Write the memo legal asked for, in the Guidelines' own words
One sentence names the strategy. Write that the exercise is supported by content validity because it is a representative sample of work the job analysis found in the role, and that every candidate reaching that stage receives it under the same instructions and scoring. Then attach the analysis, because the sentence is worthless without it.
The sentence is only as good as the paper behind it.
The job analysis goes in first, dated, naming the work behaviours in the role and who supplied them. Under each skill the exercise measures, put an operational definition, written in the words of the work rather than of a trait. The exercise goes in next, with its scoring key and the anchors a second reader would use.
Last is the administration record: who took it, under what instructions, scored by whom, and the reason written beside each outcome.
What a position statement has to show about the AI round is that same file, produced under a deadline.
Of the three strategies, content validity has the weakest outside check, and an employer who wrote the analysis to fit an exercise already built holds the same paperwork as an employer who did the analysis first. The Guidelines ask for a record, not for proof that the record was made in good faith.
Keep the productivity percentages out of the claim, or label what they are doing there. Naming the Boston Consulting Group result as background on why the role changed is fine.
Offering 12.2% more tasks completed as evidence that a higher-scoring candidate will perform better is the sentence a plaintiff's expert reads first, and the file behind it says something narrower. Whether an AI-skills assessment holds up when a rejected candidate challenges it turns on exactly that gap.
Legal asked for evidence about people and got a productivity percentage back.
The file that answers the question is one the employer builds: a dated analysis of the job, an exercise that samples the work it found, and a record of how each candidate was run through it. The Guidelines have carried that route since 1978. Write the claim they already recognise.
Common questions
Can the BCG study be cited as evidence that AI skill predicts performance?
No. The Boston Consulting Group experiment randomized access to GPT-4 across 758 consultants in two experiments and measured output on set tasks, 18 of them in the first. It never measured anyone's AI skill and never sorted anyone by it, so it supports a claim about what the tool does to a task and no claim about what a candidate's score predicts. The same paper also reports a task outside the model's competence where the AI groups were 19 percentage points less likely to be right. Cite it as background on why the role changed, and keep it out of the validity claim.
What separates content validity from criterion-related validity here?
Criterion-related evidence is empirical data showing the procedure predicts or correlates with important elements of job performance. Content evidence shows that the content of the procedure is representative of important aspects of performance on the job. The first needs a study with scored candidates, later performance measures, and enough people for the study to mean anything, which is why the Uniform Guidelines gate it on technical feasibility. The second needs a job analysis and an exercise that samples what the analysis found. For a new AI exercise at one employer, only the second is usually available.
Does an AI exercise have to be validated at all?
The validation burden under the Uniform Guidelines attaches where a selection procedure has adverse impact, and the definition of a selection procedure is broad. It covers any measure used as a basis for an employment decision, from written tests through informal or casual interviews. Calling a step informal does not move it outside that definition. Whether the burden has been triggered is a question for employment counsel on the facts, and none of this is legal advice. What is worth doing before the question arrives is holding the job analysis and the administration record, which cost little now and are hard to rebuild credibly later.
Can a vendor's validity number substitute for a job analysis?
A vendor coefficient describes its instrument on the sample it was computed on, often people already employed and doing a job somewhere else. Much validation research in this field is concurrent rather than predictive, so the number may not describe an applicant pool at all. A content validity claim is about your role, so it rests on evidence only the employer holds: what the job requires, who said so, and how the exercise samples it. Ask what the number was computed on before it goes anywhere near the memo.
How long would a proper criterion study take?
Longer than most hiring cycles and larger than most roles. The study needs candidates scored on one instrument, hired in sufficient numbers, then rated on performance once there is performance to rate, using a measure that is unbiased, reliable and relevant. A single role at a single employer rarely produces enough people, and the performance rating is usually the hardest piece to build honestly. Employers with the volume for it should still expect the first usable answer to describe their own instrument in their own setting, rather than AI skill in general.
References
- 1. Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality (Working Paper 24-013) ✓ mitsloan.mit.edu Supports both halves of the BCG result used here. The pre-registered experiment ran 758 consultants across two arms; in the inside-the-frontier arm, 385 subjects on 18 realistic tasks completed 12.2% more tasks with more than 40% higher graded quality, and on the outside-the-frontier task the AI conditions were 19 percentage points less likely to be correct (84.5% control against 60% and 70%).
- 2. Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (arXiv:2507.09089) ✓ arxiv.org Supports the claim that a randomized trial varied the tool and not the person: 16 experienced developers, 246 real tasks, 19% slower with AI tools allowed, against a 24% forecast speedup beforehand and a 20% believed speedup afterwards.
- 3. The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers ✓ economics.mit.edu Supports the pooled developer result and its width: three company-run randomized trials across 4,867 developers, a 26.08% rise in completed tasks with a standard error of 10.3% (roughly a 6% to 46% range), with less experienced developers adopting more and gaining more.
- 4. 29 CFR 1607.5 - General standards for validity studies ✓ law.cornell.edu Supports the three acceptable validity strategies and what criterion-related evidence must consist of. The page reads: "For the purposes of satisfying these guidelines, users may rely upon criterion-related validity studies, content validity studies or construct validity studies" and "Evidence of the validity of a test or other selection procedure by a criterion-related validity study should consist of empirical data demonstrating that the selection procedure is predictive of or significantly correlated with important elements of job performance."
- 5. 29 CFR 1607.14 - Technical standards for validity studies ✓ law.cornell.edu Supports the content validity test and its two limits, and the technical-feasibility gate on a criterion study. The page reads: "A selection procedure can be supported by a content validity strategy to the extent that it is a representative sample of the content of the job"; "A selection procedure based upon inferences about mental processes cannot be supported solely or primarily on the basis of content validity"; "Content validity is also not an appropriate strategy when the selection procedure involves knowledges, skills, or abilities which an employee will be expected to learn on the job"; and "Users choosing to validate a selection procedure by a criterion-related validity strategy should determine whether it is technically feasible (as defined in section 16) to conduct such a study in the particular employment context."
- 6. 29 CFR 1607.16 - Definitions ✓ law.cornell.edu Supports the three conditions that make a criterion-related study technically feasible: an adequate sample of persons available for the study, a sufficient range of scores on the selection procedure and job performance measures, and the ability to devise "unbiased, reliable and relevant measures of job performance".
- 7. 29 CFR Part 1607 - Uniform Guidelines on Employee Selection Procedures (1978), sections 1607.16(Q) and 1607.3(A) ✓ govinfo.gov Supports the breadth of the selection-procedure definition, which reaches "informal or casual interviews and unscored application forms", and the rule that a procedure with adverse impact is considered discriminatory unless validated under the Guidelines.
- 8. Revisiting Meta-Analytic Estimates of Validity in Personnel Selection: Addressing Systematic Overcorrection for Restriction of Range ✓ static1.squarespace.com Supports the revised predictor ranking and the two figures used from it: mean validity across the top five falling from .49 to .37 with the rank order largely unchanged, and the concurrent share of validation studies (98% in the work sample meta-analysis, 74% in the interview meta-analysis).
8 sources, numbered by first appearance. Every one was opened and checked against the claim it carries. How Olive sources claims
General guidance, not legal advice. Hiring rules differ by state and country and change often; check anything here against your own counsel before you act on it.
Olive assesses how a person works with AI. It does not detect AI-written documents, and it never produces a score, a ranking, or a match percentage for a person. Candidates read the same report the employer reads.