What's the Difference Between a Validated and a Bias-Audited Assessment?

What does each one actually prove?

A bias audit proves arithmetic: it reports the rate at which each sex and race/ethnicity group was selected by the tool, and the ratio of each rate to the highest one ³. Validation proves job-relatedness: evidence that the procedure samples or predicts the work of one particular job ¹. One is a statement about groups of people. The other is a statement about one job.

The confusion is profitable, which is why nobody selling an assessment volunteers the distinction. "Independently bias-audited" is a phrase a vendor earns by paying an auditor to compute ratios, often on data pooled from other customers. It is a real obligation in New York City and it is worth doing. It also says nothing whatsoever about whether the tool predicts performance in the role you are filling.

The trap is the direction of travel. Validity evidence does not generalize to a new job on its own; a bias audit's arithmetic does not need to, because it never made a claim about a job in the first place. So a vendor holding a criterion study on customer-service hires and a clean audit has told you two true things, neither of which is about the data-analyst opening on your desk. Before you buy an AI-skills assessment, sort every claim into one of those two columns and see which column is empty.

Why is validation always validation for one job?

Because the evidence is built out of a specific job's content. A criterion-related study starts with a review of job information to find the performance measures "relevant to the job or group of jobs in question" ¹. A content-validity claim holds only "to the extent that it is a representative sample of the content of the job" ¹. Change the job and the sample you would have to draw changes with it.

Criterion-related validity is a measured correlation between how people did on the procedure and how they did on the job. The Uniform Guidelines treat that relationship as established when it is statistically significant at the 0.05 level ¹, which means the study had real applicants or incumbents, in a real job, scored against a criterion someone had to define. Every one of those choices is local to the occupation studied.

Borrowing the study for a second job is allowed, and the conditions are specific: incumbents must perform substantially the same major work behaviors, shown by appropriate job analyses on both jobs ². That is a document you can ask for by name. If the answer is that the assessment measures general problem-solving and therefore applies everywhere, you have been handed a construct claim with no construct study behind it.

Work the example through. A customer-service criterion study measured a correlation between test performance and something like supervisor ratings or average handle time. A data-analyst role has neither of those criteria. The work behaviors differ, the criterion measures differ, and the labor market the sample was drawn from differs. Nothing about the earlier study is wrong. It is simply about a different job, and the vendor is not lying when they cite it. Whether one assessment can cover every department turns on exactly this paragraph of the Guidelines.

What does a bias audit measure, and what does it skip?

Selection rates and impact ratios, calculated separately for sex categories, race/ethnicity categories, and intersectional categories of the two ³. That is the whole instrument. New York City's rule requires no job analysis, no criterion study, and no showing that the tool relates to the job it screens for. A clean set of ratios is a clean set of ratios, and it is not a statement that the tool works.

The rule most vendors mean by "bias-audited" has edges worth knowing. The audit must be no more than a year old before use. The auditor has to be independent, which the rule defines negatively: not involved in developing, distributing or using the tool, no employment relationship with the vendor or the employer, no direct or material indirect financial interest in either ³. The published summary has to state the audit date, the source and explanation of the data, the number of people assessed who fell into an unknown category, and the applicant counts, rates and impact ratios for every category ³.

Two of those edges change how you read a summary. An auditor may exclude any category representing less than 2% of the audit data from the impact-ratio calculations, provided the exclusion and its justification appear in the summary ³, so the groups with the fewest applicants are the ones most likely to be missing from the table you are shown. And an employer may rely on an audit run on other employers' historical data only if it gave its own historical data to that auditor, or if it has never used the tool ³. "Our vendor is audited" does not survive that sentence once you have been using the tool for a year. Running your own adverse-impact check is a separate exercise from reading a vendor's.

The compliance claim also outruns the compliance. When researchers checked 391 employers for Local Law 144 compliance, they found 18 that had posted a bias audit report and 13 that had posted the required transparency notice ⁷. Ask for the URL of the posted summary rather than the assurance that one exists.

How do the two fit together under the law?

Sequentially. A bias audit tells you whether a selection procedure has adverse impact; validation is the defense once it does. Under the Uniform Guidelines, a procedure with adverse impact is considered discriminatory unless it has been validated ⁴. The audit is the trigger and the validity file is the answer, which is why a vendor holding one without the other has brought you half a file.

The four-fifths rule is where the audit's numbers acquire consequences. A selection rate for any race, sex or ethnic group below four-fifths of the highest group's rate is generally regarded by the federal enforcement agencies as evidence of adverse impact, and smaller differences may still constitute adverse impact where they are significant in both statistical and practical terms ⁵. An impact ratio of 0.84 is not a pass mark. It is a number you need to be able to explain.

Explaining it is the validity file's job. A selection procedure has to be job-related and consistent with business necessity for the job it is used on ⁶, and the Guidelines add a duty most buyers miss: where two procedures are substantially equally valid, use the one shown to have the lesser adverse impact ⁴. That makes the audit an ongoing design obligation rather than an annual filing. The documents to have on hand before an inquiry are the two files side by side.

Two honest limits. Local Law 144 compliance is not a Title VII defense. The city asks for arithmetic and publication, not for a validity study. And the local obligation itself keys off whether the tool substantially assists or replaces discretionary decision-making, which the rule defines as relying solely on a simplified output, weighting it above every other criterion, or using it to overrule human conclusions ³. A step that produces no simplified output has a different question to answer, and what a defense actually looks like still starts with the job analysis.

Ask the vendor for these six documents

Six documents, and the answers are short enough to give on a call. The job analysis behind the validity claim. The occupation the criterion study was run on, with its sample size. The criterion measure: what job performance was scored as. The date of the most recent bias audit. The data that audit ran on. And the categories the auditor excluded, with the reason.

1. The job analysis. Which occupation, done when, by whom. Without it there is no validity claim of any kind, because every strategy in the Guidelines starts there ¹. 2. The criterion study, with the occupation named. Sample size, the criterion measured, the significance reported ¹. "Validated across 200 companies" is not an answer; an occupation is. 3. The transportability memo, if the study was run on a different job: job analyses on both jobs, showing substantially the same major work behaviors ². 4. The bias audit summary URL. Dated, with the source of the data stated ³. If the data came from other employers, ask what happens when your own year of usage data exists. 5. The excluded categories. Any category under 2% that was dropped from the ratios, and the auditor's justification for dropping it ³. 6. The impact ratios themselves, for sex, race/ethnicity and intersectional categories ³. The table, not a headline sentence about passing.

Then ask the question that sorts vendors fastest: what would this assessment have to show for you to tell a customer not to use it for a role? A vendor with a real validity file can answer that in a sentence, because the file names the jobs it covers and therefore names the jobs it does not. The rest of the vendor question set works the same way: ask for the document, never the adjective. Olive publishes its own version of that file, including the parts not yet measured: See how Olive measures this.

See what gets scored >