Same Model, Different Configuration, Different Impact

What does a vendor's bias audit actually certify?

A narrow question that was agreed in advance. The first publicly documented cooperative audit of a commercial candidate-screening product, run in summer 2020, was scoped to five questions and found that the vendor faithfully implemented its stated fairness guarantees, with safeguards sufficient to reasonably ensure compliance with the four-fifths rule ¹. Read the scope before reading the verdict.

What sat outside that scope, by prior agreement, is most of the list a buyer cares about: the choice of fairness objective and metric, differential validity, intersectional fairness, groups outside the federal categories, and the vendor's own annual back-testing of models already deployed ¹. A pass therefore means the code implements the check the vendor says it implements and that neither human error nor a bad actor subverts it easily. It is not a finding that the tool is fair, valid, job-related, or predictive of anything at all.

New York City's rule shows the same shape written into law. Under the Department of Consumer and Worker Protection's 2023 final rules, a bias audit computes selection rates and impact ratios by sex, by race and ethnicity, and by intersectional category; an independent auditor may exclude any category representing less than 2% of the audit data; and an employer that has never used the tool may rely on an audit built entirely on other employers' historical data, or on synthetic data where too little real data exists ². Three limits pointing the same way. It measures group rates and nothing else, the carve-out most often drops the smallest groups in the pool, and a posted audit need not describe your candidates at all. The difference between a validated assessment and a bias-audited one is the distinction that collapses here most often.

Which part of the impact do you own?

The cutoff, the stage, the tuning target and the override. Each is a customer setting, each moves the selection rate a tool produces, and none of them appears in a vendor's audit, which ran on the vendor's own settings before you had any. The same weights at a different pass mark select a different set of people, so the impact ratio in a certificate is a fact about the cutoff the auditor used.

Agency guidance names the cutoff as the mechanism directly. The EEOC's 2022 technical assistance on the ADA and algorithmic assessment, which carries no force of law and has been read from an archive since the agency removed it in January 2025, says a tool screens somebody out when a disability lowers their performance on a selection criterion and they lose the opportunity as a result, and its worked example is a programmed pass mark: a business requiring 90 percent on a gamified memory assessment rejects a blind applicant who cannot play the game, though that applicant may have an excellent memory ³. That example is the agency's own hypothetical. Screening out is not automatically unlawful, and job-relatedness with business necessity remains a defence. The design point survives all three caveats, because the number that rejects people is a number somebody chose.

The same document undercuts the substitution that makes a certificate feel sufficient. A vendor's bias-free claim usually means steps taken against Title VII adverse impact on race, sex, national origin, color and religion, and those steps are typically distinct from what disability requires, because each disability is unique and an individual can still be screened out regardless of how well other people with disabilities fare on the same assessment ³. That is an agency position with no study cited for it, and it describes a risk without naming any product. It also means a group-rate audit and an individual accommodation route are two separate obligations, and clearing one says nothing about the other.

Whether the tool orders candidates or merely flags them is a configuration choice too, and it changes what the output even is. In Bloomberg's tests both models favoured whichever resume they saw first, naming the first candidate the most qualified 56% of the time for GPT-3.5 and 28% for GPT-4, against a one-in-eight chance if arrival order did not matter ⁴. The reporters disclosed that as a limitation and neutralised it by shuffling order, so it is not their headline finding, and it is specific to two 2023 model snapshots under one prompt. What it shows is that an ordered shortlist can be partly an artifact of the order you fed in, which makes the ordering a fact about how you called the tool as much as about the candidates in it.

Ask the deployment questions before the vendor questions

Five questions about your own configuration tell you more than a vendor document does, and you can answer all five without anybody's cooperation. They describe what the tool does to a real applicant pool, which is precisely the thing a certificate about an instrument was never about. Write the answers down before the renewal meeting, because they set the agenda for it.

1. What output advances a candidate? The exact threshold, in the units the tool actually emits. 2. What happens to everyone below the line? Rejected, deprioritised, or reviewed by a person anyway. These are three different products built from the same tool. 3. Which stage does it gate? A tool at the top of the funnel touches every applicant; the same tool at the final round touches a population four rules have already shaped. 4. What was it tuned to predict? If the training label was who got hired, the tool learned your previous hiring rule, including whatever was wrong with it. 5. Who may disagree with the output, and how often do they? An override path nobody uses is not a safeguard, it is a line in a contract.

Using a vendor does not move any of this outside the law. California's Civil Rights Council amended the state employment regulations to cover automated-decision systems, effective in October 2025; the regulations name screening resumes for particular terms or patterns and analysing facial expression, word choice or voice in online interviews as examples, and they make an agent acting for an employer, including a vendor running the system, an employer under the Act ⁵. That expands who can be sued and leaves the buyer's own exposure in place. It imposes no audit or notice duty of its own, and it reaches employers regularly employing five or more people. Whether it reaches your jurisdiction and your setup is a question for counsel.

The vendor questions come after, and they land harder once the configuration is written down. What to ask a screening vendor that says it is bias tested is a better conversation when you can name which cutoff, which stage and which population you are asking about.

Write your configuration down in five lines

One page, five lines, no tooling: the cutoff, the disposition of everyone below it, the stage, the tuning target, and the override path with its real usage rate. That page is the object any fairness question is actually about, and almost nobody has written it, which is how a vendor's document ends up standing in for it at renewal.

With it in hand you have something to read the vendor's evidence against. Treat any fairness claim as attached to one configuration and one population until it has been re-run on yours. Ask which cutoff the audit assumed, whose applicants it was computed on, which categories were excluded for size, and what it measured that is not a group selection rate. If those answers do not come back, that silence is itself information about what you are buying.

Then run it before it gates anybody. A pilot beside your existing round produces the two things no certificate can: your own selection rates, on your own pool, at your own cutoff, and a sense of what the output looks like when a person disagrees with it. How to pilot an assessment vendor before making it a hiring gate is the mechanics of that, and it is the step most often skipped because the audit document feels as though it already happened.

The frame has a limit worth naming. Splitting the question by who can change what is a way of deciding where to look first, not a way of allocating blame. The employer who set the cutoff and the vendor who built the model are both inside the process, and a candidate rejected at that boundary experiences a single decision.

See what gets scored >