How to Read a Published Bias Audit Before You Apply

What the Numbers Actually Measure

A bias audit posted under New York City's Local Law 144, the 2023 rule behind these postings, computes exactly two things: a selection rate for each demographic category, and an impact ratio comparing that rate to the highest-scoring group's rate ¹. It says nothing about accuracy, nothing about whether the tool is job-related, and nothing about whether any individual candidate, including you, was treated fairly. A tool can pass this audit and still be a poor measure of the job.

The number most audits report against is the four-fifths rule: a group's selection rate below 80% of the highest group's rate is generally treated by federal enforcement agencies as evidence of adverse impact ². Treat that as a screening threshold, not a verdict either way. The agencies that wrote the rule said directly that it is "not intended as a legal definition" and does not resolve whether unlawful discrimination actually occurred, so a ratio above 0.80 is not a clean bill of health and one below it is not proof of discrimination on its own ³.

Read the fine print on categories too. An auditor may drop any group representing less than 2% of the data used in the audit from the ratio calculation entirely, and that carve-out most often removes the smallest groups in the pool, which are frequently the ones a bias audit exists to protect. If a category you belong to shows no row at all in the summary, that is usually why.

A real summary usually has five parts worth finding on the page: the selection rate per category, the impact ratio, which categories were scored versus dropped, the date the audit was run, and a line naming the independent auditor. A summary missing the auditor's name, or naming the vendor auditing its own product, is telling you something before you read a single ratio.

Read the Denominator Before the Ratio

Ask what data actually produced the numbers before you trust the ratio itself. Under the rule, an employer that has never used the tool before may lawfully post an audit computed entirely on other employers' historical data, or on synthetic test data where too little real data exists ¹. A first-time buyer's posted audit can describe nobody who has ever applied to that specific company.

Check the date and the notice too. The law requires a bias audit no older than one year and a candidate notice at least ten business days before the tool is used ⁴; an audit dated eighteen months ago sits on a page that already concedes the law applies, so it tells you the employer is not current on a duty it accepted, which is itself informative before you read a single number.

Most employers post nothing at all, which is its own kind of evidence. A study that sent investigators to check 391 employers' postings under this law found only 5% had posted a bias audit report, and the researchers coined the term "null compliance" for exactly this: an absent audit often means the employer decided the law does not apply to its tool, not that it broke the law outright ⁵. Separately, when New York State auditors re-reviewed 22 employers already flagged for scrutiny, they counted at least 17 instances of potential non-compliance where the city's own review had found one: audits not done by an independent auditor, missing selection-rate and impact-ratio calculations, and unexplained reliance on historical data ⁶. Those employers were picked precisely because they raised questions, so the rate says nothing about employers in general, but it does show that a posted audit can fail the rule's own requirements and still be the public record.

None of this means an absent or thin audit says nothing. It means it says something narrower than "the tool discriminates": that the employer has not yet done, or has not yet published, the specific piece of paperwork the law requires. Whether that reflects the underlying tool, an honest oversight, or a considered judgment that the law does not reach this particular use is not something the missing page can tell you either way, and guessing which one is true rarely helps.

Does a Clean Audit Mean the Tool Is Fair?

No, and that is the honest, checkable claim worth carrying into any posting you read: a published bias audit tells you more about whether an employer takes the process seriously than about whether the tool is fair to you specifically. It is process evidence, not a personal guarantee, precisely because it reports group rates rather than individual outcomes, and a group rate can look unremarkable while still saying nothing about the particular resume you sent in.

What employers are told to do when running one is worth reading directly rather than guessing at: how to run an adverse impact audit when the vendor holds the data covers the same arithmetic from the side that commissions it, including which document a serious employer is expected to produce and which a rushed one skips.

The more useful line in the notice attached to the audit is often the one telling you which stage of the process the tool actually touches, since that tells you where to spend your effort in the loop rather than guessing at the whole pipeline. If the tool only ranks resumes before a human reviews the shortlist, the ratio you just read describes one narrow gate, not the whole decision about you.

And the audit is not the only document you can request: you can also ask for your own results directly, which is a more specific thing to hold than a group-level ratio you were never part of computing. Reading both together tells you more than either does alone: one describes a population you belong to, the other describes you.

Read the evidence >