Policy
What the Model Was Trained On Decides What It Rewards
Some hiring AI is trained on a company's own people, and what a model rewards follows its training target. Documented systems have been trained on gameplay from a client's own high performers, on how a panel of strangers perceived people in videos, and in one abandoned experiment on a decade of a company's own resumes. Each inherits whatever shaped those judgments, which is how it can count against you. A published bias audit will not say which target a tool used, so ask the employer what it was validated against.
The takeThe Amazon story is the wrong anchor for this question. It was an internal experiment, abandoned by the start of 2017, built before large language models existed, and it says nothing about the tool in front of you now. The better anchor is duller: a vendor's own account of what its model learned, plus an employer who can say which roles the thing was checked on. One of those is a headline. The other you can actually ask for.
Where Olive fits
Open a role and see what the work shows
A person writes or edits every finding in an Olive report, and no decision about a candidate is made by a machine alone: six findings in words, each quoting the evidence it rests on, with no composite number anywhere. The candidate is granted the same report the employer reads, on every tier.
Assess your shortlistWhat Were These Models Actually Trained On?
They were trained on a target somebody chose, which is not quite the same thing as the people already on the payroll. In the documented cases it is a rating, a group of incumbents, or a pile of old resumes. Whatever shaped that target travels into the model with it. So the question worth asking is what the system was fitted to reproduce.
Retorio told the German broadcaster BR in 2021 that its system was trained on videos of more than 12,000 people, with a further 2,500 people rating how they perceived each one on the Big Five dimensions 1. Those perceptions are the target the model learns. Asked why appearance changed its scores, the company said the impression a person makes on others is part of the concept 1.
Retorio said that about Retorio, not an auditor.
A 2021 peer-reviewed audit of pymetrics, co-authored with the company's own staff, describes a different target. Models were trained on gameplay from an in-group of high-performing incumbents at the client, typically 50 to 100 people identified through job analysis, against an out-group sampled from the vendor's database 2. The trained model then sorted applicants into three tiers 2. Which incumbents a client chose is not visible to the candidate 2.
That is the closest thing in the record to the fear behind the question, and it is worth stating exactly. The comparison group is a set of people the employer already has, picked by the employer. The auditors also said plainly that they did not assess whether the games predict job performance 2.
That description comes from 2020 fieldwork, and the product has since changed hands.
Neither target is announced on the screen a candidate sees.
Does Amazon's Scrapped Tool Answer This?
No, and it is the example most likely to mislead you. Reuters reported in 2018 that Amazon built an experimental recruiting engine from 2014, trained on patterns in resumes submitted to the company over a ten-year period 3. By 2015 it was penalising resumes containing the word women's and downgrading graduates of two all-women's colleges, and the team was disbanded by the start of 2017 3.
Two opposite readings of that story are both wrong. Nothing on the record says it rejected applicants: Reuters reports no candidate outcome and says recruiters never relied solely on those rankings 3.
It was not invisible either, because recruiters did look at the recommendations 3.
It is evidence about no current product.
The whole account rests on five people speaking to one Reuters reporter on condition of anonymity, and Amazon declined to comment 3. It describes technology built between 2014 and 2017, before large language models existed, so it cannot describe the system in front of you now.
The durable line is the one usually cut. Amazon edited the programs to be neutral to the flagged terms 3. Reuters wrote that this was no guarantee the machines would not devise other ways of sorting candidates that could prove discriminatory 3. Deleting a word does not delete whatever the word stood in for, and a model fitted to a company's own hiring reproduces that hiring.
Buyers are handed a neighbouring lesson in procedural form. What a hiring team is told to ask a vendor claiming it is bias tested comes down to which jobs the audit covered and whose applicants it ran on. Reading what the buyers are told is a fair way to check what you are told.
Can a Model Add a Gap the Raters Didn't Have?
In at least one case study it did, which is a sharper problem than the training data alone. A 2023 case study used 511 college students' mock video interviews, rated for hireability by at least three trained annotators 4. A model was then trained on roughly 7,877 features drawn from what the students said, how they sounded and how they looked 4. Selection was simulated on both.
Selecting the top 10% on the annotators' own ratings gave women and men the same selection ratio, an adverse-impact ratio of 1.0 4. Selecting on the model's predictions of those same ratings gave .11 for women and .08 for men, a ratio of .70 4.
The authors describe that as below the four-fifths threshold, which they call prima facie evidence of adverse impact under US rules 4.
The gap there ran against men, and the direction is not the finding. What matters is that a model introduced a group difference which was not in the human ratings it learned from. This was a demonstration on students, built to illustrate a measurement framework rather than to audit a product 4.
The authors say the size of the gap depends on the cutoff 4.
The largest independent study of video personality scoring found something related. Across 1,073 mock interviews in three samples, models trained to predict interviewees' own self-reported traits showed virtually no convergence with those self-reports in new samples, a mean r of .07 5. Models trained on interviewer ratings of the same traits converged at a mean r of .37 5.
Moderate agreement with how a rating panel saw you is not a measurement of who you are.
Those were mock interviews with students and crowd workers, and academic models rather than any vendor's product 5. The study shows what this approach can and cannot reach, which is the standard to read a vendor's claim against.
Ask What the Tool Was Validated Against
Three questions do more work than any audit summary you can find online: what the tool was validated against, on which roles, and when it was last checked. Ask them of the recruiter in writing, at the point the assessment is sent. An employer who bought the tool can answer, or can say that nobody there was told, and both replies are information.
A posted bias audit is not built to answer them. Under the 2023 rules for New York City's Local Law 144, the audit computes a selection rate and an impact ratio for each category, separately for sex, for race and ethnicity, and for intersectional categories 6.
Those are group outcomes, not the training target.
Two details matter when you read one. An independent auditor may drop any category representing less than 2% of the audit data from the impact-ratio calculation, which most often excludes the smallest groups in the applicant pool 6. And under rule 5-302, an employer that has never used the tool may rely on an audit built on other employers' historical data 6.
This arithmetic is a New York City requirement 6.
How to read a published bias audit before you apply works through one line by line. From the other chair, the difference between a validated assessment and a bias-audited one is the distinction hiring teams are told to keep straight. An audit can find a gap. Only validation speaks to whether the tool measures that job.
A recruiter who cannot say what the tool was checked against has told you something about the process, not about you. Take it as one input. It sits alongside everything else the employer has already shown about how it treats the people it assesses.
Common questions
Does a hiring AI compare me to the people the company already hired?
Sometimes, depending on the tool. A 2021 audit of pymetrics describes models trained on gameplay from 50 to 100 high-performing incumbents chosen by each client, so the comparison group there is that employer's own people. Another vendor said its model learned how 2,500 raters perceived more than 12,000 people on video, which is a different target entirely. The interface does not say which, so the question goes to the employer.
Was Amazon's AI recruiting tool used to reject people?
There is no reported evidence of that. Reuters reported in 2018 that Amazon's experimental engine, trained on ten years of resumes, penalised the word women's and downgraded graduates of two all-women's colleges, and that recruiters looked at its recommendations but never relied solely on those rankings. The team was disbanded by the start of 2017. The account came from five people speaking anonymously, and Amazon declined to comment.
If a company posts a bias audit, does that mean the tool is fair to me?
It means something narrower. A New York City bias audit reports selection rates and impact ratios by sex, by race and ethnicity, and by intersectional category. It does not report accuracy or job-relatedness, and it says nothing about what the model was trained to predict. Categories under 2% of the data can be left out of the ratio, and an employer that has never used the tool may post an audit computed on other employers' data.
Can a model create a gap the human raters did not have?
In at least one case study, yes. Selecting the top 10% of 511 mock interviews on trained annotators' hireability ratings gave women and men identical selection ratios. Selecting on a model's predictions of those same ratings produced an adverse-impact ratio of .70, below the four-fifths threshold, in that case against men. It was a demonstration on students rather than an audit of a product, and the authors note that the size of the gap depends on the cutoff.
What should I ask the recruiter?
Three questions, in writing, when the assessment arrives: what the tool was validated against, which roles it was validated on, and when it was last checked. A fourth is worth adding: who reviews the output before a decision. Asking is not an accusation, and every answer, including that nobody there was told, describes how the process is run.
My background looks nothing like this company's usual hire. Should I skip the assessment?
That is a judgment call, and the evidence does not settle it for you. What the record supports is narrower: the target a model was fitted to shapes what it rewards, and in one study a model produced a group gap that the human ratings behind it did not have. That is a reason to ask what the tool was validated against, and to weigh the answer alongside everything else you know about the employer.
References
- 1. Objective or Biased: On the questionable use of Artificial Intelligence for job applications interaktiv.br.de Supports the claim that one vendor's system was trained on videos of more than 12,000 people with 2,500 further people rating how they perceived them, and that the company describes the impression a person makes on others as part of the concept.
- 2. Building and Auditing Fair Algorithms: A Case Study in Candidate Screening ccs.neu.edu Supports the description of models trained on gameplay from 50 to 100 high-performing incumbents chosen by the client, the sorting into three tiers, and the auditors' statement that they did not assess whether the games predict job performance.
- 3. Amazon scraps secret AI recruiting tool that showed bias against women web.archive.org Supports the account of the experimental engine trained on ten years of resumes, what it penalised, that recruiters never relied solely on its rankings, the anonymity of the sources, and the line that editing out flagged terms was no guarantee against other discriminatory sorting.
- 4. Integrating Psychometrics and Computing Perspectives on Bias and Fairness in Affective Computing: A Case Study of Automated Video Interviews arxiv.org Supports the finding that selecting on human hireability ratings produced equal selection ratios while selecting on the model's predictions produced an adverse-impact ratio of .70, and the authors' caution that the gap depends on the cutoff.
- 5. Automated Video Interview Personality Assessments: Reliability, Validity, and Generalizability Investigations par.nsf.gov Supports the claim that models trained on self-reported traits showed virtually no cross-sample convergence while models trained on interviewer ratings converged at a mean r of .37, and that these were mock interviews with students and crowd workers.
- 6. Notice of Adoption of Final Rule: Use of Automated Employment Decisionmaking Tools (6 RCNY 5-300 et seq.) rules.cityofnewyork.us Supports what a Local Law 144 bias audit computes, the 2% category exclusion, and the rule 5-302 allowance for an audit built on other employers' historical data.
6 sources, numbered by first appearance. How Olive sources claims
General guidance, not legal advice. Hiring rules differ by state and country and change often; check anything here against your own counsel before you act on it.
Olive assesses how a person works with AI. It does not detect AI-written documents, and it never produces a score, a ranking, or a match percentage for a person. Candidates read the same report the employer reads.