How Do You Defend an AI-Skills Assessment to Legal and to Candidates?

What does legal actually need to see before signing off?

Four things, and all four are documents. A job-relatedness file showing the assessment measures what this occupation performs. A record of selection rates by race, sex and ethnic group. A named human who makes the decision the assessment informs. And the notice you gave candidates before they sat it. Legal's question is not whether the tool is good. It is whether you can produce those four on request.

Start with the standard, because it names the burden. When a selection procedure produces disparate impact, the employer has to show it is job-related and consistent with business necessity, which the EEOC states as necessary to the safe and efficient performance of the job, and which it ties to "the particular job in question" rather than to general skill ². Most rollouts fail on that phrase, not on the choice of vendor.

The Uniform Guidelines put the documentation burden on the user. Any selection procedure that is part of a process with adverse impact has to be validated, with the evidence maintained and available, through criterion-related, content or construct validity ³. A vendor's technical report is an input to that file. It is not the file.

The impact record is separate, and it starts on day one. Section 1607.4 asks each user to keep records disclosing the impact its selection procedures have on people by identifiable race, sex or ethnic group, and treats a selection rate below four-fifths of the highest group's rate as evidence of adverse impact ¹. You cannot reconstruct that later if nobody recorded who was invited and who advanced. Stand the adverse-impact numbers up before the first cohort, not after the first complaint.

The fourth document is the one people skip: who decided. Keep the assessment an input a named person reads, with their reasons written down. Where a tool is used to make or substantially inform a hiring decision, notice and audit duties can attach. New York City requires a published bias audit summary and at least ten business days' notice to candidates before use ⁵.

Why job-relatedness has to be rebuilt for every role

Because validity evidence does not transport. The Uniform Guidelines Q&A says a procedure found valid in one situation does not necessarily have validity in different circumstances, and sets four conditions before evidence can be borrowed from another job ⁴. So the file you assembled for software engineering does not defend the same assessment used on underwriters. Each occupation gets its own task analysis and its own record.

The Q&A is also specific about job analysis: a full job analysis is required for all content and construct validity studies, and the analysis has to describe the important work behaviors, their relative importance and their difficulty ⁴. An AI-skills assessment is a work sample, so content validity is the natural strategy, which means the job analysis is not paperwork around the defense. It is the defense.

In practice that means the assessment's material has to be the occupation's material. For an underwriter, an application and a survey that do not describe the same building. For a financial analyst, a source packet whose claims only settle when someone opens them. The behaviors under assessment can stay constant across roles; the task, the answer key and the evidence file cannot. One assessment stretched across every department is the hardest version of this to defend.

Borrowing evidence between two job families is allowed and it is narrow. The Q&A's conditions include showing the jobs comparable on a work-behavior basis and accounting for variables such as performance standards, work methods and how representative the study sample was ⁴. If you cannot write that comparison down, you have two families and two files.

How do you write the evidence file for one role?

Build it as one folder per role, assembled before the first invite goes out. It holds the task analysis for the occupation, the assessment content mapped task by task, the rubric and its version, the names of the people who review, the decision rule, and the running record of who was invited and who advanced by group. Write it while the reasoning is still cheap to recall.

What goes in, concretely:

• The task analysis. The work behaviors this role performs, their importance and their difficulty, sourced from people doing the job rather than from the job posting.
• The content map. Each part of the assessment against the behavior it samples. A row with no behavior beside it is a row to cut.
• The rubric, versioned. What counts as demonstrated, what does not, and which version was in force on the date each candidate sat it.
• The reviewer record. Who read the session, what they wrote, and the calibration they went through before reading anything that counted.
• The decision rule. What the hiring team does with the result, written before the results exist. "Considered alongside the interview" is a decision rule; so is a cut score, and a cut score needs a source.
• The impact log. Invited, started, advanced, by group, from the first cohort.

The rule of thumb: anything that would sound like a judgment call in a deposition should be a dated document instead. That is also the practical difference between a defensible AI-skills assessment and one that merely produces a defensible-sounding report: the file exists before the dispute, or it does not exist.

If you bought the assessment rather than built it, the same folder still belongs to you. Ask the vendor for the job analysis behind each bank, the rubric version history, and the reliability evidence between reviewers. Then get it in writing before the pilot, because a sales deck is not documentation.

What do you tell candidates, and when?

Before they start, in plain language: what the task is, what is captured, which behaviors are assessed, who reads the result, and how long it is kept. After it is reviewed, give them the result itself. New York City requires at least ten business days' notice before an automated employment decision tool is used, plus a published bias audit summary ⁵. Treat that as the floor everywhere, not as a New York obligation.

The objection you are answering is specific, and it is not an objection to being assessed. Pew Research Center found Americans oppose the use of AI in making final hiring decisions by 71% to 7%, while 47% said AI would do better than humans at treating all applicants similarly ⁶. People are not rejecting structure. They are rejecting being decided by a machine with nobody in the room.

So the sentence you owe a candidate is short: a person read your session, wrote what they saw, and here is what they wrote. Report parity is the cheapest defensible thing on this list: the identical document, with the identical excerpts, goes to the candidate. It also disciplines the writing, because nothing gets recorded about a candidate that you would not hand to them. Decide what a candidate-facing report exposes before the first release, not after someone forwards one to a lawyer.

Say how to request an accommodation, and say it before the assessment rather than inside the rejection. A timed task, a microphone, a screen-capture requirement and a fixed session length are each a place where the process can turn a disability into a score. Offer the alternate path up front and record that it was offered.

A finding carries the moment it rests on, so it can be discussed, disputed and corrected. A composite number standing for the person carries none of that. It never goes in the report, and it is the single artifact most likely to be read back to you.

Which questions will legal ask that you can't answer yet?

Five, and they arrive in the same order. Who makes the decision. What the cut is and where it came from. What the selection rates look like by group. What the vendor will put in writing. And whether a less discriminatory alternative existed, because a job-related procedure still loses if a challenger shows one did ². Most rollouts have an answer for the first four and nothing for the fifth.

The fifth question is answerable, but only with evidence you have to go and collect. Run the assessment beside your existing round rather than in place of it, keep both sets of results, and record what each one added. That comparison is what turns "we chose this" into "we tested this," and piloting before the gate is cheaper than discovering the answer during discovery.

On the cut: a threshold invented in a meeting is the weakest link in the file. If the decision rule is a bar, the bar needs a source: a job analysis that says what level of the behavior the work requires, or expectancy evidence from the pilot. Where neither exists yet, keep the result an input a human weighs and say so in the rule.

Two answers legal will accept faster than any argument about the tool: the reviewer's notes, and the version strings. Evidence anchoring means every conclusion points at a moment somebody can open: a timestamp, a transcript turn, a diff, a written answer. Versioning means the report cannot quietly change after release. Together they turn a defense of the assessment into a defense of a specific record about a specific person, which is the only defense that survives being tested one candidate at a time.

Read the evidence >