Will an AI-Skills Assessment Hold Up if a Candidate Challenges It?

What Does 'Hold Up' Actually Mean?

It means surviving a three-step sequence, not passing an inspection. A complaining party first demonstrates that a particular employment practice caused a disparate impact. The burden then moves to you to demonstrate the practice is job related for the position in question and consistent with business necessity. If you carry that, the candidate can still win by naming an equally valid alternative with less impact that you refused to adopt ¹.

Most challenges never reach a courtroom. What arrives is an agency charge, a demand letter, a city or state inquiry, or an internal complaint from someone who already read their report. All four are answered out of the same file, and that file either exists or it doesn't. A defense written after the challenge lands is a memo, not a record.

One clause decides how much of your process goes on trial. The complaining party normally has to point at the specific practice that caused the impact, but where the elements of your decision-making process are not capable of separation for analysis, the whole process may be analyzed as one practice ¹. If nobody wrote down how the assessment fed the decision, you have handed over that separation.

The Uniform Guidelines put it more bluntly than the statute does: a selection procedure with adverse impact is considered discriminatory unless it has been validated ². So the arithmetic on your own funnel matters long before anyone challenges anything, and so does knowing that a bias audit and a validation study answer different questions: an audit measures who came through at what rate, and validity evidence is what explains why the thing being measured belongs in the decision at all.

Why Job-Relatedness Carries the Whole Defense

Because it is the only step where evidence about your job matters. Adverse impact is arithmetic on your funnel; the alternative-practice step belongs to the candidate. The middle step is yours, and the Uniform Guidelines describe exactly what fills it: a job analysis of the important work behaviors, and a selection procedure that is a representative sample of them ³.

Content validity is the strategy most work samples rely on, and the standard is specific. The behaviors shown in the assessment should be a representative sample of the behaviors of the job, or the assessment should yield a representative sample of the work product. Where it samples a work behavior, the manner and setting of the procedure and its level and complexity should closely approximate the work situation. The further it drifts from that, the less likely it is to be content valid and the more other evidence you need ³.

Two lines in that same section quietly disqualify much of what gets sold as an AI-skills test. A content strategy is not appropriate for procedures that purport to measure traits or constructs such as intelligence, aptitude, personality, commonsense, judgment, leadership and spatial ability ³. Anything pitched as measuring AI judgment as a trait sits inside that sentence. What survives is observable behavior on occupational material: what the candidate asked for before generating anything, which claim they demanded a source for, what they refused, what they checked against something outside the chat.

The second line catches tool training. Content validity is not an appropriate strategy where the knowledge, skill or ability is one an employee will be expected to learn on the job ³. If a new hire would be handed your prompt library and your approved tools in week one, that content cannot carry a hiring gate. Move it to onboarding, and test the judgment that does not arrive in a week. That is also the honest route to a defensible proficiency bar for one role: the bar comes off the job analysis, not off a vendor's normal curve.

Which Evidence Actually Defends the Decision?

Six artifacts, and all of them have to exist before the first invite goes out. A job analysis naming the important work behaviors. The map from each assessment task to one of them. The reviewer's written findings for the candidate who challenged. The comparator records for everyone else in that requisition. Your impact figures at the threshold you actually ran. The notice and alternative-process record.

1. The job analysis. It names the critical or important work behaviors and the tasks associated with them, and where a behavior produces something, the work product too ³. Written by someone who talked to people doing the job. A competency framework bought off a shelf is not this. 2. The task-to-behavior map. One line per assessment task, naming the work behavior it samples and where that behavior appears in the analysis. This is the artifact counsel asks for first and the one most employers cannot produce. 3. The findings for this candidate. What the assessment showed, in writing, with the moment it rests on. A dimension name plus a number is not a finding; a finding is a described act that a second reader can locate in the record. 4. The comparator records. The same evidence for the other candidates in that requisition, at the same standard. A challenge is comparative by nature, and a file that is thorough for one rejected candidate and thin for the hired one reads exactly as badly as it sounds. 5. Impact figures at your threshold. Selection rates by group at the cutoff you ran, on your applicants, not the vendor's pool. Whether a rate difference is substantial is judged on the numbers you produced. 6. Notice, accommodation and the alternative process. Where New York City's rule applies, notice runs at least ten business days before use and must include instructions for requesting an alternative selection process or an accommodation ⁷. Even outside it, what you offer a candidate who cannot take the assessment as designed is part of the record.

The vendor holds a lot of this and owes you none of it by default. The EEOC's long-standing position on tests is that a vendor's validity documentation may help, but the employer is still responsible for ensuring its tests are valid ⁵; its 2023 technical assistance on algorithmic tools went further, saying an employer may be responsible even where the tool was developed by an outside vendor, and that if the vendor is wrong about its own assessment the employer can still be liable ⁶. That document no longer resolves on eeoc.gov, which is its own lesson: cite the regulation in your file, not an agency web page. The documents to demand from a vendor in writing belong in the contract, with dates.

Why a Generic AI Score Is the Weakest Artifact

Because it carries none of the evidence the middle step needs, and it attracts the rules that bite hardest. The Guidelines rule out by name any assumption of validity based on a procedure's name or descriptive labels, promotional literature, data on how widely it is used, and the testimonials and credentials of sellers or consultants ⁴. A number called an AI-readiness index is a descriptive label until a study says otherwise.

It also decides which regime you are under. New York City defines substantially assisting or replacing discretionary decision making as relying solely on a simplified output (a score, tag, classification or ranking) with no other factors considered; using that output as one of a set of criteria where it is weighted more than any other; or using it to overrule conclusions derived from other factors, including human decision-making ⁷. A number that decides is the trigger. A written finding a person weighed among other things is a different shape, and only your own records can show which one you ran.

Where the rule applies the duties are concrete, and they land on you rather than on the vendor: a bias audit within one year of use, a summary of results published on the employment section of your own site before use (including the selection or scoring rates and the impact ratios for every category), and candidate notice at least ten business days ahead ⁷.

A challenge goes after the evidence first. A finding that a candidate carried an unsourced claim into a client-facing memo at minute 34 is evidence about a work behavior, and it points at a moment two people can go and read. A 68 is a summary of evidence somebody else is holding. When counsel asks what the 68 measured, the answer has to come out of a validity study, and if no study names your occupation, there is nothing to say.

What to Do the Week a Challenge Arrives

Preserve first, reconstruct second. Once a charge is filed, every personnel record relevant to it is kept until final disposition, expressly including the test papers completed by the unsuccessful applicant and by every other candidate for the same position ⁸. Send the legal hold to the assessment vendor the same day, because a deletion default that fires mid-charge destroys your evidence, not theirs.

1. Freeze everything, vendor included. Exports, not dashboard views. A record you can only see while the subscription is live is not a record you can produce. 2. Pull the whole requisition. Every candidate, every stage, every note, including the informal ones. A challenge is answered comparatively or not at all. 3. Write the decision memo today, dated today. What the assessment showed, what else was weighed, who decided. Never backdate anything; a document with the wrong date on it converts a defensible process into a credibility problem. 4. Recompute the rates at the threshold you ran. On your applicants, for this requisition and for the last twelve months of the role. Do it before you are asked, and do it with someone who will report the number honestly. 5. Answer through counsel. Everything above is the file counsel works from. None of it is legal advice, and the moment a charge is real, the reply is not yours to draft alone.

If the file has holes, the useful move is forward rather than backward. Fix the open requisitions (write the job analysis, map the tasks, record the human decision) and treat the closed one honestly. Most people who challenge a rejection want an explanation they can act on, and a straight explanation of how the assessment fed the decision closes more of these than any citation does. The file is for the ones it doesn't close.

Read the evidence >