Your AI Exercise Became a Test the Day It Moved a Decision

When does an informal exercise become a selection procedure?

The first time it changes who advances. Federal selection law draws the line at use rather than at form. The Uniform Guidelines define a selection procedure as any measure, combination of measures, or procedure used as a basis for any employment decision, and name informal or casual interviews and unscored application forms inside the same definition ¹.

Nothing in that sentence turns on a score, a vendor, an invoice or a name.

The Guidelines date to 1978, and the text cited here is the 2023 annual edition of the Code of Federal Regulations, revised as of July 1, 2023 ¹. They name no software and anticipated no chatbot. What the definition carries instead is a reach wide enough that a manager reading a finalist's prompt log over coffee sits inside it, on the same footing as a proctored aptitude test.

The belief that a test needs a vendor is not irrational. In many hiring stacks the word describes exactly that: an instrument somebody bought, with a score, a cut mark and an invoice. But the regulation predates all of it, and its category is wider than the market's.

So the question is not whether the exercise is formal enough to count. Ask instead whether a finalist has ever moved, or failed to move, because of how the half hour went.

One yes settles it, and the yes may be two rounds old by the time anybody asks.

What attaches the day it moves a decision?

The exercise should be job related. It should reach everyone at the same stage on the same terms. And it should leave enough of a record that somebody can say afterwards what was asked and what was seen. The definition cited here spells out none of those three, and none of them waits on a number. Validation is the fourth, and it is the one that does.

The number is adverse impact, and the Guidelines treat a selection procedure that produces it as discriminatory unless it has been validated ¹.

Adverse impact is measured on rates, not on impressions. One paragraph of the Guidelines says a selection rate for any race, sex or ethnic group below four-fifths of the rate for the highest group will generally be regarded by federal enforcement agencies as evidence of it ².

The same paragraph says smaller differences can still be adverse impact where they are significant in statistical and practical terms, and that larger ones may not be where the numbers are small and not statistically significant ².

Reading four-fifths as a pass mark inverts the rule. The agencies that wrote the rule said in a 1979 interpretive document that the figure is not intended as a legal definition, but a practical means of keeping enforcement attention on serious discrepancies ³. Asked directly whether the Guidelines therefore tolerate up to 20 percent discrimination, they answered no.

One duty attaches with no arithmetic at all.

An applicant needs no legal vocabulary to request a reasonable accommodation. The EEOC's 2022 technical assistance on algorithms in assessment is guidance rather than law, and the agency took it off its site in January 2025, so it is read here from an Internet Archive capture. Under it, somebody who says a medical condition may make a test difficult to take has already requested one. Saying the phrase, or naming the statute, is not required ⁴.

An exercise nobody described in advance gives a candidate nothing to say that sentence about. The gap gets expensive when the tool the round bans is how someone manages a disability.

Does New York's tool rule reach a manager with a whiteboard?

No, and the mismatch is why the federal definition surprises people. New York City's Local Law 144 governs automated employment decision tools used on candidates in the city. Using one is barred unless a bias audit was run within the previous year, a summary of the results is posted publicly, and the candidate got notice at least 10 business days ahead ⁵.

The law took effect on January 1, 2023, and enforcement began that July 5 ⁵.

A person running an exercise by hand triggers none of those duties. There is no tool, so there is nothing to audit and nothing to post.

Working out whether a tool counts as an automated employment decision tool is a live question for the software in the stack and a dead one for a manager with a document and an hour.

But the older rule never had a technology test in it. Teams that read the two together can take the wrong lesson: the city rule reaches a narrow class of software, and the 1978 definition reaches the conversation. An exercise can sit outside one and inside the other on the same afternoon.

An unwritten half hour carries every duty and no evidence

Every duty above already applies, and the record is the part of this a team controls most directly. An unwritten exercise is not outside the Guidelines. It sits inside them with no evidence of what was asked, what was observed, or whether the same thing happened twice, which is the position hardest to answer from.

Documentation does not create the obligation. It answers one.

Written anchors have also been associated with a better instrument, which is the part nobody expects.

Levashina and colleagues, reviewing the structured interview literature, report a meta-analysis by Taylor and Small covering 19 past-behavior interview studies ⁶. Interviews using anchored rating scales showed criterion-related validity of .35, against .26 for interviews without them. Agreement between raters moved less, .77 against .73.

The .35 and the .26 come from different studies rather than from one that ran both, so anchored scales travel with whatever else careful teams already do. The 19 studies covered past-behavior questions only, not work samples or take-homes.

California has made the testing record evidence in claims about automated decisions. The Civil Rights Council's final text on automated-decision systems makes evidence of anti-bias testing, or the lack of it, relevant to a claim that such a system discriminated and to any defense, down to the quality, recency and scope of the effort and the response to its results ⁷.

The same text stretches employment-records retention from two years to four ⁷.

The sharp edge points at inaction rather than at inquiry. Having looked, found a gap and done nothing counts against an employer under that language, and holding no record at all is no safe harbor either. Confirm current status and dates with employment counsel, because none of this is legal advice.

What can a manager say when a rejected finalist asks?

Only what was written down at the time. Writing it is a short job. Name the task and what it is supposed to show about the job. Name what a strong answer looks like and what a weak one looks like. Say who gets the exercise and at which stage. Say what gets written down afterwards, in every case, including the rounds that go well.

Consistency is the part that fails first. An exercise given to the two finalists a manager is unsure about, and skipped for the one they already liked, is a different test given to different people. The difference shows in the record whether or not anyone meant it that way.

Adding the exercise partway through a live pipeline raises the same question. Bringing back candidates who finished before the round existed is worth deciding before the next requisition opens rather than after.

Nothing in the four-fifths rule forces any of this. A finalist round holds too few people for a selection rate to mean much, and the Guidelines' own paragraph says larger gaps may not be adverse impact where the numbers are small and not statistically significant ². On numbers that small a validation duty may never technically bite.

What shows up instead is a rejected finalist asking what the exercise was, and a manager who cannot say. Written down, the same half hour becomes a defensible assessment rather than a memory.

The half hour was always a test. What it leaves behind is decided this week, by the person who runs it next.

Read the evidence >