Vary the Data, Never the Task or the Bar

Does everyone have to get the same exercise?

Yes, for candidates at the same stage of the same role. A work sample is a selection procedure in the federal sense: the Uniform Guidelines define one as any measure or procedure used as a basis for an employment decision, and say the term covers the full range of assessment techniques, from paper tests through informal or casual interviews and unscored application forms ¹. On that definition, two versions of it are two procedures.

Two qualifications keep that from being a slogan. The Guidelines impose a validation burden only where adverse impact appears, and they are a 1978 regulation applied by analogy to tools nobody had then. The statutory test sits elsewhere: under 42 U.S.C. 2000e-2(k), once a complaining party shows a particular practice causes disparate impact, the employer has to demonstrate the practice is job related for the position in question and consistent with business necessity ².

That is Title VII, so it reaches race, color, religion, sex and national origin; age runs under the ADEA and disability under the ADA, on different standards. None of it is legal advice and all of it belongs in front of counsel before it becomes policy. What it establishes for a hiring team is the shape of the question you will be asked, which is normally about a particular practice applied to particular people.

An exercise varied per candidate answers that question badly. The practice is now several practices, the comparison between two finalists rests on nobody's calibration, and the reason one person got a different brief is whatever the recruiter remembers. Consistency is the cheaper posture, and it is also what makes your own data readable when you ask whether the assessment would hold up under challenge.

Which parts can you vary safely?

Inputs, freely. Task, standard and stated time, not at all. Run any proposed change through one question: is a reviewer scoring two submissions applying the same rubric to the same demand? A different dataset from the same generator, a different customer inside the same scenario, or a different quarter of the same synthetic ledger all pass that test. A different question does not, and neither does a different deliverable.

Safe to vary:

• the dataset instance, provided it comes from one generator and carries the same difficulty
• names, identifiers, and the specific customer or ticket at the centre of the case
• the order of the supporting documents
• the calendar window a candidate has to complete it in

Not safe to vary:

• the deliverable, or the number of deliverables
• the stated time
• the rubric, its weights, or the bar for a pass
• whether an assistant is permitted
• how many people read the submission

The generator is what makes the first list workable. Build the exercise so instances can be minted from one specification and varying the data costs nothing and changes nothing about what is measured. Build it around a single hand-made spreadsheet and every variation becomes a redesign, which is how teams end up holding three versions of unknown relative difficulty and no record of which finalist sat which.

Run rotation on a schedule, not on request

Rotation is legitimate when the schedule exists before the candidate does. Decide the window, generate the versions together, calibrate them against the same rubric with the same reviewers, then keep every comparison inside a single version. Announced in advance and applied to everyone inside the window, that is one procedure with instances. Swap it for one person mid-process and you have a second procedure plus an exception with a name attached to it.

Calibration is the step teams skip, and the interview literature records the same habit. A content analysis of 104 interviews reported in 103 articles published between 1997 and 2010 found that studies describing a structured interview used an average of 5.74 of 15 structure components, and two of the scoring-side components were among the least common: detailed notes in 19% of the interviews, statistical prediction in 11% ³. Those counts describe what researchers built into published designs, so read them as a ceiling on ordinary practice.

What calibrating two versions actually involves:

1. Score a handful of past submissions against both versions, with the same two reviewers. 2. Compare the pass rates, and then the disagreements. Versions producing different disagreement patterns are not equivalent, whatever the pass rates say. 3. Fix the version, not the rubric. If version B needs a softer bar to produce the same pass rate, version B is harder and should be rewritten. 4. Record which version each candidate received, in the same place you record the findings.

The alternative is the treadmill, which is where teams land when leaks set the calendar. Rotating faster than you can calibrate produces a stack of versions of unknown difficulty, and the same trap catches question banks, so it is worth reading how the rotating question treadmill plays out before committing to a cadence.

How do accommodations fit without breaking consistency?

Accommodations are not a variation you chose, and they do not break consistency. Under 29 CFR 1630.11 an employer must select and administer tests so the results reflect what the test claims to measure rather than an applicant's impaired sensory, manual or speaking skills, except where those skills are the factor the test purports to measure ⁴. The working rule that follows: adjust the format or the clock, and keep the task and the bar identical.

The EEOC's 2022 technical assistance on algorithmic assessment names extended time and an alternative version of the test, including one compatible with a screen reader, among accommodations that may be effective ⁵. That document was removed from the agency's site in January 2025 and is read here from an archive, so it is not current agency position. The regulation underneath it has not changed, and what is owed on any particular request is a question for counsel on your own process.

Three practical rules keep the record clean:

• A request needs no particular words. An applicant who says a medical condition may make the test hard to take has already asked ⁵. Where the need is not obvious you may ask for reasonable supporting documentation, and a broad pre-offer demand for medical detail is a separate problem of its own.
• Write down what was adjusted and why, in the same file as the submission, so a later reader can see the task and the bar were untouched.
• Never treat an accommodated submission as a different tier. Same exercise, same criteria, finding written the same way.

Extended time is the request worth settling before it arrives, because the answer turns on what your exercise claims to measure. If speed is genuinely part of the requirement, say so in the brief and be ready to defend it. If it is not, and for most take-homes it is not, the clock is a convenience you imposed rather than a construct you measured. Working that out in advance is most of what ADA accommodations mean for an AI-based assessment.

Before the next debrief, write down which version every live finalist received. If two of them are being compared across two versions nobody calibrated, you have two findings sitting in one spreadsheet and no comparison.

Read the evidence >