Who Scores the Take-Home Decides Which Rules It Falls Under

What changes the moment a model produces the score?

The instrument changes category. A take-home read and scored by a person is a selection procedure, and has been one since 1978. A take-home scored by a model is that same procedure plus, wherever the grader meets a local definition, an automated employment decision tool. In New York City, being one adds a bias audit, a posted summary and advance notice to candidates, and takes nothing off what is already owed.

New York City has run that rule since January 1, 2023.

Local Law 144 bars an employer from using an automated employment decision tool on a city candidate unless three things are true first. A bias audit was done within the prior year, a summary of the results is posted publicly, and the candidate had notice at least 10 business days beforehand ¹.

The hinge in that law is a phrase rather than a technology. The rule reaches a tool that substantially assists or replaces discretionary decision-making ¹, so an employer whose model only informs a human reviewer argues itself out of scope. Whether a particular grader clears that line is the AEDT question itself.

Human scoring is the property that made a human-scored work sample the safer instrument in the first place.

But the argument that a human formally decides has already failed once in Europe, under a different law and on a different fact pattern. On 7 December 2023 the Court of Justice of the EU held that a credit agency's automated probability value about a person is itself automated individual decision-making under GDPR Article 22(1). The Court's condition was that the third party receiving the value draws strongly on it ².

No court has applied that holding to a candidate's grade, and "draws strongly on" is a fact-specific test rather than a bright line. What travels is the structure. The scorer can be the decision-maker even though somebody else formally decides.

Two more jurisdictions arrive in 2027. Colorado's 2024 AI act never took effect; the governor signed a replacement on May 14, 2026 that names employment a consequential decision ³.

That statute requires notice at the point of interaction, a plain-language description of the technology's role within 30 days of an adverse outcome, and a right to request meaningful human review ³. January 1, 2027 is the date its summary attaches to the developer's documentation duty and to the attorney general's rules on that post-outcome description, rather than to each consumer duty ³. Where the replacement covers the grader, the model that produced the grade is the technology being described.

The EU's dates moved, and most compliance notes still carry the old ones. The European Commission's implementation timeline now shows the high-risk rules in Annex III, the category holding recruitment and workforce AI, applying from 2 December 2027 rather than 2 August 2026 ⁴.

None of those three dates has passed, and all three point the same way.

Where do model graders fail, and in which direction?

In directions you can name in advance, which is the useful part. A published evaluation of models used as judges reports position effects, verbosity effects and self-enhancement: the same answer scores differently depending on where it sits in the comparison, how long it is, and which model produced it ⁵.

Position, length and which model wrote the answer are not random noise. Each points one way, so each can be tested for, and each test costs some of the reviewer time the model was bought to save.

But the paper naming those three is about chatbots, not candidates. Lianmin Zheng and his co-authors, in the work that introduced MT-Bench and Chatbot Arena, tested whether a strong model can stand in for human preference judges and reported over 80% agreement with human preferences alongside the three biases ⁵.

No equivalent study has been run on take-home submissions scored against a hiring rubric.

Order effects do have a hiring-shaped measurement, on resumes rather than submissions. Bloomberg's 2024 tests found both models favoring whichever resume they saw first. GPT-3.5 named the first candidate the most qualified 56% of the time and GPT-4 28% of the time, against the one-in-eight share expected if order did not matter ⁶.

Shuffling the order fixes that one and touches nothing else.

Self-preference has one hiring-shaped number, and it comes from a simulation. In a controlled resume experiment, models preferred resumes they had generated themselves over human-written ones at rates from 67% to 82% with content quality held constant.

Across simulated pipelines in 24 occupations, candidates using the same model as the evaluator were 23% to 60% more likely to be shortlisted than equally qualified applicants with human-written resumes ⁷.

No real applicant was shortlisted or rejected in that study, and the authors report that simple interventions cut the effect by more than half.

Read the preference as a property of how these systems are being deployed today. If your candidates draft with models too, a grader that rewards its own house style would be scoring the tool rather than the work.

All of which assumes the rubric underneath was sound. Whether two of your own reviewers score the same submission the same way is the prior question, and an AI-use rubric two reviewers score the same way is harder to write than it looks. A model cannot inherit an agreement that never existed.

Keep the extraction, give up the score

Keep the model on the reading and take it off the judgment. A model can be asked to pull every factual claim, every cited source, every stated check and every unverified number out of a long submission and lay them out in one comparable shape. A person then reads eight one-page extracts instead of eight documents, and writes what the evidence shows.

The hours come back, and the finding stays human.

The line between the two is thinner than it looks. Once the extract is ordered, ranked or summarized as an assessment, the reviewer is reading a judgment before making one, which is what an AI summary in the reviewer's hands does to the record.

Five rules keep the split where you put it.

• The model returns spans of the candidate's own text, never a number and never an adjective.
• Every extracted item carries the page or line it came from, so the reviewer can go and look.
• Submissions go in one at a time, in a fixed template, so nothing is compared inside the model's window.
• The reviewer writes the finding in their own words, with the excerpt attached.
• Nothing the model wrote is copied into the rejection record, where the wording carries its own exposure.

What that buys is a record with two authors and one decision-maker. The extract shows what was in the submission; the finding shows what a named person concluded from it. An appeal can be answered by rereading both.

The saving is real and smaller than the pitch. Extraction removes the slow part of reading, which is finding things, and leaves the part that was always the job. Measure it on your own pile before the second round, and measure it against the afternoon it replaced.

Test the grader on the same pile twice

Run twenty submissions through the grader, shuffle the order, run the same twenty a week later, and count how many scores cross the pass line between the two runs. That number is most of the case for or against the grader.

Federal selection law is indifferent to who applied the standard: the Uniform Guidelines call any measure used as a basis for an employment decision a selection procedure ⁸.

That definition is scope, not a duty. The Guidelines impose a validation burden where adverse impact appears, and the enforcement agencies' own reading is that with no adverse impact there is no validation requirement ⁸.

The Guidelines' reach cuts the other way too, which is the part employers miss.

An informal or casual interview is a selection procedure by the same definition ⁸, so retreating from the model grader back to a conversation moves nothing outside the rules.

So run the reproducibility test before the grader ever touches a candidate, and write that number down.

A grade that moves when nothing about the work moved is not a grade.

Then hold the same pile against your reviewers: two of them, the same twenty submissions, scored independently, and the same count. What legal will ask for is that pair of numbers, plus the evidence behind any individual score.

Your own reviewers may never have been measured either. They can drift and tire, and the fourth submission of an afternoon may not be read the way the first was, which is what the two-reviewer count is there to show.

The submissions are still on the desk at the end of this, and the model can carry most of the afternoon: the reading, the pulling apart, the laying side by side. What it cannot carry is the sentence saying what the work showed, because that sentence is the one a candidate, an auditor or a court will ask a person to stand behind. Write it yourself.

Read the evidence >