The Week Your Engineers Spend Grading Is a Design Choice

Where does the grading week actually go?

Into rereading, not into judging. A senior engineer opens a submission with no page limit, no stated standard and no note saying what the candidate checked, so the first pass is reconstruction: what was asked, what came back, and what the person meant by it.

The judgment itself is quick. Working out what you are judging is not, and it happens again on the next submission.

The open brief is a response to something real. In a 2022 evaluation against 115 Python problem statements taken from HackerRank, OpenAI's Codex solved 96% of them with no examples and all of them with a few ⁶. The authors' own finding was clear signs that the model was reproducing memorized code, and the memorization matters more for hiring than the score: a task assembled from a public problem bank tests a lookup whether or not a candidate uses a model.

Codex was an old model on a curated benchmark, so treat 96% as a floor under current capability rather than a headline about coding tests.

So teams swapped the fixed problem for an open brief. Build something, use whatever tools you want, show the reasoning. The reading got longer, because an open brief has no defined end and every submission stops somewhere different.

Nobody was assigned those hours.

Nobody counts them either, because the number the round would be defended with does not contain them. SHRM's benchmarking report says cost-per-hire may include third-party agency fees, advertising, job fairs, job boards, referral costs, applicant and staff travel, relocation, recruiter pay and benefits, and talent acquisition system costs, divided by the number of hires ¹.

Engineering hours are not on that list.

The same report puts median cost-per-hire at $1,244 across 472 organizations, against a mean of $4,683 ¹. That distribution is heavily right-skewed, and the data come from SHRM member organizations in 2021. So any single figure from it misleads.

Set a ceiling on what comes back

Bound the submission in the brief, in units a candidate can count. A file limit, a page limit on the write-up, a fixed list of questions to answer, and one sentence saying that anything past the cap is not read. A ceiling on the deliverable is a ceiling on the read, and it is the only ceiling that works, because a reviewer cannot impose one after the fact.

The cap also changes what the exercise measures, in the direction you want. An unbounded brief rewards whoever had the freest evening. A bounded one asks what somebody did with a fixed hour, and a cap on the candidate's hour is also a cap on the grader's.

Capping both sides is how a round tests AI skills without adding an hour to the loop.

The cheapest cap is a starter repo. Hand over a small codebase with the task already scoped inside it, and the edge of the repo becomes the edge of the submission.

Decide what you plant in the starter repo while you are still writing the brief. Whatever goes in there fixes in advance where the evidence will sit, and a reviewer who knows where to look is a reviewer who stops looking.

Then ask for the working, in writing, capped as well. In one study, engineering students were permitted to use ChatGPT during take-home open-book exams and required to submit their interaction transcripts alongside their solutions. The researcher found the strongest evidence of reasoning where students evaluated incorrect or incomplete AI responses ⁴.

The paper's conclusion is that correctness of final answers alone may no longer provide sufficient evidence of comprehension.

That study is qualitative, has no control group and puts no number on what it found. A course is not a job. A transcript can also be curated after the fact, and asking for one is a disclosure some candidates will decline.

Treat the finding as a direction for the deliverable, not a validated method.

Write the key before the brief goes out

The key is the document that decides how long the read takes, and writing it after submissions arrive is what turns every one of them into a fresh argument. Draft it beside the brief: three or four things a submission has to show, each with a written description of what a strong, an adequate and a thin version looks like. Then send nobody the brief until the key exists.

Teams skip the written anchors, and they are the part of the key with some evidence behind them, if only from interviews. Levashina, Hartwell, Morgeson and Campion's review of the structured interview literature reports Taylor and Small's 2002 meta-analysis of 19 past-behaviour interview studies. Interviews using anchored rating scales showed higher interrater reliability, .77 against .73, and higher criterion-related validity, .35 against .26 ³.

The reliability gain there is small and the validity gain is the larger of the two. Those studies cover interviews, not take-homes, and the comparison runs across studies rather than the same interview scored twice. So some of the gain may belong to whatever else careful teams already do.

Take the direction and leave the numbers where they were measured.

For an AI-open exercise, four rows do most of the work, and each one names an act, not a quality:

• What was framed before anything was generated. The problem as stated, the constraint, what got ruled out.
• What the deliverable rests on, and what that was checked against. A source, a test, a run, or nothing.
• What was kept away from the model, and why. A judgment the candidate made without help.
• What was rejected, on what grounds. Output the candidate refused, with a reason attached.

Each row takes a yes, a no, or a line quoted straight out of the submission. A row that reads "strong engineering instincts" is a row two readers will fill differently.

The key also settles the argument you would otherwise have at the end of the pile: whether half the work, all of it checked, beats all of the work with errors in it. Pick one and write it into the key before anybody submits.

Which pass needs a senior engineer?

Only the second one. Split the read in two: a mechanical pass asking whether the submission builds, whether the required note is attached, whether the cap was respected and whether every named question got answered, then a judgment pass on whatever survives. The first takes minutes and needs no seniority at all. The second is the thing senior engineers are being paid for.

A script does most of the mechanical pass and a coordinator does the rest. Senior time never meets a submission that was going to fail on something a checklist could see.

The tempting next step is handing the judgment pass to a model, and that is a larger decision than it looks. Once a model produces the score instead of an extraction, who scores the take-home may change which rules the round falls under, which is a question for counsel in your jurisdiction.

Keep the model on the mechanical pass, where its output is a fact a person can confirm in a second.

Two readers earn their keep at the start and stop earning it later. Run the first several submissions past two people independently, compare the filled rows, not the verdicts, and rewrite whichever anchor produced the gap.

Bring the second reader back when the key or the role changes. Past that point, how many graders an assessment needs is answered by whether the key still holds.

Disagreement at the boundary is common enough to plan for.

In Ashby's 2026 recruiting operations benchmarks, around 38% of scorecard pairs included at least one point difference between interviewers. Nearly half of those one-point differences fell between 2 and 3, crossing the yes/no threshold on Ashby's 1 to 4 scale ².

A pair there is two scorecards compared against each other, so 38% is a rate of disagreeing pairs and not a share of interviews or candidates. The dataset holds no outcome measure, so nothing in it says which reader was right.

What the benchmark does say is that ratings move exactly where the decision gets made, which is an argument for calibrating early and not for reading everything twice.

Put grader hours in the funnel report

Count them in hours per hire, on a line beside the costs already reported. Reviewer hours spent on the exercise, divided by the hires made out of that pipeline. A round with a number attached gets argued about at the staffing meeting like everything else that has one. A round whose cost lives only in a staff engineer's calendar gets dropped quietly.

What belongs in that figure is a methodology question before it is an accounting one. Cost per hire counts the recruiter and skips the panel, so grader hours only appear if somebody puts them there.

Keep grader hours as their own line instead of converting them into money. Hours are what actually gets traded against something else.

If the exercise buys less than the week it costs, the cap and the key are beside the point. Sackett, Zhang, Berry and Lievens put work sample validity at .33 rather than the .54 that still gets quoted, drawing on Roth and colleagues' 54 studies of 10,469 people ⁵.

Fifty-three of those 54 tested people already doing the job, and none of that evidence speaks to an unpaid multi-hour take-home.

Either way, the cost-per-hire line in your report stays where it is, with its list of third-party agency fees, job boards and relocation ¹.

What changes is the line underneath it.

A round that cost a named number of senior hours per hire is a round somebody can defend, cut or redesign on evidence. A round nobody costed gets defended only by whoever liked it most.

See a sample report >