Move a Cut Score for Evidence, Never for Yield

What does a seventy percent fail rate actually measure?

A pass rate measures three things at once: the exercise you wrote, the line you drew, and the people who happened to apply. None of those is proficiency. Move any one of them and the number moves, which is why yield can't tell you whether the bar sits in the right place.

A fail rate can tell you something is worth looking at. It cannot tell you which thing.

Start with the exercise, because task choice can move a result a long way.

Pre-registered and run in 2023, one field experiment put 758 Boston Consulting Group consultants through eighteen tasks inside GPT-4's capability and one deliberately chosen to sit outside it ¹.

On that outside task, 84.5% of the control group reached the right answer. The two groups working with the model scored 60% and 70% ¹.

One task and a 2023 model is thin evidence about AI in general.

What decided that result was the task: the tool that helped inside its reach cost accuracy on the one task outside it ¹.

The task is only half of what sets the difficulty. The model your candidates hold is the other half, and it changes without asking.

A release your answer key predates can make the task easier for everyone who has it. Pass rates that jump after a model release are the same problem read from the far end.

Your own read on the difficulty is worth less than it feels.

A randomized trial put 16 experienced open-source developers on 246 real tasks in repositories they had maintained for about five years. Beforehand they forecast that allowing AI tools would cut completion time by 24% ².

Afterwards they estimated a 20% saving. The measurement was 19% slower ².

The result is narrow: 16 developers, all of them on code they knew intimately, and nothing in it about juniors or unfamiliar work.

The error pointed one way, and that much survives a result this small. People misjudged how AI changed their own work, before and after doing it, and a panel agreeing that an exercise is fair has produced the same kind of estimate.

Ask what your bar describes, and who ever cleared it

A cut score answers to the work, not to the applicant pool. The Uniform Guidelines say that where cutoff scores are used, they "should normally be set so as to be reasonable and consistent with normal expectations of acceptable proficiency within the work force" ³.

The comparison in that sentence is the people already doing the job, so the first question is what your line was set against.

Two bars fail that test in opposite directions.

One describes the best AI work anyone on the team has ever done, usually the person who built the internal tool. A ceiling is not a bar.

The other describes a wish, and nobody currently employed could pass it either. Checking that is cheap.

So run the exercise on the people already doing the job.

Read that result with a correction: the current team was hired under the old process, has had months of practice with your tools, and is missing whoever left. Running the exercise on your own team locates the line relative to work you have already accepted. It does not settle where the line belongs.

If nobody set the bar against anything, the exercise was never the problem, and setting a defensible bar for the role is the step that got skipped. A number nobody can source is not a standard. It is a preference with a threshold on it.

Read the failed submissions for their cause

The failed submissions are a sample worth reading. Pull twenty of them and write down, for each one, the specific thing that went wrong: the claim nobody checked, the brief they misread, the tool they had never opened, the clock. Then sort the list. If most of the failures cluster on something other than AI skill, the exercise is measuring that other thing.

Four causes are worth checking first.

The brief. A task written by someone who already knows the answer reads as unambiguous to everyone who already knows it.

Candidates who fail on scope were told the scope badly.

The clock. A time limit turns a verification exercise into a typing exercise, and the candidates it fails first are the ones who check their work.

The tooling. A candidate whose entire AI history is one assistant, working in yours for the first time, spends the first fifteen minutes on the interface rather than the task.

The rubric. Where the scores line up with how well the submission is written, the rubric is grading polish.

The fail rate is then a writing-quality distribution with a pass mark on it.

And the reviewers, who are less a cause than a multiplier: where two people score the same submission differently, there is no line for anyone to be above.

One cause carries a legal edge, and it is the cut score itself.

In guidance published in 2022 and since taken off its site, the EEOC worked through a business requiring 90 percent on a gamified memory assessment. A blind applicant who cannot play the game is rejected, and might still have a very good memory ⁴.

The pass mark is the mechanism in that example. Screening someone out is not automatically unlawful, and the hypothetical was never litigated.

But a rate that looks fine in aggregate can still fail one person for a reason unrelated to the job, and the accommodation question is easier to answer before the round than after it.

Decide whether this is a bar problem or a pipeline problem

If the exercise survives repair and the bar survives its check, the fail rate is telling you about the pipeline. A thin pipeline is a sourcing result, and no scoring table fixes one. Widen the search, rewrite the posting, pay differently, or hire for the trajectory and train the rest. Lowering the line changes who counts as qualified and nothing else.

Splitting the bar is the move most often confused with lowering it.

A hurdle says a candidate cannot come in below this line at all. A trade says weakness here can be bought back with strength somewhere else.

Choosing between a hurdle and a trade is a real decision, and running both at once means the line does no work.

A rank order is the third option, and the most honest one when the evidence is thin. Taking the strongest four of forty rather than everyone above a threshold makes a weaker claim about each candidate, which is sometimes the only claim an hour of work supports.

The objection here is the calendar. A hiring manager with an open req and a quarter to fill it does not have six weeks to rebuild an exercise, and a bar defended into a vacancy carries costs that never appear in the file.

Sometimes the honest move is to hire on the evidence in hand and write down that you did.

Write down what moved the line, and when

A moved cut score needs four things in the file: the old line, the evidence that moved it, the effective date, and who approved. Write them before the change rather than after, because a reason recorded afterwards reads as a reason invented afterwards. Then apply the new line to everyone from that date forward.

In 1991, Congress added a sentence to Title VII that leaves no room.

The statute makes it unlawful, in selection or referral for employment or promotion, "to adjust the scores of, use different cutoff scores for, or otherwise alter the results of, employment related tests on the basis of race, color, religion, sex, or national origin" ⁵.

So a separate pass mark for any group named there is off the table, and so is re-scoring one group's submissions. What stays available is repairing the exercise for everyone.

Measuring where the failures land is a different job from moving the line, and it comes first.

Selection rates and the four-fifths rule are where that measurement starts.

None of this turns on how formal the exercise looked. The Uniform Guidelines define a selection procedure as any measure used as a basis for an employment decision, reaching from paper-and-pencil tests through "informal or casual interviews and unscored application forms" ⁶.

A hiring manager's afternoon exercise sits inside that sentence, and an employment lawyer should see any change before it runs.

Seventy percent is still the number on the page. After the diagnosis it belongs to something: an exercise that asked for the wrong thing, a pipeline that never carried the skill, or a bar that was right and unpopular.

The regulation's phrase for what a cutoff should track is "normal expectations of acceptable proficiency within the work force" ³. Yours is down the hall.

See what gets scored >