Proving a Hiring Process Change Actually Worked

Why a Before-and-After Chart Proves Nothing

Because everything else moved too. Between the two quarters on that chart the labor market changed, the req mix changed, the sourcing channels changed, two interviewers left, and the panel got better at the new questions simply by asking them fifty times. Any one of those can produce the improvement you are crediting to the change, and no amount of analysis afterwards can separate them.

Measuring a comparable effect properly takes a scale no hiring pilot reaches. Three company-run randomized trials at Microsoft, Accenture and an anonymous Fortune 100 firm had to be pooled across 4,867 developers before they produced a readable result, a 26.08% increase in completed tasks with a standard error of 10.3%, because each experiment on its own was too noisy to carry a finding ¹. That is a randomized design with thousands of participants, measuring something as countable as tasks finished per week, and it still could not stand alone.

Set that beside a recruiting change evaluated on forty hires across two quarters, with no control and an outcome nobody defined in advance. The comparison is not close. It does not mean the change was worthless; it means the chart cannot tell you either way, and a chart that cannot tell you either way tends to get read as confirmation.

Run the Old Process on Some of the Reqs

Assign whole requisitions, never individual candidates. Two candidates for the same role getting different loops is a fairness problem, and the first question to put to counsel. Two reqs in the same family running different loops is ordinary variation in how a company works. Alternate by req as they open, or by week, and write the assignment rule down before the first req lands so nobody can choose sides case by case.

Hold constant what you can. Same role family, same level band, same sourcing channels, same recruiters where possible. If the new process is only running on engineering reqs and the old one on everything else, you have measured engineering. Where the volume is too thin to split within one family, alternate in short blocks across time, which at least mixes the seasonality through both arms.

Size it honestly before you start. Count how many reqs the family opens in a quarter, halve it, and ask whether the resulting number of hires could distinguish anything. Often it cannot, and knowing that in advance changes the plan in a useful way: extend the window, widen the families, or pick a nearer outcome. Discovering it afterwards produces an argument instead.

The pilot version of this is smaller and worth doing first. Piloting an assessment before making it a hiring gate runs the new stage alongside the existing loop without letting it decide anything, which gives you a clean read on what it would have changed.

Name the Outcome Before You Start

Write the outcome, the window and the decision rule in one paragraph, dated, and share it before any data exists. This is the single cheapest step in the whole method and the one most often skipped, because naming the outcome forecloses the move everyone makes later, which is to search the dashboard for whichever line went up and present that one.

Pre-registration is standard practice in the studies people quote at each other in these arguments. The Boston Consulting Group experiment on AI and consultant output was pre-registered across 758 consultants on 18 tasks, and its published result includes both directions: inside the tool's capability, output and quality rose sharply, and on one task chosen to sit outside it, the same group did worse ². A study free to report only the tasks that worked would have published half of that.

Your paragraph needs four things. The primary outcome, stated as a number somebody else could compute from your systems. The window, in weeks, with a start date. The decision rule, saying what result would make you keep the change, revert it, or extend the test. And the exclusions, naming which reqs will not count and why. Anything you add to the analysis afterwards is exploratory, which is fine as long as it is labelled that way and does not become the headline.

One rule for reading the result: a difference you would not have predicted, in a measure you did not name, on a subgroup you did not plan to split, is a hypothesis rather than a finding.

Which Interim Measures Are Honest Proxies?

The ones that measure the mechanism you changed. If the change was a scored work sample, the honest interim measures are agreement between reviewers, how complete the evidence is on each scorecard, and candidate drop-off inside the new stage. Each of those tells you the stage is working as designed. None of them tells you the hires are better.

You need interim measures because the real outcome is slow. Median time-to-fill for nonexecutive roles ran 44 days in SHRM's 2021 benchmarking of its member organizations, with the slowest quarter at 73 days or more, counted from opening the req to an accepted offer ³. Add a start date, then a ramp, then a retention window matched to the role, and the first honest read on hire quality sits two or three quarters out.

So publish the proxy with its label attached. Write "interim proxy, not quality of hire" in the same sentence as the number, every time, including in the slide that goes to leadership. Proxies drift into headlines by repetition, one slide at a time, and the label is what stops it.

Two proxies are worth more than one because they fail differently. Reviewer agreement rises when a rubric is clearer and also when it is blander, so pair it with something that moves the other way under blandness, like the share of scorecards carrying a specific piece of evidence. Writing a rubric two reviewers score the same way is the work behind the first of those. And before you read any stage-level rate as a result, check that the denominator held still: which funnel metrics still mean anything is the same trap one level down. Once the window closes, measuring quality of hire on outcomes the manager does not own is what the interim numbers were standing in for.

See a sample report >