How Do You Pilot an Assessment Vendor Before Making It a Hiring Gate?

What does a shadow-mode pilot actually look like?

A shadow pilot runs the assessment on real candidates and lets none of it reach a decision. The cohort gets invited, the vendor returns results, and those results go into a folder nobody on the hiring loop can open. You interview, decide and hire exactly as you would have without it. When the pilot closes, you open the folder and compare.

Three things break it, and all three are procedural rather than technical. A result leaks to someone in the loop, usually on a debrief call where a manager asks how the finalist did. The cohort is picked for convenience, so it fills up with the candidates a recruiter already liked. Or nobody writes down what the pilot was supposed to tell you, and it ends when someone's patience does.

Shadow mode is also the cheapest legal posture you will ever hold with a new instrument. The Uniform Guidelines define a selection procedure as any measure, combination of measures, or procedure used as a basis for an employment decision ¹. While the results reach no decision, the assessment is not one. The moment a result changes who advances, it becomes one, and it has to be job-related and consistent with business necessity, a bar the EEOC applies to work samples and simulations by name ⁴.

Tell the candidates. A pilot people were not told about is a study nobody consented to, and the first candidate who works out that their hour changed nothing will say so in public. One sentence in the invite carries it: this step is being trialed, it will not affect this application, and here is who sees the record.

Log the funnel while you are at it, because the pilot measures your process as much as the instrument. Invite-to-start, start-to-finish, median minutes, and every drop-off with a reason where you can get one. An instrument with a 40% completion rate has already answered the adoption question whatever its results say, which is why testing for AI skills without lengthening the loop is a design constraint and not a nice-to-have.

Decide the sample size before the first invite

Write the number down before anyone is invited, because an open-ended pilot ends the moment the results flatter someone. The Uniform Guidelines put that call on the employer: the number of persons needed for a meaningful criterion-related study is determined by the user, from the selection procedure, the potential sample and the employment situation ². Pick a number, a window and a stop date, and hold all three.

Then be honest about what that number can carry. A criterion-related validity study is technically feasible only where an adequate sample of persons is available to achieve findings of statistical significance ¹, and thirty candidates in one quarter is not that. A pilot at that scale answers operational questions (can two reviewers read the same session and agree, do candidates finish, does the instrument disagree with your process about specific people), and it does not produce a local validity coefficient. Vendors rarely volunteer that distinction. Ask which of the two you are buying.

Six things get fixed in writing before the first invite:

• The cohort. One occupation, one seniority band, one requisition family. Not "whoever we can get."
• The number, and the window it has to arrive in. A pilot with no end date is a rollout nobody approved.
• The criterion measure, and the name of the person who records it.
• The comparison you will run, written as a sentence, while no result exists to shape it.
• The decision rule. What result makes this a gate, what makes it an input, what makes it a no.
• Who may see results during the pilot. A named person, not a team.

Keep the impact counts by race, sex and ethnic group from the first invite; the Guidelines expect a user to hold records disclosing what its selection procedures do to identifiable groups ³. Expect those counts to be uninformative. Where impact evidence rests on numbers too small to be reliable, the Guidelines look instead to impact over a longer period or to the same procedure used in similar circumstances elsewhere ³, which in practice means the vendor's own audit rather than the eighteen people in your cohort. A validated assessment and a bias-audited one are different claims, and a pilot is where you find out which one the vendor actually has.

What criterion do you compare the results against?

Your own performance data, on the people you hired anyway. Choose the measure before any result is visible, and choose one that represents work: the Guidelines say the criteria used should represent important or critical work behaviors or work outcomes ². Ramp time to independent output, first-quarter deliverables accepted without rework, or a structured manager rating on named behaviors: one of those, defined in writing, with a name attached to it.

Every available criterion is contaminated, and the skill is picking the contamination you can name. An overall manager rating is the easiest to collect and the weakest signal: that manager interviewed the candidate, remembers the debrief, and is rating a person they already decided to hire. Retention measures compensation, commute and a reorg. A structured rating on the same behaviors the assessment claims to measure is the best of a bad set at this size, and it only works if the anchors are written before anyone is rated.

The ceiling on any shadow pilot is that only hires have criterion data. You never observe how the rejected candidates would have done, so the comparison runs on the survivors of the screen the new instrument was supposed to improve. That does not make the pilot worthless; it makes one particular result unavailable. Say it out loud in the plan, because someone will otherwise present a correlation as though it covered everyone who applied. Whether assessment results predict performance at all is not a question a thirty-person cohort settles.

So read the disagreements one by one. The most useful output of a shadow pilot is the short list of candidates where the instrument and your process reached opposite conclusions: the strong hire whose session was thin, the candidate you passed on whose work held up. Ten of those, read by the hiring manager against the evidence behind each one, are worth more than an aggregate you cannot compute at this sample size. That reading is only possible if a result carries evidence rather than a number. Check which you are getting before you sign, not after.

Why doesn't one role's pilot license a rollout?

Because the criterion was one occupation's performance data, and evidence does not travel with the contract. The Guidelines expect a study sample to be representative of the candidates normally available in the relevant labor market for the job in question ², and the EEOC's own best practice is that a procedure be properly validated for the positions and purposes it is used for ⁴. A pilot on analysts says something about analysts.

Purposes is the half buyers skip. An instrument piloted as a late-stage input and then deployed as an early-stage cut is a different procedure with a different impact profile, on identical content from the same vendor. Change the stage, the pass rule or the population and you are back to an untested configuration, whatever the first pilot showed.

The rollout question is also a maintenance question. Every role added is another case, another set of rating anchors and another cohort someone has to watch, and one instrument across every department is a rubric question rather than a purchase. Two or three roles, sequenced by where a bad hire costs the most, is a program. Twelve at once is a rollout carrying one role's evidence.

Run the second role as its own pilot, with its own cohort, criterion and stop date. It costs less than the first because the protocol exists and the reviewers are calibrated, and it is the only thing that puts real ground under the third. Where two roles genuinely do the same work on the same material (a pricing analyst and a sizing analyst reading the same kind of packet), write that down as a job analysis and reuse the case. Where they do not, do not.

How do you decide whether to make it a gate?

Apply the rule you wrote at the start, and notice that gate is not the only option. Three outcomes are available: make it a gate, keep it as an input a human reads alongside everything else, or drop it. The middle one is usually right and almost nobody writes it into the plan, so the decision defaults to adopt-or-abandon on evidence that supports neither.

Gating changes what you owe. A result that decides who advances is a selection procedure ¹, and it has to be job-related and consistent with business necessity, the bar the EEOC applies to work samples and simulations alike ⁴. Keeping the result as an input a hiring manager reads is a smaller claim, a smaller exposure, and for most instruments at pilot scale the only description the evidence supports.

Name the failure conditions in advance and let the pilot fail. Completion below a stated rate. Two reviewers reaching different conclusions on the same session. Results that track exactly what your resume screen already tracked, which means you bought a slower version of it. A candidate complaint you cannot answer. Stop when one of those lands, rather than extending the window until it clears.

The vendor's documentation is not your defense either. The EEOC is explicit that the employer is responsible for ensuring its selection procedures are properly validated for the positions and purposes they are used for, whatever the publisher supplied ⁴. Ask for the technical report, the sample it was built on and the audit before the pilot rather than after, because the questions to put to an assessment vendor are the ones a regulator asks later, in a worse room.

Keep the pilot's own file: the plan as written, the cohort, the criterion, the results, the disagreements, the decision, with dates. It runs to two pages and it is the only thing that will answer, two years from now, why this step exists and what it was tested on. The evidence a vendor should hand you belongs in the same folder.

See what gets scored >