How Do You Evaluate an AI-Skills Assessment Before Buying?

What Is the Assessment Benchmarked Against?

Ask for the job analysis. Under the Uniform Guidelines, a work-sample claim rests on an analysis of the important work behaviors of the job in question, and the procedure has to be a representative sample of them ¹. A vendor benchmarking against a general AI aptitude construct has no such analysis, because the construct is not an occupation. Get that answer in writing before the demo.

The Guidelines are unusually blunt about the difference. A content-validity strategy is not appropriate for a procedure that purports to measure a trait or construct, and the examples given name aptitude and judgment directly ¹. What content validity does support is a sample: the manner, setting and level of complexity of the procedure should closely approximate the work situation ¹. "AI aptitude" is the label that cannot be defended by pointing at the task; "this is the memo the job produces" is the one that can.

That gives you a test to run inside the demo. Ask which occupation the case was written for, who wrote it, and what in the role it was drawn from. A diligence memo over a source packet and a change to an unfamiliar codebase are different work situations, and no single case approximates both. An answer along the lines of "it works across every role" is the vendor telling you the benchmark is the construct.

Ask too where the validity evidence came from, because most of it came from someone else's applicants. Borrowing a study is permitted, but only where incumbents in your job and in the studied job perform substantially the same major work behaviors, shown by job analyses on both jobs ². The liability does not travel with the study either: the Guidelines caution that users are responsible for compliance, whatever the publisher's manual says ². Settle what the assessment is supposed to measure before you compare two of them on price.

What Does the Candidate See, and When?

Ask three things: what the candidate is told before the session starts, how much notice they get, and whether they ever see the result. Where New York City's rules apply, an employer must give notice 10 business days before an automated employment decision tool is used, and the bias audit summary has to be publicly available ⁵. A vendor whose invite flow cannot support that timing constrains your process, not theirs.

The notice duty attaches to the employer. So an assessment sent the same afternoon it is scheduled cannot satisfy it, however compliant the tool is. Ask for the invite mechanics in the RFP: how far ahead a link can be issued, how long it stays valid, whether an alternative process exists for a candidate who asks, and what disclosure the candidate reads before anything is captured.

Then ask what the candidate is shown at the end. A candidate-facing report is rare and is not required in most jurisdictions, which is exactly why it is worth pricing: a finding the candidate will read gets written differently from one only your hiring committee sees. Ask for a redacted sample of both documents, and read them side by side. If a sentence about a candidate would embarrass you in front of that candidate, it will eventually be read by one.

Accessibility belongs in this conversation rather than in a security review six weeks later. A timed browser task with audio, screen capture and a fixed window has several distinct accommodation surfaces, and "we extend the clock" covers one of them. Ask which the vendor has actually built, and how a request is made without disclosing a diagnosis to the hiring manager. The detail is worked through in accommodations on an AI-open assessment.

Does It Produce a Single Number?

Ask what comes out, then ask what supports using it that way. The Uniform Guidelines separate the two uses: evidence sufficient to support a procedure on a pass/fail basis may be insufficient to support the same procedure on a ranking basis, and a user choosing to rank should hold evidence supporting that use ³. A composite score is a ranking instrument whether or not the vendor describes it as one.

So the useful follow-up is not whether the number is accurate. It is what the number means and how it was assembled. A blended score has weights, and the weights are a claim about the job: that framing counts twice as much as verification, say, for this occupation. Ask what the weights are, who set them, and against what evidence. "The model learned them" means they were learned from someone's past decisions, which is a model of the old screen rather than of the work.

Cutoff scores carry their own standard. Where they are used, the Guidelines say they should normally be set so as to be reasonable and consistent with normal expectations of acceptable proficiency within the workforce ³. A vendor-supplied default cutoff was not set against your workforce, and adopting it makes it yours. Ask what happens to a candidate one point below it, and whether anyone reads the underlying work.

The alternative is a report that stays disaggregated: separate findings on separate behaviors, each readable on its own, none of them added together. That output is slower to sort and harder to put in a spreadsheet, which is the trade you are making. It also survives the question a single number cannot answer, which is what specifically this person did. Whether the number predicts anything at all is a separate claim with its own evidence.

How Is a Divergent Result Evidenced?

Ask what sits behind a finding when two candidates come out differently. The documentation standard is specific: the user should identify the work behavior each part of the procedure is intended to sample, and reliability estimates should be reported for the procedure where available ⁴. If a vendor cannot say what an item measures, nobody in your company can defend the gap between two candidates, not to the candidates and not to counsel.

Ask for the technical report by name rather than the one-page summary. For a content-validity claim the Guidelines mark particular items essential: the method used to analyze the job, a complete description of the work behaviors and how their importance was determined, an explicit description of what the procedure samples, and a comparison of the manner, setting and complexity of the procedure against the work situation ⁴. Those are the pages anyone reviewing the decision later will actually want.

Then ask about agreement. Two reviewers reading the same session, or the same scorer run twice, should agree at a rate the vendor can state as a number with a method behind it. "Our rubric is standardized" is a description of intent, not a measurement. If nobody has measured it, that is a real answer. Write it down, and treat the assessment as an input rather than a gate until it exists. What rubric reliability takes to establish is the same work whether you buy it or build it.

A "bias tested" badge is a different document answering a different question, and vendors routinely offer the first when asked for the second. An audit compares selection or scoring rates across groups; a validity study argues the procedure relates to the job. Ask for both, ask which jobs each one covered, and check the audit date against the last model change ⁵. The distinction is worth settling before the procurement call.

Run the Checklist Against One Real Req

Take the four questions to a live requisition rather than to a category review. Pick the role you are hiring for now, ask the vendor to name the case its bank would open on, and read that case against the job you posted. A vendor selling a work sample can do this inside a week. One selling a construct will offer a general demo and a customization roadmap instead.

Put the questions in the RFP, not the demo. A demo is answered by a salesperson in the room; an RFP is answered by someone whose answer gets filed. Four documents, each with a date on it: the job analysis, the technical report, the bias audit with its scope note, and a redacted sample of both the employer report and the candidate report.

Then run it beside your existing round before it replaces anything. A pilot on one requisition, where both instruments see the same candidates, tells you what the assessment adds and what it duplicates, and it produces the record you will want if anyone later asks how the tool was chosen. Scope the pilot before you sign, because a vendor who will not pilot on your req has told you something about the evidence. The mechanics are in running a pilot before the assessment becomes a gate.

Buying is not the only path, and the comparison is closer than the pricing suggests. The expensive parts of an assessment are the authored case, the answer key and the reviewer time, not the software that serves it, which is why the build-or-buy question turns on authoring capacity rather than on engineering.

See what gets scored >