Do Not Require a Skill Nobody Can Score: Borrow the Evaluator First

Get the evaluator before you write the requirement

Start with the person, not the wording. One practitioner who does this work well, for a paid afternoon, will tell you more about what the role needs than a month of reading other companies' job descriptions. Sources, roughly in order of how fast they answer: an investor's network, a former colleague, a contractor you have paid before, and a peer at a company one size up.

Be specific about what you are buying: a description of the work, and a way to tell good from plausible. A referral, a read on the market, or an opinion about whether a candidate seems smart will not get you there. Two hours is enough if you arrive with a real task and your questions already written.

Relevance is what you are paying for, and a measured gap sits behind it. In the job knowledge literature the 2022 re-analysis relies on, all 164 studies together produced a mean observed validity of .22, while the subset using knowledge tests built for the job in question produced .31, rising to .40 once corrected for unreliable performance ratings ¹. Those are two subsets of one meta-analysis with no controlled experiment behind them, and job knowledge tests assume candidates who already have the knowledge, which makes this evidence about hiring experienced people. The direction still matters for a buyer with no expertise: what an assessment is about does more work than what format it takes, and only somebody who knows the work can set that.

One rule while you shop. If a vendor cannot tell you who wrote their key and what a wrong answer looks like on it, you have paid somebody else to have your problem. That is the substance of the build-or-buy decision for an AI exercise, which applies to any domain you cannot read.

What a borrowed evaluator has to hand you

Three artifacts, and one page holds all of them. First, a real task from the next quarter of the job, written the way a candidate would receive it. Second, a strong answer and a plausible-but-weak answer described side by side, in enough detail that a non-practitioner can tell them apart. Third, the two follow-up questions that separate the two.

The second artifact is the one people skip and the one that does the work. A description of a strong answer on its own is unusable, because almost everything looks strong to a reader who cannot see what is missing. The pair makes the difference legible: one answer names the constraint and checks its number against a second source, the other asserts both.

Then turn the pair into a written scale, and decide in advance how the marks get added up. A meta-analysis of selection and admissions studies found applicant data combined by a stated rule correlated .44 with job performance, against .28 when the same kinds of data were combined by expert judgment, which the authors describe as a population-level improvement in prediction of more than 50% ². Read that as a relative gain in a correlation: the job-performance comparison rests on nine studies and the gap is smaller on other criteria, so do not oversell it. A mechanical rule here can be as plain as adding unit-weighted marks on a written scorecard, which is exactly what a borrowed evaluator page makes possible.

What the evaluator should not hand you is a verdict. Having the expert sit in and give a gut read at the end is the one move the evidence argues against: adding an intuitive judgment on top of two mechanical predictors moved the correlation with the criterion from .45 down to .35, and the review reporting it notes that the idea of a naturally better interviewer is not supported by the evidence on variance in interviewer validity ³. That study drew on 1943 university admissions data and was never about hiring, so do not overclaim it. The division of labour still holds: experts collect and define, a written rule combines. Getting non-practitioner managers to apply that rule consistently is its own problem, worked through in how to get managers who do not use AI to judge AI-assisted work.

Why an unjudgeable requirement is also an undefendable one

Because the standard a selection procedure has to meet is stated in terms of the job, and nobody in the room can say what the job requires. Under Title VII in the United States, once a complaining party shows a particular practice causes disparate impact, the employer has to demonstrate it is job related for the position in question and consistent with business necessity ⁴. That demonstration starts from a job analysis, and a job analysis is what is missing.

The principle is older than the statute. In Griggs v. Duke Power Co., decided in 1971, a unanimous Supreme Court held that Title VII reaches practices that are fair in form but discriminatory in operation, that the touchstone is business necessity, and that nothing forbids testing, only giving a test controlling force unless it is demonstrably a reasonable measure of job performance ⁵. The Court's summary line is the one to keep: a test must measure the person for the job and not the person in the abstract.

Calling the stage informal does not move it outside any of this. The Uniform Guidelines define a selection procedure as any measure, combination of measures, or procedure used as a basis for any employment decision, and say so explicitly for the full range from paper tests through informal or casual interviews and unscored application forms ⁶. The same Guidelines say a validity study rests on a review of information about the job and should include a job analysis, which for a content-valid procedure has to identify the important parts of the work required for successful performance and their relative importance ⁶. So a founder who decides on impression in a conversation has not avoided running a test. They have run one with no record of how it was applied and nothing written down about the job it was for.

What follows is public legal fact rather than advice: the Uniform Guidelines date from 1978, the burden-shifting text is 42 U.S.C. 2000e-2(k) as added in 1991, and Griggs is 1971. Title VII covers race, color, religion, sex and national origin; age and disability claims run under separate statutes. Whether a requirement is defensible for a particular role belongs in front of counsel. The operational point needs no lawyer: the artifact that makes a requirement judgeable is the one that makes it defensible.

The AI-era version of this runs in one direction only. Plausible expert-looking work is now cheap to produce in a domain nobody on your side can check, which converts an evaluation gap from a slow hire into a wrong one. Nobody in the room can tell that the architecture is conventional, the model misspecified, or the filing headed for rejection. That failure surfaces in month four, long after the interview.

Run the task when no evaluator exists at any price

Write the task instead of the requirement, pay for it, and judge the work against the outcome it has to produce. If nobody reachable knows what good looks like, the only honest evaluator left is reality: a scoped, paid piece of real work with a stated outcome, a deadline and a definition of done a non-practitioner can check without an opinion about craft.

Pick an outcome with an external verdict attached. The migration ran and the numbers reconciled. The filing was accepted. The integration passed the partner's test suite. Each is checkable by somebody who could not have produced the work, which is the point of choosing it.

Be clear about what this does not buy. It is a sample of one on a task you scoped, so it says little about how the person handles a task you did not anticipate. It costs money, and it should: unpaid multi-hour work is a different conversation and a worse one. And it tells you only whether each candidate cleared the outcome, which is a narrow basis for comparing two of them. That is a real limitation, and still better than a bar nobody can apply.

One last pass before the requisition opens. Name the evaluator for every line on it, with a person's name. Any line with no name is a preference, and the posting should say so. The one- or two-person version of this is worked through in making the first hire into a role you cannot judge, and the answer is the same: borrow the reader, or post the task.

See what gets scored >