Assessment design
The AI Line Is a Hurdle or a Trade, and Most Teams Never Chose
How much a weak AI exercise should count depends on a choice most teams never made: whether the AI line is a hurdle nobody can trade against, or a weight that domain strength can offset. For most roles open today the honest structure is compensatory with a floor. Domain skill carries more of the weight because its validity is measured and the AI line's is not, and the floor exists to catch the candidate who cannot check anything. Pick one, write it next to the line, before the results exist.
The takeThe failure here is rarely a wrong number. It is a team that behaves compensatorily until one candidate makes the room uncomfortable, then reasons as though a hurdle had been there the whole time. That switch is invisible from the inside and obvious from the outside, and it lands on whoever happened to trigger the discomfort. Writing down which model is running, before anybody has a name attached to a result, gives up the freedom to be talked into either answer afterwards. That freedom was never worth what it costs.
Where Olive fits
Open a role and see what the work shows
Olive returns six separately-evidenced findings rather than one number, each anchored to a moment in the session, so a weak result names which act was missing. The candidate is granted the same report, on every tier.
Assess your shortlistWhat Are the Two Structures, and Which One Is Running?
A hurdle and a compensatory model are the two available structures, and only one of them is running in your loop right now. A hurdle means the AI competency has to be cleared on its own terms, with no credit from anywhere else. A compensatory model means every dimension is weighed together, so a strong domain result can offset a weak AI one. Most teams have never picked, which is why the question feels unanswerable.
The tell is what happens when a strong candidate does badly on one line.
Behave compensatorily and the finalist advances, because the domain result is doing the work the domain does. Then somebody says the AI exercise was bad enough to matter on its own, and the panel starts arguing as though the line had always been a hurdle. Neither rule was written down, so both stay available.
A hurdle is not a stricter version of a weight; it rejects a different set of people.
Pick the structure before the loop opens and before any result exists. The order matters more than the number, because weights chosen after the results are visible get chosen to reach somebody, and everyone in the room can feel that happening even when nobody says it.
The same choice shows up one level down, in what a round is allowed to produce. Whether you rank candidates against each other or set a bar they either clear or do not decides what a low AI result can be traded against. A process that has not settled that question has not settled this one either.
Why a Hurdle Costs More to Defend Than a Weight
A hurdle rejects people a compensatory model would hire, and federal selection law asks a specific question about that. The Uniform Guidelines on Employee Selection Procedures have been in force since 1978. Where cutoff scores are used, they say, the scores should normally be set so as to be reasonable and consistent with normal expectations of acceptable proficiency within the work force 3.
Read that standard closely and the burden is not statistical. The Guidelines point at the job: the proficiency the work actually needs, not the result the panel wishes candidates had produced.
The same paragraph allows a higher cutoff where ranking leaves those below it little chance of selection anyway, and still asks that the degree of adverse impact be considered 3.
A pass mark is also the exact mechanism that creates disability exposure. The EEOC's 2022 guidance on software and algorithms in hiring, read now through the Internet Archive rather than the agency's site, works the point through a business requiring 90 percent on a gamified memory assessment. A blind applicant who cannot play the game is rejected, and may still have an excellent memory 4.
Screening somebody out is not automatically unlawful. It becomes a problem where the person screened out could perform the essential functions of the job, with a reasonable accommodation if one is legally required 4. A pass mark is what turns the question into one yes-or-no instead of a weighing.
Where a hurdle produces a group disparity, Title VII sets the terms. Since 1991 the statute has said a practice causing disparate impact is unlawful unless the employer demonstrates it is job related for the position in question and consistent with business necessity. It also normally makes a claimant point at each particular practice 5.
A hurdle is easy to point at. It carries a name and a number on the document the interviewers filled in.
None of that is an argument against a hurdle. It is an argument for setting one the way any pass mark gets set, which is the subject of how to set a defensible bar for good enough at AI in a specific role.
How Much Should the Domain Side Weigh?
More than the AI line, for most roles open today, and the reason is evidence rather than preference. What an assessment is about does more work than what format it takes. In the job knowledge research behind the 2022 re-analysis of selection validity, 164 studies pooled to a mean observed validity of .22. The 59 studies using tests built for the job in question reached .31, or .40 once corrected for unreliable performance ratings 1.
Those tests presuppose people who already hold the knowledge, so the finding describes hiring experienced candidates rather than entry-level ones. It also compares two subsets of one meta-analysis, which is not an experiment 1.
The table those numbers sit in was itself corrected in 2022. Paul Sackett, Charlene Zhang, Christopher Berry and Filip Lievens found that earlier meta-analyses had systematically overcorrected for range restriction, and the mean validity across the top five predictors fell from .49 to .37 1.
All five of the earlier top five are still on the list, in what the authors call a top five plus a tie, which is the part people get backwards. The repair itself is set out in the validity table everyone quotes was corrected in 2022.
Structured interviews came out top at .42, work sample tests at .33 1. Treating that gap as the exercise losing to the interview overreads a table assembled from separate meta-analyses.
A scorecard is a composite, which is the part most weighting arguments skip.
In the same authors' 2023 follow-up, combining predictors reaches about .61, and giving cognitive ability a weight of zero costs .05, dropping the composite to .56 2.
Dropping that one predictor costs less than the room assumes. But that figure holds only where predictors are combined mechanically with sensible weights and the assumed correlations hold, which four unstructured conversations stacked on each other do not 2.
None of those coefficients measured AI skill, and no published estimate yet says what an AI exercise predicts about job performance at the person level. So the weight on the AI line does not rest on the same kind of evidence as the weight on the domain. That gap is the honest version of what evidence there is that being good with AI predicts job performance.
Read the Exercise Again Before You Re-Weight Anything
A weak AI result is not one finding, and the weighting argument usually starts before anyone has asked which finding it was. Three different candidates produce the same low number. One did not know the domain well enough to catch the planted error. Another knew the domain and did not check. A third checked carefully and ran out of time.
Only the middle one is the thing the line was built to catch.
A finalist strong on the domain who missed the error is the case worth reading twice, because the two explanations point in opposite directions.
Missing an error inside a domain they know is a habit. Missing one in a domain they have never worked in closes with time on the job.
The exercise itself is the other suspect. One that rewards finishing over checking measures speed. A candidate who slowed down to verify comes out below one who did not, which is a defect in the instrument rather than a fact about the person.
Seniority moves the floor, not the structure. What the line is worth for a new grad and what it is worth for a director are the same shape at different heights, worked through in how much AI skill should weigh, level by level.
Teams whose work is now mostly reviewing what a model produced will say the compensatory model has already been overtaken, and on those roles they are right. A domain expert who cannot check anything is not a strong hire with one weak line. There the hurdle is the honest structure, and a floor set low enough that everyone clears it is a hurdle in name only.
Write the Rule Next to the Line on the Scorecard
One sentence next to the AI row settles the argument the panel is about to have. Say whether the line is a hurdle or a weight, say the number and the evidence that clears it if it is a hurdle, and say who is allowed to overrule it. Interviewers read the scorecard, not the meeting where the decision got made.
Three lines, close enough to paste:
- The AI line is compensatory. A strong domain result can offset a weak one inside the composite.
- The floor is a hurdle: nobody advances who cannot say what they checked and what they checked it against.
- The floor sits at what the work needs, and it moves only on evidence about the work.
One adjustment is out regardless of which structure got picked. The same 1991 statute makes it an unlawful employment practice to adjust the scores of, use different cutoff scores for, or otherwise alter the results of employment related tests on the basis of race, color, religion, sex or national origin 5. Confirm the current rules with employment counsel; none of this is legal advice.
The other adjustment is lawful on its own and still corrosive: moving a weight once the results are in, to reach a person. It leaves nothing on the scorecard and everything in the room, and the next candidate gets the version nobody moved for them.
Under a compensatory model with a floor, the same finalist produces a different conversation. The panel is not arguing about how much a weak exercise weighs; it is asking whether the floor was cleared, which is a question with an answer. The next requisition can open with that sentence already in the file, before anyone has a name attached to a result. Nobody argues with a rule that was already there.
Common questions
What is the difference between a hurdle and a compensatory scoring model?
A hurdle is a line that has to be cleared on its own, with no credit from any other part of the scorecard. A compensatory model weighs every dimension together, so a strong domain result can offset a weak AI one. The two select different people from the same pool: a hurdle rejects candidates a compensatory model would hire. Most hiring teams have never chosen between them, and behave compensatorily until a low result makes somebody uncomfortable, at which point the discussion proceeds as though a hurdle had been there all along.
Can a strong domain score offset a weak AI exercise?
Under a compensatory model, yes, and that is what most roles open today call for. Domain relevance is what the job knowledge research rewards: tests built for the position in question reach .31 against .22 pooled across every such test, or .40 once corrected for unreliable performance ratings. Under a hurdle, no, by construction. The structure decides the answer, which is why it has to be picked before any candidate has a result attached to their name. A floor sits underneath either version: nobody advances who cannot say what they checked and what they checked it against.
How high should the AI floor be set?
At the level the work actually needs, and no higher. The Uniform Guidelines say a cutoff score should normally be set so as to be reasonable and consistent with normal expectations of acceptable proficiency within the work force. A higher cutoff may be appropriate where applicants are ranked and those below it have little or no chance of selection anyway, and even then the degree of adverse impact should be considered. In practice that means writing down what the job cannot be done without, then setting the line there. A floor everybody clears is not a floor. A floor that half the qualified market misses is a claim about the job somebody will eventually ask you to support.
Is it legal to weight an AI exercise differently for different candidates?
Weighting by the five classes Title VII names is prohibited outright. Since 1991 federal law has made it an unlawful employment practice to adjust the scores of, use different cutoff scores for, or otherwise alter the results of employment related tests on the basis of race, color, religion, sex or national origin. Different weights for different roles or levels are ordinary practice, as long as the weighting is set in advance and applied to everyone in that role. Moving a weight mid-loop to reach one finalist is not unlawful on its own, and it is the practice that makes the rest of the record hard to defend. Ask employment counsel about your jurisdiction.
What if the candidate missed the error because they do not know the domain yet?
Then the result is about knowledge rather than habit, and the two decay differently. Somebody who skipped a check inside a domain they know well has shown you a working habit. Somebody who checked carefully and still missed a domain-specific error has shown you a gap that closes with exposure. Reading the session for which of those happened costs twenty minutes and changes the weighting argument completely. Ask what they checked, what they checked it against, and what they would have needed to know to catch the thing they missed.
References
- 1. Revisiting Meta-Analytic Estimates of Validity in Personnel Selection: Addressing Systematic Overcorrection for Restriction of Range ✓ static1.squarespace.com Supports the job knowledge comparison used to argue that domain relevance carries the weight (164 studies at a mean observed validity of .22, the 59 job-specific studies at .31, rising to .40 once corrected for unreliable performance ratings), the fall in mean validity across the top five predictors from .49 to .37 with all five of the earlier top five remaining in what the authors call a top five plus a tie, and the ranking figures of .42 for structured interviews and .33 for work sample tests.
- 2. Revisiting the design of selection systems in light of new findings regarding the validity of widely used predictors ✓ cambridge.org Supports the claim that a scorecard is a composite: a combined predictor set reaches about .61, and giving cognitive ability a weight of zero costs .05, dropping it to .56, on the assumption that predictors are combined mechanically with sensible weights.
- 3. 29 CFR 1607.5 - General standards for validity studies, paragraph (H) Cutoff scores ✓ law.cornell.edu Supports the cutoff-score standard the article rests the hurdle argument on: where cutoff scores are used they should normally be set so as to be reasonable and consistent with normal expectations of acceptable proficiency within the work force, and a higher cutoff score may be appropriate where applicants are ranked and those below it have little or no chance of selection, but the degree of adverse impact should be considered.
- 4. The Americans with Disabilities Act and the Use of Software, Algorithms, and Artificial Intelligence to Assess Job Applicants and Employees ✓ web.archive.org Supports the claim that a pass mark is the mechanism creating disability exposure: the agency's worked example of a business requiring 90 percent on a gamified memory assessment rejecting a blind applicant who may have an excellent memory, and the screen-out standard turning on whether the person could perform the essential functions of the job with a reasonable accommodation if one is legally required.
- 5. 42 U.S.C. 2000e-2 - Unlawful employment practices, subsections (k) and (l) ✓ uscode.house.gov Supports two claims: subsection (k)'s disparate-impact burden, under which a practice is unlawful unless the employer demonstrates it is job related for the position in question and consistent with business necessity, with the claimant normally pointing at each particular practice; and subsection (l)'s prohibition on adjusting the scores of, using different cutoff scores for, or otherwise altering the results of employment related tests on the basis of race, color, religion, sex or national origin.
5 sources, numbered by first appearance. Every one was opened and checked against the claim it carries. How Olive sources claims
General guidance for hiring teams. What works at one company and one volume may not transfer to yours.
Olive assesses how a person works with AI. It does not detect AI-written documents, and it never produces a score, a ranking, or a match percentage for a person. Candidates read the same report the employer reads.