Assessment design
An AI-Use Score That Tracks Polish Is Measuring Polish
An AI-use score that moves in near lockstep with a presentation rating is measuring presentation. The likeliest reason two rubric lines agree that closely is that one impression produced both, and a reviewer forms that impression from prose. The test runs on scores already collected: have a second person rate presentation alone, then hunt for the candidates who scored high on one line and low on the other. If none did, the AI line is separating nothing.
The takeA rubric this well-behaved looks like a good one. Clean columns, raters who agree, no arguments in the debrief. But a scale that works produces disagreement, because the two things it measures come apart on real people. A column that never surprises anyone has stopped carrying information. Treat a suspiciously tidy correlation as a defect report rather than as validation, and go looking for the candidate it got wrong.
Where Olive fits
Open a role and see what the work shows
Olive returns six findings per session, each written or edited by a person and anchored to a timestamped excerpt from the work, with outcomes recorded as demonstrated, partly demonstrated or not demonstrated. How the writing reads is not one of the six.
Assess your shortlistWhy do the two columns move together?
Two explanations fit, and only one of them is good news. Either writing quality and judgment about AI work are close to the same thing in your applicant pool, or a single reader formed one impression of the submission and wrote it into both lines. The second is the likelier explanation, because the scorer reads both lines off the same document, in the same sitting, usually in that order.
Rating research calls the second case halo, a general impression standing in for a specific one.
In 2010 Murray Barrick, Brian Swider and Greg Stewart published a study of 189 accounting students put through structured mock interviews. Before a single structured question was asked, an interviewer had formed an impression during the rapport-building small talk. That impression correlated .42 with the same interviewer's structured score 1.
The second number is the one to carry. When a different interviewer supplied the structured score, the correlation fell to .25, and to .24 when that score came from a third interviewer who skipped rapport building altogether 1. Part of the .42 was one person's impression predicting their own later rating, and the rest survived the change of rater.
The sample was students in mock interviews, and the authors do not conclude that interviewers make up their minds in the first three minutes.
What the study shows is narrower and enough: a rating drifts toward whatever the same rater already thought.
Your spreadsheet has the same shape, with one feature that makes it worse. Nothing sits between the two lines at all. One person reads one artifact and fills in both, so no second rater exists whose disagreement could show up.
The repair starts in the scorecard's own design, where some lines survive this problem and some create it: four lines worth scoring on AI use, and two to delete.
Find the candidate who wrote badly and worked well
The re-rating check needs no new candidates and no new instrument. Pull thirty to fifty scored submissions from the last two rounds. Have someone who never saw the original scores rate each one for presentation alone, then set the new presentation ratings beside the AI-use scores recorded at the time. An afternoon covers it, and the answer sits in the disagreements rather than in the correlation.
Two kinds of candidate decide the answer.
The first wrote badly and worked well. The submission is disorganized and the sentences are clumsy. But the figure at the center of it was traced back to the filing it came from, with the discrepancy flagged in a footnote nobody asked for.
The second wrote beautifully and checked nothing: a clean deck resting on a number nobody opened.
If the file holds neither after fifty submissions, the AI line is not separating candidates. No published rule says how many disagreements a working scale should produce, and inventing a threshold here would repeat the mistake the rubric already made. Zero needs no threshold to read.
Two rules keep the re-rating honest. The second rater cannot be the person who scored the round, and cannot see the AI scores, because a rater's own earlier impression drifts into the score they give later, which is the leak the interview study measured 1.
Rate presentation on its own, against its own written definition, and record it before anyone opens the old file.
Agreement between the two raters is a separate question with its own trap. Raw percent agreement credits every band two people would have landed on by luck, which is why eight of ten is not agreement until you discount chance.
Write the anchors around acts a clumsy writer can still perform
An anchor that survives the re-rating check names an act a reviewer can point at, and one a clumsy writer can still perform. Traced the headline figure back to its source. Rejected the model's first framing and said what was wrong with it. Tested the claim against something outside the conversation. Each of those is a yes or a no, readable off a badly written page.
Set that against anchors of the familiar kind. Clear communication. Strong judgment. Well-structured approach. Every one of those is read off the prose, because the prose is the only place any of it is visible.
Of everything in this repair, only the anchors have a measured payoff behind them, and it is uneven.
Levashina, Hartwell, Morgeson and Campion's 2014 review of the structured-interview literature reports Taylor and Small's meta-analysis of 19 past-behaviour interview studies. Interviews using anchored rating scales showed criterion-related validity of .35, against .26 for interviews without them 2.
The gain in interrater reliability is the smaller of the two: .77 against .73 2. Anchors buy more in what a rating predicts than in how closely two raters land on each other.
Different studies produced those two numbers, not the same interview scored twice, so anchored scales travel with whatever else careful teams do. And they cover past-behaviour questions only, so nothing in them speaks to work samples. In its own words, the review calls the science of rating scales underresearched.
One finding inside that review matters more for a scorecard than the headline does. Anchoring every point of a five-point scale produced ratings more resistant to disability bias than anchoring only the top and the bottom 2. A scale with words at 1 and 5 and nothing in between is not the thing that was measured.
Anchors can only read what the assignment produced. A brief asking for a finished artifact hands every reviewer a finished artifact, which is why grading a take-home when every submission comes back polished is a question about the brief before it is a question about the scale.
The line nobody calls a test is still scoring English
The AI-use column is a line in a spreadsheet, filled in by one reader who also filled in the line beside it. Nothing in that description sounds like a test. But the Uniform Guidelines define a selection procedure as any measure, combination of measures, or procedure used as a basis for any employment decision, and they reach informal or casual interviews and unscored application forms 5.
The definition turns on what a rating does, however informal it looks: anything that moves the decision is a selection procedure.
The exposure runs to national origin first.
A column keyed to fluent English is keyed to something the job may not have asked for, and language background sits behind fluent English.
Title VII's disparate-impact provision, 42 U.S.C. 2000e-2(k), was added by the Civil Rights Act of 1991. It puts the burden on the employer to show that a challenged practice with disparate impact is job related for the position in question and consistent with business necessity 4.
Validation duties attach only where adverse impact appears, and where your own process falls is a question for employment counsel.
The sharpest measurement of what happens when a verdict rides on prose comes from text detectors. Seven detectors run over 91 human-written TOEFL essays by non-native English speakers produced an average false-positive rate of 61.3% 3.
The same seven were near-perfect on 88 essays written by eighth-graders in the United States 3. The error did not fall evenly across writers. It fell on the writers working in a second language.
One line in that paper describes a rubric better than it describes a detector. Rewriting the same TOEFL essays with richer vocabulary cut the average false-positive rate to 11.6% 3. The content did not change. The prose did, and the verdict followed it.
A detector is not a human rater, and 91 essays from one language background is a narrow base.
What transfers is the direction of the error rather than its size: a judgment keyed to how writing reads lands hardest on people writing in a second language. The candidate-side version of the same problem is why formal writers and ESL applicants get flagged as AI.
Defend the rubric for a moment. Clear thinking and clear writing are not independent, and work done with a model arrives as text, so an evaluation of that work is partly an evaluation of text. A correlation between the two columns is not by itself proof that anything went wrong.
So go back to the two columns and read down them looking for the candidate who broke the pattern. One is enough to show the AI line doing work the presentation line is not. If fifty rows produce none, the file holds one column entered twice.
Rewrite the second column before the next round opens.
Common questions
How high does the correlation have to be before it is a problem?
There is no published cutoff for how closely an AI-use score may track a writing-quality rating, and inventing one would repeat the rubric's own mistake: a number standing in for a measurement nobody made. Read the disagreements instead. Count the candidates who landed at least a band apart on the two scales, then look at what those submissions actually did. A file with several holds a scale that separates two things. A file with none, after fifty scored submissions, holds one column recorded twice, whatever the coefficient says. Zero is the reading that needs no threshold to interpret.
Could writing quality and AI judgment genuinely be the same thing?
Partly, and the overlap is real: work done with a model arrives as text, and people who think clearly often write clearly. The question a rubric has to answer is narrower. Does the AI line add anything you could not have predicted from the presentation rating alone? If a second reader can guess the AI score from the presentation score on nearly every submission, the line carries no information of its own. It spends reviewer time and creates exposure to buy nothing.
Who should do the second rating, and what do they see?
Someone who did not score the original round, working from the submissions with the old scores stripped out. Give them a written definition of presentation and nothing else: no AI-use anchors, no notes from the first reviewer, no names. The design matters because a rater's own earlier impression drifts into the score they give later, which is the leak the interview research measured. Record the new ratings before anyone opens the old file, so the comparison sits between two independent readings rather than one reading confirmed twice.
Should writing quality come off the rubric?
Only where the job does not need it. Plenty of roles are paid partly for prose, and scoring that openly, on its own line, with its own weight, is the honest version. The problem is one number wearing two labels. Split them, define each on its own terms, weight them deliberately, and expect the two to disagree about the same candidate sometimes. Disagreement is the evidence that both lines are doing work.
Is an internal rubric line regulated at all?
Under federal selection law, informality is no exemption. The Uniform Guidelines define a selection procedure as any measure, combination of measures, or procedure used as a basis for any employment decision, and say the definition reaches informal or casual interviews and unscored application forms. A hiring manager's private AI rating falls inside that definition once it moves a decision. Validation duties attach where adverse impact appears rather than automatically, and what that means for a specific process is a question for employment counsel.
References
- 1. Initial Evaluations in the Interview: Relationships with Subsequent Interviewer Evaluations and Employment Offers ✓ homepages.se.edu Supports the claim that an impression formed before the scored portion leaks into the score: 189 accounting students in structured mock interviews, an initial-impression correlation of .42 with the same interviewer's structured rating, falling to .25 and .24 when different interviewers supplied that rating.
- 2. The Structured Employment Interview: Narrative and Quantitative Review of the Research Literature ✓ doi.org Supports the payoff from written anchors, reported inside the review: Taylor and Small's meta-analysis of 19 past-behaviour interview studies found validity of .35 against .26 and interrater reliability of .77 against .73 for anchored rating scales, and the review's separate report that anchoring every scale point produced ratings more resistant to disability bias than anchoring only the endpoints.
- 3. GPT detectors are biased against non-native English writers ✓ pmc.ncbi.nlm.nih.gov Supports the parallel case that a verdict keyed to prose misfires on second-language writers: seven detectors over 91 human-written TOEFL essays gave an average false-positive rate of 61.3% with 97.8% flagged by at least one tool, near-perfect accuracy on 88 US eighth-grade essays, and a drop to 11.6% when the same essays were rewritten with richer vocabulary.
- 4. 42 U.S.C. 2000e-2(k) - Burden of proof in disparate impact cases ✓ uscode.house.gov Supports the statutory test named in the exposure section: the employer must demonstrate that a challenged practice with disparate impact is job related for the position in question and consistent with business necessity.
- 5. 29 CFR Part 1607 - Uniform Guidelines on Employee Selection Procedures (1978), sections 1607.16(Q) and 1607.3(A) ✓ govinfo.gov Supports the claim that an informal internal rating is still a selection procedure: the Guidelines define one as any measure, combination of measures, or procedure used as a basis for any employment decision, reaching informal or casual interviews and unscored application forms.
5 sources, numbered by first appearance. Every one was opened and checked against the claim it carries. How Olive sources claims
General guidance for hiring teams. What works at one company and one volume may not transfer to yours.
Olive assesses how a person works with AI. It does not detect AI-written documents, and it never produces a score, a ranking, or a match percentage for a person. Candidates read the same report the employer reads.