Trade the One-to-Five AI Rating for One Sentence

What does a one-to-five AI rating measure?

A self-rating measures self-assessment, and self-assessment tracks demonstrated skill loosely at best. The research that matters here hands the same person both instruments, a self-report scale and a knowledge test built on one framework, then asks whether the two agree. They mostly do not.

So the digit on the form carries information about a candidate's read of themselves, and very little about what they can do with a model on a Tuesday.

In February 2025, researchers gave 288 Taiwanese K-12 teachers, close to half of them still in training, two AI-literacy instruments built on a single Concept, Use, Evaluate and Ethics framework: a self-report questionnaire and a knowledge-based test. Correlations between the three objective factors and the four self-reported factors ran from 0.07 to 0.24 ¹.

A correlation that low means the two instruments are not measuring the same thing. It is not evidence that people inflate. The same analysis sorted the teachers into six profiles, and underestimating themselves was one of them.

Those were teachers, in one country, recruited off mailing lists with a small gift card. None of them was applying for a job, and an applicant has reasons a survey respondent does not.

But a gap between belief and measurement also shows up on a person's own work.

On repositories they had known for about five years, 16 experienced open-source developers worked 246 real tasks between February and June 2025. They forecast that allowing AI tools would cut completion time by 24%, and afterwards still believed the tools had saved them 20%.

Measured, the tools cost 19% more time ².

A single setting cannot produce a productivity result about anybody else. The direction of the error carries further than its size. Belief and measurement pointed opposite ways, and the people holding the belief were experts on their own code.

A one-to-five box asks for the number that belief produces. It never asks for the measurement.

When does the one-to-five box become a selection procedure?

The one-to-five box becomes a selection procedure the moment a decision rests on it, which can be the first time anyone sorts or filters by it, and that moment usually arrives without a meeting. The U.S. Uniform Guidelines on Employee Selection Procedures, adopted in 1978, define a selection procedure as any measure or procedure used as a basis for any employment decision. The definition runs from paper tests through informal interviews all the way to unscored application forms ³.

Sitting inside that definition is not the same as owing a validation study. The Guidelines put the validation burden on procedures that show adverse impact.

What the definition does settle is that the word unscored buys nothing. A field nobody scores still counts once a decision rests on it, which is the entire design of a sort-by column.

Nothing in the definition is specific to a rating, so the same reading covers an informal AI exercise.

Noise on its own would only cost accuracy.

The expensive case is noise with a direction that differs by group. A field carrying a group-patterned error becomes a group-patterned filter the first time somebody sorts on it, and a sort leaves no record of what it selected for.

One nearby measurement is of willingness to apply, a different question from rating a skill. In a 2022 online experiment with 10,468 working-age participants, the Behavioural Insights Team measured what share of a posting's requirements a person had to meet before saying they would apply: women needed 55.7%, men 52.1% ⁴.

The gap is small. It appeared only among the less qualified half of the sample, and it is roughly a tenth of the sixty-versus-a-hundred folklore it replaced. The authors trace what they found to less qualified men rating their own fit higher, which is a self-perception result and not a finding about who could do the job.

None of the research cited here measures group differences in one-to-five AI self-ratings on real application forms. The honest position is that the direction is plausible but not shown, and an unmeasured field is a poor place to rest a decision.

Replace the scale with one box you can grade

Ask for three facts instead of a digit: the last real task done with an AI tool, which tool, and what got checked before the output left the candidate's hands. Cap the box at sixty words and say on the form that a reader checks it for those three facts.

A short factual answer is gradeable by two people who never speak to each other. A number is not.

What separates a strong answer from a thin one is whether the check has an object. "I always verify the output" names nothing. "Reconciled the three segment figures against the 10-K before the deck went to the client" names a document, and a reader can ask which figure moved.

The box, close to verbatim:

• The last task you did with an AI tool, in one sentence.
• Which tool, and roughly when.
• What you checked before the result went anywhere, and what you checked it against.

Grade presence, not prose. A named task, a named tool and a named check is three yeses, and two readers filling those three columns separately should land in the same place.

A five never gave anyone that.

Run the rest of the form through a test as well. The working rule for which fields on the application still produce signal is whether a good answer needs something only that applicant knows.

An attestation is not the fallback either.

In a 2021 experiment, 315 participants took a knowledge test unproctored online, then sat a parallel form of it supervised in a lab. Self-report indicators, including an honesty scale and an overclaiming scale, did not predict who had inflated the unproctored score. The test data and the response-timing data did ⁵.

That study inferred cheating for a gift-card draw, which is neither a skill rating nor a job offer, and the authors are explicit that their behavioural signals are indicators rather than proof. The pattern is still the one to expect. What somebody said about a disposition and what showed up in the record came apart, and the record was the half carrying information.

The box does its real work later, at the screen. That is where verifying an AI-proficient line without taking their word for it can start from the task the candidate named, and a sentence on a form becomes something to test.

Write down what the self-rating is not allowed to do

When the self-rating field stays, write its limits into the screening rubric, where decisions get made: the rating is intake context, no stage advances or holds on it, and the number is never a tiebreak. The reasons it stays are ordinary. The applicant tracking system already has it, a report already counts it, and removing a question costs more meetings than keeping it.

A rubric written after the first hundred applications is a description of what already happened. Setting the screening criteria before the first application opens is the version that constrains anything.

One line about the self-rating belongs in that document rather than in somebody's memory.

What the field must not become is an input to something that orders the pipeline. Feed the number to a tool, or hand the job to a vendor that infers AI capability from resume text, and the output carries every problem above plus whatever the tool adds on its own.

The box can stay on page two of the application. Change what it asks for, and the same spot on the form buys a sentence somebody can check, for a little more of a candidate's time.

See a sample report >