How to Run an AI Skills Assessment on the Team You Have

Why not just ask people to rate themselves?

Because self-rating and demonstrated ability turn out to be barely the same measurement. Among 288 teachers who took both a self-report and a knowledge-based test of AI literacy built on the same framework, correlations between the two ran from 0.07 to 0.24, and the profiles the researchers found included people who overrated themselves and people who underrated themselves ¹. Confidence is not a proxy you can correct for.

A Taiwanese teacher sample is not your finance team, and one study is not a law. What travels is narrower and better evidenced: people are unreliable narrators of their own performance with these tools, sometimes about the direction and not just the size. Sixteen experienced developers in a randomized trial forecast that AI tooling would cut their task time by 24%, estimated afterwards that it had cut it by 20%, and were measured 19% slower ². Sixteen people working on codebases they had known for years is a small, specific setting. It is still the cleanest demonstration available that the belief and the measurement can disagree about the sign.

Even a scrupulously honest answer misleads you here. The person who accepts the first draft, ships it and hears no complaints has a genuinely good experience of the tool and no evidence that anything went wrong. Asking that person to rate their AI skill collects an accurate report of a good experience.

So collect a record, and make it cheap enough to collect again next year.

What should the exercise ask people to do?

One piece of work the team genuinely produces, done in under an hour with an assistant available and nothing about the assistant prescribed. A quarterly client summary, a vendor comparison, a bug triage, a policy exception memo. The brief carries a real constraint, a real deadline, and at least one claim that would be expensive to get wrong.

Grounding matters more than difficulty. Researchers building an AI literacy assessment for a US Navy robotics training program reported that a scenario task simulating AI use on the job outperformed the tests they had adopted from earlier research or written themselves, though they published no sample size or effect size behind that comparison ³. The general version is unsurprising and still widely ignored: a question about what a language model is tells you nothing about whether someone will check its output at four in the afternoon.

Three things belong in every brief.

• A constraint the assistant cannot see. An internal policy, a client preference, a number that lives only in your systems. This is what separates specification from transcription.
• One load-bearing claim. Something the finished piece rests on that a careful person would go and confirm.
• A plausible trap. A sub-task adjacent to the ones these tools handle well, where the confident answer is wrong.

Keep the brief identical inside a function. Across functions, change the case rather than the format, so the underwriter and the analyst are read on the same four behaviors in their own occupational language. If it is not yet settled which functions need this at all, work out which roles actually need AI skills before building the exercise.

Which behaviors are worth reading off the record?

Four, and they show up in the trail rather than in the finished document. Whether the person wrote down what they wanted before generating anything. Whether they opened a source when a claim carried the decision. Whether they rejected something the assistant produced and said why. Whether they checked an output against something outside the conversation.

Each is a yes-or-no read with an excerpt attached. No scale, no weighting, no total. A reviewer marking the first is looking for a sentence written before the first prompt: the constraint, the audience, what a wrong answer would look like. A reviewer marking the fourth is looking for a moment where the person left the chat window and came back with something the chat window could not have given them.

What stays out of the read matters as much. Prose quality, prompt syntax, tool trivia, how fast the work came in, and how much of the assistant's output survived are all observable and none of them are the skill. Someone who used the assistant for nine tenths of a draft and caught its one wrong claim did better work than someone who typed every word and left the same claim standing. If the instinct is to put a stopwatch on this, read the case on speed against judgment first.

The trail is the whole instrument, which means the exercise has to produce one. Ask for the working record alongside the deliverable: the transcript, the notes, the discarded version. People need to be told that at the start, in writing, along with what happens to it afterwards.

How do you run it without turning it into a rating?

Say what the results will and will not be used for before anyone opens the brief, then hold to it. A capability read is not a performance rating, and the fastest route to honest work is a written promise that no one's answer lands in a compensation file. It carries more weight signed by whoever owns the budget than by whoever is running the exercise.

That promise costs less than it sounds. What a training program needs from this is a picture of the function: how many people framed before generating, how many opened a source, where the practice already sits. Individual records go back to the individual. The aggregate goes to whoever is designing the training.

Three guards worth writing down.

• Announce the format, the length and the four behaviors a week out. Surprise adds nothing here and costs trust.
• Offer the accommodation before anyone asks. Extra time, a different input method, a written rather than spoken commentary.
• Let people decline. A refusal is feedback on the program, and coercing participation contaminates every record collected.

Put two readers on a sample. Have them mark the same ten records independently and compare before either sees the other's marks; where they disagree, the rubric is ambiguous, and fixing that now is cheaper than defending it later. Then fix the format so next year is comparable: same length, same four behaviors, same kind of artifact, a fresh case.

The bar being cleared is low. In SHRM's benchmarking survey, 12% of member organizations reported using work sample interviews for executive candidates, 11% for middle management and 9% for individual contributors, against 79%, 78% and 76% for in-person interviews, all self-reported and collected in 2021 ⁴. Those are hiring figures rather than internal ones, and they are the closest available read on how common a graded work sample is anywhere. An hour of real work with a written rubric behind it is rarer than it sounds. Then use it: build the upskilling program around what the records showed, budget for the reading as carefully as for the courses because training and verification are two different line items, and if the real question is whether one person can move into an AI-heavy job, internal mobility is a separate decision with separate rules.

See what gets scored >