Four Levels of AI Skill, Written So Two Managers Agree

Why the usual four levels stop separating people

Because they grade attitude and tool count rather than behavior. Aware, applied, integrated, strategic: none of those names a thing a person could be seen failing to do, so the placement falls back on how often someone talks about AI and how many products they have open. Everyone using an assistant daily lands in the middle, the distribution collapses, and the scale stops working exactly where you needed it.

The frameworks people borrow from are usually honest about this, and the borrowing is where it goes wrong. The AI Fluency Framework, in its current version, sets three named sub-competencies under each of its four competencies, twelve in all, against three modalities of human-AI interaction ¹. It is a taxonomy: no levels, no scoring anchors, no proficiency thresholds published with it. Turning a taxonomy into a five-point scale by adding adverbs produces something that looks like a rubric and behaves like a survey.

A subtler failure sits underneath. A ladder written in adjectives gets applied to the person, so it absorbs everything a manager already believes about them. A better adjective will not fix that. Move the unit of judgment from the person to a task, and the question becomes what happened in this piece of work.

Accept one consequence up front: a behavioral ladder places fewer people confidently than an attitude ladder does, because half your team has never been observed doing the thing. That is not a defect in the ladder. It is the information the old one was hiding.

What each rung looks like under a real task

Take one piece of work someone actually did with an assistant and ask four questions in order. Did they hand it over and accept what came back. Did they supply context and edit the result. Did they check a claim against something outside the model and catch something. Did they decide part of the task should not be delegated at all. The highest question answered yes is the rung.

Spelled out, with the failure that defines each boundary:

• One, accepts. The task goes over intact, the output comes back, it ships roughly as written. Fails at the first factual error nobody looked for.
• Two, edits. Context supplied up front, output reworked before it goes out. Better prose, same exposure: editing improves what the reader sees without testing whether it is true.
• Three, checks. At least one claim gets traced to something outside the conversation, and at some point that check catches an error. This is the first rung with evidence attached rather than effort.
• Four, refuses. Some part of the work is kept back on purpose, with a reason: the judgment belongs to a person, the source material cannot go into a prompt, the model has no way to know what matters here.

Two details make this hold up in practice. The rungs are cumulative, so a person at four still does the checking at three. And the boundary between two and three is the one that matters most and gets blurred most, because heavy editing feels like verification and is not. If nobody left the conversation to confirm anything, the work is at rung two however polished it reads.

Check that two managers place the same person on the same rung

Run the ladder twice on the same work before anyone's level goes in a system. Give two managers the same three pieces of output, without names, and ask each to place them. Where the placements differ, the disagreement is almost always about rung three, and the fix is to write down what counts as a check in your work rather than adding another adjective to the definition.

The research on this is thinner than anyone would like, and it points one way. A team building an AI literacy assessment for a US Navy robotics programme reported that a scenario task simulating AI use on the job outperformed the knowledge tests they had adopted from prior work or written themselves, and argued that prevailing assessments overweight technical foundations against practical judgment ². One programme, one context, a preprint with no effect size in its abstract: it supports a design choice rather than a benchmark. The design choice is to anchor the rung to a task rather than to a description.

What calibration usually surfaces is not a scoring disagreement but a missing definition. One manager counts "asked a colleague" as a check; another requires a document. Both are defensible and the pair has to pick one, in writing, per role. That single sentence does more for consistency than the rest of the framework combined, and it is the piece most competency models never write down.

A rubric two reviewers apply the same way is its own problem with its own mechanics, worked through in how to write a rubric for AI use that two reviewers score the same way.

Should a self-rating ever set the level?

Use it to find out what people think, never to place them. Self-report and demonstrated ability come apart in the measured cases, and they come apart in both directions, so a self-rating is not even reliably inflated. It is a different quantity, useful for spotting where confidence and evidence disagree, and unusable as the input to a level that affects pay, promotion or a hiring decision.

The cleanest evidence sits outside the workplace. Among 288 Taiwanese teachers who took both a self-report and a knowledge-based test of AI literacy built on the same framework, correlations between the objective and self-reported factors ran from 0.07 to 0.24, and the analysis found six distinct profiles including both overestimators and underestimators ³. Teachers in Taiwan are not your engineering team, and a weak correlation means the two instruments measure different things, which is not the same as saying people flatter themselves. Both readings support the same operational rule: the two numbers are not interchangeable.

Belief about one's own AI use is unreliable even among experts working in their own domain. In a randomized trial, sixteen experienced open-source developers working on repositories they had maintained for years forecast that AI tools would cut their completion time by 24%, estimated afterwards that it had cut it by 20%, and were measured 19% slower ⁴. Small sample, one setting, and the magnitude does not transfer. What transfers is that the estimate was wrong in direction, not only in size.

So run the survey if it is useful for planning, and keep it out of the ladder. Where the two disagree is where to look first, and what the minimum every employee needs looks like sets the floor the whole ladder sits on. Whether AI should be a competency of its own at all is a prior question, handled in should AI be its own competency or folded into the ones you have.

See the benchmarks >