What Separates Heavy AI Use From Good AI Use

Why does the usage dashboard say nothing about quality?

Because it counts actions, and quality lives in the judgment around them. Microsoft's own admin report calls someone an active Copilot user when they take one intentional action inside the chosen window, and a single submitted prompt in twenty-eight days qualifies ¹. That number moves when a person presses a button. It does not move when someone reads an output and decides it is wrong.

Self-report is not a fallback either. In a randomized trial, 16 experienced developers working on repositories they had known for years forecast that AI tools would cut completion time by 24%, still believed afterwards that it had cut it by 20%, and were measured 19% slower across 246 real tasks ². Sixteen developers in one setting is not a general productivity finding, and the authors do not offer it as one. What travels is the gap: people were wrong about the direction, not only the size.

Asking the team to rate its own AI skill has the same problem in a different coat. When 288 teachers took both a self-rating and a knowledge test built on the same four-part framework, correlations between what they claimed and what they demonstrated ran between 0.07 and 0.24 ³. That is a study of teachers rather than employees, and it does not show that people flatter themselves; the profiles included underestimators. It shows the two instruments measure different things, which is enough to stop using one as a proxy for the other.

The three easiest signals to collect, then, are seat activity, self-reported speedup and self-rated skill. They are also the three that carry the least.

What four traces does good AI work leave?

Four, and each one is visible in the artifact rather than in a tool. A frame written before anything was generated. A source actually opened for the claim the decision rested on. Something the model produced that was thrown away, with a reason attached. A check run against something outside the conversation: a query, a test, a person who would know.

• A frame written first. Criteria, constraints, what a good answer would have to contain. It can be four lines in a scratch file. Its absence looks like a deliverable that answers a question nobody quite asked.
• One source opened. Not every claim. One: the claim the decision rested on. The trace is a link, a page number, a screenshot, or an account of what the source actually said next to what the model said it said.
• Something rejected. A draft, a paragraph, an approach, a whole first attempt, with a sentence about why it went. A body of work with no rejections in it was accepted rather than edited.
• A check against the outside. A number confirmed against the system that holds the number. The cited case read. The claim put to someone who would know if it were false.

None of the four requires anyone to use AI less. All of them are compatible with running every step through a model, and none of them is produced by running every step through a model. That is what makes them worth asking for: the answer cannot be inferred from how much the tool was used, in either direction.

Run the ten-minute review on two recent deliverables

Pick two things that shipped in the last month, sit down with whoever wrote them, and ask four questions about each. It takes about ten minutes per deliverable and needs no tooling, no dashboard and no new policy. The questions are about the work rather than about AI, which is what keeps the conversation from turning into a defence.

Ask what the criteria were before drafting started. Ask which single claim the decision rested on, and what was opened to check it. Ask what got thrown away, and why. Ask what would have had to be true for the conclusion to be wrong, and how that was tested.

A good answer is specific and slightly boring: the number came out of the model's summary, it was checked against the export, it was off by a rounding rule, here is the corrected version. An empty answer generalises. It talks about being careful, about always double-checking, about the tool being useful for first drafts. Careful is a disposition. A check is an event with a time on it. Hearing the difference between the two takes practice on the reviewer's side as well, which is the case for managers putting in hands-on hours of their own.

Run this on the strongest person on the team first, so the pattern being looked for is visible before it is being judged. These four questions are also what a good interviewer is reaching for when they try to see what good AI use looks like in a candidate, and the same distinction sits under the argument about measuring speed or measuring judgment.

What about the person who barely uses it?

They might be the strongest signal on the team, or a genuine holdout, and the same ten minutes tell you which. The distinction is whether the person can name the step where the model was the wrong instrument and say what it would have got wrong. That is an account. Discomfort with an unfamiliar tool produces no account of any particular step, only a preference.

Low use among strong performers is what the evidence predicts. In a rollout to 5,179 customer support agents, a conversational assistant raised issues resolved per hour by 14% on average, with a 34% improvement for novice and low-skilled agents and almost nothing for experienced, highly skilled ones ⁴. One firm, one occupation, one assistant trained on that firm's own transcripts, and the outcome measured was issues resolved rather than quality of judgment. The direction still holds: the people with least to gain from a given assistant are sometimes the ones who know most.

The costlier failure runs the other way. In the field experiment with Boston Consulting Group consultants, on one task chosen to sit outside what the model could do, 84.5% of the control group reached the correct answer against 60% and 70% in the two AI conditions ⁵. Consultants could not tell which side of the line the task sat on, and the group given a prompt-engineering overview did worse than the group given none. One task, one 2023 experiment, and the frontier has moved with every model release since.

An artifact shows both of those patterns, and a counter shows neither. No dashboard can show you the person who used the model on a step where it should not have been used. A ten-minute conversation about one deliverable can, and it works the same way on the person who used it least.

See the benchmarks >