A Manager Who Has Never Used AI Cannot Review AI Work

What does a manager actually need to be able to do?

Recognize three failure shapes on sight, and ask for the trail behind a claim without the question landing as an accusation. That is the job. The three shapes are the fabricated citation that resolves to nothing, the plausible number that is wrong in the second decimal or the second order of magnitude, and the fluent answer to a question nobody actually asked.

The first is the easiest to catch and the least common in practice, because it fails the moment anyone opens the link. The second is the expensive one: a figure with the right units, the right rough size and the wrong value, sitting in a paragraph that reads like it was checked. The third is the hardest of all, because nothing in it is false. The brief asked what the renewal risk was and the deliverable explains, very well, what renewal risk is.

None of the three is visible from prose quality. All three are visible to someone who has produced them and noticed. The whole argument for hands-on hours sits there: the failure shapes are learned the way any other professional pattern is learned, by having been taken in and then working out how.

Why doesn't a certificate cover it?

Because a certificate exam tests recognition of vocabulary under time pressure, and the two guides quoted here set out no performance task. Google Cloud states plainly that its Generative AI Leader certification is for anyone in any job role, with or without hands-on technical experience ¹. That is not a criticism of the exam. It describes what the exam is for, and it is not this.

The technical-sounding ones say the same thing in more detail. The AWS Certified AI Practitioner exam guide describes a target candidate with up to six months of exposure to AI on AWS who uses but does not necessarily build AI solutions, and lists developing or coding models as explicitly out of scope ². The question types are multiple choice, multiple response, ordering, matching and case study. A pass evidences familiarity with concepts under time, which is a real thing to know and a different thing from judgment in use.

Even the regulator that pushed hardest declined to make a credential the answer. In the EU, the European Commission's own questions and answers on the AI Act's literacy obligation, which has applied since 2 February 2025, state that Article 4 does not entail an obligation to measure the knowledge of AI of employees, that there is no need for a certificate, and that no mandatory trainings are imposed ³. The Q&A is guidance and not statute. Its wording here is the August 2026 version of a page revised once already, so any compliance conclusion belongs with counsel. It cuts both ways too: no vendor's badge confers compliance, and the Commission says so about its own repository of practices.

So fund a course if the vocabulary is genuinely missing. Do not report it upward as capability, and do not accept it in place of the hours.

Give each manager two hours a week on their own work

Two hours a week, on work they were going to do anyway, for about a month. Not a sandbox, not a prompt library, not a workshop: their own real deliverables, where they already know what a right answer looks like and will therefore notice a wrong one. Nobody measured that dose; it is a judgment, and it is small enough that no calendar argument survives contact with it.

Three exercises fill the hours better than any curriculum:

• Ask for sources on a claim you already know cold, then open two of them. The gap between what the source says and what the summary said it says is the lesson, and it only lands the first time on a topic where you can see the gap.
• Give it something you know sits outside what it can do. Watch it answer anyway, at full confidence, in your own subject. Run this one first. A confident wrong answer inside your own expertise is the clearest version of what these hours are for.
• Take one output you nearly shipped and write down what would have gone wrong. Not that it was wrong. What it would have cost, and at which meeting someone would have found out.

Keep a running file of the things it got wrong in your own work. That file is the curriculum, it is specific to the domain, and it is the only artifact from this worth showing anyone else. Two managers comparing their files learn more in twenty minutes than either learns from a vendor deck.

How should a manager ask about the trail?

Use the vocabulary of the work itself. What did you check this against. What did the first version say that this one does not. Which part are you least sure about. All three are questions a decent manager asked long before AI existed, which is exactly why none of them reads as an accusation, and why all three are answerable by someone who did the work properly.

The question that does read as an accusation is whether AI was used, and it is also useless, because style is not evidence of anything. Asked to tell GPT-3 text from human writing across stories, news and recipes, non-expert evaluators performed at random chance, and three quick training methods lifted them only to about 55%, inconsistently across the three domains ⁴. The study is from 2021, the judges were crowdworkers and not hiring managers, and the passages were short. Models have improved since, which makes unaided guessing worse rather than better, though that is an inference and not a measurement in the paper.

Guessing has to give way to asking, which makes this a management problem rather than a detection one. It is the same underlying issue as managers who don't use AI having to judge AI-assisted work in interviews, and the same reason to stop rejecting people for sounding like AI.

Once a manager can ask the three questions and read the answers, the review structure is straightforward: two recent deliverables, ten minutes each, looking for the traces good AI work leaves in the artifact. Without the hours, the same review runs and finds nothing, because the person running it cannot tell a specific answer from a confident one.

See the benchmarks >