Is AI Training Enough, or Do You Have to Check the Work?

Does finishing AI training mean someone can do the work?

No. Completion is an attendance record, and self-report is not much better. In a randomized trial, sixteen experienced developers working real issues in their own repositories took 19% longer with AI tools allowed, and still believed afterwards that the tools had made them 20% faster ². If practitioners misread their own throughput by that margin, a post-course survey is not evidence of anything.

Completion does tell you three real things, none of which is capability: a person had access to the material, they spent the time, and they could answer questions about the material immediately afterwards. Recall decays. Judgment was never measured in the first place.

The metrics most rollouts report make the gap harder to see rather than easier:

• Seat time and completion rate. An attendance figure wearing a percentage sign.
• Satisfaction scores. They track how a session felt, and the sessions that feel best are frequently the ones that asked least of the room.
• License usage. Volume of use is not a virtue. Someone who ran forty prompts and shipped every answer untouched used AI more than a colleague who ran four and threw out three.
• Self-reported confidence. The developers above are the clearest available warning, and they were measuring their own work in codebases they knew well ².

Keep all four on the dashboard, by all means. None of them answers the question being asked in the budget meeting, which is whether the work got better. Whether to train the capability or hire it is a separate decision with its own arithmetic; this article assumes the training already happened and the invoice is already paid.

Why does transfer differ so much between roles?

Because some jobs use AI as procedure and some use it as judgment. A support agent working from a suggested reply is doing procedure, and the gain arrives fast: access to a conversational assistant raised issues resolved per hour by 14% on average, and by 34% among novice and low-skilled agents, with minimal effect on the most experienced ¹. Judgment-heavy work moves the other way.

The METR participants were maintainers working on repositories they knew intimately, where most of the value sits in knowing what not to change. AI access made them slower while feeling faster ². One curriculum, delivered to both populations, would produce a 34% throughput gain in one and a 19% slowdown in the other, and a single completion number would show the same green bar for both.

So the transfer rate is a property of the role, not of the training vendor, and no case study from a different company answers it for yours. Before deciding whether a check is worth running, write down what AI is actually doing in each job you trained: drafting, summarizing, interpreting, or committing the firm to a position. What the AI does in the role is the input to that decision; a course catalogue is not.

What does an after-training check have to observe?

Five acts, all of them visible while the work happens: how the problem was framed before anything was generated, what evidence was demanded for the claim the answer rests on, what the person kept rather than handed over, what they refused from the model and why, and what they tested against something outside the conversation. A check that observes none of these is grading output, which is the part that got cheap.

A survey of 319 knowledge workers describing 936 real tasks found the effort does not disappear when a model arrives. It moves: from gathering information toward verifying it, from problem-solving toward integrating a partial response, and from execution toward stewarding a task the model is doing part of ³. Those three destinations are what an after-training check is for.

• Framing before generating. The first move names what is being decided and what would make an answer wrong. The failure shape is asking for the deliverable in the first message and letting the framing arrive with it.
• Evidence for the claim that matters. Not a general request for citations, but a source demanded for one load-bearing claim, and opened. The failure shape is confident assertions flowing into the work as facts.
• The delegation boundary. Something of consequence was kept and done by hand, deliberately. The failure shape is a record in which the person did nothing themselves.
• A refusal with a reason. A direction was rejected on substance, and the reason is stated. Accepting everything and tidying the edges is the failure shape, and it is the most common one.
• A test against the world. A figure recomputed, a script run, a page opened, a customer called. Describing a check is not running one.

The last act is field-specific and cannot be borrowed from a generic rubric: an analyst ties a number back to the filing, an engineer runs the code, a paralegal pulls the case. The same survey found that higher confidence in the tool predicted less critical thinking, while higher confidence in one's own ability predicted more ³, so the most enthusiastic graduate of your program is not automatically the one doing the checking. A rubric for AI-assisted work has to score the act rather than the polish, which is the same reason a finished-looking take-home tells you so little.

Build the check as a work sample, not a quiz

A quiz measures recall of what the course said. A work sample measures the thing you paid for. Give one task drawn from the team's own queue, with the assistant available, forty-five to sixty minutes, and a rubric written before anyone sits down. Work samples earn their standing precisely because the person performs the work instead of describing it ⁴.

Five decisions make or break the exercise:

1. Use real material with a wrong answer in it. The task should contain something a model will assert fluently and incorrectly: a figure that does not tie out, a source that says less than it appears to, a requirement that contradicts another. If nothing can go wrong, nothing can be observed. 2. Leave the assistant switched on. A check that bans AI measures a job nobody is doing any more. The question is not whether the person can work without it. 3. Write the rubric first, in observable terms. "Demanded a source for the revenue claim and opened it" is scorable. "Shows good judgment" is a memory of how the session felt. 4. Capture the working, not just the deliverable. Two people can hand in the same document having done entirely different work. The intermediate artifacts and the exchange with the assistant are the evidence; the deliverable alone is not. 5. Have a second person score a sample of them. Agreement between two scorers on the same session is the cheapest reliability evidence you will ever collect, and disagreement usually means the rubric is describing a feeling.

If you would rather adopt an existing frame than write one, the four Ds rubric is the most widely used starting point and maps onto the five acts above without much translation. Either way, run it once as a diagnostic before it decides anything. Piloting the assessment before it becomes a gate is what separates a measurement from a grievance.

What changes if the check decides anything?

It becomes a selection procedure. The moment a result affects who is promoted, reassigned, or moved onto an AI-heavy team, it sits under the same expectations as a hiring test: administered consistently, related to the job, and explainable if the pattern of who passes turns out to be uneven ⁵. That is an argument for writing the rubric down, not an argument for skipping the check.

Four things follow, and none of them is expensive:

• Same task, same time, same rubric, same accommodations. Consistency is most of defensibility, and it costs nothing at the point of design.
• Keep the evidence. The session, the rubric, and the scorer's notes. A result you cannot reconstruct in a year is a result you cannot defend in a year.
• Separate diagnosis from consequence. A first run that names which of the five acts is missing is a training input. The same instrument attached to a promotion decision is a test, and it should be announced as one before anyone takes it.
• Ask any vendor for the same evidence you would have to produce. Validity for the job, adverse-impact analysis, and what the tool actually observes. The questions worth putting to a vendor are the ones you will be asked yourself.

The close is simple. Training buys exposure; the check is what tells you whether exposure became capability, and in which roles it did not. Run the check where the job's AI use is judgment, skip it where the throughput number already answers the question, and write the standard down before the first person sits it rather than after the first argument about a result. If the bar itself is the open question, setting a defensible AI proficiency bar is the piece to settle first.

See what gets scored >