How Do You Get Managers Who Don't Use AI to Judge AI-Assisted Work?

What is the manager actually missing?

Not AI knowledge. An answer key. A manager who has run a marketing team for eight years already knows which statistic in a positioning brief is the one nobody opened, and which one sinks the launch. What they lack is a written standard for what a candidate should have done with an assistant, and the confidence that their own domain judgment is the right instrument for reading it.

The risk is not that they know too little about AI. It is that fluent, well-explained output is more persuasive than correct output. In a study of people making decisions alongside an AI system, explanations increased the chance a person accepted the AI's recommendation regardless of whether it was right, and did not improve the team's accuracy beyond what the AI's recommendations without explanations already produced ¹. That finding is about people trusting a model, but the mechanism, a confident explanation buying acceptance, is the same one running on a panel listening to a candidate narrate AI-assisted work.

Encouragingly, the evidence has a second half. In a survey of 319 knowledge workers describing 936 first-hand examples of AI-assisted work, higher confidence in the AI predicted less critical thinking, while higher confidence in one's own ability to do the task predicted more ². Domain confidence is the asset the manager already has. Build the round to use it rather than to replace it with a month of tool exposure.

What happens without a written standard is predictable. Managers fall back on the one thing they can pattern-match, which is prose style, and rejecting candidates for sounding like AI produces confident decisions with nothing recorded behind them.

Ask these three questions

Three, in this order, about one piece of work the candidate finished in the last month. What did you ask for first. Which specific claim did you check, and where. What did you throw out, and why. Each is a question about an act with a time and a place attached, which is why a manager can score it without ever having opened the tool.

1. "Say the first thing you typed to the assistant, the real message rather than the tidy one." Strong: the opening move states the problem, a constraint, or what would make an answer wrong. Weak: the opening move is the deliverable. "Write the Q3 forecast" hands over the framing before the framing exists. Scoring this needs no knowledge of prompting, because the question is whether the candidate had thought about the problem before anyone typed. 2. "Which specific claim in that output did you check, and where did you check it?" Strong: a claim, a location, and a result. "The churn figure: I opened the filing and it was annual, not quarterly, so the number was off by four." Weak: "I reviewed all of it" or "it looked right." A real check has a place, and the candidate can name it. 3. "What did you throw out, and on what grounds?" Strong: a direction refused, with the reason stated. Weak: everything kept and edited. Editing is tidying; refusing a framing is judgment. Be more worried, not less, when a candidate treats "I decided the model was the wrong tool and did that part by hand" as a confession, because how much AI someone used is not the measure.

Ask the same three, in the same order, of every candidate, and have each interviewer write the reason for the rating rather than the rating alone. Panels drift within two candidates otherwise, and getting a panel to judge AI use the same way is mostly a matter of everyone holding the same sheet of paper.

What does a strong answer sound like in each field?

The words change, the structure doesn't. Each family has one artifact an assistant produces fluently and one check only someone in the field would think to run. A strong answer names that check and says what it moved. A weak answer describes the output as good (well-structured, on-brand, clean), which is a judgment about the surface, not about whether anything underneath is true.

• Software engineering. Strong: generated code that ran clean and was still wrong, and how they found out: a test they wrote themselves, an input they tried, a dependency they read. Weak: "it compiled and the tests passed," when the tests were generated too. There is evidence behind this one. In a controlled study, participants with an AI code assistant wrote significantly less secure code than those without and were more likely to believe their code was secure, while the participants who trusted the assistant least and reworked their prompts produced fewer vulnerabilities ³.
• Marketing. Strong: a statistic in the brief traced back to the primary source, and what happened when the source turned out to say something narrower. Weak: "the copy was on-brand and I fixed the tone." The most on-message number in any AI-written brief is the one least likely to have been opened.
• Finance and accounting. Strong: a figure recomputed by hand or against the filing, and the recommendation that moved because of it. Weak: a model whose logic is described confidently but reconciles to nothing. Ask which cell they rebuilt, and what it changed.
• Legal operations. Strong: a citation pulled and read before it went into a draft. Weak: trusting a research tool because the vendor said it does not make things up. Legal research tools sold on that promise were measured producing hallucinated output between 17% and 33% of the time ⁴.

None of the four is a question about AI. Each is the check the manager would already run on a junior's work, asked at the point where an assistant makes it skippable. That is also why a list of tools on the resume tells you almost nothing: the tools are common to all four families and the tell is common to none of them.

Build the answer key in thirty minutes

Sit the manager down with one task their team shipped last quarter and ask them the same three questions about it. Their own answers are the key. Write each one as three lines (strong, adequate, weak) in the manager's words, and hand it to every interviewer on the loop before the first candidate. That is the whole training, and it takes half an hour.

Three lines each, anchored to acts. Strong: names a specific claim, a source, and what changed. Adequate: names a check but not what it moved. Weak: describes the output rather than any act performed on it. If two interviewers cannot score the same answer the same way, the wording is carrying too much, and a rubric two reviewers score the same comes down to anchoring every level to something the candidate did.

Write it before the round rather than after. An interview used to make a hiring decision is a selection procedure, and the Uniform Guidelines count even an informal or casual interview as one ⁵. Under the EEOC's guidance on employment tests and selection procedures, a procedure that screens out a protected group has to be shown job-related and consistent with business necessity ⁶. "The panel felt he wasn't AI-native" cannot be shown. A dated sheet with three anchors and a written reason per candidate can.

Keep three things off the sheet. Volume of AI use, because a candidate who judged the model was the wrong instrument for a step has demonstrated the thing being measured. Prompt vocabulary, which is a month of exposure rather than a year of judgment. And fluency describing AI, which is precisely what a coached candidate arrives with. What good AI use looks like is a set of acts, and acts are what a non-user can score.

What can't three questions tell you?

Nothing about follow-through. Every answer here is a report on a check, given by someone with an obvious interest in the report, and self-report about AI work is unreliable even when nobody is being interviewed. Sixteen experienced developers in a randomized trial took 19% longer to finish issues with AI tools allowed, and still believed afterwards that the tools had sped them up by about 20% ⁷.

Two cheap closures. Extend the round by fifteen minutes and hand the candidate a short piece of AI-assisted work with one error planted in it (your error, in your field, so no preparation reaches it), then watch what they do. Or move the whole thing to a work sample, where an assistant is available and the confident answer is wrong in a way only checking reveals. What a work sample should test now changed for exactly this reason.

Say in the invitation that the round covers how the candidate works with AI and that using it is expected rather than tolerated. An unstated rule gets guessed at, and the guessing measures interview coaching rather than judgment. If the manager still wants a shortcut, scoring an answer produced with AI is a shorter document than the AI-fluency curriculum they were about to sit through.

See how it works >