Four Questions for a Manager Whose Team Ships With AI

Ask what the team automated and what they refused

Start with the pair, never the first half alone. Asked what the team automated, most candidates have a rehearsed answer waiting. Asked what they refused to automate, and why, they have to name a decision. A manager who can give both halves, with a reason attached to the refusal, drew that line themselves.

The distinction has a name in the labor data. Employment of 22-to-25-year-olds in the two most AI-exposed occupational quintiles fell about 11% between November 2022 and June 2026 while the same age group in the three least-exposed quintiles grew about 10%, and the declines concentrate in occupations where AI substitutes for human tasks rather than complementing workers ¹. Those measures come from occupation-level usage patterns, not from how any single employer deploys anything, and the authors describe the work as descriptive rather than causal. It says nothing about your company.

What it does give you is the right axis for the question. A manager who automated the parts that were substituting for judgment has made a different bet from one who automated the parts that were eating time before the judgment started, and the two produce very different teams within a year. Ask which of those they did.

The follow-up worth asking every time: what did that refusal cost. A manager who refused to automate something and cannot say what it cost has probably refused nothing. If the role is mostly about supervising automated work, assessing candidates who manage agents is a narrower version of this question.

How did they find out somebody was over-relying on a model?

Listen for the moment they noticed, because this is the question with the fewest good fake answers. Over-reliance is invisible in the artifact and visible only in what happens next: an explanation that falls apart on the second question, a figure nobody can trace, a fix that takes longer than the draft saved. A manager who has met it once can describe how they noticed, and what they changed the following week.

The useful research here is about who the tool helps. In a randomized trial with 640 Kenyan small-business owners given a GPT-4 assistant over messaging, there was no detectable average effect on business performance, but high performers at baseline gained just over 15% while low performers did about 8% worse, and the gap came from which advice owners chose to act on rather than from differences in the advice itself ². Kenyan small businesses in 2023 and 2024 are not a knowledge-work team, the average effect could not be distinguished from zero, and this cuts against rather than refutes the studies where AI lifted weaker performers most.

The transferable part is the mechanism a manager is being hired to handle. The same assistant, given to two people on the same team, can raise one and sink the other, and the difference is whether the person can judge what came back. A manager who understands that will describe an intervention aimed at judgment: pairing, a required check, a rule about tracing figures.

Weak answers here reach for tooling or for a ban. Both are the same move, which is turning a coaching problem into a policy. What to listen for instead is a manager who names the person, the moment and the change, without turning it into a story about somebody being caught out. Structuring the first ninety days of an AI-heavy hire is what a good answer to this question usually looks like written down.

How do they judge a junior when every draft arrives clean?

They read the process. The rough edges in a draft used to show where somebody was struggling, and a clean submission said they had it, and that reading stopped working once first drafts began arriving polished regardless of understanding. Strong answers name what replaced it: questions asked, checks run, what got escalated and when.

Do not accept an answer built on being able to tell. In an ACL 2021 study, non-expert evaluators asked to distinguish GPT-3 text from human writing across stories, news articles and recipes performed at random chance, and three quick training methods lifted accuracy only to about 55%, inconsistently across the three domains ³. That was 2021-era output judged by crowdworkers rather than managers reading work in their own field, so it is not a measurement of your reviewers. It is enough to disqualify a manager whose whole development plan rests on spotting the difference.

The answers that hold up move the coaching upstream of the artifact. A manager who asks a junior to bring the question they started from, or who reviews the checks a junior ran, or who sets a weekly session where the junior explains one decision in detail, has rebuilt a signal that a polished draft cannot fake. So has one who simply changed what they assign: smaller pieces, tighter feedback, more work where the answer is verifiable.

This is also where the manager's own habits show. Hiring managers who do not use these tools themselves tend to grade the artifact because it is the only thing they know how to read, and a candidate for a management role should be able to say what they read instead.

What separates a decision from a policy statement?

A cost. A policy statement describes what should happen and carries no evidence anybody tested it. A decision names what was chosen, what was given up, and how it turned out. Ask what their review process caught last quarter and the difference appears at once: either there is an example with a date attached, or there is a description of a process nobody has run.

Be skeptical of productivity claims, because the honest aggregate numbers are small. A nationally representative US survey asked people how many extra hours the previous week's work would have taken without generative AI: mean self-reported savings were 5.4% of work hours among people who use it at work, which works out to 1.4% of total work hours across all workers ⁴. That is self-reported time saved rather than measured output, from late-2024 US data, and the authors call it a rough estimate. A manager candidate claiming their team got 40% faster is quoting a feeling.

What a strong answer sounds like, in one sentence each. The team automated first-draft release notes and refused to automate incident write-ups, because a write-up nobody thought through is worse than none. Somebody was accepting generated figures without tracing them, which showed up in a client meeting, so tracing became a required step and reviews got ten minutes longer. Juniors now bring the question they started from, because the draft stopped telling anyone anything.

When a manager's answer is that the capability has to come from outside, hiring for AI skills against training the team you have is the decision sitting behind the requisition.

Ask the same four questions of every manager candidate for the role and write the answers against the same four headings. Without that, the debrief compares how well each of them talks. It is the same discipline as giving each panel seat one AI question and grading each level against its own bar.

See the benchmarks >