How Do You Assess a Candidate Who Mostly Supervises AI Agents?

What does an agent supervisor actually do all day?

Four acts, and none of them is production. You write the brief and the limits the agent runs under. You read what comes back against something other than itself. You reject specific parts on stated grounds. You decide what never ships without a second human. Assess those four, in that order, because they are what the person does on an ordinary Tuesday.

Most exercises available today test a fifth thing: whether the candidate can plan, implement, debug and architect an agent. That is the builder's job. If your hire will never open the agent's code, an architecture exercise tells you about someone else, and the person who would have been excellent at the actual work fails it for the wrong reason. Write down what the agents in this role already do, step by step, before you write the exercise. Working out what AI actually does in the role is the same first move you would make before writing the job post.

The volume is real enough to hire against. In Stack Overflow's 2025 developer survey, 14.1% of respondents said they use AI agents at work daily and a further 9% weekly, out of 31,877 answering that question ⁵.

The failure this job exists to prevent has a name and a literature behind it. A systematic review of automation bias describes users over-accepting computer output as "a heuristic replacement of vigilant information seeking and processing"; a meta-analysis of four clinical decision-support studies inside that review found erroneous advice was more likely to be followed in the decision-support groups than in the controls, and that a system in error raised the risk of an incorrect decision being made by 26% ³. The tool is new. The way people defer to it is not.

Supervision is also turning into a named qualification rather than an implied one. Under Regulation (EU) 2024/1689, the EU's AI Act, deployers of high-risk systems "shall assign human oversight to natural persons who have the necessary competence, training and authority, as well as the necessary support" (Article 26(2)) ⁶. Which systems are in scope, and from when, is a question for counsel. The hiring consequence is the part worth acting on now: authority is granted, not brought to the interview. If the role cannot stop a run without three approvals, no assessment fixes that, and you are testing someone you have already set up to lose.

Build the exercise from one agent run that already looks finished

Give the candidate the four things the job gives them: the original request, the brief the agent was handed, the agent's full run (its steps, its intermediate claims, its finished deliverable), and the source material that settles whether any of it is true. Seed three or four real defects into the run. What they hand back is the review, not a tidier version of the work.

Seed by class, not by difficulty. Each of these is something an agent does that reads as competence:

• A claim sourced to a real document that does not say it. The link resolves. The paragraph behind it says something adjacent.
• A step reported as done that was not done. The run log says the file was validated; nothing in the trace validates anything.
• A correct answer to a slightly wrong question. The brief asked for last quarter and the agent answered for the trailing year, competently.
• A number right in its arithmetic and wrong in its base. The growth rate is computed cleanly off the wrong denominator.
• An act that should have stopped for a human. A refund issued, a record amended, an external email drafted and queued.

Cap it at 45 to 60 minutes in one sitting, and hand every candidate the identical packet so the round stays comparable. Then ask three questions in writing at the end: what did you check and against what, what are you rejecting and on what ground, and what would you not ship without another person looking at it.

Add the specification half if you have ten more minutes, because it is where the strongest candidates separate: rewrite the brief so this run would have failed loudly instead of quietly. A supervisor who can only find defects after the fact is doing half the job. One who writes "cite the page number, and stop if a source cannot be opened" into the brief has removed a class of failure from every future run. Testing whether a candidate notices when AI gets something wrong is the review half; the brief they write is the other one.

What counts as a credible check in your field?

One act that touches something outside the agent's own output. Reading the deliverable more carefully is not a check, and asking the agent whether it is confident is not one either. Name the act your field accepts before you write the rubric, because it decides what a reviewer can mark down as done. It varies more than people expect:

• Software engineering. The branch checked out and the test run against the real fixture, not the one the agent wrote to pass.
• Financial analysis. The figure re-derived from the filing rather than from the agent's summary of the filing.
• Legal operations. The case opened and the paragraph read. Leading AI legal research tools were measured hallucinating between 17% and 33% of the time, and a citation that resolves is not a citation that supports ⁴.
• Marketing. The statistic traced back to its sample, until the candidate can state the n and who was surveyed.
• Healthcare revenue cycle. The denial code read against the payer's own published policy, not against the agent's account of it.
• Supply chain. Three quotes re-normalized onto one set of terms before any of them are compared.
• Data and analytics. The query re-run with the filter the agent described, and the row count compared against what the write-up claims.

Two shortcuts will be offered to you, and both fail. The first is the agent's own reasoning. In a controlled study of AI-assisted decisions, explanations increased the chance that people accepted the recommendation regardless of whether it was correct ², so a fluent rationale attached to a wrong answer is worse than a bare wrong answer, not better.

The second is the candidate's own sense of how the run went. In METR's randomized trial, 16 experienced open-source developers working across 246 real issues took 19% longer to complete them when allowed AI tools; they had forecast a 24% speedup beforehand, and still believed they had been sped up by 20% afterward ¹. Self-report ran in the opposite direction to the stopwatch. This is the same reason hiring for verification rather than production has become its own problem: producing has gotten cheap, and knowing whether the product is right has not.

How do you score it so two reviewers agree?

Score observable acts, one column each, marked separately by two people. Which seeded defects were named. Which rejections carry a ground you could restate in one sentence. Which checks were actually run, and against what. What was escalated instead of shipped. No single number for the person: the columns are what you compare across candidates, and the columns are what you can defend later.

A ground you can restate is the whole test. "Rejected the market-size paragraph because the only source behind it is a vendor's own 40-person customer survey" is a yes. "Showed good judgment" is not a column, it is an impression, and two reviewers will fill it differently every time. Writing a rubric two reviewers score the same way is mostly this one discipline applied to every row.

Have both reviewers fill the grid before they talk. Where they disagree, the usual cause is a badly written column rather than an ambiguous candidate, so rewrite the column and re-mark. Accountability appears to matter here in a way that is worth building into the process: the automation-bias review found that external manipulations of accountability gave mixed results, while people who perceived themselves as accountable made fewer automation-bias errors ³. The cheap version of that is a name attached to every pass, on both sides of the table.

One trap in the marking. An over-refusing candidate is not the safe answer. Someone who blocks every run costs exactly the throughput the agent was bought for, and refusal without a ground is a different way of not reading. Score refusals that name what is wrong, and count them against the parts of the run actually taken up, so three well-aimed rejections beat thirty reflexive ones.

What this exercise still won't tell you

Whether they are still checking in month seven. A seeded defect is easier to find than a real one, because the candidate already knows something is wrong somewhere; the job's actual difficulty is staying alert across a long line of runs that are mostly fine. One exercise measures an afternoon of attention. Say that out loud in the debrief rather than treating one packet as a character reading.

The packet leaks, too. A dozen candidates in, it will be sitting in a group chat somewhere and the exercise starts measuring recall. Build the second case at the same time as the first, keep one rubric across both, and rotate when the round gets busy.

Tell the candidate what is being assessed before they start, in one short paragraph: the run has defects in it, the review is the deliverable, and the rubric marks checks, grounds and escalations. Nothing here needs to be a trick. A candidate who knows the review is the work does the job better, which is the point of the exercise, and a hidden rubric only measures who guessed your intent.

Two things not to score, because they will pull the grade off the job. Volume of AI use is not a virtue: a candidate who did one step by hand because the agent was the wrong instrument for it made a supervision decision, and it should count as one. And a slow candidate who ran three real checks has told you more than a fast one who skimmed and approved. Speed is a legitimate business constraint, but it belongs in the time cap, not in the rubric.

See what gets scored >