What a Good AI Answer Sounds Like at Junior, Mid and Senior

What should a junior candidate be able to show?

Verification, in one concrete instance. A junior does not yet own a delegation strategy, and asking for one produces a memorised answer. What they can honestly show is a moment when the output was wrong, how they noticed, and what they did next. If they cannot name a single instance, that is the finding, and it does not depend on how long they have worked.

The reason to set the bar there rather than at output is that the tool does most of its measured lifting at this level. Across the staggered rollout of a conversational assistant to 5,179 customer support agents at one software firm, issues resolved per hour rose 14% on average, with a 34% improvement for novice and low-skilled agents and almost nothing for experienced ones ¹. Pooling three company-run randomized trials across 4,867 developers, completed tasks rose 26.08% with a standard error of 10.3%, and less experienced developers both adopted the tool more and gained more from it ².

Read those two carefully before using them. The first is one firm, one occupation, a pre-ChatGPT assistant trained on that firm's own transcripts, measured on problems with correct answers. The second measures weekly completed pull requests, which says nothing about defect rates or review burden, and its standard error puts the honest range somewhere between roughly 6% and 46%. Neither is a general law about juniors.

What they do establish is the shape of the hiring risk. If a tool raises a junior's output substantially, output stops separating candidates at that level, and the thing that still separates them is whether they can tell a good output from a confident wrong one. Hiring for verification rather than production is that argument at full length, and it matters most for new graduates who never learned to work without these tools.

Why is delegation the mid-level bar?

Because a mid-level engineer, analyst or marketer owns a whole task and has to divide it. The judgment being hired for is where the line falls: which parts go to the model, which parts stay, and what makes a part belong on one side. A candidate at this level who describes handing over everything or nothing has not been asked to make the call yet.

The follow-up is what does the grading. Ask why that part and not the next one. A weak answer is about speed or about what the tool is good at. A strong answer is about consequence: this part gets read by a client, this part feeds a model downstream, this part is where I am the only person who would notice the error. The candidate is describing a map of where damage lives in their work, and that map is the thing that transfers to a new company.

A second useful probe at this level is what they changed their mind about. Somebody who has been dividing tasks for two years has moved the line at least once, usually after something went out wrong. The answer names the incident, the new rule, and what the new rule costs them in time. Nobody rehearses that shape, because it requires a real loss.

A mid-level candidate who lacks a team-wide policy view has not failed this bar. That view belongs to the level above, and to a different question. Whether to hire the junior who is fast with AI over the senior who is not is the trade this bar exists to make legible.

What does a senior owe that a mid-level does not?

An answer about work they did not do. A senior signs off on other people's output, so the bar is what they read line by line, what they accept on trust, and what they escalate. That is a standing rule with a cost attached, and the strong version names the cost: what got through the last time the rule sat too loose, and what changed after.

Treat a claimed speed improvement as the least reliable input in the set. A reported speedup describes what somebody believes about their own work, and belief and measurement can come apart even for experts. Sixteen experienced open-source developers, working on repositories they had known for about five years, forecast that AI tools would make them 24% faster, believed afterwards they had been 20% faster, and were measured 19% slower ³. Sixteen people in one setting, on mature codebases, with early-2025 tooling. The magnitude does not transfer. What travels is that these developers had the direction wrong, not just the size.

So grade the rule. A senior who says the model saved the team a day a week has told you what they believe. A senior who says code touching billing gets read twice regardless of who wrote it, because of an incident in March, has told you what they do.

The people-management version of this question is genuinely separate and should be asked separately: what a manager lets their team ship, how they found out somebody was over-relying on a model, and what they did about it. Four questions for a manager whose team ships with AI covers that seat rather than this one.

Calibrate the bar to the level, not to the candidate

Put the three bars in the scorecard before the first interview, written as content rather than as a rating. Junior: names a specific wrong output and what happened next. Mid: draws a line through a task and defends where it sits. Senior: states a review rule and what it cost when it was wrong. An interviewer holding that page stops grading fluency.

The failure this prevents is the debrief where levels get compared. Four interviewers who each heard a different level of candidate, with no written bar, converge on whoever sounded most current, and current vocabulary is the cheapest signal in the room. Writing the bar down before the loop is the only cheap fix, and it is the same discipline that makes a defensible bar for good enough at AI in a specific role hold up if anyone questions it later. Written in decisions rather than in AI answers, the same page answers the general question of what level a candidate really is when every title in the stack says senior.

Never score a self-rating on any of the three. In a study of 288 Taiwanese teachers who took both a self-report and a knowledge-based test of AI literacy built on the same framework, correlations between the objective and self-reported factors ran from 0.07 to 0.24, and the six profiles the authors found included people who underestimated themselves as well as people who overestimated ⁴. Teachers in Taiwan are not job candidates, and a weak correlation is not proof that anyone flatters themselves. It is proof that a claimed level and a demonstrated one are two different measurements.

If the same question is being split across a panel, put the level bar in every seat's copy of the scorecard rather than in one interviewer's head. Giving each panel seat its own AI question divides the work; the level bar is what every seat grades against.

See how it works >