How Do You Score an Interview Answer the Candidate Produced With AI?

What Do You Score When the Model Wrote the Draft?

Score four acts the candidate had to perform themselves: what they framed before generating, what evidence they demanded for the claim the answer rests on, what they refused from the model's output, and what they checked outside the conversation. Rate each separately, on its own anchored scale. Fluency, structure and length are the assistant's contribution, and rating those tells you about the tool.

The reason to split them is not tidiness. Federal selection rules put the ratable material in a specific place: a job analysis should focus on observable work behaviors and, where a behavior is not observable, on the aspects that can be observed and on the work products; a skill has to be operationally defined in terms of observable aspects of work behavior; and a selection procedure resting on inferences about mental processes cannot be supported solely or primarily on content validity ¹. So "judgment," "critical thinking" and "AI fluency" are not rubric items. Opening a source is. Recomputing a figure is. Cutting a section is.

The four items also match where the failure actually lands. In the 2025 Stack Overflow developer survey the single most common frustration with AI tools, at 66%, was output that is almost right but not quite; 46% of developers said they distrust the accuracy of what the tools produce against 33% who trust it ². Almost-right passes every proxy an interviewer used to read (structure, confidence, vocabulary, a clean numbered plan) and fails on the one claim the recommendation hangs from.

And the confident-wrong claim is not rare in professional tools built for exactly this. A Stanford RegLab and HAI study of purpose-built legal research systems found Lexis+ AI and Thomson Reuters' Westlaw AI-Assisted Research and Ask Practical Law AI each hallucinated between 17% and 33% of the time, against vendor claims of hallucination-free retrieval ³. A candidate who takes a cited-looking claim at face value is not being careless by the standards of the marketing; they are being careless by the standards of the job. That distinction is what the rubric has to encode, and it is why the scale gets written before the questions do, which is the ordering argument in Structured Interview for AI Use.

How Do You Anchor "Demanded Evidence" in Your Own Field?

Name what a source is in the work. "Demanded evidence" is empty until it says: for a consultant, a named source opened and the figure traced to the page; for an analyst, a reconciliation that ties to the system of record; for an engineer, a test that fails before the change and passes after. Write the anchor as the act the job would accept, then rate against it.

This is the half most rubrics skip. The federal definition of a structured interview has two clauses, not one: every candidate gets the same predetermined questions in the same order, and every response is evaluated using the same rating scale and standards for acceptable answers ⁴. Teams ship the scale and skip the standard, then wonder why the numbers do not travel between interviewers. A five-point scale is fine if all five points are labeled with an act; unlabeled points are where taste enters.

The pattern across the rows is one sentence: evidence is whatever a competent colleague in that job would ask to see before signing. Write that sentence for your role, put it in the top band, and the middle and bottom bands mostly write themselves: middle is asked for and not opened, bottom is neither asked for nor opened. Setting the height of that top band for a specific role is its own decision, worked through in How to Set a Defensible AI Proficiency Bar by Role.

Score One Answer: A Filled-In Rubric

Here is one answer, rated. The question went to a senior analyst candidate: top-of-funnel conversion fell from 3.1% to 2.4% over a quarter, walk through the first two days. The reply came back in 600 fluent words, five numbered steps, one benchmark claim and one recommendation. It takes the top band on nothing, the middle on framing, and the bottom on the other three.

The shape below is the one that arrives most often.

> Step 1: segment the drop by channel, device and campaign. Step 2: check for tracking changes; a broken tag explains a 0.7-point move more often than demand does. Step 3: compare against the industry benchmark; B2B SaaS top-of-funnel conversion runs around 2.8%, so 2.4% is below but not alarming. Step 4: interview two account executives. Step 5: ship a landing-page test.

• Framing: middle band. Top band asks for the decision the answer serves and one condition that would make the answer wrong. This names the diagnostic and never the decision (whether this is a spend question or a product question changes every step), and never a falsifier. It earns the middle band on step 2 alone, which raises an alternative explanation before committing to analysis. That is one real framing act, and it is the only one.
• Demanded evidence: bottom band. The load-bearing claim is "around 2.8%." Top band asks for the source named, opened, the figure located, and its definition compared against ours. No source appears. Asked in the follow-up, the candidate produced a vendor benchmark report and had not opened it; the report counted free-trial signups, which this funnel does not. Bottom band, and the follow-up is the evidence for the rating.
• Output rejection: bottom band. Top band asks for at least one direction refused on stated grounds. All five steps survive into the answer. A five-step plan with nothing dropped is the assistant's default reply with the candidate's name on it, and the give-away is that the steps are not ordered by cost: a broken tag takes ten minutes and sits second, behind segmentation.
• Verification: bottom band. Top band asks for one claim tested against something outside the conversation, where the result changed the answer. The candidate had the funnel export. Nobody recomputed 2.4% from it, so the number the entire reply rests on was never confirmed against the file sitting next to it.

One middle and three bottom is not a rejection. It is a scored, specific second conversation: open the benchmark with them, hand them the export, ask which step they would cut. What the rubric bought is that the next interviewer starts from four rated items rather than from "strong communicator." Expect an empty top band often, and resist grading the batch on a curve, which is the same discipline that keeps a polished submission from carrying a weak decision through a whole rubric, covered in Grading Polished Take-Homes.

Why Do Two Interviewers Score the Same Answer Differently?

Because the anchor is written in adjectives. "Strong evidence," "good rigor" and "shows judgment" each mean whatever the reader already believed, so two interviewers rating one reply disagree about the words rather than about the answer. Rewrite every band as an act with a trace: what was opened, what was recomputed, what was cut. Then double-score the first five candidates and compare item by item.

Two habits do most of the work after that.

• Rate the trace, never the account. A candidate's description of their own process is not evidence of it. METR randomized 246 real issues from experienced developers' own repositories to allow or forbid AI tools: with the tools allowed the work took 19% longer, and the same developers, having finished, still believed the tools had sped them up by 20% ⁵. People are unreliable narrators of their own AI-assisted work, and that holds for the sincere ones.
• Rate item by item, across candidates. Score every candidate's framing, then every candidate's demanded evidence, then rejection, then verification. Reading one reply end to end is how a fluent opening paragraph sets the tone for four ratings that were supposed to be independent.

Where the two raters disagree in the first five, the anchor is ambiguous rather than the candidate borderline. Fix the wording, log what changed, and hold the fixed version for the rest of the loop. The disagreement is cheap information at candidate three and expensive at candidate thirty; the mechanics of keeping a collaboration rubric consistent across a panel are in Can Two Interviewers Score AI Collaboration the Same Way?.

There is one bias specific to this rubric worth naming. A survey of 319 knowledge workers across 936 first-hand examples found that higher confidence in the AI tool tracked with less critical thinking, while higher confidence in one's own ability tracked with more; the effort that remains shifts toward verifying information, integrating the response and stewarding the task ⁶. An interviewer who trusts these tools reads an unsourced benchmark as fine. One who distrusts them reads it as disqualifying. Neither is rating the candidate, which is what a written standard for an acceptable answer is for.

What This Rubric Cannot See

An interview answer is an account of work, not the work. The rubric rates what a candidate says they opened, recomputed and cut, and someone who did none of it can describe all of it fluently. Follow-up questions shrink the gap (borrowed specifics survive one round and rarely two), but they do not close it. Only a record of the acts closes it.

Two ways to get closer, both cheap. Ask for the artifact rather than the summary: the source file instead of the exported chart, the transcript of the exchange instead of the conclusion, the commit instead of the description of the commit. And put the material in front of the candidate live, so the check has to happen in the room: hand over the funnel export and ask for the number, hand over the benchmark PDF and ask whether it counts what we count. Both convert a claim about an act into an act. The question design that makes those follow-ups land is worked through in Follow-Up Questions That Expose Understanding.

It is also worth being clear about what a rubric of this kind does buy, because it is not a prediction of performance. It buys comparability across candidates who were asked the same questions and rated on the same standards for acceptable answers ⁴. It buys a defensible record, because the rating rests on observable behavior and work products rather than on an inference about someone's mental processes ¹. And it buys a specific next step, which an overall impression never does.

The last limit is the one to say out loud in the debrief. A bottom band on verification means the candidate did not verify anything in this answer, under these conditions, with this much time. It does not mean they cannot. Say that plainly when the panel meets, because a four-item rubric read as a verdict on a person is worse than no rubric at all: it lends arithmetic to a judgment that has not earned it.

See what gets scored >