Give Each Panel Seat One AI Question Nobody Else Asks

Why does the whole panel end up asking the same question?

Because the panel was split by competency, and AI is not one. Give four interviewers a list that says technical depth, collaboration, ownership and culture, then add one new topic nobody owns, and all four reach for it. Each gets a version of how do you use AI in your work, each hears the same rehearsed ninety seconds, and the debrief turns into an argument about tone.

Redundancy used to be the point. Two interviewers hearing one answer and rating it separately is a reliability check, and it is cheap. It stopped being a check when the answer became something a candidate can draft, refine and practise the night before, because four samples of a rehearsed paragraph are still one sample of the person.

The selection literature is clear about where combining evidence pays. A composite of genuinely different predictors reaches about .61, an estimate that assumes those predictors are combined mechanically with sensible weights, which is not what a team gets from stacking four unscored conversations ¹. A panel is one method however its seats are arranged, so no split of it buys that number. What a split buys is coverage: the same hour comes back carrying four parts of the job.

The second cost is the one nobody budgets for. A freeform conversation can push out information the process already has. Undergraduates who interviewed a classmate predicted that classmate's GPA at r = .31, worse than the r = .65 they managed from prior GPA alone, and in a follow-up study 96 of 169 participants chose an interview in which the answers were generated at random over conducting no interview at all ². Students predicting grades are not managers predicting job performance, and the size of that gap does not transfer. The mechanism does: people build a coherent story out of whatever they are handed, including noise.

Give each seat one stage of the work

Four stages, four seats, in the order the work happens. Seat one takes the delegation decision: what went to the model and what stayed. Seat two takes verification: how the output got checked, and against what. Seat three takes recovery: the day it came back wrong with the deadline already gone. Seat four takes the refusal: the piece this person will not hand over, and the reason.

1. Which parts of that project did you keep, and which did you hand over? Follow up on the boundary itself. A useful answer names a specific task and a reason drawn from the work. Speed on its own is the answer to push on. 2. How did you know the output was right? The strong answers name a source outside the model: a filing, a test suite, a payer policy, a person who would know. The weak ones describe reading it over. 3. Tell me about a time it was confidently wrong and the deadline was that afternoon. This is the seat that finds out whether verification is a habit or a slogan. In one field experiment, on a task deliberately chosen to sit outside the model's capability, consultants working with GPT-4 reached the correct answer 60% and 70% of the time against 84.5% in the control group, and the condition given a prompt-engineering overview did worse than the one given none ³. One task, one sample, a 2023 model. What it illustrates is the shape of the failure: nobody could tell which side of the line the task was on. 4. What will you not put through a model, at this company, whatever the deadline? Listen for a line drawn around consequence rather than around policy.

Seat three and seat four are the hardest to rehearse, because both require an actual episode with a cost in it. If the loop only has three interviewers, cut seat one: what somebody delegates is also the easiest thing to read off a work sample later.

Write the scorecard before the loop opens

Each seat needs three things on paper before anyone meets the candidate: its own question, the evidence it is listening for, and what an acceptable answer contains. That third item is the one loops skip. A federal guide to structured interviewing defines the format as the same questions asked in the same order, evaluated on a common rating scale, with interviewers who agreed in advance what an acceptable answer looks like ⁴.

Write the acceptable answer as a description of content, never as a rating. For the verification seat: names at least one source outside the model, and says how the check would have caught a specific error. For the recovery seat: describes what they did in the hours after, including what they told whoever was waiting. An interviewer who has that sentence in front of them stops grading confidence.

One rule holds the split together: each seat records evidence only for its own question. If the delegation seat hears a good verification story, it goes in the notes as context and never in the rating. Without that rule the split collapses, because every interviewer wants to answer the whole question. Getting a panel to judge AI use the same way is the briefing problem underneath this one, and running a structured interview about AI use is the mechanics of asking a single question well.

Set the level bar in the same document. An acceptable verification answer from a new graduate looks different from one at staff level, and the three bars at junior, mid and senior belong in the scorecard. Left to the debrief, they turn into a negotiation.

What does the debrief do with four different answers?

It assembles them rather than averaging them. Nobody rates the candidate on AI. Each seat reports what its own question produced, in the order the work happens, and the room reads the four together: what this person hands over, how they check it, what they do when the check fails, and where they stop. Treat disagreement between seats as information about the candidate.

A candidate who delegates cleanly and verifies thinly is a real profile, and a coachable one. A candidate who verifies obsessively and delegates nothing needs a different conversation. Averaging four ratings would have erased both, and reading the seats in sequence puts both in front of the room.

Hold one thing back from the room until every seat has read out. A summary delivered first hands every other seat a frame to read its own notes through. Read in sequence, then discuss.

The honest limit: none of this converts a rehearsed paragraph into a demonstration. Four questions get four accounts, and an account is what a conversation can produce. When the decision genuinely turns on catching a wrong output, the loop needs a piece of work in it, and scoring an answer the candidate produced with AI is the narrower version of that problem. Panels that also want the hiring managers on them to read AI-assisted work consistently should start with the managers who do not use these tools themselves.

See how it works >