A Good Question Is One a Prepared Stranger Cannot Answer

What separates a good interview question from a bad one?

Two properties: the answer has to depend on something only this candidate did, and the difference between a strong and a weak answer has to be visible to the person listening. Neither is the question's category. A question that fails the first collects preparation. A question that fails the second collects a conversation, and a conversation is not evidence anyone can argue with in a debrief.

The second property is the one hiring teams under-build. Written anchors saying what strong, adequate and weak answers actually contain are what turn a rating into something two people can check against each other. In a meta-analysis of selection and admissions decisions, the average correlation with job performance was .44 when applicant data were combined by an explicit rule and .28 when the same kinds of data were combined by expert judgment, a relative improvement the authors put at more than 50% ¹. That comparison rests on nine studies, and it covers only the last step, how the evidence gets added up once it is in. It says nothing about which evidence to gather. A plain unit-weighted written scorecard counts as such a rule, and nothing in the result licenses a number standing for a person.

The first property is why format alone settles nothing. The 2022 re-analysis of the selection literature puts structured interviews at .42 and unstructured ones at .19 ². The gap between two interviews is wider than the gap between most pairs of methods. What those studies coded as structure is narrow: every candidate faces the same set of questions and is rated on a common scale. Whether the questions are open or behavioral is not part of that coding.

Interviewer agreement in a traditional unstructured interview is low enough that, even against a perfectly reliable measure of job performance, judgments from it could never account for more than 10% of the variance ³. That is a ceiling on how much such judgments could ever explain, implied by reliability rather than measured directly, and it comes from a review reporting a 1995 meta-analysis. It still sets the price of asking whatever comes to mind.

Why has the fabrication test stopped separating good questions from bad?

Because the property that test rested on was cost, and the cost is gone. Producing a detailed, plausible, well-structured account of a past project used to take real effort, which is the reason open behavioral questions were held to be hard to prepare for. A candidate can now assemble one in minutes, and preparing answers is what candidates have always been told to do.

The nearest measurement of how that reads to a person comes from education rather than hiring. In a blind study run through the real examinations system of a UK university, unedited GPT-4 answers submitted under 33 fake student accounts across five undergraduate psychology modules drew no concern of any kind from markers in 94% of cases, and were graded on average above the real students ⁴. Coursework is not an interview, the tasks were essays and short answers, the markers had no instruction to look for anything, and the authors describe their own method as using AI in the most detectable way possible. What travels is the direction: a plausible written answer, produced quickly, reads as competent to a reader who is not testing it.

So the criterion the published lists are built on has stopped doing the job it was hired for. The questions themselves are still sound. What changed is that the first answer carries much less information than it used to, which is a different problem with a different fix. Every candidate suddenly giving the same polished STAR answer is the shape this takes in a real loop.

The replacement criterion is survivability. Ask what happens when you push the account one level past what a prepared answer contains: the number they were working against, who disagreed, what they would do differently, what the second version cost. A real instance has texture at that depth because the person was there. A rehearsed one gets general at that depth, because a script has nothing underneath the first answer.

Run the four checks against the bank you already have

Open your existing interview guide and put every question through four checks in order. The bank you already run is the one producing your debriefs, so it is the one worth testing. Expect the failures to cluster in the questions people enjoy asking, because a question that is pleasant to ask is usually one any prepared candidate can answer.

1. Could a well-prepared stranger answer this convincingly? If yes, it is a preparation question. "Tell me about a time you handled conflict" passes for anyone who has read a prep guide. "Walk me through the last disagreement you had with someone whose work you depended on" is harder to answer from nothing, because it points at one occasion in that person's history. 2. Does it ask for an instance or invite a policy statement? "How do you approach code review?" gets a philosophy. "What was the last change you asked someone to make in review, and what happened next?" gets an event. The words *the last*, *the most recent* and *the one you regret* are doing most of the work here. 3. Is there a follow-up only someone who was there could survive? Write it down next to the question. If you cannot think of one, the question has no floor, and the strong answer and the rehearsed answer will arrive looking identical. Which follow-ups expose whether someone understands the answer they just gave is the harder half of this check. 4. Is there a written anchor for strong, adequate and weak? Two or three sentences each, naming what the answer contains. A question with no anchor produces a conversation and a rating nobody can reconstruct a week later.

A question that fails checks one and two is usually rescuable by rewriting it toward a specific occasion. A question that fails three and four is a scoring problem rather than a wording problem, and rewriting the question will not touch it.

What do you keep after cutting the bank in half?

Three or four questions with room to go deep on each, plus one opener you expect people to prepare for. Depth is what the cut buys: the minutes freed by deleting six questions are the minutes that let a single answer be followed to the point where preparation runs out. Coverage is the thing teams overbuy, and depth is the thing they never budget.

Keep the preparable opener on purpose. How a candidate spends preparation time is readable in its own right, and the person who arrives having read your changelog, your docs and two of your customers' reviews has told you something real about how they work. Record it as context in the notes rather than as a rated criterion, because rating it would reward available time as much as judgment.

Put the recovered minutes somewhere specific. Three planned questions with three follow-ups each is the arithmetic on the other side of the cut, and the ladder belongs in the guide next to its question so every candidate has the same rungs available.

One thing to hold on to as the bank shrinks: the questions that survive should be about the work this role actually does. A generic question well-formed enough to pass all four checks still tells you less than a specific one about the thing the person will be doing on their second Tuesday. Cutting also needs somebody who is allowed to cut, so who owns the question bank, and what each surviving row has to carry is where that authority gets written down.

See how it works >