How Do You Redesign an Interview So AI Assistance Becomes Signal?

Why does an AI-open interview usually measure nothing?

Because the question can be finished from its own wording. Ask a candidate to design a rate limiter, size a market, or outline a churn analysis with an assistant open, and the model has the entire brief in the prompt. What comes back is the model's competence, formatted. The redesign is not a format swap; it is putting something into the task the model has no access to.

The evidence sits where you would expect. METR ran a randomized trial with 16 experienced open-source developers on 246 real issues, in repositories they had contributed to for years and that average over a million lines of code. With AI tools allowed, the work took 19% longer, and afterwards the developers still believed the tools had sped them up by 20% ¹. The researchers' own reading is that assistance does worst where a task carries many implicit requirements: testing conventions, documentation standards, the local rules that take a person months to absorb ¹.

Read that as an interview design note rather than a verdict on AI. The implicit requirements are exactly what a model cannot infer, and they are the only place a candidate's judgment becomes visible while an assistant is running. A generic question hands the candidate a problem with no implicit requirements at all, which is why the answers arrive identical and unfalsifiable. If your current round is a set of spoken questions about AI habits, the questions themselves can be sharpened first, but a described habit and a demonstrated one are different evidence.

The constraint also has to be one the candidate can discover during the task. A secret you never let them reach is a trick question. A failing test they can run, a table they can pivot, a budget line printed on the brief: findable in minutes, impossible to guess.

Rewrite the engineering question around your own test suite

Take the question off the whiteboard and put it in your repository. The before is "add pagination to this endpoint, use any tools you like": a brief the assistant satisfies alone. The after hands the candidate a branch of your actual service with one integration test that fails, forty minutes, and the standing rule that the other nine hundred tests still have to pass.

Before: "Implement a retry with exponential backoff for our payments client." The model writes it correctly in nine seconds, and you learn nothing about the person.

After: "Here is the repo at this commit. `test_capture_idempotency` has been flaking for three weeks. Make it deterministic. Anything you use is fine, and the assistant will offer you a fix in the first minute."

What the model cannot know is the whole point: that your fixtures share a Redis instance across the parallel test workers, that the retry decorator it will reach for is banned in this codebase because it swallows the timeout, that the flake is in the fixture rather than in the code under test. The assistant produces a confident patch. The candidate either runs the suite and watches it fail differently, or ships the patch and tells you it is fixed.

That second outcome is not rare, and it is measurable. In a controlled study, participants given an AI assistant wrote significantly less secure code than the control group, and were more likely to believe their code was secure ². Confidence is not the signal. The check is.

Watch four things: whether they ran the failing test before changing anything, whether they asked what the failure actually was before generating a fix, what they refused from the assistant and on what grounds, and whether the final diff contains a line they wrote themselves. The mechanics of running this live, with the assistant on screen, are their own problem: an AI-open coding round has setup costs a whiteboard does not.

Rewrite the analyst question around last quarter's churn

Hand over the real cohort table instead of the concept. The before, "how would you analyze churn?", is a question with a memorized answer. The after is a sanitized export of your actual retention data for one segment, a one-paragraph recommendation due in thirty minutes, and one fact you do not mention on the brief: a pricing change landed in the middle of the period.

Before: "Walk me through how you would investigate a rise in monthly churn." You get a framework, correctly recited, from the candidate or the model or both.

After: "This is our SMB retention by cohort for the last four quarters. Tell me in one paragraph what happened and what you would do about it, with the number you are relying on."

The assistant will read the columns and write a fluent narrative: seasonality, onboarding friction, a competitor. It cannot know that 900 of those accounts were migrated between billing systems in month two and counted as cancellations, or that the segment definition changed in March. A candidate who asks what changed in the period before analyzing anything has already told you more than a framework answer ever will.

Score what happens at the moment the assistant is confidently wrong. Did they recompute one figure by hand against the raw export? Did they name the thing they could not settle from the data and say so in the paragraph? Did they push back on the model's most quotable sentence, or ship it? The follow-up matters as much as the task: a second question aimed at the reasoning behind the answer is what separates a candidate who understood the table from one who narrated it.

Sanitizing is real work and worth doing properly. Replace account names, shift the dates, round the revenue. Keep the wart intact, because the wart is the question.

Rewrite the consulting question around the client's real budget

Give the candidate the client's real number. The before is "how would you take this product to the SMB market?" The after is a two-page brief from an engagement you actually ran, with the budget line left in: $40,000, one part-time marketer, and a channel the client has already spent on and abandoned. Ask for the plan, and in writing, what they cut to fit.

Before: "Design a go-to-market for a new product in a crowded category." Every answer names the same four channels in the same order.

After: "This client has $40,000 for the quarter, half a marketer, and a paid-search program they killed in April after spending $60,000. Recommend a plan and tell me what you dropped."

The assistant produces a full-funnel plan costing roughly $300,000 and leads with paid search, because it is the modal answer to the generic version of the question and nothing in its training says otherwise. The budget is the constraint that makes the answer refusable, and refusing on stated grounds is the behavior you are hiring for. Watch whether the budget appears in the candidate's first move or only in the final edit. Framing before generating is the cheapest thing to see and the hardest to fake.

One consequence of doing this three times: the questions do not transfer. The failing test says nothing about the marketing hire, and the budget brief says nothing about the engineer. That is the real cost of the redesign and the honest answer to whether one assessment can cover every role. The rubric transfers. The task does not.

What do you score, and how do you write it down?

Settle the rubric first, then write the question, and grade acts rather than impressions. Six behaviors are worth writing down: what was framed before anything was generated, which claim got a source demanded, what the candidate kept for themselves, what existed between the brief and the deliverable, what they refused and on what grounds, and what they tested against something outside the conversation.

Each one needs an observable, not an adjective. "Good judgment" is unscoreable. "Ran the failing test before editing" is a yes or a no, and two interviewers will agree on it. A worked version of this (what the rubric rows actually say for an AI-assisted answer) is worth writing before the first candidate rather than after the third.

Quality is not the only reason to build it this way; the compliance case runs the same direction. A step that decides who advances is a selection procedure, and it has to be job-related and consistent with business necessity; work samples and simulations are named examples ³. The Uniform Guidelines put it more sharply for this exact design: content validity holds to the extent the procedure is a representative sample of the content of the job, and a procedure resting on inferences about mental processes cannot be supported by content validity alone ⁴. Asking someone how they would verify a claim is an inference about a mental process. Handing them a claim and a suite is a work sample.

Two cautions. Do not score how much AI the candidate used. A person who judged the model was the wrong instrument for a step and did it by hand has demonstrated the thing you are testing. And do not score the polish of the deliverable, which is the one variable the assistant reliably controls. What good AI use actually looks like is a shorter list than most rubrics assume.

Where does an interview redesign stop?

It stops at scale. One redesigned question per field is an afternoon of authoring plus a rubric, and it goes stale. Hand the same failing test to twenty candidates and someone posts the fixture. Budget for a rotation you will have to write yourself, an interviewer who can read the work in real time, and a panel that agrees what "refused it on substance" means before anyone is graded.

The live format has a second ceiling: forty minutes with someone watching is not how the job is done, and it selects for composure alongside judgment. Moving the same constraint into an unwatched task trades that for a provenance question you then have to answer: a take-home with AI allowed buys realism and costs you the record of how the answer was reached, unless you ask for that record explicitly.

The last limit is the one worth saying out loud to your own hiring manager. A redesigned question shows you one candidate on one afternoon against one constraint you chose, and a bad choice of constraint produces a confident, well-graded wrong answer at your end too. Run the new question against two people already doing the job before it decides anything about a stranger. If both of them fail it, the question is wrong, not the field.

Building this yourself is a legitimate choice and often the right one at small volume. Olive is one instrument in the same category and says so; multiple-choice AI literacy tests, code-collaboration graders and unwatched take-homes are all real approaches with different trade-offs.

See what gets scored >