Change the Constraint Halfway and See If the Design Moves

Why did the round stop separating anyone once the answer was free?

Because it was always weakly anchored, and the canon getting cheap made that visible. A system design prompt rarely carries a number anyone is actually held to: no traffic figure you have to hit, no bill you have to stay under, no deadline that already passed. The candidate usually supplies the constraints and then satisfies them. Grading an answer built on the candidate's own constraints has meant comparing a recital to a picture the interviewer holds in their head.

Structure is the missing part, and the gap it leaves is measurable. In a 2022 re-analysis of the selection literature, Paul Sackett and colleagues put structured interviews at .42 and unstructured ones at .19, pooling two earlier meta-analyses that had each coded interviews by degree of structure ¹. Before any statistical correction the observed figures are .32 and .13.

More than double the predictive power, bought with paperwork.

How much of that paperwork does a design round carry? Less than the word structured implies. Levashina and colleagues content-analysed 104 interviews reported in 103 articles published between 1997 and 2010, and found the studies describing a structured interview used an average of 5.74 of the 15 components in Campion, Palmer and Campion's framework ².

Two of the rarer components sit on the evaluation side: detailed notes in 19% of those interviews, no discussion between interviews in 4% ².

Those counts partly measure what the studies left undescribed, and practice is unlikely to be more structured than published research. So do not assume a design round keeps detailed notes or rules out a hallway debrief.

What changed is supply, not the round. The reference answer to each well-known system design prompt is written up, indexed, and now produced on demand, in much the same order with much the same diagram.

Somebody who spent a weekend on it now sounds like somebody who spent three years on it, and no record in the room tells them apart. Whether structured interviews still separate AI-coached candidates is the general form of the problem.

Don't answer a public canon with a private puzzle

The obvious fix is to invent a scenario nobody has written up, and it fails in two directions. Obscurity selects for whoever happened to touch that corner of the field, which is trivia wearing a systems costume. It also punishes the candidates who prepared honestly, since preparation is what you asked them to do when you named the round in the recruiter email.

Harder buys less than it looks, on one family of exercises where somebody counted. In a study that generated model solutions automatically and submitted them to LeetCode, a site engineers use to practise for technical interviews, GPT-3.5-turbo solved 92% of easy problems, 79% of medium and 51% of hard ³. Pasting the failed test cases back in a second prompt improved results by 38 to 60%.

The gradient is real, and it is a slope rather than a wall.

Read those figures for direction only. LeetCode problems are public and algorithmic, correctness was checked by submitting them rather than by reading the code, and the model tested has since been superseded. The property they share with the design canon is the one that matters: the answers were published before the candidate arrived.

A scenario invented for one candidate also stops being one test.

Improvising it does not put the round outside the rules the scored parts of your process answer to. The Uniform Guidelines, adopted by four US federal agencies in 1978, define a selection procedure as any measure or procedure used as a basis for an employment decision. The definition runs from performance tests through 'informal or casual interviews and unscored application forms' ⁴, and a separate section puts a validation burden on a procedure only where adverse impact shows up.

So under those Guidelines, the puzzle you thought up on the walk to the room counts as a selection procedure exactly as the scored assessment down the hall does, and it leaves nothing behind to compare.

Obscurity was never the target anyway. A good question is one a prepared stranger cannot answer, and a stranger who has read everything published about your problem is still a stranger to the number you are held to.

Anchor the round to a constraint from a system you run

Take a constraint off a system you own and put it in the prompt. A latency budget you are actually held to. A monthly bill somebody actually signs. A migration you finished last year, and the reason you did it that way. A queue that backed up on a Tuesday. Anchors like those cost nothing to produce, because you already paid for them.

Content relevance moves the number, and one measured comparison sits behind that. Sackett and colleagues report a job knowledge meta-analysis of 164 studies with a mean observed validity of .22. The 59 studies whose tests were built for the job in question came in at .31, rising to .40 once corrected for unreliable performance ratings ⁵.

The comparison runs between a whole meta-analysis and one subset of it rather than inside an experiment, and job knowledge tests assume candidates who already hold the knowledge. Carry the direction only: in that comparison, what the test was about changed what it predicted.

A public postmortem is the wrong place to shop for an anchor.

The famous outage write-up is published, which is the property that broke this round. Your own incident board is not. It holds what a public postmortem rarely does: what you decided afterwards, what you decided against, and the constraint you still live with because of it.

An anchor is usable when it carries three things.

• A number in the unit your dashboard uses. A p99 target in milliseconds on a named path, not the sentence that it has to be fast.
• What the number is a number of. Requests a second at peak and off peak, because the ratio between the two is where the design lives.
• The decision it forced. What went in, what you gave up to put it there, and the month you stopped defending the choice.

Reuse the same anchor for every candidate on the role, and change it when the system changes. Reuse makes two candidates comparable, and it tells you the anchor has gone stale when a fourth person walks it in three minutes. Twelve interview questions that survive an AI-prepped candidate runs on the same rule: fix the question, spend the round on the follow-up.

What a candidate says when the number moves

A real reversal sounds like something coming out of the design, named, with a cost attached. Halfway through, move the constraint: cut the budget, triple the read volume, take the second region away, or say the bill has to fall by half.

A candidate who understood their own design removes a component and says what gets worse. A candidate reciting one adds a component and names no cost.

Three answers are worth writing down after you move the number. Which part of the first design is now wrong, and why it was right before. What they would measure to know whether the replacement holds. What they would refuse to change until they had seen that measurement.

No canonical version exists for any of the three, because the reversal depends on which number you moved.

The AI question lands inside the reversal without needing a round of its own. Ask what they would hand to a model at that point and what they would keep.

Useful answers are specific and slightly unflattering: generate the migration script, distrust the capacity estimate, because a model has no idea what the traffic looks like on the first Monday of the month.

Listen for the line being drawn rather than for enthusiasm in either direction. Reading the answer about what a candidate won't delegate is the same read run on a different prompt: the boundary is the claim, and the reason behind it is the evidence.

Parochialism is what the anchored round costs you. Someone who has never run traffic shaped like yours can be genuinely strong and still miss the thing your team learned the expensive way.

You will hear the miss as a weak answer, because you already know the ending. Anchoring a round to your system anchors it to your history too.

Write the reversal's anchors before the session, not the debrief

Two written anchors per dimension, drafted before anybody sits down, are what keep the second half from being a mood. Decide in advance what a strong reversal looks like and what a weak one looks like, in observable terms: removed a component and named the cost, against added a component and named none.

Interviewers who write the strong and weak versions first are scoring behavior.

Written anchors buy something measurable, and less than their popularity suggests. Levashina and colleagues' review reports Taylor and Small's 2002 meta-analysis of 19 past-behavior interview studies. Interviews using anchored rating scales showed higher criterion-related validity, .35 against .26, and higher agreement between raters, .77 against .73 ².

Those 19 studies all used tell-me-about-a-time questions, so the figures say nothing direct about a design problem, and the comparison runs across studies rather than testing one interview with the anchors and without.

The reliability gain is small. The validity gain is the larger of the two.

Anchor the reversal, not the design. Named a component to remove and stated the cost is something two interviewers can mark the same way. Showed strong systems thinking is not, and it is the phrase that turns a debrief into whoever speaks first winning.

Every number on the scale needs an observable behavior applies here more than anywhere, because the reversal is the only part of this round with no published answer.

The blog post is still in the room, and it should be. Somebody who has read the standard answer to a sharding problem starts from a better place than somebody who has not, exactly as they always did.

What changed is that the recital now earns a few minutes of the round. Spend the rest on the number you moved.

See how it works >