Hand an Ops Candidate a Runbook That Fails Quietly

Why Does a Drafted Runbook Read Right and Still Fail?

A drafted runbook reads right because it is fluent by construction, and fails because it is complete only by accident. A model writes the path where everything works, in the register of a procedure that has already been through review. The sentences that would have said what to do when a step fails were never in the source material.

Those sentences live in an organization's own incident history, and most of that history was never written down.

Three omissions do most of the damage.

The first is the missing confirmation. The page says restart the service, clear the exception queue, reissue the credit. Nothing on it says how anyone knows that worked.

The second is the missing rollback. Every step points forward, and no sentence begins with what to do if this one does not take.

The third reads specific and is not. Escalate if the backlog is heavy. Escalate if latency is high. Neither is a number anybody can look up at three in the morning, so nobody escalates until somebody senior happens to be awake.

What a procedure prints changes what gets skipped, and there is an old estimate of how much.

In their 1990 NASA report on flight-deck checklists, Asaf Degani and Earl Wiener carry a figure from nuclear power plant human-reliability work. An item with no check-off provision runs a per-item error probability of about one in a hundred, against three in a thousand where a check-off exists ¹.

The estimate is old, it describes cockpits and control rooms rather than a warehouse or a billing queue, and it was never measured on an ops runbook. Take the direction and not the decimal.

A line that has to be ticked is a line that gets skipped less often.

A drafted page hides its own gaps because people are poor at seeing where a model stopped being reliable. In a 2023 field experiment, Harvard Business School gave 758 Boston Consulting Group consultants a set of tasks with one of them placed beyond what the model could handle ².

On average, the consultants using the model did not see the edge. On that task the control group was right 84.5% of the time, and the two groups working with GPT-4 averaged 19 percentage points lower ².

The coaching ran the wrong way too. The group handed a prompt-engineering overview finished further behind the control than the group handed none ².

The whole result comes from one task, one sample, one 2023 model that no longer exists in that form. What holds is narrower and enough: the people using it could not tell which side of the line they were standing on.

Write the Gap In Before You Send the Page

Take a runbook you already run, cut it to one page, and remove exactly one thing. Removing one gives you an answer key, so you know the right answer before anybody sits down, and so does the second grader. Removing three gives you a puzzle rather than an answer key.

Use a procedure you actually run rather than an invented scenario. The details that make a page believable are the ones nobody thinks to invent, and a candidate who runs floors can tell. A work sample is better drawn from a real task you already solved for the same reason.

Three cuts are worth keeping in rotation, one per opening, so the answer does not travel between candidates.

• The confirmation. Delete the line saying how anyone knows the change took. Leave the change itself intact.
• The rollback. Keep every forward step and delete the one sentence about backing out.
• The threshold. Replace a number with a word. "Escalate if the queue is backed up" survives a fluent reading and cannot be acted on.

A fourth cut is the owner. Name a role in the escalation line that does not exist at two in the morning, then watch whether the candidate asks who is actually awake.

Write down what a correct answer says, in one line, before the first session. That is the answer key you write before you send the assignment. The key is the whole reason two people who have never worked with a model themselves can grade this and land in the same place.

Say in the brief that the page was drafted by a model and edited by a person. It is true of the page the hire will inherit, and nothing in the exercise depends on hiding it.

Walk It Against One Bad Day

Give the procedure a specific bad day and let the candidate read it aloud against that day. Pick one that has already happened to you: the vendor API returned success and did nothing, the export ran twice, the queue drained into the wrong account. Twenty minutes is enough, and the instruction is one sentence long.

Walk this page start to finish as if today is that day, and stop wherever it stops telling you what to do.

Then say nothing. The silence is the measurement. A candidate who is going to ask what happens when the restart does not take will ask without being prompted, so note whether and when that question comes.

Decide before the session whether an interviewer may help, and hold that line across every candidate. Same page, same bad day, same silence.

What makes an interview structured, in the federal Office of Personnel Management's guide, is the same questions in the same order, a common rating scale, and agreement in advance about what an acceptable answer looks like ³.

The engineering version of this exercise plants its defects in a repository instead of on a page, and the grading problem is identical: what you seed is what the exercise can test.

Let them open a model during it if they want to.

Watching somebody paste the runbook into an assistant and then argue with what comes back is worth more than watching them work unaided.

What Separates a Real Catch From a Style Note?

A real catch names the step, the failure the step does not cover, and what a person should do instead. A style note says the page could be clearer. Both sound competent read aloud, and only one changes what happens on the floor.

Score the first, write the second down, and leave it out of the count.

Three columns, filled in separately by two people:

• Found the missing check. Yes or no, against the line you deleted.
• Asked about failure, not success. The question they asked, quoted.
• Named who and when. A person or a role, plus a number somebody could look up.

Each column asks for a behavior you can point at rather than an impression, the same discipline as giving every number on the scale an observable behavior. Two graders can disagree about whether a candidate is thorough. They cannot disagree about whether the words "what confirms the queue drained" were said out loud.

The scoring side is the side even published research most often leaves out. A content analysis of 104 interviews reported in the literature between 1997 and 2010 found that studies calling an interview structured used an average of 5.74 of 15 recognized structure components ⁴.

The rarest were the evaluation ones. Detailed notes appeared in 19% of those interviews ⁴.

Read that as a ceiling rather than a floor, since it measures what researchers built into studies they chose to publish, and practice is unlikely to be more structured than the literature.

The runbook you handed out is missing its verification step on purpose. Most of the interviews in that count did not report theirs, and nobody planted that gap.

None of this makes the exercise a strong predictor on its own.

A 2022 re-analysis by Paul Sackett and colleagues puts work sample validity at .33, well under the .54 that circulated for decades ⁵.

The studies behind that estimate mostly tested people already doing the job, so it is not a promise about applicants. One exercise on one afternoon is one observation, and it belongs beside the rest of the round rather than in front of it.

Where the Exercise Stops Telling You Anything

The exercise stops telling you anything at the candidate who has never seen your systems. Somebody who cannot picture the queue, the ledger or the shift handoff will miss a gap a peer catches in ten seconds, and missing it says almost nothing about how they read a procedure.

Credit the question they asked instead of the catch they didn't make, and keep the two apart in the notes.

A missed catch can be a knowledge gap rather than a verification failure, and the two are worth telling apart before anyone gets rejected over it.

Catching an error on paper and catching one at two in the morning under a customer's phone call are different acts. The exercise measures the first and says nothing about the second, which is the shortest argument against running it at all.

The page you hand out has one gap in it, and you know where it is. The page your team ran last Tuesday came out of the same drafting tool, and nobody has counted its gaps.

Hire the person who counts.

See what gets scored >