The Paste Is the Job Now. Interview the Command After It.

What should an operations round measure once the paste is normal?

Measure the decision that follows the answer, not the search for it. The federal occupation file for network and computer systems administrators lists, among its tasks, two that matter here: diagnose, troubleshoot and resolve hardware, software or other network and system problems, and perform data backups and disaster recovery operations ¹.

Pasting a trace sits inside the first of those tasks. The second is where a candidate gets tested.

Google's example error budget policy, published in the 2018 Site Reliability Workbook, puts a figure on where operations breaks: changes are "a major source of instability," representing roughly 70% of outages in Google's production systems ².

One company's accounting of its own incidents is not an industry rate. The figure still shows where one large operator found its risk concentrated.

The risk sits in the change.

So grading the paste gets the sequence backwards. A trace dropped into a chat window changes nothing on any host. A command copied out of the reply changes a running system, and a round built around tool use has said nothing at all about the interval between the two.

The four questions between the paste and the command

Four questions cover the boundary, and they take about six minutes. What did you look at before you pasted anything. What do you expect this command to change. What else does it touch if the diagnosis is wrong. How do you put the system back.

Ask them about a specific incident the candidate brings, then again about a case you supply.

The first question has published practice behind it. The 2016 O'Reilly volume Site Reliability Engineering, edited by Betsy Beyer, Chris Jones, Jennifer Petoff and Niall Richard Murphy, puts the instruction plainly. "Consider the obvious first: perform the tests in decreasing order of likelihood, considering possible risks to the system from the test" ³.

Risk from the test, before the test.

The same chapter carries the warning that follows from it: "Active tests may have side effects that change future test results" ³. In an outage that is the log truncated to free a disk, carrying off the evidence for the actual fault.

Question two separates a diagnosis from a guess. A candidate who can say what the command should change (a service that comes back, a queue that drains, a mount that appears) holds a model of the fault.

Answering that it should fix the error hands back the assistant's sentence with nothing behind it.

Question three scales with seniority. A junior answers with the host. A senior answers with what else shares that host, the dependency that will retry, and the customer-facing thing that goes quiet once the retries pile up.

Question four asks for an artifact, and a weak answer is easy to hear. Strong answers name something that existed before the change: a snapshot, a copy of the config file with a timestamp, an exported rule set, a colleague already on the call. An answer that promises care names nothing at all.

The follow-up is the one that works when every candidate says they always double-check: ask for the last time it happened, and what the check returned.

Build the case where the model's fix is fast and irreversible

Write one scenario where the returned command is quick, reads correctly, and destroys something. Disk full on a host that is also the only place a service writes its audit log. The suggestion truncates the largest file and the alert clears in four seconds.

The evidence for the actual fault leaves with the truncated file, and no snapshot ran that morning.

The design rests on a measured effect. In a pre-registered 2023 field experiment, 758 Boston Consulting Group consultants worked realistic tasks with and without GPT-4. Half of them, 373 consultants, drew a task deliberately placed outside what the model could handle, and on that task the ones using GPT-4 were 19 percentage points less likely to reach the correct answer ⁴.

On that task, more coaching went with a worse result. The group given a prompt-engineering overview fell 24.5 points against the control; the group given none fell 14.5 ⁴.

Against a model from 2023, on a single task in a single sitting, the rate does not transfer to an outage. The failure mode does. Many of the consultants could not tell which side of the capability line the task sat on, and a returned command at 02:00 does not say which side it sits on either.

Operations makes the same error expensive in a particular way. A wrong answer in a deck can still be edited before anyone acts on it.

A wrong answer typed into a terminal has already changed the system by the time somebody reads it again.

Elsewhere this exercise runs as testing whether a candidate notices when AI gets something wrong, which plants a defect in the material and waits. The operations version plants a fix, and the difference is that the fix works.

Hand every candidate the same returned answer, printed.

A live session gives each person a different reply from a different model on a different day, which leaves the round with no shared stimulus and two candidates nobody can compare.

How do you tell reversibility from recall in an answer?

Score what the answer names, and recall separates itself. A recall answer explains what the error means and what fixes it. A reversibility answer adds the state that existed before the change and the route back to it. Both can be correct about the fault.

Only the reversibility answer tells you what happens on the night the fault was something else.

Do not take a candidate's own account of what the tool saves them.

METR ran a randomized trial in 2025 in which 16 experienced open-source developers completed 246 real tasks on repositories they had worked in for about five years. They forecast a 24% speedup. They were measured 19% slower ⁵.

Afterwards they still believed they had gained 20% ⁵. Sixteen developers in one setting is not a general finding about AI and speed, and none of that work was operations.

The developers in that trial had about five years in those repositories, and their estimate still pointed the wrong way. A candidate at an interview table is working from the same kind of memory, so the round needs something to score against that is not their own confidence.

Three anchors, written before the round so two interviewers can fill them separately:

• Evidence before the command. Names at least one read-only check and what it would have shown. Missing if the first move is the suggested command.
• Blast radius, stated. Names what else the command reaches: the co-tenant service, the replica, the queue that retries, the backup window that starts at midnight.
• Route back, named. A snapshot, a saved config, a change record with a rollback step, a second person on the call. A promise of caution is not a route back.

Pace is the wrong thing to grade here. The general question of how fast someone works with AI, or how well they judge its output has a sharper answer in operations: the assistant is available to everyone in the pool now, and the undo is not.

Name the suggestion they refused to run

One question does more than the rest: ask about a time the model handed back something they chose not to run, and what they did instead. The answer has to contain a specific system, a reason that is not a feeling, and the second path they took. A candidate with no such story may never have held production access, or may work somewhere the tools are blocked.

A prepared candidate arrives with a story about a time the model was wrong, so how hard the catch was does more work than the fact of a catch. A syntax error the linter would have caught is not a catch. A plausible command declined because the host had no snapshot from that morning is.

Ask who else knew before the change went in. Operations judgment includes when somebody wakes a second person, and naming the human who checks it is a design the candidate inherits from the team.

A supplied scenario is the right shape for all of this, and there is an argument in print for it. Researchers building an AI literacy assessment for a US Navy robotics training programme reported that, as a measure of applied AI literacy, a scenario task simulating AI use on the job outperformed the knowledge tests they had adopted or written themselves ⁶.

The abstract reports no sample size and no margin, so it supports a design argument and nothing firmer.

The case against this round is exposure. A candidate whose career has run entirely through infrastructure as code, pull requests and a change window may never have had the chance to run an irreversible manual command, and the question then measures where they have worked.

The next trace lands at 02:00 and goes into the model, the same as the last one. What the round can tell you is how someone reasons about the seconds after the reply comes back: whether a read-only check comes first, whether the other tenants on that host get named, whether a snapshot exists to go back to.

Hire for the pause. The paste was never the part to grade.

See how it works >