An AI SRE Is Hired to Overrule the Investigation Agent

The AI SRE Exists Because Your System Failed Politely

At 3:12am the retrieval index behind your support assistant went stale. Nothing crashed. Latency held, the error rate stayed at zero, and the assistant answered every question with total composure using last month's refund policy. No pager fired, because nothing being watched was broken. That is the class of failure an AI SRE is hired for, and it is why an on-call rotation built for exceptions covers none of it.

Three traits separate a real candidate from someone who has run a monitoring stack, and each has a tell you can check in a single conversation.

The first is treating a model-backed component as a probabilistic dependency rather than a flaky one. Describe an intermittent bad answer and watch what they ask for. A real one asks how often, across what inputs, compared to what baseline, and whether anyone has run the same request enough times to see the spread. A performed one asks for the stack trace and then asks which model you use.

The second is what they do with a diagnosis they did not produce. Reliability teams at large companies now run agents that investigate incidents on their own and hand the responder a written explanation ². Ask what they do with that paragraph. The answer worth hiring names the evidence: which logs the agent actually read, which it inferred, and what one cheap check would falsify it. The answer to worry about is that it saves time.

The third is a willingness to be unpopular at the moment it costs something. Ask about a time they turned a feature off, and press for the conversation afterward. People who have done it remember who objected. People who have not describe a process.

Running underneath all three is a habit of asking how anyone would find out. Walk a real candidate through your AI feature and they keep circling back to detection until you either name the alert or admit it does not exist.

Which Backgrounds Produce an AI SRE Who Can Judge a Diagnosis?

Site reliability and platform engineers convert fastest, because error budgets, on-call discipline and blameless review transfer whole. What they have to add is statistical thinking: one reproduction is not a bug report when the same input yields different outputs, and a green test run is not a release gate. The best of them find that adjustment interesting rather than insulting, and that reaction is worth watching for.

Data reliability engineers are the strongest non-obvious feeder. Anyone who has run a nightly pipeline against a vendor feed that changes shape without warning already writes assertions about data rather than code, already thinks in drift, and already knows the worst outages are the ones where the job reported success. Retrieval staleness is that problem wearing new clothes, which is also where this role starts overlapping the context engineer who owns what the model is allowed to see.

Test engineering is the third, and the most overlooked. A person who built a rig for a flaky mobile app has thought harder about non-determinism than most backend engineers, and evaluating a model's output is test design against a fuzzy oracle. Where that work becomes continuous rather than pre-release, it belongs to an agent quality analyst instead, and the two roles argue productively about where the boundary sits.

One genuinely unexpected background: operators from process control, dispatch, or clinical monitoring. They arrive fluent in alarm fatigue, in the difference between an alert and an interruption, and in what happens to a human who has been told six times that the automated recommendation is usually right. That is exactly the failure mode of an on-call rotation restructured around AI-generated diagnoses, and almost nobody from a software background has lived it.

Two profiles read well and often disappoint. Research-leaning candidates reach for a better model when the fix is a smaller scope and a tighter tool definition. And infrastructure generalists who have never owned a correctness question will happily keep the AI feature at four nines of availability while it gives wrong answers all quarter.

Ask How the AI SRE Candidate Got Good With Their Own Agents

Ask it plainly: how did you get good at this. The answer worth hearing describes a change in practice after a specific embarrassment, not a course. Strong candidates let an assistant write a diagnosis, acted on it, watched it be confidently wrong, and now run a check they can name every time because of that morning. They remember the wrong answer.

Good answers share a shape. Someone kept a folder of incidents where the investigation agent's first hypothesis was wrong, then read the folder for a pattern and found one, usually that the agent over-trusted whichever signal was easiest to query. Someone else scored the agent against closed postmortems before letting it near a live page, and can say roughly how often it was right and, more usefully, on which kinds of incident it was reliably wrong. A third moved a step back out of the agent because the human check was cheaper than the review it required.

The underlying skill is checking a claim against something outside the conversation. The agent asserts a deploy correlates with the latency shift, and the engineer opens the deploy log instead of nodding. It is a habit that survives an interview question badly, since describing it takes eight seconds and doing it takes a real session with real ambiguity.

Press on what they instrument for the agents themselves. AI agents are now the single most demanded explicit AI skill in site reliability postings, named in 6.1 percent of them, ahead of both large language models and generative AI as terms ¹. That demand is not for someone who calls an agent; it is for someone who applies reliability discipline to the agent, which means a rate limit on its actions, a cost ceiling, a blast radius, and a rule for what it may do at 3am without waking anyone.

Avoid an architecture whiteboard here. It rewards vocabulary, and vocabulary is cheap. Hand over a real incident with the logs, a stale runbook and an agent-written summary that is partly wrong, then watch.

Find AI SRE Candidates Where Incident Reviews Get Published

Look where postmortems get written rather than where AI platforms get announced. SREcon talks and their hallway tracks, the Learning From Incidents community, issue trackers for OpenTelemetry, Prometheus and Grafana, and public incident write-ups from companies that shipped a model-backed feature and then said honestly what broke. A detailed bug report with a reproduction tells you more than a portfolio site.

The write-up route is cheap and underused. The population of people who have publicly described an AI feature failing in production is small, self-selecting, and reachable by a specific email about the specific incident they published. That message gets answered at a rate ordinary sourcing does not.

Adjacent titles to approach: site reliability engineer, platform engineer, data reliability engineer, observability lead, and whoever currently owns the on-call rotation for a machine learning service. Watch the title drift while sourcing, since the same job is posted as AIOps lead, reliability engineer for agent operations, and plain SRE with an AI paragraph buried in the requirements. Deloitte's 2026 review of the IT function names AIOps leadership among the roles organizations are actively standing up ³, which mostly means the work is being funded before anyone agrees what to call it.

Closing follows a pattern, and so does losing. The offer dies when the scope turns out to be babysitting: an agent that files tickets, a dashboard to maintain, and no standing to turn a feature off. It dies again in week two when the candidate learns there is no staging path for the model, so every experiment is a production experiment, and once more when they discover the investigation agent already executes remediations that nobody on the reliability team approved.

Three things close the hire. Give the role explicit authority to disable an AI feature on a correctness signal, in writing. Name who owns the inference budget and what happens when it is blown. And be honest about what the agents in your stack are permitted to do unattended today, including the parts you are not comfortable with, because the strong candidates ask that question early and can tell when the answer is being managed.

What Does an AI SRE Cost, and Does the Job Sit On-Premise?

No wage series covers this title, and no source found for this piece publishes a defensible band for it. So the honest answer is qualitative: price it against your own senior SRE or platform band, then expect to pay above the midpoint of that band, because the candidates who clear the correctness bar are usually already senior and usually have another offer. Any single number quoted for a title this new is a negotiating anchor wearing a benchmark's clothes.

What you can reason from is pressure rather than dollars. AI agents lead explicit AI skill demand in site reliability postings ¹, large reliability functions are already running autonomous investigation inside their incident process ², and the AIOps lead is being named as an emerging role rather than a niche one ³. Demand ahead of supply usually means your process speed matters more than your midpoint. Two weeks of scheduling gaps loses more of these candidates than five thousand dollars does.

On location, the work is remote-friendly by nature, since a pager does not care where it rings and the systems are distributed anyway. The part that resists remote is the first ninety days: learning which alerts your team has quietly agreed to ignore, which AI feature the CEO demos, and which product manager will argue when a release gets blocked. A fully remote hire with no concentrated onsite stretch tends to arrive with excellent instrumentation and no standing to use it.

On-premise constraints show up where the model runs inside your boundary for regulatory or data reasons. That changes the job more than it changes the address: self-hosted inference, GPU capacity planning, and latency budgets that are yours to defend rather than a vendor's. The pool narrows sharply to people who have operated models rather than only called them, and it starts overlapping the AI engineer hiring pool, so decide which of the two you are actually funding before the loop starts.

One governance note, offered as reasoning rather than legal advice. Where an AI feature shapes a decision about a person, several jurisdictions now impose notice, explanation and record-keeping duties, the rules differ by jurisdiction, and they are still changing through 2026. Whatever tracing your AI SRE builds for debugging is also the evidence trail those duties assume already exists, which is one reason the role often reports near an AI oversight director rather than purely into infrastructure. Check with counsel in your jurisdiction instead of reasoning from a summary.

See a sample report >