An AgentOps Engineer Is Hired to Notice What Your Dashboard Cannot

Silent Failure Is the AgentOps Engineer's Whole Job

Your support agent stopped escalating. Not loudly: the escalation tool started returning a 200 with an empty body after an API change, the agent read that as success, and for nine days every angry customer got a polite closing message instead of a human. Dashboards were green. Latency was fine. Cost went down, which looked like a win. That is the failure class an AgentOps engineer exists to catch.

Three traits separate a real one from someone who has watched a dashboard, and each has a tell you can check inside an hour.

The first is instrumenting before building. Ask what they logged on their first agent project. A real answer names the unit: a full trace per run, carrying tool inputs and outputs, model version, token count and the parent span, kept long enough to replay a week later. A performed answer names a vendor and stops.

The second is a taste for the boring failure. Push past the interesting cases (a fabricated citation, a prompt injection) and ask about the dull ones: a rate limit that arrived as a truncated response, a retry that duplicated a write, a tool that returned an error string inside a success status. Anyone who has carried a pager for an agent has stories in this register, because the boring failures are the ones that stay hidden.

The third is an opinion about what should never have been an agent at all. The strongest candidates argue for deleting agents. They can name a step they moved back into ordinary code and the number that justified the move.

The tell running through all three is that a real candidate asks how you find out. Describe your architecture and they keep returning to detection until you either name the alert or admit there is none. That instinct also marks the boundary with the AI business process consultant, who decides which process an agent should touch in the first place.

The substrate is familiar reliability work: service levels, error budgets, rollback, on-call. What is new is a component that degrades without breaking, so the alert has to be built out of behavior rather than exceptions. Accenture's live posting for the title lists agent accuracy, latency, availability and exception rates side by side with release rollback and cost visibility, which is the problem written down as a job requirement ¹.

Which Backgrounds Produce an AgentOps Engineer Who Can Debug a Fleet?

Site reliability engineers convert fastest, because most of the job is on-call discipline wearing new vocabulary. The unexpected feeders are better than the obvious ones: data engineers who have babysat pipelines fed by dirty upstream data, QA leads who built test rigs for flaky systems, and platform engineers who ran an internal developer platform nobody thanked them for.

The SRE case is easy to state. An agent is a dependency that answers wrongly with perfect grammar, and reasoning about partial, slow or incorrect dependencies is the entire craft. What an SRE has to add is statistical thinking: the same input produces different outputs, so a single reproduction is not a bug report and a green test run is not a release gate.

Data engineers bring the second half, and they are undervalued. Someone who has run a nightly pipeline against a vendor feed that changes shape without warning already understands drift, already writes assertions about data rather than code, and already knows that the worst outages are the ones where the job succeeded. That is the same instinct an eval suite needs.

QA is the third, and the most overlooked. A test lead who built a test rig for a flaky mobile app has thought harder about non-determinism than most backend engineers have. Agent evals are test design with a fuzzy oracle, and people who have written good test plans transfer better than people who have written good prompts.

Two profiles read well and often disappoint. Research-heavy candidates tend to reach for a better model when the fix is a tighter tool definition and a smaller scope. And pure prompt-engineering resumes usually stop at the boundary where production begins: no deployment story, no cost accounting, no pager.

On the question teams actually type into a search box, whether DevOps can just cover this: sometimes, for one agent. The trouble starts at scale. Active agents in the Microsoft 365 ecosystem grew fifteen times year over year, rising to eighteen times in large enterprises ³, and Deloitte's 2026 review describes IT leaders who are now managing fleets of agents rather than instances, with AI architect roles expected to nearly double from 30 percent of organizations to 58 percent over two years ⁴. A fleet needs someone whose job is the fleet. Where those agents talk to customers, that person works next to an AI support agent manager rather than instead of one.

Ask How the AgentOps Candidate Built Their Own Eval Habit

Ask directly: how did you get good at this? The answer worth hearing describes practice rather than coursework. Strong candidates built something with an agent, watched it produce a confident wrong answer, and changed how they work. They can name the moment, name the wrong answer, and name the check they now run every time because of it.

Good answers share a shape. Someone describes running the same task twenty times to see the spread rather than once to see it work. Someone else describes writing the eval before the feature, because a task with no grading rule can be admired but not improved. A third keeps a folder of transcripts where the agent went sideways, which is the closest thing this discipline has to a lab notebook.

The skill underneath all of it is checking a claim against something outside the conversation. A model asserts that an endpoint returns a field; the engineer opens the schema. A model proposes a rollback procedure; the engineer runs it in staging before writing it into the runbook. This habit survives an interview question badly, because describing it is easy and performing it is not, and it shows up in a work sample within minutes.

Press on eval design specifically. Ask what they would measure to know an agent got worse, and listen for whether they separate a regression in the model from one in the prompt, the tools, the retrieval corpus or the traffic mix. The job board that publishes AgentOps role definitions puts eval gating in continuous integration at the center of the description, alongside tracing for tool-using, non-deterministic agents ². A candidate who treats evals as a reporting artifact rather than a deploy gate has not run this in anger.

One warning about format. Whiteboarding an agent architecture rewards vocabulary. A candidate saying "supervisor pattern, structured tool schemas, span-level tracing" may have run three systems or read one post, and the transcript reads identically. Hand them a real trace from a run that went wrong, with the logs, and two hours.

Find AgentOps Engineers Where Agent Postmortems Get Written

Look where agent postmortems get written rather than where agent launches get announced. Observability and eval tooling communities, issue trackers for orchestration libraries, incident-review write-ups from companies that shipped an agent and said what broke. Adjacent titles worth approaching: SRE, platform engineer, data reliability engineer, and whoever currently owns the on-call rotation for a machine learning service.

The issue-tracker route is cheap and underused. A person who filed a detailed bug against a tracing library, with a reproduction and a span dump, has demonstrated more than a portfolio page does. So has anyone who published a postmortem about an agent that misbehaved in production. That population is small and self-selecting, and it is reachable by a specific email about the specific incident they wrote up.

Feeder companies are the consultancies and managed-service arms that run other people's agents, because that is where the title already exists as a requisition rather than an aspiration ¹. AI-native startups produce the other half: engineers who built the eval suite because nobody else would.

Closing follows a pattern, and so does losing. The offer dies when the scope is reliability theater: a dashboard to build, no authority to block a release, and a product team that ships prompt changes on Friday afternoons. It dies again when the candidate learns in week two that there is no staging environment for the agent, so every experiment is a production experiment.

Three things close the hire. Give the role a stated mandate to gate deploys on evals, not just to report on them. Name the budget for tokens and latency, and say who owns it when it is exceeded. And answer honestly about what the agent is allowed to do without a human approving, including what happens on the day the answer changes. Candidates ask because they have watched projects stall on exactly that question.

What Does an AgentOps Engineer Cost, and Can the Role Be Remote?

No government wage series covers the title yet, so treat any single number as a negotiating band rather than a benchmark. As of mid-2026, one agentic-engineering job board publishing ranges synthesized from aggregated public postings puts United States total cash at roughly 150,000 to 200,000 dollars for mid-level AgentOps engineers, 185,000 to 250,000 for senior, and 240,000 to 330,000 at staff level and above ².

Two cautions about those figures. They come from a job board's own synthesis of public postings rather than from a survey or a wage series, so the sample is whatever was advertised and the midpoints move with the market. And they describe United States total cash including equity, which makes them close to unusable for comparing against a base-salary band in another country. If you need a defensible internal number, price the role against your own senior platform or SRE band and add whatever premium your last two AI offers actually required.

The demand pressure behind the band is not in dispute even where the numbers are. Accenture posts the title in its AI managed services arm today ¹, and the agent counts that create the work are still climbing steeply ³. That combination usually means candidates have options and your process speed matters more than your midpoint.

On location, the work is remote-friendly and often has to be, because on-call for a distributed system does not care where the pager rings. The part that resists remote is the first ninety days: learning which of your agents matter, which alerts your team already ignores, and which product manager will argue about a blocked deploy. Teams that hire fully remote and skip a concentrated onsite stretch tend to get a person with excellent dashboards and no political standing to stop a release.

On-premise constraints show up in regulated environments where the model runs inside your boundary. That changes the role more than the location does: self-hosted inference, GPU capacity planning and latency budgets join the job, and the candidate pool narrows to people who have operated models rather than only called them. In healthcare that overlap runs straight into the data platform, which is why the FHIR and ML platform engineer is often the adjacent hire and sometimes the same hire.

One compliance note worth flagging without pretending to legal advice. Where an agent takes or materially shapes a decision about a person, several jurisdictions now impose notice, explanation and record-keeping duties, and the rules differ by jurisdiction and are changing. Whatever your AgentOps engineer builds for tracing is also the evidence trail those duties assume exists. Check with counsel in your jurisdiction rather than reasoning from a summary.

Read the evidence >