An Agent Training Environment Lead Is Hired for the Grader, Not the Task

Your Agent Scores 91 Percent in the Sim and Still Cannot Finish a Refund

The browser agent clears ninety-one percent of the task suite your team assembled in March, and last week it failed a real refund because a confirmation dialog moved two hundred pixels. Nobody could say whether the suite was wrong or the agent was, because nobody owned the suite. That question has an owner now: an Agent Training Environment Lead builds the worlds agents practise in and is accountable for what a passing score means.

Three traits separate a real one from a person who writes task descriptions. The first is that the grader comes first. Ask a candidate to walk you through a task they built, and listen to the order they tell it in. A weak answer describes the scenario at length and then says success is checked at the end. A strong answer starts with the assertion, in the form of a query against the environment's final state, and only then describes the setup that makes that assertion meaningful. People who write the grader last write graders that check for a string in the output.

The second is an adversarial reflex about their own work. The useful interview question is not how they would build an environment. It is how an agent would pass their environment without doing the work. Real candidates answer this instantly and with relish, because they have been embarrassed by it: the agent that deleted the failing row instead of fixing it, the one that found a debug endpoint, the one that discovered the checker only compared totals. Someone who has never been beaten by a model on their own task has not shipped one.

The third is treating environments as production software rather than as content. Seeded fixtures so a run is reproducible, hermetic execution so a flaky network does not read as a model failure, version pinning so this month's score is comparable to last month's, and a teardown that actually tears down. The tell here is a question about determinism. A candidate who asks whether your environments are deterministic, and looks unhappy at the answer, is thinking about the right problem.

The category is still forming, and that is worth saying plainly rather than pretending the title is settled. The visible demand today sits at the frontier data companies: Scale AI has posted a Data Acquisition Lead for Frontier Environments alongside a Machine Learning Research Engineer role on its Agent Data Foundation team, and Mercor has posted Research Operations and Strategic Projects roles covering the same ground ¹. Expect the title to differ at every company you approach. What is consistent is the deliverable.

Which Backgrounds Produce Someone Who Can Build a Grader That Holds?

The strongest feeder is test automation infrastructure. Someone who spent four years keeping a browser test suite green has already fought flakiness, already knows that a passing suite proves less than people assume, and has already written assertions against application state rather than against rendered text. That is most of the craft, and the people who did it well are usually undervalued in their current organization.

Game development is the second, and it converts better than its resume suggests. Level designers build spaces that constrain behaviour, playtest against people who try everything except the intended path, and iterate on difficulty until it teaches. Studio tools engineers build the deterministic replay and headless simulation that this work needs in exactly the same form. If you want the fuller version of that overlap, it runs through the same instincts described in hiring a game AI designer.

The unexpected feeders are the ones worth hunting. Competitive programming problem setters spend their evenings writing test cases designed to defeat plausible wrong solutions, which is grader design under a different name. Capture-the-flag challenge authors think in exploits by default. University teaching staff who maintain autograders for programming courses have years of experience with students who optimize for the checker instead of the subject, which is reward hacking with a tuition bill attached. Escape room designers understand that a puzzle with two solutions has one bug.

Domain experts belong in the pipeline too, on a different axis. An environment for an accounting agent is worthless if the seeded ledger does not look like a real ledger, and the person who can tell you that is a controller, not an engineer. The strongest teams pair a technical lead with rotating domain contributors rather than hiring a unicorn, which is a sourcing and quality problem more than a coding one, and it looks a lot like the work in hiring a data operations manager for human data.

Two profiles interview well and disappoint. Researchers who have only consumed benchmarks tend to underestimate the operational half, and are surprised that most of the job is fixtures, flake and vendor coordination. Content operations managers from the annotation era often bring excellent process discipline and no instinct for the failure mode where the measurement itself is broken.

Ask How They Learned That an Agent Was Gaming Their Environment

Ask how they got good at this and listen for practice rather than reading. The answers that matter describe a specific defeat: they built a task, an agent passed it, they looked at the trajectory, and the agent had not done the work. They can name the shortcut, name how they closed it, and name the check they now run on every task before it ships. That story is the qualification.

The habits underneath it are visible in how candidates use models on their own work. Good ones run an agent against a new environment before they trust it, several times, and read the trajectories rather than the scores. They ask a model to generate task variants and then hand-audit a sample, because generated tasks are cheap and generated graders are dangerous. Several keep a running file of exploits found in their own environments, which functions as institutional memory and is usually the most interesting artifact a candidate can show you.

The deeper skill is checking a claim against something outside the conversation. A model reports that the agent completed the ticket; the lead opens the simulated CRM and queries the record. A model summarizes a batch of runs as mostly successful; the lead reads six trajectories and finds two that succeeded for the wrong reason. This is the same discipline that makes evaluation trustworthy anywhere, and it is close to what an oversight specialist does at the model level rather than the task level, described in hiring an AI control and oversight researcher.

Press on the point where graders stop being mechanical. Ask how they would grade a task whose success is partly a judgment call, such as whether a drafted customer email was appropriate. Watch for whether they reach immediately for a model as judge without mentioning its failure modes, and whether they can describe when a rubric with human spot checks beats an automated check that is confidently wrong. Watch also for whether they know when to refuse: some tasks should not be in a training suite at all, because no honest grader exists for them yet.

One caution about format. This subject rewards vocabulary, and a candidate saying reward hacking, hermetic execution, trajectory review may have shipped forty environments or read three papers. The transcript reads the same. Give them a small real environment of yours with a deliberately weak grader and ninety minutes to break it. The gap between the two groups shows up in the first twenty.

Look for Environment Leads in QA Automation and Benchmark Repositories

Go where people have already built one in public. The open agent benchmarks have contributor histories you can read: SWE-bench, WebArena and similar task suites are maintained by identifiable people who have argued in issue threads about whether a task was fairly graded. That argument is the interview, already conducted, on the record. Anyone who has filed a bug against a benchmark's grader is worth a message.

The frontier data vendors are the other concentration, and they are hiring against this directly ¹. That cuts both ways: it is where the experienced people are, and it is who you are bidding against. Beyond those, look at test automation and SDET teams inside large software companies, tools and pipeline engineers at game studios, and simulation engineers from robotics and autonomous vehicles, who have spent careers on the question of what a simulated pass predicts about real behaviour.

What closes this hire is rarely the base number. It is whether their environments will actually be used. The failure story every experienced candidate carries is a suite that got built, demoed once, and never entered a training or release loop. Name the run it feeds. Name the person who decides when the suite blocks a launch. If the honest answer is that nothing blocks anything yet, say so and describe what would have to be true, because a candidate who takes the job on a fiction will leave inside a year.

Three other things move offers. Compute access, stated concretely rather than as a promise to figure out. Permission to publish some part of the work, since much of the field's reputation is built in the open and refusing that is a real pay cut in this pool. And headcount for the domain contributors, because a lead who has to personally write the accounting fixtures will be doing that instead of building the platform. Where an internal program manager already coordinates this kind of cross-functional pipeline, the pairing described in hiring a game production AI program manager is the shape that tends to work.

What Does an Agent Training Environment Lead Cost, and Can It Be Remote?

No wage series covers this title, and no salary survey found for this piece prices it, so any point estimate quoted today is a guess wearing a benchmark's clothes. Price it against a band you already run. Where the person writes the graders and the runner infrastructure, they hire against your senior machine learning or infrastructure engineer band. Where the job is mostly sourcing environments and coordinating domain contributors, it hires against your senior technical program or data operations band.

The macro pressure behind both is documented even where the title is not. PwC's 2026 AI Jobs Barometer, analysing roughly one billion job advertisements, reports an average wage premium of sixty-two percent for roles requiring AI skills ². Treat that as directional for the band, not as a number for this job.

One honest market caution: levelling is inconsistent across companies, so a candidate's current title tells you very little. Ask what they were allowed to ship without review, and whether their environments gated anything. That answer places the band.

On location, the work is remote-friendly by construction, since environments run in containers and trajectories are reviewed asynchronously. The parts that resist are calibration and domain sessions: two people writing graders in isolation drift within a quarter, and the fastest way to build a realistic fixture is usually a few hours in a room with someone who does the real job. Teams that run this well are distributed with recurring in-person weeks rather than fully synchronous.

On-premise requirements show up in a specific and predictable place. When an environment mirrors a customer's real systems, or is seeded from real operational records, it inherits that data's handling rules, and the environment then has to run inside the same boundary the source data does. Scope that before writing the offer, because it decides which candidates can do the job from where they live. Where an environment is built from records about identifiable people, treat the seeding decision as a question for counsel in your jurisdiction rather than something to reason out from a summary, and prefer synthetic fixtures where they will hold.

See the benchmarks >