Hiring an AI Infrastructure Engineer to Own GPUs, Vector Stores and the Inference Bill

Why Does the Inference Bill Double the Month After Launch?

Because nobody owns the seam. The bill arrives as one line, finance forwards it to whoever seems closest to it, and that person spends a week discovering that three squads each wired straight to a vendor API, each with its own retry policy, each retrieving from a vector store somebody stood up in a hackathon. Call her the accidental owner. She is the reason this role exists.

What she finds is that the only lever anyone can reach is asking teams to please use the cheaper model. The job that would have handed her a real one is narrow and heavy. One gateway that every call passes through, so routing, retries and fallbacks are a policy rather than a habit. Capacity: reserved GPU hours, vendor rate limits, and the queue that decides which workload waits. Retrieval: which embedding model, how documents get chunked, when the index gets rebuilt, and what happens to the answers when it does. Caching at the prompt and the semantic layer, with a stated correctness cost. And attribution, so "the AI feature spent $41,000 last month" becomes a per-team, per-endpoint number a director can act on.

This is the MLOps job rebuilt around inference instead of training. Training pipelines, feature stores and experiment tracking still exist, but the daily pressure has moved to serving: latency budgets, token spend, vendor deprecations, and the fact that a model version change is a silent behavior change in production. Deloitte's 2026 survey work puts the direction plainly: the share of organizations with AI architect roles is expected to almost double, from 30 percent to 58 percent within two years, while the average share of tech budget going to AI rises from 8 percent to 13 percent ¹. Money at that scale grows an owner whether you hire one or not.

If your agents are already in production and misbehaving in ways no dashboard explains, that is a different and adjacent hire; see the agentops engineer for where the two roles divide.

What Separates a Real AI Infrastructure Engineer From a Resume Full of Model Names?

A real one talks about constraints and tradeoffs. Ask what they cut and they name a number, a method and a cost: cache hit rate went from 4 percent to 31 percent, spend fell by roughly a third, and two support tickets a week now come from stale answers. The performed version lists vendors, frameworks and vector databases, and goes quiet the moment you ask what broke.

The tells are specific. A strong candidate can explain why they picked a smaller model for one path and kept the large one for another, and can say what they measured to defend it. They have an opinion about evaluation that is not "we ran an eval suite": they can describe the golden set, who wrote it, and how often it goes stale. They know their retrieval quality is a data problem before it is a model problem, and they have re-chunked a corpus by hand at least once. They treat rate limits as capacity planning, with headroom and a degradation path, not as an error to retry.

Watch how they handle a version bump. Ask what happens when a vendor moves a model underneath them, and the useful answer arrives as a procedure with no drama in it: pin the version, run the new one in shadow against a saved set of prompts, compare, keep a rollback that does not require a deploy. It takes about forty seconds to say and it is the whole answer.

Then ask what they refused to build. Platform engineers who last kept the surface small, told two teams no, and can explain why the third exception was worth it. A candidate who has never turned down a request has not been trusted with a budget.

How Does Anyone Learn to Ration GPUs When No Course Teaches It?

Mostly by running an expensive system while somebody watched the bill. The skill is not taught anywhere yet, so the practice behind it is what you screen for: a period of ownership over serving infrastructure where cost, latency and correctness were in genuine conflict and the candidate had to choose. That is why so many arrive from SRE, from data platform teams, and from companies that ran their own inference before it was cheap.

How they use AI in their own work is the sharper signal, and it is easy to watch for. The good ones use models heavily and distrust them precisely. They will generate a Terraform module or a load-test rig in minutes, then say which part they read line by line and why: the IAM policy, the retry math, the thing that would be expensive to get wrong at three in the morning. They ask the model for the source of a claim about a vendor's rate-limit behavior and then check the vendor's own documentation, because they have been burned by a confident answer about a quota that did not exist.

Fluency with no check behind it is the pattern that costs money: fast output, plausible architecture, no account of what was verified. In a domain where one bad caching assumption is a correctness bug and one bad routing default is a five-figure month, verifying the claim that matters is the whole job. The same habit is what a software engineer working as an agent orchestrator gets hired for, and for the same reason.

Ask for the practice, not the philosophy. "Show me something you built with an assistant last month and tell me which line you did not trust" gets you further than any question about prompt technique.

Who Has Already Run a GPU Fleet While Somebody Watched the Bill?

In adjacent roles rather than under the title. The title is new enough that searching for it returns mostly people who renamed themselves last year. The reliable pools are platform and infrastructure engineers at companies that shipped an AI product, SREs who ran GPU fleets, data engineers who own a retrieval or search stack, and MLOps engineers whose training work quietly turned into serving work.

McKinsey's survey of AI adoption reports software engineers and data engineers as the most commonly hired AI-related roles, with MLOps specialists among the roles large organizations are notably more likely to be hiring than small ones ². Read that as a sourcing map: the people you want are currently employed as something else, at a company slightly ahead of yours.

The unexpected backgrounds are worth the extra screening time. High-performance computing and scientific computing people know GPU scheduling and queueing better than most web engineers ever will. Search and information-retrieval engineers understand why your embeddings are underperforming. Game backend and streaming engineers have real instincts about latency budgets and cost per session. FinOps practitioners who moved into engineering can build attribution nobody argues with.

Venues that actually work: conference talks and their speaker lists (KubeCon, Ray Summit, MLOps World), the issue trackers and Discord servers of the serving projects themselves (vLLM, Ray, LangChain, LlamaIndex, Weights and Biases), and the maintainers of open-source gateways and proxies. A person who has filed a substantive bug against an inference server has told you more about their competence than a resume can. Referrals from your own senior platform engineers outperform every channel here, because the pool is small and reputational.

Expect to Pay at the Senior Platform Band, and to Close on Blast Radius

No published salary series exists for this exact title yet, so treat any precise figure with suspicion, including the ones below. As of mid-2026, a recruiting analysis from Nexus IT Group, drawing on LinkedIn talent market data, puts average AI engineer pay at about $206,000, entry-level roles near $143,000, and senior engineers who have operationalized AI systems at scale at roughly $220,000 to $350,000 in total compensation ³. Postings for infrastructure-flavored versions cluster at the senior end of that spread.

Two cautions on using those numbers. They aggregate very different jobs under one label, from research engineering to application work, so the average is a weak anchor for a platform role. And they are US-weighted. Build the band against what you already pay senior platform and SRE staff, add for scarcity, and be ready to defend the delta internally, because the person you hire will sit next to engineers who did not get one.

The accidental owner from the first paragraph is often already in the building, and she is both the cheapest version of this hire and the easiest one to lose, because a year of cleaning up someone else's wiring teaches a person exactly what to ask for next. What closes her, or anyone else, is rarely the top of the band. It is blast radius: how many teams the platform serves, whether they own the budget or only report on it, and whether they can say no. Ask three candidates what killed their last offer and you will hear the same answers. Being hired to run someone else's architecture. A platform mandate with no authority attached. A vague reporting line into an executive who has no view of infrastructure, which is one reason the shape of your chief AI officer's remit matters to this hire. Being told the role is strategic and then handed a ticket queue.

On location: this work is remote-friendly by default, since gateways, quotas and vendor APIs are all reachable from anywhere, and the strongest candidates expect that. The exception is real. Regulated deployments, self-hosted GPU fleets and on-premise inference come with data-center access, hardware failure and physical security, and those roles are hybrid or on-site for good reasons. Say which one you are hiring for in the first conversation, because discovering it in week three ends the relationship.

Screen for Inference Judgment, Not Vector Store Trivia

Give them a scenario with a number in it and no clean answer. A retrieval-backed feature costs $60,000 a month, p95 latency is four seconds, and quality complaints are rising. Ask what they measure first, what they would change this week, and what they would refuse to change. The response separates judgment from recall within ten minutes, which no quiz on index types will do.

Good follow-ups are concrete and cheap. Hand them your actual architecture diagram and ask what they would delete. Ask them to price a feature before it ships and to say what assumption would make the estimate wrong. Ask what they would put behind a feature flag and what they would never put behind one. Ask how they would tell whether a model version change made things worse before a customer does.

Skip the take-home that asks for a small RAG app. It measures scaffolding speed, which every candidate now has, and it tells you nothing about the judgment you are actually buying. If you want work-sample evidence, make the sample the thing the job is: a cost and capacity decision with incomplete information, done in front of you, with an assistant available and its use in the open.

One last thing before the offer. Ask what they would want on day 30 to know the platform is healthy. The answer you are hoping for is short and it is about the seam: one dashboard with spend per team, one alert on quality regression, one number for cache effectiveness. When it comes back as a tool list instead, you have learned where the attention goes, which is worth knowing before the accidental owner hands the invoice over for good.

See a sample report >