Hire Data Annotators In House When The Judgment Is Yours To Own

What Does A Data Annotator Actually Do On Your Team?

Your eval suite says the model is fine. Your support lead says three refund answers last week were wrong in a way no test caught. Somebody has to sit with two hundred transcripts, decide which answers were actually wrong and why, and write that decision down in a form a model can be trained or measured against. That person is a data annotator.

The public image of the job is boxes drawn around cars. That version still exists, and most of it has gone to vendors and to tooling. The version worth hiring for now is adjudication: reading a model's output next to a real question and deciding whether it was right, why, and what the better answer would have been. LinkedIn ranks data annotator the fourth fastest-growing US job of 2026, and reports a median 3.5 years of prior experience behind people entering it ¹. One caveat, said once and applying to nearly every figure below: that ranking is a platform reading its own postings and members, not a labor survey, so it describes the slice of the market that hires on LinkedIn. Take the shapes and leave the decimals. Three and a half years is not an entry-level number on any reading of it. People arrive having already done something else well.

A week in the role looks concrete. Forty transcripts adjudicated against a written rubric. Six edge cases escalated with a paragraph each on why the rubric does not cover them. One rubric amendment drafted. A disagreement session with a second annotator where both of you defend your calls and one of you changes a mind on the record. The output that matters is not the pile of labels. It is the rubric getting sharper and the disagreement rate falling for a reason somebody can name.

That output is the raw material an evals engineer turns into a test suite. Hire one without the other and the test suite encodes whatever the loudest engineer believed on the day it was written.

Which Traits Separate A Real Data Annotator From A Fast Clicker?

Three traits, and each has a tell. The person is comfortable being uncertain in writing, so their notes say which reading of the rubric they used. They notice that two guidelines conflict before they reach case fifty. And they hold one standard steady across four hours, which is a stamina question more than a clever question. Raw speed is the anti-signal.

The tells are easier to see in work than in an interview. Give a paid sample of twenty-five real items with a rubric you have deliberately left incomplete, including two guidelines that contradict each other on about a fifth of the set. Then read for four things. Did the escalations name the conflict, or did the candidate quietly pick a side and keep going. Do the notes distinguish a hard call from a close call. On the ten items where you already know the answer, is the candidate consistently strict, consistently lenient, or neither, because a steady bias is correctable and drift is not. And when you show one counter-example, does the person revise that call with a reason, or revise everything downstream of it in a panic.

Performed expertise sounds fluent. It uses inter-annotator agreement, Krippendorff's alpha and gold sets in the right places, and has no story about a specific case it got wrong. Ask for one. A real annotator answers immediately, usually with irritation still attached, because the case that broke their rubric is the case they remember.

The AI question is its own tell, and it is not about whether the person uses a model. Nearly all of them do. Ask what happens when the model's answer is more articulate than their own. The habit you want is ordered: form the judgment first, then read the model's, then keep or revise with the reason written down. The habit you do not want is the reverse, where the model's framing arrives first and the human supplies agreement. People who got good at this practiced it on their own work long before anyone asked, usually by grading a model against something they already knew cold.

Hire Data Annotators From The Backgrounds That Already Grade Work

Look for people whose last job was judging someone else's output against a written standard. That same platform ranking has new data annotators arriving most often from content manager, editor and data analyst roles ¹, which reads as a description of who was already on LinkedIn rather than an exhaustive map. Add the backgrounds it does not name: paralegals, medical coders, translators, standardized-test scorers, content moderation reviewers, archivists, and lab technicians who have kept a protocol notebook.

The first place to look is inside the company. Support escalation leads, QA analysts, claims reviewers and clinical documentation staff already adjudicate ambiguous cases against a policy, already know the domain, and already have the access approvals. An internal transfer skips the two slowest parts of onboarding, which are the vocabulary and the security review.

Outside, three channels do most of the work. Adjacent-title search on the roles above, filtered for people who mention rubrics or guidelines rather than throughput. Alumni of the annotation and human-data vendors, where Scale AI, Surge AI, Appen and Sama are the names most candidates will have on a resume, and where the useful ones ran quality or wrote guidelines rather than only labeling. And the domain's own professional communities, which for specialist annotation are worth more than any AI forum: coding associations, translator associations, subject teacher groups. If the model is being trained to reason about tax, the annotator pool is tax people who write clearly, not machine learning people who are curious about tax.

One pipeline fact worth using, with the same caveat attached. The ranking puts the role's workforce at 62% female, the most gender-balanced entry on its 2026 list ¹. Job families that skew this way tend to be the ones where a self-taught path is legible and a credential wall is absent, which is also why a resume screen is close to useless here. The work sample is the screen.

If the mandate is curating an existing corpus rather than labeling a stream of new output, the neighboring hire is a research data curator, and the two job descriptions should not be written by copying one into the other.

What To Pay A Data Annotator, And Whether The Work Is Remote

Say the honest thing in the posting. As of mid-2026 no BLS occupational series and no major published salary report covers this title, so there is no credible range to anchor against. Postings split into two shapes instead. Hourly project work priced like contract QA, and salaried in-house roles priced against your own analyst or subject-expert bands. Pick the second when the judgment is the product.

The hourly shape is not an accident. That same ranking describes annotation as often organized per project ¹, which is how vendors and marketplaces price it and why so much of the visible pay data is an hourly rate for a task rather than a band for a job. Treat that data as evidence about the vendor market, not about the salaried role you are trying to fill.

For the salaried version, price the domain and not the tooling, because the domain is the only part a vendor cannot sell you. A licensed nurse adjudicating clinical summaries costs what a nurse costs plus a premium for leaving the floor. A tax attorney grading tax reasoning costs what that person bills. The annotation platform takes an afternoon to learn and should carry no weight in the band. If you are building a mixed team, the band that fails is the one that pays every annotator the same regardless of what they had to know to make the call.

If the plan is contractors at volume, worker classification rules differ by jurisdiction and change often. Check with counsel before designing the arrangement, not after the first invoice.

On location, this work is less remote than its reputation. One platform's posting data puts 27.5% of these listings remote and 29.4% hybrid, with Austin, New York City and San Francisco as the top hiring metros ¹, which is a hint about a distribution rather than the distribution. The rest are on-site, and the reason is usually data residency: protected health information, unreleased product, customer records or source code that cannot leave a controlled environment. Decide the residency question before you set the band, because a secure-room requirement roughly halves the pool and a remote posting for regulated data is a promise you will have to break.

Close A Data Annotator By Naming The Ladder Out Of Annotation

What kills the offer is the suspicion that the job is a treadmill. What closes it is a written path: which decisions the person owns by month six, the rubric that carries their name, and the roles the work feeds into. Microsoft's 2026 Work Trend Index counts data annotators among 1.3 million AI-related job openings in categories that did not exist five years ago ². Candidates know the door is new, and they know new doors sometimes close.

Four things they actually care about, in the order they raise them. Attribution, meaning the gold set and the guidelines are credited rather than absorbed. Access, meaning a seat in the review where model failures get discussed instead of a queue delivered by a ticket. Measurement they can defend, meaning quality and calibration rather than items per hour. And a title that will read well in two years, which is a real negotiating point and costs nothing.

The offer dies on the opposite four. A throughput quota in the job description. Temp classification with no system access, so the person cannot see whether the labels changed anything. A manager who describes the work as cleaning up after the model. And silence on what comes next, which candidates read correctly as an answer.

What comes next is worth writing down because it exists. Annotators who write guidelines become the people who design evals. Annotators who track disagreement become quality leads. Annotators who learn the pipeline move toward data engineering, and the ones who end up teaching the rest of the company how to work with model output are doing the job of an AI enablement lead before anyone gives them the title. Name two of those paths in the closing conversation, with the person who took them, and the treadmill worry goes away.

See a sample report >