How Do You Write an AI-Use Rubric Two Reviewers Score the Same Way?

Why Two Reviewers Disagree About the Same Session

Because the scale asks for a judgment nobody defined. 'Uses AI effectively, 1 to 5' is an empty container each reviewer fills from their own practice, so a session that looks disciplined to someone who works that way looks slow to someone who does not. Reliability, in assessment terms, is exactly this: rating consistency among the people doing the rating ¹.

The federal structured-interview guidance is blunt about what raters need instead: a concrete behavior written out for each proficiency level to refer to rather than an adjective, and a note documenting the reasoning behind every rating ¹. A one-line dimension name is neither. It is a prompt for an opinion, and two competent people will hold different ones.

Three things go wrong, and each has a different fix.

• The scale rates the artifact. A finished deck, a clean pull request and a fluent memo cost almost nothing to produce now, so a rubric that rewards a good deliverable is measuring the assistant. Reviewers who weight the artifact differently will disagree forever, because the artifact was never the thing in dispute.
• Volume reads as effort. Forty prompts look like engagement to one reviewer and thrashing to another, and both readings are defensible under a scale that never said which. How much AI someone used is not a level of anything. A candidate who decided the model was the wrong instrument for a step and did it by hand has done the thing you are trying to see.
• Halo travels down the grid. Read one candidate end to end and the first strong rating colors the next three. This is the cheapest of the three to fix and almost nobody does it: rate every candidate's first dimension, then every candidate's second, and never open a session already knowing what you gave it last time.

Getting a whole panel to converge is this same problem one layer out, worked through in How to Get an Interview Panel to Judge AI Use the Same Way.

Write the Anchors in the Role's Own Evidence

An anchor is a sentence describing what the act looks like in this occupation, with the artifact that proves it happened. 'Demanded evidence' is unscorable. 'Opened the source behind the statistic the recommendation rests on, found the sample was three years old, and changed the claim' is scorable, because two reviewers can point at the same moment and agree it is or is not there.

The method for writing them is old and dull and it works. Federal structured-interview guidance has subject-matter experts sit down per competency and write, for each proficiency level, how an actual employee at that level would answer. The stated purpose is to give raters concrete behavior to refer to rather than an adjective, and so a common frame for reading different candidates ¹. Do that, one dimension at a time, with people who hold the job.

The anchors cannot be generic, because the act is not generic. Take one dimension (tested a claim against something outside the conversation) across three roles.

That engineering anchor is written where the defect actually lives. In Stack Overflow's 2025 developer survey the most common frustration with AI tools, at 66%, was output that is almost right but not quite, and 46% of developers said they distrust the accuracy of what the tools produce against 33% who trust it ⁶. Almost-right compiles. It passes the happy path. It fails the case nobody wrote a test for, which is why the anchor names the test and not the diff.

Two rules keep an anchor scorable. Every level describes an act with a before and an after, so a reviewer can point at where it happened or concede it did not. And no level mentions how much the assistant was used, in either direction. Anchoring outside engineering takes more work rather than less, and How to Screen for AI Judgment in Finance and Marketing Roles works through where the evidence hides in those roles.

What Goes on the Scale: Four Dimensions, Three Levels

Four dimensions, three levels, and no average across them. Three levels (not demonstrated, partly demonstrated, demonstrated), because a five-point scale invites the middle and forces distinctions the record cannot carry. Four dimensions because each has to be independently visible in the session, and the fifth candidate dimension is almost always a restatement of one already on the sheet.

Four that hold up across roles, each with the question it asks and the evidence that settles it.

• Framing. Did the first move go after understanding the problem, or straight at the deliverable? Settled by the opening exchange, before any output exists.
• Evidence. Was a source demanded for the specific claim the answer rests on, and opened? Settled by what was retrieved, not by what was cited.
• Boundary. What was kept, and what was handed over? Settled by the acts in the record the candidate did themselves.
• Refusal and verification. Was anything the assistant produced rejected on substance, and was any claim tested against something outside the conversation? Settled by a change you can trace: a section cut, a number moved, a limit added.

Rate refusal as a rate rather than a count, refusals per answer actually taken up, or three well-aimed prompts lose to thirty scattered ones for a reason nobody would defend out loud.

Do not average the four into one figure. The average is the part reviewers disagree about least and the part that tells you least: two candidates with the same mean can be opposite people, and the disagreement between your reviewers, which is the information you were collecting, vanishes into it. There is a paperwork cost too. Where a selection procedure assigns weights to its parts, the documentation standards expect the weights and the validity of the weighted composite to be reported ⁵. Scoring a single AI-assisted answer runs on the same rules, and How to Score an Interview Answer Produced With AI covers that case.

Run the Calibration Session: A 90-Minute Script

Both reviewers score the same three sessions alone, then meet for ninety minutes and work only the disagreements. Pick sessions spanning the range you expect (one weak, one middling, one strong), and require a note citing the moment behind every rating, because a rating with no cited moment cannot be argued about, only defended.

The note is not paperwork. The federal guidance asks for notes of sufficient quality and quantity to document the reasoning behind each rating on each competency, and treats them as the record supporting the employment decision ¹. In a calibration session they are the whole substrate. Without them the meeting is two people restating impressions at each other.

• 0-10 minutes: compare grids only. Both scoring sheets side by side with the reasons covered. Count exact matches per dimension. That number is the honest starting point and it is usually worse than either reviewer expected.
• 10-45: work the splits, one dimension at a time. Take each disagreement and have both reviewers read out the moment they rated. One rule makes this productive: whoever changes their mind has to say which sentence of the anchor moved them. If neither can point at a sentence, the anchor is at fault rather than the reviewer.
• 45-70: rewrite what split. Edit the anchor in the room, and write the disputed session's actual behavior into it as the worked example. An anchor that survived a real disagreement is worth ten written in the abstract.
• 70-85: re-score session three under the new wording. If agreement does not move, the dimension is doing two jobs at once. Split it or cut it.
• 85-90: version the rubric and log what changed. Every rating from here carries the anchor version it was made under. A rubric that changed mid-round cannot be compared across the candidates who took it before and after.

Then keep the order. Each reviewer observes, records and rates independently, and only once both grids exist do they discuss and settle a rating together ¹. Discussion before scoring does not produce agreement. It produces one reviewer's opinion held by two people, and it will read as agreement in every measurement you take afterwards.

Recalibrate when a reviewer joins, when the task changes, and quarterly regardless. Run the whole thing over historical sessions before it gates anybody: How to Pilot an Assessment Before Making It a Hiring Gate covers the shape of that trial.

Measure Agreement Before You Trust the Rubric

Count agreement with Cohen's kappa, per dimension, never averaged across them. Two reviewers who both mark 'demonstrated' most of the time will match on raw percentage while agreeing about almost nothing, which is the reason kappa exists: it subtracts the agreement that chance alone would have produced ³.

The benchmarks worth quoting in a review: 0.60 to 0.79 is moderate agreement, 0.80 to 0.90 is strong, and any kappa below 0.60 indicates inadequate agreement, with little confidence to be placed in what rests on it ³. Per dimension. An average across four will happily hide one sitting at 0.2, and that dimension is where every disputed decision comes from.

The sample size is where most teams find out they cannot make the claim yet. The same paper puts the heuristic floor at about 30 comparisons ³. Thirty double-marked sessions per dimension is more than a first round produces, so if you have eight, report exact agreement and the list of what split, and call it a calibration check rather than a reliability estimate. A kappa computed on eight sessions carries no information.

The regulatory expectation runs the same direction and is modest. Where a procedure rests on content validity, its reliability should be a matter of concern to the user, and appropriate statistical estimates of it should be made whenever that is feasible ². The documentation standards expect records showing the ratings given by each rater to be kept and produced on request ⁵. And a scored rubric that decides who advances is a selection procedure, which puts the burden of showing it is job-related and consistent with business necessity on the employer ⁴.

One limit sits underneath all of it, and it decides what your agreement is worth. A rubric can only rate what the record contains. If what reaches your reviewers is a finished artifact plus the candidate's own account of how it was made, two reviewers can reach a kappa of 0.85 about a document showing none of the four acts the rubric names. Agreement on the wrong material is still agreement, and it is the failure that survives every calibration session, because calibration measures whether two people read the same evidence the same way, never whether the evidence was there.

See what gets scored >