AI-Collaboration Rubric: Demonstrated, Partly, Not

Five behaviors, three levels, one blank sheet per candidate. Every level is written as an act with a trace, so two reviewers can point at the same moment and agree it is there or concede it is not.

The five behaviors

Fluency, structure and length are the assistant's contribution and earn nothing. Neither does volume, in either direction: how much of the work was handed over is not a level of anything.

Refusal and verification sit on one row in the rubric these come from, and are split here because their traces differ. Cut a row a role's record cannot show, rather than rating it blind, and do not add a sixth that restates one already on the sheet.

The grid

An adjective is what two reviewers disagree about, because each reads it from their own practice. Strong evidence, good rigor and shows judgment are not levels. Every cell above names something a reviewer can point at in the record, or concede is not there.

Anchors in the role's own evidence

The grid gets you a shared shape, not a shared standard, and the standard is the half that does not copy: what a competent colleague in that job asks to see before signing. Write the Demonstrated cell in that evidence first, with people who hold the job, so a reviewer has a concrete behavior to refer to rather than an adjective ¹. The other two follow: partly demonstrated is asked for and not opened, not demonstrated is neither.

A structured interview has two clauses, not one: the same predetermined questions in the same order, and every response evaluated against the same rating scale and the same standards for acceptable answers ². Teams ship the scale and skip the standard. The Demonstrated column is the standard.

Almost right is the failure these anchors are pointed at. In a 2025 developer survey the most common reported frustration with these tools was output that is almost right but not quite ⁵, and purpose-built legal research systems from two major vendors returned hallucinated content between 17% and 33% of the time in a 2024 study, against vendor claims of hallucination-free retrieval ⁶. A cited-looking claim taken at face value is careless by the standards of the job, not by the standards of the marketing.

The rating sheet

Copy this into your own file, one sheet per candidate, and write beside each behavior the level and the moment it rests on.

Put the anchor version and the date at the top. A rubric that changed mid-round cannot be compared across the candidates who took it before and after, and a level with no moment cited cannot be argued about, only defended.

The sheet carries no number. Three named levels across five behaviors do not add up to anything, and folding them into one figure hides the disagreement between your reviewers, which is the information you were collecting.

Order of work

Step two is the one teams skip, and it decides what every agreement figure afterwards is worth: talking first produces one reviewer's opinion held by two people. Step three is the cheapest fix on the list, because reading one candidate end to end lets the first strong level color the next four.

Rate the trace, never the account. In a randomized trial of 246 real issues from experienced developers' own repositories in 2025, the work took 19% longer with the tools allowed, and the same developers still believed afterwards that the tools had sped them up by 20% ⁷.

Calibration

Both reviewers rate the same three sessions alone, then meet and work only the disagreements. Whoever changes their mind says which sentence of the anchor moved them, and if neither can point at a sentence the anchor is at fault rather than the reviewer. Rewrite it in the room, with the disputed session's actual behavior in it as the worked example. Recalibrate when a reviewer joins, when the task changes, and quarterly regardless.

Measure agreement per behavior, never averaged across the five, because an average will hide one behavior sitting at 0.2 and that is where every disputed decision comes from. Two reviewers who both mark Demonstrated most of the time reach a high raw agreement rate while agreeing about almost nothing, which is what Cohen's kappa corrects for: 0.60 to 0.79 is moderate, 0.80 to 0.90 is strong, and below 0.60 the two are measuring different things ⁴. Under the same paper's floor of about 30 comparisons, report exact agreement and the list of what split, and call it a calibration check rather than a reliability estimate ⁴.

Across 19 past-behavior interview studies reviewed in 2014, anchored scales showed higher validity, .35 against .26, and higher agreement between reviewers, .77 against .73 ³. A comparison across studies, not a controlled test.

The candidate's copy

Send the five behaviors and the three level names with the exercise brief. Keep the anchors you wrote in your own evidence, and any weight you put on a behavior, with the reviewers.

Sending the criteria does not weaken the instrument, and the reason is in the grid: every Demonstrated cell is an act with a trace, so someone who reads the sheet and then goes and opens the source has done the thing the level is for. What a published rubric usually loses is completeness, and completeness is on no row here.

Sending the anchors and the weights does weaken it, because together they are a specification, and a specification handed to a candidate working with a model comes back satisfied line by line. Where the tool is permitted, correctness of the final answer stops being sufficient evidence and the judging becomes the readable part ⁸.

The tell that you sent too much is convergence: submissions arriving in the same structure under the same headings, nothing argued. At that point the exercise measures who thought to paste the sheet in.

After the decision, send the rest. There is nothing left to optimize to, and in a 2025 vignette experiment with 921 working-age Austrians an unexplained automated rejection scored lowest on every outcome measured, while an explained one drew the same fairness ratings as a human rejection carrying no explanation ⁹. Hypothetical rejections on an online panel: an ordering among unhappy outcomes rather than a route out of them.

Duties

Where this sheet decides who advances, the burden below is the employer's.

Once a complaining party shows a particular practice causes disparate impact, demonstrate that it is job related for the position in question and consistent with business necessity. 42 U.S.C. 2000e-2(k), effective November 21, 1991.

So every cell names an act from the job rather than an adjective. An adjective cannot be shown to be job related, because nobody can say what it asked for.

The session is administered material, and the duty below reaches how it is run.

Select and administer any test so the result reflects the skill it measures, not a candidate's impaired sensory, manual or speaking skills, unless those are what it measures. 29 CFR 1630.11.

In practice: offer the same material in a form the candidate can use, and rate the act rather than the pace or the manner of it. Someone who reads the source with assistive software has opened the source.

Limits

An interview answer is an account of work, not the work. This sheet rates what the record contains, and someone who did none of it can describe all of it fluently. Two repairs, both cheap: ask for the artifact rather than the summary, and put the material in front of the candidate live, so the check has to happen in the room.

Agreement is not validity. Two reviewers can be trained into near-perfect agreement about a document showing none of the five acts, and the figure that comes out will look like rigor while describing nothing. Calibration measures whether two people read the same evidence the same way, never whether the evidence was there.

Nothing here predicts performance. What it buys is comparability across candidates asked the same questions against the same standards, a record resting on observable behavior rather than on an inference about mental processes, and a specific next step. Not demonstrated on a behavior means the act is not in this record, under these conditions, with this much time. It does not mean the person cannot. Say that out loud in the debrief, because a sheet of five levels read as a verdict on a person is worse than no sheet at all: it lends arithmetic to a judgment that has not earned it.