An AI Work Sample Tests Whether You Catch the Bad Answer

What Is an AI Work Sample, Actually?

An AI-collaboration exercise gives you a real task, an assistant that will do most of the drafting if you let it, and a time box. It differs from a plain work sample less in what you produce than in what gets watched while you produce it: which parts of the assistant's output you accepted, which you rejected, and why. The deliverable matters less than the trail behind it.

Work samples earned their reputation honestly, if not for the exact number usually quoted at you. The widely repeated claim that they predict job performance at .54 is stale; a 2022 re-analysis of the meta-analytic literature puts work sample validity at .33, a figure the original authors themselves had already revised downward once before ¹. An AI-collaboration exercise is a newer variant on the same idea, built for a task that now assumes an assistant is present rather than absent.

The lineage stops short of the exercise in your invitation, though, and it is worth knowing where. That .33 comes from studies that almost all tested people already doing the job, and no comparable figure has been measured for an AI-collaboration exercise against real first-year performance, because that would need performance data collected after the hire. A number borrowed from the older work-sample literature describes a different exercise. What carries over is the design idea, that a job-shaped task tells you more than a conversation about the job. The coefficient does not carry over, and nobody should quote you one.

Expect the Prompt to Contain a Trap

Assume the material you're handed contains something wrong, because a well-built version of this exercise is designed that way. Designers who build these seed a bad number, a shaky claim, or a plausible-sounding conclusion that does not hold up, then watch who finds it. An exercise where nothing is almost-right measures nothing, since everybody passes it. Spending your first minutes hunting for that flaw beats polishing prose the assistant already wrote acceptably well.

The instinct to trust a fluent answer is documented and not a personal failing. In a field experiment with 758 Boston Consulting Group consultants, those given GPT-4 on the one task chosen to sit outside the tool's range were 19 percentage points less likely to reach the correct answer than the control group working without it ³. That result does not show the tool making people worse in general. It shows that nobody in the room could tell which side of the line the task sat on, which is the gap an exercise like yours is built to look for.

Read the material once for content, then read it again looking specifically for the sentence that sounds most confident, since a confidently stated wrong answer is easier to skim past than a hedged one. The flaw is rarely hidden in the hardest part of the task. It is usually sitting in plain sight, in the part that reads so cleanly nobody thinks to check it.

Show Your Reasoning, Not Just the Output

Where the format allows notes, a chat transcript, or a short write-up, use it to show three things: what you checked, what you rejected, and what you would still verify with more time. A rubric built around this axis is looking for visible judgment, and a clean final answer with no trace of that process reads as luck rather than skill, even when the answer happens to be correct.

Employers deciding whether a work sample or an AI-collaboration exercise predicts a first-year hire's performance better are told to look at how much of the real job an assistant now drafts first, which is the same shift this exercise is built around: the task assumed a person would produce the first pass, and now it assumes the assistant does, so the thing worth measuring moved with it.

A useful shape for that trail, when the instructions don't dictate one: name the claim you decided mattered most, say what you checked it against, and say what you did with the answer. Three lines will do it. That is the difference between a grader inferring your judgment from a finished artifact and reading it directly, and a good artifact and a lucky one look identical on the page.

Does Speed Actually Matter Here?

Rarely, and self-reported speed is an especially bad guide to trust. In a randomized trial, experienced developers using AI tools on code they already knew well finished 19% slower than they would have unaided, despite forecasting a 24% speedup beforehand and still believing afterward that they had gained about 20% ⁴. If professionals tracking their own output got the direction wrong, a candidate guessing at how fast to move is guessing blind too.

Employers running these exercises are advised to measure how well someone judges the output rather than how fast they produce it, letting the clock cap the exercise instead of scoring it directly. The most quoted developer speed number in circulation, a 55.8% reduction in completion time with an AI coding tool, comes from a single trial where 95 freelance developers recruited on Upwork were asked to write one HTTP server from scratch, and it describes only the ones who finished ². Nothing about a real assessment task resembles that setup closely enough to borrow the number, and nothing about your invitation email is asking you to beat it.

Don't Polish Past the Point of Being Checkable

A submission that looks finished can still fail if there is nothing in it an evaluator can verify. Leave the seams visible: a comment noting what you didn't have time to check, a flagged assumption, a line explaining why you rejected the assistant's first draft. All of that reads as judgment. Smoothing it away in the name of a clean deliverable removes the exact evidence the rubric is looking for.

A submission with two visible corrections and an honest open question usually reads stronger than one that looks effortless, because effortless is precisely what a rubric built around this axis has learned to distrust.

If the invitation looks more like an interview than a written assignment, what an AI interview actually scores covers the adjacent format, since the two are usually graded on close to the same axis.

See what gets scored >