What Replaces an Online Assessment ChatGPT Solves in Thirty Seconds?

What does the replacement task look like?

Occupational material with something wrong in it, plus an assistant the candidate is welcome to use. The model answers confidently in seconds, that answer is wrong or incomplete, and everything worth measuring happens after it arrives. Three acts carry the signal: the problem framed before anything is generated, evidence demanded for the claim the answer rests on, and one check run against something outside the chat.

The defect is the design. A task with a single correct answer sitting in the model's training data measures retrieval, and retrieval got free. A task whose inputs do not support the obvious conclusion measures whether anyone looked. Developers already live in that gap: in Stack Overflow's 2025 survey, 66% named "almost right, but not quite" as their top frustration with AI tools, and 45.2% said debugging AI-generated code costs them more time than writing it ³. The failure mode is not gibberish. It is plausible.

It is worst where the tooling looks most trustworthy. Purpose-built legal research systems, sold to lawyers on the promise of grounded answers, returned incorrect information in 17% to 33% of benchmark queries in a 2024 Stanford evaluation ⁴. Nothing on the screen separated the usable memo from the sanctionable one. Someone opening the cited case did.

So build the task around that gap instead of around the answer. Hiring for verification rather than production is the same argument applied to the job description, and the three observable moves are the same three whether you are watching them in an interview or reading them off a submission.

Don't reach for proctoring first

Proctoring restores the conditions of a test whose content stopped discriminating. Lock the browser, and strong candidates still spend an hour on a task the job would let them do with an assistant. You have measured tolerance for being watched, not judgment. Detection is worse for a campus funnel, because the tools misfire hardest on writing by people who learned English second ⁵.

The numbers are not marginal. Seven detectors run over 91 human-written TOEFL essays produced a 61.3% average false-positive rate, and 97.8% of those essays were flagged by at least one tool, while the same detectors were near-perfect on US eighth-grade essays ⁵. A campus pipeline is exactly where that falls: students on temporary visas earned 36% of US science and engineering master's degrees and 59% of computer science doctorates in 2019 ⁶. A rule that misfires on one group and not another, applied at campus volume, is not a fairness problem you can average away.

There is a plainer objection too. If the job allows AI (and for most roles a campus program feeds, it does), then a banned-AI assessment tests conditions the hire will never work under. Whatever you run instead, the EEOC's ask of any step that decides who advances is the same: apply it identically to every candidate, and be able to say what it measured ⁷.

Open the assistant, say so in the invitation, and put the difficulty somewhere the assistant cannot reach. The proctoring-versus-AI-open decision turns on one question: is the hard part producing the answer, or noticing the answer is wrong?

How do you build one in an afternoon?

Take a real artifact your team produced last quarter, break one thing in it on purpose, and write the answer key before you write the prompt. The key lists moves rather than a solution: what a candidate has to notice, what they would have to open to notice it, and what an honest submission says about the limit it ran into. Ninety minutes of your time, once per role.

Four decisions do most of the work:

1. Pick the defect from the occupation. A dataset whose sample cannot support the conclusion. A brief missing the one constraint that changes the recommendation. A quoted figure that does not appear in the source cited beside it. A model will sail past all three, fluently. 2. Ask for the intermediate artifact, not only the deliverable. A plan, a criteria list, an outline, made before the deliverable and visibly shaping it. It is the cheapest thing to require and the hardest thing to reconstruct afterwards. 3. Require the transcript. Whatever assistant the candidate uses, ask for the exchange pasted in beside the work. You are not policing it; you are reading it. 4. Say what happens to it. One paragraph in the invitation: how long the task takes, who reads it, what gets graded, and that AI is allowed.

Keep it recognizable as the job. Content validity, under the guidelines an agency would apply, rests on the selection procedure being a representative sample of the behavior of the job, not on the task being hard ⁸. A puzzle nobody at your company has solved on an ordinary Tuesday is not a work sample, however cleanly it resists ChatGPT.

If you already run a take-home, this is a rewrite rather than a new step, and redesigning the work sample is usually cheaper than defending the old one to a hiring manager who watched a candidate finish it in a minute.

What do you grade when every submission is polished?

Grade the working, not the prose. Fluency has stopped varying between candidates, so the deliverable carries almost no information and the record of how it was made carries nearly all of it. Four observable acts are enough: a constraint named before anything was generated, a source opened for the claim the answer rests on, a piece of model output refused with a reason, and a check run against something outside the conversation.

Write each one as a yes-with-evidence rather than a rating. "Refused something" is a claim; "rejected the second recommendation because the segment size came from a sample of eleven" is evidence, and two graders will agree on it. Count refusals as a rate against the answers actually taken up, so three well-aimed prompts are never beaten by thirty.

Two cautions. Volume of AI use is not a virtue: a candidate who decided the model was the wrong instrument for a step and did it by hand has demonstrated the thing being tested. And the job-specific version beats the generic one: the 2023 re-estimation of selection-method validity put job-knowledge tests at .40 and work samples at .33, and found job-specific measures outperforming general ones ¹. A rubric written against your own material is worth more than a bought one written against nobody's.

Every submission will look good. Grading polished take-homes is the same problem one step downstream, and the answer is the same: stop reading the artifact as evidence about the person. See how Olive measures this.

How does this survive campus volume?

Run one task per role rather than one per candidate, cap it at an hour, and put it beside your current assessment for a cycle before it gates anybody. Work samples carry high content and criterion-related validity and candidates react well to them, but they fit only where the competency is expected on entry rather than trained after the hire ². For campus, that means testing judgment on the material, not craft a first-year analyst has not learned yet.

The arithmetic is the real constraint. A machine-graded assessment costs nothing per candidate, which is why it lasted this long; a read work sample costs someone ten to fifteen minutes. Two ways out, both unglamorous: move it later in the funnel so fewer people take it, or split grading across the hiring team after one calibration session on five submissions everyone reads.

Then hold it steady. Same task, same rubric, same time allowance, same accommodation offer for every candidate on that role. A step that decides who advances is a selection procedure whether or not a machine scores it, and consistency is what makes it explainable a year later ⁷. Rotating the task per candidate to stop leaks costs you the comparison, which was the whole point of running it.

Pilot it before you gate on it. Running the new assessment beside your existing round gives you the completion rate and the grading spread before anyone is rejected on it, and keeping the loop from getting longer is the constraint campus programs actually fail on.

See what gets scored >