A Prototype Nobody Asked For Is a Test Nobody Else Took

What an unsolicited prototype proves, and what it cannot

A working prototype of your product proves the applicant will start without being asked, and that they can carry an idea from nothing to something that runs. Both are real. What it cannot prove is that the same person would do well on the task you set, because nobody set this one. The prototype answers a brief the candidate wrote for themselves.

Initiative is the finding. Record it as initiative.

Authorship is a separate question. Assume a model generated most of the code, and read telling who did the work on an AI-built portfolio for the moves that test it.

One study points at where the evidence sits instead. Engineering students permitted to use ChatGPT during take-home open-book exams were required to hand in their chat transcripts alongside their solutions. The researcher reported that some of the strongest evidence of reasoning appeared where students evaluated wrong or incomplete AI responses ⁵.

The study is qualitative, with no control group and no comparison of grades, so it validates no method. What travels is the direction: the paper concludes that a correct final answer may no longer be sufficient evidence of comprehension. The finished thing hides the working, and a polished demo hides it best.

Why does a self-chosen build stop being a work sample?

A self-chosen build stops being a work sample the moment the candidate picks the problem, the scope and the finish line. A work sample earns its place in a round by being shared: every candidate gets the same task under the same constraints, so two submissions can be set beside each other and mean something.

Remove the shared task and the comparison collapses. Two people who solved different problems have not been ranked.

Since 1978 the federal Uniform Guidelines have set a related standard for validity studies, one that asks how closely a procedure resembles the job. Under 29 CFR 1607.14(C)(4), a user demonstrating content validity should show that "the behavior(s) demonstrated in the selection procedure are a representative sample of the behavior(s) of the job in question". A representative sample of the job's work product counts too. Where a procedure purports to sample either one, the same subsection asks that "the manner and setting of the selection procedure and its level and complexity should closely approximate the work situation" ¹.

A prototype built at a kitchen table against an invented spec copies the product without approximating the manner or setting of the work situation.

The number under work samples is also smaller than the folklore. Sackett, Zhang, Berry and Lievens re-estimated the selection literature in 2022 and put work sample validity at .33 rather than the .54 in wide circulation, drawing on a review of 54 studies covering 10,469 people ².

In 53 of those 54, the people tested were already doing the job, so even .33 is not a statement about applicant-stage exercises.

A build nobody set cannot borrow even that number.

So the prototype cannot be scored against the round and cannot replace it. The rule for keeping a round comparable is already written down: vary the data, never the task or the bar. A prototype varies the task, the bar, the clock and the tooling at once.

Ask which constraint they guessed at

The constraint they had to guess at is the question worth asking, and it is not about the code. Ask which one it was, and what they would change if the guess turned out wrong. Building a version of somebody else's product means inventing the parts a real brief hands over: who the user is, what the data looks like, what counts as done.

Guessing at a spec is the step a real brief exists to remove. Real briefs arrive from a customer, a regulator or a manager, they are sometimes wrong, and part of the work is arguing with them rather than authoring them.

In 2023, 758 Boston Consulting Group consultants sat a pre-registered field experiment on realistic consulting tasks, randomly assigned to work with no AI, with GPT-4, or with GPT-4 plus a prompt-engineering overview. On the 18 tasks chosen to sit inside the model's capability, the consultants with access completed 12.2% more of them, finished 25.1% faster, and scored more than 40% higher on grader-rated quality ³.

On one task deliberately picked to sit outside that capability, the direction flipped. Roughly 84.5% of the control group reached the correct answer, against 60% and 70% in the two AI conditions, an average of 19 percentage points worse ³.

Read the reversal narrowly. It covers one task and a 2023 model, and it is evidence that the people using the tool could not tell which side of the line their task sat on.

A candidate choosing their own project can land on the friendly side of that line without meaning to.

The demo may run partly because the choice was good, and the choice was theirs.

Spend the conversation on the parts they invented. Which constraint did you guess. What did you leave out on purpose. What breaks first at a hundred times the load.

The same questions work on any artifact somebody brings unasked, and they rest on one principle: a portfolio piece counts when its decisions can be defended and not otherwise.

Who in the pool can spend a weekend on spec?

The applicants best placed to build one are the applicants with a free weekend and nobody depending on it. Crediting an unsolicited build adds an unpaid stage that only part of the pool could ever sit. Not the person working out a notice period, not the one holding a second job, not the parent with no childcare on a Saturday, not the applicant whose visa clock is running.

The point needs no study behind it. Unpaid work is rationed by who has hours to give away.

Informality does not put the decision outside the rules either. A selection procedure, in the Uniform Guidelines, is any measure, combination of measures, or procedure used as a basis for any employment decision. The definition runs all the way down to "informal or casual interviews and unscored application forms" ⁴.

Taken at its word, the definition would reach a prototype you looked at and then advanced somebody on; how it applies to your own process is a question for counsel.

Nothing follows automatically from that. The Guidelines attach a validation burden only where adverse impact shows up, and this is a 1978 regulation applied by analogy to a case it never imagined. What it removes is the comfortable belief that an informal step is not a step.

If built work is what the decision needs, buy it. A paid trial still asks finalists for time, but it pays for the hours, and whether a paid trial beats an assessment is worth settling before you commission one.

The honest case against all of this sits at the small end. At a five-person company the round is two conversations and a reference call, and the prototype is the most concrete thing anyone will put on the table. A founder there is not weighing it against a better instrument. They are weighing it against two conversations.

Reply without commissioning the next prototype

Three sentences do it. Thank them for the build, say the round is the same for every applicant, and name the two questions you will ask about it in the next conversation. Nothing in the reply should suggest the prototype advanced them, because the next applicant hears about it and the one after that spends a weekend.

Check the posting too. To an applicant with a model open in the next tab, a careers page asking for people who show initiative reads as an invitation to build the product on spec.

Say what evidence the process wants instead, in the words the process actually uses.

There is a product finding in here as well, and it belongs in a different meeting. A stranger reached a usable copy of the thing from outside the company. What that says about the surface area is worth an hour with the team, and it says nothing about the applicant either way.

The build is still worth opening. Read it once, then close it and run the round you would have run anyway, carrying two extra questions about the constraints somebody invented on a Saturday. The demo was never the whole evidence. The answers about it are the rest.

See what gets scored >