How Do You Grade a Take-Home When Every Submission Is Polished?

Why Polish Stopped Being a Signal

Because the part you were reading (structure, tone, formatting, a confident executive summary) is the part a model produces for free. Presentation used to correlate with effort, and effort with care, and that chain broke. What survives is whatever the candidate had to decide: which problem to solve, what to believe, what to throw away. None of that shows in a finished file.

The clearest measurement of how far presentation has come loose from performance is METR's 2025 trial. Sixteen experienced open-source developers worked 246 real issues from their own repositories, each issue randomly assigned to allow or forbid AI tools. With the tools allowed they took 19% longer. They had forecast a 24% speedup going in, and after finishing they still believed the tools had sped them up by 20% ¹.

Two things follow for a take-home. Self-report about AI-assisted work is not evidence, so a candidate who tells you the tool saved three hours is describing an experience rather than an outcome. And the defect you are looking for is not obvious badness. In Stack Overflow's 2025 developer survey the most common frustration with AI tools, at 66%, was output that is almost right but not quite; 46% of developers said they distrust the accuracy of what the tools produce, against 33% who trust it ².

Almost-right survives every proxy you used to grade on. It compiles, it reads well, the chart renders, the summary is coherent. It fails on the one number, the one edge case, or the one claim the whole deliverable rests on. A rubric that rewards completeness and presentation now points confidently in the wrong direction. The interview round has the same problem in a different format, worked through in Why Every Candidate Gives the Same Polished STAR Answer.

What to Grade Instead: Three Decisions

Three, and each one leaves a trace the artifact does not. What the candidate assumed and on what basis. What they went outside the prompt to verify, and what the check returned. What they rejected from a draft, and on what grounds. Those are acts with a before and an after, so they can be evidenced, disputed and rated separately. Fluency cannot.

• What they assumed. Every real brief is underspecified, and the first act of competent work is naming what you decided to take as given. "I assumed the churn figure excludes trials, because the definition note in tab 3 says so, and the recommendation flips if it does not" is a decision with a consequence attached. A candidate who assumed nothing either had a fully specified brief, which you did not give them, or resolved the ambiguity silently.
• What they went and got. Not what they cited, but what they opened. The distinction matters because a generated draft arrives full of plausible references. Ask for one claim the submission depends on, the source opened to check it, and what the check returned, including the case where it returned nothing useful. A survey of 319 knowledge workers across 936 real tasks found the work does not disappear when a tool is used; it moves, from producing an answer toward verifying information, integrating a response and stewarding the task. Higher confidence in the tool tracked with less critical thinking, and higher confidence in one's own ability with more ³.
• What they rejected. The strongest single question on a take-home is what did you throw away, and why. Rejection is expensive and specific: a section cut because it was doing no work, a recommendation dropped when the number moved, an approach abandoned after ten minutes. A submission that accepted everything reads exactly like one where nothing was worth refusing, and the two separate only when you ask.

Rate the three separately and never blend them into one impression. How to Write a Rubric for AI-Assisted Answers covers the anchoring.

How to Get the Decision Trail Without Adding Hours

Ask for one page beside the deliverable and cap it. A decision log of about 300 words: three assumptions with their basis, one claim checked outside the prompt with the result, one thing cut with the reason. That is ten minutes of the candidate's time, and it turns an invisible process into gradeable text. Then say in the brief that the log is rated and the artifact is not rated alone.

The brief carries the whole design, and it needs four lines.

• AI is allowed, and the brief says so. A ban you cannot enforce tests who believed you, and the candidates you lose are the ones who follow instructions. Should Candidates Use AI on a Take-Home? covers the policy side.
• The log is required and rated. Name the three items and the word cap. Left unnamed, you get a process narrative; named, you get decisions.
• One claim must be checked outside the prompt. State it as a requirement rather than a suggestion, and ask what the check returned, including when it returned nothing.
• The deliverable is capped too. Length is not effort. Five slides, two pages, one file. A cap also stops the candidate spending their budget on formatting, which is the part you have stopped grading.

Keep the whole exercise under two hours and mean it. A take-home is a selection procedure, which puts it under the same expectation as any other test: it has to be job-related and consistent with business necessity, and the burden of showing that sits with the employer ⁴. Unpaid hours you cannot justify against the job are both a legal exposure and a completion problem.

What Polished-but-Empty Looks Like in Each Format

Different in every output, which is why one generic rubric misses it. A deck hides the analysis that was never run behind a clean waterfall chart. A pull request hides the untested edge case behind passing happy-path tests. A campaign brief hides an audience claim nobody opened the source for. Name the format's specific hiding place in the rubric, then look there first.

• A deck or a memo. The hiding place is the working behind the chart. Ask for the source file rather than the export, pick one number, and trace it end to end. A candidate who cannot say where that number came from in one sentence did not put it there.
• A pull request. The hiding place is the untested case, and the code that already existed. An analysis of 211 million changed lines found duplicated blocks rising from 8.3% to 12.3% of changes between 2021 and 2024, while refactored code fell from about 25% to under 10% ⁵. Read the diff for what was copied rather than reused, and for the test that covers only the path the assistant wrote.
• A campaign or positioning brief. The hiding place is the audience claim nobody opened. Take the most on-message statistic in the document and ask for the source, the sample and the year.
• A spreadsheet model. The hiding place is the assumption cell with no note on it. Ask which input the recommendation is most sensitive to, and what they did to find that out.
• A written analysis or recommendation. The hiding place is the counter-argument that was never considered. Ask what would have to be true for the opposite recommendation to win.

The question underneath is identical in all five and only its location moves: which part of this did the candidate have to decide, and what would I be looking at if they had not decided it.

Score the Trail Before You Open the Artifact

Read the decision log first, rate it, then open the deliverable. Keep that order and enforce it. A strong artifact contaminates every judgment that follows it, and polish is precisely the property that now produces a strong first impression. Rate each item on the same anchored scale for every candidate, and put two readers on the first few submissions.

Federal assessment guidance is specific about what makes a work sample worth anything: trained assessors rate observed behavior or measured task outcomes against standard criteria, and the method fits roles where the competency is expected on arrival rather than trained afterward ⁶. Applied to a take-home, that is three requirements you can meet in an afternoon.

• Write the key before the first submission arrives. Two or three anchors per item, describing what a weak, an adequate and a strong answer contains for this specific case. A key written after reading three submissions is a description of those three submissions.
• Rate item by item, across candidates. Grade every candidate's assumptions, then every candidate's verification, then every candidate's rejections. Reading one full submission end to end is how a handsome artifact carries a weak decision through an entire rubric.
• Double-read the first five. Two raters, independently, then compare. Disagreement in the first five is information about the key rather than about the candidates, and it is cheap to fix at that point.

One limit is worth stating plainly, because it decides whether this is enough for you. A decision log is written after the fact, by a candidate who knows it is being graded, with the same assistant still open. It is better evidence than the artifact, because it is harder to fake convincingly, and it can be checked against the deliverable, which the deliverable cannot be checked against anything. But it is an account of the work rather than a record of it. If you need the record, the exercise has to happen somewhere the decisions are visible while they are being made, and Take-Home or Live Session? works through that trade.

See what gets scored >