Take-Home Assignment or Live Working Session: Which Shows AI Judgment?

Which Format Fits the Task Horizon?

Match the format to how long the real work takes. When the job's unit of work finishes inside an hour and leaves visible moves (a draft, a query written and run, a triage call), a live session shows AI judgment directly. When the real work runs across days, a live session catches the setup and nothing that follows, and a short take-home with a walkthrough gets closer.

Sort your own roles into two buckets before arguing about formats.

• Fast and observable. A marketer building campaign copy from a research packet. A support lead working a ticket queue. An analyst writing one query and reading the result. A recruiter drafting a screen. The whole loop (frame, generate, check, cut) fits in 45 minutes, and every move in it is something you can watch someone make.
• Long-horizon. A three-day financial model. A migration across a codebase nobody on the team wrote this year. A diligence memo over a 200-page packet. The judgment you care about is spread across hours: which of six threads to abandon on day two, which assistant output to stop trusting after it was wrong twice.

What makes this more than a preference is that people are unreliable narrators of their own long-horizon AI work. METR randomized 246 real issues from experienced open-source developers' own repositories to allow or forbid AI tools. With the tools allowed the developers took 19% longer, and after finishing they still believed the tools had made them 20% faster ¹. So a candidate's account of a three-day build is worth something, but not as measurement, and neither is 45 minutes of watching the first hour of it. Measure How Fast They Work With AI, or How Well They Judge It? works through the trap next door.

What a Live Session Shows, and What It Costs

It shows the acts a finished document cannot: the first thing asked of the assistant, whether a source was opened or only cited, what got refused, what got checked. The cost is a measurement error most loops never account for. In a randomized trial with 48 computer science students, median correctness fell by more than half when an interviewer simply watched the candidate work ².

The same trial measured higher stress and cognitive load in the watched condition, and none of the women in the public setting solved the problem while all of the women in the private setting did ². Its authors propose two fixes, and both survive the move to AI-assisted work: give the candidate private time on the problem first, then bring them back to narrate what they did.

A live session earns its place when the work is observable, on three conditions.

• Private minutes first. Ten minutes alone with the problem and the assistant, then you join. You lose the opening move in real time; you gain a candidate who is solving rather than performing.
• The assistant stays open. If they will use it on Tuesday, they use it here. Banning it turns the session into a memory test and hides the delegation you called the meeting to see. How to Run a Coding Interview With an AI Assistant Open covers the mechanics for engineering roles.
• Nothing is timed to the minute. Speed under observation is the one thing this format measures reliably, and it is the thing you least want to select on.

What a Take-Home Still Tells You

It tells you the shape of long-horizon work: how the candidate scoped a task with nobody standing over them, what they sequenced first, and what got cut when the time ran out. What it cannot tell you is who decided any of it. A finished artifact reads the same whether the candidate wrote it, edited it, or accepted it whole.

The defect you are hunting is not obvious badness. In Stack Overflow's 2025 developer survey the most common frustration with AI tools, at 66%, was output that is almost right but not quite ³. Almost-right compiles, renders and reads well, then fails on the single figure the recommendation rests on. A read-through does not find it; tracing one number end to end does.

Three things make a take-home worth the candidate's unpaid hours.

• A cap you enforce. Ninety minutes, stated in the brief, with a deliverable small enough to fit inside it. A take-home is a selection procedure like any other test, so it has to be job-related and consistent with business necessity, and that burden sits with the employer ⁴. Hours you cannot justify against the job are an exposure and a completion problem at once.
• A stated AI policy. Allowed, or not, in one line. A rule you cannot enforce mostly selects for who ignored it, and Should Candidates Use AI on the Take-Home? works through where each answer holds.
• A trail written while the work happens. Assumptions and their basis, one claim checked outside the assistant, one thing cut and why. How to Grade a Take-Home When Every Submission Is Polished has the grading side of it.

Run the Hybrid: Short Take-Home, Then a 20-Minute Walkthrough

Budget about two hours of candidate time and split it. Ninety minutes on a capped take-home with the assistant allowed, then twenty minutes live on the candidate's own submission. The walkthrough is not a re-test and not a defense. It is you watching someone account for decisions they made three days ago, with the file open in front of both of you.

Five prompts fill the twenty minutes, and each asks for an act rather than an explanation.

• Open the file and show me where this number came from. Pick the figure the recommendation rests on. Someone who put it there can trace it in a sentence.
• Which claim did you go outside the assistant to check, and what came back? Including the case where the check returned nothing useful, which is a real answer and a common one.
• Show me something you cut. Rejection is expensive and specific. A submission that accepted everything looks identical to one where nothing was worth refusing, and this is the question that separates them.
• The deadline just halved. What goes? A fresh decision, made in front of you, on material they already know cold. This is the part of the live format worth keeping.
• Where would you not have used the assistant? The answer that counts names a step and a reason rather than a policy.

That set aims at whatever is left of knowledge work once a tool drafts. A survey of 319 knowledge workers across 936 first-hand tasks found the effort moving from producing an answer toward verifying information, integrating a response and stewarding the task, with higher confidence in the tool tracking with less critical thinking and higher confidence in one's own ability tracking with more ⁵.

One limit, stated plainly: a walkthrough is a retrospective account, not a record. It is harder to fake than the artifact, because it can be checked against the artifact and the artifact can be checked against nothing. It is still not the same evidence as watching the decision get made.

Score Both Formats Against One Rubric

Rate the same four things whichever format produced the evidence: how the problem was framed before anything was generated, what was checked outside the assistant, what was kept by hand rather than handed over, and what was refused. The format decides where the evidence comes from. It does not change what you are grading, and two rubrics are how two candidates become incomparable.

Structure is the part carrying the weight, and it is worth knowing how much weight there is. Sackett and colleagues re-ran the meta-analytic estimates behind the standard validity table and found the usual correction for range restriction had substantially overcorrected: most procedures kept their rank order but lost .10 to .20 in mean validity, and structured interviews came out top-ranked ⁶. Selection methods still work. They work less well than the table on your wall says, and the structured ones hold up best.

Four things to fix before the first candidate rather than after the third.

• Write the anchors first. Two or three worked descriptions per item, for this specific case. A key written after reading three submissions is a description of those three submissions.
• Hold the case fixed. Same task, same cap, same walkthrough prompts in the same order, for everyone at that stage. Both formats are selection procedures under the same expectation of job-relatedness ⁴.
• Rate item by item across candidates. Grade every framing, then every outside check, then every rejection. Reading one submission end to end is how a handsome artifact drags a weak decision through an entire rubric.
• Double-read the first five. Two raters, independently, then compare. Disagreement early is information about the key rather than about the candidates, and it is cheap to fix at that point.

See what gets scored >