Assessment design
What Missing Empty and Error States Tell You About a Designer
Missing empty, error and loading states in AI-made screens are usually neither a craft gap nor a prompting gap. A generator renders the case the request described, and design requests describe the working case, so the absence is a property of the brief. What separates designers is the list they produce, before you point at anything, of what their own screens do not yet handle. A designer who names the states, the words each one needs and the case they would still test knows the job's surface area.
The takeDesign rounds still reward the rendered frame, which is the exact part that got cheap this year. A candidate can put eight polished screens on the table in an afternoon now, and the polish says nothing about whether they know what a product costs to run. The expensive skill was never the frame. It was knowing that a list can arrive empty, that a request can hang forever, that somebody will paste in a name with an apostrophe in it. Grade the inventory instead of the pixels.
Where Olive fits
Open a role and see what the work shows
If you build an exercise that grades the inventory, the costly parts are the authored case and the evidence trail behind each judgment. Olive returns six separately-evidenced findings per session, each anchored to a timestamped moment rather than to a score, and the candidate is granted the same report.
Assess your shortlistWhy do generated screens stop at the happy path?
Because a generator renders the case the request described, and the request described a working one. Ask for a settings page and a populated settings page arrives. Nothing in the phrasing asked what the screen shows before the first record exists, while the query is still running, or after the server returns a 500. Those are separate specifications, and nobody wrote them.
So the eight screens on the desk are a picture of the brief, not a reading of the designer.
In April 1994, Jakob Nielsen published the ten usability heuristics, refined from a set he and Rolf Molich developed in 1990. The first asks a design to keep users informed about what is going on, through appropriate feedback within a reasonable amount of time. The ninth asks error messages to state the problem in plain language and suggest a fix 1.
Both describe conditions. A loading state is the first heuristic with a clock attached, and an error state is the ninth with a sentence attached. Neither is visible in a rendered frame, because the frame is the thing the condition is absent from.
For input errors, the error condition is a conformance requirement.
WCAG 2.2, in the W3C Recommendation dated 12 December 2024, makes Error Identification a Level A criterion: an automatically detected input error has to be identified and described to the user in text. Error Suggestion sits at Level AA and requires that the suggested correction be provided when one is known, unless providing it would jeopardize the security or purpose of the content 2.
The model would have produced every one of those states on request. Type "now the empty state, the loading state and the two failure cases" and they arrive inside a minute.
The gap sits between what the request covered and what the product needs, and people are measurably bad at seeing that edge.
A 2023 Harvard Business School field experiment put a number on the blindness. On one task deliberately chosen to sit outside what the model could do, consultants using GPT-4 reached the correct answer 19 percentage points less often on average than a control group that got it right 84.5% of the time 3.
Read that gap narrowly. It came off one task, one sample, and a model since replaced, so the size does not travel. The direction does: the consultants could not tell which side of the line they were working on, and neither can a designer looking at eight screens that all render.
Ask what is missing before you say what you noticed
Order decides what the question measures. Say "there are no error states here" and the answer has been handed over, so everything after that is agreement. Ask instead for a list of what these screens do not yet handle, with nothing pointed at, and the list belongs to the candidate. It arrives in about ninety seconds or it does not arrive.
The prompt is one sentence, and it is the same sentence for every candidate who submits: name every state these screens still owe, and what each one has to say.
A full inventory runs longer than most people expect:
- Empty, twice. First run, before anything exists, and empty after a filter, which needs different words and a way back.
- Loading, with a ceiling. A skeleton under two seconds, something else past that, and a screen for the request that never resolves.
- Error, split by recoverability. The one a retry fixes, and the one where the work is gone and the copy has to say so.
- Permission and account state. Signed out, wrong role, read-only, expired trial, offline.
- The ugly data. A sixty-character name, a list of four hundred rows, a currency with no decimal places.
A published vocabulary already exists for what that list measures.
Discernment, in the AI Fluency Framework written by Rick Dakan of Ringling College of Art and Design and Joseph Feller of University College Cork, is accurately assessing the usefulness of AI outputs. Diligence is the next of its four competencies: taking responsibility and vouching for the final product, down to a sub-competency defined as verifying and vouching for AI-assisted outputs 4.
It is a vocabulary and not a rubric. The framework publishes no levels, no thresholds and no scoring anchors, so nothing in it turns a designer's list into a number. The use here is naming the act you are watching for.
Then push once, the way follow-up questions expose whether someone understands the answer they gave. Take the state they named first and ask what that screen says, in words, and what the user does next.
A designer who has shipped it answers with a sentence. Someone reading off a checklist answers with a category.
How do you read the list they hand back?
On three things: whether the entries are screens or categories, whether each carries the words it needs, and whether the designer says which case they would still have to test. "Error handling" is a category. "The upload fails at 90 percent and the file is gone, so the copy has to say the file is gone" is a screen somebody has built before.
Eight or ten states, ranked, is what a strong answer looks like. First-run empty comes first, because every new account sees it. The unrecoverable failure comes next, because the copy is the only thing standing between the user and a support ticket. Ranking is the part a checklist cannot fake.
A middle answer names the categories and stops there. Real gap, cheap to close: where a design system ships empty, loading and error as components, the question becomes which ones this flow needs, and a designer picks that up inside a quarter.
A thin answer says the tool did not generate them.
Take that answer literally. If the specification was the whole of the contribution, the next brief this person writes will produce the same eight screens, and somebody else will find the missing four in staging.
An answer like that tells you what to expect, which is more than the artifact told you.
Strong, middle or thin, the answer is not a verdict on the portfolio, and reading absence as one is the mistake sitting underneath the question.
The general version, how to evaluate a portfolio when AI clearly made half of it, turns on the same move: grade the account a candidate gives of the work rather than the finish on the work.
The best case against treating the absence as neutral gets stronger with seniority. A designer with ten years of shipped product has probably been paged at 2am over an empty state that rendered the word undefined, and the memory usually survives a change of tool.
A senior portfolio with no failure case anywhere in it is saying something about what shipped.
Write the brief so the states are in scope
One line settles what the exercise measures: the working screens are provided, and the deliverable is everything that happens when the working case does not hold. Hand over the happy path yourself, generated or drawn, and the round stops measuring who owns which tool. What it measures instead is who can enumerate a product's failure surface.
Plant three defects in what you hand over. An error toast carrying a code and no next step. A list with no empty case. A form that validates only on submit.
Ask which one gets fixed first, and what each costs to leave in. For engineers, seeding a repo makes an exercise test checking rather than typing, and the design version is the same move.
Cap the whole thing at forty-five minutes and let candidates use whatever they use.
Run the same brief for everyone, because an informal design exercise is a selection procedure under the same 1978 US federal rules that cover a scored test.
Any measure, combination of measures, or procedure used as a basis for any employment decision is a selection procedure in the Uniform Guidelines, reaching from paper-and-pencil tests through informal or casual interviews and unscored application forms 5.
The duty that follows is narrower than the definition. Validation is required where a procedure shows adverse impact, not everywhere, and the 1978 text names no software. The definition settles the category, and an unscored conversation about missing states sits inside it. What your own round owes as a result is a question for counsel.
The eight screens go back on the desk at the end, unchanged. What is different is the column beside them: the states the designer named, the ones they missed, and the one they said they would build first.
That column is the work sample. The screens were only the prompt for it.
Common questions
Is a missing error state a craft gap or a prompting gap?
Usually neither, on its own. A generated screen renders the case the request described, and design briefs describe the working case, so the absence is what the request produced rather than a reading of the designer. The gap shows up in the next move: ask the designer to list what their own screens do not yet handle, before pointing at anything. Someone who names the states, the words each one needs and the case they would still test knows the job's surface area. Someone who names a category instead has a craft gap that a design system closes inside a quarter.
What exactly should you ask a designer about missing states?
One sentence, the same for everyone who submits: name every state these screens still owe, and what each one has to say. Ask it before naming anything you noticed, because a prompt that names the gap collects agreement instead of evidence. Then push once on the entry they listed first. What does that screen say, in words, and what does the user do next? Someone who has shipped it answers with a sentence. Someone reading off a checklist answers with a category.
How many states should a designer be able to name?
No fixed number, and counting is the wrong instrument. A typical data-backed flow owes at least these: empty on first run, empty after a filter, loading under two seconds, loading that never resolves, a recoverable error, an error where the work is lost, signed out, wrong role, and the ugly-data cases such as a very long name or a list of hundreds of rows. What matters more than the count is whether the entries are screens with words attached rather than category labels, and whether the designer ranks which one to build first.
Does a design system make this question pointless?
It changes the question rather than removing it. Where empty, loading and error ship as components, nobody has to invent them, so the skill on show is choosing which apply to this flow and writing the words inside them. Ask which component the designer would reach for when a request never resolves, and what the copy says when the work is unrecoverable. A design system supplies the container. It never supplies the sentence, and the sentence is where the judgment becomes visible.
Should the design exercise ban AI tools?
Only if the job does. Banning the tool at the exercise while the team uses it daily tests something the role never asks for. The more useful change is to the brief: hand over the working screens yourself, generated or not, and make the deliverable everything that happens when the working case fails. That version cannot be finished by prompting for a prettier frame, and it stays comparable across candidates who arrive with different tools.
Is "the tool did not generate them" a disqualifying answer?
No, and treating it as one wastes the information in it. Read it literally: the specification was the whole of the contribution, which tells you what the next brief this person writes will produce and what somebody else will find in staging. That is a scope statement about how they work, not a character finding. Whether it decides the hire depends on level. For a junior designer it is a gap a states checklist closes. For a senior one, on a flow they chose to submit, it is the finding.
References
- 1. 10 Usability Heuristics for User Interface Design ✓ nngroup.com Supports the April 1994 publication date, the 1990 origin with Rolf Molich, heuristic 1 ("The design should always keep users informed about what is going on, through appropriate feedback within a reasonable amount of time") and heuristic 9 (error messages "expressed in plain language (no error codes), precisely indicate the problem, and constructively suggest a solution").
- 2. Web Content Accessibility Guidelines (WCAG) 2.2 ✓ w3.org Supports the claim that error states are conformance requirements: Success Criterion 3.3.1 Error Identification is Level A ("If an input error is automatically detected, the item that is in error is identified and the error is described to the user in text") and 3.3.3 Error Suggestion is Level AA. W3C Recommendation dated 12 December 2024.
- 3. Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality (Working Paper 24-013) ✓ mitsloan.mit.edu Supports the claim that people cannot tell when a task sits outside what the model handles: on one task chosen to lie outside AI capability, consultants using GPT-4 were correct 19 percentage points less often on average than a control group correct 84.5% of the time.
- 4. Framework for AI Fluency ✓ ringling.libguides.com Supports the named competencies the designer's list is being read for: Discernment as "accurately assessing the usefulness of AI outputs", Diligence as "taking responsibility and vouching for final products created using AI", and Deployment Diligence as verifying and vouching for AI-assisted output. Also supports the caveat that the framework publishes no levels or scoring anchors.
- 5. 29 CFR Part 1607 - Uniform Guidelines on Employee Selection Procedures (1978), sections 1607.16(Q) and 1607.3(A) ✓ govinfo.gov Supports the claim that an informal design exercise is a selection procedure: the Guidelines define one as any measure, combination of measures, or procedure used as a basis for any employment decision, reaching from paper-and-pencil tests through informal or casual interviews and unscored application forms, with validation required where adverse impact appears.
5 sources, numbered by first appearance. Every one was opened and checked against the claim it carries. How Olive sources claims
General guidance for hiring teams. What works at one company and one volume may not transfer to yours.
Olive assesses how a person works with AI. It does not detect AI-written documents, and it never produces a score, a ranking, or a match percentage for a person. Candidates read the same report the employer reads.