Assessment design
Grade a Data Analyst on What the Model Cannot See
Evaluate a data analyst on the part of the work the model cannot see: what the table means, which rows never arrive, whether a join multiplied a total, and what sits in the denominator. Build the exercise on a dataset carrying a real defect (a soft-delete flag, a day boundary in the wrong timezone, a segment redefined mid-year) rather than a puzzle in the SQL. Ask for the number and for the reason to doubt it. A figure returned with the caveat that would change it is the pass.
The takeThe reflex is to make the SQL harder, and it buys almost nothing. A query a candidate cannot write in an hour is a query the assistant writes in twenty seconds, and difficulty in the language was never what separated good analysts from bad ones. What separated them was the willingness to be the one person in the room who asks what the denominator counts. Hire for that and accept that it looks slower on the day. The analyst who ships a clean chart from a query that ran is the one whose mistakes reach a board slide.
Where Olive fits
Open a role and see what the work shows
Olive's assignment is grounded in one occupation, and data and analytics is one of the twelve. What comes back is six findings a person wrote or edited, each anchored to a timestamped moment in the session, granted to the candidate in the same form the employer reads.
Assess your shortlistThe Model Took the Typing and Left the Denominator
A model turns a question into syntactically valid SQL faster than anyone on the team can, and it draws a passable chart from the result without being asked twice. What it does not hold is the thing the query is about: which rows the table stopped receiving in March, whether a join fanned one order out into four, and what the denominator counts.
Fangyu Lei and colleagues built Spider 2.0 out of 632 text-to-SQL problems taken from real enterprise workflows, on databases that often carry more than 1,000 columns. A code agent framework running on o1-preview solved 21.3% of them 1.
The same framework solved 91.2% of Spider 1.0, the earlier academic benchmark 1.
So the gap does not sit in the language. What Spider 2.0 adds is the database: problems drawn from enterprise use cases, on tables sourced from real data applications that often carry more than 1,000 columns, which is where the difficulty went.
An earlier benchmark named the reason in its own abstract. BIRD, built from 12,751 question-and-SQL pairs across 95 databases, put its emphasis on database values: dirty contents, and knowledge a question needs that the schema never holds 2.
ChatGPT reached 40.08% execution accuracy on BIRD against 92.96% for a human 2.
Both results are dated and both describe a benchmark. Model versions have moved since o1-preview, the tasks were assembled by researchers rather than taken from an employer's own data, and no benchmark score predicts what one candidate's assistant will do on your tables.
What survives the caveats is the shape of the two numbers: the tidy task is solved and the messy one is not.
In December 2025, Indeed Hiring Lab found nearly 45% of US data and analytics postings carried an AI-related term, against about 15% in marketing 5. A mention is only a mention. But it puts AI language in nearly half the postings for the role you are hiring for, which is where the evaluation problem starts.
Build the Exercise on a Defective Table, Not a Harder Query
Hand the candidate a dataset with something wrong in it, and a question whose honest answer depends on catching the flaw. The point of the exercise is to find out whether a person reads the data before trusting a number, and a puzzle in the SQL cannot test that, because the assistant solves the puzzle.
Five defects earn their place, because each one produces a number that looks right:
- A soft delete. A deleted_at column nobody mentioned in the brief. Every count runs high and nothing in the result says so.
- A day boundary in the wrong timezone. The warehouse stores UTC and the business reports local. Daily totals shift, and the month-end boundary moves with them.
- A join that fans out. One order matched to three shipment rows becomes three orders. Revenue triples and the chart still slopes the right way.
- A definition that changed mid-year. Active user was rewritten in June. The year-over-year line now compares two different populations.
- A partial final period. The last month is nine days old. Every trend ends in a cliff that is not in the business.
None of them is a trick. Every one is an ordinary Tuesday in a real warehouse, which is why a candidate who has done the job recognises the shape before they find the instance.
The engineering version of the same design is worked out in seeding a repo so the exercise tests checking rather than typing. Analytics needs one change to it. The defect belongs in the rows, not in the code, because a planted bug in a query is a code-reading exercise and the model reads code.
Raising the difficulty of the query aims at the wrong half. The academic benchmark a hard SQL question most resembles is the one a code agent already clears at 91.2% 1. A question phrased to make the assistant stumble is a question phrased unlike anything the job will ask.
Say in the brief that the assistant is allowed and expected. A rule invented after reading one submission reaches one candidate, which turns the round into two different tests.
Ask for the Number and the Reason to Doubt It
Ask for two deliverables and grade them separately: the figure, and the one thing that would move it. A candidate who reports a number, names the rows excluded from it, and says which direction the true value sits in has done the analyst's job. A clean chart with no caveat under it has not, however well the SQL reads.
The evidence for grading the caveat comes from a study of consultants. In a Harvard Business School field experiment, 758 Boston Consulting Group consultants worked realistic tasks, one of which was deliberately chosen to sit outside what the model could do 3.
Correct answers came 19 percentage points less often from the consultants using GPT-4 on that task than from the control group 3.
The direction inside the result is worth more than the headline. The group given a prompt-engineering overview did worse than the group given none, down 24 points against 13 3. More coaching on the tool, a worse answer, on the one task where the tool was wrong.
The experiment is narrow by construction: one task, one sample, a 2023 model. What it shows is also the whole design problem, because the people holding the output could not tell which side of the line their task sat on.
A 2026 study points the same way from the other side. Engineering students given ChatGPT during open-book take-home exams had to submit their interaction transcripts with their solutions, and some of the strongest evidence of reasoning turned up where a student pushed back on an incorrect or incomplete answer. The author concludes that correctness of the final answer alone may no longer be sufficient evidence of comprehension 4.
There was no control group, and the setting was a single course. A transcript in a hiring round can also be curated afterwards. Read the study as a direction, not as a method.
For the figure itself, reperform it.
Rerunning one number against the source beats reading somebody's workbook, which is the move an accounting lead makes on a candidate's automated reconciliation.
One scoring case is worth settling before you meet the candidate who forces it: an unfinished submission against a finished wrong one. Decide it in the rubric before the debrief.
How Do You Read a Chart the Model Drew?
Read the axes, the filter and the last bar first, and read the shape of the line last. A chart the model drew is a rendering of whatever the query returned, and the query returned whatever the schema allowed it to. Ask which category is missing, whether the final period is complete, and where the y-axis starts.
The written summary is where the confident wrong sentence lives. A model handed a table narrates it fluently, cause included, and the cause is sometimes an artifact of the join. Ask the candidate which sentence in their own summary they would delete if the number moved by a tenth.
A candidate who cannot name one has not read their own summary.
Three questions do most of the work in a twenty-minute walkthrough. What one row in this table means. Which rows the query never saw. What the recommendation becomes if the denominator is the other one.
Even the unscored version of that walkthrough sits inside US federal selection law. The Uniform Guidelines define a selection procedure as any measure used as a basis for an employment decision, and say the definition reaches everything from performance tests through informal or casual interviews and unscored application forms 6.
Same table, same question, same rubric, for everyone who gets that far, and keep what each reader wrote. Confirm current obligations with employment counsel.
A planted defect tests familiarity as much as habit, which is where the design is weakest. Someone three days from your schema can miss a soft-delete column for reasons that have nothing to do with whether they check their work. Sorting a knowledge gap from a verification failure is its own question, and one exercise does not settle it.
What the round selects for, in every version of it, is closer to hiring someone who can tell when the model is wrong than to hiring someone who can produce a query. The clean chart still arrives, in seconds, from a query that ran.
What gets read now is the sentence underneath it, naming the rows that never came.
Common questions
What should a data analyst take-home ask for now?
A specific business question, a dataset with a known defect in it, and two deliverables: the figure, and the one thing that would move the figure. Cap the take-home at ninety minutes and say the assistant is allowed. The defect should be ordinary rather than clever, such as soft-deleted rows, a timezone boundary, a definition that changed mid-year, or a join that fans out. Grade the caveat first and the query second. A submission that reports a number without naming what it excludes is a fluent answer to a question nobody can act on.
Should you ban AI on an analytics exercise?
Not if the job permits the assistant. Banning the assistant tests something the role never asks for and pushes the exercise toward speed of typing, which is the part that got cheap. State in the brief that the assistant is allowed, ask for the reasoning alongside the result, and grade the reasoning. The one real exception is a data policy no approved model satisfies, which is a condition of the job and belongs in the brief as one line.
How do you tell a real caveat from one the model wrote?
A real caveat names rows and a direction. It says which records were excluded, why, and whether including them would push the number up or down. A generated caveat names a category of risk instead: data quality issues, potential limitations, results should be interpreted with caution. Ask one follow-up: how much. Someone who traced the exclusion can size it or say plainly that they could not, and someone who inherited the sentence from a summary moves to generalities and stays there.
Can you still ask SQL questions in the interview?
Yes, but not about syntax. Ask what one row in a given table represents, which rows a filter silently drops, what a join multiplies, and what changes if the grain of the table changes. Those questions are about the data model rather than the language, and an assistant cannot answer them without the context that lives in the candidate's head or in your documentation. A whiteboard window function tests recall of something the person will look up in four seconds on the job.
What if the candidate misses the planted defect?
Ask before scoring. Missing a soft-delete column can mean the person does not check their work, or it can mean they have been in your schema for forty minutes and had no reason to suspect it. The follow-up separates the two cheaply: show them the column and ask what it does to the number they reported. Someone who recomputes on the spot and names the direction has the habit and lacked the context. Someone who defends the original figure has neither.
How many people should read the submission?
Two, filling the same short rubric separately before they speak to each other. The columns that survive a second reader are concrete ones: which rows were excluded, what the exclusion did to the figure, what the candidate checked the number against, and which claim in the summary is unsupported. A column for good judgment cannot be scored twice the same way. Track how often the two readers disagree, because a rubric nobody can apply consistently is a rubric that is measuring the reader.
References
- 1. Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows ✓ arxiv.org Supports the claim that the difficulty in analytics moved out of SQL syntax and into understanding real data. The abstract states: "We introduce Spider 2.0, an evaluation framework comprising 632 real-world text-to-SQL workflow problems derived from enterprise-level database use cases. The databases in Spider 2.0 are sourced from real data applications, often containing over 1,000 columns... Our evaluations indicate that based on o1-preview, our code agent framework successfully solves only 21.3% of the tasks, compared with 91.2% on Spider 1.0 and 73.0% on BIRD."
- 2. Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs ✓ arxiv.org Supports the 12,751 pairs across 95 databases, the emphasis on database values, and the accuracy gap. The abstract states: "we present Bird, a big benchmark for large-scale database grounded in text-to-SQL tasks, containing 12,751 pairs of text-to-SQL data and 95 databases... Our emphasis on database values highlights the new challenges of dirty database contents, external knowledge between NL questions and database contents, and SQL efficiency... even the most effective text-to-SQL models, i.e. ChatGPT, only achieves 40.08% in execution accuracy, which is still far from the human result of 92.96%."
- 3. Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality (Working Paper 24-013) ✓ mitsloan.mit.edu Supports the claim that people cannot reliably tell when a task sits outside the model's capability: on one task chosen to sit outside it, consultants using GPT-4 were 19 percentage points less likely to be correct than the control group, and the condition given a prompt-engineering overview fell further (24 points) than the condition given none (13 points).
- 4. Reimagining Assessment in the Age of Generative AI: Lessons from Open-Book Exams with ChatGPT ✓ arxiv.org Supports the claim that the graded object shifts from the answer to the judgment applied to it: engineering students permitted ChatGPT on open-book take-home exams submitted interaction transcripts, the strongest reasoning evidence appeared where students evaluated incorrect or incomplete responses, and the paper concludes correctness of final answers alone may no longer provide sufficient evidence of comprehension.
- 5. January 2026 US Labor Market Update: Jobs Mentioning AI Are Growing Amid Broader Hiring Weakness ✓ hiringlab.indeed.com Supports the December 2025 figure that nearly 45% of US data and analytics postings contained an AI-related term, against about 15% in marketing, used only to show the tool sits inside the role's postings and not as a measure of required skill.
- 6. 29 CFR Part 1607 - Uniform Guidelines on Employee Selection Procedures (1978), sections 1607.16(Q) and 1607.3(A) ✓ govinfo.gov Supports the claim that an analytics exercise, and an unscored conversation about the same work, are the same category of thing under federal selection law: a selection procedure is any measure used as a basis for an employment decision, covering the range from performance tests through informal or casual interviews and unscored application forms.
6 sources, numbered by first appearance. Every one was opened and checked against the claim it carries. How Olive sources claims
General guidance for hiring teams. What works at one company and one volume may not transfer to yours.
Olive assesses how a person works with AI. It does not detect AI-written documents, and it never produces a score, a ranking, or a match percentage for a person. Candidates read the same report the employer reads.