Assessment design
Give a Legal Candidate a Memo to Check, Not One to Write
A legal hiring exercise should hand the candidate a memo to check rather than one to write. Build a two-page memo carrying eight cited propositions: some supported, some where the case is real and the proposition is not, one authority since superseded. Ask the candidate to mark each proposition, name the source they would pull to confirm it, and say what they would do at deadline with two still unresolved. Grading takes minutes, because the key was written first.
The takeThe drafting test survived because it looks like the job. It stopped being the job around the time a model could produce a competent first draft in ninety seconds. What a legal lead grades now is mostly format, plus a claim about checking that nobody can audit. Verification stayed scarce and stayed expensive. Test the scarce thing. A candidate who cannot say which of eight propositions is misgrounded on a Tuesday morning will not catch it on a Thursday afternoon either.
Where Olive fits
Open a role and see what the work shows
The two expensive parts to build yourself are the authored case and the evidence behind each judgment. Olive ships twelve authored cases per occupation, legal operations among the twelve occupations it covers, and returns six separately-evidenced findings anchored to moments in the session, with the candidate granted the same report.
Assess your shortlistWhy has the drafting exercise stopped separating anyone?
Because drafting is the part a model does competently, and the exercise ends up grading polish. A memo written with an assistant comes back structured, hedged in the right places, and cited in the right format. What a legal lead needs to know sits underneath the polish, in whether the citations hold, and reading for that means doing the candidate's checking over again.
Nobody has those hours.
A partner reading four submissions with eight citations apiece is being asked to pull thirty-two authorities to grade an exercise that existed to save time. So the grader falls back on a proxy, and the round measures how a candidate writes.
The US federal selection guidelines name neither act, drafting or checking, but they set the limit an exercise has to respect. A selection procedure holds up under a content strategy "to the extent that it is a representative sample of the content of the job," and one "based upon inferences about mental processes cannot be supported solely or primarily on the basis of content validity" 3.
So name the content. In a legal function the memo is a deliverable, and drafting it is now cheap. The act nobody may delegate is deciding that a cited proposition is safe to put in front of a client.
Trading the drafting test for a checking test is already written up in the general case: test the checking, not the production. The legal version is narrower, and narrow enough to key in advance.
Which wrong citation actually costs you?
The one that resolves. A fabricated case gets caught the first time anybody clicks it. The expensive failure is quieter: a citation that opens onto a real, good-law authority saying something adjacent to the proposition it was attached to, or saying it in a jurisdiction that does not govern the client.
Varun Magesh and Faiz Surani, researchers at Stanford, measured that failure with four co-authors and preregistered the dataset before running the queries. The dataset holds over 200 legal queries 1.
The queries ran between March and May 2024, against three commercial legal research tools: Lexis+ AI, Westlaw's AI-Assisted Research, and Thomson Reuters's Ask Practical Law AI 1.
Lexis+ AI answered 65 percent of the queries accurately, the best of the three 1.
Westlaw's AI-Assisted Research answered 42 percent accurately, and a third of its responses contained a hallucination 1.
The authors count a response as a hallucination when it contains either incorrect information or a false assertion that a source supports a proposition 1. The second half of that definition is the harder half to spot.
One example in the paper is exactly the artifact a legal lead worries about.
Asked what standard of review applies to abortion regulations, Lexis+ AI cited Planned Parenthood v. Reynolds, a real case that has not been overturned, for that case's description of Planned Parenthood v. Casey, which has been 1.
The authors are direct about what catching a misgrounded citation costs.
Such errors are "potentially more dangerous than fabricating a case outright, because they are subtler and more difficult to spot," and finding them requires a reader to click through to the cited source, read it, assess its authority, and compare it to the proposition 1.
A hiring round cannot afford that reading. An exercise can measure it in forty minutes.
Two limits ride with those numbers. The evaluation of two products closed in April 2024, LexisNexis released a second-generation tool afterwards, and the paper was a preprint under review when it was posted 1.
But the failure it names does not expire with a product version. A retrieval system that returns a real document, plus a model that overstates what the document holds, keeps producing citations that resolve and do not support.
Build the memo, then plant eight propositions in it
Write one memo, two pages, on a question your function actually gets, and carry eight cited propositions in it. Four should be clean. Two should cite a real authority for something it does not hold. One should rest on an authority since superseded. One should cite the right rule from a jurisdiction that does not govern the client.
Every candidate gets the same memo.
Draft it yourself, or draft it with the assistant and then check every line. Either way the key exists before the exercise does, and that ordering is what makes the grading cheap.
Eight rows and one column for what would settle each: that is the key here. Write the answer key before you send the assignment.
Plant the errors. Do not inherit them.
A planted error can be described in advance, at a difficulty you chose, in a row of the key. An error you happened to find in a model's draft cannot be calibrated, and a second memo built the same way will not test the same thing.
Source the question from a matter that closed six to twelve months ago and strip the client detail.
Assume the packet leaks. A dozen candidates in, the memo is posted somewhere, so build the second one now.
What forty minutes should ask, and in what order
Three things, in order: a verdict on each proposition, the source that would settle it, and one decision under time pressure. The verdict is a mark in one of three columns, supported, not supported, or cannot tell from the memo alone. The middle column is where a candidate who knows the practice separates from one pattern-matching on citation format.
Ask for a source, not a method. Saying they would Shepardize the citation is a sentence anyone can produce on demand.
Naming Reynolds as a case that describes Casey, then pulling Casey to see what survived Dobbs, is the work.
Close on the question the job actually asks. The client needs the memo in an hour and two citations are still unresolved. The candidate has to say what goes in the memo, what goes in the cover note, and what will not be sent at all.
Listen for the propositions the candidate will carry unverified and flag on the face of the document. That is the strong answer. A weak one sends everything or refuses everything, and both are the same failure, which is declining to price the risk.
California's bar has put the duty to check in a form a hiring exercise can borrow.
In the 2026 revision of its practical guidance on generative AI, the State Bar's professional responsibility committee says a lawyer must exercise independent professional judgment "by reviewing, verifying, and correcting AI-generated outputs" 2. Professional judgment, the same passage adds, "cannot be delegated to AI and remains the lawyer's responsibility at all times" 2.
The same guidance reaches past the fake-citation problem, saying competence "requires more than the mere detection and elimination of false or fabricated AI-generated results" 2. Check the rules of the jurisdictions the role covers, since this guidance is California's.
Grade the marks, not the prose
Score the eight rows and nothing else. Each proposition is right, wrong or flagged, and the key already says which. A second grader fills the same eight rows without conferring, and a disagreement then points at a row rather than at whether the memo felt thorough.
Nothing about the writing enters the score.
Agreement is worth measuring. The authors hand-scored every response against a written rubric, then had a fourth labeler re-score a random 48 of them without access to the first labels, reaching 85.4 percent agreement on the outcome and a Cohen's kappa of 0.77 1.
Raw agreement flatters, which is why eight of ten is not agreement until chance is discounted.
Settling the acceptable answer in advance is what makes an exercise structured at all.
Structured is a defined word. The US Office of Personnel Management's 2008 guide gives it three properties: the same questions in the same order, a common rating scale, and interviewers who agree beforehand on what an acceptable answer looks like 5. The definition was written for interviews and carries over to a written exercise without strain.
Somebody who can check a citation and cannot frame the question, or cannot tell a client no, will pass this exercise anyway. It tests one act well and the rest of the job not at all.
Work samples in general predict less than the figure everyone quotes. Sackett and colleagues put work sample validity at .33 in 2022, correcting a .54 that had circulated since a 1974 narrative review 4.
Keep the exercise as one hurdle inside a round.
Expect a lot of candidates to miss the misgrounded citation on the first run. A high failure rate is information about where the bar sits, and not yet a reason to move it: move a cut score for evidence, never for yield.
Reynolds had not been overturned when the paper ran, and it appears in the Lexis+ AI citation list with a positive Shepardization symbol 1. Everything wrong with the citation lives in the sentence it was attached to.
Put that sentence in front of a candidate for forty minutes, and you have watched the one act you cannot do for them at volume.
Common questions
How long should a legal verification exercise take?
Forty minutes of candidate time, and about ten minutes of grading. Eight cited propositions in a two-page memo give you four clean rows and four planted failures, enough spread to separate people without becoming an unpaid afternoon. If it runs longer than an hour, pay for it. The length is set by the key rather than by the memo: eight rows a grader can fill from the key is the unit, and adding propositions past that buys precision nobody uses.
What if the candidate uses AI to check the citations?
Let the candidate use a research tool, and say so in the brief. The exercise asks which propositions hold and what would settle each one, and a candidate who gets there with a research tool has done the job the way the job is done. The planted failures are the kind a research tool can itself produce, a citation that resolves to a real authority saying something adjacent, so an unchecked pass through an assistant can return a confident wrong answer rather than a shortcut.
Does a keyed exercise hold up if a rejected candidate challenges it?
A keyed exercise gives you more to point at than a graded memo does. The US federal Uniform Guidelines say a selection procedure can be supported under a content strategy to the extent it samples the content of the job, and that a procedure resting on inferences about mental processes cannot be supported that way alone. Checking a citation is an observable act with a right answer written down before anyone sat the exercise. Keep the key, the marks and the date. Whether that holds up in a challenge is a question for employment counsel, since none of this is legal advice.
Can a non-lawyer run this exercise?
A non-lawyer can administer the exercise and score it, because scoring is a comparison against a key somebody with the practice knowledge wrote. Building the memo is the part that needs a lawyer: choosing a question the function actually gets, picking an authority that reads as supportive and is not, and deciding which superseded case is fair to plant. Budget two hours of senior time once, then reuse the memo until it leaks.
What does a good answer to the deadline question sound like?
A good answer separates the propositions the memo can carry from the ones it cannot, and it puts the separation in writing. A candidate might send the memo with two propositions marked as unconfirmed in the cover note, name what would confirm them, and give an hour estimate. What matters is that something gets flagged to the client rather than absorbed silently. An answer that sends everything, and an answer that refuses to send anything, both decline to price the risk.
References
- 1. Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools ✓ arxiv.org Supports the preregistered dataset of over 200 legal queries run between March and May 2024, the 65 percent accuracy for Lexis+ AI and 42 percent for Westlaw AI-Assisted Research with one third of Westlaw responses hallucinated, the definition of a hallucination as incorrect information or a false assertion that a source supports a proposition, the Planned Parenthood v. Reynolds example cited for its description of the overturned Casey, the quoted sentence that such errors are subtler and more difficult to spot, the Reynolds positive Shepardization symbol footnote, the 85.4 percent inter-rater agreement and Cohen's kappa of 0.77 on 48 re-scored responses, and the paper's own limits (April 2024 close, LexisNexis second-generation release, preprint under review).
- 2. Practical Guidance for the Use of Generative Artificial Intelligence in the Practice of Law ✓ calbar.ca.gov Supports the quoted duty to exercise independent professional judgment by reviewing, verifying, and correcting AI-generated outputs, the statement that professional judgment cannot be delegated to AI and remains the lawyer's responsibility at all times, and the line that competence requires more than the mere detection and elimination of false or fabricated results. This 2026 revision replaces the 2023 guidance and is California's, not a national rule.
- 3. 29 CFR Part 1607 - Uniform Guidelines on Employee Selection Procedures (1978), section 1607.14(C)(1) ✓ govinfo.gov Supports both quoted clauses: that a selection procedure can be supported by a content validity strategy to the extent that it is a representative sample of the content of the job, and that a procedure based upon inferences about mental processes cannot be supported solely or primarily on the basis of content validity.
- 4. Revisiting Meta-Analytic Estimates of Validity in Personnel Selection: Addressing Systematic Overcorrection for Restriction of Range ✓ gwern.net Supports the correction of work sample validity from .54 to .33, and that the .54 traces to a 1974 narrative review rather than to a current estimate. Cited here only to keep the exercise sized as one hurdle in a round.
- 5. Structured Interviews: A Practical Guide ✓ opm.gov Supports the three defining properties of a structured procedure used here: the same questions in the same order, a common rating scale, and agreement in advance on what an acceptable answer looks like. The guide is written for US federal interviewing and is guidance rather than statute.
5 sources, numbered by first appearance. Every one was opened and checked against the claim it carries. How Olive sources claims
General guidance for hiring teams. What works at one company and one volume may not transfer to yours.
Olive assesses how a person works with AI. It does not detect AI-written documents, and it never produces a score, a ranking, or a match percentage for a person. Candidates read the same report the employer reads.