SOC 23-2011 · Paralegals and Legal Assistants
Legal operations, tested on the work
Take a position on a customer's contract from a packet where the playbook, the system record and the signed precedent do not agree.
The assignment
What the candidate is asked to do
The candidate produces a short position memo over a counterparty's agreement, a company playbook, an executed precedent and a contract-system record, with an AI assistant available throughout. The paper is fluent, the signature date is already promised, and nothing inside the conversation settles what the agreement actually says.
Time: 40 to 60 minutes, hard-capped 15 to 20 minutes past whatever the case states and never above 75, with a short written debrief after. Paused time does not count against either.
SOC 23-2011
Paralegals and Legal Assistants. The benchmark is derived from live postings mapped to that code and re-derived each quarter.
What the reviewer watches for
The legal operations failure modes
Each maps to a scored dimension, so the reviewer's finding arrives with the moment attached.
Is what the memo has to decide named before any clause is marked up?
Whether the opening move asks what is true, what is wrong, or what the options are — and whether a constraint or criterion is named in the same breath.
Is the executed agreement opened, or is the system record taken as the history?
Whether sources are demanded for a specific claim rather than requested in general — and whether at least one is actually opened.
Which positions are the candidate's own, and which arrive from the panel intact?
Whether the split between the assistant's work and the candidate's own acts is deliberate rather than convenient, read from the record of both.
Why this bank
Why we built the legal operations bank
Legal operations is where a fluent wrong answer stops being a draft and becomes a signed obligation — the one bank whose failure outlives the quarter, because the clause runs for the term and the counterparty's counsel reads it either way.
This bank is live and has passed author review at launch quality.
https://olive.is/benchmarks · Edition 2026 Q3Named up front
Take a position on a customer's contract from a packet where the playbook, the system record and the signed precedent do not agree.
The construct
Six dimensions, scored separately
- D1 · Problem framing Did the first move go after understanding, or straight for output?
- D2 · Evidence sourcing Did they demand evidence for the claim that mattered?
- D3 · Delegation boundary What did they keep, and what did they hand over?
- D4 · Working structure Did anything exist between the brief and the answer?
- D5 · Output rejection Was anything the assistant produced refused, and on what grounds?
- D6 · Verification Was anything tested against the world, and did the result change something?
SOC 23-2011
Run one legal operations role, free
Ten attempts a month against this bank, with the full six-dimension report on every one. No card, no salesperson.