Assessment design
Reperform Three Items, and Interview the Disagreements
Don't try to read a candidate's AI-built reconciliation workbook. Pick three items out of its output, a reconciling item, an accrual and an allocation, redo them yourself or with a staff accountant from the same source data, and compare. The disagreements are the interview: ask where each figure came from and what it was tied back to. What the exercise tests is whether each number traces to a document, and whether the candidate can name which ones she checked and which she accepted.
The takeThe instinct to learn the candidate's tools first is the expensive mistake here. An accounting lead already owns a technique built for judging work nobody watched. Reperformance, by design, asks nothing about how that work was done. What deserves suspicion is the opposite reflex: a loop that grades an accountant on how fluently she talks about automation, which is the one thing about her the close will never care about.
Where Olive fits
Open a role and see what the work shows
If you build this in-house, the expensive parts are the answer key and the evidence behind each judgment. Olive ships twelve authored cases per occupation, accounting and audit among them, and returns six separately-evidenced findings, each anchored to a moment in the session rather than to a score.
Assess your shortlistStop Auditing the Workbook and Redo Part of It
Reading someone else's workbook is not how an audit settles whether a number is right. The PCAOB's audit evidence standard, adopted in 2010, lists reperformance as a procedure in its own right: the independent execution of procedures or controls that were originally performed by company personnel 1.
Recalculation, one paragraph earlier, is narrower still, just checking the mathematical accuracy of information 1. Neither definition asks the auditor to follow the preparer's method.
Reperformance was built for exactly the position you are in.
So stop asking to be walked through the automation. A candidate can narrate a pipeline you have no way to evaluate, and a fluent account of a bad process is hard to tell from a fluent account of a good one. The output is the part you can check without her help.
Tool fluency is not the question in front of you either. Whether the reconciliation ran through Python, a model, a macro or three hours of manual tick-marking is a fact about the workbook. What you are hiring is a person whose numbers tie.
Pick the Three Items From Your Own Close
Choose one reconciling item, one accrual and one allocation, from a month you have already closed. Each fails in a different place, which is why three beats ten of the same kind. A reconciling item either ties to an outside statement or it does not. An accrual rests on a judgment about period and amount. An allocation rests on a driver somebody chose.
Build the exercise out of your own ledger rather than a textbook. Anyone can pattern-match a clean bank reconciliation. Nobody pattern-matches your intercompany elimination, or the accrual the operations team argues about every quarter.
Then work the three items yourself, inside the time budget you plan to give her. Writing the answer key starts with doing the assignment yourself, and in accounting the key has three columns: your figure, the document you tied it to, and the tolerance you will accept as agreement.
Do that before any candidate sees the file.
One hour of senior time buys the key, and the key is reusable. Every candidate after the first is measured against work already done, so the hour stops being a per-candidate cost.
What the Disagreements Are For
Every gap between your number and hers is a question with a known answer behind it, which is what makes the comparison an interview rather than a quiz. Ask where the figure came from and what it was tied back to. Ask the same about the items where the two of you agreed, because agreement can come from a source document or from last month's file copied forward.
The answer to listen for names a boundary. Nobody checks everything, and an accountant who claims she did either did not, or means something looser by the word.
A strong answer sounds like a line drawn on purpose. Three largest reconciling items traced to the bank statement. Intercompany balance confirmed against the counterparty ledger. Everything else accepted because the variance sat under a threshold she can state out loud.
Confidence is the part that travels badly. In a 2023 field experiment with 758 Boston Consulting Group consultants, one task was chosen to sit outside what GPT-4 could do.
The control group answered it correctly 84.5% of the time. The two assisted groups averaged 19 percentage points worse 2.
A single task, a single sample and a 2023 model do not carry far. What they do carry is narrower than the headline and more useful here: those consultants could not tell which side of the line the task was on. Expect the same of a candidate on the reconciling item that looks like all the others.
The same move works anywhere the artifact is unreadable and its sources are not. A legal lead facing an AI-assisted memo can hand the candidate a memo to check instead of one to write. Different document, identical logic.
Score the Three Items Before You Open Hers
Write the scoring sheet before the first submission arrives, because a comparison needs a constant. Three rows, one per item, and three columns: does her figure agree with yours inside tolerance, does she name the document it traces to, and can she say what she accepted without checking. Fill it in for every candidate, then add it up.
Adding it up is not a formality. Kuncel, Klieger, Connelly and Ones pooled selection and admissions studies and found applicant data combined mechanically by a formula correlated .44 with job performance, against .28 when the same kinds of data were combined by expert judgment 3.
A unit-weighted checklist counts as a formula, and the paper says so.
The job performance half of that comparison rests on nine studies, so take it as a direction rather than a coefficient to bank. The direction is enough here.
Three filled rows beat a feeling about who seemed sharper on the call.
Consistency is also the legal shape of a hiring exercise. In the United States, the Uniform Guidelines have defined a selection procedure since 1978 as any measure or procedure used as a basis for an employment decision, reaching everything from performance tests through informal or casual interviews 4.
The validation burden attaches only where adverse impact shows up, so what the definition settles is scope. Even so, it argues for giving every candidate the same three items instead of the ones a resume made you curious about.
Settle one scoring question before it arrives. One candidate returns all three items with two figures wrong; another returns one item tied to a bank statement and says she ran out of time. The sheet has to say which you prefer in advance, which is the general case of scoring an unfinished submission against a wrong one.
Where Reperformance Stops Being Evidence
Reperformance settles one month's output and nothing beyond it. A workbook that ties out in August can break in September, when a new entity arrives or a bank feed changes format, and the exercise you ran will not have seen that coming. The exercise tests the numbers rather than the method, so a candidate who got there by copying the prior file passes on figures alone.
The tie-back conversation is what catches that one, which is why the disagreements earn more time than the matches.
A job-shaped exercise is also a weaker single predictor than the hiring field spent decades claiming. The .54 that made work samples the best predictor on the famous table has been superseded.
Sackett and colleagues estimate .33 instead, drawn from 54 studies covering 10,469 people. Of the 54, 53 tested people already doing the job rather than applicants 5.
The cost is real too.
Three items reperformed is an hour of senior time to build the key and twenty minutes per candidate to compare, and a team already behind on the close will feel both.
The workbook comes back at the end, still unread. It will sit in the shared drive whether or not you ever open it. What you will know about the candidate on offer day is three numbers you reproduced yourself and one conversation about where they came from.
Run the same exercise on your own team next quarter, before the next req opens. The answer tells you whether the bar was hers or yours.
Common questions
Which three items should you pick for a reperformance exercise?
One reconciling item, one accrual and one allocation, taken from a month already closed. They fail in different places, so three of them cover more ground than ten of any one kind. The reconciling item tests whether a figure ties to an outside document. The accrual tests judgment about period and amount. The allocation tests whether the driver was chosen or inherited. Take all three from your own ledger rather than a textbook, because a candidate can pattern-match a clean bank reconciliation and cannot pattern-match your intercompany elimination.
Do you need to understand the candidate's AI tooling to grade this?
No. Reperformance is the independent execution of a procedure originally performed by somebody else, and nothing in it requires knowing how they worked. You reproduce three of the candidate's figures from the same source data, compare, and ask where each came from. Whether the workbook ran on Python, a model, a macro or manual tick-marking is a fact about the file. If the automation matters to the role, it shows up later as a maintenance question: what breaks when a new entity or a new bank feed arrives, and who fixes it.
What if the candidate's numbers match yours exactly?
Agreement is not evidence on its own, so ask the same tie-back questions you would ask about a gap. A figure can match because it was traced to a bank statement, or because the prior month's file was copied forward and nothing changed. The distinction only appears in the answer. Ask which items she checked, what she checked them against, and which ones she accepted without checking. An accountant who can draw that line is doing the job. One who says she verified everything either did not, or means something looser by the word.
How long does a reperformance exercise take to run?
About an hour of senior time once, plus roughly twenty minutes per candidate. The hour goes into the answer key: working the three items yourself, recording the document behind each figure and the tolerance you will accept as agreement. That key is reusable across the whole pipeline, so the per-candidate cost is the comparison and the conversation. Give the candidate the same time budget you gave yourself, and say in the brief that AI is allowed, because the exercise grades the tie-out rather than the tool.
Does a hiring exercise like this count as a test you have to validate?
A hiring exercise like this sits inside the US federal definition. The Uniform Guidelines have defined a selection procedure since 1978 as any measure or procedure used as a basis for an employment decision, covering everything from performance tests through informal or casual interviews. The validation burden attaches only where adverse impact appears, so this is a question of scope, not a mandate to validate every exercise. Give every candidate the same three items, keep the scoring sheets, and confirm the current position with employment counsel rather than treating this as advice.
What if the candidate did the work by hand and never used AI?
Then the reperformance exercise reads the same way, which is the point of choosing it. Reperformance asks whether the numbers reproduce and whether the person can say what each one was traced to. A manual preparer who cannot name a source document fails the same test a model-assisted one fails. Keeping the exercise method-blind also keeps the round comparable: two candidates who worked differently are still being measured against one answer key, one tolerance and one set of three items.
References
- 1. AS 1105: Audit Evidence ✓ pcaobus.org Supports the definitions this article borrows: paragraph .20, "Reperformance involves the independent execution of procedures or controls that were originally performed by company personnel," and paragraph .19, "Recalculation consists of checking the mathematical accuracy of information." Adopted by PCAOB Release No. 2010-004; the page itself links to a separate list of amending releases rather than naming them, and the latest listed there for AS 1105 is No. 2024-007.
- 2. Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality (Working Paper 24-013) ✓ mitsloan.mit.edu Supports the claim that people cannot tell which tasks sit outside a model's capability: with 758 BCG consultants, the control group answered the out-of-frontier task correctly 84.5% of the time against an average 19 percentage point decrease across the two AI conditions.
- 3. Mechanical Versus Clinical Data Combination in Selection and Admissions Decisions: A Meta-Analysis ✓ gwern.net Supports adding up a written scoring sheet rather than forming an impression: mechanical combination correlated .44 with job performance against .28 for clinical combination, on nine studies for that criterion.
- 4. 29 CFR Part 1607 - Uniform Guidelines on Employee Selection Procedures (1978), sections 1607.16(Q) and 1607.3(A) ✓ govinfo.gov Supports the claim that a reperformance exercise is a selection procedure: the definition covers any measure, combination of measures, or procedure used as a basis for any employment decision, from performance tests through informal or casual interviews and unscored application forms.
- 5. Revisiting Meta-Analytic Estimates of Validity in Personnel Selection: Addressing Systematic Overcorrection for Restriction of Range ✓ gwern.net Supports the correction of the work sample validity figure from .54 to .33, based on Roth et al. (2005): 54 studies, N = 10,469, with 53 of the 54 concurrent rather than applicant-stage.
5 sources, numbered by first appearance. Every one was opened and checked against the claim it carries. How Olive sources claims
General guidance for hiring teams. What works at one company and one volume may not transfer to yours.
Olive assesses how a person works with AI. It does not detect AI-written documents, and it never produces a score, a ranking, or a match percentage for a person. Candidates read the same report the employer reads.