Assessment design
Build the Support Hiring Test From the Tickets Your Bot Got Wrong
Build the support hiring test out of your own escalation queue. Take three tickets the bot answered wrong and hand the candidate the customer's message, the bot's reply and the account record. Ask what the customer actually needs, what reply they would send, and what they would change upstream so the ticket stops coming back. Grade the diagnosis first, the catch that the bot was confidently wrong second, and tone last. Building the exercise also shows what the knowledge base is missing.
The takeThe canned ticket was never measuring much, and deflection exposed that rather than caused it. Grammar and warmth were always the easy half, and a model now supplies the easy half free. What is left in the queue is the ticket where the customer described the wrong problem, so the person to hire is the one who reads past the description. A team cannot write this exercise until someone has read the escalation queue closely enough to say what the bot keeps getting wrong. Reading it is the first half of the job.
Where Olive fits
Open a role and see what the work shows
Building this in house, your escalation queue supplies the case, and the costly part is the record behind each judgment. Olive ships authored cases for twelve occupations, customer support not among them, and returns six separately evidenced findings, each anchored to a timestamped moment in the session and granted to the candidate in the same form the employer reads.
Assess your shortlistWhat did the bot take off the queue?
The bot took the tickets with one correct answer and a documented place to find it. Password resets, shipping status, the refund window, the plan comparison that already lives on a pricing page. What escalates now is the ticket where the article was missing, or was right and did not fit, or where the customer's stated problem was not their real one. The old exercise tested the first pile.
The nearest measurement comes from support agents working beside an assistant.
Generative AI at Work, a 2023 National Bureau of Economic Research working paper, followed the staggered rollout of a conversational assistant to 5,179 customer support agents at one software firm. Issues resolved per hour rose 14% on average: 34% for novice and low-skilled agents, and close to nothing for the experienced, highly skilled ones 1.
The split is the part to read. The assistant supplied what a new agent did not have yet, and it added almost nothing where experience was already carrying the work.
The paper measured an assistant beside the agent rather than a bot answering customers, and the rollout ran across 2020 and 2021. So treat the direction as an inference from the paper, and a plausible one: automation likely absorbs the part of the job a new hire would otherwise be learning on.
Your ninety-day ramp was built out of exactly that part.
So the canned ticket has stopped separating anyone. What a work sample should test now that AI can produce it is the general form of the problem. Support meets it in the plainest form, because deflection deleted the tested task outright.
Take three escalations the bot got wrong
Open the escalation queue and pull three real cases where the bot answered confidently and the answer was wrong. Skip the ones where it said it could not help, because those escalated correctly. You want the reply that sounded right, went out, and came back. Strip names, account numbers, anything else that identifies a customer, and anything a contract keeps private. Keep everything else, timestamps included.
Each case travels as four documents: the customer's opening message, the bot's reply, whatever the customer wrote back, and the account record an agent would have had open. No summary. A summary is your diagnosis, and the diagnosis is the thing being tested.
Real tickets beat invented ones for a reason a regulator would recognize.
Representative sample is the operative phrase. Under the US federal Uniform Guidelines on Employee Selection Procedures, adopted in 1978 and still in force, a selection procedure can be supported by a content validity strategy to the extent that it is a representative sample of the content of the job 2.
The same section asks a user to show that the behaviors demonstrated in the selection procedure are a representative sample of the behaviors of the job in question 2.
An invented scenario can be representative too, but only as far as your picture of the queue is accurate.
A real task you already solved, with the outcome kept, is the standing answer to where a work sample comes from. Support has the cheapest version of it. The queue is already written down and the resolution is already sitting in the thread.
Pulling the three cases costs an afternoon and pays for itself before anyone is interviewed.
Sorting escalations by what the bot got wrong is also a read of the knowledge base. An article that was missing, one that was right and unfindable, one that was right and out of date: three different fixes, and only the sort tells you which.
What should a support candidate hand back?
Three things per case, in this order: what the customer actually needs, the reply they would send, and the one upstream change that stops the ticket coming back. The diagnosis goes first on the page, before the reply, so it can be read on its own. Ninety minutes for three cases, an AI assistant allowed and expected, and a line saying where it helped and where it was wrong.
A case makes the order obvious. A customer writes that the export has been broken for two days. The export works.
Their saved filter is set to a date range that closed on Friday. The bot matched on the word broken and sent the export troubleshooting article with a cheerful sign-off.
The stated problem was the export. The real one was the filter.
A reply that fixes the export fixes nothing. The ticket comes back on Thursday under a new subject line.
Allow the assistant, because the job allows it. Someone who pastes the customer message into a model and sends back what it produces has shown you the thing you most need to see, on a case where you already know the resolution.
The note about where the model was wrong is graded.
The upstream question is the one most support exercises skip. Ask what the candidate would change so the ticket stops arriving: the article, the macro, the bot's fallback, the wording inside the product.
A candidate who only writes replies writes a good reply and nothing else, and a queue that never shrinks is made of good replies.
Grade the diagnosis before the reply
Score three columns, weighted in this order: whether the candidate named the problem underneath the stated one, whether they caught that the bot's reply was confidently wrong, and whether their reply commits the company only to what it can deliver. Tone comes fourth, because a warm, grammatical paragraph is now free.
Here the answer key is unusually cheap to write, because the real ticket already resolved and the thread says what closed it. Write the answer key before the assignment goes out, put that resolution in it, and the graders' disagreements shrink to whether the candidate reached it.
Catching a confident wrong answer is harder than it sounds, and one measurement of it comes from a consulting firm.
In a 2023 Harvard Business School field experiment, on one task deliberately chosen to sit outside the model's capability, consultants using GPT-4 answered correctly 60% and 70% of the time against 84.5% in the control group 3.
Behind those figures sit one task, one sample, one 2023 model. What a queue can borrow from it is narrower: on that task, the consultants could not tell which side of the capability line it sat on.
Coaching did not help on this task. The group given a prompt-engineering overview came out worse, down 24 points, than the group given none, down 13 3.
A candidate who diagnoses two cases carefully and leaves the third open tells you more than one who answers all three fluently and misses the filter. Scoring an unfinished submission against a wrong one is its own decision. Settle it in the key, before the debrief.
What the exercise is about may do more work than what shape it takes, though the evidence for that comes from knowledge tests, not work samples. Sackett and colleagues' 2022 re-analysis reports 164 job knowledge studies at a mean observed validity of .22, and the 59 of them using tests built for the job in question at .31, rising to .40 once corrected for unreliable performance ratings 4.
The .31 sits next to the .22 inside one meta-analysis rather than in a controlled trial, and job knowledge tests presuppose people who already hold the knowledge. Read it as a direction.
Where this exercise stops telling you anything
The queue records one bot configuration, not the job. Retune the bot next quarter and some of your three cases stop being live failures. And the queue only holds the errors that came back: a confident wrong answer the customer accepted, acted on and never disputed leaves no escalation at all.
Those are the expensive ones, and the exercise cannot see them.
Expect less of the result than the older literature promised. The claim that work samples predict best at .54 is superseded: the 2022 re-analysis puts work sample validity at .33, from 54 studies of which 53 tested people already doing the job 4.
Two failures also look identical on the page. A candidate who missed the bot's error because they have never seen your product is not the same as one who never checks. Missing a planted error can be a knowledge gap rather than a habit.
Ask what they would have needed in order to catch it. A weak answer names nothing. A strong one names the account record and the field inside it.
One line in the Guidelines belongs in the rubric review. A selection procedure based upon inferences about mental processes cannot be supported solely or primarily on the basis of content validity 2. So write the rubric in behaviors, and take the validation question itself to counsel:
- Weak, because it is an inference: the candidate noticed that the bot was wrong.
- Strong, because it is on the page: the candidate wrote that the closed date filter, and not the export, was the cause.
A candidate can dispute the second one. Two graders can agree on it.
The second case you pulled has an article behind it that says the wrong thing. Fix the article this week and that case stops being a live failure. The exercise decays on purpose, so rebuild it each quarter out of whatever the bot is getting wrong then.
The hire you want is the person who would have written that fix.
Common questions
Should candidates be allowed to use AI during the support exercise?
Yes, if the job allows it, and the job almost certainly does. Banning the assistant tests a version of the work nobody performs. Say in the brief that an assistant is permitted and that a short note is part of the deliverable: what was generated, what was checked, and against what. The note is graded and the polish is not. A candidate who pastes the ticket into a model and sends back the first answer has told you something useful, and you already know how that ticket resolved.
What if there is no escalation queue to pull from?
Use the closest record that already exists: chats where the customer restated the problem, tickets reopened within a week, or refunds issued after a support contact. Each one is a first answer that failed. If none of those exists, the exercise is not the first problem to solve. A support team that cannot say which answers came back has no way to grade a candidate's diagnosis either, because there is no key to grade it against.
Does grading the bot's mistake test the candidate or the bot?
The candidate, as long as the rubric records a behavior rather than an impression. Do not score whether they noticed anything. Score whether they wrote down what the bot got wrong and what the correct answer was. A candidate who accepts the bot's reply and builds on it leaves a visible artifact: a response that inherits the error and repeats it to the customer. That sits on the page, two graders can agree about it, and a candidate can argue with it.
How long should the support exercise run?
Ninety minutes for three cases, and pay for anything longer. Three cases is enough to show a pattern and short enough that a working candidate can finish in one evening. If speed under pressure is what you want to measure, measure it directly with one live case and a watching interviewer, and say so in the brief. A long unpaid assignment selects for who had a free weekend, which is not the skill in question.
Do you still test writing at all?
Test it, and weight it last. A support reply still has to be clear, and someone who cannot write a plain sentence will struggle on the queue. But grammar and warmth are the parts a model supplies free, so they no longer separate one candidate from another. What still separates them is whether the reply says something true about the account and promises only what the company can deliver.
What does a strong answer to the upstream question look like?
It names the artifact and the edit. For a ticket where a customer reported a broken export and a saved date filter was hiding the rows, update the export article so it says that a saved date filter hides rows: that is specific enough to assign to someone. Improve the knowledge base is not. The strongest answers rank the fix against the ticket volume it would remove, and say what they would check a week later to see whether those tickets stopped arriving. That answer also tells you whether the person will be useful outside the queue.
References
- 1. Generative AI at Work (NBER Working Paper 31161) ✓ nber.org Supports the staggered rollout of a generative conversational assistant to 5,179 customer support agents at one software firm, and the 14% average rise in issues resolved per hour, 34% for novice and low-skilled workers and minimal for experienced, highly skilled ones. Cited for the split across experience levels, not as a general productivity figure.
- 2. 29 CFR 1607.14 - Technical standards for validity studies (Uniform Guidelines on Employee Selection Procedures) ✓ law.cornell.edu Supports two sentences on this page. 1607.14(C)(1): 'A selection procedure can be supported by a content validity strategy to the extent that it is a representative sample of the content of the job,' and 'A selection procedure based upon inferences about mental processes cannot be supported solely or primarily on the basis of content validity.' 1607.14(C)(4): 'To demonstrate the content validity of a selection procedure, a user should show that the behavior(s) demonstrated in the selection procedure are a representative sample of the behavior(s) of the job in question.'
- 3. Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of Artificial Intelligence on Knowledge Worker Productivity and Quality (Working Paper 24-013) ✓ mitsloan.mit.edu Supports the out-of-frontier task result: 84.5% correct in the control group against 60% and 70% in the two AI conditions, an average drop of 19 percentage points, with the prompt-engineering group down 24 points and the group given no overview down 13. One task, one sample, a 2023 model.
- 4. Revisiting Meta-Analytic Estimates of Validity in Personnel Selection: Addressing Systematic Overcorrection for Restriction of Range ✓ static1.squarespace.com Supports two claims. The job knowledge comparison: 164 studies at a mean observed validity of .22, the 59 using job-specific knowledge tests at .31, rising to .40 corrected for unreliable performance ratings. And the work sample revision: the .54 estimate is superseded by .33, based on Roth et al. (2005), 54 studies of which 53 were concurrent.
4 sources, numbered by first appearance. Every one was opened and checked against the claim it carries. How Olive sources claims
General guidance for hiring teams. What works at one company and one volume may not transfer to yours.
Olive assesses how a person works with AI. It does not detect AI-written documents, and it never produces a score, a ranking, or a match percentage for a person. Candidates read the same report the employer reads.