Assessment design
Grade a Security Candidate on Triage, Not on the Top Ten
Assess a security candidate on model-introduced flaws by handing them a diff to triage rather than a taxonomy to recite. Seed a real change from your own repository with weaknesses drawn from a published list, and give an hour. Grade three things: the order the findings are worked in, what the candidate declines to fix and why, and what they say they would have to check before deciding.
The takeThe recall round survived this long because it was cheap to run, not because it worked. Asking somebody to name the top ten measured whether they had read a list, and a model can reproduce a published list on request. What changed is the surface: a weak line can now arrive with a comment above it explaining why the code is fine. So the scarce skill was never spotting the flaw. Deciding which of eleven real findings is worth holding Thursday's release for, and which one the team can live with, is the part a recall round never tests.
Where Olive fits
Open a role and see what the work shows
If you build this in-house, the expensive parts are the seeded case and the record of what the candidate actually did with it. Olive ships twelve authored cases per occupation and returns six findings a person wrote or edited, each anchored to a timestamped moment in the session, with the candidate granted the same report.
Assess your shortlistWhy Does a Recall Round Miss What a Model Introduces?
Because naming a weakness class and finding one in a change are different acts, and only the second was ever the job. A candidate who can recite the top ten has demonstrated that they read a list a model reproduces on request. The round that still measures something puts a flaw in front of them, inside code that works.
The answers to public programming problems can already be inside the model. Against 115 Python problem statements taken from HackerRank, OpenAI's Codex solved 96% of them with no examples and 100% with a few, and the authors report clear signs that the model was reproducing memorised code rather than writing it 1.
A 2022 model on public problems is a floor, not a ceiling.
So the round has to hand over code. A flaw a model introduces can be ordinary, plausible and confidently commented: the missing bounds check, the query built by string concatenation, the certificate verification switched off to make a test pass.
Self-assessment does not catch a flaw that looks fine. In a Stanford user study of people writing security-relevant functions, participants given an AI assistant wrote significantly less secure code than participants without one, and were more likely to believe their code was secure 2.
Participants who trusted the assistant less, and engaged more with the wording of their prompts, produced fewer vulnerabilities 2.
An error that announces itself is a different exercise. Testing whether a candidate notices when AI gets something wrong can be done with a wrong number, because the check that catches it is a check anybody can run. A weakness passes every test the candidate was handed, and the test suite agrees with it.
The round shape is not new, and the case for asking the candidate to review AI-written code instead of writing any holds here.
Security narrows what the round grades.
Build the Diff From a Published Weakness List
Take a change your team actually merged, rewind it, and put the weaknesses back in from a published ranking rather than from memory. The 2024 CWE Top 25 puts cross-site scripting first, an out-of-bounds write second and SQL injection third 3. None of those is exotic.
Ordinariness is the property you want, because ordinary is what a model produces.
A list you assemble yourself encodes what your own team has been burned by. A published ranking encodes more than one team's history, which is a wider sample than any one engineer's memory and a shorter argument in the debrief.
Seed at least two, and make them differ in one specific way: one weakness sits on a path untrusted input can reach, the other sits somewhere nothing outside the process can touch.
The pair is the whole round.
A candidate who reports both at the same volume has searched rather than triaged. The one worth hiring says the second finding is real, says it is not worth a release, and names the condition that would change the answer.
Preparing the repository is where the hours go, and it is done once for a hiring season rather than once per candidate. What to plant so the exercise tests checking and not typing speed is the general version of that work, and the answer transfers.
Leave the assistant open. A scanner and a model will both be on the desk on the first Monday, so a round that bans them grades a way of working the job never uses.
What Separates a Triage From a List of Findings?
Order, and the reasons under it. Two candidates can return the same eleven findings and be nowhere near the same hire. The stronger one puts the injection on the request path above the hardcoded credential in a test fixture, says why in a sentence, and names the fact about your deployment that would reverse the order.
Exploitability asks what an attacker gets. Reachability asks whether an attacker can get there at all.
A finding that scores high on exploitability and zero on reachability belongs under a duller one that scores moderately on both. Saying so out loud is the skill. Nothing in the diff contains the answer, because the answer is about your system.
The best answers arrive as questions. Is the endpoint reachable without a session. Does anything upstream normalise the path. Is the pinned library version the one that carries the patch.
A candidate who calls a finding critical without asking one of them is reading the snippet.
Permitting the tool, and asking for the record of using it, makes the reasoning visible. In one study, engineering students used ChatGPT during take-home open-book exams and handed in their interaction transcripts alongside their solutions 4.
The researcher found some of the strongest evidence of reasoning where students evaluated incorrect or incomplete AI responses, and concluded that the correctness of final answers alone may no longer provide sufficient evidence of comprehension 4.
One course, one qualitative reading, no control group and no comparison of grades.
Ranking is also what makes a partial answer scoreable. Scoring an unfinished submission against a wrong one is the same argument in a different round: three findings ordered and justified beat eleven listed flat.
Grade the Refusals, and What They Could Not Answer
Write down what the candidate declined to fix and the reason given. A security hire who fixes everything on the list has not yet shown you how they price a fix, because every fix has a cost and somebody prices it. The gradeable answer names the finding, states what the fix would break, and names who should make the call.
Decide which refusals are correct before the session, not during the debrief. Writing the answer key before the assignment goes out is what turns a refusal into a score rather than an argument between two graders.
Do not let the candidate's account of their own care stand in for the record. In a randomized trial, 16 experienced open-source developers worked on 246 real tasks in repositories they had contributed to for around five years. Allowed to use early-2025 AI tools, they were 19% slower 5.
The same developers had forecast a 24% speedup beforehand, and still estimated a 20% speedup once the work was done 5.
The 19% came out of one setting with sixteen developers in it, so the size of that effect travels nowhere. The direction of the belief does.
Two senior engineers who agree on every fact in the diff can still rank the same eleven findings differently, and both can defend the order they chose. Triage order is contested, and a rubric that scores the exact sequence is likely to fail its own agreement check.
A round nobody can score twice the same way produces an argument, not a record.
The commit you rewound in September ages badly. Its library gets patched, and its weakness class becomes the one the scanner bought in March catches unaided. The finding that was worth an argument turns into the one every candidate makes.
Rewind another commit and plant it again. The round is worth what the code inside it still costs to get wrong.
Common questions
Should the candidate be allowed to use AI during the exercise?
AI stays open during the exercise, and the brief should say so. The job runs with an assistant and a scanner open, so a round that bans both grades a workflow nobody uses. Allowing the tool also changes what you can see: the prompts, the output the candidate rejected, and the moment they stopped believing a confident explanation. Ask for the session record alongside the write-up, say in advance that the record is part of what gets read, and tell the candidate what it will be used for. What you give up is the idea that the artifact alone shows who did the work, which was never true.
How long should a security triage exercise run?
About an hour of the candidate's time, on a diff of a few hundred lines carrying two or three seeded weaknesses. Longer buys less than it costs: past the first hour you are measuring stamina and unpaid labour rather than judgment. Cap the write-up at a page. Ask for three things, in this order: the findings ranked the way the candidate would work them, the reason for the top one, and anything they would need to check before calling it exploitable. Pay for anything that runs beyond a couple of hours.
What if the candidate misses one of the planted weaknesses?
Check which one was missed, then read the miss for what it is. A miss on a weakness that needs no knowledge of your domain is evidence about how the person reads code. A miss on one that turns on your framework, your deployment or a library version they have never touched is evidence about exposure, and a new hire buys that in a month. Plant at least one weakness any competent reviewer can see without knowing your stack, so a single miss can be interpreted at all. Score misses separately from ordering; they answer different questions.
Can a scanner do this instead of a person?
A scanner can find some classes of seeded weakness faster than a candidate, which is exactly why finding is the cheap half. Static analysis returns a list sorted by its own severity model, with at best a partial view of whether the path is reachable in your deployment or whether the fix breaks a caller. Ranking that list against a release going out on Thursday is the work you are hiring for. Run the scanner over the seeded diff yourself first: whatever it catches unaided is no longer worth grading a person on.
Does a triage round replace the round where they write code?
No. A triage round says nothing about whether the candidate can land the fix, and the two come apart in real people. Keep one short round that asks for a patch: hand back the finding they ranked first, ask for the change, the test that would have failed before it, and the reason the test sits where they put it. Twenty minutes in the same repository is enough. A security hire who can rank a list and never close an item on it is half a hire.
References
- 1. Codex Hacks HackerRank: Memorization Issues and a Framework for Code Synthesis Evaluation ✓ arxiv.org Supports the claim that a question drawn from a public bank tests a lookup: Codex solved 96% of 115 HackerRank Python problem statements zero-shot and 100% few-shot, with clear signs of reproducing memorised code.
- 2. Do Users Write More Insecure Code with AI Assistants? ✓ arxiv.org Supports both the security failure mode and the confidence reversal. The abstract states that participants with access to an AI assistant 'wrote significantly less secure code than those without access', 'were more likely to believe they wrote secure code', and that participants who trusted the AI less and engaged more with their prompts 'provided code with fewer security vulnerabilities'.
- 3. 2024 CWE Top 25 Most Dangerous Software Weaknesses ✓ cwe.mitre.org Supports the three ranked classes named in the article: cross-site scripting (CWE-79) ranks first in the 2024 edition, out-of-bounds write (CWE-787) second and SQL injection (CWE-89) third.
- 4. Reimagining Assessment in the Age of Generative AI: Lessons from Open-Book Exams with ChatGPT ✓ arxiv.org Supports the claim that permitting the tool and collecting the transcript is what makes the reasoning observable: engineering students submitted interaction transcripts alongside solutions, the strongest evidence of reasoning appeared where students evaluated incorrect or incomplete AI responses, and the paper concludes correctness of final answers alone may no longer provide sufficient evidence of comprehension. Qualitative, one course, no control group.
- 5. Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (arXiv:2507.09089) ✓ arxiv.org Supports the gap between believed and measured performance: 16 experienced open-source developers on 246 real tasks in repositories they had worked in for about five years were 19% slower with early-2025 AI tools, after forecasting a 24% speedup and while still estimating a 20% speedup afterwards.
5 sources, numbered by first appearance. Every one was opened and checked against the claim it carries. How Olive sources claims
General guidance for hiring teams. What works at one company and one volume may not transfer to yours.
Olive assesses how a person works with AI. It does not detect AI-written documents, and it never produces a score, a ranking, or a match percentage for a person. Candidates read the same report the employer reads.