Policy
A Candidate's Agent Needs the Review You Give a Vendor
Before a candidate's agent gets credentials to your sandbox, run the review you would run on any third-party integration. Scope access to the task, use synthetic records rather than real ones, and leave no route to production. Set the expiry when the credential is minted, and write down what the session logs, who reads it and how long it is kept. Give every candidate an equivalent exercise that needs no credentials. Decide in advance what a destructive action means, because it is an ordinary outcome of the format.
The takeThe failure here is a process problem, not a technical one. The person approving the access is usually the person who wants the exercise to happen this week, and nobody with a security title ever sees the request. Move the approval to whoever signs off on a contractor's laptop, and accept that it costs the round two days. A sandbox nobody would stand up for a paid vendor is not one to stand up for a stranger who is still deciding whether to work here.
Where Olive fits
Open a role and see what the work shows
Olive runs its assessment in its own workspace with its own AI assistant, and each candidate enters through an invite link that is theirs alone and expires in 14 days. Each of the six findings is written or edited by a person and carries the timestamped excerpt it rests on, and every released report exports with its rubric, scorer and bank versions attached.
Assess your shortlistWhat does the agent actually need to touch?
Read the exercise backwards from the evidence you want, then grant only what that requires. Most agent tasks need a repository, a runner and a few dozen records to act on. They rarely need a directory account, a shared secret, or anything that resolves to a hostname in production. Scope the credential to the smallest surface on which the task is still real work.
NIST's control catalog states the rule in one sentence: employ the principle of least privilege, allowing only authorized accesses for users, or processes acting on behalf of users, that are necessary to accomplish assigned organizational tasks 1.
An agent working a take-home is a process acting on behalf of a user you have never met.
Manufacture the data. An exercise that hands over last quarter's customer rows to find out whether a candidate is careful with customer rows has already spent the thing it was testing. And anything you send a candidate should be assumed to go into a model.
Put the boundary in the network.An agent instructed to stay out of production will sometimes reach production, because an instruction is not a permission. OWASP's 2025 risk list for LLM applications puts the remedy downstream: implement authorization in downstream systems rather than relying on an LLM to decide if an action is allowed 2.
The egress rules are the policy. The system prompt is a request.
Write down what the session logs before anyone starts
Say what is captured, who reads it and when it is deleted, in the brief the candidate sees before accepting. An agent run against systems you own leaves a fuller record than any take-home did: every command issued, every file touched, every prompt in between, timestamped. A candidate who finds that out afterwards is right to be angry.
Illinois put a narrow version of that notice into force in 2020. The Artificial Intelligence Video Interview Act requires an employer using AI to analyse a recorded video interview to notify the applicant beforehand, explain what the system evaluates, and obtain consent, and it bars evaluating anyone who has not consented 3.
The Act reaches recorded video interviews and nothing else.
It names no penalty and gives applicants no private right of action, so it sets a floor in one state rather than a rule that covers a sandbox.
Write the retention line as a date. "Kept for the duration of the hiring process" is how a transcript stays on a laptop for three years. How long a screen recording of a candidate can be kept is the same question one format over.
Then decide what leaves the sandbox. The run log is evidence about the exercise, and the prompts inside it are the candidate's working. So what you may collect and keep from a candidate's AI session is a narrower question than what the sandbox happens to record.
Set the teardown time when you mint the credential
Give the credential an expiry at the moment it is created, and make the expiry shorter than the exercise plus a day. A team that plans to revoke access after the debrief will not, because the debrief moves and nobody owns cleanup. NIST's account-management control asks for temporary accounts to be removed or disabled automatically after a defined period, rather than at an administrator's convenience 1.
The reason to automate removal is on the public record, inside a hiring system.
On 30 June 2025 the security researchers Ian Carroll and Sam Curry disclosed to Paradox and McDonald's that McHire, McDonald's Paradox-built application chatbot, accepted 123456 as both username and password on its restaurant-owner administration interface 4.
An insecure direct object reference on an internal API then let a logged-in account read applicants' contact details and chat histories.
Paradox published its own account on 9 July 2025. By that account, the researchers had logged into a test account tied to a single client instance, using a legacy password that had never been updated, on an account last logged into in 2019 that should have been decommissioned 5.
A test credential outlived its test by six years.
Carroll and Curry put the exposed dataset at more than 64 million applicants, inferred from sequential applicant IDs, while Paradox says five candidates in total had information viewed, and only by the researchers 45.
Records reachable and records opened are not the same measure.
A sandbox credential is that account on a shorter clock. It gets created in a hurry, for one person, under a deadline, by whoever built the exercise, and it appears on no inventory anyone reviews.
Give every candidate a version that needs no credentials
Build a second path to the same evidence that requires no access at all. That path can be a recorded agent run to read and mark up, a repository dump to work against locally, or a transcript with the decisions left in and the answers taken out. Publish it in the brief as a normal option. Nobody should have to explain why they would rather not connect their own tooling to systems you own.
Disability accommodation is one narrow version of that second path. The EEOC's 2022 technical assistance on algorithmic assessment names extended time and an alternative version of the test, including one compatible with a screen reader, among the accommodations that may be effective 6.
Those are examples of what might work rather than a menu anyone is owed, and the document was removed from the EEOC's site in January 2025.
The design lesson survives the removal: an alternative has to exist before anyone asks, or the request itself becomes a disclosure.
Most declines have nothing to do with disability. A contractor bound by a client agreement, a candidate whose agent runs on an employer's licence, someone who read the McHire write-up: each has a reason that is nobody's business. A single path makes all of them volunteer it.
Whose AI account the work happens in changes what you can read afterwards, and what you end up storing. The no-credentials path costs the live run and keeps most of what you were going to grade, especially where the alternative is a candidate opening a real session from their own work and talking it through.
Who decides when the agent breaks something?
Name that person before the first invite goes out, and write down what the answer costs the candidate. An agent that drops a table, exhausts a rate limit or commits a secret into the exercise repository is producing an ordinary outcome of the format. Read it as a fact about the sandbox first, and about the person only if the record supports that.
OWASP's list defines the class by what the system was allowed to do. Excessive agency, in that document, is the vulnerability that enables damaging actions to be performed in response to unexpected, ambiguous or manipulated outputs from an LLM 2. Its three named roots are excessive functionality, excessive permissions and excessive autonomy.
One remedy the list gives is a checkpoint: require a human to approve high-impact actions before they are taken 2.
In a hiring exercise that checkpoint is also the observation. Whether the candidate builds one, and what they do when it fires, is part of what the sandbox was stood up to show, and the checkpoints in an agent run are the evidence.
Settle the intervention rule in the same sitting. When you are allowed to intervene is a decision to make before the session, not one to improvise while a candidate watches a table disappear.
Against all of it sits a week of platform work: a scoped sandbox, manufactured data, an expiring credential and a written logging notice, for a round you may use four times. The version that gets built under deadline is a copy of staging with the passwords changed.
The credential you mint for one exercise is the whole decision compressed into a line. It carries a scope, a clock and a set of records that will outlast the round, the candidate's interest in the job, and probably the person who created it. Put the expiry in the same line that creates it.
A month after the last candidate, go and look for it.
Common questions
Can a hiring exercise require the candidate to connect their own AI account?
A hiring exercise can require that, and a second path should exist alongside it. Requiring one route makes anyone who declines explain themselves, and the reasons are usually a client agreement, an employer-licensed account, or plain caution about a stranger's systems. Build an equivalent exercise that needs no credentials, publish it as a normal option rather than an exception, and grade both against the same rubric. Whose account the work happens in also changes what you are able to read afterwards, and what you are then storing.
How long should a candidate's sandbox credential live?
Shorter than the exercise plus a day, with the expiry set at the moment the credential is created rather than added later. An enhancement to NIST's account-management control asks for exactly this: temporary accounts removed or disabled automatically after a defined period rather than at an administrator's convenience. Manual cleanup depends on a debrief that moves and an owner who has gone back to a full week of work. An expiry that fires without anyone remembering is the only kind that fires.
What has to be in the brief about logging?
Four things, in the brief the candidate reads before accepting: what the session captures, who inside the company reads it, how long it is kept, and the date it is deleted. An agent run records every command, every file touched and the prompts in between, which is more than a take-home ever produced. Since 2020, Illinois has required notice, an explanation and consent for the narrower case of AI analysis of a recorded video interview for an Illinois-based position, and bars evaluating an applicant who has not consented. A retention answer written as a date is the one that gets honoured.
Does the exercise still measure anything if the data is synthetic?
Yes, as long as the manufactured records carry the same shape and the same traps as real ones: the ambiguous join, the field that is null more often than the schema suggests, the row that contradicts its own timestamp. What the exercise measures is what a candidate does when the model is confidently wrong about a record. Production rows add no signal to that, and they add a disclosure problem, an access review and a deletion obligation that synthetic rows do not.
What should happen if the candidate's agent destroys something?
Decide before the invite goes out, and write the answer into the brief. OWASP's list of LLM application risks names damaging actions from an autonomous process as a risk class, rooted in too much functionality, too many permissions and too much autonomy. A sandbox that one agent run can wreck is first a fact about the sandbox. Restore from the snapshot, keep the run record, and read what the candidate did at the moment things went wrong. That moment is usually the most informative part of the session, whichever way it went.
Who should approve access for a hiring exercise?
Whoever approves access for a contractor or a third-party integration, on the same form. A hiring manager approving credentials for a stranger's tooling is making an access decision inside a hiring loop, where the deadline belongs to the round and the risk belongs to the company. Routing it through the normal review costs a couple of days and produces two useful artefacts: a scope somebody outside the loop agreed to, and a record of who granted what, which is the thing anyone asks for later.
References
- 1. Security and Privacy Controls for Information Systems and Organizations (NIST Special Publication 800-53, Revision 5) ✓ nvlpubs.nist.gov Supports the least-privilege sentence quoted from control AC-6, and the automatic removal or disabling of temporary accounts after a defined period rather than at an administrator's convenience, from control enhancement AC-2(2).
- 2. LLM06:2025 Excessive Agency ✓ genai.owasp.org Supports the quoted definition of excessive agency, its three named roots (excessive functionality, permissions and autonomy), the recommendation to implement authorization in downstream systems rather than relying on an LLM, and human approval of high-impact actions before they are taken.
- 3. Artificial Intelligence Video Interview Act, 820 ILCS 42 ✓ ilga.gov Supports the notice, explanation and consent duties before AI analysis of a recorded video interview, and the bar on evaluating an applicant who has not consented. Cited as a narrow, single-state precedent, not as a rule reaching sandbox exercises.
- 4. Would you like an IDOR with that? Leaking 64 million McDonald's job applications ✓ ian.sh Supports the default 123456 credentials on the McHire owner administration interface, the insecure direct object reference on an internal API, the 30 June 2025 disclosure date, and the more-than-64-million figure as the researchers' inference from sequential applicant IDs.
- 5. Responsible Security Update ✓ paradox.ai Supports the vendor's 9 July 2025 account: a test account tied to one client instance, a legacy password never updated, a last login in 2019 on an account that should have been decommissioned, and five candidates whose information was viewed, only by the researchers.
- 6. The Americans with Disabilities Act and the Use of Software, Algorithms, and Artificial Intelligence to Assess Job Applicants and Employees ✓ web.archive.org Supports extended time and an alternative version of the test, including one compatible with a screen reader, as accommodations that may be effective in an algorithmic assessment, and the document's removal from eeoc.gov in January 2025.
6 sources, numbered by first appearance. Every one was opened and checked against the claim it carries. How Olive sources claims
General guidance, not legal advice. Hiring rules differ by state and country and change often; check anything here against your own counsel before you act on it.
Olive assesses how a person works with AI. It does not detect AI-written documents, and it never produces a score, a ranking, or a match percentage for a person. Candidates read the same report the employer reads.