Assessment design
Public Questions Are Real Practice and a Bad Prediction
Some of them are. A company tag on a practice site is built from what past candidates remembered and posted, so it is real preparation and an unreliable prediction. Public exercises are also ones models have plausibly already seen and solve at high rates, so some employers are changing the exercise or adding a round with a person in it. Practice the patterns. Expect the specific question to have moved.
The takeThe forum argument is stuck between grind-the-list and the employer line that every question is fresh, and both are selling a certainty nobody has. The honest position sits in between and keeps moving. Some items are genuinely public, they age at a rate you cannot see from outside, and the exercise is being redesigned faster than any list can track. Preparation that survives all of that is the kind you can explain to a person who asks why you did it that way.
Where Olive fits
Open a role and see what the work shows
If the assessment an employer sends is Olive, the AI assistant is meant to be used: a human reviewer writes or edits six findings about how you framed the problem, what you delegated and what you verified, in words rather than a number. The candidate is granted the identical report, free.
Assess your shortlistWhere Do Company-Tagged Questions Come From?
Mostly from candidates who sat the test and typed up what they remembered afterwards. A company tag on a practice site is a crowd's recall, pooled from those reports. It is not a feed from inside the company. Some of what it lists is still in rotation, and some may have been retired a year ago while the tag kept its label.
Nothing on the list marks which is which.
At least one platform markets leak alerts as a feature. HackerRank's Screen page lists leaked question alerts among the integrity signals an employer can switch on, alongside tab proctoring, copy and paste tracking and full session replay 4. That is a vendor's own page, with no accuracy figures attached to any of it.
An item that trips an alert is also an item worth replacing.
Age matters more than accuracy here. A question posted eighteen months ago may have been exactly right on the day somebody wrote it down and still be gone. From outside, you cannot date the rotation, which is the practical reason to treat the list as practice material and not as a forecast.
The forums know this too. A thread saying a list is stale and a thread swearing it is current sit next to each other, and both are somebody's single data point.
Why a Public Question Measures Less Than It Did
Because public exercises are ones models have plausibly seen in training. Evaluated on 115 Python problem statements taken from HackerRank, OpenAI's Codex solved 96% of them zero-shot, and the authors report clear signs the model was reproducing memorized code rather than writing it 1. That was a 2022 model on a curated public set, so the number reads as a floor.
Difficulty buys a shrinking margin. In an automated study submitting model-written solutions to LeetCode, GPT-3.5-turbo solved 92% of easy, 79% of medium and 51% of hard problems 2. Correctness there means passing the site's tests, not design or explanation.
Half the hard problems fell to a model since superseded.
Employers see the same thing from their side, and at least one has responded by changing the exercise rather than policing the people taking it. Anthropic published an account of a performance-engineering take-home that more than 1,000 candidates have completed since early 2024 3. Its own models kept beating it: under the same time limit, Claude Opus 4 outperformed most human applicants, and Claude Opus 4.5 matched even the strongest 3.
The response, across three redesigns, was a different exercise. On that particular take-home, AI use is explicitly permitted 3.
One team, one exercise, one company, written up by the engineer who designed it. The employer-side version of the problem is written down too. What a hiring team is told to do when a model release moves the pass rate on a work sample is to re-anchor the cut score against freshly marked work rather than assume the assessment broke.
What Employers Changed in Response
Two moves show up in what is on record: change the exercise, and add a stage with a person in it, checking fundamentals or probing your reasoning. Asked in June 2025 how Google's coding interviews had changed, Sundar Pichai said Google is making sure it will introduce at least one round of in-person interviews, to check the fundamentals are there 5. That is a chief executive's stated intent on a podcast rather than a published policy.
He said in the same answer that if you can use these tools to generate better code, that is an asset 5.
The added round is a place to show fundamentals, not a trap for people who practiced.
One vendor is building the same assumption into a product. Karat launched NextGen Interviews in December 2025: engineers work on complex multi-file projects with an integrated AI assistant while a live interview engineer probes reasoning, trade-offs and judgment in real time 6. That is a launch announcement about one product line, and whether you meet the format is the employer's choice.
The place a memorized solution gets priced is the conversation after the assessment.
The employer-side advice here is public, and reading it beats guessing. A hiring team asking what replaces an online assessment a model solves in seconds is told to hand over occupational material with a defect built into it. Let candidates use AI openly, then grade how the problem was framed and what got checked.
Where Should the Hours Go?
Into patterns and into explaining a solution out loud, with the public list used as practice material rather than as a forecast. Working through company-tagged problems is legitimate preparation and always was. What has stopped working is the expectation that the specific item will appear, and a plan built on that expectation leaves you flat when it does not.
Three habits are worth more than another twenty solved items.
- Say the constraint you are solving before you write anything, in the words you would use with a colleague.
- Take a problem you already solved and explain the trade-off out loud to somebody who has not seen it.
- Read the instructions on the test itself for the tool policy, because it varies from test to test.
That last one is not a formality. HackerRank's page frames the rule per test: if a test is designed to allow AI it is fair for everyone to use it, and if it says not to, using AI is flagged as suspicious activity 4. A flag there is something an employer reviews rather than a verdict, and the same platform sits on both sides of the line depending on what was switched on. The general company policy is not the answer. The instructions on the test are.
One line is worth keeping straight. A question a past candidate remembered and posted is public, whatever the company would prefer. A bank taken from inside a company is not public, and working from one is a different act with different consequences, whatever a forum thread says about it.
If the round ahead allows the assistant, the preparation shifts again. What an interviewer watches when AI is allowed in the coding round is what you questioned before generating, and what you tested against a case you picked yourself.
The question you can explain is the one that keeps its value when the item changes.
Common questions
Are company-tagged questions on practice sites real?
Often real in the sense that somebody reports having been asked them, since the tags are built from what past candidates remembered and posted. What a tag cannot tell you is whether the item is still in rotation. At least one assessment platform markets leak alerts: HackerRank's Screen page lists leaked question alerts among the integrity signals an employer can switch on 4. An item that trips one is an item worth retiring, so a well-known question may not have long left.
Is grinding LeetCode still worth it before an online assessment?
Yes as practice in the patterns, and no as a prediction of the specific questions. The public set is also the part models handle best: an automated study found GPT-3.5-turbo solving 92% of easy, 79% of medium and 51% of hard LeetCode problems 2. That is a superseded model on public algorithmic problems, judged by whether the submission passes tests. Some employers reading numbers like those have been changing the exercise, so hours spent on explaining a solution tend to survive better than hours spent memorizing one.
Will employers just make the assessment harder?
Harder is one option, and it buys less than it looks, because difficulty on a public corpus buys a shrinking margin rather than a wall. The employer-side advice on this site recommends changing the kind of task instead: hand over occupational material with a defect in it, allow AI openly, and grade how the problem was framed. Anthropic's published account of its own take-home describes three redesigns after its models beat most applicants, and on that exercise AI use is explicitly permitted 3.
Am I allowed to use AI on an online assessment?
The test itself sets that rule, and its instructions screen is where to find out. HackerRank's page states it per test: a test designed to allow AI is fair for everyone to use, and where it says not to, using AI is flagged as suspicious activity 4. A company's general policy can differ from what one assessment says. If nothing is stated anywhere, ask the recruiter before you start.
Does an in-person round mean employers assume candidates are cheating?
The one on-record example gives a different reason. Asked in June 2025 how Google's coding interviews had changed, Sundar Pichai said Google is making sure it will introduce at least one round of in-person interviews to check the fundamentals are there 5. He added in the same answer that using these tools to generate better code is an asset. That is stated intent on a podcast rather than a published policy, it does not rule out other motives, and it reads as a reason to practice explaining your work to a person.
References
- 1. Codex Hacks HackerRank: Memorization Issues and a Framework for Code Synthesis Evaluation arxiv.org Supports the claim that public exercises are the ones models solve best, with the memorization finding attached.
- 2. Evaluating ChatGPT-3.5 Efficiency in Solving Coding Problems of Different Complexity Levels: An Empirical Analysis arxiv.org Supports the claim that raising difficulty on a public corpus buys a shrinking margin rather than a wall.
- 3. Designing AI-resistant technical evaluations anthropic.com Supports the claim that employers are redesigning the exercise rather than policing candidates, and that AI use can be permitted per assessment.
- 4. HackerRank Screen (product page) hackerrank.com Supports the claim that platforms watch for leaked items, and that the AI rule is set per test rather than per company.
- 5. Transcript for Sundar Pichai: CEO of Google and Alphabet | Lex Fridman Podcast #471 lexfridman.com Supports the claim that an in-person round is being added at one named employer, as stated intent rather than policy.
- 6. Karat launches NextGen Interviews: the first human-led, AI-enabled talent evaluation solution karat.com Supports the claim that an AI-allowed coding round now includes explaining decisions to a live interviewer.
6 sources, numbered by first appearance. How Olive sources claims
General guidance for hiring teams. What works at one company and one volume may not transfer to yours.
Olive assesses how a person works with AI. It does not detect AI-written documents, and it never produces a score, a ranking, or a match percentage for a person. Candidates read the same report the employer reads.