Screening
What a Hackathon Win and a Kaggle Rank Leave Untested
A Kaggle rank shows skill at optimizing against a target somebody else chose. A hackathon win shows which demo a judging panel preferred. Both are dated, checkable, and scored by somebody with no stake in the application, which puts them above almost every other line on a resume. Neither one tests whether the target was right, because no stakeholder disagreed and a wrong answer cost nothing. Ask the candidate what they would have changed about the metric.
The takeThe placement is the least interesting thing in the story behind it. A competition entry holds a hundred decisions the rank never records: what got tried and abandoned, where the leak was, which validation split earned trust and why. Somebody who can walk through three of them is showing the work, and the placement stops mattering halfway through the second one. Somebody who leads with the number and has nothing behind it has told you the placement was the achievement. Both findings are reachable in ten minutes, which makes this the cheapest question on the screen.
Where Olive fits
Open a role and see what the work shows
A competition result is somebody else's assessment of somebody else's problem, which is what makes it cheap to check and narrow to read. Olive assesses the person against one occupation instead: a 40-to-60-minute assignment done with an AI assistant, returned as six findings with the timestamped excerpt behind each one, and the candidate is granted the identical report.
Assess your shortlistWhat does a leaderboard rank actually verify?
A leaderboard rank verifies a result, which is more than almost anything else on the page does. A competition placement is dated, produced under conditions the candidate did not control, and scored by somebody with no stake in whether they get this job. Few other lines in a resume screen clear that bar.
A bullet about impact is a claim. A leaderboard position is a result somebody else recorded and published.
The score also survives the obvious objection. In 2019, Rebecca Roelofs and six co-authors published the first large meta-analysis of test-set reuse, built on over one hundred machine learning competitions hosted on Kaggle 1. Entrants spent each contest measuring themselves against a holdout set that produced a public ranking, while a separate test set, used only once, produced the final one.
They found little evidence of substantial overfitting 1.
Grinding the visible leaderboard did not, on the whole, buy a position that fell apart when the private set was scored.
But the two lines at the top of that resume are not the same kind of evidence. A Kaggle placement is a number a script computes against data the entrant never saw. A hackathon win is a panel of people picking a favourite from demos after a weekend.
Both still beat the unscored artifacts sitting beside them. Whether a virtual job simulation on a resume is evidence of anything turns on a question a competition has already answered. Nobody graded the simulation. Somebody with no stake in the result scored the competition.
So verify the two lines differently. The competition name and a profile link settle the first in about a minute.
For the hackathon, ask who ran it, how many teams entered, and what the judges were told to reward. A win at a fifty-team event judged on working software is a different fact from a win at a six-team internal one judged on the pitch.
Which of the seventeen tasks did it score?
The clearest one is comparing models on a statistical metric, and the work places it between two others. O*NET, the occupational database the U.S. Department of Labor publishes, lists seventeen task statements for Data Scientists under code 15-2051.00. The one a leaderboard scores most directly is "Compare models using statistical performance metrics, such as loss functions or proportion of explained variance" 2.
Read those two next to a placement. "Identify business problems or management objectives that can be addressed through data analysis" comes before any modelling starts. "Recommend data-driven solutions to key stakeholders" comes after it 2.
No leaderboard contains either one.
The metric is chosen and published before anyone enters. The evaluation set is held out and identical for everybody. Nobody disagrees with the objective halfway through, moves the deadline, or asks what happens to the people the model gets wrong.
So the question to put to a placement is which tasks it covered, not how high the score was.
The nearest number on that question comes from a different kind of test altogether. In the job knowledge meta-analysis Sackett and colleagues rely on, 164 studies together produced a mean observed validity of .22 against job performance 3.
The 59 of those studies using knowledge tests built for the job in question produced .31.
The two sets differ in whether the test's content was relevant to the job, and may differ in other ways too.
Those are knowledge tests, and the .31 against .22 is one comparison between subsets of a single 1993 meta-analysis, with no experiment behind it. It settles nothing about a leaderboard by itself.
The seventeen tasks are the sturdier tool here. They name what a leaderboard left out.
The half of the work that got cheap
Optimizing against a stated objective is technical work that models can already do on public problem sets. Competitions were never designed to certify anything else. The argument is about what a placement is now evidence of: the skill it certifies is one that got cheaper to reach.
The person reading the resume is hiring for the other half.
In 2022, Karmakar, Prenner, D'Ambros and Robbes ran OpenAI's Codex against 115 Python problem statements taken from HackerRank, a competitive programming portal. It solved 96% of them with no worked examples and 100% with a few 4.
The memorization half of that paper is the half that gets dropped.
HackerRank's problems are public, so they were plausibly in the training data. The authors' own contribution is a framework for re-testing models on mutated problem statements, precisely because the raw pass rate overstates capability 4.
Codex is a 2022 model on a curated benchmark, so treat 96% as a floor on what models can do today. Nothing in it speaks for coding tests in general, a real repository, a bespoke brief, or a weekend with a live client in the room.
What stayed scarce is knowing which problems sit beyond what a model can do.
In 2023, a Harvard Business School field experiment randomly assigned 758 Boston Consulting Group consultants to work with GPT-4 or without it, on tasks including one deliberately chosen to sit outside the model's capability 5.
84.5% of the control group got that task right.
The two AI conditions scored 60% and 70%, an average drop of 19 percentage points 5.
People could not tell which side of the line the task was on. That is the finding, not that AI makes people worse in general. A public leaderboard tells you, submission by submission, how far off you are.
And the 2023 model in that experiment is not a current one.
Ask what they would have changed about the metric
One question closes most of the gap and it takes under two minutes. Ask what they would have changed about the competition's metric, or which part of their solution they would refuse to put in front of a paying customer. Somebody who thought about the objective has an answer.
Somebody who only chased the number has nothing to say about the target, and that absence is the finding.
Strong answers are concrete and name a flaw. The metric rewarded a threshold no operations team would ever use. The evaluation set held a column that would not exist at prediction time. The winning ensemble takes four hours to score and nobody could ship it.
Weak answers stay on the leaderboard. More tuning. Better hardware. A team with more people on it. None of those say anything about the target, which is the one thing nobody asked the entrant to think about.
Then walk one decision backwards. Ask what got tried and abandoned, and why.
The rule is the same one that governs any built artifact: a portfolio piece counts when its decisions can be defended. A competition entry is a portfolio piece with a public score attached.
Ask both questions of everybody who put a placement on the page, and write the answers where the next reader will find them. A follow-up asked only of the candidate whose result impressed you is a different screen given to different people.
How much should a placement move the decision?
Less than its precision suggests and more than zero. Treat it as a reason to give somebody a conversation rather than an offer, and put that conversation on the record beside everyone else's. A placement is one observation, from one problem, on one metric, which is too little to decide a hire on.
The number people quote for work samples has moved.
The familiar .54 traces to a 1974 narrative review. Sackett and colleagues replaced it in 2022 with .33, drawn from a meta-analysis of 54 studies covering 10,469 people 6.
Read that .33 with its limits attached.
53 of the 54 studies tested people already doing the job, and the studies say nothing about unpaid exercises like a competition entry, so it is thin evidence about anything applied at the resume stage.
The .33 is also not a demotion. The ranges around it and around structured interviews overlap heavily, and the whole validity table was corrected downward in 2022.
Where a placement earns its keep is the shortlist you cannot separate. A dated third-party result is at least a fact about the person. But breaking a tie among applicants who all clear the bar stays harder than it looks.
Discounting the placement quietly favours candidates who already had a job doing that work. Unpaid time spent on a hard problem is itself scarce.
The two lines at the top of the resume are unchanged.
Beside them now sits an answer about the metric this person would have picked, and one decision they abandoned with the reason for it, written where the next reader can see it. The competition asked for neither.
Common questions
Is a Kaggle rank worth anything on a resume?
Yes, more than most lines on the page. A Kaggle rank is dated, it came out of a process the candidate did not control, and it takes a minute to check by asking for the competition and a profile link. What it certifies is narrow: hitting a target somebody else chose, on an evaluation set held out and used once. It says nothing about whether the target was the right one, because a Kaggle competition does not score that. Credit it as verified evidence of one half of the work and get the other half from a follow-up question.
How do you verify a hackathon win?
Ask three things about the hackathon: who ran it, how many teams entered, and what the judges were told to reward. A hackathon result is a panel picking a favourite after a weekend, so without those answers you know only that somebody liked the demo. A win at a fifty-team event judged on working software is a different fact from a win at a six-team internal one judged on the pitch. Neither is disqualifying. Both go into the notes at the same level of detail, for every candidate who lists one.
Does a competition result replace a work sample?
No, because the competition specified the problem for the entrant. An exercise you write can put the ambiguity back, with messy source material, an objective nobody has settled, or a stakeholder who wants the wrong measure. A placement tells you somebody optimized well against a clean target. Your own exercise is where you find out whether they notice a target is wrong. Keep both when you have them, and score them for different things.
What should the follow-up question about a Kaggle placement be?
What would you have changed about the metric? Then: which part of your solution would you refuse to put in front of a paying customer? Strong answers are concrete, naming a threshold no operations team would use, a column that would not exist at prediction time, or an ensemble too slow to ship. Weak answers stay on the leaderboard, about more tuning or better hardware. Ask both of every candidate who lists a placement, and write the answers down so the round stays comparable.
Should a Kaggle rank count for a role that is not data science?
Barely, and say so rather than letting it work quietly. The result came from a modelling problem with a published metric, so it carries information about persistence and about optimizing toward a stated target. It says little about the tasks a different occupation lists. If the role does not involve building predictive models, treat the placement as evidence that this person has stayed with a hard problem, which is worth something, and stop there.
Do AI tools make competition results meaningless?
They make results on public problem sets cheaper to reach, not meaningless. Competitive programming problem banks are public, and a 2022 evaluation found OpenAI's Codex solving 96% of 115 HackerRank Python problems with no worked examples, with clear signs it was reproducing memorized code. Treat that as a floor, since it is an old model on a curated benchmark. The practical consequence is that a result on a public problem set certifies less than it did and the follow-up matters more. Deciding which problem to solve did not get cheaper.
References
- 1. A Meta-Analysis of Overfitting in Machine Learning ✓ proceedings.neurips.cc Primary source, fetched 2026-09-01. Supports both the structure of a Kaggle competition and the overfitting result. The abstract states: "Our analysis is based on over one hundred machine learning competitions hosted on the Kaggle platform over the course of several years. In each competition, numerous practitioners repeatedly evaluated their progress against a holdout set that forms the basis of a public ranking available throughout the competition. Performance on a separate test set used only once determined the final ranking" and "Our study shows, somewhat surprisingly, little evidence of substantial overfitting."
- 2. Summary Report for: 15-2051.00 - Data Scientists ✓ onetonline.org Primary source, fetched 2026-09-01; page marked Updated 2026. Supports the count of seventeen task statements and the three quoted verbatim: "Compare models using statistical performance metrics, such as loss functions or proportion of explained variance", "Identify business problems or management objectives that can be addressed through data analysis", and "Recommend data-driven solutions to key stakeholders."
- 3. Revisiting Meta-Analytic Estimates of Validity in Personnel Selection: Addressing Systematic Overcorrection for Restriction of Range ✓ static1.squarespace.com Supports the job knowledge comparison between two sets of the same test format that differ in whether the content was relevant to the job: 164 studies with a mean observed validity of .22, and the 59 using job-specific knowledge tests at .31. Cited with the paper's own limit, that this is a comparison between subsets of the 1993 Dye et al. meta-analysis rather than an experiment.
- 4. Codex Hacks HackerRank: Memorization Issues and a Framework for Code Synthesis Evaluation ✓ arxiv.org Supports the claim that a public competitive programming bank is partly a lookup: Codex solved 96% of 115 HackerRank Python problem statements zero-shot and 100% few-shot, with the authors reporting clear signs of memorized code and proposing mutated problem statements as the fix.
- 5. Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of Artificial Intelligence on Knowledge Worker Productivity and Quality (Working Paper 24-013) ✓ mitsloan.mit.edu Supports the claim that the scarce skill is telling which side of the capability line a task sits on: in a 758-consultant BCG field experiment, on one task chosen to sit outside GPT-4's capability, the two AI conditions scored 60% and 70% against 84.5% in the no-AI control group, an average drop of 19 percentage points.
- 6. Revisiting Meta-Analytic Estimates of Validity in Personnel Selection: Addressing Systematic Overcorrection for Restriction of Range ✓ gwern.net Supports the weighting argument: the familiar .54 for work samples traces to a 1974 narrative review and was replaced with .33 from Roth et al. (2005), 54 studies covering 10,469 people, of which 53 were concurrent designs testing people already doing the job.
6 sources, numbered by first appearance. Every one was opened and checked against the claim it carries. How Olive sources claims
General guidance for hiring teams. What works at one company and one volume may not transfer to yours.
Olive assesses how a person works with AI. It does not detect AI-written documents, and it never produces a score, a ranking, or a match percentage for a person. Candidates read the same report the employer reads.