Pipeline
Cutting Twelve Hundred Applications to a Shortlist You Can Defend
Twelve hundred applications for one opening reduce to a shortlist with a written cut rule: one sentence naming what has to be true for a candidate to advance. Read a random sample rather than the top of the queue, because queue order records arrival time, and arrival time tracks alerts and tooling rather than fit. Cap the read at what two calibrated people can do in a week, put the decision on a short piece of work, and blind re-read thirty rejections to find what the cut sorted on.
The takeDefensibility here is a quality test before it is a compliance one, and the only thing that tells you whether the shortlist is any good. A resume screen is the highest-volume decision in the whole process and the least examined one, and at twelve hundred applications it disposes of more people in an afternoon than every interview round will touch all year. If nobody can say in one sentence what the cut rule was, the honest description of the shortlist is the twelve applications read while attention held. That is a schedule, not a decision.
Where Olive fits
Open a role and see what the work shows
Olive sits on the far side of a cut like this rather than inside it: one attempt returns six findings on one candidate, each carrying the moment in the session it rests on, and a person writes every word of them. Nothing in the report orders a pile or stands in for a person as a number.
Rank your shortlistWrite the Cut Rule Before You Open the Pile
One sentence, written down, before the first application is opened: what has to be true for someone to reach a conversation. Two or three conditions, each checkable from what you asked for, each connected to the work. Anything you cannot check from the application is a preference, and you will apply it unevenly across twelve hundred people while getting tired.
A usable rule looks like this: advance anyone who has shipped a forecast into a business decision, can name the data they used, and is inside the pay band. A rule that does not work looks like this: advance the strongest candidates. The second one is not a rule, it is the absence of one, and it is what most piles are actually cut with.
Test it on fifty applications before running it on the rest. You are checking two things: how many advance, and whether the rule is doing the deciding. If it advances 40% the bar is too low and the rule needs a third condition. If it advances two people out of fifty, the rule encodes a candidate who does not exist and the requisition is the problem. Adjust it once, write down what you changed, and then stop adjusting it.
Why Does Reading From the Top of the Queue Select for Speed?
Because the top of the queue is whoever arrived first, and arrival time tracks alerts, aggregators and the software some candidates have submitting for them. Fit appears nowhere on that list. A timestamp tells you when an application landed, never what produced it, so the first fifty on a well-syndicated posting are a sample of who was reached first rather than of who fits. Read down that queue and you have run a speed contest nobody entered on purpose.
Any default ordering the system hands you carries the same problem: recency, keyword match, a vendor's ordering with no published basis. Each of them encodes something, and none of them encodes what you decided the cut rule would be.
Randomize instead. Shuffle the pile, read a fixed number, apply the rule, and stop. It costs nothing beyond a sort column, and it removes the one bias in this process that is guaranteed to be there. Whether a req that filled its cap in two days selected for the fastest bot is the sharp version of this problem, and why applications per opening rose so far so fast is where the volume came from.
Two pools are worth pulling out before you shuffle, and only two. Referrals and internal applicants are usually few enough to read in full, and they arrive with a person attached who can answer a question about them. Read those separately, against the same written rule, and record that you did. Everything else goes into the shuffle, including applications from candidates who emailed to follow up, since a follow-up is not something the written rule can check.
Read a Sample, Not the Whole Pile
Decide the read budget in hours and work backwards. Four minutes is a considered read of one application, so eight reader-hours split across two people is about 120 of them, which is a realistic week. Twelve hundred applications is not a reading problem to solve, it is a number to sample from. Take 120 at random, advance whoever meets the rule, and go back for a second sample only if the first falls short.
Calibration is what makes two readers one instrument. Both read the same fifteen applications, decide independently, then compare and reconcile the rule rather than the individual calls. Repeat it once mid-way, because agreement drifts as fatigue sets in.
What this gives up is the theoretical best candidate hiding at position 940. What it buys is a decision made on a stated rule, by rested readers, in a week instead of a month. That trade is usually right, and it stops being right only when the pool is small enough to read properly, which twelve hundred is not.
Then put the real decision on work rather than on documents. A short role-shaped exercise, offered to more people than a resume screen would have let through, moves the judgment onto something the applicant produced for this role. Which application friction pays back covers how to size that step without adding an hour to everyone's evening.
What Makes the Cut Defensible Later?
A written rule, a record of how it was applied, and a check that it sorted on what you meant. Federal selection law is broader than most teams assume: the 1978 Uniform Guidelines (29 CFR part 1607) define a selection procedure as any measure used as a basis for an employment decision, covering everything from tests through informal interviews and unscored application forms 1. A resume screen is a selection procedure, and so is the unwritten judgment that replaced it.
The check that matters most is the cheapest one. Pull thirty rejected applications at random, strip the names and schools, and re-read them against the written rule. You are reading for three things: the rule applied inconsistently, the rule that was never the real basis, and the cut that sorted on how a document reads rather than on the candidate. All three are fixable, and none of them are visible from the shortlist alone.
Run the selection rates too, by whatever categories you already hold. The four-fifths rule is a screening device rather than a pass mark: the agencies that wrote it said in 1979 that it is 'not intended as a legal definition' 2, and it neither clears you when you pass nor convicts you when you do not. It tells you where to look.
Local rules can add duties on top of that. New York City has required since 1 January 2023 that an automated employment decision tool substantially assisting a hiring decision carry a bias audit from within the prior year, a publicly posted summary of the results, and notice to the candidate at least ten business days before the tool is used 4. That duty is New York City's alone, and it lands on the employer using the tool rather than on the vendor selling it, so put the question to counsel before a tool of that kind touches the pile.
Last, do not stack four versions of the same evidence. Combining two or three genuinely different predictors is what recovers most of the accuracy the old single-method numbers promised: a mechanically combined composite reaches a validity of about .61 3. Two resume reads and two unstructured chats are one predictor measured four times. Whether the resume screen is throwing away the wrong people is the question this whole audit answers.
Common questions
Is it acceptable to reject people nobody read?
Rejecting people nobody read is what already happens at twelve hundred applications, so the honest choice is between doing it by an unstated rule and doing it by a stated one. A published cap on how many applications get read, applied to a random sample, is more defensible and more truthful than a queue that quietly ran out of attention at application 200. Tell candidates the posting closed at a cap, and close it rather than collecting more.
Should I buy a tool that ranks the pile instead?
Ask the vendor what the ordering is computed from, whether you can see it per candidate, and what happens to people the ordering places last. If the answer is a proprietary fit number, you have replaced an unstated human rule with an unstated machine one and added a compliance surface. Where a tool substantially assists the decision, local rules can attach duties to it: New York City has required a posted bias audit and ten business days of candidate notice since 1 January 2023 4, so check what applies where the candidates sit.
How large should the random sample be?
Large enough to produce the shortlist you need at the rate your rule advances people. If the rule advances roughly one in ten and you want twelve candidates in a first round, 120 applications is the right order of magnitude. Read them in one or two sittings, then check the advance rate against what you expected. A rate far off the estimate means the rule needs adjusting, not the sample.
What do I keep as a record?
The cut rule as written, the date it was fixed, who read what, the advance counts, and the notes from the blind re-read. That is a page, not a project. It is also the only thing that answers a question from a candidate, a manager or a lawyer a year later without anyone reconstructing the reasoning from memory. Keep it with the requisition rather than in an inbox.
Does the shortlist have to be ranked?
No, and ranking usually adds false precision. A shortlist is a set of people who cleared a bar, and ordering them by a number invites the team to treat small differences as real ones. Present the set with the evidence behind each person, and let the debrief argue about the evidence. If a tie has to be broken, break it on the work sample rather than on the resume.
References
- 1. 29 CFR Part 1607 - Uniform Guidelines on Employee Selection Procedures (1978), sections 1607.16(Q) and 1607.3(A) govinfo.gov Supports the claim that a resume screen and an unscored application form are selection procedures under federal law, alongside tests and informal interviews.
- 2. Questions and Answers to Clarify and Provide a Common Interpretation of the Uniform Guidelines on Employee Selection Procedures (Q.11, Q.19) eeoc.gov Supports the claim that the four-fifths rule is a rule of thumb for directing attention rather than a compliance threshold, in the words of the agencies that wrote it.
- 3. Revisiting the design of selection systems in light of new findings regarding the validity of widely used predictors cambridge.org Supports the claim that combining two or three different kinds of evidence outperforms repeating one kind: a mechanically combined composite reaches about .61.
- 4. Automated Employment Decision Tools: Frequently Asked Questions nyc.gov Supports the claim that New York City has required, since 1 January 2023, a bias audit within the prior year, a posted summary of it, and ten business days of candidate notice before an automated employment decision tool is used.
4 sources, numbered by first appearance. How Olive sources claims
General guidance for hiring teams. What works at one company and one volume may not transfer to yours.
Olive assesses how a person works with AI. It does not detect AI-written documents, and it never produces a score, a ranking, or a match percentage for a person. Candidates read the same report the employer reads.