Build a Pipeline of Dated Evidence, Not a List of Interested People

What is worth keeping in a pipeline?

Four fields per person, and a rating is not one of them: the artifact they produced, the date, the rules in force when they produced it, and one sentence naming the capability it showed. Everything else in a typical talent-community record is either recoverable from the person later or already stale by the time a requisition opens against it.

The reason to build anything at all is elapsed time. Across Ashby's own customer base, which skews to venture-backed technology employers, median time to first fill runs 56 days for business roles and 76 for technical ones, roughly eight to ten weeks ¹. Read that clock carefully: it stops at the first hire on a requisition, and SHRM's 39 calendar day median for nonexecutive positions, from a survey of over 4,600 organisations, runs a different clock over a different population ². Nothing published apportions the gap between those two causes, so read each figure on its own terms. Either way, the wait between opening a role and having somebody in it is long enough that a warm start is worth something, which is the honest case for a pipeline.

What that warm start consists of is where the vendors and the evidence part company. A stored record is worth having when it lets a future reader form their own view. So:

• The artifact. The exercise they submitted, the portfolio link, the writing sample, the thing itself.
• The date. When the work was made, which is rarely the day you added them to a list.
• The rules in force. What tools were permitted, how long they had, whether the brief was yours or theirs.
• One sentence. Which capability the work demonstrated, in the vocabulary of the job, written by whoever read it.

How fast does stored hiring evidence spoil?

About a year for a judgment about somebody's output, and indefinitely for the underlying facts. What a person built and where they worked ages slowly. What an interviewer concluded about their output ages with the tools, because a written sample or a coding exercise from two years ago was produced under different conditions than one produced this month, which makes the two incomparable even when the exercise is word for word identical.

The size of that shift is easy to underestimate. Evaluated against 115 Python problem statements taken from HackerRank, OpenAI's Codex solved 96% of them with no examples and all of them with a few, and the authors report clear signs that the model was reproducing memorised code rather than synthesising it ³. The memorisation half is the load-bearing half and it usually gets dropped: the paper's own contribution is a way of re-testing on mutated problems precisely because the raw pass rate overstates capability. For a pipeline, though, it reads as a straightforward warning. An exercise drawn from a public bank was already testing a lookup in 2022, on a model several generations old, so a stored pass on one from 2023 is thin evidence about anybody and no evidence at all about this year's pool.

Hence the shelf life, and shorter still where the exercise came from a public source, since that one was testing a lookup on the day it was set. Mark the lapsed record expired and keep it: the artifact and the date stay useful as context, and the expiry flag is what stops a future reader taking an old judgment for a current one.

The specific case of reopening an old finalist pool, where every record is expired at once, is worked through in whether 2024 interview evidence still holds.

Write down what the person actually did

Write the observation: which capability the person demonstrated, on what task, and what specifically they did that showed it. A number is the part of a record nobody can argue with later, which is exactly why it is the wrong thing to store. A sentence naming a capability still means something to somebody who was not in the room. Nobody can reconstruct what a four out of five was for once its author has left.

Interviewer notes are a rarer object of study than interviewer ratings. Researchers text-mining post-interview notes on 7,650 candidates who were hired at a large Chinese internet technology company found that the number of job-related capabilities an interviewer named in the notes related positively to later job performance and promotions, and negatively to turnover, with a standard deviation rise in the note-to-job-analysis matching score corresponding to roughly a 2 percent rise in performance ⁴. Take that with its limits. It is correlational, at one firm, in one country, between 2016 and 2018, and it can only see candidates who were hired, so it says nothing about the people the notes described and the company rejected. The effect is small, and it points the right way.

Nobody can say what a stored rating was worth, including the people who stored it. In LinkedIn's 2025 survey of 1,271 recruiting professionals in management roles across 23 countries, only 25% said they felt highly confident in their organisation's ability to measure quality of hire effectively ⁵. That is a self-report from a convenience sample of platform-active talent professionals, not an audit of what anybody measures, and quality of hire is a downstream outcome, some distance from the interview score itself. Still, an employer that cannot tell how its own hiring judgments turned out has nothing to calibrate a stored score against, and a written observation needs no calibration to be read.

So write the sentence in the job's own vocabulary. Not strong candidate. Rebuilt the reconciliation from the raw ledger and found the two entries that did not net, without being told they were there.

Keep the nurture small and the promise honest

Two or three genuine updates a year is the whole of it. Somebody who agreed to stay in touch expects to hear something they could not read on the careers page: what the team shipped, what changed about the role, what the hiring plan actually is now. Anything more frequent teaches them to filter you, and anything vaguer is not worth the time it takes to write.

The re-approach is where the record earns its keep, and honesty is the cheapest thing in it. Say what you have: your exercise is from early 2024, the tools you were allowed then are not the tools now, so treating it as current would not be fair to you. Would you do a fresh one. Almost nobody objects to that, and somebody who would rather not has told you they are not available for this round, which is worth knowing on day one.

Three things are worth deciding once, in writing, before the pipeline has anybody in it:

1. What you keep, and for how long. Write it down and apply it. Record-retention duties differ by jurisdiction and can set a floor under your own preference, so settle the number with counsel before it becomes policy. A record you would be uncomfortable showing the person it describes should not be in there. 2. Who can read it. A stored observation written for a hiring team reads differently when a manager finds it two years later attached to an internal applicant. 3. How somebody leaves. One reply should be enough to take a person off the contact list for good, and it should actually take them off.

Before any of this, know what the role will require, because a pipeline built against a job description that has stopped describing the work stores the wrong artifacts carefully. That groundwork is finding out what AI actually does in the role, and it is the difference between a pipeline of evidence and a list of names that were relevant once.

See a sample report >