Three Years of Watching Versus One Hour of Evidence

Why doesn't holding both to the same bar fix this?

Because the standard was never the lopsided part. Open the two files and the bar is identical while the contents are not: one holds three years of reviews written by the person about to make the call, the other holds a single scored session marked against a scale. Judging both strictly leaves the mismatch underneath untouched.

The two files also describe different jobs.

Three years of watching an analyst covers the job that analyst held, and that job may well not have included this competency until recently. If it did not, what you saw was somebody working the old way, well. The session, whatever its faults, is at least aimed at the work the opening is for.

In 1943 Theodore Sarbin reported what happened when counselors' judgment was added to two mechanical predictors of college success. High school rank plus a college aptitude test correlated .45 with academic achievement. The same two predictors plus the counselors' intuitive judgment correlated .35 ¹.

Sarbin studied University of Minnesota undergraduates admitted in 1939, so this is an admissions result and not a hiring one.

Scott Highhouse, a psychologist at Bowling Green State University who reproduces both figures in his 2008 review of why employers keep trusting intuition in selection, also states that the evidence does not support stable differences between good and bad interviewers ¹.

What carries over is narrower than either finding: a judgment layered on top of evidence is not free. It can subtract.

So the repair runs the other way. Rather than equalizing how hard the two are judged, equalize what you are holding, then spend the tenure on the questions a session cannot ask.

Run the same exercise, and let someone else score it

Give the internal candidate the identical session: same task, same clock, same scale, same instructions about what the assistant is allowed to do. The US Office of Personnel Management's practical guide defines a structured interview by three properties, and all three transfer to an exercise.

Every candidate gets the same questions in the same order, everyone is rated on a common scale, and the raters agree in advance what an acceptable answer looks like ².

Then hand the marking to somebody who does not manage them.

A first impression carries into the same rater's structured score, even one formed in a few minutes of small talk. Murray Barrick, Brian Swider and Greg Stewart put 189 accounting students through structured mock interviews and found that the interviewer's impression from the rapport-building small talk, formed before any structured question, correlated .42 with that same interviewer's structured rating ³.

When a different interviewer supplied the structured score, the correlation fell to .25. Much of the .42 was one person's impression predicting their own later rating. Three years of working alongside somebody is not a scaled-up version of three minutes of small talk, and this study does not measure the larger case.

But the protection the smaller one points at costs nothing.

Vary the data and never the task or the bar, which is the same rule that decides whether every candidate has to get the same exercise. Here it earns its keep twice over: internal candidates talk to the team that wrote the case, so the version they sit should carry a different dataset inside the same structure.

Check first whether they have already seen it. A team that validated the exercise by having the current team take it first handed a preview to precisely the people most likely to apply. Write a second case rather than discounting the score afterwards, which is a correction nobody can size.

The shortcut everyone reaches for is the tool logs. Seat usage data measures adoption, not the person's skill, and it answers a question you did not ask: heavy use of an approved assistant can mean delegating the judgment that should have been kept.

What are the three years actually evidence of?

Conduct, under conditions no assessment reproduces. A session shows how somebody works for an hour when the stakes are an offer and the case was written to be fair to everyone. Three years shows what they did the week a project slipped, whether they escalated before or after the slip became visible, and what they said about a deadline they were going to miss.

Those are the questions an exercise cannot ask, which makes them the ones the tenure should be spent on. Write them as episodes with dates.

A hypothetical entry, to show the form: in March the forecast went out with a figure nobody had traced, and the analyst caught it on the read-back and said so in the channel where the team could see it. An episode with a date is evidence.

An adjective is not an episode.

Hold the external candidate to the same form, and the comparison finally becomes one. References supply episodes too, and the useful ones name what the person did rather than what the referee thought of them.

Two records of behaviour, unequal in length and equal in kind, can be read side by side. A rating and a score cannot.

None of this replaces the prior question of whether an internal candidate can really move into an AI-heavy role, which gets answered on that person's own merits before any head-to-head opens. Familiarity does not make a candidate viable who was not.

Which record is weaker than it looks?

Both are, and the external candidate's paper record is the weaker one. The line everyone reads as their advantage, years of relevant experience, is the single piece of evidence on the table with a measured near-zero relationship to later performance. What is strong in that file is the session, not the background sitting behind it.

Chad Van Iddekinge and three co-authors pooled 81 independent samples and found prehire work experience correlated .06 with later job performance, across the 44 samples and 11,785 people that measured it ⁴. Experience specifically with relevant tasks, jobs or occupations did no better, at roughly .07.

The .06 has a boundary, and it cuts toward the internal candidate. What the meta-analysis pooled is prehire experience, the amount, duration or type of experience acquired before somebody enters the organization, which is what a resume screen reads ⁴. It measured nothing about years spent inside the job the internal candidate already holds, so nothing in the .06 says the three years are worthless.

In 2011 Matthew Bidwell, a management professor at the Wharton School, published personnel records from the US investment banking arm of one financial services company, covering 2003 to 2009. Workers promoted into a job performed better than workers hired into similar jobs for their first two years, and left less often, both voluntarily and involuntarily ⁵.

The external hires were paid around 18 percent more, and arrived with more education and more experience ⁵. Stronger on paper, weaker in the seat, for two years.

The complication sits inside that same finding, and it runs the other way from the obvious guess. Performance at Bidwell's firm is rated inside the firm, by the kind of manager whose familiarity with a promoted worker is the problem this page opened on. Bidwell names that worry and tests it. The promoted worker's advantage was slightly larger among traders and salespeople, not smaller as a subjective rating bias would predict, and his reading is that supervisory bias may actually narrow the difference between promoted workers and external hires ⁵.

The limit that does bite is a different one. This is one investment bank between 2003 and 2009, and the paper's discussion says the performance gap should be smaller where a job demands less firm-specific knowledge ⁵.

Anyone who already prefers the internal candidate can quote the two years. Nobody can quote a rating bias that produced them.

Write down which evidence decided it

Set the combination rule before either record is in front of you, then record what actually moved the call. Name the four or five things the role needs, say which piece of evidence covers each one, and say in advance what happens when the two candidates split across them. A rule invented after the debrief is a preference wearing a rule's vocabulary.

Nathan Kuncel and colleagues meta-analysed selection and admissions studies and found that applicant data combined by an explicit formula correlated .44 with job performance, against .28 when the same kinds of data were combined by expert judgment ⁶. A mechanical method here can be as plain as adding up a written scorecard, and the paper says so.

The job-performance comparison rests on nine studies, which is thin, and none of it licenses a number standing for a person.

The classification underneath is the more useful part. That same paper counts a group consensus meeting as a holistic method, because what makes a method holistic is combining evidence by judgment rather than by a rule applied the same way every time ⁶.

A debrief that ends in a shared feeling is the thing being warned about. The debrief is not.

So write the note the way you would want it read back to you. Write what you heard, never why you think it happened carries more weight here than in an ordinary round, because the losing candidate is at their desk on Monday and their manager was in the room.

By the end the two files should look alike. Each holds one scored session on the same case, a short list of dated episodes, and one line naming which of them decided the call.

The internal file is still three years thicker. Only a page or two of that decided anything, and those pages are now the part somebody else can read.

See the benchmarks >