What an AI Interview Score Predicts, and Who Measured It

What Does the .24 Actually Mean?

It means that in the study behind that figure, the machine's rating and the employee's job performance moved together weakly. A 2024 study of automated video interview scores reports an uncorrected sample-weighted correlation of .24 across five organisational samples totalling 1,124 people ¹. The samples ran from .20 in a maintenance role to .27 in a call centre ¹.

That is a real signal, and it is a small one.

Who produced the number matters as much as its size. Four of the study's five authors were employed by HireVue, the vendor whose algorithms were evaluated, and the assessments studied are that vendor's product ¹. Peer review is not the same thing as independence, and nobody has replicated the result from outside. The authors also say they are unaware of any previous literature examining scores of this kind against key organisational criteria ¹. So the published record on job performance starts here rather than settling here.

The comparison the authors reach for is a human interviewer, scored on the same raw basis.

They set their .24 beside .32, the uncorrected figure for human-rated structured interviews, and note that their estimate rests on five studies against the 105 effect sizes behind that comparison ¹. Both figures are raw, uncorrected for the unreliability of performance ratings and for range restriction, which makes the comparison internally fair. Neither one is the .42 usually quoted for structured interviews ³.

Why the Structured-Interview Number Doesn't Transfer

Because that .42 was never measured on a machine-scored interview. Sackett and colleagues, whose 2022 re-analysis puts structured interviews at the top of its table, note that interviews become more amenable to mass administration as their administration and scoring move toward automation ². Validity, they add in the same sentence, needs evaluating under those changed circumstances ².

That last clause is doing a lot of work and it is easy to miss. It comes from the paper that ranks structured interviewing first, and it says validity under automated conditions still needs evaluating ². Not that automated scoring failed. Not that it passed. The .42 belongs to interviews humans conducted and humans rated, and moving it across to a machine-scored version is an assumption dressed as a finding. The authors flag the gap; they do not fill it ².

An open question is not a verdict, in either direction.

Two kinds of number circulate in this field, and they do not mix. Before any correction, the mean observed correlation between structured interview scores and rated job performance was .28 in one meta-analysis and .36 in another ³. The familiar .51 came from correcting .28 up to .37 for unreliable performance ratings, then up to .51 for range restriction, a step that added 38% on its own ³. Corrections exist for good reasons. A corrected figure still cannot be set beside a raw one.

Published estimates move too, including the famous ones. Work samples were called the best predictor of job performance at .54 for decades, and the same 2022 paper puts them at .33, a drop the .54's own lineage had already made ⁶. Read that as a benchmark being corrected, not a method being demoted ⁶.

Would the Score Be the Same Tomorrow?

Not exactly, and there is a measurement of how far apart two sittings land, taken on research models rather than a vendor's product. In an academic study, 99 students recorded the same six interview questions twice, on average 15.6 days apart, and video-interview personality models scored both recordings ⁴. Test-retest correlations averaged .46 for the models trained on self-reports and .50 for those trained on interviewer reports ⁴.

Correlations of that size mean a good part of what one sitting produces does not carry over to the next.

The detail underneath the average is the part worth reading. Individual trait models ranged from .01 to .85, and the self-report models were least stable on conscientiousness at .30, openness at .32 and extraversion at .34 ⁴. The authors describe their figures as higher than earlier interview retest studies over longer gaps, and noticeably lower than self-report personality scales ⁴. The traits that sound most like a reason to hire somebody are the ones the self-report models held least steadily between sittings.

None of that says which of the two sittings was the right one.

The setting limits what the finding covers. Those were students in a lab answering identical questions for course credit, and the models were built by researchers rather than sold to employers ⁴. No vendor's live score has been retested from outside.

So this describes a method, not the product you sat in front of last week.

Ask What the Score Is Used For

The more useful question is where the score sits in the process. A weak signal used to order a large pile is a different thing from a weak signal used to decide, and both get called a score. Sackett and colleagues make the practical point in passing: the strongest predictor in their table, in its traditional in-person form, is generally not viable for high-volume jobs ².

A 2020 review of vendor websites found most did not show their working. It looked at 18 algorithmic pre-employment assessment vendors, working only from what their websites disclosed at the time ⁵. The sites generally did not make clear whether the vendors validate their models, by what method, or on what data ⁵. Fifteen referred to bias at least in the abstract, and seven discussed compliance or adverse impact for the assessments they sold ⁵.

One published validation studies for its psychometric component but not for its algorithmic recommendations ⁵. That snapshot is six years old and the market has turned over since.

Absence of a validation claim on a website is not proof that no validation happened ⁵.

Employers get told to ask vendors for evidence too, and you can read what they get told. How to evaluate an AI-skills assessment before buying sets out what a buyer is supposed to put to a vendor. Whether assessment scores still predict performance once AI is in the workflow works a related validity problem from the hiring team's chair. Neither is written for you, which is what makes them worth reading.

Your own version of the question is smaller and still worth asking: what does this assessment measure, who reviews the result, and does a person speak with you before a decision. What an AI interview actually scores covers what one large vendor publishes about what its system reads.

A .24 will not tell you why you were passed over.

See what gets scored >