Brilliant in the Screen, Flat Onsite: Which Candidate Is Real?

Which round should you believe?

The one whose conditions look like the job. If the work happens at a screen with a browser, an editor and an assistant open, the remote round is the closer sample and the conference room is the artificial one. If the job is a room, a whiteboard and four people disagreeing, the onsite is the sample. Pick before you reread your notes, so the choice is not a rationalization of the impression you already formed.

Which round is closer changes by occupation, and it is worth being literal about it:

• Software engineering. The daily instrument is an editor, a terminal, documentation and an assistant that will write the whole change if nobody stops it. A marker at a whiteboard tests recall of syntax nobody types from memory any more.
• Financial analysis. The memo gets built in a spreadsheet against a filing somebody has to open. A case discussion with no file open measures how well a person talks about analysis, which is a different skill and a real one.
• Underwriting, claims and revenue cycle. Queue work, alone, at a screen, all day. A panel round measures almost nothing the job asks for.
• Management consulting. Rooms are half the job, so the onsite is a genuine sample of that half. The other half is done on a laptop at 11pm.
• Sales, teaching, clinical work, anything where the room is the deliverable. The onsite is the sample and the remote screen was the artificial one. The gap runs the other way and means the opposite thing.

If the role genuinely contains both, then you have one observation of each and no contradiction between them. What you know is that this candidate is stronger with tools and time than in front of strangers with neither. That is information about how to deploy someone, not a verdict on which version was authentic. It is also why candidates who interview brilliantly can struggle in their first quarter: the round that impressed you was not measuring the conditions the work imposes.

How much of the gap is your interview format?

More of it than a debrief usually credits, and the direction is not fixed. A meta-analysis of twelve studies (13 samples, 1,557 interviewees) found interviewer ratings lower in technology-mediated interviews than in face-to-face ones, a moderate negative effect (d = -.41), with applicant reactions lower too (d = -.36) ¹. The medium moved the ratings before any candidate did anything.

Nerves in the room is the next explanation a panel reaches for, and one controlled comparison does not support it. Eighty-eight interviewees were randomly assigned to a face-to-face, telephone or videoconference interview, with strain measured physiologically as well as by self-report. There were no differences between the three media on strain or interview anxiety, and ratings of interviewee performance were still lower in the technology-mediated conditions ². Strain did not separate the formats. The ratings did.

Which direction the format pushes belongs to your design, not to the medium. Across 1,026 applicants to one pharmacy program, candidates in the virtual multiple mini interview scored higher than the in-person cohort on three stations, with medium effects on two teamwork stations (D = 0.44 and D = 0.47) and a small one on integrity (D = 0.28) ³. Same intent, opposite direction from the meta-analysis. Nobody can tell you in advance which way your own loop leans, and the only thing that settles it is your own ratings.

And the medium was not the only thing that changed between the two rounds. Count the rest:

• Who asked. One recruiter who runs screens daily, then four people who interview twice a year.
• What was asked. A screening script, then role-specific depth nobody calibrated in advance.
• What was allowed. Notes, a browser, a second monitor, then a marker.
• The audience. One listener, then a panel watching each other watch her.
• The clock. Forty-five minutes, then six hours and a lunch that is also being observed.

Everything moved at once and one impression came back different. There is no honest way to attribute that to the candidate.

What if the remote round had AI in it?

Assume it did, and notice how little that settles. Unless the invitation said not to, in writing, before the call, there was no rule to break. A second window with an assistant in it is ordinary equipment. What you want to know is what the candidate did with what it produced, and the onsite round is the one that removed the evidence rather than revealing it.

How ordinary depends on the job, which is worth checking instead of assuming. In nationally representative US surveys reported in 2024, 23% of employed respondents had used generative AI for work at least once in the previous week and 9% used it every work day ⁴. That is a real share and not a universal one. If it describes the role you are filling, the assisted round is the closer sample of the work. If the role's work genuinely happens without a model in the loop, the onsite is.

The delta itself tells you nothing about assistance. A candidate who prepared hard, who writes better than she speaks, who had a bad morning, or who faced four people at once produces exactly the same shape. Inferring AI use from a performance drop is the same move as inferring it from prose style, and it lands hardest on people whose fluency does not match what an interviewer expected. If the question is live and specific, answer it directly: whether candidates are using AI to answer your questions during live interviews has a real answer, and a candidate's eyes moving off camera is not it.

What is readable is the substance of the remote answers, reread with the polish set aside. A fluent answer with nothing behind it fails on its own terms: no claim traced to a source, no alternative weighed, false precision where she could not have had the data. Name that if it is there, and name it about the answer rather than about the tool. Every candidate suddenly giving the same polished STAR answer is the version of this problem waiting in your next loop anyway.

Ask about the delta, don't infer it

Book twenty minutes and put the two rounds side by side out loud, with no accusation in the sentence. Say what you observed, then ask about the work rather than about the tool: what a specific claim from the screen answer rests on, what she would check first if that number were wrong, and what was different for her between the two rooms. Specifics arrive fast from someone who did the thinking.

Three questions carry it:

• "In the screen you said the migration would take a quarter. What is that resting on, and how would you check it?"
• "What did you use to prepare, and where did it get something wrong?"
• "Which of the two conversations was closer to the work you actually do?"

Ask all three of every finalist, not only of the one whose rounds disagreed. A question asked of one person is a different test given to one person, and it is the version that becomes expensive to explain later. The answers are also usable either way: someone who did the thinking names the source and the correction inside a minute, and someone who did not moves to generalities and stays there.

Better than asking is watching a little of it happen. A short working session in the format the job uses, with the tools the job supplies, run identically for every finalist, retires the question entirely, because both rounds become the same test. Redesigning the interview so AI assistance becomes signal instead of cheating is the general form of that fix. See how Olive measures this.

Write down what would change your mind before the session, not after. "She defends the screen answer with a source she opens in front of me" is a check that can fail. "She seemed more confident this time" is the impression you already had, arriving twice.

Fix the loop before the next candidate splits

Hold the medium constant inside a stage. The meta-analysis that measured the mode effect closes on exactly this point: organizations should be wary of varying interview mode across applicants, because inconsistency in administration raises fairness questions ¹. If some finalists get a video call and others get a room, part of what you are comparing is your own calendar.

If a loop runs two formats on purpose, ask one identical question in both. Without a shared item, "stronger in the screen" has no unit: two different sets of raters heard two different conversations, and subtracting them produces a number about the loop. One repeated question turns a mood into an observation with something to hold against it.

Then keep one rubric across both rounds, written before either happens, with the evidence each rating needs named in it. Consistency is doing more work than it used to, because the answers themselves have converged: a well-prepared candidate now arrives with a coached version of every standard question, which is why a structured interview no longer separates candidates the way it did without something behind the questions.

Do all of that and neither round is predictive on its own. A screen is forty-five minutes on a laptop, an onsite is one afternoon in front of strangers, and both are samples of behavior under conditions you built. The useful thing to do with a candidate who split across them is to write down which conditions the job actually imposes, run one round under those conditions for everyone, and let the split stop being mysterious.

See how it works >