Interviewing
Leading an AI Rollout Is Adoption Work, Not Proof of Craft
Leading a company's AI rollout is evidence of change management, not evidence that the person is good at the work itself. The deliverables of that job are licenses issued, sessions run and an adoption curve, and none of them require craft with the tool. Ask the two halves separately: what changed in how the organization worked, and what the candidate personally made with the tool and threw out. A candidate who answers only the first may still be the right hire for a change role.
The takeThe rollout line is an easy credential to over-read, because it sounds like leadership and expertise arriving together. Read as craft, it flatters the wrong candidate. Read as nothing, it costs you a capable change lead. Enablement people can be misfiled in both directions: hired into craft roles nobody tested them for, rejected from the change job they might be excellent at. Decide which job is open before you decide what the line is worth.
Where Olive fits
Open a role and see what the work shows
An interview can capture a candidate describing the rollout they ran; it cannot capture them doing the work the rollout was for. Olive puts an occupational assignment in front of them, with an AI assistant that will overreach if nobody stops it, and a human reviewer reads what happened and writes or edits six findings, with the timestamped excerpt behind each one.
Assess your shortlistWhat does leading a rollout actually put on the record?
Running a rollout puts change management on the record. Licenses procured, a security review survived, a training curriculum written, three skeptical directors talked round, an adoption curve that went up and stayed up. Every one of those is a real accomplishment, and not one of them requires the person to have been good at the work the tool does.
The deliverable is that other people used the tool.
The metrics follow from the deliverable: seats activated, sessions attended, weekly active users, a support queue that shrank.
What an adoption count does not measure is whether the work got better. Adoption is the number on the dashboard because adoption is the number that is cheap to count, and a license purchase plus a mandatory training week can move the first three.
The word rollout also covers a great deal of ground.
Between November 2025 and January 2026, the US Census Bureau's Business Trends and Outlook Survey found 18 percent of firms using AI in a business function 1. Among the firms that had adopted it anywhere, 57 percent used it in three or fewer of the fifteen functions the survey asks about.
So ask which functions.
A candidate who took AI through sales and marketing and a candidate who took it through claims processing and legal review did different jobs, and the second one had to argue with people who could say no.
An adoption curve also says nothing about who benefited. In a staggered rollout of a conversational assistant to 5,179 customer support agents at one software firm, issues resolved per hour rose 14 percent on average: 34 percent for novice and low-skilled agents, and almost nothing for the experienced ones 2.
Those 5,179 agents sat in one occupation at one firm, using one assistant trained on that firm's own transcripts. The number belongs to that setting, but its pattern is worth carrying: the same tool changed different people by different amounts, so a company average describes who was in the building and says nothing about the person who ran the program.
Seat usage data measures adoption, not the person's skill, even when the only seat on the report is the candidate's own.
Ask the program question and the craft question separately
Ask two questions, in two turns, and do not let the first answer serve the second. One: what changed in how the organization worked after the rollout, and how did anyone know. Two: name something you made with the tool in the last three months, what you kept from what it gave you, and what you threw out.
The order matters. Put the program question first and let the candidate have it in full, because someone defending the rollout and proving craft at the same time will merge the two, and then you cannot tell which one you heard.
Asking beats inferring from the line on the resume.
In a meta-analysis of 81 independent samples, the 44 that reported job performance put its correlation with prehire work experience at .06 3. Experience with tasks, jobs or occupations relevant to the position did no better, at about .07. Both figures already carry the standard statistical corrections, so the near-zero is not a product of leaving those out.
The finding is about experience measured at hire, which is exactly what a resume line is. It does not say the rollout lead is no good. Applied to a rollout claim, it suggests the line by itself carries little information about performance, and asking for the work behind it is a direct way to get more.
If the job you are filling is a management one, the program half is the more relevant half, and the questions change shape. Ask what the team automated, what the manager refused, and what the review caught last quarter.
The craft answer is an episode, and its speedup claim needs a stopwatch
Read the craft answer for an episode. A candidate with craft answers the second question with a specific thing in a specific week: the draft they rewrote, the number the model got wrong, the paragraph they cut because nothing outside the conversation supported it. A candidate without it answers in the vocabulary of the rollout, and stays there when you push.
Be careful with the part of the answer that is about how much faster everything got.
On repositories they had worked on for about five years, 16 experienced open-source developers completed 246 real tasks in a randomized trial 4. They forecast a 24 percent speedup from early-2025 AI tools. Afterwards they estimated a 20 percent speedup. They were measured 19 percent slower.
The study is small and the developers knew the code intimately, so the magnitude does not transfer. The gap does. People can misjudge what a tool is doing to their own speed, and these ones got even the direction wrong.
So do not score the claim that adoption made the team 40 percent faster. Score whether anybody measured it, what the number was before, and who held the stopwatch.
An episode is still heard by a person, and two people hear the same one differently.
Settle what a strong answer to each question contains before the first interview, and put anchors on the scale. In a meta-analysis of 19 past-behavior interview studies, interviews using anchored rating scales tracked job performance more closely than interviews without them, .35 against .26, and two interviewers agreed slightly more closely, .77 against .73 5. Anchors are meant to make "strong" mean one thing across a whole loop, which is the point of running a structured interview about AI use so candidates are actually comparable.
One follow-up belongs to the checking. Every candidate says they verify what the model produces, and the follow-up that tests whether that is real asks what the last check was against and when it happened.
Treat a program lead as a different hire, not a failed one
Treat the two outcomes as two decisions. A candidate who answers the program question well and the craft question thinly has told you something true and useful: they are good at getting an organization to change, which is hard, and easy for a loop built around craft to miss. What they have not shown is that they can do the work the posted job is made of.
What a test is about changes what it predicts. In the job knowledge meta-analysis behind the 2022 revision of the selection validity table, all 164 studies together produced a mean observed validity of .22, while the 59 that used tests built for the job in question produced .31 6.
Subsets of one meta-analysis produced that comparison, rather than a controlled test, and job knowledge tests presuppose people who already hold the knowledge.
Read narrowly, the comparison says job-specific knowledge tests out-predicted the full set of 164. Carried over to interviews and exercises, that is an extension the study did not test, though it is the direction the evidence points.
So aim the assessment. A craft opening makes the rollout answer background and lets an exercise decide it. An enablement or change opening makes the rollout answer the substance, and the craft question becomes a check that the person can tell good work from bad.
A reference call can corroborate what an interview cannot, if the question is narrow enough. Ask for one incident the reference watched, an AI output that was confidently wrong, and what the candidate did in the next hour.
Where the rollout lead was already the best practitioner in the building, the separating costs little, because both answers come back strong.
In plenty of companies the enablement job went to that person for exactly that reason, and the two kinds of evidence really do travel together there.
The line on the resume has not changed. It now describes one job, done at one company, with a second question sitting next to it. A candidate who did both halves has been waiting for somebody to ask the second one.
So ask it.
Common questions
Does leading an AI rollout mean a candidate is good with AI?
Leading a rollout is evidence the candidate can get other people to use AI, which is a different skill and a valuable one. The deliverables of a rollout are procurement, training, security sign-off and an adoption curve, and a person can produce all four without having made anything good with the tool themselves. Treat the claim as evidence for one job and untested for the other, then ask a separate question about their own work: something they made in the last three months, what they kept, and what they threw out.
What single question separates rollout leadership from personal skill?
Ask for one thing the candidate produced with the tool recently, and then ask what they rejected from it and why. Program answers describe populations, timelines and change plans. Craft answers describe an artifact and a decision: this draft, that wrong figure, the section cut because nothing outside the model supported it. Someone who did both can move between the two registers when asked. Someone who only ran the program will answer the second question in the vocabulary of the first, and will stay there when pressed for the specific week it happened.
Should an enablement lead be rejected for a hands-on role?
Not automatically, but do not let the rollout stand in for the exercise. Run the same work sample every other finalist runs and read it on its own terms. The honest risk is a ramp: someone whose last two years were spent in training rooms and steering committees may be rusty at the work itself, which is a timing question rather than a capability question. If the loop shows craft, the change-management history is a bonus almost no other candidate carries.
How do you check whether the rollout actually worked?
Ask what was measured, what the number was before, and who owned the measurement. Most rollouts report adoption, which counts usage rather than results, and a license purchase plus a mandatory training week moves adoption on its own. A candidate who can name a work outcome, a baseline and the person who held the data ran a serious program. A candidate who can only supply weekly active users ran a deployment, which is still real work and a much smaller claim.
What if the candidate cannot give examples because of a confidentiality agreement?
Ask for the shape rather than the content. A candidate can describe the type of document, the kind of error, the check they ran and the decision they changed without naming a client, a dataset or a figure. If that still fails, move the evidence to an exercise on your own material, where nothing is confidential and everyone gets the same brief. Never treat a confidentiality agreement as evasion by itself, and never make the candidate choose between an offer and their obligations to a former employer.
Is a rollout lead the right person to set AI policy for a team?
Often, yes, and that is a distinct question from craft. Policy work needs someone who has argued with security, written rules people follow, and handled the person who refused to change. Rollout leads have usually done all three. What to test is whether they can tell strong work from weak work in the discipline they will be writing rules for, because a policy written by someone who cannot read the output tends to govern usage instead of quality.
References
- 1. The Microstructure of AI Diffusion: Evidence from Firms, Business Functions, and Worker Tasks (CES 26-25) ✓ www2.census.gov Supports the claim that a company AI rollout usually covers a narrow slice of the business: 18 percent of firms used AI in a business function between November 2025 and January 2026, and 57 percent of adopting firms used it in three or fewer of the fifteen functions surveyed.
- 2. Generative AI at Work (NBER Working Paper 31161) ✓ nber.org Supports the claim that a rollout's average gain describes a population rather than a person: 5,179 support agents at one firm, 14 percent more issues resolved per hour on average, 34 percent for novice and low-skilled agents and minimal impact for experienced ones.
- 3. A meta-analysis of the criterion-related validity of prehire work experience ✓ digitalcommons.unf.edu Supports the claim that the type of experience named on a resume carries almost no information about later performance: 81 independent samples, corrected correlation .06 with job performance across the 44 that reported it, about .07 for experience relevant to the position.
- 4. Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (arXiv:2507.09089) ✓ arxiv.org Supports the claim that people can misjudge what an AI tool does to their own speed: 16 developers, 246 tasks, a forecast 24 percent speedup, a 20 percent speedup believed afterwards, and a measured 19 percent slowdown.
- 5. The Structured Employment Interview: Narrative and Quantitative Review of the Research Literature ✓ doi.org Supports the claim that anchored rating scales go with higher validity and slightly closer agreement between interviewers: across 19 past-behaviour interview studies, validity .35 against .26 and interrater reliability .77 against .73 for interviews using anchors.
- 6. Revisiting Meta-Analytic Estimates of Validity in Personnel Selection: Addressing Systematic Overcorrection for Restriction of Range ✓ static1.squarespace.com Supports the claim that knowledge tests built for the job being filled predicted better than the full set of job knowledge studies: 164 job knowledge studies produced a mean observed validity of .22, while the 59 using tests built for the job in question produced .31.
6 sources, numbered by first appearance. Every one was opened and checked against the claim it carries. How Olive sources claims
General guidance for hiring teams. What works at one company and one volume may not transfer to yours.
Olive assesses how a person works with AI. It does not detect AI-written documents, and it never produces a score, a ranking, or a match percentage for a person. Candidates read the same report the employer reads.