Interviewing
Ask a VP Candidate What They Stopped, and What It Cost
An executive candidate's AI judgment shows in allocation decisions they already made, not in the policy they can describe. Ask which AI project they stopped and what had been spent when they stopped it. Ask which work they refused to route through a model and who argued back. Ask what changed after the first deployment came in worse than promised. Vision language is free and everybody has read the same material. A decision with a date, a number and a price has to have happened.
The takeThe vision round is not merely weak evidence at this level. It is evidence that can pull a prediction away from better evidence, and a panel is unlikely to feel the pull. Everyone in the room read the same essays this quarter, so fluency in that vocabulary flatters every candidate and separates none of them. The hour is better spent on a decision that cost the candidate something: the project they cancelled after real money went in, the automation they blocked and then had to defend to a peer. Nobody who has been close to this work has been right about all of it.
Where Olive fits
Open a role and see what the work shows
An interview can hold a candidate's account of a decision they made two years ago; it cannot hold them making one. Olive puts a role-grounded assignment from one of twelve occupations in front of the person as work, with an assistant that will overreach if nobody stops it, and returns six findings a person wrote or edited, each anchored to the moment in the session it rests on.
Assess your shortlistWhy a strategy hour can make your prediction worse
The reading list is short and public. The same essays, conference talks and survey decks that shaped your own view of AI this quarter shaped the candidate's. A fifty-minute conversation about strategy mostly measures who read them most recently. Fluency in a shared vocabulary is real, and it is not rare, and it separates nobody in a finalist pool already filtered for seniority.
A 2013 paper in Judgment and Decision Making measured what an unstructured conversation does to a prediction. Seventy-six undergraduates predicted a fellow student's semester grade average. Predictions made after interviewing that student correlated .31 with the actual result. The student's prior cumulative average, used by itself, correlated .65 2.
The interview made the prediction worse.
The third study in that paper is the one to carry into an executive loop. Asked to rank four ways of making the prediction, 96 of the 169 participants ranked an interview in which the interviewee answered at random above having no interview at all 2. Interviewers make sense of whatever they are handed.
Undergraduates predicting a fellow student's grades are not a panel predicting a VP, and the samples are small.
But the mechanism is what transfers, not the size of the effect. Weak information dilutes strong information, and the dilution is easy to miss while it happens.
Executive loops can add a step of the same kind. When the last conversation is a principal's own read, it can outrank everything written down before it.
Scott Highhouse, writing in Industrial and Organizational Psychology in 2008, reproduces the result of a 1943 study of Minnesota undergraduates admitted in 1939. High school rank plus an aptitude test predicted academic achievement at .45, and the same two predictors plus the counselors' intuitive judgment predicted it at .35 3.
Adding the judgment lowered the prediction.
The same review reports that measured differences between interviewers look like sampling error, so the unusually good interviewer is not a category this evidence supports 3.
Ask for three decisions, with dates and prices attached
Three questions do most of the work, and all three ask for something that already happened. Which AI project did they stop, and what had been spent when they stopped it. Which work did they decide would not go through a model, and who argued the other side. What changed after the first deployment came in worse than the business case promised.
Ask all three of every finalist, in the same words, and have each panellist score them against a written scale before anyone compares notes.
Structured interviews sit at the top of the ranking in the 2022 re-analysis of the personnel selection literature by Sackett, Zhang, Berry and Lievens. Their estimate of each format's correlation with supervisor-rated job performance is .42 for structured interviews against .19 for unstructured ones. Before any statistical correction the observed figures were .32 and .13 1.
Structured there is a research coding rather than a product: the same questions, rated on a common scale. The paper puts the standard deviation of that .42 at .19 and the lower bound of its 80 percent credibility interval at .18, so it describes a body of studies, and your Thursday panel is one draw from it.
A structured interview in a particular setting can still land below the unstructured average.
A strong answer arrives with the boring parts intact. A date. The amount already committed, or its rough size where the figure is confidential. Who argued the other way, by role if a name would breach a confidence, and what they argued. What the stop cost, in money or in a quarter of roadmap.
What a candidate refused to delegate is worth asking at every level. At this one the refusal shows up as a budget line and a written policy sentence.
Those three questions are not the manager version. Four questions for a manager whose team ships with AI go at how a team works week to week, and that shows in how the manager describes review and handoff.
An executive's evidence is money, scope and policy, which is older, larger and easier to check.
How do you know a stop was real?
Push twice on each decision and watch whether the specifics hold. The first follow-up asks for the number the decision rested on and where that number came from. The second asks what would have had to be different for the call to go the other way. Rehearsed talking points thin out at the second push, because they have no second layer under them.
Follow-up questions that expose whether someone understands their own answer work here, with one addition at this level: ask how they learned the deployment was underperforming.
How they knew is the whole question. An executive who says the team flagged it has described a reporting line. One who names the measure, the baseline it was read against and the month it turned has described a decision they were actually inside.
Then ask for one they got wrong.
A candidate who names no wrong call on AI has given the panel nothing to test here, so ask for a smaller one.
The useful version is specific: what they funded that did not work, what the first signal was that they had misjudged it, and how many months ran between that signal and the stop.
Watch for the answer that gets more particular under pressure. Talking points go vaguer at the second push while lived decisions go finer, and the difference is legible to a panel that agreed in advance what it was listening for.
What the job owns: scope, and in the EU a duty already in force
The job is scope, and what the function is already on the hook for. The Census Bureau's Center for Economic Studies found 18 percent of US firms using AI in a business function between November 2025 and January 2026. Weighted by employment, the share was 32 percent 4. Among the firms that had adopted it at all, most had not spread it far.
Fifty-seven percent of adopting firms used AI in three or fewer of the fifteen business functions the survey lists. The most common were Sales and Marketing at 52 percent, Strategy and Business Development at 45 percent and IT at 41 percent 4.
The VP is being hired to decide where in the function AI stays out, and for how long.
In the EU, deploying AI in any function already carries a duty. Article 4 of the EU AI Act has applied since 2 February 2025, and it is not limited to high-risk systems. It requires providers and deployers of AI systems to take measures on the AI literacy of their staff and of anyone operating AI on their behalf 5.
An employer in the EU running a bought AI tool is a deployer, so the duty reaches ordinary functions and not only vendors.
The wording has already moved once. Amendments from the Digital Omnibus on AI entered into force in mid-July 2026. The European Commission's own questions and answers now state that AI literacy remains an obligation for providers and deployers, while no specific level is mandated 6.
Widely used trackers were still reproducing the pre-amendment sentence as current after the change took effect.
Whether the function's written policy quotes the current text is the new VP's to own.
Close the round with the forward version of the same question. Ask what they would need to see in their first ninety days to reach a fund-or-stop decision on the function's largest AI commitment, and what they would read to see it.
What HR has to build in the 90 days after an AI-first mandate is a related ninety-day problem, seen from HR's chair.
Ask the references about the same three decisions
Reference the decisions, not the person. An executive reference call built on an overall impression gets a summary back. A former board member's summary of a good operator is not evidence about a specific call. Name the project, the stop and the refusal the candidate described, and ask each reference what they remember about those three.
Asking a reference for one episode instead of an overall impression is the general form of the question.
The executive version carries one extra hazard: the references are chosen, senior and practised, and an impression question invites a summary they have given before.
Ask each of them the same three, and ask what they personally saw.
When two references describe the same stop differently, that is information. A reference who cannot place the project at all is more information than a warm paragraph about judgment.
By the third call the shared reading list has stopped mattering. What is left is a set of dates, one number the candidate could source, one person who argued the other way, and one call that went wrong. All of it said twice, once by the candidate and once by someone who watched it happen.
The essay everyone read this quarter will be superseded before the new VP's first quarter closes.
Common questions
What should you ask a VP candidate who will own AI policy?
Three past decisions, asked of every finalist in the same words. Which AI project they stopped and what had been spent when they stopped it. Which work they decided would not go through a model, and who argued the other side. What changed after the first deployment came in worse than the business case promised. Score each against a written scale before the panel compares notes. Then push twice on each answer: the number the call rested on, and what would have had to be different for it to go the other way.
How is this different from interviewing a manager whose team uses AI?
The evidence is a different size and lives in different places. A first-line manager's AI judgment is visible in how their team works week to week, so the questions go at review, handoff and what gets checked before it ships. An executive's judgment is budget, scope and written policy, so the questions go at what was funded, what was cancelled and what was refused. The executive record is older and larger, which makes it easier to check with references and harder to improvise in the room.
Is it fair to ask an executive candidate about a decision that went wrong?
Yes, and the answer is usually the most informative thing in the round. Ask for one AI call they got wrong, what the first signal was that they had misjudged it, and how long it took to act on that signal. A candidate who names no wrong call has left that part of the round without evidence, so ask for a smaller one. Keep the question identical for every finalist, since a failure question asked only of some candidates is a different interview given to different people.
Should the panel score AI answers on a scale?
Score them, and write the anchors before the first interview. The 2022 re-analysis of the personnel selection literature by Sackett and colleagues put structured interviews at the top of its ranking, with an estimated correlation with job performance of .42 against .19 for unstructured ones. Structured there means the same questions rated on a common scale. Those are pooled research estimates rather than a promise about one panel. The practical gain is narrower and still worth having: two people who scored the same answer separately can find out where they disagree, which a discussion cannot.
Does the EU AI Act change what to ask a VP about AI policy?
It changes what the job owns rather than what the interview should test. Article 4 has applied since 2 February 2025 and reaches deployers, meaning ordinary employers running bought AI tools in the EU, not only vendors. The European Commission states that amendments which entered into force in mid-July 2026 kept the obligation while removing any mandated level. Quoting the current text is not judgment, and being behind on it is not a disqualifier. Ask instead what the candidate would put in a written policy, and what they would refuse to allow in it.
What if the candidate has never stopped an AI project?
Ask what they scoped down, deferred or declined to fund, and treat those as the same class of evidence. Not every function has had a project big enough to cancel. What matters is whether the candidate has ever taken a position that cost them something with a peer or a budget holder, and whether they can say who was on the other side of it. An executive who reports only approvals has described an approval process rather than a judgment they exercised.
References
- 1. Revisiting Meta-Analytic Estimates of Validity in Personnel Selection: Addressing Systematic Overcorrection for Restriction of Range ✓ gwern.net Supports the claim that asking every finalist the same scored questions is the best-evidenced interview format: structured interviews estimate at .42 and unstructured at .19, with raw observed validities of .32 and .13, and Table 3 giving the .42 a standard deviation of .19 and a lower 80 percent credibility value of .18, which is the only bound the paper prints.
- 2. Belief in the unstructured interview: The persistence of an illusion ✓ sjdm.org Supports the claim that an unscored conversation dilutes better evidence: 76 undergraduates' GPA predictions correlated .31 after an unstructured interview, while prior cumulative GPA used by itself correlated .65 with the same outcome, and, ranking four prediction methods, 96 of 169 participants placed an interview with randomly generated answers above no interview.
- 3. Stubborn Reliance on Intuition and Subjectivity in Employee Selection ✓ edbatista.com Supports the claim that a human judgment added on top of mechanical predictors can lower accuracy, shown in an admissions study rather than a hiring study, which is the risk the body names for a principal's closing read: Sarbin's 1943 Minnesota study predicted achievement at .45 from high school rank plus an aptitude test and at .35 once counselors' intuitive judgment was added, and the review reports interviewer-level differences in validity as sampling error.
- 4. The Microstructure of AI Diffusion: Evidence from Firms, Business Functions, and Worker Tasks (CES 26-25) ✓ www2.census.gov Supports the claim that the executive's real decision is scope: 18 percent of firms used AI in a business function from November 2025 to January 2026, 32 percent employment-weighted, and 57 percent of adopting firms used it in three or fewer of fifteen functions, most often Sales and Marketing (52 percent), Strategy and Business Development (45 percent) and IT (41 percent).
- 5. Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act) ✓ publications.europa.eu Supports the claim that the AI literacy duty already reaches ordinary employers: Article 4 binds providers and deployers of AI systems, applies from 2 February 2025, and is not limited to high-risk systems.
- 6. AI Literacy - Questions & Answers ✓ digital-strategy.ec.europa.eu Supports the claim that the most-quoted Article 4 sentence is no longer the operative wording: the Commission states that the Digital Omnibus on AI amendments entered into force in mid-July 2026 and that AI literacy remains an obligation for providers and deployers while no specific level is mandated.
6 sources, numbered by first appearance. Every one was opened and checked against the claim it carries. How Olive sources claims
General guidance for hiring teams. What works at one company and one volume may not transfer to yours.
Olive assesses how a person works with AI. It does not detect AI-written documents, and it never produces a score, a ranking, or a match percentage for a person. Candidates read the same report the employer reads.