Screening
A Manager's Complaint About ChatGPT Is Not Yet Evidence
A former manager's complaint that a finalist leaned on ChatGPT is not evidence about the finalist until one episode is attached to it. Ask when it mattered, what went out the door, what was wrong with it if anything, who caught it, and what happened next. An answer with an outcome belongs in the file. An answer that stays at the level of impression describes the manager's own norm and the last employer's tool policy.
The takeTwo people can hear that sentence and both be right, which is why it should never be scored as it stands. The strength of a reference's language says little about how much it proves: a flat, dull answer with a date in it beats a vivid verdict. The likelier reading is that a manager who says somebody leaned on ChatGPT for everything is reporting a working style that irritated them, and sometimes reporting a real failure they can name. The difference costs one follow-up question.
Where Olive fits
Open a role and see what the work shows
A reference can tell you what one manager remembers about how somebody worked, and cannot show you the work. Olive assesses the person directly: a 40-to-60-minute occupational assignment done with an AI assistant, returned as six findings with the timestamped excerpt behind each one, and the candidate is granted the same report the employer reads.
Assess your shortlistWhat Does Leaning on ChatGPT for Everything Measure?
Almost nothing on its own. A remark like that carries no task, no date and no outcome, so there is nothing in it to weigh. It also arrives through the least checkable instrument in the file. Reference checks were one of eight selection procedures the 2022 re-analysis of Schmidt and Hunter's validity table set aside for insufficient information, leaving the method with no revised estimate at all 1.
The .26 that circulates for reference checks was published in 1998, and the 2022 authors could not evaluate the sources behind it, which is why the method has no updated figure rather than a lower one 1.
Set aside is not the same as disproved.
One episode moves the remark: a date, a piece of work that went out the door, whatever was wrong with it, the person who caught it, and what the manager did next. Five answers, none of them about character.
The same gap opens from the other direction when a reference will only say they were great. A warm summary and a cold one carry the same quantity of evidence, which is none, and the same follow-up fixes both.
Turn the ChatGPT Complaint Into a Date and a Deliverable
Five questions turn the remark into something you can weigh: when it mattered, what went out the door, what was wrong with it if anything, who caught it, and what the manager did about it afterwards. Ask them in that order, and ask the same five of every reference on the requisition, including the ones who volunteer nothing but praise.
The questions are reaching for something somebody has already defined.
Deployment Diligence is the name for it in the AI Fluency Framework, written by Rick Dakan of Ringling College of Art and Design and Joseph Feller of University College Cork. The term covers taking responsibility for verifying and vouching for AI-assisted output, including fact-checking, testing for accuracy and validating claims 4.
Nothing in the framework turns an answer into a number, because it publishes no levels and no scoring anchors 4.
It supplies the subject matter instead. Not how much the candidate generated, but what they did before signing their name to it.
Asking a reference for one bounded episode with a date on it is the general form of this call, and repetition is what makes it work. A follow-up saved for the candidate somebody already doubts produces a file that records the doubt.
Push Twice for a Date, Then Stop Scoring the Candidate
Push twice for a date and an outcome. If the second push returns another version of the same impression, stop scoring the candidate on it and start reading it as a description of the room they worked in. What it describes then is a manager's norm about AI and a workplace's policy, which are facts about the last employer rather than about your finalist.
As of 3 May 2026, the Census Bureau's Business Trends and Outlook Survey put AI use among US businesses at 19.8 percent 5. The unit counted is the firm, and the question asks only about the past two weeks, so the figure says nothing about how heavily anyone used it.
The figure cannot tell you whether this finalist touched the tool. The manager's own complaint already asserts that.
What it can size is the range of rooms a complaint like this comes out of. The spread is steep, though the two figures are different cuts: 37 percent of firms with at least 250 employees reported using it, and about 14 percent of firms in retail trade 5. Adoption is not the same thing as a norm, but the norm a manager carries was formed inside one workplace, and those workplaces are not alike.
What the manager said is about to sit in your file as free-text commentary about a person, and that material is not neutral. One measurement of it comes from academic hiring letters; no employment reference was in the sample.
Madera and colleagues coded 624 authentic letters written for 174 candidates for eight assistant professor positions. Letters for women carried at least one doubt raiser 54 percent of the time, against 51 percent for men 2. Two or more appeared in 13 percent of letters for women and 7 percent of letters for men 2.
The gap in those headline percentages is small, and letters written by men and by women did not differ.
The companion experiment adds the price: with 305 professors as raters, letters containing negativity or hedging drew lower ratings of research competence, and not of teaching competence 2.
Reading a manager's disapproval as a fact about the candidate makes the same move as sourcing from companies known to be AI-forward on the theory those people have the skills. Both price an employer and bill a person.
The Former Manager's Idea of a Good Analyst Is Not Your Scorecard
Weigh whatever the reference gives you against the standard written for this role before the finalist applied, never against how the former manager would have done the job. A manager who thinks a good analyst opens the filing personally and a manager who thinks a good analyst reaches an answer in a third of the time are describing two different jobs. Only one of them is yours, and it sits on your scorecard.
Using the model settles quality in neither direction, and one exercise inside a 2023 field experiment shows why. Researchers randomly assigned 758 Boston Consulting Group consultants to work with no AI, with GPT-4, or with GPT-4 plus a prompt-engineering overview across a set of realistic tasks, and deliberately included one chosen to sit outside what the model could do.
Consultants working with the model were 19 percentage points less likely to reach the correct answer on that task 3. The control group got it right 84.5 percent of the time, against 60 and 70 percent in the two AI conditions 3.
On the 18 tasks chosen to sit inside the model's capability, the assisted consultants beat the control group on every measure 3.
Assistance making people worse is not the finding. One task, one sample, a model from mid-2023. People could not tell which side of the line the task was on, and that is the failure a reference call can genuinely help you find.
The distinction the complaint is groping at is what separates heavy AI use from good AI use, and heavy is the half a former manager can see from across an office.
But a manager who watched somebody every day for two years knows things an hour of assessment cannot reach. And pressing a reluctant reference for specifics is a good way to get nothing from a person who has been told to give dates and titles.
Write the Episode Into the File, Not the Verdict on ChatGPT
Write what the reference described, in the reference's own terms, into the same file the interview notes live in. A line saying the manager thought the candidate over-relied on ChatGPT records an impression. A line saying a client deck was drafted in March with an unchecked figure in it, caught by a second reader before it reached the client, records an event.
The call sits inside the same legal category as your interviews and tests, under a US federal regulation from 1978.
The wording is wide. Selection procedure, as the Uniform Guidelines use the term, covers any measure, combination of measures, or procedure used as a basis for any employment decision. The list runs from paper-and-pencil tests through what the regulation calls informal or casual interviews and unscored application forms 6.
A validation burden attaches only where adverse impact appears, so a reference call with no adverse impact carries no obligation to be validated. The definition settles one thing: a call that moves a decision is not outside the process. Take anything with legal weight to employment counsel; the text above is public regulation, not advice.
Keep the note to what you heard, and leave out why you think it happened. What the manager said. What happened to the work. Who caught it, and what changed afterwards.
The complaint is still in the file at the end, and it should be. By then it is attached to a date, an outcome and a name, or it is attached to nothing and marked as one manager's view of what a good week looks like.
Somebody who was not on the call can read either version back in a year.
The tone alone could not be read back at all.
Common questions
Can you reject a finalist over a reference's remark about AI use?
Only if the remark resolves into something you can name. A rejection resting on the sentence itself records that a former manager disliked how somebody worked, which is not a finding about the finalist and reads badly the first time anyone asks for the reason. Push for the episode: the date, the work, the error if there was one, who caught it, and what happened next. Reject on the episode if it misses the bar written for this role. Apply the same follow-up to every reference on the requisition, so the file shows one standard rather than one suspicion.
What if the manager will not give an example?
Take a repeated non-answer as an answer and move on. Some managers have been instructed to give dates and titles and nothing else, some are describing a preference rather than a problem, and neither case gives you anything to score. Ask one clarifying question about what the work was and what happened to it. If the second answer restates the first, close the topic and record that no example was offered. A reference who cannot name an outcome has described a workplace and a working style, and the file should say that much and no more.
Does using AI for everything mean the work is worse?
Not by itself, because the task decides. In a 2023 field experiment at Boston Consulting Group, one exercise was chosen to sit outside the model's capability, and consultants using GPT-4 were 19 percentage points less likely to get it right. The control group scored 84.5 percent against 60 and 70 percent in the two AI conditions. On the 18 tasks chosen to sit inside its capability, the assisted consultants beat the control group on every measure. One study, one sitting, a model from mid-2023. The usable lesson is that people could not tell which kind of task they had.
Should you tell the candidate what the reference said?
Put the substance to them without naming the source, and do it before the decision rather than after. Describe the situation neutrally: a former manager mentioned heavy use of AI tools on client work. Then ask what the candidate remembers about how that work was checked. A finalist who can name the same episode gives you a second account of one event, which is worth more than either account alone. Check the disclosure question with employment counsel where a reference asked for confidentiality; practice varies by jurisdiction and by what was promised on the call.
How many references should you ask about this?
All of them on the requisition, with the same questions in the same order. A follow-up used only where somebody already holds a doubt produces a record of the doubt rather than a record of the candidate, and the difference shows the first time anyone else reads the file. Two accounts of the same working period also reveal whether the complaint belongs to one manager or to the job. If one reference raises it and the others describe that period without it, the remark starts to look like a fact about one working relationship.
Is a reference check worth running at all for AI skill?
Run it for episodes, not for verdicts. The method's own evidence base is thin: reference checks were among eight procedures a 2022 re-analysis set aside for insufficient information, so the only meta-analytic validity figure for them is a 1998 number the 2022 authors could not evaluate. Read that as a reason to ask better questions, and keep the stage. A former manager supplies what nothing else in the loop can: a bounded event from a real working week, with an outcome attached and a second person who saw it.
References
- 1. Revisiting Meta-Analytic Estimates of Validity in Personnel Selection: Addressing Systematic Overcorrection for Restriction of Range ✓ static1.squarespace.com Supports the claim that reference checks were one of eight selection procedures excluded from the 2022 re-analysis for insufficient information, leaving the method with no revised validity estimate and only the 1998 figure of .26, whose sources the authors could not evaluate.
- 2. Raising Doubt in Letters of Recommendation for Academia: Gender Differences and Their Impact ✓ s3.wp.wsu.edu Supports the claim that free-text commentary carries systematic differences before any reader sees it: 624 letters for 174 candidates across eight assistant professor positions, 54 percent of letters for women with at least one doubt raiser against 51 percent for men, 13 percent with two or more against 7 percent, no difference by letter-writer gender, and the companion experiment with 305 professors in which negativity or hedging lowered ratings of research competence but not of teaching competence.
- 3. Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality (Working Paper 24-013) ✓ mitsloan.mit.edu Supports the claim that using the model settles quality in neither direction: 758 Boston Consulting Group consultants, 84.5 percent of the control group correct on the task chosen to sit outside AI capability against 60 and 70 percent in the two AI conditions, a 19 percentage point average drop, alongside the gains on the 18 tasks chosen to sit inside it.
- 4. Framework for AI Fluency ✓ ringling.libguides.com Supports the definition of Deployment Diligence as taking responsibility for verifying and vouching for AI-assisted outputs, including fact-checking, testing for accuracy and validating claims, and the point that the framework publishes no levels or scoring anchors.
- 5. Large Firms With at Least 20 Employees Biggest AI Users ✓ census.gov Supports the claim that firm-level AI adoption differed across employers: 19.8 percent AI use among US businesses as of 3 May 2026, 37 percent among firms with at least 250 employees (a size cut), and about 14 percent in retail trade (a sector cut), measured at the firm level over a two-week reference period.
- 6. 29 CFR Part 1607 - Uniform Guidelines on Employee Selection Procedures (1978), sections 1607.16(Q) and 1607.3(A) ✓ govinfo.gov Supports the claim that a reference call which moves a decision falls inside the definition of a selection procedure: any measure, combination of measures, or procedure used as a basis for any employment decision, a list that runs through informal or casual interviews and unscored application forms.
6 sources, numbered by first appearance. Every one was opened and checked against the claim it carries. How Olive sources claims
General guidance for hiring teams. What works at one company and one volume may not transfer to yours.
Olive assesses how a person works with AI. It does not detect AI-written documents, and it never produces a score, a ranking, or a match percentage for a person. Candidates read the same report the employer reads.