A Professor Saw the Drafts, Not the Deadline

What Did the Professor Actually Watch Happen?

A professor read the proposal, sent it back, read the revision, and watched the student defend a claim in front of people who could push back, which is more than the reference list suggests. The academic referee's limit is not thin observation but pressure: none of what they watched happened under a client deadline.

The missing manager reference is a smaller loss than it feels.

The validity figure usually quoted for reference checks is .26, and it comes from ten samples gathered before 1984 and pooled by Hunter and Hunter. The authors of the 2016 working paper that reprints the number say it may not at present be fully accurate, because many former employers in the United States now release only dates of employment and job titles ¹.

The 2022 re-analysis then dropped reference checks altogether. Sackett, Zhang, Berry and Lievens list eight procedures excluded for insufficient information, reference checks among them, so the paper carries the 1998 value of .26 forward with no revised estimate beside it ².

Both findings leave the evidence for the professional version thinner than its reputation, without making a reference call worthless.

But the coursework a professor graded says little about who wrote it. Researchers submitted entirely AI-written GPT-4 answers under 33 fake student accounts across five undergraduate psychology modules at a UK university. Of those submissions, 94% drew no concern of any kind from the markers ³.

The AI grades averaged about half a classification boundary above real students.

The submission leaves open who made it, which is the same gap behind what is left to screen on when every new-grad resume is AI-built. What a professor can add is what they watched being made.

Ask About the Draft, the Bad Source and the Unrehearsed Question

Three questions fit what a professor was in a position to observe. Ask what changed between the first draft and the final version, and who decided each change. Ask what the student did when a source turned out not to support the claim built on it. Ask how they handled a defense question nobody had prepared them for.

Each one asks for an event rather than an assessment.

An assessment carries the writer's habits into the record, and those habits have been measured against the candidate. In an experiment with 305 professors, Madera, Hebl, Dial, Martin and Valian found that letters of recommendation carrying hedging drew lower ratings of research competence ⁴.

In that academic-hiring experiment, a hedge cost the candidate whether the writer meant it to or not.

So ask for what happened. A referee's verdict on AI carries the referee's own view of the tool, which is the trouble when a former manager says the finalist leaned on ChatGPT for everything.

One more question depends on the course. Ask whether the assignment made the student hand in their working next to the answer.

Engineering students in a 2026 study were permitted to use ChatGPT on take-home open-book exams and required to submit their interaction transcripts. The researcher found some of the strongest evidence of reasoning where a student evaluated an incorrect or incomplete AI response, and concluded that a correct final answer alone may no longer be sufficient evidence of comprehension ⁵.

One qualitative study of 94 students across three courses is only a direction.

But a professor whose course asked for the working watched something a manager almost never sees.

Why the AI-Use Question Comes Back as a Syllabus

Because the professor answering it may be describing a policy they wrote themselves. At some universities the generative-AI rule belongs to the individual instructor. Vanderbilt's guidance states that the university empowers instructors to establish their own policies on the use of generative AI in the classroom, within the guidelines of the Honor Code ⁶. Ask whether a student used AI too much, and the answer reports where that instructor set the limit.

The range those instructors can choose from is wide. One example policy that Carnegie Mellon's Eberly Center publishes for syllabi states: "I specifically forbid the use of ChatGPT or any other generative artificial intelligence (AI) tools at all stages of the work process" ⁷.

Another on the same page allows the tools, provided any such use is acknowledged and cited under APA style ⁷.

A student who cleared the second policy would have failed the first, having done nothing differently.

A verdict on AI use grades the student against a line the student never drew. Two finalists from two departments are not comparable on that answer, and neither is comparable to a candidate whose AI use was governed by an employer's rules.

Nothing in a syllabus speaks to talent either. A reference confirming that a student followed the course rule is not evidence for or against the familiar assumption that the twenty-four-year-old is better with AI than the forty-five-year-old on the same shortlist.

Cut the Pressure Questions, Keep the One About Week Three

A client's deadline, a brief that moves halfway through, an ambiguous instruction from someone three levels up: a professor watched none of them. A confident answer about any one is the referee filling a gap in good faith, and it will sound every bit as specific as the answers worth having. Cut those questions before the call rather than discounting the answers after it.

The tell is that the answer arrives with no event attached to it.

Two more belong on the cut list. A professor did not watch the student be wrong in front of a paying client, and did not watch them choose between the correct answer and the cheap one. Those are workplace facts, and a reference list made only of professors does not hold them.

One pressure the setting did test is the deadline a student sets themselves.

A term runs fifteen weeks with no stand-up in it. A professor who watched one student start in week three and another start in week eleven saw something real about how work gets scheduled, so ask when this student started.

An oral defense is rehearsed, which is the best argument against everything above. A student who defends a thesis knows it is coming and can prepare for the questions a committee is likely to ask. A defense like that is not the unprepared moment a hiring manager pictures.

Give the Finalist a Task, Then Write the Setting Beside Every Line

One work sample reaches part of what a reference call cannot, so it earns the time a fourth professor's call would take. Give the finalist a task grounded in the actual role, under an hour, with the tool available on the same terms the job allows it. Ask them to say what they checked and what they refused to hand over.

Run the calls with structure whichever setting they come from. The US Office of Personnel Management's reference-checking guidance for federal hiring says that adding structure can enhance effectiveness, and that checks are typically collected by phone in a structured interview format ⁸.

The same page recommends a minimum of three contacts at about 20 minutes each, and reports that written requests for work histories typically produce low response rates and less useful information ⁸.

Three professors answering the same three questions is a reference stage worth having, for the same reason two structured reference calls beat five courtesy calls.

The rest of a new-grad file deserves the same reading. A capstone, an internship, a Kaggle placing and a certificate each come out of a setting with its own rules, and so do Forage-style virtual job simulations on a student's resume.

Then write the setting next to each line on the reference list. "Revised the argument twice after a source failed, in a fifteen-week seminar that allowed AI with citation" is a sentence a second reader can weigh six months from now. "Strong with AI" is not.

See a sample report >