The 61% Bias Study Describes Seven Detectors, Not the Flagged Writer

Originality.AI, Quill.org, Sapling, OpenAI's classifier, Crossplag, GPTZero and ZeroGPT, run over 91 TOEFL essays written by people ¹. Turnitin wasn't one of them, and neither was any tool released after early 2023. On those seven the average false positive rate was 61.22 percent in the arXiv version of the paper, while on essays by US eighth graders the same tools were close to perfect ¹.

All seven flagged 18 of the 91 essays as machine-written, and 89 of the 91 were flagged by at least one of them ¹.

One of the seven is gone. OpenAI withdrew its own classifier on 20 July 2023 and gave low accuracy as the reason ². On its own challenge set that tool had caught 26 percent of machine-written English text, while labeling human writing as machine-written 9 percent of the time ². The same page had already said the classifier shouldn't be a primary decision-making instrument, only a complement to other ways of working out where a text came from ². That was the maker of ChatGPT, describing its own detector.

So the roster matters more than the average. A reply built on the line that detectors flag non-native writers 61 percent of the time invites one answer: the tool in this case wasn't one of those detectors. A reply that names the seven and asks which one produced this flag gets no such easy exit.

A question about the tool and the population. Never a verdict on one essay. The figure averages seven products over a single set of 91 essays, and the study's own discussion calls the sample sizes relatively small ¹. What it supports is narrower than it sounds: this class of tool misread this population in 2023. That's a different sentence from anything about the paper on the desk.

The mechanism is the part worth quoting. The authors' explanation, offered as one possible cause, is that detectors of that generation read low perplexity, roughly how predictable the next word is, as a machine signal, and that writing in a second language tends to have lower perplexity ¹.

Then they moved the rate in both directions. When they had ChatGPT rewrite the same TOEFL essays with richer word choices, keeping the students' ideas and structure, the average false positive rate dropped from 61.22 percent to 11.77 percent ¹. When ChatGPT simplified the word choices in the eighth graders' essays, misclassification rose from 5.19 percent to 56.65 percent ¹. The flag tracked vocabulary, and the authors note the paradox: a writer may need an AI tool to refine vocabulary to escape a false flag ¹.

That pair of numbers belongs in a written reply because it turns the accusation into a claim about prose style, which can be argued about, rather than a claim about honesty, which can't. The authors went further and strongly cautioned against using detectors in evaluative or educational settings, with the work of non-native English speakers as the sharpest case ¹. Once a committee accepts the first half of that, it has already accepted that the reading is about sentences and not about a person.

What that written reply should actually say is a separate question from what the study proves, and the two get mixed up often.

A timestamped record of the writing, which was the evidence in the one reported case below. William Quarterman, a senior at UC Davis, was referred for academic dishonesty in 2023 after his professor ran an exam answer through detection software that included GPTZero ³. The edit history of his Google document showed the work being built over two or three hours ⁴. He shared that history with the university, along with research on GPTZero's errors, and the university dropped the case ³.

The same family also ran well-known human writing through the tool the professor had used, which the report does not name; it flagged Martin Luther King's I Have A Dream speech and passages from the Book of Genesis ⁴.

Version history has one mechanical trap, and it's worth checking before any meeting. In Google Docs only people who can edit the file can see it, so a document shared as view-only hasn't shared its history ⁵. Give the instructor edit access. Failing that, make a copy of a dated earlier version or take dated screenshots of the history.

How a version history reads as proof covers what that record shows and where it misleads.

Work done in a second language leaves evidence of its own. An outline in the writer's first language, a marked-up printout, a reading list with dates on it, a tutoring appointment logged on a writing center calendar: each of those is dated, and dates are something a committee can check without trusting anybody. None of it has to prove authorship alone. Two dated records that agree, say a version history beside a calendar entry, are harder to arrange than either one alone.

Ask for the tool's name, then its version and threshold. Vendors publish different limits, and each of the three cited here publishes something useful. Turnitin puts its sentence-level false positive rate at around 4 percent, roughly one highlighted sentence in 25, and its document-level rate under 1 percent for papers marked at 20 percent AI writing or more ⁶.

Those are Turnitin's own figures about Turnitin, published on its blog in 2023 ⁶. They describe documents Turnitin scored at a setting Turnitin chose. That's why the version and threshold matter.

GPTZero's FAQ has a sentence worth quoting in full: "these results should not be used to punish students" ⁷.

Why bring a vendor page to a hearing, when accuracy is the thing in dispute? Because it's the seller's own written statement about what its number can carry, and a committee reading it is weighing the company rather than the accused.

At least one detector on sale now publishes against the 2023 benchmark directly, and that cuts both ways. Pangram's model card lists a 91-sample human-authored non-native English benchmark it attributes to Liang et al. 2023 at 0.000 percent ⁸. That figure comes from Pangram's own benchmark table, which makes it a public answer to the 2023 result without making it an outside audit. A 2023 measurement is evidence about 2023 software, in both directions. An honest reply names the year of the study and the year of the tool.

What else Pangram publishes about learner English is worth reading before quoting a 2023 study at a 2026 product.

Open with the timeline. A reply that leads with a 2023 paper reads as an argument about science, while one that leads with the hours the essay took reads like the person who wrote it. Put the dates in one paragraph and the artifacts in a short list below it. Then ask for the detector's name and setting, and attach the study at the end.

Keep the request narrow enough to answer in two lines. An instructor who can't name the model version isn't necessarily hiding anything. Note the gap anyway.

The formal route runs on a different clock. An appeal is a different document from a reply, with named grounds and a filing deadline, and it will run on whatever evidence got collected this week.

Dates travel between offices. Arguments about perplexity don't.

Attach the study's own caution, not its headline number. Its authors wrote that they "strongly caution against the use of GPT detectors in evaluative or educational settings, particularly when assessing the work of non-native English speakers" ¹.

Two pages, four dates, one quotation.