Pangram Reports Zero Flags on the 91 Essays of the 2023 Bias Study

Seven GPT detectors, and Pangram was not one of them. A Stanford group ran 91 TOEFL essays written by people through seven tools available in early 2023 and recorded an average false positive rate of 61.22 percent, against near-perfect accuracy on essays by US eighth graders ¹. Pangram's technical report on arXiv came out in February 2024, after that testing was finished ⁶.

Most of the tools it tested were built on GPT-2 and read low perplexity, roughly how predictable the next word is, as a machine signal; the authors' explanation, offered as a hypothesis, is that a writer working with a narrower range of English produces that pattern ¹. That hypothesis is the part worth carrying into a department meeting.

Then the authors tested that hypothesis. Rewriting the same essays with richer vocabulary dropped the average false positive rate from 61.22 percent to 11.77 percent, with the authorship unchanged ¹. Pushed the other way, simplifying the word choices in the eighth graders' essays lifted misclassification from 5.19 percent to 56.65 percent ¹.

The signal followed the vocabulary.

The study's own limits section says the sample sizes were relatively small ¹, which is the sentence most secondary coverage drops. Ninety-one essays is a pilot signal about seven specific products. The authors still concluded that detectors should stay out of evaluative and educational settings, above all when the writer is not a native English speaker ¹.

The student-facing version of this problem covers what a writer can do once the flag has already landed.

Four learner-English corpora and one benchmark row, all in-house. Pangram's ESL page, updated May 2026, puts 0.012 percent on 25,021 held-out learner essays under Pangram 3.3 from four sets: ELLIPSE, ICNALE, PELIC and the 91 Liang TOEFL essays ⁸. The Pangram 4 model card repeats the Liang row at 0.000 percent and puts academic writing at 0.019 percent across 62,971 documents ². A separate Pangram post reports zero false positives across the 25,996 PERSUADE 2.0 student essays ³.

Before any of those tables existed, the 2024 technical report made the claim in words: the classifier is not biased against nonnative English speakers and generalizes to domains and models it never saw in training ⁶.

Version matters here.

Pangram 4 is dated 29 July 2026 and replaced Pangram 3.3.2 ². A school that signed in 2024 or 2025 was not running the model these numbers describe, and a procurement file quoting them needs to say which version it means.

One figure on that card runs the other way.

Pangram's miss rate across 18 languages is 1.24 percent, against 0.3396 percent in English ². That is machine text getting through rather than a student getting accused, so it bears on what the tool catches in a multilingual classroom and says nothing about who it wrongly names.

One of them, and the set was small. Epoch AI tested three detectors in July 2026 on Pangram 3.3.2, the version the model card says Pangram 4 replaced, and recorded no Pangram false positives on 495 human-written passages ⁴. Nature's August 2026 survey of the field puts the general position plainly: only the accuracy rates the firms announce from internal testing can be truly current, and those are not externally verified ⁵.

Outside testing lags the products. Pangram 4 replaced its predecessor in July 2026 ², the one outside test cited here ran on the version it replaced ⁴, and the model card's headline 0.0041 percent false positive rate on a million FineWeb documents ² stays a company's own reading until somebody outside reproduces it on somebody else's data.

Reproduction is the missing step.

The same Epoch AI test found errors running the other direction, and in quantity: 25 of 99 machine-written scientific passages composed in imitation of a named author went undetected ⁴. A detector can clear human writing almost perfectly and still be wrong often in the direction that costs a student nothing.

The arithmetic behind a false positive rate converts a percentage into a number of wrongly flagged papers per term, which is the form a department chair can act on.

Short work first, edited work second. Pangram states that its detector is intended for prose of at least 50 words written primarily in complete sentences ², so anything under 50 words falls outside it. Pangram also says short conversational replies are outside its primary scope or more error-prone ², which is the shape of a discussion-board post, and it says the same of templated or automated writing ².

The second weakness is procedural. A student who machine-translates a first-language draft, or runs a paragraph through a grammar tool before submitting, has produced machine-edited text, the class Nature says the tools shed least light on because human and machine text intertwine ⁵.

Nature reported that when Pangram tested consumer AI substantially modifying student essays, the model still called 41 percent of the results fully human ⁵. The error leans toward clearing edited work, which is safe for the accused student, and no cited measurement says how the tool reads a lighter edit, a grammar pass or a translation.

Then there is the law, which does not wait for a model card.

A December 2023 brief from the Center for Democracy and Technology notes that English learners in US schools are protected from discrimination under Title VI on the basis of both race and national origin ⁷. The statute itself bars discrimination on the ground of race, color or national origin in any program receiving federal funds ⁹. CDT argues that an agency which knows those error rates and deploys the tool anyway arguably meets the requirements of a disparate-impact claim under Title VI ⁷. On that argument the question is a civil rights one before it is a technical one.

What belongs in the policy text itself is the administrator's half of this question.

Take twenty papers from the international cohort with known provenance, written before November 2022 or with a draft history the department already holds, and run them through the tool. Count the flags. That count is on writing the department knows to be the students' own, for the writers in the room, on the product the school bought, and it is a number the department measured itself.

Twenty papers is not a study.

It is enough to catch the failure that matters. A tool flagging three of the twenty is not usable on that cohort whatever any published rate says, and learning that in August costs an afternoon.

Known provenance is the load-bearing detail. Anything submitted since the cohort started using translation or grammar software is machine-edited text, and a flag on that says little about honesty.

Write the rule down before the first flag.

Settle which decisions a reading is allowed to touch while nothing is riding on the answer. The model card itself states that false accusations of AI usage can lead to reputational damage and emotional trauma, and that the model has a non-zero error rate ².

The twenty-paper test works the same way on Human, which reads college-level academic writing sentence by sentence, returns a document verdict of Human, Mixed or AI, and gives no reading at all below 50 words. The anonymous check takes up to 1,200 words, one paper a day, with no account; a department running twenty papers needs the school plan, which opens up to 50 papers a month to a school or university address. Human's detector scores the text in memory and de-identifies any personal information for additional training. Human's own held-out measurement, as of September 2026, records 0 wrongly flagged by Human's default model of 600 human-written papers it had never been trained on; the ceiling on that is 0.498%. This is an estimate from our detector. A flag should be reason to look closer and not verified proof of cheating. That is an in-house measurement on Human's own data rather than an independent audit, which puts it in the same column of the table above as every Pangram row.