AI Detector Accuracy Is Two Numbers, and the Vendor's Own Tests Set Both

Accuracy here means two separate measurements. One is the catch rate, the share of machine-written text a tool identifies. The other is the false-flag rate, the share of human writing it wrongly calls machine-made. A tool can raise either one by giving up some of the other. So a single percentage on a marketing page tells a teacher nothing to act on.

OpenAI published both numbers for its own classifier. On a challenge set of English texts the classifier caught 26% of AI-written text and labeled human writing as AI 9% of the time ¹. OpenAI withdrew the classifier on 20 July 2023, giving its low rate of accuracy as the reason ¹. Nine wrong flags in every hundred human-written texts: that's the second rate when a company actually prints it.

The two rates answer different questions, too. Who cares most about the catch rate? An administrator worried that machine writing is slipping through unnoticed. The false-flag rate belongs to the sophomore whose own paper lands in the wrong column, and to the teacher who has to open that conversation on a Tuesday afternoon with nothing but a number.

Mostly the vendor: the figures come from its own internal testing. Nature reported in August 2026 that only the accuracy rates the firms announce from internal testing can be current, and that those aren't externally verified ². The University of Pittsburgh's teaching center notes that claims of 98 to 99 percent are common ³.

Pittsburgh also names the gap: many of the software companies involved refuse to offer a specific false positive rate for their products at all ³. A claim with no denominator can't be checked.

Turnitin is one that does publish, and on two different units ⁴. Its document-level false positive rate is under 1%, and that figure holds only for documents its model reads as 20% or more AI writing ⁴. Its sentence-level rate is around 4%, which the same page spells out as a 4% likelihood that any specific highlighted sentence was written by a person ⁴.

Picture that on a returned paper. At Turnitin's published rate, each highlighted sentence carries about a 4% chance of being the student's own, more in a paper that mixes human and machine writing ⁴. That is a rate, not a count for any one paper. On a paper a student wrote alone, every highlighted sentence is a wrong one. The sub-1% claim belongs to the document score, a different measurement on a different unit. The reading itself is unpacked in what a Turnitin AI score means.

Turnitin also moved its own thresholds after launch, which matters before an old figure gets quoted in a hearing. In May 2023 it added an asterisk to AI scores under 20% to flag that the score is less reliable in that range, and raised the minimum length for a reading from 150 words to 300 ⁵. Since 8 July 2024 it goes further: no score and no highlights are attributed under 20% at all, and the asterisk stands where the number used to ¹¹.

Independent numbers exist, and they move with the text being scored. The sharpest is a 2023 Stanford study published in Patterns. Seven detectors were near-perfect on essays by United States eighth graders. On 91 TOEFL essays written by people, they misclassified more than half, an average false positive rate of 61.22% ⁶.

The same study showed the flag was tracking style rather than authorship: rewriting those human essays with richer vocabulary cut the average false positive rate from 61.22% to 11.77% ⁶. The writers didn't change. Carry that result into any argument about detectors and non-native English writers.

Bloomberg Businessweek ran a cleaner test on ordinary English prose. It took 500 college application essays submitted to Texas A&M University in the summer of 2022, shortly before ChatGPT was released. The essays came through a public records request, so they sat outside any training data. GPTZero and Copyleaks falsely flagged 1% to 2% of them, in some cases claiming near 100% certainty ⁷.

Epoch AI's July 2026 test ran three detectors on 495 human-written passages. Pangram and GPTZero flagged none of them; Originality.ai flagged 19, or 3.8% ¹⁰. The misses were another matter: Pangram missed 30 of 297 AI passages written in the style of a particular author, 10.1% ¹⁰. Nature's summary of the same test put the share that passed as human at some 8% ². Nature also reported a figure from Pangram's own testing, not an independent one: consumer AI used to substantially modify human-written student essays still read as fully human 41% of the time ².

Multiply the false-flag rate by the papers submitted, and a small percentage turns into a count of wrong flags. Vanderbilt did that arithmetic in public: 75,000 papers went to Turnitin in 2022, and at the 1% rate the vendor claimed, around 750 student papers could have been incorrectly labeled in a single year ⁸.

The multiplication works at classroom scale too. A teacher marking 120 essays a term, four sections of thirty, runs the same lottery 120 times. Assume all 120 were written by the students and that the vendor's under-1% benchmark rate carries over to this classroom unchanged; that is an assumption, not a measurement. The expected count is then up to about one wrongly flagged paper a term. At the sentence level the rate is about 4% per highlighted sentence, higher in documents that mix human and machine writing ⁴. Neither figure says which paper.

Vanderbilt disabled the detector rather than tuning it, and stated that it does not believe AI detection software is an effective tool that should be used ⁸. Its second reason belongs in any procurement meeting: Turnitin gave no detailed information as to how it determines whether a piece of writing is AI-generated ⁸. The term-by-term maths is worked through in the false positive rate arithmetic.

Turnitin's own guidance on the sentence-level rate points the same way. It tells instructors to use the information to initiate a conversation, not to draw a conclusion ⁴.

The number that matters is the rate for documents like the one on the desk, and that rate moves with what the document is. Length moves it. So does editing, and so does whether the writer learned English as a second language. A published benchmark average says very little about a 400-word reflection written by a sophomore in a second language at eleven at night.

Length is the cheapest thing to check before trusting a reading. OpenAI called its classifier very unreliable below 1,000 characters ¹. Turnitin raised its minimum to 300 words in May 2023 and said it might adjust the figure later ⁵. Pangram's model card says Pangram 4 is intended for samples of at least 50 words, written primarily in complete sentences ⁹. A short discussion post can sit under Turnitin's 300-word floor ⁵, and a reading near any tool's floor is weaker than it looks.

Editing can produce a wrong flag on work a student wrote. In its own test, with no student involved, Bloomberg Businessweek found that using Grammarly to improve an essay or make it sound academic turned work that passed as fully human-written into work that read as fully AI-written, while standard spelling and grammar fixes barely moved it ⁷.

Human, at human.olive.is, is tuned for college-level academic writing and publishes two figures from its own testing. Length still applies to it: a reading on 50 to 149 words is weak, and under 50 words Human gives no reading. Its in-house testing, measured September 2026, recorded 0 wrongly flagged by Human's default model of 600 human-written papers it had never been trained on; the ceiling on that is 0.498%. This is an estimate from our detector. A flag should be reason to look closer and not verified proof of cheating. The same testing recorded 0 misses in 481 wholly machine-written documents; the ceiling on that is 0.62%. Those are in-house measurements on held-out sets, not an independent audit.

Non-native writing is where the published spread is widest, and it's the case a teacher is most likely to meet without recognizing it. The Stanford average of 61.22% is the highest published figure cited here, and the rewriting experiment in the same paper points to vocabulary range as one driver ⁶. A reading on that paper earns a request for drafts and a few minutes talking through the argument.