Ask Any AI Detector Four Questions Before Trusting Its Score

Four questions, and a free web checker answers to them as a district license does. The first two are about the number itself: how many human documents produced the false-positive rate, and whether anyone outside the company measured it. The third asks what the tool reports below its minimum word count and on writing a student edited after the fact. The fourth lives in the documentation rather than on the sales page: what the score may not be used for.

Nearly every result on that search is a product page or a ranked list, and almost none of them tests the claim printed on it.

Two vendors publish a headline number on their own pages. Pangram Labs' model card gives a false positive rate of 0.0041 percent, roughly one flag in 24,000, measured on a million human-written English documents ¹. Turnitin's own blog gives a document-level rate under 1 percent for documents its model reads as 20 percent or more AI ². Both numbers are true about a test the company designed. Neither says anything about the stack of essays due Friday.

Which tool colleagues already run is a different question, and what teachers actually use answers it. Administrators ask a longer version, which is the questions to put to a detection vendor.

Set size decides what a rate can mean. Pangram's 0.0041 percent comes from 1,000,000 human-written English documents drawn from a web corpus, and its academic-writing column is separate: 0.019 percent on 62,971 papers ¹. Turnitin's figure carries a condition too. The rate under 1 percent holds only for documents its model already reads as 20 percent or more AI writing ², and Turnitin says it was validated on 800,000 academic writing samples from before ChatGPT ³.

Below 20 percent Turnitin now attributes no score and no highlights at all, showing an asterisk where the percentage would be ⁸; the 2023 report printed the number with an asterisk beside it and called it less reliable ³.

Highlights are what a teacher actually reads, and the rate behind them is a sentence-level one that Turnitin puts near 4 percent, meaning roughly one highlighted sentence in twenty-five is human-written ². Carry that rate onto a five-page paper with thirty highlighted sentences and about one is human-written; that is an assumption, since the rate was measured on Turnitin's own set, and no product page carries the arithmetic. A teacher reading highlights at four in the afternoon needs it more than the headline, because a wrong highlight is the same color as a right one.

Human states the set size beside its own figure, measured as of September 2026: 0 wrongly flagged by Human's default model of 600 human-written papers it had never been trained on; the ceiling on that is 0.498%. This is an estimate from our detector. A flag should be reason to look closer and not verified proof of cheating. That figure is Human's own in-house measurement on a held-out set, not an outside audit, and the ceiling is the more useful half of it, because a zero counted on any finite set is a bound rather than a promise.

A rate becomes real only once it meets a class roster, which is what the false-positive arithmetic works out.

Almost every current number is the vendor's own. Nature reported in August 2026 that only the accuracy rates firms announce from internal testing can be truly current, and that those figures are not externally verified ⁴. The two independent tests this article found are smaller by orders of magnitude. That gap is the trade a teacher is making.

Epoch AI reported in July 2026 that Pangram returned zero false positives on 495 human-written texts ⁴. An outside test in an education setting is smaller still, and it is the one that should slow a purchase down. A 2023 study in Patterns ran seven detectors over 91 TOEFL essays written by non-native English speakers and found they wrongly flagged more than half, an average false-positive rate of 61.3 percent ⁶. Every detector in that study agreed on flagging 19.8 percent of those human essays, and at least one flagged 97.8 percent of them ⁶. That set is 91 essays. The study says so.

Those results are from 2023 and the tools have moved since. The set size hasn't.

So a teacher is weighing a vendor test on a million web documents against an outside test on 91 student essays. Neither one is the classroom. Read both as bounds, and ask which generation of the tool was measured. A rate taken on general web text says nothing about a sophomore's lab report.

Each detector named here has a floor, and short classroom writing sits near it. Turnitin raised its minimum from 150 words to 300 in May 2023, giving accuracy on longer text as the reason ³. Pangram Labs states that Pangram 4 is intended for samples of at least 50 words written primarily in complete sentences ¹. Under 50 words, Human returns nothing at all.

A discussion post often sits below or barely above those floors, and so do a paragraph of reflection and many short-answer exams.

Edited text is the harder case. Nature reported that when Pangram used consumer AI to substantially modify human-written student essays, the model still labeled the results fully human 41 percent of the time ⁴. A tool that reads a wholly machine-written document correctly can still read a machine-edited one as a person's work, and a submission folder can hold that draft.

Human's own guidance runs the same way: a reading on 50 to 149 words is weak, because the detector has less to read.

Ask a vendor for the minimum word count, then ask what the tool reports just above it.

The limits live in the help pages, not on the product pages. Turnitin tells instructors to "use the information to initiate a conversation, not to draw a conclusion" ². GPTZero's own FAQ states that its results should not be used to punish students ⁵. Pangram Labs admits in its model card that the model carries a non-zero error rate ¹.

Three vendors, three sentences, and a search returns the product pages rather than these. One line of faculty guidance covers the same ground: AI detectors are not reliable enough to act as proof, the University of Pittsburgh's teaching center states ⁷.

GPTZero's published accuracy figures apply to its high confidence band, where it reports 99.1 percent of human articles classified as human and 98.4 percent of AI articles classified as AI ⁵. A band is a condition. A rate inside the band says nothing about the reading outside it, so ask a vendor for the rate outside the band and note whether an answer comes back at all.

Thresholds are the version of that question a teacher can act on, and Human publishes three, as of September 2026, beside a verdict of Human, Mixed or AI: accusation-safe flags a paper above 15% machine content, standard above 6% (the default), sensitive above 2%. No setting turns an AI verdict into a Human one; an AI verdict needs at least 80% of the document to read as machine-written. This is an estimate from our detector. A flag should be reason to look closer and not verified proof of cheating. Where a limit like that is written down, it can be checked before a purchase rather than discovered inside an integrity hearing.

Whether to run a detector at all is a separate decision, and whether teachers should use AI detectors is worth settling before a license is signed.