No AI Detector Is 100 Percent Accurate, and a Zero Count Is Not a Zero Rate

None of the technical documents cited here claims 100 percent accuracy for a whole detector. The closest is a zero count on one small set, and the vendor publishing it says in the same card that its error rate is not zero ⁵. Nature reported in August 2026 that detectors clear purely human writing almost every time, while noting that no tool can be perfect ¹. A procurement decision lives in the gap between the landing page and the model card.

The claim is common enough to be a research finding in its own right. The authors of the RAID benchmark open their 2024 paper by noting that many detectors advertise 99 percent accuracy or better ². Then they report what happened to the twelve tools they tested. Adversarial edits fooled them. So did changes in sampling, and so did generators the detectors had never seen ².

Faculty at the University of Pittsburgh get it in plainer words from the teaching center: claims of 98 to 99 percent and above are not unusual, and third-party evaluation finds that most of those tools have high false-positive rates ³.

Detection reads statistical traces in prose. Every reading is a probability about text rather than a fact about a person. Tune a detector to rarely flag human writing and it starts missing machine writing. Tune it the other way and human papers come back flagged. One knob, two costs. Perfection would need both at once, on writing nobody has seen yet.

Narrow the question and an answer appears. Whether detectors are reliable enough for a particular decision can be argued from published rates. A single accuracy percentage can be argued from nothing.

A count of zero errors bounds a rate only as tightly as the set it was counted on. The arithmetic is three decades old and fits on one line. When nothing goes wrong in n observations, the upper limit of the 95 percent confidence interval is about 3 divided by n, for n above 30 ⁴. Zero errors in 91 documents is a ceiling near 3.3 percent.

That 91 comes off a real page. Pangram's model card for Pangram 4 reports 0.000 percent false positives on the non-native English benchmark drawn from the 2023 Stanford study, and that benchmark holds 91 human-authored samples ⁵. The zero is a real count on a named set. What it supports is a rate under about 3 percent on writing like that, which is a weaker sentence than the one most readers carry away.

One company prints different figures on different pages. Pangram's page for teachers states 99.98 percent accuracy and a false-positive rate of 1 in 10,000 ⁶. The July 2026 technical report from the same company gives 0.0041 percent false positives and 0.3396 percent false negatives, at an AUROC of 0.9916 ⁷.

Neither page says whether the figures come from the same set. The technical report names its corpus ⁷ and the model card names its sets ⁵, and those are the figures a committee can check.

A published rate that carries its set reads like this one. Human, at human.olive.is, reports 0 wrongly flagged by Human's default model of 600 human-written papers it had never been trained on; the ceiling on that is 0.498%, measured September 2026. Both figures are Human's own in-house measurement on a held-out set rather than an independent audit, and the ceiling is the same upper bound arithmetic run on a much larger count. This is an estimate from our detector. A flag should be reason to look closer and not verified proof of cheating.

Three kinds of page carry accuracy claims, and they are worth different amounts. A technical report names a corpus and the date and method behind it. Model cards go one level down, into per-set numbers. Marketing pages name a percentage. A bid that cites the third and cannot produce the first has offered a slogan. Nature reports that a vendor's internal figures are the only current ones, and that nobody outside the company checks them ¹.

The last row is the common one.

Pittsburgh reports that many of the companies selling these tools decline to offer a specific false-positive rate for their products at all ³. A rubric that awards that blank partial credit has scored a sentence nobody wrote.

Read a vendor's model card beside its own marketing. Pangram's says that false accusations of AI use cause real harm and that the model has a non-zero error rate ⁵. That is a sentence a contract can be built on.

The ten questions worth putting to a detection vendor start with set size for the same reason.

A percentage becomes a number of papers once it meets a submission count. Vanderbilt worked its own case in 2023. The university had sent 75,000 papers to Turnitin in 2022, so a 1 percent false-positive rate could have wrongly labeled around 750 of them, and it switched the detector off ⁸. The arithmetic does not care which vendor is in the contract.

Run the ceiling through the same multiplication, not the headline. Suppose the 0.498% ceiling above transferred to a district's own writing, which is an assumption: it is Human's own in-house figure, measured on a held-out set of Human's rather than on that district's papers. On that assumption, of 40,000 human-written papers a year, the ceiling budgets for as many as 199 wrongly flagged papers. Against an unstated ceiling a committee budgets for nothing, because there is no figure to multiply.

Volume is half of it. The rate itself moves with who wrote the documents.

The 2023 study in Patterns ran seven detectors over 91 TOEFL essays by non-native English writers and found an average false-positive rate of 61.3 percent ⁹. Those same detectors were close to perfect on essays by US eighth graders ⁹.

A rate measured on one cohort says little about another, which is why a flag on a multilingual student's essay calls for a different reading from the same flag elsewhere.

Confidence and accuracy are separate readings. Bloomberg Businessweek tested two detectors on 500 college application essays submitted to Texas A&M in mid-2022, before ChatGPT was released. It found 1 to 2 percent falsely flagged, some with claimed certainty close to 100 percent ¹⁰.

A product can be wrong and sure at once.

An accuracy claim is worth what a contract can hold it to. Move each number out of the evaluation criteria and into the requirements, where a blank makes a bid non-responsive instead of costing it a point. A percentage becomes enforceable once four things sit beside it. Name the count of documents behind it and who wrote them, then the date of measurement and the threshold the product ships with.

One sentence does most of the work. Every accuracy figure in the response states the number of documents it was measured on and the 95 percent upper bound that follows from it ⁴. A vendor with a real corpus answers in a line. The vendor with a slogan has to produce a set or withdraw the figure, and either outcome beats scoring the slogan.

Two further terms cost nothing to add. The first dates each measurement, since Nature reports that only a vendor's own internal figures can be current, and quotes a researcher's view that outside evaluations will always lag the latest detector ¹.

The second names the threshold the score is reported against, since a false-positive rate quoted without the cut-off it was measured at states nothing a district can plan against.

The clauses that belong in a detector RFP cover the data terms and the student's route to the report as well. Drafting starts with the cheap one: require the set size beside every percentage; a 100 percent claim has to produce a set or withdraw.