Ten Questions for an AI Detection Vendor, Starting With Set Size

Ask for the size of the human set behind the false-positive rate, in writing, before anything else. The accuracy questions that follow all lean on that one. Ten questions cover a detection contract, and each has a shape of answer that counts as an answer.

General AI procurement checklists cover data governance, bias and vendor viability. A detector needs questions of its own.

Each row becomes a clause. Detector requirements written into an RFP is the sibling page that does that translation.

A teacher choosing between two web checkers runs a smaller version of the same test, set out in four questions to ask any detector.

The number of documents is the answer, and a rate quoted without the set it came from is a sales figure. Published practice varies. Pangram Labs states 0.0041 percent on 1,000,000 human-written English documents, roughly one flag in 24,000 ¹. Turnitin stated in 2023 a document-level rate under 1 percent for documents its model reads as 20 percent or more AI writing ². A condition is not a set size.

Turnitin's sentence-level figure is a different measure again. It put that near 4 percent in 2023, and glossed it as a 4 percent likelihood that a given highlighted sentence might be human-written ².

Pangram's domain table shows what a real answer to question one looks like. Academic writing sits at 0.019 percent on 62,971 documents, measured separately from the web corpus ¹. Breaking a rate down by domain means the vendor measured it by domain. The alternative is one number covering lab reports, reflections, discussion posts and dissertations.

Then ask who ran the test. Nature reported in August 2026 that only the accuracy rates the firms announce from internal testing can be truly current, and that those figures are not externally verified ⁵. That sets the bar for question two: naming an outside evaluation puts a vendor somewhere different from one that can name none.

Some vendors publish nothing at all. Many of the software companies involved refuse to offer a specific false positive rate for their products at all, according to the University of Pittsburgh's teaching center ⁴. A refusal is itself a finding. Record it in the evaluation file.

A published answer to question one, in the shape a procurement file can score, reads like this. Human reports 0 wrongly flagged by Human's default model of 600 human-written papers it had never been trained on; the ceiling on that is 0.498%. Those are Human's own in-house measurements on a held-out set, as of September 2026, not an independent audit. This is an estimate from our detector. A flag should be reason to look closer and not verified proof of cheating. Human is tuned for college-level academic writing, and a rate measured on that set applied to a district's own papers is an assumption, not a measurement. The count and the ceiling travel together.

Working that rate against a real submission volume comes next, and what a false-positive rate does to a year of papers runs the arithmetic.

Ask for a subgroup figure with its own count. A 2023 Stanford study ran seven detectors over 91 TOEFL essays written by people and recorded an average false positive rate of 61.22 percent, while the same detectors were near-perfect on essays by US eighth graders ⁶. Any tool bought after that study needs an answer for multilingual cohorts.

The follow-up inside that study is the part a committee reads twice. Having ChatGPT rewrite the same essays with richer vocabulary cut the average false positive rate to 11.77 percent, and simplifying the word choices in the eighth-grade essays pushed misclassification from 5.19 percent to 56.65 percent ⁶. Word choice moved the verdict. Authorship never changed.

Current vendors report better, and both figures are real answers. Turnitin published a comparison for English language learners, 0.014 against 0.013 for native English writers, on documents meeting its 300-word requirement ³. Pangram reports no false positives on the same 91-sample non-native benchmark the 2023 study used ¹. Ask the vendor that publishes one subgroup for the next one.

Short writing and edited writing are the other two shapes a district submits, and each needs its own answer. Pangram states that its model is intended for samples of at least 50 words written primarily in complete sentences, and names source code, reference sections and templated writing as outside its scope ¹. Turnitin's floor is 300 words ³. A short discussion post can fall under both floors, so a license covering every assignment type can cover work the tool was not built to read.

Edited text is harder. Nature reported that when Pangram tested consumer AI used to "substantially modify" human-written student essays, the model still labeled 41 percent of the results fully human ⁵. A vendor with no measurement on edited text has not measured a case a school will meet.

Two of these answers shape policy more than procurement: the starting position that no detector is 100 percent accurate, and what a flag on an international cohort may be allowed to decide, worked out in detectors and non-native English writers.

Ask for both in writing, naming the role that holds each one. A threshold is the line at which a document gets flagged, and whether a district can set it affects how many papers reach a dean's desk in March. Report access decides whether a parent, or an accused student who is 18 or in postsecondary education, can see the evidence. Both belong in the detector's own RFP section.

Vendors handle the line differently, and their own documentation is where the difference shows. Pangram's document verdict is arithmetic on characters: Human at 90 percent or more human characters, AI at 80 percent or more AI characters, anything else Mixed ¹.

Turnitin's 2023 published rate applies at 20 percent or more AI writing ². That is a threshold statement in its own right, since below the line the vendor's own number stops describing the tool.

Pangram also states that the High, Medium and Low confidence label beside a segment measures the peakedness of a posterior and is not a calibrated probability estimate ¹. A label that reads like certainty is documented as not being certainty, so ask how the vendor explains it to instructors.

Human publishes where its flag line sits: accusation-safe flags a paper above 15% machine content, standard above 6% (the default), sensitive above 2%. Those are Human's own in-house measurements rather than an outside audit, as of September 2026. This is an estimate from our detector. A flag should be reason to look closer and not verified proof of cheating. The limit on the setting is published beside it. No setting turns an AI verdict into a Human one; an AI verdict needs at least 80% of the document to read as machine-written.

Report access is a records question before it is a product question. In the United States, FERPA (34 CFR Part 99) gives the right to inspect education records to a parent, or to the student once 18 or enrolled in postsecondary education. A school must comply with a request within a reasonable period of time and no later than 45 days ⁷. The US Department of Education's Privacy Technical Assistance Center, in guidance dated February 2014, states that a school or district must be able to provide a requesting parent or eligible student with access to records a provider holds ⁸.

A detection report kept in a student's disciplinary file is likely to count as an education record under the 34 CFR 99.3 definition: a record directly related to a student and maintained by the institution or by a party acting for it ⁹. District counsel settles that. The contract needs an export route and a turnaround shorter than the statutory ceiling, and the records side has a page of its own: FERPA and detector reports on student papers.

Close by asking what the documentation says the score may not be used for, and take the sentence verbatim. Turnitin tells instructors to "use the information to initiate a conversation, not to draw a conclusion" ². Pangram Labs acknowledges in its model card that false accusations of AI usage can lead to serious consequences and that its model has a non-zero error rate ¹. Pittsburgh's teaching center states the institutional version in one line: detectors are not reliable enough to act as proof ⁴.

Ask whether the writing trains the model, how long a copy is kept, and who can order it deleted. A submission is student work leaving the building, and in the United States it may be an education record under 34 CFR 99.3 ⁹. Pangram Labs states that Pangram 4 is not trained on user-submitted text or customer API data ¹.

A sentence of that kind belongs in the agreement rather than on a marketing page, and a marketing page can change without notice.

FERPA reaches the arrangement when a district discloses student work under the school official exception. The US Department of Education's February 2014 guidance describes that route as more likely to apply to online educational services ⁸. That guidance notes that FERPA regulations do not require a written agreement for disclosures under the exception, while districts wishing to outsource services will usually establish the direct control it demands through a contract ⁸.

The same guidance lists, among the conditions, that the provider has been determined to meet the criteria for a school official with a legitimate educational interest set out in the school's annual FERPA notification ⁸. Check that the notification's criteria are written widely enough to cover a contracted detection provider.

Retention is the question that gets skipped in the demo. Ask for a deletion term in days, a route for a district-wide deletion request, and a written answer on whether a deleted submission leaves the training pipeline as well as the database. Those can be three different systems, and one yes covers none of them reliably.

Score the answers instead of collecting them. Ten real answers look like this: a set size with an interval, a named evaluator, a subgroup figure for multilingual writers, a word floor, a measured figure on edited text, a scope list, a named role for the threshold, an export route, a written answer on training use and a deletion term in days, a quoted line on what the score may not decide. A vendor that returns those has answered in a form a hearing can read later. A vendor that sends a sales engineer to explain why the questions do not apply has answered one.