AI detectors flag 9 to 61% of human writing as machine-written — vendor claims against measured rates

Written by

in

8 min read

Two commercial AI detectors were run over 642 journal abstracts this year. On untouched abstracts published between 2023 and 2025, Pangram marked 14.9% as AI-written and GPTZero marked 8.9%[1]. Those abstracts were written by researchers who submitted them to journals.

In the same study, abstracts lightly edited by an AI, the kind of help most universities and journals explicitly allow, were detected at 80.1% and 48.5%. AI-written text first passed through a commercial humanizer, a service that rewrites a passage until detectors stop flagging it, stayed flagged under 4% of the time[1].

Detector scores now decide grades, admissions and hiring outcomes. Published false positive rates run from under 1% in vendor material[3] to 61.3% in a peer-reviewed study of non-native English essays[2], a spread that comes mostly from measurement conditions.

Detectors score how predictable a passage is

Nothing is stamped into AI-written text. A language model picks each next word by probability, so its sentences tend to be built from whatever word was most likely to come next. Detectors score that property. The usual measure is perplexity, which captures how hard a passage is to predict, plus how much sentence length and structure vary.

In the 2023 Stanford study, the human essays that all seven detectors called AI had significantly lower perplexity[2]. In the 2026 Notre Dame study, what drove scores up was the density of long words and academic vocabulary[1]. Neither quantity tracks who typed the text. Second-language writers, formulaic abstracts and template-shaped application essays all score high on both, which is where the false positives concentrate.

Vendor numbers come with conditions attached

Turnitin publishes its own rates: under 1% false positives at the document level, and around 4% at the sentence level. The document figure applies to “documents with 20% or more AI writing”[3]. Sentences are the unit instructors actually see highlighted, so the 4% is the number that shows up on screen.

A single accuracy figure folds false positives and misses together, and it moves with the mix of the test set: a sample weighted toward AI text produces a higher number. Accuracy quoted without the sample, the threshold and the generating model behind it is not comparable across tools.

OpenAI pulled its own detector after six months

OpenAI shipped an AI text classifier in January 2023 and published its performance: it caught 26% of AI-written text on a challenge set of hard cases, and labelled human writing as AI 9% of the time. The same announcement now carries a line saying the tool was withdrawn on July 20, 2023 “due to its low rate of accuracy”[4].

Non-native essays were flagged 61.3% of the time

Stanford researchers ran seven widely used detectors over 91 essays from the TOEFL English proficiency exam, written by non-native English speakers, and 88 US eighth-grade essays. Every essay was human-written. The US essays were classified correctly almost every time. The TOEFL essays drew an average false positive rate of 61.3%. All seven detectors agreed on 19.8% of them, and at least one detector flagged 97.8%[2].

The researchers then asked ChatGPT to “enhance the word choices to sound more like that of a native speaker” and ran the improved essays again. False positives fell from 61.3% to 11.6%. Simplifying the American essays in the opposite direction pushed their false positives up. Changing the vocabulary alone was enough to flip the verdict on the same writers. The paper argues against deploying these tools in evaluative or educational settings.

Playing by the rules scored worse than hiding the AI

Three conditions scored by the same two detectors. Untouched abstracts: Pangram 14.9%, GPTZero 8.9%. Abstracts refined by AI within permitted limits: 80.1% and 48.5%. AI-written text run through a humanizer: under 4% for both
Three conditions scored by the same two detectors. Untouched abstracts: Pangram 14.9%, GPTZero 8.9%. Abstracts refined by AI within permitted limits: 80.1% and 48.5%. AI-written text run through a humanizer: under 4% for both

The Notre Dame team varied each abstract three ways. The “refine abstract only” condition stands in for permitted AI assistance, where the author writes the substance and a model tidies the sentences. Pangram flagged 80.1% of those, GPTZero 48.5%. Humanized AI text, the deliberate evasion case, stayed under 4%[1].

In the paper’s words, “Honest AI-editing results in a higher sanction risk than humanizer-assisted evasion,” and it concludes that “detector scores should not serve as standalone misconduct evidence”[1]. It is a preprint, first posted August 6, 2026 and revised October 13.

Share of human-written text classified as AI. Vendor figures: Turnitin under 1% at document level, about 4% at sentence level, OpenAI's own classifier 9%. Independent measurements: GPTZero 8.9% and Pangram 14.9% on 2023 to 2025 abstracts, and 61.3% averaged over seven detectors on 91 non-native TOEFL essays
Share of human-written text classified as AI. Vendor figures: Turnitin under 1% at document level, about 4% at sentence level, OpenAI’s own classifier 9%. Independent measurements: GPTZero 8.9% and Pangram 14.9% on 2023 to 2025 abstracts, and 61.3% averaged over seven detectors on 91 non-native TOEFL essays

Some universities stopped using the scores years ago

Vanderbilt University disabled Turnitin’s AI detector in August 2023 and published the arithmetic behind the decision. The university submitted 75,000 papers to Turnitin in 2022; a 1% false positive rate would have mislabelled roughly 750 of them. The announcement also cited the bias against non-native writers and the privacy problem of feeding student work to a third-party vendor[5].

At that volume a 1% error rate leaves 750 students to defend work they wrote themselves.

Korean universities are buying detection anyway

In a computer science course at Seoul National University last semester, 36 of 62 students were found to have used generative AI on an assignment. Students who came forward received a zero on the assignment; those identified in interviews received a D-. The instructor told the class that “every student who was penalised admitted it themselves,” asking what clearer basis for judgement exists than an admission. The student council objected that the standard for deciding AI use was unclear[6]. No detector appears in the reporting.

Commercial detection is growing anyway. Muhayu’s GPT Killer, the dominant tool at Korean universities, claims 98% accuracy, a company figure with no published method, and reported a 3.9x year-on-year jump in documents checked during October 2025[7].

If a detector flags your work

  • Keep process evidence that accumulates on its own: version history in Google Docs or Word, Git commits, timestamps in a notes app. Text pasted in at once leaves none of it.
  • Ask for the conditions. Which tool, which threshold, document level or sentence level, and what date it was run. The same text scores differently across tools and settings.
  • Disclose any AI use when you submit rather than afterwards. On the 2026 numbers, permitted light editing is the condition most likely to be flagged.
  • Expect higher scores when you write in a second language. English coursework, papers and applications fall into exactly the category the Stanford study measured, and machine-translated phrasing pushes vocabulary toward the statistical middle.

If you are the one running the detector

  • Treat a score as a reason to look, not as a finding. The Notre Dame authors say it cannot stand alone; Vanderbilt switched the feature off entirely.
  • Write the permitted range down before the work starts, in terms as specific as “spelling and grammar allowed, first drafts not allowed”. When no policy exists in advance, the detector’s output ends up standing in for one.
  • Assess work in ways that do not depend on a single artefact. Drafts, oral checks and in-class writing cost more staff time and remove the need to trust a score.
  • Do not escalate sentence-level highlights into a document-level verdict. At a 4% sentence false positive rate, long documents always light up somewhere.
  • If you screen job applications this way, tell applicants which tool and which threshold you use. Without that, a rejected applicant has no way to challenge the result.

What these numbers do not settle

No independent measurement of detector accuracy on Korean-language text turned up. Korean vendor figures are self-reported with undisclosed methods. The Stanford work reflects 2023 detectors and 2023 models; the Notre Dame work covers two detectors and academic abstracts only. Coursework, cover letters and blog posts may behave differently.

Measuring false positives is itself unreliable now. For anything written after 2023 there is no way to confirm the author used no AI at all, which is why the Notre Dame authors declined to call their 9 to 15% confirmed false positives. With AI authorship traces showing up in about a third of the post-ChatGPT web, assembling a clean human control set is the hard part.

The short version

  • Vendor-published false positive rates: under 1% per document and about 4% per sentence, conditions attached.
  • Measured rates: 9 to 15% on human-written abstracts in 2026, and 61.3% on non-native English essays in 2023.
  • Detectors score how predictable a passage is, which penalises plain vocabulary and formulaic structure.
  • Permitted AI editing was flagged 48 to 80% of the time; humanized AI text under 4%.
  • Researchers and universities have already moved away from scores as evidence. Process records and written rules are what is left.

Sources


Notes

  1. Jonathan A. Karr Jr et al., “Why AI Detection Fails for Academic Integrity”, arXiv:2608.11256v3 (posted August 6, 2026; revised October 13, 2026). Pangram 3.2 and GPTZero, 642 abstracts across chemistry, computer science, political science and theology, threshold 0.5. Originals flagged at 14.9% and 8.9%; “refine abstract only” at 80.1% and 48.5%; under 4% after humanization. arXiv ↩
  2. Weixin Liang et al., “GPT detectors are biased against non-native English writers”, Patterns 4(7), 2023. Seven detectors, 91 TOEFL essays and 88 US eighth-grade essays; 61.3% average false positive rate, 19.8% unanimous, 97.8% flagged by at least one, 11.6% after vocabulary enhancement. PMC10382961 ↩
  3. Turnitin blog: “Our document false positive rate … is less than 1% for documents with 20% or more AI writing” and “Our sentence-level false positive rate is around 4%”. Turnitin ↩
  4. OpenAI, “New AI classifier for indicating AI-written text” (January 31, 2023), with the withdrawal note dated July 20, 2023. Archived copy ↩
  5. Vanderbilt University Brightspace notice, August 16, 2023, including the 75,000 papers and roughly 750 mislabelled papers calculation. Vanderbilt University ↩
  6. Seoul National University computer science course, 36 of 62 students, zeros for self-reported use and D- grades for those identified in interviews, instructor notice and student council objection. Segye Ilbo (July 30, 2026) ↩
  7. Muhayu GPT Killer, 98% accuracy per the company, and documents checked in October 2025 at 3.9 times the year-earlier figure. Asiae (November 14, 2025) ↩

Source: https://arxiv.org/abs/2608.11256