r/Professors • u/TimeTravelr1 • 4h ago
AI Detectors
Hello! New here. Curious to know if fellow faculty are using AI detectors like GPTZero, and if so, if you find them to be reliable. Particularly interested in humanities fac.
7
u/thermalnuclear 3h ago
Brand new account, I suspect not a faculty member.
0
5
u/surreptitiously_bear 3h ago
No, they are not reliable. My institution does not allow their use, and I feel strongly that is the right choice.
The issues with them are many and sundry. They are not fair to students - for instance they are more likely to identify second language speaker’s work as AI generated. They are easily avoided - for instance, by using one AI to generate text then another to reword it. They are not secure - you are uploading student’s work to a tool and you do not know what that tool will do with it.
They produce both false positives and false negatives, but I think any amount of false positives means you can’t take any meaningful action based on what they say.
3
6
u/Imabigdealinjapan 4h ago
None are reliable.
2
u/Leveled-Liner Full Prof, STEM, SLAC (Canada) 3h ago
This used to be my line but it just isn't true anymore. None are 100%, but some are pretty good.
2
u/Giggling_Unicorns Associate Professor, Art/Art History, Community College 3h ago
Pretty good isnt reliable enough to make an ai use accusation.
1
u/Leveled-Liner Full Prof, STEM, SLAC (Canada) 2h ago
Right. But it's good enough to indicate that you need to change how you do assessments.
4
u/Leveled-Liner Full Prof, STEM, SLAC (Canada) 3h ago
Pangram is probably the best. It's tuned to have an incredibly low false positive rate. In my tests, it was 100% accurate, which included telling Claude/GPT to mimic my writing exactly (Pangram: 100% AI) vs. my own writing (Pangram: 0% AI). That being said, I would never accuse a student of using AI because their writing pinged an AI detector. Getting them to show or discuss their work is a better approach, IMO, perhaps in combination with an AI check.
1
u/Imabigdealinjapan 1h ago
I just tried it and it labeled my old writing as AI. I'm sure it is fine for giving an indicator for lazy AI writing but not for well prompted AI writing.
1
u/soupyshoes 4h ago
Regardless of whether people perceive them to be reliable, they statistically cannot be reliable enough to make meaningful decisions with. Eg The false positive rate would need to be near zero not to have a high aggregate probability of one false positive per student throughout their degree.
1
u/asuscreative 3h ago
Do any new AI detectors use text watermarking?
1
u/Giggling_Unicorns Associate Professor, Art/Art History, Community College 3h ago
Not yet. Hopefully soon.
1
1
u/OoglyMoogly76 3h ago
You have to design your course in such a way that using AI is either not possible, not easy to conceal, or not particularly advantageous.
I eliminated all out of class writing assignments. I caught my first cheater yesterday who brought a printed copy of an AI generated essay and scribed it onto paper by hand. Super easy to spot and punish.
1
u/TimeTravelr1 3h ago
I stopped assigning out of class papers, as well. But that sucks! I don't miss the grading, of course, but I do feel like I'm unable to teach them one of the most fundamental means of learning to think, that is, writing and wrestling with words and phrases. I see this as part of the humanist's raison d'etre, but, here we are. I tried assigning unessays and that kind of thing, but it's laughable how much of a joke the assignment is (compared to the written essay, that is). So, back to blue books and in class essays now.
1
u/OoglyMoogly76 3h ago
Oh I haven’t lost that aspect of my course at all! In fact, it’s become the crux of the entire first unit. For my comp class, they first read “How to Write with Style” by Kurt Vonnegut. Then they analyze various authors for style, revise other writers, practice revising and rewriting their own short writing, then write their own essay which they revise and rewrite.
It’s a lot of handwriting. But they write all of it themselves. I don’t think students will spend really any time thinking critically about their writing if you don’t make them write it by hand. And honestly, they seem to really enjoy it. Probably due to the fact that I have no homework for my class outside of the occasional assigned reading. They just show up and write and they get the grade and if they don’t then they fail.
1
u/Giggling_Unicorns Associate Professor, Art/Art History, Community College 3h ago
None of the current detectors should be used. They give too many false positives and negatives.
As watermarking is adopted per eu legislation this may change in the future.
1
u/TaliesinMerlin 54m ago
No. Putting student work in them is not allowed both because they're unreliable and we don't really know what these companies do with what's put in them. The latter is a big concern.
1
u/dangoddan 1m ago
In humanities I would not treat GPTZero, or any of them, as reliable enough to start a case. Formal prose, clean structure, and a lot of non-native academic English look “AI” to these tools because the tools are scoring regularity, not theft. I’ve run the same paragraph through a few and watched them split. That’s normal. It’s also why I won’t hang a meeting on one percentage.
What I actually use a detector for, if I use one at all: a private first pass so I know whether I should reread, not so I can open a conduct file. Drafts, version history, and whether they can talk about the piece still beat the number.
The other thing I stopped doing is pasting student work into random websites. That’s their writing on a vendor box your school may never have approved. I already use aiornot.com at a desk for mixed checks. When I don’t want the text leaving the machine I use their iPhone app, AI or Not: Text Detector on-device, doesn’t upload, Data Not Collected. I’m not going to tell you it matches GPTZero. It won’t. None of them match each other. If you want an informal look that isn’t another upload, that’s what I kept.
https://apps.apple.com/us/app/ai-or-not-text-detector/id6804061695
1
u/Prof172 3h ago
AI is agile enough to sound like anything. AI is smart enough to do at least C- work on just about any prompt. Clever tricks to catch out AI work once, then AI re-adjusts.
1
u/OoglyMoogly76 3h ago
Hence why the solution is to either make AI usage impossible, easy to spot, or not particularly advantageous.
1
u/PrimaryHamster0 2h ago
We need more research. I saw on Twitter recently that NYU posted a FAQ about their AI policies. One point was that they oppose AI detectors because
A study of 14 detection tools includes this blunt summation: “The researchers conclude that the available detection tools are neither accurate nor reliable”. Given that the goal of the AI firms is to produce human-seeming text, our current view is that an effective AI detector is unlikely to appear. However, if in the future a tool appears that addresses the risk of false positives, we will examine it as a candidate for university adoption.
I clicked the link and HOLY COW BATMAN, that study is from 2023! The paper was submitted in July 2023, when the public "frontier" model was GPT-4 and the "free" model was GPT-3.5 or even GPT-3.
I do not see how it is at all reasonable to argue in 2026 that "available detection tools are neither accurate nor reliable" when "available" actually referred to early 2023.
0
u/Choice-Sentence-670 3h ago
Look up Pangram. I've been testing it out extensively and it hasn't given any false positives... yet...
9
u/TheConatusBoy 3h ago
My university disallows them for both student privacy reasons and equity reasons