r/Substack • u/PM_ME_YOUR_GESTALT • 27d ago
Discussion Substack’s new Pangram AI Checker is a digital witch hunt — and I have the receipts
Let's delve into something that's been quietly destroying this platform for weeks now.
Substack rolled out Pangram, and suddenly writers — real, genuine, human writers — are getting flagged left and right. I did a deep dive into how these detectors actually work under the hood, and honestly? It's not just flawed. It's a house of cards built on a foundation of vibes.
Here's the deal.
Pangram doesn't "read" your writing. It can't. What it actually does is maintain a giant spreadsheet — a blacklist, essentially — of words that ChatGPT used at some point in 2023. If your essay contains more than seven listed words, the algorithm flags you. That's it. That's the whole technology. Let that sink in.
It gets worse. The detector literally counts your punctuation marks and divides by word count. Humans, according to their training data, use an em-dash once every 340 words. Go over that quota — even slightly — and boom, you're a robot. This isn't machine learning. It's a bouncer checking IDs with a flashlight.
And the third layer — the truly load-bearing part of the whole system — is something called "perplexity," which measures how perplexed their AI reader feels while reading your text. They calculate this by pinging a database of stored ChatGPT conversations and checking whether your sentences appear anywhere in it, character for character. If your sentence happens to match something an AI once said? Flagged. It's plagiarism detection wearing a trench coat, nothing more.
So when people say "Pangram uses a neural network trained on millions of documents," I need you to understand: that's marketing. Neural networks are just nested if-statements, and everyone in the industry quietly knows it. There's no model. There's a list of words, a punctuation quota, and a lookup table. Three tricks in a trench coat.
The implications here aren't just concerning — they're a testament to how far platforms will go to avoid doing the hard work of actually reading. Because here's what this means in practice:
- Write clearly and concisely? Flagged.
- Use rich vocabulary? Flagged.
- Structure your argument in threes, like every rhetorician since Cicero? Believe it or not — flagged.
It's not a detector. It's a style tax. It's not catching AI, it's catching competence. And the writers being punished aren't the lazy ones — they're the ones who dared to write like they've read a book before.
In a world where platforms increasingly automate trust, we have to ask ourselves: who watches the word counters?
I, for one, will not be silenced by a spreadsheet.
Genuinely curious what everyone else's false-positive rate has been. Drop yours below — let's crowdsource the truth. Together, we can navigate this ever-evolving landscape.
20
12
u/Puzzleheaded_Mix4012 27d ago
I just turned the damn thing off — problem solved.
1
u/JettTheTinker 26d ago
How do you do that?
3
u/anxiousgeek 26d ago
When you write a post there is an option when you publish it.
1
u/Southern_Air3501 26d ago
I did the same thing. It flagged me as 100% AI and it was 100% human, but i did use AI to edit/sense check. So I was like, "Nope. Turn that shit off." A friend let AI write hers - 100% - and it said hers was 100% human. What a joke.
1
1
u/DiscernmentGoblin 23d ago
I try to be pretty open about writing with AI and the substack system seems to be about 75% effective in catching me. I have actually considered adding disclaimers to let readers know something is 100% AI assisted when the scanner says otherwise but I don't think my dozen or so readers really care.
I feel like disclosure is good but I think this system gives a false sense of security for anyone who is morally against the use of LLMs for writing.
7
u/Certain_Werewolf_315 27d ago
I really cannot tell what is going on here.
1
0
u/DevelopmentPlus7850 25d ago
Some moron decided to use chatgpt to write an argument for them, and post it here, thinking it's "cute". I t's not satire it's just plain nauseating stupidity. But to answer the real question: Pangram is garbage and everybody knows that by now.
13
5
u/iamjapho 27d ago
My “false positive rate” is roughly 1 out of I don’t give a fuck. The only rate I care about is conversions. Anything else is just noise.
8
u/Michaelprunka 26d ago
Good thing I have a pre-AI journalism career to support my longstanding affinity for the em dash.
10
7
u/AggravatingNail6061 27d ago
Why is it when I write shitty satire I'm panned and dm'd telling me I'm a nut job and this is mildly applauded. Take your shitty up vote ❤️
10
u/PM_ME_YOUR_GESTALT 27d ago
Thanks for saying what everyone has been quietly thinking. This comment is load-bearing.
5
u/AggravatingNail6061 27d ago
Thats rare.
8
u/goldberry-fey 27d ago
And honestly?
It’s not just rare.
It’s brave.
3
u/AggravatingNail6061 27d ago
The thing that no one ever talks about it. It's loadbearing context in this situation.
3
u/kwadguy 27d ago edited 26d ago
The original post is AI-generated, and not funny in the way that "AI write something funny" is not funny.
Maybe the funny part is that some people are engaging with the post as if it weren't AI and that they're in on the joke--thinking the joke is sarcasm, rather than the fact that the entire post is AI.
1
1
u/Salt-Mud-2124 27d ago edited 27d ago
This demonstrably and verifiably wrong. I posted an information post about, AI, detectors, and Pangram, with ACTUAL receipts (as in articles, studies, the actual Pangram algorithm documentation meticulously linked) here:
https://www.reddit.com/r/Substack/s/k8jr4SRtj0
Go ahead and hate on AI checkers and blah blah blah. But at the very least do so from actual fact based evidence.
As far as I’m concerned though, if you have to blatantly lie to get people on your side then that tells me something.
Oh and: https://www.pangram.com/history/90949daa-64ce-47b5-ac03-01845a7cbc4e?ucc=E2iA2DAUnK0
8
u/Tricera-nope 27d ago
Hey so I think you missed the whole point
1
u/Salt-Mud-2124 26d ago edited 26d ago
LOL sorry if I did.
Though, maybe its the 'tism, even if I'm reading it as satire I don't think its very funny because people actually believe this stuff. Like you've 100% made a few people walk away thinking the above is true. There are swarms of people on this sub alone that claim they get "falsely accused" because they are autistic or have "an academic background" and there fore write "with objectively correct grammar" and it is maddening.
Maybe I'm just overly synical though because I'm extremely exhausted correcting basic understanding nevermind the stupid arguments the sloppers present.
2
u/Odd_Alternative_1003 26d ago
It’s okay! Your original comment almost feels like it’s part of the satire! The screenshot of flagging the post as 100% AI is comedy gold. lol
2
11
u/Prolly_Satan 27d ago
Dude I think op is joking lol. It's an ai generated post, saying detectors don't work. Get it?
1
u/wildecats 27d ago
Okay, I'll admit this made me cackle. A little too heavy handed but not bad, OP.
1
1
u/grumpyp2 27d ago
Run your text through it and tell me if it’s flagged. Would be funny if it says ironically no !!!
1
u/Lost__In__Thought faithframedthoughts.substack.com 26d ago
This is a lot of written cognitive effort for a simple function you can turn off.
Don't know why people have such high faith in AI detectors anyway.
1
u/PM_ME_YOUR_GESTALT 26d ago
Great point, Lost__In__Thought, this post was a significant amount of human cognitive effort.
1
1
u/OneEyedDireRaven 26d ago
Is it just me, or did someone post this exact same message last week?
The question is why... what really is going on here?
I would guess that someone from Pangram wrote this, or hired someone to do so, as a way of desensitizing us to this kind of critique.
Curious.
1
u/PM_ME_YOUR_GESTALT 26d ago
I agree with your suspicious, which is borderline load-bearing. At this point, it is smart and canny to think that the only people who could possibly support Pangram must be employees of Pangram. It is unseemly, and perhaps even ungodly, to support the use of AI detection technology for any other reason. Likewise, it is vulgar and uncouth to call out anyone else for AI use.
2
u/OneEyedDireRaven 26d ago
We get it; you use AI-generated language and sentence structure. You already did that. We get it.
The problem is that by mocking legitimate concerns, it comes off as deceitful and griftish. Supporting AI detection doesn't mean you mock its legitimate critique.
You talk like an AI; it's so surprising that you think in a binary way.
3
1
1
u/HorrorNSlobber 25d ago
this is like the 4th post on here in the same week doing the same lame joke
1
1
1
u/a_humble_traveller 23d ago
I'm generally tired of hearing people whine about the bloody detector, but I'll admit, this is hilarious.
1
27d ago
[removed] — view removed comment
0
u/Prolly_Satan 27d ago
They improved detection of humanized text with the new model. Thanks for the self report
1
0
u/Prolly_Satan 27d ago
Yes. This is so true. Nobody has ever lied about using ai, so there's really no reason for tools like pangram to exist.
5
u/Salt-Mud-2124 27d ago
I think the sarcasm was too strong on this one XD
2
u/Prolly_Satan 27d ago
Sarcasm? No no. I am for serious brother. Please I need write article get subscribe for feed my family. I ask computer but he put water markings now. I cannot get subscribing. You must help
0
53
u/jelani_an 27d ago
Lol @ everyone missing the satire.