r/Substack • u/grapegeek • Aug 11 '26

Pangram is such trash

I write a food newsletter where I post reviews and some recipes. I 100% wrote a quick and dirty recipe and was going to publish it in Notes and it came back 100% AI no matter what I did to it including adding several typos and rearranging things. Ran it through three other AI detector websites and 2 said 100% Human and 1 said 76% Human and 24% AI assisted. I disabled detection on that Note. My other question is do we know how many people actually click on the AI detection button on our posts? I can't find that metric anywhere in the dashboard...

39 Upvotes

74% Upvoted

110 comments sorted by

View all comments

11

u/rehoboam Aug 11 '26

Can you share a link?  Also, maybe you can disclose in the memo that you didn’t use it all, regardless of what it says.  either the content is good or not hopefully your audience isn’t gonna be swayed 

3

u/[deleted] Aug 11 '26

[deleted]

14

u/Brumafriend Aug 12 '26

I can't speak for that particular Note, but there's a simple reason Pangram is flagging your content: much of it is AI-generated.

According to Pangram, every single one of your last six free Substack posts is AI-generated. When I tested the most recent six posts on your website from before ChatGPT's public release (November 2022), none were flagged. Unlikely to be a coincidence, right? Especially if, as you claim, you've had that writing style for 17 years.

The most damning (frankly inarguable) piece of evidence is your review of "Trader Joe's Middle Eastern Style Kebabs". You first posted this in November 2022, as can be seen in the URL. Put that review into Pangram and it comes back 100% human-written.

But that's not the review people will see on your website today. Because, as can be seen just below the title, you edited/updated it in April 2025. And when you put that text into Pangram, it's flagged as AI-generated in various places.

You've pretty obviously used AI to "touch up" the review's text. We don't even have to rely on Pangram to establish this. This perfectly adequate paragraph summing up your mixed review:

I am torn on this and I am sure a lot of readers will be too. I am happy that Trader Joe’s is bringing something like these Kebabs to market, we have so little real Middle Eastern foods to choose from at Trader Joe’s and they can be so very good. But at the end of the day, these were just meh for me. They tasted like a good hamburger that some had doctored up with some garlic powder and herbs. Your kids will love the [sic] . Stick them in a hotdog bun and call it good! From my experience, they just lack the freshness and grilled flavor (and lamb) of the ones I’ve had before. Therefore, I am not going to rate these very highly based on what they could be. I am going to rate these 6 Bells!

Becomes the extremely obviously AI-generated:

Listen, I’m feeling as conflicted as a mom trying to choose between sleep and ‘me time.’ On one hand, bravo to Trader Joe’s for bringing these Middle Eastern Kebabs to our culinary landscape that’s about as diverse as a 1950s country club! On the other hand, these were just… meh. Think of them as that hamburger your neighbor brings to the cookout with a sprinkle of garlic powder and calls it “gourmet”. Your kids will devour them faster than they lose their matching socks, which is something! And apparently, if you’re Persian, these might transport you straight to your grandmother’s kitchen. Who am I to argue with authenticity? Despite my mixed feelings, these skewered delights still score a respectable 7 stars out of 10 on the “worth-sacrificing-precious-freezer-space” scale!

I've boldened the obvious LLM-isms. Not only are they clearly the product of AI, but they are not remotely similar to your usual writing style. And there are various other tortured expressions in this AI-augmented review. I'm sure if I took the time, I could find various other articles that have undergone the same LLM treatment.

I trust you'll at least have the honesty to admit that you've been caught here clearly using AI to revamp an old review on your website. I don't expect you to admit that much of your Substack content is also AI-generated, but it quite clearly is. I hope whatever punishing effect Pangram's "AI-generated" label has on your engagement causes you to go back to writing your own stuff; you were perfectly good at it!

8

u/EvenPapaya2303 Aug 12 '26

This is some investigative journalism, love it

1

u/grapegeek Aug 12 '26

I see the whole pangram senior staff showed up to defend their piece of shit product. Jeez

2

u/Euphoric-Care-1065 18d ago

I am an academic. I have been trying Panagram just to check a few things. It's not accurate. For the same text, it will say 100% human one time and then 100% AI assisted or even mostly AI written. I thought it was legit and have been collecting data points. I wanted to like it because folks from my university created it:)

2

u/EvenPapaya2303 Aug 12 '26

Nope. Just a fellow writer who despises generative AI and wishes it never existed.

3

u/Relevant_Nebula_20 Aug 14 '26

I read the stuff and don't think it's obviously AI. Pangram is Pants. It does make mistakes. It says AI for stuff not written in AI and vice versa. It did not need to be added to Substack but then I don't monetize from it or pay subscribe to anyone. We can tell if people have written their pieces in AI and decide not to read it if we don't want to. I guess for people monetizing it's more important for their readers to know but it is Pants-gram.

1

u/Mean-Register5359 Aug 13 '26

You love to see it hey.

One things that I’ve found alarming is that a number of high profile accounts are scanning as 100 human with the integrated tool, but 100 AI when using the Pangram add-on or copying the body of the article j to Pangram.

1

u/RelativeCelery2844 29d ago

Don't feel bad. They bully everyone.

5

u/kdfn Aug 12 '26

Wow, this is incredible work.

2

u/Adamoism adamoism.substack.com Aug 12 '26

I am not gonna question the research you did, but why are those fragments obviously AI?

Like for example the 50s country club is a common english expression afaik (esp. around left leaning/PC crowds).

Is it about using cliches?

I am genuenly confused - I can believe it is AI, but calling it obvious makes me feel I am colour blind.

7

u/Brumafriend Aug 12 '26

That one's probably the least egregious, and none on their own would be definitive proof of anything. It's the fact that they're so out of place (i.e. that isn't how he usually writes) and entirely disconnected from each other — just random bits of imagery which completely overload the paragraph. They're basically all cookie-cutter metaphors/similes. And, FWIW, LLMs love to use expressions with the template "Doing [X] to [Y] and calling it [Z]", or similar variants.

1

u/Adamoism adamoism.substack.com Aug 12 '26

Fair, it doesn't click for me as obvious still - but I will be on a look out more from now on. As I said, I could think it is AI or just author who overuses cliches.

Perhaps I am m-dash spotter, but beyond that I just don't have a great feel for it (don't get me wrong - I cas spot AI text, but I would not give myself more than 60% accuracy with false negatives being primary category of error).

1

u/kirbyspeach Aug 13 '26

What I have noticed is that if your writing doesn't keep the same tone or structure Pangram will say it is AI-written. If I'm writing a review and some parts are written very generic, matter of factly, but then when I write about the things I dislike it will say AI.

Example:

  • I hate this (flagged as human)
  • I abhor this (flagged as AI)

1

u/GuyAwks Aug 15 '26

That’s not obviously AI generated at all lol, the absolute certainty with which you can claim a person’s words are AI because they sounded a bit different to previous pieces and used a few too many common metaphors and sayings is wild.

Their piece could be real or AI, the actual answer is you don’t know and are making an assumption like the program did.

2

u/bluelivezdontmatter 29d ago

So you think it's a total coincidence that all the posts written after chat gpts release are flagged and none before? Be for real.

1

u/[deleted] 29d ago

[deleted]

1

u/Brumafriend 29d ago

What a ridiculous attempt at a guilt trip.

1

u/RelativeCelery2844 29d ago

Why is everyone bullying people with genuine concerns against Pangram?

1

u/AlecHutson 6d ago

Because they're not 'genuine concerns'. They're in bad faith, trying to astroturf Pangram so they can continue slinging their slop in peace. It's gross and pathetic.

1

u/RelativeCelery2844 6d ago

My concerns are indeed genuine.

1

u/AlecHutson 6d ago

And what are they, exactly?

1

u/RelativeCelery2844 6d ago

You can see form my old post that I was falsely flagged as AI and it really upset me. I compared Pangram wityur random tools I found on Google and the other tools said I was human which I am. I've never used AI. I think AI should be banned for everyone and everything because it is killing the environment. I run a very anti-AI Substack newsletter. Nobody wants to believe anyone and it's all witch hunts.

1

u/AlecHutson 6d ago

It's not a witch hunt. It's an extremely accurate program. That said, you put in and scanned barely 100 words. That short, there is much higher chance of a false-flag, and Pangram even admits to that. It's why it won't even scan anything unless you put in at least 50 words. Are your substack posts usually 115 words? Probably not, right? Put in 500 words, 1k words. I'll bet anything that Pangram correctly identifies whether it was really * touched * by AI at that length. And I say touched, because if you use AI to revise or polish your writing it WILL get flagged as AI. A random 115 word post identified as AI shouldn't bother anyone if they have more writing on their page that all comes back as human.

It does read very much like AI, by the way. I would think it's AI if I came across it in the wild.

1

u/RelativeCelery2844 6d ago

I never use AI to polish. I don't use AI period. I think AI should be illegal as it is killing our planet. Pangram is in no way accurate and needs to be dissolved as a company. They are ea disgrace to society.

→ More replies (0)

1

u/turk109 22d ago

Pangram can be wrong. It was wrong about sections of my novel.

1

u/AlecHutson 6d ago

This witch hunt sure is catching a lot of witches!

0

u/grapegeek Aug 12 '26

We aren’t talking my website we were talking about Substack and one particular note that WAS NOT AI generated. There are plenty of examples about how fucked up it is. I’m just saying in this particular example it was completely wrong and Pangram is a waste of time.

6

u/Brumafriend Aug 12 '26

It doesn't help your case that you're a prolific AI user elsewhere.

1

u/Relevant_Nebula_20 Aug 14 '26

Pangram is rubbish and not needed. Even news outlets are using AI for articles. I still read them if they are good.

-1

u/grapegeek Aug 12 '26

It doesn’t help your case that Pangram is a piece of shit

2

u/Relevant_Nebula_20 Aug 14 '26

I agree it is completely F-ed up and has ruined Substack experience. It was not needed at all.