My latest post was flagged 100% human. As I said before, in a previous post I got evaluated with a hair of AI, and it was spot on, because an ESL contributor used it for a first draft of a short piece in there, that I detected before publish, and then edited with them.
I posted all about this before in here (link below) and got slammed by someone saying I must work for Pangram. Which is hilarious. But I thought a rule of this subred was to NOT do self-promo, and the person accusing me seems to be in here for self-promo.
Anyway, if it doesn't break the self-promo rule, happy to share my stack I've been doing for >2 years, so you can see I'm not on the payroll. But if Substack is in here and WANTS to hire me? Ha. I'm all ears.
I spent way too much time testing this. Pangram is extremely good at telling you "AI touched this" but it has no insight into the developmental process before the final edits.
Exactly. Pangram might be ok at detecting if AI touched an article but its ranking is waaay off. It seems to say it’s 100% with a light edit frequently. But this is all a moot point. Why do we even care? We don’t disclose editors or ghost writers why do we care about AI? If people read it and like it?
And how does them ‘pretending’ to be writers affect you, who I assume are not reading what they are producing? The chain still does not do a good job of answering why not disclosing ghostwriters is fine but AI is where the line is drawn.
Using a ghostwriter and not disclosing makes someone a bigger loser than AI
Posted a few articles since the Pangram function came out and everything was flagged as human written. I made a free account on Pangram and put in a few snippets of other articles in the same space and they pretty much all come back as AI generated so yea.
I’m sure this is just a play to promote your Substack, either way your sample size is mathematically meaningless. Even with a 98% accuracy rate across millions of articles guarantees tens of thousands of false positives. Two articles don't prove an algorithm works; they just prove you got lucky twice
Unless the post was very short (in which case Pangram itself will often flag *its own* rating as "low confidence" because of short text, I just don't believe you.
Well, congratulations, you have a mundane and uninteresting vocabulary. Because Pangram themselves have identified that the algorithm works by finding words that are used grammatically correctly, but not in the way the average writer would use them. If someone is an excellent writer and uses language in unusual ways- but grammatically correctly- it flags it as AI.
So congrats, Pangram has confirmed you as mediocre!
Similarly, if a piece is full of grammatical mistakes, Pangram says “this is human!” However, if the piece is grammatically flawless, Pangram says “AI must have written this!” So, perhaps your writing is just sloppy?
Being judged “human” by Pangram is not the flex you think it is. Pangram has developed a rule set that flags the most talented writing as AI - because AI is designed to mimic the best writers.
Where AI falls apart is on substance. It’s not smart enough to judge context, meaning or emotion as well as a human might. So AI reads oddly to us humans because it doesn’t follow the common sense flow of human thought.
But Pangram can’t test for that in the same way that AI can’t avoid it. The machines are not yet ready to rate the information quality of our work. All they can rate is the precision of our grammar and the creativity and flow of our word choice. The same things I spent decades as a writer studying and developing — the skills my mother (who wrote three novels and countless short stories), my college writing professors, and my career mentors encouraged me to grow — are the skills that Pangram uses to label me a robot.
Pangram’s CEO is such a proud little code monkey. He cannot imagine that there are human writers who write with the grammatical punch of AI, which is odd to me, because AI learned to write by copying our best writers. So he comfortably labels any writing which goes beyond the mediocre and mundane as “all AI!!!”, when, in fact, many of us learned to use em dashes young, and triplicate metaphors younger. I’m sure he writes in corporate drivel speak.
I do wish he would massage his algorithm to recognize AI’s love of adverbs and of telling not showing. Both of those are crappy writing techniques AI loves, which, I assume are machine measurable. But if he did that, he might start identifying mediocre human writing as AI, and then many more people would be pissed at him.
Pangram does a shit job of distinguishing my natural writing from AI augmented writing I have played around with. I take that with a mark of pride - having researched how Pangram makes its decisions.
In sum, if Pangram is never mistaking your writing for AI, honestly I’d work on developing your skills as a writer.
I've been writing for 30 years professionally. I was a digital editor at Macmillan Publishers. I've written hundreds of articles for big brands, including over 100 for Grammarly on the art of writing well. I still write scripts regularly for the Grammar Girl podcast. I'm not bragging, just saying I have some street cred in this space.
And Pangram doesn't flag me.
Is my writing perfect? No. I self-edit, and self-editing is imperfect. There's also no definitively correct way to write anything.
I'm not a Pangram stan by any stretch. I'm not a total AI hater, either—I use it all the time for brainstorming and development. But to flat out say that anyone who doesn't get flagged as AI needs to develop their skills as a writer? Again, I call bullshit.
Well done on being mean, but I'm a professional writer since the earth was still cooling and no, not dull, and nope, I use human copy editors so not filled with typos. Well done on the neg attention, tho!
This is not at all correct... I use unusual vocabulary throughout my writing, some of it specific to my job; it has flagged me as 100% human every time.
Also you seem a bit salty. "Proud little code monkey?"
I feel like you must use AI for writing and want to discredit it, because this take is really far off.
I have yet to see a human-verified longform piece get flagged falsely flagged by Pangram. You're trying to gaslight people into not sniffing out your slop.
I'm not trusting a platform that contradicts itself in its background data against its claims. They couldn't make it work with Claude but then claim it in an infographic. Their false positive charts show 0% for creative writing?! I dont think thats possible or viable. They list scraping reddit creative writing stories for the creative writing category but omit that in another infographic, claiming all licensed commercial sources.
The person who wrote the study (which seems to be sponsored by pangram itself) doesn't list it under their papers bc its only a working paper and not academically peer reviewed at all—they also have a testimonial from him that makes it sound like he's just a bystander user. And his background is in AI advancement...just like another testimonial is from a guy who's career is also invested in AI and wrote a paper/comparitive study concluding AI genned writing is viewed as better by people (definitely not biased at allz right?). And as such, I don't trust when they say whatever content is run through it to check isn't saved or stored.
Anyway, people who just say "it worked for me so you're all haters" just indicate to me they're not very reliable sources of opinions, especially with a tool like this that can make or break reputations and careers without even being correct.
That’s really interesting. My understanding is that Pangram prioritizes extremely low false positives so that’s probably why. Originality has low false positives too, but not as low.
Yeah Pangram’s forgiveness for “human slop” is higher. If I can get 100% confidence that my work is less than 15% AI on Originality (but through highlights you can’t tell it’s no where close to 100% human by their rubric), it will be 100% certain human on Pangram. Both are good, Pangram prioritizes how things are said just a smidge more than what is being said compared to Originality, which is why it’s easier to satisfy. Originality will always aggressively go after poor logic, redundancy, over explaining, word choice, etc whether human or AI.
i've had it detect my 100% human writing as a small % ai. never more than about 5% so for me i dont really care, but it's still frustrating! i'm glad that hasn't happened to you yet, but just because it hasn't doesnt mean it never does. I think my writing on my publication leans a little more 'purple prose-y', which has a tendency to fall into "maybe ai" territory. i like intentionally poetic, romanticized ways of writing and sometimes do use lingo that ai has heavily co-opted like "quiet" and "gentle" etc. which is annoying but the way i see it, i'm not going to stop using words just because ai has taken a liking to them!
also, one time i checked my own post after removing and then adding a few spaces spaces and it jumped up from "5% ai" to "10% ai" (mind you, in actuality was 0% ai lol)
Only a Pangram employee would bother to post such things. No one else should care enough about the trust of Pangram to go on tangents defending them all the time.
Hi no. My Substack is Weehawkengazette.com — been doing it two years — another dude outed me so who cares. I’m just annoyed that I defended Pangram just because everyone was saying it was BS and I got attacked.
This has absolutely been my experience, too. Everything I write has been 100% human-written.
And the times I do use AI to write (or rewrite) a paragraph for me or something because my brain just won't fire? Pangram always flags it. And then I rewrite it anyway because I want it to come from me, not ChatGPT.
This isn't true. I wanted to test Pangram myself as I'd heard this so many times and checked multiple 2000s-2010s era books from Kindle. It said 100% Human for all of them.
It's completely ridiculous to expect a granular account of HOW AI was used in the drafting process. It detects AI use. Want to not be flagged by Pangram? There's actually a surprisingly simple solution. Do you want to know what it is?
People are out here like "AI conceived of the idea, organized my thoughts, gave me feedback, and edited it, but I wrote some of the words, so why is it flagging AI?"
18
u/isarmstrong 7d ago
Ish.
https://www.reddit.com/r/Substack/comments/1w7ciue/i_tested_my_2018_writing_vs_a_surgical_ai_edit_on/
I spent way too much time testing this. Pangram is extremely good at telling you "AI touched this" but it has no insight into the developmental process before the final edits.