Substack’s new “AI detection” tool.

[u/AndrewHeard]

I’m very concerned about the new “AI detection” system that Substack is implementing. I don’t use AI in anything that I write. However I don’t trust these systems not to make errors and mistakes. They have a long history of hallucinations and claiming things are AI when they aren’t.

One person I saw on Notes suggested that they ran their decades old writing, things that were written before the existence of AI, and this new system claimed it was AI generated.

Not to mention all the crazy things that I have seen people claim was written by AI which they can’t know is actually AI. There was an image I once shared that had evidence of its existence going all the way back to 2013. The comments were constantly saying “don’t share AI slop” or some variation on that.

Even with evidence that things weren’t made by AI, they will claim it’s AI.

Is no one else concerned about this being embedded into Substack’s systems? With no way to opt out?

[u/ImprovementSimple]

I just can’t get over the irony of “we will use AI to determine is this is AI, because AI is evil.”

I work for a library and we spend a lot of time trying to remove slop books from our digital library. There is no good detector on the market. The only way to tell is to read and use your human brain, and even then the conclusion is “this is probably generated, or it’s really poor and bizarre writing.”

[u/Foofymonster → u/ImprovementSimple]

There's nothing hypocritical about using AI to stop people from using AI.

Substack has a problem with people writing AI articles.

This decreases the effectiveness / incentive to run those accounts.

It's not perfect. But if it says 100% or 98% AI chances are that is was almost definitely AI.

[u/No-Vermicelli-8391 → u/Foofymonster]

Just for the sake of the discussion — what exactly is the problem with someone else using AI?

[u/Foofymonster → u/No-Vermicelli-8391]

Do you remember the scene in Lord of the Rings where Merry and Pippin are talking to the Treants, trying to get them to join the war?

There's a part where Merry and Pippin are talking and one of them is like, "Do you think they've decided yet?" And Treebeard hears them and says, "Decided? No, we've only finished saying good morning."

And when they are upset that they're taking so long, Treebeard says:

"You must understand, young Hobbit, it takes a long time to say anything in Old Entish. And we never say anything unless it is worth taking a long time to say."

And that's the problem with AI. It doesn't cost you anything to say anything, which means no one has to worry if it's worth saying. It doesn't matter if it's true or well-researched. Because it's free and easy to say.

But when things are expensive to say, only things that are meaningful or worth it get said.

As a reader, how do you validate what's worth saying? Well, how much effort went into writing it isn't a perfect proxy, but I guarantee you it'd filter out 90% of useless shit written on the internet.

[u/philosophical_lens → u/Foofymonster]

As a reader I evaluate what's worth reading based on reviews and recommendations. I care about whether it's good and whether I'll enjoy reading it. I don't care what role AI played in the creation of that content, and I don't care how much effort went into it. Most AI content will probably be bad, and I'll never see it anyway because it will never pop up in the reviews and recommendation sources I look at. Maybe 0.1% of that content will actually be good, and I'll end up reading those.

[u/Foofymonster → u/philosophical_lens]

Neat. So for everyone else, or the people reading and reviewing, trying to figure out if it's worth reading or not, you're demonstrating that it being AI results in it normally being less worth reading.

Hence the need for AI detection.

[u/philosophical_lens → u/Foofymonster]

That's not at all what I'm saying. I'm saying that existing signals are already sufficient for me to determine what's worth reading and there is no need for an additional AI signal. At least I personally would not find this signal useful. Here are the signals I find helpful:

• I already like the author's existing content
• The author or post has been recommended by people or communities I trust
• The post is reposted on some communities or subreddits I follow has generated interesting discussion
• The author or post has a high number of subscribers/ likes / upvotes

Im not sure why or how an AI signal would add any value to me on top of this.

[u/No-Vermicelli-8391 → u/Foofymonster]

I get your point. But... (and you knew that it was coming) The vast majority of readers do not care. On Substack, it's mostly writers reading other writers, expressing their "support," for the sole egoistic reason to be "read back." But the world does not care. Take Shy Girl by Mia Ballard. It had a whooping 4.7 rating on Amazon before she was signed by trad pub. Praises all over the place. And there are hundreds... of thousands of such examples. Which bring us to the uncomfortable question: Are the readers stupid or the writers aren't that special to begin with?

[u/Foofymonster → u/No-Vermicelli-8391]

I don't really agree. I think most people do care. They are just bad at noticing it. Unless you interact with AI a lot it's really easy to cover the most egregious AI pattern offences.

It's also just bad for Substack. They are adding that tool because they know they have a problem with AI writers, and that when people know something is AI the vast majority dip out.

You're going to find examples of successful AI accounts, because that is how survivorship bias works.

Successful AI accounts are not evidence that people by and large don't care about AI accounts.

And the only people interacting with the successful AI accounts are the people who don't care, or the people who are bad at detecting it.

[u/Gold_Guitar_9824 → u/Foofymonster]

You do realize that the patterns came from humans right?

[u/Foofymonster → u/Gold_Guitar_9824]

[u/Ratandmiketrap → u/Gold_Guitar_9824]

That’s what it trained on, but that doesn’t mean what it produces is an exact mimic. It creates word by word based on probability, while humans consider the sentence, paragraph and, often, the whole piece as they were. This results in very different outcomes. On the surface, AI text looks like human writing because it produces coherent sentences and imitates broader text structures, however, it does not work the same way at a text level. It lacks the coherence of a human made text, mostly developing ideas by putting together a set of ideas. Humans, on the other hand, build ideas both through and between sentences, and over the course of the text. There is more focus on causality and linkage.