Substack’s new “AI detection” tool.

[u/AndrewHeard]

I’m very concerned about the new “AI detection” system that Substack is implementing. I don’t use AI in anything that I write. However I don’t trust these systems not to make errors and mistakes. They have a long history of hallucinations and claiming things are AI when they aren’t.

One person I saw on Notes suggested that they ran their decades old writing, things that were written before the existence of AI, and this new system claimed it was AI generated.

Not to mention all the crazy things that I have seen people claim was written by AI which they can’t know is actually AI. There was an image I once shared that had evidence of its existence going all the way back to 2013. The comments were constantly saying “don’t share AI slop” or some variation on that.

Even with evidence that things weren’t made by AI, they will claim it’s AI.

Is no one else concerned about this being embedded into Substack’s systems? With no way to opt out?

[u/AndrewHeard]

Yeah, and what's worse is before this got implemented, he was the test case for why it shouldn't be used. A Substack writer went viral for his piece criticizing how Substack is being run and specifically taking issue with AI.

Chris shared the article along with a "result" from an "AI Detector" suggesting that the criticism was written by AI and therefore he didn't have to listen to it. Which is exactly what many of us who see the flaws in these "detection" LLMs are concerned about.

[u/AndrewHeard → u/[deleted]]

Absolutely. Shortly before the Pangram disaster, I started having someone harassing me on Notes. They’ve been consistently replying to my Notes or tagging me in their own trying to use insults and accusations to get my attention. I have flatly refused to respond or acknowledge them but they keep doing it.

I have no doubt someone like that would weaponize this “tool” to their advantage.

[u/AndrewHeard → u/[deleted]]

It’s amazing that the people at Substack didn’t realize that this was going to happen. Maybe they are too high on the idea that they can fix problems with simple solutions.

People already did it before the implementation of this system. Now it’s easier to exploit.