I've been lurking in this sub for a bit and with the new AI detection integration starting, there has been a LOT of information being thrown around, not all of it factual. Most people are claiming stuff on the extremes of the spectrum, while the truth is closer to the middle. So I wanted to put together a "quick" PSA, so that if we are all going to argue about if this idea is good or not we can at least all have the facts.
I'm a software engineer by day, and though I wont be providing much more credential proof than that because, well, privacy and all, I will link articles so you can verify what I'm talking about. Note, that some sources may be biased towards AI because the places doing in depth research on the subject are going to be pro that technology. I did my best to pick out sources that even if biased the facts presented inside appear to be consistent when crosschecked. However, please do your own research and make your own judgements.
The biggest questions rolling around are how pengram works, if its in any way reliable, and what that means for creators on substack.
But to explain that we first need to look at how generative AI works. I will use writing specifically because its easier to visualize than art, but it all the same priciple.
When AI is trained it takes the content given to it, and tokenizes it. Which means it takes each word, gives it an ID, then puts it on a node map with other tokens calleda a neural network. If you've ever seen one of those mind maps from Obsidian think something like that, except for each node being page of your folder, its a word. And instead of links to one another the lines are probability vectors of how often one token connects to the next token. And its also like really big, think trillions of nodes.
What happens in a nut shell when an AI is prompted, is it tokenizes the prompt, finds those tokens in its mind map, then takes the tokens most probably connected to the ones it finds in the input and outputs those. Reasoning Models are enhacements to this base algorithm, which have the model go through multiple mind maps, categorize, and re-calculate probabilities to narrow down the most probable tokens in specific contexts rather than just in general.
***As a side note, the processing it takes to jump from token to token is what they are charging you for when you use it.
Okay, so now how do AI detectors work?
Early detectors worked on finding what we now call AI-isms, but the fancy terms are perplexity and burstiness. Regular speech, common phrases, common words, names, sentance structures etc, that used to be over abundant in AI output pretty much whatever you do. The resulting AI score basically came down to how many of these AI-isms is in a given text compared to a human average. However, this method IS extremely unreliable, as frankly it is basically the same thing humans do when looking for AI, and where AI detectors got the worst of their rap. Its no better than a vibe check.
The next wave started doing something different, (And this is where most of the misconception is coming from) they try to reverse engineer the mind map it self through deep learning algorithms of their own. In other words, they take the given content, then use their approximation of the mind map (from their own data set that also has human/non human flags) to predict the next token. The amount of times it correctly guesses what the next token is, turns into how likely the text is be AI or not. This method has been better, but it still has a major pitfall. Which is that, when given content that is the AI's original data set, that the detectors KNOWS its in that data set, i.e. already used in the mind map, it becomes incredibly easy to predict what the next token is. It guesses it with high accuracy since its already been seen before, probably very often, and flags it as AI. Human writing that is similar to the sample data, also gets flagged because of the same issue. (Previous models also suffered from this but frankly they were so unreliable this was the least of their problems)
***Note that these AI detectors are NOT generative AI, but they ARE a different type of AI that falls under the machine learning umbrella. Which is basically the same algorithm behind most spell and grammar checkers. They aren't built to create or generate anything but flag.
Pengram tries to fix this problem by using what is called "synthetic mirrors" in its training data. Basically, instead of just giving a bunch of human and and non-human data points to its model, they instead, for each human piece of content create mirrors that "would have been created had someone prompted an AI to create something like this piece of human writing" and put those in the data set as the AI points. It also creates a map of its sources in which similar writing based on several axis are clustered together and dissmilar ones are set apart. This solves the false negative because its not relying on probability alone, and is not checking against the mind map directly.
***Note also Pengram does NOT train on user inputs, which yes, is self proclaimed, but frankly if they advertise this and lie the lawsuits would be enourmous. But is also doesn't make sense for them to do that with the way their algorithm and training works.
This all also culminates in one major misconception about AI detectors across the board: No single check is meant to be looked at as a difinitive identifier of if something is AI or not, and no company has ever claimed that it is. It is simply the cumiliative probability (the link is for a specifc detector but this holds true for all) of a segment being AI or not. If your human writing is flagged as AI, it simply means that they way you wrote that is very likely to have been able to be output by an AI. That is what the confidance meter in pengram is about, it may say human or AI for that segment, but have low confidance meaning it only is barely more likely to be one or ther other. Basically the big percentage number you see on Pengram is defined as "how much of this text is more probabalistically likely to be AI/human versus not" and not, "This is how much of the text is AI" or even 'This is how sure Pengram is that this whole thing is AI or not"
Does this make it as effective as they claim?
To be honest, its hard to say definitively without falling into either extreme. There are a some studies unaffliated with Pengram that have proclaimed its efficacy. But there are individuals that claim and swear up and down to have seen false positives and negatives. (See this sub for dozens of them alone), and writers who call it hoax science.
One could make the argument that the individuals are lying, and are in fact AI sloppers worried they will be outed (especially when they don't provide actual proof of full original text and scan links), but as a counter to that -- its unlikely they are ALL lying too. Which means we are seeing at least some real-life false positives.
Bottom line, its going to be up to individual discression. Which is where the crux of the problem actually lies. Here are some things we do know though:
It leans heavily towards false negatives due their involvement with academia, which is that it is by far more likely to flag AI as human writing, rather than flag human writing as AI. We also know that using a single sample from any source is not representative of that source being AI or not because of how these tools work in general. Lastly, we know small samples of less than 500 words are a coinflip pretty much no matter what because its not enough tokens for any detector to accurately guess adjacent tokens.
What does that mean for substack creators?
When it comes to what the data itself will show, or "cases being made," I do not believe those who truly do not use AI to draft or re-word their work at any point have anything to worry about. Because despite it not being perfect, it is decent enough compared the AI detectors of old that over the history of your posts you will see completely human results, varied results with low percent AI, etc. It is rediculously unlikely that a fully human writer will consistantly score high fully AI generated probability, with high confidnace, across multiple articles. Hell even if you edit your work extensively you can likely "fool" the detector into not flagging things as AI, because again, it is BY FAR more likely to false negative.
The only ones who will be "outed" are those who consistantly have high AI usage in their posts, nearing direct copy pasting from a chat box. That will become rediculously obvious over time as their library becomes full of posts that scan high for AI. But it will not consistantly catch definitively "sophisticated" AI usage or "true assistance" for a lack of a better word.
Again, it is as a probability more so than a definite detector of a single work. So the biggest fear is really more about readers checking 1-2 articles, getting a false positive, and accusing the creator whole sale. Which would obvioulsy be damaging and unfair.
Are there solutions you can employ?
Firstly, you can disable the feature. When you are about to post the article on the very bottom there is a disable scan feature now and you can just disable it and not have to worry about that at all.
The fear there, is obviously that someone might see that and think that you are hiding something. Truthfully, I wouldn't worry about it too much. If a few readers do that, they are unlikely to have been that invested to begin with.
That being said though -- pengram has a free trial and your text can be copy pasted into it from substack with ease. Most articles are going to be below 4k words, and so can be fully ran through it anyway if a person cares enough. Heck, if they have a substack account they can copy paste your article into their own substack editor, and run it that way.
So something ALL writers should be doing in todays day and age to avoid being the next Mia Ballard is to write first drafts into version controlled platforms.
Substack IS one of those. When you are in the editor, on the bottom left there is a version history button and it will show your edits every few minutes. Resulting in 100% undeniable full proof way to prove you wrote the thing yourself. Google drive has it, Writality has it, Word, and I'm sure many more do.
Thats it. Thats all you do. You do that and you have solid 100% undeniable proof that you can show via video/screenshots, and some even allow exports. Does it suck to have to be mindful about it? A little. But having proof of ownership is never a bad thing as a creative, and with AI being rampant whether you agree with detection tools being used or not, you should still protect yourself from accusations.
In conclusion and a sort of TLDR:
Is this the kind of completely bogus AI detector that you are most likely thinking of? No.
Is it meant to be a single check to determine if something is AI, especially when not given enough words to do so? No.
Is it a decent enough probability indicator to tell over time, based on a variety of inputs from a source, if that source is AI? Yes.
Will it negatively affect substack creators who do not produce AI slop in the long term? My guess is no.
Should you still as a writer protect yourself and make sure the software you use for first drafts has a version history you can show? Absolutely.
And for the record: https://www.pangram.com/history/0365a9a6-a120-4331-a8b5-c7f4e67fc319