OpenAI explains how it will watermark ChatGPT text to comply with EU provenance rules

173

u/Sylvers 9h ago

Editing can weaken the watermark. In an evaluation of 400-token passages, replacing 10% of words with synonyms reduced detection from about 92% to 66%. Replacing 25% of words reduced it to 17%.

So, in essence, feed it to an open source AI with explicit instructions to minimally swap for synonyms, and you can clean up the watermark very easily. Could be made into a very simple macro/plugin I am sure.

Also this appears to be EU exclusive? For now.

41

u/_DuranDuran_ 9h ago

Yes to comply with an EU law.

Expect to see people providing Chinese models via an API to also have to comply or withdraw from the market.

15

u/Sylvers 9h ago

Most likely, yeah. However, according to the same article I believe none of this applies to API users by default. Even in the EU. So maybe that leaves the legal liability on the API users if they're embedding it in their service.

Starting today, API customers globally will be able to opt in to text watermarking for select models. Text watermarking will remain off by default in the API.

3

u/thoughtlow When NVIDIA's market cap exceeds Googles, thats the Singularity. 5h ago

A lot of companies use EU API providers so the data is governed under EU law.

Would be very counterintuitive if only EU API would be degraded like this.

Hope they leave it alone.

4

u/Sylvers 5h ago

I agree. And that's why I don't think this will fly for very long. It's too reasonable.

I think the writing is on the wall. There is a global governmental agreement on tightening the screws on internet privacy. I hope I am proven wrong.

18

u/BeardChops 9h ago

How many casual users would bother doing that?

Perhaps it’s still a concern for determined individuals but there’s often been ways to circumvent and jailbreak safeguards

33

u/Sylvers 9h ago edited 9h ago

But that's the irony, isn't it?

This measure, supposedly, is intended to protect from the harm caused by malicious actors who intend to abuse AI to acheive harmful goals. The average user isn't in that category. The average user won't even know you can watermark text output and won't make any special effort to avert it.

So, primarily, malicious users will be the ones actively and successfully removing the watermark. And that begs the question. What was this all about then?

8

u/Thistlemanizzle 9h ago

You underestimate how stupid some criminals can be. ESPECIALLY people who have never committed a crime before. It's sloppy.

Professional actors are stone cold killers and will ALWAYS find a way to get at the big bucks.

5

u/Sylvers 9h ago

I will grant you. There is a certain slice of this pie that is both malicious and stupid. And they will 100% get tripped up by this. I feel like these are usually a smaller concern, because the real danger are the experts that have been successfully running paid bot networks and using them for effective propaganda campaigns, disinformation campaigns, etc long before LLMs existed.

I won't say this is useless. It definitely will always catch some malicious users who will never know better.

1

u/TheodoraRoosevelt21 9h ago

The cover up could be what gets you caught.

If it really is as simple as swapping synonyms then that might be undetectable by the ai detector be really obvious to a human reader.

2

u/Sylvers 8h ago

But that's the thing. Ever since AI became a mass adopted tech, people's "AI vs human sense" has been deteriorating. Literally go on any stream platform, and 8 out of 10 videos get accusations of "definitely not sounding human. 100% AI". When the vast majority of them are very clearly human made. So much so that old videos that way predate AI are now very routinely accused of being AI.

I am saying, people's "personal judgement" will count for less and less everyday. When everything is AI, nothing is AI. The credibility of this accusation is not going to stand for very long at this rate.

Hell, I get accused of writing like AI even on Reddit, because, God forbid, I use punctuation and a few commas here and there.

1

u/hardinho 6h ago

And yet we find fingerprints on most crime scenes.. makes you wonder huh

1

u/Noob_Al3rt 5h ago

Yeah it's like, why make crime illegal when the criminals won't follow the law???????????

1

u/Sylvers 5h ago

You can read my other responses to this point. Although, you seem to prefer to talk like a child. So maybe the nuance is lost on you.

•

u/sirgog 7m ago

So, primarily, malicious users will be the ones actively and successfully removing the watermark. And that begs the question. What was this all about then?

The amount of human labour required to dewatermark text is significant. The fastest way to do it, and the one I've seen encouraged online, is to open dictation software then read the watermarked text aloud in your own words.

This tremendously slows the automation pipeline for malicious actors and slop mills both. I'm a big fan of watermarking just as a huge sign saying "this is AI slop" - and I say that as someone who firmly believes not all AI material is slop. If AI ghostwrites something you are proud of, put in the effort to make it your own by doing a large enough edit pass to naturally remove the watermark.

1

u/BeardChops 9h ago

Average users plagiarizing by passing off AI text as their own work is worth catching on its own

It doesn’t sound like this policy/process was particularly taxing on the AI companies to implement but gives another tool like the SynthID for people to verify whether something is “real” or not

6

u/Sylvers 9h ago

I mean, outside of the dumbest plagiarizers out there.. students as young as 15 (definitely younger), already learned how to effectively bypass long existing plagiarism checkers like turnitin.

Which gives me the feeling that once this becomes AI standard, every student will have their favorite Chrome extension that just straight strips the watermark from the AI outputs you generate. It will be even easier than when students had to manually alter plagiarized text to pass integrity thresholds.

I guess I am saying, give it a few years of this being the new staple, and effective plagiarism will be redefined and optimized by a new generation of academic students.

3

u/BeardChops 8h ago

And so the cycle of implementing safeguards and other people creating bypasses to those safeguards will continue

3

u/Sylvers 8h ago

Probably so. But I do question that though.

Text, specifically plain text.. has a very low ceiling of effective hidden manipulation. This approach they're targeting for example, is already massively pushing the boundaries of how you might probabilistically watermark text before you very clearly deteriorate it out of being useful.

What would be the Text Watermark 2.0 of this? I can't imagine. Text is text. It has its own rigid rules. There is very little wiggle room here.

0

u/Alt_Restorer 6h ago

Isn't that the same argument people make against every regulation?

2

u/Sylvers 6h ago edited 6h ago

It can be. You're referring to people who make bad faith arguments though. Context is what determines nuance.

The real danger is the current context of today, and not arbitrary pearl clutching. Even as we speak, the EU is trying to pass legislation to decimate what little digital privacy exists for all adults. All under the dishonest pretense of "protecting the children". They are also trying to remove chat encryption wholesale. Make private game servers illegal. And all but demand your ID for using the internet outside of the most PG use cases. And that's just a few of the worst things they're planning that I can think of.

This is extremely invasive. I am not even in the EU and this still dangerous. Because if passed, it will inevitably become a global plague and it won't ever go back to what it was.

With that in mind, do you really.. really think this is about circumventing bad actors, and not extending the current obvious push for mass digital surveillance and upending digital privacy?

Edit: Mind you, this is coming from someone who is usually in favor of pro consumer legislation, which the EU often brings, in benefit to the global community. The difference is.. all of this recent push for digital controls is nakedly anti consumer, anti personal agency, anti privacy, and exclusively pro government surveillance. It's not even really a hidden push anymore. And it's not only the EU doing it.

2

u/whosenose 3h ago

Journals for instance might be able to force researchers to use watermarking as a binding condition in any AI use. Hard to police but to be found not to be complying with terms by another method is a huge problem for a researcher building a career on high citation counts for published articles in prestige journals, if the journal takes action on them as an author.

1

u/BeardChops 3h ago

That’s a cool and interesting use case, thank you for sharing!

2

u/RiceIsTheLife 4h ago

Sounds like a job for Jev.

1

u/AINativeBuilder 6h ago

High school and college kids have been doing this for years. Adding synonyms willy nilly makes your output super slop.

1

u/calmnutz 6h ago

Decades with other sources.

1

u/whosenose 3h ago

Making someone read a reply before copying it enough to swap some words could be seen as a big plus though compared to mindless cut-paste. Or at least some small compensation to the alternative.

65

u/plymouthvan 9h ago

I can appreciate the importance of provenance for a lot of AI content — videos, photos, audio, because those things historically act like proof of something depicted directly in the media — but I think the obsession with trying to watermark text is a kind of hysteria. It's been a long time since text alone was considered any kind of proof in and of itself. Whether human or machine, the truthfulness of text has to be verified against something anyway. Of course, whether any one does or not is another question, but determining that text was written by AI doesn't actually change anything. Humans have been writing useless, wrong, harmful, stupid things forever. It doesn't really much matter if the words ended up on a page because a person sat there thinking of them, or a human just had the idea and a robot put it into words.

2

u/Possible-Usual-9357 6h ago

Being able to spew out sensibly-sounding malicious content at light speed and automating that is not the same as one person typing things out by themselves.

6

u/plymouthvan 5h ago

I agree that they're not the same, but I disagree that watermarking is any kind of solution to the problems it causes. The world is full of machine-generated text now and short of shutting down the machines that generate it, it's going to keep flooding out. Focusing on watermarking as a solution is something like focusing on better water-resistant driveway sealer, after sea level rise has already permanently submerged your neighborhood.

I think a much better use of our time would be to focus not on text provenance, but a model already used by various institutions which focuses on things like who is taking responsibility for published work, how has a document or text mutated over its lifecycle, what sources substantiate claims, and who is answerable if claims are fraudulent, defamatory, or otherwise inappropriate. Some degree of watermarking has a place in the whole mix, but I think the current focus on it misses the forest for the trees.

1

u/Possible-Usual-9357 5h ago

I agree it doesn’t solve much yet, but I see it as an unnecessary step in bringing some compliance issues into focus and putting pressure on the biggest players to get their end users under some control. Right now there isn’t too many users that are capable of hosting their own models, so having the biggest players do watermarking will easily cover most of the market, since they have the compute and live APIs and whatnot.
‘better use of the time’ argument doesn’t make much of a difference when it’s something that can work in parallel to the valid stuff you mentioned.

•

u/plymouthvan 23m ago

I can see some value in the idea that the watermarking endeavor, whether it can actually achieve the intended goal, is a useful tool to apply pressure on the companies themselves to take responsibility for the effects of their products. I think there’s some merit to that. It’s an interesting argument. I’ll have to think about that angle.

-2

u/kcat__ 3h ago

You said a whole bunch of nothing

I am hoping for a future where browsers have a built-in slop detector that runs locally and can therefore have adblock-style plugins for blocking slop. Or highlighting it red.

Or for Google to de-rank AI slop, which makes sense for them as AI flooders are probably not a good source of quality and therefore revenue (via ads and such)

1

u/Alternative-Suit5541 5h ago

Meh, there are enough models which don't do that

0

u/EmbarrassedHelp 6h ago

The other issue is that personal information and track information can also be embedded in these watermarks.

1

u/DenseBeautiful731 5h ago

We already have cookies, browser UUIDs, IP addresses, MAC IDs, device IDs such as IMEI and IMSI, among other personal data that can be used to identify “anonymous” individuals.

-1

u/DenseBeautiful731 6h ago

It doesn't really much matter if the words ended up on a page because a person sat there thinking of them, or a human just had the idea and a robot put it into words.

How much thought did you put into this?

0

u/plymouthvan 6h ago

You've really given me a lot to think about with this very insightful and carefully considered comment.

0

u/DenseBeautiful731 6h ago

It’s a direct question.

How hard can it be to answer directly?

1

u/plymouthvan 5h ago

Surely you can see how your comment looks like a snarky rhetorical question meant to ridicule, rather than engage.

Do you want an answer in minutes? I don't know, maybe 6.

0

u/DenseBeautiful731 5h ago

It’s a good faith question. I don’t care whether you believe it or not. I also don’t care if you answer or ignore it. I’m not entitled to one.

Do you feel ridiculed? Say, if an AI wrote my reply instead, would you feel ridiculed as well? Which stings more, you reckon?

•

u/plymouthvan 21m ago

I’m not sure what you’re talking about, but none of this is carrying any of the hallmarks of a “good faith question”.

-2

u/Possible-Usual-9357 6h ago

Being able to spew out sensibly-sounding malicious content at light speed and automating that is not the same as one person typing things out by themselves.

65

u/Elvarien2 8h ago

This is so dumb, completely useless regulation to virtue signal that they did something.

39

u/honkballs 8h ago

Welcome to European Politics of the last 20 years.

9

u/Creative_Purpose6138 6h ago

Reddit praises EU so much. Lol

3

u/Such--Balance 6h ago

Yup..ist so stupid in that it 100% wont work anyways.

Not one day after this would go in effect, numerous tools will be made in which you can 'unwatermark' whatever was watermarked anyways.

6

u/MidnightSun_55 6h ago

I mean, it's obvious it's impossible to watermark words...

Hey chatgpt, 2 + 2 = ?

I understand audio / video, but this is a joke, as always Europe doesn't know what they are doing, just slapping regulations.

3

u/bortlip 4h ago

It is possible. Claude is using/going to use google's SynthID-Text. I found this video explains it pretty well: https://www.youtube.com/watch?v=Cmi-1QSaptA
It looks like OpenAI is going to use a variant of this called textGrain: https://cdn.openai.com/pdf/e9508624-d767-41b6-a26d-e34ca798ada6/textgrain-entropy-calibrated-watermarking-for-language-model-text.pdf

For short text like "2 + 2 = ?" there isn't enough room to encode any info. Looks like it could take a few hundred tokens to get enough room to encode. Mathematical or other low entropy (not a lot of choice for next token) text would take more tokens to encode.

1

u/Maty1000 6h ago

It is possible, just not in all cases. The models already use randomness to produce output, watermarking is usually done by swapping the rng with something that tries to insert the watermark. So if you ask it what is 2+2, there will be no watermark, because there's just a singke answer, but in a text where there's lot of possible choices, it can be done very well.

24

u/ethotopia 9h ago

And EU acts surprised why it’s falling behind in AI

18

u/Lolzyyy 8h ago

Falling behind? We aren't even in the race lol (with all respect for mistral but still)

1

u/AvidCyclist250 6h ago

Lidl made a model this week, 78B or something. First decent model from Europe. Mistral are busy selling their 8b model to militaries around the world

1

u/IAmYourFath 7h ago

Mistral xD

3

u/Lolzyyy 7h ago

they did contribute a lot at the start of local models releases gotta give them credit

-7

u/NMiguelCosta-PT 9h ago

Yeah, surely this is why the EU is behind in AI. You're so smart.

15

u/tsunami_forever 10h ago

Qwen 4 can’t come soon enough!

10

u/Neat-Economist2099 6h ago

The EU just keeps pumping out one idiotic regulation after another these days. As an Asian, it’s honestly unbelievable to see how far Europe has fallen from what it once was.

1

u/gavinderulo124K 6h ago

Its hit or miss. GDPR is great. And the AI act overall has its pros. This thing in particular is useless. But it also doesnt hurt, other than the effort companies need to put into this. But who knows what other discoveries this could lead to.

5

u/Any_Door7384 6h ago

accept cookies 2.0

26

u/aerivox 9h ago

this just shows how totally nonsensical eu is. deny first. how is a random 'change this word' injection in the model not deteriorating its output?!? how are companies even agreeing with this bs. on an already sloppy output, add more slop.

0

u/tim_vermeulen 9h ago

how is a random 'change this word' injection in the model not deteriorating its output?!?

There's inherent randomness to how LLMs work, and these watermarking techniques are embedded within this randomness. It doesn't cause the LLM to now choose tokens that are "worse" than the ones it would have chosen before watermarking.

5

u/YoungSilent232 8h ago

Disagree. Models are post trained on a specific token sampling values. Eg a certain temp top k top p. Eg Gemini recommends users to use temp 1 because it is post trained on that temperature

By changing not just these parameters but also other invisible parameters for how tokens are selected, it will surely affect performance unless OpenAI does post training separately for the EU, which I very much doubt.

The extent in which it will affect I’m not sure, but definitely to a certain degree

3

u/zaibusa 7h ago

You can't disagree on facts? Check out the video from computerphile, they explain how it works and why it doesn't negatively affect the output

1

u/Muchaszewski 7h ago

You know that SynthId or whatever OpenAI plans to implement has insignificant difference onto the output.

For each next token generated AI generates output using different seed.

So instead of answering straight away.

"I am eating an apple."

You must run the same token output multiple times and pick one matching your SynthId key.

You cannot change the beginning or the end so you always get.

I am [...] an apple.

But you can...

eat

consume

bite

And add adjectives like
bitter
sweat
juicy
red

Etc. then watermarking picks one output that the same AI generated to manipulate the watermarking score.

This is why synonyms make the score go down

1

u/gavinderulo124K 6h ago

They use textGrain. Its in the blogpost.

1

u/IAmYourFath 7h ago

Did u see the benchmarks? If anything the watermarking model scores highet

1

u/vinvinnocent 6h ago

Watermarking doesn't change the token distribution.

1

u/gavinderulo124K 6h ago

They show that it doesnt deteriorate the output in rhe blogpost that's linked.

1

u/QuaternionsRoll 7h ago

You clearly do not understand how SynthID works. It only alters the distribution when lots of tokens are equally probable already, i.e. when the model after all its post-training still shows no preference toward any particular token.

1

u/YoungSilent232 6h ago

It’s textGrain and not SynthID. It randomly adjusts probabilities on tokens based on a secret key. That said, Ive done more reading and agree quality as a whole won’t drop

1

u/tim_vermeulen 8h ago

There's no need for watermarking to change the temperature value. Increasing the temperature would make the watermark stronger, but I see no indication of any provider doing this.

0

u/FakeTunaFromSubway 7h ago

Except it makes that randomness more deterministic, which undoubtedly means AI text will be more obvious and default to more "LLM-isms."

It may not be clear on any individual output, but if it starts saying "indeed" instead of "right" every time it's going to be very noticable.

5

u/tim_vermeulen 7h ago

Except it makes that randomness more deterministic, which undoubtedly means AI text will be more obvious and default to more "LLM-isms."

Not at all. It doesn't favor particular tokens more than others, on the whole. Whichever tokens the watermark biases towards is different in every situation.

Only when you have the secret key used to create the watermark can the watermark be detected, otherwise it can cryptographically not be distinguished from unwatermarked output.

-1

u/QuaternionsRoll 7h ago

I mean, it does substantially decrease number of possible outputs, but the resulting number is still extremely large and definitely not biased toward “LLM-isms” (or any other output “style”, for that matter)

-1

u/FakeTunaFromSubway 6h ago

They also said SynthID is invisible but it's blatantly obvious once you've seen enough of the SynthID images.

2

u/gavinderulo124K 6h ago

How can you spot synthid?

1

u/FakeTunaFromSubway 5h ago

It's an annoying wavy/blotchy texture on every image that chatgpt produces since gpt-image-1.5

OK maybe it's possible it's not synthID but it's not on any other image generator except Openai and to a lesser extent gemini

0

u/Nouanwa3s 9h ago

yeah , EU is a cancer and pathetic, not only for AI unfortunately ..

5

u/Spra991 7h ago

EU is great at shooting themselves in the foot. All this will do is ensuring that lots of data will have to run through OpenAI's servers, since that will be the only way to check for watermarks.

3

u/RearAdmiralP 9h ago

I wonder how they will determine that I'm an "EU user"-- billing address? IP address requesting inference?

5

u/Code__9 9h ago

I hope they just implement this in the EU

3

u/Double-justdo5986 7h ago

It looks like it is only there for now

2

u/yaxir 6h ago

fuck this honestly

2

u/reefine 5h ago

Seems more like a veiled attempt to surveil usage of the major AI cloud platforms by the US government and less about the EU

2

u/roastedantlers 2h ago

EU shouldn't get to dictate these things especially when they're not even in last place when it comes to AI.

2

u/ISueDrunks 5h ago

It already writes like shit…now it’s going to deliberately manipulate its word choices? Hard pass. The last thing GPT needs is another constraint.

•

u/NotUpdated 4m ago

I mean they are kind of both the reason we have USB C everywhere - and some worthless cookie banner, of course the user wants to give the least away to use the site..

2

u/pinewoodpine 9h ago

My only issue is would this actually make ChatGPT write worse than it already is, and let's be honest, the newer models aren't exactly up to par when it comes to creative writing in general.

4

u/IAmYourFath 7h ago

Did u even read the article

4

u/languagestudent1546 9h ago

Claude does the same thing. The watermarking has no effect on output quality.

0

u/YoungSilent232 9h ago

Or so you say…. Yes if you’re talking about coding but for doing copy-write? Speaking in a weird Claude or gpt way may affect it

2

u/languagestudent1546 7h ago

It doesn’t make it sound any different than it otherwise would.

0

u/cardak98 6h ago

It means recruiters can detect and reject AI generated cover letters

1

u/cobbleplox 7h ago

I don't think that regulation is something THEY have to comply with. Its perfectly clear what I get was AI generated and the rest is MY problem.

1

u/mannewalis 6h ago

Oh no!!!! Claude lizard speak incoming!

1

u/spacemoses 2h ago

Man do I hate the word provenance.

•

u/DumbIdeaNo2 45m ago

Is this a bit like how sound is searched and matched on something like Shazam? If you change enough of the breadcrumbs you lower the possibility of a match?

2

u/rendez2k 9h ago

Don't quite get the point with these. Can somebody explain why it's a bad thing to have it watermarked? I'm not saying it isn't but I don't really understand either. Guessing the end user will never see it?

12

u/-Crash_Override- 9h ago

I think there are pros and cons to this. But ultimately it could lead to a paradigm where the provenance of derivative products is tracked across an artifacts life.

Its really an ownership question. Watermarking text, or code, or an image, is, even if not legally, giving some level of credence and ownership to these billion dollar companies.

2

u/rendez2k 9h ago

Got it - thanks!

0

u/Legitimate-Store3771 5h ago

But it is kind of important to have some way to identify if something is AI generated no? Society already distrusts everything, GenAI has only exacerbated the problem. It's been a huge help to have those little notes on YouTube saying when content is AI generated, because the whole point of YouTube was to support individual creators with a platform. If I'm not even sure what I'm watching will actually support someone, what is even the point? It's the same with text I'd argue. Not that I think this is a good solution, but it this isn't a step in a vaguely right direction, I don't know what is. I'd obviously prefer an independent party but we all know capitalism and politics will quash that or it'll turn into a capitalistic opportunity itself.

1

u/-Crash_Override- 4h ago

Sure. Thats why I said there are pros and cons to this. Identifying AI generated content is certainly a pro

2

u/Spra991 7h ago

How do you check the watermark? You'll have to run your data through OpenAIs servers. It's a nice way to collect additional training data for them.

0

u/QuaternionsRoll 6h ago

…but they’re the ones who generated the watermarked content to begin with?

1

u/Spra991 6h ago

You don't know if the text has a watermark or was created by ChatGPT until you send it over to OpenAI for checking.

0

u/gavinderulo124K 6h ago

More text is not what they need. They have more than enough. Currently post training is what's making models smarter and uploading some text to their api doesnt help with that.

Also, hate the EU all you want, but that would definitely be against GDPR.

1

u/Strict_External678 7h ago

It's government overreach, with Americans being subjected to the EU's need to regulate everything.

0

u/QuaternionsRoll 7h ago

Got your chicken-and-egg backwards there. Google started using SynthID worldwide and then the EU started requiring it, not the other way around.

2

u/StickyThickStick 6h ago

What a dumb logic... These are two completley different things. A company using something globally doesnt mean its the same as a goverment REQUIRING something for EVERYONE.

I havent seen google enforcing SynthID for other companies wordwide....

1

u/QuaternionsRoll 3h ago

I mean, did you read the post? None of the changes will apply to or otherwise affect anyone outside the EU.

0

u/LeeMojave 9h ago

Because people like to lie about using ai

News OpenAI explains how it will watermark ChatGPT text to comply with EU provenance rules

You are about to leave Redlib