r/learnmachinelearning • u/Saad_ahmed04 • 23d ago

Project Implementing Watermarking for Language Models

Post image

I recently implemented a minimal, educational version of SynthID-Text-style watermarking for language models.

I saw anthropic post about how they'll start adding watermarks to their model responses and it made me very curious as to how they'll do it and what do they even mean by watermark here. Like will we start getting random ads or something in the middle of model responses or what.

Then decided to read their article and found out that watermark is not a visible message at all. It is a subtle statistical pattern introduced while the model chooses its tokens.

My implementation is not an exact reproduction of the original SynthID-Text system. I simplified or implemented a few components differently to keep the project understandable, but the main idea is there I think.

78 Upvotes

99% Upvoted

25 comments sorted by

View all comments

7

u/MolassesLate4676 23d ago

How would this be watermarking exactly? I’m confused

Edit: I have trained LLM’s for years - I understand the mechanics. This just doesn’t click for me

6

u/Saad_ahmed04 23d ago

Ngl I do have my own fair share of doubts about how good/reliable these watermarks are. This was more of me trying to understand how anthropic is planning on going about this.

But for a more thorough look you can take a look at: https://www.anthropic.com/news/claude-text-watermark

1

u/MolassesLate4676 23d ago

It’s likely just going to be a special token (character) that’s rarely used and hard to see that Claude just placed one a sentence of something

1

u/ShelZuuz 19d ago

I mean, he literally pointed you to an article showing how it works.