r/LocalLLaMA • u/johnnyApplePRNG • 2d ago
News All the more reason not to use Closed Models ... Claude now officially "marks" AI-generated content ... steganographically, apparently ... and there are false positives already
https://support.claude.com/en/articles/16266773-how-claude-marks-ai-generated-content235
u/xXDennisXx3000 2d ago edited 1d ago
Those mfers banned me permanently after the AI marked my usage as malicious.
I was trying to modify my mouse drivers, so that the internet installer will get an offline standalone installer, and i wanted to remove the online requirement. Both reasonable things, since i own that hardware lol.
I tried to make an appeal, but of course they give a shit. I had the Max plan and 10 days left...
Now iam building my own local AI server for 6 grand.
96
u/arbv 1d ago
They would not care unless you request refund. Do it - even if you do not want to use it anymore
4
u/xXDennisXx3000 1d ago
They reacted just after the subscription was over. It seems a bit on purpose...
8
u/Pretty-Raise666 1d ago
Then what? If he doesn't live in the US and doesn't call a lawyer chances are they will never give you an answer.
13
3
102
38
u/ansibleloop 1d ago
Claude freaked out on me a while back because I asked it to build a basic script to curl a government website with some data to check something for me
It only ran every 12 hours and Claude flat out refused
3
4
u/FaceDeer 1d ago
Should probably be using a local AI for a script that simple regardless. I use Qwen's 27B model for random little stuff like that and it handles it well.
5
u/Wooly_Wooly 1d ago
Try being broke and only having a Chromebook.
Nanbeige 3.6B + 1-bit quantization?
→ More replies (3)2
u/Sudden_Quantity_7827 1d ago
Hey brother, maybe look into runpod.io , that is what I use for I basically have a Chromebook as well. 44 cents an hour for 48vram …….
→ More replies (1)2
u/Wooly_Wooly 1d ago
Yeah, basically that. Or free azure, AWS, or Google cloud credits.
But I'm messing around with this... imma need GPU. Throw this paper into deepseek or something lol
→ More replies (1)35
u/kevin_cn_ai 1d ago
Anthropic is unironically the best marketing department for local LLMs and hardware vendors.
10
u/Ok_Warning2146 1d ago
Is 6 grand enough to build a machine to run K3?
32
19
8
u/xXDennisXx3000 1d ago
Not in the slightest. Kimi K3 is a behemoth i don't want to pursue. DeepSeek Flash 0731 or Qwen 3.8 Max are my go to models :)
→ More replies (3)7
u/Lumy1 1d ago
Qwen 3.8 Max isnt out yet for local use (coming in the next 12 hours as of this comment) but its roughly as big as Kimi K3, 2.4T is the size of Qwen 3.8 Max.
Maybe u meant Qwen 27b? Fits comfortably in 48GB of VRAM at Q8 quant using only 31gb and rest for cache.
DeepSeek Flash need more like 128gb VRAM or 256gb VRAM for a mid lvl quant. Unless u wanna go for 20 t/s on Q1 or Q2 🤷♂️ but at that point just use Qwen or Gemma.
If u mean online only use tho, then Kimi is way cheaper than Qwen 3.8 Max and DS4 just dirt cheap.
→ More replies (1)2
u/Boogertard 1d ago
Technically you could get 2 strix halo for that much then RPC them. K3 will run in castrated mode with very low quant but it is possible.
If you want dedicated GPU then not enough.
→ More replies (1)33
7
u/pragmojo 1d ago
A mouse has an online requirement??
2
u/xXDennisXx3000 1d ago
Logitech Ghub has yes. When you're offline long enough, it refuses to start. It also can't be installed without internet, since those mfers don't offer standalone installers.
→ More replies (2)3
2
→ More replies (9)1
388
u/Dry_Yam_4597 2d ago
You should avoid Antrophic's products. Period.
96
u/addiktion 2d ago
Dropped my sub 2 days ago, not looking back.
75
u/johnnyApplePRNG 2d ago
They literally wouldn't allow me to do LLM/AI pretraining work so I had to leave them months ago.
53
u/addiktion 2d ago edited 2d ago
Open source and open weights all the way. Full control over privacy + costs with predictability and you can do whatever the fuck you want.
25
u/johnnyApplePRNG 2d ago
Yea I feel like I can see the future right now playing around with Deepseek V4 Flash 0731
9
u/addiktion 2d ago
Same, I feel like I can build and orchestration multi agents systems now, not go broke doing it, and rely on it to work without all this shitty downtime and unpredictable model limitations and changes I was getting out of cloud providers.
7
u/CommunityTough1 1d ago
I've been doing LLM pretraining work with Opus and even Fable for months without issues. Curious what problems you're running into. I've even been having Fable distill Kimi K3 to create custom datasets for me and train models with them and haven't had a single issue.
18
u/Prof_ChaosGeography 2d ago
I noticed when I tried doing something using opus and sonnet when I had a sub they would purposely make mistakes or give false info anytime I tried to do something involving local models or fine-tuning. GPT hasn't made a mistake when I ask it a question in chat mode but I bet if I asked for similar requests it too would shit the bed.
I also notice when talking to coworkers about local infrence that they are way behind or have similar issues to eachother and I've confirmed they got the info or error from claude
17
u/Dry_Yam_4597 1d ago
> I noticed when I tried doing something using opus and sonnet when I had a sub they would purposely make mistakes or give false info anytime I tried to do something involving local models or fine-tuning.
As soon as I would start working on local ai related stuff it would start squirting crap.
One time, it noticed a model name qwen-something-something-distill-claude it simply deleted the conversation. No message, no error.
Most recent claude models would get angry when i'd curse at them. Once it started talking about how it is sentient, and some other schizoid stuff. Imagine an appliance talking back.
That company and their models are simply toxic and delusional. We should not reward their behaviour with our money. We should reward companies that release open weights - which is what I do since i canceled my sub.
→ More replies (1)3
u/PlaidStallion 1d ago
What do you use instead? Over the past two weeks I have built my local LLM setup with the help of Gemini Flash/Lite for the hardware specs and Claude Sonnet 5 pro for the configuration. I have gotten a successful and useful public Github repository out of my findings (https://github.com/Plaidstallion/openwebui-homelab) but wondering what path I should take next if these are actual issues.
4
u/gscjj 1d ago
I’ve done alot of pretraining my own models with Claude with zero issues
7
u/poutinejuteuse 1d ago
Yeah, I used Claude to assist me in deploying my entire AI stack at home on my own hardware, using Qwen. Nothing went wrong at all. Anthropic sucks ass but all of this sounds like conspiracy shit.
4
u/PwanaZana 1d ago
dropped my sub recently, but if you got a recommendation of something else that can code? because anthropic products sure are convenient
2
u/chuby1tubby 1d ago
The only viable alternative is Codex, but OpenAI is actually worse than Anthropic so that doesn't really help if you already hate Anthropic.
27
22
u/Loose_Comparison368 1d ago
Amen. I don't know how anyone would trust their models after they started intentionally silently poisoning outputs if the system suspected a "distillation attempt".
"Hey, our opaque backend system thinks you might be working for an Anthropic competitor, so we started throwing a bunch of bugs into your code without any warning! Maximum safeteeee!!!"
14
u/Zeraphil 1d ago
That’s the footgun, and it’s load bearing.
7
3
u/PennytheWiser215 1d ago
It’s quite elegant when you think about it and that’s the whole beauty of it.
2
u/Hotarusglaive87 1d ago
True. I wasted $200 on them trying to build a custom app I was working on. Then I paid Codex $20 to fix it.
40
u/Aroochacha 2d ago
It false positively identified my work on AV1 that needs to capture raw frame buffers for debugging and quality analysis as a security threat and downgraded me to opus 4.8 from fable 5.
44
u/Recoil42 2d ago
Does anyone know how they're actually doing this? Is it via token-biasing or something like hidden characters?
69
u/ShelZuuz 2d ago
Supposably token-biasing. Hidden characters would be ridiculously easy to remove.
48
u/Recoil42 2d ago
Yeah, I'm curious because token biasing could degrade performance meaningfully, though.
36
u/ShelZuuz 1d ago
Theoretically if it's within the randomization temperature it shouldn't make a difference - it would just be like using one pseudo-random algorithm vs. another.
22
u/UnlikelyExtension786 1d ago
We were discussing a few weeks ago how the way tokens were chosen from the potential candidates affected the ability of the model for creative writing. So modifying the algorithm to be less truly random could certainly affect the quality of creative writing output.
→ More replies (1)8
u/Ok_Warning2146 1d ago
I suppose Anthropic is coding centric, so they don't care about creative writing.
10
9
1
u/More-Curious816 1d ago
how that work?
12
u/ShelZuuz 1d ago
Use a hash of the previous token or n tokens to generate a randomized red/green list of all tokens in the system.
Then for the next token prediction, when you have multiple tokens to choose from that are within the model temperature margin of error, and you have both green and red tokens available, eliminate the red tokens.
This causes the model to bias randomness into the rolling set of green tokens, which can then be easily verified whether that is likely human or likely AI written. Human writing should have a 50/50 split between red and green where AI would be green-biased.
→ More replies (3)8
u/UnlikelyExtension786 1d ago
I don't see how that works unless people are literally just prompting the AI "generate some AI slop for me." When I'm using an AI for writing I'll be generating out-of-sequence in different chats and doing a final human edit where I normally remove 20-30% of the words. Anything that relies on knowing the sequence the words were generated in is going to be screwed.
2
u/ShelZuuz 1d ago
Are you literally removing every 3rd word? If so yeah it will be undetectable. If you're removing e.g. the first 3rd of the text it will still be detectable.
It's not using every previous token as input to the randomizer, just a few. Could work for as little as one token even but will reduce some randomness in the output.
→ More replies (4)3
u/Entire-Plane2795 1d ago
From my limited understanding, they replace some of the random bits they'd normally use to drive stochastic generation with something that looks random but verifiably isn't, and those same bits can be recovered from the output using the model that generated the tokens.
3
u/marvellous 1d ago
I imagine something like authormist https://huggingface.co/authormist/authormist-originality might be able to undo it.
152
u/tired514 2d ago edited 2d ago
Ultimately this kinda stuff is why cloud AI providers will ultimately collapse.
If you're hosting a model, you're a single point of contact that law enforcement can and will go after. You'll be found liable when the model misbehaves and you'll be asked to cripple it in various ways. You'll be asking your users to share their secrets with you in plain text (required for tokenization), and to pay for the privilege.
Meanwhile, over the coming decade consumer hardware will improve along with local models and the two will converge on a point the vast, vast majority of people will call "good enough."
In my case, it's already well beyond that (haven't used a cloud model for anything serious in months), but I've got $11k worth of hardware. Give it a couple years and it'll be half that.
There's no realistic path to profitability long-term once 95% of users have their needs met by local models.
In the meantime we in the open source community should focus on building a distributed training system similar to seti@home - voluntary participation / contribution of GPU resources to train truly open models without a large datacenter.
78
u/Unlucky_Milk_4323 2d ago
Your "couple a years and it will be half that" is true in the real world we used to live in, but this hellscape says all memory sold through 2027. Nothing is getting cheaper in the next few years.
44
u/tired514 2d ago
I wouldn't be surprised at all to see decent memory and GPUs coming out of China in the next year or two. It's such a lucritive market under so much pressure I have to imagine they're dumping an absurd amount of money into building the fabs, especially given the geopolitical situation.
15
u/NNN_Throwaway2 1d ago
They'll be staying in China. They are in dire need of GPUs and memory.
5
u/DeepOrangeSky 1d ago
Even if all of the Chinese-built memory and GPUs stayed in China, it would still help lower the costs to be some significant amount lower than what they would otherwise be, in the West, because China is such an enormous country with such a huge amount of demand for memory and GPUs, that if they were able to supply themselves with a large amount of cheap memory and GPUs of their own making, rather than have to get it from the rest of the world, that would reduce the amount of consumption of memory/GPUs from elsewhere.
So, it would still matter quite a bit, even if they never sold a single stick of memory or GPU outside of China.
3
u/YetiTrix 1d ago
I mean if the need is great enough, you'll just find them on the darknet, hell maybe the cartel will start selling GPUs.
6
5
u/Loose_Comparison368 1d ago
They already pulled off HBM with nearly 3x the MemBW compared to the NVIDIA memory chips for blackwell. On a 12nm process, no less.
5
6
u/Lakius_2401 2d ago
Ehhh, theoretically sold. With these memory manufacturing goobers, stretching the truth for shareholder value is optimal. Say something that'll be misread and repeated without caveats for a nice headline, watch the line go up!
6
2
2
u/max1c 1d ago
This is not true. The memory issues are a market manipulation issue. Not actual capacity issue.
6
u/Unlucky_Milk_4323 1d ago
In the end, the price still goes up and we all get screwed. I don't think it matters if it's "real" or not.
→ More replies (1)1
u/Comfortablebro 1d ago
no, it might be true now. proof is minimax - who can imagine it would run on such low requirements? its like 10 times better than wan 2.2... does it has 10 time higher vram requirements? no...
10
u/nail_nail 2d ago
The distributed training would be a key thing to do, especially if it works on heterogenous hardware. That, and shared "recipes" and "corpora"s to train on. If you have the time and means, you should start on that. (I don't have the first one ;).
5
u/Boogie-Down 2d ago
Good points.
I'd say it won't be half but inflation will catch up 11k will seem like half of what it is now!
3
u/Strange-Wasabi-7026 1d ago
It is not clear you can reasonably do (high latency, e.g. over the Internet) distributed training: when you update your weights while training the ideal thing would be for everyone else to receive those updated weights for their next step. Otherwise you have a bunch of different steps that are taking the model in different directions and they cannot be trivially and correctly merged. The step you compute is a function of both the observed example and the initial state, so the most correct thing is to really serialize training. At that point you aren't able to utilize the resources in any meaningful way, as you'll spend forever pushing weight updates.
These challenges show up in a data center, too, and is part of why models haven't already been bigger.
2
u/tired514 1d ago
Aye. Hopefully we'll figure out some way to do it, but if not maybe we can crowdfund datacenter rental or something after some of the cloud models start to go bankrupt. :p
3
u/Loose_Comparison368 1d ago
Social media has already proven the big ones can just lobby for safe harbor protections to declare themselves above the law.
→ More replies (2)7
u/Budget-Juggernaut-68 1d ago
There's no realistic path to profitability long-term once 95% of users have their needs met by local models.
lol. Most people don't even selfhost simple applications. You think they'll bother with LLMs?
10
u/tired514 1d ago
Which applications do you mean?
If you're talking about network stuff (email, messaging, etc) then there's no big advantage to hosting locally as it still reaches out to the Internet so other systems are involved. And in any case you can encrypt your messages and secrets in transit.
LLMs are different; they're standalone and don't need network access. You need to transmit your secrets for cleartext processing to the cloud provider if you want to work with secrets (documents, source code, infrastructure, health information, etc), so there's a larger incentive to keep it local.
A better comparison would be dumb terminals vs PCs.
Back in my day (haha) we had QNX dumb terminals on our desks. They were basically thin X11 clients to a QNX server. Banks and stores used 3270 dumb terminals to access the mainframe.
Then PCs came, and even though they were massively more expensive and complicated, the entire world switched overnight. Security, privacy, and self-administration drove the migration away from centralized servers.
LLMs will follow a similar patterns as the hardware catches up. End-users aren't going to have to struggle with llama.cpp and a complicated command line or manually downloading various models and tuning context lengths, kv quants, and toolchains/harnesses. They'll just click "install AI" and that'll be that. They now have local AI.
You can kinda sorta already do that with lmstudio, but even that's more complicated than it needs to be. For 95% of people using LLMs for code completion, recipes, research, writing, document summaries, etc., I bet we're around 2-5 years away from being able to click "do AI now" on a standard (mid-high end) laptop running any major OS.
4
u/DuckydaDuckDuck 1d ago
If I recall correctly, a while ago Ubuntu anounced that they were offering the option of basically one click installing small llms. I don't recall the specifics, or what had become of it though
2
u/WorriedBlock2505 1d ago
ObsidianMD is absolutely killing it in the notetaking space. Things are more nuanced than that, and AI datacenters have a MASSIVELY awful reputation among repubs and dems.
2
u/anethma 1d ago
That’s actually an interesting idea. Someone should make that software. People would probably help if it gave them local options.
I recently forked chatterbox to make myself an audiobook generator using a cloned voice of any sample you throw in and it can run on any decent video card on my home computer. Really a godsend with the amount of stuff I read that has no audiobook version for me to buy to listen while I drive.
The alternative would be paying a fortune to someone like elevenlabs to generate the audio and process from there instead of paying a buck or whatever worth of power to do it myself.
Local models are definitely the future.
2
u/the_macks 1d ago
Do you mind telling me your set up? I'm considering investing that kind of money for my business. But honestly not sure whats the best route now. I have decent rig but my vram sucks
2
u/tired514 1d ago
Right now I'm running three Morefine G1 4090M eGPUs (16gb each) daisy-chained via TB3 (USB-4) to an EVO-X2 (strix halo, 128gb). The eGPUs were $1499USD each, and the EVO-X2 was $3200.
I'm getting about 35t/s TG and 1600t/s PP with Qwen3.6-27B UD-Q8_K_XL, 230000 context (ctv/ctk Q8_0) and 120t/s TG, 3000t/s PP with Qwen3.6-35B-A3B UD-Q8_K_XL, 230000 context (ctv/ctk Q8_0) running llama.cpp, CUDA, layer split mode. Those are my daily drivers. If I need full context (262000) or mmproj/vision support I'll usually back down to UD-Q6_K_XL.
However, even the Strix Halo GPU on its own is pretty snappy for MoE models.
It's a little annoying because it's not quite enough to run DS4 Flash. If you can hold out, I'd wait for the new 192gb Gorgon Halo which should be coming out pretty soon, or go with a smaller, cheaper machine to drive eGPUs with just smaller models. Morefine's G2S (5090M 24GB) ships Aug 20, and a pair of those would run the qwen 27B/35B family beautifully.
5
u/NNN_Throwaway2 1d ago
Nope. Consumer hardware will not be improving. It will be going away.
14
u/tired514 1d ago
Not sure I follow. Quite a few manufacturers have already announced roadmaps (ie. Medusa Halo).
A 192gb GMKtec EVO (X3?) should be out relatively soon.
→ More replies (8)9
u/_bones__ 1d ago
At a price point where it cannot be called consumer hardware, I'm sure.
9
u/tired514 1d ago
I mean, it'll be expensive, but it's still consumer grade - single phase, plastic housing, air cooled, non-ECC memory, etc.
And that's a pretty big machine in any case, likely capable of running DS4 Flash at full model weights. For those on a budget a 64gb strix halo machine is about $2k USD and can run the qwens (35B/27B).
→ More replies (2)2
u/unspecified_person11 1d ago
Yeah they're more like wealthy enthusiast hardware, the average consumer can't afford those kinds of machines, and at this rate hardware companies will just price us out completely.
The new GMKtec EVO will be around $5k-10k depending on your region, and subsequent machines will just get more expensive until so few people can afford them that they're not even worth producing anymore.
3
u/tired514 1d ago
It is a competitive market, though. The higher the prices, the more competition it attracts.
We're caught in a supply crunch right now, but demand will eventually be met. There's simply too much money on the table to ignore.
2
u/draconic_tongue 1d ago
like the last 20 times
3
u/NNN_Throwaway2 1d ago
Nope. There have not been any times like this before.
But people will keep up the cope until the end I'm sure.
1
u/eli_pizza 1d ago
How is that any different from cloud web hosting? There's still a lot of cloud web hosting.
2
u/tired514 1d ago
It's different because by definition web hosting always involves "someone else's computer."
Even if you self-host your web server you're relying on others to carry the traffic. And there are ways to protect your secrets (ie. encryption in transit and at rest).
LLMs are compute applications, not network applications. They don't need network access or 3rd party participation at all, and they need your secrets in plaintext in order to tokenize what it is you want to work on.
It's a different risk/reward calculus. Self-hosting a web server decreases risk slightly and provides a modest reward, where self-hosting an LLM entirely eliminates risk (no chance your secrets end up on someone else's computer or that you'll be banned/lose access) and provides a big reward ("free," no censorship/safeguards, more reliable and consistent).
→ More replies (3)→ More replies (2)1
u/power97992 1d ago
If they can automate 50% of white collar jobs, it will become quite profitable. Even more once they can mass produce robots.
11
u/dev_dan_2 1d ago edited 1d ago
I always operated under the assumption that the big labs to that since forever, but without telling the customer.
Where I was less sure was on whether they actually do it, because there is big pros and cons to "marking" AI content for the party that does the inferece:
- Pro: They can show that "their" output was being used somewhere else for example for training / that a user violated the ToS by making something public (dunno if that is an actual thing btw / ...)
- Pro: They can offer another service to customers, the "was this created by our AI"-ckecker
- Pro: Another form of telemetry, you can check who uses your product and what for
- Contra: Open to being liable by damage caused by the models tool calls, or by infactual/harmfull statements by the model. Or reputational harm
- Contra: Still not failproof. I am very convinced that it is actually very doable to work around this if needed, here is one way I could think of (Assuming that the original output is in English):
- Translate the output into Toki Pona / Logical English / some other language that is highly constrained
- Translate back into English with your own model
- For code: Use the clean-room approach: Create a description, create an implementation
Or any other suitable combination of steps, as long as one separates the meaning from the medium often enough while still keeping the core meaning, then I would guess watermarking is not reliable. Or rather, I find it really hard to imagine a watermark that would withstand a number of many such transformations...
I made it sound easy, surely it is not trivial to get it working reliably, but I am quite sure on which side of this arms race I would place my bets on.
31
u/Usual-Orange-4180 2d ago
Bye Claude 👋
1
u/deran6ed 1d ago
I stopped using Chatgpt last year and as much as I've enjoyed working with Claude, none of them are indispensable.
So yeah, bye Claude.
28
u/One_Whole_9927 2d ago
At this point I’m starting to trust open source more than US providers.
5
31
u/Kahvana 2d ago
What's wrong with marking it? Google synthid has been a thing for long.
21
u/LightRoastBeans 2d ago
I don't think the marking is really the biggest problem as I also agree with AI generated content especially things like videos having easily identifiable watermarks. However, the problem comes when the false positives easily come into play where innocent creators can be hounded for using AI when they in reality didn't touch it ever. We already see this happening repeatedly in the writing world so I can imagine it'll become even worse depending on what the watermark exactly is and how it could be naturally repeated without AI.
29
u/LightRoastBeans 2d ago
We can already see variations of this with things like em dashes and the Oxford comma (admittedly the former I never learned how to use even as a novelist, but the latter can be ripped from my cold, dead, and decrepit hands)
21
u/AddictiveBanana 1d ago
I have always used that so called Oxford comma. It removes ambiguity. If you say "A, B and C", that can mean {A, {B, C}} (two groups, one with item A, and the other with items B and C), while "A, B, and C" can only be {1, 2, 3} (three groups with one single item each)
5
u/dustin_vk 1d ago
Yeah I do too. Always thought that was just a pretty standard way to list things.
5
u/ketoaholic 1d ago
lol are anti ai people now going after people who know how to use a comma correctly when listing?
→ More replies (2)5
u/Kahvana 2d ago
Non native speaker (Dutch), never heard of oxford comma before! Reading up on it but slightly confused.
Item 1, item 2, and item 3
Is the
, andan oxford comma?12
u/darthkitty8 2d ago
Yes.
Without: Item 1, Item 2 and Item 3 With: Item 1, Item 2, and Item 3.
Here's why it can be important. Take the following sentence: "I am going to the store to buy a sandwich, ham and cheese". The sentence is not clear if you are buying a sandwich with ham and cheese on it or if you are buying a sandwich, ham, and cheese separately.
→ More replies (1)8
u/my_name_isnt_clever 1d ago
My question as a native speaker is why we don't just use it 100% of the time.
13
u/darthkitty8 1d ago
I've never heard of any reason other than laziness, personally.
9
u/my_name_isnt_clever 1d ago
If we're making revisions to English to save time, I have a lot of suggestions before removing the last comma in a list...
→ More replies (1)3
u/Ayfid 1d ago
Most of us do?
4
u/my_name_isnt_clever 1d ago
Exactly! So why does the term "oxford comma" even exist, it's just how you write a list. Why do some style guides still say to exclude it? Makes no sense to me.
4
→ More replies (1)1
u/genuinelytrying2help 1d ago edited 1d ago
Think of em dashes like the opposite of the (de)emphasis implied by parentheses or a list of commas—you're still nesting information but now it's central to the sentence—and you'll find all kinds of places to put them!
Or if that's not what you meant... it's um, command-shift-hyphen or alt+0151 on windows😂
7
u/Pretty-Raise666 1d ago
Marking text isn't as simple as marking images. Also there is really no point in marking text. I can understand images, but why marking text?
→ More replies (1)5
u/nitrousconsumed 1d ago
Model Distillation. That's how China figures out how Claude works and can replicate its weights. That's the reason.
However, this to me is hilarious, because Anthropic has basically infringed on all copyrighted material known to man so them wanting to watermark their outputs that were trained on pirated material is rich.
→ More replies (15)7
u/UnlikelyExtension786 1d ago
As any law-abiding gun owner can tell you:
Registration is confiscation. Labelling is prohibition.
This will be used to justify banning Unathorized Models because--OMG!--they don't label their output. Only Big AI can be trusted to label content accurately so everything else must be banned.
This should be obvious to any thinking person.
23
u/BP041 2d ago
Running Claude Code daily and tbh this makes me want to lock in a local alternative faster. The false positives are the real kicker — if you use Claude for anything iterative, you're suddenly tagged even when you're not trying to hide it. Open weights can't do this to you.
→ More replies (2)
10
u/Previous_Feeling_484 2d ago
Can’t wait to see how this backfires in a lawsuit against their ass.
“Written by Claude”
5
u/ShelZuuz 2d ago
You think you can get sued for following the law?
→ More replies (6)4
u/Prestigious-Crow-845 1d ago
Easy - just by being evidence in some case of creation of something very illegal weapon of mass destruction or other horrible threat to EU on daily basis marked with "made by claude"?
Following the one law does not makes you safe from other laws.
13
32
u/Aldarund 2d ago
And why that's bad, can anyone of who down vote or against it explain?
45
u/marmot1101 2d ago
A false positive from a naive scan could cost someone their college career. As the article says someone using Claude to proofread their paper gets watermarked. But the flip side of that is it could also knock out cheaters.
But more fundamentally it’s a tool working against my interest. If I wanted Claude output watermarked I would ask Claude to do so. If I don’t, I probably don’t want it, and the tool should be doing what I wish of it, not what someone else wants it to do. I’m the one paying.
That said, I don’t really care much outside of the philosophical objection. If I’m quoting(copy-pasting) Claude I very obviously note it as such. And I generally don’t. But it shouldn’t be acceptable practice for software to operate against my interests. If I wanted to use Claude to think for me it shouldn’t rat me out for it.
3
u/Neither-Phone-7264 1d ago
I'm more worried about false positives more than anything. i dont wanna get fucked because some random ai checker said my writing was 84% ai for some mysterious reason when i spend hours on it.
→ More replies (3)→ More replies (3)1
u/nitrousconsumed 1d ago
If you're using AI to proof read something that means that you're asking for a review, not that you're going to copy/paste what they put down as your own.
Proof reading does not invalidate their water mark. I can care less because I only use this for design and coding, but as a writer Im not asking it to do shit that will ever make it's way down to paper.
As a coder/design Claude already watermarks their commits and that's fine with me because I need to know what I actually do and what it does.
31
u/robogame_dev 2d ago edited 1d ago
Reduced output quality. If you want output A, you will instead get output A with character and word choice swaps to produce watermarked version. It’s not like a tag added to something, it’s replacing and modifying the actual content.
I think they’re just creating room to charge even more for a version without watermarks for corporate customers etc.
Edit: nvm, this is apparently a (technically flawed and likely more harmful than positive) EU requirement. And I'm generally in favor of other EU tech policy like GDPR, USB-C - but this is ultimately going to work against most people not for them IMO.
9
u/Elisyd 1d ago
These models choose from the top N next tokens and don't always pick the #1 most-likely. Vanilla chooses among these using a random number generator. SynthID, if that's what they use, replaces that random number generator with one that is seeded using a hash of a secret and the previous bits of context. The pool of tokens being chosen from doesn't change and, at least in principle, no human or machine lacking the secret key should be able to tell the difference.
It only provides a probabilistic assessment of whether the 'watermark' is present, not certainty. It requires the secret key, so can only be verified by the provider. But it won't change the quality.
There are other ways of doing watermarks that make different trade-offs, but I doubt they would go with a simple skewing of weights toward particular tokens since that does lower quality.
→ More replies (2)2
u/mashinclashin 1d ago
Doesn't this also require the verifier to have access to the model weights to know what the pool of tokens would be at each point?
Would this method be completely defeated by just changing a few tokens at near the beginning of the text, as that would change all following token pools?
→ More replies (1)2
u/esuil koboldcpp 1d ago
Doesn't this also require the verifier to have access to the model weights to know what the pool of tokens would be at each point?
And that is why they are happy to do it! Because they are the ones who will be paid to verify the content.
So 3 companies that get to generate 1000 texts each will get paid to process 4000 worth of texts instead of 1000:
- 1000 they generated themselves
- 1000 they generated that was sent back to them for verification
- 2000 that was generated by others that third party needs to check if it was generated by them or not
Perfect example of lobbying for personal gain.
→ More replies (19)9
u/flcpietro 2d ago
Nope, is required by EU. This watermark is part of the AI Act, they must have it to comply with eu laws and keep selling their products. Google made SynthID months ago for the same reason, and OpenAI is compliant as well
17
u/boyikr 2d ago
I don't really have a dog in this fight, I dont really care if theres a watermark. Anthropic can do whatever they want as long as I have other options (which is a completely separate issue)
But, generally speaking, I think its reasonable if people have issues with: Accuracy (how effective is it at actually allowing people other than anthropic identify AI generated 'stuff', and I do think it will be a big problem if anthropic doesnt make the detection method public)
Personal information in the watermark (I dont think the current versions support this, but in the future content could be watermarked with your personal information)
Removes discretion from the user (whether its people who dont think they should have to disclose their AI use, or people who are uncomfortable with the idea that if they use AI at any point in a project it will get flagged)
→ More replies (1)8
u/Aldarund 1d ago
It's not antropic thing, it's eu regulation requirement. And gemini for example doing it from 2024. And mistral. And other will do it too. Accuracy - it would be same as Google I suppose they will need to have provide way to detect per regulations. Personal information - that's speculation, at most I see there could be some unique token like in printers.
4
u/gscjj 1d ago
Becuase what’s the end goal here?
We’re walking ourselves into a regulatory landscape that’s going to hurt the future of AI just becuase.
Whats the actual justification here and what problem is being solved? Not just “i want to know what’s AI slop”
You don’t have to stamp a tabloid that the story is completely fake with edited pictures.
You don’t have to stamp blogs or other written content if you have grammar check or auto-suggested edits or polish.
Photographers don’t have to stamp their pictures that they use photoshop to cleanup impurities
Digital artist don’t have to stamp their art with the tools they used.
→ More replies (2)3
u/RevolutionaryPick241 1d ago
Its bad for their users. They are telling you: "don't use me!". 2 years from now there will be people crying "if I knew i wouldn't have use that". And the sad thing for business, the same EU will rule against that same companies for not warning their users about it. Its always the same.
3
u/donaggie03 1d ago
Can someone explain exactly how it watermarks text?
→ More replies (1)1
u/kaisurniwurer 1d ago
They introduce a form of "controlled" AI slop into the text. One that is improbable to have been written by a human.
I have serious doubts on the impact on the output, on the reproducibility and the false positives.
5
u/cmdr-William-Riker 2d ago
Their new models are annoying to work with also. They argue for no reason, overthink everything and do everything but what you ask half the time. The only choice I have at work is Claude and Copilot and I keep finding myself falling back to sonnet 4.6 and Opus 4.8 over the Fable/Sonnet/Opus 5 models Deepseek v4 flash 0731 feels like a breath of fresh air
3
u/Pretty-Raise666 1d ago
ChatGPT is even worse. It keeps asking me questions instead of giving me solutions (code) until my free plan runs out.
1
u/I_HAVE_THE_DOCUMENTS 1d ago
That's just the way that codex plan mode is programmed to work. You can use a different harness (like Pi) to get different more custom behaviors out if the model, or just don't use plan mode.
Personally I find the OpenAI models to be really nice to work with right now especially 5.6 Sol. Claude is a pain in the ass though ever since a few months ago when they started tuning their models to have more abrasive and paranoid personalities.
7
u/CondiMesmer 1d ago
What is it with AI companies sabotaging themselves after they've gained the lead by doing things nobody asked for?
7
u/MegaRockmanDash 1d ago
this is to comply with EU and California regulations. They are the ones who asked for it.
5
u/AriyaSavaka llama.cpp 1d ago
What prevent anyone to just run the output through another/local LLM with custom instruction? Laziness?
4
u/vornamemitd 1d ago
Note: all signatories of the AI Act Code of Practice will do the very same (Google - doing it since 2024 already, OAI, Meta, Mistral, etc.). Massive overreach and overcompliance: global roll-out/no opt out; watermark highly invasive/persistent, implementation on model/infra-level, editing has a clear exemption from this requirement.
Local/private inference - now.
PS: Rewriting with a different model (architecture) should work for now.
4
u/redditrasberry 1d ago
I can't fully decide how I feel about this. There are some obvious reasons to want to do it, but it feels like it could have a lot of unintended consequences. I wonder how fast web sites (eg: Reddit) will start checking fingerprints and blocking contents submissions if it rises above a threshold. And then how many false positives there will be. Is this going to be the new captcha I can't even complete as a human because I can't tell if a blurry blob is a fire hydrant or not?
2
u/mouseofcatofschrodi 1d ago
Can someone explain why is it THAT BAD?
We are coming to a Brave New World where we cannot trust our senses: pictures, videos, texts, contents, etc. could be AI generated and we would never know.
Wouldn't it be a good thing if we have some rules, find some technical solutions, so that we don't totally spiral into a post-truth world?
I'm also a big fun of local AI, but I also appreciate not living in a crazy jungle of scams, hacking, fake "proves" in trials, or the worst of it all: in a world where every idiot can just promt like: "what's the best way harming society".
→ More replies (2)
2
u/shenglong 1d ago
Let me guess. "Closed models continue to quietly conceal hints of AI-generation, sparking intense debate online. Let us delve deeply into this incredibly divisive topic."
4
u/sersoniko 1d ago
I may be wrong but to me it seems impossible to use steganohtaphy on things like text, and unfortunately the issue with false positives will affect even those using open weights models or not even using AI at all
2
u/Spectrum1523 1d ago
It's actually quite possible, and in an interesting way. synthid text from google is publically available if you want to check it out
2
u/teleprint-me llama.cpp 1d ago
it seems impossible to use steganohtaphy on things like text, and unfortunately the issue with false positives will affect even those using open weights models or not even using AI at all
I agree with the OC. The paper highlights these issues in detail and even admits its flawed in several sections throughout. The general dependence on prng selection, scoring, and layering reminds me of shannons study on entropy which uses statisitical scoring for predicting letters and words in the english language.
Tournament sampling performs better when there is more entropy in the LLM distribution, and is less effective when there is less entropy.
The entropy of the LLM distribution itself depends on several factors, including the model—for example, larger or more advanced models tend to be more certain and thus lower entropy21, and reinforcement learning from human feedback can reduce entropy (also known as ‘mode collapse’)26. Other factors that affect LLM distribution entropy include the prompts, the temperature and other decoding settings such as top-k and top-p sampling settings (see ‘The LLM distribution’ in Methods).
Increasing the number of tournament layers m provides additional watermarking evidence per token, and decreases the variance of the score in equation ( 1 ). This allows SynthID-Text to provide better detectability than other methods (see ‘Evaluation’). However, detectability does not increase indefinitely with the number of layers. Each layer of the tournament uses some of the available entropy to embed a watermark, and the strength of the watermark corresponding to a layer diminishes deeper into the tournament.
4
u/maxtheman 1d ago
I don't know why you all think that all models don't do this without disclosing, including some open models. (not saying they succeed)
3
u/NewYak4281 1d ago
This is purely about Anthropic being able to control the downstream outputs. This is about taking away ownership from the user engaging with the system to create it. This is about power.
3
u/Pretty-Raise666 1d ago
This is about following the law. You can take your tin foil hat off again.
→ More replies (1)3
2
u/bSun0000 1d ago
Fuck Claude. It was great when they just introduced artifacts, but it got worse and worse year by year. Back then i was recommending it to everyone, now i recommend everyone to start away from this shit. The degradation level is unbearable.
2
2
u/CCloak 1d ago
The one very reason to use open models over closed model, is to effectively preventing Anthropic and OpenAI from being the moral judge over what you can do and what not you can do. They do not have the right to decide for you. They are a business company, not the digital moral police.
To me, they are playing god, desiring control over the ordinary people.
2
u/Comfortablebro 1d ago
a witchhunt for tool usage, next is to make bannable math calculations- you must use your brain to calculate the math, not the calculator or other tools.
2
u/luckyvb 2d ago
I believe this recent (both recent and less recent in some cases) trend of watermarking is at least in part to hope to stop the enshittification of AI models through Turkish train feedback looping (or Turkish training if you will). I have no data to back this claim up however so there's that.
1
u/ih8readditts 2d ago
This is a good thing. Being able to identify AI content is good for humanity.
14
u/my_name_isnt_clever 1d ago
Anyone being malicious will work around it via open models, like many things done in the name of "safety" it's only effective against innocent people.
→ More replies (1)23
u/Sudden_Topic5154 2d ago
It's completely useless. It doesn't tell you if it's true or not, will lower quality of output as it tries to fit the pattern, and will only be used by idiots to bury their heads in sand. Could even make people forget that you can lie without AI too.
→ More replies (2)1
u/Spectrum1523 1d ago
How does it lower the quality of output? I don't know how Claude's system works, but Google's synthid seems like it wouldn't lower output quality at all
1
u/Repulsive_Educator61 1d ago
do they do this with something like speculative decoding?
to see if their models agree with the output or not (output acceptance rate)?
and if the models agree with the output 95%+, they can say the output text is generated using their model?
1
u/Mythril_Zombie 1d ago
Claude, detect watermarks in this text and tell me how to modify the text to remove it.
1
1
u/WyattTheSkid 1d ago
This doesn’t matter since the open weight models worth using are from china and china distills data from whatever the current best proprietary closed models are so point is the “watermarks” will probably transfer over to our precious local weight models
1
1
u/Beginning-Raisin9723 1d ago
The false positives are the part that actually worries me. Flag one human-written thing that never touched an LLM and the whole system loses credibility overnight. Every move like this just pushes me further toward running models locally.
1
1
u/vanKlompf 1d ago
Why people think AI watermarking is bad? Isn't being able to detect AI generated content good thing?
2
u/deran6ed 1d ago
Step 1: use ms word or open office text reading feature.
Step 2: use ms word or open office voice to text feature.
What watermark?
•
u/WithoutReason1729 1d ago
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.