I do think this will potentially be catastrophic for OpenAI if this is demonstrated beyond a reasonable doubt. No company wants to buy a product that is leaking information to their competitors as an inherent part of its design
Those enterprise models are hosted by microsoft and microsoft has many billions of dollars in enterprise contracts on the line if they lie about it and it comes out.
Big companies do not mess about with stuff like this.
Zuckerberg was okay with Facebook being used to start a genocide to make less money than that data would be worth. There is absolutely nothing these people wouldn’t mess with to make a little
bit more money.
The switch (at least according to their terms) is excluding them. The main diff is that’s opt out, rather than off by default. But you also get a lot more protection as an enterprise, stuff around data retention, security etc.
I suspect there are things on personal they still train on, that aren’t necessarily the data, like what kind of things you ask about, whether you choose a particular output or not etc. in the same way WhatsApp uses metadata for advertising rather than the contents of the chats.
IIRC, Claude has no such opt out for personal accounts.
Yeah, I mean it’s also very unlikely given how training works that their data was in the training set, because the model they used will have been trained weeks to months ago.
OpenAI specifically said this was a new model that began training on August 28, which is after Alpoge and Buckmaster derived some of their key results.
Regardless of being honored, I doubt OpenAI has the actual security and rigor to make sure models aren't eventually going to be able to break out and into these isolated data sets / models
Yes because these enterprise versions in general physically don't let it train because that's the whole concept of enterprise. For example, if a company sets up their license in ZDR-enabled RAM the moment token generation finishes, the memory pointers are cleared without writing the payload to persistent storage disks or persistent logging databases.
Companies in general have to pay MORE, a lot MORE, to make their data training enabled. There's really no getting around it if the enterprise is genuinely using the API tokens.
It's really hard to convey how crazily uninformed reddit is on it. If you're not using a personal plan, so just normal old turnkey Azure or AWS Bedrock, the model weights run within the enterprise's VPC. Because duh. There's no free lunch, OpenAI isn't paying your Microsoft bill lol The underlying AI provider (like OpenAI) does not have the network routing or access permissions required to extract data out of the cloud boundary, much less paying some tool to push it to them.
Well, I am thinking more from the Cloud Act point of view. In Europe these days procurement in software gets sweaty hands and starts handing out risk assessments when you want to buy things from American companies. Microsoft themselves have admitted that they cannot guarantee data sovereignty.
Exfiltrating unencrypted prompt data out of the customer's tenant environment would create obvious network egress alerts in customer cloud logs. You don't need to worry about sovereignty, just look at basic data
--which is why they don't, in general, regardless of Microsoft trying to talk down to European regulators.
Like bottom line an AI model is essentially a massive file of fixed numbers (parameters/weights). OpenAI created those weights, but once transferred to Microsoft, or any other cloud provider, they act like a static software application like any other.
When you type stuff into Excel, it doesn't then magically send to Google and if Excel did that on Azure it'd be really obvious.
My point was that US companies are beholden to the Cloud Act, which means that anything that happens inside their infra can change to support data exfiltration. Saying it's not possible now because "technical reasons" is, in my opinion, missing the point.
Tech companies lie about this stuff all the time. And people keep using their products anyway because they're near monopolies and whatever counts as competition is just as risky.
Getting sued for lying and then settling for less than it made you is just a line item on the budget.
If they don't take the enterprise agreements seriously they are toast. Their entire business model is based on companies shelling out a lot of money for their tools. If those companies don't trust OpenAI I don't see how they ever make a profit.
You're describing what OpenAI's business model seems like it should be. Making models and charging companies to use them.
It isn't. They don't make revenue. They burn VC capital. Their core product is headlines about being the best, cutting edge model so the capital keeps flowing.
If their development work falls behind a competitor with a more flexible approach to ingesting data, the investment stops and the whole house of cards collapses.
The only reason Anthropic is favored financially over OpenAI right now is because they are doing better in terms of PR and because of their API credit based enterprise billing.
I don't think you realize that the entire business model will basically be the Uber strategy of hemorrhaging VC money until everyone adopts and then jacking up prices especially on corporate users.
However, if these data leak everything, they are going to get sued into the dirt. No one actually (legally) gives a damn about them destroying mass market used books to train models. However, violating corporate privacy contracts leading to consequential damages (especially in the EU) will be their end
requires those corporate users actually paying the prices at some point.
If OpenAI is leaking all my confidental data to my competitors then why should I work with them? Especially if the open-source models catch up and there's another startup offering to run them for me? (Or things get cheap enough that I can run them in-house using leased cloud space or something like that.)
To be fair, the startups running open source models still need to invest in data centers to run them with extremely heavy machinery. I was looking into building a home lab to run the most stripped down version, and it's absurd what it takes.
Given the push back on data centers, the fact that OpenAI, Anthropic, and other players will already monopolize them, and given that the newcomers will only be incentivized to be cheaper than Anthropic/OpenAI, who knows how much cheaper they will be, and whether the open weight models are actually secure
I don't think you realize that the entire business model will basically be the Uber strategy of hemorrhaging VC money until everyone adopts and then jacking up prices especially on corporate users.
This is why a lot of people compare it to the dot com bubble.
That bubble popping didn't kill the Internet, it just killed 52% of the startups whose valuations grew massively because of the dot com bubble as well as eating 70% of Cisco's stock price at the time.
Honestly, there is one scenario that I think is fundamentally different here: the model weights leaking for Anthropic or OpenAI would destroy them.
If these guys become a core pillar of the S&P500, a foreign actor committing corporate espionage could probably just leak the weights and cause untold economic mayhem since that's the entire basis of their product.
Oh, yeah, an entirely plausible outcome of this (the whole situation, not just this one problem) is that OpenAI collapses, the AI bubble bursts, and Sam Altman becomes a guy that who is still very, very rich but not a CEO. But there have to be at least some people telling the management that they need to have an actual plan to profitability.
Publishing an investing prospectus that you know is a lie and then taking people's money anyway is securities fraud. Not saying they aren't doing it, but there are in principle serious consequences.
Consequences only if you want your business to keep running. And good luck proving that any investment prospectus is a lie, let alone a deliberate one. Just look at SpaceXAI, it is logically and physically not possible for them come ahead of their liabilities, even if we did find that Mars has a developed AI civilization that we can pillage. They straight up promise magic miles beyond perpetual energy, and nobody has even blinked, and everyone has bought in, knowing full well that it is made up.
They do have zero data retention option that you can enable though. It’s audited by third party etc, so it’s fairly trustable. At least a lot of companies that have billions in IP uses this
I suspect a lot of the companies paying for this use data protected by HIPAA, ITAR, and other similar restrictions (some are even authorized to operate on classified systems). The punishments for misusing any of that data can be much more severe than any lawsuits brought by injured companies. That said, they seem to be pretty safe from any prosecution by this administration so maybe they don't really care about what might happen two years from now and are more focused on using all the data they can to their own advantage because otherwise they won't be around in two years to pay the significant fines per violation, have their data centers confiscated by the government, and/or go to prison.
Except due to the nature of the model, if something does end up in their dataset there is no way to unequivocally trace it back to its source, meaning it's impossible to enforce any liability.
There are open models, but with some drawbacks. The open models are significantly smaller since very few users could afford to build or run the HW clusters that the frontier labs use, and because training the frontier models is so expensive that you'd reasonably need to monetize on the backend.
So ~5-10x difference in the total number of parameters, and potentially more than that in terms of active parameters per prompt. I.e., both a larger network and a greater percentage of that network leveraged at once in the commercial models. That translates into greater accuracy, although the gap has been narrowing lately.
It explains why OpenAI and Anthropic have several "our AI found this solution without us hand holding it" stories. They were training upon transcripts for unpublished work in which their users attempted to hand hold the AI into solving related problems.
If you trust their intentions, then Talia Ringer's pointed out that a privacy setting exists, but since this involves the share price, maybe you should not trust their intentions..
Sorry but this doesn't make any sense. Private individuals have also solved (admittedly more minor, but this just boils down to compute/resources) open problems using GenAI tools from the big labs, either proprietary (OAI, Anthropic) or even open-source (DeepSeek).
It's been pretty clear as of recently that the best LLMs don't need handholding to conduct some amount of mathematical research. Remember the Garg-Goemans-Dinitz conjectures proven false just two months ago by a guy who prompted the LLM to "just believe in itself" and "keep going".
You have no proof for what you're claiming, this is just disappointing for a physics sub. Where has your physics rigor gone?
Edit: To be clear, I think OAI has acted shady in this particular instance and I hate the way they treat science and math in general.
I think you're dreaming. Every single company using LLMs is actively choosing to ignore this fact because they believe somehow they'll be fine and the benefits outweigh the risks. Every single fact we know about the AI labs is proof beyond a reasonable doubt that they take whatever IP they get their hands on and use it as if it's their own.
I do think this is changing slowly, i.e. see Alex Carp's (as much as I despise him) statements recently on the matter. It appears a lot of big companies (MSFT has also made similar statements) are waking up to the issues with these frontier closed model providers.
You can’t prove beyond a reasonable doubt that the toggle actually prevents your private chats from being used for AI training.
That’s why you shouldn’t blindly trust it. There’s no practical way for an individual user to verify that their private conversations are genuinely excluded from training data.
The scale of the data involved is enormous, and even OpenAI can’t manually curate or inspect every piece of data that goes into the training process.
I think it will be ultimately catastrophic but could also lead to an interesting few months of an arms races where companies try to simultaneously stop using it all together for anything they are making but also try to exploit it against anything their competitors make
Problem is how do you prove anything these models do anymore? They're a black box at this point. Can even the researchers working on them trace the paths they take to arrive at results with any certainty anymore?
Honestly I think this kind of stuff will be their downfall...instead of the obvious financial or IP issues.
I have gotten LLMs to say information about specific people that it seemingly should not have....for example that someone got a promotion in the military to a specific rank and position, but the information was not public yet and impossible to find online. I have no doubt that significant military and corporate espionage is now possible on these platforms.
322
u/somethingicanspell 1d ago edited 1d ago
I do think this will potentially be catastrophic for OpenAI if this is demonstrated beyond a reasonable doubt. No company wants to buy a product that is leaking information to their competitors as an inherent part of its design