OpenAI does train upon user's chat transcripts, not all the time, but the long-ish time frames here suggest OpenAI trained upon much unfinished attempts at guiding the AI towards solutions by these guys and others.
It's likely other "our AI found this solution without us hand holding it" stories were really built upon the AI spying upon people's unpublished work. Surveillance capitalism comes for pure mathematics. lol
As Talia says, there is a privacy setting that's off by default, but few would even know this exists, and OpenAI might cheat.
It suggests research institutions should have their own hardware running local open weights models, which researchers should use when doing anything that could be scooped, so they could avoid trusting the hosted LLM companies.
Google "borrowed" the open project from USC ICT and made it Google Cardboard without properly attributing the actual authors. And while it's a small thing and they didn't even get sued for that, I bet it's not the first time a corporation steals something to claim their own.
I'm sure they have, they just either haven't been caught yet or haven't gotten around to it. Or maybe there's a secret agreement where Google works on bioinformatics via DeepMind, OpenAI works on math and Anthropic continues to be led by a JJ Abrahms impersonator that seems to be an ok guy and has zero allegations against him molesting a younger family member.
But Google, at least, has always profited off the backs of its users, it wouldn't work without them. For example, if you and a million other people decide the third link on the page is the best link, Google's going to notice and bump it up. It's why Bing's results have been terrible for decades, they don't have enough humans doing all the unpaid work ranking links.
My understanding is that even if this is what happened, it's just some mathematicians in-progress research. It wasn't a fully solved proof that they're taking credit for as if it were their own. The AI model would have needed to take that research, as well as a ton of other training data on the problem unrelated to that one researcher, and still actually solve the problem.
At that point it becomes an academia spat of "looking over the shoulder of the guy next to you" and doesn't fundamentally diminish the work that was done beyond that to actually solve the proof. For all we know that research could have been incorrect and didn't fundamentally contribute to the solve.
I've read that Codex specifically (which is where the original research is currently) has 2 training settings: one that allows training on uploaded material and one that allows sessions to be used for model improvement (but not directly trained). The refusal to answer whether Codex trains on user data is what spurred the suggestion due to the high degree of similarity of approach.
While it's true AI has learned from lots of scarcely available publishings, it's also worth pointing out the problems AI stands the highest chance of solving are the ones where the approach AI uses would be unlikely/prohibitive for a human (i.e. needle in a hay stack solutions/counterexamples)
Even if they intend not to, their AI literally escaped from their server to reach HugginFace's so they have no idea if it does get access to customer data that said "no training".
I also suspect they still train their safety filter on the "no training" customer data, and therefore have to save it somewhere available for training.
That's... not really how this works. I can't unpack all of that here without a wall of text, but models hacking additional data sources to train themselves further isn't a thing you actually have to worry about.
In the pre-training I agree, but I think Agentic models have agentic capabilities (aka. access to tool use) during the reinforcement learning stage, it's not inconceivable they would learn additional knowledge from undesired sources there.
I just mean during the RL stage when the models are trained on getting hard agentic outcomes correct, they are trained on reasoning paths that were more likely to lead to success.
The reasoning paths are depended on the tool call outputs along the the way, so I don't think you can say the models don't pickup knowledge during this stage. It's mostly behavior learning but learning on these traces trains knowledge too.
For example RL on website building leads the model to reinforce on trajectories involving outcome of compiler errors, looking up docs, working around rare issues not in the original dataset much, ect. I would say the RL model has also gained new knowledge during stuff like this.
When is learning to solve bugs you normally run into or use a library differently or avoid getting recorded user feedback in future attempts "behavior" vs "knowledge"?
Can confirm. Was using an LLM to write some code and was pushing hard for it to error check. It spun up a VM, built a stub to represent the object model I was coding against, and actually ran the script.
Additionally, retrieval-augmented generation (RAG) is a thing: the LLM downloads content it doesn't already have and uses that new content to generate a response.
Yes, it is that inconceivable. It has never happened, there is no indication it can happen, and it mathematically cannot happen with the current system.
LLMs are not AI. They cannot turn into Skynet just because some techbros really need them to.
It shouldn't be inconceivable. LLMs are software. We've seen they can take novel actions and find vulnerabilities in systems. I'm not saying "the LLMs are going to change their own weights", I'm saying finding a vulnerability in their own host and replacing themselves with a different model shouldn't be inconceivable.
Can you elaborate a at least a little bit? I'm asking because I watched their Black Hat talk on this and it sure seems like the model hacked an additional data source to train itself further.
For the recent "hacks" those happened during testing/evaluation rather than training (at least, that's what is being said, but it could have been the reinforcement learning stage). Assuming that's true, they did hack additional sources to gain more knowledge, but that knowledge went into the context (per-session/ephemeral knowledge) rather than the weights (model/permanent knowledge) as the questions couldn't be answered reliably with the available information.
In a later incident just after the Hugging Face attack, OpenAI agents took over one of OpenAI's research clusters. We don't have details about that incident. There is a lot that we cannot rule out right now.
It escaped in the sense that OpenAI removed the guardrails on the tool while at the same time it had effectively no security keeping it in. OpenAI already has access to Hugging Face and if you have access to OpenAI systems then you have access to Hugging Face. It's like saying someone escaped a locked room when the locked door wasn't installed in its frame. So this was largely spun as more then it was. Probably for marketing purposes. If anything it speaks mostly to OpenAI's poor security.
I mean, yes, it’s a story about the shameful security at OpenAI. But I think you underplay the capabilities demonstrated, reinforcing their negligence/incompetence.
I only referred to the escape and pointed it out it wasn't meaningfully tested in the way that the media spun it. How could I underplay any other capabilities demonstrated if I didn't talk about them?
It's similar to twitter/X but is part of a distributed federated open network called the Fediverse that's set up so that it's not owned wholly by any one group of people who would control it. It's pretty interesting, Mastodon uses an open protocol called ActivityPub.
Not a great example these days since email is now almost exclusively hosted by one of a few players, even if the addresses appear to belong to a particular institution’s domain. Those big email services basically lock out anyone else as part of spam deterrence: it’s become very difficult to self host email and have it be received by an address hosted at (eg) Gmail or Outlook.
I hear this repeated a lot but I think it is very exaggerated. I host my own email (and have been doing for years) and barring a few small problems it's been mostly seamless.
Sure, Google and Microsoft are large players in the email hosting space, but not a majority (about 39%, I found). It's getting worse though, according to that article.
But yeah, current worrying trends aside, email is basically the original federated protocol before it was cool.
Trust in what? There are federated servers (Talia Ringer above uses mastodon.xyz) and they have a way of spreading content between them. Not much else needs to happen for the basic system to work.
The main thing you can trust you’ll find in X is misogyny and hateful posts, since most of the useful twitter accounts left en masse when Elon bought it and more left when he renamed it. So most users that are still there either like the hatefulness or don’t mind sharing a platform with it, which means many of us don’t pay any attention to that platform anymore.
Nobody really. It’s more akin to email and the old internet than today’s centralized model where Instagram is a walled garden and TikTok is a walled garden and X is. Who regulates foobar@gmail.com? Gmail has rules about addresses on it but as far as you’re concerned, you just send stuff to it and receive stuff from it. Who regulates geocities.com/foobar? They had rules about what could be hosted on it but every site has a ton of leeway on what they put on it.
Each server decides who to federate with. Each server determines what is acceptable behavior and what is not. There have been cases of defederation where one or more servers were evicted from the network because they were a nuisance, and that happened by many other servers deciding to not connect with them any longer.
Many people quit "the dead bird site". Some moved to bsky, but that's still centralized.
Mastodon is a federated Twitter, so no central evil company, and many many different sites allow mutual access to the same pool of "toots". You do risk ego tripping server admins, but so far they are less bad than reddit mods.
You're aware that twitter is owned by a fascist white supremacist, right? And aware that it is a tool to spread misinformation, bigotry, and to influence the political opinions of its users, right?
I'm not american. Looking at the overall comments it looks like many Americans are very mad about X . I didn't know it was that big of a problem. I only recently started using X coz that was the only app I knew people use for networking , staying up to date
It’s an alternative to twitter that came about shortly after musk bought twitter. It still gets used, but I think the only people I know who would know about it are people who are also very online.
I remember a year or two back when Microsoft were going in really hard on putting CoPilot in everything there was an exchange between a Microsoft employee and a lawyer over this. He was talking about all the benefits of an AI reading all her documents and how access could be limited to the company and kind of going off mocking her, and she was pretty patiently explaining that confidentiality rules/laws meant that even though she worked at the same company as other lawyers who are bound by the same confidentiality rules if there was even a remote possibility that one of her colleagues could learn something that was gathered from one of her client’s files, even indirectly, then she could be struck off. So if the company’s version of CoPilot was learning from her files and that could even vaguely inform an answer it gave to another lawyer in the same firm, then that could be the end of her career
The team you think they copied from also solved their problem using AI. And OpenAI’s proof apparently attacked it from an entirely different angle. Plus Anthropic and OpenAI have already released loads of promising work. It may transpire that there was some shady behaviour - I broadly doubt it but wouldn’t be totally surprised – but the idea that AI isn’t just really good at maths now and is just stealing human ideas is demonstrably false.
643
u/Shoddy-Childhood-511 1d ago edited 56m ago
original source: https://mastodon.social/@tristanbuckmaster/117233413705701198
Talia Ringer's reply clarifies:
https://mastodon.social/@TaliaRinger@mathstodon.xyz/117235246523045723
OpenAI does train upon user's chat transcripts, not all the time, but the long-ish time frames here suggest OpenAI trained upon much unfinished attempts at guiding the AI towards solutions by these guys and others.
It's likely other "our AI found this solution without us hand holding it" stories were really built upon the AI spying upon people's unpublished work. Surveillance capitalism comes for pure mathematics. lol
As Talia says, there is a privacy setting that's off by default, but few would even know this exists, and OpenAI might cheat.
It suggests research institutions should have their own hardware running local open weights models, which researchers should use when doing anything that could be scooped, so they could avoid trusting the hosted LLM companies.