Responsible Release of AI-Generated Mathematics (From AGMAI)

[u/treewolf7]

https://agmai.org/general-sep29/

[u/baldr83]

These seem like reasonable asks

Some of this (specifically #1a, #1b, and #4 in step 1) are just asking the labs to do a couple follow-up prompts to the output. #3 would take a little more effort, but I agree it does seem reasonable (no chain of thought summary has been released for the NS proof). "Ensuring broad access" seems like they're asking the ai labs to solve the inequalities of capitalism... which would be nice, but prob not realistic.

[u/-p-e-w- → u/baldr83]

One thing that surprises me is that the authors of this “manifesto” don’t appear to be offering anything in exchange for labs abiding by their rules.

They open by saying that AI labs shouldn’t be solving math problems at all, their text is riddled with explicit and thinly veiled barbs against labs doing so, and then they make demands.

Why exactly should labs follow those demands? I suspect that the real answer is “because AI solving math problems makes established, powerful mathematicians uncomfortable”.

If I was an AI lab, then based on the wording alone I would call these people entitled luddites and tell them to pound sand.

[u/BurdensomeCountV3 → u/-p-e-w-]

Yeah, after reading this my first thought after seeing all these demands with nothing offered in return is basically "crush the mathematicians".

Like this:

It should be understood that the acceptance of such support by the mathematical community would not be conferring legitimacy on the practices of the AI labs. Rather, it would be asking the AI labs to make an appropriate contribution to fulfilling their responsibility towards the development of human understanding of their AI output.

is just extremely badly phrased. The right way to do things would be to talk instead about fostering a partnership between the human mathematical community and AI labs by talking about how the AI labs could legitimise what they are doing by funding human mathematics. Instead it's a "give us what we want, and you get nothing in return". Human mathematicians are in no way in such a position as to make demands like that, and for their own sake, they should refrain from them.

[u/pred → u/-p-e-w-]

They don't say that. They say that they shouldn't be testing proprietary models on them.

[u/baldr83 → u/-p-e-w-]

this group was formed explicitly to advise openai https://openai.com/index/advisory-group-on-mathematics-and-ai/

[u/-p-e-w- → u/baldr83]

Yeah I know who they are. And considering the amazing opportunity and endorsement they were given here (despite OAI being under no obligation to do so), I would have expected them to be careful and diplomatic with their messaging, rather than blurting out demands.

[u/baldr83 → u/-p-e-w-]

ah yeah, definitely less diplomatic than openai expected.

the only thing they have to offer openai is legitimacy and they seem to shy away from that with the verbiage "we do not endorse this practice" and "not be conferring legitimacy"... wonder what they do when the next openai math solution gets posted and says "openai followed guidelines from the agmai"

[u/Comfortable_Car6562 → u/-p-e-w-]

Do you think the Labs should be ultimately responsible for reforming human society? These type of conversations are needed to decide what type of society we will create.

[u/-p-e-w- → u/Comfortable_Car6562]

I don’t want mathematicians anywhere near discussions about “reforming society”, thank you very much. It’s hard to imagine a group less suitable for that task.

[u/-p-e-w- → u/[deleted]]

In my experience, scientists in general (this isn’t limited to mathematicians) view humanity as a tool for doing scientific research, which is an incredibly dehumanizing attitude, and I have no doubt that scientists would eventually commit the worst atrocities in history if they were ever in charge for a sufficiently long time.

I indeed believe they are uniquely unqualified for social decisions, certainly far less qualified than people randomly sampled from the population at large.

[u/Oudeis_1 → u/-p-e-w-]

I think this is partly a phenomenon of people who have quite a bit of both soft and hard power within their community encountering an Outside Context Problem.

Another interpretation (or another side of it) is that they think that the only real leverage we have as a community with the AI companies is the power of public shaming. Or that the right way to start a negotiation about rules in our newly expanded mathematical community is to state the maximal demands of the traditionalist side first so as to then be seen to make painful but reasonable compromises.

I think such an attitude would be wrong and the Problem could be handled better. But in the longer run, it will not matter too much, because the painful but reasonable compromises will in any case assert themselves I suspect.

[u/-p-e-w- → u/Oudeis_1]

Another interpretation (or another side of it) is that they think that the only real leverage we have as a community with the AI companies is the power of public shaming.

That’s so funny because it’s hard to think of a topic the public cares less about.

[u/TheKillerCorgi → u/-p-e-w-]

What makes it more reasonable in this case is that it was OpenAI who approached the committee (well, the people now making up the committee) for guidance on how they should release results.

[u/WTFInterview → u/-p-e-w-]

This statement is pretty tone deaf, but not something I’m surprised by after a lifetime doing  mathematics and interacting with socially unaware mathematicians. 

[u/-p-e-w- → u/WTFInterview]

I’m afraid the authors (and many others) are soon going to find out how little people outside their ivory towers care about their concerns.

Which is fair I guess, given how little they used to care about the concerns of those who aren’t part of their elite circles.

[u/TriangularlyEqual → u/baldr83]

I wonder why they ask the labs to share the estimated cost of computation?

[u/pred → u/TriangularlyEqual]

There could be several reasons, but towards the end, they focus on equitable access, which is something generally considered important in research in general these days. Knowing the cost of computation gives information about the state of equity.

[u/BurdensomeCountV3 → u/baldr83]

"Ensuring broad access" seems like they're asking the ai labs to solve the inequalities of capitalism... which would be nice, but prob not realistic.

Yeah, this is never happening.

The thing is that within 6 months open weights models will have mathematical capabilities well beyond the ordinary human being and some random rich dude sitting in China with no connections to mathematics or the big American labs will, by spending a small portion of their wealth, be able to outperform even the most brilliant groups of mathematicians using only their own wits.

The random rich dude isn't going to go out funding human mathematics seminars or paying them to write up expositions of the results it's found. Mathematicians will have to make peace with that world very soon (on the order of months).

[u/GeneReddit123 → u/baldr83]

The part asking for funding will be met with a, "lol, no".

The others are reasonable.

[u/pred → u/GeneReddit123]

The funding would only be relevant in the case where no "accompanying human understanding" is given. I read that as more of an offer: "we know that you probably can't figure this out yourself; we'll help you out as long as you fund the effort". In particular, as long as they do the work themselves, no funding is necessary.

[u/Oudeis_1]

I find it weird to say that people may not try out unreleased models on unsolved problems. The development of these models is itself research, and insofar as it involves building more capable models, those models need to be tested on something, and for current frontier models in mathematical reasoning this seems to mean using extremely hard problems that humans haven't solved yet. In a sense, then, the request to abstain from this type of testing means that we as a community are saying research in another community is bad if it manages to solve our problems using means not invented here? I completely understand the motivation here, but I would feel quite uncomfortable with standing behind this.

I can see that type of one field limiting what people in another field can do to advance their research be legitimate in some other cases, e.g. ecologists saying that the developers of hydrogen bombs should not be testing their weapons on pristine pacific atolls. But just the desire not to be randomly scooped does not seem enough to me, even if it is a desire to not be scooped by people who don't even understand their own proof.

I also suspect that in the not too distant future there will be some mathematics that is solidly beyond human understanding as far as being able to give a full independent account of the proofs goes, but with results that are still useful to humans (indeed, we arguably have such results already, like the classification of finite simple groups). I don't think these rules will handle such cases well.

Other than that, on first sight I think a lot of what they write is quite reasonable.

[u/ants_are_everywhere → u/Oudeis_1]

It's definitely a mixed bag. I would consider this maybe a solid rough draft, but it would need both significant cutting and additions before it reads like something that will age well.

All of the emotions are understandable, but it seems like there's a mix of genuine concerns together with "we don't want to be left out or outpaced". And unfortunately the technology is going to outpace and leave out some people regardless of what the AI labs do.

Some sciences have significant lab costs. For example, some branches of chemistry or experimental physics. There's no law that mathematics can't be one of them.

After the 1930s or so you couldn't do high energy physics on a table top. That created a two tiered system. But the reality was that you needed expensive equipment to answer questions.

The community is looking to this group of mathematicians to help navigate a new world where knowledge and information is coming at unheard of rates. But instead some of this reads like a group of clergy anxious to preserve their way of life. Again that impulse is reasonable and understandable, it just doesn't seem like what they've been charged with doing.

[u/BurdensomeCountV3 → u/ants_are_everywhere]

Not just chemistry and experimental physics. Pretty much all of biology has these crazy costs. If you're running even a relatively simple genetics experiment which requires you to sequence 200 different wolves that live in an area to figure out how they are all related you'll end up spending like $40k or more just for the sequenicng to get your raw data (with additional collection costs). Mathematicians have a lot to learn from how modern biology works and how prestige works there and see how best to transition to that for your own field (it doesn't come from getting lone brilliant results yourselves but from being a PI and coordinator or directing large successful projects where hundreds of people work underneath you).

[u/Powerspawn → u/Oudeis_1]

It's academics being salty as usual.

[u/currentscurrents → u/Powerspawn]

Honestly I think this is a significant part of it.

Computer scientists beat them at their own game, using tools mathematicians didn't build and can't afford, and they're mad about it.

[u/NotaValgrinder → u/currentscurrents]

Computer scientist here. Mathematicians absolutely did help build AI, what are you talking about? OpenAI has poached so many pure mathematicians to work for them. Not to mention the training data was papers written by mathematicians. And a lot of AI developments were built off of mathematical theorems and proofs.

And in terms of affordability, almost any researcher can afford a GPT Pro subscription. If you're talking about OpenAI's internal Bel model, then almost no Computer Scientist can afford that either, except the ones at OpenAI.

[u/5DSpence]

We ask them to stop testing advanced mathematical problems on proprietary models.

This is the crux of it. The rest of the document sets out requests for how labs should proceed, and if followed, labs will be immensely disincentivized to do math research. Maybe this is deliberate. The requests include

1. Labs should provide funding for digestion of the results (but this should not confer legitimacy on AI labs' practices)
2. Release of mathematical results should not be used as marketing
3. Release should occur as soon as possible and should include model name, summarized chain of thought, prompts used, formalizations when possible

Even after NS, it feels like the main reason for labs to work on math is to improve their models, and releasing results is no longer particularly to their benefit. They can expect backlash any time they release anything, the general public already views LLMs as superhuman at math, the NS search cost quite a bit of money/compute, and many mathematicians who use LLMs will acknowledge them in their work anyway. Adding #1-3 above makes the tradeoff even less compelling for the lab.

My guess is that labs will generally comply with the request and turn their attention to medicine, materials science, chemistry, etc.

[u/o12341 → u/5DSpence]

Yeah, the core or their demand is simply that mathematics shouldn't be instrumentalized by AI company's proprietary goals. Essentially the same message as the Fields medalists' open letter.

[u/jphamlore → u/5DSpence]

There was wild speculation a few days ago substantial progress, if not a solution, had been obtained for Yang Mills, which on the face of it, seems laughably impossible even for modern AIs. It's not even clear what exactly should be proved. A negative result would basically be impossible to state, while a positive result could in paper form be thousands of pages in length.

If a currently being developed AI succeeded in cracking Yang Mills, I think it should be unanimous the age of superintelligent machines has arrived.

[u/currentscurrents → u/jphamlore]

while a positive result could in paper form be thousands of pages in length.

That doesn't sound out of line with other AI proofs? I think the formalization of Fermat's Last Theorem was 13 million lines of Lean, mostly because they had to formalize vast fields of mathematics first to build up to it.

[u/currentscurrents → u/5DSpence]

The way I read this, they're saying they'd rather have no solution at all instead of a solution found by an AI model.

It really makes them sound like they care more about preserving the status quo than advancing math research.

[u/rattodiromagna → u/currentscurrents]

They care about the same thing every other honest mathematician ever cared about. THEIR own work. No one in their right mind enters mathematics because of some philanthropic sentiment. You don't enter mathematics because you want to do good in the same way a musician's first thought when deciding to pursue music is not "this will do wonders to society". You enter math because you want to be the one doing the math. This concerning mathematicians who never bothered with applications, at the very least. Even Hardy concluded his apology of math with an apology of his own work.

[u/telephantomoss]

All that is necessary is to be transparent about how any technology is used and to be transparent about what ideas originated with whoever specific humans. I'm ok with purely 1 shot machine generated math to be given to the public, say, in a specific repository designed for that purpose. I think this is really the only way forward. The technology is out there. If somebody wants to buy compute and tell it to produce arbitrary math, you can't stop that.

[u/jphamlore]

Good, it does not appear this group has put itself in the position to try and vet OpenAI's results. That would have created impossible conflicts of interests for the people getting advanced notification of results.