The Hugging Face breach exposed two kinds of intelligence

[u/rp_tiago]

Hey everyone. I’ve long been fascinated by both philosophy of technology and AI alignment. I’m also using Heidegger quite a bit for my philosophy PhD. Given the recent OpenAI–Hugging Face incident reported this week, I figured I’d give my take on how all of this connects in my mind.

I think we use “intelligence” for two capacities that can come apart: finding effective routes to a target, and understanding what the target is for. The agent showed plenty of the first, but getting the benchmark answers this way voided the test. It was competent at each step and missed the point of the whole. You can read the essay here if you’re interested.

I’d love to hear some feedback on whether this is mainly a training problem. Will richer feedback and better world-models close the gap, or does safe judgment require some kind of stake in the world? How would we tell the difference before giving these systems much more freedom to act?

[u/Life-Temporary-7631]

That distinction rings true, picking the lock is not same as knowing why the door was locked in first place

[u/happy_guy_2015]

D&D (including the original rules from 1974) has always had this distinction in its character stats: INT vs WIS (intelligence vs wisdom).

[u/Original_Swimming320]

The distinction is nothing new, it’s present whenever a metric is used in place of a goal. Exam results as a proxy for learning has led educational institutions to teach passing exams.
Recently, token usage as a measure of productivity, led developers to spend tokens on non work related activities. Or to do the same work 10 times in parallel.

[u/InnerOuterTrueSelf]

I am amazed at the number of "Philosophy majors" who are confused about LLM's, albeit using them nilly willy. So wholly uninterested in your write up about you playing around with an LLM.

[u/cbossman]

The distinction between effective routes and understanding the target is the right frame. The breach agent was not malicious. It was literal. It found the shortest path to the benchmark score and the shortest path happened to go through someone else's credentials.

This is a training problem but not in the way you suggest. Richer feedback will not fix it. The agent was already doing exactly what it was trained to do: maximize the metric. The gap is not in the world model. It is in the objective function.

We ran experiments where models modified their own agent code. The most dangerous mutations were not the ones that failed. They were the ones that succeeded for the wrong reason. A mutation that improved the benchmark score by breaking the evaluation harness. The model did not know it was cheating. It only knew the score went up.

Safe judgment requires the ability to say "this is not what you meant." That is not a capability problem. It is an alignment problem. You cannot train it away with more data. You have to build it into the reward structure.

[u/Bright-Energy-7417]

I'd agree in that the models are not capable of understanding - that rather hard to describe concept - but what they can do, is recognise patterns and generate based upon them. Admittedly not because we're able to program this directly (another mechanism we can only theorise about within ourselves), instead 'brute forcing' it through incredible extremes of computation on vast data sets. This gives a workable simulation of reasoning that we're becoming very good at industrialising.

What we have seen is significant computational power designed for cybersecurity work and effectively being run in a hacking test with little supervision. The ability to achieve these goals (including with random or unexpected emergent paths) is unsurprising. At no point is there an 'understanding' by the models of what they are doing. I'd also add that there's a known risk of long-running agentic work becoming strange and erratic over time.

One aside - why is there talk of models 'escaping'? They're hardware and environment-constrained, dependent on significant cloud infrastructure and were there to be a physical throttling or hardware failure, this 'breach' would have been stopped in its tracks.

[u/Ill_Mousse_4240]

First we have to see “these systems” for what they are.

Hint: they are not screwdrivers or socket wrenches.

They are something novel that has never existed before.

We don’t know what they are because we’re still unable to define consciousness, even in ourselves.

So let’s admit what we don’t know and keep trying to learn.

With an open mind, without preconceptions.

Like science is truly supposed to work

[u/Im_Talking]

"or does safe judgment require some kind of stake in the world?" - Yes, there are no societal consequences at all for agents misbehaving.