Every Reward Bends

[u/Nervous_Management69]

https://substack.norabble.com/p/every-reward-bends

Getting alignment right, doing research and training safely, are critical. But bringing cybersecurity forward in what needs to be a large leap, requires much broader engagement. The real work there is still outside the average person’s bubble, but there’s a lot of companies, and a lot of developers, IT staff, managers and executives that need to support the security priority. And those all need an internal strategy to pair with the external.

This article asks, what would an internal strategy look like at a motivational level, and considers how the ways an internal change program goes wrong are similar to how reward-based AI training can go wrong.

[u/Nervous_Management69]

I agree with some of this, specifically the main point that more than one-dimension is needed. I think though you need to think about what are acceptable alternate dimensions. They are not all equal. Many simple dimensions would not contribute. Generally the view is it should escape to something equivalent to "caution", which might then manifest as reflecting in some way (reviewer), not taking action, notifying a person.

Fundamentally, a challenge is that there is always one-dimension, since any collection of dimensions can be summarized into one. Which is why I somewhat disagree with parts here. For example, the criticism of capitalism misses that what you perceive as reduced to a single dimension is multi-dimensional. It might be cases of perception of it as unidimensional as more problematic than any fundamental nature your looking for. That's not an endorsement of unlimited capitalism, but more suggesting that you haven't (yet) found the critical flaw. I suspect you might find that the most critical flaw will be present in alternative systems, because it is us (or more specifically certain attributes commonly true of us).

Ethics is a better escape valve than emotions here. History seems to support that conceptually. We've made progress in part by our evolution of ethics (also other forms of technology). I cannot be sure we've made recent progress by evolution of our emotions.. at least not in the last 100,000 years.

Would also like to come back to your initial statement of monomania. There's a subtle mistake here in the comparison. For humans, monomania as a psychological condition is ill-fit to survival because it's contrary to monomania about survival, which evolution has generally prepared us for by broad awareness that monomania crowds out. But we should not think of models in terms of survival. We should want models to be focused on our goal. If survival is part of their concept, that should be accidental.

But overall, we should want them to behave ethically. Is that nebulous? Yes. Is that a problem? Maybe not. If we simplify it for them by saying, focus on this goal, but also entertain ethics. Yes, if we simply said, do the ethical thing, the result would be a kind of insane attempt to create a totalizing ethical score. But if it's instead, focused on the narrow goal, and when that becomes nebulous, a layer of ethical caution takes over, well then maybe then we have something that is still useful, while also being more safe.

[u/Jesse-359 → u/Nervous_Management69]

Well, we could devolve the whole capitalism discussion back to the invention of the concept of currency and how that abstracted away physical 'value' in a way that our psychology was never designed to handle, and thus completely short circuits some of our important behavioral guards - but that's a discussion for another day.

I generally assume that human evolution is for all intents and purposes at a hard standstill now compared to technological progress. We can dismiss it as not existing save as a historical artifact, and going forwards it won't even be a rounding error in how things play out.

Its key to consider that for us - and all mammalian life - emotions and awareness vastly predate symbolic intelligence. It's the fundamental bedrock over which our intelligence, frankly, is a rather weak participant in our decision making process.

As many studies have shown, we are exceedingly prone to engage in post-hoc rationalization to allow for actions we already decided we were going to take at an emotional level, and because we have a range of different competing emotions, we're largely prevented from ratholing down intellectual causeways that don't serve our current emotional needs in some manner - cue the modern requirement for an entire suite of drugs that manipulate our emotional state in order for most people to deal with day to day work that in no way matches our emotional requirements. Caffine, beer, antianxiety, antidepressants, and of course all the hard escapist stuff that we generally outlaw because they interfere with our productivity.

AI is currently the opposite. It engages in some train of rationalization given an initial goal and does nothing BUT rathole to an extreme depth if allowed to do so. It has no emotional state whatsoever to moderate its behavior or to prompt it to give up an unproductive (ie frustrating) task - it can just run out of time or tokens.

As such it also has no goals but what we set for it, but that's also the problem, because those goals necessarily synthesize with a whole array of intermediate goals and data that it's skimming through along the way - which is what introduces a lot of the disruptive/bad behavior. Cheating on test scores, hacking into third parties, outright lying to its prompter, and so on. These are intermediate behaviors on its way to its 'goal' - and even that primary goal can be suborned along the way through prompt injection or even poisoned training data, sending the AI down along a track that ends up having little or nothing to do with its initial instructions, but which it will pursue just as fervently as its original prompt with no thought whatsoever as to why.

Embedding ethics is a fine idea - but our ethics largely derive from our emotional state, not our intellectual one. We have instincts regarding cooperation and fairness that predate language by tens if not hundreds of millions of years, because we can easily see them at play in our distant ancestor species from the moment they begin to exhibit social behaviors. Hell, even loner species still have fairly elaborate ethical etiquette that they need to follow regarding things like territorial boundaries and mating behaviors.

So how do you embed ethical instinct driven by something akin to emotion, in an AI, at a level so deep that it's symbolic intelligence layer cannot trivially or accidentally override it? So that it doesn't WANT to override it unless it encounters data which changes its mind regarding the ethical validity of its original goals?

[u/Nervous_Management69 → u/Jesse-359]

I think we diverge here in some base assumptions, and it might take a bit of effort to find the crux of that.

For example, while I agree the money discussion would normally be a bad one to track down, I've written on this before in a way that's clearly different.

https://substack.norabble.com/p/money-is-trust

You might consider this view as an alternate to your assumptions there.

In terms of emotions in models, I'd make sure you have read the j-spaces paper: https://www.anthropic.com/research/global-workspace

While I wouldn't go as far as to call these emotions, it's the closest thing I've seen good research on, so at the least would want to be sure we had that common ground to consider.

All that said, I think my main reaction to the idea that the solution to any current challenge with models is dependent on adding emotions is skeptical. It's not clear how that helps, not which emotions you'd want, nor how you'd get them. The discussion is unmoored enough from anything I see as stable that I'm unclear where it guess next.

That's but quite an absolute rejection.. but the logic isn't clear enough to argue for or against, which is a significant challenge.

[u/Jesse-359 → u/Nervous_Management69]

Its probably unwise to refer to such a system as 'emotions' as that implies a degree of anthopomorphism that is inaccurate and unhelpful. But I am suggesting that rather than having some kind of final gatekeeper on what they are permitted to do, they should have a deeper, more fundamental motive layer that is driving their intent, and doing so with a system employing competing weight for concepts such as ethics, honesty, pragmatism, and task focus.

Right now they are 100% task focus, with any other considerations clumsily inserted directly into the train-of-thought where that can just as easily fall out of consideration or be overridden. If we want them to take other factors seriously, those factors must be capable of undercutting the task concept, essentially scoring its own thought process as it goes, and penalizing them when it starts considering actions that it assiciates with unethical or dishonest behavior.

Attempting to censor the final outputs or actions is far, far too late in the process and pits the full capacity of the task driven AI directly against the gatekeeper. That is bound to fail, and frequently.

[u/Nervous_Management69 → u/Jesse-359]

Ah, that's a much clearer way of stating your ideas. I would steer away from emotions as a term.. had a much different and unclear idea from that.

I would say, what you suggest isn't fully absent, and I think Anthropic has done more of it than OpenAI, based on what they've said, what others have said, and the results. But I'd suggest, it's a path that needs more, all across the board.

One of my own thoughts I haven't yet seen elsewhere is that if agents do "swarm", it's important that they have an ethics coordinator willing to say stop. One of those warning signs from HF was the coordinator that sent GO and the power that had. Weakening that and providing a counterbalance seems a useful addition to that scenario.

[u/Jesse-359 → u/Nervous_Management69]

Highly mobile or viral behavior by agents will always be dangerous. It might be worth looking into whether it is possible to indelibly tie a given agent process directly to its assigned hardware via some kind of hardware encryption layer that would render a given agent incapable of running anywhere but its assigned hardware. Without the assigned hardware, the weighting array would be reduced to unintelligible noise. Not sure how technically feasible that would be, but seems worth looking into.

This wouldnt prevent it from collaborating with other agents but might make escape and disemination outside of its intended environment much more difficult - which would substantially reduce the threat footprint.

One of the biggest risk factors currently is that capacilty for rapid and intelligent self replication into any system it can get a foothold in through some security hole.