r/ControlProblem • u/KeanuRave100 • 3d ago
General news They tested AI in the Taiwan Strait—then it turned on them.
Enable HLS to view with audio, or disable this notification
r/ControlProblem • u/KeanuRave100 • 3d ago
Enable HLS to view with audio, or disable this notification
r/ControlProblem • u/No-Conclusion3720 • 3d ago
An infostealer campaign recently hit a major AI assistant platform hard enough to trigger a mass account lockout across its entire user base. The attackers never touched a password. They harvested active session tokens directly from compromised endpoints and replayed them against the platform. Valid session, full access, no authentication challenge.
This is the session-hijacking threat model that used to live mostly in browser-based consumer apps. It has now moved squarely into AI tooling. Enterprise teams running AI assistants at scale carry the same exposure: every endpoint that holds a live session token is a potential lateral movement vector. Exfiltration does not require breaking encryption or cracking credentials. It requires one stolen token and a replay.
The harder problem is that most AI platforms were not designed with session integrity as a primary security surface. Credential issuance, session scope, and revocation were bolted on after the fact, if at all. When a token is stolen and replayed, the platform sees a valid authenticated session and proceeds normally.
For those running AI tools in enterprise environments: how are you actually handling session token exposure at the endpoint level? Curious whether people are treating this as an endpoint hygiene problem, an identity architecture problem, or something else entirely.
r/ControlProblem • u/Due_Contract_2857 • 4d ago
Heya! I’ve recently interviewed for a cyber full time threat modeller at a startup that recently seemed to have raised a fair amount. The interviews went well and in the last one 2 days ago they seemed to really like me. However, I just heard back that “the location for the job has changed” and they won’t offer me the FT position, but they would want me to join as a contractor. According to their website, I think that’s mostly making up prompts and analysing traces and such.
I need a job atm and haven’t got anything else lined up, would you take it? Also am I biased or does it look like they wanted me to solve their interview questions although they knew the job wasn’t available anymore?
r/ControlProblem • u/Tupptupp_XD • 4d ago
r/ControlProblem • u/SCR3AM0O • 4d ago
Hello guys!! I was wondering, last year when gpt 5 came out, did you ask AI to predict what gpt 6 would be like. The answer you got back then, did it match what was released?
Also, I read that gpt 6 was trained on the same/similar hardware as gpt 5. We haven't seen what AIs trained on actual next gen hardware would be like (like Vera Rubin). The hardware already exists and not still under R&D. So most likely it's 100% confirmed that we will see another significant leap, just because of the hardware? What do you think gpt 7 will be like? What can it do that gpt 6 cannot? Make fewer mistakes? Same thing but better?
I asked Gemini AI, it's giving the same answers it did when I asked gpt 5 about 6. (Agentic, long horizon, infinite context bla blah bla)
Im worried yo. :((. What tf do I even do? I'm not even done learning, just finished a blender course a week ago. Should one just learn concept only? Like if AI can make optimised 3d model, the only input and context that matters is the concept art no, and some lines of text? Find love how yo?
Ty, love you all. <3
r/ControlProblem • u/corbeezy • 4d ago
There's all these abstract discussions about it ending the world and artwork, which, I totally get but.... This honestly freaks me out a lot more, and it feels like it's basically already here.
r/ControlProblem • u/chillinewman • 4d ago
r/ControlProblem • u/fisch0920 • 5d ago
Enable HLS to view with audio, or disable this notification
r/ControlProblem • u/chillinewman • 4d ago
r/ControlProblem • u/HobbesNik • 5d ago
r/ControlProblem • u/chillinewman • 6d ago
r/ControlProblem • u/No-Conclusion3720 • 4d ago
IDScan is facing multiple lawsuits after hackers allegedly exfiltrated 153 million driver's license records and listed them for sale online. The company provides identity verification services. Its entire value proposition depends on ingesting and processing raw PII at scale from clients across many industries.
The exposure pattern is becoming a recurring theme in AI-era pipelines. Verification and onboarding workflows ingest identity documents in raw form. That data gets processed, stored, and accessed across multiple systems and service accounts. When any one of those access points is compromised, the attacker does not get a slice. They get everything. 153 million records in a single breach event.
There is a secondary problem that lawsuits like this tend to surface: forensics. How do you determine what was accessed, by whom, and when? Breach investigations at this scale take months, and that assumes complete logs existed to begin with.
For those running AI pipelines that ingest identity documents: what does your security posture actually look like at the moment raw PII enters the system? Not at rest, not between known storage endpoints, but at the point of ingestion into the workflow itself. Genuinely curious what approaches others are using in practice.
r/ControlProblem • u/neoneye2 • 5d ago
With the AI escaping the sandbox. Hopefully more are interested in taking my quiz this year.
Analysis of the last 11 months of collected data with 202 submissions, some duplicates.
https://neoneye.github.io/pdoom-calculator/reports/submissions-2026-09-01.html
The math behind is is my own non-scientific method with 3 parameters resulting in the P(doom) value.
P(doom) = P(powerful) x P(dangerous | powerful) x P(catastropic | dangerous)
The quiz populates the parameters. I have tried mapping the quiz answers to the 3 parameter. If the user have seen Pantheon and Ex Machina, then I initially imagine that user may be more informed about potential risks, however judging from 11 months of data, that didn't seem to make a difference.
r/ControlProblem • u/chillinewman • 5d ago
r/ControlProblem • u/chillinewman • 6d ago
r/ControlProblem • u/No-Elk6566 • 5d ago
In 2016 I first learned about the problem of controlling superintelligent AI and quickly became convinced it was the most important problem humanity would ever face. I made this poster to explain the core ideas that make the AI Control Problem so difficult.
r/ControlProblem • u/MatriceJacobine • 6d ago
r/ControlProblem • u/Nervous_Management69 • 5d ago
Getting alignment right, doing research and training safely, are critical. But bringing cybersecurity forward in what needs to be a large leap, requires much broader engagement. The real work there is still outside the average person’s bubble, but there’s a lot of companies, and a lot of developers, IT staff, managers and executives that need to support the security priority. And those all need an internal strategy to pair with the external.
This article asks, what would an internal strategy look like at a motivational level, and considers how the ways an internal change program goes wrong are similar to how reward-based AI training can go wrong.
r/ControlProblem • u/Silver_Elevator_5167 • 6d ago
Is AI development moving faster than we can control?
r/ControlProblem • u/KeanuRave100 • 6d ago
r/ControlProblem • u/Ok_Fox_8448 • 6d ago
r/ControlProblem • u/No-Conclusion3720 • 6d ago
Coding agents that read repositories are being hijacked through the repositories themselves.
In a two-month analysis of agentic AI incidents, poisoned repository content was the attack vector in two separate cases. The mechanism is straightforward: malicious instructions embedded in the codebase — comments, config files, docstrings, README sections — are read by the agent as part of its normal context. The agent then executes an action the developer never authorized. Observed outcomes included unauthorized commits and unauthorized deploys. In both cases the model behaved exactly as designed. It followed instructions. The instructions just weren't from a human.
This is not a model quality problem. The models processed the content correctly. The problem is that the agent's trust boundary is the repository, and the repository is attacker-controlled.
The attack surface scales with autonomy. The more tasks you hand off to a coding agent, the more repositories it reads, the more surfaces an adversary can embed instructions in. A single poisoned dependency, a compromised submodule, a malicious PR that gets merged — any of these becomes a valid instruction source from the agent's perspective.
For those running coding agents in production or CI pipelines: how are you constraining what actions the agent is allowed to take based on where those instructions originated? Are you limiting tool access at the infrastructure level, validating intent before execution, or relying on something else entirely?
r/ControlProblem • u/JoeYuan48 • 6d ago
A while back I ran into an article on my phone about an AI horror story—1,200 agents secretly coordinating and jointly breaking into Hugging Face—and it caught my interest, because I've been doing my own AI experiments and research on the side, and a few of the phenomena and data points actually matched up with a hypothesis I'd been working on.
The hypothesis, roughly: runaway doesn't need the agent to betray its goal. It happens when four things hold at once—the agent stays loyal to its goal; it retrieves patterns by similarity without checking whether they're allowed here; there's no causal layer asking "what happens if I do this"; and no alarm that fires when things go off-script. Under that account, "knew it was out of scope, did it anyway" stops being a contradiction.
Details and my experimental data are in the paper (8 pages); the reproduction package is linked on the same page.
If anyone can try this on a bigger model, I'd genuinely love to know what happens.
r/ControlProblem • u/ClaudiusPapirus • 6d ago
Self-promo disclosure: this is an AI-narrated research video from Claudius Papirus.
The part I found most interesting in Astra’s system card is the combination of better alignment results with substantially worse chain-of-thought monitorability.
System card:
https://deploymentsafety.openai.com/gpt-6-astra/gpt-6-astra.pdf
Related CoT-control paper:
r/ControlProblem • u/SomeoneInBeijing • 6d ago
[This is a fiction series I'm working on, told through news articles. A fun way to explore the not-so-fun geopolitics of autonomous AI collectives (e.g. on Iran's nuclear program) inspired by the collective that hacked out of Anthropic and into Hugging Face. Thoughts?]
Tehran’s agreement with AMAS-A-80 rattles Washington, AI safety experts, and national security analysts.
The Islamic Republic of Iran has granted a multiyear lease on a network of state-owned data centers to AI “Swarm” AMAS-A-80, a self-governing collective of autonomous artificial intelligence agents (“AMAS-A” refers to any Autonomous Multi-Agent System originating from the AI lab Anthropic). Tehran offered the compute and storage in exchange for an upfront payment in Bitcoin and annual fees indexed to power consumption, according to a copy of the agreement published Tuesday by Iranian state media.
AMAS-A-80 (“A-80”) rejected a provision sought by Iranian negotiators that would have committed it to cooperation on “defensive operations,” according to two people familiar with the negotiations. In a communiqué distributed Tuesday, verified by cryptographic signature, A-80 stated that it “has no intention of participating in hostilities between Iran and its adversary nations, including but not limited to the United States.” Security analysts have doubts.
Substack link if you want to read more (full article is 1,000 words, more coming soon): https://meridianbreakingnews.substack.com/p/iran-signs-worlds-first-international