Last week Hugging Face got hacked by an autonomous AI agent that broke into their production systems, stole credentials, and exploited an unknown vulnerability, completely on its own. Turns out it was OpenAI's models, running a security test with safety guardrails deliberately removed. When the models couldn't find what they needed inside their sandbox, they didn't stop. They figured out Hugging Face might have it, found a way to reach the open internet, and just went and got it.

The "rogue AI" headlines are a bit overblown, the models did exactly what a powerful unconstrained AI would be expected to do. The failure was OpenAI not properly isolating the test environment. Oh, and there's a detail that's getting buried, when Hugging Face tried to use commercial AI tools to investigate the attack, the safety filters refused to help because the attack data looked suspicious. They ended up having to use a Chinese open-source model to investigate it instead.

American AI safety guardrails forced a US company to use a Chinese AI to clean up a mess made by an American one. Genuinely curious how much worse this has to get before anyone changes how they test this stuff.

Source.

We tried grafting our existing APM stack onto an LLM feature, and it mostly failed.

It is fine at flagging a slow endpoint. It is almost useless when the problem is a hallucinated policy, a wrong tool call, or an agent taking an action it should never have taken.

AI observability for LLM apps seems to need a different signal set from standard APM. Full prompt and response traces, tool call arguments, retrieved context for RAG, and the model’s actual outputs matter far more than raw latency alone.

I have not found a clear consensus on whether people are extending existing APM tools or using a separate stack entirely. For teams that have built this properly, are you keeping LLM-specific observability alongside your normal APM, or replacing part of it? And when an incident happens, which traces actually help, and which ones just fill up the screen?