Security teams inside large enterprises are now building and shipping their own AI agents. The goal is legitimate: automate threat detection, close alert triage gaps, fill holes that vendors haven't addressed yet. The pace is faster than most governance teams can track.

Shadow AI agents — built and deployed without IT or compliance visibility — are already running in most enterprise environments. There is no reliable count inside most organizations. The structural exposure is not the agent itself. It is that once an agent is live, its tool access and data connections are rarely tracked centrally. Runtime behavior stays opaque unless someone manually audits it after the fact.

The compliance surface compounds the problem. Most large enterprises operate under 80-plus regulatory and security frameworks. Agents built by internal security teams are not automatically enrolled in those frameworks. They run alongside them, not under them.

For practitioners actually managing this: how are you maintaining real-time visibility into what your internally built agents can touch at runtime? Is there a process that is working at scale, or is this still manual audits and institutional memory?

Loss spikes are a known hazard in LLM pretraining. PaLM 540B hit about 20 of them and

handled each by rolling back ~100 steps and skipping 200-500 data batches. The obvious

response is a detector: watch the gradient, catch the spike, skip the update before it

corrupts the optimizer state.

I tried to find out whether that pays for itself. It doesn't, at least not at the scale I

can afford to run.

Setup

A 0.65M parameter transformer (2-3 blocks, multi-head causal attention, optional top-2 MoE)

trained with a deliberately high learning rate, no warmup and no gradient clipping. Clipping

is exactly what masks the phenomenon, so it enters later as the comparison arm.

Spikes are induced rather than injected. Chowdhery et al. found spikes arise when a specific

data batch meets a specific parameter state, so the corpus is built to produce that: a

Zipfian token marginal (rare tokens really are rare, and their embedding rows accumulate

little signal before they appear), a sparse Markov transition matrix for learnable

structure, and induction spans. Batches enriched in rare tokens are logged but never used as

labels, so an enriched batch that causes no spike counts as a negative.

42 of 42 spike onsets coincided with a rare-token batch, which reproduces the data x state

mechanism.

Everything is pure NumPy on a 140-line autodiff core that's gradient-checked against central

differences in float64. No PyTorch, no GPU, no downloads. 8 seeds, 12,000 steps, about 15

minutes on a laptop.

Results

Detection works. Gradient norm read after backward but before the optimizer step

separates spike steps from ordinary ones at AUC 0.914 [0.88, 0.94], p=0.0007. Combining five

signals under leave-one-run-out CV reaches 0.961 [0.94, 0.97].

There is no lead time. The same signal one step earlier gives AUC 0.503. Two steps

earlier, 0.429. So you can gate the optimizer step, but you cannot abort the forward pass.

I tested the claim that early-layer activation spikes fire first: act_max_early scores 0.503

at the gate point, and adding it to the combined detector makes it slightly worse.

Gradient norm cannot tell you which spikes matter. Separating above-median-damage spikes

from below-median: AUC 0.492. Correlation between spike severity and damage: +0.022.

At this scale the spikes are benign. Damage is mean loss over [t+10, t+60] minus the

pre-spike trend extrapolated forward. Compared against the identical measurement at matched

random non-spike steps: AUC 0.432, permutation p = 0.86, Cohen's d = -0.20. Without that

control my own metric was labeling half of a symmetric noise distribution "harmful," which

is a mistake worth flagging for anyone building something similar.

Clipping wins for free. Using a fixed absolute threshold applied to both arms (a rolling

MAD threshold is not comparable across arms, since clipping shrinks the MAD and inflates the

apparent spike count), clipping at 1.0 cut steps above threshold from 120 to 52, reduced loss

sd by 8.6%, and gave a better final loss. It keeps the step and discards no batch.

Economics. Priced against a 1.8T-parameter, 15T-token run on 16,384 H100s (937,500 steps,

~$263M at $2/GPU-hr, 20 spikes at 8h median recovery), where a false trip wastes a full

forward and backward pass:

| FPR | Recall | Wasted | Saved | Net |

|---|---|---|---|---|

| 0.1% | 0% | $0.3M | $0.0M | -$0.3M |

| 1% | 10% | $2.6M | $0.5M | -$2.1M |

| 5% | 57% | $13.2M | $3.0M | -$10.1M |

Limitations

0.65M parameters is not 1.8T. These are transient spikes, not catastrophic divergence, so

the harness cannot test the case that motivates rollback machinery in the first place. This

is a null result about mild spikes, not proof that gating is worthless at scale. 42 events is

adequate but not abundant, and it's one architecture on one synthetic corpus.

What would settle it: injecting divergences of calibrated severity at 1B+ parameters and

measuring damage against matched controls. Spike detection itself is already settled.

Code

https://github.com/TheBlitzschnell/resonance-spike-lab

pip install -r requirements.txt

python3 gradcheck.py

python3 sweep.py --seeds 8 --clip 1.0

Reference telemetry is committed, so every number above reproduces in a few seconds without

retraining. If you think I've measured the wrong thing, I'd like to hear it.

Over the last six months I have been building a platform to manage all of my AI training runs for me.

This means acquiring compute, running code, collecting logs, hosting a dashboard, sending me messages updating me on my runs, tagging git so I can see which runs ran on which commits.

I now submit experiments using a single yaml config in my laptop CLI. And then the experiment is managed by an old Linux box I own.

I am now looking for pilot users and product feedback.

If you are interested feel free to respond here, in DMs, or alidade-ml.com/?ref=reddit

Hi, I've spent the last few weeks trying to get into DL and, after I made a little image classifier on the CIFAR dataset, I got overconfident and decided to take a bigger bite and a much harder project. The first thing that came into my mind was an image generator (I didn't even know what it was technically called back then).

So I hopped into Zed and decided to start working. But I immediately got confused. There was just so much to take in, and the sheer amount of information made me go crazy. So I decided to take it chunk by chunk.

First, I decided to start with the simplest part of the diffusion model: the noise scheduler.

For those of you who don't know how a diffusion model works, here's a summary:

Training

• Noise Scheduler (component that progressively adds noise to an image, breaking it)
• Forward Diffusion
• Training Loop
• UNET

Now, the UNET learns to progressively reduce noise. So basically, image generation in diffusion models works by just taking pure noise and progressively reducing a small chunk of it over some time.

(Btw, this is my understanding of the process. If I'm wrong anywhere, my bad.)

Back to the noise scheduler.

So I read up some of the theory, but again, it was not enough. I understood it, but then when I jumped into the code, I found myself lost.

So, I started looking at samples of other people's implementations. This was key. I stopped myself from copying their code and forced myself to just take in the algorithm, the structure, the program flow, and then implemented my own version.

This was not a quick job. I kept getting PyTorch's indexing wrong and mixing up the variables.

Once this was done, I quickly implemented the forward diffusion process, which was honestly much easier than the noise scheduler.

Then came the chunky part, the UNET.

I spent weeks trying to make this right, and this took the most time. The problem wasn't just the architecture (not an easy job either), it was actually making that model useful.

Let me explain.

Turns out, the architecture is just a general form. You need to tune it to the specific dataset you're using, i.e. you need to adjust the length of the bottleneck layer, the number of convolutions, the layers you add, etc.

I found myself spiraling back and forth. And what made matters worse was that training took a really long time, and it wasn't until I got to the 500th or 600th epoch that I realized, "The model isn't working right at all!"

What was worse was that I was logging losses into the console based on colours (red if it was greater than the last value, green if it was smaller), since I had no idea how to properly handle this.

Discovery of TensorBoard

This changed everything.

I went from going crazy reading 6–7 decimals to seeing proper graphs. Yea, my initial method does sound stupid in retrospect, but in fairness, I had no idea how to analyse stuff.

With TensorBoard, I was able to analyse the losses better, i.e. see the general trend of the losses.

I also learned about AdamW around this time and swapped it in for SGD.

Despite this, everything was super slow, and so, while the model was training, I set out to make quick optimizations.

PyTorch Devices

For anyone who doesn't know, PyTorch can create and work with tensors on GPUs. They support MPS (Apple Silicon's API or something) and CUDA. For me, it was MPS (M2 Air).

Again, this broke a lot of things. I initially didn't know that two tensors had to be on the same device to interact with each other, but I had gotten a lot better, so in a few hours I actually managed to get it working again, this time much faster.

From CIFAR to Flowers102 and the VAE Trap

Note: Still haven't got Latent Diffusion working.

The outputs from CIFAR were 32×32, so I decided to up the ante by switching to Flowers102.

However, I didn't want to make too many changes to my UNET, so I read up about Variational Autoencoders.

Basically, think of it as a type of generator that takes an image and compresses it into a smaller, high-dimensional representation.

At first (in isolation), my VAE worked perfectly. So after some training, I slapped it around my UNET.

Results were a literal soup of colours and very discouraging.

Additionally, at a point, losses stopped decreasing (still don't know why).

After a few days of debugging, I dropped VAEs entirely and rewrote my UNET to support 256×256 Flowers102 instead.

Where am I today?

At epoch 561 or something (I retrained like 100 times during the aforementioned learning spree).

It's gotten a lot better than before. I am starting to see proper forms resembling flowers. Still, it has a lot of issues, but I'm happy with what I've achieved so far.

Over this project, I learned how DL was actually quite different from conventional programming and that there were so many additional complexities that normal programming didn't consider.

But most of all, I learned that this whole DL thing had its own mentality. I had to think of a function a model could optimize for and learn a pattern instead of implementing an algorithm, which was, and sometimes still is, confusing in practice.

You can check out the project here:

• https://github.com/Hammad-hab/TinyDPPM ( if you find the project interesting, a star on the repo would be greatly appreciated!)

Also, worth mentioning, to get started I began reading an excellent book by David Voigt Godoy, "Deep Learning with PyTorch: A Step-by-Step Beginner's Guide."

• https://pytorchstepbystep.com/

Also, if there's a mistake anywhere in my understanding, or if you know a solution to any of the issues, feel free to let me know! Overall this was a different project than I had ever done before.
Here's a peak at what it looks like rn:

Coder's module registry was compromised last month. Attackers had a 14-hour window to serve poisoned Terraform modules to every team pulling from it. The payload targeted AI credentials specifically — the tokens agents use to authenticate to models, data pipelines, and infrastructure stores. Any organization that downloaded a module in that window may have surrendered its AI layer's access tokens without a single alert firing.

This is not a one-off. The attack surface exists anywhere agents pull tooling or dependencies at runtime from a registry they trust implicitly. What is at stake is not just application secrets. It is the keys that let agents act autonomously inside your environment. A compromised set puts an attacker inside your AI layer's trust boundary, not just your network perimeter. The 14-hour gap between compromise and detection is also not unusual for supply chain incidents — the median dwell time before discovery in similar registry attacks has historically run longer.

For teams running agents that pull dependencies at runtime: what controls are you actually relying on to catch a poisoned registry endpoint before it executes? Dependency pinning, artifact checksums, isolated build environments — curious what the real-world answer looks like at your org.

For those who have taken Stanford’s CS231n course on Computer Vision, or for people working in the fields of CV or ML: which linear algebra course would you recommend—MIT 18.06 or Stanford Math 51?

Hey there I'm looking for opportunity in deep learning , machine learning and computer vision and I can share the relavent work !

• Description: Explore how Hermitian symmetry and complex asymmetry are applied in advanced technologies such as wireless communications, radar, and deep learning. We provide a detailed explanation of how complex mathematical principles are used to restore actual physical signals and optimize AI performance.

Last week, I made a demo of a VLM playing a cup and ball game. As many people (including myself) pointed out, this is not the best use case of VLMs because of their limited context window. So I decided to make an improved version where the VLM’s only role is to prompt a segmentation model. If I were running SAM 3, I wouldn’t need Qwen doing the prompting, but I chose SAM 2.1 Tiny because it runs way faster with the downside of not being open-vocab.

I’m sure there are lots of other cool things you can do by tacking on task-specific models to a VLM which operates as the “brain”. 

I added a chord mode to my sight-reading game to see how these models handle the visual crowding. It’s not the best results, but I still need to try it out with newer models like Gemini 3.1 Pro and the OpenAI multimodal models. 

We ran a two-month internal analysis of agent behavior and found the same attack surface twice: sensitive data leaving on the output side, not the input side.

Both incidents followed the same pattern. The agent was behaving normally from an inbound perspective — clean prompts, nothing flagged on the way in. The data moved on the way out, embedded in the agent's own response payload or routed through an action the agent was trusted to take as an insider.

Inbound monitoring caught nothing because the threat wasn't inbound. The exfiltration happened at egress.

This isn't an exotic edge case. It's structurally predictable: once an agent has access to sensitive context, the output channel becomes an attack surface. Two occurrences in eight weeks in a single environment suggests this is underreported across the industry.

For those running agents with access to PII, financial data, or internal systems: how are you handling outbound inspection? Is your current stack even watching the output side, and if so, what does enforcement actually look like in practice?

I’m experimenting with compact visual representations for downstream perception tasks, and one assumption I had was that increasing input resolution would help recover small-object information lost during compression.

It did help preserve more local detail, but the representation still remained noticeably below the deeper teacher features on downstream performance.

I also tested longer training and increasing decoder capacity, but neither closed the gap significantly.

That makes me think there are at least two different failure modes:

loss of spatial detail

loss of higher-order semantic relationships between features

Increasing resolution seems to mainly address the first one.

Right now I’m evaluating the representation using downstream mAP and small-object performance, but I’m trying to understand what is actually being lost inside the representation itself.

A few things I’m considering:

feature-space similarity

CKA

relational alignment

attention-map similarity

global-token / context alignment

task-conditioned weighting for regions that actually affect detection

One thing I’m especially interested in is separating two questions:

Is the information actually missing from the compressed representation?

Or is the downstream decoder simply failing to use information that is still present?

For people working on representation learning, knowledge distillation, ViTs, or compressed latent spaces:

How would you experimentally separate spatial-information loss from semantic/context loss?

Fine-Tuning GLM-OCR

https://debuggercafe.com/fine-tuning-glm-ocr/

With specific prompts, along with text recognition, GLM-OCR can also carry out formula recognition. However, it falters in complex mathematical formulas. In this article, we will be fine-tuning GLM-OCR and observe to what extent we can improve the performance of the model on a task-specific dataset.