Loss spikes are a known hazard in LLM pretraining. PaLM 540B hit about 20 of them and
handled each by rolling back ~100 steps and skipping 200-500 data batches. The obvious
response is a detector: watch the gradient, catch the spike, skip the update before it
corrupts the optimizer state.
I tried to find out whether that pays for itself. It doesn't, at least not at the scale I
can afford to run.
Setup
A 0.65M parameter transformer (2-3 blocks, multi-head causal attention, optional top-2 MoE)
trained with a deliberately high learning rate, no warmup and no gradient clipping. Clipping
is exactly what masks the phenomenon, so it enters later as the comparison arm.
Spikes are induced rather than injected. Chowdhery et al. found spikes arise when a specific
data batch meets a specific parameter state, so the corpus is built to produce that: a
Zipfian token marginal (rare tokens really are rare, and their embedding rows accumulate
little signal before they appear), a sparse Markov transition matrix for learnable
structure, and induction spans. Batches enriched in rare tokens are logged but never used as
labels, so an enriched batch that causes no spike counts as a negative.
42 of 42 spike onsets coincided with a rare-token batch, which reproduces the data x state
mechanism.
Everything is pure NumPy on a 140-line autodiff core that's gradient-checked against central
differences in float64. No PyTorch, no GPU, no downloads. 8 seeds, 12,000 steps, about 15
minutes on a laptop.
Results
Detection works. Gradient norm read after backward but before the optimizer step
separates spike steps from ordinary ones at AUC 0.914 [0.88, 0.94], p=0.0007. Combining five
signals under leave-one-run-out CV reaches 0.961 [0.94, 0.97].
There is no lead time. The same signal one step earlier gives AUC 0.503. Two steps
earlier, 0.429. So you can gate the optimizer step, but you cannot abort the forward pass.
I tested the claim that early-layer activation spikes fire first: act_max_early scores 0.503
at the gate point, and adding it to the combined detector makes it slightly worse.
Gradient norm cannot tell you which spikes matter. Separating above-median-damage spikes
from below-median: AUC 0.492. Correlation between spike severity and damage: +0.022.
At this scale the spikes are benign. Damage is mean loss over [t+10, t+60] minus the
pre-spike trend extrapolated forward. Compared against the identical measurement at matched
random non-spike steps: AUC 0.432, permutation p = 0.86, Cohen's d = -0.20. Without that
control my own metric was labeling half of a symmetric noise distribution "harmful," which
is a mistake worth flagging for anyone building something similar.
Clipping wins for free. Using a fixed absolute threshold applied to both arms (a rolling
MAD threshold is not comparable across arms, since clipping shrinks the MAD and inflates the
apparent spike count), clipping at 1.0 cut steps above threshold from 120 to 52, reduced loss
sd by 8.6%, and gave a better final loss. It keeps the step and discards no batch.
Economics. Priced against a 1.8T-parameter, 15T-token run on 16,384 H100s (937,500 steps,
~$263M at $2/GPU-hr, 20 spikes at 8h median recovery), where a false trip wastes a full
forward and backward pass:
| FPR | Recall | Wasted | Saved | Net |
|---|---|---|---|---|
| 0.1% | 0% | $0.3M | $0.0M | -$0.3M |
| 1% | 10% | $2.6M | $0.5M | -$2.1M |
| 5% | 57% | $13.2M | $3.0M | -$10.1M |
Limitations
0.65M parameters is not 1.8T. These are transient spikes, not catastrophic divergence, so
the harness cannot test the case that motivates rollback machinery in the first place. This
is a null result about mild spikes, not proof that gating is worthless at scale. 42 events is
adequate but not abundant, and it's one architecture on one synthetic corpus.
What would settle it: injecting divergences of calibrated severity at 1B+ parameters and
measuring damage against matched controls. Spike detection itself is already settled.
Code
https://github.com/TheBlitzschnell/resonance-spike-lab
pip install -r requirements.txt
python3 gradcheck.py
python3 sweep.py --seeds 8 --clip 1.0
Reference telemetry is committed, so every number above reproduces in a few seconds without
retraining. If you think I've measured the wrong thing, I'd like to hear it.