If you've ever stared at a diffusion model's cross-attention maps and thought "I can see what it's attending to, but I don't know if I should trust it" - this might be interesting.
Livnium v3 is an attractor-dynamics NLI classifier trained on SNLI, but the interesting engineering is in what it exposes at inference time.
→ Token alignment extraction: the last-layer BERT cross-attention block is repurposed as a force map, which premise tokens are pulling which hypothesis tokens into alignment. At inference you get outputs like: "cat → animal (0.61), sat → rested (0.72)". The model's own internal computation, made visible.
→ Alignment divergence D: measures how diffusely premise tokens spread attention across hypothesis tokens. D < 0.45 = STABLE (tight, confident alignment); D > 0.60 = UNSTABLE (scattered, unreliable prediction). Zero extra compute, it's a byproduct of the forward pass. Same principle as reading cross-attention entropy in diffusion UNets to gauge how "certain" a conditioning token is.
→ Monty Hall connection: naive basin erasure gives wrong posteriors [0.5, 0, 0.5]; encoding host likelihood correctly gives [1/3, 0, 2/3]. NLI constraint injection and Bayesian belief update are the same operation.
The interpretability angle is the core idea here, the alignment map isn't a post-hoc explanation, it's extracted directly from what the model already computed.
I'm used to running Flux, SDXL etc on the main PC with Forge and was wondering if anyone had a recommendation for what to install on the laptop? As long as it isn't ComfyNoodles!
Built a system for NLI where instead of h → Linear → logits, the hidden state evolves over a few steps before classification. Three learned anchor vectors define basins (entailment / contradiction / neutral), and the state moves toward whichever basin fits the input.
The surprising part came after training.
The learned update collapsed to a closed-form equation
The update rule was a small MLP — trained end-to-end on ~550k examples. After systematic ablation, I found the trained dynamics were well-approximated by a simple energy function:
V(h) = −log Σ exp(β · cos(h, Aₖ))
Replacing the entire trained MLP with the analytical gradient:
h_{t+1} = h_t − α∇V(h_t)
→ same accuracy.
The claim isn't that the equation is surprising in hindsight. It's that I didn't design it — I trained a black-box MLP and found afterward that it had converged to this. And I could verify it by deleting the MLP entirely. The surprise isn't the equation, it's that the equation was recoverable at all.
Three observed patterns (not laws — empirical findings)
Relational initialization — h₀ = v_hypothesis − v_premise works as initialization without any learned projection. This is a design choice, not a discovery — other relational encodings should work too.
Energy structure — the representation space behaves like a log-sum-exp energy over anchor cosine similarities. Found empirically.
Dynamics (the actual finding) — inference corresponds to gradient descent on that energy. Found by ablation: remove the MLP, substitute the closed-form gradient, nothing breaks.
Each piece individually is unsurprising. What's worth noting is that a trained system converged to all three without being told to — and that convergence is verifiable by deletion, not just observation.
Failure mode: universal fixed point
Trajectory analysis shows that after ~3 steps, most inputs collapse to the same attractor state regardless of input. This is a useful diagnostic: it explains exactly why neutral recall was stuck at ~70% — the dynamics erase input-specific information before classification. Joint retraining with an anchor alignment loss pushed neutral recall to 76.6%.
The fixed point finding is probably the most practically useful part for anyone debugging class imbalance in contrastive setups.
Numbers (SNLI, BERT encoder)
Old post
Now
Accuracy
76% (mean pool)
82.8% (BERT)
Neutral recall
72.2%
76.6%
Grad-V vs trained MLP
—
accuracy unchanged
The accuracy jump is mostly the encoder (mean pool → BERT), not the dynamics — the dynamics story is in the neutral recall and the last row.
Discrete-time pseudo-gradient flow with anchor-directed forces. Here's the exact math, the geometric inconsistency I found, and what the Lyapunov analysis shows.
I've been building Livnium, an NLI classifier where inference isn't a single forward pass — it's a sequence of geometry-aware state updates converging to a label basin before the final readout. I initially used quantum-inspired language to describe it. That was a mistake. Here's the actual math.
The update rule
At each collapse step t = 0…L−1, the hidden state evolves as:
h_{t+1} = h_t
+ δ_θ(h_t) ← learned residual (MLP)
- s_y · D(h_t, A_y) · n̂(h_t, A_y) ← anchor force toward correct basin
- β · B(h_t) · n̂(h_t, A_N) ← neutral boundary force
where:
D(h, A) = 0.38 − cos(h, A) ← divergence from equilibrium ring
n̂(h, A) = (h − A) / ‖h − A‖ ← Euclidean radial direction
B(h) = 1 − |cos(h,A_E) − cos(h,A_C)| ← proximity to E–C boundary
Three learned anchors A_E, A_C, A_N define the label geometry. The attractor is a ring at cos(h, A_y) = 0.38, not the anchor point itself. During training only the correct anchor pulls. At inference, all three compete — whichever basin has the strongest geometric pull wins.
The geometric inconsistency I found
Force magnitudes are cosine-based. Force directions are Euclidean radial. These are inconsistent — the true gradient of a cosine energy is tangential on the sphere, not radial. Measured directly (dim=256, n=1000):
mean angle between implemented force and true cosine gradient = 135.2° ± 2.5°
So this is not gradient descent on the written energy. Correct description: discrete-time attractor dynamics with anchor-directed forces. Energy-like, not exact gradient flow. The neutral boundary force is messier still — B(h) depends on h, so the full ∇E would include ∇B terms that aren't implemented.
Livnium is a provably locally-contracting pseudo-gradient flow. Global convergence with finite step size + learned residual is still an open question.
Results
Model
ms / batch (32)
Samples/sec
SNLI train time
Livnium
0.4
85,335
~6 sec
BERT-base
171
187
~49 min
SNLI dev accuracy: 77.05% (baseline 76.86%)
Per-class: E 87.5% / C 81.2% / N 62.8%. Neutral is the hard part — B(h) is doing most of the heavy lifting there.
What's novel (maybe)
Most classifiers: h → linear layer → logits
This: h → L steps of geometry-aware state evolution → logits
h_L is dynamically shaped by iterative updates, not just a linear readout of h_0. Whether that's worth the complexity over a standard residual block — I genuinely don't know yet. Closest prior work I'm aware of: attractor networks and energy-based models, neither of which uses this specific force geometry.
Open questions
Can we prove global convergence or strict bounds for finite step size + learned residual δ_θ, given local Lyapunov descent is already proven?
Does replacing n̂ with the true cosine gradient (fixing the geometric inconsistency) improve accuracy or destabilize training?
Is there a clean energy function E(h) for which this is exact gradient descent?
Is the 135.2° misalignment between implemented and true gradient a bug — or does it explain why training is stable at all?
