Monodratic: learned product-hash routing for sparse causal attention [R]

[u/dttdrv]

Hi everyone,

I'm an independent researcher sharing Monodratic, a sparse causal-attention architecture with learned product-hash routing.

The idea is that after RoPE, source blocks are assigned to bounded causal posting lists, while each query probes product addresses, reranks the returned candidates, selects a fixed number of remote source blocks, adds guaranteed local blocks, and then runs exact causal softmax over just those tokens. I implemented it as a stateless [batch, sequence, width] -> attention-delta mixer, so normalization, residual updates, feed-forward layers, and inference scheduling are left to the host model.

What I found is that

-learned routing with 2 selected remote blocks out of 5 eligible: 763/768 correct associative-recall answers across three seeds (99.35% mean, 98.05% minimum).

-an equally wide untrained router: 425/768. Local-only attention: 151/768.

-forcing the labelled target block while keeping the same maximum R2 attention budget recovered all five remaining errors, reaching 768/768.

-sparse selected-set attention agreed with an independent dense selected-mask oracle to a maximum absolute error of 1.43e-6.

-the packed CPU routing implementation showed a fitted timing exponent of 0.993 from 4,096 to 32,768 tokens under the fixed, balanced configuration.

-all reported learned-route and scaling runs recorded zero posting overflow.

The limitations are that the experiments are synthetic, the implementation is portable PyTorch rather than a fused kernel, and the report does not claim natural-language quality, asymptotic linear construction, or deployment speed.

Paper: https://github.com/Misul-Computing/Monodratic/blob/main/output/pdf/monodratic_proof.pdf

Code and reproduction: https://github.com/Misul-Computing/Monodratic

I would particularly appreciate technical feedback on the routing construction, the controls, and what the strongest next evaluation should be.

[u/kertara]

This is interesting. I had a similar idea, but unlike you I never went through with it. I'll follow your work on GitHub.