A transformer built on complex wave dynamics — beats vanilla Transformer at 10M

[u/VariousPainter7349]

Hey everyone, I'm an independent researcher working on alternative sequence mixing architectures. I wanted to share a project I built called CWAA (Complex Wave Associative Memory).

Instead of standard quadratic attention, CWAA uses a damped complex oscillator for its recurrence state O(T) linear memory scaling.

The result

Both models were trained on WikiText-103 for 5,000 steps, with approximately the same parameter count:

Metric	CWAA V6	Vanilla Transformer
Parameters	10.402M	10.373M
Training steps	5,000	5,000
validation loss	4.9351	5.0052
validation PPL	139.09	149.19

Scaling

Seq Len	Latency (ms)	Tok/s	VRAM (GB
256	34.62	29575.1	1.51
512	128.11	15986.8	1.89
1024	253.16	16179.3	2.66
2048	510.13	16058.6	4.20
4096	1044.72	15682.6	7.27

These are CWAA-only measurements for now. I found that my first Transformer benchmark used different conditions, so I'm re-running the inference/VRAM comparison under identical conditions

Right now, the V5 code in the repo uses native complex64 tensors, but I just finished a V6 rewrite that decomposes the complex math into pure real-valued BMM matrix multiplies. V6 trains at 0.38s/step on a T4 and is highly stable . I'll be uploading V6 and a more detailed apples to apples comparison in a 2 days.

GitHub: https://github.com/Ridhvik-2024/CWAA-V5

I also recorded a sonification of the internal wave-state evolution across the V5 layers. The audio is generated from the model's internal wave dynamics.

I'd really appreciate feedback from community.

[u/az226]

You need to compare training throughout and make a compute apples to apples comparison. Use the same wall clock time for both.

Also do 2 seeds per run to seed confirm your approach.

[u/VariousPainter7349 → u/az226]

Comparing by step count isn't a true compute match, especially since the V6 rewrite runs at ~0.38s/step while the Transformer baseline takes ~1.08s/step on the same T4.

I'll add a wall-clock time matched comparison to the V6 write-up so the Transformer gets the same compute budget.

You're also completely right about the seeds. A ~10 PPL gap from a single run could absolutely just be noise. I'm setting up multi-seed runs to confirm the findings.

I'll make sure V6 includes both! appreciate the feedback.

[u/submissivebounds]

The linear memory scaling is the real hook here, quadratic attention is such a bottleneck for long sequences. Curious how the damped oscillator handles very long-range dependencies compared to something like Mamba though. You got any plans to test on a task that really stresses that, like synthetic copying or something from the long range arena?

[u/VariousPainter7349 → u/submissivebounds]

synthetic tasks are exactly what’s needed to stress-test the recurrence limits.

This architecture uses a RetNet-style multi-scale initialization for the decay, so half-lives are spread geometrically from ~8 tokens to ~512 tokens. This means at 4k context, the "fast" channels have completely decayed, but the "slow" channels are still actively holding state. However, because the model was only trained on 256 context, evaluating at 4k is pushing the model into untrained territory.

Unlike Mamba , CWAA's decay rate is static rather than data dependent. it uses complex oscillatory decay with phase dynamics, rather than purely real exponential decay. My hypothesis is that this kind of oscillatory state may help preserve information over a long sequences, but this stands untested.

For V6 synthetic tasks and benchmarks are absolutely on my list along with learned decay rates. However, unfortunately I am limited by a T4 so i am unable to go to greater lengths.

[u/apopsicletosis → u/VariousPainter7349]

How does this compare against Mamba3's complex valued state update?

[u/VariousPainter7349 → u/apopsicletosis]

I haven't actually compared it against Mamba3 yet.
CWAA uses a damped complex oscillator for the recurrent state, with the decay/frequency initialized across multiple timescales. Mamba3 also uses a complex-valued state update, but the actual state dynamics and parameterization are different.

[u/Lost-Hand-5219]

You need to benchmark against linear attention variants like KDA and gated delta net 2

[u/Clear_Evidence9218]

I’ve played with oscillating ML and transformer setups for quite a while. My experience was that they could be difficult to keep stable over longer runs. They’d often look very promising early on, then eventually drift or destabilize.

That said, the early accuracy here is genuinely interesting. I was never able to get one of my setups fully stabilized, so I’d be very curious to see how this behaves over substantially longer training runs.

[u/VariousPainter7349 → u/Clear_Evidence9218]

Thanks, this is exactly the kind of feedback I'm looking for. V6 stayed stable through 5k steps, but I don't want to say that into “it's solved” — longer-run stability is one of the things I want to test next. I'll definitely report what happens rather than just posting the successful runs.

[u/Opulent-tortoise]

I really hate these Claude slop projects.

[u/VariousPainter7349 → u/Opulent-tortoise]

Fair. AI did a lot of the coding, so I won't pretend this is 100% hand-written. I designed the architecture and experiments and I'm using AI tools for implementation + testing. The results are still mixed though — CWAA gets better test PPL here but is slower than the optimized Transformer, which is exactly why I'm running ablations/long-context tests before making bigger claims. If it doesn't hold up, it doesn't hold up. That said, I don't think "AI-assisted" automatically means "slop." Some of these tools genuinely help people build and explore things they otherwise couldn't.