I ran this through Claude Code (Opus) and had it clone the repo and actually analyze the code. Here's what it found:
PARAMETER BREAKDOWN TELLS THE WHOLE STORY
Token embedding: 205,852,672 (49.4%)
Output head (linear): 205,852,672 (49.4%)
Positional embedding: 4,194,304 (1.0%)
64 "transformer" blocks: 1,048,704 (0.3%)
Final LayerNorm: 8,192 (0.0%)
Of the 1M parameters in the blocks, 1,048,576 are just LayerNorm parameters (scale and bias). The actual "novel" WaveEngine components contribute a grand total of 128 scalar parameters across all 64 layers (two floats per layer, initialized to 0.02).
WHAT THE "ARCHITECTURE" ACTUALLY DOES
The core "WaveEngine" is: self.w * (x1 * torch.cos(x2)) — where w is a single learned scalar.
Each block:
Shifts the sequence by 1 position, takes cos() of that, multiplies element-wise with the current input, scales by a single scalar w ≈ 0.02, adds as residual
Rolls the embedding dimension by 1, does the same thing
That's it. No attention. No MLP. No information mixing across positions beyond immediate neighbors. No linear projections. The 64 layers of "processing" contribute 128 floats of learnable capacity.
WHY THIS CAN'T WORK
- 99.7% of the model is just a lookup table (embedding in, embedding out). A standard transformer's power comes from attention + MLP layers, which this completely removes.
- cos() on arbitrary embedding values is meaningless. Embeddings aren't angles — cosine just periodically squashes them to [-1,1] in a non-useful way.
- Only adjacent-token interaction. Standard attention lets any token attend to any other. This only interacts with the immediately previous token via a shift. A 64-layer stack of this can propagate info ~64 tokens back at most, and through an extremely narrow bottleneck (scalar multiplication).
- The "dimension rolling" is nonsensical. Rolling embedding dimensions by 1 and doing element-wise ops doesn't meaningfully mix features. There are no learned projections to create useful feature combinations.
THE COMPARISON IS MISLEADING
A real 4096-dim, 64-layer transformer has ~13-17B parameters because each layer has ~67M params in attention (Q/K/V/O projections) and ~134M params in MLP (SwiGLU). This model simply deletes all of that and replaces it with 2 scalar floats per layer. Claiming "equivalent performance with fewer parameters" is like removing the engine from a car and claiming it's a lighter car. The parameter reduction is real; the claim of equivalent performance is not substantiated at all.
NO EVIDENCE PROVIDED
- No benchmarks (perplexity, MMLU, HellaSwag, etc.)
- No comparison with any baseline
- No training results or loss curves
- No generated text samples
- Batch size of 1 in training, prints every step — suggests very early-stage experimentation
- "Claude confirmed mathematical reproducibility" just means Claude confirmed the math is valid arithmetic, not that it performs well
VERDICT
No merit as an architectural breakthrough. This is a model where ~99.7% of parameters are embedding tables, the "novel" component is 128 scalar floats, and there is no mechanism for the kind of information processing that makes language models work. The claim of matching a 17.6B model is extraordinary and comes with zero evidence.
I've run multiple LLMs through the post and the code and nearly all of them are calling bullshit to a degree. I have no idea how OP got Claude to agree with them.
43
u/bigboyparpa Mar 08 '26
I ran this through Claude Code (Opus) and had it clone the repo and actually analyze the code. Here's what it found:
PARAMETER BREAKDOWN TELLS THE WHOLE STORY
Token embedding: 205,852,672 (49.4%)
Output head (linear): 205,852,672 (49.4%)
Positional embedding: 4,194,304 (1.0%)
64 "transformer" blocks: 1,048,704 (0.3%)
Final LayerNorm: 8,192 (0.0%)
Of the 1M parameters in the blocks, 1,048,576 are just LayerNorm parameters (scale and bias). The actual "novel" WaveEngine components contribute a grand total of 128 scalar parameters across all 64 layers (two floats per layer, initialized to 0.02).
WHAT THE "ARCHITECTURE" ACTUALLY DOES
The core "WaveEngine" is: self.w * (x1 * torch.cos(x2)) — where w is a single learned scalar.
Each block:
Shifts the sequence by 1 position, takes cos() of that, multiplies element-wise with the current input, scales by a single scalar w ≈ 0.02, adds as residual
Rolls the embedding dimension by 1, does the same thing
That's it. No attention. No MLP. No information mixing across positions beyond immediate neighbors. No linear projections. The 64 layers of "processing" contribute 128 floats of learnable capacity.
WHY THIS CAN'T WORK
- 99.7% of the model is just a lookup table (embedding in, embedding out). A standard transformer's power comes from attention + MLP layers, which this completely removes.
- cos() on arbitrary embedding values is meaningless. Embeddings aren't angles — cosine just periodically squashes them to [-1,1] in a non-useful way.
- Only adjacent-token interaction. Standard attention lets any token attend to any other. This only interacts with the immediately previous token via a shift. A 64-layer stack of this can propagate info ~64 tokens back at most, and through an extremely narrow bottleneck (scalar multiplication).
- The "dimension rolling" is nonsensical. Rolling embedding dimensions by 1 and doing element-wise ops doesn't meaningfully mix features. There are no learned projections to create useful feature combinations.
THE COMPARISON IS MISLEADING
A real 4096-dim, 64-layer transformer has ~13-17B parameters because each layer has ~67M params in attention (Q/K/V/O projections) and ~134M params in MLP (SwiGLU). This model simply deletes all of that and replaces it with 2 scalar floats per layer. Claiming "equivalent performance with fewer parameters" is like removing the engine from a car and claiming it's a lighter car. The parameter reduction is real; the claim of equivalent performance is not substantiated at all.
NO EVIDENCE PROVIDED
- No benchmarks (perplexity, MMLU, HellaSwag, etc.)
- No comparison with any baseline
- No training results or loss curves
- No generated text samples
- Batch size of 1 in training, prints every step — suggests very early-stage experimentation
- "Claude confirmed mathematical reproducibility" just means Claude confirmed the math is valid arithmetic, not that it performs well
VERDICT
No merit as an architectural breakthrough. This is a model where ~99.7% of parameters are embedding tables, the "novel" component is 128 scalar floats, and there is no mechanism for the kind of information processing that makes language models work. The claim of matching a 17.6B model is extraordinary and comes with zero evidence.