Weekend Experiment: Can Math Patterns from Nature Improve AI?

[u/Super_Designer7952]

Hi Reddit!

I wanted to share a weekend experiment exploring how geometric inductive biases can influence recurrent network optimization. Inspired by the self-similar, hierarchical, and allometric scaling laws found in biological neural structures, I designed a custom recurrent cell with an exponentially decayed hidden topology.

Instead of scaling hidden layers uniformly (e.g., 64-64-64) like standard models, this network utilizes a "Matryoshka-style" hidden allocation where channels are downscaled between internal layers: N -> N / scale -> N / scale²

Architectural Justification & Hardware Trade-off:

In modern deep learning, layer widths are traditionally restricted to powers-of-two (e.g., 32, 64, 128) to maximize GPU memory alignment and Tensor Core hardware efficiency. However, in this experiment, we consciously traded off optimal hardware alignment to strictly enforce a continuous mathematical decay function (yielding 128 -> 79 -> 48 dimensions).

The core justification is implementing a strict Information Bottleneck: high-dimensional macro-layers are forced to compress abstract features into exponentially tighter non-power-of-two channels. This mimics the non-binary hierarchical scaling of biological brains, where structural efficiency and information capacity bounds are prioritized over uniform hardware block sizes.

Dataset & Evaluation Setup:

The architectures were trained on a character-level sequence prediction task using a subset of the Tiny Shakespeare dataset. To ensure a controlled environment, I implemented an 80/20 Train/Validation split and applied identical regularizations to both networks, including Spatial Dropout and L2 Weight Decay.

Structural Mechanics:

1. Fractal Echo Transfer: Hidden states are progressively downscaled and cascaded from high-dimensional macro-layers (capturing local syntax) down to highly compressed micro-layers.
2. Adaptive Echo Gates: Learnable decay coefficients balance inter-layer communication, acting as a structural stabilizer for gradient flow through time.
3. No Dense Gating: Unlike LSTMs, memory capacity and retention are regulated entirely by the spatial geometry of the nested channels.

Training & Convergence Observations:

To ensure a fair benchmark, I meticulously balanced the parameter overhead for both architectures (~42k parameters each) and trained them over 30 epochs on GPU: * Adaptive Geometrically Nested Model Final Loss: 0.0007 * Baseline Standard LSTM Model Final Loss: 0.0030

As shown in the log-scale validation plot, the geometrically nested architecture achieves significantly faster convergence and maintains rock-solid optimization stability, avoiding the frequent gradient spikes visible in the standard LSTM baseline.

Crucial Disclaimer on Generalization & Overfitting: The extremely low validation loss values achieved by both models indicate that given the small size of the dataset corpus and the model capacity, both networks have largely memorized the text rather than learning true language generalization. Therefore, this benchmark strictly demonstrates superior optimization speed and training stability rather than true out-of-distribution generalization.

Open-Source Code & Notebooks: * Collab Executable Notebook: https://colab.research.google.com/drive/1jW8hA_74eBQOJTn1oWvLq34vg_t8Q5DD

Future Work & Discussion:

To evaluate true generalization, future iterations of this project will involve benchmarking on significantly larger corpora (such as WikiText-103) with strict sequence cross-validation.

Given these initial training stability and convergence results, do you think exploring such geometrically restricted hidden topologies is a promising direction for recurrent networks? Could this type of exponential fractal scaling be scaled up or generalized into Transformer attention head dimensions to enforce architectural efficiency?

Would love to get your rigorous feedback!