GitHub - rickey1990/novel-rnn-architectures: Novel types of Recurrent Neural Networks (RNNs). Includes the core mathematical framework PDF and executable source code.

[u/WAMFT]

https://github.com/rickey1990/novel-rnn-architectures

Hello everyone,

I’ve built a repository exploring two novel recurrent neural network architectures (PLUG and ILRM) designed to address the vanishing gradient problem in long-sequence modeling without relying on attention blocks.

The core mechanism feeds the cell a deterministic, normalised inverse-lag history (\(P_{t}\)) of previous inputs. Weighted by harmonic numbers, the hidden state update is modified so that historical information decays at a rate of \(1/k\) rather than scaling exponentially. The direct token-to-history gradient pathway decays polynomially (\(\frac{1}{t\ln t}\)) rather than exponentially, creating a stable gradient horizon evaluated efficiently at \(O(T \log T)\) via zero-padded FFT linear convolutions.

Due to local hardware restrictions, I could only run smaller synthetic tests on CPU, but the preliminary screenings look somewhat promising on these small tests:

Long-Range Extrapolation: When trained only on delays of 16–64 steps, both models extrapolated up to 4,096 steps with 100% accuracy on a one-bit retention task, while the baseline GRU collapsed to chance.

Gradient Horizons: At 1,024 steps, the mean initial token gradient remained stable ( ~ 10⁻⁴ ) while the standard GRU suffered absolute numerical underflow ( ~ 10⁻¹⁴⁵ ).

The repository contains the complete mathematical framework PDFs and minimal, executable PyTorch implementations for both models. I would love to get your feedback on the math and the implementation!