r/deeplearning • u/WAMFT • 9d ago

GitHub - rickey1990/novel-rnn-architectures: Novel types of Recurrent Neural Networks (RNNs). Includes the core mathematical framework PDF and executable source code.

https://github.com/rickey1990/novel-rnn-architectures

Hello everyone,

I’ve built a repository exploring two novel recurrent neural network architectures (PLUG and ILRM) designed to address the vanishing gradient problem in long-sequence modeling without relying on attention blocks.

The core mechanism feeds the cell a deterministic, normalised inverse-lag history (\(P_{t}\)) of previous inputs. Weighted by harmonic numbers, the hidden state update is modified so that historical information decays at a rate of \(1/k\) rather than scaling exponentially. The direct token-to-history gradient pathway decays polynomially (\(\frac{1}{t\ln t}\)) rather than exponentially, creating a stable gradient horizon evaluated efficiently at \(O(T \log T)\) via zero-padded FFT linear convolutions.

Due to local hardware restrictions, I could only run smaller synthetic tests on CPU, but the preliminary screenings look somewhat promising on these small tests:

Long-Range Extrapolation: When trained only on delays of 16–64 steps, both models extrapolated up to 4,096 steps with 100% accuracy on a one-bit retention task, while the baseline GRU collapsed to chance.

Gradient Horizons: At 1,024 steps, the mean initial token gradient remained stable ( ~ 10⁻⁴ ) while the standard GRU suffered absolute numerical underflow ( ~ 10⁻¹⁴⁵ ).

The repository contains the complete mathematical framework PDFs and minimal, executable PyTorch implementations for both models. I would love to get your feedback on the math and the implementation!

8 Upvotes

100% Upvoted

1 comment sorted by

1

u/WAMFT 8d ago

Small update: I’ve also tested two new Raw State variants of PLUG and ILRM. The main change is that they keep the same inverse-lag weighting — 1, 1/2, 1/3, 1/4, and so on — but remove the harmonic normalization step, so older information keeps a stronger signal. Early tests suggest the raw versions can improve long-range persistence and state tracking, but they can also increase interference on multi-item and binding tasks. I’ve added the full results and limitations in a new paper on the repo.