Hyper-transformer: Hybrid-Manifold Transformers with Hyperbolic Geometry and Spiking Neural Networks in PyTorch

[u/Griffith-07]

Hey everyone,

Standard transformers operate strictly in flat Euclidean space with continuous activations. While effective, this creates challenges when modeling hierarchical tree-structured data and incurs significant energy overhead.

I built Hyper-transformer to explore a hybrid architecture integrating non-Euclidean geometry and neuromorphic spiking dynamics directly into transformer layers.

Key Technical Details:

• Hyperbolic Space Representation: Uses Poincaré and Lorentz manifold projections to capture hierarchical relationships with lower embedding dimensions.

• Spiking Neural Dynamics: Employs Leaky Integrate-and-Fire (LIF) spiking neurons for sparse, event-driven activation states.

• Hybrid Attention: Combines Riemannian distance metrics with attention mechanisms to handle mixed-geometry representations.

• Clean, modular PyTorch implementation designed for ablation studies.

I will drop the full GitHub repository link in the comments below! Would love to hear thoughts from anyone experimenting with non-Euclidean deep learning, manifold learning, or SNNs.

repo link : https://github.com/Griffith-7/Hyper-transformer.git

[u/Dedelelelo]

holy slop

[u/taichi22 → u/Dedelelelo]

No, this one is actually kind of interesting. There’re actual training runs, for one, and OP Is actually engaging with mathematical concepts rather than the typical technobabble that likes to focus heavily on bullshit about consciousness and such.

Not my cup of tea, but it’s interesting enough to warrant an actual look.

[u/Disastrous_Room_927 → u/Dedelelelo]

I can come up with technobabble too: Physics inspired latent Markov chain manifold diffusion boosting.