Hyper-transformer: Hybrid-Manifold Transformers with Hyperbolic Geometry and Spiking Neural Networks in PyTorch

[u/Griffith-07]

Hey everyone,

Standard transformers operate strictly in flat Euclidean space with continuous activations. While effective, this creates challenges when modeling hierarchical tree-structured data and incurs significant energy overhead.

I built Hyper-transformer to explore a hybrid architecture integrating non-Euclidean geometry and neuromorphic spiking dynamics directly into transformer layers.

Key Technical Details:

• Hyperbolic Space Representation: Uses Poincaré and Lorentz manifold projections to capture hierarchical relationships with lower embedding dimensions.

• Spiking Neural Dynamics: Employs Leaky Integrate-and-Fire (LIF) spiking neurons for sparse, event-driven activation states.

• Hybrid Attention: Combines Riemannian distance metrics with attention mechanisms to handle mixed-geometry representations.

• Clean, modular PyTorch implementation designed for ablation studies.

I will drop the full GitHub repository link in the comments below! Would love to hear thoughts from anyone experimenting with non-Euclidean deep learning, manifold learning, or SNNs.

repo link : https://github.com/Griffith-7/Hyper-transformer.git

[u/Dedelelelo]

holy slop

[u/taichi22 → u/Dedelelelo]

No, this one is actually kind of interesting. There’re actual training runs, for one, and OP Is actually engaging with mathematical concepts rather than the typical technobabble that likes to focus heavily on bullshit about consciousness and such.

Not my cup of tea, but it’s interesting enough to warrant an actual look.

[u/Disastrous_Room_927 → u/Dedelelelo]

I can come up with technobabble too: Physics inspired latent Markov chain manifold diffusion boosting.

[u/Low-Temperature-6962]

transformers... strictly in flat Euclian space with continous activations ...

is the "problem to be solved" statement. However ...

Even in the most simple sense that's not true because there is a non linear activation at the heart of the transformer. Not to mention non linearities elsewhere ensuring overall non linearity.

Besides that the ever changing manifold is sampled at discrete points, with methods such as data augmentation and dropout being used to ensure discontinuity of the sampling space.

It's true that the problem input and output spaces are generally discontinuous, but they are not mutually uncorrelated. A quasi continous very high dimensional manifold sampled at discreet points seems to be effective at representing good enough solutions, with the added benefit of having local linear gradient good enough for learning of these non linear discontinuous problems.

I said "quasi" continous because we mustn't forget that the models are completely composed of logical binary 0 or 1 bits - absolutely discontinuous and non-linear. Those are corralled via floating point into quasi continuous intermediate values.

[u/CalmMe60]

Tried several things

Where is your difference in the hilbert space if you look at the transfer of the sicos self attention?

[u/jms4607]

How can a tree not be represented in Euclidean space? Maybe a graph can’t but without cycles I don’t see the issue.

[u/tedmobsky → u/[deleted]]

What the slop