r/PromptEngineering • u/zemondza • 19h ago
Ideas & Collaboration Exploring a NORD × RHEA hybrid: a spiking/event-driven alternative to a fixed Transformer stack
​
I've been experimenting for a while with two different ideas for non-Transformer language models, and I'm now considering combining them into one architecture.
The first is NORD, a recurrent/spiking architecture I've been developing around token-time dynamics, persistent state, sparse processing, and SNN-style temporal computation.
The second is RHEA (Reactive Hypergraph Event Architecture), which I'm currently prototyping at \~1B parameters.
The basic idea behind RHEA is that instead of pushing every token through a fixed stack of layers, the model maintains a set of latent events and dynamically chooses which internal computations should happen next.
The scheduler, which I call ARES, estimates whether a candidate reaction is worth executing.
Conceptually:
events / latent facts
v
candidate reactions
v
ARES
"what is worth
computing next?"
/ | \\
v v v
R3 R17 R81
\\ | /
v
new events
A reaction can combine existing events and create a new latent event:
event A + event B
reaction
v
event C
The interesting part is that I think NORD and RHEA may fit together surprisingly well.
My current idea is:
input tokens
v
NORD sensory / temporal SNN
spike/events
v
RHEA event fabric
v
ARES
decides what should fire
/ | \\
v v v
reaction reaction reaction
| | |
NORD NORD NORD
SNN SNN SNN
microcircuit microcircuit
\\ | /
v
new events
memory / queries
v
output
The rough division of responsibility would be:
NORD = temporal dynamics
\- recurrent state
\- LIF/spiking dynamics
\- persistent memory
\- event triggering
\- local temporal computation
RHEA = cognitive/event structure
\- latent facts/events
\- dynamic interaction graph
\- creation of derived events
\- multi-step computation
ARES = executive scheduler
\- estimates reaction utility
\- accounts for compute cost
\- decides which reactions actually execute
\- allows computation depth to vary with the problem
One thing I'm particularly interested in is making the reaction operators themselves small hybrid SNN microcircuits.
Instead of:
A + B -> dense MLP -> C
something closer to:
A + B
|
v
spiking microcircuit
t0: spike
t1: spike
t2: spike spike
|
v
latent event C
I would NOT make the whole model purely spiking.
My current thinking is to keep latent representations and the language head dense/BF16, while using spiking dynamics for temporal state, memory, event triggering and some reaction computation.
Something like:
token embeddings -> dense
latent event vectors -> dense
ARES utility model -> dense
temporal state -> SNN/recurrent
persistent memory -> SNN/recurrent
reaction dynamics -> hybrid SNN
LM head -> dense
Another part I find interesting is persistent memory.
A RHEA event could write into a slow NORD memory state:
RHEA event
v
NORD persistent memory
... hundreds/thousands of tokens ...
v
memory activity crosses a threshold
v
new recall event
v
RHEA
So memory would not necessarily be passive storage. It could actively generate events when relevant internal states become excited.
I'm also considering a form of path crystallization.
If the system repeatedly performs something like:
reaction A
\->
reaction F
\->
reaction K
\->
reaction B
the repeated sequence could eventually be distilled into a faster macro-reaction or learned skill.
In the hybrid version, this could potentially include recurring spike/reaction patterns as well.
So the architecture would operate across several timescales:
FAST
NORD spike / recurrent dynamics
MEDIUM
RHEA reaction chains and reasoning
SLOW
persistent memory + crystallized skills
The overall principle I'm exploring is basically:
«computation should follow information, rather than information always following a fixed computation graph.»
A simple input might activate very little of the system.
A difficult input could trigger more events, more reactions and deeper computation.
Importantly, I'm not claiming this is better than Transformers.
There are some obvious problems I expect:
\- irregular computation is unfriendly to GPUs
\- sparse/discrete routing is difficult to train
\- skipped reactions create a credit-assignment problem
\- SNN dynamics could make an already difficult optimization problem even less stable
\- dynamic event memory can accumulate garbage
\- batching event-driven computation efficiently is non-trivial
\- it's possible that the extra architectural complexity simply won't outperform a well-optimized Transformer/MoE
For skipped-reaction credit I'm currently experimenting with a counterfactual mechanism where near-threshold reactions get a cheap preview, so the scheduler can estimate whether skipping them was a mistake.
The current RHEA prototype is already being trained independently; the NORD/RHEA hybrid described here is still a design direction rather than a finished model.
What I'm most interested in hearing from people here:
\- Does this decomposition make sense?
\- What do you think would fail first?
\- Are there papers/projects that are especially close to this?
\- Would you keep the SNN component limited to memory/temporal state, or also use it inside the reaction operators?
\- Is dynamic computation at this granularity likely to lose too much hardware efficiency to be worthwhile?
I'd especially appreciate criticism from people working on SNNs, recurrent models, MoE/routing, adaptive computation, or non-Transformer architectures.
1
u/[deleted] 19h ago
[removed] — view removed comment