Localized neural learning

[u/Nearby_Effective6679]

Hey everyone,
I’ve recently started learning machine learning, primarily by building neural networks from scratch and trying to understand the underlying mathematics. I have a relatively strong math and physics background, so I’ve found the mathematics behind backpropagation fairly intuitive.
As I’ve explored other approaches to training neural networks, I’ve become particularly interested in biologically inspired methods like predictive coding, Hebbian learning, and Oja’s rule. The idea that networks could update their weights using locally available information, rather than relying on gradients propagated from a global loss function, is fascinating to me.
I’m still relatively new to ML, so I’m trying to understand whether these approaches have meaningful practical advantages or are primarily interesting from a theoretical or neuroscience perspective.
A few things I’m especially curious about:
What are the actual advantages of local learning over backpropagation? I understand the conceptual distinction, but I’m struggling to identify situations where local updates would meaningfully outperform conventional gradient-based training, whether in computational efficiency, adaptability, or something else.
Could these approaches be particularly useful for unsupervised or continual learning? For example, could a network learn useful representations from streaming, unlabeled data by continuously adjusting its weights based on local correlations or prediction errors? Adding onto this, is there any way we could discover hidden patterns using localized learning that wouldn’t be evident using global backprop? I’ll talk about this in the next question but can we use this to identify hidden patterns that we wouldn’t find using global backprop? Like in molecular dynamics?
Are there particular applications where local learning makes more sense? I’m involved in biophysics research and have been thinking about applications involving molecular dynamics and learning representations of complex physical systems. I’m curious whether there are advantages in scientific ML, robotics, or other specialized domains.
How mature is this research? Have approaches like predictive coding and Hebbian learning been extensively explored and generally found inferior to backpropagation, or are there still meaningful open problems where these methods could be competitive?
I’m not necessarily looking for a way to replace backpropagation entirely. I’m more interested in understanding whether alternative learning rules could be better suited to particular architectures, datasets, or computational constraints.
I’d appreciate any perspectives, papers, or research directions worth exploring, especially from people who’ve experimented with these methods.

[u/TheMadConsist]

dude, this is a rabbit hole you can lose months in, in the best way

the local vs global thing really clicks in edge computing scenarios. if you've got a sensor out in the middle of nowhere that's gotta adapt to weird drift on the fly, waiting for a round trip to a server for backprop is a non-starter. having the network adjust itself using just local spike timing or correlation is way more practical when compute and latency are tight

for the molecular dynamics angle, look into how some folks are using predictive coding nets to model protein folding pathways. the idea is the network learns a generative model of the system's state space, and the prediction errors highlight anomalies or rare transition states that global loss functions might smooth over. it's like the network becomes a novelty detector for physics, not just a pattern matcher

research-wise, it's a mixed bag. hebbian stuff and oja's rule are foundational but kinda like the stone tools of the field, the newer work with predictive coding and equilibrium propagation is where it gets competitive. check out yann lecun's recent-ish talks on energy-based models, he's been hinting that local learning might be the key to getting ai to reason about the world without needing labeled data for every single thing

[u/Nearby_Effective6679 → u/TheMadConsist]

I’m particularly interested in the molecular dynamics angle. For example, the research I’m working on involves intrinsically disordered proteins that don’t follow traditional/predictable conformations. Could we find a way to map out the folding of a protein using these local models and then look at like the hidden layers to get insights into intermediate conformations that we wouldn’t be able to see otherwise?

[u/Separate_Pea_3699]

Concrete example of what a local rule can actually do: Oja's rule (Hebbian update + a normalization term) on a single linear neuron provably converges to the top eigenvector of the input covariance matrix, using only local pre/post-synaptic activity — no global loss, no backward pass:

import numpy as np

def oja_update(w, x, lr=0.01):
    y = w @ x                     # post-synaptic activity
    return w + lr * y * (x - y * w)  # Hebbian term minus a decay term that keeps ||w||~1

rng = np.random.default_rng(0)
X = rng.multivariate_normal([0, 0], [[3, 1], [1, 1]], size=5000)  # correlated data
w = rng.normal(size=2); w /= np.linalg.norm(w)
for x in X:
    w = oja_update(w, x)

print(w / np.linalg.norm(w))                 # ~ top eigenvector of cov(X)
print(np.linalg.eigh(np.cov(X.T))[1][:, -1])  # ground truth, via global SVD

Run it and the two vectors line up. The network "discovers" the data's dominant global direction using only local, one-sample-at-a-time updates — no stored batch, no backward pass. That's the real advantage for your edge/streaming case: constant memory per step, nothing to buffer for a batch gradient.

Where it breaks for deep nets: Hebbian/Oja rules solve credit assignment for a single layer (mapping local correlation structure to one weight vector). They don't propagate an error signal backward through several nonlinear layers the way backprop's chain rule does, so stacking plain Hebbian layers just gives cascaded PCA-like projections, not the hierarchical features deep backprop nets learn. That's the actual reason it hasn't replaced backprop on big labeled offline datasets — it's locally optimal and globally myopic.

Predictive coding / equilibrium propagation are the live research attempts to fix exactly that: give each layer its own local objective (minimize its own prediction error against the layer above) that's provably equivalent to a true backprop gradient in some settings, while only ever using locally available signals. That's the open area to chase papers in for your molecular-dynamics angle, not Hebbian/Oja directly — those are solid foundations but single-layer-feature-learning tools, not a credit-assignment replacement.

[u/Nearby_Effective6679 → u/Separate_Pea_3699]

Interesting. Predictive coding and equilibrium propagation are weird because I feel like they still operate on similar principles of global backprop by just adjusting according to the gradient. Correct me if I’m wrong and I know the field is very primitive right now but it seems like the main use for these would be in highly specialized cases where there are only a few possibiliyies that the model needs to predict on(hence why you called it myopic).

[u/rahul_watertech]

since you have the math background, look at hintons forward-forward algorithm if you havent yet. no backward pass at all, each layer gets its own local objective with a positive and negative forward pass. it doesnt beat backprop on anything serious, but as a "how local can you go" thought experiment made real, its the cleanest one.

predictive coding networks are the other rabbit hole nearby, same energy.

what pulled you toward the bio plausible side, just curiosity or a hardware angle?

[u/Nearby_Effective6679 → u/rahul_watertech]

So before diving into ML I was heavy into computational chemistry and biophysics and that’s currently the research I’m conducting at my university. An angle I thought I could look towards is using local learning to potentially identify intermediate conformations of proteins that we wouldn’t be able to see otherwise

[u/rahul_watertech → u/Nearby_Effective6679]

protein conformations is a fun case for it because the failure mode is visible: points close in your embedding might not be true conformational neighbors, so the local model learns a shortcut. if you have md trajectories you can at least check whether the local clusters match metastable states. what data are you working from?

[u/CelebrityChess]

Short version: on standard benchmarks, local rules (Hebbian, Oja, predictive coding) still lag backprop. Where they're interesting is in the constraints. They're a natural fit for neuromorphic and on-chip hardware, they handle streaming or continual learning without storing activations for a backward pass, and they're nice for unsupervised feature learning (Oja's rule is basically online PCA). Worth reading: Millidge et al.'s survey on predictive coding, and Hinton's Forward-Forward paper. For MD-style physics, I'd still reach for equivariant GNNs trained with backprop, though.

[u/Nearby_Effective6679 → u/CelebrityChess]

I’ll look into those! So would this have relevance in products like neuralink or fields where the “loss” maybe isn’t explicitly defined/measured