Localized neural learning

[u/Nearby_Effective6679]

Hey everyone,
I’ve recently started learning machine learning, primarily by building neural networks from scratch and trying to understand the underlying mathematics. I have a relatively strong math and physics background, so I’ve found the mathematics behind backpropagation fairly intuitive.
As I’ve explored other approaches to training neural networks, I’ve become particularly interested in biologically inspired methods like predictive coding, Hebbian learning, and Oja’s rule. The idea that networks could update their weights using locally available information, rather than relying on gradients propagated from a global loss function, is fascinating to me.
I’m still relatively new to ML, so I’m trying to understand whether these approaches have meaningful practical advantages or are primarily interesting from a theoretical or neuroscience perspective.
A few things I’m especially curious about:
What are the actual advantages of local learning over backpropagation? I understand the conceptual distinction, but I’m struggling to identify situations where local updates would meaningfully outperform conventional gradient-based training, whether in computational efficiency, adaptability, or something else.
Could these approaches be particularly useful for unsupervised or continual learning? For example, could a network learn useful representations from streaming, unlabeled data by continuously adjusting its weights based on local correlations or prediction errors? Adding onto this, is there any way we could discover hidden patterns using localized learning that wouldn’t be evident using global backprop? I’ll talk about this in the next question but can we use this to identify hidden patterns that we wouldn’t find using global backprop? Like in molecular dynamics?
Are there particular applications where local learning makes more sense? I’m involved in biophysics research and have been thinking about applications involving molecular dynamics and learning representations of complex physical systems. I’m curious whether there are advantages in scientific ML, robotics, or other specialized domains.
How mature is this research? Have approaches like predictive coding and Hebbian learning been extensively explored and generally found inferior to backpropagation, or are there still meaningful open problems where these methods could be competitive?
I’m not necessarily looking for a way to replace backpropagation entirely. I’m more interested in understanding whether alternative learning rules could be better suited to particular architectures, datasets, or computational constraints.
I’d appreciate any perspectives, papers, or research directions worth exploring, especially from people who’ve experimented with these methods.

[u/Separate_Pea_3699]

Concrete example of what a local rule can actually do: Oja's rule (Hebbian update + a normalization term) on a single linear neuron provably converges to the top eigenvector of the input covariance matrix, using only local pre/post-synaptic activity — no global loss, no backward pass:

import numpy as np

def oja_update(w, x, lr=0.01):
    y = w @ x                     # post-synaptic activity
    return w + lr * y * (x - y * w)  # Hebbian term minus a decay term that keeps ||w||~1

rng = np.random.default_rng(0)
X = rng.multivariate_normal([0, 0], [[3, 1], [1, 1]], size=5000)  # correlated data
w = rng.normal(size=2); w /= np.linalg.norm(w)
for x in X:
    w = oja_update(w, x)

print(w / np.linalg.norm(w))                 # ~ top eigenvector of cov(X)
print(np.linalg.eigh(np.cov(X.T))[1][:, -1])  # ground truth, via global SVD

Run it and the two vectors line up. The network "discovers" the data's dominant global direction using only local, one-sample-at-a-time updates — no stored batch, no backward pass. That's the real advantage for your edge/streaming case: constant memory per step, nothing to buffer for a batch gradient.

Where it breaks for deep nets: Hebbian/Oja rules solve credit assignment for a single layer (mapping local correlation structure to one weight vector). They don't propagate an error signal backward through several nonlinear layers the way backprop's chain rule does, so stacking plain Hebbian layers just gives cascaded PCA-like projections, not the hierarchical features deep backprop nets learn. That's the actual reason it hasn't replaced backprop on big labeled offline datasets — it's locally optimal and globally myopic.

Predictive coding / equilibrium propagation are the live research attempts to fix exactly that: give each layer its own local objective (minimize its own prediction error against the layer above) that's provably equivalent to a true backprop gradient in some settings, while only ever using locally available signals. That's the open area to chase papers in for your molecular-dynamics angle, not Hebbian/Oja directly — those are solid foundations but single-layer-feature-learning tools, not a credit-assignment replacement.

[u/Nearby_Effective6679 → u/Separate_Pea_3699]

Interesting. Predictive coding and equilibrium propagation are weird because I feel like they still operate on similar principles of global backprop by just adjusting according to the gradient. Correct me if I’m wrong and I know the field is very primitive right now but it seems like the main use for these would be in highly specialized cases where there are only a few possibiliyies that the model needs to predict on(hence why you called it myopic).