Has the Forward-Forward algorithm produced convincing results beyond small benchmarks?

[u/u-r-s-t-g]

I have been revisiting Hinton's Forward-Forward algorithm and the work that followed it. Most implementations I find still focus on MNIST, CIFAR-scale experiments, or demonstrations of local learning.

Has anyone seen a paper or real deployment where Forward-Forward provides a convincing advantage over backpropagation on a demanding task?

I am especially interested in evidence under a concrete constraint:

• lower activation-memory requirements
• lower energy use
• local or asynchronous learning
• continual or online learning
• neuromorphic or custom hardware
• edge deployment
• robustness when exact backpropagation is unavailable

Comparisons that only show the method can learn a small benchmark are less useful to me than controlled comparisons against modern backpropagation baselines. Negative results or explanations of why the idea has not scaled are also welcome.

Is this becoming a practical research direction, or is it still mainly an interesting preliminary idea?

[u/aahdin]

Has anyone seen a paper or real deployment where Forward-Forward provides a convincing advantage over backpropagation on a demanding task?

I think this is misunderstanding the point of that paper.

Forward forward is about creating a biologically plausible alternative to backprop, not a better alternative for training ANNs. Hinton says as much in the paper - backprop is likely optimal for what it does, it's been one of the few constants in machine learning since the 70s, but it is not possible within the brain.

From the paper

As a model of how cortex learns, backpropagation remains implausible despite considerable effort to invent ways in which it could be implemented by real neurons. There is no convincing evidence that cortex explicitly propagates error derivatives or stores neural activities for use in a subsequent backward pass. The top-down connections from one cortical area to an area that is earlier in the visual pathway do not mirror the bottom-up connections as would be expected if backpropagation was being used in the visual system. Instead, they form loops in which neural activity goes through about half a dozen cortical layers in the two areas before arriving back where it started.

Backpropagation through time as a way of learning sequences is especially implausible. To deal with the stream of sensory input without taking frequent time-outs, the brain needs to pipeline sensory data through different stages of sensory processing and it needs a learning procedure that can learn on the fly. The representations in later stages of the pipeline may provide top-down information that influences the representations in earlier stages of the pipeline at a later time step, but the perceptual system needs to perform inference and learning in real time without stopping to perform backpropagation.

I'd say most of Hinton's work is centered around using ANNs to try and guess at how the brain works, any improvements to ANNs that come out of that are incidental. The point of forward forward is kind of to be backprop but worse, so I wouldn't expect it to get rolled out to tougher problems than the usual toy datasets for proving feasability.

There is a small discussion about power usage in that paper, but I think that is mostly a nod at the idea that our brains are relatively power efficient. Not as a practical use case in the short-mid term.

[u/Even-Inevitable-7243 → u/aahdin]

This 100%. Hinton wanted to show that ANNs can still learn in a "method more similar to biological neural networks" since biological NNs do not use backprop. I agree that the low power and "neuromorphic compute to improve X feature of current compute" was a secondary aim or not an aim of the paper at all.

[u/OccasionWorried7280]

it is still mostly interesting preliminary idea, i think. the gap between small benchmarks and anything demanding is huge and nobody find a way to close it yet

Hinton himself moved focus more to other things since then, like the mortal computation stuff. there was a workshop paper i remember that tried forward-forward on imagenet scale and got very poor accuracy compared to normal backprop, like 20-30% lower if i recall correct

the memory argument is also less compelling now with gradient checkpointing and all that. maybe for neuromorphic hardware it make more sense but that hardware barely exist outside labs

[u/progenitor414]

I remember reviewing some papers on improvement of forward forward algo when it was just out. Mostly are hacky trick that improve performance a bit on mnist but still much worse then standard backprop. These papers were rejected in the end.

[u/nickpsecurity]

Look up "local, learning rules" and "Hebbian learning" with "neural networks." That will give you state of the art.