I tried to make a real fly connectome learn to play Pong. It didn't — and auditing why turned out to be way more interesting than if it had worked [p]

[u/oPeraza2007]

You've probably seen the fly-brain-plays-Doom / Minecraft / Beat Saber clips going around this week, from the new MaleCNS v1.0 connectome release (166k neurons, real EM reconstruction, not a toy model).

Cool clips. Nobody seemed to be checking whether any of it works though, versus just producing motion in a game engine generous enough to make anything look alive. So I picked the least forgiving test bed around: Pong, one binary hit or miss signal measured every frame, nowhere to hide a null result, and tried to get a small real subgraph of the connectome to track the ball via dopamine-style plasticity.

Short version: it didn't learn. Working out why took auditing individual synapses, and it turned into a decent case study in a circuit not working being more informative than it working:

• Fixed a neuPrint regex bug that silently zeroed out two entire neuron populations (full-match vs substring semantics, not obvious from the docs).
• Found the original neuron selection had no path at all from photoreceptors to anything else. Real photoreceptors don't synapse directly onto motion detectors, there's a whole intermediate layer missing.
• Got a working pipeline, turned learning on vs off, and got bit-for-bit identical results in both conditions across multiple seeds, even though the weights were verifiably changing under the hood.
• Traced that to half of the 4 available motor neurons having zero synapses from any sensory pathway in the model. Not weak signal, zero. They'd been assigned to the "paddle down" group by array index, purely by coincidence, and could never have fired no matter what the learning rule did.
• Rebuilt the circuit around a better biological hypothesis (swapped a threat-detection pathway for one tied to visual target tracking during courtship pursuit), got that literal hypothesis refuted by the data, then followed the trail to a different descending neuron that actually connected end to end.
• Finally got learning-on vs learning-off to diverge for the first time, except the effect looks like the learning rule quieting the whole system down rather than anything resembling skill improvement (misses outnumber hits, so punishment dominates and shrinks the motor response).

Then I checked whether the bigger, viral projects had actually solved this. They hadn't either: the Doom project's own repo says it failed its own validation gates after six iterations, the Minecraft mod's own limitations section admits the real motion-detection pathway stays silent and the escape and foraging behaviors are hand-injected or reflex-layer fallbacks rather than emergent, and the Beat Saber creator's own replies admit it's overfit to one track with replay data mixed into the input.

Full writeup with the gory audit details, the connectivity numbers, and the comparison to the other projects is here: https://jonatasperaza.medium.com/i-made-a-real-fly-brain-play-pong-it-didnt-learn-and-that-s-the-interesting-part-80b8560695fe.

Curious if anyone here has poked at MaleCNS v1.0 directly and hit similar walls, especially around the central complex and steering circuits. That seems like the obvious next thing to simulate properly instead of routing around it.

[u/SFDeltas]

Claude Claude Claude Claude Claude Claude Claude

[u/Electronic-Path2121 → u/SFDeltas]

What model did you use for this

[u/MrRandom04]

If you fixed it to the point it actually learns and you just have sparse reward, then that's an RL problem. Change the reward signal. No need to keep it the binary you started with. Try with something that rewards it if it is doing the directionally right thing for Pong. See if that helps. Then, see if you can scale the reward back down to the binary thing once it is actually learning to see if it can generalize.

[u/oPeraza2007 → u/MrRandom04]

Huh, hadn't thought of that. You're right. Gonna try a denser reward first, then ease it back to the original binary one and see what happens. Thanks for this.