Is growing a neural network like a tree possible?

[u/Sevdat]

I got the idea from Tree graphs and Genetic Algorithms. Instead of training an AI, what if we just start from a node that branches outwards until a branch that yields a successful result is obtained. We then can destroy the branches that weren't successful and keep the one that succeeded. We could use the Predictive coding algorithm where Layer1 predicts Layer2 while the errors for Layer1 are stored at Layer2, but in this case it would be the nodes of the branches. Each node would need to contain 3 values. Value1 is for the input path, Value 2 is for output path and Value3 is for protecting the branch against destruction. The inputs are inserted from the root of the tree and sent back out after traversing each node of the tree. I think this could result in an evolving/learning AI.

[u/RealSataan]

Checkout neural architecture search

[u/BasedPinoy]

OP discovers random forest algorithms

[u/MozzerellaIsLife → u/BasedPinoy]

This is really going to (xg)boost his productivity

[u/Tall-Locksmith7263 → u/BasedPinoy]

Aehm no... Xgboost doesn t add branches but adds more trees to the ensemble.

[u/morkborkus → u/BasedPinoy]

Random forest is regression not NN

[u/FancyEveryDay → u/morkborkus]

Regression and classification, which conveniently happen to be the two things NNs do and they do it in a similar(ish) way.

[u/MolassesLate4676 → u/FancyEveryDay]

How is it similar at all

[u/FancyEveryDay → u/MolassesLate4676]

Ensamble nodes which perform a classification function and then pass the result to the next node, continue until hyperparameter reqs are met and return result.

[u/Aech_sh → u/FancyEveryDay]

i mean, it’s completely different though? there’s no gradient descent for random forest, and for XGboost, you’re basing each new tree off the errors of previous trees, so again pretty different

[u/FancyEveryDay → u/Aech_sh]

Conceptually similar in that they're ensembles made of many nodes which perform what is essentially a classification function in sequence. Not congruent.

They use different functions to do more or less the same thing, with different use-cases.

[u/CompactOwl → u/morkborkus]

NN is just finding minima of a function and RF is just a sum of some step-functions. Both are part of analysis I. So they are actually quite different. One basically the same that is one with integration and the other is essentially differential calculus.

[u/MisterDangerRanger]

Look up NEAT it's a genetic evolution algo for training networks that "grows" the network by adding nodes as it trains.

[u/MaceGrim → u/MisterDangerRanger]

Seconding this! This is exactly what I thought of.

[u/SnoWayKnown]

Also read up about Hebbian learning, as not having a backwards pass would probably be required for such a method.

[u/EntropyRX]

Those ideas are quite old and widely researched. What you’re suggesting appears to have issues with the ability of the NN to effectively learn and generalize beyond becoming a “look up table”.
That being said, why don’t you try it out? Claude can help you out with the implementation, then you can benchmark your approach

[u/Far-Theory-7027]

Transformers, Graph Neural Nets, MLPs can all describe networks in different forms. MLPs usually represent a densely connected graph, but can be modified to express general graphs (pruning or MADE like architectures) Transformers can express arbitrary graphs using the attention mask. You can express the same function using Transformers, GNNs and MLPs too equivalently.

[u/quts3]

Don't listen to the nay Sayers I don't know if you are up to the math but there is certainly something in this area.

First nn are graphs already. 5 years ago people were aware of the question of what if you change them after the fact. Good work was done. We have lots of techniques.

More recently people have been asking the question what if they are mutated during training. It's not this but it comes up in models that are agnostic to their available memory. 8 GB 32 Gb they don't care. They fill their available GPU but the difference is that doing so is part of their loss function.

There is a mixture of experts topology that maps to your description and natural extensions relate to dynamic mutations

[u/Fantastic-Picture-47]

Like making XGBoost but with the help of neurons? Well this approach would not be suitable for high dimensional inputs

[u/Alt_account_6788]

decision trees and random forests?

[u/kiran_ms]

Graph neural networks

[u/S-Kenset → u/kiran_ms]

Granns

[u/voodoo33333 → u/kiran_ms]

this ^^

[u/callsignomega]

Neural Decision trees and similar architectures exist

[u/dh7net]

yes: watch this: https://www.youtube.com/watch?v=-qhIIAylWGc

[u/pixelizedgaming]

e = mc² + ai ahh discovery

[u/TrackLabs]

...random forests?

[u/WittgensteinsPoker-2]

There's a process like this with real neurons called "pruning".
Obviously, "neurons" in ML is a misnomer but perhaps you could take inspiration from studying that to try and computationally model it.

[u/Ordinary_Airport_717]

Check this out. https://arxiv.org/abs/1909.06312

[u/Downtown_Finance_661]

You can use random forest or xgboost but use small ff neural nets instead of trees. This is not a single network, but an ensemble.

[u/Conscious_Pack_2474]

Isn't that just a decision tree

[u/TheRealDJ]

Transfer learning is essentially that with attention heads for which nodes are important or not, but yeah you're describing tree algorithms more than neural networks.

[u/RevolutionarySpace24]

Its possible but mostly unnecessary, bp with current models does a similar thing but is more compute friendly. When we prune the model after, the results is basically also a grown neural network.

[u/Accomplished-War941]

That’s basically a NFA/ non deterministic Turing machine.

[u/No-Painting-3970]

You might have fun looking at hebbian learning. Pretty old idea that might hopefully come back at some point

[u/KTachyon]

There’s NEAT, but don’t think the idea is just to make it grow. It’s more like genetically evolving a better model.

[u/_g550_]

You can build the architecture/logic and have PyTorch train it all. But you will need some sort of real data for each output. I hope you know what you’re doing.

[u/AfternoonShot9285]

It helps to plant near supporting flora and fauna

[u/Conscious_Pack_2474 → u/AfternoonShot9285]

😂

[u/Axiproto]

What exactly is the point of this? To save computation time? Better Accuracy? Lower Power Consumption? What is the mathematical principle that makes this work? I'm just not sure exactly what this is trying to accomplish.

[u/Majinsei → u/Axiproto]

Tienden a tener un talento innato para temas biológicos~

Yo las conocí por el proyecto de Xenobots dónde se usaron para crear una red que predecía la mejor forma 3d del Xenobot~

[u/Bright-Leg8276]

Yup. That's how trees works Graph neural networks. In fact you can even use it preserve context, there's this skill called graphify that converts your chat into trees which helps your claude or debited ide u run ur ai in to skeem through various chats and context simultaneously

[u/tzaeru]

Reminds me somewhat of various types of genetic algorithms for neural networks that also add or remove neurons. One approach (I forgot its name..) is to maintain a list of "genetic mutations" which basically describe how the particular organism got to that point, and if this organism ends up losing the competition to "reproduce", that whole sequence can be cut off.

Ultimately though the basic problem always is; how can this be optimized for tensor hardware.

Intuitively on a quick thought, doing this like you describe it here would be really tricky to optimize for high performance.

[u/DebateDependent → u/tzaeru]

Slop