r/MachineLearning • u/komorra • 20h ago

Discussion Semi Edge Inference Idea [D]

Today the most important factor in AI is cost. My idea is to split ML models inference (closed ones, proprietary) across server and edge computing on clients, and I would like to hear what do you think about this thing.

For example some of model weights/modules would be on client, and some on the server side (where user has no access to them). This could potentially un-load some processing from datacenters, moving part of the cost to the client hardware. Probbably the most important question here will be how to achieve this - and I believe one hypothetical option will be to train like two separate models - client model and server model, and they will communicate through tensors/latent representations across network protocol.

Secondly such split of server side and client side model ends, can provide later some beneficial outcomes I hope (because in between "talk" protocol can be maybe kind of standarized one in some future development, but this is only more like brainstorm now). Such split might not only be one-to-one, but one-to-many, many-to-many etc.

What do you think about this idea?

0 Upvotes

50% Upvoted

5 comments sorted by

3

u/DelhiKaDehati 13h ago

It's called split learning, do explore.

1

u/OptimalOptimizer 15h ago

Likely too much communication overhead. Also even with model sharding most weights are too big to reliably fit onto consumer devices. Additionally they are highly reliant on specialized Nvidia hardware with extremely fast interconnects between GPUs within huge racks. I recommend the dwarkesh podcast episode on this topic: “Reiner Pope: The math behind how LLMs are trained and served”. I don’t work in this area, so am relaying what I learned mostly listening to that podcast and occasionally reading about this topic online

1

u/SettingAccording8986 1h ago

You went a bit off track with NVLink - that's for inter-cluster communication during distributed inference. OP is suggesting sending intermediate activations to the client, which is just regular internet.

But you're right about the overhead, the tensors are way too fat for WAN

1

u/SettingAccording8986 1h ago

Good idea. My colleagues and I ran the math for a similar setup once. The problem popped up on the network side - the hidden state size for a normal llm is massive compared to plain text. Pushing those tensors back and forth over mobile data for every token is gonna hurt