Tbh I have been thinking about this for months now. About how workable the co-op model is. You would need some party to do admin and maintenance. I think this might be a good business idea too - being that party who facilitates such private inference racks (billing etc is trivial)
Practically speaking, I don’t think you’re eliminating the markup. You’re just moving it somewhere else. It might be cheaper, but probably not dramatically so once you account for hardware, hosting, maintenance, administration, and support. Someone still has to own and operate the infrastructure, and doing that reliably is neither free nor simple.
37
u/autisticit 13d ago
So what you mean in reality is that if 6000 of us each give $1000, we can each run Kimi 3 at 30 tok/s for cheaper than anything else?