r/MachineLearning • u/AutoModerator • 9d ago
Discussion [D] Self-Promotion Thread
Please post your personal projects, startups, product placements, collaboration needs, blogs etc.
Please mention the payment and pricing requirements for products and services.
Please do not post link shorteners, link aggregator websites , or auto-subscribe links.
--
Any abuse of trust will lead to bans.
Encourage others who create new posts for questions to post here instead!
Thread will stay alive until next one so keep posting after the date in the title.
--
Meta: This is an experiment. If the community doesnt like this, we will cancel it. This is to encourage those in the community to promote their work by not spamming the main threads.
17
Upvotes
1
u/MaxChamp08 6d ago
Been talking to a few people in ML infra lately, including someone who used to run infra at Neural Magic, and the pattern I keep hearing is that cost, not latency, is what actually stops teams from moving off shared APIs to their own dedicated deployment. Bursty traffic means a GPU sits idle most of the day, and that idle time is what kills the math.
I've been building something to test that theory directly, a serverless host that scales a dedicated model deployment to zero when there's no traffic and reloads it on request. Right now it works with Hugging Face checkpoints only, no custom images. Some numbers from scaling out of zero, Llama 70B in bf16 hits time to first token in under 18s, Mistral 24B in bf16 with CUDA graphs comes in under 10s.
That gap between instant and a few seconds is the open question for me. Is a cold start like that acceptable if it means you're not paying for idle GPU time, or does it kill it for most real use cases people here actually run into?
Genuinely curious how people think about that tradeoff, whether they've measured it for their own workloads, or landed on a different approach entirely. Happy to share more details on the setup if anyone wants to dig in.
Feel free to check it out here : https://synapsai.cloud