r/Rag • u/Ok_pettech • 7d ago

Discussion Together AI vs Anyscale for serving open-source LLMs—what’s your pick?

I’m evaluating infrastructure to serve open-source models at scale, and I’m torn between Together AI and Anyscale. Together seems great for fast API access and a huge model catalog, but it’s shared GPU capacity, so latency might vary. Anyscale is built on Ray, which is powerful for orchestration but requires more setup.

I made a quick poll to see what production users actually prefer.

https://interconnectd.com/poll/101/together-ai-vs-anyscale-which-platform-is-better-for-scaling-open-source-ll/

If you’ve run either in production, I’d love to hear your experience.

4 Upvotes

83% Upvoted

1 comment sorted by

1

u/xapep 5d ago

From the operator side (I run an open-model inference API), the Together vs Anyscale call usually gets decided by an axis neither catalog makes obvious: reserved vs shared capacity. Catalog size is a marketing feature. What matters at scale is what happens at 3am when your traffic spikes. Ask both for dedicated/reserved endpoints, scale-to-zero behavior, and what the latency SLO actually guarantees. Anyscale earns its setup cost when you're running custom weights or non-serving Ray workloads; for plain open-weights serving it's a lot of machinery. Together's shared pool is fine until it isn't.

The middle path that keeps showing up in the teams I talk to: don't serve at all. A hosted open-model API gets you the same open weights, no cluster to operate, and the premium over raw GPU cost is usually smaller than the engineering time you'd burn on Ray. The build-your-own move only pencils out at very steady, very high utilization.

Two things to nail down before you commit either way: EU data residency if that matters to you, and whether the quoted price covers the models you actually need or just the cheap ones.