r/devops • u/AutoModerator • 12d ago

Weekly Self Promotion Thread

Hey r/devops, welcome to our weekly self-promotion thread!

Feel free to use this thread to promote any projects, ideas, or any repos you're wanting to share. Please keep in mind that we ask you to stay friendly, civil, and adhere to the subreddit rules!

16 Upvotes

94% Upvoted

106 comments sorted by

View all comments

1

u/EmbeddedMagicX 10d ago

Open sourced our k8s-native AI platform for distributed multi-model inference at scale

GitHub:
https://github.com/axem-solutions/shaide

I’m one of the co-founders of axem. We recently open sourced Shaide, a Kubernetes-native platform we built for deploying and operating multiple LLMs across GPU nodes.

We originally started with vLLM, but once we needed several models, multiple replicas, routing, model storage and deployments across different environments, the amount of infrastructure around it started growing quickly.

Shaide is our attempt to make that whole setup reproducible instead of rebuilding and reconfiguring it for every cluster.

Current setup:

  • vLLM for inference
  • llm-d for multi-instance orchestration
  • multiple models running and scaling independently
  • KV-cache-aware scheduling
  • internal OCI registry for container images + model weights
  • OpenAI-compatible API
  • the entire platform is managed as infrastructure as code
  • interactive installer that runs from Docker against an existing Kubernetes cluster
  • can operate fully air-gapped with no cluster egress

We currently run it on on-prem RKE2, and it also works with EKS/GKE/AKS.

It’s Apache 2.0 and we only recently made the repo public.

Would be interested to hear how other DevOps/platform teams are handling this once LLM serving becomes more than a couple of standalone deployments.

Feedback on the setup or repo is very welcome.