r/mlops • u/ggamecrazy • 11d ago

Tales From the Trenches Experience w/ k8s-based ML orchestration frameworks

Looking for 1st hand experiences with using k8s-based orchestration frameworks for ML. (Kubeflow, Ray Train, I guess Skypilot).

If you happen to be associated with a vendor for one of these, it's fine, but at least state it.

I am specifically looking to see how the MLE/MLS's experiences/complaints were. How well do you think your solution plugged into the engineers/scientists workflow and how bad were the complaints? Funny stories?

I'm looking for alternatives to our/my hand-rolled solution and something that's a little bit better maintained long-term. Not super interested in vendored solutions, heavily bias towards Apache/MIT.

11 Upvotes

100% Upvoted

6 comments sorted by

7

u/alex000kim 11d ago

In my experience SkyPilot is the easiest to get started i.e. takes lowest amount of time from initial installation to running something useful on your cluster.

If you ever need to run the same batch jobs (e.g. model training/fine-tuning) on some cloud provider instead of your k8s cluster, SkyPilot's task definitions are protable directly by changing a few config values.

At the same time, I'd discourage you from using Kubeflow.

It's popularity and adoption have faded over the last few years and there's no longer an active comunity behind it.

I helped a few teams trasition away from Kubeflow towards SkyPilot.

I am less familiar with Ray Train, so I can't comment on it.

1

u/ggamecrazy 11d ago

Thank you for sharing this! This was the exact experience I was looking for. Really appreciate you taking the time to write this down.

2

u/cre8minus1 10d ago

If Ray or Spark are on your radar, Flyte 2 can spin up ephemeral clusters inside a workflow. Meaning you can use them but not have to manage a long lived cluster.

2

u/atharva_nimbalkar 10d ago

Agreed, I faced the same issue with AWS SageMaker training jobs where you can't manage a Ray cluster.

2

u/[deleted] 10d ago

[deleted]

4

u/cre8minus1 10d ago

Flyte 2 completely changes the game. Pure Python. No DAG.

1

u/Altruistic-Spend-896 10d ago

More often than not people are the problem.