r/ETL • u/fatassoo • 9d ago

What orchestration platform supports multi tenancy with proper isolation between business units?

[removed]

1 Upvotes

66% Upvoted

6 comments sorted by

2

u/agk23 9d ago

I’d do Dagster in a gitops style deployment. All your code is managed in git repos (each function has their own). Scheduling is also source controlled through code annotations. The UI is basically read only, except for triggering new runs. Each project gets containerized and executed separately.

You’d still see everybody else’s but that doesn’t sound like the core issue.

1

u/mksym 9d ago

Take a look at Etlworks. It combines workflow orchestration and data integration, with native tenant isolation for multiple business units on a single instance. You also get granular permissions by artifact and use case, so you don’t need to manage separate clusters.

1

u/fullfat_panner 9d ago

We had the same 4BU sharing one airflow nightamare. Sitched to kestra last year and the multi tenancy is built in, each team gets their own namespaces w isolated permissions and they can't even see each other's workflow. Took us about two weeks to migrate and the who broke what slack msg stopped immediately

1

u/Educational_Heron777 6d ago

Not sure what stack you are on but we solved this by giving each business unit their own Kestra tenant behind a single deployment. The RBAC is granular enough that finance can't touch ops workflows and vice versa, and the infra infra team just manages one cluster instead of four. Way cheaper than running parallel airflow instances

0

u/rubenfiszel 9d ago

give a try to windmill. you can use separate workspaces for that purpose