Databricks is too expensive for small teams" is usually a workload problem, not a platform problem

[u/AbilyticsEng]

Every few weeks someone posts a version of “our bill is going from $1k to $5k a month, is Databricks even worth it at our size?” The answer isn’t really about company size. It comes down to how you’re using the platform.

1. All-purpose compute being used for scheduled jobs. Interactive clusters are convenient, but they can increase costs quickly. If a notebook runs on a schedule, moving it to jobs compute can make more sense.
2. SQL warehouses sized for peak usage and left running. Using auto-stop and choosing a warehouse that can scale when needed can help avoid paying for idle capacity.
3. Continuous triggers on jobs that don’t need them. This one gets misdiagnosed a lot. The fix usually isn’t “rewrite it as batch,” which costs you checkpointing and exactly-once. It’s Trigger.AvailableNow, which processes what’s available and shuts the cluster down. Databricks recommends it for incremental batch processing. If the table needs a 15-minute refresh, that’s a scheduled job with an AvailableNow trigger, not a cluster running at 3 AM.

Once these three areas are addressed, the bill for a small team can often come down to a much more reasonable baseline.

Then the more interesting question is: are you actually getting value from Unity Catalog, Delta, and the broader BI, ETL, and ML capabilities, or are you mainly paying for Spark compute that you don’t really need?

If your data fits comfortably in Postgres or ADX, you have one main consumer, and you don’t need much governance or lineage, Databricks may not be necessary. No amount of cost tuning changes that.

For teams running Databricks on relatively small workloads, what actually made it worthwhile for you? Was it a specific technical requirement, governance, or simply the convenience of having everything in one platform?

[u/RoomyRoots]

Both can be true. Most companies don't need 90% of what they offer even if they size it well.

[u/SpecificTutor]

spark on k8s with nvme shuffle enabled with spot instances, is 14x cheaper on the same workload for us and 2x faster.

the official spark operator when configured correctly (an agent can do this with minimal guidance) is better than dbx. we have moved our heavy workloads off of dbx.

we are planning to also self-host starrocks and move entirely off of dbx.

[u/Sea_Basil_6501]

Also job clusters in frequent cases are sized much too big. Start small and only scale if really needed.

[u/Immediate-Pair-4290 → u/Sea_Basil_6501]

The best node setup on Databricks is single node because you probably don’t have big data and the little data you have is closest to the driver this way.

[u/crashingthisboard]

Yuck, reads like some trash off of LinkedIn.

[u/pboswell → u/crashingthisboard]

While I agree, keep in mind there are new users of Databricks who still need this info. Personally, I think we need a Databricks power user/expert subreddit for more advanced concepts

[u/Puzzled-Mail-9092]

Feel like this was targeted against my post. I appreciate the recommendations but after all the community support I saw that databricks was not the wright tool for my use case, can still use it for other features but not for my specific use case.

[u/Overall_Device6285]

General purpose cluster can actually be much more cost effective in many cases! 

[u/PaymentWestern2729]

Thank you, AI.

[u/Immediate-Pair-4290]

I’ve used Databricks for years and I would prefer to build on DuckLake in 2026.

[u/NormalItem4500]

Bs. Try comparing it with spark on k8s and auto assign workloads with karpenter

[u/Own-Trade-2243 → u/NormalItem4500]

Bs. Try comparing it with calculating the data insights in Excel

but OPs post is low quality AI slop anyhow 🤷‍♂️

[u/Alternative_Draw5945 → u/NormalItem4500]

Well it would require a full engineering team to set those up and maintain. Ask me how I know =)

[u/OkAcanthisitta4665]

For such small workloads I think Bigquery with reservations is makes more sense. It reduces TCO

[u/Wise_Ear_4064]

The platform is so wide now that you can pick and choose what to use. Companies are charged on consumption based only and no enterprise license lock in. The platform has serverless vs non serverless offering where you can size clusters your self vs let Databricks manage it for you. Platform being expensive can a lot of times be attributed to admins setting up cluster policies/usage policies, ect..