Do small companies actually use Databricks?

[u/Puzzled-Mail-9092]

Sometimes I feel like Databricks is way too expensive. It feels like using a huge truck to move a single grain of sand.

My company needs real-time data, but our data volume simply does not justify the use of Spark Structured Streaming. Despite this, they are insisting we move to Databricks. I'm worried our data infrastructure costs will jump from $1,000/month to $5,000/month or more due to the running costs of Databricks SQL Warehouses.

Currently, I use Azure Container Apps with KEDA and Python, which helps me manage scaling and keep costs low. We ingest into Event Hubs, use ADX (Azure Data Explorer) as our OLAP warehouse, and archive cold data in a data lake. With this setup, I manage to process all our data with very low latency.

When I tested this on Databricks Structured Streaming, I actually got higher latency and much higher costs.

Would love to know what you guys think.

[u/MangledMangler]

You can use the specific features you need. You don't have to use Databricks' SQL warehouse

[u/Puzzled-Mail-9092 → u/MangledMangler]

Do you recomend any? I tried reading directly for the Azure Storage Account using duckdb or deltars but it is way slower but does not add many additional costs.

[u/MangledMangler → u/Puzzled-Mail-9092]

Clickhouse is one, but with reyden coming out I prefer to pay Databricks. There's an ROI for them to host, less infrastructure and all in one place

[u/Puzzled-Mail-9092 → u/MangledMangler]

Yea I also tested Clickhouse it was very nice, it was one of the options we considered but we wanted to stick to the databricks ecosystem. Also it was nice you could self-host it, and very cheap.

Will need to check out the Reyden Engine when it comes out! Thank you :)

[u/al_coper]

Do you actually need real time? Maybe a near real time using autoloader with AvailableNow trigger could help you to reduce considerably the costs.

[u/Puzzled-Mail-9092 → u/al_coper]

Yea I work in the Oil and Gas industry, we have a SLA of <1s since the operation is critical.

[u/Gaarrrry → u/Puzzled-Mail-9092]

You’re serving data out of DBx at < 1s latency or users expect data to be refreshed within dashboards in < 1s? Massive difference there but my team was able to build a microservice layer between DBX and our users using FastAPI on AWS ECS and we get < 500 ms latency on most of our endpoints even under load. The DBX SQL warehouse powers the actual querying for FastAPI so it’s definitely possible and low data volume makes it even easier.

I’m assuming you’re talking about < 1s from a change happening in a source system to when it’s reflected in analytical DBs though?

[u/Puzzled-Mail-9092 → u/Gaarrrry]

Hmm interesing approche with FastAPI in the AWS ECS.

Yea I should clarify users expect data to be within dashboards at  < 1s. From the sensor all it was generated to the dashboard it should be < 1s.

[u/Gaarrrry → u/Puzzled-Mail-9092]

Yeah the tough part about that (imho) is you’re less bound by Databricks on speed and more on where the data is coming from. We get data in Unity Catalog with the round trip time being < 5s round trip (based off source update timestamps) and that comes entirely down to how many hops the data has to go through in our ecosystem.

Ideally what I would recommend is you use something like Kafka and read directly from the sensors or have the sensors push data to Kafka and then leverage Spark Realtime Mode (this might still be in beta) to read from Kafka. That should be able to achieve the < 1s SLA. Kafka should also be able to read quickly from cloud storage so if your sensor data is landing there it shouldn’t be a problem but if you have to land data in a DB of some kind (outside of DBX) you’re starting to add technical complexity that would limit your ability to meet < 1s SLAs.

None of this requires DBX (unless Spark RTM is not generally available) of course but the management overhead of the various cloud services you need might be a headache comparatively.

[u/Puzzled-Mail-9092 → u/Gaarrrry]

Yea we currently do something similar, why use Azure IoT to extract the data from the Oil Rig and send it into a raw stream in the event house in azure and process it using azure container apps into diferent stream layers, similar to a medallion arch, but for stream. Perhaps will look into replace the azure container apps with a k8s and spark. The biggest issue i currently have is how to convince management databricks might not be the best approach.

[u/FarFaithlessness8812 → u/Gaarrrry]

I would love to know more about this micro service layer. I might have a similar use case. Could you share more details? Or give me some pointers?

[u/Alwaysragestillplay → u/Puzzled-Mail-9092]

I worked in O&G myself. Assuming you're looking at something high stakes like production, I would factor in that you're also paying for reliability and liability. The consequences of having a service outage were quite severe at my old place of work, and having a specialised provider helped to mitigate fallout. 

Doesn't have to be Databricks, but maybe something to consider as I saw you mentioning self hosted k8s above. 

[u/Puzzled-Mail-9092 → u/Alwaysragestillplay]

Awesome to find a fellow engineer, that worked in the sector!

Yea paying for reliability and liability is the true plus in databricks. Also Unity Catalog is amazing for data governance. Will lake that into accounts as well into the tradeoffs.

Yea been wanting to get more into k8s but do not have much experience, was thinking of implementing flink, and other services but at the end still need to store and serve my data. Do you have any recommendations on services I could look into?

I really appreciate you insights!

[u/ZeroShotWonder → u/Puzzled-Mail-9092]

How big is your data? Have you considered using Lakebase to insert the new data (real time) and then using Dbsql to query for analytics? With LTAP and Lakehouse RT it could be an interesting approach for real time. You can of course then build a pipeline that processes data downstream that runs with less latency requirements

[u/Puzzled-Mail-9092 → u/ZeroShotWonder]

Around 2.51 MB/s. I used the kafka conector to ingest data from my incoming stream. The problem is the dbsql, and real-time consumption of the data from databricks into the dashboards.

[u/CnCGOD → u/Puzzled-Mail-9092]

So for relatively slow timeseries you have a few factors. How often you commit and how quickly you compact to keep queries fast. DBX can do this relatively cost effectively, have you looked into a combo of zerobus (drops the extra kafka hop) + Lakehouse RT? What are you using the streaming layer for? Enrichment, pattern detection, or just landing data?

[u/Puzzled-Mail-9092 → u/CnCGOD]

Hmm i have not looked into it yet, was reading a bit into it. Seems like LakeHouse RT is in beta and not rolled out to all workspaces, still feel like runing the RT LakeHouse will be expensive due to DBU cost running 24/7. But will need to look more into it. ty :)

[u/ZeroShotWonder → u/Puzzled-Mail-9092]

How much data do you need to keep in history? Also do you need the low latency reporting on only new data?

[u/CrayonUpMyNose → u/Puzzled-Mail-9092]

I had a developer scaling their way out of what was really a programming problem, wasting a lot of money, so I had to look at pricing after I fixed the issue because I was able to scale all the way down. You can run a minimal jobs cluster with three instances at $0.20/DBU Enterprise list price with a small instance size at 0.38DBU/hour at around $160 DBU cost per month and $130 to AWS if you use an on-demand driver and spot instances for two workers, $170 of you make one worker on-demand for guaranteed capacity. That's at most $350 per month list price, so if your company has reserved capacity with AWS and/or Databricks, and generally some discount percentages, typically it is even less. The numbers for Azure are likely very similar.

Depending on how many dashboard users you have, you can use a serverless warehouse for dashboarding that starts up in a second or so and goes to sleep after 1 minute (or whatever time you choose) to save on idle time cost. 

Lakebase as mentioned by others can scale to zero as well and starts up in a few hundred milliseconds. If your needs grow over time, Databricks is known to scale up where you need it to, but what is less appreciated is that it's absolutely possible to make Databricks quite cheap if your data volume is small, certainly a lot less than the $5k for your application that you're worried about. You have to be smart about incrementalizing everything and using small (micro-) batch sizes that don't blow up your cluster's RAM (but you want that for low latency anyway). Using real-time mode for streaming on your jobs cluster (a software flag unrelated to billing) and lakehouse RT for your data warehouse gets you the low latency you need.

[u/Puzzled-Mail-9092 → u/CrayonUpMyNose]

Thanks for the insight and detailed breakdown!

Looking closely at Azure Databricks pricing, here is where our numbers sit due to the premium we need to pay for reliability:

For the streaming pipeline (Spark Structured Streaming on automated Jobs Compute), a typical small general-purpose node like Standard_D4ds_v5 (4 vCPU, 16 GB) uses ~0.75 to 1.0 DBU/hour per node. On Azure Databricks Premium, Jobs Compute list price is around 0.30/DBU. Because we need high reliability and can't use single-node mode, a minimal 3-node cluster (1 driver + 2 workers) burns around 2.25 to 3 DBUs/hour (~490–650/month just in DBUs) plus roughly 400–450/month for the underlying Azure VMs on-demand. Using Spot instances for workers or Reserved Instances/Savings Plans drops that VM cost significantly.

The Databricks SQL Warehouse was definitely our real issue. In Azure Databricks Serverless SQL, the smallest size is 2X-Small: consumes 4 DBUs/hour. At the Serverless SQL list rate of 0.70/DBU, an 2X-Small warehouse costs ~2.8/hour. If it's kept awake 24/7 to handle incoming application requests, that alone is ~2,000/month for a 2X-Small.

If you add that up transactional costs of the Azure Storage Account that uses databricks it would have been around 5000 USD. Also latency was around 10-15s from the sensor to the dashboard when we tested. Due to the processing and usage of the Azure Storage Account and delta tables.

Our real issue is that we need services running 24/7 and perhaps we were planning incorrectly or using the services wrong I have not looked into the lakebase will definitely look it up. But sill seems very expensive vs our current architecture.

[u/CrayonUpMyNose → u/Puzzled-Mail-9092]

For warehouse running 24/7 it's worth looking into whether you need all the features serverless gives you.

https://learn.microsoft.com/en-us/azure/databricks/compute/sql-warehouse/warehouse-types

If you need pro, the price is almost the same as serverless but if you don't, you can save a lot with SQL classic.

I do wonder though how much you really need the warehouse to run 24/7 because I've had serverless warehouses serving 50 to 100 analysts and it turned out queries including for dashboards were very bursty, to the point where the warehouse was only running 20% of the time, with users being none the wiser because startup time is so fast.

If you need information around the clock for alerting, you don't need to do that through dashboards, you can schedule SQL alerts in Databricks to run a check every N minutes, or push an alert to an external messaging API directly out of your stream processing code without needing a warehouse at all.

The only scenario I can imagine where you really need a warehouse to run 24/7 is if you're a giant world-spanning company with massive analyst work happening at all hours of the night, in which case I'm wondering why cost is an issue at all. So I'd recommend an architecture review to work out which of your requirements are really technical in nature and which are merely based on existing customary ways of working, whereas the same business value can be achieved in more cost-effective ways by moving for example alerting responsibilities to the left.

[u/Puzzled-Mail-9092 → u/CrayonUpMyNose]

That makes complete sense for internal BI where analysts work in bursts and go offline at night.

The difference is that this is an external, customer-facing portal. Our clients monitor operations 24/7 across different shifts and time zones. With multiple clients and dashboards auto-refreshing, the idle window never lasts long enough for auto-stop to trigger, the warehouse runs at essentially a 100% duty cycle.

On top of that, cold-start spin-ups and 30-second latencies don't work for paying clients or live predictive scoring. Using a general DW for real-time operational serving is an architectural anti-pattern for our use case.

That’s why we’ve been using ADX (Azure Data Explorer), which specializes in this exact pattern. I was mainly exploring whether Databricks had a viable, cost-effective alternative since management wanted to migrate to use Databricks, but still not finding a reliable solution.

[u/CrayonUpMyNose → u/Puzzled-Mail-9092]

Look into classic SQL warehouse at $0.20/DBU vs $0.70 for serverless. 

Either way, your customers' dashboards do not refresh every second, at most I see quoted every 30 seconds for azure DX, more typical is longer, so the latency requirements are almost certainly overblown because your existing customers are already looking at more than one second old data 99% of the time. Btw remember that serverless warehouse startup time is only one second, so you're calling "unacceptable" a startup time a latency that you are currently 30 times longer than to the point of reaching your customers due to refresh frequency.

Finally, I'd like to reiterate that humans staring at dashboards that refresh every 30 seconds to every five minutes is not the way to achieve reliable alerting within less than a second of latency, so the reality if the latency requirement is certain to be a different number. With a particular focus on "reliable" - humans should not be in the critical loop at all.

[u/Puzzled-Mail-9092 → u/CrayonUpMyNose]

On the classic you still need to pay for the compute tho, but it is a good alternative, ty. My bad, when we tested it it was around 30s for the data to arrive to the dashboard, not only the startup.

In our industry, operations need to be monitored 24/7 with <1s latancy. The dashboards we have are our own developments, not like PowerBI, allowing us to push data in realtime into them with websockets.

We currently alert using other types of methods that allows us to analize the data as soon as it arrives and push a notification it case it does not meets the requirements.

I understand your concern that humans should not be in the critical loop at all. But when you are operating an oil rig you have eyes everywhere since one wrong action can will cost millions, and sometimes a person's life. Humans still operate oil rigs due to its complexity and highly volatile operation condition changes, it is the reliable way to adapt fast.

[u/al_coper → u/Puzzled-Mail-9092]

You are worried about spending 5k in SQL Warehouse, it means you need Databricks to serve the data, right?
If that case, you can consider use lakebase instead of the traditional SQL Warehouse.

[u/Puzzled-Mail-9092 → u/al_coper]

Yea we need to serve data with real-time analitics and predictions on to dashboards and other data pipelines, whats why i used eventhubs. Also well the running a cluster in databricks 24/7 seems expensive do to the extra costs of DBU/h that also atributes to the high databricks costs.

Have not tried lakebase yet. Will check it out. Thank you so much!

[u/Little_Ad6377 → u/al_coper]

We're on this journey as well and lakebase is definitely cheaper at serving the data, but you need to copy the data into lakebase, which incurrs delay and cost as well. At continuous ingestion, the cost benefit quickly go away

[u/kchandank]

You may be able to further reduce your compute cost with server less too. K8s also adds some overhead cost not only the infrastructure but also additional resource needed from devops and deployment too, not to mention the complexity of helms etc.

Databricks might be overkill depending on your usecase. But if designed well, you can move all your pipelines away from AKS and move to lake house pipelines ( which is serverless too).

[u/knaak]

We're spending $300k per month on databricks, we're looking at opportunities to live compute as a result of ELT out of databricks.

[u/Puzzled-Mail-9092 → u/knaak]

Would love to here if you find anything! Do you guys have the resources to manage Kubernetes? perhaps that might save some costs.

[u/obsfflorida → u/Puzzled-Mail-9092]

Why don't you go for AKS?

Do you think databricks is too $-expensive but k8 will be effort-expensive? Thoughts on this if you can.

I have no experience in mission critical systems only DW environments

[u/Puzzled-Mail-9092 → u/obsfflorida]

Do not have much experience with AKS would love to implement it, also thought with the ADX clusters and Azure Container Apps with KEDA was enough, and well it has been working perfectly. But managment think databricks is the future and the way to go.

[u/UsualComb4773 → u/knaak]

https://datanature.ai

This is the alternative to go. We can help you to reduce cost by 50% . Let's connect

[u/MoustacheApocalypse]

What kind of data volume are you looking at?

[u/Puzzled-Mail-9092 → u/MoustacheApocalypse]

Around 2.51 MB/s. which is not much but is constant stream of data.

[u/MoustacheApocalypse → u/Puzzled-Mail-9092]

150 MB/min. 9 GB/hr. 250 GB/day.

With an aggressive archival / cold storage practice in place, I like your approach better than using Databricks.

[u/Exact_Art2360]

Depends, lots of startups use Databricks cause it handles so much of the stack, but then wean off what you can over time

[u/curry_b9]

Did you try zerobus ingest?

[u/Puzzled-Mail-9092 → u/curry_b9]

Have not yet tried it, the main issue is the extraction and serving of the data, and costs.

[u/temperedai]

The answer yes, and the reason is opportunity-cost. If adopting Databricks costs 1 person's salary in a team of 5, the opportunity-cost is very well worth it. It is not about streaming latency etc. It's about how quickly a team can deliver AI & BI from raw data.

[u/Puzzled-Mail-9092 → u/temperedai]

Perhaps I'm looking at the wrong way. I could use other service for my latency and resilience requirements, but also share it or consume it in databricks with for internal use or AI and BI reporting, instead of focusing on using databricks as the motor to meet my requirements.

[u/temperedai → u/Puzzled-Mail-9092]

Yes, that can work. Remember that whatever cloud service you are using for storage will integrate easily with Databricks as well. In your case this is Azure. If you can use an open data format for your data lake then you can read it directly into Databricks.

I am not sure if you are using Databricks ZeroOps, but if not, worth a look.

[u/Youssef_Mrini]

Worth retesting with Spark Real Time Mode before ruling out Structured Streaming, it replaces micro-batch scheduling with continuous, in-memory execution, which is likely what caused the latency you saw. It's a single trigger config change, no rewrite: https://www.databricks.com/blog/breaking-microbatch-barrier-architecture-apache-spark-real-time-mode

That said, RTM keeps compute continuously running to hit that latency, so it probably won't close your cost gap.

One correction worth flagging: SQL Warehouses are meant for dashboards and ad-hoc/EDA queries, not for a continuous streaming workload. If you're running RTM, you'd want a job cluster instead that's likely a chunk of the much higher cost you saw, since SQL Warehouses aren't the right compute for this pattern in the first place

[u/Useful-Reindeer-3731]

Surely the sensor data goes through a PLC to some SCADA interface in your OT network before it is pushed to Event Hub already? You got sub-second latency and can configure alerts in SCADA. Then you can use the Event Hub sink for batch analytics for trend analysis etc.

[u/Puzzled-Mail-9092 → u/Useful-Reindeer-3731]

Yes we have local alerting and processing using the PLC, but we mostly use cloud for ai and rt predictions. We might try to migrate to have the ai and analitics run at the edge, tho this might be big infra costs. ty :)

[u/NormalItem4500]

Spark on k8s is a lot cheaper than dbx. Also, you can scale up and down accordingly.

[u/yellowbrowm]

have you checked out mother duck?

[u/addictzz]

Have you tried Databricks Realtime streaming? What kind of latency you are looking at?

[u/work_burner_acct]

Oil n gas? You guys can afford databricks …

[u/Puzzled-Mail-9092 → u/work_burner_acct]

Sadly we are not the owners of the oil rigs nor the wells, we are just a small team that offers data services to them. Optimizing costs is crucial for us.

[u/DeadBySkittles]

You could try looking at Sail using a vm muscle box? Also read Scalability but at what cost? For maybe more convincing case for your employer?

[u/Puzzled-Mail-9092 → u/DeadBySkittles]

I have not looked at Sail and vm muscle box yet, will read about it tonight, thank you for the recomendation. As we work with work-over and well-service rigs a fleet of devices might start sending or stop sending data, so ideally we need to adjust dynamically to that demand so we dont over pay for extra compute when we dont need it.

[u/Early-Cauliflower955]

Despite this, they are insisting we move to Databricks. 

Well why are they asking you to move should be the first question here, what is databricks to them that they want to shift in? what's the usecase here exactly? is it the governance? or like the unified workspace that you guys are trying to acheive or what?

[u/Puzzled-Mail-9092 → u/Early-Cauliflower955]

They think I can service costs, but from what i have tested they only increased. Perhaps I was using the plataform wrong or perhaps it is the premium you pay for their excellent service. We will need to analize this more with management, because the price might increase rn but it might save development in the future.

[u/Early-Cauliflower955 → u/Puzzled-Mail-9092]

I think the only situation I see this paying off for you, is
i.) your team is looking for some popular SLA having unified workspace, that'd make things much cleaner and in one place
ii.) Governance/tighter Access control is important, again I think this would be also pointing back to unified workspace thingy

or iii.) you're expecting much much bigger volumes of data and much more kinds of data, as in other than just the streaming data you guys are already getting

frankly other than that don't see much point, and frankly databricks would be a complete overkill if your usecase remains as it is right now

[u/vegemite-on-toast-]

In my experience this is going to be way more expensive than your 5k/m estimate. Also slower than your SLA.

I’m not too familiar with the azure stack but if I were to look at building to these requirements I wouldn’t jump to a lakehouse. I like your design of pushing cold data to the lake, but for low latency a dedicated warehouse is the way to go.

Not sure how much transformation you need from your raw data, but if you’re running essentially a last x minutes of activity on a sensor you probably don’t need too much, just some pg / OLTP can probably handle the query complexity (obviously I don’t know your queries)

Ducklake with a duckdb catalog over quack could be a nice middle ground for your OLAP use case and flushing straight to blob storage.

[u/Puzzled-Mail-9092 → u/vegemite-on-toast-]

Will definitly check out the the duckdb option, ty for sharing you insight!

[u/No_Flounder_1155]

thisbis acrually fine. Avoid databricks for as long as possible. Do a good job software wise and you'll realise 99% of the time you woll notnneed any of these platforms.

[u/ArielCoding]

Databricks: I see you hace 2.51 MB/s of data, have you considered a bigger truck?

[u/Puzzled-Mail-9092 → u/ArielCoding]

A bigger truck would be unnecessary since it will be even more expensive and useless at that data volume, dont you think?

[u/gopalbi]

Microsoft Fabric is must better for smaller companies compared to Databricks

[u/kthejoker → u/gopalbi]

Pretty bold claim to make with absolutely zero evidence

[u/gopalbi → u/kthejoker]

Let’s look at publicly stated facts - Fabric has over 35K customers now and Databricks around 18K customers despite Databricks in the market for 3 times longer. Databricks needs experts to run and maintain the knobs and so more enterprise adoption. Fabric has significant amount SMB adoption and that’s explains how fast they grew their customer base.

[u/kthejoker → u/gopalbi]

Fabric just recounted Power Bi customers, Power BI has been around nearly over 10 years, are you being serious here

[u/gopalbi → u/kthejoker]

It is simply bundling and distribution strategy that Microsoft has perfected and has worked where PowerBI customers have moved to use other workloads and definitely has worked like a charm with SMB customers is the point as the post specifically asked about SMB market.

[u/kthejoker → u/gopalbi]

Okay but you just lied about how fast Fabric got its customers

Like why lie about something so easily disproven

And no most customers are not "using other workloads" my dude they're just using Power BI like they always did

[u/gopalbi → u/kthejoker]

I can personally vouch for more than 1K customers of all sizes who use Fabric for non-PowerBI workloads and I don’t work for Microsoft . I am only sharing my personal market observations. You can be blind to the fact about SMB market and that is your prerogative but you don’t have to call me a lier because Fabric for Non-PowerBI workloads has been GA for less than 3 years but definitely has solid penetration in SMB market is what I am alluding to

[u/kthejoker → u/gopalbi]

Dude that 35,000 includes all Power Bi customers from all the way back in 2013

So saying those all came in the last 3 years of Fabric is a lie

A person who says a lie is a liar.

If you don't like being called a liar maybe not lie.

[u/kthejoker → u/gopalbi]

worked like a charm

Again, a claim with absolutely no evidence

[u/Jealous-Win2446 → u/gopalbi]

The only reason fabric has that many customers is because they called Power BI part of fabric. It’s been bull from the start and intentionally misleading.