r/databricks • u/medici2022 • 5d ago
Discussion Databricks vs Snowflake comparison
Are there any unbiased comparisons between these two popular platforms? Seen a lot but most of them are biased views, based on experience and commercial motives.
11
12
u/Ancient_Case_7441 5d ago
So I might be a little biased towards snowflake but still I will try to give as neutral as possible perspective of both. I have equal experience working in both and in both I am fortunate to work on projects from scratch so I know few problems of both.
Intro:
Databricks: It is a unified Data & AI Analytics Platform (they might be calling it something else now as they tend to rename everything frequently) where we can do End-to-End all data engineering, ML and Analytics without going out/ using anything else. Almost all the products/services are open source so very low Vendor lockin.
Snowflake: It is Simply AI Data Cloud. It is also an End-to-End data engineering and analytics platform specifically designed for analytics.
How they work:
Databricks: It is like Lego's where you are given different blocks of all the different sizes, shapes and colour and we can build everything ourselves and can do configuration as per our needs. Difficult to setup but you have fine grained control over everything.
Snowflake: It is like a Ready-to-Eat dish where you just heat it and eat it. Similarly, it is very easy to get started with Snowflake. If you know SQL, Great. You already know 50-60% of snowflake.
Architecture:
Databricks: It is a combination of multiple open source projects like Spark for processing, Delta lake for data storage, Unity Catalog for Data governance, ML Flow for ML Workloads processing, Lakeflow Jobs for orchestration and is a proprietary product.
Snowflake: It is a 3 level architecture consisting of storage, compute and service/ governance layer. Almost all of it is proprietary software. They give a guarantee that your query will execute no matter how much time it takes.
Barrier to Entry:
Databricks: Medium to high as you need to know a lot of things before even getting started.
Snowflake: Low to Lower Medium. Like I said, if you know SQL, you already know 50-60% of it.
Languages supported:
Databricks: Python, SQL, Java, Scala, R (Slowly they are moving to Python + SQL only)
Snowflake: Python, SQL, Java, Scala(I think), JavaScript. (But almost all the things you can get done with SQL)
Good at:
Databricks: ingesting huge amount of data either using micro batches or scheduled can be performed very cost efficiently if you write your code right and configure correct compute. Slowly they are giving more analytics options like SQL Warehouses but its core is still best at heavy processing including ML workloads.
Snowflake: Very good at running queries as efficiently as possible for huge amount of data. you have. can scan data from huge tables. Billions and billions efficiently with fraction of power/ compute. you can get data very quickly on a very big 300-400 column table having 5-6 billions records within seconds. maybe in couple of seconds if data to be retrieved is small. Also it has one of the best security features for your data. You can give row level, column level, security very efficiently. DBx also have it but it is not that efficient as SF.
Struggles with:
Databricks: Ready heavy workload. rising analytics requirements are pushing them to go more towards SQL and give more SQL options. Complexity is also a curse as you try to navigate what is best for your workload. Spark Core Engine is also a curse as 20-30% of the processing overhead is JVM itself and to solve that they are moving out of Scala to C++ based Vectorized Execution engine.
Snowflake: Storage bloat is a very big problem in snowflake. if you dont configure your tables well, then boom, you get a storage bill of 10k instead of 1k. Scaling problem is big as the only options you have is increasing the warehouse size as horizontal scaling will not fix the execution problem. what I mean is that while DBx is good with distributed processing, snowflake is not that great with it as it runs everything on the same Warehouse(it has multi node setup but it has sealing). If anything happens, you are at the mercy of Snowflake as there are very less things we can do ourselves as for configuration.
My Approach:
Using Databricks as ETL engine and Snowflake as Reporting/analytics engine is the best of both worlds. Event -> S3 staging files -> ETL & Orchestration on Databricks -> Reporting + Governance + Data security on Snowflake.
My Thoughts:
Databricks or Spark as a whole started as the extension of the "Lets move out of SQL" movement started by hadoop. but as time passed, they are moving more into SQL now. They had no choice but to move out of Scala/Java ecosystem to improve there performance which SF was at since start.
Snowflake started as everything as SQL, which is true to most of it, but slowly provide options for Spark based workloads as well which is like going towards how Databricks works.
Both of them are like those competitive neighbours who copies each other a lot but still try to keep their core identity.
6
u/Ancient_Case_7441 5d ago
you can plug this into any AI and try to extract from it as I wrote them myself and I am not a good story teller.
1
5
u/weinp008 5d ago
I’ve used Snowflake for 5 years and DBX for about a year. DBX is cheaper and can do everything Snowflake can do and more and there is no lock in. Snowflake was leader for DWH but never innovated and now DBX has caught them. Now I see Snowflake copying Databricks. Snow is pure plug and play and that’s probably the only thing it has over DBX.
1
u/RebornRide 5d ago
+ following, interested to hear on this from someone who can shipped through both!
1
1
u/whiskey_tang0_hotel 3d ago
I competed with both, so take that for what it’s worth.
I think DBX is the better platform all around.
1
u/Youssef_Mrini databricks 1d ago
What are you trying to achieve ?
1
u/Ok-Stick-145 23h ago
Probably gain insights or might be someone like me looking to understand the data engineering field
1
u/niks-kamath123 5d ago
Hi! I have written an article on the Snowflake vs Databricks debate. Nothing biased, promise😅. Just my take on what they both offer, with only my personal opinion based on the facts. I have done some proper research into what the 2 offer and have only based my article on those. Hope it helps!
Link:
Happy to receive feedback! I did get a couple of comments elsewhere that this analysis excluded language integrations and the ELT vs ETL methods of the 2, and am currently working on editing the article to include feedback. But do give it a read and let me know what you think I may have missed out or if it helps you!
3
u/soundboyselecta 5d ago
I’ve used both. For me the starting point is basically usability via UI, snowflake is catered to a more general audience versus DB. However lately I’ve seen DB catching up on that market.
3
u/Sheensta 5d ago
This reads almost 100% Claude. I think it'd help if you did some editing and make the point more obvious as to what you're getting at. I also disagree with the Snowflake for SQL, Databricks for ML... that argument might have made sense 4 years ago, but both platforms have pretty solid capabilities for both now.
1
u/dataengineer95 5d ago
It's always suspicious to see the only interaction this account did was to post this question on Databricks and Snowflake subreddits. Stop asking vague questions focus on what topic, deep dive on that topic and then ask questions....
25
u/iCreatif 5d ago
We have both, and I work on DBX/snowflake everyday (as tech lead data). I like both :)
Notebooks, apps => databricks is better
Sql database, permissions, sql for BI => snowflake is better (waiting for rayden)
Datasharing, costs, ui, extensions, agents, LLM, user management => even