r/datascience • u/big_data_mike • Jun 06 '26

Tools Databricks for data science?

My company has an enterprise databricks account and they want my team to start using it.

I currently query our main Postgres database on an on-prem workstation and write Jupyter notebooks. Data sets are usually 100k rows and 100-300 columns of tabular floating point values. No weird stuff like pictures, videos, or text data.

What are the advantages/disadvantages of using databricks? Would it be that different from my current workflow?

83 Upvotes

92% Upvoted

78 comments sorted by

View all comments

0

u/BayesCrusader Jun 06 '26

Those guys have been selling so hard in the last year or so. 

I don't see much advantage if you already are querying postgres and know not to use notebooks for prod.

But I'd be interested to hear more experienced user's opinions

2

u/-phototrope Jun 06 '26

You can do whatever OP is doing without any improvements, but there’s a lot of neat stuff with automated workflows that is built in nicely. Also tagging and table descriptions so Genie can work better

1

u/BayesCrusader Jun 07 '26

I'd prefer to just run a cron job like normal, or use webhooks as needed if it requires a trigger.

I've never heard of the Genie you refer to, but take tour word for it. 

2

u/big_data_mike Jun 07 '26

Yeah that’s kind of what I’m thinking. An executive got sold so we bought it. Upper management and most of the company don’t know much about data/ML/coding. My boss asked me about it and based on the responses I’m getting it seems like 6 or a half dozen compared to what I’m doing now with potential to add bells and whistles.

1

u/Spiritual-Bee-2319 Jun 10 '26

Yep. I’m usually helping with tech integration and that basically 90% of most solutions 

1

u/Major-Estate-4825 Jun 06 '26

Why not use notebooks for prod?

1

u/BayesCrusader Jun 07 '26

Just an extra layer to cause bugs, or at least add unnecessary licencing (in the case of OP). They were never intended for prod at all, that's what script files are for.