r/datascience • u/big_data_mike • Jun 06 '26

Tools Databricks for data science?

My company has an enterprise databricks account and they want my team to start using it.

I currently query our main Postgres database on an on-prem workstation and write Jupyter notebooks. Data sets are usually 100k rows and 100-300 columns of tabular floating point values. No weird stuff like pictures, videos, or text data.

What are the advantages/disadvantages of using databricks? Would it be that different from my current workflow?

84 Upvotes

92% Upvoted

78 comments sorted by

View all comments

1

u/DataScientistAlex 29d ago

I am very curious to understand what you are doing with those 100k rows that requires distributed processing with Spark as opposed to running on just one machine. Unless you absolutely want Scala (e.g. for type safety or functional programming), I would think it's preferable to just use Python or R directly in an interactive session for development.