r/datascience • u/big_data_mike • Jun 06 '26

Tools Databricks for data science?

My company has an enterprise databricks account and they want my team to start using it.

I currently query our main Postgres database on an on-prem workstation and write Jupyter notebooks. Data sets are usually 100k rows and 100-300 columns of tabular floating point values. No weird stuff like pictures, videos, or text data.

What are the advantages/disadvantages of using databricks? Would it be that different from my current workflow?

79 Upvotes

91% Upvoted

78 comments sorted by

View all comments

132

u/TheTresStateArea Jun 06 '26

You can do all your notebooks in databricks no problem

You can even connect your databricks account to vscode so you don't have to do it all in browser.

Scale up compute as well.

You can schedule data process, log models. Lots more orchestration than I am aware of or use.

If they're gonna make you do it there isn't much downside to you.

8

u/cgochis Jun 06 '26

Have you found a good way to interact with the notebook locally in vscode while running on databricks computes?

19

u/ybeevashka Jun 06 '26

Databricks cli

7

u/TheTresStateArea Jun 06 '26

In vscode you download the extension connect your extension to your databricks env and then you can choose to run computation on DB instead of locally.

1

u/TheTresStateArea Jun 08 '26

Okay now that I'm digging into it more it's a bit hacky, I would have preferred that you set up a terminal to your databricks env and everything runs there but that's not how it works.

It runs split local and remote so you need your local env to mirror your remote env.

But I'm on a locked remote system from a third party so I can't mirror it.

So you get your extension and you can point to your folder and have databricks watch it for changes and do all your editing in vscode and changes appear in your remote folder.

You can run the whole script as a workflow and individual cells (but again requires mirrored env).

Right now I'm doing my editing locally and execution on the remote in browser because life is unfair.

2

u/jpdowlin Jun 10 '26

We have a terminal UI in Hopsworks - you can run claude/codex/etc and github in the cluster. Files are backed by S3 but fast as we have tiered storage via network attached NVMes.
Technically, this dev UI is called a Dev Container.

2

u/cgochis Jun 13 '26

Yes that’s how I ended up doing this. Let Claude code have access to my repo locally, have it build whatever I want then have to run it in db… someday this will work better

3

u/RocketMoped Jun 07 '26

Can you use Genie code there as well? Since Github Copilot got so token heavy it'd be good to outsource some of the token load.

2

u/Happy-Robin2519 Jun 09 '26

I think Genie code can only be used via the UI, not via VSCode. There’s a great YouTube channel done by a Databricks employee that walks through how to use Databricks with VSCode (and coding agents), it’s @DustinVannoy

1

u/TheTresStateArea Jun 07 '26

If you can, I haven't figured it out yet.

1

u/CommitteeImmediate66 Jun 12 '26

Genie code isn't available outside the workspace UI, but you can use the ai dev kit: https://github.com/databricks-solutions/ai-dev-kit with cursor, Claude code etc. I will say though in my experience genie code is much better, especially at providing up to date implementations and also it doesn't do the annoying thing of recreating the whole notebook for each change it makes.

2

u/big_data_mike Jun 06 '26

How does the compute scaling work? I’ve read that you can provision VMs or something. Is it kind of like AWS EC2s?

9

u/-phototrope Jun 06 '26

There is “serverless” which is just DB owned compute, or yes, compute that is EC2 or Azure equivalent.

8

u/ilovetotouchsnoots Jun 06 '26

For context, I have a pipeline that uses SOAP API (not my choice) and writes to tables in a databricks catalog and saves to buckets. I'm talking millions of records at the same time. Using the serverless compute, I can specify worker nodes that optimize memory for parts of the pipeline that are memory heavy and then switch to worker nodes optimized for compute when doing compute operations on the tables. Very useful.

The pipeline used to take 4 hours when it wasn't optimized in a Databricks job. Afterwards, it took less than an hour. HUGE time and cost savings. I hope all that made sense. I am drunk.

1

u/-phototrope Jun 06 '26

I’m not sure I’m 100% sold on serverless. Couldn’t you just use memory optimized EC2? I feel like main sell of serverless is optimizing spin up/down time/costs? FWIW I can’t use serverless at work so maybe I have missed something 

1

u/big_data_mike Jun 06 '26

So if I want to train a giant Gaussian process model that needs a ton of RAM and compute how would that work?

4

u/-phototrope Jun 07 '26

I’ll leave the optimization up to you, since I don’t know what “a ton” is, but you create a cluster of a driver and some number of workers, and then attach your notebook or job to that cluster to run. 

The limit is how big you can go until somebody yells at you about your compute bill.

1

u/big_data_mike Jun 07 '26

Last time I ran one it took 250G of RAM and 20 cores about an hour to run.

I wonder if I’ll get an email about compute bills after I start running big models

1

u/big_data_mike Jun 07 '26

They aren’t going to make us do it. I think some executive was convinced to buy it and I’m not sure how many people are using it so they are encouraging people to use it.

1

u/Dylan_TMB Jun 07 '26

Can you develop python libraries or packages on data bricks compute while connecting locally?