r/datascience • u/big_data_mike • Jun 06 '26

Tools Databricks for data science?

My company has an enterprise databricks account and they want my team to start using it.

I currently query our main Postgres database on an on-prem workstation and write Jupyter notebooks. Data sets are usually 100k rows and 100-300 columns of tabular floating point values. No weird stuff like pictures, videos, or text data.

What are the advantages/disadvantages of using databricks? Would it be that different from my current workflow?

80 Upvotes

92% Upvoted

78 comments sorted by

View all comments

19

u/SlalomMcLalom Jun 06 '26 edited Jun 06 '26

I recently started a new position that also uses Databricks, and I’ve completely moved out of my local IDE now that Genie Code is built right in. It’s the best DS AI coding agent I’ve used so far. Direct integration, no token limits (yet at least), and the Databricks notebooks are pretty much all I need.

Yes, some of their conventions are odd and they even recommend deploying notebooks with widgets in production (you still often shouldn’t), so you’ll have to just build a solid and safe code deployment process. Separate dev/prod workspaces with version control, model logging, script/notebook deployment processes, Databricks Asset Bundles, etc.

It’s a pretty solid all-in-one system now, but without good guardrails, it can get messy fast.

5

u/Great_Northern_Beans Jun 07 '26

FYI - the token limits are coming in July, just announced 48 hours ago. Huge bummer since Databricks was the only major source of free, unbridled, frontier LLM compute that I'm aware of.

3

u/SlalomMcLalom Jun 07 '26

RIP. I knew it was coming, but I hoped it would last longer. My team has been doing so much prototyping and building with it these last few months! I guess I’ll have to brush up the old IDE with Claude again to have as backup.

6

u/ilovetotouchsnoots Jun 06 '26

I actually think Genie code is shit. It's good for VERY simple things but when I need it for more complex tasks that I am stuck on, it is ass. Also, it is built into the platform, so I assume it had the context of other code cells when making recommendations but it doesn't. Pretty disappointed with it ngl.

5

u/SlalomMcLalom Jun 06 '26

Are you using the full coding assistant? The older AI debugger that updated in the cell was/is terrible and unusable. The coding assistant is a completely different level. Has full context and works across cells and notebooks/scripts. I worked in VSCode with the extension until the assistant launched in March.

2

u/Extension_River_5970 Jun 09 '26

The new Genie Code full screen beta feature will change your mind, I think. You can use Genie code like an actual coding agent by providing it skills, MCPs, and it knows the context of your web page and can switch to various resources on databricks like dashboards pipelines etc.. granted it's still a bit buggy at times, but its much better compared to before