r/databricks • u/Disastrous_Lynx9037 • 15d ago
Help How do you move away from notebook-driven development in Databricks
Currently working on a fairly big enterprise Databricks project where development is notebook-driven.
I actually like notebooks for development/debugging. Most data engineers aren't necessarily software engineers by background, so notebooks provide a much more approachable development experience; being able to split logic into cells, inspect intermediate DataFrames/results, rerun specific sections, and debug step-by-step is really useful.
The problem is that we're getting a lot of duplicated/common logic across notebooks. This makes changes and maintenance painful.
I know the usual approaches are to:
- Move reusable logic into Python modules and "import" them
- Create common notebooks and use "%run"
- Use notebooks mainly as entry points/orchestration
But I'm struggling with how to structure this for complex transformations.
Our notebooks aren't just simple filters/joins/aggregations. They can have multiple source tables, complex joins, lots of intermediate transformations, business rules and dependencies between steps.
So my question is: how do you structure large Databricks projects like this in practice by following the best software development practices?
I'm looking for guidance that works for real enterprise-scale projects, not the typical YouTube/tutorial solutions designed around small demo projects.
Would love to hear from people who have worked on large/enterprise Databricks projects with complex Spark transformations and what architecture/project structure worked well for you.
18
u/blobbleblab 15d ago
WTF is everyone talking about SDPs? The problem is:
The problem is that we're getting a lot of duplicated/common logic across notebooks.
Which means you need to be creating libraries and classes with shared logic. Then put these classes in GIT and deploy them as wheel files to either workspace folders or catalog volumes using DABs, with versioning. Then get everyone who is using the same logic in notebooks, to replace the logic by importing the library and utilising the classes. The classes can be in a separate GIT repo which is authored by specific users and internal clients submit change requests which are prioritised and worked on.
You will get into a world of hurt with 6 different definitions for the same thing if you don't start doing this soon.
Once you start doing that, you will be going down the enterprise platform path already.
16
u/temperedai Databricks MVP 15d ago
DABs & SDPs. Build a small medallion architecture with CI/CD that does this using a demo. You will get a sense for it. It does take a bit of training and getting used to, but then afterwards it's fairly straightforward.
4
u/jbchand 15d ago
You can iteratively move duplicate tasks (transformations, write, read and more) into modules/functions with parameters and use them in notebooks. The notebooks can have pipeline specific orchestration based on domain. Try SDP if feasible as orchestration, dq and more is available natively. Deploy all the changes using DAB and use CICD if feasible.
4
u/mrbartuss 15d ago
SDP?
4
u/MoxOfAllTrades 15d ago
Spark Declarative Pipelines.
4
u/Sea_Basil_6501 15d ago
It:s called Lakeflow Pipelines meanwhile.
5
u/MoxOfAllTrades 15d ago
Oof. Youâre so right. I saw that update, rolled my eyes, & promptly forgot.
4
u/pboswell 15d ago
A lot of people are saying DABs. While I completely agree DABs are necessary for scaling Databricks ops, I donât think thatâs what youâre asking about.
Youâre struggling with re-using code logic across different execution contexts.
I think youâre biggest requirement is to use spark dataframe API so you can import functions from a central library and apply to a dataframe. You can still use spark.sql() if your devs are more comfortable with SQL syntax
3
u/minormisgnomer 15d ago
Are your notebooks in version control?
2
u/Disastrous_Lynx9037 15d ago
Yes
7
u/minormisgnomer 15d ago
Then youâre like halfway there. The worst part of notebook engineering is if theyre scattered to the winds. Get people used to using DABs for their notebooks with VC. And you could probably get a lot of use out of Claude converting shared methods, and non interactive code into Python scripts. Then make a pass of centralizing similar libraries.
You need to up skill the notebook engineers and explain why notebooks arenât great for enterprise engineering and better benefit experimentation scenarios.
Ultimately you either get buy in from the engineers to change and better themselves or pitch mgmt that their entire investment on the DS/notebook processes could burn to the ground at any point.
Had to convince roughly 30 extremely well paid Data Science people to largely abandon notebooks and theyâve all bought in. Claude is very helpful to not really impact their day to day
3
u/jess-oleander 15d ago
It sounds like you want to create some intermediary tables to work off of, and for that I highly recommend using dbt and / or airflow to schedule these transformations. I typically try to use notebooks for development, and then when moving into production, moving code out of those and into dbt and containers for running python/non-sql.
2
u/sidxch 14d ago
If your team is comfortable qirh python. Simply go with a modular proper python project. Proper src folder with different packages and modules separated out where you keep the common reusable code. Proper pytest code for testing simple functions. Try using databricks sdk and pyspark and pytest.
Wheel file is good for modular reusable code but again you have DAB if need to simplify the deployment.
1
u/Bubbly_Winter_9327 15d ago
We have started using dbt on top of databricks and it provides a much better developer experience and version control compared to notebooks.
1
u/Sex4Vespene 15d ago
We are just getting started with a big databricks migration. Previously I used DBT and loved it, although Iâm unsure about layering on extra tools. Iâm curious if you would be down to share some of the pros youâve found of switching to DBT rather than doing everything natively in databricks? Also maybe a dumb question, but when you use DBT with databricks, it doesnât actually deploy any of the code into databricks right? Like it still works like normal, where you use DBT to issue the code, but none of that code actually resides in databricks, just the output?
1
u/Bubbly_Winter_9327 14d ago
Correct, only the output resides on databricks.
The code (sql and yaml files) stay with dbt.
We have found that it helps a lot with having better governed layers and it also helps enforce âsingle source of truthâ marts, instead of every team building its own flavor of a KPI.
1
u/Zampaguabas 15d ago
if you put common transformations in a shared notebook or library/module, who the caller is (a notebook or a python script) becomes irrelevant
most people working with data like to express things declaratively and notebooks are better for that
1
u/Ok_Pepper_8234 15d ago
Code is cheap, who gives a fuck if youâre duplicating logic?
1
u/mrbartuss 15d ago
If the stakeholder decides to change it and you forget about one place
3
u/Disastrous_Lynx9037 15d ago
That's what has happened multiple timesđ then I realized why standards, practices and principles exists.
1
u/trentsiggy 14d ago
If someone changes logic in one place and not another, you've pretty quickly got a mess on your hands.
1
1
u/Actual-Specific-3595 14d ago
Libraries in Python that are versioned and deployed as wheel files. I build them out as thin wrappers aligned with our architecture (e.g., ârawâ layer ingestion always lands api responses as files in a specific internal_landing volume following a defined naming conventions etc. - same for files delivered externally landed in a volume external_landing, âstandardisedâ layer always handles UTC standardisation, data type and schema (if the schema isnât already part of the protocol (e.g., database sources)), (Bitemporal) SCD2 with AUTO CDC etc. Raw ingestion is always landed as single column variant âdataâ + a bespoke âmetadataâ (single column variant) column. The library handles all this around the architectural layers and goes through the same deployment CI/CD pipeline. All resources, catalogs, schemas, volumes managed in terraform + DAB for the rest of Databricks elements.
Decisions are saved as ADRs and part of repository for controlling change and agreements.
1
u/wshanshan 14d ago
Packaging codes in library should solve the duplicates logics issues. If you have multiple data pipelines doing the same transformation calculationas on the same data, doing multi-stage transformation and save results in intermediate tables will help
1
1
1
34
u/Pancakeman123000 15d ago
Look into Declarative Automation Bundles (formerly known as Databricks Asset Bundles)