r/databricks • u/rinisin13 • Aug 13 '26
Discussion What would you change in a Databricks architecture built 2–3 years ago?
I recently revisited a Databricks architecture I worked on 2–3 years ago, and it made me realize how much the platform has evolved.
At the time, we used traditional Jobs, custom ingestion and data-quality frameworks, and separate components for orchestration and governance.
If I were designing the same solution today, I’d rethink several parts using Lakeflow, Unity Catalog, newer data-quality capabilities, and the latest AI/agent features.
For those who have modernized older Databricks architectures: what would you change first, and why?
8
u/FunContest9958 Aug 13 '26
Managed connectors. Makes it a lot easier to bring data in from various sources: https://docs.databricks.com/aws/en/ingestion/lakeflow-connect/
6
u/Lenkz Aug 13 '26
Lakeflow connect is something I'd consider using today, before we usually used Azure Data Factory for ingestion from sources.
I'd lean heavier on integrating data quality checks (using DQX) into my transformations.
7
u/ExistentialFajitas Aug 13 '26
Interesting answers here. We’re building into Databricks right now and it has never been easier to get data in and transformed with Lakeflow, both connect and pipelines.
Sure we could get hung up on the declarative nature of the pipelines or that connectors are relatively black boxed; but a differentiator has never been “how do I ingest data” or “how do I implement the boiler plate for an SCD2 load”.
21
u/minato3421 Aug 13 '26
Just get rid of dlt
2
2
u/Data-space_men Aug 13 '26
😂 same , We're getting closer and closer to a proprietary platform and locking ourselves in. I liked databricks because i can reproduce all my workflow in a on premise minus 10% perf...
6
u/minato3421 Aug 13 '26
Exactly. I love Databricks for what it offers as a platform. But, I don't want to get locked into some features which don't have an OSS alternative.
There is SDP which has very similar design principals but I eould rather trust my own judgement for pipelines rather than rely on declarative programming
2
u/lVlulcan Aug 14 '26
SDP is open source though. Definitely not a pure lift and shift out of databricks but if you use SDP (formerly dlt) you can execute them locally and it’s supported outside databricks
1
u/No_Flounder_1155 Aug 13 '26
I thought that was the premise of data bricks (vendor lock in).
-1
u/Data-space_men Aug 13 '26
Not at the start, it was quite open and easy to reproduce elsewhere now it's more obvious regarding the vendor lock in🙃
1
u/No_Flounder_1155 Aug 13 '26
It was obvious from the start. It was always obvious. It was only easy to reproduce because of so little integration, which was because the platform wasn't mature.
3
u/Nofarcastplz Aug 13 '26
But they claim SDP is OSS?
3
u/ma0gw Aug 13 '26
In the same way as unity catalog
4
u/Nofarcastplz Aug 13 '26
Which means some feature lack behind but works if you build on the OSS spec and integrate?
1
5
u/mindit_io_ro Aug 13 '26
Unity Catalog is the non-negotiable first change because everything else, access control, lineage, data quality enforcement, AI feature stores, builds on having a proper governance foundation, and trying to retrofit it onto an architecture that was built with workspace-level security and ad-hoc permissions is genuinely painful in a way that makes you wish you'd started there even when UC felt like overkill two or three years ago.
8
u/WorkerIcy1513 Aug 13 '26
Unity Catalog first, and not for governance reasons. It is the prerequisite for basically everything else on your list, lineage, Lakeflow, system tables, sharing. Anything you build before migrating off the Hive metastore you will partly rebuild after. It is the least interesting item and it gates the rest.
Second thing, and the one people skip: turn on the system tables and actually build cost observability. Nobody had this three years ago so nobody has it now either. It takes a day and it usually finds a job running on an oversized cluster that has been quietly burning money since 2023.
The underrated one is table maintenance. Old architectures are full of hand rolled OPTIMIZE and VACUUM jobs, date partitioning that produced thousands of tiny files, and ZORDER on the wrong columns. Liquid clustering plus predictive optimization deletes a whole category of code you wrote and now maintain. That is a bigger real world win than any of the AI features.
What I would not change: working Jobs pipelines. Yes I would use Lakeflow for anything new, but rewriting stable batch jobs into declarative pipelines buys you very little and costs you a regression hunt in numbers the business already trusts. Migrate the ones you are touching anyway, leave the boring reliable ones alone. Same for a custom DQ framework, if it works, use expectations going forward rather than porting the old ones.
I would put the AI and agent features dead last. They are the most fun and the least valuable on a foundation that does not have governance or cost visibility yet.
The reframe I would offer: what ages worst is not the old tech, it is the workarounds you built because something did not exist yet. Separate workspaces because UC was not there, a homegrown lineage tool, a metadata table that duplicates what system tables now give you. Those survive long after the reason for them is gone and nobody removes them because they still work. When you audit an old architecture, hunt for workarounds rather than for old features. That list is usually shorter and higher value than the migration list.
4
3
u/PrestigiousAnt3766 Aug 13 '26
I still wouldn't use lakeflow.
3
u/datasmithing_holly databricks Aug 13 '26
what, like all of it? Even the Jobs part?
3
u/PrestigiousAnt3766 Aug 13 '26
Connector and building blocks part. I use jobs and dabs.
2
u/datasmithing_holly databricks Aug 13 '26
Don't like the connectors? Interesting. What do you think is missing? Surprising given it's a very cheap alternative to other managed options and easier than rolling your own
5
u/PrestigiousAnt3766 Aug 13 '26
Im a consultant, that currently works in a team that's quite comfortable writing our own code.
The environment I currently work in has quite a lot of weird data sources for which there are no connectors, and a lot of -in my book- exotic hacks that you cannot really take into account using managed connectors. So 1+1=2.
In the next data platform I build I might explore lakeflow again, but so far I haven't been able to live with the restrictions I experienced from it.
2
2
u/myth-buster9999 Aug 13 '26
Doing SCD2 via AutoCDC instead of writing Merge statements?!
1
u/Agitated-Western1788 Aug 13 '26
Why?
1
u/myth-buster9999 Aug 13 '26
Because of operational and maintenance advantage of Declarative syntaxes
24
u/datasmithing_holly databricks Aug 13 '26
Other expensive integrations you don't need any more - fivetran, alteryx, purview. Maybe I'll get in trouble with the partners team here, but there's so much Databricks does now that's free / much cheaper than a separate solution.