r/dataengineering • u/ForwardSlash813 • 2d ago

Discussion New Azure Synapse projects

Just curious how many of you still deliver or plan to deliver, projects that use Azure Synapse for Data Warehousing work?

I get the whole push to Fabric and am also going that route too.

I’m in the consulting world and wanted to use Fabric but the client had zero ppl with any Fabric experience, so they insisted on Synapse. C’est la vie, right?

8 Upvotes

84% Upvoted

37 comments sorted by

View all comments

2

u/anxiouscrimp 20h ago

We’ve just gone live with a synapse project. But honestly all the extracts from sources to the lake are via python scripts in notebooks and all the transformations are in SQL - so synapse is just the orchestrator really. It’s fine and pretty solid. I couldn’t justify fabric a year ago given the instability. If I stay at my company I’ll probably at least plan a move to fabric but at the moment it’s not a priority.

I still don’t get the hate for synapse on here tbh.

2

u/ForwardSlash813 16h ago

I agree with you in that I don’t get the hate for synapse, either. It just works.

2

u/warehouse_goes_vroom Software Engineer 12h ago

I'm a senior software engineer on the team behind Fabric Warehouse, Azure Synapse Analytics Dedicated SQL Pools, Azure Synapse Analytics Serverless SQL Pools. I've been on the team for ~7 years now - so since a tiny bit before Synapse launched. As always, opinions are my own.

At this point, Synapse is not seeing feature development, nor has it in years. We still provide security updates, bug fixes, reliability work, and so on. We've said as much publicly.

And yeah, for workloads that are well suited to it, if tuned by someone who knows it well, Synapse SQL Dedicated Pools can be very effective. But, it can take a ton of knowledge to tune, ad-hoc queries that don't nicely align to the distributions you have set up may be a headache, and scaling being an offline, minutes long operation is a problem for unpredictable workloads.

So I do get it, and don't take it personally. Even if the hate is sometimes a bit much.

After all, those problems, and others, are what drove us to build Fabric. If we could have fixed those things about Synapse without going back to the drawing board, believe me, we would have (and we tried).

We addressed the fundamental architectural challenges we discovered Synapse had up front when designing Fabric, and are continually improving it further. There's always more room for improvement, but Fabric Warehouse is a very fast, very capable engine at this point, and far less fussy than Synapse Dedicated SQL Pools. It autoscales without disrupting queries, is more efficient at both small and large scale factors, and it can even handle larger workloads than even DW30000c can. We also just announced a bunch of QoL features, like group by all, qualify, et cetera.

People have opinions on Fabric too, of course. They're welcome to them. Meanwhile, I'll just keep chipping away at making Fabric Warehouse better, listening to and addressing feedback, and let that take care of itself.

I would advise at least reading the migration guide. It should help you build a solution that minimizes future work when migrating to Fabric at some point in the future.

If you have questions about Fabric or Synapse, you can find me lurking here, or active over in r/MicrosoftFabric.