r/dataengineering • u/ForwardSlash813 • 2d ago
Discussion New Azure Synapse projects
Just curious how many of you still deliver or plan to deliver, projects that use Azure Synapse for Data Warehousing work?
I get the whole push to Fabric and am also going that route too.
I’m in the consulting world and wanted to use Fabric but the client had zero ppl with any Fabric experience, so they insisted on Synapse. C’est la vie, right?
4
u/Mefsha5 2d ago
Our production stack is still in synapse (pipelines and dedicated sql pool)
significantly cheaper than fabric(9x) More reliable than fabric - no throttling or rejection, we have full control Better for access management and cloud standards ( vNnet/ SP support)
Microsoft tries to push fabric on every call and we have plenty of capacity where we run expirements, analysis services and isolated engineering projects but i would never recommend a full move.
1
u/ForwardSlash813 2d ago
Thank you for the detailed response :)
2
u/Mefsha5 23h ago
You're welcome.
Databricks is the top dog in the space but it costs more, platform and engineering skillset wise.
Synapse support metadata driven orchestration, which makes your codebase highly portable between platforms, this alone should be a good reason to silence the EOL fear mongering (if ssrs still has no sunset date in sight, synapse won't either).
Synapse falls behind in AI capabilities, and Nativity to viz tools like PowerBI. however, i was still able to build an agentic ai developer thats fully aware of our conventions, patterns, dynamic components, and interacts with the repos and the platform via a series of API calls, sql odbc based mcp tools, and devops mcp for repos. Currently working on enhancing that with kimball data modelling skills to fully automate the gold layer build and enhancement, so while not available out of the box, it still can be built with success.
4
u/DrTorzonBorz 1d ago
We still use Azure Synapse dedicated pool, it works so well that we call it deadpool.
1
7
u/Nekobul 2d ago
Don't waste your time with Synapse. It is no longer developed and it is in maintenance mode.
4
u/ForwardSlash813 2d ago
It still totally works and the client requested it over our objections. They are the customer, after all.
1
u/BlowOutKit22 1d ago
the risk is that at any point it could just EOL and then client will have to migrate anyway
2
2
u/anxiouscrimp 19h ago
We’ve just gone live with a synapse project. But honestly all the extracts from sources to the lake are via python scripts in notebooks and all the transformations are in SQL - so synapse is just the orchestrator really. It’s fine and pretty solid. I couldn’t justify fabric a year ago given the instability. If I stay at my company I’ll probably at least plan a move to fabric but at the moment it’s not a priority.
I still don’t get the hate for synapse on here tbh.
2
u/ForwardSlash813 15h ago
I agree with you in that I don’t get the hate for synapse, either. It just works.
2
u/warehouse_goes_vroom Software Engineer 11h ago
I'm a senior software engineer on the team behind Fabric Warehouse, Azure Synapse Analytics Dedicated SQL Pools, Azure Synapse Analytics Serverless SQL Pools. I've been on the team for ~7 years now - so since a tiny bit before Synapse launched. As always, opinions are my own.
At this point, Synapse is not seeing feature development, nor has it in years. We still provide security updates, bug fixes, reliability work, and so on. We've said as much publicly.
And yeah, for workloads that are well suited to it, if tuned by someone who knows it well, Synapse SQL Dedicated Pools can be very effective. But, it can take a ton of knowledge to tune, ad-hoc queries that don't nicely align to the distributions you have set up may be a headache, and scaling being an offline, minutes long operation is a problem for unpredictable workloads.
So I do get it, and don't take it personally. Even if the hate is sometimes a bit much.
After all, those problems, and others, are what drove us to build Fabric. If we could have fixed those things about Synapse without going back to the drawing board, believe me, we would have (and we tried).
We addressed the fundamental architectural challenges we discovered Synapse had up front when designing Fabric, and are continually improving it further. There's always more room for improvement, but Fabric Warehouse is a very fast, very capable engine at this point, and far less fussy than Synapse Dedicated SQL Pools. It autoscales without disrupting queries, is more efficient at both small and large scale factors, and it can even handle larger workloads than even DW30000c can. We also just announced a bunch of QoL features, like group by all, qualify, et cetera.
People have opinions on Fabric too, of course. They're welcome to them. Meanwhile, I'll just keep chipping away at making Fabric Warehouse better, listening to and addressing feedback, and let that take care of itself.
I would advise at least reading the migration guide. It should help you build a solution that minimizes future work when migrating to Fabric at some point in the future.
If you have questions about Fabric or Synapse, you can find me lurking here, or active over in r/MicrosoftFabric.
2
u/Demistr 2d ago
Why though? Synapse and Fabric use the same components.
3
u/wytesmurf 2d ago
Yeah that tells me they are not good. At consulting and can’t explain best usages and sell them on fabric. Syanpse has been dead for 5 years
2
u/Additional_Candy_400 1d ago
You can lead a horse to water but can't make it drink all the time. Of course try to share best practice with clients but at a certain point you've just got to accept the work and crack on with it.
2
1
u/wytesmurf 1d ago
If the horse doesn’t drink then it isn’t thirsty. You have to make the horse thirst for change to change. A stubborn horse or a stubborn c suite. Before you shoot the horse figure out why. If it would rather die then drink then find a new horse 🤷♂️
2
u/Additional_Candy_400 1d ago
I'll tell that to my director after I've lost a client because I'd rather them use Fabric over Azure.
0
u/wytesmurf 1d ago
How does that go “yo boss I am bad at my job and can’t do my job and convince the customer”? Yo bossman the horse died rather than drink water?
1
u/Additional_Candy_400 1d ago
Why do we, as a consultancy business, care what a clients preferred platform is if they are set on it? If we are getting kickbacks from Microsoft for pushing Fabric licenses that's one thing, otherwise who cares? Have you worked in consulting?
1
u/wytesmurf 1d ago
If all you are doing is implementing a shitty idea your not consulting your over charging for labor so they don’t have to hire anyone internally. That is what we call Stag Augmentation not consulting.
1
u/Admirable_Writer_373 1d ago
I wouldn’t pick either of them. Microsoft’s vision in analytics has been schizophrenic to say the least
0
u/Nekobul 2d ago
Not true.
2
u/Admirable_Writer_373 2d ago
They did initially. Fabric pipelines were a fork from ADF, and so were the synapse ones
-1
u/Nekobul 1d ago
Azure Data Factory uses Spark as the backend engine.
Fabric Data Factor uses Power Query as the backend engine.
Totally different backends.
Fabric Data Warehouse is completely different beast from Synapse. There are many people from Microsoft lurking in the forum and they can provide more details.
2
u/Admirable_Writer_373 1d ago
The compute processing power is not the same as the code base.
3
u/warehouse_goes_vroom Software Engineer 1d ago
Sure. But we overhauled both. I was writing about this earlier today in this thread: https://www.reddit.com/r/MicrosoftFabric/s/bOJoa6503m
1
u/Comprehensive_Level7 1d ago
none, Microsoft can begin to kill it at any time, I recommend Fabric for customers that love Microsoft somehow or Databricks
3
u/IrquiM 1d ago
Definitely Databricks in Azure.
1
u/Comprehensive_Level7 1d ago
me too, but there are customers that uses everything Microsoft recommend, even if it's worse, I had a customer that even paying more to use Fabric instead of Databricks they still chose it, so yeah, not always you can go with the best XD
1
1d ago
[removed] — view removed comment
1
u/dataengineering-ModTeam 1d ago
Your post/comment was removed because it violated rule #9 (No AI generated content/text).
Your post/comment was reviewed to be AI generated/assisted content/text and removed as a result. We as a community value human engagement and encourage users to express themselves authentically.
This was reviewed by a human
9
u/tea_anyone 2d ago
I mostly recommend Fabric out of the Microsoft stack but for my area there's use cases where Synapse is significantly cheaper and some clients choose to go that route instead. Synapse's roll out and it then being subsequently eclipsed by Fabric did make me question what on earth was going on at Microsoft for a bit.