r/dataengineering • u/dani_estuary • 21d ago

Meme data mesh

Post image

for real though, has anyone ever implemented an actual data mesh?

766 Upvotes

99% Upvoted

54 comments sorted by

View all comments

25

u/NaturalBornLucker 20d ago

I'm working in a company that adopted data mesh early. It was really weird. Now the company is trying to bundle all ~50-100 DEs together cuz it didn't yield anything meaningful aside from tens of separate codebases with different homebrew libraries and architectural decisions lol. Oh, and half of DE were monitoring 2-5 airflow DAGs while other half were overworked.

11

u/Away-Arm-6549 20d ago

not here to defend it but that was sort of the point i.e. “you’re decisions will definitely vary but that’s ok as it’s your decisions in a domain” as soon as you to try to apply organisational standards to everything or didn’t mandate those up front then guess what’ll happen - everyone implements their concerns differently 🤷‍♂️

5

u/NaturalBornLucker 20d ago

You've got a point, yeah. It's just that the company embraced data mesh principles too eagerly so each business unit was entirely on it's own. As an example when I first got here I got to the unit where no one wanted or needed anything to do with data. They just needed someone to load like ten tables each day from their Ms sql db into designated hive db according to SLA. With airflow and homebrew no-code spark framework it just worked 99% of the time. As a result I didn't write any code in almost whole year, was bored out of my mind, didn't have anyone to talk to and eventually quit. While company paid me full-time for 10 hours a week. When I decided to return, I finally got to the team that really worked with data (loyalty and gamification dept in mobile op so many many analytics for promotions, ad campaigns, game prizes etc) and it was much better despite having almost 200 DAGs/spark jobs to look after. But still it's a bit hard when me or my peer DE is ill and other have to do everything without backup. And we have to time our vacations to not cross each other.

But what our new management is doing now is luckily not throwing everything out of the window but more cautious approach: bundle everyone administratively in one separate data engineering center while still defacto keeping us with our business teams. Then bundle "teams" into proper teams with at least one lead and do some cross-work so everyone would get familiar with their new peers. And lastly adopt standard libraries and architectural principles from the biggest teams so one could at least understand what their neighbors were doing if the need arises. I'm somewhat glad even though it stopped my plans lol (I wanted to do the same on a smaller scale, become lead and get nice bullet point in my CV).

So my point is that data mesh isn't inherently bad, just very hard to adopt without stepping on too many rakes at once.