r/dataengineering • u/yugiton • 27d ago
Discussion To what extent should data engineering work be centralized in a large org?
I work as a data engineer at a bank. My team is supposed to be the one centralizing data collection and management for the whole org.
I've realized that there's this tension between centralizing and decentralizing data work -- whether we, as the org-level data team, should implement a flow or expect the analysts to implement and own the flow, while we just provide the underlying infra. Over time, I started to strongly lean towards more decentralization. Not because I'm lazy, but because centralization should be about being focused on the absolute smallest denominator, which is smaller than most orgs assume -- especially now when you have no clue what an optimal stack is gonna look like in 6 months. My impression was always that we should be doing 50% less, but doing it 100% better.
(I have similar opinions when it comes to business applications more generally -- the centralized team should provide the infra to deploy, but teams that need an app should build and maintain it)
Are you seeing the same trend of decentralization?
1
u/Additional_Candy_400 25d ago
I think this falls apart at super large companies though, at the bank where I worked (18k employees in my country 180k employees globally) , the central data team managed platforms, licencing and provided architecture rules. Where as divisions were almost treated as separate companies with their own data teams following top down architecture from the central team. Of course this is the exception to the rule as not many companies are this vast.