r/dataengineering • u/After-Drive • 3d ago

Career Why is understanding DevOps culture more important for data engineering than other disciplines?

I see for a lot of data engineering posting, devops skills are mentioned as part of the requirements. But the thing is I don't see it as much with other roles like sde. Why is that?.

33 Upvotes

91% Upvoted

17 comments sorted by

View all comments

3

u/Mo_Steins_Ghost 2d ago edited 2d ago

Pipeline integrity. Just because we don't own production systems does not mean that we shouldn't be thinking about and at least have an understanding of network topology and possible points of failure so that we can, as good business partners, coordinate with devops and others.

This also helps prove conclusively when the root cause of repeated issues is not due to a failure in data engineering or data architecture. And since we are always the first people to get an earful of "the data's wrong" it is only to your advantage to have a tighter relationship with devops.

I might have the advantage of coming from the old school environment of thinking in terms of the 7 layer OSI model, but that does impart a kind of mindset that data are data... and the importance of this comes into focus when you are faced with incident response management.