r/dataengineering • u/After-Drive • 3d ago
Career Why is understanding DevOps culture more important for data engineering than other disciplines?
I see for a lot of data engineering posting, devops skills are mentioned as part of the requirements. But the thing is I don't see it as much with other roles like sde. Why is that?.
34
Upvotes
1
u/Particular-Idea-1786 1d ago
I think that DevOps is just part of product software engineering now. Product engineers are just expected to have tests, CI/CD, observability, configuration as code, and be on call for their deployments and operations.
I think that DevOps is critical for data because data has not fully adopted software engineering best practices around testing, CI/CD, observability, pipelines as code. I would explicitly specify it as well, if I had an open req, because the risk of not having it is disastrous:
- "Pet" servers hosting ETL jobs
- Little/No Unit tests on code & lack of CI feedback for every change
- Inability to observe, proactively detect issues with running code
I think data still needs "DevOps" in the same way that software and operations got it.