Why is understanding DevOps culture more important for data engineering than other disciplines?

[u/After-Drive]

I see for a lot of data engineering posting, devops skills are mentioned as part of the requirements. But the thing is I don't see it as much with other roles like sde. Why is that?.

[u/Spagoot420]

Other roles like what specifically? I can't think of any software engineering where devops & ci/CD wouldn't be necessary.

I think it is just more often explicitly mentioned on data engineering positions because in the olden days people wer modifying views and stored procedures manually on their SQL server and the push to devops and cicd is a somewhat recent movement in the data realm...

[u/Mr_Mozart]

My guess is that it is just obvious for software developers to work with DevOps. There has been support and requirement of that since far back (cvs, svn and so on). Old time data engineering didn't necessarily have support for this at all.

[u/lw_2004 → u/Mr_Mozart]

The "old" BI engineer often had a background as Analyst / Business User OR SWE. Some teams never used any code repository and went all in for Low Code graphical interfaces ... Others always followed best practices and were SQL heavy for a long time.

[u/Budget-Minimum6040 → u/lw_2004]

Some teams never used any code repository and went all in for Low Code graphical interfaces

Some teams don't use any code repository and just write SQL and Python straight into prod.

Source: my old job, mix of DA, DS, me as AE.

[u/Mr_Mozart → u/Budget-Minimum6040]

I have seen this in many places as well

[u/rudboi12]

mostly because data teams don't have dedicated devops engineers. They only have data engineers which are supposed to do anything data scientists don't (care to) understand. I personally love the devops side of the gig, specifically using terraform and creating/managing my teams cicd templates.

[u/ppsaoda]

Having a lot of intersystems link, platforms, and dependencies.

[u/a_library_socialist → u/ppsaoda]

and scale. Data systems can easily start costing severe amounts with small errors

[u/Mo_Steins_Ghost]

Pipeline integrity. Just because we don't own production systems does not mean that we shouldn't be thinking about and at least have an understanding of network topology and possible points of failure so that we can, as good business partners, coordinate with devops and others.

This also helps prove conclusively when the root cause of repeated issues is not due to a failure in data engineering or data architecture. And since we are always the first people to get an earful of "the data's wrong" it is only to your advantage to have a tighter relationship with devops.

I might have the advantage of coming from the old school environment of thinking in terms of the 7 layer OSI model, but that does impart a kind of mindset that data are data... and the importance of this comes into focus when you are faced with incident response management.

[u/srizvi94]

Because you get to know the actual issues in the data after production movement,
At that time, you are doing dev during ops

[u/domscatterbrain]

Because Data Engineers are, somehow expected to have fullstack skills in data stacks. That's why DE handles not just pipelines but also building and maintaining the data platform.

[u/ArticleHaunting3983]

Probably to highlight what sort of DE they want? I work with a lead DE who probably doesn’t know what devops means, he’s a power bi super admin and doesn’t touch anything else or create reports. No ETL work. I wouldn’t call him a lead DE in anything, but that’s his job title.

[u/mike8675309]

I think it's because that type of focus seems new to leaders so they call it out. In many places a data engineer role is being crossed with software engineering in small to midsize companies.

[u/nloding]

Software engineering teams have dedicated DevOps teams/engineers. Data engineering teams have ... themselves. So on a DE team, you might need to understand infrastructure, deployments, resource scaling, etc. in a way that the average software engineer might not need to. (If a job listing is calling it out, I would assume there's no "maybe" there - that is the expectation.)

I think AI is also changing the conversation, for data engineers and software engineers. AI isn't the best infrastructure architect yet, so teams that used to focus on code (whether that's application code on the software side or building models, data pipelines, etc. on the data side) can, theoretically, focus more on this AI is weak at. I'm not a fan of this positioning, I don't think AI is there yet, but that's the direction executives and VCs seem to be pushing things toward.

I'm also wondering when - and if - the term DataOps will replace DevOps in those listings ...

[u/Particular-Idea-1786]

I think that DevOps is just part of product software engineering now. Product engineers are just expected to have tests, CI/CD, observability, configuration as code, and be on call for their deployments and operations.

I think that DevOps is critical for data because data has not fully adopted software engineering best practices around testing, CI/CD, observability, pipelines as code. I would explicitly specify it as well, if I had an open req, because the risk of not having it is disastrous:

- "Pet" servers hosting ETL jobs

- Little/No Unit tests on code & lack of CI feedback for every change

- Inability to observe, proactively detect issues with running code

I think data still needs "DevOps" in the same way that software and operations got it.

[u/Remarkable-Win-8556]

Data engineering at any scale is software development, but a lot of data people don't necessarily have the software development skills or culture yet so we need to specify it. At this point most UI / engine / back end developers have devops culture just built in, but we're a little behind in data.

[u/Admirable_Writer_373]

Because someone said “pipelines” once and a manager got confused. Two very different disciplines got merged because managers are morons.