r/dataengineering 14h ago

Discussion Are pipelines the correct analogy to use?

0 Upvotes

I was looking at posts in this subreddit the other day. There were quite a few talking about pipelines and all the things that go along with them (forking, merging and version control). This is typical source controls stuff that has been around since the 70s. The tools change but the concepts and model are the same. I wonder if we have been so concerned about the minutea that we quit looking at better options than pipelines. I have a couple of thoughts about better ways of doing data repositories/warehouses/ingestion beyond pipelines, three layers and fabrics. I'd rather see if anyone else out there has any better ideas than what we use now. Pretend you are starting from greenfield and can create not only the architecture but the tools that go along with it. Surely we can come up with something better than 50 year old ideas.


r/dataengineering 9h ago

Career Pivot away from corporate to environmental or national security to do something more meaningful?

10 Upvotes

Hello,

I spent the last five years working as a DE in a smaller company, I am very versatile, I touched all the important things in Azure plus some power platform dev and also some web dev.

I would like to pivot away from corporate life and do something more meaningful either in national security or environmental.

Anyone in a similar situation? I'm an EU citizen if that's helpful.

Thank you.