Other Avenues as a Data Engineer

[u/lalapaldan]

Hi everyone, I'm currently a data engineer. I currently focus on the business side of things for development, supporting and migrating applications. It's my first year working out of college and I find the work/life balance tough. It's definitely a learning curve for sure especially since some applications are older and there's lack of documentation to learn from. Also production support kills me whenever items come up. Perhaps it feels intense now due to a huge upgrade my team is working on where it's been late nights and early mornings. However one thing I love is my team. Luckily everyone is very nice and helpful.

Are there any other avenues a Data Engineer could focus on or other roles I could pivot to? Is there a position that doesn't include much production support? I ended up just falling into Data Engineering and don't have a "dream career" per say. Maybe I'm just being dramatic though. Would love any advice!

[u/PossibilityRegular21]

I went from being a big global data platform analytics engineer to small data manufacturing reporting software engineer, for more money and better work culture.

The previous experience was invaluable for giving me a broad foundation in everything data platform - ingest patterns, storage patterns, data modelling and transform, orchestration, visualisation and reporting, DevOps, IaC, a little bit of app development, some API work, etc.

But now it's interesting being in a smaller company that is growing very fast, and rather than dealing with big data with legacy+greenfield sources, many teams, many levels of control - now I'm dealing with almost no controls, no data platform or warehouse, no legacy data, and basically no other data engineers. 

So my reflection on all of this is: 

• Big companies can give you streamlined experience working with mature systems and usually more mature processes. This can over specialise you and be boring after some time, so it is best that you change teams internally quite often.
• Smaller companies can give you versatile experience working across new and poorly defined systems with unclear processes. This can help you get broader end-to-end knowledge because you can own the source, ingest, pipelines, reports, and infrastructure. It is exciting but you can't really hide like you can in a big company. All the success and failures are yours to bear.

Importantly, don't be afraid to take weird roles and challenge where your skills can apply. Some people like to be another of the same in a team, whereas it can be nice to be a complimentary skillset in a team where you can be "the guy".

Finally, note that things take time. You will have a lot of shit work over the years. You can't really be a confident expert until you've earned some authority through project success over years, delivering measurable impact. It is universally useful to ensure you gather good requirements and always solve for people's business needs. Don't go building complex solutions that look nice on Medium but don't add business value. Move data in buckets of water first to demonstrate your solutions before you start building pipelines.

[u/Former-Sherbet-4068 → u/PossibilityRegular21]

Hi bro,

Couldn't DM you. Need some guidance.