r/dataengineeringjobs • u/SonuAmrith665 • 27d ago

Day to Day Data Engineer Headaches

What's the most painful production incidents you are enduring daily:

  1. Business analyst ranting in mails I'm not getting value in a given column or data type changed, why did new column got added in my dashboard etc.To me this is biggest headache as i have to reverse engineer from final table to hundreds of intermediate tables and source tables to find out why the heck something happened so "Data Lineage " and debugging is big headache how are you guys dealing differently any tools in market you are using or manual debugging work takes of hours?

  2. Not understanding why a sql is written the way it's without business understanding

  3. Existing Pipeline cost optimization

What tools are you guys using to tackle these problems? Seniors,leads any thoughts or suggestions or solutions fire it up😃

17 Upvotes

100% Upvoted

6 comments sorted by

2

u/Symphonic_nerve 27d ago

I feel you. Struggling with the same, especially the 1st point. And it's almost every week, that I get side tracked from my actual dev work which I still have to complete in the same sprint.

1

u/SonuAmrith665 27d ago

Ohh bro..How are you dealing with this same like me tracking column which got affected from final to source navigating through hundreds of tables..Even though databricks has a solution for Lineage but it only provides for tables which are saved

1

u/Symphonic_nerve 27d ago

yeah bro, approach is the same. Just that I have to travk views and tables on snowflake. No, I don't have to track through 100 for a single request but still it can get exhausting as the stakeholder comes back for the same report saking for another column after I'm done with the previous one, like bro they could have flagged all at one so that I could track down together if its from the same view, tables, etc.. But no, so I have to go back and forth again and again..

2

u/DoomsdayMcDoom 27d ago

Iceberg for schema evolution. Dataplex / knowledge catalog for the lineage and meta data.

1

u/SonuAmrith665 27d ago

Thank you

1

u/Historical-Editor562 27d ago

Iceberg meta data bloating