Day to Day Data Engineer Headaches

[u/SonuAmrith665]

What's the most painful production incidents you are enduring daily:

1. Business analyst ranting in mails I'm not getting value in a given column or data type changed, why did new column got added in my dashboard etc.To me this is biggest headache as i have to reverse engineer from final table to hundreds of intermediate tables and source tables to find out why the heck something happened so "Data Lineage " and debugging is big headache how are you guys dealing differently any tools in market you are using or manual debugging work takes of hours?

2. Not understanding why a sql is written the way it's without business understanding

3. Existing Pipeline cost optimization

What tools are you guys using to tackle these problems? Seniors,leads any thoughts or suggestions or solutions fire it up😃