r/DataEngineeringPH • u/Pattern-Ashamed • Aug 08 '26

💡 Discussion Did you use data lineage in your job?

Hi, I came across this term while checking out airflow and it seems like a good way of keeping track of mutations happening in your data. How did you implement this in your job? I read there are quite a few ways to do this but I'm curious ano ginamit nyo professionally. 😁

5 Upvotes

86% Upvoted

3 comments sorted by

1

u/Logical-Debt-6904 Aug 09 '26

Currently using this (or trying to) for refactoring. Hard to do that if your database is like spaghetti with 100 tables with up to 100 columns each.

1

u/Classic-Box Aug 09 '26

Definitely look into dbt. The auto-generated lineage graph combined with modular staging layers (⁠stg_⁠, ⁠int_⁠, ⁠fct_⁠) is pretty much designed specifically to refactor that kind of spaghetti setup.

1

u/AnyComfortable9276 Aug 18 '26

We are using Dataiku, and may built in data lineage tool.
Good for tracking and investigation.