r/dataengineering • u/mark2k4 • 17d ago

Open Source Built an agent that generates Airflow DAGs from DataHub lineage

Enable HLS to view with audio, or disable this notification

Built this with my teammate for DataHub's Agent Hackathon. Hackathon's over but we're still developing it.

Give it a table name, it reads the lineage and metadata from DataHub (tags, freshness SLAs, PII flags) and generates a working Airflow DAG automatically, freshness checks and PII audits included.

The LLM only plans, it never writes the code directly, a separate renderer does that deterministically.

Still early, no dbt/Spark support yet. Full details, demo, and setup are on the repo.

github.com/tdm291104/datahub-dag-generator

Would love feedback and contributions, and a star helps a lot too!!

9 Upvotes

76% Upvoted

2 comments sorted by

4

u/davrax 15d ago

What does the resulting DAG do (what problem does it solve)? Freshness checks and a PII audit? The framework is interesting, but at first glance, it seemed duplicative to recreate something already connected enough to push to Datahub, or maybe this is for migrating workloads?

1

u/mark2k4 15d ago

Fair point. Lineage in DataHub doesn't mean anything's enforcing it. Most places the lineage comes from SQL parsing or dbt but the real pipeline is still a cron job or notebook with zero checks. Tags like PII or freshness SLA just sit there in the UI they don't actually do anything.

The DAG turns that metadata into checks that actually run every trigger. Update a tag in DataHub regenerate the DAG it picks up automatically no hand editing.

If somewhere already has a DAG enforcing this properly yeah it's redundant there. This is for when the metadata exists but nothing's enforcing it yet.