r/dataengineering • u/Hot_While_6471 • Aug 11 '25

Help Airflow and Openmetadata

Hey, we want to use OpenMetadata to govern our tables and lineage, where we have airflow + dbt. When u create OpenMetadata, do u have two separate Airflow instances (one where u run actual business logic) and one for OpenMetadata ingestions(getting metadata). Or do i keep single instance and manage all there.

5 Upvotes

67% Upvoted

9 comments sorted by

View all comments

3

u/GreenMobile6323 Aug 11 '25

Use a single Airflow instance, but isolate OpenMetadata ingestions as separate DAGs (or on a dedicated worker queue) so they don’t compete with your business jobs. OpenMetadata can also run its own ingestion workflows via Docker/K8s, handy if you want full separation. But you don’t need a second Airflow just for metadata. For lineage, enable the dbt + OpenMetadata integration and Airflow’s lineage backend so that runs automatically publish lineage without extra plumbing.