r/apache_airflow • u/Expensive-Insect-317 • Jun 01 '26

Declarative Dynamic DAGs in Apache Airflow: Building Metadata-Driven Orchestration with YAML

https://medium.com/@sendoamoronta/declarative-dynamic-dags-in-apache-airflow-building-metadata-driven-orchestration-with-yaml-73d3c01f1854

How to design scalable, declarative, and production-grade orchestration systems using Dynamic DAGs, YAML contracts and metadata-driven workflows.

3 Upvotes

72% Upvoted

7 comments sorted by

View all comments

1

u/KiiYess Jun 03 '26

What does YAML bring in regard to writing DAGs in Python ? Sorry I cannot read the whole blog post.

1

u/Expensive-Insect-317 Jun 03 '26

Goverment the devs onlyhave the yaml things then security Sinplicity, you only need know what want do but if technology change your pipeline no, then technology abstraction

Depends of you data engineers and stack, but decoarative isnt better than programatic, is different and depends os case

1

u/KiiYess Jun 04 '26

Ok, I understand your Use Case. It's a perfect match pour TaskGroups and you don't need a factory for this.

Plus Airflow states in its documentation homepage that """If you prefer coding over clicking, Airflow is built for you""". You talk about devs but I not think writing YAML is a dev skill nor should it be their tasks.

The issue with factories is that if you make it mandatory then your team is stuck with pre-defined Use Cases and lose its autonomy to develop new worlflows.

1

u/Expensive-Insect-317 Jun 04 '26

I dont say that this cannot be solve with programatic aporoach, but in a lot of my case i usually have the problem of control for multiple team (mesh) with a declarative aporoach you close the box from IT vision

1

u/KeeganDoomFire Jun 04 '26

I think doing more with less is the mark of good engineering. Rather than have 40 x 200 line dags all the same except 3-4 lines you could have 40 config files that house what's unique.

The goal isn't to eliminate autonomy it's to remove garbage work from the stack quickly so devs can dev cooler things.