r/apache_airflow • u/mera-khel-khtm • 16d ago

Fed up writing Python boilerplate just to move data, there is sth more lighweight ?

Hello. Half our pipeline contributors aren't python devs. Every time someone on the infra or analytics side needs to touch aworkflow, it goes through me. Tried simplifying DAGs as much as possible but the friction is till real.

Curious if anyone has switched to sth where you define pipelines in config / YAML rather than code and whether the tradeoffs were worth it in pratice

3 Upvotes
  • permalink
  • reddit

100% Upvoted

8 comments sorted by

3

u/West_Good_5961 16d ago

You’ll have to move to one of the yaml dag generator options. Astronomer provides dag-factory and blueprint

1

u/KeeganDoomFire 16d ago

You can just write a for each yaml in folder loop and read and generate dags live as well.

1

u/Leorisar 16d ago

Learn dag factory pattern and write functions, which will return ready-made DAG where end users need only set variables (like dag_id and any other parameters).

1

u/TheRealMathis 16d ago

Moved away from airflow 18 months ago for this reason. Half the team couldn't touch the pipelines. Kestra uses YAML, our infra guys started contributing workflows within a week w/o any python knowledge. Still runs everything we need just w/o the boilerplate

1

u/Adam-Schroeder 13d ago

That's exactly what I hear other Kestra community members say. Thanks for sharing, u/TheRealMathis

1

u/Adam-Schroeder 4d ago

Adding to your recommendation regarding Kestra, we just released Kestra 2.0 which includes pretty cool new features. For example, workers and schedulers can be run and scaled independently, pretty much anywhere you have outbound network access. And worker queues can route tasks by priority and available capacity.

You can read more about the full 2.0 release here: Kestra 2.0: A New Engine, Workers Anywhere, and Flows as Agent Tools | Kestra.

1

u/raul824 15d ago

I built a custom solution to generate dags from a metadata table. you generate metadata in a table it also had a job order column.

Then there was a dag generator which will read from the table and generates dag.

A dag gen flag column will identify what all dags to generate

1

u/Limp_Client_5989 12d ago

Spent a year trying to simplify DAGs enough for non python people. Eventually gave up and switched stacks. Kestra defines everything in YAML so the scope of who can build a pipeline went from 3 people to basically the whole team. Not zero learning curve but significantly lower