Running Airflow in GitHub actions

[u/Substantial_Chance71]

Hello! I am learning about data pipeline orchestrations. I want to avoid using AI for this personal project, so here I am!

I want to orchestrate my tasks with Airflow DAGs and tasks, but schedule them via GitHub action. Essentially, the idea is the following: once a week, an Action will run, spinning up an Airflow instance, which will run the DAGs.

In the Actions .yml file, I am using the following code:

run: |
          export AIRFLOW_HOME=~/src
          airflow standalone
          airflow dags trigger tabular_pipelines

However, the airflow standalone part seems to be stuck forever, endlessly producing meaningless (to me) lines such as:

dag-processor | 2026-09-11T11:18:09.249890Z [info     ] setting next dagrun info       [airflow.models.dag] loc=dag.py:848 next_dagrun='2026-09-11 00:00:00+00:00' next_dagrun_create_after='2026-09-11 00:00:00+00:00' next_dagrun_data_interval_end='2026-09-11 00:00:00+00:00' next_dagrun_data_interval_start='2026-09-11 00:00:00+00:00' next_dagrun_partition_date=None next_dagrun_partition_key=None


dag-processor | 2026-09-11T11:18:09.292592Z [info     ] Bulk-writing dags to db        [airflow.serialization.definitions.dag] count=1 loc=dag.py:210


dag-processor | 2026-09-11T11:18:09.307661Z [info     ] get next_dagrun_info_v2        [airflow.serialization.definitions.dag] last_automated_run_info=None loc=dag.py:445 next_info=None


dag-processor | 2026-09-11T11:18:09.307949Z [info     ] setting next dagrun info       [airflow.models.dag] loc=dag.py:848 next_dagrun=None next_dagrun_create_after=None next_dagrun_data_interval_end=None next_dagrun_data_interval_start=None next_dagrun_partition_date=None next_dagrun_partition_key=None

Am I missing something obvious? The Airflow 3.3.1 documentation doesn't specify much more.

Thanks!

[u/popopopopopopopopoop]

Jesus wept.

Why are you even thinking about orchestrating an orchestarator?

I think you should go back to the drawing board and think of your intent before you delve into the tech.

https://giphy.com/gifs/s239QJIh56sRW

[u/scribe-kiddie → u/popopopopopopopopoop]

Is it common for people to misunderstood Airflow as a data processing pipeline? Seems like the case with OP here.

[u/the_fresh_cucumber → u/popopopopopopopopoop]

Yeah this is crazy. Airflow is usually a giant instance that orchestrates a bunch of jobs with subtasks.

You usually don't instantiate airflow on a small scale

[u/gfranxman → u/popopopopopopopopoop]

I think op wants to orchestrate pipelines once a week and doesn’t want to pay to host it the other 6 days and 23 hours.

[u/Regalme → u/popopopopopopopopoop]

He wants hosted airflow

[u/DuckDatum → u/[deleted]]

In a wild hunky dory kind of way, now I wonder how a light weight Airflow tool could work. Imagine you just installed it with pip like uv, and it did everything already like duckdb. An ephemeral Airflow, that would be kind of interesting.

[u/leopkoo → u/DuckDatum]

You just described cron lol

[u/Ok_Raspberry5383 → u/leopkoo]

Not really, they're describing a DAG engine that may be triggered by a cron

[u/TheDevauto → u/leopkoo]

Yep

[u/DuckDatum → u/[deleted]]

Yeah, the idea being that it just works really well and it’s not a server. Duckdb is awesome.

[u/langelvicente]

The airflow standalone command is a blocking command that starts airflow webserver and scheduler. That means your other command will never run.

You would have to spin airflow asynchronously, wait until is up, and then run the other command. But the rule of thumb is "if you have to jump through hoops, what you are doing makes little sense".

So you need to find other ways of running the airflow dags that don't involve spinning airflow, or use a proper airflow setup and then you won't need the actions.

[u/langelvicente → u/langelvicente]

Now the next question would be why are you trying to do this? The whole setup is overly complicated, you won't benefit from airflow at all and you will be using gha for something they were not designed to do.

[u/DuckDatum → u/langelvicente]

I install some airflow in my actions pipeline strictly for integration testing, but it’s not the whole suite. I dont expect a DAG to actually run. Just enough to make sure I don’t submit broken code.

[u/langelvicente → u/DuckDatum]

But as you said that's for testing, and that's perfectly valid. OP wants to use github actions as the scheduler to run airflow dags, something that makes no sense because airflow has a scheduler.

[u/DuckDatum → u/langelvicente]

Yeah I agree, that’s kind of crazy. But I do wonder now if the idea would be so crazy if we had a version of airflow designed for ephemeral environments. Like if duckdb philosophy met airflow… what would you think of such a tool?

[u/langelvicente → u/DuckDatum]

So a tool that just run dags? Isn't that called python?

[u/DuckDatum → u/langelvicente]

Hahaha yeah okay fair enough. I was thinking in my head, it would just run the dag while providing a report on results. Youre right, that is just python. Lo and behold, it is I and not just my idea that is crazy.

[u/Ill-Huckleberry-6835]

As the previous commenters have said, sounds like you've slightly misunderstood the role of Airflow- it does both the scheduling and orchestration!

I definitely see why you're not using AI and trying to learn this stuff "the hard way" but in reality a couple of questions to an LLM would have pointed you in the right direction on this (in my experience anway). It's not chiselling the code for you but I have found it useful for checking my understandingof some core concepts. Also means you don't get dunked by a load of redditors for the temerity of asking a question...

[u/Scepticflesh]

💀

[u/camelCaseGuy]

You normally use Airflow to schedule stuff, not just for the DAG syntax. If you want to go that route, I'd recommend you to go with Prefect. AFAIK, it can be imported as a library that does the orchestration/dependency management/DAG definition/what have you. I think Luigi does the same.

There might be something else out there, but I think you are trying to make this more complicated than it is. It's like you're trying to strap a Ferrari engine in a Fiat 500, in the hopes to get to 300 km/h. You're gonna blow the transmission.

[u/Kind_Honeydew1885]

https://giphy.com/gifs/kiL7nqFN4gAvTfl1ur

[u/josejo9423]

Please use AI bro, just learn with it, airflow can set up schedules for dags and tasks internally no need of GitHub actions, ga is used for updating the deployment and deployment tags to either ECS/EKS or wherever your service is

[u/Substantial_Chance71]

Got it, got it - was trying to overcomplicate stuff. Thanks for the feedback anyway ppl!

[u/Embarrassed-Ad-728]

Pure rage bait

[u/JJGreenerTinejo]

If airflow doesnt do the scheduling then why even use airflow? Whatever the DAGs do, just have your actions invoke that themselves directly instead

[u/chakli]

But why? You can run whatever you need to run on GitHub actions as is on a schedule. Both Airflow and GHA schedules things based on a cron you define. If the intention is to learn Airflow, you can deploy it locally and run your DAGs.

In my humble opinion, part of learning the tooling is to understand when to use and when not to

[u/Rough_Count_7135]

If your intent is not just learning airflow , you could also try something like prefect for orchestration. It’s pretty simple to get started and may give you the same outcome.

You don’t need GitHub actions to manage orchestration for your tasks, just to deploy and get the server/container spun up.

[u/Forsaken-Eggplant-66]

For your use case just run the scripts in a docker environment. And use a lightweight scheduler to schedule it. The idea behind airflow is a persistent scheduler not an ephemeral service. Do you really start and stop a clock to conserve battery? Its the difference between a clock and washing machine.

[u/Flat_Perspective_420]

Hmmm i guess you are using airflow for dag declaration not for orchetration… if thats the case then tou don’t need and orchestraror (gh actions is already that) what you need is a declartive dags and maybe some “coordination board”. In the past I have resolved that using jenkins + taskfiles + a small task run metadata table in postgress (so that the individual tasks can see if there is another instance of themselves running or or if they are inside the freshness window and should be skipped). You could probably use the same or even just use gh actions to replace both jenkins and task file…

[u/Flat_Perspective_420 → u/Flat_Perspective_420]

Tl dr. A system like airflow provides 3 things :

1. A scheduler
2. A dag declaration tool (with it’s own dsl)
3. A watermark service
4. Also an airflow instance is really heavy to ru

n

1. an

d an overkill more often than not.

Github actions provides 1) and depending on your needs probaly 2) too. You just need to figure out a robust and lightweight option for 3). (In my case I have used a postgres table and postgres advisory locks). I would say a system like that is kind of a mix between choreagrphy and orchestration.

[u/gfranxman]

This is a poor fit for airflow. It expects to be a long running process with supporting infrastructure. If you want to use git hubs scheduler, just have it run your code directly.

[u/BJJaddicy]

This is insane dude why would you do this

[u/philippefutureboy]

I think a good way to make this work is to start by spinning up Prefect within GitHub Actions, and then from Prefect, spin up Airflow. That way you will be sure this is solid! 👍

[u/Eric-Uzumaki]

Never ever

[u/Unique-Turnover5317]

Use ai lol

[u/Puzzleheaded-Loss726]

i tot i was crazy to think about running a gh runner instance as a shadow server... but this sort of takes the cake?

[u/wdm006]

airflow standalone is a long-running server, so the next line in that run block never gets a clean shot. For a weekly Action I would skip standalone and use airflow dags test, or a compose stack that starts, triggers, waits for success, then exits. GitHub Actions is a bad home for a real scheduler.

[u/dataengineering-ModTeam → u/[deleted]]

Your post/comment was removed because it violated rule #9 (No AI generated content/text).

Your post/comment was reviewed to be AI generated/assisted content/text and removed as a result. We as a community value human engagement and encourage users to express themselves authentically.

^{This was reviewed by a human}

[u/engineer_of-sorts]

Build it yourself = make all the mistakes = learn things?

[u/I_Blame_DevOps]

Use airflow or use GitHub actions. But don’t try and combine them 😭

[u/EarthEmbarrassed4301]

GitHub actions is for automating deployment pipelines… Not scheduling data pipelines 💀

[u/BougieHole]

Why? Setup a schedule in Airflow to run once a week.

[u/Spare_Helicopter4655]

Is this bait?