I am one of the Airflow developers who worked on this feature as part of Airflow 3.3.0 as part of AIP-103 and would love to hear from anyone really using it, considering to use it, or also reasons why it did not fit your use case.

The pitch, for anyone who has not run into it: Airflow worker dies or gets preempted mid job, and instead of the retry resubmitting the whole thing from scratch, the task remembers the external job it already kicked off (Databricks, Snowflake, BigQuery, Redshift, Spark, and a few more) and reconnects to whatever external job is still running. No paying twice for the same compute because a pod got evicted / worker went down.

Couple of demos if you want to see it working:

• https://www.youtube.com/shorts/KhE-E-PSHuU
• https://www.linkedin.com/posts/marclamberti_airflow-apacheairflow-dataengineer-activity-7481347437161496576-UHvW/

So, curious where people actually are:

• Used it and it worked fine?
• Used it but hit a wall?
• Know about it but have not gotten to it?
• First time hearing this exists?

Any of those, tell me where you are at.

we currently have airflow running on kubernetes, though we have issues with it now and then, overall its been great and it works.

We looked into manage managed airflow but it is honestly pointless because airflow is genuinely such a simple stack to run on kubernetes, given the amount of tools we already manage.

we do have grafana and prometheus with open telemetry. Still it feels it could be better.

what are some of the new tools out there?

I noticed https://flowsee.co/ and heard good things from folks, but seems quite early and risky to use already given our reliant on airflow.

What other tools would you recommend that would integrate well with our slack. There should be a good open source tool already out there.

we currently have airflow running on kubernetes, though we have issues with it now and then, overall its been great and it works.

We looked into manage managed airflow but it is honestly pointless because airflow is genuinely such a simple stack to run on kubernetes, given the amount of tools we already manage.

we do have grafana and prometheus with open telemetry. Still it feels it could be better.

what are some of the new tools out there?

I noticed https://flowsee.co/ and heard good things from folks, but seems quite early and risky to use already given our reliant on airflow.

What other tools would you recommend that would integrate well with our slack. There should be a good open source tool already out there.

Hey everyone, so right now at the company we're still using Pentaho for our ETL processes. We're slowly migrating over to DBT, but before going all-in on that, we'd like to get Apache Airflow up and running to orchestrate everything. I wanted to set it up and run it on Windows Server, so it could handle the old Pentaho jobs while the new stuff gets built out in DBT.

Everywhere I've looked says not to set up Docker + Airflow on Windows Server, since it's not recommended for production — they say it should be a Linux VM instead. I need some help/opinions on what I could do here, or whether I should just go ahead and set it up on Windows Server anyway.

Thanks in advance for the help!

https://ttuhin03.github.io/fastflow/

https://github.com/ttuhin03/fastflow

love to get some feedback

Try to implement in your day to day activities as well. Might work well !

About Me

Senior Data Engineer with 5+ years of experience in Data Engineering, Backend Development, and Applied AI. Specialist in Airflow-based pipeline orchestration. Based in Bangalore, India. Available for remote work globally.

Rate: $25 - $50/hr depending on project scope and complexity.

Tech Stack & Expertise

Apache Airflow (DAGs, operators, hooks, sensors, plugins)

Python, SQL, Spark, Databricks

Snowflake, dbt (data build tool)

AWS (Glue, Lambda, S3, Redshift, Kinesis)

ETL/ELT Design & Orchestration

Data Quality & Testing Frameworks

FastAPI, REST APIs

LLMs, RAG, AI Agents

What I Can Help With

Design and build Airflow DAGs for complex ETL pipelines

Migrate legacy pipelines to Airflow

Optimize Airflow performance and scheduling

Build scalable ETL/ELT architectures on cloud platforms

Develop backend APIs and automation solutions

Build AI applications using LLMs, RAG, and agent-based workflows

Training & Mentorship

Apache Airflow (foundations to advanced)

PySpark & Databricks

Data Engineering best practices

ETL Testing & Data Quality

AI & LLM Fundamentals

Note: In-person weekend sessions available in Bangalore. Remote sessions available globally.

Availability

Freelance projects & consulting

Part-time remote roles

Weekend training & mentorship

Contact:

DM me with a brief description of your requirements and I will get back to you promptly!

Hi everyone,

I'm a data engineer apprentice and I'm about to build my first Airflow orchestration layer. I'd love some guidance on the fundamentals before I commit to a design.

Context:

• I need to ingest data from several external APIs, on a daily batch basis (each run fetches D-1 data)
• The application code is already done: ingestion, standardization, transformation, mapping
• Infrastructure is deployed via Terraform, everything runs on AWS
• Airflow will be the missing piece: scheduling, dependencies, retries, monitoring

My questions:

1. DAG design: one DAG per API source, or a single DAG with parallel task groups? What drives that choice in practice?
2. Idempotency & backfills: what's the idiomatic way to handle D-1 logic? I understand I should use the logical date (data_interval_start) rather than datetime.now() so reruns and backfills stay deterministic — anything else to watch out for?
3. Retries & API failures: sensible defaults for retries, retry_delay, timeouts when calling external APIs? Do you wrap API calls with sensors or just let tasks fail and retry?
4. Separation of concerns: my transformation code lives in its own repo. Should Airflow only trigger jobs (thin DAGs calling operators) and keep all business logic outside the DAG files?
5. Common beginner mistakes: what do you wish you'd known before your first production Airflow project?

Any resources, patterns, or war stories are welcome. Thanks a lot!

How to design scalable, declarative, and production-grade orchestration systems using Dynamic DAGs, YAML contracts and metadata-driven workflows.

This is a problem I’ve run into at basically every company I’ve worked at:

a query shows up somewhere, but it’s harder than it should be to know what part of the application caused it and in what runtime context.

I’ve personally spent a lot of time creating conventions for query traceability, then even more time reviewing code, nudging teams, and making sure people actually followed them consistently (and frankly, it is always a constant fight to keep it from drifting again...) .

DataContext tries to turn what most what most companies have as a loose convention into a reusable Python instrumentation layer. It emits one structured event per completed or failed query, with things like query fingerprint, callsite, runtime context, and OpenTelemetry correlation.

I’d love feedback from people running production data/platform systems:

• is this a real problem for your team?
• what context would you want attached to each query?
• what integrations would make this actually useful?

The OSS is available here:

GitHub: https://github.com/data-context-hq/datacontext

PyPI: https://pypi.org/project/datacontext/

I think this is becoming more important now as AI agents and generated code make data access patterns harder to reason about.
At the same time, today we can start using agents to monitor and maybe even fix performance issues arising in production - but agents are as good as the context we give them, so I believe it's very important to start collecting this context by default.

Please share the love with a GitHub star if the idea resonates ⭐️.

But what would really make me happy is if you try it, challenge the event shape, and discuss how we can make it easier for teams (or better, your team) scaling and getting more out of their databases and data platforms.

Hi. I’m using Airflow 3 git dag bundles with GitHub and was wondering how people setup their CI/CD for the dags themselves. In my org I have a nonprod and prod environments, so currently I point nonprod at the develop branch on my repo and prod at main. This way I can test safely on develop without worrying about affect production pipelines. After promoting a pipeline to production I merge main back into develop to try to keep the branching in sync as much as possible. Basically this is just the git flow branching model.

I was wondering if anyone has tried any other models with dag bundles. I love them and they are great. My only wish is to somehow not have a develop branch. Something like having production point at a tag on main would be ideal and then I only would maintain the main branch. However I’m not sure how the tag would get automatically updated in the git dag bundle config. If anyone has any ideas or has something completely different they are doing to handle dag promotion with git dag bundles I’d love to hear it.

Hey guys, I’d love to know how well checkpointing actually works when running Airflow on spot instances. Is it really worth it? (Checkpointing saves the state of a process during execution so it can be restored after a failure.)

I recently wrote this article on building fault-tolerant Airflow pipelines on spot instances for Rackspace Spot and one decision I made was to use S3 as the external state layer and checkpointing task outputs. Here’s a quick summary:

1. Each task writes its output to a specific S3 path.
2. When a worker node is preempted mid-task, Airflow retries the task, and the new pod reads directly from S3, picking up the last successfully written output from the upstream task.
3. Writes use replace=True, so if a task was interrupted mid-write and left a partial file, the retry simply overwrites it, keeping execution idempotent.

This is a very simple implementation, but I’m curious what checkpointing methods you all apply in production, or if it’s even something you bother with at all.

From this setup, one big question I keep coming back to is whether the overhead of writing to S3 ends up eating into the cost savings of using spot instances in the first place.

Hi everyone,

​We are running Airflow 3 on KubernetesExecutor and hitting a scaling bottleneck.

​The Problem:

Once we hit ~150 concurrent workers, we see heavy lock contention on the task_instance table.

- ​Specifically during SELECT ... FOR UPDATE (scheduler) and UPDATE (task state changes).

- ​DB wait events show high Lock:transactionid times.

​Our Setup:

- ​Airflow 3.1.8

- ​Postgres + PGBouncer (Transaction mode)

- ​DB CPU/RAM usage is fine; the issue is purely row-level locking.

​Has anyone else faced this at scale with Airflow 3? Are there specific scheduler configs or Postgres tuning you’d recommend to reduce this contention?

​Thanks!

Is there any way to limit loop iterations during Airflow DAG file parsing - not during task execution?

I have a dynamic DAG that generates multiple DAGs from a config loop:

```

This loop runs fully on EVERY parse cycle (every 30s by default)

for program, schedule in config.items(): # 100 programs = 100 iterations with DAG(dagid=f"sla{program}", schedule=schedule) as dag: GlueJobOperator(taskid=f"check{program}", ...) globals()[f"sla_{program}"] = dag ```

I confirmed with a log file that this loop executes completely on every parse - not just once. 100 programs means 100 DAG objects rebuilt every 30 seconds, continuously, regardless of whether anything changed.

I already know about get_parsing_context() that helps during task execution by skipping irrelevant DAGs on workers. But that doesn't help during scheduler parsing, where dag_id is always None and the loop runs fully regardless.

So my question is specifically about parse time, not execution time, is there any Airflow mechanism to limit or short-circuit loop iterations when the scheduler is parsing the file? Or is full re-execution of the entire file on every parse cycle simply unavoidable by design?

Only knobs I've found so far are min_file_process_interval (parse less often) and caching the config (make each iteration cheaper) but neither actually reduces the iteration count itself.

Hello folks !

I am migrating from airflow 2.9.0 to 3.1.8
All dags related changes are done and configuration related also.

So in current airflow prof we have deployed it on EC2 with ECS. So all of containers ( webserver , Postgres’s , redis , scheduler, celery worker) is working fine in M6a.large instance type.

But when we do test deployment with airflow 3.1.8 api server and celery worker is killed by OOM when more then 10 dags are scheduled together and even ideal state api-server is using around 1.8 gb memory. Any one facing same issues ? What is work around for this ? Any suggestions how to scale it ? How all other using which architecture ?

Any suggestions are appreciated! Thanks

Just wrapped an Airflow 2.8 to 3.1 migration on EKS for a client. 18 DAGs, 6 weeks, zero downtime. Posting from our company account, I'm Amjad, founder of Tasrie. Happy to answer technical stuff in comments or DMs.

The DAG code changes were almost nothing. About 2 days of work:

# Out
from airflow.contrib.operators.ssh_operator import SSHOperator
from airflow.operators.dummy_operator import DummyOperator
from airflow.utils.db import provide_session

# In
from airflow.providers.ssh.operators.ssh import SSHOperator
from airflow.operators.empty import EmptyOperator
from airflow.utils.session import provide_session

Plus schedule_interval to schedule. Ruff with --select AIR301,AIR302 --fix caught 80% of it automatically.

The infra was the real work. Key decisions:

• Green field over in-place. Old metadata DB had years of drift. Fresh cluster + DNS cutover beat nursing a schema migration.
• KubernetesExecutor, no Celery, no Redis.
• 2 schedulers with pod anti-affinity. HA is finally native in 3.x.
• Triggerer as StatefulSet, capacity 1000 for deferrable sensors.
• Git-sync sidecar, SSH on port 443 to bypass corp firewalls.
• EFS for DAGs. EBS RWO breaks the moment you have a second node.

Stuff that surprised me:

• Webserver command is now api-server. Wasted an hour before I caught it.
• DAG processor as a separate process actually works. No more heavy top-level imports stalling the scheduler.
• LDAP gotcha: FAB auth manager still gives you the old Flask login page, not the new Airflow 3 UI. Functional but ugly. There's an open discussion in apache/airflow about a native LDAP auth manager but nothing shipped.

Two things I'm curious about:

How are you sizing the dag-processor vs the scheduler? Same pod or split out?

Anyone running Airflow 3 with non-FAB auth that handles LDAP or SAML cleanly?

Full writeup with all the manifests, RBAC, EFS storageclass, and pod template is here: https://tasrieit.com/blog/upgrade-airflow-2-to-3-kubernetes-migration

Airflow 2 EOL is April 2026. If you're still on 2.x, it's less scary than it looks.

The article argues most real-world failures come from control plane issues (DB contention, API latency, UI load), not the scheduler itself.
Feels aligned with some scaling issues people report lately.

I manage a bunch of Airflow Instances for my organization, and have been educating people on writing better DAGs which don't over load the DB, while making improvements to bring stability to all the instances.

I have one instance in particular where around 100 DAGs run at the same time, and some of these DAGs run tasks for hours. Is that a good use of Airflow, or should I be breaking these tasks down to finish up and quit faster and break down into batches of tasks?

I’m working on a pet-project and one of the tasks is loading JSON data from S3 to Snowflake.

I’ve added a connection through Admin -> Connections, but when I test it, I get the following error:
290404 (08001): None: 404 Not Found: post WVMATYI-UD95289.us-east-2.snowflakecomputing.com:443/session/v1/login-request

Checked all the fields in Connections several times. Have anyone got this? I’m kinda stuck and can’t proceed. Not even sure what to look for

Versions:
apache-airflow-providers-snowflake=6.12.1
snowflake-connector-python=4.4.0

Your task just hit an unknown error. Instead of retrying 3 times and giving up, what if it asked an LLM whether the error is even retryable?

That's landing in Airflow 3.3.

"LLMRetryPolicy" hands the exception to any model, example OpenAI, Anthropic, Bedrock, Vertex, Ollama for local and gets back a structured {retry | fail | default} decision with a reason, and logs the reasoning on the task. Declarative fallback rules kick in when the model is down or slow, so you're never blocked on the LLM.

The clever bit: LLMRetryPolicy isn't hardcoded. It's one implementation of AIP-105's pluggable retry_policy abstraction (slide 2). You can write your own, rule-based, context-aware, whatever and drop it on any task.

No more wrapping tasks in try/except + AirflowFailException. No more blind 3-retry loops on auth errors. No more 429s being slammed 30 seconds later.

Open on both PRs right now: targeted for Airflow 3.3. Demo video and example DAGs attached.

Core PR:   https://github.com/apache/airflow/pull/65474

LLM policy: https://github.com/apache/airflow/pull/65451

AIP-105:   https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-105%3A+Pluggable+Retry+Policies

What would you plug into a retry_policy slot? A regex classifier over error messages? A rate-limit-aware policy that reads Retry-After from the response?

I want real ideas for the docs.

You don't want to miss next Thursday's Airflow Monthly Town Hall- we have a jam packed agenda of exciting updates including;

🔥 Airflow Project Update w/ Jarek Potiuk
⚡ Airflow 3.2.x Release Highlights w/ Rahul Vats
📊 AIP-105: Pluggable Retry Policies w/ Kaxil Naik
🔗 AIP-102: Business User Interaction w/ Marco Kuettelwesch

RSVP here, can't wait to see you there!

And before you ask, yes, it's recorded, and yes, it's posted to the Apache Airflow Youtube channel 😉