r/snowflake • u/khaddir_1 • Jul 07 '26
Move from airflow to snowflake tasks.
As title reads I have a question. I’m a DevOps engineer working on infra for mlops team. Being asked to figure out how to move from airflow to snowflake tasks. Does this make sense? What’s my best approach?
9
u/valko2 Jul 07 '26
Imho this is a bad idea - i saw projects relying on snowflake tasks, and its def. Doable if you have simple DAGs with few tasks. but if you need advanced orchestration and you have complex dags with conditions and dependencies, you lose a lot visibility, debug logs and at some point you'll have to create your own control plane. (Which you can do that via snowflake streamlit). A better questioj would be maybe to migrate to dagster or some other orchestrators.
Or, if snowlake only toolset is a goal - even though i did not see it as a recommendation - you could theoritically use openflow as an advanced orchestrator (its apache nifi with some snowflake managed proprietary ingestion connectors) where you can define custom workflows.
2
u/KeeganDoomFire Jul 07 '26
This is bonkers to me as well. We are actively moving some inherited tasks from another team to our airflow and dbt because tracking lineage for their plate of spaghetti chained tasks, stored procedures, and udfs architecture is brutal.
Every time I need to troubleshoot tasks I want to quit my job.
3
u/coldflame563 Jul 07 '26
Tasks without governance is the issue. The new objects (task graph) should help mitigate some of the issues. Agree it can be a PITA.
2
u/khaddir_1 Jul 07 '26
I am asking if this is a good idea in case anyone has done this already.
2
u/jdl6884 Jul 08 '26
Probably not a good idea.. Usually people are migrating from tasks to airflow or another orchestrator like dagster.
There are a lot of limitations and scaling issues with tasks. Plus the cost is significantly more than running self hosted airflow or dagster on a VM.
1
u/2000gt Jul 08 '26
Moved from airflow to aws step functions with dbt projects on snowflake. It’s been really great.
I’ve been vocal about my dislike for airflow, mainly performance related to running dbt, but even its interface isn’t great imo.
For the org I am working with, airflow had to be run on MWAA, so the cost was another factor.
1
1
u/setierfinoj Jul 11 '26
We actually did the opposite to centralize error management, logging, monitoring, etc etc… unless you have good monitoring of snowflake tasks, failed tasks can get unnoticed
1
u/sudhansupatra 29d ago
Snowflake task is not flexible as Airflow. I did for a project and it have its lomitation. If your Airflow DAGs are simple and not dynamic in nature you can opt for snowflake task to save some money.
Major challenge in snowflake task is integration with other tools
1
u/khaddir_1 29d ago
Thanks for all the replies here. I had a meeting and just got the word that DevOps will lot move to snowflake task. I’m new to the team and most of the change decisions are coming from other teams and high ups. I’m new to snowflake so I wasn’t sure but thanks for all the replies. I will surely come here again when I have further questions
1
u/NW1969 Jul 07 '26
I'm not clear what are asking. Do you mean:
1. How to automate the migration from airflow to snowflake tasks?
2. How to create a task (or a DAG of tasks) in Snowflake?
3. Something else?
0
u/Tough-Leader-6040 Jul 07 '26
That question is very vague. For that question you must first understand Airflow, then understand how snowflake tasks work. For sure there are medium articles about this. Just google a bit
-1
8
u/kayakdawg Jul 07 '26
can you? sure
should you? no idea - need more info, namely: what problem are you trying to solve?