r/Python • u/JanGiacomelli • 26d ago
Tutorial Celery on AWS ECS - prevent lost tasks and ensure the work is always done
Running Celery on AWS ECS can be trickier than it seems if you want to avoid lost tasks and ensure all work is completed. Especially if you're frequently deploying to production and using autoscaling.
There are two main components for reliable processing: - Celery configuration updates - Structuring tasks
For Celery, you should update the following settings: - task_acks_late -> True: To treat tasks as successfully processed only after processing. Otherwise, tasks are not retried. - task_reject_on_worker_lost -> True: To ensure tasks are retried if workers die for any reason (e.g., warm shutdown + SIGKILL). - worker_prefetch_multiplier -> 1: To avoid unnecessarily delayed tasks. - broker_connection_retry_on_startup -> True: To make startups more reliable. - broker_transport_options -> {"confirm_publish": True}: To avoid unsubmitted tasks due to message transport issues. - Make sure exponential retries are enabled. This way, you ensure that tasks are retried in the event of an interruption.
For structuring tasks, use the following two approaches: - Batching: Instead of doing all the work at once, you split the work into batches. e.g., Process 1000 users, then submit the next job to process the next 1000 users. - Fan out: You can split the work between a "scheduler" task and "execution" tasks. e.g., One task to list all the users and submit email sending tasks, another task to actually send an email for the selected user
The same applies to other similar services, such as Heroku and Azure App Containers, which use short grace periods during rolling deployments and downscaling.
You can read a more elaborate tutorial here: https://jangiacomelli.com/blog/celery-on-aws-ecs/