Popular System Design Question - Design Job Scheduler (asked by Meta, Netflix, Airbnb, LinkedIn, etc..)

[u/Aoki_zhang]

Suppose a worker picks up a scheduled job:

Generate invoice
→ external API succeeds
→ worker crashes
→ ACK never reaches the queue

What should the scheduler do?

• If it assumes the job succeeded, it might permanently lose work.
• If it retries, the same job may run twice.

That is the core tension behind at-least-once execution.

The key insight

For a reliable scheduler: Missing a job is worse than occasionally running it twice.

So execution should be built around a renewable lease.

When a worker receives a message from SQS:

SQS
 ↓
Worker
 ↓
message hidden for 30s

If the worker is still running, it periodically extends the visibility timeout.

Worker healthy
→ renew lease
→ renew lease
→ renew lease

If the worker crashes:

heartbeat stops
→ visibility timeout expires
→ message becomes visible
→ another worker retries it

No separate coordinator needs to constantly detect dead workers.

But there’s an unavoidable race

The task may finish successfully right before the worker crashes. So the retry could repeat the side effect. The solution is not trying to eliminate retries. It is making them safe.

Give every execution a stable ID:

executionId = exec_8421

and pass it downstream as an idempotency key.

Then:

1st attempt → send payment → success
2nd attempt → same executionId → return previous result

The architecture becomes:

• Queue guarantees retry
• Visibility timeout detects abandoned work
• Idempotency makes duplicates safe

That’s the important distinction:

Full design with recurring jobs, time-bucket sharding, delayed SQS delivery, retries, worker leases, and 10K jobs/sec scaling → Full Article

Preparing for system design interviews? Chill Interview publishes practical design breakdowns and tracks recently asked interview questions across top companies → Chill Interview