Introduction
Not every operation belongs inside an HTTP request. Sending emails, generating reports, processing uploaded files, and synchronizing external systems can take too long to complete while a user waits.
Background workers let an application accept a task quickly and process it separately. But reliable background processing requires more than moving a function into another process: tasks need clear ownership, retry behavior, monitoring, and failure handling.
1. Know when to use a background task
A useful rule is to keep the request focused on work that must finish before the response can be returned.
For example, an API might save an order and schedule a confirmation email. The order should be persisted before the API reports success, while email delivery can usually happen asynchronously.
Do not move essential work to a background task without considering consistency. If a payment must be confirmed before an order is accepted, that operation cannot simply be deferred without changing the business rules.
2. Separate task creation from execution
A common architecture contains three components:
- The API or application service creates a task.
- A queue stores pending work.
- A worker consumes tasks and executes them.
Popular Python options include Celery, RQ, and Dramatiq. They differ in their supported brokers, scheduling capabilities, configuration, and operational model.
A simplified Celery task might look like this:
from celery import Celeryapp = Celery( "tasks", broker="redis://localhost:6379/0",)@app.taskdef send_order_confirmation(order_id: int) -> None: # Load the order and send the confirmation. print(f"Processing order {order_id}")The example illustrates task registration, not a complete production email implementation. The worker must be running, and the broker must be configured and available.
Pass stable identifiers such as order_id rather than serializing an entire database object into the task message.
3. Design for retries and duplicate execution
A worker can crash after performing an operation but before acknowledging the message. The task may then run again.
This means many queue-based systems provide at-least-once delivery semantics rather than a guarantee that each task executes exactly once.
Make tasks idempotent wherever possible. An idempotent task can safely be repeated without creating an unwanted duplicate effect.
For example, before sending an invoice, record a durable invoice status or use a unique operation identifier. For external services, use their idempotency mechanisms when available.
Retry transient failures, such as temporary network errors, with bounded exponential backoff. Do not endlessly retry invalid input or permanent authorization failures.
4. Handle the database and queue boundary
A subtle failure occurs when the database transaction succeeds but the task message is never published—or when a task is published before the database transaction commits.
For workflows where losing a task is unacceptable, consider the transactional outbox pattern. Store the business change and an outgoing event in the same database transaction. A separate publisher reads the outbox and delivers events to the queue.
This adds complexity, so reserve it for workflows whose reliability requirements justify it.
5. Monitor workers as production services
Track queue depth, task age, execution duration, retry counts, and failures. A queue that continues accepting tasks can still be unhealthy if workers are falling behind.
Set sensible timeouts, concurrency limits, and dead-letter or failure-handling procedures. Make task failures visible to operators and provide a safe way to replay tasks after correcting the underlying issue.
Conclusion
Background processing improves responsiveness and separates slow work from request handling, but it introduces distributed-systems concerns. Choose a queue that fits your workload, make tasks repeatable, define retry behavior, and monitor the entire pipeline.
