Python: Structured Logging for Easier Production Debugging

Logs are often the first place developers look when a production problem occurs. Yet logs that consist entirely of unstructured sentences can be difficult to search, filter, and correlate across services.

Structured logging records information as fields, making it easier for logging systems to query events and group related failures.

From Text Messages to Structured Events

A basic logging statement might look like this:

import logginglogging.warning(    "Payment failed for order 123")

A person can understand this message, but a monitoring system may struggle to extract the order ID reliably.

A structured event represents the message and its context separately. For example, a JSON log record could contain:

{
  "event": "payment_failed",
  "order_id": "123",
  "reason": "provider_timeout",
  "level": "WARNING"
}

The fields can then be filtered independently.

Use Python’s Logging Framework

Python’s built-in logging module supports consistent log levels and named loggers:

import logginglogger = logging.getLogger(__name__)def process_payment(order_id: str) -> None:    try:        # Perform payment processing.        pass    except TimeoutError:        logger.warning(            "Payment provider timed out",            extra={                "event": "payment_failed",                "order_id": order_id,                "reason": "provider_timeout",            },        )        raise

The extra dictionary adds attributes to the LogRecord. It does not automatically serialize the entire record as JSON. Configure a formatter or structured logging library that explicitly emits the desired fields.

Libraries such as structlog can make structured event creation more convenient in applications that need richer context handling.

Add Context That Helps Diagnosis

Useful fields may include:

• Request or correlation ID.
• Operation name.
• Dependency name.
• Duration.
• Result status.
• Retry count.
• Tenant or account identifier, where appropriate and safe.

Context should make a failure easier to investigate. Avoid adding every available field to every event, because noisy logs increase storage costs and make important information harder to find.

Use Log Levels Consistently

A practical logging policy distinguishes between severity levels:

• DEBUG: detailed information useful during development or diagnosis.
• INFO: meaningful application events.
• WARNING: unexpected situations that may be recoverable.
• ERROR: operations that failed and require attention.
• CRITICAL: severe failures that may threaten continued operation.

The correct level depends on the application’s operational policy. Do not log every expected validation failure as an error, and do not hide important service failures at debug level.

Protect Sensitive Information

Structured logs can make secrets easier to search—and easier to expose.

Never log passwords, access tokens, private keys, or full authorization headers. Carefully evaluate personal information, payment details, request bodies, and database parameters before recording them.

Apply redaction at appropriate boundaries, restrict access to log systems, and define retention rules.

Correlate Events Across Requests

In a service handling many concurrent requests, a correlation ID helps connect logs that belong to one request or workflow.

Accept externally supplied correlation IDs only according to a safe validation policy, or generate your own identifier. Propagate it to downstream services where supported.

For asynchronous jobs, include a job or operation identifier so that the original request and later processing can be connected.

Conclusion

Structured logging turns application output into queryable operational data. Use consistent event names, useful context fields, appropriate log levels, and strict redaction rules.

When an incident occurs, well-designed logs help developers answer what happened, where it happened, and which request or operation was affected.