Author(s): Ray Hu

Originally published on Towards AI.

Three retries, one alert, and the request is gone. A DLQ took us to 0.1%.

Here’s a moment every backend engineer who has shipped an LLM feature will recognize.

The article explains a common LLM failure mode: retry loops exist for transient errors, but once retries are exhausted the request can vanish without a dead letter queue (DLQ), leaving no one to handle the failure. It contrasts how LLM failures differ from traditional service failures and argues that a robust design needs multiple exit paths—async compensation, graceful degradation for synchronous user-facing flows, auto-repair/redrive for structural issues, and a human review queue for high-value or judgment-requiring failures. It then provides practical implementation guidance (including Redis Streams), highlights production “traps” such as DLQs with no consumers, tangled routing logic, treating permanent/transient failures the same, missing context in DLQ messages, and lack of idempotency in compensation, along with suggested DLQ monitoring metrics. Finally, it shares the production outcome of adding a DLQ (dropping request loss from 2.3% to 0.1%) and emphasizes the core takeaway: retries are not enough—requests must not disappear when the retry budget runs out.

Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI