A consumer that retries in place blocks its partition. Every message behind the failing one waits, and a single poisoned record can stall a whole stream. The usual fix is to move the failure off the hot path.
The shape of the solution
Failed messages go to a dedicated retry topic with a retry-after header. A scheduler
re-drives them once that timestamp passes. Messages that exhaust their attempts land in a
dead-letter queue for inspection.
The main consumer never waits, and ordering is preserved for everything that succeeds.
Producing to the retry topic
public record RetryEnvelope(String payload, int attempt, Instant retryAfter) {
static final int MAX_ATTEMPTS = 5;
boolean exhausted() {
return attempt >= MAX_ATTEMPTS;
}
Duration backoff() {
return Duration.ofSeconds((long) Math.pow(2, attempt));
}
}Exponential backoff keeps a struggling downstream from being hammered while it recovers.
What to watch
| Signal | Why it matters |
|---|---|
| Retry topic lag | A growing backlog means the downstream is not recovering |
| DLQ arrival rate | A spike usually means a bad deploy, not a transient failure |
| Attempt distribution | Everything failing at attempt 1 suggests a systemic problem |
A retry mechanism you cannot observe is a place for messages to disappear quietly. Wire up the metrics before you need them.