Retries and dead letters

Retries and circuit breaking#

Failures use full jitter over an exponential delay, starting with a 1-second ceiling and capped at 60 seconds. SQS visibility deferral uses at least one second. MAX_DELIVERY_ATTEMPTS defaults to 8.

Workers persist the failed state and next attempt time, then call ChangeMessageVisibility on the existing message. An open destination circuit or an exhausted delivery token bucket can defer work without an HTTP attempt. The default circuit opens after 5 failures and allows a probe after a 30-second cooldown.

Browse exhausted deliveries#

On exhaustion, the worker ensures an S3 payload archive exists, persists a PostgreSQL dead-letter record and marks the delivery dead-lettered before deleting its queue message. Dead letters are not a separate SQS queue.

curl --fail-with-body --max-time 15 \
  'http://127.0.0.1:3000/tenants/REPLACE_WITH_TENANT_UUID/dead-letters?limit=20'

curl --fail-with-body --max-time 20 \
  'http://127.0.0.1:3000/tenants/REPLACE_WITH_TENANT_UUID/dead-letters/REPLACE_WITH_DEAD_LETTER_UUID/payload'

Replace both UUID placeholders with existing records. List and payload lookups read the replica. Archived payload inspection is limited to 4 MiB and has a 15-second read deadline. The response uses Cache-Control: no-store.

Request an explicit replay#

curl --fail-with-body --max-time 20 \
  --request POST \
  'http://127.0.0.1:3000/tenants/REPLACE_WITH_TENANT_UUID/dead-letters/REPLACE_WITH_DEAD_LETTER_UUID/replay' \
  --header 'Idempotency-Key: REPLACE_WITH_REQUEST_UUID' \
  --header 'Content-Type: application/json' \
  --data '{}'

Use a fresh UUID for Idempotency-Key, then reuse that same key when retrying the same replay request. The body must be empty or {}; replacement payloads and destinations are rejected.

A 202 response returns replayId, deadLetterId, originalEventId, the new eventId and createdAt. The original delivery remains unchanged. Replay reads primary state, verifies the archive and creates the new event, outbox entry and audit record in one database transaction.

The same key for the same source returns the existing replay. A key reused for another source, a different key for an already replayed source, or a mismatched configured destination returns 409. Each dead-letter record can have at most one replay. Replay is not automatic and does not guarantee delivery.

Failure responses#