16 · Idempotency and retries

Send once, even when everything retries.

Your job runner retries, the network drops a response, a queue replays a message. None of that should notify a user twice.

Deterministic Idempotency-Keys

POST /notifications requires an Idempotency-Key. The first request with a key is stored; any later request with the same key and the same body returns that notification (202) without sending again, however much later it comes. The same key with a different body is 409 IDEMPOTENCY_KEY_REUSE, never a silent drop. This holds even if our cache is down: the key is unique in the database.

Build the key from your event, such as invoice-paid:inv_123, not from a random value. Keys are 1 to 256 of A-Z a-z 0-9 _ - :; a dot is refused with 400, so write a type key like invoice.paid with hyphens inside the key. The SDK generates a random key per call if you pass none: that protects its own retries, but not a retry of your whole job.

on-invoice-paid.ts

// Derive the key from the event that caused the send, never from a random
// value: then a retry from a crashed job, a second worker or a replayed
// queue message is the same request.
async function onInvoicePaid(invoice: { id: string; customerId: string; total: string }) {
  await notify.notifications.send(
    { type: "invoice.paid", recipients: [invoice.customerId], data: { amount: invoice.total } },
    { idempotencyKey: `invoice-paid:${invoice.id}` },
  );
}
await onInvoicePaid({ id: "inv_123", customerId: "user_123", total: "12.00" });
await onInvoicePaid({ id: "inv_123", customerId: "user_123", total: "12.00" }); // same notification, nothing sent twice

Your retries

retries.ts

import { Notify } from "@productcraft/notify";

// Network errors, 408, 429 and 5xx are retried with backoff (2 retries by
// default), honouring Retry-After, with the same Idempotency-Key.
const notify = new Notify({ maxRetries: 4, timeout: 10_000 });
await notify.notifications.send(
  { type: "invoice.paid", recipients: ["user_123"] },
  { idempotencyKey: "invoice-paid:inv_124", signal: AbortSignal.timeout(30_000) },
);

Our retries

A provider error, timeout or 429 is retried after 15 s, 1 min, 5 min, 30 min and 2 h (with jitter), until the type’s TTL. Retry-After from Apple or Google pauses that credential, not just one message.
If a provider keeps failing, sends through that credential pause briefly (circuit_open) and resume on their own.
Dead tokens, payload errors and rejected credentials are never retried.

Exactly once where it can be

Inbox items and webhooks are exactly once per recipient. A push can, rarely, reach a provider twice (if we never heard its answer); each carries the notification id as its collapse id (APNs apns-collapse-id, FCM collapse_key, web push Topic), so the device shows it once. Emails carry the delivery id as Mail’s idempotency key.