Exponential Backoff That Survives App Restarts

A diary edit fails to sync because the train dipped into a tunnel. The app schedules a retry in fifteen seconds — then the user switches apps, iOS kills Storyie, and fifteen seconds later nothing happens. The mutation is still on disk. The intent to retry was not: it lived in a setTimeout that died with the process.

That is the gap this post is about. We already persist every offline mutation in a local queue (Designing an Offline Mutation Queue on SQLite). Durability of the payload is not enough. The backoff schedule — how many times we have tried, and the earliest moment we are allowed to try again — has to survive a force-quit the same way the draft does. Storyie does that by treating retryCount and nextRetryAt as data on the row, then rebuilding "what is due?" from those columns on every cold start.

TL;DR

• In-memory backoff is a restart bug. A setTimeout (or any process-local timer) evaporates when the OS kills the app; the mutation remains, the schedule does not.
• Persist the schedule, not the timer. Each queue item stores retryCount and nextRetryAt next to the payload so a cold start can resume the same exponential backoff.
• Ready work is a query. After launch, getPendingSyncQueueRows / getRetryableEntityItems / getNextEntityItem select items whose deadline has passed — no timer reconstruction.
• Terminal failure clears the deadline. Past MAX_RETRIES = 5, nextRetryAt becomes null so the item stops cycling and waits for a human.
• Time tests assert windows, not sleeps. Specs pin Date.now, compare past/future ISO stamps, and check the first delay lands near now + 5s.

Memory retry is a schedule that forgets itself

The instinctive backoff looks like this: catch the network error, setTimeout(() => flush(item), delay), bump a counter in a module-level Map. It works until the process disappears. On mobile that is common — memory pressure, background eviction, a deliberate force-quit after typing on the subway. When Storyie comes back:

• the local draft is still there (local-first autosave wrote it synchronously — see Local-First Autosave Without Losing the Last Keystroke);
• the pending mutation row is still there;
• every in-flight timer is gone, and so is any "attempt 3 of 5, wait until 12:04:15" knowledge that lived only in RAM.

Worse, a naive restart that retries immediately on launch turns every cold start into a thundering herd against a still-broken API. You need the opposite of "forget and hammer": remember the deadline, and only fire when it has passed.

So the design rule is simple: if the OS can kill the process between attempts, the backoff deadline must be a durable field, not a timer handle.

retryCount and nextRetryAt are the schedule

Storyie keeps a fixed delay table and a hard ceiling in one place:

// apps/expo/constants/storageKeys.ts (excerpt)
/** Retry delays in milliseconds (exponential backoff) */
export const RETRY_DELAYS = [5000, 15000, 45000, 135000, 405000] as const;

/** Maximum number of sync retries */
export const MAX_RETRIES = 5;

Those five steps (~5s → ~6.75 minutes cumulative) are deliberate product numbers: short enough that a brief tunnel recovers quickly, long enough that a multi-hour outage does not spin the radio forever. The table is data, not a recursive delay * 2 in a closure — so the next process can recompute the same next attempt from retryCount alone if it ever needs to.

When the entity draft queue fails a flush, markEntityFailed writes the schedule into storage before returning:

// apps/expo/services/syncQueueService.ts (excerpt)
export function markEntityFailed(
  entityType: DraftEntityType,
  entityId: string,
  error: string
): void {
  // ...
  const newRetryCount = item.retryCount + 1;
  const isPermanentlyFailed = newRetryCount >= MAX_RETRIES;

  const backoffIndex = Math.min(newRetryCount - 1, RETRY_DELAYS.length - 1);
  const backoffMs = RETRY_DELAYS[backoffIndex];
  const nextRetryAt = isPermanentlyFailed ? null : new Date(Date.now() + backoffMs).toISOString();

  queue[key] = {
    ...item,
    status: "failed",
    retryCount: newRetryCount,
    errorMessage: error,
    nextRetryAt,
    timestamp: new Date().toISOString(),
  };
  setEntitySyncQueue(queue);
}

The SQLite mutation-queue path does the same idea at the row layer: bump retryCount, compute nextRetryAt via calculateNextRetryTime, and — for that queue — flip status back to pending so the drain has one shape to look for:

// apps/expo/services/syncQueueService.ts (excerpt)
export function calculateNextRetryTime(retryCount: number): string {
  const clampedRetry = Math.min(Math.max(retryCount - 1, 0), RETRY_DELAYS.length - 1);
  const delayMs = RETRY_DELAYS[clampedRetry];
  const nextTime = new Date(Date.now() + delayMs);
  return nextTime.toISOString();
}

Either way, after a kill the interesting state is still on disk: which attempt we are on, and the earliest wall-clock time we may try again. No timer to restore.

Ready work is whatever the deadline says is due

On the next launch, useSyncProcessor does not ask "which timeouts did I schedule?" It asks "is there anything ready?" and drains while online:

// apps/expo/hooks/useSyncProcessor.ts (excerpt)
useEffect(() => {
  if (!autoProcess) return;

  if (isOnline && hasEntityItemsToProcess()) {
    logger.debug("[useSyncProcessor] Online with pending items, processing...");
    processQueue();
  }
}, [isOnline, autoProcess, processQueue]);

hasEntityItemsToProcess / getNextEntityItem prefer fresh pending items, then failed items whose backoff window has already elapsed:

// apps/expo/services/syncQueueService.ts (excerpt)
export function getRetryableEntityItems(): EntitySyncQueueItem[] {
  const queue = getEntitySyncQueue();
  const now = Date.now();

  const items = Object.values(queue).filter((item) => {
    if (item.status !== "failed" || !item.nextRetryAt) {
      return false;
    }
    const retryTime = new Date(item.nextRetryAt).getTime();
    return retryTime <= now;
  });

  items.sort(
    (a, b) => new Date(a.nextRetryAt ?? 0).getTime() - new Date(b.nextRetryAt ?? 0).getTime()
  );

  return items;
}

The SQLite drain encodes the same predicate in SQL so a future nextRetryAt never leaves the database as "ready":

// apps/expo/lib/db/queries/sync-queue.ts (excerpt)
// status = 'pending' AND (nextRetryAt IS NULL OR nextRetryAt <= now)
export async function getPendingSyncQueueRows(): Promise<LocalSyncQueueRow[]> {
  const db = await getLocalDb();
  const now = new Date();
  return db
    .select()
    .from(localSyncQueue)
    .where(
      and(
        eq(localSyncQueue.status, "pending"),
        or(isNull(localSyncQueue.nextRetryAt), lte(localSyncQueue.nextRetryAt, now))
      )
    )
    .orderBy(asc(localSyncQueue.createdAt));
}

A helper on the item shape makes the rule testable without standing up the whole drain:

// apps/expo/services/syncQueueService.ts (excerpt)
export function isReadyForRetry(item: SyncQueueItem): boolean {
  if (item.status !== "pending") return false;
  if (!item.nextRetryAt) return true;
  return new Date(item.nextRetryAt) <= new Date();
}

The processor also polls every 30 seconds while online. That interval is not the backoff — it is only how often we re-evaluate deadlines that may have matured while the app stayed alive. After a restart, the mount effect covers the same job immediately: read durable deadlines, enqueue what is due, leave the rest alone until their nextRetryAt passes.

Terminal failure: stop scheduling, keep the evidence

Unbounded retries are a battery and API tax. When newRetryCount >= MAX_RETRIES, Storyie refuses to invent another deadline:

// apps/expo/services/syncQueueService.ts (excerpt)
// inside markEntityFailed — permanently failed path
const nextRetryAt = isPermanentlyFailed ? null : new Date(Date.now() + backoffMs).toISOString();

nextRetryAt: null on a failed row is load-bearing. getRetryableEntityItems requires a non-null deadline, so permanently failed items never reappear as "ready" after a restart — which is exactly what you want when the last five attempts already burned the budget. A separate helper surfaces them for UI or cleanup:

// apps/expo/services/syncQueueService.ts (excerpt)
export function getPermanentlyFailedEntityItems(): EntitySyncQueueItem[] {
  const queue = getEntitySyncQueue();
  return Object.values(queue).filter(
    (item) => item.status === "failed" && item.retryCount >= MAX_RETRIES && !item.nextRetryAt
  );
}

Manual retry is a different product verb. Resetting retryCount to 0 and clearing nextRetryAt treats the tap as fresh user intent with a full budget — not as "continue attempt 6 of 5." Automatic backoff protects the server; the Retry button serves the person staring at a stuck diary.

Time tests pin the contract without sleeping

Backoff that depends on wall clocks is easy to test badly (real sleep(5000) flakes) or not at all (mock away Date so thoroughly the schedule never meets reality). Storyie's service tests do three cheap things instead.

First, assert the first failure lands in a tight window around the table's first entry:

// apps/expo/services/__tests__/syncQueueService.test.ts (excerpt)
it("should set nextRetryAt with exponential backoff", () => {
  mockQueue = {
    "diary:diary-1": createQueueItem({ entityId: "diary-1", retryCount: 0 }),
  };

  const now = Date.now();
  markEntityFailed("diary", "diary-1", "Error");

  const nextRetryAt = new Date(mockQueue["diary:diary-1"].nextRetryAt as string).getTime();
  // First retry: 5 seconds delay
  expect(nextRetryAt).toBeGreaterThanOrEqual(now + 4900);
  expect(nextRetryAt).toBeLessThanOrEqual(now + 5100);
});

Second, spy Date.now when checking pure schedule math, and use past/future ISO strings when checking readiness filters — no fake timers required for "due vs not due":

// apps/expo/services/__tests__/syncQueueService.test.ts (excerpt)
it("calculates backoff delay based on retry count", () => {
  const now = Date.now();
  jest.spyOn(Date, "now").mockReturnValue(now);

  const next = calculateNextRetryTime(1);

  expect(new Date(next).getTime()).toBe(now + 5000);
  jest.restoreAllMocks();
});

it("should not return items with nextRetryAt in the future", () => {
  const futureTime = new Date(Date.now() + 60000).toISOString();
  // ... mark failed with futureTime ...
  expect(getRetryableEntityItems().length).toBe(0);
});

Third, treat "gave up" as a null-deadline assertion, not as waiting out the delay table:

// apps/expo/services/__tests__/syncQueueService.test.ts (excerpt)
it("should not set nextRetryAt when max retries exceeded", () => {
  mockQueue = {
    "diary:diary-1": createQueueItem({ entityId: "diary-1", retryCount: 4 }),
  };

  markEntityFailed("diary", "diary-1", "Error");

  expect(mockQueue["diary:diary-1"].retryCount).toBe(5);
  expect(mockQueue["diary:diary-1"].nextRetryAt).toBeNull();
});

Those three cases — delay stamped correctly, future deadlines excluded, exhausted retries unscheduled — are the restart contract. If they hold, a cold start that re-runs the same queries will behave like the timers never died.

Takeaways

• Persist the backoff deadline. retryCount + nextRetryAt on the queue item beat any in-process timer on a phone that can be killed mid-wait.
• Select ready work from storage. Mount and periodic drains should ask "whose deadline has passed?", not "which timeout ids did I keep?"
• Null means stop. Clearing nextRetryAt at MAX_RETRIES is how permanent failure survives restarts without silently retrying forever.
• Test clocks as data. Window assertions and past/future stamps verify the schedule without flaky real delays.

The queue that holds these fields is covered in Designing an Offline Mutation Queue on SQLite. The write path that feeds it without dropping the last keystroke is Local-First Autosave Without Losing the Last Keystroke.