Reconnect with backoff and resume from the last event

How to do it

When a server restarts, every client loses its connection at the same moment. If all reconnect at once, they knock the new server over. Delay the first try by a random time between 0 and 5 seconds. Grow later gaps as retry with backoff and jitter says. Give every event an ID in order, and keep recent events on the server. On reconnect the client sends its last ID and gets only newer events. Event streams do this with Last-Event-ID. When the ID is older than the kept events, send a reset event so the client reloads the full state.

Examples

JavaScript example

const FIRST_DELAY_MAX_MS = 5_000;
const MAX_DELAY_MS = 60_000;
let lastEventId = null;

function connect(attempt = 0) {
  const socket = new WebSocket(`wss://api.example.com/v1/live?after=${lastEventId ?? ''}`);
  socket.onopen = () => { attempt = 0; };
  socket.onmessage = message => { const event = JSON.parse(message.data); lastEventId = event.id; apply(event); };
  socket.onclose = () => {
    const ceiling = Math.min(MAX_DELAY_MS, FIRST_DELAY_MAX_MS * 2 ** attempt);
    setTimeout(() => connect(attempt + 1), Math.random() * ceiling);
  };
}

Python example

import asyncio, json, random
import websockets

FIRST_DELAY_MAX = 5
MAX_DELAY = 60

async def follow(last_event_id=""):
    attempt = 0
    while True:
        try:
            async with websockets.connect(f"wss://api.example.com/v1/live?after={last_event_id}") as socket:
                attempt = 0
                async for raw in socket:
                    event = json.loads(raw)
                    last_event_id = event["id"]
                    apply(event)
        except (websockets.ConnectionClosed, OSError):
            pass
        await asyncio.sleep(random.uniform(0, min(MAX_DELAY, FIRST_DELAY_MAX * 2 ** attempt)))
        attempt += 1

How to check

Restart the server with 1,000 test clients connected. Their reconnects spread over at least 5 seconds. Each client then holds every event exactly once.

Sources

• RFC 6455 section 7.2.3 Recovering from abnormal closure
• WHATWG HTML the Last-Event-ID header

Related rules

• Retry with backoff and jitter in API design
• Deliver events through a shared channel in API design