Background jobs and queues

Required

• Make every job safe to run twice

  SCL-17

  Write each job so that running it twice has the same effect as running it once. Queues deliver at least once, and a worker can crash after the work but before it marks the job done.

• Bound the backlog and push back on producers

  SCL-19

  Set a maximum backlog for every queue, as a count or an age. When it is reached, slow or refuse new work instead of letting the backlog grow without end.

Recommended

• Move slow work out of the request

  SCL-16

  Run work the visitor does not need in the answer as a background job. Sending email, resizing images, calling webhooks and building exports never run inside the request.

• Give every job a time limit

  SCL-18

  Set a maximum run time on each job type. Keep the queue's lock or visibility time longer than that limit, so no second worker starts a job that still runs.

• Keep urgent jobs in their own queue

  SCL-20

  Put jobs a person waits for, such as sign-in codes and password resets, in a separate queue with its own workers. Bulk work, such as exports and newsletters, never delays them.

• Put record IDs in jobs instead of copies

  SCL-21

  Store the record's ID and the few inputs the job needs in the job. The worker reads the current record when it runs.

• Spread scheduled work across the hour

  SCL-22

  Give each scheduled job its own start minute instead of the full hour. Split jobs that touch every customer into batches with gaps between them.

• Set aside messages that keep failing

  SCL-34

  After a set number of failed attempts, move a message or job to a separate dead letter queue and alert. Never retry it forever or drop it.

• Run each scheduled job once at a time

  SCL-38

  Take a lock before each scheduled job starts, so a second copy skips its run. Jobs that charge, send or delete must also be safe to run twice.