Size workers and pools from rate times duration

How to do it

This is Little's law. At 200 requests per second that each take 50 ms, 10 requests are in flight on average. Peaks and slow requests need more, so use the peak rate and the 99th percentile time. A pool far larger than this only moves the queue into the database. Take both numbers from rate, errors and duration. Keep the total of every pool under the database limit, as Fit the connection budget to the autoscaling maximum says.

Examples

JavaScript example

const HEADROOM = 1.5;

export function poolSize({ peakPerSecond, p99Seconds }) {
  return Math.ceil(peakPerSecond * p99Seconds * HEADROOM);
}

poolSize({ peakPerSecond: 200, p99Seconds: 0.12 }); // 36

Python example

import math

HEADROOM = 1.5

def pool_size(peak_per_second: float, p99_seconds: float) -> int:
    return math.ceil(peak_per_second * p99_seconds * HEADROOM)

pool_size(200, 0.12)  # 36

How to check

Each pool's size is within a factor of 2 of the peak rate times the 99th percentile time over the last 30 days. Under the load test, the 95th percentile wait for a pool connection stays under 10 ms.

Related rules

• Measure rate, errors and duration in Operations
• Fit the connection budget to the autoscaling maximum in Scaling