Hold normal load for hours to find leaks

How to do it

Run the load test first, so you know the service holds its peak. Then hold the average daily load, not the peak, for hours. Some leaks show only with time. Memory, open connections, file handles, disk and queues grow until the service falls over. Watch the server's metrics during the run. Compare the last 30 minutes with the first 30 minutes after warm-up. The response time budget and the memory ceiling come from the load test rule. Run it against staging, never production. Use the result to set headroom.

Examples

JavaScript example

// k6 run soak.js
import http from 'k6/http';
import { check } from 'k6';

export const options = {
  stages: [
    { duration: '10m', target: 50 },
    { duration: '4h', target: 50 },
    { duration: '5m', target: 0 },
  ],
  thresholds: { http_req_failed: ['rate<0.01'] },
};

export default function () {
  const res = http.get('https://staging.example.com/products');
  check(res, { 'status is 200': r => r.status === 200 });
}

Python example

# locust -f soak.py --headless --users 50 --spawn-rate 5 --run-time 4h --host https://staging.example.com
from locust import HttpUser, between, task

WARM_UP_SAMPLES = 10
WINDOW_SAMPLES = 30
MAX_GROWTH = 1.10

class Shopper(HttpUser):
    wait_time = between(1, 5)

    @task
    def browse_products(self):
        self.client.get("/products")

def grew_too_much(samples_per_minute: list[float]) -> bool:
    early = samples_per_minute[WARM_UP_SAMPLES:WARM_UP_SAMPLES + WINDOW_SAMPLES]
    late = samples_per_minute[-WINDOW_SAMPLES:]
    return sum(late) / len(late) > MAX_GROWTH * sum(early) / len(early)

How to check

The soak run lasts four hours or more at the average load. The 95th percentile response time in its last 30 minutes is within 10 percent of the first 30 minutes after warm-up. Memory, open connections and disk use grow less than 10 percent over the same span. The script exits with an error otherwise.

Sources

• k6 soak testing

Related rules

• Test load against a budget in Testing
• Keep headroom in Operations
• Use Locust for load tests in Software choices