Test load against a budget

How to do it

Write a short load script with a tool such as k6, Gatling or Locust. Run it against staging, never production without warning. Set limits for the slowest responses and the error rate, such as a 95th percentile under 500 ms and under 1 percent errors. Then raise the load in steps until errors or response times pass the limits, and record that point as the service's capacity. Page speed budgets live in the website playbook, and security tests in the security testing playbook. Measure the server's memory after a warm-up over every kind of page. Set a ceiling below the machine's limit, such as 80 percent of it.

Examples

JavaScript example

// k6
import http from 'k6/http';
export const options = {
  vus: 50,
  duration: '2m',
  thresholds: { http_req_duration: ['p(95)<500'], http_req_failed: ['rate<0.01'] },
};
export default () => { http.get('https://staging.example.com/products'); };

Python example

# Locust
from locust import HttpUser, between, events, task

class Shopper(HttpUser):
    wait_time = between(1, 3)

    @task
    def products(self):
        self.client.get("/products")

@events.quitting.add_listener
def fail_over_budget(environment, **kwargs):
    stats = environment.stats.total
    if stats.fail_ratio > 0.01 or stats.get_response_time_percentile(0.95) > 500:
        environment.process_exit_code = 1

How to check

The load test exits with an error when a limit is passed. The latest run before the launch passed. The run also fails when memory passes its ceiling.

Sources

• Azure Well-Architected performance testing
• k6 thresholds

Related rules

• WEB-58 Fail the build when pages get heavier in Website
• SEC-7 Turn every fix into a check in Adversarial testing
• TST-53 Run load tests from a written runbook in Testing
• TST-51 Hold normal load for hours to find leaks in Testing
• TST-61 Catch speed regressions on every pull request in Testing
• OPS-38 Walk through a launch checklist before users arrive in Operations and observability
• OPS-39 Keep headroom in Operations and observability
• OPS-41 Size the runtime's memory to the box in Operations and observability
• INF-62 Test the service against every platform limit in Infrastructure and environments
• STK-97 Use Locust for load tests in Software choices
• SCL-1 Find the bottleneck before you add servers in Scaling
• SCL-3 Write down the load you plan for in Scaling
• SCL-40 Spread writes that all land on one row in Scaling