Agent Wranglers

Required

Roll out settings and data files like code

Validate every change to settings, rules and generated data files before it ships, and roll it out in stages like code. A service that cannot load a new file keeps running on the last good one.

By Agent Wranglers. Updated .

Risk if broken

The app down or unusable

How to do it

Many outages start with a settings or data file pushed everywhere at once. Check each file against a schema and its limits, such as size and number of entries, in CI and again when the service loads it. Send it through the same stages as a release, as Release to a small share first describes. Keep the last file that loaded. When a new one fails, keep using the old one and alert a person. Keep every version in version control, so a rollback is one step. Feature flags follow Separate deploying from releasing.

Examples

Using these examples: they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

JavaScript example

import { readFileSync } from 'node:fs';

const MAX_RULES = 200;
let currentRules = loadRules('/etc/shop/rules.json');

function loadRules(file) {
  const rules = JSON.parse(readFileSync(file, 'utf8'));
  if (!Array.isArray(rules) || rules.length > MAX_RULES) throw new Error(`Invalid rules file ${file}`);
  return rules;
}

export function reloadRules(file) {
  try {
    currentRules = loadRules(file);
  } catch (error) {
    alertOnCall(`Kept the last good rules. ${error.message}`);
  }
}

Python example

import json

MAX_RULES = 200

def load_rules(path):
    with open(path) as file:
        rules = json.load(file)
    if not isinstance(rules, list) or len(rules) > MAX_RULES:
        raise ValueError(f"Invalid rules file {path}")
    return rules

current_rules = load_rules("/etc/shop/rules.json")

def reload_rules(path):
    global current_rules
    try:
        current_rules = load_rules(path)
    except (OSError, ValueError) as error:
        alert_on_call(f"Kept the last good rules. {error}")

How to check

Push a broken settings file to staging, such as one with twice the allowed entries. The service keeps serving with the old file, logs the rejection and sends an alert. The release history shows every settings change in production passed the same stages as code.

Sources

For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/engineering/build-test-and-deploy/staged-config-rollouts">Agent Wranglers engineering playbook</a>. In code you write, add a comment next to it with the same link.