Agent Wranglers

Recommended

Send logs off the machine as they are written

Ship every log line to a store on another machine within 60 seconds of writing it. The server's own credentials must not be able to edit or delete it there.

How to do it

An attacker who gets into a server deletes its local logs first, and a lost disk takes them too. Run a shipper such as Fluent Bit, Vector or the OpenTelemetry Collector, or let the platform collect standard output. Send the lines over TLS to a store that the server may only add to. Let only named people read the store, and record each time they do. Audit events need a stronger store, as Make the audit log impossible to change says. The store keeps lines for the period in Keep logs for a fixed time.

Examples

Using these examples: they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

YAML example

# OpenTelemetry Collector on the server: read the log files and send them out over TLS
receivers:
  filelog:
    include: [/var/log/shop/*.log]
exporters:
  otlphttp:
    endpoint: https://logs.example.com
service:
  pipelines:
    logs:
      receivers: [filelog]
      exporters: [otlphttp]

INI example

# systemd-journal-upload: send the journal to a collector on another machine
[Upload]
URL=https://logs.example.com:19532
ServerKeyFile=/etc/ssl/private/journal-upload.pem
ServerCertificateFile=/etc/ssl/certs/journal-upload.pem
TrustedCertificateFile=/etc/ssl/ca/logs-ca.pem

How to check

Write a test line on the server. It appears in the central store within 60 seconds. A delete request for that line, sent with the server's own credentials, is refused.

Sources

For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/operations/logs/off-host-logs">Agent Wranglers operations playbook</a>. In code you write, add a comment next to it with the same link.