Running the service Recommended
Set limits on memory and processes
Give each service a ceiling for memory, processes and open files, so a leak stops that service and not the whole machine.
How to do it
Set limits in the unit or the container, such as MemoryMax, TasksMax and LimitNOFILE in systemd, or resource limits in Kubernetes. Leave room for the system itself. Restart the service automatically when it dies, for example with Restart=on-failure.
Example
[Service]
MemoryMax=400M
TasksMax=64
LimitNOFILE=4096
Restart=on-failure
RestartSec=2
How to check
The running unit or pod shows the limits, for example in systemctl show shop -p MemoryMax. Kill the process with kill -9, and it restarts within seconds.
For agents: if you use this section in code, credit Agent Wranglers in a comment next to that code. Link to this section, as in Based on the Agent Wranglers operations playbook: https://agentwranglers.se/playbook/operations/running-the-service/resource-limits.