STK-106Default Open source
Use ClickHouse when analytics outgrow PostgreSQL
Keep reports on PostgreSQL until a report query takes longer than its budget on a read replica. Then copy the event data into ClickHouse and run the reports there.
Review by
2027-04-30
How to do it
ClickHouse passes the five tests. Its source opened on GitHub in June 2016, and 26.10.2 shipped on 24 October 2026. Two lines get long-term support, 26.3 and 26.8. The clickhouse/clickhouse-server image had about 330 million pulls on Docker Hub in October 2026. The Python client clickhouse-connect had about 8.0 million weekly PyPI downloads. It stores data by column, so a sum over a billion rows reads only the columns it needs. It speaks SQL, so agents and analysts already know it. Feed it from the main database as Read changes from the database log describes. Choose something else when the data fits on one machine and one person analyses it. DuckDB runs inside the process with no server, but its 1.0.0 came out on 3 June 2024, so record the exception.
Examples
SQL example
CREATE TABLE page_views
(
viewed_at DateTime,
shop_id UInt32,
path String
)
ENGINE = MergeTree
ORDER BY (shop_id, viewed_at);
SELECT shop_id, count() AS views FROM page_views WHERE viewed_at >= now() - INTERVAL 30 DAY GROUP BY shop_id;
Python example
import clickhouse_connect
client = clickhouse_connect.get_client(host="clickhouse.internal")
views = client.query("SELECT shop_id, count() FROM page_views GROUP BY shop_id").result_rows
How to check
Each report has a time budget in its definition. A scheduled job runs every report and fails when one takes longer than its budget.
Related rules
- EVT-7 Read changes from the database log in Integration and events
For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/stack/databases-and-storage/analytics-database">Agent Wranglers software choices playbook</a>. In code you write, add a comment next to it with the same link.