Build large exports ahead of time

How to do it

A file built and compressed on each request costs the same work every time, and a large one can hold a worker for seconds. Write each file and its .gz and .br copies during the build, at the highest compression level, since it runs once. Let the web server send the right copy without calling the app, for example with gzip_static in nginx or precompressed in Caddy. Rebuild the files when the content changes, or on a schedule as a background job. Files that differ per customer go to object storage from a job, with a link sent when ready, as Run long jobs asynchronously says. Pages follow Render at build time and serve from memory. Prove one request cannot stall the site finds the files that need this.

Examples

JavaScript example

import { writeFile } from 'node:fs/promises';
import { brotliCompressSync, constants, gzipSync } from 'node:zlib';

export async function writePrecompressed(path, text) {
  const bytes = Buffer.from(text);
  await writeFile(path, bytes);
  await writeFile(`${path}.gz`, gzipSync(bytes, { level: 9 }));
  await writeFile(`${path}.br`, brotliCompressSync(bytes, { params: { [constants.BROTLI_PARAM_QUALITY]: 11 } }));
}

await writePrecompressed('dist/feed.xml', buildFeed(articles));

Python example

import gzip
from pathlib import Path

import brotli


def write_precompressed(path: Path, text: str) -> None:
    data = text.encode()
    path.write_bytes(data)
    Path(f"{path}.gz").write_bytes(gzip.compress(data, compresslevel=9))
    Path(f"{path}.br").write_bytes(brotli.compress(data, quality=11))


write_precompressed(Path("dist/feed.xml"), build_feed(articles))

How to check

The build output holds each large file with .gz and .br copies. Requests for those files never appear in the app's access log and carry no app time in Server-Timing. Their first byte leaves the server within 20 ms.

Sources

• nginx gzip_static module

Related rules

• TST-54 Prove one request cannot stall the site in Testing
• API-37 Run long jobs asynchronously in API design
• WEB-18 Render at build time and serve from memory in Website