Measuring it yourself

The method

1. Copy the repository once, so edits made during the run can't leak into it.
2. For each experiment, copy it again. Apply one small edit script that removes a feature, or adds one back.
3. Build the copy and start the server inside a container. Run Lighthouse 3 times on two pages, phone and desktop, and keep the median. One run on one page is enough for SEO and accessibility scores.
4. Lighthouse can't see clicks, prefetch, the service worker or repeat visits. For those, run a Playwright script in 7 fresh browsers per variant. Add 100 ms of round trip and a 20 Mbit/s limit.
5. Compare each variant with the baseline built the same way.

Lab image

cat > Dockerfile <<'EOF'
FROM mcr.microsoft.com/playwright:v1.57.0-noble
RUN apt-get -qq update && apt-get -qq install -y iproute2 curl >/dev/null
WORKDIR /lab
RUN npm init -y >/dev/null && npm i --silent playwright@1.57.0 lighthouse@12.8.2 \
 && ln -s /ms-playwright/chromium-*/chrome-linux*/chrome /usr/local/bin/chrome
ENV CHROME_PATH=/usr/local/bin/chrome
EOF
docker build -t web-lab .

Give the container --shm-size=1g, or Chrome's tab crashes at desktop size. To add delay, also give it --cap-add NET_ADMIN and run tc qdisc add dev lo root netem delay 50ms rate 20mbit. That is 50 ms each way on the loopback device, so a 100 ms round trip.

Lighthouse runner

import lighthouse from 'lighthouse';
import * as chromeLauncher from 'chrome-launcher';
import desktop from 'lighthouse/core/config/desktop-config.js';
const [,, base, runs = '3', ...pages] = process.argv;
const med = a => [...a].sort((x, y) => x - y)[Math.floor(a.length / 2)];
for (const form of ['mobile', 'desktop']) for (const p of pages) {
  const rs = [];
  for (let i = 0; i < +runs; i++) {
    const chrome = await chromeLauncher.launch({ chromeFlags: ['--headless=new', '--no-sandbox', '--disable-dev-shm-usage'] });
    const { lhr } = await lighthouse(base + p, { port: chrome.port, output: 'json' }, form === 'desktop' ? desktop : undefined);
    await chrome.kill();
    rs.push({ perf: lhr.categories.performance.score * 100, lcp: lhr.audits['largest-contentful-paint'].numericValue, cls: lhr.audits['cumulative-layout-shift'].numericValue });
  }
  console.log(form, p, Object.fromEntries(Object.keys(rs[0]).map(k => [k, med(rs.map(r => r[k]))])));
}

Click and repeat-visit test

In each fresh browser, load the start page. Record HTML arrival, FCP and LCP from connectStart. Wait 2.5 s, so prefetch and the service worker can warm up. Click a link and record performance.timeOrigin + FCP − click time. Then open the start page again in a new tab of the same browser. Record its FCP as the repeat visit.

Useful tricks

• Prefetch did not fire? In the Chrome DevTools Protocol, call Preload.enable and log Preload.prefetchStatusUpdated events.
• Does a resource hold up paint? Abort it with page.route(...) and compare.
• Test the TCP start window on its own. Run ip route change $(ip route show default | sed "s/ initcwnd.*//") initcwnd 10 initrwnd 10, then restore it.
• Check production from outside. Use curl -w '%{http_version}' for the HTTP version. curl --tlsv1.1 --tls-max 1.1 --ciphers DEFAULT@SECLEVEL=0 -v must fail with the server's protocol version alert. Without the cipher option, OpenSSL 3 refuses TLS 1.1 itself, so the test proves nothing. Look up AAAA, CAA, DS and _agents records with DNS over HTTPS by querying https://cloudflare-dns.com/dns-query?name=…&type=CAA with accept: application/dns-json.