Add load tests, a browser swarm and a public benchmark site

.ai/PERFORMANCE.md

@@ -95,6 +95,46 @@ production bundle, runs the harness against `baseline.json`, and:
 Thresholds live per-scenario in `scenarios.py` (`warn_ms` / `fail_ms`, keyed by
 metric). Keep them generous: this is a smoke alarm, not a microbenchmark.
 
+## Streaming load test and the public site
+
+`benchmarks/streaming/` measures streaming-callback capacity per server setup
+(see `benchmarks/README.md` for how to run it). Simulated browsers follow the
+renderer's `StreamClient`: one downlink per browser, a long NDJSON response on
+ASGI and polls with the same backoff on WSGI. Frames carry the server's
+`time.time()`, so latency is measured end to end on one clock.
+
+Pitfalls the harness guards against:
+
+- One Python client process tops out at a few hundred browsers and then
+  reports its own lag as server latency. Clients run 100 browsers per process,
+  on CPUs separate from the server, and a point is flagged `client_bound` when
+  a client hits 85% CPU.
+- Client output goes to temp files, not pipes: a full pipe blocks the client
+  forever because the runner only reads after the clients exit.
+- Multi-worker setups need shared storage (Redis); with a per-process secret
+  the workers also reject each other's stream tokens, so the runner sets
+  a shared secret (`LOAD_SECRET`). The server imports dash from the checkout
+  under test (`PYTHONPATH`), not whatever is installed.
+
+`benchmarks/callbacks/` runs the same way for plain callbacks, HTTP vs
+websocket per backend; both load tests share `benchmarks/loadkit.py`.
+
+`benchmarks/swarm/` drives real headless Chromium (Playwright) against a
+deployed app, on as many machines as needed. Pitfalls it guards against:
+
+- Timings are taken in the page (`performance.now()` from the click to the DOM
+  update), so machines never need agreeing clocks; only the start is shared,
+  and an agent that got its browsers up after it is flagged `late`.
+- A failed reload restarts the page's click counts; the expected output text
+  is derived from them, so a stale count would time out every later click.
+- Steps stop when the swarm stops delivering its load (failed agents, users
+  that never loaded, agents over 85% CPU), not only when the server slows.
+- `page.on("websocket")` does not see sockets opened by a SharedWorker; to
+  confirm a click went over the websocket, check that it made no HTTP POST.
+
+`benchmarks/publish.py` turns results into the site on `gh-pages`
+(`.github/workflows/benchmarks-publish.yml`).
+
 ## Profiling a slow scenario
 
 The runner can capture a **Chrome DevTools CPU profile** of a scenario and print

.github/workflows/benchmarks-publish.yml

@@ -0,0 +1,200 @@
+name: Publish Benchmarks
+
+# Measures the benchmarks on the dev branch and publishes them to GitHub
+# Pages (the gh-pages branch): every chart as an embeddable page, the raw JSON
+# and its history. Renderer timings run on every push to dev; the streaming
+# and callback load sweeps take 30-60 minutes each, so they run weekly and on
+# demand.
+
+on:
+  push:
+    branches: [dev]
+    paths:
+      - 'dash/**'
+      - 'benchmarks/**'
+      - 'components/**'
+      - '@plotly/**'
+  schedule:
+    - cron: '0 6 * * 1'
+  workflow_dispatch:
+    inputs:
+      streaming:
+        description: Run the streaming load sweep too
+        type: boolean
+        default: true
+      callbacks:
+        description: Run the HTTP vs websocket callback sweep too
+        type: boolean
+        default: true
+
+permissions:
+  contents: read
+
+concurrency:
+  group: benchmarks-publish
+  cancel-in-progress: false
+
+jobs:
+  renderer:
+    name: Renderer benchmarks
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: '24'
+          cache: npm
+      - run: npm ci
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+          cache: pip
+      - name: Install Dash
+        run: |
+          python -m pip install --upgrade pip
+          python -m pip install "setuptools<80.0.0"
+          python -m pip install -e .[ci,dev,testing]
+      - run: npm run build
+      - uses: browser-actions/setup-chrome@c785b87e244131f27c9f19c1a33e2ead956ab7ce # v1
+        with:
+          chrome-version: stable
+      - name: Set up virtual display
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y xvfb
+          sudo Xvfb :99 -ac -screen 0 1400x1000x24 &
+          echo "DISPLAY=:99" >> $GITHUB_ENV
+      - run: python -m benchmarks.run --out benchmarks/results.json
+      - uses: actions/upload-artifact@v4
+        with:
+          name: renderer-results
+          path: benchmarks/results.json
+
+  streaming:
+    name: Streaming load sweep
+    if: github.event_name == 'schedule' || (github.event_name == 'workflow_dispatch' && inputs.streaming)
+    runs-on: ubuntu-latest
+    timeout-minutes: 120
+    services:
+      redis:
+        image: redis:7-alpine
+        ports:
+          - 6379:6379
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: '24'
+          cache: npm
+      - run: npm ci
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+          cache: pip
+      - name: Install Dash
+        run: |
+          python -m pip install --upgrade pip
+          python -m pip install "setuptools<80.0.0"
+          python -m pip install -e .[ci,dev,testing,async,fastapi,quart,redis]
+          python -m pip install gunicorn uvicorn aiohttp psutil
+      - run: npm run build
+      - name: Run the sweep
+        env:
+          BENCH_RUNNER: GitHub Actions ubuntu-latest
+        run: |
+          ulimit -n 65536
+          python -m benchmarks.streaming.load \
+            --redis-url redis://127.0.0.1:6379/0 \
+            --out benchmarks/streaming/results.json
+      - uses: actions/upload-artifact@v4
+        with:
+          name: streaming-results
+          path: benchmarks/streaming/results.json
+
+  callbacks:
+    name: Callback load sweep
+    if: github.event_name == 'schedule' || (github.event_name == 'workflow_dispatch' && inputs.callbacks)
+    runs-on: ubuntu-latest
+    timeout-minutes: 120
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: '24'
+          cache: npm
+      - run: npm ci
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+          cache: pip
+      - name: Install Dash
+        run: |
+          python -m pip install --upgrade pip
+          python -m pip install "setuptools<80.0.0"
+          python -m pip install -e .[ci,dev,testing,async,fastapi,quart]
+          python -m pip install gunicorn uvicorn aiohttp psutil
+      - run: npm run build
+      - name: Run the sweep
+        env:
+          BENCH_RUNNER: GitHub Actions ubuntu-latest
+        run: |
+          ulimit -n 65536
+          python -m benchmarks.callbacks.load --out benchmarks/callbacks/results.json
+      - uses: actions/upload-artifact@v4
+        with:
+          name: callbacks-results
+          path: benchmarks/callbacks/results.json
+
+  publish:
+    name: Publish to GitHub Pages
+    needs: [renderer, streaming, callbacks]
+    # Only dev's numbers go public, even when a dispatch runs elsewhere.
+    if: >-
+      always() && github.ref == 'refs/heads/dev' &&
+      (needs.renderer.result == 'success' || needs.streaming.result == 'success' ||
+       needs.callbacks.result == 'success')
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+      - name: Check out gh-pages
+        run: |
+          if git ls-remote --exit-code --heads origin gh-pages; then
+            git fetch origin gh-pages
+            git worktree add site origin/gh-pages
+            git -C site switch -c gh-pages
+          else
+            git worktree add --detach site
+            git -C site switch --orphan gh-pages
+          fi
+      - uses: actions/download-artifact@v4
+        with:
+          path: results
+      - name: Build the site
+        env:
+          BENCH_RUNNER: GitHub Actions ubuntu-latest
+        run: |
+          args=""
+          [ -f results/renderer-results/results.json ] && args="$args --renderer results/renderer-results/results.json"
+          [ -f results/streaming-results/results.json ] && args="$args --streaming results/streaming-results/results.json"
+          [ -f results/callbacks-results/results.json ] && args="$args --callbacks results/callbacks-results/results.json"
+          python -m pip install -e .
+          python -m benchmarks.publish --site site --commit "${GITHUB_SHA::9}" \
+            --base-url "https://${{ github.repository_owner }}.github.io/${{ github.event.repository.name }}" $args
+      - name: Push
+        run: |
+          cd site
+          git config user.name "github-actions[bot]"
+          git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
+          git add -A
+          if git diff --cached --quiet; then
+            echo "Nothing changed."
+            exit 0
+          fi
+          git commit -m "Publish benchmarks for ${GITHUB_SHA::9}"
+          git push origin gh-pages

benchmarks/.gitignore

@@ -3,3 +3,6 @@ results.json
 summary.md
 *.cpuprofile
 __pycache__/
+streaming/results.json
+site/
+callbacks/results.json

benchmarks/README.md

@@ -39,3 +39,110 @@ scenario(
 `b` is the browser helper (`b.timed`, `b.render_time`, `b.reload`, `b.state`,
 `b.graph_time`). Every layout must end with the shared `READY` sentinel so the
 harness can detect "fully hydrated". Then regenerate `baseline.json`.
+
+## Streaming load test (`streaming/`)
+
+How many browsers a server setup can stream to. Each setup (Flask on
+gunicorn, FastAPI and Quart on uvicorn, 1 or 4 workers) serves
+`streaming/load_app.py`, and simulated browsers (`streaming/client.py`) stream
+from it while the count climbs until p95 frame latency passes 1 s or errors
+pass 1%.
+
+```bash
+ulimit -n 65536
+python -m benchmarks.streaming.load --redis-url redis://127.0.0.1:6379/0
+python -m benchmarks.streaming.load --setup fastapi-uvicorn-w1 --browsers 100 1000
+```
+
+Multi-worker setups share frames through Redis and are skipped without
+`--redis-url`. The server and the clients get separate CPUs (`--server-cpus`,
+half the machine by default); a point where a client process hit 85% CPU is
+flagged `client_bound` and ends that setup's sweep, since past that the numbers
+measure the load generator.
+
+## Callback load test (`callbacks/`)
+
+Plain callbacks over HTTP vs websocket callbacks, by backend: Flask (HTTP
+only), FastAPI and Quart (both), 1 and 4 workers. Simulated browsers click
+and wait for the response; the count climbs until p95 round trip passes 1 s
+or errors pass 1%. Shares its runner with the streaming test
+(`loadkit.py`).
+
+```bash
+python -m benchmarks.callbacks.load
+python -m benchmarks.callbacks.load --setup fastapi-uvicorn-w1-ws --browsers 100 1000
+python -m benchmarks.callbacks.load --kind async --work-ms 50   # like a query
+```
+
+A websocket stays on the worker that accepted it, so with several workers
+each browser's callbacks all run on one worker. `--kind sync` (default) runs
+plain `def` callbacks; `--work-ms` makes each
+one burn that much CPU (or `asyncio.sleep` with `--kind async`).
+
+## Browser swarm (`swarm/`)
+
+The simulated clients above speak the wire protocols. The swarm drives real
+headless Chromium instead, many users per machine and many machines, against
+a deployed app, so the numbers include the renderer, its SharedWorkers and
+the browser's own connection limits. Each user clicks a plain HTTP callback,
+the same callback over the websocket, or a streaming callback
+(`swarm/app.py`), timed in the page from the click to the DOM update.
+
+Serve the app where it will be tested, the way it would be deployed:
+
+```bash
+SWARM_BACKEND=fastapi SWARM_REDIS_URL=redis://redis:6379/0 \
+    uvicorn benchmarks.swarm.app:server --host 0.0.0.0 --port 8050 --workers 4
+```
+
+On Flask (no websocket) pass `--scenario http,stream`.
+
+Try the swarm on one machine first (agents as local processes):
+
+```bash
+pip install playwright psutil && playwright install --only-shell chromium
+python -m benchmarks.swarm.run --target http://127.0.0.1:8050 --local 2 \
+    --users 10 25 --ramp 10 --duration 20
+```
+
+On a cloud, build the agent image (`swarm/Dockerfile`), push it to a
+registry, start N machines with Docker that can reach the app, pull the image
+on each, and list them in a hosts file (one `user@host` per line). Then from
+any machine with SSH access to them:
+
+```bash
+docker build -t REGISTRY/dash-swarm-agent benchmarks/swarm && docker push REGISTRY/dash-swarm-agent
+python -m benchmarks.swarm.run --target http://APP:8050 --hosts hosts.txt \
+    --image REGISTRY/dash-swarm-agent --users 25 50 100 --out swarm.json
+```
+
+`--users` is per machine and per step; every agent starts at the same
+wall-clock time (`--lead` seconds after the command, NTP keeps cloud clocks
+close enough) and reports come back over SSH. Steps stop climbing at 1%
+errors or a p95 over `--max-p95` ms (whole streams excluded: they take
+frames x interval by design). An agent whose machine passed 85% CPU is
+reported as `AGENT-BOUND`: give it fewer users or a bigger machine, its
+numbers describe the browsers, not the server. Measured locally at 1 s think
+time, a user costs about 0.05 core and up to 170 MB (RSS, an overcount);
+start at about 30 users per 4 vCPU / 8 GB machine and watch the flag.
+
+Reports from any number of agents merge exactly (`swarm/aggregate.py`):
+
+```bash
+python -m benchmarks.swarm.aggregate reports/*.json --out swarm.json
+```
+
+## Publishing (`publish.py`)
+
+`.github/workflows/benchmarks-publish.yml` measures on `dev` and pushes a static
+site to the `gh-pages` branch: renderer timings on every push, the streaming
+and callback sweeps weekly and on demand. Each chart has its own page under `embed/` to drop
+into an `<iframe>` (`?theme=dark` or `?theme=light` to force a theme), the raw
+numbers and their history are under `data/`, and `badges/` holds shields.io
+endpoint badges. To build the site locally:
+
+```bash
+python -m benchmarks.publish --site /tmp/site \
+    --renderer benchmarks/results.json --streaming benchmarks/streaming/results.json \
+    --callbacks benchmarks/callbacks/results.json
+```