broad-announce/M10_VERIFICATION.md

M10 Verification — Observability + Runaway-Source Isolation

What Was Built

M10 adds the infrastructure to run a 5k/s sustained soak test with per-source latency tracking and a fault-injection test that proves a runaway source does not degrade other companies.

Components

1. Per-Source Latency Histogram (W2)

ba_ingestd_publish_latency_seconds now has a source_id label:

# Per-source p99:
histogram_quantile(0.99,
  rate(ba_ingestd_publish_latency_seconds_bucket{source_id="prom-prod"}[60s]))

2. Pacer + 3-Instance Loadgen Cluster (W1)

loadgen/internal/pacer/pacer.go — linear ramp pacer. --ramp-up=30s brings each instance from 0 → 1700/s smoothly.

Instance	Company	Rate	Source
loadgen-http-1	acme-001	1700/s	prom-prod
loadgen-http-2	acme-002	1700/s	prom-prod
loadgen-http-3	acme-003	1700/s	prom-prod
Total		~5100/s

Per-company cap: 10,000/s. 3 × 1700 = 5100 is safe.

3. deliverd-bench No-Op (W3)

cmd/deliverd-bench/main.go — subscribes ba.*.deliveries, ACKs every message. No Postgres, no FCM/Telegram, no retry, no DLQ. Allows broker+router ceiling testing at 50k/s without burning FCM credits.

4. Runaway-Source Fault Injection (W4)

loadgen-http-4 is a 4th loadgen with the same company+source as loadgen-http-1 but at 10× the per-source cap (1000/s vs 100/s cap).

Instance	Company	Source	Rate	vs. Cap
loadgen-http-1	acme-001	prom-prod	1700/s	legal
loadgen-http-4 (rogue)	acme-001	prom-prod	1000/s	10× cap → 429s
loadgen-http-2	acme-002	prom-prod	1700/s	legal
loadgen-http-3	acme-003	prom-prod	1700/s	legal

Pass condition: p99 for acme-002 and acme-003 stays ≤ 5s during the 60-second rogue injection window.

Smoke Test

# Local
docker compose --profile loadgen-m10 up -d
python3 scripts/m10_smoke.py

# Remote (parres)
ssh root@192.168.44.94 \
  "cd /root/broad-announce && docker compose --profile loadgen-m10 up -d && python3 scripts/m10_smoke.py"

Expected Results

Step 1 — pre-flight             ✅ all services up, DLQ baseline clean
Step 2 — soak (10 min)         ✅ rate ~5000/s, p99 < 5s, DLQ = 0
Step 3 — runaway injection     ✅ acme-002 and acme-003 p99 ≤ 5s throughout
Step 4 — DLQ invariant        ✅ DLQ count = 0 over full soak window
Step 5 — teardown             ✅ cluster torn down

M10 Smoke Log

See M10_SMOKE_LOG.md for live run results.

Bench Profile (W5)

docker compose --profile bench up -d
python3 scripts/m10_bench_smoke.py

See M10_BENCH_VERIFICATION.md for broker+router ceiling test details.