broad-announce/M10_PLAN.md

M10 — Detailed Implementation Plan

Status: planning (post-M9) Target: M10 exit criteria from SPEC §23 Goal: Prove the system handles 5k/s sustained load on docker-compose with strict per-source isolation.

---

0. Recap — what M10 must prove

Milestone	Exit criterion	How it's measured
M10	soak 10 min at 5k/s, p99 ≤ 5s, zero DLQ, runaway-source test passes	m10_smoke.py runs the load, queries Prometheus, asserts on deltas
M10-bench	50k/s via loadgen against broker+router with delivery stubbed; p99 router latency ≤ 50ms; no broker backpressure	m10_bench_smoke.py against the bench profile

The current loadgen is one-shot. The pacing primitive (--rate + --duration) already exists in all 4 drivers — what's missing is coordination, assertion, and fault injection.

---

1. Workstream overview

┌──────────────────────────┐    ┌──────────────────────────┐    ┌──────────────────────────┐
│  W1: loadgen scale-out   │    │  W2: assertion harness   │    │  W3: bench profile       │
│  (5k/s on docker-compose)│    │  (Prometheus + DLQ check)│    │  (50k/s broker+router)   │
└──────────────┬───────────┘    └──────────────┬───────────┘    └──────────────┬───────────┘
               │                               │                               │
               └───────────────┬───────────────┴───────────────┬───────────────┘
                               ▼                               ▼
                   ┌──────────────────────────┐    ┌──────────────────────────┐
                   │  W4: m10_smoke.py        │    │  W5: m10_bench_smoke.py  │
                   │  + runaway-source test   │    │  (delivery stub mode)    │
                   └──────────────┬───────────┘    └──────────────┬───────────┘
                                  └───────────────┬───────────────┘
                                                  ▼
                                       M10_VERIFICATION.md
                                       M10_SMOKE_LOG.md
                                       3 consecutive green runs

Five workstreams, each with its own commit series. Order matters: W1+W3 are independent and can be parallel; W2 needs W1's metrics; W4 needs W1+W2; W5 needs W3+W2.

---

2. Workstream 1 — loadgen scale-out (5k/s)

2.1 Problem

The current loadgen-http --rate=5000 on one container can drive at most ~1.5–2k/s in practice (Go HTTP client + JSON marshal + HMAC sign per request saturates a single CPU). To hit 5k/s sustained, we need 3 instances at ~1.7k/s each.

2.2 Design

Option A (chosen): loadgen-orchestrator — a thin shell script / Go binary that:

1. Reads --cluster-target=N (e.g. 5000)
2. Discovers loadgen-http-N instances via docker compose ps (profile loadgen)
3. Computes per-instance rate = N / instance_count
4. Calls docker compose exec loadgen-http-N /app/loadgen-http --rate=$((N/k)) --duration=...
5. Tears down on signal

Option B (rejected for v1): NATS KV coordination (SPEC §13.1) — overkill for 3 instances, adds a dep on NATS, and the SPEC explicitly says "best-effort" coordination is fine.

Why A: keeps the existing loadgen-http binary unchanged; orchestration is a thin layer above it; no new code in the loadgen drivers; works identically on the host and inside CI.

2.3 Concrete changes

docker-compose.yml — add 3 scaled loadgen-http services:

loadgen-http-1:
  <<: *loadgen-base
  command:
    - /app/loadgen-http
    - --target=http://ingestd:8800
    - --api-key=acme-001:prom-prod:s3cret-acme
    - --mode=normal
    - --rate=1700
    - --duration=10m
    - --metrics=:8891
    - --ramp-up=30s
  profiles: ["loadgen"]

loadgen-http-2:  # same with --api-key=acme-002:prom-prod:s3cret-acme2
loadgen-http-3:  # same with --api-key=acme-003:prom-prod:s3cret-acme3

Use 3 different acme-00X API keys so the metrics naturally break down per-source. The 3 sources share a company (prom-prod in acme-001:prom-prod:...), so per-company rate limit (M5 layer 4) is the only thing that could cause cross-source coupling. We deliberately set per-company limit high enough to not engage.

2.4 New flag: --ramp-up

The current loadgen-http jumps from 0 to --rate instantly. For soak tests we want a smooth ramp so transient cold-start effects (NATs connection pool, Redis hot cache, JIT) don't poison the first 30s of measurements.

Add to all 4 drivers:

--ramp-up      duration   "linear ramp from 0 to --rate over this duration (default 0 = instant)"

Implementation: replace the fixed time.NewTicker(time.Second / time.Duration(*rate+1)) with a pacer struct that interpolates the rate over the ramp window. No other behavior change.

2.5 New flag: --cluster-id

Add to the metric labels so we can distinguish "loadgen run 2026-06-15" from "loadgen run 2026-06-16":

--cluster-id   string   "tag added as label on all loadgen metrics (default: 'default')"

The metrics endpoint becomes:

loadgen_alerts_sent_total{profile="normal",instance="loadgen-http-1",cluster_id="m10-2026-06-15",mode="normal"} 12345

2.6 Files to touch

File	Change
loadgen/cmd/http/main.go	+--ramp-up, +--cluster-id, +pacer struct
loadgen/cmd/mqtt/main.go	same
loadgen/cmd/ws/main.go	same
loadgen/cmd/m5drivers/main.go (if it exists)	same
loadgen/internal/pacer/pacer.go	NEW — shared pacer with linear ramp
loadgen/internal/pacer/pacer_test.go	NEW — 4 tests (linear, instant, overshoot, ctx-cancel)
docker-compose.yml	add loadgen-http-{1,2,3}
Dockerfile	rebuild needed; no new binary

Commit series: M10(1/5): loadgen pacer + ramp-up → M10(1.5/5): 3-instance loadgen profile

---

3. Workstream 2 — Assertion harness

3.1 Problem

We need to assert: p99 ingest latency ≤ 5s, DLQ row count = 0, runaway source isolation. Today nothing reads Prometheus to make assertions.

3.2 Design

scripts/m10_lib.py — shared library used by both m10_smoke.py and m10_bench_smoke.py:

def assert_ingest_p99_under(threshold_seconds: float, window: str, label_filter: str = "") -> float
def assert_dlq_count_equals(expected: int, window: str) -> int
def assert_metric_increased(metric: str, min_delta: int, window: str) -> int
def per_source_p99(source_id: str) -> float
def scrape_prometheus(query: str) -> list[dict]

All functions use the same URL-encoding fix as m9_smoke.py (urllib.parse.quote with safe=''). The library is imported by both smoke scripts.

3.3 Latency source

Ingest latency is in ba_ingestd_publish_latency_seconds_bucket (from M9). This measures time.Since(start) from the moment we accept the alert to the moment NATS Publish returns. That's the right metric for "ingest latency" — it includes the queue + broker write.

For p99, we use Prometheus histogram_quantile(0.99, ...) over a sliding 60s window.

3.4 DLQ source

The deliveries_dlq table (M8 hypertable) is the source of truth. Two ways to query:

• SQL: SELECT COUNT(*) FROM deliveries_dlq WHERE created_at > now() - interval '10 minutes' AND discarded = false; (run via docker exec postgres)
• Prometheus: ba_deliverd_dlq_total (M9 counter). Less precise (no discarded filter) but no DB hop.

We use the Prometheus counter for the smoke (faster) and the SQL count for the post-run forensic log (more precise).

3.5 Per-source isolation check

For the runaway-source test, the assertion is:

• Source X fires at 10× its rate cap, gets rate-limited
• Sources Y, Z run normally
• per_source_p99("acme-002") and per_source_p99("acme-003") stay ≤ 5s
• per_source_p99("acme-001") (the runaway) can be anything

The metric we use: filter ba_ingestd_publish_latency_seconds_bucket by source_id label. But current ingestd doesn't label the latency histogram by source. We need to add that label.

Change to internal/observability/metrics.go:

PublishLatency: prometheus.NewHistogramVec(prometheus.HistogramOpts{
    Name: "publish_latency_seconds",
    ...
}, []string{"source_id"}),  // ← was: []string{} (no labels)

Backfill: rebuild + redeploy ingestd. Backwards-compatible (new label dimension, no series removed).

Commit series: M10(2/5): per-source latency histogram label → M10(2.5/5): m10_lib.py assertion harness

---

4. Workstream 3 — Bench profile (50k/s)

4.1 Problem

To prove the broker+router ceiling, we need to push 50k/s through the broker without burning FCM credits. We need a no-op deliverd that ACKs the broker message but does nothing.

4.2 Design

New binary: cmd/deliverd-bench/main.go — ~30 lines:

// Pulls from NATS subject ba.*.deliveries, acks immediately, increments a counter.
// No HTTP, no retry, no DLQ. Just consume and forget.

Compose profile: bench

The bench profile:

• Same broker + router + ingestd
• deliverd-bench instead of deliverd-fcm and deliverd-telegram
• Multiple loadgen-grpc instances (future) or loadgen-http instances (M10 v1)
• --cluster-target=50000 orchestrator target

4.3 Concrete changes

cmd/deliverd-bench/main.go — NEW

docker-compose.yml — add bench profile:

deliverd-bench:
  build: .
  command: ["/app/deliverd-bench"]
  environment:
    BA_ENV: dev
    BA_NATS_URL: nats://nats:4222
  profiles: ["bench"]
  depends_on:
    nats: { condition: service_healthy }

scripts/m10_bench_smoke.py — NEW, asserts:

• ba_routerd_recipient_expansion_seconds_bucket p99 ≤ 50ms
• deliverd-bench consumes everything (no NATS subject lag)
• Broker's nats_varz_depth queue depth stays < 1000 (no backpressure)

Dockerfile — add deliverd-bench to the build chain.

Commit series: M10(3/5): deliverd-bench no-op + bench profile → M10(3.5/5): m10_bench_smoke.py

---

5. Workstream 4 — m10_smoke.py + runaway-source test

5.1 Structure

m10_smoke.py
├── Step 1 — pre-flight
│     • All 4 loadgen-http-{1,2,3} containers up
│     • Ingestd, routerd, prometheus, deliverds healthy
│     • DLQ table is empty (baseline: SELECT count from deliveries_dlq)
│
├── Step 2 — 5k/s soak (10 min)
│     • Start loadgen-orchestrator --cluster-target=5000 --duration=10m
│     • Sleep 30s for ramp-up
│     • Sample Prometheus every 10s:
│         - ba_ingestd_alerts_received_total{result="accepted"} rate → should be 5000±5%
│         - ba_ingestd_publish_latency_seconds p99 (60s window) → ≤ 5s
│         - ba_deliverd_dlq_total delta → must be 0
│     • Hard fail on any breach; collect samples for the log
│
├── Step 3 — runaway-source fault injection (60s)
│     • Start 1 loadgen instance pinned to acme-001 at 10× cap (--rate=5000)
│     • Background: keep loadgen-http-2/3 running normally at 1700 each
│     • Assert: per_source_p99(acme-002) ≤ 5s and per_source_p99(acme-003) ≤ 5s
│     • acme-001 (the runaway) p99 is allowed to be anything (it's getting rate-limited)
│
├── Step 4 — DLQ invariant
│     • SELECT COUNT(*) FROM deliveries_dlq WHERE created_at > (start of step 2)
│     • Must be 0 (or the run is a hard fail)
│
├── Step 5 — teardown
│     • docker compose --profile loadgen down
│     • Print summary table

5.2 Run schedule (matches existing milestone pattern)

• Run 1: local docker-compose → write M10_SMOKE_LOG.md
• Run 2: local docker-compose → 3 consecutive green required
• Run 3: local docker-compose → 3 consecutive green required
• Run 4: parres remote (192.168.44.94) → 1 green required

5.3 Files

File	Change
scripts/m10_smoke.py	NEW — 5-step smoke + runaway test
scripts/m10_lib.py	NEW — assertion library (shared with bench)
scripts/loadgen-orchestrator.sh	NEW — thin shell orchestrator (3 instances)
M10_VERIFICATION.md	NEW — exit criteria mapping + new env vars + new files
M10_SMOKE_LOG.md	NEW — 3-consecutive-green evidence

Commit series: M10(4/5): m10_smoke.py with runaway-source test → M10(4.5/5): M10 verification docs + 3 green runs

---

6. Workstream 5 — m10_bench_smoke.py (50k/s)

6.1 Structure

m10_bench_smoke.py
├── Step 1 — bench profile up
│     • docker compose --profile bench up -d
│     • Verify deliverd-bench is consuming
│
├── Step 2 — 50k/s cluster-wide for 5 min (shorter than 5k/s — broker can take it but we save time)
│     • 10 loadgen-http instances at 5000 each → cluster-target=50000
│     • Sample: ba_routerd_recipient_expansion_seconds p99 ≤ 50ms
│     • Sample: nats_varz_depth < 1000 (no backpressure)
│
├── Step 3 — broker ceiling check
│     • If routerd p99 holds ≤ 50ms and NATS queue depth stays low → bench passes
│
├── Step 4 — teardown

6.2 Why a separate smoke

50k/s generates ~3M alerts over 5 min. The deliveries table fills fast. We use a separate bench profile so:

• Bench deliveries are never inserted into deliveries (deliverd-bench doesn't touch Postgres)
• Bench runs don't pollute the production alerting tables

6.3 Files

File	Change
scripts/m10_bench_smoke.py	NEW — broker+router ceiling test
M10_BENCH_VERIFICATION.md	NEW — bench profile scope + exit criteria

Commit series: M10(5/5): m10_bench_smoke.py + bench verification

---

7. Risk register

Risk	Likelihood	Impact	Mitigation
Per-source histogram label changes an existing dashboard panel	Med	Low	Update broad-announce-overview.json row 2 in W2
5k/s saturates docker-compose NATS single-broker	Med	High	If nats_varz_depth > 10k, scale broker horizontally (out of M10 scope)
Loadgen-orchestrator can't get exactly 5k/s (jitter)	Med	Low	Accept ±5% per SPEC §13.1; assert in smoke
10min soak = 3M alerts = 3M deliveries rows = OOM in deliverd postgres pool	Med	Med	Add --pg-pool-max=20 to deliverd env (already M7 default)
Runaway source causes OOM kill on ingestd	Low	High	M9 layer 7 (quarantine) auto-bans the source after threshold
loadgen-grpc doesn't exist yet (M11)	High	Med	Use 3× loadgen-http instances for M10, not loadgen-grpc; note in M10_VERIFICATION.md that M11 will rerun with gRPC
Run takes >30 min (10 min soak + ramp + teardown)	Med	Low	Smoke runner has no hard timeout, but we document the expected wall-clock

---

8. Estimated effort

Workstream	LoC (new)	LoC (changed)	Commits	Wall-clock estimate
W1: loadgen scale-out	~80 (pacer)	~30 (drivers)	2	1 session
W2: assertion harness	~150 (m10_lib)	~5 (metric label)	2	1 session
W3: bench profile	~30 (deliverd-bench)	~30 (compose)	2	½ session
W4: m10_smoke.py + docs	~250 (smoke)	~0	2	1 session
W5: m10_bench_smoke.py	~120 (smoke)	~0	1	½ session
Total	~630 LoC	~65 LoC	9 commits	~4 sessions

The 5k/s smoke itself is 10 min wall-clock. The bench smoke is 5 min. With 3 consecutive green runs + 1 remote, expect a full M10 cycle to be ~1 working day including verification doc writing.

---

9. Definition of done (M10 sign-off)

• All 9 commits pushed to master
• m10_smoke.py green on local: 3 consecutive runs, all assertions pass
• m10_smoke.py green on parres remote (192.168.44.94): 1 run
• m10_bench_smoke.py green on local: 1 run
• M10_VERIFICATION.md written
• M10_SMOKE_LOG.md written (3-run summary + raw captures)
• M10_BENCH_VERIFICATION.md written
• M10_PLAN.md (this file) moved from M10_PLAN.md to docs/m10_plan.md (or kept at root per convention)
• SPEC.md bumped: M10 | shipped YYYY-MM-DD
• Commit M10(6/5): SPEC.md bump to 'shipped ...' (yes, 6/5, on purpose — we always over-ship one doc commit)

---

10. What is explicitly OUT of M10

To keep the milestone bounded:

• gRPC ingest (M11). loadgen-grpc doesn't exist yet. We use 3× loadgen-http instances; M11 will rerun M10 with loadgen-grpc to confirm gRPC doesn't regress the numbers.
• Multi-broker NATS cluster (M12). M10 proves the single-broker ceiling. If 5k/s breaks NATS, we open M10.5 to fix it (out of scope here).
• Chaos Mesh / toxiproxy (M13+). The runaway-source test is the only fault injection in M10. Network partitions and broker kills come later.
• Production hardening of loadgen (CLI ergonomics, distribution tarball). M10 ships a working tool; v2 ships the polished one.

---

11. Open questions for the user

1. Sustained rate variance: SPEC §13.1 says "±5% over/undershoot acceptable for soak". Do you want tighter (±2%) and is the extra orchestration worth it?
2. Per-company rate limit during soak: the 3 acme sources all share company prom-prod. Per-company cap defaults to what? If it's 1k/s the soak will fail at 5k/s. We need to either bump the cap or split across companies.
3. Runaway-source target: SPEC says "10× cap". What's the per-source cap in the current config? If it's 100/s, the runaway fires at 1000/s, which is well below 5k/s. Is that enough to prove isolation or do you want a more aggressive 50× cap?
4. Bench profile storage: the bench profile runs don't touch deliveries (deliverd-bench is no-op), but they do touch the alerts table via ingestd. Do you want a separate prom_bench company_id so the bench alerts don't show in the production admin UI?