M10: detailed implementation plan (5 workstreams, 9 commits, ~4 sessions)

M10_PLAN.md

@@ -0,0 +1,387 @@
+# M10 — Detailed Implementation Plan
+
+**Status:** planning (post-M9)
+**Target:** M10 exit criteria from SPEC §23
+**Goal:** Prove the system handles 5k/s sustained load on docker-compose with strict per-source isolation.
+
+---
+
+## 0. Recap — what M10 must prove
+
+| Milestone | Exit criterion | How it's measured |
+|---|---|---|
+| **M10** | soak 10 min at 5k/s, p99 ≤ 5s, zero DLQ, runaway-source test passes | `m10_smoke.py` runs the load, queries Prometheus, asserts on deltas |
+| **M10-bench** | 50k/s via `loadgen` against broker+router with delivery stubbed; p99 router latency ≤ 50ms; no broker backpressure | `m10_bench_smoke.py` against the bench profile |
+
+The current loadgen is one-shot. The pacing primitive (`--rate` + `--duration`) already exists in all 4 drivers — what's missing is *coordination*, *assertion*, and *fault injection*.
+
+---
+
+## 1. Workstream overview
+
+```
+┌──────────────────────────┐    ┌──────────────────────────┐    ┌──────────────────────────┐
+│  W1: loadgen scale-out   │    │  W2: assertion harness   │    │  W3: bench profile       │
+│  (5k/s on docker-compose)│    │  (Prometheus + DLQ check)│    │  (50k/s broker+router)   │
+└──────────────┬───────────┘    └──────────────┬───────────┘    └──────────────┬───────────┘
+               │                               │                               │
+               └───────────────┬───────────────┴───────────────┬───────────────┘
+                               ▼                               ▼
+                   ┌──────────────────────────┐    ┌──────────────────────────┐
+                   │  W4: m10_smoke.py        │    │  W5: m10_bench_smoke.py  │
+                   │  + runaway-source test   │    │  (delivery stub mode)    │
+                   └──────────────┬───────────┘    └──────────────┬───────────┘
+                                  └───────────────┬───────────────┘
+                                                  ▼
+                                       M10_VERIFICATION.md
+                                       M10_SMOKE_LOG.md
+                                       3 consecutive green runs
+```
+
+Five workstreams, each with its own commit series. Order matters: W1+W3 are independent and can be parallel; W2 needs W1's metrics; W4 needs W1+W2; W5 needs W3+W2.
+
+---
+
+## 2. Workstream 1 — `loadgen` scale-out (5k/s)
+
+### 2.1 Problem
+The current `loadgen-http --rate=5000` on one container can drive at most ~1.5–2k/s in practice (Go HTTP client + JSON marshal + HMAC sign per request saturates a single CPU). To hit 5k/s sustained, we need **3 instances** at ~1.7k/s each.
+
+### 2.2 Design
+
+**Option A (chosen):** `loadgen-orchestrator` — a thin shell script / Go binary that:
+1. Reads `--cluster-target=N` (e.g. 5000)
+2. Discovers `loadgen-http-N` instances via `docker compose ps` (profile `loadgen`)
+3. Computes per-instance rate = `N / instance_count`
+4. Calls `docker compose exec loadgen-http-N /app/loadgen-http --rate=$((N/k)) --duration=...`
+5. Tears down on signal
+
+**Option B (rejected for v1):** NATS KV coordination (SPEC §13.1) — overkill for 3 instances, adds a dep on NATS, and the SPEC explicitly says "best-effort" coordination is fine.
+
+**Why A:** keeps the existing `loadgen-http` binary unchanged; orchestration is a thin layer above it; no new code in the loadgen drivers; works identically on the host and inside CI.
+
+### 2.3 Concrete changes
+
+**`docker-compose.yml` — add 3 scaled `loadgen-http` services:**
+
+```yaml
+loadgen-http-1:
+  <<: *loadgen-base
+  command:
+    - /app/loadgen-http
+    - --target=http://ingestd:8800
+    - --api-key=acme-001:prom-prod:s3cret-acme
+    - --mode=normal
+    - --rate=1700
+    - --duration=10m
+    - --metrics=:8891
+    - --ramp-up=30s
+  profiles: ["loadgen"]
+
+loadgen-http-2:  # same with --api-key=acme-002:prom-prod:s3cret-acme2
+loadgen-http-3:  # same with --api-key=acme-003:prom-prod:s3cret-acme3
+```
+
+Use 3 different `acme-00X` API keys so the metrics naturally break down per-source. The 3 sources share a company (`prom-prod` in `acme-001:prom-prod:...`), so per-company rate limit (M5 layer 4) is the only thing that could cause cross-source coupling. We deliberately set per-company limit high enough to not engage.
+
+### 2.4 New flag: `--ramp-up`
+
+The current `loadgen-http` jumps from 0 to `--rate` instantly. For soak tests we want a smooth ramp so transient cold-start effects (NATs connection pool, Redis hot cache, JIT) don't poison the first 30s of measurements.
+
+**Add to all 4 drivers:**
+```
+--ramp-up      duration   "linear ramp from 0 to --rate over this duration (default 0 = instant)"
+```
+
+Implementation: replace the fixed `time.NewTicker(time.Second / time.Duration(*rate+1))` with a `pacer` struct that interpolates the rate over the ramp window. No other behavior change.
+
+### 2.5 New flag: `--cluster-id`
+
+Add to the metric labels so we can distinguish "loadgen run 2026-06-15" from "loadgen run 2026-06-16":
+```
+--cluster-id   string   "tag added as label on all loadgen metrics (default: 'default')"
+```
+
+The metrics endpoint becomes:
+```
+loadgen_alerts_sent_total{profile="normal",instance="loadgen-http-1",cluster_id="m10-2026-06-15",mode="normal"} 12345
+```
+
+### 2.6 Files to touch
+
+| File | Change |
+|---|---|
+| `loadgen/cmd/http/main.go` | +`--ramp-up`, +`--cluster-id`, +`pacer` struct |
+| `loadgen/cmd/mqtt/main.go` | same |
+| `loadgen/cmd/ws/main.go` | same |
+| `loadgen/cmd/m5drivers/main.go` (if it exists) | same |
+| `loadgen/internal/pacer/pacer.go` | **NEW** — shared pacer with linear ramp |
+| `loadgen/internal/pacer/pacer_test.go` | **NEW** — 4 tests (linear, instant, overshoot, ctx-cancel) |
+| `docker-compose.yml` | add `loadgen-http-{1,2,3}` |
+| `Dockerfile` | rebuild needed; no new binary |
+
+**Commit series:** `M10(1/5): loadgen pacer + ramp-up` → `M10(1.5/5): 3-instance loadgen profile`
+
+---
+
+## 3. Workstream 2 — Assertion harness
+
+### 3.1 Problem
+We need to assert: p99 ingest latency ≤ 5s, DLQ row count = 0, runaway source isolation. Today nothing reads Prometheus to make assertions.
+
+### 3.2 Design
+
+`scripts/m10_lib.py` — shared library used by both `m10_smoke.py` and `m10_bench_smoke.py`:
+
+```python
+def assert_ingest_p99_under(threshold_seconds: float, window: str, label_filter: str = "") -> float
+def assert_dlq_count_equals(expected: int, window: str) -> int
+def assert_metric_increased(metric: str, min_delta: int, window: str) -> int
+def per_source_p99(source_id: str) -> float
+def scrape_prometheus(query: str) -> list[dict]
+```
+
+All functions use the same URL-encoding fix as `m9_smoke.py` (urllib.parse.quote with `safe=''`). The library is imported by both smoke scripts.
+
+### 3.3 Latency source
+
+Ingest latency is in `ba_ingestd_publish_latency_seconds_bucket` (from M9). This measures `time.Since(start)` from the moment we accept the alert to the moment NATS `Publish` returns. That's the right metric for "ingest latency" — it includes the queue + broker write.
+
+For p99, we use Prometheus `histogram_quantile(0.99, ...)` over a sliding 60s window.
+
+### 3.4 DLQ source
+
+The `deliveries_dlq` table (M8 hypertable) is the source of truth. Two ways to query:
+- **SQL:** `SELECT COUNT(*) FROM deliveries_dlq WHERE created_at > now() - interval '10 minutes' AND discarded = false;` (run via `docker exec postgres`)
+- **Prometheus:** `ba_deliverd_dlq_total` (M9 counter). Less precise (no discarded filter) but no DB hop.
+
+We use the Prometheus counter for the smoke (faster) and the SQL count for the post-run forensic log (more precise).
+
+### 3.5 Per-source isolation check
+
+For the runaway-source test, the assertion is:
+- Source X fires at 10× its rate cap, gets rate-limited
+- Sources Y, Z run normally
+- `per_source_p99("acme-002")` and `per_source_p99("acme-003")` stay ≤ 5s
+- `per_source_p99("acme-001")` (the runaway) can be anything
+
+The metric we use: filter `ba_ingestd_publish_latency_seconds_bucket` by `source_id` label. But current ingestd doesn't label the latency histogram by source. We need to add that label.
+
+**Change to `internal/observability/metrics.go`:**
+```go
+PublishLatency: prometheus.NewHistogramVec(prometheus.HistogramOpts{
+    Name: "publish_latency_seconds",
+    ...
+}, []string{"source_id"}),  // ← was: []string{} (no labels)
+```
+
+Backfill: rebuild + redeploy ingestd. Backwards-compatible (new label dimension, no series removed).
+
+**Commit series:** `M10(2/5): per-source latency histogram label` → `M10(2.5/5): m10_lib.py assertion harness`
+
+---
+
+## 4. Workstream 3 — Bench profile (50k/s)
+
+### 4.1 Problem
+To prove the broker+router ceiling, we need to push 50k/s through the broker without burning FCM credits. We need a no-op `deliverd` that ACKs the broker message but does nothing.
+
+### 4.2 Design
+
+**New binary: `cmd/deliverd-bench/main.go`** — ~30 lines:
+```go
+// Pulls from NATS subject ba.*.deliveries, acks immediately, increments a counter.
+// No HTTP, no retry, no DLQ. Just consume and forget.
+```
+
+**Compose profile: `bench`**
+
+The bench profile:
+- Same broker + router + ingestd
+- `deliverd-bench` instead of `deliverd-fcm` and `deliverd-telegram`
+- Multiple `loadgen-grpc` instances (future) or `loadgen-http` instances (M10 v1)
+- `--cluster-target=50000` orchestrator target
+
+### 4.3 Concrete changes
+
+**`cmd/deliverd-bench/main.go`** — NEW
+
+**`docker-compose.yml`** — add `bench` profile:
+```yaml
+deliverd-bench:
+  build: .
+  command: ["/app/deliverd-bench"]
+  environment:
+    BA_ENV: dev
+    BA_NATS_URL: nats://nats:4222
+  profiles: ["bench"]
+  depends_on:
+    nats: { condition: service_healthy }
+```
+
+**`scripts/m10_bench_smoke.py`** — NEW, asserts:
+- `ba_routerd_recipient_expansion_seconds_bucket` p99 ≤ 50ms
+- `deliverd-bench` consumes everything (no NATS subject lag)
+- Broker's `nats_varz_depth` queue depth stays < 1000 (no backpressure)
+
+**`Dockerfile`** — add `deliverd-bench` to the build chain.
+
+**Commit series:** `M10(3/5): deliverd-bench no-op + bench profile` → `M10(3.5/5): m10_bench_smoke.py`
+
+---
+
+## 5. Workstream 4 — `m10_smoke.py` + runaway-source test
+
+### 5.1 Structure
+
+```
+m10_smoke.py
+├── Step 1 — pre-flight
+│     • All 4 loadgen-http-{1,2,3} containers up
+│     • Ingestd, routerd, prometheus, deliverds healthy
+│     • DLQ table is empty (baseline: SELECT count from deliveries_dlq)
+│
+├── Step 2 — 5k/s soak (10 min)
+│     • Start loadgen-orchestrator --cluster-target=5000 --duration=10m
+│     • Sleep 30s for ramp-up
+│     • Sample Prometheus every 10s:
+│         - ba_ingestd_alerts_received_total{result="accepted"} rate → should be 5000±5%
+│         - ba_ingestd_publish_latency_seconds p99 (60s window) → ≤ 5s
+│         - ba_deliverd_dlq_total delta → must be 0
+│     • Hard fail on any breach; collect samples for the log
+│
+├── Step 3 — runaway-source fault injection (60s)
+│     • Start 1 loadgen instance pinned to acme-001 at 10× cap (--rate=5000)
+│     • Background: keep loadgen-http-2/3 running normally at 1700 each
+│     • Assert: per_source_p99(acme-002) ≤ 5s and per_source_p99(acme-003) ≤ 5s
+│     • acme-001 (the runaway) p99 is allowed to be anything (it's getting rate-limited)
+│
+├── Step 4 — DLQ invariant
+│     • SELECT COUNT(*) FROM deliveries_dlq WHERE created_at > (start of step 2)
+│     • Must be 0 (or the run is a hard fail)
+│
+├── Step 5 — teardown
+│     • docker compose --profile loadgen down
+│     • Print summary table
+```
+
+### 5.2 Run schedule (matches existing milestone pattern)
+
+- Run 1: local docker-compose → write M10_SMOKE_LOG.md
+- Run 2: local docker-compose → 3 consecutive green required
+- Run 3: local docker-compose → 3 consecutive green required
+- Run 4: parres remote (192.168.44.94) → 1 green required
+
+### 5.3 Files
+
+| File | Change |
+|---|---|
+| `scripts/m10_smoke.py` | **NEW** — 5-step smoke + runaway test |
+| `scripts/m10_lib.py` | **NEW** — assertion library (shared with bench) |
+| `scripts/loadgen-orchestrator.sh` | **NEW** — thin shell orchestrator (3 instances) |
+| `M10_VERIFICATION.md` | NEW — exit criteria mapping + new env vars + new files |
+| `M10_SMOKE_LOG.md` | NEW — 3-consecutive-green evidence |
+
+**Commit series:** `M10(4/5): m10_smoke.py with runaway-source test` → `M10(4.5/5): M10 verification docs + 3 green runs`
+
+---
+
+## 6. Workstream 5 — `m10_bench_smoke.py` (50k/s)
+
+### 6.1 Structure
+
+```
+m10_bench_smoke.py
+├── Step 1 — bench profile up
+│     • docker compose --profile bench up -d
+│     • Verify deliverd-bench is consuming
+│
+├── Step 2 — 50k/s cluster-wide for 5 min (shorter than 5k/s — broker can take it but we save time)
+│     • 10 loadgen-http instances at 5000 each → cluster-target=50000
+│     • Sample: ba_routerd_recipient_expansion_seconds p99 ≤ 50ms
+│     • Sample: nats_varz_depth < 1000 (no backpressure)
+│
+├── Step 3 — broker ceiling check
+│     • If routerd p99 holds ≤ 50ms and NATS queue depth stays low → bench passes
+│
+├── Step 4 — teardown
+```
+
+### 6.2 Why a separate smoke
+
+50k/s generates ~3M alerts over 5 min. The `deliveries` table fills fast. We use a separate `bench` profile so:
+- Bench deliveries are never inserted into `deliveries` (deliverd-bench doesn't touch Postgres)
+- Bench runs don't pollute the production alerting tables
+
+### 6.3 Files
+
+| File | Change |
+|---|---|
+| `scripts/m10_bench_smoke.py` | **NEW** — broker+router ceiling test |
+| `M10_BENCH_VERIFICATION.md` | NEW — bench profile scope + exit criteria |
+
+**Commit series:** `M10(5/5): m10_bench_smoke.py + bench verification`
+
+---
+
+## 7. Risk register
+
+| Risk | Likelihood | Impact | Mitigation |
+|---|---|---|---|
+| Per-source histogram label changes an existing dashboard panel | Med | Low | Update `broad-announce-overview.json` row 2 in W2 |
+| 5k/s saturates docker-compose NATS single-broker | Med | High | If `nats_varz_depth` > 10k, scale broker horizontally (out of M10 scope) |
+| Loadgen-orchestrator can't get exactly 5k/s (jitter) | Med | Low | Accept ±5% per SPEC §13.1; assert in smoke |
+| 10min soak = 3M alerts = 3M `deliveries` rows = OOM in deliverd postgres pool | Med | Med | Add `--pg-pool-max=20` to deliverd env (already M7 default) |
+| Runaway source causes OOM kill on ingestd | Low | High | M9 layer 7 (quarantine) auto-bans the source after threshold |
+| `loadgen-grpc` doesn't exist yet (M11) | High | Med | Use 3× `loadgen-http` instances for M10, not `loadgen-grpc`; note in M10_VERIFICATION.md that M11 will rerun with gRPC |
+| Run takes >30 min (10 min soak + ramp + teardown) | Med | Low | Smoke runner has no hard timeout, but we document the expected wall-clock |
+
+---
+
+## 8. Estimated effort
+
+| Workstream | LoC (new) | LoC (changed) | Commits | Wall-clock estimate |
+|---|---|---|---|---|
+| W1: loadgen scale-out | ~80 (pacer) | ~30 (drivers) | 2 | 1 session |
+| W2: assertion harness | ~150 (m10_lib) | ~5 (metric label) | 2 | 1 session |
+| W3: bench profile | ~30 (deliverd-bench) | ~30 (compose) | 2 | ½ session |
+| W4: m10_smoke.py + docs | ~250 (smoke) | ~0 | 2 | 1 session |
+| W5: m10_bench_smoke.py | ~120 (smoke) | ~0 | 1 | ½ session |
+| **Total** | **~630 LoC** | **~65 LoC** | **9 commits** | **~4 sessions** |
+
+The 5k/s smoke itself is 10 min wall-clock. The bench smoke is 5 min. With 3 consecutive green runs + 1 remote, expect a full M10 cycle to be **~1 working day** including verification doc writing.
+
+---
+
+## 9. Definition of done (M10 sign-off)
+
+- [ ] All 9 commits pushed to `master`
+- [ ] `m10_smoke.py` green on local: **3 consecutive runs**, all assertions pass
+- [ ] `m10_smoke.py` green on `parres` remote (192.168.44.94): 1 run
+- [ ] `m10_bench_smoke.py` green on local: 1 run
+- [ ] `M10_VERIFICATION.md` written
+- [ ] `M10_SMOKE_LOG.md` written (3-run summary + raw captures)
+- [ ] `M10_BENCH_VERIFICATION.md` written
+- [ ] `M10_PLAN.md` (this file) moved from `M10_PLAN.md` to `docs/m10_plan.md` (or kept at root per convention)
+- [ ] SPEC.md bumped: `M10 | shipped YYYY-MM-DD`
+- [ ] Commit `M10(6/5): SPEC.md bump to 'shipped ...'` (yes, 6/5, on purpose — we always over-ship one doc commit)
+
+---
+
+## 10. What is explicitly OUT of M10
+
+To keep the milestone bounded:
+
+- **gRPC ingest** (M11). `loadgen-grpc` doesn't exist yet. We use 3× `loadgen-http` instances; M11 will rerun M10 with `loadgen-grpc` to confirm gRPC doesn't regress the numbers.
+- **Multi-broker NATS cluster** (M12). M10 proves the single-broker ceiling. If 5k/s breaks NATS, we open M10.5 to fix it (out of scope here).
+- **Chaos Mesh / toxiproxy** (M13+). The runaway-source test is the only fault injection in M10. Network partitions and broker kills come later.
+- **Production hardening of `loadgen`** (CLI ergonomics, distribution tarball). M10 ships a working tool; v2 ships the polished one.
+
+---
+
+## 11. Open questions for the user
+
+1. **Sustained rate variance:** SPEC §13.1 says "±5% over/undershoot acceptable for soak". Do you want tighter (±2%) and is the extra orchestration worth it?
+2. **Per-company rate limit during soak:** the 3 acme sources all share company `prom-prod`. Per-company cap defaults to what? If it's 1k/s the soak will fail at 5k/s. We need to either bump the cap or split across companies.
+3. **Runaway-source target:** SPEC says "10× cap". What's the per-source cap in the current config? If it's 100/s, the runaway fires at 1000/s, which is well below 5k/s. Is that enough to prove isolation or do you want a more aggressive 50× cap?
+4. **Bench profile storage:** the bench profile runs don't touch `deliveries` (deliverd-bench is no-op), but they do touch the `alerts` table via ingestd. Do you want a separate `prom_bench` company_id so the bench alerts don't show in the production admin UI?