# M10 — Detailed Implementation Plan

**Status:** planning (post-M9)
**Target:** M10 exit criteria from SPEC §23
**Goal:** Prove the system handles 5k/s sustained load on docker-compose with strict per-source isolation.

---

## 0. Recap — what M10 must prove

| Milestone | Exit criterion | How it's measured |
|---|---|---|
| **M10** | soak 10 min at 5k/s, p99 ≤ 5s, zero DLQ, runaway-source test passes | `m10_smoke.py` runs the load, queries Prometheus, asserts on deltas |
| **M10-bench** | 50k/s via `loadgen` against broker+router with delivery stubbed; p99 router latency ≤ 50ms; no broker backpressure | `m10_bench_smoke.py` against the bench profile |

The current loadgen is one-shot. The pacing primitive (`--rate` + `--duration`) already exists in all 4 drivers — what's missing is *coordination*, *assertion*, and *fault injection*.

---

## 1. Workstream overview

```
┌──────────────────────────┐    ┌──────────────────────────┐    ┌──────────────────────────┐
│  W1: loadgen scale-out   │    │  W2: assertion harness   │    │  W3: bench profile       │
│  (5k/s on docker-compose)│    │  (Prometheus + DLQ check)│    │  (50k/s broker+router)   │
└──────────────┬───────────┘    └──────────────┬───────────┘    └──────────────┬───────────┘
               │                               │                               │
               └───────────────┬───────────────┴───────────────┬───────────────┘
                               ▼                               ▼
                   ┌──────────────────────────┐    ┌──────────────────────────┐
                   │  W4: m10_smoke.py        │    │  W5: m10_bench_smoke.py  │
                   │  + runaway-source test   │    │  (delivery stub mode)    │
                   └──────────────┬───────────┘    └──────────────┬───────────┘
                                  └───────────────┬───────────────┘
                                                  ▼
                                       M10_VERIFICATION.md
                                       M10_SMOKE_LOG.md
                                       3 consecutive green runs
```

Five workstreams, each with its own commit series. Order matters: W1+W3 are independent and can be parallel; W2 needs W1's metrics; W4 needs W1+W2; W5 needs W3+W2.

---

## 2. Workstream 1 — `loadgen` scale-out (5k/s)

### 2.1 Problem
The current `loadgen-http --rate=5000` on one container can drive at most ~1.5–2k/s in practice (Go HTTP client + JSON marshal + HMAC sign per request saturates a single CPU). To hit 5k/s sustained, we need **3 instances** at ~1.7k/s each.

### 2.2 Design

**Option A (chosen):** `loadgen-orchestrator` — a thin shell script / Go binary that:
1. Reads `--cluster-target=N` (e.g. 5000)
2. Discovers `loadgen-http-N` instances via `docker compose ps` (profile `loadgen`)
3. Computes per-instance rate = `N / instance_count`
4. Calls `docker compose exec loadgen-http-N /app/loadgen-http --rate=$((N/k)) --duration=...`
5. Tears down on signal

**Option B (rejected for v1):** NATS KV coordination (SPEC §13.1) — overkill for 3 instances, adds a dep on NATS, and the SPEC explicitly says "best-effort" coordination is fine.

**Why A:** keeps the existing `loadgen-http` binary unchanged; orchestration is a thin layer above it; no new code in the loadgen drivers; works identically on the host and inside CI.

### 2.3 Concrete changes

**`docker-compose.yml` — add 3 scaled `loadgen-http` services:**

```yaml
loadgen-http-1:
  <<: *loadgen-base
  command:
    - /app/loadgen-http
    - --target=http://ingestd:8800
    - --api-key=acme-001:prom-prod:s3cret-acme
    - --mode=normal
    - --rate=1700
    - --duration=10m
    - --metrics=:8891
    - --ramp-up=30s
  profiles: ["loadgen"]

loadgen-http-2:  # same with --api-key=acme-002:prom-prod:s3cret-acme2
loadgen-http-3:  # same with --api-key=acme-003:prom-prod:s3cret-acme3
```

Use 3 different `acme-00X` API keys so the metrics naturally break down per-source. The 3 sources share a company (`prom-prod` in `acme-001:prom-prod:...`), so per-company rate limit (M5 layer 4) is the only thing that could cause cross-source coupling. We deliberately set per-company limit high enough to not engage.

### 2.4 New flag: `--ramp-up`

The current `loadgen-http` jumps from 0 to `--rate` instantly. For soak tests we want a smooth ramp so transient cold-start effects (NATs connection pool, Redis hot cache, JIT) don't poison the first 30s of measurements.

**Add to all 4 drivers:**
```
--ramp-up      duration   "linear ramp from 0 to --rate over this duration (default 0 = instant)"
```

Implementation: replace the fixed `time.NewTicker(time.Second / time.Duration(*rate+1))` with a `pacer` struct that interpolates the rate over the ramp window. No other behavior change.

### 2.5 New flag: `--cluster-id`

Add to the metric labels so we can distinguish "loadgen run 2026-06-15" from "loadgen run 2026-06-16":
```
--cluster-id   string   "tag added as label on all loadgen metrics (default: 'default')"
```

The metrics endpoint becomes:
```
loadgen_alerts_sent_total{profile="normal",instance="loadgen-http-1",cluster_id="m10-2026-06-15",mode="normal"} 12345
```

### 2.6 Files to touch

| File | Change |
|---|---|
| `loadgen/cmd/http/main.go` | +`--ramp-up`, +`--cluster-id`, +`pacer` struct |
| `loadgen/cmd/mqtt/main.go` | same |
| `loadgen/cmd/ws/main.go` | same |
| `loadgen/cmd/m5drivers/main.go` (if it exists) | same |
| `loadgen/internal/pacer/pacer.go` | **NEW** — shared pacer with linear ramp |
| `loadgen/internal/pacer/pacer_test.go` | **NEW** — 4 tests (linear, instant, overshoot, ctx-cancel) |
| `docker-compose.yml` | add `loadgen-http-{1,2,3}` |
| `Dockerfile` | rebuild needed; no new binary |

**Commit series:** `M10(1/5): loadgen pacer + ramp-up` → `M10(1.5/5): 3-instance loadgen profile`

---

## 3. Workstream 2 — Assertion harness

### 3.1 Problem
We need to assert: p99 ingest latency ≤ 5s, DLQ row count = 0, runaway source isolation. Today nothing reads Prometheus to make assertions.

### 3.2 Design

`scripts/m10_lib.py` — shared library used by both `m10_smoke.py` and `m10_bench_smoke.py`:

```python
def assert_ingest_p99_under(threshold_seconds: float, window: str, label_filter: str = "") -> float
def assert_dlq_count_equals(expected: int, window: str) -> int
def assert_metric_increased(metric: str, min_delta: int, window: str) -> int
def per_source_p99(source_id: str) -> float
def scrape_prometheus(query: str) -> list[dict]
```

All functions use the same URL-encoding fix as `m9_smoke.py` (urllib.parse.quote with `safe=''`). The library is imported by both smoke scripts.

### 3.3 Latency source

Ingest latency is in `ba_ingestd_publish_latency_seconds_bucket` (from M9). This measures `time.Since(start)` from the moment we accept the alert to the moment NATS `Publish` returns. That's the right metric for "ingest latency" — it includes the queue + broker write.

For p99, we use Prometheus `histogram_quantile(0.99, ...)` over a sliding 60s window.

### 3.4 DLQ source

The `deliveries_dlq` table (M8 hypertable) is the source of truth. Two ways to query:
- **SQL:** `SELECT COUNT(*) FROM deliveries_dlq WHERE created_at > now() - interval '10 minutes' AND discarded = false;` (run via `docker exec postgres`)
- **Prometheus:** `ba_deliverd_dlq_total` (M9 counter). Less precise (no discarded filter) but no DB hop.

We use the Prometheus counter for the smoke (faster) and the SQL count for the post-run forensic log (more precise).

### 3.5 Per-source isolation check

For the runaway-source test, the assertion is:
- Source X fires at 10× its rate cap, gets rate-limited
- Sources Y, Z run normally
- `per_source_p99("acme-002")` and `per_source_p99("acme-003")` stay ≤ 5s
- `per_source_p99("acme-001")` (the runaway) can be anything

The metric we use: filter `ba_ingestd_publish_latency_seconds_bucket` by `source_id` label. But current ingestd doesn't label the latency histogram by source. We need to add that label.

**Change to `internal/observability/metrics.go`:**
```go
PublishLatency: prometheus.NewHistogramVec(prometheus.HistogramOpts{
    Name: "publish_latency_seconds",
    ...
}, []string{"source_id"}),  // ← was: []string{} (no labels)
```

Backfill: rebuild + redeploy ingestd. Backwards-compatible (new label dimension, no series removed).

**Commit series:** `M10(2/5): per-source latency histogram label` → `M10(2.5/5): m10_lib.py assertion harness`

---

## 4. Workstream 3 — Bench profile (50k/s)

### 4.1 Problem
To prove the broker+router ceiling, we need to push 50k/s through the broker without burning FCM credits. We need a no-op `deliverd` that ACKs the broker message but does nothing.

### 4.2 Design

**New binary: `cmd/deliverd-bench/main.go`** — ~30 lines:
```go
// Pulls from NATS subject ba.*.deliveries, acks immediately, increments a counter.
// No HTTP, no retry, no DLQ. Just consume and forget.
```

**Compose profile: `bench`**

The bench profile:
- Same broker + router + ingestd
- `deliverd-bench` instead of `deliverd-fcm` and `deliverd-telegram`
- Multiple `loadgen-grpc` instances (future) or `loadgen-http` instances (M10 v1)
- `--cluster-target=50000` orchestrator target

### 4.3 Concrete changes

**`cmd/deliverd-bench/main.go`** — NEW

**`docker-compose.yml`** — add `bench` profile:
```yaml
deliverd-bench:
  build: .
  command: ["/app/deliverd-bench"]
  environment:
    BA_ENV: dev
    BA_NATS_URL: nats://nats:4222
  profiles: ["bench"]
  depends_on:
    nats: { condition: service_healthy }
```

**`scripts/m10_bench_smoke.py`** — NEW, asserts:
- `ba_routerd_recipient_expansion_seconds_bucket` p99 ≤ 50ms
- `deliverd-bench` consumes everything (no NATS subject lag)
- Broker's `nats_varz_depth` queue depth stays < 1000 (no backpressure)

**`Dockerfile`** — add `deliverd-bench` to the build chain.

**Commit series:** `M10(3/5): deliverd-bench no-op + bench profile` → `M10(3.5/5): m10_bench_smoke.py`

---

## 5. Workstream 4 — `m10_smoke.py` + runaway-source test

### 5.1 Structure

```
m10_smoke.py
├── Step 1 — pre-flight
│     • All 4 loadgen-http-{1,2,3} containers up
│     • Ingestd, routerd, prometheus, deliverds healthy
│     • DLQ table is empty (baseline: SELECT count from deliveries_dlq)
│
├── Step 2 — 5k/s soak (10 min)
│     • Start loadgen-orchestrator --cluster-target=5000 --duration=10m
│     • Sleep 30s for ramp-up
│     • Sample Prometheus every 10s:
│         - ba_ingestd_alerts_received_total{result="accepted"} rate → should be 5000±5%
│         - ba_ingestd_publish_latency_seconds p99 (60s window) → ≤ 5s
│         - ba_deliverd_dlq_total delta → must be 0
│     • Hard fail on any breach; collect samples for the log
│
├── Step 3 — runaway-source fault injection (60s)
│     • Start 1 loadgen instance pinned to acme-001 at 10× cap (--rate=5000)
│     • Background: keep loadgen-http-2/3 running normally at 1700 each
│     • Assert: per_source_p99(acme-002) ≤ 5s and per_source_p99(acme-003) ≤ 5s
│     • acme-001 (the runaway) p99 is allowed to be anything (it's getting rate-limited)
│
├── Step 4 — DLQ invariant
│     • SELECT COUNT(*) FROM deliveries_dlq WHERE created_at > (start of step 2)
│     • Must be 0 (or the run is a hard fail)
│
├── Step 5 — teardown
│     • docker compose --profile loadgen down
│     • Print summary table
```

### 5.2 Run schedule (matches existing milestone pattern)

- Run 1: local docker-compose → write M10_SMOKE_LOG.md
- Run 2: local docker-compose → 3 consecutive green required
- Run 3: local docker-compose → 3 consecutive green required
- Run 4: parres remote (192.168.44.94) → 1 green required

### 5.3 Files

| File | Change |
|---|---|
| `scripts/m10_smoke.py` | **NEW** — 5-step smoke + runaway test |
| `scripts/m10_lib.py` | **NEW** — assertion library (shared with bench) |
| `scripts/loadgen-orchestrator.sh` | **NEW** — thin shell orchestrator (3 instances) |
| `M10_VERIFICATION.md` | NEW — exit criteria mapping + new env vars + new files |
| `M10_SMOKE_LOG.md` | NEW — 3-consecutive-green evidence |

**Commit series:** `M10(4/5): m10_smoke.py with runaway-source test` → `M10(4.5/5): M10 verification docs + 3 green runs`

---

## 6. Workstream 5 — `m10_bench_smoke.py` (50k/s)

### 6.1 Structure

```
m10_bench_smoke.py
├── Step 1 — bench profile up
│     • docker compose --profile bench up -d
│     • Verify deliverd-bench is consuming
│
├── Step 2 — 50k/s cluster-wide for 5 min (shorter than 5k/s — broker can take it but we save time)
│     • 10 loadgen-http instances at 5000 each → cluster-target=50000
│     • Sample: ba_routerd_recipient_expansion_seconds p99 ≤ 50ms
│     • Sample: nats_varz_depth < 1000 (no backpressure)
│
├── Step 3 — broker ceiling check
│     • If routerd p99 holds ≤ 50ms and NATS queue depth stays low → bench passes
│
├── Step 4 — teardown
```

### 6.2 Why a separate smoke

50k/s generates ~3M alerts over 5 min. The `deliveries` table fills fast. We use a separate `bench` profile so:
- Bench deliveries are never inserted into `deliveries` (deliverd-bench doesn't touch Postgres)
- Bench runs don't pollute the production alerting tables

### 6.3 Files

| File | Change |
|---|---|
| `scripts/m10_bench_smoke.py` | **NEW** — broker+router ceiling test |
| `M10_BENCH_VERIFICATION.md` | NEW — bench profile scope + exit criteria |

**Commit series:** `M10(5/5): m10_bench_smoke.py + bench verification`

---

## 7. Risk register

| Risk | Likelihood | Impact | Mitigation |
|---|---|---|---|
| Per-source histogram label changes an existing dashboard panel | Med | Low | Update `broad-announce-overview.json` row 2 in W2 |
| 5k/s saturates docker-compose NATS single-broker | Med | High | If `nats_varz_depth` > 10k, scale broker horizontally (out of M10 scope) |
| Loadgen-orchestrator can't get exactly 5k/s (jitter) | Med | Low | Accept ±5% per SPEC §13.1; assert in smoke |
| 10min soak = 3M alerts = 3M `deliveries` rows = OOM in deliverd postgres pool | Med | Med | Add `--pg-pool-max=20` to deliverd env (already M7 default) |
| Runaway source causes OOM kill on ingestd | Low | High | M9 layer 7 (quarantine) auto-bans the source after threshold |
| `loadgen-grpc` doesn't exist yet (M11) | High | Med | Use 3× `loadgen-http` instances for M10, not `loadgen-grpc`; note in M10_VERIFICATION.md that M11 will rerun with gRPC |
| Run takes >30 min (10 min soak + ramp + teardown) | Med | Low | Smoke runner has no hard timeout, but we document the expected wall-clock |

---

## 8. Estimated effort

| Workstream | LoC (new) | LoC (changed) | Commits | Wall-clock estimate |
|---|---|---|---|---|
| W1: loadgen scale-out | ~80 (pacer) | ~30 (drivers) | 2 | 1 session |
| W2: assertion harness | ~150 (m10_lib) | ~5 (metric label) | 2 | 1 session |
| W3: bench profile | ~30 (deliverd-bench) | ~30 (compose) | 2 | ½ session |
| W4: m10_smoke.py + docs | ~250 (smoke) | ~0 | 2 | 1 session |
| W5: m10_bench_smoke.py | ~120 (smoke) | ~0 | 1 | ½ session |
| **Total** | **~630 LoC** | **~65 LoC** | **9 commits** | **~4 sessions** |

The 5k/s smoke itself is 10 min wall-clock. The bench smoke is 5 min. With 3 consecutive green runs + 1 remote, expect a full M10 cycle to be **~1 working day** including verification doc writing.

---

## 9. Definition of done (M10 sign-off)

- [ ] All 9 commits pushed to `master`
- [ ] `m10_smoke.py` green on local: **3 consecutive runs**, all assertions pass
- [ ] `m10_smoke.py` green on `parres` remote (192.168.44.94): 1 run
- [ ] `m10_bench_smoke.py` green on local: 1 run
- [ ] `M10_VERIFICATION.md` written
- [ ] `M10_SMOKE_LOG.md` written (3-run summary + raw captures)
- [ ] `M10_BENCH_VERIFICATION.md` written
- [ ] `M10_PLAN.md` (this file) moved from `M10_PLAN.md` to `docs/m10_plan.md` (or kept at root per convention)
- [ ] SPEC.md bumped: `M10 | shipped YYYY-MM-DD`
- [ ] Commit `M10(6/5): SPEC.md bump to 'shipped ...'` (yes, 6/5, on purpose — we always over-ship one doc commit)

---

## 10. What is explicitly OUT of M10

To keep the milestone bounded:

- **gRPC ingest** (M11). `loadgen-grpc` doesn't exist yet. We use 3× `loadgen-http` instances; M11 will rerun M10 with `loadgen-grpc` to confirm gRPC doesn't regress the numbers.
- **Multi-broker NATS cluster** (M12). M10 proves the single-broker ceiling. If 5k/s breaks NATS, we open M10.5 to fix it (out of scope here).
- **Chaos Mesh / toxiproxy** (M13+). The runaway-source test is the only fault injection in M10. Network partitions and broker kills come later.
- **Production hardening of `loadgen`** (CLI ergonomics, distribution tarball). M10 ships a working tool; v2 ships the polished one.

---

## 11. Open questions for the user

1. **Sustained rate variance:** SPEC §13.1 says "±5% over/undershoot acceptable for soak". Do you want tighter (±2%) and is the extra orchestration worth it?
2. **Per-company rate limit during soak:** the 3 acme sources all share company `prom-prod`. Per-company cap defaults to what? If it's 1k/s the soak will fail at 5k/s. We need to either bump the cap or split across companies.
3. **Runaway-source target:** SPEC says "10× cap". What's the per-source cap in the current config? If it's 100/s, the runaway fires at 1000/s, which is well below 5k/s. Is that enough to prove isolation or do you want a more aggressive 50× cap?
4. **Bench profile storage:** the bench profile runs don't touch `deliveries` (deliverd-bench is no-op), but they do touch the `alerts` table via ingestd. Do you want a separate `prom_bench` company_id so the bench alerts don't show in the production admin UI?