# M10 Verification — Observability + Runaway-Source Isolation

## What Was Built

M10 adds the infrastructure to run a 5k/s sustained soak test with per-source
latency tracking and a fault-injection test that proves a runaway source does
not degrade other companies.

## Components

### 1. Per-Source Latency Histogram (W2)
`ba_ingestd_publish_latency_seconds` now has a `source_id` label:

```promql
# Per-source p99:
histogram_quantile(0.99,
  rate(ba_ingestd_publish_latency_seconds_bucket{source_id="prom-prod"}[60s]))
```

### 2. Pacer + 3-Instance Loadgen Cluster (W1)
`loadgen/internal/pacer/pacer.go` — linear ramp pacer.
`--ramp-up=30s` brings each instance from 0 → 1700/s smoothly.

| Instance | Company | Rate | Source |
|---|---|---|---|
| loadgen-http-1 | acme-001 | 1700/s | prom-prod |
| loadgen-http-2 | acme-002 | 1700/s | prom-prod |
| loadgen-http-3 | acme-003 | 1700/s | prom-prod |
| **Total** | | **~5100/s** | |

Per-company cap: 10,000/s. 3 × 1700 = 5100 is safe.

### 3. deliverd-bench No-Op (W3)
`cmd/deliverd-bench/main.go` — subscribes `ba.*.deliveries`, ACKs every message.
No Postgres, no FCM/Telegram, no retry, no DLQ.
Allows broker+router ceiling testing at 50k/s without burning FCM credits.

### 4. Runaway-Source Fault Injection (W4)
`loadgen-http-4` is a 4th loadgen with the **same company+source as
loadgen-http-1** but at **10× the per-source cap** (1000/s vs 100/s cap).

| Instance | Company | Source | Rate | vs. Cap |
|---|---|---|---|---|
| loadgen-http-1 | acme-001 | prom-prod | 1700/s | legal |
| loadgen-http-4 (rogue) | acme-001 | prom-prod | 1000/s | 10× cap → 429s |
| loadgen-http-2 | acme-002 | prom-prod | 1700/s | legal |
| loadgen-http-3 | acme-003 | prom-prod | 1700/s | legal |

**Pass condition:** p99 for acme-002 and acme-003 stays ≤ 5s during the
60-second rogue injection window.

## Smoke Test

```bash
# Local
docker compose --profile loadgen-m10 up -d
python3 scripts/m10_smoke.py

# Remote (parres)
ssh root@192.168.44.94 \
  "cd /root/broad-announce && docker compose --profile loadgen-m10 up -d && python3 scripts/m10_smoke.py"
```

### Expected Results

```
Step 1 — pre-flight             ✅ all services up, DLQ baseline clean
Step 2 — soak (10 min)         ✅ rate ~5000/s, p99 < 5s, DLQ = 0
Step 3 — runaway injection     ✅ acme-002 and acme-003 p99 ≤ 5s throughout
Step 4 — DLQ invariant        ✅ DLQ count = 0 over full soak window
Step 5 — teardown             ✅ cluster torn down
```

## M10 Smoke Log

See `M10_SMOKE_LOG.md` for live run results.

## Bench Profile (W5)

```bash
docker compose --profile bench up -d
python3 scripts/m10_bench_smoke.py
```

See `M10_BENCH_VERIFICATION.md` for broker+router ceiling test details.