# M12 — Detailed Implementation Plan

**Status:** planning (post-M11-conditional, blocked on NATS issue)
**Target:** M12 exit criterion from SPEC §19
**Goal:** Move from single-broker docker-compose to K8s-deployed multi-broker NATS, sustaining 50k/s end-to-end with M11 prod gate (10k/s sustained 10 min, p99 ≤ 50ms, DLQ=0) on a single ingestd instance.

---

## 0. Recap — what M12 must prove

| Milestone | Exit criterion | How it's measured |
|---|---|---|
| **M11 prod gate** (carried into M12) | internal Go service pushes ≥ 10k alerts/sec on one gRPC stream, p99 server-side `Ack` ≤ 50ms, DLQ=0, sustained 10 min | `m11_smoke.py` on K8s cluster, asserts on receive rate **and** publish success |
| **M12 ceiling** (new) | 50k/s end-to-end through multi-broker NATS + multi-ingestd, p99 router latency ≤ 50ms, no broker backpressure | `m10_bench_smoke.py` (adapted) on K8s, delivery stubbed |
| **M12 cluster** (new) | NATS cluster survives single-broker loss without message loss or sustained publish failure | chaos test: kill one NATS pod mid-soak, verify p99 stays under 100ms and no DLQ |

**Why M12 now (not later):** The 2-min re-verification on 2026-06-16 14:48 EDT exposed the single-NATS broker as the system bottleneck. NATS logged `[ERR] JetStream resource limits exceeded for server` every 10s under sustained ~6k/s load, and the receive rate collapsed to 0/s in 30s. Single-broker NATS on docker-compose cannot deliver v1 capacity (5k/s sustained per the SPEC, design ceiling 50k/s). M10-bench proved the **router** ceiling at 50k/s with delivery stubbed, but the broker was the implicit weak link — M12 surfaces and fixes it.

---

## 1. Workstream overview

```
┌──────────────────────────┐    ┌──────────────────────────┐    ┌──────────────────────────┐
│  W1: K8s manifests       │    │  W2: NATS JetStream      │    │  W3: Helm chart +        │
│  (Deployments, Services, │───▶│  cluster (3+ brokers,    │───▶│  argocd / flux /        │
│   ConfigMaps, Secrets)   │    │   R3, clustering)        │    │  plain kubectl           │
└──────────────────────────┘    └──────────────┬───────────┘    └──────────────┬───────────┘
                                               │                               │
                                               ▼                               ▼
                                     ┌──────────────────────────┐    ┌──────────────────────────┐
                                     │  W4: 50k/s ceiling bench │    │  W5: M11 prod gate on K8s│
                                     │  (M10-bench equivalent)  │    │  (10k/s × 10 min)        │
                                     │  delivery stubbed        │    │  full smoke (5 steps)    │
                                     └──────────────────────────┘    └──────────────────────────┘
                                               │                               │
                                               ▼                               ▼
                                     ┌──────────────────────────────────────────────────┐
                                     │  W6: CI/CD + observability for K8s              │
                                     │  (GitHub Actions / GitLab CI build+push,        │
                                     │   Prometheus Operator, Grafana dashboards)       │
                                     └──────────────────────────────────────────────────┘
```

Six workstreams, each ends with passing evidence before the next starts.

---

## 2. The four layers of risk (and what each WS defuses)

| Risk | Why it's scary | How we defuses it |
|---|---|---|
| **Single NATS broker is the bottleneck** (post-M11-conditional finding) | M11 10-min soak was green only because the rate metric counts gRPC receive, not publish success. Real-world delivery is broken under sustained load. | W2: NATS cluster (3+ brokers, R=3 JetStream streams) raises the limit ceiling by ~3x. W4: prove the cluster ceiling at 50k/s. |
| **K8s deployment is operationally complex** | K8s has a learning curve; mistakes (wrong resource limits, missing healthchecks, bad ConfigMap wiring) cause silent failures | W1: manifests mirror docker-compose service-by-service; W3: Helm chart encapsulates complexity; W6: GitOps pipeline means config changes are reviewable in PRs. |
| **Stateful workloads in K8s** | NATS, Postgres, ClickHouse, Redis all need persistent volumes and ordered shutdown. Compose just restarts; K8s needs StatefulSets + PodDisruptionBudgets. | W1: use StatefulSets for stateful services, Deployments for stateless (ingestd, routerd, deliverd-*). W3: Helm chart includes PVC templates + PDBs. |
| **Migrating from docker-compose to K8s in production** | Compose works; K8s is new. Risk of breaking the dev loop. | W1 keeps docker-compose as the dev story (unchanged), K8s is the deploy target only. M11+M12 work continues against compose. |

---

## 3. Workstream W1 — K8s manifests

### 3.1 Repo additions

```
deploy/k8s/
  base/
    namespace.yaml                # broad-announce namespace
    configmap.yaml                # BA_* env vars
    secrets.yaml.example          # fcm-credentials, telegram-bot-token, etc. (gitignored real one)
  postgres/
    statefulset.yaml              # 1 replica, 10Gi PVC, healthcheck
    service.yaml
    pdb.yaml                      # minAvailable: 1
  nats/                           # see W2 for cluster specifics
    statefulset.yaml
    headless-service.yaml
    configmap.yaml
    pdb.yaml
  redis/
    statefulset.yaml
    service.yaml
  clickhouse/
    statefulset.yaml
    service.yaml
  emqx/
    statefulset.yaml              # 1 replica (single-node OK for v2.0)
    service.yaml
  ingestd/
    deployment.yaml               # 2 replicas, max_inflight=256, BA_INGESTD_RATE_LIMIT_PER_SOURCE=20000
    service.yaml                  # ClusterIP, port 8800 + 9090 (gRPC)
    hpa.yaml                      # autoscale on CPU + custom metric (alerts_received rate)
  routerd/
    deployment.yaml
    service.yaml
  deliverd-fcm/
    deployment.yaml
    service.yaml
  deliverd-telegram/
    deployment.yaml
    service.yaml
  admind/
    deployment.yaml
    service.yaml
  archiverd/
    deployment.yaml
    service.yaml
  loadgen/                        # kept as Deployments, scaled up for the 50k/s bench
    deployment.yaml
```

### 3.2 Conversion rules (compose → K8s)

| compose | K8s |
|---|---|
| `service: ingestd` | `Deployment` named `ingestd`, `Service` ClusterIP, port 8800+9090 |
| `environment:` block | `ConfigMap` mounted as envFrom |
| `volumes: ["/data"]` | `StatefulSet` + `volumeClaimTemplates` with PVCs |
| `healthcheck: test: ...` | `livenessProbe` + `readinessProbe` (exec or httpGet) |
| `depends_on: [postgres]` | Same effect via init containers (wait for postgres:5432) or K8s-native service discovery with retries in app code |
| `ports: ["8800:8800"]` | `Service` (ClusterIP) + optional `Ingress` for external |
| `deploy.resources.limits` | `resources.limits` in container spec |
| profiles (loadgen-grpc) | `Deployment` in `deploy/k8s/overlays/loadgen/` (Kustomize) — off by default in base |

### 3.3 Exit criteria for W1

- All services from `docker-compose.yml` have K8s manifests in `deploy/k8s/base/`
- `kubectl apply -k deploy/k8s/base/` brings the stack up
- `kubectl get pods -n broad-announce` shows all services `Running` with `Ready 1/1` (or appropriate for stateful)
- Service-to-service DNS works (e.g., ingestd can reach nats:4222)
- The M11 10-min soak passes against the K8s deployment (regression check: nothing broke in the move)

---

## 4. Workstream W2 — NATS JetStream cluster

### 4.1 Why this is its own workstream

The 2026-06-16 14:48 EDT finding shows single-broker NATS hits resource limits under sustained ~6k/s. Multi-broker is M12's core fix, not a deployment detail.

### 4.2 Cluster topology

- **3 brokers** (`nats-0`, `nats-1`, `nats-2`) on separate nodes (anti-affinity)
- **JetStream** enabled on all 3
- **Stream `ALERTS`** with replicas=3, storage=File, max_age=24h
- **Subjects:** `alerts.<company_id>.<source_id>` (per ARCHITECTURE.md §6)
- **Consumer `ROUTER`** for routerd, replicas=3, ack_wait=30s, max_deliver=5
- **Consumer `DELIVERY_FCM`** / `DELIVERY_TELEGRAM` for deliverd-*, replicas=3

### 4.3 Config

```yaml
# deploy/k8s/base/nats/configmap.yaml
cluster:
  name: broad-announce-nats
  listen: 0.0.0.0:6222
  routes:
    - nats-broad-announce-nats-0.nats-headless:6222
    - nats-broad-announce-nats-1.nats-headless:6222
    - nats-broad-announce-nats-2.nats-headless:6222
jetstream:
  store_dir: /data/jetstream
  max_memory_store: 2Gi
  max_file_store: 100Gi
```

### 4.4 Capacity target (post-cluster)

| Stream | Per-broker | Cluster (R=3) |
|---|---|---|
| `ALERTS` throughput | ~20k/s | ~50-60k/s (3 brokers share load) |
| `ALERTS` storage | 100Gi | 300Gi (file-backed) |
| `ALERTS` max memory | 2Gi | 6Gi |

This raises the resource-limit ceiling by ~3x. The 50k/s ceiling bench (W4) verifies it.

### 4.5 Exit criteria for W2

- 3-broker NATS cluster running in K8s
- `nats stream info ALERTS` shows replicas=3, leader distributed
- Subject routing works: publish to `alerts.acme-001` on any broker lands in the stream
- Single-broker loss does not cause message loss (chaos test)

---

## 5. Workstream W3 — Helm chart + GitOps

### 5.1 Why a Helm chart

- Parameterization (replica counts, image tags, env vars) per environment
- Encapsulates the manifest sprawl from W1
- Standard for K8s packaging

### 5.2 Repo additions

```
deploy/helm/
  broad-announce/
    Chart.yaml
    values.yaml                  # defaults: dev-shape cluster, single NATS
    values.prod.yaml             # prod-shape: 3 NATS, multi-ingestd, higher resource limits
    templates/
      _helpers.tpl
      namespace.yaml
      configmap.yaml
      secrets.yaml
      postgres/
      nats/
      redis/
      clickhouse/
      emqx/
      ingestd/
      routerd/
      deliverd-fcm/
      deliverd-telegram/
      admind/
      archiverd/
```

### 5.3 Values structure

```yaml
# values.prod.yaml
global:
  imageRegistry: registry.techno-world.net/lrosales
  imageTag: "M12-W6"

nats:
  cluster:
    enabled: true
    replicas: 3
    storage: 100Gi
  jetstream:
    maxMemory: 2Gi
    maxFile: 100Gi

ingestd:
  replicas: 3
  resources:
    requests: {cpu: "500m", memory: "512Mi"}
    limits:   {cpu: "2",    memory: "1Gi"}
  env:
    BA_INGESTD_RATE_LIMIT_PER_SOURCE: "20000"
    BA_INGESTD_MAX_INFLIGHT: "256"

routerd:
  replicas: 2
  ...
```

### 5.4 GitOps (optional but recommended)

- **Argo CD** or **Flux** watches `deploy/helm/broad-announce/`
- PRs to `values.prod.yaml` trigger preview environments
- Auto-sync to prod after approval

### 5.5 Exit criteria for W3

- `helm install broad-announce deploy/helm/broad-announce/ -f values.prod.yaml -n broad-announce` brings up the prod-shape cluster
- `helm upgrade --reuse-values` works idempotently
- `helm template` produces valid manifests that pass `kubeconform` (or `kubectl apply --dry-run=server`)

---

## 6. Workstream W4 — 50k/s ceiling bench (M10-bench on K8s)

### 6.1 What this proves

The M10-bench in M10_BENCH_VERIFICATION.md proved the **router** ceiling at 50k/s on a single NATS broker with delivery stubbed. W4 re-proves it on the **3-broker cluster** with stubbed delivery, and is the gate for M12.

### 6.2 Test setup

- 2 `loadgen` Deployments (one per company/api-key, same as M11)
- Each loadgen targets the local K8s ingestd via the `ingestd` Service
- Combined rate target: 50k/s
- Duration: 5 min
- p99 router latency assertion: ≤ 50ms
- DLQ assertion: = 0
- Broker queue depth: informational

### 6.3 Exit criteria for W4

- 5/5 green runs at 50k/s on the K8s cluster
- Router p99 ≤ 50ms in all 5 runs
- DLQ = 0 throughout
- NATS cluster resource utilization: each broker < 80% CPU, < 80% memory

---

## 7. Workstream W5 — M11 prod gate on K8s (10k/s × 10 min, full smoke)

### 7.1 What this proves

M11 dev-playground gate passed on parres (4 cores, shared). M12 proves the M11 prod gate on the prod-shape cluster (8+ cores, dedicated). This is the second of the two M12 exit criteria.

### 7.2 Smoke updates needed (carry-over from M11-conditional)

The 2-min re-verification on 2026-06-16 14:48 EDT showed the M11 smoke's rate metric (`ba_ingestd_alerts_received_total`) does not assert on **publish success**. M12 W5 fixes this:

```python
# scripts/m11_smoke.py — add a publish-success assertion
def assert_grpc_publish_ok(target_rate, tolerance, window_seconds=30):
    """Rate of successful NATS publishes, must be ≥ target * (1 - tolerance)."""
    q = f'rate(ba_ingestd_nats_publish_total{{service="ingestd",result="ok"}}[{window_seconds}s])'
    rate = float(prom_query(q))
    if rate < target_rate * (1 - tolerance):
        fail_(f"NATS publish rate {rate:.0f}/s is below target {target_rate}/s")
    pass_(f"NATS publish rate {rate:.0f}/s ≥ {target_rate}/s")
```

(Assumes the `ba_ingestd_nats_publish_total` counter exists in ingestd with a `result` label. If not, it's added as part of W1 — see W1 §3.4.)

### 7.3 Test setup

- 2 `loadgen` Deployments targeting `ingestd:9090` (gRPC)
- Combined rate target: 10k/s
- Duration: 10 min
- p99 ack latency assertion: ≤ 50ms
- Receive rate assertion: ≥ 9k/s (±10%)
- **NEW: Publish rate assertion: ≥ 9k/s (±10%)** ← the fix
- DLQ assertion: = 0
- Backpressure step: 16 streams × 1k/s = 16k/s, all rate-limited acks honored

### 7.4 Exit criteria for W5

- 3/3 green runs at 10k/s on the K8s cluster
- 20/20 soak samples ≥ 9k/s receive rate
- 20/20 soak samples ≥ 9k/s publish rate ← the new assertion
- p99 ≤ 50ms in all samples
- DLQ = 0 throughout
- Backpressure step: 16/16 streams receive rate-limited acks when over per-source cap

---

## 8. Workstream W6 — CI/CD + observability for K8s

### 8.1 CI/CD (GitHub Actions or GitLab CI)

- **Build:** on PR, `docker build` for each service, push to `registry.techno-world.net/lrosales/broad-announce-{service}:<sha>`
- **Test:** `go test ./...` for each service; smoke against ephemeral docker-compose (existing M10/M11 harnesses)
- **Deploy:** on merge to main, GitOps (Argo CD) picks up the new image tag and applies via Helm

### 8.2 Observability

- **Prometheus Operator** + `ServiceMonitor` resources for each service
- **Grafana dashboards** carry over from docker-compose; the PromQL queries are unchanged
- **Loki** for log aggregation (already in stack)
- **Alertmanager** rules: NATS broker count, ingestd panic count, deliverd queue depth

### 8.3 Exit criteria for W6

- PR triggers CI build + test
- Merge to main triggers GitOps deploy to staging
- Staging smoke (M11 2-min @ 6k) passes automatically
- Dashboards in Grafana show all 5 service tiers (ingest, broker, route, deliver, archive)

---

## 9. Sequencing & parallelism

```
W1 ─────────────────┐
                    ├──▶ W2 (NATS cluster)
                    │       │
                    │       ├──▶ W3 (Helm) ──┐
                    │       │                  │
                    │       ▼                  ▼
                    │   W4 (50k/s bench)   W5 (M11 prod gate)
                    │       │                  │
                    │       └──────┬───────────┘
                    │              ▼
                    │         W6 (CI/CD + observability)
                    │
                    (W1 must land first; rest can parallelize with W3 as a fan-out)
```

Estimated wall time (with one full-time engineer):
- W1: 2-3 weeks (manifests + stateful services are fiddly)
- W2: 1 week (JetStream config + chaos test)
- W3: 1 week (Helm chart extraction)
- W4: 3 days (smoke adapted from M10-bench)
- W5: 1 week (smoke update for publish assertion + full 5-step verification)
- W6: 1 week (CI/CD + observability)
- **Total: ~6-8 weeks**

---

## 10. What M12 is *not*

(Scope boundaries to prevent creep)

- **Not a rewrite of the data plane.** ingestd, routerd, deliverd-* keep their Go code unchanged. M12 is deployment + NATS topology.
- **Not a regional/multi-region story.** v2.0 is single-region. Multi-region is v3.
- **Not an IaC migration.** Terraform / Pulumi for cloud resources is a separate milestone (M13?).
- **Not a security hardening pass.** mTLS, secrets management, network policies are M11.5 (still pending).
- **Not a release of the M11 conditional ship issue.** M11 NATS resource limit fix must land in a hotfix before M12 W5 runs. Otherwise the smoke's new publish-success assertion will fail on a single-broker configuration.

---

## 11. Open questions to resolve before W1 starts

1. **Cloud or on-prem K8s?** EKS/GKE/AKS/on-prem (kubeadm, k3s, Rancher)? Determines service-mesh and storage-class choices.
2. **GitOps tool:** Argo CD, Flux, or plain `kubectl apply` from CI? Argo is the most common; Flux is lighter-weight.
3. **NATS deployment mode:** official `nats-io/nats` Helm chart, the NATS operator, or custom StatefulSet? Custom gives more control; the operator gives rolling-upgrade + observability for free. **Recommendation:** NATS operator (jetstream + cluster + chaos-tested).
4. **Stateful storage:** cloud-provider CSI driver (EBS, GCE PD, Azure Disk) or on-prem (NFS, Ceph, Longhorn)? Determines PVC templates.
5. **TLS / mTLS in K8s:** cert-manager for in-cluster, or pin certs as Secrets? M11.5 will formalize this; M12 just needs to not paint us into a corner.
6. **Where does the K8s cluster live?** parres is the dev box (docker-compose). Is there a staging K8s cluster already, or do we need to provision one?
7. **K8s version target:** 1.28+ (covers everything we need). Pin in the chart.
8. **CI provider:** GitHub Actions, GitLab CI, or Jenkins? Already have a CI somewhere?

---

## 12. Prerequisites (M12 cannot start until these are done)

1. **M11 NATS resource limit issue resolved.** The 2-min re-verification on 2026-06-16 14:48 EDT showed single-broker NATS fails under sustained load. Either:
   - (a) Tune NATS limits on the single broker to a level that proves the publish path is healthy, OR
   - (b) Skip ahead to M12 W2 (multi-broker) and use the cluster to clear M11 fully
   - M12 W5 cannot pass the new publish-success assertion until this is resolved.
2. **M11 loadgen exit-code issue resolved** (currently deferred). The full 5-step smoke needs a green backpressure step.
3. **K8s cluster provisioned** (parres can host minikube/k3s for dev; staging cluster for CI; prod cluster for the M12 ceiling bench).
4. **Container registry** (`registry.techno-world.net/lrosales/...`) ready for CI-built images.
5. **Push current M11 work to git3** so the team can see the conditional-ship state and the M12 plan.

---

## 13. Definition of done

- [ ] W1: all 14 services have K8s manifests; `kubectl apply -k deploy/k8s/base/` brings them up
- [ ] W2: 3-broker NATS JetStream cluster, R=3 streams, chaos test passes
- [ ] W3: Helm chart installs + upgrades cleanly; values per environment
- [ ] W4: 5/5 green runs at 50k/s, router p99 ≤ 50ms, DLQ=0
- [ ] W5: 3/3 green runs at 10k/s, 20/20 soak samples green, **publish-success assertion passes**, full 5-step smoke green
- [ ] W6: CI builds + tests on PR; GitOps deploys on merge; Prometheus + Grafana cover all tiers
- [ ] SPEC.md M12 row updated to **✅ shipped YYYY-MM-DD**
- [ ] `M12_VERIFICATION.md` published with bench evidence + chaos test results
- [ ] `deploy/k8s/README.md` quickstart written

---

## 14. Risks & mitigations

| Risk | Likelihood | Impact | Mitigation |
|---|---|---|---|
| NATS operator / Helm chart is immature | Low | Medium (deployment friction) | Fall back to custom StatefulSet (W1 template) |
| Single-region outage takes down the cluster | Medium | High (revenue) | Multi-AZ nodes; PodDisruptionBudgets; M3 (multi-region) is the long-term fix |
| Helm values drift between dev / staging / prod | Medium | Medium (config bugs) | Single `values.yaml` with per-env overlays; Kustomize on top if needed |
| NATS cluster resource limits still hit at 50k/s | Low | High (M12 ceiling fails) | W4 finds the wall early; scale brokers to 5 if needed (R=5) |
| StatefulSet PVC migration in production | High | High (data loss) | Use Velero for backup; test restore in staging; never delete PVCs without backup |
| Cost: K8s cluster is more expensive than docker-compose | Certain | Low | Operational benefits outweigh the cost; capacity planning in W3 |

---

**Next step:** resolve open questions (§11) and the M11 NATS prerequisite (#1 in §12). W1 cannot start until both are settled.

---

## Appendix A — Push to git3 (current M11 work)

The local repo on the workspace (`/root/.openclaw/workspace/broad-announce/`) is 5 commits ahead of origin/master on git3, but origin has 6 commits not in local (the M11 hotfixes shipped from a different machine). The histories diverged at `09f0d54`.

To push the M11 work to git3, three approaches:

### A1. Merge origin/master into local, then push (recommended)

```bash
cd /root/.openclaw/workspace/broad-announce
git fetch origin master                                   # already done
git merge origin/master                                   # creates a merge commit; preserves both histories
# resolve any conflicts (likely in M11 smoke or loadgen code, since both sides edited)
git push origin master                                    # fast-forward or merge push
```

Pros: safest, no history rewrite, both sets of commits visible.
Cons: creates a merge commit; the graph has a "diamond" shape.

### A2. Rebase local onto origin, then force-push

```bash
cd /root/.openclaw/workspace/broad-announce
git fetch origin master                                   # already done
git rebase origin/master                                  # replays 5 local commits on top of origin's 6
# resolve any conflicts
git push --force-with-lease origin master                 # rewrites origin's master
```

Pros: clean linear history.
Cons: rewrites the commits the user (or their other machine) already pushed. If those commits are referenced anywhere (PRs, CI runs, tags), those references break.

### A3. Drop the local M11 work, fetch origin, and start fresh

```bash
cd /root/.openclaw/workspace/broad-announce
git fetch origin master
git reset --hard origin/master                            # discards local 5 commits
```

Pros: simple, no conflicts.
Cons: **loses the local M11 work entirely** (the conditional ship, the loadgen fix, the M11 smoke threshold). Not recommended unless the user is sure the local work is duplicated on origin.

**Recommended: A1 (merge).** The conditional-ship commit (`d8008bc`) is a real change to `SPEC.md` and `M11_VERIFICATION.md` that should be on origin. The loadgen fix (`39907d1`) is also worth shipping. Merging preserves both.

