# M12 — Detailed Implementation Plan **Status:** planning (post-M11-conditional, blocked on NATS issue) **Target:** M12 exit criterion from SPEC §19 **Goal:** Move from single-broker docker-compose to K8s-deployed multi-broker NATS, sustaining 50k/s end-to-end with M11 prod gate (10k/s sustained 10 min, p99 ≤ 50ms, DLQ=0) on a single ingestd instance. --- ## 0. Recap — what M12 must prove | Milestone | Exit criterion | How it's measured | |---|---|---| | **M11 prod gate** (carried into M12) | internal Go service pushes ≥ 10k alerts/sec on one gRPC stream, p99 server-side `Ack` ≤ 50ms, DLQ=0, sustained 10 min | `m11_smoke.py` on K8s cluster, asserts on receive rate **and** publish success | | **M12 ceiling** (new) | 50k/s end-to-end through multi-broker NATS + multi-ingestd, p99 router latency ≤ 50ms, no broker backpressure | `m10_bench_smoke.py` (adapted) on K8s, delivery stubbed | | **M12 cluster** (new) | NATS cluster survives single-broker loss without message loss or sustained publish failure | chaos test: kill one NATS pod mid-soak, verify p99 stays under 100ms and no DLQ | **Why M12 now (not later):** The 2-min re-verification on 2026-06-16 14:48 EDT exposed the single-NATS broker as the system bottleneck. NATS logged `[ERR] JetStream resource limits exceeded for server` every 10s under sustained ~6k/s load, and the receive rate collapsed to 0/s in 30s. Single-broker NATS on docker-compose cannot deliver v1 capacity (5k/s sustained per the SPEC, design ceiling 50k/s). M10-bench proved the **router** ceiling at 50k/s with delivery stubbed, but the broker was the implicit weak link — M12 surfaces and fixes it. --- ## 1. Workstream overview ``` ┌──────────────────────────┐ ┌──────────────────────────┐ ┌──────────────────────────┐ │ W1: K8s manifests │ │ W2: NATS JetStream │ │ W3: Helm chart + │ │ (Deployments, Services, │───▶│ cluster (3+ brokers, │───▶│ argocd / flux / │ │ ConfigMaps, Secrets) │ │ R3, clustering) │ │ plain kubectl │ └──────────────────────────┘ └──────────────┬───────────┘ └──────────────┬───────────┘ │ │ ▼ ▼ ┌──────────────────────────┐ ┌──────────────────────────┐ │ W4: 50k/s ceiling bench │ │ W5: M11 prod gate on K8s│ │ (M10-bench equivalent) │ │ (10k/s × 10 min) │ │ delivery stubbed │ │ full smoke (5 steps) │ └──────────────────────────┘ └──────────────────────────┘ │ │ ▼ ▼ ┌──────────────────────────────────────────────────┐ │ W6: CI/CD + observability for K8s │ │ (GitHub Actions / GitLab CI build+push, │ │ Prometheus Operator, Grafana dashboards) │ └──────────────────────────────────────────────────┘ ``` Six workstreams, each ends with passing evidence before the next starts. --- ## 2. The four layers of risk (and what each WS defuses) | Risk | Why it's scary | How we defuses it | |---|---|---| | **Single NATS broker is the bottleneck** (post-M11-conditional finding) | M11 10-min soak was green only because the rate metric counts gRPC receive, not publish success. Real-world delivery is broken under sustained load. | W2: NATS cluster (3+ brokers, R=3 JetStream streams) raises the limit ceiling by ~3x. W4: prove the cluster ceiling at 50k/s. | | **K8s deployment is operationally complex** | K8s has a learning curve; mistakes (wrong resource limits, missing healthchecks, bad ConfigMap wiring) cause silent failures | W1: manifests mirror docker-compose service-by-service; W3: Helm chart encapsulates complexity; W6: GitOps pipeline means config changes are reviewable in PRs. | | **Stateful workloads in K8s** | NATS, Postgres, ClickHouse, Redis all need persistent volumes and ordered shutdown. Compose just restarts; K8s needs StatefulSets + PodDisruptionBudgets. | W1: use StatefulSets for stateful services, Deployments for stateless (ingestd, routerd, deliverd-*). W3: Helm chart includes PVC templates + PDBs. | | **Migrating from docker-compose to K8s in production** | Compose works; K8s is new. Risk of breaking the dev loop. | W1 keeps docker-compose as the dev story (unchanged), K8s is the deploy target only. M11+M12 work continues against compose. | --- ## 3. Workstream W1 — K8s manifests ### 3.1 Repo additions ``` deploy/k8s/ base/ namespace.yaml # broad-announce namespace configmap.yaml # BA_* env vars secrets.yaml.example # fcm-credentials, telegram-bot-token, etc. (gitignored real one) postgres/ statefulset.yaml # 1 replica, 10Gi PVC, healthcheck service.yaml pdb.yaml # minAvailable: 1 nats/ # see W2 for cluster specifics statefulset.yaml headless-service.yaml configmap.yaml pdb.yaml redis/ statefulset.yaml service.yaml clickhouse/ statefulset.yaml service.yaml emqx/ statefulset.yaml # 1 replica (single-node OK for v2.0) service.yaml ingestd/ deployment.yaml # 2 replicas, max_inflight=256, BA_INGESTD_RATE_LIMIT_PER_SOURCE=20000 service.yaml # ClusterIP, port 8800 + 9090 (gRPC) hpa.yaml # autoscale on CPU + custom metric (alerts_received rate) routerd/ deployment.yaml service.yaml deliverd-fcm/ deployment.yaml service.yaml deliverd-telegram/ deployment.yaml service.yaml admind/ deployment.yaml service.yaml archiverd/ deployment.yaml service.yaml loadgen/ # kept as Deployments, scaled up for the 50k/s bench deployment.yaml ``` ### 3.2 Conversion rules (compose → K8s) | compose | K8s | |---|---| | `service: ingestd` | `Deployment` named `ingestd`, `Service` ClusterIP, port 8800+9090 | | `environment:` block | `ConfigMap` mounted as envFrom | | `volumes: ["/data"]` | `StatefulSet` + `volumeClaimTemplates` with PVCs | | `healthcheck: test: ...` | `livenessProbe` + `readinessProbe` (exec or httpGet) | | `depends_on: [postgres]` | Same effect via init containers (wait for postgres:5432) or K8s-native service discovery with retries in app code | | `ports: ["8800:8800"]` | `Service` (ClusterIP) + optional `Ingress` for external | | `deploy.resources.limits` | `resources.limits` in container spec | | profiles (loadgen-grpc) | `Deployment` in `deploy/k8s/overlays/loadgen/` (Kustomize) — off by default in base | ### 3.3 Exit criteria for W1 - All services from `docker-compose.yml` have K8s manifests in `deploy/k8s/base/` - `kubectl apply -k deploy/k8s/base/` brings the stack up - `kubectl get pods -n broad-announce` shows all services `Running` with `Ready 1/1` (or appropriate for stateful) - Service-to-service DNS works (e.g., ingestd can reach nats:4222) - The M11 10-min soak passes against the K8s deployment (regression check: nothing broke in the move) --- ## 4. Workstream W2 — NATS JetStream cluster ### 4.1 Why this is its own workstream The 2026-06-16 14:48 EDT finding shows single-broker NATS hits resource limits under sustained ~6k/s. Multi-broker is M12's core fix, not a deployment detail. ### 4.2 Cluster topology - **3 brokers** (`nats-0`, `nats-1`, `nats-2`) on separate nodes (anti-affinity) - **JetStream** enabled on all 3 - **Stream `ALERTS`** with replicas=3, storage=File, max_age=24h - **Subjects:** `alerts..` (per ARCHITECTURE.md §6) - **Consumer `ROUTER`** for routerd, replicas=3, ack_wait=30s, max_deliver=5 - **Consumer `DELIVERY_FCM`** / `DELIVERY_TELEGRAM` for deliverd-*, replicas=3 ### 4.3 Config ```yaml # deploy/k8s/base/nats/configmap.yaml cluster: name: broad-announce-nats listen: 0.0.0.0:6222 routes: - nats-broad-announce-nats-0.nats-headless:6222 - nats-broad-announce-nats-1.nats-headless:6222 - nats-broad-announce-nats-2.nats-headless:6222 jetstream: store_dir: /data/jetstream max_memory_store: 2Gi max_file_store: 100Gi ``` ### 4.4 Capacity target (post-cluster) | Stream | Per-broker | Cluster (R=3) | |---|---|---| | `ALERTS` throughput | ~20k/s | ~50-60k/s (3 brokers share load) | | `ALERTS` storage | 100Gi | 300Gi (file-backed) | | `ALERTS` max memory | 2Gi | 6Gi | This raises the resource-limit ceiling by ~3x. The 50k/s ceiling bench (W4) verifies it. ### 4.5 Exit criteria for W2 - 3-broker NATS cluster running in K8s - `nats stream info ALERTS` shows replicas=3, leader distributed - Subject routing works: publish to `alerts.acme-001` on any broker lands in the stream - Single-broker loss does not cause message loss (chaos test) --- ## 5. Workstream W3 — Helm chart + GitOps ### 5.1 Why a Helm chart - Parameterization (replica counts, image tags, env vars) per environment - Encapsulates the manifest sprawl from W1 - Standard for K8s packaging ### 5.2 Repo additions ``` deploy/helm/ broad-announce/ Chart.yaml values.yaml # defaults: dev-shape cluster, single NATS values.prod.yaml # prod-shape: 3 NATS, multi-ingestd, higher resource limits templates/ _helpers.tpl namespace.yaml configmap.yaml secrets.yaml postgres/ nats/ redis/ clickhouse/ emqx/ ingestd/ routerd/ deliverd-fcm/ deliverd-telegram/ admind/ archiverd/ ``` ### 5.3 Values structure ```yaml # values.prod.yaml global: imageRegistry: registry.techno-world.net/lrosales imageTag: "M12-W6" nats: cluster: enabled: true replicas: 3 storage: 100Gi jetstream: maxMemory: 2Gi maxFile: 100Gi ingestd: replicas: 3 resources: requests: {cpu: "500m", memory: "512Mi"} limits: {cpu: "2", memory: "1Gi"} env: BA_INGESTD_RATE_LIMIT_PER_SOURCE: "20000" BA_INGESTD_MAX_INFLIGHT: "256" routerd: replicas: 2 ... ``` ### 5.4 GitOps (optional but recommended) - **Argo CD** or **Flux** watches `deploy/helm/broad-announce/` - PRs to `values.prod.yaml` trigger preview environments - Auto-sync to prod after approval ### 5.5 Exit criteria for W3 - `helm install broad-announce deploy/helm/broad-announce/ -f values.prod.yaml -n broad-announce` brings up the prod-shape cluster - `helm upgrade --reuse-values` works idempotently - `helm template` produces valid manifests that pass `kubeconform` (or `kubectl apply --dry-run=server`) --- ## 6. Workstream W4 — 50k/s ceiling bench (M10-bench on K8s) ### 6.1 What this proves The M10-bench in M10_BENCH_VERIFICATION.md proved the **router** ceiling at 50k/s on a single NATS broker with delivery stubbed. W4 re-proves it on the **3-broker cluster** with stubbed delivery, and is the gate for M12. ### 6.2 Test setup - 2 `loadgen` Deployments (one per company/api-key, same as M11) - Each loadgen targets the local K8s ingestd via the `ingestd` Service - Combined rate target: 50k/s - Duration: 5 min - p99 router latency assertion: ≤ 50ms - DLQ assertion: = 0 - Broker queue depth: informational ### 6.3 Exit criteria for W4 - 5/5 green runs at 50k/s on the K8s cluster - Router p99 ≤ 50ms in all 5 runs - DLQ = 0 throughout - NATS cluster resource utilization: each broker < 80% CPU, < 80% memory --- ## 7. Workstream W5 — M11 prod gate on K8s (10k/s × 10 min, full smoke) ### 7.1 What this proves M11 dev-playground gate passed on parres (4 cores, shared). M12 proves the M11 prod gate on the prod-shape cluster (8+ cores, dedicated). This is the second of the two M12 exit criteria. ### 7.2 Smoke updates needed (carry-over from M11-conditional) The 2-min re-verification on 2026-06-16 14:48 EDT showed the M11 smoke's rate metric (`ba_ingestd_alerts_received_total`) does not assert on **publish success**. M12 W5 fixes this: ```python # scripts/m11_smoke.py — add a publish-success assertion def assert_grpc_publish_ok(target_rate, tolerance, window_seconds=30): """Rate of successful NATS publishes, must be ≥ target * (1 - tolerance).""" q = f'rate(ba_ingestd_nats_publish_total{{service="ingestd",result="ok"}}[{window_seconds}s])' rate = float(prom_query(q)) if rate < target_rate * (1 - tolerance): fail_(f"NATS publish rate {rate:.0f}/s is below target {target_rate}/s") pass_(f"NATS publish rate {rate:.0f}/s ≥ {target_rate}/s") ``` (Assumes the `ba_ingestd_nats_publish_total` counter exists in ingestd with a `result` label. If not, it's added as part of W1 — see W1 §3.4.) ### 7.3 Test setup - 2 `loadgen` Deployments targeting `ingestd:9090` (gRPC) - Combined rate target: 10k/s - Duration: 10 min - p99 ack latency assertion: ≤ 50ms - Receive rate assertion: ≥ 9k/s (±10%) - **NEW: Publish rate assertion: ≥ 9k/s (±10%)** ← the fix - DLQ assertion: = 0 - Backpressure step: 16 streams × 1k/s = 16k/s, all rate-limited acks honored ### 7.4 Exit criteria for W5 - 3/3 green runs at 10k/s on the K8s cluster - 20/20 soak samples ≥ 9k/s receive rate - 20/20 soak samples ≥ 9k/s publish rate ← the new assertion - p99 ≤ 50ms in all samples - DLQ = 0 throughout - Backpressure step: 16/16 streams receive rate-limited acks when over per-source cap --- ## 8. Workstream W6 — CI/CD + observability for K8s ### 8.1 CI/CD (GitHub Actions or GitLab CI) - **Build:** on PR, `docker build` for each service, push to `registry.techno-world.net/lrosales/broad-announce-{service}:` - **Test:** `go test ./...` for each service; smoke against ephemeral docker-compose (existing M10/M11 harnesses) - **Deploy:** on merge to main, GitOps (Argo CD) picks up the new image tag and applies via Helm ### 8.2 Observability - **Prometheus Operator** + `ServiceMonitor` resources for each service - **Grafana dashboards** carry over from docker-compose; the PromQL queries are unchanged - **Loki** for log aggregation (already in stack) - **Alertmanager** rules: NATS broker count, ingestd panic count, deliverd queue depth ### 8.3 Exit criteria for W6 - PR triggers CI build + test - Merge to main triggers GitOps deploy to staging - Staging smoke (M11 2-min @ 6k) passes automatically - Dashboards in Grafana show all 5 service tiers (ingest, broker, route, deliver, archive) --- ## 9. Sequencing & parallelism ``` W1 ─────────────────┐ ├──▶ W2 (NATS cluster) │ │ │ ├──▶ W3 (Helm) ──┐ │ │ │ │ ▼ ▼ │ W4 (50k/s bench) W5 (M11 prod gate) │ │ │ │ └──────┬───────────┘ │ ▼ │ W6 (CI/CD + observability) │ (W1 must land first; rest can parallelize with W3 as a fan-out) ``` Estimated wall time (with one full-time engineer): - W1: 2-3 weeks (manifests + stateful services are fiddly) - W2: 1 week (JetStream config + chaos test) - W3: 1 week (Helm chart extraction) - W4: 3 days (smoke adapted from M10-bench) - W5: 1 week (smoke update for publish assertion + full 5-step verification) - W6: 1 week (CI/CD + observability) - **Total: ~6-8 weeks** --- ## 10. What M12 is *not* (Scope boundaries to prevent creep) - **Not a rewrite of the data plane.** ingestd, routerd, deliverd-* keep their Go code unchanged. M12 is deployment + NATS topology. - **Not a regional/multi-region story.** v2.0 is single-region. Multi-region is v3. - **Not an IaC migration.** Terraform / Pulumi for cloud resources is a separate milestone (M13?). - **Not a security hardening pass.** mTLS, secrets management, network policies are M11.5 (still pending). - **Not a release of the M11 conditional ship issue.** M11 NATS resource limit fix must land in a hotfix before M12 W5 runs. Otherwise the smoke's new publish-success assertion will fail on a single-broker configuration. --- ## 11. Open questions to resolve before W1 starts 1. **Cloud or on-prem K8s?** EKS/GKE/AKS/on-prem (kubeadm, k3s, Rancher)? Determines service-mesh and storage-class choices. 2. **GitOps tool:** Argo CD, Flux, or plain `kubectl apply` from CI? Argo is the most common; Flux is lighter-weight. 3. **NATS deployment mode:** official `nats-io/nats` Helm chart, the NATS operator, or custom StatefulSet? Custom gives more control; the operator gives rolling-upgrade + observability for free. **Recommendation:** NATS operator (jetstream + cluster + chaos-tested). 4. **Stateful storage:** cloud-provider CSI driver (EBS, GCE PD, Azure Disk) or on-prem (NFS, Ceph, Longhorn)? Determines PVC templates. 5. **TLS / mTLS in K8s:** cert-manager for in-cluster, or pin certs as Secrets? M11.5 will formalize this; M12 just needs to not paint us into a corner. 6. **Where does the K8s cluster live?** parres is the dev box (docker-compose). Is there a staging K8s cluster already, or do we need to provision one? 7. **K8s version target:** 1.28+ (covers everything we need). Pin in the chart. 8. **CI provider:** GitHub Actions, GitLab CI, or Jenkins? Already have a CI somewhere? --- ## 12. Prerequisites (M12 cannot start until these are done) 1. **M11 NATS resource limit issue resolved.** The 2-min re-verification on 2026-06-16 14:48 EDT showed single-broker NATS fails under sustained load. Either: - (a) Tune NATS limits on the single broker to a level that proves the publish path is healthy, OR - (b) Skip ahead to M12 W2 (multi-broker) and use the cluster to clear M11 fully - M12 W5 cannot pass the new publish-success assertion until this is resolved. 2. **M11 loadgen exit-code issue resolved** (currently deferred). The full 5-step smoke needs a green backpressure step. 3. **K8s cluster provisioned** (parres can host minikube/k3s for dev; staging cluster for CI; prod cluster for the M12 ceiling bench). 4. **Container registry** (`registry.techno-world.net/lrosales/...`) ready for CI-built images. 5. **Push current M11 work to git3** so the team can see the conditional-ship state and the M12 plan. --- ## 13. Definition of done - [ ] W1: all 14 services have K8s manifests; `kubectl apply -k deploy/k8s/base/` brings them up - [ ] W2: 3-broker NATS JetStream cluster, R=3 streams, chaos test passes - [ ] W3: Helm chart installs + upgrades cleanly; values per environment - [ ] W4: 5/5 green runs at 50k/s, router p99 ≤ 50ms, DLQ=0 - [ ] W5: 3/3 green runs at 10k/s, 20/20 soak samples green, **publish-success assertion passes**, full 5-step smoke green - [ ] W6: CI builds + tests on PR; GitOps deploys on merge; Prometheus + Grafana cover all tiers - [ ] SPEC.md M12 row updated to **✅ shipped YYYY-MM-DD** - [ ] `M12_VERIFICATION.md` published with bench evidence + chaos test results - [ ] `deploy/k8s/README.md` quickstart written --- ## 14. Risks & mitigations | Risk | Likelihood | Impact | Mitigation | |---|---|---|---| | NATS operator / Helm chart is immature | Low | Medium (deployment friction) | Fall back to custom StatefulSet (W1 template) | | Single-region outage takes down the cluster | Medium | High (revenue) | Multi-AZ nodes; PodDisruptionBudgets; M3 (multi-region) is the long-term fix | | Helm values drift between dev / staging / prod | Medium | Medium (config bugs) | Single `values.yaml` with per-env overlays; Kustomize on top if needed | | NATS cluster resource limits still hit at 50k/s | Low | High (M12 ceiling fails) | W4 finds the wall early; scale brokers to 5 if needed (R=5) | | StatefulSet PVC migration in production | High | High (data loss) | Use Velero for backup; test restore in staging; never delete PVCs without backup | | Cost: K8s cluster is more expensive than docker-compose | Certain | Low | Operational benefits outweigh the cost; capacity planning in W3 | --- **Next step:** resolve open questions (§11) and the M11 NATS prerequisite (#1 in §12). W1 cannot start until both are settled. --- ## Appendix A — Push to git3 (current M11 work) The local repo on the workspace (`/root/.openclaw/workspace/broad-announce/`) is 5 commits ahead of origin/master on git3, but origin has 6 commits not in local (the M11 hotfixes shipped from a different machine). The histories diverged at `09f0d54`. To push the M11 work to git3, three approaches: ### A1. Merge origin/master into local, then push (recommended) ```bash cd /root/.openclaw/workspace/broad-announce git fetch origin master # already done git merge origin/master # creates a merge commit; preserves both histories # resolve any conflicts (likely in M11 smoke or loadgen code, since both sides edited) git push origin master # fast-forward or merge push ``` Pros: safest, no history rewrite, both sets of commits visible. Cons: creates a merge commit; the graph has a "diamond" shape. ### A2. Rebase local onto origin, then force-push ```bash cd /root/.openclaw/workspace/broad-announce git fetch origin master # already done git rebase origin/master # replays 5 local commits on top of origin's 6 # resolve any conflicts git push --force-with-lease origin master # rewrites origin's master ``` Pros: clean linear history. Cons: rewrites the commits the user (or their other machine) already pushed. If those commits are referenced anywhere (PRs, CI runs, tags), those references break. ### A3. Drop the local M11 work, fetch origin, and start fresh ```bash cd /root/.openclaw/workspace/broad-announce git fetch origin master git reset --hard origin/master # discards local 5 commits ``` Pros: simple, no conflicts. Cons: **loses the local M11 work entirely** (the conditional ship, the loadgen fix, the M11 smoke threshold). Not recommended unless the user is sure the local work is duplicated on origin. **Recommended: A1 (merge).** The conditional-ship commit (`d8008bc`) is a real change to `SPEC.md` and `M11_VERIFICATION.md` that should be on origin. The loadgen fix (`39907d1`) is also worth shipping. Merging preserves both.