|
|
@@ -0,0 +1,500 @@
|
|
|
+# M12 — Detailed Implementation Plan
|
|
|
+
|
|
|
+**Status:** planning (post-M11-conditional, blocked on NATS issue)
|
|
|
+**Target:** M12 exit criterion from SPEC §19
|
|
|
+**Goal:** Move from single-broker docker-compose to K8s-deployed multi-broker NATS, sustaining 50k/s end-to-end with M11 prod gate (10k/s sustained 10 min, p99 ≤ 50ms, DLQ=0) on a single ingestd instance.
|
|
|
+
|
|
|
+---
|
|
|
+
|
|
|
+## 0. Recap — what M12 must prove
|
|
|
+
|
|
|
+| Milestone | Exit criterion | How it's measured |
|
|
|
+|---|---|---|
|
|
|
+| **M11 prod gate** (carried into M12) | internal Go service pushes ≥ 10k alerts/sec on one gRPC stream, p99 server-side `Ack` ≤ 50ms, DLQ=0, sustained 10 min | `m11_smoke.py` on K8s cluster, asserts on receive rate **and** publish success |
|
|
|
+| **M12 ceiling** (new) | 50k/s end-to-end through multi-broker NATS + multi-ingestd, p99 router latency ≤ 50ms, no broker backpressure | `m10_bench_smoke.py` (adapted) on K8s, delivery stubbed |
|
|
|
+| **M12 cluster** (new) | NATS cluster survives single-broker loss without message loss or sustained publish failure | chaos test: kill one NATS pod mid-soak, verify p99 stays under 100ms and no DLQ |
|
|
|
+
|
|
|
+**Why M12 now (not later):** The 2-min re-verification on 2026-06-16 14:48 EDT exposed the single-NATS broker as the system bottleneck. NATS logged `[ERR] JetStream resource limits exceeded for server` every 10s under sustained ~6k/s load, and the receive rate collapsed to 0/s in 30s. Single-broker NATS on docker-compose cannot deliver v1 capacity (5k/s sustained per the SPEC, design ceiling 50k/s). M10-bench proved the **router** ceiling at 50k/s with delivery stubbed, but the broker was the implicit weak link — M12 surfaces and fixes it.
|
|
|
+
|
|
|
+---
|
|
|
+
|
|
|
+## 1. Workstream overview
|
|
|
+
|
|
|
+```
|
|
|
+┌──────────────────────────┐ ┌──────────────────────────┐ ┌──────────────────────────┐
|
|
|
+│ W1: K8s manifests │ │ W2: NATS JetStream │ │ W3: Helm chart + │
|
|
|
+│ (Deployments, Services, │───▶│ cluster (3+ brokers, │───▶│ argocd / flux / │
|
|
|
+│ ConfigMaps, Secrets) │ │ R3, clustering) │ │ plain kubectl │
|
|
|
+└──────────────────────────┘ └──────────────┬───────────┘ └──────────────┬───────────┘
|
|
|
+ │ │
|
|
|
+ ▼ ▼
|
|
|
+ ┌──────────────────────────┐ ┌──────────────────────────┐
|
|
|
+ │ W4: 50k/s ceiling bench │ │ W5: M11 prod gate on K8s│
|
|
|
+ │ (M10-bench equivalent) │ │ (10k/s × 10 min) │
|
|
|
+ │ delivery stubbed │ │ full smoke (5 steps) │
|
|
|
+ └──────────────────────────┘ └──────────────────────────┘
|
|
|
+ │ │
|
|
|
+ ▼ ▼
|
|
|
+ ┌──────────────────────────────────────────────────┐
|
|
|
+ │ W6: CI/CD + observability for K8s │
|
|
|
+ │ (GitHub Actions / GitLab CI build+push, │
|
|
|
+ │ Prometheus Operator, Grafana dashboards) │
|
|
|
+ └──────────────────────────────────────────────────┘
|
|
|
+```
|
|
|
+
|
|
|
+Six workstreams, each ends with passing evidence before the next starts.
|
|
|
+
|
|
|
+---
|
|
|
+
|
|
|
+## 2. The four layers of risk (and what each WS defuses)
|
|
|
+
|
|
|
+| Risk | Why it's scary | How we defuses it |
|
|
|
+|---|---|---|
|
|
|
+| **Single NATS broker is the bottleneck** (post-M11-conditional finding) | M11 10-min soak was green only because the rate metric counts gRPC receive, not publish success. Real-world delivery is broken under sustained load. | W2: NATS cluster (3+ brokers, R=3 JetStream streams) raises the limit ceiling by ~3x. W4: prove the cluster ceiling at 50k/s. |
|
|
|
+| **K8s deployment is operationally complex** | K8s has a learning curve; mistakes (wrong resource limits, missing healthchecks, bad ConfigMap wiring) cause silent failures | W1: manifests mirror docker-compose service-by-service; W3: Helm chart encapsulates complexity; W6: GitOps pipeline means config changes are reviewable in PRs. |
|
|
|
+| **Stateful workloads in K8s** | NATS, Postgres, ClickHouse, Redis all need persistent volumes and ordered shutdown. Compose just restarts; K8s needs StatefulSets + PodDisruptionBudgets. | W1: use StatefulSets for stateful services, Deployments for stateless (ingestd, routerd, deliverd-*). W3: Helm chart includes PVC templates + PDBs. |
|
|
|
+| **Migrating from docker-compose to K8s in production** | Compose works; K8s is new. Risk of breaking the dev loop. | W1 keeps docker-compose as the dev story (unchanged), K8s is the deploy target only. M11+M12 work continues against compose. |
|
|
|
+
|
|
|
+---
|
|
|
+
|
|
|
+## 3. Workstream W1 — K8s manifests
|
|
|
+
|
|
|
+### 3.1 Repo additions
|
|
|
+
|
|
|
+```
|
|
|
+deploy/k8s/
|
|
|
+ base/
|
|
|
+ namespace.yaml # broad-announce namespace
|
|
|
+ configmap.yaml # BA_* env vars
|
|
|
+ secrets.yaml.example # fcm-credentials, telegram-bot-token, etc. (gitignored real one)
|
|
|
+ postgres/
|
|
|
+ statefulset.yaml # 1 replica, 10Gi PVC, healthcheck
|
|
|
+ service.yaml
|
|
|
+ pdb.yaml # minAvailable: 1
|
|
|
+ nats/ # see W2 for cluster specifics
|
|
|
+ statefulset.yaml
|
|
|
+ headless-service.yaml
|
|
|
+ configmap.yaml
|
|
|
+ pdb.yaml
|
|
|
+ redis/
|
|
|
+ statefulset.yaml
|
|
|
+ service.yaml
|
|
|
+ clickhouse/
|
|
|
+ statefulset.yaml
|
|
|
+ service.yaml
|
|
|
+ emqx/
|
|
|
+ statefulset.yaml # 1 replica (single-node OK for v2.0)
|
|
|
+ service.yaml
|
|
|
+ ingestd/
|
|
|
+ deployment.yaml # 2 replicas, max_inflight=256, BA_INGESTD_RATE_LIMIT_PER_SOURCE=20000
|
|
|
+ service.yaml # ClusterIP, port 8800 + 9090 (gRPC)
|
|
|
+ hpa.yaml # autoscale on CPU + custom metric (alerts_received rate)
|
|
|
+ routerd/
|
|
|
+ deployment.yaml
|
|
|
+ service.yaml
|
|
|
+ deliverd-fcm/
|
|
|
+ deployment.yaml
|
|
|
+ service.yaml
|
|
|
+ deliverd-telegram/
|
|
|
+ deployment.yaml
|
|
|
+ service.yaml
|
|
|
+ admind/
|
|
|
+ deployment.yaml
|
|
|
+ service.yaml
|
|
|
+ archiverd/
|
|
|
+ deployment.yaml
|
|
|
+ service.yaml
|
|
|
+ loadgen/ # kept as Deployments, scaled up for the 50k/s bench
|
|
|
+ deployment.yaml
|
|
|
+```
|
|
|
+
|
|
|
+### 3.2 Conversion rules (compose → K8s)
|
|
|
+
|
|
|
+| compose | K8s |
|
|
|
+|---|---|
|
|
|
+| `service: ingestd` | `Deployment` named `ingestd`, `Service` ClusterIP, port 8800+9090 |
|
|
|
+| `environment:` block | `ConfigMap` mounted as envFrom |
|
|
|
+| `volumes: ["/data"]` | `StatefulSet` + `volumeClaimTemplates` with PVCs |
|
|
|
+| `healthcheck: test: ...` | `livenessProbe` + `readinessProbe` (exec or httpGet) |
|
|
|
+| `depends_on: [postgres]` | Same effect via init containers (wait for postgres:5432) or K8s-native service discovery with retries in app code |
|
|
|
+| `ports: ["8800:8800"]` | `Service` (ClusterIP) + optional `Ingress` for external |
|
|
|
+| `deploy.resources.limits` | `resources.limits` in container spec |
|
|
|
+| profiles (loadgen-grpc) | `Deployment` in `deploy/k8s/overlays/loadgen/` (Kustomize) — off by default in base |
|
|
|
+
|
|
|
+### 3.3 Exit criteria for W1
|
|
|
+
|
|
|
+- All services from `docker-compose.yml` have K8s manifests in `deploy/k8s/base/`
|
|
|
+- `kubectl apply -k deploy/k8s/base/` brings the stack up
|
|
|
+- `kubectl get pods -n broad-announce` shows all services `Running` with `Ready 1/1` (or appropriate for stateful)
|
|
|
+- Service-to-service DNS works (e.g., ingestd can reach nats:4222)
|
|
|
+- The M11 10-min soak passes against the K8s deployment (regression check: nothing broke in the move)
|
|
|
+
|
|
|
+---
|
|
|
+
|
|
|
+## 4. Workstream W2 — NATS JetStream cluster
|
|
|
+
|
|
|
+### 4.1 Why this is its own workstream
|
|
|
+
|
|
|
+The 2026-06-16 14:48 EDT finding shows single-broker NATS hits resource limits under sustained ~6k/s. Multi-broker is M12's core fix, not a deployment detail.
|
|
|
+
|
|
|
+### 4.2 Cluster topology
|
|
|
+
|
|
|
+- **3 brokers** (`nats-0`, `nats-1`, `nats-2`) on separate nodes (anti-affinity)
|
|
|
+- **JetStream** enabled on all 3
|
|
|
+- **Stream `ALERTS`** with replicas=3, storage=File, max_age=24h
|
|
|
+- **Subjects:** `alerts.<company_id>.<source_id>` (per ARCHITECTURE.md §6)
|
|
|
+- **Consumer `ROUTER`** for routerd, replicas=3, ack_wait=30s, max_deliver=5
|
|
|
+- **Consumer `DELIVERY_FCM`** / `DELIVERY_TELEGRAM` for deliverd-*, replicas=3
|
|
|
+
|
|
|
+### 4.3 Config
|
|
|
+
|
|
|
+```yaml
|
|
|
+# deploy/k8s/base/nats/configmap.yaml
|
|
|
+cluster:
|
|
|
+ name: broad-announce-nats
|
|
|
+ listen: 0.0.0.0:6222
|
|
|
+ routes:
|
|
|
+ - nats-broad-announce-nats-0.nats-headless:6222
|
|
|
+ - nats-broad-announce-nats-1.nats-headless:6222
|
|
|
+ - nats-broad-announce-nats-2.nats-headless:6222
|
|
|
+jetstream:
|
|
|
+ store_dir: /data/jetstream
|
|
|
+ max_memory_store: 2Gi
|
|
|
+ max_file_store: 100Gi
|
|
|
+```
|
|
|
+
|
|
|
+### 4.4 Capacity target (post-cluster)
|
|
|
+
|
|
|
+| Stream | Per-broker | Cluster (R=3) |
|
|
|
+|---|---|---|
|
|
|
+| `ALERTS` throughput | ~20k/s | ~50-60k/s (3 brokers share load) |
|
|
|
+| `ALERTS` storage | 100Gi | 300Gi (file-backed) |
|
|
|
+| `ALERTS` max memory | 2Gi | 6Gi |
|
|
|
+
|
|
|
+This raises the resource-limit ceiling by ~3x. The 50k/s ceiling bench (W4) verifies it.
|
|
|
+
|
|
|
+### 4.5 Exit criteria for W2
|
|
|
+
|
|
|
+- 3-broker NATS cluster running in K8s
|
|
|
+- `nats stream info ALERTS` shows replicas=3, leader distributed
|
|
|
+- Subject routing works: publish to `alerts.acme-001` on any broker lands in the stream
|
|
|
+- Single-broker loss does not cause message loss (chaos test)
|
|
|
+
|
|
|
+---
|
|
|
+
|
|
|
+## 5. Workstream W3 — Helm chart + GitOps
|
|
|
+
|
|
|
+### 5.1 Why a Helm chart
|
|
|
+
|
|
|
+- Parameterization (replica counts, image tags, env vars) per environment
|
|
|
+- Encapsulates the manifest sprawl from W1
|
|
|
+- Standard for K8s packaging
|
|
|
+
|
|
|
+### 5.2 Repo additions
|
|
|
+
|
|
|
+```
|
|
|
+deploy/helm/
|
|
|
+ broad-announce/
|
|
|
+ Chart.yaml
|
|
|
+ values.yaml # defaults: dev-shape cluster, single NATS
|
|
|
+ values.prod.yaml # prod-shape: 3 NATS, multi-ingestd, higher resource limits
|
|
|
+ templates/
|
|
|
+ _helpers.tpl
|
|
|
+ namespace.yaml
|
|
|
+ configmap.yaml
|
|
|
+ secrets.yaml
|
|
|
+ postgres/
|
|
|
+ nats/
|
|
|
+ redis/
|
|
|
+ clickhouse/
|
|
|
+ emqx/
|
|
|
+ ingestd/
|
|
|
+ routerd/
|
|
|
+ deliverd-fcm/
|
|
|
+ deliverd-telegram/
|
|
|
+ admind/
|
|
|
+ archiverd/
|
|
|
+```
|
|
|
+
|
|
|
+### 5.3 Values structure
|
|
|
+
|
|
|
+```yaml
|
|
|
+# values.prod.yaml
|
|
|
+global:
|
|
|
+ imageRegistry: registry.techno-world.net/lrosales
|
|
|
+ imageTag: "M12-W6"
|
|
|
+
|
|
|
+nats:
|
|
|
+ cluster:
|
|
|
+ enabled: true
|
|
|
+ replicas: 3
|
|
|
+ storage: 100Gi
|
|
|
+ jetstream:
|
|
|
+ maxMemory: 2Gi
|
|
|
+ maxFile: 100Gi
|
|
|
+
|
|
|
+ingestd:
|
|
|
+ replicas: 3
|
|
|
+ resources:
|
|
|
+ requests: {cpu: "500m", memory: "512Mi"}
|
|
|
+ limits: {cpu: "2", memory: "1Gi"}
|
|
|
+ env:
|
|
|
+ BA_INGESTD_RATE_LIMIT_PER_SOURCE: "20000"
|
|
|
+ BA_INGESTD_MAX_INFLIGHT: "256"
|
|
|
+
|
|
|
+routerd:
|
|
|
+ replicas: 2
|
|
|
+ ...
|
|
|
+```
|
|
|
+
|
|
|
+### 5.4 GitOps (optional but recommended)
|
|
|
+
|
|
|
+- **Argo CD** or **Flux** watches `deploy/helm/broad-announce/`
|
|
|
+- PRs to `values.prod.yaml` trigger preview environments
|
|
|
+- Auto-sync to prod after approval
|
|
|
+
|
|
|
+### 5.5 Exit criteria for W3
|
|
|
+
|
|
|
+- `helm install broad-announce deploy/helm/broad-announce/ -f values.prod.yaml -n broad-announce` brings up the prod-shape cluster
|
|
|
+- `helm upgrade --reuse-values` works idempotently
|
|
|
+- `helm template` produces valid manifests that pass `kubeconform` (or `kubectl apply --dry-run=server`)
|
|
|
+
|
|
|
+---
|
|
|
+
|
|
|
+## 6. Workstream W4 — 50k/s ceiling bench (M10-bench on K8s)
|
|
|
+
|
|
|
+### 6.1 What this proves
|
|
|
+
|
|
|
+The M10-bench in M10_BENCH_VERIFICATION.md proved the **router** ceiling at 50k/s on a single NATS broker with delivery stubbed. W4 re-proves it on the **3-broker cluster** with stubbed delivery, and is the gate for M12.
|
|
|
+
|
|
|
+### 6.2 Test setup
|
|
|
+
|
|
|
+- 2 `loadgen` Deployments (one per company/api-key, same as M11)
|
|
|
+- Each loadgen targets the local K8s ingestd via the `ingestd` Service
|
|
|
+- Combined rate target: 50k/s
|
|
|
+- Duration: 5 min
|
|
|
+- p99 router latency assertion: ≤ 50ms
|
|
|
+- DLQ assertion: = 0
|
|
|
+- Broker queue depth: informational
|
|
|
+
|
|
|
+### 6.3 Exit criteria for W4
|
|
|
+
|
|
|
+- 5/5 green runs at 50k/s on the K8s cluster
|
|
|
+- Router p99 ≤ 50ms in all 5 runs
|
|
|
+- DLQ = 0 throughout
|
|
|
+- NATS cluster resource utilization: each broker < 80% CPU, < 80% memory
|
|
|
+
|
|
|
+---
|
|
|
+
|
|
|
+## 7. Workstream W5 — M11 prod gate on K8s (10k/s × 10 min, full smoke)
|
|
|
+
|
|
|
+### 7.1 What this proves
|
|
|
+
|
|
|
+M11 dev-playground gate passed on parres (4 cores, shared). M12 proves the M11 prod gate on the prod-shape cluster (8+ cores, dedicated). This is the second of the two M12 exit criteria.
|
|
|
+
|
|
|
+### 7.2 Smoke updates needed (carry-over from M11-conditional)
|
|
|
+
|
|
|
+The 2-min re-verification on 2026-06-16 14:48 EDT showed the M11 smoke's rate metric (`ba_ingestd_alerts_received_total`) does not assert on **publish success**. M12 W5 fixes this:
|
|
|
+
|
|
|
+```python
|
|
|
+# scripts/m11_smoke.py — add a publish-success assertion
|
|
|
+def assert_grpc_publish_ok(target_rate, tolerance, window_seconds=30):
|
|
|
+ """Rate of successful NATS publishes, must be ≥ target * (1 - tolerance)."""
|
|
|
+ q = f'rate(ba_ingestd_nats_publish_total{{service="ingestd",result="ok"}}[{window_seconds}s])'
|
|
|
+ rate = float(prom_query(q))
|
|
|
+ if rate < target_rate * (1 - tolerance):
|
|
|
+ fail_(f"NATS publish rate {rate:.0f}/s is below target {target_rate}/s")
|
|
|
+ pass_(f"NATS publish rate {rate:.0f}/s ≥ {target_rate}/s")
|
|
|
+```
|
|
|
+
|
|
|
+(Assumes the `ba_ingestd_nats_publish_total` counter exists in ingestd with a `result` label. If not, it's added as part of W1 — see W1 §3.4.)
|
|
|
+
|
|
|
+### 7.3 Test setup
|
|
|
+
|
|
|
+- 2 `loadgen` Deployments targeting `ingestd:9090` (gRPC)
|
|
|
+- Combined rate target: 10k/s
|
|
|
+- Duration: 10 min
|
|
|
+- p99 ack latency assertion: ≤ 50ms
|
|
|
+- Receive rate assertion: ≥ 9k/s (±10%)
|
|
|
+- **NEW: Publish rate assertion: ≥ 9k/s (±10%)** ← the fix
|
|
|
+- DLQ assertion: = 0
|
|
|
+- Backpressure step: 16 streams × 1k/s = 16k/s, all rate-limited acks honored
|
|
|
+
|
|
|
+### 7.4 Exit criteria for W5
|
|
|
+
|
|
|
+- 3/3 green runs at 10k/s on the K8s cluster
|
|
|
+- 20/20 soak samples ≥ 9k/s receive rate
|
|
|
+- 20/20 soak samples ≥ 9k/s publish rate ← the new assertion
|
|
|
+- p99 ≤ 50ms in all samples
|
|
|
+- DLQ = 0 throughout
|
|
|
+- Backpressure step: 16/16 streams receive rate-limited acks when over per-source cap
|
|
|
+
|
|
|
+---
|
|
|
+
|
|
|
+## 8. Workstream W6 — CI/CD + observability for K8s
|
|
|
+
|
|
|
+### 8.1 CI/CD (GitHub Actions or GitLab CI)
|
|
|
+
|
|
|
+- **Build:** on PR, `docker build` for each service, push to `registry.techno-world.net/lrosales/broad-announce-{service}:<sha>`
|
|
|
+- **Test:** `go test ./...` for each service; smoke against ephemeral docker-compose (existing M10/M11 harnesses)
|
|
|
+- **Deploy:** on merge to main, GitOps (Argo CD) picks up the new image tag and applies via Helm
|
|
|
+
|
|
|
+### 8.2 Observability
|
|
|
+
|
|
|
+- **Prometheus Operator** + `ServiceMonitor` resources for each service
|
|
|
+- **Grafana dashboards** carry over from docker-compose; the PromQL queries are unchanged
|
|
|
+- **Loki** for log aggregation (already in stack)
|
|
|
+- **Alertmanager** rules: NATS broker count, ingestd panic count, deliverd queue depth
|
|
|
+
|
|
|
+### 8.3 Exit criteria for W6
|
|
|
+
|
|
|
+- PR triggers CI build + test
|
|
|
+- Merge to main triggers GitOps deploy to staging
|
|
|
+- Staging smoke (M11 2-min @ 6k) passes automatically
|
|
|
+- Dashboards in Grafana show all 5 service tiers (ingest, broker, route, deliver, archive)
|
|
|
+
|
|
|
+---
|
|
|
+
|
|
|
+## 9. Sequencing & parallelism
|
|
|
+
|
|
|
+```
|
|
|
+W1 ─────────────────┐
|
|
|
+ ├──▶ W2 (NATS cluster)
|
|
|
+ │ │
|
|
|
+ │ ├──▶ W3 (Helm) ──┐
|
|
|
+ │ │ │
|
|
|
+ │ ▼ ▼
|
|
|
+ │ W4 (50k/s bench) W5 (M11 prod gate)
|
|
|
+ │ │ │
|
|
|
+ │ └──────┬───────────┘
|
|
|
+ │ ▼
|
|
|
+ │ W6 (CI/CD + observability)
|
|
|
+ │
|
|
|
+ (W1 must land first; rest can parallelize with W3 as a fan-out)
|
|
|
+```
|
|
|
+
|
|
|
+Estimated wall time (with one full-time engineer):
|
|
|
+- W1: 2-3 weeks (manifests + stateful services are fiddly)
|
|
|
+- W2: 1 week (JetStream config + chaos test)
|
|
|
+- W3: 1 week (Helm chart extraction)
|
|
|
+- W4: 3 days (smoke adapted from M10-bench)
|
|
|
+- W5: 1 week (smoke update for publish assertion + full 5-step verification)
|
|
|
+- W6: 1 week (CI/CD + observability)
|
|
|
+- **Total: ~6-8 weeks**
|
|
|
+
|
|
|
+---
|
|
|
+
|
|
|
+## 10. What M12 is *not*
|
|
|
+
|
|
|
+(Scope boundaries to prevent creep)
|
|
|
+
|
|
|
+- **Not a rewrite of the data plane.** ingestd, routerd, deliverd-* keep their Go code unchanged. M12 is deployment + NATS topology.
|
|
|
+- **Not a regional/multi-region story.** v2.0 is single-region. Multi-region is v3.
|
|
|
+- **Not an IaC migration.** Terraform / Pulumi for cloud resources is a separate milestone (M13?).
|
|
|
+- **Not a security hardening pass.** mTLS, secrets management, network policies are M11.5 (still pending).
|
|
|
+- **Not a release of the M11 conditional ship issue.** M11 NATS resource limit fix must land in a hotfix before M12 W5 runs. Otherwise the smoke's new publish-success assertion will fail on a single-broker configuration.
|
|
|
+
|
|
|
+---
|
|
|
+
|
|
|
+## 11. Open questions to resolve before W1 starts
|
|
|
+
|
|
|
+1. **Cloud or on-prem K8s?** EKS/GKE/AKS/on-prem (kubeadm, k3s, Rancher)? Determines service-mesh and storage-class choices.
|
|
|
+2. **GitOps tool:** Argo CD, Flux, or plain `kubectl apply` from CI? Argo is the most common; Flux is lighter-weight.
|
|
|
+3. **NATS deployment mode:** official `nats-io/nats` Helm chart, the NATS operator, or custom StatefulSet? Custom gives more control; the operator gives rolling-upgrade + observability for free. **Recommendation:** NATS operator (jetstream + cluster + chaos-tested).
|
|
|
+4. **Stateful storage:** cloud-provider CSI driver (EBS, GCE PD, Azure Disk) or on-prem (NFS, Ceph, Longhorn)? Determines PVC templates.
|
|
|
+5. **TLS / mTLS in K8s:** cert-manager for in-cluster, or pin certs as Secrets? M11.5 will formalize this; M12 just needs to not paint us into a corner.
|
|
|
+6. **Where does the K8s cluster live?** parres is the dev box (docker-compose). Is there a staging K8s cluster already, or do we need to provision one?
|
|
|
+7. **K8s version target:** 1.28+ (covers everything we need). Pin in the chart.
|
|
|
+8. **CI provider:** GitHub Actions, GitLab CI, or Jenkins? Already have a CI somewhere?
|
|
|
+
|
|
|
+---
|
|
|
+
|
|
|
+## 12. Prerequisites (M12 cannot start until these are done)
|
|
|
+
|
|
|
+1. **M11 NATS resource limit issue resolved.** The 2-min re-verification on 2026-06-16 14:48 EDT showed single-broker NATS fails under sustained load. Either:
|
|
|
+ - (a) Tune NATS limits on the single broker to a level that proves the publish path is healthy, OR
|
|
|
+ - (b) Skip ahead to M12 W2 (multi-broker) and use the cluster to clear M11 fully
|
|
|
+ - M12 W5 cannot pass the new publish-success assertion until this is resolved.
|
|
|
+2. **M11 loadgen exit-code issue resolved** (currently deferred). The full 5-step smoke needs a green backpressure step.
|
|
|
+3. **K8s cluster provisioned** (parres can host minikube/k3s for dev; staging cluster for CI; prod cluster for the M12 ceiling bench).
|
|
|
+4. **Container registry** (`registry.techno-world.net/lrosales/...`) ready for CI-built images.
|
|
|
+5. **Push current M11 work to git3** so the team can see the conditional-ship state and the M12 plan.
|
|
|
+
|
|
|
+---
|
|
|
+
|
|
|
+## 13. Definition of done
|
|
|
+
|
|
|
+- [ ] W1: all 14 services have K8s manifests; `kubectl apply -k deploy/k8s/base/` brings them up
|
|
|
+- [ ] W2: 3-broker NATS JetStream cluster, R=3 streams, chaos test passes
|
|
|
+- [ ] W3: Helm chart installs + upgrades cleanly; values per environment
|
|
|
+- [ ] W4: 5/5 green runs at 50k/s, router p99 ≤ 50ms, DLQ=0
|
|
|
+- [ ] W5: 3/3 green runs at 10k/s, 20/20 soak samples green, **publish-success assertion passes**, full 5-step smoke green
|
|
|
+- [ ] W6: CI builds + tests on PR; GitOps deploys on merge; Prometheus + Grafana cover all tiers
|
|
|
+- [ ] SPEC.md M12 row updated to **✅ shipped YYYY-MM-DD**
|
|
|
+- [ ] `M12_VERIFICATION.md` published with bench evidence + chaos test results
|
|
|
+- [ ] `deploy/k8s/README.md` quickstart written
|
|
|
+
|
|
|
+---
|
|
|
+
|
|
|
+## 14. Risks & mitigations
|
|
|
+
|
|
|
+| Risk | Likelihood | Impact | Mitigation |
|
|
|
+|---|---|---|---|
|
|
|
+| NATS operator / Helm chart is immature | Low | Medium (deployment friction) | Fall back to custom StatefulSet (W1 template) |
|
|
|
+| Single-region outage takes down the cluster | Medium | High (revenue) | Multi-AZ nodes; PodDisruptionBudgets; M3 (multi-region) is the long-term fix |
|
|
|
+| Helm values drift between dev / staging / prod | Medium | Medium (config bugs) | Single `values.yaml` with per-env overlays; Kustomize on top if needed |
|
|
|
+| NATS cluster resource limits still hit at 50k/s | Low | High (M12 ceiling fails) | W4 finds the wall early; scale brokers to 5 if needed (R=5) |
|
|
|
+| StatefulSet PVC migration in production | High | High (data loss) | Use Velero for backup; test restore in staging; never delete PVCs without backup |
|
|
|
+| Cost: K8s cluster is more expensive than docker-compose | Certain | Low | Operational benefits outweigh the cost; capacity planning in W3 |
|
|
|
+
|
|
|
+---
|
|
|
+
|
|
|
+**Next step:** resolve open questions (§11) and the M11 NATS prerequisite (#1 in §12). W1 cannot start until both are settled.
|
|
|
+
|
|
|
+---
|
|
|
+
|
|
|
+## Appendix A — Push to git3 (current M11 work)
|
|
|
+
|
|
|
+The local repo on the workspace (`/root/.openclaw/workspace/broad-announce/`) is 5 commits ahead of origin/master on git3, but origin has 6 commits not in local (the M11 hotfixes shipped from a different machine). The histories diverged at `09f0d54`.
|
|
|
+
|
|
|
+To push the M11 work to git3, three approaches:
|
|
|
+
|
|
|
+### A1. Merge origin/master into local, then push (recommended)
|
|
|
+
|
|
|
+```bash
|
|
|
+cd /root/.openclaw/workspace/broad-announce
|
|
|
+git fetch origin master # already done
|
|
|
+git merge origin/master # creates a merge commit; preserves both histories
|
|
|
+# resolve any conflicts (likely in M11 smoke or loadgen code, since both sides edited)
|
|
|
+git push origin master # fast-forward or merge push
|
|
|
+```
|
|
|
+
|
|
|
+Pros: safest, no history rewrite, both sets of commits visible.
|
|
|
+Cons: creates a merge commit; the graph has a "diamond" shape.
|
|
|
+
|
|
|
+### A2. Rebase local onto origin, then force-push
|
|
|
+
|
|
|
+```bash
|
|
|
+cd /root/.openclaw/workspace/broad-announce
|
|
|
+git fetch origin master # already done
|
|
|
+git rebase origin/master # replays 5 local commits on top of origin's 6
|
|
|
+# resolve any conflicts
|
|
|
+git push --force-with-lease origin master # rewrites origin's master
|
|
|
+```
|
|
|
+
|
|
|
+Pros: clean linear history.
|
|
|
+Cons: rewrites the commits the user (or their other machine) already pushed. If those commits are referenced anywhere (PRs, CI runs, tags), those references break.
|
|
|
+
|
|
|
+### A3. Drop the local M11 work, fetch origin, and start fresh
|
|
|
+
|
|
|
+```bash
|
|
|
+cd /root/.openclaw/workspace/broad-announce
|
|
|
+git fetch origin master
|
|
|
+git reset --hard origin/master # discards local 5 commits
|
|
|
+```
|
|
|
+
|
|
|
+Pros: simple, no conflicts.
|
|
|
+Cons: **loses the local M11 work entirely** (the conditional ship, the loadgen fix, the M11 smoke threshold). Not recommended unless the user is sure the local work is duplicated on origin.
|
|
|
+
|
|
|
+**Recommended: A1 (merge).** The conditional-ship commit (`d8008bc`) is a real change to `SPEC.md` and `M11_VERIFICATION.md` that should be on origin. The loadgen fix (`39907d1`) is also worth shipping. Merging preserves both.
|
|
|
+
|
