Почетна Преглед Помоћ
Регистрација Пријавите се
lrosales
/
broad-announce
1
0
Креирај огранак 0
Датотеке Дискусије 0 Захтеви за спајање 0 Вики
Дрво: 57f838089e
Гране Ознаке
broad-announce
/
M12_PLAN.md

M12_PLAN.md 23 KB
Историја Датотека

M12 — Detailed Implementation Plan

Status: planning (post-M11-conditional, blocked on NATS issue) Target: M12 exit criterion from SPEC §19 Goal: Move from single-broker docker-compose to K8s-deployed multi-broker NATS, sustaining 50k/s end-to-end with M11 prod gate (10k/s sustained 10 min, p99 ≤ 50ms, DLQ=0) on a single ingestd instance.

0. Recap — what M12 must prove

Milestone Exit criterion How it's measured
M11 prod gate (carried into M12) internal Go service pushes ≥ 10k alerts/sec on one gRPC stream, p99 server-side Ack ≤ 50ms, DLQ=0, sustained 10 min m11_smoke.py on K8s cluster, asserts on receive rate and publish success
M12 ceiling (new) 50k/s end-to-end through multi-broker NATS + multi-ingestd, p99 router latency ≤ 50ms, no broker backpressure m10_bench_smoke.py (adapted) on K8s, delivery stubbed
M12 cluster (new) NATS cluster survives single-broker loss without message loss or sustained publish failure chaos test: kill one NATS pod mid-soak, verify p99 stays under 100ms and no DLQ

Why M12 now (not later): The 2-min re-verification on 2026-06-16 14:48 EDT exposed the single-NATS broker as the system bottleneck. NATS logged [ERR] JetStream resource limits exceeded for server every 10s under sustained ~6k/s load, and the receive rate collapsed to 0/s in 30s. Single-broker NATS on docker-compose cannot deliver v1 capacity (5k/s sustained per the SPEC, design ceiling 50k/s). M10-bench proved the router ceiling at 50k/s with delivery stubbed, but the broker was the implicit weak link — M12 surfaces and fixes it.

1. Workstream overview

┌──────────────────────────┐    ┌──────────────────────────┐    ┌──────────────────────────┐
│  W1: K8s manifests       │    │  W2: NATS JetStream      │    │  W3: Helm chart +        │
│  (Deployments, Services, │───▶│  cluster (3+ brokers,    │───▶│  argocd / flux /        │
│   ConfigMaps, Secrets)   │    │   R3, clustering)        │    │  plain kubectl           │
└──────────────────────────┘    └──────────────┬───────────┘    └──────────────┬───────────┘
                                               │                               │
                                               ▼                               ▼
                                     ┌──────────────────────────┐    ┌──────────────────────────┐
                                     │  W4: 50k/s ceiling bench │    │  W5: M11 prod gate on K8s│
                                     │  (M10-bench equivalent)  │    │  (10k/s × 10 min)        │
                                     │  delivery stubbed        │    │  full smoke (5 steps)    │
                                     └──────────────────────────┘    └──────────────────────────┘
                                               │                               │
                                               ▼                               ▼
                                     ┌──────────────────────────────────────────────────┐
                                     │  W6: CI/CD + observability for K8s              │
                                     │  (GitHub Actions / GitLab CI build+push,        │
                                     │   Prometheus Operator, Grafana dashboards)       │
                                     └──────────────────────────────────────────────────┘

Six workstreams, each ends with passing evidence before the next starts.

2. The four layers of risk (and what each WS defuses)

Risk Why it's scary How we defuses it
Single NATS broker is the bottleneck (post-M11-conditional finding) M11 10-min soak was green only because the rate metric counts gRPC receive, not publish success. Real-world delivery is broken under sustained load. W2: NATS cluster (3+ brokers, R=3 JetStream streams) raises the limit ceiling by ~3x. W4: prove the cluster ceiling at 50k/s.
K8s deployment is operationally complex K8s has a learning curve; mistakes (wrong resource limits, missing healthchecks, bad ConfigMap wiring) cause silent failures W1: manifests mirror docker-compose service-by-service; W3: Helm chart encapsulates complexity; W6: GitOps pipeline means config changes are reviewable in PRs.
Stateful workloads in K8s NATS, Postgres, ClickHouse, Redis all need persistent volumes and ordered shutdown. Compose just restarts; K8s needs StatefulSets + PodDisruptionBudgets. W1: use StatefulSets for stateful services, Deployments for stateless (ingestd, routerd, deliverd-*). W3: Helm chart includes PVC templates + PDBs.
Migrating from docker-compose to K8s in production Compose works; K8s is new. Risk of breaking the dev loop. W1 keeps docker-compose as the dev story (unchanged), K8s is the deploy target only. M11+M12 work continues against compose.

3. Workstream W1 — K8s manifests

3.1 Repo additions

deploy/k8s/
  base/
    namespace.yaml                # broad-announce namespace
    configmap.yaml                # BA_* env vars
    secrets.yaml.example          # fcm-credentials, telegram-bot-token, etc. (gitignored real one)
  postgres/
    statefulset.yaml              # 1 replica, 10Gi PVC, healthcheck
    service.yaml
    pdb.yaml                      # minAvailable: 1
  nats/                           # see W2 for cluster specifics
    statefulset.yaml
    headless-service.yaml
    configmap.yaml
    pdb.yaml
  redis/
    statefulset.yaml
    service.yaml
  clickhouse/
    statefulset.yaml
    service.yaml
  emqx/
    statefulset.yaml              # 1 replica (single-node OK for v2.0)
    service.yaml
  ingestd/
    deployment.yaml               # 2 replicas, max_inflight=256, BA_INGESTD_RATE_LIMIT_PER_SOURCE=20000
    service.yaml                  # ClusterIP, port 8800 + 9090 (gRPC)
    hpa.yaml                      # autoscale on CPU + custom metric (alerts_received rate)
  routerd/
    deployment.yaml
    service.yaml
  deliverd-fcm/
    deployment.yaml
    service.yaml
  deliverd-telegram/
    deployment.yaml
    service.yaml
  admind/
    deployment.yaml
    service.yaml
  archiverd/
    deployment.yaml
    service.yaml
  loadgen/                        # kept as Deployments, scaled up for the 50k/s bench
    deployment.yaml

3.2 Conversion rules (compose → K8s)

compose K8s
service: ingestd Deployment named ingestd, Service ClusterIP, port 8800+9090
environment: block ConfigMap mounted as envFrom
volumes: ["/data"] StatefulSet + volumeClaimTemplates with PVCs
healthcheck: test: ... livenessProbe + readinessProbe (exec or httpGet)
depends_on: [postgres] Same effect via init containers (wait for postgres:5432) or K8s-native service discovery with retries in app code
ports: ["8800:8800"] Service (ClusterIP) + optional Ingress for external
deploy.resources.limits resources.limits in container spec
profiles (loadgen-grpc) Deployment in deploy/k8s/overlays/loadgen/ (Kustomize) — off by default in base

3.3 Exit criteria for W1

  • All services from docker-compose.yml have K8s manifests in deploy/k8s/base/
  • kubectl apply -k deploy/k8s/base/ brings the stack up
  • kubectl get pods -n broad-announce shows all services Running with Ready 1/1 (or appropriate for stateful)
  • Service-to-service DNS works (e.g., ingestd can reach nats:4222)
  • The M11 10-min soak passes against the K8s deployment (regression check: nothing broke in the move)

4. Workstream W2 — NATS JetStream cluster

4.1 Why this is its own workstream

The 2026-06-16 14:48 EDT finding shows single-broker NATS hits resource limits under sustained ~6k/s. Multi-broker is M12's core fix, not a deployment detail.

4.2 Cluster topology

  • 3 brokers (nats-0, nats-1, nats-2) on separate nodes (anti-affinity)
  • JetStream enabled on all 3
  • Stream ALERTS with replicas=3, storage=File, max_age=24h
  • Subjects: alerts.<company_id>.<source_id> (per ARCHITECTURE.md §6)
  • Consumer ROUTER for routerd, replicas=3, ack_wait=30s, max_deliver=5
  • Consumer DELIVERY_FCM / DELIVERY_TELEGRAM for deliverd-*, replicas=3

4.3 Config

# deploy/k8s/base/nats/configmap.yaml
cluster:
  name: broad-announce-nats
  listen: 0.0.0.0:6222
  routes:
    - nats-broad-announce-nats-0.nats-headless:6222
    - nats-broad-announce-nats-1.nats-headless:6222
    - nats-broad-announce-nats-2.nats-headless:6222
jetstream:
  store_dir: /data/jetstream
  max_memory_store: 2Gi
  max_file_store: 100Gi

4.4 Capacity target (post-cluster)

Stream Per-broker Cluster (R=3)
ALERTS throughput ~20k/s ~50-60k/s (3 brokers share load)
ALERTS storage 100Gi 300Gi (file-backed)
ALERTS max memory 2Gi 6Gi

This raises the resource-limit ceiling by ~3x. The 50k/s ceiling bench (W4) verifies it.

4.5 Exit criteria for W2

  • 3-broker NATS cluster running in K8s
  • nats stream info ALERTS shows replicas=3, leader distributed
  • Subject routing works: publish to alerts.acme-001 on any broker lands in the stream
  • Single-broker loss does not cause message loss (chaos test)

5. Workstream W3 — Helm chart + GitOps

5.1 Why a Helm chart

  • Parameterization (replica counts, image tags, env vars) per environment
  • Encapsulates the manifest sprawl from W1
  • Standard for K8s packaging

5.2 Repo additions

deploy/helm/
  broad-announce/
    Chart.yaml
    values.yaml                  # defaults: dev-shape cluster, single NATS
    values.prod.yaml             # prod-shape: 3 NATS, multi-ingestd, higher resource limits
    templates/
      _helpers.tpl
      namespace.yaml
      configmap.yaml
      secrets.yaml
      postgres/
      nats/
      redis/
      clickhouse/
      emqx/
      ingestd/
      routerd/
      deliverd-fcm/
      deliverd-telegram/
      admind/
      archiverd/

5.3 Values structure

# values.prod.yaml
global:
  imageRegistry: registry.techno-world.net/lrosales
  imageTag: "M12-W6"

nats:
  cluster:
    enabled: true
    replicas: 3
    storage: 100Gi
  jetstream:
    maxMemory: 2Gi
    maxFile: 100Gi

ingestd:
  replicas: 3
  resources:
    requests: {cpu: "500m", memory: "512Mi"}
    limits:   {cpu: "2",    memory: "1Gi"}
  env:
    BA_INGESTD_RATE_LIMIT_PER_SOURCE: "20000"
    BA_INGESTD_MAX_INFLIGHT: "256"

routerd:
  replicas: 2
  ...

5.4 GitOps (optional but recommended)

  • Argo CD or Flux watches deploy/helm/broad-announce/
  • PRs to values.prod.yaml trigger preview environments
  • Auto-sync to prod after approval

5.5 Exit criteria for W3

  • helm install broad-announce deploy/helm/broad-announce/ -f values.prod.yaml -n broad-announce brings up the prod-shape cluster
  • helm upgrade --reuse-values works idempotently
  • helm template produces valid manifests that pass kubeconform (or kubectl apply --dry-run=server)

6. Workstream W4 — 50k/s ceiling bench (M10-bench on K8s)

6.1 What this proves

The M10-bench in M10_BENCH_VERIFICATION.md proved the router ceiling at 50k/s on a single NATS broker with delivery stubbed. W4 re-proves it on the 3-broker cluster with stubbed delivery, and is the gate for M12.

6.2 Test setup

  • 2 loadgen Deployments (one per company/api-key, same as M11)
  • Each loadgen targets the local K8s ingestd via the ingestd Service
  • Combined rate target: 50k/s
  • Duration: 5 min
  • p99 router latency assertion: ≤ 50ms
  • DLQ assertion: = 0
  • Broker queue depth: informational

6.3 Exit criteria for W4

  • 5/5 green runs at 50k/s on the K8s cluster
  • Router p99 ≤ 50ms in all 5 runs
  • DLQ = 0 throughout
  • NATS cluster resource utilization: each broker < 80% CPU, < 80% memory

7. Workstream W5 — M11 prod gate on K8s (10k/s × 10 min, full smoke)

7.1 What this proves

M11 dev-playground gate passed on parres (4 cores, shared). M12 proves the M11 prod gate on the prod-shape cluster (8+ cores, dedicated). This is the second of the two M12 exit criteria.

7.2 Smoke updates needed (carry-over from M11-conditional)

The 2-min re-verification on 2026-06-16 14:48 EDT showed the M11 smoke's rate metric (ba_ingestd_alerts_received_total) does not assert on publish success. M12 W5 fixes this:

# scripts/m11_smoke.py — add a publish-success assertion
def assert_grpc_publish_ok(target_rate, tolerance, window_seconds=30):
    """Rate of successful NATS publishes, must be ≥ target * (1 - tolerance)."""
    q = f'rate(ba_ingestd_nats_publish_total{{service="ingestd",result="ok"}}[{window_seconds}s])'
    rate = float(prom_query(q))
    if rate < target_rate * (1 - tolerance):
        fail_(f"NATS publish rate {rate:.0f}/s is below target {target_rate}/s")
    pass_(f"NATS publish rate {rate:.0f}/s ≥ {target_rate}/s")

(Assumes the ba_ingestd_nats_publish_total counter exists in ingestd with a result label. If not, it's added as part of W1 — see W1 §3.4.)

7.3 Test setup

  • 2 loadgen Deployments targeting ingestd:9090 (gRPC)
  • Combined rate target: 10k/s
  • Duration: 10 min
  • p99 ack latency assertion: ≤ 50ms
  • Receive rate assertion: ≥ 9k/s (±10%)
  • NEW: Publish rate assertion: ≥ 9k/s (±10%) ← the fix
  • DLQ assertion: = 0
  • Backpressure step: 16 streams × 1k/s = 16k/s, all rate-limited acks honored

7.4 Exit criteria for W5

  • 3/3 green runs at 10k/s on the K8s cluster
  • 20/20 soak samples ≥ 9k/s receive rate
  • 20/20 soak samples ≥ 9k/s publish rate ← the new assertion
  • p99 ≤ 50ms in all samples
  • DLQ = 0 throughout
  • Backpressure step: 16/16 streams receive rate-limited acks when over per-source cap

8. Workstream W6 — CI/CD + observability for K8s

8.1 CI/CD (GitHub Actions or GitLab CI)

  • Build: on PR, docker build for each service, push to registry.techno-world.net/lrosales/broad-announce-{service}:<sha>
  • Test: go test ./... for each service; smoke against ephemeral docker-compose (existing M10/M11 harnesses)
  • Deploy: on merge to main, GitOps (Argo CD) picks up the new image tag and applies via Helm

8.2 Observability

  • Prometheus Operator + ServiceMonitor resources for each service
  • Grafana dashboards carry over from docker-compose; the PromQL queries are unchanged
  • Loki for log aggregation (already in stack)
  • Alertmanager rules: NATS broker count, ingestd panic count, deliverd queue depth

8.3 Exit criteria for W6

  • PR triggers CI build + test
  • Merge to main triggers GitOps deploy to staging
  • Staging smoke (M11 2-min @ 6k) passes automatically
  • Dashboards in Grafana show all 5 service tiers (ingest, broker, route, deliver, archive)

9. Sequencing & parallelism

W1 ─────────────────┐
                    ├──▶ W2 (NATS cluster)
                    │       │
                    │       ├──▶ W3 (Helm) ──┐
                    │       │                  │
                    │       ▼                  ▼
                    │   W4 (50k/s bench)   W5 (M11 prod gate)
                    │       │                  │
                    │       └──────┬───────────┘
                    │              ▼
                    │         W6 (CI/CD + observability)
                    │
                    (W1 must land first; rest can parallelize with W3 as a fan-out)

Estimated wall time (with one full-time engineer):

  • W1: 2-3 weeks (manifests + stateful services are fiddly)
  • W2: 1 week (JetStream config + chaos test)
  • W3: 1 week (Helm chart extraction)
  • W4: 3 days (smoke adapted from M10-bench)
  • W5: 1 week (smoke update for publish assertion + full 5-step verification)
  • W6: 1 week (CI/CD + observability)
  • Total: ~6-8 weeks

10. What M12 is not

(Scope boundaries to prevent creep)

  • Not a rewrite of the data plane. ingestd, routerd, deliverd-* keep their Go code unchanged. M12 is deployment + NATS topology.
  • Not a regional/multi-region story. v2.0 is single-region. Multi-region is v3.
  • Not an IaC migration. Terraform / Pulumi for cloud resources is a separate milestone (M13?).
  • Not a security hardening pass. mTLS, secrets management, network policies are M11.5 (still pending).
  • Not a release of the M11 conditional ship issue. M11 NATS resource limit fix must land in a hotfix before M12 W5 runs. Otherwise the smoke's new publish-success assertion will fail on a single-broker configuration.

11. Open questions to resolve before W1 starts

  1. Cloud or on-prem K8s? EKS/GKE/AKS/on-prem (kubeadm, k3s, Rancher)? Determines service-mesh and storage-class choices.
  2. GitOps tool: Argo CD, Flux, or plain kubectl apply from CI? Argo is the most common; Flux is lighter-weight.
  3. NATS deployment mode: official nats-io/nats Helm chart, the NATS operator, or custom StatefulSet? Custom gives more control; the operator gives rolling-upgrade + observability for free. Recommendation: NATS operator (jetstream + cluster + chaos-tested).
  4. Stateful storage: cloud-provider CSI driver (EBS, GCE PD, Azure Disk) or on-prem (NFS, Ceph, Longhorn)? Determines PVC templates.
  5. TLS / mTLS in K8s: cert-manager for in-cluster, or pin certs as Secrets? M11.5 will formalize this; M12 just needs to not paint us into a corner.
  6. Where does the K8s cluster live? parres is the dev box (docker-compose). Is there a staging K8s cluster already, or do we need to provision one?
  7. K8s version target: 1.28+ (covers everything we need). Pin in the chart.
  8. CI provider: GitHub Actions, GitLab CI, or Jenkins? Already have a CI somewhere?

12. Prerequisites (M12 cannot start until these are done)

  1. M11 NATS resource limit issue resolved. The 2-min re-verification on 2026-06-16 14:48 EDT showed single-broker NATS fails under sustained load. Either:
    • (a) Tune NATS limits on the single broker to a level that proves the publish path is healthy, OR
    • (b) Skip ahead to M12 W2 (multi-broker) and use the cluster to clear M11 fully
    • M12 W5 cannot pass the new publish-success assertion until this is resolved.
  2. M11 loadgen exit-code issue resolved (currently deferred). The full 5-step smoke needs a green backpressure step.
  3. K8s cluster provisioned (parres can host minikube/k3s for dev; staging cluster for CI; prod cluster for the M12 ceiling bench).
  4. Container registry (registry.techno-world.net/lrosales/...) ready for CI-built images.
  5. Push current M11 work to git3 so the team can see the conditional-ship state and the M12 plan.

13. Definition of done

  • W1: all 14 services have K8s manifests; kubectl apply -k deploy/k8s/base/ brings them up
  • W2: 3-broker NATS JetStream cluster, R=3 streams, chaos test passes
  • W3: Helm chart installs + upgrades cleanly; values per environment
  • W4: 5/5 green runs at 50k/s, router p99 ≤ 50ms, DLQ=0
  • W5: 3/3 green runs at 10k/s, 20/20 soak samples green, publish-success assertion passes, full 5-step smoke green
  • W6: CI builds + tests on PR; GitOps deploys on merge; Prometheus + Grafana cover all tiers
  • SPEC.md M12 row updated to ✅ shipped YYYY-MM-DD
  • M12_VERIFICATION.md published with bench evidence + chaos test results
  • deploy/k8s/README.md quickstart written

14. Risks & mitigations

Risk Likelihood Impact Mitigation
NATS operator / Helm chart is immature Low Medium (deployment friction) Fall back to custom StatefulSet (W1 template)
Single-region outage takes down the cluster Medium High (revenue) Multi-AZ nodes; PodDisruptionBudgets; M3 (multi-region) is the long-term fix
Helm values drift between dev / staging / prod Medium Medium (config bugs) Single values.yaml with per-env overlays; Kustomize on top if needed
NATS cluster resource limits still hit at 50k/s Low High (M12 ceiling fails) W4 finds the wall early; scale brokers to 5 if needed (R=5)
StatefulSet PVC migration in production High High (data loss) Use Velero for backup; test restore in staging; never delete PVCs without backup
Cost: K8s cluster is more expensive than docker-compose Certain Low Operational benefits outweigh the cost; capacity planning in W3

Next step: resolve open questions (§11) and the M11 NATS prerequisite (#1 in §12). W1 cannot start until both are settled.

Appendix A — Push to git3 (current M11 work)

The local repo on the workspace (/root/.openclaw/workspace/broad-announce/) is 5 commits ahead of origin/master on git3, but origin has 6 commits not in local (the M11 hotfixes shipped from a different machine). The histories diverged at 09f0d54.

To push the M11 work to git3, three approaches:

A1. Merge origin/master into local, then push (recommended)

cd /root/.openclaw/workspace/broad-announce
git fetch origin master                                   # already done
git merge origin/master                                   # creates a merge commit; preserves both histories
# resolve any conflicts (likely in M11 smoke or loadgen code, since both sides edited)
git push origin master                                    # fast-forward or merge push

Pros: safest, no history rewrite, both sets of commits visible. Cons: creates a merge commit; the graph has a "diamond" shape.

A2. Rebase local onto origin, then force-push

cd /root/.openclaw/workspace/broad-announce
git fetch origin master                                   # already done
git rebase origin/master                                  # replays 5 local commits on top of origin's 6
# resolve any conflicts
git push --force-with-lease origin master                 # rewrites origin's master

Pros: clean linear history. Cons: rewrites the commits the user (or their other machine) already pushed. If those commits are referenced anywhere (PRs, CI runs, tags), those references break.

A3. Drop the local M11 work, fetch origin, and start fresh

cd /root/.openclaw/workspace/broad-announce
git fetch origin master
git reset --hard origin/master                            # discards local 5 commits

Pros: simple, no conflicts. Cons: loses the local M11 work entirely (the conditional ship, the loadgen fix, the M11 smoke threshold). Not recommended unless the user is sure the local work is duplicated on origin.

Recommended: A1 (merge). The conditional-ship commit (d8008bc) is a real change to SPEC.md and M11_VERIFICATION.md that should be on origin. The loadgen fix (39907d1) is also worth shipping. Merging preserves both.