Pārlūkot izejas kodu

feat(spec): loadgen traffic generators + M10 split into M10 + M10-bench

- SPEC §20: add loadgen/ to repo layout
- SPEC §21 (new): traffic generators. Four Go binaries
  (loadgen-http/ws/mqtt/grpc), shared CLI surface, five built-in
  profiles (normal/burst/chatty-iot/pager-storm/stress),
  distributed cluster mode via NATS KV, its own Go module so
  it can't end up in service images
- SPEC §22: M10 split — M10 = in-cluster 5k/s via loadgen,
  M10-bench = 50k/s against broker+router with delivery stubbed
  (option C from the discussion)
- README: add loadgen/ to layout; mention gRPC in ingest line
- PROMPT: log option C and the loadgen rationale
Luis Rosales 2 mēneši atpakaļ
vecāks
revīzija
16d2457b31
3 mainītis faili ar 150 papildinājumiem un 4 dzēšanām
  1. 11 0
      PROMPT.md
  2. 2 1
      README.md
  3. 137 3
      SPEC.md

+ 11 - 0
PROMPT.md

@@ -71,3 +71,14 @@
   `internal/grpcclient/`. Auth = API key in metadata + optional
   mTLS. Per-stream rate limit + 256 in-flight cap = natural
   backpressure.
+
+**2026-06-13 — M10 split: option C**
+- M10 (in-cluster, real) stays at **5k/s on docker-compose**.
+- Added **M10-bench**: 50k/s against the broker + router with
+  the delivery tier stubbed. Proves the ceiling without
+  committing to K8s. Green here means K8s is *implementable*,
+  not required.
+- Implication: we need traffic generators that can hit the
+  in-cluster path at 5k/s (M10) and the broker+router path at
+  50k/s (M10-bench) with the same source-protocol clients we
+  ship. The `loadgen/` tool is the new home for that.

+ 2 - 1
README.md

@@ -26,10 +26,11 @@ SPEC.md           — requirements, entities, severity, retention
 ARCHITECTURE.md   — diagrams, sequences, SLOs, capacity model
 PROMPT.md         — build log, decisions, open questions
 migrations/       — (coming) golang-migrate SQL files
-ingestd/          — (coming) HTTP/WS/MQTT ingest binary
+ingestd/          — (coming) HTTP/WS/MQTT/gRPC ingest binary
 routerd/          — (coming) recipient resolution binary
 deliverd/         — (coming) per-channel delivery workers
 admind/           — (coming) admin HTTP API + UI host
+loadgen/          — (coming) per-protocol traffic generators
 internal/         — (coming) shared Go packages
 deploy/           — (coming) prometheus, grafana, loki configs
 docker-compose.yml — (coming) one-shot local stack

+ 137 - 3
SPEC.md

@@ -504,7 +504,6 @@ internal Go service can publish ≥ 10k alerts/sec on one stream
 sustained 10 min, p99 server-side `Ack` latency ≤ 50ms.
 
 ## 20. Repo layout (planned)
-
 ```
 broad-announce/
 ├── SPEC.md                ← this file
@@ -518,6 +517,7 @@ broad-announce/
 ├── routerd/               ← binary: recipient expansion
 ├── deliverd/              ← binary: per-channel delivery workers
 ├── admind/                ← binary: admin HTTP API + (later) UI host
+├── loadgen/               ← (coming) traffic generators per protocol
 ├── internal/
 │   ├── alert/             ← Alert v1 type, validation
 │   ├── broker/            ← NATS wrapper
@@ -535,7 +535,140 @@ broad-announce/
     └── loki/
 ```
 
-## 21. Milestones
+## 21. Traffic generators (loadgen)
+
+Four small Go binaries in `loadgen/` — one per source protocol — that
+behave as **reference producers** of alerts. Same wire format, same
+auth, same client libs an internal service would use. Copy the
+binaries to as many hosts as you want, point them at a target, and
+they generate load.
+
+### Goals
+
+- **Realistic traffic** ("normal"): matches the shape of real-world
+  sources — bursty, dedupe-heavy, mix of severities, mix of
+  company_ids.
+- **Stress traffic** ("stress"): maximum sustained rate, no dedupe,
+  fat payloads, with fault-injection knobs (drop %, latency spikes,
+  random disconnects).
+- **Distributed by design**: any number of `loadgen` instances on
+  any number of hosts coordinate via a tiny shared state in
+  NATS JetStream KV (`loadgen.coordination`). No single host is
+  in charge.
+- **Self-contained**: no separate deploy, no extra observability
+  stack. Emit Prometheus metrics on `:9091`, structured logs to
+  stdout.
+
+### Binaries
+
+| binary | protocol | client lib | what it generates |
+|---|---|---|---|
+| `loadgen-http` | HTTP POST | `internal/httpclient/` | signed webhooks, configurable concurrency, N concurrent connections |
+| `loadgen-ws` | WebSocket | `internal/wsclient/` | one long-lived socket per instance, alert frames every X ms |
+| `loadgen-mqtt` | MQTT | `internal/mqttclient/` | QoS 1, per-company topic, configurable in-flight |
+| `loadgen-grpc` | gRPC bidi | `internal/grpcclient/` | one stream per instance, batched send with bounded in-flight |
+
+### CLI surface (same shape across all four)
+
+```bash
+loadgen-http \
+  --target https://broad-announce:8080 \
+  --api-key $SOURCE_API_KEY \
+  --mode normal | stress \
+  --rate 1000               # alerts/sec target per instance
+  --duration 10m \
+  --company-prefix acme-    # company_ids are "acme-001" .. "acme-100" cycled
+  --severities info:70,critical:5,inminent_colapse:1 \
+  --dedupe-ratio 0.30       # 30% of alerts share a dedupe_key within 60s
+  --payload-bytes 512       # avg data{} size
+  --concurrency 64          # concurrent HTTP connections
+  --drop-pct 0.0            # stress only: drop this % of sends
+  --latency-spike-ms 0      # stress only: inject X ms of extra sleep
+  --reconnect-every 0       # stress only: drop+reconnect every X seconds
+  --metrics :9091
+  --coordinator nats://nats:4222  # for distributed mode
+```
+
+### Traffic profiles (built-in)
+
+| profile | rate per instance | dedupe | severity mix | payload | use case |
+|---|---|---|---|---|---|
+| `normal` | 500/s | 30% | 70/25/4/1 | 512B | soak tests, capacity planning |
+| `burst` | peak 5k/s for 10s, idle 50s | 50% | 50/30/15/5 | 1KB | alert-storm resilience |
+| `chatty-iot` | 50/s × 1000 sources | 0% | 95/5/0/0 | 128B | IoT fleet size test |
+| `pager-storm` | 10k/s | 0% | 0/0/50/50 | 256B | inminent_colapse path |
+| `stress` | unlimited (CPU bound) | 0% | 25/25/25/25 | 4KB | find the breaking point |
+
+### Distributed mode
+
+Each instance advertises its rate in NATS KV. A coordinator (any one
+instance, elected) sums the rates and assigns per-instance targets
+to hit a **cluster-wide** rate. Use case: spin up 10 hosts, each
+running 4 instances of `loadgen-grpc`, target 50k/s cluster-wide.
+
+```
+loadgen-grpc --mode stress --cluster-target 50000 --cluster-id bench-2026-06-13
+```
+
+Coordination is **best-effort** (no leader lock, no consensus) — if
+the coordinator drops, the next instance takes over. Over- and
+under-shoot within ±5% is acceptable for soak tests.
+
+### Metrics
+
+- `loadgen_alerts_sent_total{profile,instance,mode}` counter
+- `loadgen_alerts_failed_total{reason}` counter (drop, timeout, 4xx, 5xx)
+- `loadgen_alert_send_latency_seconds_bucket{mode}` histogram
+- `loadgen_dedupe_hits_total` counter (computed server-side; we just
+  read it from `Ack` and increment locally)
+- `loadgen_cluster_target_rate` and `loadgen_cluster_actual_rate`
+  gauges (only in distributed mode)
+
+### What it is NOT
+
+- Not a Chaos Mesh / Litmus replacement. No pod kills, no network
+  partitions. (We can add that layer later with toxiproxy.)
+- Not a load test for the **delivery** tier. Stubbing the delivery
+  tier in `M10-bench` is the way to test the broker+router ceiling;
+  the full delivery path is tested in M10 with the real sinks
+  pointed at a fake-FCM / fake-Telegram server.
+- Not for **production use**. The bin won't even compile with the
+  release tag — it has its own `loadgen` Go module so it can't
+  accidentally end up in a service image.
+
+### Layout
+
+```
+loadgen/
+├── go.mod                       # separate module
+├── cmd/
+│   ├── http/main.go
+│   ├── ws/main.go
+│   ├── mqtt/main.go
+│   └── grpc/main.go
+├── profiles/                    # YAML profiles; --mode picks one
+│   ├── normal.yaml
+│   ├── burst.yaml
+│   ├── chatty-iot.yaml
+│   ├── pager-storm.yaml
+│   └── stress.yaml
+├── internal/
+│   ├── client/                  # wrappers around the same clients the services use
+│   ├── coord/                   # NATS KV coordination
+│   ├── traffic/                 # profile + dedupe + severity mix
+│   └── metrics/                 # prom + slog
+└── README.md                    # cookbook: "how to soak at 5k/s"
+```
+
+### Milestone
+
+A "loadgen usable" check is part of **M10 exit**:
+`loadgen-http --mode normal --rate 5000` sustained 10 min against
+the running stack with p99 ingest latency ≤ 100ms, then
+`loadgen-grpc --mode stress --cluster-target 50000` against
+broker+router stubbed for **M10-bench**.
+
+## 22. Milestones
 
 | # | milestone | exit criterion |
 |---|---|---|
@@ -549,5 +682,6 @@ broad-announce/
 | M7 | Timescale + ClickHouse | 7d retention + archive job |
 | M8 | DLQ + replay UI | operator can replay a failed delivery |
 | M9 | Observability (Prom/Grafana) | 1 dashboard per tier + per-company drilldown |
-| M10 | Load test 5k/s on docker-compose | soak 10 min, p99 ≤ 5s, zero DLQ |
+| M10 | Load test 5k/s on docker-compose | soak 10 min, p99 ≤ 5s, zero DLQ, run via `loadgen` |
+| M10-bench | Broker + router ceiling bench | 50k/s via `loadgen` against broker+router (delivery stubbed); p99 router latency ≤ 50ms; no broker backpressure |
 | M11 | gRPC bidi-streaming ingest | internal Go service pushes ≥ 10k alerts/sec on one stream, p99 server-side `Ack` ≤ 50ms |