|
|
@@ -504,7 +504,6 @@ internal Go service can publish ≥ 10k alerts/sec on one stream
|
|
|
sustained 10 min, p99 server-side `Ack` latency ≤ 50ms.
|
|
|
|
|
|
## 20. Repo layout (planned)
|
|
|
-
|
|
|
```
|
|
|
broad-announce/
|
|
|
├── SPEC.md ← this file
|
|
|
@@ -518,6 +517,7 @@ broad-announce/
|
|
|
├── routerd/ ← binary: recipient expansion
|
|
|
├── deliverd/ ← binary: per-channel delivery workers
|
|
|
├── admind/ ← binary: admin HTTP API + (later) UI host
|
|
|
+├── loadgen/ ← (coming) traffic generators per protocol
|
|
|
├── internal/
|
|
|
│ ├── alert/ ← Alert v1 type, validation
|
|
|
│ ├── broker/ ← NATS wrapper
|
|
|
@@ -535,7 +535,140 @@ broad-announce/
|
|
|
└── loki/
|
|
|
```
|
|
|
|
|
|
-## 21. Milestones
|
|
|
+## 21. Traffic generators (loadgen)
|
|
|
+
|
|
|
+Four small Go binaries in `loadgen/` — one per source protocol — that
|
|
|
+behave as **reference producers** of alerts. Same wire format, same
|
|
|
+auth, same client libs an internal service would use. Copy the
|
|
|
+binaries to as many hosts as you want, point them at a target, and
|
|
|
+they generate load.
|
|
|
+
|
|
|
+### Goals
|
|
|
+
|
|
|
+- **Realistic traffic** ("normal"): matches the shape of real-world
|
|
|
+ sources — bursty, dedupe-heavy, mix of severities, mix of
|
|
|
+ company_ids.
|
|
|
+- **Stress traffic** ("stress"): maximum sustained rate, no dedupe,
|
|
|
+ fat payloads, with fault-injection knobs (drop %, latency spikes,
|
|
|
+ random disconnects).
|
|
|
+- **Distributed by design**: any number of `loadgen` instances on
|
|
|
+ any number of hosts coordinate via a tiny shared state in
|
|
|
+ NATS JetStream KV (`loadgen.coordination`). No single host is
|
|
|
+ in charge.
|
|
|
+- **Self-contained**: no separate deploy, no extra observability
|
|
|
+ stack. Emit Prometheus metrics on `:9091`, structured logs to
|
|
|
+ stdout.
|
|
|
+
|
|
|
+### Binaries
|
|
|
+
|
|
|
+| binary | protocol | client lib | what it generates |
|
|
|
+|---|---|---|---|
|
|
|
+| `loadgen-http` | HTTP POST | `internal/httpclient/` | signed webhooks, configurable concurrency, N concurrent connections |
|
|
|
+| `loadgen-ws` | WebSocket | `internal/wsclient/` | one long-lived socket per instance, alert frames every X ms |
|
|
|
+| `loadgen-mqtt` | MQTT | `internal/mqttclient/` | QoS 1, per-company topic, configurable in-flight |
|
|
|
+| `loadgen-grpc` | gRPC bidi | `internal/grpcclient/` | one stream per instance, batched send with bounded in-flight |
|
|
|
+
|
|
|
+### CLI surface (same shape across all four)
|
|
|
+
|
|
|
+```bash
|
|
|
+loadgen-http \
|
|
|
+ --target https://broad-announce:8080 \
|
|
|
+ --api-key $SOURCE_API_KEY \
|
|
|
+ --mode normal | stress \
|
|
|
+ --rate 1000 # alerts/sec target per instance
|
|
|
+ --duration 10m \
|
|
|
+ --company-prefix acme- # company_ids are "acme-001" .. "acme-100" cycled
|
|
|
+ --severities info:70,critical:5,inminent_colapse:1 \
|
|
|
+ --dedupe-ratio 0.30 # 30% of alerts share a dedupe_key within 60s
|
|
|
+ --payload-bytes 512 # avg data{} size
|
|
|
+ --concurrency 64 # concurrent HTTP connections
|
|
|
+ --drop-pct 0.0 # stress only: drop this % of sends
|
|
|
+ --latency-spike-ms 0 # stress only: inject X ms of extra sleep
|
|
|
+ --reconnect-every 0 # stress only: drop+reconnect every X seconds
|
|
|
+ --metrics :9091
|
|
|
+ --coordinator nats://nats:4222 # for distributed mode
|
|
|
+```
|
|
|
+
|
|
|
+### Traffic profiles (built-in)
|
|
|
+
|
|
|
+| profile | rate per instance | dedupe | severity mix | payload | use case |
|
|
|
+|---|---|---|---|---|---|
|
|
|
+| `normal` | 500/s | 30% | 70/25/4/1 | 512B | soak tests, capacity planning |
|
|
|
+| `burst` | peak 5k/s for 10s, idle 50s | 50% | 50/30/15/5 | 1KB | alert-storm resilience |
|
|
|
+| `chatty-iot` | 50/s × 1000 sources | 0% | 95/5/0/0 | 128B | IoT fleet size test |
|
|
|
+| `pager-storm` | 10k/s | 0% | 0/0/50/50 | 256B | inminent_colapse path |
|
|
|
+| `stress` | unlimited (CPU bound) | 0% | 25/25/25/25 | 4KB | find the breaking point |
|
|
|
+
|
|
|
+### Distributed mode
|
|
|
+
|
|
|
+Each instance advertises its rate in NATS KV. A coordinator (any one
|
|
|
+instance, elected) sums the rates and assigns per-instance targets
|
|
|
+to hit a **cluster-wide** rate. Use case: spin up 10 hosts, each
|
|
|
+running 4 instances of `loadgen-grpc`, target 50k/s cluster-wide.
|
|
|
+
|
|
|
+```
|
|
|
+loadgen-grpc --mode stress --cluster-target 50000 --cluster-id bench-2026-06-13
|
|
|
+```
|
|
|
+
|
|
|
+Coordination is **best-effort** (no leader lock, no consensus) — if
|
|
|
+the coordinator drops, the next instance takes over. Over- and
|
|
|
+under-shoot within ±5% is acceptable for soak tests.
|
|
|
+
|
|
|
+### Metrics
|
|
|
+
|
|
|
+- `loadgen_alerts_sent_total{profile,instance,mode}` counter
|
|
|
+- `loadgen_alerts_failed_total{reason}` counter (drop, timeout, 4xx, 5xx)
|
|
|
+- `loadgen_alert_send_latency_seconds_bucket{mode}` histogram
|
|
|
+- `loadgen_dedupe_hits_total` counter (computed server-side; we just
|
|
|
+ read it from `Ack` and increment locally)
|
|
|
+- `loadgen_cluster_target_rate` and `loadgen_cluster_actual_rate`
|
|
|
+ gauges (only in distributed mode)
|
|
|
+
|
|
|
+### What it is NOT
|
|
|
+
|
|
|
+- Not a Chaos Mesh / Litmus replacement. No pod kills, no network
|
|
|
+ partitions. (We can add that layer later with toxiproxy.)
|
|
|
+- Not a load test for the **delivery** tier. Stubbing the delivery
|
|
|
+ tier in `M10-bench` is the way to test the broker+router ceiling;
|
|
|
+ the full delivery path is tested in M10 with the real sinks
|
|
|
+ pointed at a fake-FCM / fake-Telegram server.
|
|
|
+- Not for **production use**. The bin won't even compile with the
|
|
|
+ release tag — it has its own `loadgen` Go module so it can't
|
|
|
+ accidentally end up in a service image.
|
|
|
+
|
|
|
+### Layout
|
|
|
+
|
|
|
+```
|
|
|
+loadgen/
|
|
|
+├── go.mod # separate module
|
|
|
+├── cmd/
|
|
|
+│ ├── http/main.go
|
|
|
+│ ├── ws/main.go
|
|
|
+│ ├── mqtt/main.go
|
|
|
+│ └── grpc/main.go
|
|
|
+├── profiles/ # YAML profiles; --mode picks one
|
|
|
+│ ├── normal.yaml
|
|
|
+│ ├── burst.yaml
|
|
|
+│ ├── chatty-iot.yaml
|
|
|
+│ ├── pager-storm.yaml
|
|
|
+│ └── stress.yaml
|
|
|
+├── internal/
|
|
|
+│ ├── client/ # wrappers around the same clients the services use
|
|
|
+│ ├── coord/ # NATS KV coordination
|
|
|
+│ ├── traffic/ # profile + dedupe + severity mix
|
|
|
+│ └── metrics/ # prom + slog
|
|
|
+└── README.md # cookbook: "how to soak at 5k/s"
|
|
|
+```
|
|
|
+
|
|
|
+### Milestone
|
|
|
+
|
|
|
+A "loadgen usable" check is part of **M10 exit**:
|
|
|
+`loadgen-http --mode normal --rate 5000` sustained 10 min against
|
|
|
+the running stack with p99 ingest latency ≤ 100ms, then
|
|
|
+`loadgen-grpc --mode stress --cluster-target 50000` against
|
|
|
+broker+router stubbed for **M10-bench**.
|
|
|
+
|
|
|
+## 22. Milestones
|
|
|
|
|
|
| # | milestone | exit criterion |
|
|
|
|---|---|---|
|
|
|
@@ -549,5 +682,6 @@ broad-announce/
|
|
|
| M7 | Timescale + ClickHouse | 7d retention + archive job |
|
|
|
| M8 | DLQ + replay UI | operator can replay a failed delivery |
|
|
|
| M9 | Observability (Prom/Grafana) | 1 dashboard per tier + per-company drilldown |
|
|
|
-| M10 | Load test 5k/s on docker-compose | soak 10 min, p99 ≤ 5s, zero DLQ |
|
|
|
+| M10 | Load test 5k/s on docker-compose | soak 10 min, p99 ≤ 5s, zero DLQ, run via `loadgen` |
|
|
|
+| M10-bench | Broker + router ceiling bench | 50k/s via `loadgen` against broker+router (delivery stubbed); p99 router latency ≤ 50ms; no broker backpressure |
|
|
|
| M11 | gRPC bidi-streaming ingest | internal Go service pushes ≥ 10k alerts/sec on one stream, p99 server-side `Ack` ≤ 50ms |
|