# broad-announce

Multi-tenant notification router. Receives alerts from many source
systems (HTTP/JSON webhooks, WebSockets, MQTT, gRPC bidi-streaming),
normalizes them, resolves recipients via `companies` → `groups` →
`individuals` + `subscriptions`, and delivers to FCM (Android),
Telegram, SMS, email, voice, Slack, MS Teams, and arbitrary outbound
webhooks.

> **Status**: M0 + M1 + M2 + M3 + M4 + M5 + M6 + M6.5 + M7 + M8 **shipped** 2026-06-14. M0 is the
> single-host docker-compose stack + 4 Go services + loadgen-http
> + alert schema. M1 is the end-to-end: signed webhook → broker →
> router → deliverd-fcm → fakefcmd (live-verified, 1530 deliveries
> in the loadgen burst, 0 failures). M2 is the recipient-resolution
> rules engine (source.allowed_targets + routing_rules +
> subscriptions with min_severity, quiet hours, inminent_colapse
> bypass; hard-fail on zero recipients). M3 is Telegram delivery +
> bot commands (deliverd split into per-channel binaries,
> telegramd long-polling bot with /start /subscribe /unsubscribe
> /preferences /status /mute /unmute, faketgmd fake Bot API,
> 14 deliveries in 8 sendMessage calls, 0 failures). M4 is MQTT
> ingest (EMQX 5.10.4 broker with per-source ACL on
> `ba/<co>/<src>/incoming`, ingestd's MQTT subscriber reuses the
> same `ProcessAlert` pipeline as HTTP, loadgen-mqtt publisher,
> 12 deliveries in 5-step smoke with 0 failures). M5 is
> WebSocket ingest (`GET /v1/ingest/ws` with `{api_key}` auth
> frame, 35-conn load test, per-IP cap 32) + live tail
> (`GET /v1/tail/ws?token=***&company_id=...` for operators,
> in-process pub/sub fan-out, 13/13 checks green across 3
> consecutive smoke runs). M6 is dedupe + ×N display
> (sliding-window TTL via single Lua script, default 300s;
> dedupe runs before rate limit so duplicates are "free";
> recipient sees `(×N)` inline suffix on the title;
> per-source `dedupe_collapsed_total` and
> `dedupe_count_max_observed` metrics, 11/11 checks green
> across 3 consecutive smoke runs). M6.5 is router-level
> dedupe collapse (a 100-alert burst produces 1 message
> to the recipient, not 100; the tail still sees the
> full storm; debounce with max-wait 2s, configurable
> via `BA_ROUTERD_DEDUPE_FLUSH_MS`; 9/9 checks green
> across 3 consecutive smoke runs). M7 is the data tier:
> `deliveries` becomes a Timescale hypertable with a 7-day
> retention policy, and a new `archiverd` service ships rows
> older than 7 days into ClickHouse (`ba_archive.deliveries_archive`
> MergeTree, 365-day TTL, plus a per-company daily SummingMergeTree
> MV for M9 dashboards); idempotent via `pg_try_advisory_lock` +
> `FOR UPDATE SKIP LOCKED`; `m7_smoke.sh` 4-step, 9-check
> green × 3 consecutive runs on the remote playground
> `parres` (192.168.44.94). M8 is the DLQ + replay UI:
> `deliveries_dlq` Timescale hypertable (7d retention,
> mirror of `deliveries` plus `original_subject` and
> `discarded` columns); in-process retry with bounded
> exp backoff (10 attempts, base 100ms, cap 2s,
> ~12s total wall clock for a fully failing target,
> env-driven via `BA_DELIVERD_*`); `internal/dlq.Write()`
> records the terminal failure; `archiverd` ships DLQ
> rows to ClickHouse `ba_archive.deliveries_dlq_archive`
> (2y TTL, longer than live's 1y because DLQ is forensic);
> `admind` exposes `GET /v1/dlq` (list/filter, 30d
> window), `POST /v1/dlq/{id}/replay` (re-publishes
> the original NATS envelope onto the original subject
> then marks the row discarded), `POST /v1/dlq/{id}/discard`
> (mark discarded, idempotent), and a minimal HTML
> UI at `GET /dlq` with filter form + inline
> replay/discard buttons; `m8_smoke.sh` 4-step,
> 12-check green × 3 consecutive runs on the local
> docker-compose stack.
> See `M0_VERIFICATION.md` … `M8_VERIFICATION.md` and
> `M1_SMOKE_LOG.md` … `M8_SMOKE_LOG.md` for the smoke tests.
> Spec is in `SPEC.md`, diagrams in `ARCHITECTURE.md`, build log
> in `PROMPT.md`.

## v1 in one paragraph

Six Go services (`ingestd`, `routerd`, `deliverd-fcm`,
`deliverd-telegram`, `telegramd`, `admind`) wired together by NATS
JetStream. Postgres + Timescale for live data, ClickHouse for
archive, Redis for dedupe + rate limits, EMQX for MQTT. Strict
app-level multi-tenant isolation. ~5k alerts/sec on Docker
Compose, 50k/sec design ceiling for v2 K8s.

## Repo layout

```
SPEC.md             - requirements, entities, severity, retention
ARCHITECTURE.md     - diagrams, sequences, SLOs, capacity model
PROMPT.md           — build log, decisions, open questions
M0_VERIFICATION.md  — M0 smoke test (signed webhook → 202)
M1_VERIFICATION.md  — M1 smoke test (end-to-end → fakefcmd)
M2_VERIFICATION.md  — M2 smoke test (recipient resolution)
M3_VERIFICATION.md  — M3 smoke test (Telegram delivery + bot)
M4_VERIFICATION.md  — M4 smoke test (MQTT ingest + EMQX ACL)
M5_VERIFICATION.md  — M5 smoke test (WS ingest + live tail + per-IP cap)
M6_VERIFICATION.md  — M6 smoke test (dedupe + ×N + sliding TTL + free-for-dupes)
M6.5_VERIFICATION.md — M6.5 smoke test (router-level dedupe collapse)
M7_VERIFICATION.md  — M7 smoke test (Timescale 7d + ClickHouse archive)
M8_VERIFICATION.md  — M8 smoke test (DLQ + retry + replay UI)
M1_SMOKE_LOG.md     — M1 live run results
M2_SMOKE_LOG.md     — M2 live run results
M3_SMOKE_LOG.md     — M3 live run results
M4_SMOKE_LOG.md     — M4 live run results
M5_SMOKE_LOG.md     — M5 live run results (3 consecutive green)
M6_SMOKE_LOG.md     — M6 live run results (3 consecutive green)
M6.5_SMOKE_LOG.md   — M6.5 live run results (3 consecutive green)
M7_SMOKE_LOG.md     — M7 live run results (3 consecutive green, on parres)
M8_SMOKE_LOG.md     — M8 live run results (3 consecutive green, local)
docker-compose.yml  — single-host M0–M8 stack
Dockerfile          — multi-stage build for all 8 binaries (M0–M7 + archiverd)
.env.example        — every BA_* knob documented
cmd/ingestd/        — HTTP POST handler (M0) + MQTT subscriber (M4) + WS ingest (M5) + dedupe before rate limit (M6); M11 = TLS
cmd/routerd/        — consumer (M0) + recipient resolution (M2) + M6.5 dedupe Collapser with max-wait debounce
cmd/routerd/        - M2 rules engine + M3 channel union
cmd/deliverd-fcm/   - M1 FCM HTTP v1 delivery + M8 retry + DLQ write
cmd/deliverd-telegram/ - M3 Telegram Bot API delivery + M8 retry + DLQ write
cmd/telegramd/      - M3 long-polling bot loop + command handler
cmd/admind/         - /v1/ping (M0) + /v1/dlq + /v1/dlq/{id} + /v1/dlq/{id}/{replay,discard} + /dlq HTML UI (M8)
cmd/archiverd/      - M7 hourly Timescale→ClickHouse archiver + M8 DLQ drain
internal/dlq/       - M8 deliveries_dlq writer
internal/retry/     - M8 bounded exp-backoff retry helper
loadgen/cmd/http/   - HTTP traffic generator (M0)
loadgen/cmd/mqtt/   - MQTT traffic generator (M4)
loadgen/cmd/ws/     - WebSocket traffic generator (M5)
internal/alert/     - Alert v1 type + Validate() + Severity.Rank
internal/broker/    - NATS JetStream wrapper
internal/config/    - env-driven config
internal/dedupe/    - 60s SET NX EX + INCR
internal/ratelimit/ - per-second INCR bucket
internal/httpserver/ - /health + /metrics scaffold
internal/observability/ - slog + Prometheus
internal/postgres/  - pgxpool wrapper
internal/routing/   - Resolver (M2 rules engine, M3 channel union)
internal/telegram/  - BotClient + command parser + handler
internal/store/     - Redis + (later) Postgres
deploy/prometheus/  - prometheus.yml
migrations/         - 001-004 + seed/seed_m2/seed_m3.sql
testfakes/          - fakefcmd (M1), faketgmd (M3)
```

## License

Private. © 2026 Techno-World.