broad-announce

broad-announce

Multi-tenant notification router. Receives alerts from many source systems (HTTP/JSON webhooks, WebSockets, MQTT, gRPC bidi-streaming), normalizes them, resolves recipients via companies → groups → individuals + subscriptions, and delivers to FCM (Android), Telegram, SMS, email, voice, Slack, MS Teams, and arbitrary outbound webhooks.

Status: M0 + M1 + M2 + M3 + M4 + M5 + M6 + M6.5 + M7 + M8 + M11 + F2 + M13a + M14-backend W1 shipped 2026-06-17. M0 is the single-host docker-compose stack + 4 Go services + loadgen-http

• alert schema. M1 is the end-to-end: signed webhook → broker → router → deliverd-fcm → fakefcmd (live-verified, 1530 deliveries in the loadgen burst, 0 failures). M2 is the recipient-resolution rules engine (source.allowed_targets + routing_rules + subscriptions with min_severity, quiet hours, inminent_colapse bypass; hard-fail on zero recipients). M3 is Telegram delivery + bot commands (deliverd split into per-channel binaries, telegramd long-polling bot with /start /subscribe /unsubscribe /preferences /status /mute /unmute, faketgmd fake Bot API, 14 deliveries in 8 sendMessage calls, 0 failures). M4 is MQTT ingest (EMQX 5.10.4 broker with per-source ACL on ba/<co>/<src>/incoming, ingestd's MQTT subscriber reuses the same ProcessAlert pipeline as HTTP, loadgen-mqtt publisher, 12 deliveries in 5-step smoke with 0 failures). M5 is WebSocket ingest (GET /v1/ingest/ws with {api_key} auth frame, 35-conn load test, per-IP cap 32) + live tail (GET /v1/tail/ws?token=***&company_id=... for operators, in-process pub/sub fan-out, 13/13 checks green across 3 consecutive smoke runs). M6 is dedupe + ×N display (sliding-window TTL via single Lua script, default 300s; dedupe runs before rate limit so duplicates are "free"; recipient sees (×N) inline suffix on the title; per-source dedupe_collapsed_total and dedupe_count_max_observed metrics, 11/11 checks green across 3 consecutive smoke runs). M6.5 is router-level dedupe collapse (a 100-alert burst produces 1 message to the recipient, not 100; the tail still sees the full storm; debounce with max-wait 2s, configurable via BA_ROUTERD_DEDUPE_FLUSH_MS; 9/9 checks green across 3 consecutive smoke runs). M7 is the data tier: deliveries becomes a Timescale hypertable with a 7-day retention policy, and a new archiverd service ships rows older than 7 days into ClickHouse (ba_archive.deliveries_archive MergeTree, 365-day TTL, plus a per-company daily SummingMergeTree MV for M9 dashboards); idempotent via pg_try_advisory_lock + FOR UPDATE SKIP LOCKED; m7_smoke.sh 4-step, 9-check green × 3 consecutive runs on the remote playground parres (192.168.44.94). M8 is the DLQ + replay UI: deliveries_dlq Timescale hypertable (7d retention, mirror of deliveries plus original_subject and discarded columns); in-process retry with bounded exp backoff (10 attempts, base 100ms, cap 2s, ~12s total wall clock for a fully failing target, env-driven via BA_DELIVERD_*); internal/dlq.Write() records the terminal failure; archiverd ships DLQ rows to ClickHouse ba_archive.deliveries_dlq_archive (2y TTL, longer than live's 1y because DLQ is forensic); admind exposes GET /v1/dlq (list/filter, 30d window), POST /v1/dlq/{id}/replay (re-publishes the original NATS envelope onto the original subject then marks the row discarded), POST /v1/dlq/{id}/discard (mark discarded, idempotent), and a minimal HTML UI at GET /dlq with filter form + inline replay/discard buttons; m8_smoke.sh 4-step, 12-check green × 3 consecutive runs on the local docker-compose stack. See M0_VERIFICATION.md … M8_VERIFICATION.md and M1_SMOKE_LOG.md … M8_SMOKE_LOG.md for the smoke tests. Spec is in SPEC.md, diagrams in ARCHITECTURE.md, build log in PROMPT.md.

v1 in one paragraph

Six Go services (ingestd, routerd, deliverd-fcm, deliverd-telegram, telegramd, admind) wired together by NATS JetStream. Postgres + Timescale for live data, ClickHouse for archive, Redis for dedupe + rate limits, EMQX for MQTT. Strict app-level multi-tenant isolation. ~5k alerts/sec on Docker Compose, 50k/sec design ceiling for v2 K8s.

Repo layout

SPEC.md             - requirements, entities, severity, retention
ARCHITECTURE.md     - diagrams, sequences, SLOs, capacity model
PROMPT.md           — build log, decisions, open questions
M0_VERIFICATION.md  — M0 smoke test (signed webhook → 202)
M1_VERIFICATION.md  — M1 smoke test (end-to-end → fakefcmd)
M2_VERIFICATION.md  — M2 smoke test (recipient resolution)
M3_VERIFICATION.md  — M3 smoke test (Telegram delivery + bot)
M4_VERIFICATION.md  — M4 smoke test (MQTT ingest + EMQX ACL)
M5_VERIFICATION.md  — M5 smoke test (WS ingest + live tail + per-IP cap)
M6_VERIFICATION.md  — M6 smoke test (dedupe + ×N + sliding TTL + free-for-dupes)
M6.5_VERIFICATION.md — M6.5 smoke test (router-level dedupe collapse)
M7_VERIFICATION.md  — M7 smoke test (Timescale 7d + ClickHouse archive)
M8_VERIFICATION.md  — M8 smoke test (DLQ + retry + replay UI)
M1_SMOKE_LOG.md     — M1 live run results
M2_SMOKE_LOG.md     — M2 live run results
M3_SMOKE_LOG.md     — M3 live run results
M4_SMOKE_LOG.md     — M4 live run results
M5_SMOKE_LOG.md     — M5 live run results (3 consecutive green)
M6_SMOKE_LOG.md     — M6 live run results (3 consecutive green)
M6.5_SMOKE_LOG.md   — M6.5 live run results (3 consecutive green)
M7_SMOKE_LOG.md     — M7 live run results (3 consecutive green, on parres)
M8_SMOKE_LOG.md     — M8 live run results (3 consecutive green, local)
docker-compose.yml  — single-host M0–M8 stack
Dockerfile          — multi-stage build for all 9 binaries (M0–M8 + archiverd + authd)
.env.example        — every BA_* knob documented
cmd/ingestd/        — HTTP POST handler (M0) + MQTT subscriber (M4) + WS ingest (M5) + dedupe before rate limit (M6); M11 = TLS
cmd/routerd/        — consumer (M0) + recipient resolution (M2) + M6.5 dedupe Collapser with max-wait debounce
cmd/routerd/        - M2 rules engine + M3 channel union
cmd/deliverd-fcm/   - M1 FCM HTTP v1 delivery + M8 retry + DLQ write
cmd/deliverd-telegram/ - M3 Telegram Bot API delivery + M8 retry + DLQ write
cmd/telegramd/      - M3 long-polling bot loop + command handler
cmd/admind/         - /v1/ping (M0) + /v1/dlq + /v1/dlq/{id} + /v1/dlq/{id}/{replay,discard} + /dlq HTML UI (M8); M13a W3 = JWT gate
cmd/archiverd/      - M7 hourly Timescale→ClickHouse archiver + M8 DLQ drain
cmd/authd/          - M13a W1: in-house multi-tenant auth IdP (JWT + refresh-token store)
internal/authd/     - M13a: JWT (HS256) + magic links + refresh-token CRUD + audit hooks
internal/auth/      - M14 W2: mTLS client cert verifier
internal/dlq/       - M8 deliveries_dlq writer
internal/retry/     - M8 bounded exp-backoff retry helper
loadgen/cmd/http/   - HTTP traffic generator (M0)
loadgen/cmd/mqtt/   - MQTT traffic generator (M4)
loadgen/cmd/ws/     - WebSocket traffic generator (M5)
internal/alert/     - Alert v1 type + Validate() + Severity.Rank
internal/broker/    - NATS JetStream wrapper
internal/config/    - env-driven config
internal/dedupe/    - 60s SET NX EX + INCR
internal/ratelimit/ - per-second INCR bucket
internal/httpserver/ - /health + /metrics scaffold
internal/observability/ - slog + Prometheus
internal/postgres/  - pgxpool wrapper
internal/routing/   - Resolver (M2 rules engine, M3 channel union)
internal/telegram/  - BotClient + command parser + handler
internal/store/     - Redis + (later) Postgres
deploy/prometheus/  - prometheus.yml
migrations/         - 001-004 + seed/seed_m2/seed_m3.sql
testfakes/          - fakefcmd (M1), faketgmd (M3)

License

Private. © 2026 Techno-World.