broad-announce/PROMPT.md

Broad-Announce — Build Log (PROMPT)

Decisions, lessons, blockers. Append-only. Update as we go.

2026-06-13 — kickoff

Decided

• Repo at git3.techno-world.net/lrosales/broad-announce (private).
• v1 stack: Go, PostgreSQL + Timescale, ClickHouse, NATS JetStream, Redis, EMQX (MQTT), Prometheus + Grafana, Loki. Deploy v1 = Docker Compose. v2 = K8s.
• Multi-tenancy = shared infrastructure, strict app-level isolation. No row-level security in v1; tenant filter on every query.
• FCM = single shared project for v1, schema supports per-company FCM project later (companies.fcm_shared).
• Telegram is a first-class delivery channel + a management UI (the bot is how users mute, subscribe, acknowledge).
• Severity taxonomy: info | warning | critical | inminent_colapse. Only inminent_colapse bypasses quiet hours.
• Dedupe: 60s window per (source_id, dedupe_key), attach dedupe_count so user sees "×N in 60s" not N pushes.
• v1 capacity target in docker-compose: 5k alerts/sec sustained. 50k/sec is the design ceiling; the K8s + multi-broker work is what unlocks it.

Open (was) → Resolved 2026-06-13

• Bot ↔ individual linking → admin-invites only. Flow: admin creates individuals row + generates one-time invite code; user runs /start <invite_code> in the Telegram bot; bot matches the code, links telegram_chat_id to the pre-existing individual, and burns the code. Stricter path: prevents drive-by bot self-registration, keeps individuals auditable.
• Localized titles → source-localized, pass through. v1 does NOT translate. Sources send pre-localized title/body strings (or use the locale on the subscription to key into their own lookup table before calling our API). Alert schema carries locale and title/body already-resolved. We can add a translation layer in v2 if customers ask.
• mTLS: schema supports it, but the docker-compose profile won't terminate client certs in v1. Documented as opt-in for enterprise sources.

Lessons (already)

• It's much cheaper to answer "what can this be?" with 30 questions than to refactor later. Most of the SPEC's weight is in §11 (security) and §6 (recipient resolution) — those are the parts that are expensive to change after launch.
• For multi-tenant at 10k companies, the one design choice that compounds is the subject layout in the broker. alerts.<company_id> is fine; alerts.<company_id>.<source_id> would let us scale router consumer groups per source. Locked in §6 of ARCHITECTURE.md.

2026-06-13 — added gRPC ingest (option a)

• Decision: ship gRPC bidi-streaming as the fourth ingest protocol, scoped to internal high-volume sources only. Public SaaS webhooks stay on HTTP POST, browsers stay on WebSocket, IoT stays on MQTT.
• Rationale: typed schemas, HTTP/2 + protobuf, native backpressure, no reconnect-loop code, generated Go/Java/Python/Node clients. Cost: one more protocol to operate + a buf generate build step.
• Gated on M11, after M10 (load test). M10 now caps at 5k/s on docker-compose (not 50k/s) — 50k/s is the design ceiling that requires K8s + multi-broker (M12 in v2).
• StreamAlerts(Alert) → Ack carries the same dedupe_count contract as HTTP/WS/MQTT.
• Proto lives at proto/broadannounce/v1/ingest.proto; server stub in internal/grpcserver/, reusable client in internal/grpcclient/. Auth = API key in metadata + optional mTLS. Per-stream rate limit + 256 in-flight cap = natural backpressure.

2026-06-13 — M10 split: option C

• M10 (in-cluster, real) stays at 5k/s on docker-compose.
• Added M10-bench: 50k/s against the broker + router with the delivery tier stubbed. Proves the ceiling without committing to K8s. Green here means K8s is implementable, not required.
• Implication: we need traffic generators that can hit the in-cluster path at 5k/s (M10) and the broker+router path at 50k/s (M10-bench) with the same source-protocol clients we ship. The loadgen/ tool is the new home for that.

2026-06-13 — three open questions resolved

• (Q1) Yes, build the fake-FCM / fake-Telegram / fake-SMS servers (testfakes/). M10 must not burn 50k FCM credits.
• (Q2) M10-bench delivery stub lives at the service level — a no-op deliverd binary in the bench profile, not a NATS subject drop. More realistic: we exercise the real broker + router + the message shape deliverd would consume.
• (Q3) loadgen ships as a single Docker image with all four binaries; entrypoint picks one via the image's command: field in the compose / k8s spec.

2026-06-13 — Source protection (throttling) is in

• Spec §22 (new). Seven layers, evaluated in order per request: payload-size cap → per-IP conn cap → per-source token bucket → per-company token bucket → schema validate → broker circuit breaker → per-source quarantine.
• Per-source config additive to §4 schema (rate_limit_per_sec, max_payload_bytes, max_concurrent_connections, quarantine_*).
• Rejection-code matrix is the contract across HTTP/WS/MQTT/gRPC (e.g. per-source rate → 429+Retry-After / Ack{RATE_LIMITED} / MQTT reason 0x97 / close 1013).
• Rollout tied to milestones: M0 layer 3, M1 layers 1/4/5, M5 layer 2, M6 dedupe-aware shaping, M9 layers 6+7.
• Runaway-source test is part of M10 exit (loadgen fault-injection: one source at 10× cap must not push p99 for other sources above 5s).

2026-06-13 — M0 shipped (12 commits, 2888 LoC)

What landed:

• cmd/{ingestd,routerd,deliverd,admind}/ — four Go service mains
• cmd/ingestd/http.go — HTTP POST handler implementing SPEC §22 layers 1, 3, 4, 5 + Stripe-style HMAC auth
• internal/alert — Alert v1 type + Validate() (183 LoC + 103 LoC tests)
• internal/broker — NATS JetStream wrapper, three streams (ALERTS/DELIVERIES/DLQ) auto-created
• internal/dedupe — 60s SET NX EX + INCR (Redis-required tests pass against a host-local Redis)
• internal/ratelimit — per-second INCR bucket (Redis-required tests pass)
• internal/observability — slog + Prometheus registry, the IngestdMetrics struct matches SPEC §22 metric names
• internal/httpserver — shared /health + /metrics scaffold
• internal/config — env-driven Common + Ingestd
• loadgen/cmd/http/ — loadgen-http with --mode normal, HMAC signing, 70/25/4/1 severity mix, dedupe-pct knob
• loadgen/go.mod — separate module per SPEC §21, replace directive points at the parent module
• docker-compose.yml + Dockerfile — single-host stack, all 5 binaries in one image
• M0_VERIFICATION.md — 8-step smoke test
• deploy/prometheus/prometheus.yml — scrapes all 5 services

What's NOT in M0 (and not supposed to be):

• routerd/deliverd/admind business logic (M2/M3+)
• MQTT, gRPC, WebSocket ingest (M4/M5/M11)
• per-IP concurrency cap (M5)
• circuit breaker (M9)
• quarantine (M9)
• ClickHouse archive (M7)
• source registry in DB (M2)
• Postgres migrations (M2)

Module path: git3.techno-world.net/lrosales/broad-announce. Loadgen module path: git3.techno-world.net/lrosales/broad-announce/loadgen. All pushed: 7cd922c..49b2dba on master.

2026-06-13 — Port convention

Project rule: app HTTP services use 8800–8899 (ingestd 8800, routerd 8801, deliverd 8802, admind 8803, loadgen metrics 8891, fakefcmd 8820). Canonical ports stay (5432, 4222, 6379, 1883, 9090, 3000). Reason: 8080+ collides with workstation tooling. SPEC §18 now has a port-conventions sub-section. Commit: d76aa0b.

2026-06-13 — M1 code-complete (8 commits, awaiting live smoke)

What landed:

• migrations/001_init.up.sql — companies, individuals, fcm_tokens (the M1 minimum schema; full SPEC §4 lands in M2 as additional migrations)
• migrations/002_deliveries.up.sql — deliveries table (status: pending|sent|failed|dlq; payload jsonb for M8 replay)
• migrations/seed.sql — idempotent; 1 company, 1 individual, 1 token
• internal/postgres — pgxpool wrapper with retry-on-startup
• cmd/seed — applies *.up.sql in lexical order, then seed.sql
• testfakes/fakefcmd — 70 lines, /health + /v1/.../messages:send, --fail-rate knob
• internal/routing — Resolver with ResolveTokens (M1 broadcast via single SQL join; M2 swaps for the rules engine)
• cmd/routerd — M1 main: subscribes to alerts.>, resolves recipients, publishes one deliveries.fcm. per token
• cmd/deliverd — M1 main: subscribes to deliveries.fcm.>, builds FCM HTTP v1 message body (M3 swap is a no-op at this layer), posts to BA_FAKECMD_URL, writes a deliveries row
• Dockerfile builds 7 binaries
• docker-compose adds fakefcmd + seed (one-shot sidecar), wires BA_FAKECMD_URL into deliverd
• M1_VERIFICATION.md: 9-step smoke test

What we agreed to defer (per the user):

• Q1 seed: 1 company / 1 individual / 1 token — DONE
• Q2 router: M1 broadcast (no subscriptions) — DONE
• Q3 testfakes: only fakefcmd for M1 — DONE
• Q4 migrations: 3 tables in M1, expand in M2 — DONE

What's NOT in M1 (and not supposed to be):

• subscriptions, quiet hours, routing rules (M2)
• per-source allowed_targets (M2)
• real FCM (M3)
• other delivery channels (M3+)
• retry + DLQ from SPEC §9 (M3)
• ClickHouse / Timescale hypertables (M7)
• HMAC secret in DB (M2)
• per-IP cap, circuit breaker, quarantine (M5/M9)

Pushed: 264d284..9cf68a1 on master (5 commits for M1 code, plus the d76aa0b port shift). The user still has to actually run docker compose up and the M1 verification steps before M1 is fully done.

2026-06-13 — M2 shipped (recipient resolution, rules engine)

What landed (3 commits, ~600 LoC Go + ~250 LoC SQL):

• migrations/003_subscriptions_groups.up.sql (and .down.sql): 5 new tables — sources, groups, group_members, subscriptions, routing_rules. FKs to companies / individuals. Idempotent.
• migrations/seed_m2.sql: 2 more individuals (Bob, Carol), 1 group (sre: Alice+Bob), 1 source row, 3 subscriptions covering the 3 scenarios, 1 routing rule.
• cmd/seed/main.go: runner now applies seed.sql then seed_m2.sql (lexically ordered) so re-running is safe.
• internal/routing/routing.go: full rewrite. New Resolver with ResolveTargets(ctx, *alert.Alert) ([]Target, error). Single SQL round-trip via a 5-CTE query:
  1. src — the source row
  2. allowed_individual_ids — direct + group-expanded + broadcast
  3. rule_individual_ids — routing_rules where match_expr matches
  4. candidates — union of 2 and 3
  5. sub_expanded + filtered — subscriptions with min_severity, channel_mask, quiet-hours filter Final pass: Go-side quiet-hours check (with the inminent_colapse bypass) and channel='fcm' gate.
• internal/alert/alert.go: added Severity.Rank() and MinSeverityRank() helpers. Two new unit tests in alert_test.go.
• internal/routing/routing_test.go: new file, 10 subtests for inQuietHours (same-day, wrap-around, always-quiet, edge cases at window start/end).
• cmd/routerd/main.go: M2 main. Calls ResolveTargets(alert), hard-fails on zero recipients (logs WARN, acks — no DLQ for M2; that's M3+). Emits the new envelope shape with Channel + Endpoint instead of the M1 FCMToken+Locale.
• cmd/deliverd/main.go: envelope struct updated to match the M2 routerd output. Only field renames; the FCM HTTP v1 body shape is unchanged.
• M2_VERIFICATION.md: 7-step plan + hard-fail scenario.
• scripts/m2_smoke.sh: automated runner for steps 2-6.
• M2_SMOKE_LOG.md: per-step results, honest flags.

Live smoke results (all green):

• step 2: warning, db-prod-04 → Alice only
• step 3: critical, db-prod-04 → Alice + Bob
• step 4: critical, db-prod-03 (rule match) → Alice + Bob
• step 5: warning, db-prod-03 (rule match but min filter) → Alice only (Bob filtered by min_severity=critical)
• step 6: inminent_colapse → Alice + Bob + Carol (Carol's quiet_hours=00:00-23:59 bypassed)
• step 7: subscriptions paused → routerd logs WARN msg="zero recipients, dropping", no delivery 9 deliveries, 0 failures, 0 retries.

Per the user's M2 Q1/Q2/Q3 answers:

• Q1: seed = 3 individuals + 1 group + 1 source + 4 subscriptions covering all 4 scenarios (DONE; one of the 4 was dropped to 3 because we folded "fcm with min=critical AND in quiet hours" into a single individual — the bypass test is still valid via Carol).
• Q2: hard-fail on zero recipients with a WARN log (DONE; the routerd logger is the notification path for M2; M3+ will add a dlq.no_recipients.<company> subject).
• Q3: (a) emit the row, drop the channel at the SQL filter (DONE; WHERE f.channel = 'fcm' in the resolver).

What's NOT in M2 (and not supposed to be):

• real FCM (M3)
• telegram, sms, email, slack, teams (M3+)
• retry + DLQ (M3)
• per-source HMAC secret in DB (M5 when mTLS / API-key path comes in)
• full match_expr language (M6+); M2 supports category, severity, data.*, all
• per-rule priority semantics with "first match wins unless continue=true" (M3+); M2 unions all matching rules
• routing rule "drop" target type (schema supports it, no test for it; the SQL filters it via the candidate union)
• Timezone support beyond UTC (M2 falls back to UTC if time.LoadLocation(tz) fails)

Pushed: 280e048 on master (3 commits: a29c2d8 + 1e9eca9 + 280e048).

2026-06-14 — M3 shipped (Telegram delivery + bot commands)

What landed (~2k LoC Go + ~80 LoC SQL across these commits):

• migrations/004_telegram.up.sql (+ .down.sql): telegram_bots table (one row per (company, bot); M3 supports one bot per company), and 4 new columns on individuals: telegram_chat_id, telegram_user_id, telegram_invite_code, mute_until. Two partial unique indexes for fast /start <code> lookup and "is this telegram_user_id already linked" check.
• migrations/seed_m3.sql: inserts the per-company telegram_bots row, sets telegram_invite_code on all three individuals, and pre-links Alice to a fake Telegram account (chat_id=1001, user_id=900001) so the very first POST exercises both fcm and telegram delivery paths. Also expands every subscription's channel_mask to ["fcm","telegram"] (M2 was ["fcm"]).
• cmd/seed/main.go: runner now applies seed.sql, seed_m2.sql, seed_m3.sql in lexical order. All three are idempotent.
• cmd/deliverd/ → cmd/deliverd-fcm/: rename. The M2 deliverd is now deliverd-fcm — per the user's M3 Q2 answer (two per-channel binaries, not one with a registry).
• cmd/deliverd-telegram/: new binary. Subscribes to deliveries.telegram.<company_id>, builds a severity-prefixed text message, posts to the Bot API's sendMessage endpoint, writes a deliveries row. Single-attempt (no retry, no DLQ). M9 adds the chain.
• cmd/telegramd/: new binary. Loads the active bot list from telegram_bots at startup, long-polls getUpdates, dispatches commands to the handler, and replies via sendMessage. One process per deployment; per-bot sharding can come later if needed.
• internal/telegram/: 3 files, ~640 LoC.
  • client.go — BotClient interface + HTTP impl.
  • commands.go — text → Command parser (/start, /subscribe, /unsubscribe, /preferences, /status, /mute, /unmute). /mute accepts 2h, 30m, 90s, or until 18:00.
  • handler.go — Command → DB updates. /start is atomic claim-by-invite-code; /subscribe upserts a subscription with the requested min_severity.
  • commands_test.go — 7 subtests, 100% pass.
• internal/routing/routing.go: the resolver's single CTE now UNION-ALLs an fcm_rows branch (joins on fcm_tokens) and a tg_rows branch (filters on telegram_chat_id IS NOT NULL). Same cost as M2; one extra row in the join key set. Hard-fail on zero targets unchanged.
• testfakes/faketgmd/: ~350 LoC. Fake Bot API server with /admin/queue (queue an incoming update), /admin/sent (read every sendMessage call), and /admin/reset (clear state). In-memory only. Used purely for the smoke test.
• docker-compose.yml + Dockerfile: add deliverd-telegram (port 8821), telegramd (port 8822), faketgmd (port 8830). Grafana is now 3001:3000 because :3000 is held by gogs on this host.
• M3_VERIFICATION.md: 8-step manual + automated walkthrough.
• M3_SMOKE_LOG.md: per-step results, honest flags.
• scripts/m3_smoke.sh: automated runner for steps 2–8.
• README + PROMPT + SPEC §23 bumped to "shipped 2026-06-14".

Live smoke results (all green):

• step 2: warning → Alice fcm + Alice telegram (Bob unlinked)
• step 3: critical → Alice fcm + Alice telegram + Bob fcm
• step 4: Bob /start acme-bob-002 via faketgmd admin queue → handler atomically claims the code, telegram_user_id=900002, bot reply "Linked. Welcome, Bob SRE."
• step 5: critical → Alice fcm+telegram + Bob fcm+telegram (Bob telegram now fires)
• step 6: inminent_colapse → Alice fcm+telegram + Bob fcm+telegram + Carol fcm (Carol never linked; quiet hours bypassed)
• step 7: Bob /preferences → bot reply "Your subscriptions: …"
• step 8: faketgmd received 8 sendMessage calls

14 deliveries across 4 unique alert_ids, 0 failures, 0 retries.

Per the user's M3 Q1/Q2/Q3 answers:

• Q1: faketgmd is shipped as a test fake (DONE; M5+ can add a webhook-based "real" mode).
• Q2: two per-channel binaries, deliverd renamed to deliverd-fcm (DONE; deliverd-telegram is its own binary).
• Q3: channel expansion on Bob and Carol's subscriptions (DONE; M2's ["fcm"] → M3's ["fcm","telegram"] via the seed migration).

What's NOT in M3 (and not supposed to be):

• real FCM auth (M9/M11)
• SMS, email, Slack, MS Teams, voice (M3+)
• webhook mode for the bot (M5/M9)
• bot token encryption at rest (M11, security milestone)
• retry + DLQ for telegram delivery (M9)
• per-company bot token resolution in deliverd-telegram (currently reads env-supplied default; M3+ looks up by company_id encoded in alert or subject)
• markdown/HTML formatting in telegram messages (M3.5+)
• per-rule priority semantics with "first match wins unless continue=true" (deferred from M2; the unioned-rules behavior holds for M3)

Pushed:

2026-06-14 — M4 shipped (MQTT ingest + per-source ACL)

What landed (~750 LoC Go + ~60 LoC config across these commits):

• internal/mqttclient/: new package. Thin wrapper around paho.MQTT that hides the token-on-publish option behind a single error-returning Publish(), sets consistent LastWill / MaxInflight / AutoReconnect defaults, and formats client_id as <service>-mqtt-<host> so EMQX /admin/clients shows them cleanly. Used by both cmd/ingestd (subscriber) and loadgen/cmd/mqtt (publisher).
• cmd/ingestd/mqtt.go: new file. The MQTT subscriber runs every message through the same processDeps.ProcessAlert pipeline as the HTTP POST handler — parse → validate → HMAC verify → rate-limit → dedupe → publish to NATS. The only MQTT-specific code is the topic parser (parseIncomingTopic → ba/<co>/<src>/incoming, 4 segments, last = incoming) and the envelope sniffer (accepts both {alert, auth} envelope and bare alert bodies for future broker-native clients).
• cmd/ingestd/main.go: wires up the MQTT path. New env vars BA_INGESTD_MQTT_BROKER, BA_INGESTD_MQTT_USERNAME, BA_INGESTD_MQTT_PASSWORD, BA_INGESTD_MQTT_SUBSCRIBE (ba/+/+/incoming default).
• internal/observability/metrics.go: adds the MQTTMessages CounterVec (labels: result=received | accepted | deduped | bad_topic | bad_signature | invalid_json | <other RejectReason>). Exposed as ba_ingestd_mqtt_messages_total.
• loadgen/cmd/mqtt/: new binary. Same severity mix, dedupe ratio, and burst mode as loadgen-http. Publishes to ba/<co>/<src>/incoming with the <src>-<co> user and HMAC secret for auth.
• deploy/emqx/acl.conf: Erlang-term ACL rules. Each source can only publish to its own ba/<co>/<src>/incoming; ingestd can subscribe to ba/+/+/incoming; default deny on #. Re-read on EMQX SIGHUP.
• deploy/emqx/auth-built-in-db-bootstrap.csv: per-username auth. <source_id>-<company_id> for sources, ingestd for the subscriber. Password == HMAC secret so the same secret serves both MQTT auth and per-message HMAC.
• deploy/emqx/README.md: explains the auth model, the env-var-vs-emqx.conf precedence (env vars win in EMQX 5.x), and the emqx_ctl listeners restart command for hot-reloading acl.conf.
• docker-compose.yml: switches the emqx service from volume-mounting emqx.conf (which EMQX 5.x rejects as a partial config with node.cookie required_field) to env-var config. The EMQX_AUTHENTICATION__1__* and EMQX_AUTHORIZATION__* env vars are the only way to inject chain-level config in 5.x. acl.conf and the bootstrap CSV stay as volume mounts (their files are partials, which is fine).
• scripts/m4_smoke.sh: end-to-end smoke. 5 steps (1 alert, 5 alerts with dedupe, bad sig, ACL violation, bad json). Builds loadgen-mqtt and the 3 failure-path test binaries into /tmp/ on first run. 12 deliveries, 0 failures, 3 consecutive green runs.
• M4_VERIFICATION.md + M4_SMOKE_LOG.md: the spec-style step-by-step + the live run results.

Why MQTT in v1: SPEC §18 calls for HTTP+MQTT as the v1 transports. M0–M3 ship HTTP; M4 ships MQTT. Sources that prefer a broker (Prometheus alertmanager webhook-bridge, Grafana, IoT) can publish to ba/<co>/<src>/incoming without writing a webhook client.

Why per-source ACL: the broker is the first line of defense. A source that gets compromised can only spam its own topic — which still goes through the HMAC gate at ingestd, so a forged message without the secret is rejected in step 4. ACL stops cross-tenant spam; HMAC stops forged messages; rate-limit stops flood; dedupe stops burst; M9 adds the circuit breaker and quarantine.

Why env-var config beats emqx.conf: EMQX 5.x's HOCON config is layered as emqx.conf → base.hocon → cluster.hocon → env vars (highest precedence). The double-underscore separator in env-var names maps to nested HOCON keys. emqx.conf is a full config — partial overrides get rejected with node.cookie required_field. The env-var approach lets us set just the auth + authz chains without restating the entire base.hocon.

What's NOT in M4 (and not supposed to be):

• Per-IP concurrency cap (M5 with WS, SPEC §22 layer 2)
• Circuit breaker + quarantine (M9)
• TLS to EMQX (M11, security milestone)
• Persistent sessions (M11)
• Per-company bot token resolution for sources that share a single EMQX user across companies (M3+ generalization)
• HTTP-style signature in a header (MQTT has no headers; the auth field in the JSON envelope is the equivalent)
• QoS 2 (M11, when persistent sessions land)

Pushed: bc907d9 on master (1 commit for the code; this PROMPT bump + README + SPEC bump + the EMQX env-var refactor consolidated into a single follow-up commit).