broad-announce/PROMPT.md

Broad-Announce — Build Log (PROMPT)

Decisions, lessons, blockers. Append-only. Update as we go.

2026-06-13 — kickoff

Decided

• Repo at git3.techno-world.net/lrosales/broad-announce (private).
• v1 stack: Go, PostgreSQL + Timescale, ClickHouse, NATS JetStream, Redis, EMQX (MQTT), Prometheus + Grafana, Loki. Deploy v1 = Docker Compose. v2 = K8s.
• Multi-tenancy = shared infrastructure, strict app-level isolation. No row-level security in v1; tenant filter on every query.
• FCM = single shared project for v1, schema supports per-company FCM project later (companies.fcm_shared).
• Telegram is a first-class delivery channel + a management UI (the bot is how users mute, subscribe, acknowledge).
• Severity taxonomy: info | warning | critical | inminent_colapse. Only inminent_colapse bypasses quiet hours.
• Dedupe: 60s window per (source_id, dedupe_key), attach dedupe_count so user sees "×N in 60s" not N pushes.
• v1 capacity target in docker-compose: 5k alerts/sec sustained. 50k/sec is the design ceiling; the K8s + multi-broker work is what unlocks it.

Open (was) → Resolved 2026-06-13

• Bot ↔ individual linking → admin-invites only. Flow: admin creates individuals row + generates one-time invite code; user runs /start <invite_code> in the Telegram bot; bot matches the code, links telegram_chat_id to the pre-existing individual, and burns the code. Stricter path: prevents drive-by bot self-registration, keeps individuals auditable.
• Localized titles → source-localized, pass through. v1 does NOT translate. Sources send pre-localized title/body strings (or use the locale on the subscription to key into their own lookup table before calling our API). Alert schema carries locale and title/body already-resolved. We can add a translation layer in v2 if customers ask.
• mTLS: schema supports it, but the docker-compose profile won't terminate client certs in v1. Documented as opt-in for enterprise sources.

Lessons (already)

• It's much cheaper to answer "what can this be?" with 30 questions than to refactor later. Most of the SPEC's weight is in §11 (security) and §6 (recipient resolution) — those are the parts that are expensive to change after launch.
• For multi-tenant at 10k companies, the one design choice that compounds is the subject layout in the broker. alerts.<company_id> is fine; alerts.<company_id>.<source_id> would let us scale router consumer groups per source. Locked in §6 of ARCHITECTURE.md.

2026-06-13 — added gRPC ingest (option a)

• Decision: ship gRPC bidi-streaming as the fourth ingest protocol, scoped to internal high-volume sources only. Public SaaS webhooks stay on HTTP POST, browsers stay on WebSocket, IoT stays on MQTT.
• Rationale: typed schemas, HTTP/2 + protobuf, native backpressure, no reconnect-loop code, generated Go/Java/Python/Node clients. Cost: one more protocol to operate + a buf generate build step.
• Gated on M11, after M10 (load test). M10 now caps at 5k/s on docker-compose (not 50k/s) — 50k/s is the design ceiling that requires K8s + multi-broker (M12 in v2).
• StreamAlerts(Alert) → Ack carries the same dedupe_count contract as HTTP/WS/MQTT.
• Proto lives at proto/broadannounce/v1/ingest.proto; server stub in internal/grpcserver/, reusable client in internal/grpcclient/. Auth = API key in metadata + optional mTLS. Per-stream rate limit + 256 in-flight cap = natural backpressure.

2026-06-13 — M10 split: option C

• M10 (in-cluster, real) stays at 5k/s on docker-compose.
• Added M10-bench: 50k/s against the broker + router with the delivery tier stubbed. Proves the ceiling without committing to K8s. Green here means K8s is implementable, not required.
• Implication: we need traffic generators that can hit the in-cluster path at 5k/s (M10) and the broker+router path at 50k/s (M10-bench) with the same source-protocol clients we ship. The loadgen/ tool is the new home for that.

2026-06-13 — three open questions resolved

• (Q1) Yes, build the fake-FCM / fake-Telegram / fake-SMS servers (testfakes/). M10 must not burn 50k FCM credits.
• (Q2) M10-bench delivery stub lives at the service level — a no-op deliverd binary in the bench profile, not a NATS subject drop. More realistic: we exercise the real broker + router + the message shape deliverd would consume.
• (Q3) loadgen ships as a single Docker image with all four binaries; entrypoint picks one via the image's command: field in the compose / k8s spec.

2026-06-13 — Source protection (throttling) is in

• Spec §22 (new). Seven layers, evaluated in order per request: payload-size cap → per-IP conn cap → per-source token bucket → per-company token bucket → schema validate → broker circuit breaker → per-source quarantine.
• Per-source config additive to §4 schema (rate_limit_per_sec, max_payload_bytes, max_concurrent_connections, quarantine_*).
• Rejection-code matrix is the contract across HTTP/WS/MQTT/gRPC (e.g. per-source rate → 429+Retry-After / Ack{RATE_LIMITED} / MQTT reason 0x97 / close 1013).
• Rollout tied to milestones: M0 layer 3, M1 layers 1/4/5, M5 layer 2, M6 dedupe-aware shaping, M9 layers 6+7.
• Runaway-source test is part of M10 exit (loadgen fault-injection: one source at 10× cap must not push p99 for other sources above 5s).

2026-06-13 — M0 shipped (12 commits, 2888 LoC)

What landed:

• cmd/{ingestd,routerd,deliverd,admind}/ — four Go service mains
• cmd/ingestd/http.go — HTTP POST handler implementing SPEC §22 layers 1, 3, 4, 5 + Stripe-style HMAC auth
• internal/alert — Alert v1 type + Validate() (183 LoC + 103 LoC tests)
• internal/broker — NATS JetStream wrapper, three streams (ALERTS/DELIVERIES/DLQ) auto-created
• internal/dedupe — 60s SET NX EX + INCR (Redis-required tests pass against a host-local Redis)
• internal/ratelimit — per-second INCR bucket (Redis-required tests pass)
• internal/observability — slog + Prometheus registry, the IngestdMetrics struct matches SPEC §22 metric names
• internal/httpserver — shared /health + /metrics scaffold
• internal/config — env-driven Common + Ingestd
• loadgen/cmd/http/ — loadgen-http with --mode normal, HMAC signing, 70/25/4/1 severity mix, dedupe-pct knob
• loadgen/go.mod — separate module per SPEC §21, replace directive points at the parent module
• docker-compose.yml + Dockerfile — single-host stack, all 5 binaries in one image
• M0_VERIFICATION.md — 8-step smoke test
• deploy/prometheus/prometheus.yml — scrapes all 5 services

What's NOT in M0 (and not supposed to be):

• routerd/deliverd/admind business logic (M2/M3+)
• MQTT, gRPC, WebSocket ingest (M4/M5/M11)
• per-IP concurrency cap (M5)
• circuit breaker (M9)
• quarantine (M9)
• ClickHouse archive (M7)
• source registry in DB (M2)
• Postgres migrations (M2)

Module path: git3.techno-world.net/lrosales/broad-announce. Loadgen module path: git3.techno-world.net/lrosales/broad-announce/loadgen. All pushed: 7cd922c..49b2dba on master.

2026-06-13 — Port convention

Project rule: app HTTP services use 8800–8899 (ingestd 8800, routerd 8801, deliverd 8802, admind 8803, loadgen metrics 8891, fakefcmd 8820). Canonical ports stay (5432, 4222, 6379, 1883, 9090, 3000). Reason: 8080+ collides with workstation tooling. SPEC §18 now has a port-conventions sub-section. Commit: d76aa0b.

2026-06-13 — M1 code-complete (8 commits, awaiting live smoke)

What landed:

• migrations/001_init.up.sql — companies, individuals, fcm_tokens (the M1 minimum schema; full SPEC §4 lands in M2 as additional migrations)
• migrations/002_deliveries.up.sql — deliveries table (status: pending|sent|failed|dlq; payload jsonb for M8 replay)
• migrations/seed.sql — idempotent; 1 company, 1 individual, 1 token
• internal/postgres — pgxpool wrapper with retry-on-startup
• cmd/seed — applies *.up.sql in lexical order, then seed.sql
• testfakes/fakefcmd — 70 lines, /health + /v1/.../messages:send, --fail-rate knob
• internal/routing — Resolver with ResolveTokens (M1 broadcast via single SQL join; M2 swaps for the rules engine)
• cmd/routerd — M1 main: subscribes to alerts.>, resolves recipients, publishes one deliveries.fcm. per token
• cmd/deliverd — M1 main: subscribes to deliveries.fcm.>, builds FCM HTTP v1 message body (M3 swap is a no-op at this layer), posts to BA_FAKECMD_URL, writes a deliveries row
• Dockerfile builds 7 binaries
• docker-compose adds fakefcmd + seed (one-shot sidecar), wires BA_FAKECMD_URL into deliverd
• M1_VERIFICATION.md: 9-step smoke test

What we agreed to defer (per the user):

• Q1 seed: 1 company / 1 individual / 1 token — DONE
• Q2 router: M1 broadcast (no subscriptions) — DONE
• Q3 testfakes: only fakefcmd for M1 — DONE
• Q4 migrations: 3 tables in M1, expand in M2 — DONE

What's NOT in M1 (and not supposed to be):

• subscriptions, quiet hours, routing rules (M2)
• per-source allowed_targets (M2)
• real FCM (M3)
• other delivery channels (M3+)
• retry + DLQ from SPEC §9 (M3)
• ClickHouse / Timescale hypertables (M7)
• HMAC secret in DB (M2)
• per-IP cap, circuit breaker, quarantine (M5/M9)

Pushed: 264d284..9cf68a1 on master (5 commits for M1 code, plus the d76aa0b port shift). The user still has to actually run docker compose up and the M1 verification steps before M1 is fully done.

2026-06-13 — M2 shipped (recipient resolution, rules engine)

What landed (3 commits, ~600 LoC Go + ~250 LoC SQL):

• migrations/003_subscriptions_groups.up.sql (and .down.sql): 5 new tables — sources, groups, group_members, subscriptions, routing_rules. FKs to companies / individuals. Idempotent.
• migrations/seed_m2.sql: 2 more individuals (Bob, Carol), 1 group (sre: Alice+Bob), 1 source row, 3 subscriptions covering the 3 scenarios, 1 routing rule.
• cmd/seed/main.go: runner now applies seed.sql then seed_m2.sql (lexically ordered) so re-running is safe.
• internal/routing/routing.go: full rewrite. New Resolver with ResolveTargets(ctx, *alert.Alert) ([]Target, error). Single SQL round-trip via a 5-CTE query:
  1. src — the source row
  2. allowed_individual_ids — direct + group-expanded + broadcast
  3. rule_individual_ids — routing_rules where match_expr matches
  4. candidates — union of 2 and 3
  5. sub_expanded + filtered — subscriptions with min_severity, channel_mask, quiet-hours filter Final pass: Go-side quiet-hours check (with the inminent_colapse bypass) and channel='fcm' gate.
• internal/alert/alert.go: added Severity.Rank() and MinSeverityRank() helpers. Two new unit tests in alert_test.go.
• internal/routing/routing_test.go: new file, 10 subtests for inQuietHours (same-day, wrap-around, always-quiet, edge cases at window start/end).
• cmd/routerd/main.go: M2 main. Calls ResolveTargets(alert), hard-fails on zero recipients (logs WARN, acks — no DLQ for M2; that's M3+). Emits the new envelope shape with Channel + Endpoint instead of the M1 FCMToken+Locale.
• cmd/deliverd/main.go: envelope struct updated to match the M2 routerd output. Only field renames; the FCM HTTP v1 body shape is unchanged.
• M2_VERIFICATION.md: 7-step plan + hard-fail scenario.
• scripts/m2_smoke.sh: automated runner for steps 2-6.
• M2_SMOKE_LOG.md: per-step results, honest flags.

Live smoke results (all green):

• step 2: warning, db-prod-04 → Alice only
• step 3: critical, db-prod-04 → Alice + Bob
• step 4: critical, db-prod-03 (rule match) → Alice + Bob
• step 5: warning, db-prod-03 (rule match but min filter) → Alice only (Bob filtered by min_severity=critical)
• step 6: inminent_colapse → Alice + Bob + Carol (Carol's quiet_hours=00:00-23:59 bypassed)
• step 7: subscriptions paused → routerd logs WARN msg="zero recipients, dropping", no delivery 9 deliveries, 0 failures, 0 retries.

Per the user's M2 Q1/Q2/Q3 answers:

• Q1: seed = 3 individuals + 1 group + 1 source + 4 subscriptions covering all 4 scenarios (DONE; one of the 4 was dropped to 3 because we folded "fcm with min=critical AND in quiet hours" into a single individual — the bypass test is still valid via Carol).
• Q2: hard-fail on zero recipients with a WARN log (DONE; the routerd logger is the notification path for M2; M3+ will add a dlq.no_recipients.<company> subject).
• Q3: (a) emit the row, drop the channel at the SQL filter (DONE; WHERE f.channel = 'fcm' in the resolver).

What's NOT in M2 (and not supposed to be):

• real FCM (M3)
• telegram, sms, email, slack, teams (M3+)
• retry + DLQ (M3)
• per-source HMAC secret in DB (M5 when mTLS / API-key path comes in)
• full match_expr language (M6+); M2 supports category, severity, data.*, all
• per-rule priority semantics with "first match wins unless continue=true" (M3+); M2 unions all matching rules
• routing rule "drop" target type (schema supports it, no test for it; the SQL filters it via the candidate union)
• Timezone support beyond UTC (M2 falls back to UTC if time.LoadLocation(tz) fails)

Pushed: 280e048 on master (3 commits: a29c2d8 + 1e9eca9 + 280e048).

2026-06-14 — M3 shipped (Telegram delivery + bot commands)

What landed (~2k LoC Go + ~80 LoC SQL across these commits):

• migrations/004_telegram.up.sql (+ .down.sql): telegram_bots table (one row per (company, bot); M3 supports one bot per company), and 4 new columns on individuals: telegram_chat_id, telegram_user_id, telegram_invite_code, mute_until. Two partial unique indexes for fast /start <code> lookup and "is this telegram_user_id already linked" check.
• migrations/seed_m3.sql: inserts the per-company telegram_bots row, sets telegram_invite_code on all three individuals, and pre-links Alice to a fake Telegram account (chat_id=1001, user_id=900001) so the very first POST exercises both fcm and telegram delivery paths. Also expands every subscription's channel_mask to ["fcm","telegram"] (M2 was ["fcm"]).
• cmd/seed/main.go: runner now applies seed.sql, seed_m2.sql, seed_m3.sql in lexical order. All three are idempotent.
• cmd/deliverd/ → cmd/deliverd-fcm/: rename. The M2 deliverd is now deliverd-fcm — per the user's M3 Q2 answer (two per-channel binaries, not one with a registry).
• cmd/deliverd-telegram/: new binary. Subscribes to deliveries.telegram.<company_id>, builds a severity-prefixed text message, posts to the Bot API's sendMessage endpoint, writes a deliveries row. Single-attempt (no retry, no DLQ). M9 adds the chain.
• cmd/telegramd/: new binary. Loads the active bot list from telegram_bots at startup, long-polls getUpdates, dispatches commands to the handler, and replies via sendMessage. One process per deployment; per-bot sharding can come later if needed.
• internal/telegram/: 3 files, ~640 LoC.
  • client.go — BotClient interface + HTTP impl.
  • commands.go — text → Command parser (/start, /subscribe, /unsubscribe, /preferences, /status, /mute, /unmute). /mute accepts 2h, 30m, 90s, or until 18:00.
  • handler.go — Command → DB updates. /start is atomic claim-by-invite-code; /subscribe upserts a subscription with the requested min_severity.
  • commands_test.go — 7 subtests, 100% pass.
• internal/routing/routing.go: the resolver's single CTE now UNION-ALLs an fcm_rows branch (joins on fcm_tokens) and a tg_rows branch (filters on telegram_chat_id IS NOT NULL). Same cost as M2; one extra row in the join key set. Hard-fail on zero targets unchanged.
• testfakes/faketgmd/: ~350 LoC. Fake Bot API server with /admin/queue (queue an incoming update), /admin/sent (read every sendMessage call), and /admin/reset (clear state). In-memory only. Used purely for the smoke test.
• docker-compose.yml + Dockerfile: add deliverd-telegram (port 8821), telegramd (port 8822), faketgmd (port 8830). Grafana is now 3001:3000 because :3000 is held by gogs on this host.
• M3_VERIFICATION.md: 8-step manual + automated walkthrough.
• M3_SMOKE_LOG.md: per-step results, honest flags.
• scripts/m3_smoke.sh: automated runner for steps 2–8.
• README + PROMPT + SPEC §23 bumped to "shipped 2026-06-14".

Live smoke results (all green):

• step 2: warning → Alice fcm + Alice telegram (Bob unlinked)
• step 3: critical → Alice fcm + Alice telegram + Bob fcm
• step 4: Bob /start acme-bob-002 via faketgmd admin queue → handler atomically claims the code, telegram_user_id=900002, bot reply "Linked. Welcome, Bob SRE."
• step 5: critical → Alice fcm+telegram + Bob fcm+telegram (Bob telegram now fires)
• step 6: inminent_colapse → Alice fcm+telegram + Bob fcm+telegram + Carol fcm (Carol never linked; quiet hours bypassed)
• step 7: Bob /preferences → bot reply "Your subscriptions: …"
• step 8: faketgmd received 8 sendMessage calls

14 deliveries across 4 unique alert_ids, 0 failures, 0 retries.

Per the user's M3 Q1/Q2/Q3 answers:

• Q1: faketgmd is shipped as a test fake (DONE; M5+ can add a webhook-based "real" mode).
• Q2: two per-channel binaries, deliverd renamed to deliverd-fcm (DONE; deliverd-telegram is its own binary).
• Q3: channel expansion on Bob and Carol's subscriptions (DONE; M2's ["fcm"] → M3's ["fcm","telegram"] via the seed migration).

What's NOT in M3 (and not supposed to be):

• real FCM auth (M9/M11)
• SMS, email, Slack, MS Teams, voice (M3+)
• webhook mode for the bot (M5/M9)
• bot token encryption at rest (M11, security milestone)
• retry + DLQ for telegram delivery (M9)
• per-company bot token resolution in deliverd-telegram (currently reads env-supplied default; M3+ looks up by company_id encoded in alert or subject)
• markdown/HTML formatting in telegram messages (M3.5+)
• per-rule priority semantics with "first match wins unless continue=true" (deferred from M2; the unioned-rules behavior holds for M3)

Pushed: