# Broad-Announce — Build Log (PROMPT)

> Decisions, lessons, blockers. Append-only. Update as we go.

## 2026-06-13 — kickoff

**Decided**
- Repo at `git3.techno-world.net/lrosales/broad-announce` (private).
- v1 stack: Go, PostgreSQL + Timescale, ClickHouse, NATS JetStream,
  Redis, EMQX (MQTT), Prometheus + Grafana, Loki. Deploy v1 = Docker
  Compose. v2 = K8s.
- Multi-tenancy = **shared infrastructure, strict app-level isolation**.
  No row-level security in v1; tenant filter on every query.
- FCM = **single shared project** for v1, schema supports per-company
  FCM project later (`companies.fcm_shared`).
- Telegram is a **first-class delivery channel** + a **management UI**
  (the bot is how users mute, subscribe, acknowledge).
- Severity taxonomy: `info | warning | critical | inminent_colapse`.
  Only `inminent_colapse` bypasses quiet hours.
- Dedupe: 60s window per `(source_id, dedupe_key)`, attach
  `dedupe_count` so user sees "×N in 60s" not N pushes.
- v1 capacity target in docker-compose: **5k alerts/sec sustained**.
  50k/sec is the design ceiling; the K8s + multi-broker work
  is what unlocks it.

**Open (was) → Resolved 2026-06-13**
- ~~Bot ↔ individual linking~~ → **admin-invites only**. Flow: admin
  creates `individuals` row + generates one-time invite code;
  user runs `/start <invite_code>` in the Telegram bot; bot
  matches the code, links `telegram_chat_id` to the
  pre-existing individual, and burns the code. Stricter path:
  prevents drive-by bot self-registration, keeps
  `individuals` auditable.
- ~~Localized titles~~ → **source-localized, pass through**. v1
  does NOT translate. Sources send pre-localized
  `title`/`body` strings (or use the locale on the
  subscription to key into their own lookup table before
  calling our API). `Alert` schema carries `locale` and
  `title`/`body` already-resolved. We can add a translation
  layer in v2 if customers ask.
- mTLS: schema supports it, but the docker-compose profile won't
  terminate client certs in v1. Documented as opt-in for
  enterprise sources.

**Lessons (already)**
- It's much cheaper to answer "what *can* this be?" with 30
  questions than to refactor later. Most of the SPEC's weight is
  in §11 (security) and §6 (recipient resolution) — those are
  the parts that are expensive to change after launch.
- For multi-tenant at 10k companies, the *one* design choice
  that compounds is the **subject layout in the broker**.
  `alerts.<company_id>` is fine; `alerts.<company_id>.<source_id>`
  would let us scale router consumer groups per source. Locked
  in §6 of ARCHITECTURE.md.

**2026-06-13 — added gRPC ingest (option a)**
- Decision: ship gRPC bidi-streaming as the **fourth** ingest
  protocol, scoped to **internal high-volume sources only**.
  Public SaaS webhooks stay on HTTP POST, browsers stay on
  WebSocket, IoT stays on MQTT.
- Rationale: typed schemas, HTTP/2 + protobuf, native backpressure,
  no reconnect-loop code, generated Go/Java/Python/Node clients.
  Cost: one more protocol to operate + a `buf generate` build step.
- Gated on **M11**, after M10 (load test). M10 now caps at
  **5k/s on docker-compose** (not 50k/s) — 50k/s is the design
  ceiling that requires K8s + multi-broker (M12 in v2).
- `StreamAlerts(Alert) → Ack` carries the same `dedupe_count`
  contract as HTTP/WS/MQTT.
- Proto lives at `proto/broadannounce/v1/ingest.proto`; server
  stub in `internal/grpcserver/`, reusable client in
  `internal/grpcclient/`. Auth = API key in metadata + optional
  mTLS. Per-stream rate limit + 256 in-flight cap = natural
  backpressure.

**2026-06-13 — M10 split: option C**
- M10 (in-cluster, real) stays at **5k/s on docker-compose**.
- Added **M10-bench**: 50k/s against the broker + router with
  the delivery tier stubbed. Proves the ceiling without
  committing to K8s. Green here means K8s is *implementable*,
  not required.
- Implication: we need traffic generators that can hit the
  in-cluster path at 5k/s (M10) and the broker+router path at
  50k/s (M10-bench) with the same source-protocol clients we
  ship. The `loadgen/` tool is the new home for that.

**2026-06-13 — three open questions resolved**
- (Q1) **Yes, build the fake-FCM / fake-Telegram / fake-SMS
  servers** (`testfakes/`). M10 must not burn 50k FCM credits.
- (Q2) **M10-bench delivery stub lives at the service level** —
  a no-op `deliverd` binary in the bench profile, not a NATS
  subject drop. More realistic: we exercise the real broker +
  router + the message shape `deliverd` would consume.
- (Q3) **`loadgen` ships as a single Docker image** with all
  four binaries; entrypoint picks one via the image's
  `command:` field in the compose / k8s spec.

**2026-06-13 — Source protection (throttling) is in**
- Spec §22 (new). Seven layers, evaluated in order per request:
  payload-size cap → per-IP conn cap → per-source token bucket
  → per-company token bucket → schema validate → broker
  circuit breaker → per-source quarantine.
- Per-source config additive to §4 schema
  (`rate_limit_per_sec`, `max_payload_bytes`,
  `max_concurrent_connections`, `quarantine_*`).
- Rejection-code matrix is the contract across HTTP/WS/MQTT/gRPC
  (e.g. per-source rate → 429+Retry-After / Ack{RATE_LIMITED} /
  MQTT reason 0x97 / close 1013).
- Rollout tied to milestones: M0 layer 3, M1 layers 1/4/5,
  M5 layer 2, M6 dedupe-aware shaping, M9 layers 6+7.
- Runaway-source test is part of **M10** exit (loadgen
  fault-injection: one source at 10× cap must not push p99
  for *other* sources above 5s).

**2026-06-13 — M0 shipped (12 commits, 2888 LoC)**

What landed:
- `cmd/{ingestd,routerd,deliverd,admind}/` — four Go service mains
- `cmd/ingestd/http.go` — HTTP POST handler implementing
  SPEC §22 layers 1, 3, 4, 5 + Stripe-style HMAC auth
- `internal/alert` — Alert v1 type + Validate() (183 LoC + 103
  LoC tests)
- `internal/broker` — NATS JetStream wrapper, three streams
  (ALERTS/DELIVERIES/DLQ) auto-created
- `internal/dedupe` — 60s SET NX EX + INCR (Redis-required tests
  pass against a host-local Redis)
- `internal/ratelimit` — per-second INCR bucket (Redis-required
  tests pass)
- `internal/observability` — slog + Prometheus registry, the
  IngestdMetrics struct matches SPEC §22 metric names
- `internal/httpserver` — shared /health + /metrics scaffold
- `internal/config` — env-driven Common + Ingestd
- `loadgen/cmd/http/` — `loadgen-http` with --mode normal,
  HMAC signing, 70/25/4/1 severity mix, dedupe-pct knob
- `loadgen/go.mod` — separate module per SPEC §21, replace
  directive points at the parent module
- `docker-compose.yml` + `Dockerfile` — single-host stack, all
  5 binaries in one image
- `M0_VERIFICATION.md` — 8-step smoke test
- `deploy/prometheus/prometheus.yml` — scrapes all 5 services

What's NOT in M0 (and not supposed to be):
- routerd/deliverd/admind business logic (M2/M3+)
- MQTT, gRPC, WebSocket ingest (M4/M5/M11)
- per-IP concurrency cap (M5)
- circuit breaker (M9)
- quarantine (M9)
- ClickHouse archive (M7)
- source registry in DB (M2)
- Postgres migrations (M2)

Module path: `git3.techno-world.net/lrosales/broad-announce`.
Loadgen module path: `git3.techno-world.net/lrosales/broad-announce/loadgen`.
All pushed: 7cd922c..49b2dba on master.

**2026-06-13 — Port convention**

Project rule: app HTTP services use 8800–8899 (ingestd 8800,
routerd 8801, deliverd 8802, admind 8803, loadgen metrics 8891,
fakefcmd 8820). Canonical ports stay (5432, 4222, 6379, 1883,
9090, 3000). Reason: 8080+ collides with workstation tooling.
SPEC §18 now has a port-conventions sub-section. Commit: d76aa0b.

**2026-06-13 — M1 code-complete (8 commits, awaiting live smoke)**

What landed:
- `migrations/001_init.up.sql` — companies, individuals,
  fcm_tokens (the M1 minimum schema; full SPEC §4 lands in M2
  as additional migrations)
- `migrations/002_deliveries.up.sql` — deliveries table
  (status: pending|sent|failed|dlq; payload jsonb for M8 replay)
- `migrations/seed.sql` — idempotent; 1 company, 1 individual,
  1 token
- `internal/postgres` — pgxpool wrapper with retry-on-startup
- `cmd/seed` — applies *.up.sql in lexical order, then seed.sql
- `testfakes/fakefcmd` — 70 lines, /health + /v1/.../messages:send,
  --fail-rate knob
- `internal/routing` — Resolver with ResolveTokens (M1 broadcast
  via single SQL join; M2 swaps for the rules engine)
- `cmd/routerd` — M1 main: subscribes to alerts.>, resolves
  recipients, publishes one deliveries.fcm.<co> per token
- `cmd/deliverd` — M1 main: subscribes to deliveries.fcm.>,
  builds FCM HTTP v1 message body (M3 swap is a no-op at this
  layer), posts to BA_FAKECMD_URL, writes a deliveries row
- Dockerfile builds 7 binaries
- docker-compose adds fakefcmd + seed (one-shot sidecar),
  wires BA_FAKECMD_URL into deliverd
- M1_VERIFICATION.md: 9-step smoke test

What we agreed to defer (per the user):
- Q1 seed: 1 company / 1 individual / 1 token — DONE
- Q2 router: M1 broadcast (no subscriptions) — DONE
- Q3 testfakes: only fakefcmd for M1 — DONE
- Q4 migrations: 3 tables in M1, expand in M2 — DONE

What's NOT in M1 (and not supposed to be):
- subscriptions, quiet hours, routing rules (M2)
- per-source allowed_targets (M2)
- real FCM (M3)
- other delivery channels (M3+)
- retry + DLQ from SPEC §9 (M3)
- ClickHouse / Timescale hypertables (M7)
- HMAC secret in DB (M2)
- per-IP cap, circuit breaker, quarantine (M5/M9)

Pushed: 264d284..9cf68a1 on master (5 commits for M1 code,
plus the d76aa0b port shift).
**The user still has to actually run `docker compose up` and the
M1 verification steps before M1 is fully done.**

**2026-06-13 — M2 shipped (recipient resolution, rules engine)**

What landed (3 commits, ~600 LoC Go + ~250 LoC SQL):
- `migrations/003_subscriptions_groups.up.sql` (and `.down.sql`):
  5 new tables — `sources`, `groups`, `group_members`,
  `subscriptions`, `routing_rules`. FKs to `companies` /
  `individuals`. Idempotent.
- `migrations/seed_m2.sql`: 2 more individuals (Bob, Carol), 1
  group (sre: Alice+Bob), 1 source row, 3 subscriptions
  covering the 3 scenarios, 1 routing rule.
- `cmd/seed/main.go`: runner now applies `seed.sql` then
  `seed_m2.sql` (lexically ordered) so re-running is safe.
- `internal/routing/routing.go`: full rewrite. New `Resolver`
  with `ResolveTargets(ctx, *alert.Alert) ([]Target, error)`.
  Single SQL round-trip via a 5-CTE query:
    1. src — the source row
    2. allowed_individual_ids — direct + group-expanded +
       broadcast
    3. rule_individual_ids — routing_rules where match_expr
       matches
    4. candidates — union of 2 and 3
    5. sub_expanded + filtered — subscriptions with
       min_severity, channel_mask, quiet-hours filter
  Final pass: Go-side quiet-hours check (with the
  inminent_colapse bypass) and channel='fcm' gate.
- `internal/alert/alert.go`: added `Severity.Rank()` and
  `MinSeverityRank()` helpers. Two new unit tests in
  `alert_test.go`.
- `internal/routing/routing_test.go`: new file, 10 subtests
  for `inQuietHours` (same-day, wrap-around, always-quiet,
  edge cases at window start/end).
- `cmd/routerd/main.go`: M2 main. Calls `ResolveTargets(alert)`,
  hard-fails on zero recipients (logs WARN, acks — no DLQ for
  M2; that's M3+). Emits the new envelope shape with
  `Channel` + `Endpoint` instead of the M1 `FCMToken`+`Locale`.
- `cmd/deliverd/main.go`: envelope struct updated to match
  the M2 routerd output. Only field renames; the FCM HTTP v1
  body shape is unchanged.
- `M2_VERIFICATION.md`: 7-step plan + hard-fail scenario.
- `scripts/m2_smoke.sh`: automated runner for steps 2-6.
- `M2_SMOKE_LOG.md`: per-step results, honest flags.

Live smoke results (all green):
- step 2: warning, db-prod-04 → Alice only
- step 3: critical, db-prod-04 → Alice + Bob
- step 4: critical, db-prod-03 (rule match) → Alice + Bob
- step 5: warning, db-prod-03 (rule match but min filter)
          → Alice only (Bob filtered by min_severity=critical)
- step 6: inminent_colapse → Alice + Bob + Carol
          (Carol's quiet_hours=00:00-23:59 bypassed)
- step 7: subscriptions paused → routerd logs
          `WARN msg="zero recipients, dropping"`, no delivery
9 deliveries, 0 failures, 0 retries.

Per the user's M2 Q1/Q2/Q3 answers:
- Q1: seed = 3 individuals + 1 group + 1 source + 4
  subscriptions covering all 4 scenarios (DONE; one of the 4
  was dropped to 3 because we folded "fcm with min=critical
  AND in quiet hours" into a single individual — the bypass
  test is still valid via Carol).
- Q2: hard-fail on zero recipients with a WARN log (DONE; the
  routerd logger is the notification path for M2; M3+ will
  add a `dlq.no_recipients.<company>` subject).
- Q3: (a) emit the row, drop the channel at the SQL filter
  (DONE; `WHERE f.channel = 'fcm'` in the resolver).

What's NOT in M2 (and not supposed to be):
- real FCM (M3)
- telegram, sms, email, slack, teams (M3+)
- retry + DLQ (M3)
- per-source HMAC secret in DB (M5 when mTLS / API-key
  path comes in)
- full match_expr language (M6+); M2 supports category,
  severity, data.*, all
- per-rule priority semantics with "first match wins
  unless continue=true" (M3+); M2 unions all matching rules
- routing rule "drop" target type (schema supports it, no
  test for it; the SQL filters it via the candidate union)
- Timezone support beyond UTC (M2 falls back to UTC if
  `time.LoadLocation(tz)` fails)

Pushed: <will fill in>