# Broad-Announce — Build Log (PROMPT)

> Decisions, lessons, blockers. Append-only. Update as we go.

## 2026-06-13 — kickoff

**Decided**
- Repo at `git3.techno-world.net/lrosales/broad-announce` (private).
- v1 stack: Go, PostgreSQL + Timescale, ClickHouse, NATS JetStream,
  Redis, EMQX (MQTT), Prometheus + Grafana, Loki. Deploy v1 = Docker
  Compose. v2 = K8s.
- Multi-tenancy = **shared infrastructure, strict app-level isolation**.
  No row-level security in v1; tenant filter on every query.
- FCM = **single shared project** for v1, schema supports per-company
  FCM project later (`companies.fcm_shared`).
- Telegram is a **first-class delivery channel** + a **management UI**
  (the bot is how users mute, subscribe, acknowledge).
- Severity taxonomy: `info | warning | critical | inminent_colapse`.
  Only `inminent_colapse` bypasses quiet hours.
- Dedupe: 60s window per `(source_id, dedupe_key)`, attach
  `dedupe_count` so user sees "×N in 60s" not N pushes.
- v1 capacity target in docker-compose: **5k alerts/sec sustained**.
  50k/sec is the design ceiling; the K8s + multi-broker work
  is what unlocks it.

**Open (was) → Resolved 2026-06-13**
- ~~Bot ↔ individual linking~~ → **admin-invites only**. Flow: admin
  creates `individuals` row + generates one-time invite code;
  user runs `/start <invite_code>` in the Telegram bot; bot
  matches the code, links `telegram_chat_id` to the
  pre-existing individual, and burns the code. Stricter path:
  prevents drive-by bot self-registration, keeps
  `individuals` auditable.
- ~~Localized titles~~ → **source-localized, pass through**. v1
  does NOT translate. Sources send pre-localized
  `title`/`body` strings (or use the locale on the
  subscription to key into their own lookup table before
  calling our API). `Alert` schema carries `locale` and
  `title`/`body` already-resolved. We can add a translation
  layer in v2 if customers ask.
- mTLS: schema supports it, but the docker-compose profile won't
  terminate client certs in v1. Documented as opt-in for
  enterprise sources.

**Lessons (already)**
- It's much cheaper to answer "what *can* this be?" with 30
  questions than to refactor later. Most of the SPEC's weight is
  in §11 (security) and §6 (recipient resolution) — those are
  the parts that are expensive to change after launch.
- For multi-tenant at 10k companies, the *one* design choice
  that compounds is the **subject layout in the broker**.
  `alerts.<company_id>` is fine; `alerts.<company_id>.<source_id>`
  would let us scale router consumer groups per source. Locked
  in §6 of ARCHITECTURE.md.

**2026-06-13 — added gRPC ingest (option a)**
- Decision: ship gRPC bidi-streaming as the **fourth** ingest
  protocol, scoped to **internal high-volume sources only**.
  Public SaaS webhooks stay on HTTP POST, browsers stay on
  WebSocket, IoT stays on MQTT.
- Rationale: typed schemas, HTTP/2 + protobuf, native backpressure,
  no reconnect-loop code, generated Go/Java/Python/Node clients.
  Cost: one more protocol to operate + a `buf generate` build step.
- Gated on **M11**, after M10 (load test). M10 now caps at
  **5k/s on docker-compose** (not 50k/s) — 50k/s is the design
  ceiling that requires K8s + multi-broker (M12 in v2).
- `StreamAlerts(Alert) → Ack` carries the same `dedupe_count`
  contract as HTTP/WS/MQTT.
- Proto lives at `proto/broadannounce/v1/ingest.proto`; server
  stub in `internal/grpcserver/`, reusable client in
  `internal/grpcclient/`. Auth = API key in metadata + optional
  mTLS. Per-stream rate limit + 256 in-flight cap = natural
  backpressure.

**2026-06-13 — M10 split: option C**
- M10 (in-cluster, real) stays at **5k/s on docker-compose**.
- Added **M10-bench**: 50k/s against the broker + router with
  the delivery tier stubbed. Proves the ceiling without
  committing to K8s. Green here means K8s is *implementable*,
  not required.
- Implication: we need traffic generators that can hit the
  in-cluster path at 5k/s (M10) and the broker+router path at
  50k/s (M10-bench) with the same source-protocol clients we
  ship. The `loadgen/` tool is the new home for that.

**2026-06-13 — three open questions resolved**
- (Q1) **Yes, build the fake-FCM / fake-Telegram / fake-SMS
  servers** (`testfakes/`). M10 must not burn 50k FCM credits.
- (Q2) **M10-bench delivery stub lives at the service level** —
  a no-op `deliverd` binary in the bench profile, not a NATS
  subject drop. More realistic: we exercise the real broker +
  router + the message shape `deliverd` would consume.
- (Q3) **`loadgen` ships as a single Docker image** with all
  four binaries; entrypoint picks one via the image's
  `command:` field in the compose / k8s spec.

**2026-06-13 — Source protection (throttling) is in**
- Spec §22 (new). Seven layers, evaluated in order per request:
  payload-size cap → per-IP conn cap → per-source token bucket
  → per-company token bucket → schema validate → broker
  circuit breaker → per-source quarantine.
- Per-source config additive to §4 schema
  (`rate_limit_per_sec`, `max_payload_bytes`,
  `max_concurrent_connections`, `quarantine_*`).
- Rejection-code matrix is the contract across HTTP/WS/MQTT/gRPC
  (e.g. per-source rate → 429+Retry-After / Ack{RATE_LIMITED} /
  MQTT reason 0x97 / close 1013).
- Rollout tied to milestones: M0 layer 3, M1 layers 1/4/5,
  M5 layer 2, M6 dedupe-aware shaping, M9 layers 6+7.
- Runaway-source test is part of **M10** exit (loadgen
  fault-injection: one source at 10× cap must not push p99
  for *other* sources above 5s).

**2026-06-13 — M0 shipped (12 commits, 2888 LoC)**

What landed:
- `cmd/{ingestd,routerd,deliverd,admind}/` — four Go service mains
- `cmd/ingestd/http.go` — HTTP POST handler implementing
  SPEC §22 layers 1, 3, 4, 5 + Stripe-style HMAC auth
- `internal/alert` — Alert v1 type + Validate() (183 LoC + 103
  LoC tests)
- `internal/broker` — NATS JetStream wrapper, three streams
  (ALERTS/DELIVERIES/DLQ) auto-created
- `internal/dedupe` — 60s SET NX EX + INCR (Redis-required tests
  pass against a host-local Redis)
- `internal/ratelimit` — per-second INCR bucket (Redis-required
  tests pass)
- `internal/observability` — slog + Prometheus registry, the
  IngestdMetrics struct matches SPEC §22 metric names
- `internal/httpserver` — shared /health + /metrics scaffold
- `internal/config` — env-driven Common + Ingestd
- `loadgen/cmd/http/` — `loadgen-http` with --mode normal,
  HMAC signing, 70/25/4/1 severity mix, dedupe-pct knob
- `loadgen/go.mod` — separate module per SPEC §21, replace
  directive points at the parent module
- `docker-compose.yml` + `Dockerfile` — single-host stack, all
  5 binaries in one image
- `M0_VERIFICATION.md` — 8-step smoke test
- `deploy/prometheus/prometheus.yml` — scrapes all 5 services

What's NOT in M0 (and not supposed to be):
- routerd/deliverd/admind business logic (M2/M3+)
- MQTT, gRPC, WebSocket ingest (M4/M5/M11)
- per-IP concurrency cap (M5)
- circuit breaker (M9)
- quarantine (M9)
- ClickHouse archive (M7)
- source registry in DB (M2)
- Postgres migrations (M2)

Module path: `git3.techno-world.net/lrosales/broad-announce`.
Loadgen module path: `git3.techno-world.net/lrosales/broad-announce/loadgen`.
All pushed: 7cd922c..49b2dba on master.

**2026-06-13 — Port convention**

Project rule: app HTTP services use 8800–8899 (ingestd 8800,
routerd 8801, deliverd 8802, admind 8803, loadgen metrics 8891,
fakefcmd 8820). Canonical ports stay (5432, 4222, 6379, 1883,
9090, 3000). Reason: 8080+ collides with workstation tooling.
SPEC §18 now has a port-conventions sub-section. Commit: d76aa0b.

**2026-06-13 — M1 code-complete (8 commits, awaiting live smoke)**

What landed:
- `migrations/001_init.up.sql` — companies, individuals,
  fcm_tokens (the M1 minimum schema; full SPEC §4 lands in M2
  as additional migrations)
- `migrations/002_deliveries.up.sql` — deliveries table
  (status: pending|sent|failed|dlq; payload jsonb for M8 replay)
- `migrations/seed.sql` — idempotent; 1 company, 1 individual,
  1 token
- `internal/postgres` — pgxpool wrapper with retry-on-startup
- `cmd/seed` — applies *.up.sql in lexical order, then seed.sql
- `testfakes/fakefcmd` — 70 lines, /health + /v1/.../messages:send,
  --fail-rate knob
- `internal/routing` — Resolver with ResolveTokens (M1 broadcast
  via single SQL join; M2 swaps for the rules engine)
- `cmd/routerd` — M1 main: subscribes to alerts.>, resolves
  recipients, publishes one deliveries.fcm.<co> per token
- `cmd/deliverd` — M1 main: subscribes to deliveries.fcm.>,
  builds FCM HTTP v1 message body (M3 swap is a no-op at this
  layer), posts to BA_FAKECMD_URL, writes a deliveries row
- Dockerfile builds 7 binaries
- docker-compose adds fakefcmd + seed (one-shot sidecar),
  wires BA_FAKECMD_URL into deliverd
- M1_VERIFICATION.md: 9-step smoke test

What we agreed to defer (per the user):
- Q1 seed: 1 company / 1 individual / 1 token — DONE
- Q2 router: M1 broadcast (no subscriptions) — DONE
- Q3 testfakes: only fakefcmd for M1 — DONE
- Q4 migrations: 3 tables in M1, expand in M2 — DONE

What's NOT in M1 (and not supposed to be):
- subscriptions, quiet hours, routing rules (M2)
- per-source allowed_targets (M2)
- real FCM (M3)
- other delivery channels (M3+)
- retry + DLQ from SPEC §9 (M3)
- ClickHouse / Timescale hypertables (M7)
- HMAC secret in DB (M2)
- per-IP cap, circuit breaker, quarantine (M5/M9)

Pushed: 264d284..9cf68a1 on master (5 commits for M1 code,
plus the d76aa0b port shift).
**The user still has to actually run `docker compose up` and the
M1 verification steps before M1 is fully done.**

**2026-06-13 — M2 shipped (recipient resolution, rules engine)**

What landed (3 commits, ~600 LoC Go + ~250 LoC SQL):
- `migrations/003_subscriptions_groups.up.sql` (and `.down.sql`):
  5 new tables — `sources`, `groups`, `group_members`,
  `subscriptions`, `routing_rules`. FKs to `companies` /
  `individuals`. Idempotent.
- `migrations/seed_m2.sql`: 2 more individuals (Bob, Carol), 1
  group (sre: Alice+Bob), 1 source row, 3 subscriptions
  covering the 3 scenarios, 1 routing rule.
- `cmd/seed/main.go`: runner now applies `seed.sql` then
  `seed_m2.sql` (lexically ordered) so re-running is safe.
- `internal/routing/routing.go`: full rewrite. New `Resolver`
  with `ResolveTargets(ctx, *alert.Alert) ([]Target, error)`.
  Single SQL round-trip via a 5-CTE query:
    1. src — the source row
    2. allowed_individual_ids — direct + group-expanded +
       broadcast
    3. rule_individual_ids — routing_rules where match_expr
       matches
    4. candidates — union of 2 and 3
    5. sub_expanded + filtered — subscriptions with
       min_severity, channel_mask, quiet-hours filter
  Final pass: Go-side quiet-hours check (with the
  inminent_colapse bypass) and channel='fcm' gate.
- `internal/alert/alert.go`: added `Severity.Rank()` and
  `MinSeverityRank()` helpers. Two new unit tests in
  `alert_test.go`.
- `internal/routing/routing_test.go`: new file, 10 subtests
  for `inQuietHours` (same-day, wrap-around, always-quiet,
  edge cases at window start/end).
- `cmd/routerd/main.go`: M2 main. Calls `ResolveTargets(alert)`,
  hard-fails on zero recipients (logs WARN, acks — no DLQ for
  M2; that's M3+). Emits the new envelope shape with
  `Channel` + `Endpoint` instead of the M1 `FCMToken`+`Locale`.
- `cmd/deliverd/main.go`: envelope struct updated to match
  the M2 routerd output. Only field renames; the FCM HTTP v1
  body shape is unchanged.
- `M2_VERIFICATION.md`: 7-step plan + hard-fail scenario.
- `scripts/m2_smoke.sh`: automated runner for steps 2-6.
- `M2_SMOKE_LOG.md`: per-step results, honest flags.

Live smoke results (all green):
- step 2: warning, db-prod-04 → Alice only
- step 3: critical, db-prod-04 → Alice + Bob
- step 4: critical, db-prod-03 (rule match) → Alice + Bob
- step 5: warning, db-prod-03 (rule match but min filter)
          → Alice only (Bob filtered by min_severity=critical)
- step 6: inminent_colapse → Alice + Bob + Carol
          (Carol's quiet_hours=00:00-23:59 bypassed)
- step 7: subscriptions paused → routerd logs
          `WARN msg="zero recipients, dropping"`, no delivery
9 deliveries, 0 failures, 0 retries.

Per the user's M2 Q1/Q2/Q3 answers:
- Q1: seed = 3 individuals + 1 group + 1 source + 4
  subscriptions covering all 4 scenarios (DONE; one of the 4
  was dropped to 3 because we folded "fcm with min=critical
  AND in quiet hours" into a single individual — the bypass
  test is still valid via Carol).
- Q2: hard-fail on zero recipients with a WARN log (DONE; the
  routerd logger is the notification path for M2; M3+ will
  add a `dlq.no_recipients.<company>` subject).
- Q3: (a) emit the row, drop the channel at the SQL filter
  (DONE; `WHERE f.channel = 'fcm'` in the resolver).

What's NOT in M2 (and not supposed to be):
- real FCM (M3)
- telegram, sms, email, slack, teams (M3+)
- retry + DLQ (M3)
- per-source HMAC secret in DB (M5 when mTLS / API-key
  path comes in)
- full match_expr language (M6+); M2 supports category,
  severity, data.*, all
- per-rule priority semantics with "first match wins
  unless continue=true" (M3+); M2 unions all matching rules
- routing rule "drop" target type (schema supports it, no
  test for it; the SQL filters it via the candidate union)
- Timezone support beyond UTC (M2 falls back to UTC if
  `time.LoadLocation(tz)` fails)

Pushed: 280e048 on master (3 commits: a29c2d8 + 1e9eca9 + 280e048).


**2026-06-14 — M3 shipped (Telegram delivery + bot commands)**

What landed (~2k LoC Go + ~80 LoC SQL across these commits):
- `migrations/004_telegram.up.sql` (+ `.down.sql`):
  `telegram_bots` table (one row per (company, bot); M3
  supports one bot per company), and 4 new columns on
  `individuals`: `telegram_chat_id`, `telegram_user_id`,
  `telegram_invite_code`, `mute_until`. Two partial unique
  indexes for fast `/start <code>` lookup and "is this
  telegram_user_id already linked" check.
- `migrations/seed_m3.sql`: inserts the per-company
  `telegram_bots` row, sets `telegram_invite_code` on all
  three individuals, and **pre-links Alice** to a fake
  Telegram account (chat_id=1001, user_id=900001) so the
  very first POST exercises both fcm and telegram delivery
  paths. Also expands every subscription's `channel_mask`
  to `["fcm","telegram"]` (M2 was `["fcm"]`).
- `cmd/seed/main.go`: runner now applies `seed.sql`,
  `seed_m2.sql`, `seed_m3.sql` in lexical order. All three
  are idempotent.
- `cmd/deliverd/` → `cmd/deliverd-fcm/`: rename. The M2
  `deliverd` is now `deliverd-fcm` — per the user's M3 Q2
  answer (two per-channel binaries, not one with a registry).
- `cmd/deliverd-telegram/`: new binary. Subscribes to
  `deliveries.telegram.<company_id>`, builds a
  severity-prefixed text message, posts to the Bot API's
  `sendMessage` endpoint, writes a `deliveries` row.
  Single-attempt (no retry, no DLQ). M9 adds the chain.
- `cmd/telegramd/`: new binary. Loads the active bot list
  from `telegram_bots` at startup, long-polls
  `getUpdates`, dispatches commands to the handler, and
  replies via `sendMessage`. One process per deployment;
  per-bot sharding can come later if needed.
- `internal/telegram/`: 3 files, ~640 LoC.
  - `client.go` — BotClient interface + HTTP impl.
  - `commands.go` — text → Command parser
    (`/start`, `/subscribe`, `/unsubscribe`,
    `/preferences`, `/status`, `/mute`, `/unmute`).
    `/mute` accepts `2h`, `30m`, `90s`, or `until 18:00`.
  - `handler.go` — Command → DB updates. `/start` is
    atomic claim-by-invite-code; `/subscribe` upserts a
    subscription with the requested `min_severity`.
  - `commands_test.go` — 7 subtests, 100% pass.
- `internal/routing/routing.go`: the resolver's single CTE
  now UNION-ALLs an `fcm_rows` branch (joins on
  `fcm_tokens`) and a `tg_rows` branch (filters on
  `telegram_chat_id IS NOT NULL`). Same cost as M2; one
  extra row in the join key set. Hard-fail on zero
  targets unchanged.
- `testfakes/faketgmd/`: ~350 LoC. Fake Bot API server
  with `/admin/queue` (queue an incoming update),
  `/admin/sent` (read every sendMessage call), and
  `/admin/reset` (clear state). In-memory only. Used
  purely for the smoke test.
- `docker-compose.yml` + `Dockerfile`: add
  `deliverd-telegram` (port 8821), `telegramd` (port 8822),
  `faketgmd` (port 8830). Grafana is now `3001:3000`
  because :3000 is held by gogs on this host.
- `M3_VERIFICATION.md`: 8-step manual + automated walkthrough.
- `M3_SMOKE_LOG.md`: per-step results, honest flags.
- `scripts/m3_smoke.sh`: automated runner for steps 2–8.
- README + PROMPT + SPEC §23 bumped to "shipped 2026-06-14".

Live smoke results (all green):
- step 2: warning → Alice fcm + Alice telegram (Bob unlinked)
- step 3: critical → Alice fcm + Alice telegram + Bob fcm
- step 4: Bob `/start acme-bob-002` via faketgmd admin queue
          → handler atomically claims the code, telegram_user_id=900002,
          bot reply "Linked. Welcome, Bob SRE."
- step 5: critical → Alice fcm+telegram + Bob fcm+telegram (Bob
          telegram now fires)
- step 6: inminent_colapse → Alice fcm+telegram + Bob fcm+telegram
          + Carol fcm (Carol never linked; quiet hours bypassed)
- step 7: Bob `/preferences` → bot reply "Your subscriptions: …"
- step 8: faketgmd received 8 sendMessage calls

14 deliveries across 4 unique alert_ids, 0 failures, 0 retries.

Per the user's M3 Q1/Q2/Q3 answers:
- Q1: faketgmd is shipped as a test fake (DONE; M5+ can add a
  webhook-based "real" mode).
- Q2: two per-channel binaries, deliverd renamed to
  deliverd-fcm (DONE; deliverd-telegram is its own binary).
- Q3: channel expansion on Bob and Carol's subscriptions
  (DONE; M2's `["fcm"]` → M3's `["fcm","telegram"]` via the
  seed migration).

What's NOT in M3 (and not supposed to be):
- real FCM auth (M9/M11)
- SMS, email, Slack, MS Teams, voice (M3+)
- webhook mode for the bot (M5/M9)
- bot token encryption at rest (M11, security milestone)
- retry + DLQ for telegram delivery (M9)
- per-company bot token resolution in deliverd-telegram
  (currently reads env-supplied default; M3+ looks up by
  company_id encoded in alert or subject)
- markdown/HTML formatting in telegram messages (M3.5+)
- per-rule priority semantics with "first match wins
  unless continue=true" (deferred from M2; the unioned-rules
  behavior holds for M3)

Pushed: <M3 commits land on master via 3 commits mirroring
M0/M1/M2's pattern — code, verified, bump.>


**2026-06-14 — M4 shipped (MQTT ingest + per-source ACL)**

What landed (~750 LoC Go + ~60 LoC config across these commits):
- `internal/mqttclient/`: new package. Thin wrapper around
  paho.MQTT that hides the token-on-publish option behind a
  single error-returning `Publish()`, sets consistent
  LastWill / MaxInflight / AutoReconnect defaults, and
  formats `client_id` as `<service>-mqtt-<host>` so EMQX
  `/admin/clients` shows them cleanly. Used by both
  `cmd/ingestd` (subscriber) and `loadgen/cmd/mqtt` (publisher).
- `cmd/ingestd/mqtt.go`: new file. The MQTT subscriber runs
  every message through the same `processDeps.ProcessAlert`
  pipeline as the HTTP POST handler — parse → validate →
  HMAC verify → rate-limit → dedupe → publish to NATS. The
  only MQTT-specific code is the topic parser
  (`parseIncomingTopic` → `ba/<co>/<src>/incoming`, 4
  segments, last = `incoming`) and the envelope sniffer
  (accepts both `{alert, auth}` envelope and bare alert
  bodies for future broker-native clients).
- `cmd/ingestd/main.go`: wires up the MQTT path. New env vars
  `BA_INGESTD_MQTT_BROKER`, `BA_INGESTD_MQTT_USERNAME`,
  `BA_INGESTD_MQTT_PASSWORD`, `BA_INGESTD_MQTT_SUBSCRIBE`
  (`ba/+/+/incoming` default).
- `internal/observability/metrics.go`: adds the
  `MQTTMessages` CounterVec (labels: `result=received |
  accepted | deduped | bad_topic | bad_signature |
  invalid_json | <other RejectReason>`). Exposed as
  `ba_ingestd_mqtt_messages_total`.
- `loadgen/cmd/mqtt/`: new binary. Same severity mix, dedupe
  ratio, and burst mode as `loadgen-http`. Publishes to
  `ba/<co>/<src>/incoming` with the `<src>-<co>` user and
  HMAC secret for auth.
- `deploy/emqx/acl.conf`: Erlang-term ACL rules. Each source
  can only publish to its own `ba/<co>/<src>/incoming`;
  `ingestd` can subscribe to `ba/+/+/incoming`; default
  deny on `#`. Re-read on EMQX SIGHUP.
- `deploy/emqx/auth-built-in-db-bootstrap.csv`: per-username
  auth. `<source_id>-<company_id>` for sources, `ingestd`
  for the subscriber. Password == HMAC secret so the same
  secret serves both MQTT auth and per-message HMAC.
- `deploy/emqx/README.md`: explains the auth model, the
  env-var-vs-emqx.conf precedence (env vars win in EMQX
  5.x), and the `emqx_ctl listeners restart` command for
  hot-reloading `acl.conf`.
- `docker-compose.yml`: switches the `emqx` service from
  volume-mounting `emqx.conf` (which EMQX 5.x rejects as a
  *partial* config with `node.cookie required_field`) to
  env-var config. The `EMQX_AUTHENTICATION__1__*` and
  `EMQX_AUTHORIZATION__*` env vars are the only way to
  inject chain-level config in 5.x. `acl.conf` and the
  bootstrap CSV stay as volume mounts (their files are
  partials, which is fine).
- `scripts/m4_smoke.sh`: end-to-end smoke. 5 steps (1 alert,
  5 alerts with dedupe, bad sig, ACL violation, bad json).
  Builds `loadgen-mqtt` and the 3 failure-path test
  binaries into `/tmp/` on first run. 12 deliveries, 0
  failures, 3 consecutive green runs.
- `M4_VERIFICATION.md` + `M4_SMOKE_LOG.md`: the spec-style
  step-by-step + the live run results.

**Why MQTT in v1:** SPEC §18 calls for HTTP+MQTT as the
v1 transports. M0–M3 ship HTTP; M4 ships MQTT. Sources that
prefer a broker (Prometheus alertmanager webhook-bridge,
Grafana, IoT) can publish to `ba/<co>/<src>/incoming`
without writing a webhook client.

**Why per-source ACL:** the broker is the first line of
defense. A source that gets compromised can only spam its
own topic — which still goes through the HMAC gate at
ingestd, so a forged message without the secret is rejected
in step 4. ACL stops *cross-tenant* spam; HMAC stops
*forged messages*; rate-limit stops *flood*; dedupe stops
*burst*; M9 adds the circuit breaker and quarantine.

**Why env-var config beats emqx.conf:** EMQX 5.x's HOCON
config is layered as `emqx.conf → base.hocon →
cluster.hocon → env vars (highest precedence)`. The
double-underscore separator in env-var names maps to nested
HOCON keys. `emqx.conf` is a *full* config — partial
overrides get rejected with `node.cookie required_field`.
The env-var approach lets us set just the auth + authz
chains without restating the entire base.hocon.

What's NOT in M4 (and not supposed to be):
- Per-IP concurrency cap (M5 with WS, SPEC §22 layer 2)
- Circuit breaker + quarantine (M9)
- TLS to EMQX (M11, security milestone)
- Persistent sessions (M11)
- Per-company bot token resolution for sources that share
  a single EMQX user across companies (M3+ generalization)
- HTTP-style signature in a header (MQTT has no headers;
  the `auth` field in the JSON envelope is the equivalent)
- QoS 2 (M11, when persistent sessions land)

Pushed: bc907d9 on master (1 commit for the code; this PROMPT
bump + README + SPEC bump + the EMQX env-var refactor
consolidated into a single follow-up commit).

**2026-06-14 — M5 shipped (WebSocket ingest + live tail + per-IP cap)**

What landed (~650 LoC Go + 2 SQL files across these commits):
- `migrations/005_ws.up.sql`: adds `max_concurrent_connections`
  column to `sources` and `companies` (defaults 32 and 64).
- `internal/concurrency/perip.go`: tiny `sync.Map[ip]atomic.Int64`
  with `Acquire/Release/InUse` and a janitor goroutine that
  prunes entries idle > 5 min. No Redis on this hot path —
  would self-DoS.
- `internal/tailhub/hub.go`: in-process pub/sub. `Subscribe`
  returns a `*Subscription` with a buffered `chan *Event`
  (size 64) and a per-sub `Drops` counter; `Publish` is
  best-effort, drops on full. `Filter.CompanyID` matches
  single-company or all (empty).
- `internal/wsclient/client.go`: thin `gorilla/websocket`
  wrapper. `Connect` does the auth round-trip and stashes
  the auth reply; `SendAlert` does a single WS write; `Close`
  sends `CloseMessage(1000)` then TCP close. Used by
  `loadgen-ws` and the smoke drivers.
- `loadgen/cmd/ws/main.go`: `loadgen-ws` binary. Default
  `normal` profile (30% dedupe). 3 modes: `normal`, `burst`.
- `loadgen/cmd/m5drivers/tail/main.go`: `/tmp/m5-tail-test`
  driver used by `scripts/m5_smoke.sh` steps 6 + 7.
- `cmd/ingestd/ws.go`: WS ingest at `GET /v1/ingest/ws`.
  Auth via first frame `{api_key}`. The handler does
  per-IP `Acquire` (rejects + 503 on cap exceeded), then
  loops on alert frames, sniffs `{alert, auth}` envelope
  vs bare body, and calls the shared `scoped.ProcessAlert`.
  Per-IP `Release` runs in the handler's defer.
- `cmd/ingestd/wstail.go`: live tail at `GET /v1/tail/ws`.
  Token via `Authorization: Bearer`, `X-BA-Tail-Token`, or
  `?token=`. Optional `?company_id=` filter. **Subscribe
  happens BEFORE the upgrade** so events in the
  dial→subscribe window are not lost (a real bug we
  caught in M5 verification — see PROMPT bug #1 below).
- `cmd/ingestd/process.go`: `processDeps` now has
  `Tail *tailhub.Hub` and `Transport string`. `ProcessAlert`
  calls `d.Tail.Publish(tailhub.FromAlert(&a, d.Transport))`
  on the `accepted` path, so the tail fans out alerts
  from HTTP, MQTT, and WS through the same hub.
- `cmd/ingestd/main.go`: wires `concurrency.NewPerIP`,
  `tailhub.NewHub`, and the two new dep structs.
- `internal/observability/metrics.go`: 5 new metrics —
  `WSMessages`, `WSConnections`, `ConnectionRejected`,
  `TailSubscribers`, `TailDropped` — all registered with
  the ingestd service label.
- `docker-compose.yml`: `BA_INGESTD_TAIL_TOKEN` and
  `BA_INGESTD_MAX_CONCURRENT_PER_IP` env vars on ingestd.
- `scripts/m5_smoke.sh`: 7-step live driver. Auto-builds
  `/tmp/loadgen-ws`, `/tmp/m5-tail-test`, `/tmp/m5-perip-test`,
  `/tmp/m5-badkey` on first run.

Bug #1 — Tail subscribe vs upgrade race
The first M5 verification run showed the live tail seeing
`subs=0` at publish time even though the tail's gauge
was 1. Root cause: `Subscribe` was called *after* the
WS upgrade, leaving a window where the client thought
it was connected but ingestd hadn't subscribed yet.
The WS alert sent during that window was published to
0 subscribers. Fix: subscribe first, *then* upgrade
(so the client's dial-completion implies the subscription
is in place). This is documented inline in
`cmd/ingestd/wstail.go`.

Bug #2 — Tail defer ordering
The first run of the gauge-update defer fired while the
subscription was still in the hub's map (defers run LIFO,
so the metric defer fired *before* the unsubscribe
defer). Result: the gauge stayed at 1 after the
client disconnected. Fix: combine unsubscribe + gauge
update into a single defer that runs after the stream
loop exits.

Bug #3 — Badkey envelope shape
The first M5 step-4 test sent the HMAC as
`auth: {"hmac": "v1=..."}` (a map). The server's envelope
parser declared `auth` as a `string`, so the unmarshal
failed, the parser fell back to using the whole envelope
as the alert body, and validation rejected the alert
with "company_id missing". Fix: send `auth` as a string
`"t=...,v1=..."`, same shape as the MQTT envelope.
`scripts/m4_smoke.sh` already had this correct — copy-
pasted the pattern into M5.

What's NOT in M5 (and not supposed to be):
- HTTP `POST /v1/ingest` per-IP cap (M10 — the M5-bump
  in SPEC §22; we ship the gate now and the wire-up to
  HTTP is its own small PR)
- JWT-based tail auth (M11 security milestone)
- Cross-node tail fan-out via Redis pub/sub (M15+ when
  ingestd runs as a cluster)
- A "send test alert" UI button in the tail (deferred —
  operators can `wscat` and a `loadgen-ws --count 1`)

Pushed: 3 commits on master mirroring M0–M4's pattern:
1. M5(1/3): code (migrations + 3 new packages + ws.go +
   wstail.go + process.go + main.go + metrics + loadgen-ws)
2. M5(2/3): verified (M5_VERIFICATION.md +
   scripts/m5_smoke.sh + M5_SMOKE_LOG.md, 3 consecutive
   green runs, 13/13 checks each, +36 deliveries
   cumulative)
3. M5(3/3): this PROMPT bump + README + SPEC §23.

**2026-06-14 — M6 shipped (Dedupe + ×N suffix + sliding TTL + free-for-dupes)**

What landed (~350 LoC Go + 1 SQL-friendly column + 2 new
metrics across these commits):
- `internal/dedupe/dedupe.go`: rewritten with a single Lua
  script that does SET-NX-or-INCR-EXPIRE atomically. The
  EXPIRE call on every duplicate is what makes the window
  slide — a steady stream of duplicates keeps the key alive.
  Sub-second windows in tests are handled by `math.ceil`
  in Lua (Redis EX requires an int).
- `internal/observability/maxseen.go`: a new
  `MaxSeen` type that tracks per-key monotonic max. The
  gauge API doesn't expose Get(), so the canonical
  Prometheus pattern is to use a custom Collector; instead
  we keep the max in a `sync.Map[source]uint32` and
  `Set()` the gauge only on growth. CAS-safe under
  concurrent writers.
- `cmd/ingestd/process.go`: the order of operations is
  now HMAC verify → **dedupe** → rate limit (per-source,
  per-company). A duplicate (`isNew=false`) does not burn
  a token in either bucket — the rate limit gate is
  "free for dupes" as agreed. The duplicate still
  propagates to NATS so the router can fan out the alert
  with the updated `dedupe_count`. The M2 router, M3
  deliverers, and the M5 tail hub all see the count climb.
- `cmd/ingestd/main.go`: wires `observability.NewMaxSeen()`
  into `processDeps` so HTTP, MQTT, and WS transports
  share one in-process state and one max-observed gauge.
- `cmd/deliverd-telegram/main.go`: `formatMessage` takes
  a `dedupeCount uint32` and appends ` (×N)` to the title
  when N > 1. Below 2, the title is unchanged.
- `cmd/deliverd-fcm/main.go`: notification.title gets
  the same suffix. The raw count also goes into
  `data.dedupe_count` so a native Android client can
  render it however it likes.
- `loadgen/cmd/{http,mqtt,ws}/main.go`: all three
  loadgens gain a `--dedupe-key` flag that forces a
  specific key on every alert. This is what the ×N
  smoke needs (one key, N copies, N=600 in the rate-limit
  test).
- `internal/config/config.go` + `docker-compose.yml`:
  `BA_INGESTD_DEDUPE_TTL_SECONDS` env var, default 300s
  (was 60s hard-coded).
- `internal/observability/metrics.go`: 2 new metrics —
  `ba_ingestd_dedupe_collapsed_total{source}` (counter
  that ticks on every `isNew=false` hit) and
  `ba_ingestd_dedupe_count_max_observed{source}` (gauge
  that climbs to the highest dedupe_count seen for that
  source since process start).

Bug #1 — int-seconds truncation broke sub-second test windows
The first dedupe test with a 500ms window failed: the Lua
script received `0` for the TTL and returned an error
("invalid ttl"). Go's `int(500ms.Seconds())` truncates to
0, which is below the script's `> 0` guard. Fix: pass the
TTL as a float (Go's `d.window.Seconds()` returns float64)
and round up in Lua with `math.ceil`. Test sleeps were
also bumped to 1.3s to account for the rounded TTL.

Bug #2 — Prometheus Gauge doesn't expose Get()
The "max dedupe count" gauge needs to only tick upward,
but Prometheus Gauges can be Set() to any value. The
canonical pattern is a custom Collector that knows the
in-process state. That's a lot of boilerplate for one
metric. The pragmatic alternative: keep the max in a
sync.Map ourselves, and call `Set()` on the gauge only
when the new value strictly exceeds the previous max.
The MaxSeen type is thread-safe (CAS on conflict) and
the gauge becomes a read-only view of our in-process
state. 5 race-tested unit tests cover the corner cases.

Decision: free-for-dupes applies to BOTH rate-limit buckets
The user picked "both buckets free" over "per-source only"
in the M6 design conversation. The reasoning: a duplicate
is a duplicate, and the rate limit exists to backpressure
*new* alert volume, not to charge *attempt* volume. A dupe
storm of 600/s through a 100/s bucket now costs 1 token
instead of 600. The downside (a bad source with a valid
HMAC could pump dupes for free) doesn't apply because
they already have full source-level access — rate limit
isn't a security control, it's a backpressure mechanism.

What's NOT in M6 (and not supposed to be):
- Router-level collapse on `dedupe_key` — a true
  "one message per burst" would require the router to
  collapse before fanning out, instead of fanning out
  N alerts with climbing dedupe_count. M6.5+ when the
  M9 dashboard traffic pattern justifies it.
- Per-source override of `max_concurrent_connections` —
  the migration added the column but the M6 path still
  reads the env default (32). M6.5 or M10.
- Server-derived dedupe_key from a hash of (title+body+labels)
  — currently the source must supply the key. M11 lets
  sources opt in to a server-hash fallback for
  duplicate-prone senders that don't compute their own key.
- Per-company rollup of the max-observed gauge — the
  gauge is per-source only. M9 dashboard work.

Pushed: 3 commits on master mirroring M0–M5's pattern:
1. M6(1/3): code (dedupe sliding-window Lua, MaxSeen,
   metrics, process.go reorder, deliverer ×N suffix,
   loadgen --dedupe-key, config + compose, 5 new
   maxseen tests, 2 new dedupe tests)
2. M6(2/3): verified (M6_VERIFICATION.md +
   scripts/m6_smoke.sh + M6_SMOKE_LOG.md, 3 consecutive
   green runs, 11/11 checks each, 600-alert dupe storm
   confirms free-for-dupes, 0 rate-limit hits)
3. M6(3/3): this PROMPT bump + README + SPEC §23.

**2026-06-14 — M6.5 shipped (Router-level dedupe collapse)**

The natural follow-on to M6, falling out of the M6 "What's
NOT" list. M6 gave us "the recipient sees ×N" (100 messages,
the last one tagged). M6.5 gives us "the recipient sees 1
message" (collapsed by the router, not 100).

New code (~700 LoC Go + 1 new test binary + 1 new env var):

- `internal/dedupe/collapser.go` (NEW, ~200 LoC): the
  Collapser. Public API: `NewCollapser(flushMs, onFlush)`,
  `Observe(sourceID, dedupeKey, alert) Decision`,
  `Run(ctx)`, `FlushAll()`. The Decision is one of:
    * `Passthrough`   — empty dedupe_key; deliver immediately
    * `CollapseNew`  — first arrival of (source, key);
      caller does the expensive work (resolve recipients)
      and caches it
    * `CollapseDupe` — subsequent arrival within the
      window; stored alert's dedupe_count already updated
  Per-source isolation enforced by keying on
  `(source_id, dedupe_key)`, not on `dedupe_key` alone.

- `internal/dedupe/collapser_test.go` (NEW, ~250 LoC, 7
  tests, all green): passthrough-on-empty-key,
  single-collapse-100-alerts, per-source-isolation,
  max-wait-re-flush, empty-key-doesn't-block-real-key,
  8-goroutine-concurrent-same-key-storm, FlushAll-drain.

- `cmd/routerd/collapse.go` (NEW, ~250 LoC): wires the
  Collapser into routerd. The router holds a tiny
  `fanoutState` (source+key → cached resolved targets) so
  duplicate alerts skip the DB-bound `ResolveTargets`
  call. The flush callback reads the cached targets and
  publishes one delivery per (target, channel) with the
  final dedupe_count.

- `cmd/routerd/main.go` (refactored): `handleOne` routes
  through `observeAndFanout`. The Collapser's Run loop
  starts in a goroutine in `consume()`. `FlushAll()` is
  called on graceful shutdown so the last few pending
  collapses still get delivered. Config now loaded via
  `config.LoadRouterd()` (was `LoadCommon`).

- `internal/config/config.go`: new `Routerd` struct with
  `DedupeFlushMs` field. Env var
  `BA_ROUTERD_DEDUPE_FLUSH_MS`, default 2000ms.

- `testfakes/tailcount/main.go` (NEW, ~70 LoC): a tiny Go
  helper that subscribes to the M5 tail WS and counts
  events, filtering on a substring. Used only by the
  smoke; no production code touches it.

- `loadgen/cmd/ws/main.go` (small change): `makeAlert`
  injects the `--dedupe-key` into the title as
  `LG M5 burst <key> #N` when forced. The compact tail
  event includes title but NOT dedupe_key (by design),
  so the smoke needs to filter on something the operator
  can see.

- `docker-compose.yml` + `.env.example`:
  `BA_ROUTERD_DEDUPE_FLUSH_MS=2000` documented.

Two real bugs caught + fixed in M6.5 verification:

1. **Tail filter was on `dedupe_key` but the compact tail
   event doesn't include it.** The tail event JSON is just
   `{alert_id, company_id, source_id, severity, title,
   received_at, transport, dedupe_count}` — no
   `dedupe_key` field by design (M5 spec keeps the event
   small). Fixed by making the loadgen put the key in the
   title, and the smoke filters on the title prefix
   instead. The dedupe_key field is preserved in the
   payload (for the deliverers) but not in the tail
   event.

2. **Globex had no recipients in the dev DB.** The dev
   schema only seeds acme; the m6 smoke never sent
   globex alerts. Step 5's per-source-isolation test
   surfaced this ("no recipients resolved" for grafana).
   The smoke now auto-seeds the globex company, source,
   individual, and subscription at the top (idempotent
   INSERT ... ON CONFLICT DO NOTHING). This is a smoke
   helper, not a migration — the real seed lives in the
   M2 migration files. The smoke needs globex to exist
   for Step 5; the production code doesn't care.

3. **Step 3 (tail + collapse) needed the burst to fit
   inside one flush window.** First version sent 25
   alerts at 10/s = 2.5s, which crossed the 2s flush
   boundary and produced 2 messages for one burst. Bumped
   the rate to 50/s = 0.5s, so the whole burst is inside
   a single window. The test still proves the headline
   behavior (1 message + tail sees the storm), but the
   timing has to be tuned to the configured flush window.

Design choice recap (per the M6.5 scope conversation):

- **Router-side, max-wait debounce.** Ingest publishes
  unconditionally (so alerts hit durable storage + tail).
  The collapse happens *between* NATS and the deliverers,
  i.e. in the router.
- **2s default flush window.** Long enough to coalesce
  most bursts; short enough that critical alerts feel
  instant. Continuous stream re-flushes every window.
- **Per-source isolation.** Same key from two sources is
  two separate collapses. We considered global collapse
  but it has bad blast-radius properties (one noisy source
  could shadow another). Per-source-only ships.
- **Cached resolved targets.** The first arrival does
  the DB-bound `ResolveTargets` call. Subsequent dupes
  update the dedupe_count on the stored alert and
  return. Net: 100-alert burst produces 1 DB call, not
  100. (This is the biggest perf win — without it,
  the 100-alert burst would have done 100 DB queries,
  which is the original M2–M5 cost we were trying to
  avoid.)
- **Trust the ingestd's count.** The router doesn't
  INCR itself; it takes the max of the stored and
  incoming values. Ingestd is the canonical counter;
  this avoids a second Redis round-trip per arrival.

What's NOT in M6.5 (deferred):

- **Active collapse metrics** — no Prometheus metric for
  "how many collapses happened" or "how many alerts
  were collapsed". The router logs them at info level
  with `dedupe_count`, but we don't have a
  `ba_routerd_collapse_total{source}` counter yet. M7
  dashboard work.
- **Per-source override of `BA_ROUTERD_DEDUPE_FLUSH_MS`** —
  a noisy source might want a longer window. The
  migration added `sources.dedupe_flush_ms` but the M6.5
  path uses the env default for all sources.
- **Collapse-across-sources** — currently per-source.
  Could be a config flag but the blast-radius worries
  are real.
- **Tombstone on collapse** — the collapsed delivery
  currently looks like a normal alert. A future M7 could
  add a `collapsed_count: N` field to the payload so
  deliverers can render "5 messages collapsed" in
  addition to the `(×N)` suffix.

Pushed: 3 commits on master mirroring M0–M6's pattern:
1. M6.5(1/3): code (Collapser + collapse.go wiring +
   config + compose + 7 new collapser tests)
2. M6.5(2/3): verified (M6.5_VERIFICATION.md +
   scripts/m6.5_smoke.sh + M6.5_SMOKE_LOG.md, 3
   consecutive green runs, 9/9 checks each, 100-alert
   burst → 1 message confirmed)
3. M6.5(3/3): this PROMPT bump + README + SPEC §23.