broad-announce/PROMPT.md

Broad-Announce — Build Log (PROMPT)

Decisions, lessons, blockers. Append-only. Update as we go.

2026-06-13 — kickoff

Decided

• Repo at git3.techno-world.net/lrosales/broad-announce (private).
• v1 stack: Go, PostgreSQL + Timescale, ClickHouse, NATS JetStream, Redis, EMQX (MQTT), Prometheus + Grafana, Loki. Deploy v1 = Docker Compose. v2 = K8s.
• Multi-tenancy = shared infrastructure, strict app-level isolation. No row-level security in v1; tenant filter on every query.
• FCM = single shared project for v1, schema supports per-company FCM project later (companies.fcm_shared).
• Telegram is a first-class delivery channel + a management UI (the bot is how users mute, subscribe, acknowledge).
• Severity taxonomy: info | warning | critical | inminent_colapse. Only inminent_colapse bypasses quiet hours.
• Dedupe: 60s window per (source_id, dedupe_key), attach dedupe_count so user sees "×N in 60s" not N pushes.
• v1 capacity target in docker-compose: 5k alerts/sec sustained. 50k/sec is the design ceiling; the K8s + multi-broker work is what unlocks it.

Open (was) → Resolved 2026-06-13

• Bot ↔ individual linking → admin-invites only. Flow: admin creates individuals row + generates one-time invite code; user runs /start <invite_code> in the Telegram bot; bot matches the code, links telegram_chat_id to the pre-existing individual, and burns the code. Stricter path: prevents drive-by bot self-registration, keeps individuals auditable.
• Localized titles → source-localized, pass through. v1 does NOT translate. Sources send pre-localized title/body strings (or use the locale on the subscription to key into their own lookup table before calling our API). Alert schema carries locale and title/body already-resolved. We can add a translation layer in v2 if customers ask.
• mTLS: schema supports it, but the docker-compose profile won't terminate client certs in v1. Documented as opt-in for enterprise sources.

Lessons (already)

• It's much cheaper to answer "what can this be?" with 30 questions than to refactor later. Most of the SPEC's weight is in §11 (security) and §6 (recipient resolution) — those are the parts that are expensive to change after launch.
• For multi-tenant at 10k companies, the one design choice that compounds is the subject layout in the broker. alerts.<company_id> is fine; alerts.<company_id>.<source_id> would let us scale router consumer groups per source. Locked in §6 of ARCHITECTURE.md.

2026-06-13 — added gRPC ingest (option a)

• Decision: ship gRPC bidi-streaming as the fourth ingest protocol, scoped to internal high-volume sources only. Public SaaS webhooks stay on HTTP POST, browsers stay on WebSocket, IoT stays on MQTT.
• Rationale: typed schemas, HTTP/2 + protobuf, native backpressure, no reconnect-loop code, generated Go/Java/Python/Node clients. Cost: one more protocol to operate + a buf generate build step.
• Gated on M11, after M10 (load test). M10 now caps at 5k/s on docker-compose (not 50k/s) — 50k/s is the design ceiling that requires K8s + multi-broker (M12 in v2).
• StreamAlerts(Alert) → Ack carries the same dedupe_count contract as HTTP/WS/MQTT.
• Proto lives at proto/broadannounce/v1/ingest.proto; server stub in internal/grpcserver/, reusable client in internal/grpcclient/. Auth = API key in metadata + optional mTLS. Per-stream rate limit + 256 in-flight cap = natural backpressure.

2026-06-13 — M10 split: option C

• M10 (in-cluster, real) stays at 5k/s on docker-compose.
• Added M10-bench: 50k/s against the broker + router with the delivery tier stubbed. Proves the ceiling without committing to K8s. Green here means K8s is implementable, not required.
• Implication: we need traffic generators that can hit the in-cluster path at 5k/s (M10) and the broker+router path at 50k/s (M10-bench) with the same source-protocol clients we ship. The loadgen/ tool is the new home for that.

2026-06-13 — three open questions resolved

• (Q1) Yes, build the fake-FCM / fake-Telegram / fake-SMS servers (testfakes/). M10 must not burn 50k FCM credits.
• (Q2) M10-bench delivery stub lives at the service level — a no-op deliverd binary in the bench profile, not a NATS subject drop. More realistic: we exercise the real broker + router + the message shape deliverd would consume.
• (Q3) loadgen ships as a single Docker image with all four binaries; entrypoint picks one via the image's command: field in the compose / k8s spec.

2026-06-13 — Source protection (throttling) is in

• Spec §22 (new). Seven layers, evaluated in order per request: payload-size cap → per-IP conn cap → per-source token bucket → per-company token bucket → schema validate → broker circuit breaker → per-source quarantine.
• Per-source config additive to §4 schema (rate_limit_per_sec, max_payload_bytes, max_concurrent_connections, quarantine_*).
• Rejection-code matrix is the contract across HTTP/WS/MQTT/gRPC (e.g. per-source rate → 429+Retry-After / Ack{RATE_LIMITED} / MQTT reason 0x97 / close 1013).
• Rollout tied to milestones: M0 layer 3, M1 layers 1/4/5, M5 layer 2, M6 dedupe-aware shaping, M9 layers 6+7.
• Runaway-source test is part of M10 exit (loadgen fault-injection: one source at 10× cap must not push p99 for other sources above 5s).

2026-06-13 — M0 shipped (12 commits, 2888 LoC)

What landed:

• cmd/{ingestd,routerd,deliverd,admind}/ — four Go service mains
• cmd/ingestd/http.go — HTTP POST handler implementing SPEC §22 layers 1, 3, 4, 5 + Stripe-style HMAC auth
• internal/alert — Alert v1 type + Validate() (183 LoC + 103 LoC tests)
• internal/broker — NATS JetStream wrapper, three streams (ALERTS/DELIVERIES/DLQ) auto-created
• internal/dedupe — 60s SET NX EX + INCR (Redis-required tests pass against a host-local Redis)
• internal/ratelimit — per-second INCR bucket (Redis-required tests pass)
• internal/observability — slog + Prometheus registry, the IngestdMetrics struct matches SPEC §22 metric names
• internal/httpserver — shared /health + /metrics scaffold
• internal/config — env-driven Common + Ingestd
• loadgen/cmd/http/ — loadgen-http with --mode normal, HMAC signing, 70/25/4/1 severity mix, dedupe-pct knob
• loadgen/go.mod — separate module per SPEC §21, replace directive points at the parent module
• docker-compose.yml + Dockerfile — single-host stack, all 5 binaries in one image
• M0_VERIFICATION.md — 8-step smoke test
• deploy/prometheus/prometheus.yml — scrapes all 5 services

What's NOT in M0 (and not supposed to be):

• routerd/deliverd/admind business logic (M2/M3+)
• MQTT, gRPC, WebSocket ingest (M4/M5/M11)
• per-IP concurrency cap (M5)
• circuit breaker (M9)
• quarantine (M9)
• ClickHouse archive (M7)
• source registry in DB (M2)
• Postgres migrations (M2)

Module path: git3.techno-world.net/lrosales/broad-announce. Loadgen module path: git3.techno-world.net/lrosales/broad-announce/loadgen. All pushed: 7cd922c..49b2dba on master.

2026-06-13 — Port convention

Project rule: app HTTP services use 8800–8899 (ingestd 8800, routerd 8801, deliverd 8802, admind 8803, loadgen metrics 8891, fakefcmd 8820). Canonical ports stay (5432, 4222, 6379, 1883, 9090, 3000). Reason: 8080+ collides with workstation tooling. SPEC §18 now has a port-conventions sub-section. Commit: d76aa0b.

2026-06-13 — M1 code-complete (8 commits, awaiting live smoke)

What landed:

• migrations/001_init.up.sql — companies, individuals, fcm_tokens (the M1 minimum schema; full SPEC §4 lands in M2 as additional migrations)
• migrations/002_deliveries.up.sql — deliveries table (status: pending|sent|failed|dlq; payload jsonb for M8 replay)
• migrations/seed.sql — idempotent; 1 company, 1 individual, 1 token
• internal/postgres — pgxpool wrapper with retry-on-startup
• cmd/seed — applies *.up.sql in lexical order, then seed.sql
• testfakes/fakefcmd — 70 lines, /health + /v1/.../messages:send, --fail-rate knob
• internal/routing — Resolver with ResolveTokens (M1 broadcast via single SQL join; M2 swaps for the rules engine)
• cmd/routerd — M1 main: subscribes to alerts.>, resolves recipients, publishes one deliveries.fcm. per token
• cmd/deliverd — M1 main: subscribes to deliveries.fcm.>, builds FCM HTTP v1 message body (M3 swap is a no-op at this layer), posts to BA_FAKECMD_URL, writes a deliveries row
• Dockerfile builds 7 binaries
• docker-compose adds fakefcmd + seed (one-shot sidecar), wires BA_FAKECMD_URL into deliverd
• M1_VERIFICATION.md: 9-step smoke test

What we agreed to defer (per the user):

• Q1 seed: 1 company / 1 individual / 1 token — DONE
• Q2 router: M1 broadcast (no subscriptions) — DONE
• Q3 testfakes: only fakefcmd for M1 — DONE
• Q4 migrations: 3 tables in M1, expand in M2 — DONE

What's NOT in M1 (and not supposed to be):

• subscriptions, quiet hours, routing rules (M2)
• per-source allowed_targets (M2)
• real FCM (M3)
• other delivery channels (M3+)
• retry + DLQ from SPEC §9 (M3)
• ClickHouse / Timescale hypertables (M7)
• HMAC secret in DB (M2)
• per-IP cap, circuit breaker, quarantine (M5/M9)

Pushed: 264d284..9cf68a1 on master (5 commits for M1 code, plus the d76aa0b port shift). The user still has to actually run docker compose up and the M1 verification steps before M1 is fully done.

2026-06-13 — M2 shipped (recipient resolution, rules engine)

What landed (3 commits, ~600 LoC Go + ~250 LoC SQL):

• migrations/003_subscriptions_groups.up.sql (and .down.sql): 5 new tables — sources, groups, group_members, subscriptions, routing_rules. FKs to companies / individuals. Idempotent.
• migrations/seed_m2.sql: 2 more individuals (Bob, Carol), 1 group (sre: Alice+Bob), 1 source row, 3 subscriptions covering the 3 scenarios, 1 routing rule.
• cmd/seed/main.go: runner now applies seed.sql then seed_m2.sql (lexically ordered) so re-running is safe.
• internal/routing/routing.go: full rewrite. New Resolver with ResolveTargets(ctx, *alert.Alert) ([]Target, error). Single SQL round-trip via a 5-CTE query:
  1. src — the source row
  2. allowed_individual_ids — direct + group-expanded + broadcast
  3. rule_individual_ids — routing_rules where match_expr matches
  4. candidates — union of 2 and 3
  5. sub_expanded + filtered — subscriptions with min_severity, channel_mask, quiet-hours filter Final pass: Go-side quiet-hours check (with the inminent_colapse bypass) and channel='fcm' gate.
• internal/alert/alert.go: added Severity.Rank() and MinSeverityRank() helpers. Two new unit tests in alert_test.go.
• internal/routing/routing_test.go: new file, 10 subtests for inQuietHours (same-day, wrap-around, always-quiet, edge cases at window start/end).
• cmd/routerd/main.go: M2 main. Calls ResolveTargets(alert), hard-fails on zero recipients (logs WARN, acks — no DLQ for M2; that's M3+). Emits the new envelope shape with Channel + Endpoint instead of the M1 FCMToken+Locale.
• cmd/deliverd/main.go: envelope struct updated to match the M2 routerd output. Only field renames; the FCM HTTP v1 body shape is unchanged.
• M2_VERIFICATION.md: 7-step plan + hard-fail scenario.
• scripts/m2_smoke.sh: automated runner for steps 2-6.
• M2_SMOKE_LOG.md: per-step results, honest flags.

Live smoke results (all green):

• step 2: warning, db-prod-04 → Alice only
• step 3: critical, db-prod-04 → Alice + Bob
• step 4: critical, db-prod-03 (rule match) → Alice + Bob
• step 5: warning, db-prod-03 (rule match but min filter) → Alice only (Bob filtered by min_severity=critical)
• step 6: inminent_colapse → Alice + Bob + Carol (Carol's quiet_hours=00:00-23:59 bypassed)
• step 7: subscriptions paused → routerd logs WARN msg="zero recipients, dropping", no delivery 9 deliveries, 0 failures, 0 retries.

Per the user's M2 Q1/Q2/Q3 answers:

• Q1: seed = 3 individuals + 1 group + 1 source + 4 subscriptions covering all 4 scenarios (DONE; one of the 4 was dropped to 3 because we folded "fcm with min=critical AND in quiet hours" into a single individual — the bypass test is still valid via Carol).
• Q2: hard-fail on zero recipients with a WARN log (DONE; the routerd logger is the notification path for M2; M3+ will add a dlq.no_recipients.<company> subject).
• Q3: (a) emit the row, drop the channel at the SQL filter (DONE; WHERE f.channel = 'fcm' in the resolver).

What's NOT in M2 (and not supposed to be):

• real FCM (M3)
• telegram, sms, email, slack, teams (M3+)
• retry + DLQ (M3)
• per-source HMAC secret in DB (M5 when mTLS / API-key path comes in)
• full match_expr language (M6+); M2 supports category, severity, data.*, all
• per-rule priority semantics with "first match wins unless continue=true" (M3+); M2 unions all matching rules
• routing rule "drop" target type (schema supports it, no test for it; the SQL filters it via the candidate union)
• Timezone support beyond UTC (M2 falls back to UTC if time.LoadLocation(tz) fails)

Pushed: 280e048 on master (3 commits: a29c2d8 + 1e9eca9 + 280e048).

2026-06-14 — M3 shipped (Telegram delivery + bot commands)

What landed (~2k LoC Go + ~80 LoC SQL across these commits):

• migrations/004_telegram.up.sql (+ .down.sql): telegram_bots table (one row per (company, bot); M3 supports one bot per company), and 4 new columns on individuals: telegram_chat_id, telegram_user_id, telegram_invite_code, mute_until. Two partial unique indexes for fast /start <code> lookup and "is this telegram_user_id already linked" check.
• migrations/seed_m3.sql: inserts the per-company telegram_bots row, sets telegram_invite_code on all three individuals, and pre-links Alice to a fake Telegram account (chat_id=1001, user_id=900001) so the very first POST exercises both fcm and telegram delivery paths. Also expands every subscription's channel_mask to ["fcm","telegram"] (M2 was ["fcm"]).
• cmd/seed/main.go: runner now applies seed.sql, seed_m2.sql, seed_m3.sql in lexical order. All three are idempotent.
• cmd/deliverd/ → cmd/deliverd-fcm/: rename. The M2 deliverd is now deliverd-fcm — per the user's M3 Q2 answer (two per-channel binaries, not one with a registry).
• cmd/deliverd-telegram/: new binary. Subscribes to deliveries.telegram.<company_id>, builds a severity-prefixed text message, posts to the Bot API's sendMessage endpoint, writes a deliveries row. Single-attempt (no retry, no DLQ). M9 adds the chain.
• cmd/telegramd/: new binary. Loads the active bot list from telegram_bots at startup, long-polls getUpdates, dispatches commands to the handler, and replies via sendMessage. One process per deployment; per-bot sharding can come later if needed.
• internal/telegram/: 3 files, ~640 LoC.
  • client.go — BotClient interface + HTTP impl.
  • commands.go — text → Command parser (/start, /subscribe, /unsubscribe, /preferences, /status, /mute, /unmute). /mute accepts 2h, 30m, 90s, or until 18:00.
  • handler.go — Command → DB updates. /start is atomic claim-by-invite-code; /subscribe upserts a subscription with the requested min_severity.
  • commands_test.go — 7 subtests, 100% pass.
• internal/routing/routing.go: the resolver's single CTE now UNION-ALLs an fcm_rows branch (joins on fcm_tokens) and a tg_rows branch (filters on telegram_chat_id IS NOT NULL). Same cost as M2; one extra row in the join key set. Hard-fail on zero targets unchanged.
• testfakes/faketgmd/: ~350 LoC. Fake Bot API server with /admin/queue (queue an incoming update), /admin/sent (read every sendMessage call), and /admin/reset (clear state). In-memory only. Used purely for the smoke test.
• docker-compose.yml + Dockerfile: add deliverd-telegram (port 8821), telegramd (port 8822), faketgmd (port 8830). Grafana is now 3001:3000 because :3000 is held by gogs on this host.
• M3_VERIFICATION.md: 8-step manual + automated walkthrough.
• M3_SMOKE_LOG.md: per-step results, honest flags.
• scripts/m3_smoke.sh: automated runner for steps 2–8.
• README + PROMPT + SPEC §23 bumped to "shipped 2026-06-14".

Live smoke results (all green):

• step 2: warning → Alice fcm + Alice telegram (Bob unlinked)
• step 3: critical → Alice fcm + Alice telegram + Bob fcm
• step 4: Bob /start acme-bob-002 via faketgmd admin queue → handler atomically claims the code, telegram_user_id=900002, bot reply "Linked. Welcome, Bob SRE."
• step 5: critical → Alice fcm+telegram + Bob fcm+telegram (Bob telegram now fires)
• step 6: inminent_colapse → Alice fcm+telegram + Bob fcm+telegram + Carol fcm (Carol never linked; quiet hours bypassed)
• step 7: Bob /preferences → bot reply "Your subscriptions: …"
• step 8: faketgmd received 8 sendMessage calls

14 deliveries across 4 unique alert_ids, 0 failures, 0 retries.

Per the user's M3 Q1/Q2/Q3 answers:

• Q1: faketgmd is shipped as a test fake (DONE; M5+ can add a webhook-based "real" mode).
• Q2: two per-channel binaries, deliverd renamed to deliverd-fcm (DONE; deliverd-telegram is its own binary).
• Q3: channel expansion on Bob and Carol's subscriptions (DONE; M2's ["fcm"] → M3's ["fcm","telegram"] via the seed migration).

What's NOT in M3 (and not supposed to be):

• real FCM auth (M9/M11)
• SMS, email, Slack, MS Teams, voice (M3+)
• webhook mode for the bot (M5/M9)
• bot token encryption at rest (M11, security milestone)
• retry + DLQ for telegram delivery (M9)
• per-company bot token resolution in deliverd-telegram (currently reads env-supplied default; M3+ looks up by company_id encoded in alert or subject)
• markdown/HTML formatting in telegram messages (M3.5+)
• per-rule priority semantics with "first match wins unless continue=true" (deferred from M2; the unioned-rules behavior holds for M3)

Pushed:

2026-06-14 — M4 shipped (MQTT ingest + per-source ACL)

What landed (~750 LoC Go + ~60 LoC config across these commits):

• internal/mqttclient/: new package. Thin wrapper around paho.MQTT that hides the token-on-publish option behind a single error-returning Publish(), sets consistent LastWill / MaxInflight / AutoReconnect defaults, and formats client_id as <service>-mqtt-<host> so EMQX /admin/clients shows them cleanly. Used by both cmd/ingestd (subscriber) and loadgen/cmd/mqtt (publisher).
• cmd/ingestd/mqtt.go: new file. The MQTT subscriber runs every message through the same processDeps.ProcessAlert pipeline as the HTTP POST handler — parse → validate → HMAC verify → rate-limit → dedupe → publish to NATS. The only MQTT-specific code is the topic parser (parseIncomingTopic → ba/<co>/<src>/incoming, 4 segments, last = incoming) and the envelope sniffer (accepts both {alert, auth} envelope and bare alert bodies for future broker-native clients).
• cmd/ingestd/main.go: wires up the MQTT path. New env vars BA_INGESTD_MQTT_BROKER, BA_INGESTD_MQTT_USERNAME, BA_INGESTD_MQTT_PASSWORD, BA_INGESTD_MQTT_SUBSCRIBE (ba/+/+/incoming default).
• internal/observability/metrics.go: adds the MQTTMessages CounterVec (labels: result=received | accepted | deduped | bad_topic | bad_signature | invalid_json | <other RejectReason>). Exposed as ba_ingestd_mqtt_messages_total.
• loadgen/cmd/mqtt/: new binary. Same severity mix, dedupe ratio, and burst mode as loadgen-http. Publishes to ba/<co>/<src>/incoming with the <src>-<co> user and HMAC secret for auth.
• deploy/emqx/acl.conf: Erlang-term ACL rules. Each source can only publish to its own ba/<co>/<src>/incoming; ingestd can subscribe to ba/+/+/incoming; default deny on #. Re-read on EMQX SIGHUP.
• deploy/emqx/auth-built-in-db-bootstrap.csv: per-username auth. <source_id>-<company_id> for sources, ingestd for the subscriber. Password == HMAC secret so the same secret serves both MQTT auth and per-message HMAC.
• deploy/emqx/README.md: explains the auth model, the env-var-vs-emqx.conf precedence (env vars win in EMQX 5.x), and the emqx_ctl listeners restart command for hot-reloading acl.conf.
• docker-compose.yml: switches the emqx service from volume-mounting emqx.conf (which EMQX 5.x rejects as a partial config with node.cookie required_field) to env-var config. The EMQX_AUTHENTICATION__1__* and EMQX_AUTHORIZATION__* env vars are the only way to inject chain-level config in 5.x. acl.conf and the bootstrap CSV stay as volume mounts (their files are partials, which is fine).
• scripts/m4_smoke.sh: end-to-end smoke. 5 steps (1 alert, 5 alerts with dedupe, bad sig, ACL violation, bad json). Builds loadgen-mqtt and the 3 failure-path test binaries into /tmp/ on first run. 12 deliveries, 0 failures, 3 consecutive green runs.
• M4_VERIFICATION.md + M4_SMOKE_LOG.md: the spec-style step-by-step + the live run results.

Why MQTT in v1: SPEC §18 calls for HTTP+MQTT as the v1 transports. M0–M3 ship HTTP; M4 ships MQTT. Sources that prefer a broker (Prometheus alertmanager webhook-bridge, Grafana, IoT) can publish to ba/<co>/<src>/incoming without writing a webhook client.

Why per-source ACL: the broker is the first line of defense. A source that gets compromised can only spam its own topic — which still goes through the HMAC gate at ingestd, so a forged message without the secret is rejected in step 4. ACL stops cross-tenant spam; HMAC stops forged messages; rate-limit stops flood; dedupe stops burst; M9 adds the circuit breaker and quarantine.

Why env-var config beats emqx.conf: EMQX 5.x's HOCON config is layered as emqx.conf → base.hocon → cluster.hocon → env vars (highest precedence). The double-underscore separator in env-var names maps to nested HOCON keys. emqx.conf is a full config — partial overrides get rejected with node.cookie required_field. The env-var approach lets us set just the auth + authz chains without restating the entire base.hocon.

What's NOT in M4 (and not supposed to be):

• Per-IP concurrency cap (M5 with WS, SPEC §22 layer 2)
• Circuit breaker + quarantine (M9)
• TLS to EMQX (M11, security milestone)
• Persistent sessions (M11)
• Per-company bot token resolution for sources that share a single EMQX user across companies (M3+ generalization)
• HTTP-style signature in a header (MQTT has no headers; the auth field in the JSON envelope is the equivalent)
• QoS 2 (M11, when persistent sessions land)

Pushed: bc907d9 on master (1 commit for the code; this PROMPT bump + README + SPEC bump + the EMQX env-var refactor consolidated into a single follow-up commit).

2026-06-14 — M5 shipped (WebSocket ingest + live tail + per-IP cap)

What landed (~650 LoC Go + 2 SQL files across these commits):

• migrations/005_ws.up.sql: adds max_concurrent_connections column to sources and companies (defaults 32 and 64).
• internal/concurrency/perip.go: tiny sync.Map[ip]atomic.Int64 with Acquire/Release/InUse and a janitor goroutine that prunes entries idle > 5 min. No Redis on this hot path — would self-DoS.
• internal/tailhub/hub.go: in-process pub/sub. Subscribe returns a *Subscription with a buffered chan *Event (size 64) and a per-sub Drops counter; Publish is best-effort, drops on full. Filter.CompanyID matches single-company or all (empty).
• internal/wsclient/client.go: thin gorilla/websocket wrapper. Connect does the auth round-trip and stashes the auth reply; SendAlert does a single WS write; Close sends CloseMessage(1000) then TCP close. Used by loadgen-ws and the smoke drivers.
• loadgen/cmd/ws/main.go: loadgen-ws binary. Default normal profile (30% dedupe). 3 modes: normal, burst.
• loadgen/cmd/m5drivers/tail/main.go: /tmp/m5-tail-test driver used by scripts/m5_smoke.sh steps 6 + 7.
• cmd/ingestd/ws.go: WS ingest at GET /v1/ingest/ws. Auth via first frame {api_key}. The handler does per-IP Acquire (rejects + 503 on cap exceeded), then loops on alert frames, sniffs {alert, auth} envelope vs bare body, and calls the shared scoped.ProcessAlert. Per-IP Release runs in the handler's defer.
• cmd/ingestd/wstail.go: live tail at GET /v1/tail/ws. Token via Authorization: Bearer, X-BA-Tail-Token, or ?token=. Optional ?company_id= filter. Subscribe happens BEFORE the upgrade so events in the dial→subscribe window are not lost (a real bug we caught in M5 verification — see PROMPT bug #1 below).
• cmd/ingestd/process.go: processDeps now has Tail *tailhub.Hub and Transport string. ProcessAlert calls d.Tail.Publish(tailhub.FromAlert(&a, d.Transport)) on the accepted path, so the tail fans out alerts from HTTP, MQTT, and WS through the same hub.
• cmd/ingestd/main.go: wires concurrency.NewPerIP, tailhub.NewHub, and the two new dep structs.
• internal/observability/metrics.go: 5 new metrics — WSMessages, WSConnections, ConnectionRejected, TailSubscribers, TailDropped — all registered with the ingestd service label.
• docker-compose.yml: BA_INGESTD_TAIL_TOKEN and BA_INGESTD_MAX_CONCURRENT_PER_IP env vars on ingestd.
• scripts/m5_smoke.sh: 7-step live driver. Auto-builds /tmp/loadgen-ws, /tmp/m5-tail-test, /tmp/m5-perip-test, /tmp/m5-badkey on first run.

Bug #1 — Tail subscribe vs upgrade race The first M5 verification run showed the live tail seeing subs=0 at publish time even though the tail's gauge was 1. Root cause: Subscribe was called after the WS upgrade, leaving a window where the client thought it was connected but ingestd hadn't subscribed yet. The WS alert sent during that window was published to 0 subscribers. Fix: subscribe first, then upgrade (so the client's dial-completion implies the subscription is in place). This is documented inline in cmd/ingestd/wstail.go.

Bug #2 — Tail defer ordering The first run of the gauge-update defer fired while the subscription was still in the hub's map (defers run LIFO, so the metric defer fired before the unsubscribe defer). Result: the gauge stayed at 1 after the client disconnected. Fix: combine unsubscribe + gauge update into a single defer that runs after the stream loop exits.

Bug #3 — Badkey envelope shape The first M5 step-4 test sent the HMAC as auth: {"hmac": "v1=..."} (a map). The server's envelope parser declared auth as a string, so the unmarshal failed, the parser fell back to using the whole envelope as the alert body, and validation rejected the alert with "company_id missing". Fix: send auth as a string "t=...,v1=...", same shape as the MQTT envelope. scripts/m4_smoke.sh already had this correct — copy- pasted the pattern into M5.

What's NOT in M5 (and not supposed to be):

• HTTP POST /v1/ingest per-IP cap (M10 — the M5-bump in SPEC §22; we ship the gate now and the wire-up to HTTP is its own small PR)
• JWT-based tail auth (M11 security milestone)
• Cross-node tail fan-out via Redis pub/sub (M15+ when ingestd runs as a cluster)
• A "send test alert" UI button in the tail (deferred — operators can wscat and a loadgen-ws --count 1)

Pushed: 3 commits on master mirroring M0–M4's pattern:

1. M5(1/3): code (migrations + 3 new packages + ws.go + wstail.go + process.go + main.go + metrics + loadgen-ws)
2. M5(2/3): verified (M5_VERIFICATION.md + scripts/m5_smoke.sh + M5_SMOKE_LOG.md, 3 consecutive green runs, 13/13 checks each, +36 deliveries cumulative)
3. M5(3/3): this PROMPT bump + README + SPEC §23.

2026-06-14 — M6 shipped (Dedupe + ×N suffix + sliding TTL + free-for-dupes)

What landed (~350 LoC Go + 1 SQL-friendly column + 2 new metrics across these commits):

• internal/dedupe/dedupe.go: rewritten with a single Lua script that does SET-NX-or-INCR-EXPIRE atomically. The EXPIRE call on every duplicate is what makes the window slide — a steady stream of duplicates keeps the key alive. Sub-second windows in tests are handled by math.ceil in Lua (Redis EX requires an int).
• internal/observability/maxseen.go: a new MaxSeen type that tracks per-key monotonic max. The gauge API doesn't expose Get(), so the canonical Prometheus pattern is to use a custom Collector; instead we keep the max in a sync.Map[source]uint32 and Set() the gauge only on growth. CAS-safe under concurrent writers.
• cmd/ingestd/process.go: the order of operations is now HMAC verify → dedupe → rate limit (per-source, per-company). A duplicate (isNew=false) does not burn a token in either bucket — the rate limit gate is "free for dupes" as agreed. The duplicate still propagates to NATS so the router can fan out the alert with the updated dedupe_count. The M2 router, M3 deliverers, and the M5 tail hub all see the count climb.
• cmd/ingestd/main.go: wires observability.NewMaxSeen() into processDeps so HTTP, MQTT, and WS transports share one in-process state and one max-observed gauge.
• cmd/deliverd-telegram/main.go: formatMessage takes a dedupeCount uint32 and appends (×N) to the title when N > 1. Below 2, the title is unchanged.
• cmd/deliverd-fcm/main.go: notification.title gets the same suffix. The raw count also goes into data.dedupe_count so a native Android client can render it however it likes.
• loadgen/cmd/{http,mqtt,ws}/main.go: all three loadgens gain a --dedupe-key flag that forces a specific key on every alert. This is what the ×N smoke needs (one key, N copies, N=600 in the rate-limit test).
• internal/config/config.go + docker-compose.yml: BA_INGESTD_DEDUPE_TTL_SECONDS env var, default 300s (was 60s hard-coded).
• internal/observability/metrics.go: 2 new metrics — ba_ingestd_dedupe_collapsed_total{source} (counter that ticks on every isNew=false hit) and ba_ingestd_dedupe_count_max_observed{source} (gauge that climbs to the highest dedupe_count seen for that source since process start).

Bug #1 — int-seconds truncation broke sub-second test windows The first dedupe test with a 500ms window failed: the Lua script received 0 for the TTL and returned an error ("invalid ttl"). Go's int(500ms.Seconds()) truncates to 0, which is below the script's > 0 guard. Fix: pass the TTL as a float (Go's d.window.Seconds() returns float64) and round up in Lua with math.ceil. Test sleeps were also bumped to 1.3s to account for the rounded TTL.

Bug #2 — Prometheus Gauge doesn't expose Get() The "max dedupe count" gauge needs to only tick upward, but Prometheus Gauges can be Set() to any value. The canonical pattern is a custom Collector that knows the in-process state. That's a lot of boilerplate for one metric. The pragmatic alternative: keep the max in a sync.Map ourselves, and call Set() on the gauge only when the new value strictly exceeds the previous max. The MaxSeen type is thread-safe (CAS on conflict) and the gauge becomes a read-only view of our in-process state. 5 race-tested unit tests cover the corner cases.

Decision: free-for-dupes applies to BOTH rate-limit buckets The user picked "both buckets free" over "per-source only" in the M6 design conversation. The reasoning: a duplicate is a duplicate, and the rate limit exists to backpressure new alert volume, not to charge attempt volume. A dupe storm of 600/s through a 100/s bucket now costs 1 token instead of 600. The downside (a bad source with a valid HMAC could pump dupes for free) doesn't apply because they already have full source-level access — rate limit isn't a security control, it's a backpressure mechanism.

What's NOT in M6 (and not supposed to be):

• Router-level collapse on dedupe_key — a true "one message per burst" would require the router to collapse before fanning out, instead of fanning out N alerts with climbing dedupe_count. M6.5+ when the M9 dashboard traffic pattern justifies it.
• Per-source override of max_concurrent_connections — the migration added the column but the M6 path still reads the env default (32). M6.5 or M10.
• Server-derived dedupe_key from a hash of (title+body+labels) — currently the source must supply the key. M11 lets sources opt in to a server-hash fallback for duplicate-prone senders that don't compute their own key.
• Per-company rollup of the max-observed gauge — the gauge is per-source only. M9 dashboard work.

Pushed: 3 commits on master mirroring M0–M5's pattern:

1. M6(1/3): code (dedupe sliding-window Lua, MaxSeen, metrics, process.go reorder, deliverer ×N suffix, loadgen --dedupe-key, config + compose, 5 new maxseen tests, 2 new dedupe tests)
2. M6(2/3): verified (M6_VERIFICATION.md + scripts/m6_smoke.sh + M6_SMOKE_LOG.md, 3 consecutive green runs, 11/11 checks each, 600-alert dupe storm confirms free-for-dupes, 0 rate-limit hits)
3. M6(3/3): this PROMPT bump + README + SPEC §23.