Home Explore Help
Register Sign In
lrosales
/
broad-announce
1
0
Fork 0
Files Issues 0 Pull Requests 0 Wiki
Branch: master
Branches Tags
broad-announce
/
PROMPT.md

PROMPT.md 49 KB
Permalink History Raw

Broad-Announce — Build Log (PROMPT)

Decisions, lessons, blockers. Append-only. Update as we go.

2026-06-13 — kickoff

Decided

  • Repo at git3.techno-world.net/lrosales/broad-announce (private).
  • v1 stack: Go, PostgreSQL + Timescale, ClickHouse, NATS JetStream, Redis, EMQX (MQTT), Prometheus + Grafana, Loki. Deploy v1 = Docker Compose. v2 = K8s.
  • Multi-tenancy = shared infrastructure, strict app-level isolation. No row-level security in v1; tenant filter on every query.
  • FCM = single shared project for v1, schema supports per-company FCM project later (companies.fcm_shared).
  • Telegram is a first-class delivery channel + a management UI (the bot is how users mute, subscribe, acknowledge).
  • Severity taxonomy: info | warning | critical | inminent_colapse. Only inminent_colapse bypasses quiet hours.
  • Dedupe: 60s window per (source_id, dedupe_key), attach dedupe_count so user sees "×N in 60s" not N pushes.
  • v1 capacity target in docker-compose: 5k alerts/sec sustained. 50k/sec is the design ceiling; the K8s + multi-broker work is what unlocks it.

Open (was) → Resolved 2026-06-13

  • Bot ↔ individual linking → admin-invites only. Flow: admin creates individuals row + generates one-time invite code; user runs /start <invite_code> in the Telegram bot; bot matches the code, links telegram_chat_id to the pre-existing individual, and burns the code. Stricter path: prevents drive-by bot self-registration, keeps individuals auditable.
  • Localized titles → source-localized, pass through. v1 does NOT translate. Sources send pre-localized title/body strings (or use the locale on the subscription to key into their own lookup table before calling our API). Alert schema carries locale and title/body already-resolved. We can add a translation layer in v2 if customers ask.
  • mTLS: schema supports it, but the docker-compose profile won't terminate client certs in v1. Documented as opt-in for enterprise sources.

Lessons (already)

  • It's much cheaper to answer "what can this be?" with 30 questions than to refactor later. Most of the SPEC's weight is in §11 (security) and §6 (recipient resolution) — those are the parts that are expensive to change after launch.
  • For multi-tenant at 10k companies, the one design choice that compounds is the subject layout in the broker. alerts.<company_id> is fine; alerts.<company_id>.<source_id> would let us scale router consumer groups per source. Locked in §6 of ARCHITECTURE.md.

2026-06-13 — added gRPC ingest (option a)

  • Decision: ship gRPC bidi-streaming as the fourth ingest protocol, scoped to internal high-volume sources only. Public SaaS webhooks stay on HTTP POST, browsers stay on WebSocket, IoT stays on MQTT.
  • Rationale: typed schemas, HTTP/2 + protobuf, native backpressure, no reconnect-loop code, generated Go/Java/Python/Node clients. Cost: one more protocol to operate + a buf generate build step.
  • Gated on M11, after M10 (load test). M10 now caps at 5k/s on docker-compose (not 50k/s) — 50k/s is the design ceiling that requires K8s + multi-broker (M12 in v2).
  • StreamAlerts(Alert) → Ack carries the same dedupe_count contract as HTTP/WS/MQTT.
  • Proto lives at proto/broadannounce/v1/ingest.proto; server stub in internal/grpcserver/, reusable client in internal/grpcclient/. Auth = API key in metadata + optional mTLS. Per-stream rate limit + 256 in-flight cap = natural backpressure.

2026-06-13 — M10 split: option C

  • M10 (in-cluster, real) stays at 5k/s on docker-compose.
  • Added M10-bench: 50k/s against the broker + router with the delivery tier stubbed. Proves the ceiling without committing to K8s. Green here means K8s is implementable, not required.
  • Implication: we need traffic generators that can hit the in-cluster path at 5k/s (M10) and the broker+router path at 50k/s (M10-bench) with the same source-protocol clients we ship. The loadgen/ tool is the new home for that.

2026-06-13 — three open questions resolved

  • (Q1) Yes, build the fake-FCM / fake-Telegram / fake-SMS servers (testfakes/). M10 must not burn 50k FCM credits.
  • (Q2) M10-bench delivery stub lives at the service level — a no-op deliverd binary in the bench profile, not a NATS subject drop. More realistic: we exercise the real broker + router + the message shape deliverd would consume.
  • (Q3) loadgen ships as a single Docker image with all four binaries; entrypoint picks one via the image's command: field in the compose / k8s spec.

2026-06-13 — Source protection (throttling) is in

  • Spec §22 (new). Seven layers, evaluated in order per request: payload-size cap → per-IP conn cap → per-source token bucket → per-company token bucket → schema validate → broker circuit breaker → per-source quarantine.
  • Per-source config additive to §4 schema (rate_limit_per_sec, max_payload_bytes, max_concurrent_connections, quarantine_*).
  • Rejection-code matrix is the contract across HTTP/WS/MQTT/gRPC (e.g. per-source rate → 429+Retry-After / Ack{RATE_LIMITED} / MQTT reason 0x97 / close 1013).
  • Rollout tied to milestones: M0 layer 3, M1 layers 1/4/5, M5 layer 2, M6 dedupe-aware shaping, M9 layers 6+7.
  • Runaway-source test is part of M10 exit (loadgen fault-injection: one source at 10× cap must not push p99 for other sources above 5s).

2026-06-13 — M0 shipped (12 commits, 2888 LoC)

What landed:

  • cmd/{ingestd,routerd,deliverd,admind}/ — four Go service mains
  • cmd/ingestd/http.go — HTTP POST handler implementing SPEC §22 layers 1, 3, 4, 5 + Stripe-style HMAC auth
  • internal/alert — Alert v1 type + Validate() (183 LoC + 103 LoC tests)
  • internal/broker — NATS JetStream wrapper, three streams (ALERTS/DELIVERIES/DLQ) auto-created
  • internal/dedupe — 60s SET NX EX + INCR (Redis-required tests pass against a host-local Redis)
  • internal/ratelimit — per-second INCR bucket (Redis-required tests pass)
  • internal/observability — slog + Prometheus registry, the IngestdMetrics struct matches SPEC §22 metric names
  • internal/httpserver — shared /health + /metrics scaffold
  • internal/config — env-driven Common + Ingestd
  • loadgen/cmd/http/ — loadgen-http with --mode normal, HMAC signing, 70/25/4/1 severity mix, dedupe-pct knob
  • loadgen/go.mod — separate module per SPEC §21, replace directive points at the parent module
  • docker-compose.yml + Dockerfile — single-host stack, all 5 binaries in one image
  • M0_VERIFICATION.md — 8-step smoke test
  • deploy/prometheus/prometheus.yml — scrapes all 5 services

What's NOT in M0 (and not supposed to be):

  • routerd/deliverd/admind business logic (M2/M3+)
  • MQTT, gRPC, WebSocket ingest (M4/M5/M11)
  • per-IP concurrency cap (M5)
  • circuit breaker (M9)
  • quarantine (M9)
  • ClickHouse archive (M7)
  • source registry in DB (M2)
  • Postgres migrations (M2)

Module path: git3.techno-world.net/lrosales/broad-announce. Loadgen module path: git3.techno-world.net/lrosales/broad-announce/loadgen. All pushed: 7cd922c..49b2dba on master.

2026-06-13 — Port convention

Project rule: app HTTP services use 8800–8899 (ingestd 8800, routerd 8801, deliverd 8802, admind 8803, loadgen metrics 8891, fakefcmd 8820). Canonical ports stay (5432, 4222, 6379, 1883, 9090, 3000). Reason: 8080+ collides with workstation tooling. SPEC §18 now has a port-conventions sub-section. Commit: d76aa0b.

2026-06-13 — M1 code-complete (8 commits, awaiting live smoke)

What landed:

  • migrations/001_init.up.sql — companies, individuals, fcm_tokens (the M1 minimum schema; full SPEC §4 lands in M2 as additional migrations)
  • migrations/002_deliveries.up.sql — deliveries table (status: pending|sent|failed|dlq; payload jsonb for M8 replay)
  • migrations/seed.sql — idempotent; 1 company, 1 individual, 1 token
  • internal/postgres — pgxpool wrapper with retry-on-startup
  • cmd/seed — applies *.up.sql in lexical order, then seed.sql
  • testfakes/fakefcmd — 70 lines, /health + /v1/.../messages:send, --fail-rate knob
  • internal/routing — Resolver with ResolveTokens (M1 broadcast via single SQL join; M2 swaps for the rules engine)
  • cmd/routerd — M1 main: subscribes to alerts.>, resolves recipients, publishes one deliveries.fcm. per token
  • cmd/deliverd — M1 main: subscribes to deliveries.fcm.>, builds FCM HTTP v1 message body (M3 swap is a no-op at this layer), posts to BA_FAKECMD_URL, writes a deliveries row
  • Dockerfile builds 7 binaries
  • docker-compose adds fakefcmd + seed (one-shot sidecar), wires BA_FAKECMD_URL into deliverd
  • M1_VERIFICATION.md: 9-step smoke test

    • What we agreed to defer (per the user):

    • Q1 seed: 1 company / 1 individual / 1 token — DONE
    • Q2 router: M1 broadcast (no subscriptions) — DONE
    • Q3 testfakes: only fakefcmd for M1 — DONE
    • Q4 migrations: 3 tables in M1, expand in M2 — DONE

    What's NOT in M1 (and not supposed to be):

    • subscriptions, quiet hours, routing rules (M2)
    • per-source allowed_targets (M2)
    • real FCM (M3)
    • other delivery channels (M3+)
    • retry + DLQ from SPEC §9 (M3)
    • ClickHouse / Timescale hypertables (M7)
    • HMAC secret in DB (M2)
    • per-IP cap, circuit breaker, quarantine (M5/M9)

    Pushed: 264d284..9cf68a1 on master (5 commits for M1 code, plus the d76aa0b port shift). The user still has to actually run docker compose up and the M1 verification steps before M1 is fully done.

    2026-06-13 — M2 shipped (recipient resolution, rules engine)

    What landed (3 commits, ~600 LoC Go + ~250 LoC SQL):

    • migrations/003_subscriptions_groups.up.sql (and .down.sql): 5 new tables — sources, groups, group_members, subscriptions, routing_rules. FKs to companies / individuals. Idempotent.
    • migrations/seed_m2.sql: 2 more individuals (Bob, Carol), 1 group (sre: Alice+Bob), 1 source row, 3 subscriptions covering the 3 scenarios, 1 routing rule.
    • cmd/seed/main.go: runner now applies seed.sql then seed_m2.sql (lexically ordered) so re-running is safe.
    • internal/routing/routing.go: full rewrite. New Resolver with ResolveTargets(ctx, *alert.Alert) ([]Target, error). Single SQL round-trip via a 5-CTE query:
      1. src — the source row
      2. allowed_individual_ids — direct + group-expanded + broadcast
      3. rule_individual_ids — routing_rules where match_expr matches
      4. candidates — union of 2 and 3
      5. sub_expanded + filtered — subscriptions with min_severity, channel_mask, quiet-hours filter Final pass: Go-side quiet-hours check (with the inminent_colapse bypass) and channel='fcm' gate.
    • internal/alert/alert.go: added Severity.Rank() and MinSeverityRank() helpers. Two new unit tests in alert_test.go.
    • internal/routing/routing_test.go: new file, 10 subtests for inQuietHours (same-day, wrap-around, always-quiet, edge cases at window start/end).
    • cmd/routerd/main.go: M2 main. Calls ResolveTargets(alert), hard-fails on zero recipients (logs WARN, acks — no DLQ for M2; that's M3+). Emits the new envelope shape with Channel + Endpoint instead of the M1 FCMToken+Locale.
    • cmd/deliverd/main.go: envelope struct updated to match the M2 routerd output. Only field renames; the FCM HTTP v1 body shape is unchanged.
    • M2_VERIFICATION.md: 7-step plan + hard-fail scenario.
    • scripts/m2_smoke.sh: automated runner for steps 2-6.
    • M2_SMOKE_LOG.md: per-step results, honest flags.

    Live smoke results (all green):

    • step 2: warning, db-prod-04 → Alice only
    • step 3: critical, db-prod-04 → Alice + Bob
    • step 4: critical, db-prod-03 (rule match) → Alice + Bob
    • step 5: warning, db-prod-03 (rule match but min filter) → Alice only (Bob filtered by min_severity=critical)
    • step 6: inminent_colapse → Alice + Bob + Carol (Carol's quiet_hours=00:00-23:59 bypassed)
    • step 7: subscriptions paused → routerd logs WARN msg="zero recipients, dropping", no delivery 9 deliveries, 0 failures, 0 retries.

    Per the user's M2 Q1/Q2/Q3 answers:

    • Q1: seed = 3 individuals + 1 group + 1 source + 4 subscriptions covering all 4 scenarios (DONE; one of the 4 was dropped to 3 because we folded "fcm with min=critical AND in quiet hours" into a single individual — the bypass test is still valid via Carol).
    • Q2: hard-fail on zero recipients with a WARN log (DONE; the routerd logger is the notification path for M2; M3+ will add a dlq.no_recipients.<company> subject).
    • Q3: (a) emit the row, drop the channel at the SQL filter (DONE; WHERE f.channel = 'fcm' in the resolver).

    What's NOT in M2 (and not supposed to be):

    • real FCM (M3)
    • telegram, sms, email, slack, teams (M3+)
    • retry + DLQ (M3)
    • per-source HMAC secret in DB (M5 when mTLS / API-key path comes in)
    • full match_expr language (M6+); M2 supports category, severity, data.*, all
    • per-rule priority semantics with "first match wins unless continue=true" (M3+); M2 unions all matching rules
    • routing rule "drop" target type (schema supports it, no test for it; the SQL filters it via the candidate union)
    • Timezone support beyond UTC (M2 falls back to UTC if time.LoadLocation(tz) fails)

    Pushed: 280e048 on master (3 commits: a29c2d8 + 1e9eca9 + 280e048).

    2026-06-14 — M3 shipped (Telegram delivery + bot commands)

    What landed (~2k LoC Go + ~80 LoC SQL across these commits):

    • migrations/004_telegram.up.sql (+ .down.sql): telegram_bots table (one row per (company, bot); M3 supports one bot per company), and 4 new columns on individuals: telegram_chat_id, telegram_user_id, telegram_invite_code, mute_until. Two partial unique indexes for fast /start <code> lookup and "is this telegram_user_id already linked" check.
    • migrations/seed_m3.sql: inserts the per-company telegram_bots row, sets telegram_invite_code on all three individuals, and pre-links Alice to a fake Telegram account (chat_id=1001, user_id=900001) so the very first POST exercises both fcm and telegram delivery paths. Also expands every subscription's channel_mask to ["fcm","telegram"] (M2 was ["fcm"]).
    • cmd/seed/main.go: runner now applies seed.sql, seed_m2.sql, seed_m3.sql in lexical order. All three are idempotent.
    • cmd/deliverd/ → cmd/deliverd-fcm/: rename. The M2 deliverd is now deliverd-fcm — per the user's M3 Q2 answer (two per-channel binaries, not one with a registry).
    • cmd/deliverd-telegram/: new binary. Subscribes to deliveries.telegram.<company_id>, builds a severity-prefixed text message, posts to the Bot API's sendMessage endpoint, writes a deliveries row. Single-attempt (no retry, no DLQ). M9 adds the chain.
    • cmd/telegramd/: new binary. Loads the active bot list from telegram_bots at startup, long-polls getUpdates, dispatches commands to the handler, and replies via sendMessage. One process per deployment; per-bot sharding can come later if needed.
    • internal/telegram/: 3 files, ~640 LoC.
      • client.go — BotClient interface + HTTP impl.
      • commands.go — text → Command parser (/start, /subscribe, /unsubscribe, /preferences, /status, /mute, /unmute). /mute accepts 2h, 30m, 90s, or until 18:00.
      • handler.go — Command → DB updates. /start is atomic claim-by-invite-code; /subscribe upserts a subscription with the requested min_severity.
      • commands_test.go — 7 subtests, 100% pass.
    • internal/routing/routing.go: the resolver's single CTE now UNION-ALLs an fcm_rows branch (joins on fcm_tokens) and a tg_rows branch (filters on telegram_chat_id IS NOT NULL). Same cost as M2; one extra row in the join key set. Hard-fail on zero targets unchanged.
    • testfakes/faketgmd/: ~350 LoC. Fake Bot API server with /admin/queue (queue an incoming update), /admin/sent (read every sendMessage call), and /admin/reset (clear state). In-memory only. Used purely for the smoke test.
    • docker-compose.yml + Dockerfile: add deliverd-telegram (port 8821), telegramd (port 8822), faketgmd (port 8830). Grafana is now 3001:3000 because :3000 is held by gogs on this host.
    • M3_VERIFICATION.md: 8-step manual + automated walkthrough.
    • M3_SMOKE_LOG.md: per-step results, honest flags.
    • scripts/m3_smoke.sh: automated runner for steps 2–8.
    • README + PROMPT + SPEC §23 bumped to "shipped 2026-06-14".

    Live smoke results (all green):

    • step 2: warning → Alice fcm + Alice telegram (Bob unlinked)
    • step 3: critical → Alice fcm + Alice telegram + Bob fcm
    • step 4: Bob /start acme-bob-002 via faketgmd admin queue → handler atomically claims the code, telegram_user_id=900002, bot reply "Linked. Welcome, Bob SRE."
    • step 5: critical → Alice fcm+telegram + Bob fcm+telegram (Bob telegram now fires)
    • step 6: inminent_colapse → Alice fcm+telegram + Bob fcm+telegram + Carol fcm (Carol never linked; quiet hours bypassed)
    • step 7: Bob /preferences → bot reply "Your subscriptions: …"
    • step 8: faketgmd received 8 sendMessage calls

    14 deliveries across 4 unique alert_ids, 0 failures, 0 retries.

    Per the user's M3 Q1/Q2/Q3 answers:

    • Q1: faketgmd is shipped as a test fake (DONE; M5+ can add a webhook-based "real" mode).
    • Q2: two per-channel binaries, deliverd renamed to deliverd-fcm (DONE; deliverd-telegram is its own binary).
    • Q3: channel expansion on Bob and Carol's subscriptions (DONE; M2's ["fcm"] → M3's ["fcm","telegram"] via the seed migration).

    What's NOT in M3 (and not supposed to be):

    • real FCM auth (M9/M11)
    • SMS, email, Slack, MS Teams, voice (M3+)
    • webhook mode for the bot (M5/M9)
    • bot token encryption at rest (M11, security milestone)
    • retry + DLQ for telegram delivery (M9)
    • per-company bot token resolution in deliverd-telegram (currently reads env-supplied default; M3+ looks up by company_id encoded in alert or subject)
    • markdown/HTML formatting in telegram messages (M3.5+)
    • per-rule priority semantics with "first match wins unless continue=true" (deferred from M2; the unioned-rules behavior holds for M3)

    Pushed:

    2026-06-14 — M4 shipped (MQTT ingest + per-source ACL)

    What landed (~750 LoC Go + ~60 LoC config across these commits):

    • internal/mqttclient/: new package. Thin wrapper around paho.MQTT that hides the token-on-publish option behind a single error-returning Publish(), sets consistent LastWill / MaxInflight / AutoReconnect defaults, and formats client_id as <service>-mqtt-<host> so EMQX /admin/clients shows them cleanly. Used by both cmd/ingestd (subscriber) and loadgen/cmd/mqtt (publisher).
    • cmd/ingestd/mqtt.go: new file. The MQTT subscriber runs every message through the same processDeps.ProcessAlert pipeline as the HTTP POST handler — parse → validate → HMAC verify → rate-limit → dedupe → publish to NATS. The only MQTT-specific code is the topic parser (parseIncomingTopic → ba/<co>/<src>/incoming, 4 segments, last = incoming) and the envelope sniffer (accepts both {alert, auth} envelope and bare alert bodies for future broker-native clients).
    • cmd/ingestd/main.go: wires up the MQTT path. New env vars BA_INGESTD_MQTT_BROKER, BA_INGESTD_MQTT_USERNAME, BA_INGESTD_MQTT_PASSWORD, BA_INGESTD_MQTT_SUBSCRIBE (ba/+/+/incoming default).
    • internal/observability/metrics.go: adds the MQTTMessages CounterVec (labels: result=received | accepted | deduped | bad_topic | bad_signature | invalid_json | <other RejectReason>). Exposed as ba_ingestd_mqtt_messages_total.
    • loadgen/cmd/mqtt/: new binary. Same severity mix, dedupe ratio, and burst mode as loadgen-http. Publishes to ba/<co>/<src>/incoming with the <src>-<co> user and HMAC secret for auth.
    • deploy/emqx/acl.conf: Erlang-term ACL rules. Each source can only publish to its own ba/<co>/<src>/incoming; ingestd can subscribe to ba/+/+/incoming; default deny on #. Re-read on EMQX SIGHUP.
    • deploy/emqx/auth-built-in-db-bootstrap.csv: per-username auth. <source_id>-<company_id> for sources, ingestd for the subscriber. Password == HMAC secret so the same secret serves both MQTT auth and per-message HMAC.
    • deploy/emqx/README.md: explains the auth model, the env-var-vs-emqx.conf precedence (env vars win in EMQX 5.x), and the emqx_ctl listeners restart command for hot-reloading acl.conf.
    • docker-compose.yml: switches the emqx service from volume-mounting emqx.conf (which EMQX 5.x rejects as a partial config with node.cookie required_field) to env-var config. The EMQX_AUTHENTICATION__1__* and EMQX_AUTHORIZATION__* env vars are the only way to inject chain-level config in 5.x. acl.conf and the bootstrap CSV stay as volume mounts (their files are partials, which is fine).
    • scripts/m4_smoke.sh: end-to-end smoke. 5 steps (1 alert, 5 alerts with dedupe, bad sig, ACL violation, bad json). Builds loadgen-mqtt and the 3 failure-path test binaries into /tmp/ on first run. 12 deliveries, 0 failures, 3 consecutive green runs.
    • M4_VERIFICATION.md + M4_SMOKE_LOG.md: the spec-style step-by-step + the live run results.

    Why MQTT in v1: SPEC §18 calls for HTTP+MQTT as the v1 transports. M0–M3 ship HTTP; M4 ships MQTT. Sources that prefer a broker (Prometheus alertmanager webhook-bridge, Grafana, IoT) can publish to ba/<co>/<src>/incoming without writing a webhook client.

    Why per-source ACL: the broker is the first line of defense. A source that gets compromised can only spam its own topic — which still goes through the HMAC gate at ingestd, so a forged message without the secret is rejected in step 4. ACL stops cross-tenant spam; HMAC stops forged messages; rate-limit stops flood; dedupe stops burst; M9 adds the circuit breaker and quarantine.

    Why env-var config beats emqx.conf: EMQX 5.x's HOCON config is layered as emqx.conf → base.hocon → cluster.hocon → env vars (highest precedence). The double-underscore separator in env-var names maps to nested HOCON keys. emqx.conf is a full config — partial overrides get rejected with node.cookie required_field. The env-var approach lets us set just the auth + authz chains without restating the entire base.hocon.

    What's NOT in M4 (and not supposed to be):

    • Per-IP concurrency cap (M5 with WS, SPEC §22 layer 2)
    • Circuit breaker + quarantine (M9)
    • TLS to EMQX (M11, security milestone)
    • Persistent sessions (M11)
    • Per-company bot token resolution for sources that share a single EMQX user across companies (M3+ generalization)
    • HTTP-style signature in a header (MQTT has no headers; the auth field in the JSON envelope is the equivalent)
    • QoS 2 (M11, when persistent sessions land)

    Pushed: bc907d9 on master (1 commit for the code; this PROMPT bump + README + SPEC bump + the EMQX env-var refactor consolidated into a single follow-up commit).

    2026-06-14 — M5 shipped (WebSocket ingest + live tail + per-IP cap)

    What landed (~650 LoC Go + 2 SQL files across these commits):

    • migrations/005_ws.up.sql: adds max_concurrent_connections column to sources and companies (defaults 32 and 64).
    • internal/concurrency/perip.go: tiny sync.Map[ip]atomic.Int64 with Acquire/Release/InUse and a janitor goroutine that prunes entries idle > 5 min. No Redis on this hot path — would self-DoS.
    • internal/tailhub/hub.go: in-process pub/sub. Subscribe returns a *Subscription with a buffered chan *Event (size 64) and a per-sub Drops counter; Publish is best-effort, drops on full. Filter.CompanyID matches single-company or all (empty).
    • internal/wsclient/client.go: thin gorilla/websocket wrapper. Connect does the auth round-trip and stashes the auth reply; SendAlert does a single WS write; Close sends CloseMessage(1000) then TCP close. Used by loadgen-ws and the smoke drivers.
    • loadgen/cmd/ws/main.go: loadgen-ws binary. Default normal profile (30% dedupe). 3 modes: normal, burst.
    • loadgen/cmd/m5drivers/tail/main.go: /tmp/m5-tail-test driver used by scripts/m5_smoke.sh steps 6 + 7.
    • cmd/ingestd/ws.go: WS ingest at GET /v1/ingest/ws. Auth via first frame {api_key}. The handler does per-IP Acquire (rejects + 503 on cap exceeded), then loops on alert frames, sniffs {alert, auth} envelope vs bare body, and calls the shared scoped.ProcessAlert. Per-IP Release runs in the handler's defer.
    • cmd/ingestd/wstail.go: live tail at GET /v1/tail/ws. Token via Authorization: Bearer, X-BA-Tail-Token, or ?token=. Optional ?company_id= filter. Subscribe happens BEFORE the upgrade so events in the dial→subscribe window are not lost (a real bug we caught in M5 verification — see PROMPT bug #1 below).
    • cmd/ingestd/process.go: processDeps now has Tail *tailhub.Hub and Transport string. ProcessAlert calls d.Tail.Publish(tailhub.FromAlert(&a, d.Transport)) on the accepted path, so the tail fans out alerts from HTTP, MQTT, and WS through the same hub.
    • cmd/ingestd/main.go: wires concurrency.NewPerIP, tailhub.NewHub, and the two new dep structs.
    • internal/observability/metrics.go: 5 new metrics — WSMessages, WSConnections, ConnectionRejected, TailSubscribers, TailDropped — all registered with the ingestd service label.
    • docker-compose.yml: BA_INGESTD_TAIL_TOKEN and BA_INGESTD_MAX_CONCURRENT_PER_IP env vars on ingestd.
    • scripts/m5_smoke.sh: 7-step live driver. Auto-builds /tmp/loadgen-ws, /tmp/m5-tail-test, /tmp/m5-perip-test, /tmp/m5-badkey on first run.

    Bug #1 — Tail subscribe vs upgrade race The first M5 verification run showed the live tail seeing subs=0 at publish time even though the tail's gauge was 1. Root cause: Subscribe was called after the WS upgrade, leaving a window where the client thought it was connected but ingestd hadn't subscribed yet. The WS alert sent during that window was published to 0 subscribers. Fix: subscribe first, then upgrade (so the client's dial-completion implies the subscription is in place). This is documented inline in cmd/ingestd/wstail.go.

    Bug #2 — Tail defer ordering The first run of the gauge-update defer fired while the subscription was still in the hub's map (defers run LIFO, so the metric defer fired before the unsubscribe defer). Result: the gauge stayed at 1 after the client disconnected. Fix: combine unsubscribe + gauge update into a single defer that runs after the stream loop exits.

    Bug #3 — Badkey envelope shape The first M5 step-4 test sent the HMAC as auth: {"hmac": "v1=..."} (a map). The server's envelope parser declared auth as a string, so the unmarshal failed, the parser fell back to using the whole envelope as the alert body, and validation rejected the alert with "company_id missing". Fix: send auth as a string "t=...,v1=...", same shape as the MQTT envelope. scripts/m4_smoke.sh already had this correct — copy- pasted the pattern into M5.

    What's NOT in M5 (and not supposed to be):

    • HTTP POST /v1/ingest per-IP cap (M10 — the M5-bump in SPEC §22; we ship the gate now and the wire-up to HTTP is its own small PR)
    • JWT-based tail auth (M11 security milestone)
    • Cross-node tail fan-out via Redis pub/sub (M15+ when ingestd runs as a cluster)
    • A "send test alert" UI button in the tail (deferred — operators can wscat and a loadgen-ws --count 1)

    Pushed: 3 commits on master mirroring M0–M4's pattern:

    1. M5(1/3): code (migrations + 3 new packages + ws.go + wstail.go + process.go + main.go + metrics + loadgen-ws)
    2. M5(2/3): verified (M5_VERIFICATION.md + scripts/m5_smoke.sh + M5_SMOKE_LOG.md, 3 consecutive green runs, 13/13 checks each, +36 deliveries cumulative)
    3. M5(3/3): this PROMPT bump + README + SPEC §23.

    2026-06-14 — M6 shipped (Dedupe + ×N suffix + sliding TTL + free-for-dupes)

    What landed (~350 LoC Go + 1 SQL-friendly column + 2 new metrics across these commits):

    • internal/dedupe/dedupe.go: rewritten with a single Lua script that does SET-NX-or-INCR-EXPIRE atomically. The EXPIRE call on every duplicate is what makes the window slide — a steady stream of duplicates keeps the key alive. Sub-second windows in tests are handled by math.ceil in Lua (Redis EX requires an int).
    • internal/observability/maxseen.go: a new MaxSeen type that tracks per-key monotonic max. The gauge API doesn't expose Get(), so the canonical Prometheus pattern is to use a custom Collector; instead we keep the max in a sync.Map[source]uint32 and Set() the gauge only on growth. CAS-safe under concurrent writers.
    • cmd/ingestd/process.go: the order of operations is now HMAC verify → dedupe → rate limit (per-source, per-company). A duplicate (isNew=false) does not burn a token in either bucket — the rate limit gate is "free for dupes" as agreed. The duplicate still propagates to NATS so the router can fan out the alert with the updated dedupe_count. The M2 router, M3 deliverers, and the M5 tail hub all see the count climb.
    • cmd/ingestd/main.go: wires observability.NewMaxSeen() into processDeps so HTTP, MQTT, and WS transports share one in-process state and one max-observed gauge.
    • cmd/deliverd-telegram/main.go: formatMessage takes a dedupeCount uint32 and appends (×N) to the title when N > 1. Below 2, the title is unchanged.
    • cmd/deliverd-fcm/main.go: notification.title gets the same suffix. The raw count also goes into data.dedupe_count so a native Android client can render it however it likes.
    • loadgen/cmd/{http,mqtt,ws}/main.go: all three loadgens gain a --dedupe-key flag that forces a specific key on every alert. This is what the ×N smoke needs (one key, N copies, N=600 in the rate-limit test).
    • internal/config/config.go + docker-compose.yml: BA_INGESTD_DEDUPE_TTL_SECONDS env var, default 300s (was 60s hard-coded).
    • internal/observability/metrics.go: 2 new metrics — ba_ingestd_dedupe_collapsed_total{source} (counter that ticks on every isNew=false hit) and ba_ingestd_dedupe_count_max_observed{source} (gauge that climbs to the highest dedupe_count seen for that source since process start).

    Bug #1 — int-seconds truncation broke sub-second test windows The first dedupe test with a 500ms window failed: the Lua script received 0 for the TTL and returned an error ("invalid ttl"). Go's int(500ms.Seconds()) truncates to 0, which is below the script's > 0 guard. Fix: pass the TTL as a float (Go's d.window.Seconds() returns float64) and round up in Lua with math.ceil. Test sleeps were also bumped to 1.3s to account for the rounded TTL.

    Bug #2 — Prometheus Gauge doesn't expose Get() The "max dedupe count" gauge needs to only tick upward, but Prometheus Gauges can be Set() to any value. The canonical pattern is a custom Collector that knows the in-process state. That's a lot of boilerplate for one metric. The pragmatic alternative: keep the max in a sync.Map ourselves, and call Set() on the gauge only when the new value strictly exceeds the previous max. The MaxSeen type is thread-safe (CAS on conflict) and the gauge becomes a read-only view of our in-process state. 5 race-tested unit tests cover the corner cases.

    Decision: free-for-dupes applies to BOTH rate-limit buckets The user picked "both buckets free" over "per-source only" in the M6 design conversation. The reasoning: a duplicate is a duplicate, and the rate limit exists to backpressure new alert volume, not to charge attempt volume. A dupe storm of 600/s through a 100/s bucket now costs 1 token instead of 600. The downside (a bad source with a valid HMAC could pump dupes for free) doesn't apply because they already have full source-level access — rate limit isn't a security control, it's a backpressure mechanism.

    What's NOT in M6 (and not supposed to be):

    • Router-level collapse on dedupe_key — a true "one message per burst" would require the router to collapse before fanning out, instead of fanning out N alerts with climbing dedupe_count. M6.5+ when the M9 dashboard traffic pattern justifies it.
    • Per-source override of max_concurrent_connections — the migration added the column but the M6 path still reads the env default (32). M6.5 or M10.
    • Server-derived dedupe_key from a hash of (title+body+labels) — currently the source must supply the key. M11 lets sources opt in to a server-hash fallback for duplicate-prone senders that don't compute their own key.
    • Per-company rollup of the max-observed gauge — the gauge is per-source only. M9 dashboard work.

    Pushed: 3 commits on master mirroring M0–M5's pattern:

    1. M6(1/3): code (dedupe sliding-window Lua, MaxSeen, metrics, process.go reorder, deliverer ×N suffix, loadgen --dedupe-key, config + compose, 5 new maxseen tests, 2 new dedupe tests)
    2. M6(2/3): verified (M6_VERIFICATION.md + scripts/m6_smoke.sh + M6_SMOKE_LOG.md, 3 consecutive green runs, 11/11 checks each, 600-alert dupe storm confirms free-for-dupes, 0 rate-limit hits)
    3. M6(3/3): this PROMPT bump + README + SPEC §23.

    2026-06-14 — M6.5 shipped (Router-level dedupe collapse)

    The natural follow-on to M6, falling out of the M6 "What's NOT" list. M6 gave us "the recipient sees ×N" (100 messages, the last one tagged). M6.5 gives us "the recipient sees 1 message" (collapsed by the router, not 100).

    New code (~700 LoC Go + 1 new test binary + 1 new env var):

    • internal/dedupe/collapser.go (NEW, ~200 LoC): the Collapser. Public API: NewCollapser(flushMs, onFlush), Observe(sourceID, dedupeKey, alert) Decision, Run(ctx), FlushAll(). The Decision is one of:

      • Passthrough — empty dedupe_key; deliver immediately
      • CollapseNew — first arrival of (source, key); caller does the expensive work (resolve recipients) and caches it
      • CollapseDupe — subsequent arrival within the window; stored alert's dedupe_count already updated Per-source isolation enforced by keying on (source_id, dedupe_key), not on dedupe_key alone.

    • internal/dedupe/collapser_test.go (NEW, ~250 LoC, 7 tests, all green): passthrough-on-empty-key, single-collapse-100-alerts, per-source-isolation, max-wait-re-flush, empty-key-doesn't-block-real-key, 8-goroutine-concurrent-same-key-storm, FlushAll-drain.

    • cmd/routerd/collapse.go (NEW, ~250 LoC): wires the Collapser into routerd. The router holds a tiny fanoutState (source+key → cached resolved targets) so duplicate alerts skip the DB-bound ResolveTargets call. The flush callback reads the cached targets and publishes one delivery per (target, channel) with the final dedupe_count.

    • cmd/routerd/main.go (refactored): handleOne routes through observeAndFanout. The Collapser's Run loop starts in a goroutine in consume(). FlushAll() is called on graceful shutdown so the last few pending collapses still get delivered. Config now loaded via config.LoadRouterd() (was LoadCommon).

    • internal/config/config.go: new Routerd struct with DedupeFlushMs field. Env var BA_ROUTERD_DEDUPE_FLUSH_MS, default 2000ms.

    • testfakes/tailcount/main.go (NEW, ~70 LoC): a tiny Go helper that subscribes to the M5 tail WS and counts events, filtering on a substring. Used only by the smoke; no production code touches it.

    • loadgen/cmd/ws/main.go (small change): makeAlert injects the --dedupe-key into the title as LG M5 burst <key> #N when forced. The compact tail event includes title but NOT dedupe_key (by design), so the smoke needs to filter on something the operator can see.

    • docker-compose.yml + .env.example: BA_ROUTERD_DEDUPE_FLUSH_MS=2000 documented.

    Two real bugs caught + fixed in M6.5 verification:

    1. Tail filter was on dedupe_key but the compact tail event doesn't include it. The tail event JSON is just {alert_id, company_id, source_id, severity, title, received_at, transport, dedupe_count} — no dedupe_key field by design (M5 spec keeps the event small). Fixed by making the loadgen put the key in the title, and the smoke filters on the title prefix instead. The dedupe_key field is preserved in the payload (for the deliverers) but not in the tail event.

    2. Globex had no recipients in the dev DB. The dev schema only seeds acme; the m6 smoke never sent globex alerts. Step 5's per-source-isolation test surfaced this ("no recipients resolved" for grafana). The smoke now auto-seeds the globex company, source, individual, and subscription at the top (idempotent INSERT ... ON CONFLICT DO NOTHING). This is a smoke helper, not a migration — the real seed lives in the M2 migration files. The smoke needs globex to exist for Step 5; the production code doesn't care.

    3. Step 3 (tail + collapse) needed the burst to fit inside one flush window. First version sent 25 alerts at 10/s = 2.5s, which crossed the 2s flush boundary and produced 2 messages for one burst. Bumped the rate to 50/s = 0.5s, so the whole burst is inside a single window. The test still proves the headline behavior (1 message + tail sees the storm), but the timing has to be tuned to the configured flush window.

    Design choice recap (per the M6.5 scope conversation):

    • Router-side, max-wait debounce. Ingest publishes unconditionally (so alerts hit durable storage + tail). The collapse happens between NATS and the deliverers, i.e. in the router.
    • 2s default flush window. Long enough to coalesce most bursts; short enough that critical alerts feel instant. Continuous stream re-flushes every window.
    • Per-source isolation. Same key from two sources is two separate collapses. We considered global collapse but it has bad blast-radius properties (one noisy source could shadow another). Per-source-only ships.
    • Cached resolved targets. The first arrival does the DB-bound ResolveTargets call. Subsequent dupes update the dedupe_count on the stored alert and return. Net: 100-alert burst produces 1 DB call, not
      1. (This is the biggest perf win — without it, the 100-alert burst would have done 100 DB queries, which is the original M2–M5 cost we were trying to avoid.)
    • Trust the ingestd's count. The router doesn't INCR itself; it takes the max of the stored and incoming values. Ingestd is the canonical counter; this avoids a second Redis round-trip per arrival.

    What's NOT in M6.5 (deferred):

    • Active collapse metrics — no Prometheus metric for "how many collapses happened" or "how many alerts were collapsed". The router logs them at info level with dedupe_count, but we don't have a ba_routerd_collapse_total{source} counter yet. M7 dashboard work.
    • Per-source override of BA_ROUTERD_DEDUPE_FLUSH_MS — a noisy source might want a longer window. The migration added sources.dedupe_flush_ms but the M6.5 path uses the env default for all sources.
    • Collapse-across-sources — currently per-source. Could be a config flag but the blast-radius worries are real.
    • Tombstone on collapse — the collapsed delivery currently looks like a normal alert. A future M7 could add a collapsed_count: N field to the payload so deliverers can render "5 messages collapsed" in addition to the (×N) suffix.

    Pushed: 3 commits on master mirroring M0–M6's pattern:

    1. M6.5(1/3): code (Collapser + collapse.go wiring + config + compose + 7 new collapser tests)
    2. M6.5(2/3): verified (M6.5_VERIFICATION.md + scripts/m6.5_smoke.sh + M6.5_SMOKE_LOG.md, 3 consecutive green runs, 9/9 checks each, 100-alert burst → 1 message confirmed)
    3. M6.5(3/3): this PROMPT bump + README + SPEC §23.

    M7 (data tier) — Timescale 7d hot + ClickHouse archive

    What's in M7:

    • migrations/006_timescale.{up,down}.sql — deliveries → hypertable on created_at, 1d chunks, 7-day retention policy. PK rebased from (id) to (id, created_at) because Timescale requires the partition column in any UNIQUE/PK constraint. Down-migration uses the shadow-table rename pattern so it's reversible without data loss.
    • migrations/clickhouse_schema.sql — hand-runnable ops reference for ba_archive.deliveries_archive (MergeTree, partition by toYYYYMM(created_at), TTL 365 days) + deliveries_per_company_daily_mv (SummingMergeTree aggregate for M9 dashboards). Renamed from 007_clickhouse.up.sql to keep the file out of the seed's *.up.sql glob (the seed applies files via Postgres pool.Exec, which would have failed on the pure ClickHouse DDL). The runtime source of truth is archiverd.ensureCHSchema (idempotent via IF NOT EXISTS).
    • internal/archiver/archiver.go (~350 LoC) — RunOnce(ctx, opts) → Report. Drain loop, advisory lock (pg_try_advisory_lock(0xBA21B0DA)), FOR UPDATE SKIP LOCKED select, CH HTTP POST INSERT, idempotent ensureCHSchema. Time marshalling via chTime / chTimeOrEmpty helpers — CH 24.10 cannot parse Go's RFC3339Nano for DateTime64(3, 'UTC').
    • cmd/archiverd/main.go (~150 LoC) — hourly loop (first run immediate), /health (200 if last run < 2× cadence, 503 otherwise), /metrics (ba_archiverd_rows_archived_total{table}, ba_archiverd_last_run_*).
    • internal/config/config.go — Archiverd struct + LoadArchiverd().
    • docker-compose.yml — archiverd service on host port 8804.
    • Dockerfile — archiverd in the build chain.
    • .env.example — BA_ARCHIVERD_RUN_EVERY_SECONDS=3600, BA_ARCHIVERD_OLDER_THAN_HOURS=168, BA_ARCHIVERD_BATCH_SIZE=10000.
    • scripts/m7_smoke.sh — 4-step, 9-check smoke.
    • scripts/m7_smoke_runner.sh — wrapper that stops the archiverd compose service first to release the advisory lock (so the smoke's one-shot exec is the deterministic holder).
    • M7_VERIFICATION.md + M7_SMOKE_LOG.md — three green runs, 9/9 checks each, on the remote playground parres (192.168.44.94).

    SPEC drift noted (not a bug):

    • SPEC §23 originally listed alerts as a hypertable alongside deliveries. Code inspection: alerts is never written to Postgres — alerts flow through NATS JetStream only (cmd/ingestd/process.go → nc.Publish("alerts", ...)). The alerts table is absent from migrations/001_init.up.sql. The routerd dispatcher writes per-recipient rows to deliveries, not alerts. Decision: ship M7 with deliveries only. Document drift in verification.

    Bugs caught during M7 build (all fixed in this milestone):

    1. CH TTL on DateTime64 rejected. `TTL created_at
      • INTERVAL 365 DAY→TTL toDateTime(created_at) + INTERVAL 365 DAY. CH requiresDateTimein TTL, not DateTime64`.
    2. Go time.Time JSON format. Default RFC3339Nano with T and Z not parseable by CH. Fix: chTime(t) → "2006-01-02 15:04:05.000".
    3. Redundant SET LOCAL lock_timeout. SKIP LOCKED is non-blocking by definition. Removed.
    4. Docker RUN chain masks errors. Chained && RUN builds all binaries in one cache layer. Failure produces generic "exit code 1" with no stderr. Root cause: loadgen/cmd/mqtt/main.go referenced *dedupeKey on line 114 without declaring the flag. The same fix that the M6.5 postmortem flagged for cmd/http/main.go had not been extended to cmd/mqtt/main.go. Fix: add the --dedupe-key flag to the mqtt var block.
    5. CH writes need POST, not GET. curl "$URL/?query=..." returns 405 for INSERTs in CH 24.10. Use http.NewRequestWithContext(ctx, "POST", chURL+"/", bytes.NewBufferString(s)).
    6. seed would have failed on ClickHouse DDL. The seed service globs migrations/*.up.sql and applies every file via pool.Exec. 007_clickhouse.up.sql is pure ClickHouse DDL. Renamed to migrations/clickhouse_schema.sql (no .up.sql suffix) so the seed glob skips it. The file is preserved as a hand-runnable ops reference; the runtime source of truth is archiverd.ensureCHSchema.

    Remote-playground deployment notes:

    • No docker compose plugin in the remote's apt repo. Standalone v2.27.0 binary at /usr/local/lib/docker/cli-plugins/docker-compose (with a symlink to /usr/local/bin/docker-compose for shell compat).
    • Docker Hub pulls broken on the remote (IPv6 only, IPv6 path broken). sysctl -w net.ipv6.conf.all.disable_ipv6=1 fixes it. All 9 base images pulled in 66 s after the disable.
    • Local docker save → remote docker load is too slow on the local 1-CPU QEMU host. Cold-cache ZFS reads are 200-400 KB/s; the 5.3 GB of image data would take 3+ hours. The remote (4 CPU, fast network) can pull all base images from Docker Hub directly after the IPv6 disable, and docker compose build on the remote compiles the 12 Go binaries in 11 seconds. Build on the remote, do not transfer pre-built images from local.

    M8 (DLQ + replay UI) — shipped 2026-06-14

    SPEC §23 M8 = DLQ for failed deliveries + replay UI. shipped. 6 commits mirroring M0–M7 (1/3 + 1b/3 + 1c/3 + 1d/3 are all code; 2/3 is the verification doc

    • smoke log; 3/3 is this PROMPT + README + SPEC bump). 3 consecutive 12/12 green runs of m8_smoke.sh on the local docker-compose stack; see M8_VERIFICATION.md + M8_SMOKE_LOG.md. Remote parres (192.168.44.94) was unreachable from this dev host (TCP RST on 22/80) so the live re-run there is left to the operator.

    What landed:

    • migrations/008_dlq.{up,down}.sql — deliveries_dlq Timescale hypertable, 1d chunks, 7d retention, PK (id, created_at). Extra columns: original_subject (for replay), discarded / discarded_at / discarded_by (for the operator's discard action).
    • internal/retry — bounded exp-backoff retry helper (10 attempts, base 100ms, cap 2s, budget 30s, ~12s total wall clock for a fully failing target). PermanentError short-circuit. 7 unit tests, 7/7 PASS.
    • internal/dlq — single Write() that INSERTs one row into deliveries_dlq. Per-attempt deliveries rows are left as-is so the audit trail is preserved.
    • cmd/deliverd-fcm + cmd/deliverd-telegram — refactored handleOne() to use retry.Run. On success, a 'sent' row. On exhaustion, a per-attempt 'failed' row + a final dlq.Write(). PermanentError on FCM/Telegram 4xx (excluding 408/429) so we don't burn the retry budget on a bad token or a missing chat.
    • cmd/admind — DLQ surface:
      • GET /v1/dlq?company_id=&channel=&alert_id= &include=all&limit=100&offset=0 (30d window, hides discarded by default)
      • GET /v1/dlq/{id} (single row w/ payload)
      • POST /v1/dlq/{id}/replay (re-publishes the original NATS envelope onto the original subject, then marks the row discarded)
      • POST /v1/dlq/{id}/discard (mark discarded; idempotent)
      • GET /dlq — minimal HTML page embedded via go:embed, light/dark theming, filter form, inline replay/discard buttons, plain ES5 JS.
    • internal/archiver — RunOptions now carries a []TableSpec; default plan covers both deliveries and deliveries_dlq (7d Postgres hot window). CH schema adds ba_archive.deliveries_dlq_archive (2y TTL) + ba_archive.deliveries_dlq_per_company_daily_mv (SummingMergeTree) for M9 dashboards.
    • testfakes/fakefcmd — runtime /control?fail=0|1 endpoint to flip failure mode without restarting the container (saves 30s+ per smoke cycle on the 1-CPU QEMU host).
    • docker-compose.yml + .env.example — wired BA_DELIVERD_* on both deliverd-fcm and deliverd-telegram.
    • scripts/m8_smoke.sh — 4-step, 12-check live smoke.

    Loose ends addressed:

    • archiverd drains deliveries_dlq (2y CH TTL, same 7d PG hot window as live deliveries).
    • Auth on /v1/dlq* is deferred to M11. M8 ships unauthenticated (LAN-only deploy assumed).

    Loose ends still open (deferred):

    • Materialized view deliveries_per_company_daily_mv could be exposed via admind for the dashboard (deferred to M9 observability).
    • Persist sysctl net.ipv6.conf.all.disable_ipv6=1 in /etc/sysctl.d/99-disable-ipv6.conf on the remote playground so it survives reboots.

    SPEC drift:

    SPEC §9 literally calls for backoff of 1s, 2s, 4s, … 512s = 1023s total. M8 ships defaults that cap per- attempt wait at 2s and total budget at 30s, terminating in ~12s. Operators who want the SPEC-literal behavior can opt in via the four BA_DELIVERD_* env vars. The defaults trade literal SPEC compliance for fast failure detection.

    Pushed: 6 commits on master mirroring the M0–M7 pattern (1/3 + 1b/3 + 1c/3 + 1d/3 are all code; 2/3 is the verification doc + smoke log; 3/3 is this PROMPT + README + SPEC bump):

    1. M7(1/3): Timescale 7d + ClickHouse archive + archiverd (code)
    2. M7(1b/3): move clickhouse_schema.sql out of seed's *.up.sql glob (bug 6 above)
    3. M7(1c/3): declare --dedupe-key flag in loadgen-mqtt (bug 4 above)
    4. M7(1d/3): add m7_smoke_runner.sh that stops archiverd before smoke
    5. M7(2/3): M7_VERIFICATION.md + M7_SMOKE_LOG.md (3 consecutive green runs, 9/9 checks each, on the remote playground parres 192.168.44.94)
    6. M7(3/3): this PROMPT bump + README + SPEC §23.

    M8 (this section supersedes the 'next' placeholder above):

    1. M8(1/3): DLQ schema + in-process retry + deliveries_dlq writer (8 files, ~1000 LoC)
    2. M8(1b/3): archiver DLQ drain + ClickHouse DLQ archive (2 files, ~240 LoC)
    3. M8(1c/3): admind DLQ endpoints + HTML UI (2 files, ~660 LoC)
    4. M8(1d/3): docker-compose / .env wiring (2 files, ~36 LoC)
    5. M8(2/3): M8_VERIFICATION.md + M8_SMOKE_LOG.md + m8_smoke.sh (3 consecutive 12/12 green runs on the local stack)
    6. M8(3/3): this PROMPT bump + README + SPEC §23.