broad-announce/M8_VERIFICATION.md

M8 Verification — DLQ + replay UI

Status: shipped 2026-06-14 Branch: master Commits: see git log --oneline | grep M8 Verified on: local 1-CPU QEMU host (broad-announce docker-compose stack)

This milestone closes the loop on the delivery path. Failed deliveries now go through a bounded exponential- backoff retry, terminate in a durable Dead-Letter Queue when exhausted, and the operator can inspect the DLQ via JSON API or HTML UI and either replay the failed delivery (re-publish onto the original NATS subject) or discard it. See SPEC §9 and §23.

What landed

Area	Change
Schema	migrations/008_dlq.{up,down}.sql — new deliveries_dlq Timescale hypertable (PK (id, created_at), 1d chunks, 7d retention, mirroring the live deliveries table). Extra columns: original_subject (for replay), discarded/discarded_at/discarded_by (for the operator's discard action).
Retry helper	internal/retry/retry.go — exp backoff (BaseDelay doubles, capped at MaxDelay), wall-clock Budget, ctx-cancel aware, PermanentError short-circuit. 10 attempts, base 100ms, cap 2s, budget 30s — total wall clock ~12s for a fully failing target. Configurable via env.
Retry tests	internal/retry/retry_test.go — 7 unit tests covering: first-try success, retry-then-succeed, exhaustion, permanent short-circuit, budget respect, ctx cancel, monotonic backoff. 7/7 PASS.
DLQ writer	internal/dlq/dlq.go — single Write() that INSERTs one row into deliveries_dlq. Per-attempt rows in the live deliveries table are left as-is (status='failed') so the audit trail is preserved.
Config	internal/config/config.go — four new env knobs on Common: BA_DELIVERD_MAX_ATTEMPTS (10), BA_DELIVERD_RETRY_BASE_MS (100), BA_DELIVERD_RETRY_MAX_MS (2000), BA_DELIVERD_RETRY_BUDGET_MS (30000). Other services ignore them.
FCM retry	cmd/deliverd-fcm/main.go — refactored handleOne() to use retry.Run. Per-attempt deliveries row + on exhaustion a dlq.Write(). PermanentError on FCM 4xx (excluding 408/429) so we don't burn the retry budget on a bad token. Per-attempt timeout 10s; outer Budget caps total wall clock.
Telegram retry	cmd/deliverd-telegram/main.go — same pattern. Per-attempt timeout 15s (Telegram can be slower). PermanentError on Telegram 4xx (excluding 429).
Archiver DLQ drain	internal/archiver/archiver.go — RunOptions now carries a []TableSpec; drainTable takes a TableSpec and selects/inserts/deletes per-table. Default plan covers both deliveries (live) and deliveries_dlq (forensic) with 7d Postgres hot window. ClickHouse schema adds ba_archive.deliveries_dlq_archive (2y TTL) + ba_archive.deliveries_dlq_per_company_daily_mv (SummingMergeTree) for M9 dashboards.
ClickHouse schema	migrations/clickhouse_schema.sql — hand-runnable ops reference, updated to mirror the new DLQ DDL.
Admind API	cmd/admind/main.go — GET /v1/ping (M0) + M8 endpoints: GET /v1/dlq (list/filter, 30d window, hides discarded by default), GET /v1/dlq/{id} (single row w/ payload), POST /v1/dlq/{id}/replay (re-publishes the original NATS envelope onto the original subject, then marks the row discarded), POST /v1/dlq/{id}/discard (mark discarded; idempotent).
Admind UI	cmd/admind/ui/dlq.html — minimal HTML page embedded via go:embed. Filter form (company, channel, alert_id, include-discarded), inline replay/discard buttons, /v1/dlq/{id} JSON link per row, light/dark theming via prefers-color-scheme. JS is plain ES5.
Test double	testfakes/fakefcmd/main.go — added a runtime /control?fail=0\|1 endpoint to flip the failure mode at runtime. Avoids restarting the container for the smoke test.
Compose	docker-compose.yml — wired the four BA_DELIVERD_* env vars on both deliverd-fcm and deliverd-telegram.
Env	.env.example — documented the four new knobs with the math behind the defaults.
Smoke	scripts/m8_smoke.sh — 4-step, 12-check, HMAC-signed HTTP ingest + runtime fail injection + replay + discard.

New env vars (deliverd-*)

Var	Default	Purpose
BA_DELIVERD_MAX_ATTEMPTS	10	total tries including the first
BA_DELIVERD_RETRY_BASE_MS	100	wait before the SECOND attempt; doubles each subsequent attempt, capped at RETRY_MAX_MS
BA_DELIVERD_RETRY_MAX_MS	2000	cap on the per-attempt wait
BA_DELIVERD_RETRY_BUDGET_MS	30000	wall-clock cap across all attempts; loop bails early if the next wait would exceed it

With the defaults, a fully-failing target terminates in ~12s. The smoke runs against the live defaults — the test does not slow down the run.

New endpoints (admind)

Method	Path	Purpose
GET	/v1/dlq?company_id=&channel=&alert_id=&include=all&limit=100&offset=0	list DLQ rows (30d window; hides discarded by default)
GET	/v1/dlq/{id}	single row with payload
POST	/v1/dlq/{id}/replay	re-publishes the original NATS envelope onto the original subject, then marks the row discarded (idempotent re-publish; failure leaves the DLQ row in place)
POST	/v1/dlq/{id}/discard	marks the row discarded (hidden from default list); idempotent
GET	/dlq	minimal HTML page (light/dark theming) with filter form + inline replay/discard buttons

Files

migrations/008_dlq.up.sql                 # deliveries_dlq hypertable, 7d retention
migrations/008_dlq.down.sql               # shadow-table rename (reversible)
migrations/clickhouse_schema.sql          # + deliveries_dlq_archive (2y TTL) + MV
internal/dlq/dlq.go                       # Write() — single INSERT into deliveries_dlq
internal/retry/retry.go                   # bounded exp-backoff retry helper
internal/retry/retry_test.go              # 7 unit tests, 7/7 PASS
internal/config/config.go                 # + 4 BA_DELIVERD_* knobs on Common
internal/archiver/archiver.go             # + TableSpec + DLQ drain
cmd/deliverd-fcm/main.go                  # M8 retry + dlq.Write
cmd/deliverd-telegram/main.go             # M8 retry + dlq.Write
cmd/admind/main.go                        # /v1/dlq + /v1/dlq/{id} + /v1/dlq/{id}/{replay,discard} + /dlq
cmd/admind/ui/dlq.html                    # embedded HTML page
testfakes/fakefcmd/main.go                # /control?fail=0|1 (runtime fail injection)
docker-compose.yml                        # + BA_DELIVERD_* on both deliverds
.env.example                              # + BA_DELIVERD_* with math
scripts/m8_smoke.sh                       # 4-step, 12-check live smoke
M8_VERIFICATION.md                        # this file
M8_SMOKE_LOG.md                           # 3 consecutive 12/12 runs

Behavior notes

• One row per attempt, not one row per alert. The M1 audit shape is preserved: each retry attempt INSERTs one deliveries row with attempts=N and the per-attempt last_error. Operators see "10 attempts, all failed" at a glance. The DLQ row is a summary: attempts=10 + the last error.

• No flip on the live row. Earlier drafts of dlq.Write() flipped the per-attempt deliveries rows to status='dlq'. That destroyed the audit trail — operators couldn't tell whether 1 attempt or 10 attempts failed. M8 ships without that UPDATE; the DLQ row is the source of truth for "this hit the DLQ".

• Replay is best-effort idempotent. A replay publishes the original NATS envelope onto the original subject, then Flush()es, then marks the DLQ row discarded. If the publish fails (broker down, etc.), the row stays live and the operator can retry. A second concurrent replay would publish twice — we don't dedupe, because the operator explicitly asked for a re-send. If the alert_id is in the M6 dedupe window, ingestd will treat the second arrival as a duplicate of the first, but the deliveries table gets a fresh row regardless.

• Discard is idempotent. A second discard is a 200 with {"already": true}. The handler is keyed on (id, created_at) to satisfy the Timescale hypertable composite PK.

• Retry is in-process, not JetStream redelivery. JetStream's redelivery timer is fixed at the consumer level and doesn't express per-attempt exp backoff. We want explicit config-driven backoff (SPEC §9: 1s, 2s, 4s, … up to 10 attempts). In-process retry is simpler and gives more control. The downside is that one stuck downstream can tie up a single consumer for up to Budget — but with the default 30s budget and 4-replica deliverds, that's bounded.

• PermanentError short-circuit. Both deliverd-fcm and deliverd-telegram classify 4xx (excluding 408/429) as permanent via &retry.PermanentError{Err: …}. FCM's 4xx means the token is bad, Telegram's 4xx means the chat is gone — retrying won't help. We use those 4xx codes to skip the remaining attempts.

• Auth is deferred. The /v1/dlq* endpoints are unauthenticated in M8. M11 will gate them behind an operator JWT. Until then, deploy behind a LAN boundary or a reverse proxy that enforces IP allowlists.

SPEC drift noted (not a bug)

SPEC §9 literally calls for backoff of 1s, 2s, 4s, … 512s = 1023s total. That ties up a single NATS consumer for ~17 minutes per stuck downstream. The M8 implementation caps per-attempt wait at 2s and total budget at 30s. With BA_DELIVERD_* env vars, an operator who wants the SPEC-literal behavior can opt in (base=1000, max=60000, budget=1200, attempts=10 → 1+2+4+8+16+32+60+60+60+60 = 303s). The defaults trade literal SPEC compliance for fast failure detection.

SPEC §23 doesn't list an M8.5. The deliverd-fcm and deliverd-telegram source comments still reference "M9 adds the retry + DLQ chain" — those are stale; M8 is the one. (Fixed in this milestone: see the updated headers.)

Latent bugs caught during M8 build (fixed in this milestone)

Bug 1 — dlq.Write() was over-aggressively flipping live rows

The first version of dlq.Write() did:

UPDATE deliveries SET status='dlq' WHERE alert_id=… AND status NOT IN ('sent','dlq')

A 10-attempt failure produced 10 deliveries rows (one per attempt, all status='failed') plus 1 deliveries_dlq row. The UPDATE then flipped all 10 of those failed rows to status='dlq', destroying the per-attempt audit trail. The smoke caught it: attempts=10 on the DLQ row but 0 rows in deliveries WHERE status='failed'. Fix: remove the UPDATE entirely. The DLQ row is the source of truth for "this hit the DLQ"; the per-attempt rows are the source of truth for "this attempt did X". They serve different purposes.

Bug 2 — LastError retained prior failure on a successful retry

The first version of retry.Run set res.LastError once and never cleared it. Test TestRunRetriesThenSucceeds failed: attempt 1 fails (sets LastError), attempt 2 succeeds, but res.LastError still had the attempt-1 error. Caught by unit test; fix: res.LastError = nil on success.

Bug 3 — fakefcmd had no way to fail mid-test

The M1 test double accepted --fail-rate=0.x at startup but the smoke needed to flip the failure mode after the happy-path step. Added a /control?fail=0|1 endpoint. The smoke uses this instead of restarting the fakefcmd container (which on this 1-CPU QEMU host takes 30s+ to rebuild the image).

Bug 4 — Compose's docker cp + restart doesn't pick up the new binary

docker cp new-binary old-container:/app/old-binary overwrites the file on the running container's writable layer, but the running process still has the old inode mapped. docker compose restart svc does NOT recreate the container; it just sends SIGTERM/SIGKILL to the same container, which then starts with the (new) on-disk binary — except: I observed that on this Compose 5.1.3 build, restart doesn't actually restart the container. Workaround: docker compose rm -sf svc && docker compose up -d --no-deps svc to fully recreate, then docker cp the new binary, then kill + up to pick it up. Documented in the developer runbook.