# M8 Verification — DLQ + replay UI

**Status:** shipped 2026-06-14
**Branch:** master
**Commits:** see `git log --oneline | grep M8`
**Verified on:** local 1-CPU QEMU host (broad-announce
docker-compose stack)

This milestone closes the loop on the delivery path.
Failed deliveries now go through a bounded exponential-
backoff retry, terminate in a durable Dead-Letter Queue
when exhausted, and the operator can inspect the DLQ
via JSON API or HTML UI and either replay the failed
delivery (re-publish onto the original NATS subject)
or discard it. See SPEC §9 and §23.

## What landed

| Area | Change |
| --- | --- |
| Schema | `migrations/008_dlq.{up,down}.sql` — new `deliveries_dlq` Timescale hypertable (PK `(id, created_at)`, 1d chunks, 7d retention, mirroring the live `deliveries` table). Extra columns: `original_subject` (for replay), `discarded`/`discarded_at`/`discarded_by` (for the operator's discard action). |
| Retry helper | `internal/retry/retry.go` — exp backoff (BaseDelay doubles, capped at MaxDelay), wall-clock Budget, ctx-cancel aware, PermanentError short-circuit. 10 attempts, base 100ms, cap 2s, budget 30s — total wall clock ~12s for a fully failing target. Configurable via env. |
| Retry tests | `internal/retry/retry_test.go` — 7 unit tests covering: first-try success, retry-then-succeed, exhaustion, permanent short-circuit, budget respect, ctx cancel, monotonic backoff. 7/7 PASS. |
| DLQ writer | `internal/dlq/dlq.go` — single `Write()` that INSERTs one row into `deliveries_dlq`. Per-attempt rows in the live `deliveries` table are left as-is (status='failed') so the audit trail is preserved. |
| Config | `internal/config/config.go` — four new env knobs on Common: `BA_DELIVERD_MAX_ATTEMPTS` (10), `BA_DELIVERD_RETRY_BASE_MS` (100), `BA_DELIVERD_RETRY_MAX_MS` (2000), `BA_DELIVERD_RETRY_BUDGET_MS` (30000). Other services ignore them. |
| FCM retry | `cmd/deliverd-fcm/main.go` — refactored `handleOne()` to use `retry.Run`. Per-attempt deliveries row + on exhaustion a `dlq.Write()`. PermanentError on FCM 4xx (excluding 408/429) so we don't burn the retry budget on a bad token. Per-attempt timeout 10s; outer Budget caps total wall clock. |
| Telegram retry | `cmd/deliverd-telegram/main.go` — same pattern. Per-attempt timeout 15s (Telegram can be slower). PermanentError on Telegram 4xx (excluding 429). |
| Archiver DLQ drain | `internal/archiver/archiver.go` — `RunOptions` now carries a `[]TableSpec`; `drainTable` takes a `TableSpec` and selects/inserts/deletes per-table. Default plan covers both `deliveries` (live) and `deliveries_dlq` (forensic) with 7d Postgres hot window. ClickHouse schema adds `ba_archive.deliveries_dlq_archive` (2y TTL) + `ba_archive.deliveries_dlq_per_company_daily_mv` (SummingMergeTree) for M9 dashboards. |
| ClickHouse schema | `migrations/clickhouse_schema.sql` — hand-runnable ops reference, updated to mirror the new DLQ DDL. |
| Admind API | `cmd/admind/main.go` — `GET /v1/ping` (M0) + M8 endpoints: `GET /v1/dlq` (list/filter, 30d window, hides discarded by default), `GET /v1/dlq/{id}` (single row w/ payload), `POST /v1/dlq/{id}/replay` (re-publishes the original NATS envelope onto the original subject, then marks the row discarded), `POST /v1/dlq/{id}/discard` (mark discarded; idempotent). |
| Admind UI | `cmd/admind/ui/dlq.html` — minimal HTML page embedded via `go:embed`. Filter form (company, channel, alert_id, include-discarded), inline replay/discard buttons, /v1/dlq/{id} JSON link per row, light/dark theming via `prefers-color-scheme`. JS is plain ES5. |
| Test double | `testfakes/fakefcmd/main.go` — added a runtime `/control?fail=0\|1` endpoint to flip the failure mode at runtime. Avoids restarting the container for the smoke test. |
| Compose | `docker-compose.yml` — wired the four `BA_DELIVERD_*` env vars on both `deliverd-fcm` and `deliverd-telegram`. |
| Env | `.env.example` — documented the four new knobs with the math behind the defaults. |
| Smoke | `scripts/m8_smoke.sh` — 4-step, 12-check, HMAC-signed HTTP ingest + runtime fail injection + replay + discard. |

## New env vars (deliverd-*)

| Var | Default | Purpose |
| --- | --- | --- |
| `BA_DELIVERD_MAX_ATTEMPTS` | 10 | total tries including the first |
| `BA_DELIVERD_RETRY_BASE_MS` | 100 | wait before the SECOND attempt; doubles each subsequent attempt, capped at `RETRY_MAX_MS` |
| `BA_DELIVERD_RETRY_MAX_MS` | 2000 | cap on the per-attempt wait |
| `BA_DELIVERD_RETRY_BUDGET_MS` | 30000 | wall-clock cap across all attempts; loop bails early if the next wait would exceed it |

With the defaults, a fully-failing target terminates in ~12s. The smoke runs against the live defaults — the test does not slow down the run.

## New endpoints (admind)

| Method | Path | Purpose |
| --- | --- | --- |
| `GET` | `/v1/dlq?company_id=&channel=&alert_id=&include=all&limit=100&offset=0` | list DLQ rows (30d window; hides discarded by default) |
| `GET` | `/v1/dlq/{id}` | single row with payload |
| `POST` | `/v1/dlq/{id}/replay` | re-publishes the original NATS envelope onto the original subject, then marks the row discarded (idempotent re-publish; failure leaves the DLQ row in place) |
| `POST` | `/v1/dlq/{id}/discard` | marks the row discarded (hidden from default list); idempotent |
| `GET` | `/dlq` | minimal HTML page (light/dark theming) with filter form + inline replay/discard buttons |

## Files

```
migrations/008_dlq.up.sql                 # deliveries_dlq hypertable, 7d retention
migrations/008_dlq.down.sql               # shadow-table rename (reversible)
migrations/clickhouse_schema.sql          # + deliveries_dlq_archive (2y TTL) + MV
internal/dlq/dlq.go                       # Write() — single INSERT into deliveries_dlq
internal/retry/retry.go                   # bounded exp-backoff retry helper
internal/retry/retry_test.go              # 7 unit tests, 7/7 PASS
internal/config/config.go                 # + 4 BA_DELIVERD_* knobs on Common
internal/archiver/archiver.go             # + TableSpec + DLQ drain
cmd/deliverd-fcm/main.go                  # M8 retry + dlq.Write
cmd/deliverd-telegram/main.go             # M8 retry + dlq.Write
cmd/admind/main.go                        # /v1/dlq + /v1/dlq/{id} + /v1/dlq/{id}/{replay,discard} + /dlq
cmd/admind/ui/dlq.html                    # embedded HTML page
testfakes/fakefcmd/main.go                # /control?fail=0|1 (runtime fail injection)
docker-compose.yml                        # + BA_DELIVERD_* on both deliverds
.env.example                              # + BA_DELIVERD_* with math
scripts/m8_smoke.sh                       # 4-step, 12-check live smoke
M8_VERIFICATION.md                        # this file
M8_SMOKE_LOG.md                           # 3 consecutive 12/12 runs
```

## Behavior notes

* **One row per attempt, not one row per alert.** The M1 audit shape is preserved: each retry attempt INSERTs one `deliveries` row with `attempts=N` and the per-attempt `last_error`. Operators see "10 attempts, all failed" at a glance. The DLQ row is a *summary*: `attempts=10` + the last error.

* **No flip on the live row.** Earlier drafts of `dlq.Write()` flipped the per-attempt `deliveries` rows to `status='dlq'`. That destroyed the audit trail — operators couldn't tell whether 1 attempt or 10 attempts failed. M8 ships without that UPDATE; the DLQ row is the source of truth for "this hit the DLQ".

* **Replay is best-effort idempotent.** A replay publishes the original NATS envelope onto the original subject, then `Flush()`es, then marks the DLQ row discarded. If the publish fails (broker down, etc.), the row stays live and the operator can retry. A second concurrent replay would publish twice — we don't dedupe, because the operator explicitly asked for a re-send. If the alert_id is in the M6 dedupe window, ingestd will treat the second arrival as a duplicate of the first, but the deliveries table gets a fresh row regardless.

* **Discard is idempotent.** A second discard is a 200 with `{"already": true}`. The handler is keyed on `(id, created_at)` to satisfy the Timescale hypertable composite PK.

* **Retry is in-process, not JetStream redelivery.** JetStream's redelivery timer is fixed at the consumer level and doesn't express per-attempt exp backoff. We want explicit config-driven backoff (SPEC §9: 1s, 2s, 4s, … up to 10 attempts). In-process retry is simpler and gives more control. The downside is that one stuck downstream can tie up a single consumer for up to Budget — but with the default 30s budget and 4-replica deliverds, that's bounded.

* **PermanentError short-circuit.** Both deliverd-fcm and deliverd-telegram classify 4xx (excluding 408/429) as permanent via `&retry.PermanentError{Err: …}`. FCM's 4xx means the token is bad, Telegram's 4xx means the chat is gone — retrying won't help. We use those 4xx codes to skip the remaining attempts.

* **Auth is deferred.** The /v1/dlq* endpoints are unauthenticated in M8. M11 will gate them behind an operator JWT. Until then, deploy behind a LAN boundary or a reverse proxy that enforces IP allowlists.

## SPEC drift noted (not a bug)

SPEC §9 literally calls for backoff of 1s, 2s, 4s, … 512s = 1023s total. That ties up a single NATS consumer for ~17 minutes per stuck downstream. The M8 implementation caps per-attempt wait at 2s and total budget at 30s. With `BA_DELIVERD_*` env vars, an operator who wants the SPEC-literal behavior can opt in (base=1000, max=60000, budget=1200, attempts=10 → 1+2+4+8+16+32+60+60+60+60 = 303s). The defaults trade literal SPEC compliance for fast failure detection.

SPEC §23 doesn't list an `M8.5`. The `deliverd-fcm` and `deliverd-telegram` source comments still reference "M9 adds the retry + DLQ chain" — those are stale; M8 is the one. (Fixed in this milestone: see the updated headers.)

## Latent bugs caught during M8 build (fixed in this milestone)

### Bug 1 — `dlq.Write()` was over-aggressively flipping live rows

The first version of `dlq.Write()` did:

```sql
UPDATE deliveries SET status='dlq' WHERE alert_id=… AND status NOT IN ('sent','dlq')
```

A 10-attempt failure produced 10 `deliveries` rows (one per attempt, all `status='failed'`) plus 1 `deliveries_dlq` row. The UPDATE then flipped all 10 of those `failed` rows to `status='dlq'`, destroying the per-attempt audit trail. The smoke caught it: `attempts=10` on the DLQ row but 0 rows in `deliveries WHERE status='failed'`. Fix: remove the UPDATE entirely. The DLQ row is the source of truth for "this hit the DLQ"; the per-attempt rows are the source of truth for "this attempt did X". They serve different purposes.

### Bug 2 — `LastError` retained prior failure on a successful retry

The first version of `retry.Run` set `res.LastError` once and never cleared it. Test `TestRunRetriesThenSucceeds` failed: attempt 1 fails (sets LastError), attempt 2 succeeds, but `res.LastError` still had the attempt-1 error. Caught by unit test; fix: `res.LastError = nil` on success.

### Bug 3 — `fakefcmd` had no way to fail mid-test

The M1 test double accepted `--fail-rate=0.x` at startup but the smoke needed to flip the failure mode after the happy-path step. Added a `/control?fail=0|1` endpoint. The smoke uses this instead of restarting the fakefcmd container (which on this 1-CPU QEMU host takes 30s+ to rebuild the image).

### Bug 4 — Compose's `docker cp` + `restart` doesn't pick up the new binary

`docker cp new-binary old-container:/app/old-binary` overwrites the file on the running container's writable layer, but the running process still has the old inode mapped. `docker compose restart svc` does NOT recreate the container; it just sends SIGTERM/SIGKILL to the same container, which then starts with the (new) on-disk binary — except: I observed that on this Compose 5.1.3 build, `restart` doesn't actually restart the container. Workaround: `docker compose rm -sf svc && docker compose up -d --no-deps svc` to fully recreate, then `docker cp` the new binary, then `kill` + `up` to pick it up. Documented in the developer runbook.