|
|
@@ -974,33 +974,95 @@ Remote-playground deployment notes:
|
|
|
the remote, do not transfer pre-built images from
|
|
|
local.**
|
|
|
|
|
|
-M8 (DLQ + replay UI) — next
|
|
|
+M8 (DLQ + replay UI) — **shipped 2026-06-14**
|
|
|
|
|
|
SPEC §23 M8 = DLQ for failed deliveries + replay UI.
|
|
|
-Not started; awaiting user "go". Likely scope:
|
|
|
-
|
|
|
-- `deliveries_dlq` hypertable (same shape as `deliveries`,
|
|
|
- PK `(id, created_at)`).
|
|
|
-- `deliverd-*` writes to DLQ on terminal failure
|
|
|
- (status='dlq', attempts >= max).
|
|
|
-- Replay endpoint in `admind` to re-enqueue (re-INSERT
|
|
|
- into the deliverd NATS subject).
|
|
|
-- Replay UI: list/filter/replay buttons in `admind`'s
|
|
|
- existing deliveries view.
|
|
|
-
|
|
|
-Loose ends for M8:
|
|
|
-
|
|
|
-- `archiverd` should also drain `deliveries_dlq` (likely
|
|
|
- a separate `older_than` window; the DLQ is forensic
|
|
|
- and may want a longer CH TTL — say 2 years).
|
|
|
-- Materialized view `deliveries_per_company_daily_mv`
|
|
|
- could be exposed via `admind` for the dashboard (defer
|
|
|
- to M9 observability).
|
|
|
-- Persist `sysctl net.ipv6.conf.all.disable_ipv6=1` in
|
|
|
- `/etc/sysctl.d/99-disable-ipv6.conf` on the remote
|
|
|
- playground so it survives reboots.
|
|
|
+**shipped.** 6 commits mirroring M0–M7 (1/3 + 1b/3 +
|
|
|
+1c/3 + 1d/3 are all code; 2/3 is the verification doc
|
|
|
++ smoke log; 3/3 is this PROMPT + README + SPEC bump).
|
|
|
+3 consecutive 12/12 green runs of `m8_smoke.sh` on the
|
|
|
+local docker-compose stack; see `M8_VERIFICATION.md` +
|
|
|
+`M8_SMOKE_LOG.md`. Remote `parres` (192.168.44.94) was
|
|
|
+unreachable from this dev host (TCP RST on 22/80) so
|
|
|
+the live re-run there is left to the operator.
|
|
|
+
|
|
|
+What landed:
|
|
|
|
|
|
-Pushed: 4 commits on master mirroring the M0–M6.5
|
|
|
+- `migrations/008_dlq.{up,down}.sql` —
|
|
|
+ `deliveries_dlq` Timescale hypertable, 1d chunks,
|
|
|
+ 7d retention, PK `(id, created_at)`. Extra columns:
|
|
|
+ `original_subject` (for replay), `discarded` /
|
|
|
+ `discarded_at` / `discarded_by` (for the operator's
|
|
|
+ discard action).
|
|
|
+- `internal/retry` — bounded exp-backoff retry helper
|
|
|
+ (10 attempts, base 100ms, cap 2s, budget 30s,
|
|
|
+ ~12s total wall clock for a fully failing target).
|
|
|
+ PermanentError short-circuit. 7 unit tests, 7/7 PASS.
|
|
|
+- `internal/dlq` — single `Write()` that INSERTs one
|
|
|
+ row into `deliveries_dlq`. Per-attempt `deliveries`
|
|
|
+ rows are left as-is so the audit trail is preserved.
|
|
|
+- `cmd/deliverd-fcm` + `cmd/deliverd-telegram` —
|
|
|
+ refactored `handleOne()` to use `retry.Run`. On
|
|
|
+ success, a 'sent' row. On exhaustion, a per-attempt
|
|
|
+ 'failed' row + a final `dlq.Write()`. PermanentError
|
|
|
+ on FCM/Telegram 4xx (excluding 408/429) so we don't
|
|
|
+ burn the retry budget on a bad token or a missing
|
|
|
+ chat.
|
|
|
+- `cmd/admind` — DLQ surface:
|
|
|
+ * `GET /v1/dlq?company_id=&channel=&alert_id=
|
|
|
+ &include=all&limit=100&offset=0` (30d window,
|
|
|
+ hides discarded by default)
|
|
|
+ * `GET /v1/dlq/{id}` (single row w/ payload)
|
|
|
+ * `POST /v1/dlq/{id}/replay` (re-publishes the
|
|
|
+ original NATS envelope onto the original subject,
|
|
|
+ then marks the row discarded)
|
|
|
+ * `POST /v1/dlq/{id}/discard` (mark discarded;
|
|
|
+ idempotent)
|
|
|
+ * `GET /dlq` — minimal HTML page embedded via
|
|
|
+ `go:embed`, light/dark theming, filter form,
|
|
|
+ inline replay/discard buttons, plain ES5 JS.
|
|
|
+- `internal/archiver` — `RunOptions` now carries a
|
|
|
+ `[]TableSpec`; default plan covers both `deliveries`
|
|
|
+ and `deliveries_dlq` (7d Postgres hot window). CH
|
|
|
+ schema adds `ba_archive.deliveries_dlq_archive`
|
|
|
+ (2y TTL) + `ba_archive.deliveries_dlq_per_company_daily_mv`
|
|
|
+ (SummingMergeTree) for M9 dashboards.
|
|
|
+- `testfakes/fakefcmd` — runtime `/control?fail=0|1`
|
|
|
+ endpoint to flip failure mode without restarting
|
|
|
+ the container (saves 30s+ per smoke cycle on the
|
|
|
+ 1-CPU QEMU host).
|
|
|
+- `docker-compose.yml` + `.env.example` — wired
|
|
|
+ `BA_DELIVERD_*` on both `deliverd-fcm` and
|
|
|
+ `deliverd-telegram`.
|
|
|
+- `scripts/m8_smoke.sh` — 4-step, 12-check live smoke.
|
|
|
+
|
|
|
+Loose ends addressed:
|
|
|
+
|
|
|
+- `archiverd` drains `deliveries_dlq` (2y CH TTL,
|
|
|
+ same 7d PG hot window as live deliveries).
|
|
|
+- Auth on /v1/dlq* is deferred to M11. M8 ships
|
|
|
+ unauthenticated (LAN-only deploy assumed).
|
|
|
+
|
|
|
+Loose ends still open (deferred):
|
|
|
+
|
|
|
+- Materialized view `deliveries_per_company_daily_mv`
|
|
|
+ could be exposed via `admind` for the dashboard
|
|
|
+ (deferred to M9 observability).
|
|
|
+- Persist `sysctl net.ipv6.conf.all.disable_ipv6=1`
|
|
|
+ in `/etc/sysctl.d/99-disable-ipv6.conf` on the
|
|
|
+ remote playground so it survives reboots.
|
|
|
+
|
|
|
+SPEC drift:
|
|
|
+
|
|
|
+SPEC §9 literally calls for backoff of 1s, 2s, 4s, …
|
|
|
+512s = 1023s total. M8 ships defaults that cap per-
|
|
|
+attempt wait at 2s and total budget at 30s, terminating
|
|
|
+in ~12s. Operators who want the SPEC-literal behavior
|
|
|
+can opt in via the four `BA_DELIVERD_*` env vars.
|
|
|
+The defaults trade literal SPEC compliance for fast
|
|
|
+failure detection.
|
|
|
+
|
|
|
+Pushed: 6 commits on master mirroring the M0–M7
|
|
|
pattern (1/3 + 1b/3 + 1c/3 + 1d/3 are all code; 2/3
|
|
|
is the verification doc + smoke log; 3/3 is this
|
|
|
PROMPT + README + SPEC bump):
|
|
|
@@ -1017,3 +1079,18 @@ PROMPT + README + SPEC bump):
|
|
|
consecutive green runs, 9/9 checks each, on the
|
|
|
remote playground `parres` 192.168.44.94)
|
|
|
6. M7(3/3): this PROMPT bump + README + SPEC §23.
|
|
|
+
|
|
|
+M8 (this section supersedes the 'next' placeholder above):
|
|
|
+
|
|
|
+1. M8(1/3): DLQ schema + in-process retry +
|
|
|
+ deliveries_dlq writer (8 files, ~1000 LoC)
|
|
|
+2. M8(1b/3): archiver DLQ drain + ClickHouse DLQ
|
|
|
+ archive (2 files, ~240 LoC)
|
|
|
+3. M8(1c/3): admind DLQ endpoints + HTML UI
|
|
|
+ (2 files, ~660 LoC)
|
|
|
+4. M8(1d/3): docker-compose / .env wiring
|
|
|
+ (2 files, ~36 LoC)
|
|
|
+5. M8(2/3): M8_VERIFICATION.md + M8_SMOKE_LOG.md +
|
|
|
+ m8_smoke.sh (3 consecutive 12/12 green runs on
|
|
|
+ the local stack)
|
|
|
+6. M8(3/3): this PROMPT bump + README + SPEC §23.
|
Luis Rosales