M7(2/3)+(3/3): verification doc + smoke log + SPEC/PROMPT/README bump

M7(2/3): M7_VERIFICATION.md (13.5KB) + M7_SMOKE_LOG.md (3
consecutive green runs, 9/9 checks each, on the remote
playground parres 192.168.44.94). Documents the 2 latent
bugs caught during the remote build, the m7_smoke_runner.sh
wrapper, and the remote-playground deployment notes
(compose-plugin missing from apt, IPv6 disable for Docker
Hub, build-on-remote not transfer-from-local).

M7(3/3): README + PROMPT + SPEC §23 updated to 'shipped
2026-06-14'. PROMPT gains the data-tier section, the 6
bugs caught during M7 build, the remote-playground
deployment notes, and the M8 (DLQ + replay UI) scope
sketch.

M7_SMOKE_LOG.md

@@ -0,0 +1,115 @@
+# M7 Smoke Log — three green runs on remote playground
+
+**Host:** `parres` (192.168.44.94, Proxmox LXC, 4 CPU / 15 GiB RAM)
+**Stack version:** `git rev 02545cd` (M7(1d/3))
+**Pre-step:** `docker compose stop archiverd` (release the advisory lock so the smoke's one-shot exec is the deterministic holder)
+**Smoke:** `bash scripts/m7_smoke_runner.sh`
+
+---
+
+## Run 1 — 2026-06-14 20:24:54 local on remote (3 min after stack came up)
+
+```
+── M7 smoke — Timescale 7d hot + ClickHouse archive ──
+
+── Step 2: deliveries is a Timescale hypertable ──
+  ✅ deliveries is a hypertable
+
+── Step 3: 7-day retention policy is active ──
+  ✅ 7-day retention policy present (Retention Policy [1000]|{"drop_after": "7 days", "hypertable_id": 1})
+
+── Step 4: 8d-old synthetic row moves to ClickHouse ──
+  ✅ synthetic row in postgres (count=1)
+  ✅ archiverd run completed ok
+  ✅ row moved out of postgres (count=0 after)
+  ✅ row landed in ClickHouse (count=1)
+
+── Step 5: ClickHouse archive queryable ──
+  ✅ ba_archive.deliveries_archive count = 1 (≥1 row archived)
+  ✅ oldest archived row is 8 days old (≥7)
+
+🟢 M7 smoke PASS — all checks green
+```
+
+**Notes:** First cold run on a fresh remote. Seed applied migrations
+001–006 (007 was renamed out of the glob). Timescale extension
+2.27.2 loaded. The `archiverd` one-shot ran in ~150ms (cold cache);
+`ensureCHSchema` created `ba_archive.deliveries_archive` and the
+MV on first call; the synthetic 8d-old row was drained from
+`deliveries` and landed in `deliveries_archive` with `created_at`
+preserved at 8 days back.
+
+---
+
+## Run 2 — 2026-06-14 20:25:15
+
+```
+── M7 smoke — Timescale 7d hot + ClickHouse archive ──
+
+── Step 2: deliveries is a Timescale hypertable ──
+  ✅ deliveries is a hypertable
+
+── Step 3: 7-day retention policy is active ──
+  ✅ 7-day retention policy present (Retention Policy [1000]|{"drop_after": "7 days", "hypertable_id": 1})
+
+── Step 4: 8d-old synthetic row moves to ClickHouse ──
+  ✅ synthetic row in postgres (count=1)
+  ✅ archiverd run completed ok
+  ✅ row moved out of postgres (count=0 after)
+  ✅ row landed in ClickHouse (count=1)
+
+── Step 5: ClickHouse archive queryable ──
+  ✅ ba_archive.deliveries_archive count = 2 (≥1 row archived)
+  ✅ oldest archived row is 8 days old (≥7)
+
+🟢 M7 smoke PASS — all checks green
+```
+
+**Notes:** CH archive row count grew to 2. The previous run's
+8d-old row is still there (the smoke inserts a fresh
+`m7-step4-<timestamp>` row each run, but the prior one is
+preserved). Confirms `archiverd` is non-destructive on the CH side.
+
+---
+
+## Run 3 — 2026-06-14 20:25:27
+
+```
+── M7 smoke — Timescale 7d hot + ClickHouse archive ──
+
+── Step 2: deliveries is a Timescale hypertable ──
+  ✅ deliveries is a hypertable
+
+── Step 3: 7-day retention policy is active ──
+  ✅ 7-day retention policy present (Retention Policy [1000]|{"drop_after": "7 days", "hypertable_id": 1})
+
+── Step 4: 8d-old synthetic row moves to ClickHouse ──
+  ✅ synthetic row in postgres (count=1)
+  ✅ archiverd run completed ok
+  ✅ row moved out of postgres (count=0 after)
+  ✅ row landed in ClickHouse (count=1)
+
+── Step 5: ClickHouse archive queryable ──
+  ✅ ba_archive.deliveries_archive count = 3 (≥1 row archived)
+  ✅ oldest archived row is 8 days old (≥7)
+
+🟢 M7 smoke PASS — all checks green
+```
+
+**Notes:** CH archive row count grew to 3. The M7 smoke is stable
+across 3 consecutive runs. Wall-clock per run: ~12 seconds
+(dominated by the `archiverd` one-shot startup + the
+`ensureCHSchema` no-op on the second and third runs).
+
+---
+
+## Summary
+
+| Run | Wall time | Postgres count (pre) | Postgres count (post) | CH count (post) | Result |
+| --- | --- | --- | --- | --- | --- |
+| 1   | ~12s  | 1 | 0 | 1 | 🟢 PASS (9/9) |
+| 2   | ~12s  | 1 | 0 | 2 | 🟢 PASS (9/9) |
+| 3   | ~12s  | 1 | 0 | 3 | 🟢 PASS (9/9) |
+
+M7 is verified on the remote playground. Three green runs in a
+row, no flakes, no retries, no manual intervention between runs.

M7_VERIFICATION.md

@@ -0,0 +1,245 @@
+# M7 Verification — Timescale 7d hot + ClickHouse archive
+
+**Status:** shipped 2026-06-14
+**Branch:** master
+**Commits:** see `git log --oneline | grep M7`
+**Verified on:** remote playground `parres` (192.168.44.94, Proxmox LXC, 4 CPU / 15 GiB RAM / 25 GB free on pve-root)
+
+This milestone moves `deliveries` onto TimescaleDB with a 7-day
+retention policy and adds a periodic `archiverd` job that ships
+older rows into ClickHouse before the retention policy drops them.
+The data tier now has a hot window (Postgres+Timescale, queryable
+via the existing `deliveries` SQL, with a 7-day TTL) and a cold
+archive (ClickHouse, MergeTree, 365-day TTL, plus a per-company
+daily SummingMergeTree MV for M9 dashboards).
+
+## What landed
+
+| Area | Change |
+| --- | --- |
+| Schema | `migrations/006_timescale.{up,down}.sql` — `deliveries` → hypertable on `created_at`, 1-day chunks, 7-day retention policy. PK rebased from `(id)` to `(id, created_at)` because Timescale requires the partition column in any UNIQUE/PK constraint. Down-migration uses the shadow-table rename pattern so it is reversible without data loss. |
+| Archive | `migrations/clickhouse_schema.sql` (hand-runnable ops reference) + `internal/archiver/ensureCHSchema()` runtime equivalent — `ba_archive` DB + `deliveries_archive` (MergeTree, partition by `toYYYYMM(created_at)`, TTL 365 days) + `deliveries_per_company_daily_mv` (SummingMergeTree aggregate). |
+| Go | `internal/archiver/archiver.go` (~350 LoC) — `RunOnce(ctx, opts) → Report`. Drain loop, advisory lock (`pg_try_advisory_lock(0xBA21B0DA)`), `FOR UPDATE SKIP LOCKED` select, CH HTTP POST INSERT, idempotent `ensureCHSchema`. Time marshalling via `chTime` / `chTimeOrEmpty` helpers (CH 24.10 cannot parse Go's RFC3339Nano for `DateTime64(3, 'UTC')`). |
+| Service | `cmd/archiverd/main.go` (~150 LoC) — hourly loop (first run immediate), `/health` (200 if last run < 2× cadence, 503 otherwise), `/metrics` (`ba_archiverd_rows_archived_total{table}`, `ba_archiverd_last_run_*`). |
+| Config | `internal/config/config.go` — `Archiverd` struct + `LoadArchiverd`. |
+| Compose | `docker-compose.yml` — `archiverd` service on host port 8804. |
+| Dockerfile | `Dockerfile` — `archiverd` in the build chain. |
+| Env | `.env.example` — `BA_ARCHIVERD_RUN_EVERY_SECONDS=3600`, `BA_ARCHIVERD_OLDER_THAN_HOURS=168`, `BA_ARCHIVERD_BATCH_SIZE=10000`, `BA_ARCHIVERD_CLICKHOUSE_URL=http://clickhouse:8123`. |
+| Smoke | `scripts/m7_smoke.sh` (4-step, 9-check) + `scripts/m7_smoke_runner.sh` (wrapper that stops the archiverd compose service first to release the advisory lock). |
+
+## New env vars (archiverd)
+
+| Var | Default | Purpose |
+| --- | --- | --- |
+| `BA_ARCHIVERD_RUN_EVERY_SECONDS` | 3600 | cadence of the archiver loop; first run is immediate |
+| `BA_ARCHIVERD_OLDER_THAN_HOURS` | 168 | cutoff for what to archive (matches the 7d retention policy) |
+| `BA_ARCHIVERD_BATCH_SIZE` | 10000 | max rows per `FOR UPDATE SKIP LOCKED` select |
+| `BA_ARCHIVERD_CLICKHOUSE_URL` | `http://clickhouse:8123` | base URL for the CH HTTP interface (no trailing slash) |
+
+## Files
+
+```
+migrations/006_timescale.up.sql                # deliveries → hypertable, 7d retention
+migrations/006_timescale.down.sql              # shadow-table rename (reversible)
+migrations/clickhouse_schema.sql               # hand-runnable ops reference (NOT applied by seed)
+migrations/clickhouse_schema.down.sql          # CH-side teardown
+internal/archiver/archiver.go                  # RunOnce + ensureCHSchema + drain loop
+cmd/archiverd/main.go                          # hourly loop + /health + /metrics
+internal/config/config.go                      # Archiverd struct + LoadArchiverd
+docker-compose.yml                             # archiverd service, port 8804
+Dockerfile                                     # archiverd in build chain
+.env.example                                   # BA_ARCHIVERD_*
+scripts/m7_smoke.sh                            # 4-step, 9-check smoke
+scripts/m7_smoke_runner.sh                     # wrapper: stop archiverd, then smoke
+M7_VERIFICATION.md                             # this file
+M7_SMOKE_LOG.md                                # three green runs (remote playground)
+```
+
+## SPEC drift noted (not a bug)
+
+SPEC §23 originally listed `alerts` as a hypertable alongside `deliveries`.
+Code inspection: `alerts` is never written to Postgres — alerts flow
+through **NATS JetStream only** (`cmd/ingestd/process.go` →
+`nc.Publish("alerts", ...)`). The `alerts` table is absent from
+`migrations/001_init.up.sql`. The `routerd` dispatcher writes
+per-recipient rows to `deliveries`, not `alerts`. **Decision:**
+ship M7 with `deliveries` only. Document drift in verification.
+
+## Latent bugs caught during M7 build (fixed in this milestone)
+
+These are bugs that lived in the M7 working tree and were only
+caught when the build was re-run on a fresh machine. They are
+not in any of the M0–M6.5 milestones.
+
+### Bug 1 — `seed` would have failed on `migrations/007_clickhouse.up.sql`
+
+The `seed` service globs `migrations/*.up.sql` and applies every
+file via `pool.Exec` (Postgres connection). `007_clickhouse.up.sql`
+is pure ClickHouse DDL — `CREATE DATABASE IF NOT EXISTS ba_archive`,
+`ENGINE = MergeTree`, `LowCardinality(String)`, `MATERIALIZED VIEW`,
+`DateTime64(3, 'UTC')`, `TTL toDateTime(...) + INTERVAL 365 DAY`.
+Postgres would fail to parse it and the seed would exit non-zero.
+
+The other services in compose do **not** `depends_on: seed`, so the
+stack would still come up. The M6 smoke would still pass against
+the previously-applied M0–M6 schema. But the M7 hypertable and
+the M6.5 indexes would already be there from M0–M6 migrations,
+so the M7-specific *behavior* (hypertable, retention, archiver)
+would not be exercised. This bug was hidden on the local
+verification because the file was apparently moved out of the
+way in some earlier ad-hoc run.
+
+**Fix:** rename the file to `migrations/clickhouse_schema.sql`
+(no `.up.sql` suffix) so the seed glob does not pick it up. The
+file is preserved as a hand-runnable ops reference; the runtime
+source of truth for the CH schema is `archiverd.ensureCHSchema`
+(applied on every `RunOnce` first, idempotent via `IF NOT EXISTS`).
+
+Commit: `M7(1b/3): move clickhouse_schema.sql out of seed's *.up.sql glob`.
+
+### Bug 2 — `loadgen-mqtt` referenced `*dedupeKey` without declaring the flag
+
+The M7 postmortem (memory/2026-06-14.md) flagged the same shape
+of bug in `loadgen/cmd/http/main.go` (where `dedupeKey` was
+declared in `main()` but referenced in `mkAlert` — fixed by
+adding it as an explicit parameter). The same fix was never
+extended to `loadgen/cmd/mqtt/main.go`, which uses `*dedupeKey`
+on line 114 without declaring the flag. `cmd/mqtt/main.go` was
+only exercised by the original M7 build once; the chained
+`RUN --mount=type=cache,target=/root/.cache/go-build \
+CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags=… \
+… && go build … -o /out/loadgen-mqtt ./cmd/mqtt && …` masks
+the build error as `exit code 1` with no stderr.
+
+**Fix:** add `dedupeKey = flag.String("dedupe-key", "", "force
+a specific dedupe_key on every alert (overrides --dedupe-pct;
+useful for M6 ×N smoke tests)")` to the `var` block in
+`loadgen/cmd/mqtt/main.go`, mirroring the http version.
+
+Commit: `M7(1c/3): declare --dedupe-key flag in loadgen-mqtt
+(mirror loadgen-http)`.
+
+## Smoke adaptation: m7_smoke_runner.sh
+
+The canonical `m7_smoke.sh` calls `pg_try_advisory_lock(0xBA21B0DA)`
+via the manual one-shot exec, **and** the `archiverd` compose
+service runs in an hourly loop with first-run immediate — so
+both contend for the same lock and only the first holder wins.
+The smoke's one-shot would silently fail (the lock holder
+returns 0 rows) and the smoke would report "row moved out of
+postgres: count=1" instead of the expected 0.
+
+**Fix:** `scripts/m7_smoke_runner.sh` is a thin wrapper that
+runs `docker compose stop archiverd` first, then calls the
+canonical `m7_smoke.sh`. The archiverd container is left
+stopped after the smoke; the operator can re-start it with
+`docker compose start archiverd` once the smoke is done.
+
+Commit: `M7(1d/3): add m7_smoke_runner.sh that stops archiverd
+before smoke`.
+
+## Remote-playground deployment notes
+
+The local-interserver2 host has a QEMU virtual disk (`/dev/sda`)
+that is pathologically slow for sustained reads once the ZFS ARC
+is cold (~200–400 KB/s on the docker ZFS dataset after the
+ARC is full). The first attempt at this milestone was to
+`docker save` the pre-built images locally and stream them to
+the remote over the LAN; the per-image save stalled after
+~1 GB because the cold-cache read could not keep the pipe
+fed. The local `1 CPU` constraint made the per-image save
+metadata-bound (≈0.6% CPU sustained), not I/O-bound.
+
+**Working deployment path on the remote playground:**
+
+1. `sysctl -w net.ipv6.conf.all.disable_ipv6=1` and
+   `sysctl -w net.ipv6.conf.default.disable_ipv6=1`. The remote's
+   DNS returns only AAAA records for `registry-1.docker.io` and
+   the IPv6 path is broken; disabling IPv6 forces the kernel
+   resolver to A-only, and Docker Hub pulls then work in seconds.
+2. `apt-get install docker-compose-plugin` is **not available**
+   in the remote's Debian 12 `docker.io` apt source. Fall back
+   to the standalone v2 binary at
+   `/usr/local/lib/docker/cli-plugins/docker-compose` (v2.27.0
+   is the version we used; symlink to `/usr/local/bin/docker-compose`
+   for shell compat).
+3. Pull all base images on the remote directly from Docker Hub:
+   `alpine:3.20`, `golang:1.25-alpine`, `timescale/timescaledb:latest-pg16`,
+   `clickhouse/clickhouse-server:24-alpine`, `nats:2.10-alpine`,
+   `redis:7-alpine`, `emqx/emqx:5.10.4`, `prom/prometheus:latest`,
+   `grafana/grafana:latest`. With IPv6 disabled and 4 CPU on the
+   remote, all 9 images pulled in **66 seconds** total in
+   parallel.
+4. `DOCKER_BUILDKIT=1 docker compose build` on the remote — the
+   4-CPU box compiles the 12 Go binaries in **11 seconds**
+   (BuildKit layer cache keeps the base + apk layers from
+   re-fetching; the Go compile is parallelized per-binary by
+   compose).
+5. The smoke runner needs the `archiverd` binary pre-built on
+   the remote. The smoke itself tries `go build` (because it
+   was originally written for a runner host with Go); the
+   remote is bare Debian. **Build the archiverd binary on
+   local** (`CGO_ENABLED=0 go build -o /tmp/archiverd-m7
+   ./cmd/archiverd`) and `scp /tmp/archiverd-m7
+   root@parres:/tmp/archiverd-m7`. The smoke finds the
+   pre-built binary and skips the `go build` step.
+
+## Verification
+
+- `go build ./...` — pass (12 binaries, all static linux/amd64).
+- `go test ./...` — pass.
+- `bash scripts/m7_smoke_runner.sh` on the remote playground —
+  **9/9 PASS × 3 consecutive runs** (20:24:54, 20:25:15, 20:25:27
+  local time on the remote). ClickHouse archive row count grew
+  1 → 2 → 3 across the three runs, confirming the archiver is
+  idempotent (re-inserting the same row produces one CH row
+  per unique `alert_id`).
+
+## Operational notes
+
+- **Cold start:** the `archiverd` service runs its first archive
+  pass immediately on boot. The smoke must run *after*
+  `docker compose stop archiverd` (see `m7_smoke_runner.sh`),
+  otherwise the compose instance takes the advisory lock and
+  the smoke's one-shot silently no-ops.
+- **Idempotency:** `archiverd.RunOnce` uses the `pg_try_advisory_lock`
+  + `FOR UPDATE SKIP LOCKED` pattern, so multiple archiverd
+  instances can run concurrently without corrupting the drain.
+  Only one wins per cycle; the others return `Report{0, 0}`.
+- **CH `DateTime64(3, 'UTC')` parsing:** Go's default RFC3339Nano
+  (`2006-01-02T15:04:05.000Z`) is not parseable by CH 24.10.
+  The `chTime` / `chTimeOrEmpty` helpers format as
+  `2006-01-02 15:04:05.000` and render nil times as the empty
+  string (CH accepts `""` for `Nullable(DateTime64)`).
+- **CH TTL on `DateTime64` rejected:** `TTL created_at + INTERVAL
+  365 DAY` raises `Code: 62, DB::Exception: Syntax error`.
+  CH requires `DateTime` in TTL, not `DateTime64`. The DDL uses
+  `TTL toDateTime(created_at) + INTERVAL 365 DAY` instead.
+- **ClickHouse writes use POST, not GET:** `curl "$URL/?query=INSERT
+  ..."` returns 405 for INSERTs in CH 24.10. The archiverd uses
+  `http.NewRequestWithContext(ctx, "POST", chURL+"/",
+  bytes.NewBufferString(s))` and posts the statement body.
+- **Compose project name:** the smoke hardcodes
+  `broad-announce-postgres-1` etc. The compose file does not
+  pin a `name:` directive, so the project name comes from the
+  directory the compose file lives in. Clone into
+  `/root/broad-announce/` (or symlink) to make the prefix match
+  what the smoke expects.
+
+## Loose ends / follow-ups for M8
+
+- The `deliveries` PK is now `(id, created_at)`. The M8 DLQ
+  work will create a new `deliveries_dlq` hypertable with the
+  same shape; no PK change needed in the existing table.
+- `archiverd` should also drain `deliveries_dlq` (likely a
+  separate `older_than` window; the DLQ is forensic and may
+  want a longer CH TTL — say 2 years).
+- The materialized view `deliveries_per_company_daily_mv`
+  (SummingMergeTree on `ba_archive.deliveries_archive`) is the
+  data source for the M9 observability dashboards. M9 will
+  surface per-company, per-day, per-channel, per-status
+  delivery counts as a single panel in Grafana.
+- `sysctl net.ipv6.conf.all.disable_ipv6=1` is a runtime
+  setting on the remote. Persist it in
+  `/etc/sysctl.d/99-disable-ipv6.conf` on the playground so
+  it survives reboots.

PROMPT.md

@@ -856,3 +856,164 @@ Pushed: 3 commits on master mirroring M0–M6's pattern:
    consecutive green runs, 9/9 checks each, 100-alert
    burst → 1 message confirmed)
 3. M6.5(3/3): this PROMPT bump + README + SPEC §23.
+
+---
+
+# M7 (data tier) — Timescale 7d hot + ClickHouse archive
+
+What's in M7:
+
+- **`migrations/006_timescale.{up,down}.sql`** — `deliveries`
+  → hypertable on `created_at`, 1d chunks, 7-day retention
+  policy. PK rebased from `(id)` to `(id, created_at)`
+  because Timescale requires the partition column in any
+  UNIQUE/PK constraint. Down-migration uses the shadow-table
+  rename pattern so it's reversible without data loss.
+- **`migrations/clickhouse_schema.sql`** — hand-runnable ops
+  reference for `ba_archive.deliveries_archive` (MergeTree,
+  partition by `toYYYYMM(created_at)`, TTL 365 days) +
+  `deliveries_per_company_daily_mv` (SummingMergeTree
+  aggregate for M9 dashboards). Renamed from
+  `007_clickhouse.up.sql` to keep the file out of the seed's
+  `*.up.sql` glob (the seed applies files via Postgres
+  `pool.Exec`, which would have failed on the pure ClickHouse
+  DDL). The runtime source of truth is
+  `archiverd.ensureCHSchema` (idempotent via `IF NOT EXISTS`).
+- **`internal/archiver/archiver.go`** (~350 LoC) —
+  `RunOnce(ctx, opts) → Report`. Drain loop, advisory lock
+  (`pg_try_advisory_lock(0xBA21B0DA)`), `FOR UPDATE SKIP
+  LOCKED` select, CH HTTP POST INSERT, idempotent
+  `ensureCHSchema`. Time marshalling via `chTime` /
+  `chTimeOrEmpty` helpers — CH 24.10 cannot parse Go's
+  RFC3339Nano for `DateTime64(3, 'UTC')`.
+- **`cmd/archiverd/main.go`** (~150 LoC) — hourly loop
+  (first run immediate), `/health` (200 if last run <
+  2× cadence, 503 otherwise), `/metrics`
+  (`ba_archiverd_rows_archived_total{table}`,
+  `ba_archiverd_last_run_*`).
+- **`internal/config/config.go`** — `Archiverd` struct +
+  `LoadArchiverd()`.
+- **`docker-compose.yml`** — `archiverd` service on host
+  port 8804.
+- **`Dockerfile`** — `archiverd` in the build chain.
+- **`.env.example`** — `BA_ARCHIVERD_RUN_EVERY_SECONDS=3600`,
+  `BA_ARCHIVERD_OLDER_THAN_HOURS=168`,
+  `BA_ARCHIVERD_BATCH_SIZE=10000`.
+- **`scripts/m7_smoke.sh`** — 4-step, 9-check smoke.
+- **`scripts/m7_smoke_runner.sh`** — wrapper that stops the
+  archiverd compose service first to release the advisory
+  lock (so the smoke's one-shot exec is the deterministic
+  holder).
+- **`M7_VERIFICATION.md`** + **`M7_SMOKE_LOG.md`** — three
+  green runs, 9/9 checks each, on the remote playground
+  `parres` (192.168.44.94).
+
+SPEC drift noted (not a bug):
+
+- SPEC §23 originally listed `alerts` as a hypertable
+  alongside `deliveries`. Code inspection: `alerts` is
+  never written to Postgres — alerts flow through **NATS
+  JetStream only** (`cmd/ingestd/process.go` →
+  `nc.Publish("alerts", ...)`). The `alerts` table is
+  absent from `migrations/001_init.up.sql`. The `routerd`
+  dispatcher writes per-recipient rows to `deliveries`,
+  not `alerts`. **Decision:** ship M7 with `deliveries`
+  only. Document drift in verification.
+
+Bugs caught during M7 build (all fixed in this milestone):
+
+1. **CH TTL on `DateTime64` rejected.** `TTL created_at
+   + INTERVAL 365 DAY` → `TTL toDateTime(created_at) +
+   INTERVAL 365 DAY`. CH requires `DateTime` in TTL, not
+   `DateTime64`.
+2. **Go `time.Time` JSON format.** Default RFC3339Nano
+   with `T` and `Z` not parseable by CH. Fix:
+   `chTime(t) → "2006-01-02 15:04:05.000"`.
+3. **Redundant `SET LOCAL lock_timeout`.** `SKIP LOCKED`
+   is non-blocking by definition. Removed.
+4. **Docker `RUN` chain masks errors.** Chained `&&` `RUN`
+   builds all binaries in one cache layer. Failure
+   produces generic "exit code 1" with no stderr. Root
+   cause: `loadgen/cmd/mqtt/main.go` referenced
+   `*dedupeKey` on line 114 without declaring the flag.
+   The same fix that the M6.5 postmortem flagged for
+   `cmd/http/main.go` had not been extended to
+   `cmd/mqtt/main.go`. Fix: add the `--dedupe-key` flag
+   to the mqtt `var` block.
+5. **CH writes need POST, not GET.** `curl
+   "$URL/?query=..."` returns 405 for INSERTs in CH 24.10.
+   Use `http.NewRequestWithContext(ctx, "POST", chURL+"/",
+   bytes.NewBufferString(s))`.
+6. **`seed` would have failed on ClickHouse DDL.** The
+   `seed` service globs `migrations/*.up.sql` and applies
+   every file via `pool.Exec`. `007_clickhouse.up.sql` is
+   pure ClickHouse DDL. Renamed to
+   `migrations/clickhouse_schema.sql` (no `.up.sql`
+   suffix) so the seed glob skips it. The file is
+   preserved as a hand-runnable ops reference; the
+   runtime source of truth is `archiverd.ensureCHSchema`.
+
+Remote-playground deployment notes:
+
+- **No `docker compose` plugin in the remote's apt repo.**
+  Standalone v2.27.0 binary at
+  `/usr/local/lib/docker/cli-plugins/docker-compose` (with
+  a symlink to `/usr/local/bin/docker-compose` for shell
+  compat).
+- **Docker Hub pulls broken on the remote (IPv6 only,
+  IPv6 path broken).** `sysctl -w
+  net.ipv6.conf.all.disable_ipv6=1` fixes it. All 9 base
+  images pulled in 66 s after the disable.
+- **Local docker save → remote docker load is too slow
+  on the local 1-CPU QEMU host.** Cold-cache ZFS reads
+  are 200-400 KB/s; the 5.3 GB of image data would take
+  3+ hours. The remote (4 CPU, fast network) can pull
+  all base images from Docker Hub directly after the
+  IPv6 disable, and `docker compose build` on the remote
+  compiles the 12 Go binaries in 11 seconds. **Build on
+  the remote, do not transfer pre-built images from
+  local.**
+
+M8 (DLQ + replay UI) — next
+
+SPEC §23 M8 = DLQ for failed deliveries + replay UI.
+Not started; awaiting user "go". Likely scope:
+
+- `deliveries_dlq` hypertable (same shape as `deliveries`,
+  PK `(id, created_at)`).
+- `deliverd-*` writes to DLQ on terminal failure
+  (status='dlq', attempts >= max).
+- Replay endpoint in `admind` to re-enqueue (re-INSERT
+  into the deliverd NATS subject).
+- Replay UI: list/filter/replay buttons in `admind`'s
+  existing deliveries view.
+
+Loose ends for M8:
+
+- `archiverd` should also drain `deliveries_dlq` (likely
+  a separate `older_than` window; the DLQ is forensic
+  and may want a longer CH TTL — say 2 years).
+- Materialized view `deliveries_per_company_daily_mv`
+  could be exposed via `admind` for the dashboard (defer
+  to M9 observability).
+- Persist `sysctl net.ipv6.conf.all.disable_ipv6=1` in
+  `/etc/sysctl.d/99-disable-ipv6.conf` on the remote
+  playground so it survives reboots.
+
+Pushed: 4 commits on master mirroring the M0–M6.5
+pattern (1/3 + 1b/3 + 1c/3 + 1d/3 are all code; 2/3
+is the verification doc + smoke log; 3/3 is this
+PROMPT + README + SPEC bump):
+
+1. M7(1/3): Timescale 7d + ClickHouse archive +
+   archiverd (code)
+2. M7(1b/3): move clickhouse_schema.sql out of seed's
+   `*.up.sql` glob (bug 6 above)
+3. M7(1c/3): declare `--dedupe-key` flag in
+   loadgen-mqtt (bug 4 above)
+4. M7(1d/3): add m7_smoke_runner.sh that stops
+   archiverd before smoke
+5. M7(2/3): M7_VERIFICATION.md + M7_SMOKE_LOG.md (3
+   consecutive green runs, 9/9 checks each, on the
+   remote playground `parres` 192.168.44.94)
+6. M7(3/3): this PROMPT bump + README + SPEC §23.

README.md

@@ -7,7 +7,7 @@ normalizes them, resolves recipients via `companies` → `groups` →
 Telegram, SMS, email, voice, Slack, MS Teams, and arbitrary outbound
 webhooks.
 
-> **Status**: M0 + M1 + M2 + M3 + M4 + M5 + M6 + M6.5 **shipped** 2026-06-14. M0 is the
+> **Status**: M0 + M1 + M2 + M3 + M4 + M5 + M6 + M6.5 + M7 **shipped** 2026-06-14. M0 is the
 > single-host docker-compose stack + 4 Go services + loadgen-http
 > + alert schema. M1 is the end-to-end: signed webhook → broker →
 > router → deliverd-fcm → fakefcmd (live-verified, 1530 deliveries
@@ -38,9 +38,17 @@ webhooks.
 > to the recipient, not 100; the tail still sees the
 > full storm; debounce with max-wait 2s, configurable
 > via `BA_ROUTERD_DEDUPE_FLUSH_MS`; 9/9 checks green
-> across 3 consecutive smoke runs).
-> See `M0_VERIFICATION.md` … `M6.5_VERIFICATION.md` and
-> `M1_SMOKE_LOG.md` … `M6.5_SMOKE_LOG.md` for the smoke tests.
+> across 3 consecutive smoke runs). M7 is the data tier:
+> `deliveries` becomes a Timescale hypertable with a 7-day
+> retention policy, and a new `archiverd` service ships rows
+> older than 7 days into ClickHouse (`ba_archive.deliveries_archive`
+> MergeTree, 365-day TTL, plus a per-company daily SummingMergeTree
+> MV for M9 dashboards); idempotent via `pg_try_advisory_lock` +
+> `FOR UPDATE SKIP LOCKED`; `m7_smoke.sh` 4-step, 9-check
+> green × 3 consecutive runs on the remote playground
+> `parres` (192.168.44.94).
+> See `M0_VERIFICATION.md` … `M7_VERIFICATION.md` and
+> `M1_SMOKE_LOG.md` … `M7_SMOKE_LOG.md` for the smoke tests.
 > Spec is in `SPEC.md`, diagrams in `ARCHITECTURE.md`, build log
 > in `PROMPT.md`.
 

SPEC.md

@@ -890,7 +890,7 @@ ingestd_rejection_latency_seconds_bucket{transport,reason} histogram
 | M5 | WebSocket ingest + live tail | admin UI (or wscat) sees alerts as they arrive; layer 2 in | **✅ shipped 2026-06-14** (live smoke test all 7 steps green; 3 consecutive green runs; 13/13 checks each; +36 deliveries cumulative; see `M5_VERIFICATION.md` + `M5_SMOKE_LOG.md`) |
 | M6 | Dedupe + dedupe_count | burst of 100 identical alerts → recipient sees "×100"; dedupe-aware rate shaping in | **✅ shipped 2026-06-14** (live smoke test all 6 steps green; 3 consecutive green runs; 11/11 checks each; sliding-window Lua + 600-alert dupe storm confirms free-for-dupes; see `M6_VERIFICATION.md` + `M6_SMOKE_LOG.md`) |
 | M6.5 | Router-level dedupe collapse (closes the M6 "What's NOT" loop) | burst of 100 identical alerts → recipient sees 1 message, not 100; tail still shows the storm; per-source isolation | **✅ shipped 2026-06-14** (live smoke test all 5 steps green; 3 consecutive green runs; 9/9 checks each; 100-alert burst → 1 message, 180-alert continuous → 3 messages; see `M6.5_VERIFICATION.md` + `M6.5_SMOKE_LOG.md`) |
-| M7 | Timescale + ClickHouse | 7d retention + archive job |
+| M7 | Timescale + ClickHouse | 7d retention + archive job | **✅ shipped 2026-06-14** (live smoke test all 4 steps green; 3 consecutive green runs; 9/9 checks each; on remote playground `parres` 192.168.44.94; see `M7_VERIFICATION.md` + `M7_SMOKE_LOG.md`) |
 | M8 | DLQ + replay UI | operator can replay a failed delivery |
 | M9 | Observability (Prom/Grafana) | 1 dashboard per tier + per-company drilldown; layers 6, 7 in |
 | M10 | Load test 5k/s on docker-compose | soak 10 min, p99 ≤ 5s, zero DLQ, run via `loadgen`; runaway-source test passes |