# M13c — Live tail + DLQ + Audit + K8s + E2E

> Third and final sub-milestone of M13. Delivers the ops surface
> (live tail, DLQ, audit), the K8s manifests, and the Playwright
> E2E gate. After this lands, M13 v1 is done.

**Status:** planning (post-M13b)
**Target:** `M13_FRONTEND_SPEC.md` §4.3
**Estimate:** 7-9 days with one engineer

---

## 0. Recap — what M13c ships

- **Live tail** (SSE feed) with filter bar + drill-in modal.
- **DLQ surface** (replaces `cmd/admind/ui/dlq.html`).
- **Audit log viewer** (read-only).
- **K8s manifests** for `admind` and `authd` (carry from M12 W1).
- **Playwright E2E** happy path green in CI.
- `M13_VERIFICATION.md` with screenshots + run logs.
- SPEC.md `M13c` row flipped to ✅.

**What M13c is NOT:**
- Not SSO, not SLO dashboards, not Storybook, not visual regression —
  all v2.
- Not visual rule editor, not density toggle, not SLO dashboards in UI.

---

## 1. Workstreams

```
┌──────────────────────────┐    ┌──────────────────────────┐
│  W1: Live tail (SSE)     │    │  W2: DLQ surface         │
│  (filter bar, drill-in,  │───▶│  (replaces dlq.html,     │
│   pause-on-scroll)       │    │   replay + discard)      │
└──────────────┬───────────┘    └──────────────┬───────────┘
               │                               │
               ▼                               ▼
┌──────────────────────────────────────────────────────────────┐
│  W3: Audit log viewer (read-only, filterable)                │
└──────────────────────────────────────────────────────────────┘
                                               │
                                               ▼
┌──────────────────────────────────────────────────────────────┐
│  W4: K8s manifests (admind + authd) + ConfigMap + Secret    │
└──────────────────────────────────────────────────────────────┘
                                               │
                                               ▼
┌──────────────────────────────────────────────────────────────┐
│  W5: Playwright E2E + CI pipeline + M13_VERIFICATION.md     │
└──────────────────────────────────────────────────────────────┘
```

Five workstreams. W1 → W2 → W3 → W4 → W5. W4 (K8s) is
independent of W1-W3 and can start in parallel. W5 is the final
gate.

---

## 2. Workstream details

### W1: Live tail (SSE)

**Goal:** Real-time feed of inbound alerts. Filterable.
Drill-in modal.

**Scope:**
- Route: `/tail`.
- Feature folder: `web/src/features/tail/`.
- Components: `TailFeed`, `TailFilterBar`, `TailRow`, `TailDetailModal`,
  `SSEStatusBadge`.
- Transport: native `EventSource` with `?access_token=***` query
  param (workaround for `EventSource`'s lack of custom headers).
- Heartbeat: green dot when SSE healthy, red on disconnect,
  auto-reconnect with exponential backoff (1s, 2s, 4s, 8s, max 30s).
- Pause-on-scroll-up: when user scrolls up, pause auto-scroll and
  show a "go to bottom" button. Resume when scrolled to bottom or
  button clicked.
- Filter bar: company (super), severity (≥ X), category, source,
  text search on title_key.
- Drill-in modal: full alert JSON (data, dedupe history, raw NATS
  subject).
- HTTP fallback: if SSE blocked by proxy, the filter bar shows a
  "use polling" toggle that hits `/v1/tail/recent?since=<ts>` every
  5s.

**Exit criteria:**
- [ ] `loadgen` sends 1 alert → appears in live tail within 2s.
- [ ] Burst of 100 alerts → all 100 visible in order.
- [ ] Filter by company hides other companies' alerts.
- [ ] Filter by severity hides lower severities.
- [ ] Pause on scroll up, "go to bottom" button works.
- [ ] SSE reconnect on network blip (close laptop, reopen, still
      streaming).
- [ ] HTTP fallback works when SSE blocked (manual test with
      curl).
- [ ] Tenant-admin sees only their company's alerts.
- [ ] Bundle: live-tail feature code-split (< 40 KB gzipped).

**Estimated:** 2-3 days.

---

### W2: DLQ surface (replaces `dlq.html`)

**Goal:** Replace `cmd/admind/ui/dlq.html` with the new SPA. Same
list/replay/discard flow, plus filters.

**Scope:**
- Routes: `/dlq`, `/dlq/:id`.
- Feature folder: `web/src/features/dlq/`.
- Components: `DLQList`, `DLQDetail`, `DLQReplayDialog`,
  `DLQDiscardDialog` (typed confirmation).
- List: filter by company (super), channel, status, time range.
- Detail: full alert payload + delivery history timeline (each
  attempt's error, response, latency).
- Replay: confirmation modal showing "this will redeliver to
  N recipients", audit row written.
- Discard: typed confirmation (`type "discard" to confirm`), audit
  row written.
- Optimistic updates for replay/discard (instant UI, rollback on
  error).
- **Delete `cmd/admind/ui/dlq.html`** and remove the old `GET /dlq`
  handler from `cmd/admind/main.go` (SPA fallback now serves the
  new DLQ page).

**Exit criteria:**
- [ ] DLQ list: all M8 rows visible, paginated, filterable.
- [ ] Replay: alert re-delivered (verified via deliverd metrics:
      `ba_deliverd_attempts_total{result=ok}` increments).
- [ ] Discard: row hidden from default list, `status=discarded`.
- [ ] Tenant-admin sees only their company's DLQ rows.
- [ ] Old `dlq.html` file deleted from `cmd/admind/ui/`.
- [ ] Old `GET /dlq` handler in `admind` removed.
- [ ] Bundle: dlq feature code-split (< 30 KB gzipped).

**Estimated:** 2-3 days.

---

### W3: Audit log viewer

**Goal:** Read-only table of `audit_log` rows, filterable.

**Scope:**
- Route: `/audit`.
- Feature folder: `web/src/features/audit/`.
- Components: `AuditList`, `AuditDetailModal` (expanded JSON
  metadata).
- List: filter by actor (user picker), action, entity_type, time
  range, free text on metadata.
- Read-only. No actions from this page.
- New backend endpoint: `GET /v1/audit` in `cmd/admind/main.go`.
  Reads from the `audit_log` table that M9 already created.
- New SQL: `SELECT ... FROM audit_log WHERE company_id = $1 ORDER BY
  ts DESC LIMIT $2 OFFSET $3` (super-admin passes company_id query
  param; tenant-admin forced to own).

**Exit criteria:**
- [ ] Audit list: rows from `ingestd`, `routerd`, `deliverd`,
      `admind` actions all visible.
- [ ] Filter by actor narrows the list.
- [ ] Filter by action narrows the list.
- [ ] Time range filter works.
- [ ] AuditDetailModal shows full metadata JSON pretty-printed.
- [ ] Tenant-admin sees only their company's audit rows.
- [ ] Bundle: audit feature code-split (< 20 KB gzipped).

**Estimated:** 1-2 days.

---

### W4: K8s manifests (admind + authd)

**Goal:** K8s manifests that bring up `admind` and `authd` alongside
the rest of the M12 stack. Carry-over from M12 W1 with M13-specific
additions (authd Service, JWT secret Secret).

**Scope:**
- `deploy/k8s/base/admind/`:
  - `deployment.yaml` — 2 replicas, resource limits, env from
    ConfigMap + Secret.
  - `service.yaml` — ClusterIP, ports 8803 (HTTP) + 9090 (gRPC,
    unused by frontend but kept for parity).
  - `configmap.yaml` — BA_HTTP_ADDR, BA_PG_*, BA_NATS_*.
  - `pdb.yaml` — minAvailable: 1.
- `deploy/k8s/base/authd/`:
  - `deployment.yaml` — 2 replicas, env from Secret.
  - `service.yaml` — ClusterIP, port 8804.
  - `configmap.yaml` — BA_AUTHD_HTTP_ADDR, BA_AUTH_SMTP_*.
  - `secret.yaml.example` — BA_AUTH_JWT_SECRET (real one gitignored).
  - `pdb.yaml` — minAvailable: 1.
- `deploy/k8s/base/kustomization.yaml` updated to include both.
- `deploy/k8s/README.md` — quickstart, secret-rotation note.
- Carry from M12 W1: namespace, postgres, nats, redis, clickhouse,
  emqx, ingestd, routerd, deliverd-*, archiverd, prometheus, grafana.

**Exit criteria:**
- [ ] `kubectl apply -k deploy/k8s/base/` brings up `admind` and
      `authd` alongside the rest.
- [ ] `kubectl get pods -n broad-announce` shows both Running +
      Ready.
- [ ] Kustomize output passes `kubeconform` (or
      `kubectl apply --dry-run=server`).
- [ ] Single `admind` binary size < 60 MB.
- [ ] `deploy/k8s/README.md` documents the quickstart and the
      JWT-secret rotation procedure.

**Estimated:** 1 day (most of the templates already exist from
M12 W1).

---

### W5: Playwright E2E + CI pipeline + final verification

**Goal:** Single Playwright happy-path test that runs in CI.
Final `M13_VERIFICATION.md` with screenshots.

**Scope:**
- `web/tests/e2e/happy-path.spec.ts`:
  1. Login as super-admin.
  2. Create a company.
  3. Create a source (capture one-time secrets from response).
  4. Send 1 alert via curl to ingestd (Playwright runs the curl in
     a subprocess).
  5. See the alert in Live Tail within 2s.
  6. Open DLQ, replay a row, verify deliverd metrics incremented.
  7. Open Audit, see all 6+ audit rows.
  8. Logout.
  9. Login as tenant-admin (created during the test).
  10. Cross-tenant 403 attempt: try to GET another company's
      sources → expect 403.
- `web/playwright.config.ts` — chromium only, base URL from env.
- `scripts/m13_e2e.sh` — wraps the Playwright run, brings up
  docker-compose, runs, teardown (no `-v`, learned from M11).
- `.github/workflows/m13-e2e.yml` (or GitLab CI equivalent) — runs
  the smoke on every PR that touches `cmd/**` or `web/**`.
- `M13_VERIFICATION.md`:
  - Screenshots of all 6 modules.
  - Playwright run log.
  - Tenant-isolation 403 test.
  - Binary size.
  - Bundle breakdown.
  - CI link.

**Exit criteria:**
- [ ] Playwright happy-path test green on local docker-compose.
- [ ] Playwright happy-path test green in CI for 3 consecutive
      runs.
- [ ] CI workflow runs on every PR; green required to merge.
- [ ] `M13_VERIFICATION.md` published.
- [ ] SPEC.md `M13a`, `M13b`, `M13c` rows all flipped to
      **✅ shipped YYYY-MM-DD**.

**Estimated:** 1-2 days.

---

## 3. Sequencing

```
W1 (2-3d) ──┐
            ├──▶ W2 (2-3d) ──▶ W3 (1-2d) ──▶ W5 (1-2d)
            │
W4 (1d) ────┴── (parallel with W1-W3, joins W5 at the end)
```

- **W1 alone:** 2-3 days.
- **W2 after W1:** 2-3 days.
- **W3 after W2:** 1-2 days.
- **W4 in parallel:** 1 day.
- **W5 last:** 1-2 days.
- **Total: ~7-11 days, target 7-9 with overlap.**

---

## 4. Risks specific to M13c

| Risk | Likelihood | Impact | Mitigation |
|---|---|---|---|
| SSE through corporate proxy | Medium | Medium | HTTP fallback (`/v1/tail/recent`) is in the API; UI has a "use polling" toggle |
| Playwright flakes in CI | Medium | Medium | Retry once on failure; isolate test data per run (unique company slug with timestamp) |
| `dlq.html` removal breaks a customer who's bookmarked it | Low | Low | Note in release notes; old URL → new SPA `/dlq` route |
| Audit log volume overwhelms the list | Medium | Low | Server-side pagination (25/50/100); filters required for >1k rows |
| K8s manifest drift from M12 W1 | Medium | Low | Reuse M12 templates; W4 PR diff'd against M12 W1 to catch unintended changes |
| Single binary size > 60 MB | Low | Low | Per-feature code splitting; gzip embed; CI size gate |

---

## 5. Definition of done — M13c

- [ ] All 5 workstreams have their per-workstream exit criteria
      checked.
- [ ] `cmd/admind/ui/dlq.html` removed.
- [ ] Old `GET /dlq` handler in `admind` removed.
- [ ] `admind` binary size < 60 MB.
- [ ] K8s manifests apply cleanly, both `admind` and `authd`
      Running.
- [ ] Playwright happy-path green in CI for 3 consecutive runs.
- [ ] Tenant-isolation 403 test green in CI.
- [ ] `M13_VERIFICATION.md` published.
- [ ] SPEC.md `M13a`, `M13b`, `M13c` rows all flipped to
      **✅ shipped YYYY-MM-DD**.

---

## 6. Definition of done — whole M13 (final check)

- [ ] All 3 sub-milestones (M13a, M13b, M13c) DoD checked.
- [ ] Demo to a real customer (or internal stakeholder) and signed
      off.
- [ ] SPEC.md `M13` row (top-level) updated to **✅ shipped
      YYYY-MM-DD** with a one-line summary linking to M13a/b/c
      VERIFICATION docs.

---

**Next step:** M13c W1 (Live tail). W4 (K8s) can start in parallel
from a separate branch.