# M14 — Security Hardening Plan

> Implements the security promises of SPEC §15 (mTLS) and closes
> the M11.5 wishlist (mTLS + secrets management + network policies).
> Makes broad-announce defensible to enterprise customers without
> re-architecting the data plane.

**Status:** planning (post-M11, parallel to M12/M13)
**Target:** SPEC §15 (mTLS opt-in per source) + gRPC internal mTLS
(today a TODO) + cert lifecycle UI + secrets rotation
**Estimate:** 16-24 days with one engineer; 10-15 days with two
engineers in parallel with M13c

---

## 0. Recap — what M14 must prove

| Surface | Acceptance criteria |
|---|---|
| **CA hierarchy** | Internal CA (root + intermediate) provisioned via cert-manager, root offline, intermediate serves cluster + sources |
| **Source-side mTLS** | A source with `mtls_required=true` is rejected at ingestd if the client cert is missing, expired, signed by an untrusted CA, or has the wrong CN/SAN. HMAC + API key still work for non-mTLS sources. |
| **Internal gRPC mTLS** | `ingestd`, `routerd`, `deliverd-*` talk over mTLS with cert-manager-issued certs. The `TODO(m11)` in `internal/grpcserver/server.go:61` is closed. |
| **Cert lifecycle UI** | Super-admin can: list source certs, see expiration, revoke, generate a CSR (or auto-issue), download the cert bundle. Expiration < 30 days triggers a banner. |
| **Rotation** | Intermediate CA rotates annually. Certs rotate every 90 days. No downtime during rotation. Documented runbook. |
| **PromQL alerts** | `CertExpiringSoon` (>30d, < 7d, < 24h buckets), `CertRevoked`, `mTLSHandshakeErrorsHigh` |
| **Audit log** | Cert issue, rotate, revoke, expiration all produce audit rows. |

**What M14 is NOT:**
- **Not a SOC2 / ISO 27001 certification.** M14 is a prerequisite, not
  the certification. Compliance certification is M15+ (separate
  project, separate budget, separate auditor).
- **Not a key-vault replacement.** M14 uses cert-manager secrets.
  HashiCorp Vault integration is v2.
- **Not customer-facing PKI.** Sources that need mTLS get a cert
  signed by OUR intermediate CA. We don't issue public certs.
- **Not a redesign of auth.** mTLS is layered on top of HMAC + API key
  in a "AND" relation: cert proves identity, HMAC proves payload
  integrity, API key is the rate-limit token. All three can be
  required for the strictest sources.

---

## 1. Threat model — what mTLS actually buys

| Threat | Without mTLS | With mTLS (opt-in) |
|---|---|---|
| **Stolen API key from a leaked source** | Attacker forges alerts forever | Cert pinned to the source machine — rotate the source cert, attack stops |
| **Replay of captured HMAC** | Within 5-min window per Stripe-style | Cert auth is per-connection; replay needs both cert AND HMAC |
| **Network MITM (compromised router)** | TLS protects in transit, but anyone with the API key can call | Cert pinned; MITM can't produce a valid client cert without the private key |
| **Insider abuse (operator with DB access)** | Can read API keys | Certs are on the source machine, not in the DB |
| **Credential stuffing** | Common API keys, brute force | Each cert is unique; no shared secret to stuff |

**M14 doesn't fix:** compromised source machine (attacker has the
private key), compromised root CA (game over, but offline), social
engineering of operator (cert issuance is gated by super-admin).

---

## 2. Workstream overview

```
┌──────────────────────────┐    ┌──────────────────────────┐
│  W1: cert-manager + CA   │    │  W2: ingestd mTLS        │
│  (root offline,          │───▶│  (opt-in per source,     │
│   intermediate in K8s)   │    │   cert validation,       │
│                          │    │   CN/SAN extraction)     │
└──────────────┬───────────┘    └──────────────┬───────────┘
               │                               │
               ▼                               ▼
┌──────────────────────────┐    ┌──────────────────────────┐
│  W3: gRPC internal mTLS  │    │  W4: cert lifecycle UI   │
│  (server + client,       │    │  (M13 frontend: list,    │
│   closes the TODO)       │    │   issue, rotate, revoke) │
└──────────────┬───────────┘    └──────────────┬───────────┘
               │                               │
               ▼                               ▼
┌──────────────────────────────────────────────────────────────┐
│  W5: rotation + PromQL alerts + audit + runbook              │
└──────────────────────────────────────────────────────────────┘
                                               │
                                               ▼
┌──────────────────────────────────────────────────────────────┐
│  W6: smoke E2E mTLS path + M14_VERIFICATION.md               │
└──────────────────────────────────────────────────────────────┘
```

Six workstreams. W1 is the only hard blocker. W4 (UI) can be split:
the API changes land in M14, the UI lands in M13b/c as a feature
incremental on top of the M13 sources module.

---

## 3. Workstream details

### W1: cert-manager + CA hierarchy

**Goal:** Provision a working internal PKI in the K8s cluster. Root
CA offline. Intermediate CA issues certs. cert-manager handles
renewal.

**Scope:**
- Choose CA tooling: **`step-ca` or `cfssl` for the offline root,
  `cert-manager` for the intermediate and serving certs.**
  - `step-ca` if you want a single binary and JSON-based config.
  - `cfssl` if you want the Cloudflare-style Go binary, lighter.
  - cert-manager is the standard for K8s and gives you
    `Certificate` CRDs, Issuer/ClusterIssuer, renewal controller.
- **Root CA (offline):**
  - Generate root key + cert on an air-gapped machine or a
    passphrase-encrypted USB stick.
  - 10-year validity. RSA 4096 or ECDSA P-384.
  - Stored in a K8s `Secret` of type `kubernetes.io/tls` in a
    `cert-manager` namespace, with sealed-secrets or external-secrets
    (NOT plaintext git).
  - Used ONLY to sign the intermediate. Never used to serve.
- **Intermediate CA (in-cluster):**
  - cert-manager `ClusterIssuer` backed by the root.
  - 1-year validity, rotated annually (cron triggers re-issue).
  - RSA 2048 or ECDSA P-256. ECDSA preferred (smaller, faster).
- **Serving certs (per-service):**
  - `Certificate` CRD per service (ingestd, routerd, deliverd-*,
    admind, authd).
  - DNS SANs: `<svc>.<namespace>.svc.cluster.local`, plus any
    public DNS name.
  - 90-day validity, auto-renewed at 30 days.
  - Mounted as files via `volumeMounts` from a Secret.
- **cert-manager deployment:**
  - Helm chart, `cert-manager v1.16+`.
  - RBAC for the controller.
  - DNS-01 or HTTP-01 challenge for public certs (none needed for
    in-cluster).
- **Source cert issuance path (used by W2):**
  - `Issuer` of type `CA` (cert-manager) backed by the
    intermediate.
  - Super-admin POSTs a CSR via API → cert-manager signs → cert
    bundle returned.
- **Tooling scripts:**
  - `scripts/ca-init.sh` — generates root + intermediate from a
    fresh machine. One-time use. Writes encrypted tarball.
  - `scripts/ca-rotate-intermediate.sh` — annual cron. Issues
    new intermediate from root, swaps cert-manager's
    ClusterIssuer, re-issues all serving certs.

**Exit criteria:**
- [ ] Root CA exists, offline, encrypted.
- [ ] Intermediate CA exists in K8s, valid for 1 year.
- [ ] cert-manager running, `kubectl get clusterissuer` shows
      `broad-announce-intermediate` Ready.
- [ ] 5 sample `Certificate` resources exist (ingestd, routerd,
      deliverd-fcm, deliverd-telegram, admind, authd) all
      Ready with valid serving certs.
- [ ] Auto-renewal tested: manually expire a cert at 30 days
      remaining, watch it renew without downtime.
- [ ] `scripts/ca-init.sh` and `scripts/ca-rotate-intermediate.sh`
      are documented in `cmd/certmanager/README.md`.

**Estimated:** 3-4 days.

---

### W2: ingestd source-side mTLS (opt-in per source)

**Goal:** A source with `mtls_required=true` is rejected at ingestd
if the client cert is missing, expired, signed by an untrusted CA,
or has the wrong CN/SAN. HMAC + API key still work for non-mTLS
sources.

**Scope:**
- `internal/auth/mtls.go` (new package):
  - `ClientCertVerifier` that wraps a `*x509.CertPool` (the
    intermediate) and a CN/SAN allowlist.
  - Extracts `r.TLS.PeerCertificates[0]` from the request.
  - Verifies: not expired, signed by intermediate, CN matches
    `source.<source_id>.<company_slug>` or SAN
    `source:<source_id>`, key usage `ClientAuth`.
  - Returns 401 with a structured error if any check fails.
- `cmd/ingestd/main.go`:
  - Two listeners: `:443` (TLS, with mTLS opt-in) and `:80`
    (plaintext, redirect to :443 for mTLS-required sources).
  - In practice, two HTTP servers on different ports. The
    mTLS one uses `tls.Config{ClientAuth: tls.RequireAndVerifyClientCert, ClientCAs: pool}`.
  - When `source.mtls_required=true`, route through the TLS
    server. Else, plain HTTP (existing behavior).
  - For sources with both HMAC and mTLS, both must succeed
    (AND, not OR).
- `internal/config/config.go`:
  - New env vars: `BA_INGESTD_TLS_ADDR` (default `:8443`),
    `BA_INGESTD_TLS_CERT` (path), `BA_INGESTD_TLS_KEY` (path),
    `BA_INGESTD_TLS_CA` (path to intermediate cert PEM).
  - `BA_INGESTD_REQUIRE_MTLS_DEFAULT` (default `false` —
    controlled per source).
- `internal/grpcserver/server.go`:
  - Close the `TODO(m11): grpc.Creds(tlsCredentials())` by
    wiring the cert paths from config.
  - The gRPC server is internal-only (not exposed to sources),
    so the cert is the serving cert from W1.
- Migrations: `migrations/NNNN_add_source_cert_columns.sql`:
  - `sources.mtls_required` (already in schema, just wire it).
  - `sources.mtls_cn` (CN expected, auto-derived from source_id).
  - `sources.cert_id` (FK to a new `source_certs` table).
  - `source_certs(id, source_id, serial, not_before, not_after,
    revoked_at, cert_pem)`.
- `cmd/admind/main.go` (new endpoints in M14):
  - `POST /v1/sources/:id/cert/csr` — accept a CSR (PEM), sign it
    with the intermediate, return the cert bundle.
  - `GET /v1/sources/:id/cert` — current cert + chain.
  - `DELETE /v1/sources/:id/cert` — revoke.
  - `GET /v1/certs/expiring?days=30` — list of certs expiring soon.
- New audit log events: `cert.issue`, `cert.revoke`, `cert.expire_soon`.

**Exit criteria:**
- [ ] `source.mtls_required=true` → ingestd rejects requests
      without a valid client cert (curl with `--cert` succeeds,
      curl without fails with 401).
- [ ] Cert signed by an untrusted CA → 401.
- [ ] Expired cert → 401.
- [ ] Wrong CN/SAN → 401.
- [ ] Cert revoked via `DELETE /v1/sources/:id/cert` → 401 within
      60s (cert-manager CRL updates).
- [ ] `source.mtls_required=false` → existing HMAC + API key flow
      unchanged.
- [ ] Both required (mTLS + HMAC) → both must succeed.
- [ ] Internal gRPC between ingestd, routerd, deliverd uses
      mTLS (W3) — verified with `tcpdump` or by
      `kubescape`/`kube-bench`.

**Estimated:** 5-7 days.

---

### W3: gRPC internal mTLS (server + client)

**Goal:** Close the `TODO(m11)` in `internal/grpcserver/server.go:61`
and the analogous client path. All service-to-service gRPC (and
HTTP/2 streams) use mTLS with cert-manager-issued certs.

**Scope:**
- `internal/grpcserver/server.go`:
  - Read `BA_GRPC_TLS_CERT`, `BA_GRPC_TLS_KEY`, `BA_GRPC_TLS_CA`
    from env.
  - `grpc.Creds(credentials.NewServerTLSFromCert(&cert))` — or the
    mTLS variant for client certs (if any service authenticates
    the other).
  - Hot-reload: SIGHUP or file watcher on the cert files.
    cert-manager renews → kubelet updates the Secret → file
    changes → service reloads without restart.
- `internal/grpcclient/client.go` and `options.go`:
  - Wire the `TransportCredentials` from the same env vars.
  - For mTLS to other services: load the cert-manager-issued
    client cert + CA.
- `internal/grpcclient/options.go:44`:
  - The "Production deployments should use mTLS or at minimum
    TLS" comment becomes code.
- `deploy/k8s/base/*/deployment.yaml`:
  - Mount the cert-manager Secret as a volume at
    `/etc/broad-announce/tls/`.
  - `lifecycle.preStop` hook to drain in-flight requests
    gracefully before pod termination (avoid 5xx during
    cert rotation).
- `internal/observability/metrics.go`:
  - `ba_grpc_tls_handshakes_total{result=ok|error,peer=...}`.
  - `ba_grpc_tls_cert_age_seconds{service=...}` (gauge).

**Exit criteria:**
- [ ] Internal gRPC between any two services uses TLS.
- [ ] `tcpdump` shows TLS handshake (not plaintext) for a sample
      gRPC call between ingestd and routerd.
- [ ] Cert rotation (manually expire a cert) doesn't drop a single
      in-flight request (verified with a load test).
- [ ] Hot-reload works: SIGHUP picks up the new cert without a
      pod restart.
- [ ] Metrics exported for handshake success/failure.

**Estimated:** 2-3 days.

---

### W4: cert lifecycle UI (M13 frontend)

**Goal:** Super-admin can manage source certs from the M13 UI. CSR
upload, cert download, revocation, expiration warnings.

**Scope:**
- Routes (added to M13b W2 — Sources module):
  - `/sources/:id/cert` — current cert info, expiration, status.
  - Modal: "Generate CSR" (paste a CSR, get a signed cert back) or
    "Auto-generate" (server generates a key + CSR, signs it,
    returns the cert bundle as a downloadable .zip).
  - Modal: "Revoke" with typed confirmation.
- Component: `CertStatusBadge` — green/amber/red based on
  expiration.
- Component: `CertExpirationBanner` — global banner in the top
  bar when any source cert is < 30 days from expiry. Clicks
  navigate to the cert list.
- New endpoints in `cmd/admind`:
  - `POST /v1/sources/:id/cert/csr` (from W2).
  - `GET /v1/sources/:id/cert`.
  - `DELETE /v1/sources/:id/cert`.
  - `GET /v1/certs/expiring?days=30`.
- Bundle: cert-management code-split (< 15 KB gzipped).

**Exit criteria:**
- [ ] Super-admin can upload a CSR and download a signed cert.
- [ ] Super-admin can use "Auto-generate" and get a `.zip` with
      cert + chain + private key (one-time download).
- [ ] Revoke button removes the cert; subsequent requests with
      that cert fail within 60s.
- [ ] Expiration banner appears 30 days before any cert expires.
- [ ] Tenant-admin sees cert info for their sources (read-only,
      no revoke).
- [ ] Bundle size < 15 KB gzipped.

**Estimated:** 3-5 days (in parallel with M13b, lands in M13c W7).

---

### W5: rotation + PromQL alerts + audit + runbook

**Goal:** Automated cert rotation. PromQL alerts for expiration and
revocation. Audit log for every cert event. Runbook for incident
response.

**Scope:**
- Rotation:
  - cert-manager handles serving certs automatically (90-day,
    renew at 30).
  - `cert-manager.io/renew-before: 720h` annotation.
  - Source certs: 90-day validity, super-admin is notified
    30/7/1 days before via the banner.
  - Annual intermediate CA rotation: `scripts/ca-rotate-intermediate.sh`
    (from W1).
- PromQL alerts (in `deploy/prometheus/alerts.yml`):
  - `CertExpiringSoon` (severity warning): `cert_not_after -
    time() < 30 * 86400` for any source cert.
  - `CertExpiringCritical` (severity critical): `< 7 * 86400`.
  - `CertExpired` (severity critical): `cert_not_after < time()`.
  - `CertRevoked` (severity info): counter increment.
  - `mTLSHandshakeErrorsHigh`: rate of failed handshakes > 1%
    for 5 min.
- Audit log:
  - `cert.issue` — who, which source, cert serial.
  - `cert.rotate` — auto-rotation, no actor.
  - `cert.revoke` — who, which source, reason.
  - `cert.expire` — auto, when cert is detected expired.
- Runbook (`docs/runbooks/mtls-incident.md`):
  - "Cert expired and source is down" — reissue flow.
  - "Private key compromise" — revoke + reissue + audit.
  - "CA compromise" — generate new root offline, re-issue
    intermediate, re-issue all certs. Documented as M14.5
    emergency procedure.
  - "Handshake errors spiking" — likely clock skew or CA
    pool issue. Check cert chain, check issuer.

**Exit criteria:**
- [ ] All 5 PromQL alerts fire in a test scenario (expire a
      cert, revoke a cert, simulate clock skew).
- [ ] Annual rotation script tested in a staging cluster.
- [ ] Runbook reviewed by 2 people (you + me, or you + a
      peer).
- [ ] Audit log entries for issue, rotate, revoke, expire.

**Estimated:** 2-3 days.

---

### W6: smoke E2E mTLS path + M14_VERIFICATION.md

**Goal:** End-to-end smoke that exercises the full mTLS path:
cert issuance → cert-bound source → ingestd validation → alert
accepted → cert revocation → alert rejected.

**Scope:**
- `scripts/m14_smoke.sh`:
  1. Bring up stack with cert-manager + intermediate CA.
  2. Super-admin creates a company.
  3. Super-admin creates a source with `mtls_required=true`.
  4. Super-admin generates a cert for the source via API.
  5. `curl` with `--cert` and `--key` to ingestd using that
     cert + HMAC → 200.
  6. `curl` without cert → 401.
  7. `curl` with expired cert → 401.
  8. `curl` with HMAC-only (cert stripped) → 401.
  9. Revoke the cert via API.
  10. `curl` with revoked cert → 401.
  11. Send a real alert via the working path → see it in
      live tail (M13c).
- `M14_VERIFICATION.md`:
  - Smoke log.
  - Cert chain diagram (root → intermediate → serving).
  - Bundle size breakdown.
  - The 5 PromQL alerts with sample output.
  - Runbook link.
  - Sign-off: "broad-announce v1 supports opt-in mTLS per
    source, with cert-manager-managed rotation, lifecycle
    UI, and incident response runbook."

**Exit criteria:**
- [ ] `make m14-smoke` exits 0 from a clean state.
- [ ] 3 consecutive green runs.
- [ ] M13 functionality not regressed.
- [ ] M14_VERIFICATION.md published.

**Estimated:** 1-2 days.

---

## 4. Sequencing & parallelism

```
W1 (3-4d) ──┬──▶ W2 (5-7d) ──┬──▶ W4 UI portion (3-5d, parallel with M13)
            │                 │
            │                 ├──▶ W3 (2-3d, parallel with W2)
            │                 │
            │                 └──▶ W5 (2-3d) ──▶ W6 (1-2d)
            │
            └── (W1 is the only hard blocker)
```

**With one engineer (sequential):**
- W1: 3-4d
- W2 + W3: 7-10d (can overlap, W3 short)
- W4: 3-5d (after M13b lands)
- W5: 2-3d
- W6: 1-2d
- **Total: 16-24 days**

**With two engineers (W4 lands in M13c):**
- Eng 1: W1 → W2 → W5 → W6 (9-12 days)
- Eng 2: M13c + W4 (parallel with W2) → W3 → W5 (10-15 days)
- **Total wall time: 10-15 days**

---

## 5. What M14 is NOT

(Recorded so a future reader doesn't re-litigate.)

- **Not a SOC2 certification.** M14 builds the technical controls.
  SOC2 is an audit process (M15+).
- **Not customer-issued public certs.** We sign with our internal
  intermediate. Sources that need a public cert bring their own.
- **Not Vault / KMS integration.** cert-manager secrets are K8s
  Secrets (sealed-secrets or external-secrets encrypted at rest).
  HashiCorp Vault + dynamic certs is v2.
- **Not HSM-backed root.** The root is on an encrypted USB stick
  (or air-gapped VM). HSM is v3.
- **Not a full PKI replacement.** External CAs (DigiCert, Let's
  Encrypt) are still used for any public-facing endpoints (the
  broad-announce marketing site, the public docs). M14 is for
  in-cluster + source-side certs.
- **Not a redesign of the data plane.** ingestd, routerd, deliverd
  keep their Go code. M14 layers TLS on top, doesn't rewrite.
- **Not retroactive.** Existing sources with HMAC + API key
  continue working. mTLS is opt-in per source. No flag day.

---

## 6. Dependencies

| From | What | Status | Affects |
|---|---|---|---|
| M12 W1 | K8s manifests exist | ⏳ in flight | W1 (cert-manager deploy) |
| M11 | gRPC infra stable | ✅ done | W3 (gRPC mTLS) |
| M13a | authd + JWT gate | ⏳ not started | W4 (cert UI uses auth) |
| M13b | Sources CRUD | ⏳ not started | W4 (cert UI is incremental on Sources) |
| Postgres | `source_certs` table | to be created in W2 migration | W2 |
| cert-manager | upstream chart | ✅ available | W1 |

---

## 7. Risks

| Risk | Likelihood | Impact | Mitigation |
|---|---|---|---|
| Root CA compromise | Very low | Catastrophic | Root offline + encrypted; documented re-issue procedure (full cluster cert rotation) |
| Cert rotation drops in-flight requests | Medium | High | PreStop hook + graceful drain + 30-day renewal window + W3 hot-reload |
| Operator issues cert to wrong source | Medium | Medium | Cert bound to source_id in CN/SAN; UI shows source label before issuance |
| `mtls_required=true` breaks an existing source | Medium | High | Opt-in per source; rollout plan: 1 source, 5 sources, all sources, default-on (3 quarters) |
| `cert-manager` operator upgrade breaks Issuer | Low | High | Pin chart version; test upgrade in staging first |
| CRL distribution lag → revoked cert still works | Medium | Medium | 60s SLA in W2; OCSP responder for real-time (v2) |
| Sources can't generate CSRs (legacy systems) | Medium | Medium | "Auto-generate" button in UI: server generates key+CSR, returns bundle |

---

## 8. Definition of done — M14

- [ ] All 6 workstreams have their per-workstream exit criteria
      checked.
- [ ] `make m14-smoke` green for 3 consecutive runs.
- [ ] M13 functionality not regressed.
- [ ] M14_VERIFICATION.md published.
- [ ] Runbook reviewed by 2 people.
- [ ] SPEC.md `M14` row flipped to **✅ shipped YYYY-MM-DD** with
      the entry: *"opt-in mTLS per source + cert-manager-managed
      rotation + cert lifecycle UI + internal gRPC mTLS + runbook.
      Closes M11.5 wishlist. See M14_VERIFICATION.md for evidence."*

---

## 9. What M14 buys the business

| Before (today) | After (M14) |
|---|---|
| "API key + HMAC" in marketing | "mTLS opt-in per source" in marketing |
| Stolen API key = game over | Cert rotation stops the attack |
| gRPC internal plaintext (TODO) | gRPC internal mTLS |
| Manual cert management (if any) | cert-manager + UI |
| "Enterprise" in pitch deck | "Enterprise" in feature matrix |
| No compliance trail | Audit log for every cert event |
| Not SOC2-ready | **SOC2-ready controls in place** (cert audit is one of the SOC2 controls) |

**What it does NOT buy:** SOC2 itself, FedRAMP, HIPAA. Those are
certifications, not features. M14 builds the technical controls
that the auditors will check; getting the certification is a
separate project (M15).

---

**Next step:** start W1 (cert-manager + CA). If we have 2
engineers, W4 UI work starts as soon as M13b W1 (Companies CRUD)
is done — the cert UI is a Sources incremental.