|
@@ -0,0 +1,521 @@
|
|
|
|
|
+# M14 — Security Hardening Plan
|
|
|
|
|
+
|
|
|
|
|
+> Implements the security promises of SPEC §15 (mTLS) and closes
|
|
|
|
|
+> the M11.5 wishlist (mTLS + secrets management + network policies).
|
|
|
|
|
+> Makes broad-announce defensible to enterprise customers without
|
|
|
|
|
+> re-architecting the data plane.
|
|
|
|
|
+
|
|
|
|
|
+**Status:** planning (post-M11, parallel to M12/M13)
|
|
|
|
|
+**Target:** SPEC §15 (mTLS opt-in per source) + gRPC internal mTLS
|
|
|
|
|
+(today a TODO) + cert lifecycle UI + secrets rotation
|
|
|
|
|
+**Estimate:** 16-24 days with one engineer; 10-15 days with two
|
|
|
|
|
+engineers in parallel with M13c
|
|
|
|
|
+
|
|
|
|
|
+---
|
|
|
|
|
+
|
|
|
|
|
+## 0. Recap — what M14 must prove
|
|
|
|
|
+
|
|
|
|
|
+| Surface | Acceptance criteria |
|
|
|
|
|
+|---|---|
|
|
|
|
|
+| **CA hierarchy** | Internal CA (root + intermediate) provisioned via cert-manager, root offline, intermediate serves cluster + sources |
|
|
|
|
|
+| **Source-side mTLS** | A source with `mtls_required=true` is rejected at ingestd if the client cert is missing, expired, signed by an untrusted CA, or has the wrong CN/SAN. HMAC + API key still work for non-mTLS sources. |
|
|
|
|
|
+| **Internal gRPC mTLS** | `ingestd`, `routerd`, `deliverd-*` talk over mTLS with cert-manager-issued certs. The `TODO(m11)` in `internal/grpcserver/server.go:61` is closed. |
|
|
|
|
|
+| **Cert lifecycle UI** | Super-admin can: list source certs, see expiration, revoke, generate a CSR (or auto-issue), download the cert bundle. Expiration < 30 days triggers a banner. |
|
|
|
|
|
+| **Rotation** | Intermediate CA rotates annually. Certs rotate every 90 days. No downtime during rotation. Documented runbook. |
|
|
|
|
|
+| **PromQL alerts** | `CertExpiringSoon` (>30d, < 7d, < 24h buckets), `CertRevoked`, `mTLSHandshakeErrorsHigh` |
|
|
|
|
|
+| **Audit log** | Cert issue, rotate, revoke, expiration all produce audit rows. |
|
|
|
|
|
+
|
|
|
|
|
+**What M14 is NOT:**
|
|
|
|
|
+- **Not a SOC2 / ISO 27001 certification.** M14 is a prerequisite, not
|
|
|
|
|
+ the certification. Compliance certification is M15+ (separate
|
|
|
|
|
+ project, separate budget, separate auditor).
|
|
|
|
|
+- **Not a key-vault replacement.** M14 uses cert-manager secrets.
|
|
|
|
|
+ HashiCorp Vault integration is v2.
|
|
|
|
|
+- **Not customer-facing PKI.** Sources that need mTLS get a cert
|
|
|
|
|
+ signed by OUR intermediate CA. We don't issue public certs.
|
|
|
|
|
+- **Not a redesign of auth.** mTLS is layered on top of HMAC + API key
|
|
|
|
|
+ in a "AND" relation: cert proves identity, HMAC proves payload
|
|
|
|
|
+ integrity, API key is the rate-limit token. All three can be
|
|
|
|
|
+ required for the strictest sources.
|
|
|
|
|
+
|
|
|
|
|
+---
|
|
|
|
|
+
|
|
|
|
|
+## 1. Threat model — what mTLS actually buys
|
|
|
|
|
+
|
|
|
|
|
+| Threat | Without mTLS | With mTLS (opt-in) |
|
|
|
|
|
+|---|---|---|
|
|
|
|
|
+| **Stolen API key from a leaked source** | Attacker forges alerts forever | Cert pinned to the source machine — rotate the source cert, attack stops |
|
|
|
|
|
+| **Replay of captured HMAC** | Within 5-min window per Stripe-style | Cert auth is per-connection; replay needs both cert AND HMAC |
|
|
|
|
|
+| **Network MITM (compromised router)** | TLS protects in transit, but anyone with the API key can call | Cert pinned; MITM can't produce a valid client cert without the private key |
|
|
|
|
|
+| **Insider abuse (operator with DB access)** | Can read API keys | Certs are on the source machine, not in the DB |
|
|
|
|
|
+| **Credential stuffing** | Common API keys, brute force | Each cert is unique; no shared secret to stuff |
|
|
|
|
|
+
|
|
|
|
|
+**M14 doesn't fix:** compromised source machine (attacker has the
|
|
|
|
|
+private key), compromised root CA (game over, but offline), social
|
|
|
|
|
+engineering of operator (cert issuance is gated by super-admin).
|
|
|
|
|
+
|
|
|
|
|
+---
|
|
|
|
|
+
|
|
|
|
|
+## 2. Workstream overview
|
|
|
|
|
+
|
|
|
|
|
+```
|
|
|
|
|
+┌──────────────────────────┐ ┌──────────────────────────┐
|
|
|
|
|
+│ W1: cert-manager + CA │ │ W2: ingestd mTLS │
|
|
|
|
|
+│ (root offline, │───▶│ (opt-in per source, │
|
|
|
|
|
+│ intermediate in K8s) │ │ cert validation, │
|
|
|
|
|
+│ │ │ CN/SAN extraction) │
|
|
|
|
|
+└──────────────┬───────────┘ └──────────────┬───────────┘
|
|
|
|
|
+ │ │
|
|
|
|
|
+ ▼ ▼
|
|
|
|
|
+┌──────────────────────────┐ ┌──────────────────────────┐
|
|
|
|
|
+│ W3: gRPC internal mTLS │ │ W4: cert lifecycle UI │
|
|
|
|
|
+│ (server + client, │ │ (M13 frontend: list, │
|
|
|
|
|
+│ closes the TODO) │ │ issue, rotate, revoke) │
|
|
|
|
|
+└──────────────┬───────────┘ └──────────────┬───────────┘
|
|
|
|
|
+ │ │
|
|
|
|
|
+ ▼ ▼
|
|
|
|
|
+┌──────────────────────────────────────────────────────────────┐
|
|
|
|
|
+│ W5: rotation + PromQL alerts + audit + runbook │
|
|
|
|
|
+└──────────────────────────────────────────────────────────────┘
|
|
|
|
|
+ │
|
|
|
|
|
+ ▼
|
|
|
|
|
+┌──────────────────────────────────────────────────────────────┐
|
|
|
|
|
+│ W6: smoke E2E mTLS path + M14_VERIFICATION.md │
|
|
|
|
|
+└──────────────────────────────────────────────────────────────┘
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+Six workstreams. W1 is the only hard blocker. W4 (UI) can be split:
|
|
|
|
|
+the API changes land in M14, the UI lands in M13b/c as a feature
|
|
|
|
|
+incremental on top of the M13 sources module.
|
|
|
|
|
+
|
|
|
|
|
+---
|
|
|
|
|
+
|
|
|
|
|
+## 3. Workstream details
|
|
|
|
|
+
|
|
|
|
|
+### W1: cert-manager + CA hierarchy
|
|
|
|
|
+
|
|
|
|
|
+**Goal:** Provision a working internal PKI in the K8s cluster. Root
|
|
|
|
|
+CA offline. Intermediate CA issues certs. cert-manager handles
|
|
|
|
|
+renewal.
|
|
|
|
|
+
|
|
|
|
|
+**Scope:**
|
|
|
|
|
+- Choose CA tooling: **`step-ca` or `cfssl` for the offline root,
|
|
|
|
|
+ `cert-manager` for the intermediate and serving certs.**
|
|
|
|
|
+ - `step-ca` if you want a single binary and JSON-based config.
|
|
|
|
|
+ - `cfssl` if you want the Cloudflare-style Go binary, lighter.
|
|
|
|
|
+ - cert-manager is the standard for K8s and gives you
|
|
|
|
|
+ `Certificate` CRDs, Issuer/ClusterIssuer, renewal controller.
|
|
|
|
|
+- **Root CA (offline):**
|
|
|
|
|
+ - Generate root key + cert on an air-gapped machine or a
|
|
|
|
|
+ passphrase-encrypted USB stick.
|
|
|
|
|
+ - 10-year validity. RSA 4096 or ECDSA P-384.
|
|
|
|
|
+ - Stored in a K8s `Secret` of type `kubernetes.io/tls` in a
|
|
|
|
|
+ `cert-manager` namespace, with sealed-secrets or external-secrets
|
|
|
|
|
+ (NOT plaintext git).
|
|
|
|
|
+ - Used ONLY to sign the intermediate. Never used to serve.
|
|
|
|
|
+- **Intermediate CA (in-cluster):**
|
|
|
|
|
+ - cert-manager `ClusterIssuer` backed by the root.
|
|
|
|
|
+ - 1-year validity, rotated annually (cron triggers re-issue).
|
|
|
|
|
+ - RSA 2048 or ECDSA P-256. ECDSA preferred (smaller, faster).
|
|
|
|
|
+- **Serving certs (per-service):**
|
|
|
|
|
+ - `Certificate` CRD per service (ingestd, routerd, deliverd-*,
|
|
|
|
|
+ admind, authd).
|
|
|
|
|
+ - DNS SANs: `<svc>.<namespace>.svc.cluster.local`, plus any
|
|
|
|
|
+ public DNS name.
|
|
|
|
|
+ - 90-day validity, auto-renewed at 30 days.
|
|
|
|
|
+ - Mounted as files via `volumeMounts` from a Secret.
|
|
|
|
|
+- **cert-manager deployment:**
|
|
|
|
|
+ - Helm chart, `cert-manager v1.16+`.
|
|
|
|
|
+ - RBAC for the controller.
|
|
|
|
|
+ - DNS-01 or HTTP-01 challenge for public certs (none needed for
|
|
|
|
|
+ in-cluster).
|
|
|
|
|
+- **Source cert issuance path (used by W2):**
|
|
|
|
|
+ - `Issuer` of type `CA` (cert-manager) backed by the
|
|
|
|
|
+ intermediate.
|
|
|
|
|
+ - Super-admin POSTs a CSR via API → cert-manager signs → cert
|
|
|
|
|
+ bundle returned.
|
|
|
|
|
+- **Tooling scripts:**
|
|
|
|
|
+ - `scripts/ca-init.sh` — generates root + intermediate from a
|
|
|
|
|
+ fresh machine. One-time use. Writes encrypted tarball.
|
|
|
|
|
+ - `scripts/ca-rotate-intermediate.sh` — annual cron. Issues
|
|
|
|
|
+ new intermediate from root, swaps cert-manager's
|
|
|
|
|
+ ClusterIssuer, re-issues all serving certs.
|
|
|
|
|
+
|
|
|
|
|
+**Exit criteria:**
|
|
|
|
|
+- [ ] Root CA exists, offline, encrypted.
|
|
|
|
|
+- [ ] Intermediate CA exists in K8s, valid for 1 year.
|
|
|
|
|
+- [ ] cert-manager running, `kubectl get clusterissuer` shows
|
|
|
|
|
+ `broad-announce-intermediate` Ready.
|
|
|
|
|
+- [ ] 5 sample `Certificate` resources exist (ingestd, routerd,
|
|
|
|
|
+ deliverd-fcm, deliverd-telegram, admind, authd) all
|
|
|
|
|
+ Ready with valid serving certs.
|
|
|
|
|
+- [ ] Auto-renewal tested: manually expire a cert at 30 days
|
|
|
|
|
+ remaining, watch it renew without downtime.
|
|
|
|
|
+- [ ] `scripts/ca-init.sh` and `scripts/ca-rotate-intermediate.sh`
|
|
|
|
|
+ are documented in `cmd/certmanager/README.md`.
|
|
|
|
|
+
|
|
|
|
|
+**Estimated:** 3-4 days.
|
|
|
|
|
+
|
|
|
|
|
+---
|
|
|
|
|
+
|
|
|
|
|
+### W2: ingestd source-side mTLS (opt-in per source)
|
|
|
|
|
+
|
|
|
|
|
+**Goal:** A source with `mtls_required=true` is rejected at ingestd
|
|
|
|
|
+if the client cert is missing, expired, signed by an untrusted CA,
|
|
|
|
|
+or has the wrong CN/SAN. HMAC + API key still work for non-mTLS
|
|
|
|
|
+sources.
|
|
|
|
|
+
|
|
|
|
|
+**Scope:**
|
|
|
|
|
+- `internal/auth/mtls.go` (new package):
|
|
|
|
|
+ - `ClientCertVerifier` that wraps a `*x509.CertPool` (the
|
|
|
|
|
+ intermediate) and a CN/SAN allowlist.
|
|
|
|
|
+ - Extracts `r.TLS.PeerCertificates[0]` from the request.
|
|
|
|
|
+ - Verifies: not expired, signed by intermediate, CN matches
|
|
|
|
|
+ `source.<source_id>.<company_slug>` or SAN
|
|
|
|
|
+ `source:<source_id>`, key usage `ClientAuth`.
|
|
|
|
|
+ - Returns 401 with a structured error if any check fails.
|
|
|
|
|
+- `cmd/ingestd/main.go`:
|
|
|
|
|
+ - Two listeners: `:443` (TLS, with mTLS opt-in) and `:80`
|
|
|
|
|
+ (plaintext, redirect to :443 for mTLS-required sources).
|
|
|
|
|
+ - In practice, two HTTP servers on different ports. The
|
|
|
|
|
+ mTLS one uses `tls.Config{ClientAuth: tls.RequireAndVerifyClientCert, ClientCAs: pool}`.
|
|
|
|
|
+ - When `source.mtls_required=true`, route through the TLS
|
|
|
|
|
+ server. Else, plain HTTP (existing behavior).
|
|
|
|
|
+ - For sources with both HMAC and mTLS, both must succeed
|
|
|
|
|
+ (AND, not OR).
|
|
|
|
|
+- `internal/config/config.go`:
|
|
|
|
|
+ - New env vars: `BA_INGESTD_TLS_ADDR` (default `:8443`),
|
|
|
|
|
+ `BA_INGESTD_TLS_CERT` (path), `BA_INGESTD_TLS_KEY` (path),
|
|
|
|
|
+ `BA_INGESTD_TLS_CA` (path to intermediate cert PEM).
|
|
|
|
|
+ - `BA_INGESTD_REQUIRE_MTLS_DEFAULT` (default `false` —
|
|
|
|
|
+ controlled per source).
|
|
|
|
|
+- `internal/grpcserver/server.go`:
|
|
|
|
|
+ - Close the `TODO(m11): grpc.Creds(tlsCredentials())` by
|
|
|
|
|
+ wiring the cert paths from config.
|
|
|
|
|
+ - The gRPC server is internal-only (not exposed to sources),
|
|
|
|
|
+ so the cert is the serving cert from W1.
|
|
|
|
|
+- Migrations: `migrations/NNNN_add_source_cert_columns.sql`:
|
|
|
|
|
+ - `sources.mtls_required` (already in schema, just wire it).
|
|
|
|
|
+ - `sources.mtls_cn` (CN expected, auto-derived from source_id).
|
|
|
|
|
+ - `sources.cert_id` (FK to a new `source_certs` table).
|
|
|
|
|
+ - `source_certs(id, source_id, serial, not_before, not_after,
|
|
|
|
|
+ revoked_at, cert_pem)`.
|
|
|
|
|
+- `cmd/admind/main.go` (new endpoints in M14):
|
|
|
|
|
+ - `POST /v1/sources/:id/cert/csr` — accept a CSR (PEM), sign it
|
|
|
|
|
+ with the intermediate, return the cert bundle.
|
|
|
|
|
+ - `GET /v1/sources/:id/cert` — current cert + chain.
|
|
|
|
|
+ - `DELETE /v1/sources/:id/cert` — revoke.
|
|
|
|
|
+ - `GET /v1/certs/expiring?days=30` — list of certs expiring soon.
|
|
|
|
|
+- New audit log events: `cert.issue`, `cert.revoke`, `cert.expire_soon`.
|
|
|
|
|
+
|
|
|
|
|
+**Exit criteria:**
|
|
|
|
|
+- [ ] `source.mtls_required=true` → ingestd rejects requests
|
|
|
|
|
+ without a valid client cert (curl with `--cert` succeeds,
|
|
|
|
|
+ curl without fails with 401).
|
|
|
|
|
+- [ ] Cert signed by an untrusted CA → 401.
|
|
|
|
|
+- [ ] Expired cert → 401.
|
|
|
|
|
+- [ ] Wrong CN/SAN → 401.
|
|
|
|
|
+- [ ] Cert revoked via `DELETE /v1/sources/:id/cert` → 401 within
|
|
|
|
|
+ 60s (cert-manager CRL updates).
|
|
|
|
|
+- [ ] `source.mtls_required=false` → existing HMAC + API key flow
|
|
|
|
|
+ unchanged.
|
|
|
|
|
+- [ ] Both required (mTLS + HMAC) → both must succeed.
|
|
|
|
|
+- [ ] Internal gRPC between ingestd, routerd, deliverd uses
|
|
|
|
|
+ mTLS (W3) — verified with `tcpdump` or by
|
|
|
|
|
+ `kubescape`/`kube-bench`.
|
|
|
|
|
+
|
|
|
|
|
+**Estimated:** 5-7 days.
|
|
|
|
|
+
|
|
|
|
|
+---
|
|
|
|
|
+
|
|
|
|
|
+### W3: gRPC internal mTLS (server + client)
|
|
|
|
|
+
|
|
|
|
|
+**Goal:** Close the `TODO(m11)` in `internal/grpcserver/server.go:61`
|
|
|
|
|
+and the analogous client path. All service-to-service gRPC (and
|
|
|
|
|
+HTTP/2 streams) use mTLS with cert-manager-issued certs.
|
|
|
|
|
+
|
|
|
|
|
+**Scope:**
|
|
|
|
|
+- `internal/grpcserver/server.go`:
|
|
|
|
|
+ - Read `BA_GRPC_TLS_CERT`, `BA_GRPC_TLS_KEY`, `BA_GRPC_TLS_CA`
|
|
|
|
|
+ from env.
|
|
|
|
|
+ - `grpc.Creds(credentials.NewServerTLSFromCert(&cert))` — or the
|
|
|
|
|
+ mTLS variant for client certs (if any service authenticates
|
|
|
|
|
+ the other).
|
|
|
|
|
+ - Hot-reload: SIGHUP or file watcher on the cert files.
|
|
|
|
|
+ cert-manager renews → kubelet updates the Secret → file
|
|
|
|
|
+ changes → service reloads without restart.
|
|
|
|
|
+- `internal/grpcclient/client.go` and `options.go`:
|
|
|
|
|
+ - Wire the `TransportCredentials` from the same env vars.
|
|
|
|
|
+ - For mTLS to other services: load the cert-manager-issued
|
|
|
|
|
+ client cert + CA.
|
|
|
|
|
+- `internal/grpcclient/options.go:44`:
|
|
|
|
|
+ - The "Production deployments should use mTLS or at minimum
|
|
|
|
|
+ TLS" comment becomes code.
|
|
|
|
|
+- `deploy/k8s/base/*/deployment.yaml`:
|
|
|
|
|
+ - Mount the cert-manager Secret as a volume at
|
|
|
|
|
+ `/etc/broad-announce/tls/`.
|
|
|
|
|
+ - `lifecycle.preStop` hook to drain in-flight requests
|
|
|
|
|
+ gracefully before pod termination (avoid 5xx during
|
|
|
|
|
+ cert rotation).
|
|
|
|
|
+- `internal/observability/metrics.go`:
|
|
|
|
|
+ - `ba_grpc_tls_handshakes_total{result=ok|error,peer=...}`.
|
|
|
|
|
+ - `ba_grpc_tls_cert_age_seconds{service=...}` (gauge).
|
|
|
|
|
+
|
|
|
|
|
+**Exit criteria:**
|
|
|
|
|
+- [ ] Internal gRPC between any two services uses TLS.
|
|
|
|
|
+- [ ] `tcpdump` shows TLS handshake (not plaintext) for a sample
|
|
|
|
|
+ gRPC call between ingestd and routerd.
|
|
|
|
|
+- [ ] Cert rotation (manually expire a cert) doesn't drop a single
|
|
|
|
|
+ in-flight request (verified with a load test).
|
|
|
|
|
+- [ ] Hot-reload works: SIGHUP picks up the new cert without a
|
|
|
|
|
+ pod restart.
|
|
|
|
|
+- [ ] Metrics exported for handshake success/failure.
|
|
|
|
|
+
|
|
|
|
|
+**Estimated:** 2-3 days.
|
|
|
|
|
+
|
|
|
|
|
+---
|
|
|
|
|
+
|
|
|
|
|
+### W4: cert lifecycle UI (M13 frontend)
|
|
|
|
|
+
|
|
|
|
|
+**Goal:** Super-admin can manage source certs from the M13 UI. CSR
|
|
|
|
|
+upload, cert download, revocation, expiration warnings.
|
|
|
|
|
+
|
|
|
|
|
+**Scope:**
|
|
|
|
|
+- Routes (added to M13b W2 — Sources module):
|
|
|
|
|
+ - `/sources/:id/cert` — current cert info, expiration, status.
|
|
|
|
|
+ - Modal: "Generate CSR" (paste a CSR, get a signed cert back) or
|
|
|
|
|
+ "Auto-generate" (server generates a key + CSR, signs it,
|
|
|
|
|
+ returns the cert bundle as a downloadable .zip).
|
|
|
|
|
+ - Modal: "Revoke" with typed confirmation.
|
|
|
|
|
+- Component: `CertStatusBadge` — green/amber/red based on
|
|
|
|
|
+ expiration.
|
|
|
|
|
+- Component: `CertExpirationBanner` — global banner in the top
|
|
|
|
|
+ bar when any source cert is < 30 days from expiry. Clicks
|
|
|
|
|
+ navigate to the cert list.
|
|
|
|
|
+- New endpoints in `cmd/admind`:
|
|
|
|
|
+ - `POST /v1/sources/:id/cert/csr` (from W2).
|
|
|
|
|
+ - `GET /v1/sources/:id/cert`.
|
|
|
|
|
+ - `DELETE /v1/sources/:id/cert`.
|
|
|
|
|
+ - `GET /v1/certs/expiring?days=30`.
|
|
|
|
|
+- Bundle: cert-management code-split (< 15 KB gzipped).
|
|
|
|
|
+
|
|
|
|
|
+**Exit criteria:**
|
|
|
|
|
+- [ ] Super-admin can upload a CSR and download a signed cert.
|
|
|
|
|
+- [ ] Super-admin can use "Auto-generate" and get a `.zip` with
|
|
|
|
|
+ cert + chain + private key (one-time download).
|
|
|
|
|
+- [ ] Revoke button removes the cert; subsequent requests with
|
|
|
|
|
+ that cert fail within 60s.
|
|
|
|
|
+- [ ] Expiration banner appears 30 days before any cert expires.
|
|
|
|
|
+- [ ] Tenant-admin sees cert info for their sources (read-only,
|
|
|
|
|
+ no revoke).
|
|
|
|
|
+- [ ] Bundle size < 15 KB gzipped.
|
|
|
|
|
+
|
|
|
|
|
+**Estimated:** 3-5 days (in parallel with M13b, lands in M13c W7).
|
|
|
|
|
+
|
|
|
|
|
+---
|
|
|
|
|
+
|
|
|
|
|
+### W5: rotation + PromQL alerts + audit + runbook
|
|
|
|
|
+
|
|
|
|
|
+**Goal:** Automated cert rotation. PromQL alerts for expiration and
|
|
|
|
|
+revocation. Audit log for every cert event. Runbook for incident
|
|
|
|
|
+response.
|
|
|
|
|
+
|
|
|
|
|
+**Scope:**
|
|
|
|
|
+- Rotation:
|
|
|
|
|
+ - cert-manager handles serving certs automatically (90-day,
|
|
|
|
|
+ renew at 30).
|
|
|
|
|
+ - `cert-manager.io/renew-before: 720h` annotation.
|
|
|
|
|
+ - Source certs: 90-day validity, super-admin is notified
|
|
|
|
|
+ 30/7/1 days before via the banner.
|
|
|
|
|
+ - Annual intermediate CA rotation: `scripts/ca-rotate-intermediate.sh`
|
|
|
|
|
+ (from W1).
|
|
|
|
|
+- PromQL alerts (in `deploy/prometheus/alerts.yml`):
|
|
|
|
|
+ - `CertExpiringSoon` (severity warning): `cert_not_after -
|
|
|
|
|
+ time() < 30 * 86400` for any source cert.
|
|
|
|
|
+ - `CertExpiringCritical` (severity critical): `< 7 * 86400`.
|
|
|
|
|
+ - `CertExpired` (severity critical): `cert_not_after < time()`.
|
|
|
|
|
+ - `CertRevoked` (severity info): counter increment.
|
|
|
|
|
+ - `mTLSHandshakeErrorsHigh`: rate of failed handshakes > 1%
|
|
|
|
|
+ for 5 min.
|
|
|
|
|
+- Audit log:
|
|
|
|
|
+ - `cert.issue` — who, which source, cert serial.
|
|
|
|
|
+ - `cert.rotate` — auto-rotation, no actor.
|
|
|
|
|
+ - `cert.revoke` — who, which source, reason.
|
|
|
|
|
+ - `cert.expire` — auto, when cert is detected expired.
|
|
|
|
|
+- Runbook (`docs/runbooks/mtls-incident.md`):
|
|
|
|
|
+ - "Cert expired and source is down" — reissue flow.
|
|
|
|
|
+ - "Private key compromise" — revoke + reissue + audit.
|
|
|
|
|
+ - "CA compromise" — generate new root offline, re-issue
|
|
|
|
|
+ intermediate, re-issue all certs. Documented as M14.5
|
|
|
|
|
+ emergency procedure.
|
|
|
|
|
+ - "Handshake errors spiking" — likely clock skew or CA
|
|
|
|
|
+ pool issue. Check cert chain, check issuer.
|
|
|
|
|
+
|
|
|
|
|
+**Exit criteria:**
|
|
|
|
|
+- [ ] All 5 PromQL alerts fire in a test scenario (expire a
|
|
|
|
|
+ cert, revoke a cert, simulate clock skew).
|
|
|
|
|
+- [ ] Annual rotation script tested in a staging cluster.
|
|
|
|
|
+- [ ] Runbook reviewed by 2 people (you + me, or you + a
|
|
|
|
|
+ peer).
|
|
|
|
|
+- [ ] Audit log entries for issue, rotate, revoke, expire.
|
|
|
|
|
+
|
|
|
|
|
+**Estimated:** 2-3 days.
|
|
|
|
|
+
|
|
|
|
|
+---
|
|
|
|
|
+
|
|
|
|
|
+### W6: smoke E2E mTLS path + M14_VERIFICATION.md
|
|
|
|
|
+
|
|
|
|
|
+**Goal:** End-to-end smoke that exercises the full mTLS path:
|
|
|
|
|
+cert issuance → cert-bound source → ingestd validation → alert
|
|
|
|
|
+accepted → cert revocation → alert rejected.
|
|
|
|
|
+
|
|
|
|
|
+**Scope:**
|
|
|
|
|
+- `scripts/m14_smoke.sh`:
|
|
|
|
|
+ 1. Bring up stack with cert-manager + intermediate CA.
|
|
|
|
|
+ 2. Super-admin creates a company.
|
|
|
|
|
+ 3. Super-admin creates a source with `mtls_required=true`.
|
|
|
|
|
+ 4. Super-admin generates a cert for the source via API.
|
|
|
|
|
+ 5. `curl` with `--cert` and `--key` to ingestd using that
|
|
|
|
|
+ cert + HMAC → 200.
|
|
|
|
|
+ 6. `curl` without cert → 401.
|
|
|
|
|
+ 7. `curl` with expired cert → 401.
|
|
|
|
|
+ 8. `curl` with HMAC-only (cert stripped) → 401.
|
|
|
|
|
+ 9. Revoke the cert via API.
|
|
|
|
|
+ 10. `curl` with revoked cert → 401.
|
|
|
|
|
+ 11. Send a real alert via the working path → see it in
|
|
|
|
|
+ live tail (M13c).
|
|
|
|
|
+- `M14_VERIFICATION.md`:
|
|
|
|
|
+ - Smoke log.
|
|
|
|
|
+ - Cert chain diagram (root → intermediate → serving).
|
|
|
|
|
+ - Bundle size breakdown.
|
|
|
|
|
+ - The 5 PromQL alerts with sample output.
|
|
|
|
|
+ - Runbook link.
|
|
|
|
|
+ - Sign-off: "broad-announce v1 supports opt-in mTLS per
|
|
|
|
|
+ source, with cert-manager-managed rotation, lifecycle
|
|
|
|
|
+ UI, and incident response runbook."
|
|
|
|
|
+
|
|
|
|
|
+**Exit criteria:**
|
|
|
|
|
+- [ ] `make m14-smoke` exits 0 from a clean state.
|
|
|
|
|
+- [ ] 3 consecutive green runs.
|
|
|
|
|
+- [ ] M13 functionality not regressed.
|
|
|
|
|
+- [ ] M14_VERIFICATION.md published.
|
|
|
|
|
+
|
|
|
|
|
+**Estimated:** 1-2 days.
|
|
|
|
|
+
|
|
|
|
|
+---
|
|
|
|
|
+
|
|
|
|
|
+## 4. Sequencing & parallelism
|
|
|
|
|
+
|
|
|
|
|
+```
|
|
|
|
|
+W1 (3-4d) ──┬──▶ W2 (5-7d) ──┬──▶ W4 UI portion (3-5d, parallel with M13)
|
|
|
|
|
+ │ │
|
|
|
|
|
+ │ ├──▶ W3 (2-3d, parallel with W2)
|
|
|
|
|
+ │ │
|
|
|
|
|
+ │ └──▶ W5 (2-3d) ──▶ W6 (1-2d)
|
|
|
|
|
+ │
|
|
|
|
|
+ └── (W1 is the only hard blocker)
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+**With one engineer (sequential):**
|
|
|
|
|
+- W1: 3-4d
|
|
|
|
|
+- W2 + W3: 7-10d (can overlap, W3 short)
|
|
|
|
|
+- W4: 3-5d (after M13b lands)
|
|
|
|
|
+- W5: 2-3d
|
|
|
|
|
+- W6: 1-2d
|
|
|
|
|
+- **Total: 16-24 days**
|
|
|
|
|
+
|
|
|
|
|
+**With two engineers (W4 lands in M13c):**
|
|
|
|
|
+- Eng 1: W1 → W2 → W5 → W6 (9-12 days)
|
|
|
|
|
+- Eng 2: M13c + W4 (parallel with W2) → W3 → W5 (10-15 days)
|
|
|
|
|
+- **Total wall time: 10-15 days**
|
|
|
|
|
+
|
|
|
|
|
+---
|
|
|
|
|
+
|
|
|
|
|
+## 5. What M14 is NOT
|
|
|
|
|
+
|
|
|
|
|
+(Recorded so a future reader doesn't re-litigate.)
|
|
|
|
|
+
|
|
|
|
|
+- **Not a SOC2 certification.** M14 builds the technical controls.
|
|
|
|
|
+ SOC2 is an audit process (M15+).
|
|
|
|
|
+- **Not customer-issued public certs.** We sign with our internal
|
|
|
|
|
+ intermediate. Sources that need a public cert bring their own.
|
|
|
|
|
+- **Not Vault / KMS integration.** cert-manager secrets are K8s
|
|
|
|
|
+ Secrets (sealed-secrets or external-secrets encrypted at rest).
|
|
|
|
|
+ HashiCorp Vault + dynamic certs is v2.
|
|
|
|
|
+- **Not HSM-backed root.** The root is on an encrypted USB stick
|
|
|
|
|
+ (or air-gapped VM). HSM is v3.
|
|
|
|
|
+- **Not a full PKI replacement.** External CAs (DigiCert, Let's
|
|
|
|
|
+ Encrypt) are still used for any public-facing endpoints (the
|
|
|
|
|
+ broad-announce marketing site, the public docs). M14 is for
|
|
|
|
|
+ in-cluster + source-side certs.
|
|
|
|
|
+- **Not a redesign of the data plane.** ingestd, routerd, deliverd
|
|
|
|
|
+ keep their Go code. M14 layers TLS on top, doesn't rewrite.
|
|
|
|
|
+- **Not retroactive.** Existing sources with HMAC + API key
|
|
|
|
|
+ continue working. mTLS is opt-in per source. No flag day.
|
|
|
|
|
+
|
|
|
|
|
+---
|
|
|
|
|
+
|
|
|
|
|
+## 6. Dependencies
|
|
|
|
|
+
|
|
|
|
|
+| From | What | Status | Affects |
|
|
|
|
|
+|---|---|---|---|
|
|
|
|
|
+| M12 W1 | K8s manifests exist | ⏳ in flight | W1 (cert-manager deploy) |
|
|
|
|
|
+| M11 | gRPC infra stable | ✅ done | W3 (gRPC mTLS) |
|
|
|
|
|
+| M13a | authd + JWT gate | ⏳ not started | W4 (cert UI uses auth) |
|
|
|
|
|
+| M13b | Sources CRUD | ⏳ not started | W4 (cert UI is incremental on Sources) |
|
|
|
|
|
+| Postgres | `source_certs` table | to be created in W2 migration | W2 |
|
|
|
|
|
+| cert-manager | upstream chart | ✅ available | W1 |
|
|
|
|
|
+
|
|
|
|
|
+---
|
|
|
|
|
+
|
|
|
|
|
+## 7. Risks
|
|
|
|
|
+
|
|
|
|
|
+| Risk | Likelihood | Impact | Mitigation |
|
|
|
|
|
+|---|---|---|---|
|
|
|
|
|
+| Root CA compromise | Very low | Catastrophic | Root offline + encrypted; documented re-issue procedure (full cluster cert rotation) |
|
|
|
|
|
+| Cert rotation drops in-flight requests | Medium | High | PreStop hook + graceful drain + 30-day renewal window + W3 hot-reload |
|
|
|
|
|
+| Operator issues cert to wrong source | Medium | Medium | Cert bound to source_id in CN/SAN; UI shows source label before issuance |
|
|
|
|
|
+| `mtls_required=true` breaks an existing source | Medium | High | Opt-in per source; rollout plan: 1 source, 5 sources, all sources, default-on (3 quarters) |
|
|
|
|
|
+| `cert-manager` operator upgrade breaks Issuer | Low | High | Pin chart version; test upgrade in staging first |
|
|
|
|
|
+| CRL distribution lag → revoked cert still works | Medium | Medium | 60s SLA in W2; OCSP responder for real-time (v2) |
|
|
|
|
|
+| Sources can't generate CSRs (legacy systems) | Medium | Medium | "Auto-generate" button in UI: server generates key+CSR, returns bundle |
|
|
|
|
|
+
|
|
|
|
|
+---
|
|
|
|
|
+
|
|
|
|
|
+## 8. Definition of done — M14
|
|
|
|
|
+
|
|
|
|
|
+- [ ] All 6 workstreams have their per-workstream exit criteria
|
|
|
|
|
+ checked.
|
|
|
|
|
+- [ ] `make m14-smoke` green for 3 consecutive runs.
|
|
|
|
|
+- [ ] M13 functionality not regressed.
|
|
|
|
|
+- [ ] M14_VERIFICATION.md published.
|
|
|
|
|
+- [ ] Runbook reviewed by 2 people.
|
|
|
|
|
+- [ ] SPEC.md `M14` row flipped to **✅ shipped YYYY-MM-DD** with
|
|
|
|
|
+ the entry: *"opt-in mTLS per source + cert-manager-managed
|
|
|
|
|
+ rotation + cert lifecycle UI + internal gRPC mTLS + runbook.
|
|
|
|
|
+ Closes M11.5 wishlist. See M14_VERIFICATION.md for evidence."*
|
|
|
|
|
+
|
|
|
|
|
+---
|
|
|
|
|
+
|
|
|
|
|
+## 9. What M14 buys the business
|
|
|
|
|
+
|
|
|
|
|
+| Before (today) | After (M14) |
|
|
|
|
|
+|---|---|
|
|
|
|
|
+| "API key + HMAC" in marketing | "mTLS opt-in per source" in marketing |
|
|
|
|
|
+| Stolen API key = game over | Cert rotation stops the attack |
|
|
|
|
|
+| gRPC internal plaintext (TODO) | gRPC internal mTLS |
|
|
|
|
|
+| Manual cert management (if any) | cert-manager + UI |
|
|
|
|
|
+| "Enterprise" in pitch deck | "Enterprise" in feature matrix |
|
|
|
|
|
+| No compliance trail | Audit log for every cert event |
|
|
|
|
|
+| Not SOC2-ready | **SOC2-ready controls in place** (cert audit is one of the SOC2 controls) |
|
|
|
|
|
+
|
|
|
|
|
+**What it does NOT buy:** SOC2 itself, FedRAMP, HIPAA. Those are
|
|
|
|
|
+certifications, not features. M14 builds the technical controls
|
|
|
|
|
+that the auditors will check; getting the certification is a
|
|
|
|
|
+separate project (M15).
|
|
|
|
|
+
|
|
|
|
|
+---
|
|
|
|
|
+
|
|
|
|
|
+**Next step:** start W1 (cert-manager + CA). If we have 2
|
|
|
|
|
+engineers, W4 UI work starts as soon as M13b W1 (Companies CRUD)
|
|
|
|
|
+is done — the cert UI is a Sources incremental.
|