M14 — Security Hardening Plan

Implements the security promises of SPEC §15 (mTLS) and closes the M11.5 wishlist (mTLS + secrets management + network policies). Makes broad-announce defensible to enterprise customers without re-architecting the data plane.

Status: planning (post-M11, parallel to M12/M13) Target: SPEC §15 (mTLS opt-in per source) + gRPC internal mTLS (today a TODO) + cert lifecycle UI + secrets rotation Estimate: 16-24 days with one engineer; 10-15 days with two engineers in parallel with M13c

0. Recap — what M14 must prove

Surface	Acceptance criteria
CA hierarchy	Internal CA (root + intermediate) provisioned via cert-manager, root offline, intermediate serves cluster + sources
Source-side mTLS	A source with `mtls_required=true` is rejected at ingestd if the client cert is missing, expired, signed by an untrusted CA, or has the wrong CN/SAN. HMAC + API key still work for non-mTLS sources.
Internal gRPC mTLS	`ingestd`, `routerd`, `deliverd-*` talk over mTLS with cert-manager-issued certs. The `TODO(m11)` in `internal/grpcserver/server.go:61` is closed.
Cert lifecycle UI	Super-admin can: list source certs, see expiration, revoke, generate a CSR (or auto-issue), download the cert bundle. Expiration < 30 days triggers a banner.
Rotation	Intermediate CA rotates annually. Certs rotate every 90 days. No downtime during rotation. Documented runbook.
PromQL alerts	`CertExpiringSoon` (>30d, < 7d, < 24h buckets), `CertRevoked`, `mTLSHandshakeErrorsHigh`
Audit log	Cert issue, rotate, revoke, expiration all produce audit rows.

What M14 is NOT:

Not a SOC2 / ISO 27001 certification. M14 is a prerequisite, not the certification. Compliance certification is M15+ (separate project, separate budget, separate auditor).
Not a key-vault replacement. M14 uses cert-manager secrets. HashiCorp Vault integration is v2.
Not customer-facing PKI. Sources that need mTLS get a cert signed by OUR intermediate CA. We don't issue public certs.
Not a redesign of auth. mTLS is layered on top of HMAC + API key in a "AND" relation: cert proves identity, HMAC proves payload integrity, API key is the rate-limit token. All three can be required for the strictest sources.

1. Threat model — what mTLS actually buys

Threat	Without mTLS	With mTLS (opt-in)
Stolen API key from a leaked source	Attacker forges alerts forever	Cert pinned to the source machine — rotate the source cert, attack stops
Replay of captured HMAC	Within 5-min window per Stripe-style	Cert auth is per-connection; replay needs both cert AND HMAC
Network MITM (compromised router)	TLS protects in transit, but anyone with the API key can call	Cert pinned; MITM can't produce a valid client cert without the private key
Insider abuse (operator with DB access)	Can read API keys	Certs are on the source machine, not in the DB
Credential stuffing	Common API keys, brute force	Each cert is unique; no shared secret to stuff

M14 doesn't fix: compromised source machine (attacker has the private key), compromised root CA (game over, but offline), social engineering of operator (cert issuance is gated by super-admin).

2. Workstream overview

┌──────────────────────────┐    ┌──────────────────────────┐
│  W1: cert-manager + CA   │    │  W2: ingestd mTLS        │
│  (root offline,          │───▶│  (opt-in per source,     │
│   intermediate in K8s)   │    │   cert validation,       │
│                          │    │   CN/SAN extraction)     │
└──────────────┬───────────┘    └──────────────┬───────────┘
               │                               │
               ▼                               ▼
┌──────────────────────────┐    ┌──────────────────────────┐
│  W3: gRPC internal mTLS  │    │  W4: cert lifecycle UI   │
│  (server + client,       │    │  (M13 frontend: list,    │
│   closes the TODO)       │    │   issue, rotate, revoke) │
└──────────────┬───────────┘    └──────────────┬───────────┘
               │                               │
               ▼                               ▼
┌──────────────────────────────────────────────────────────────┐
│  W5: rotation + PromQL alerts + audit + runbook              │
└──────────────────────────────────────────────────────────────┘
                                               │
                                               ▼
┌──────────────────────────────────────────────────────────────┐
│  W6: smoke E2E mTLS path + M14_VERIFICATION.md               │
└──────────────────────────────────────────────────────────────┘

Six workstreams. W1 is the only hard blocker. W4 (UI) can be split: the API changes land in M14, the UI lands in M13b/c as a feature incremental on top of the M13 sources module.

3. Workstream details

W1: cert-manager + CA hierarchy

Goal: Provision a working internal PKI in the K8s cluster. Root CA offline. Intermediate CA issues certs. cert-manager handles renewal.

Scope:

Choose CA tooling: step-ca or cfssl for the offline root, cert-manager for the intermediate and serving certs.
- step-ca if you want a single binary and JSON-based config.
- cfssl if you want the Cloudflare-style Go binary, lighter.
- cert-manager is the standard for K8s and gives you Certificate CRDs, Issuer/ClusterIssuer, renewal controller.
Root CA (offline):
- Generate root key + cert on an air-gapped machine or a passphrase-encrypted USB stick.
- 10-year validity. RSA 4096 or ECDSA P-384.
- Stored in a K8s Secret of type kubernetes.io/tls in a cert-manager namespace, with sealed-secrets or external-secrets (NOT plaintext git).
- Used ONLY to sign the intermediate. Never used to serve.
Intermediate CA (in-cluster):
- cert-manager ClusterIssuer backed by the root.
- 1-year validity, rotated annually (cron triggers re-issue).
- RSA 2048 or ECDSA P-256. ECDSA preferred (smaller, faster).
Serving certs (per-service):
- Certificate CRD per service (ingestd, routerd, deliverd-*, admind, authd).
- DNS SANs: <svc>.<namespace>.svc.cluster.local, plus any public DNS name.
- 90-day validity, auto-renewed at 30 days.
- Mounted as files via volumeMounts from a Secret.
cert-manager deployment:
- Helm chart, cert-manager v1.16+.
- RBAC for the controller.
- DNS-01 or HTTP-01 challenge for public certs (none needed for in-cluster).
Source cert issuance path (used by W2):
- Issuer of type CA (cert-manager) backed by the intermediate.
- Super-admin POSTs a CSR via API → cert-manager signs → cert bundle returned.
Tooling scripts:
- scripts/ca-init.sh — generates root + intermediate from a fresh machine. One-time use. Writes encrypted tarball.
- scripts/ca-rotate-intermediate.sh — annual cron. Issues new intermediate from root, swaps cert-manager's ClusterIssuer, re-issues all serving certs.

Exit criteria:

Root CA exists, offline, encrypted.
Intermediate CA exists in K8s, valid for 1 year.
cert-manager running, kubectl get clusterissuer shows broad-announce-intermediate Ready.
5 sample Certificate resources exist (ingestd, routerd, deliverd-fcm, deliverd-telegram, admind, authd) all Ready with valid serving certs.
Auto-renewal tested: manually expire a cert at 30 days remaining, watch it renew without downtime.
scripts/ca-init.sh and scripts/ca-rotate-intermediate.sh are documented in cmd/certmanager/README.md.

Estimated: 3-4 days.

W2: ingestd source-side mTLS (opt-in per source)

Goal: A source with mtls_required=true is rejected at ingestd if the client cert is missing, expired, signed by an untrusted CA, or has the wrong CN/SAN. HMAC + API key still work for non-mTLS sources.

Scope:

internal/auth/mtls.go (new package):
- ClientCertVerifier that wraps a *x509.CertPool (the intermediate) and a CN/SAN allowlist.
- Extracts r.TLS.PeerCertificates[0] from the request.
- Verifies: not expired, signed by intermediate, CN matches source.<source_id>.<company_slug> or SAN source:<source_id>, key usage ClientAuth.
- Returns 401 with a structured error if any check fails.
cmd/ingestd/main.go:
- Two listeners: :443 (TLS, with mTLS opt-in) and :80 (plaintext, redirect to :443 for mTLS-required sources).
- In practice, two HTTP servers on different ports. The mTLS one uses tls.Config{ClientAuth: tls.RequireAndVerifyClientCert, ClientCAs: pool}.
- When source.mtls_required=true, route through the TLS server. Else, plain HTTP (existing behavior).
- For sources with both HMAC and mTLS, both must succeed (AND, not OR).
internal/config/config.go:
- New env vars: BA_INGESTD_TLS_ADDR (default :8443), BA_INGESTD_TLS_CERT (path), BA_INGESTD_TLS_KEY (path), BA_INGESTD_TLS_CA (path to intermediate cert PEM).
- BA_INGESTD_REQUIRE_MTLS_DEFAULT (default false — controlled per source).
internal/grpcserver/server.go:
- Close the TODO(m11): grpc.Creds(tlsCredentials()) by wiring the cert paths from config.
- The gRPC server is internal-only (not exposed to sources), so the cert is the serving cert from W1.
Migrations: migrations/NNNN_add_source_cert_columns.sql:
- sources.mtls_required (already in schema, just wire it).
- sources.mtls_cn (CN expected, auto-derived from source_id).
- sources.cert_id (FK to a new source_certs table).
- source_certs(id, source_id, serial, not_before, not_after, revoked_at, cert_pem).
cmd/admind/main.go (new endpoints in M14):
- POST /v1/sources/:id/cert/csr — accept a CSR (PEM), sign it with the intermediate, return the cert bundle.
- GET /v1/sources/:id/cert — current cert + chain.
- DELETE /v1/sources/:id/cert — revoke.
- GET /v1/certs/expiring?days=30 — list of certs expiring soon.
New audit log events: cert.issue, cert.revoke, cert.expire_soon.

Exit criteria:

source.mtls_required=true → ingestd rejects requests without a valid client cert (curl with --cert succeeds, curl without fails with 401).
Cert signed by an untrusted CA → 401.
Expired cert → 401.
Wrong CN/SAN → 401.
Cert revoked via DELETE /v1/sources/:id/cert → 401 within 60s (cert-manager CRL updates).
source.mtls_required=false → existing HMAC + API key flow unchanged.
Both required (mTLS + HMAC) → both must succeed.
Internal gRPC between ingestd, routerd, deliverd uses mTLS (W3) — verified with tcpdump or by kubescape/kube-bench.

Estimated: 5-7 days.

W3: gRPC internal mTLS (server + client)

Goal: Close the TODO(m11) in internal/grpcserver/server.go:61 and the analogous client path. All service-to-service gRPC (and HTTP/2 streams) use mTLS with cert-manager-issued certs.

Scope:

internal/grpcserver/server.go:
- Read BA_GRPC_TLS_CERT, BA_GRPC_TLS_KEY, BA_GRPC_TLS_CA from env.
- grpc.Creds(credentials.NewServerTLSFromCert(&cert)) — or the mTLS variant for client certs (if any service authenticates the other).
- Hot-reload: SIGHUP or file watcher on the cert files. cert-manager renews → kubelet updates the Secret → file changes → service reloads without restart.
internal/grpcclient/client.go and options.go:
- Wire the TransportCredentials from the same env vars.
- For mTLS to other services: load the cert-manager-issued client cert + CA.
internal/grpcclient/options.go:44:
- The "Production deployments should use mTLS or at minimum TLS" comment becomes code.
deploy/k8s/base/*/deployment.yaml:
- Mount the cert-manager Secret as a volume at /etc/broad-announce/tls/.
- lifecycle.preStop hook to drain in-flight requests gracefully before pod termination (avoid 5xx during cert rotation).
internal/observability/metrics.go:
- ba_grpc_tls_handshakes_total{result=ok|error,peer=...}.
- ba_grpc_tls_cert_age_seconds{service=...} (gauge).

Exit criteria:

Internal gRPC between any two services uses TLS.
tcpdump shows TLS handshake (not plaintext) for a sample gRPC call between ingestd and routerd.
Cert rotation (manually expire a cert) doesn't drop a single in-flight request (verified with a load test).
Hot-reload works: SIGHUP picks up the new cert without a pod restart.
Metrics exported for handshake success/failure.

Estimated: 2-3 days.

W4: cert lifecycle UI (M13 frontend)

Goal: Super-admin can manage source certs from the M13 UI. CSR upload, cert download, revocation, expiration warnings.

Scope:

Routes (added to M13b W2 — Sources module):
- /sources/:id/cert — current cert info, expiration, status.
- Modal: "Generate CSR" (paste a CSR, get a signed cert back) or "Auto-generate" (server generates a key + CSR, signs it, returns the cert bundle as a downloadable .zip).
- Modal: "Revoke" with typed confirmation.
Component: CertStatusBadge — green/amber/red based on expiration.
Component: CertExpirationBanner — global banner in the top bar when any source cert is < 30 days from expiry. Clicks navigate to the cert list.
New endpoints in cmd/admind:
- POST /v1/sources/:id/cert/csr (from W2).
- GET /v1/sources/:id/cert.
- DELETE /v1/sources/:id/cert.
- GET /v1/certs/expiring?days=30.
Bundle: cert-management code-split (< 15 KB gzipped).

Exit criteria:

Super-admin can upload a CSR and download a signed cert.
Super-admin can use "Auto-generate" and get a .zip with cert + chain + private key (one-time download).
Revoke button removes the cert; subsequent requests with that cert fail within 60s.
Expiration banner appears 30 days before any cert expires.
Tenant-admin sees cert info for their sources (read-only, no revoke).
Bundle size < 15 KB gzipped.

Estimated: 3-5 days (in parallel with M13b, lands in M13c W7).

W5: rotation + PromQL alerts + audit + runbook

Goal: Automated cert rotation. PromQL alerts for expiration and revocation. Audit log for every cert event. Runbook for incident response.

Scope:

Rotation:
- cert-manager handles serving certs automatically (90-day, renew at 30).
- cert-manager.io/renew-before: 720h annotation.
- Source certs: 90-day validity, super-admin is notified 30/7/1 days before via the banner.
- Annual intermediate CA rotation: scripts/ca-rotate-intermediate.sh (from W1).
PromQL alerts (in deploy/prometheus/alerts.yml):
- CertExpiringSoon (severity warning): cert_not_after - time() < 30 * 86400 for any source cert.
- CertExpiringCritical (severity critical): < 7 * 86400.
- CertExpired (severity critical): cert_not_after < time().
- CertRevoked (severity info): counter increment.
- mTLSHandshakeErrorsHigh: rate of failed handshakes > 1% for 5 min.
Audit log:
- cert.issue — who, which source, cert serial.
- cert.rotate — auto-rotation, no actor.
- cert.revoke — who, which source, reason.
- cert.expire — auto, when cert is detected expired.
Runbook (docs/runbooks/mtls-incident.md):
- "Cert expired and source is down" — reissue flow.
- "Private key compromise" — revoke + reissue + audit.
- "CA compromise" — generate new root offline, re-issue intermediate, re-issue all certs. Documented as M14.5 emergency procedure.
- "Handshake errors spiking" — likely clock skew or CA pool issue. Check cert chain, check issuer.

Exit criteria:

All 5 PromQL alerts fire in a test scenario (expire a cert, revoke a cert, simulate clock skew).
Annual rotation script tested in a staging cluster.
Runbook reviewed by 2 people (you + me, or you + a peer).
Audit log entries for issue, rotate, revoke, expire.

Estimated: 2-3 days.

W6: smoke E2E mTLS path + M14_VERIFICATION.md

Goal: End-to-end smoke that exercises the full mTLS path: cert issuance → cert-bound source → ingestd validation → alert accepted → cert revocation → alert rejected.

Scope:

scripts/m14_smoke.sh:
1. Bring up stack with cert-manager + intermediate CA.
2. Super-admin creates a company.
3. Super-admin creates a source with mtls_required=true.
4. Super-admin generates a cert for the source via API.
5. curl with --cert and --key to ingestd using that cert + HMAC → 200.
6. curl without cert → 401.
7. curl with expired cert → 401.
8. curl with HMAC-only (cert stripped) → 401.
9. Revoke the cert via API.
10. curl with revoked cert → 401.
11. Send a real alert via the working path → see it in live tail (M13c).
M14_VERIFICATION.md:
- Smoke log.
- Cert chain diagram (root → intermediate → serving).
- Bundle size breakdown.
- The 5 PromQL alerts with sample output.
- Runbook link.
- Sign-off: "broad-announce v1 supports opt-in mTLS per source, with cert-manager-managed rotation, lifecycle UI, and incident response runbook."

Exit criteria:

make m14-smoke exits 0 from a clean state.
3 consecutive green runs.
M13 functionality not regressed.
M14_VERIFICATION.md published.

Estimated: 1-2 days.

4. Sequencing & parallelism

W1 (3-4d) ──┬──▶ W2 (5-7d) ──┬──▶ W4 UI portion (3-5d, parallel with M13)
            │                 │
            │                 ├──▶ W3 (2-3d, parallel with W2)
            │                 │
            │                 └──▶ W5 (2-3d) ──▶ W6 (1-2d)
            │
            └── (W1 is the only hard blocker)

With one engineer (sequential):

W1: 3-4d
W2 + W3: 7-10d (can overlap, W3 short)
W4: 3-5d (after M13b lands)
W5: 2-3d
W6: 1-2d
Total: 16-24 days

With two engineers (W4 lands in M13c):

Eng 1: W1 → W2 → W5 → W6 (9-12 days)
Eng 2: M13c + W4 (parallel with W2) → W3 → W5 (10-15 days)
Total wall time: 10-15 days

5. What M14 is NOT

(Recorded so a future reader doesn't re-litigate.)

Not a SOC2 certification. M14 builds the technical controls. SOC2 is an audit process (M15+).
Not customer-issued public certs. We sign with our internal intermediate. Sources that need a public cert bring their own.
Not Vault / KMS integration. cert-manager secrets are K8s Secrets (sealed-secrets or external-secrets encrypted at rest). HashiCorp Vault + dynamic certs is v2.
Not HSM-backed root. The root is on an encrypted USB stick (or air-gapped VM). HSM is v3.
Not a full PKI replacement. External CAs (DigiCert, Let's Encrypt) are still used for any public-facing endpoints (the broad-announce marketing site, the public docs). M14 is for in-cluster + source-side certs.
Not a redesign of the data plane. ingestd, routerd, deliverd keep their Go code. M14 layers TLS on top, doesn't rewrite.
Not retroactive. Existing sources with HMAC + API key continue working. mTLS is opt-in per source. No flag day.

6. Dependencies

From	What	Status	Affects
M12 W1	K8s manifests exist	⏳ in flight	W1 (cert-manager deploy)
M11	gRPC infra stable	✅ done	W3 (gRPC mTLS)
M13a	authd + JWT gate	⏳ not started	W4 (cert UI uses auth)
M13b	Sources CRUD	⏳ not started	W4 (cert UI is incremental on Sources)
Postgres	`source_certs` table	to be created in W2 migration	W2
cert-manager	upstream chart	✅ available	W1

7. Risks

Risk	Likelihood	Impact	Mitigation
Root CA compromise	Very low	Catastrophic	Root offline + encrypted; documented re-issue procedure (full cluster cert rotation)
Cert rotation drops in-flight requests	Medium	High	PreStop hook + graceful drain + 30-day renewal window + W3 hot-reload
Operator issues cert to wrong source	Medium	Medium	Cert bound to source_id in CN/SAN; UI shows source label before issuance
`mtls_required=true` breaks an existing source	Medium	High	Opt-in per source; rollout plan: 1 source, 5 sources, all sources, default-on (3 quarters)
`cert-manager` operator upgrade breaks Issuer	Low	High	Pin chart version; test upgrade in staging first
CRL distribution lag → revoked cert still works	Medium	Medium	60s SLA in W2; OCSP responder for real-time (v2)
Sources can't generate CSRs (legacy systems)	Medium	Medium	"Auto-generate" button in UI: server generates key+CSR, returns bundle

8. Definition of done — M14

All 6 workstreams have their per-workstream exit criteria checked.
make m14-smoke green for 3 consecutive runs.
M13 functionality not regressed.
M14_VERIFICATION.md published.
Runbook reviewed by 2 people.
SPEC.md M14 row flipped to ✅ shipped YYYY-MM-DD with the entry: "opt-in mTLS per source + cert-manager-managed rotation + cert lifecycle UI + internal gRPC mTLS + runbook. Closes M11.5 wishlist. See M14_VERIFICATION.md for evidence."

9. What M14 buys the business

Before (today)	After (M14)
"API key + HMAC" in marketing	"mTLS opt-in per source" in marketing
Stolen API key = game over	Cert rotation stops the attack
gRPC internal plaintext (TODO)	gRPC internal mTLS
Manual cert management (if any)	cert-manager + UI
"Enterprise" in pitch deck	"Enterprise" in feature matrix
No compliance trail	Audit log for every cert event
Not SOC2-ready	SOC2-ready controls in place (cert audit is one of the SOC2 controls)

What it does NOT buy: SOC2 itself, FedRAMP, HIPAA. Those are certifications, not features. M14 builds the technical controls that the auditors will check; getting the certification is a separate project (M15).

Next step: start W1 (cert-manager + CA). If we have 2 engineers, W4 UI work starts as soon as M13b W1 (Companies CRUD) is done — the cert UI is a Sources incremental.

M14_SECURITY_PLAN.md 23 KB История Исходник

M14 — Security Hardening Plan

0. Recap — what M14 must prove

1. Threat model — what mTLS actually buys

2. Workstream overview

3. Workstream details

W1: cert-manager + CA hierarchy

W2: ingestd source-side mTLS (opt-in per source)

W3: gRPC internal mTLS (server + client)

W4: cert lifecycle UI (M13 frontend)

W5: rotation + PromQL alerts + audit + runbook

W6: smoke E2E mTLS path + M14_VERIFICATION.md

4. Sequencing & parallelism

5. What M14 is NOT

6. Dependencies

7. Risks

8. Definition of done — M14

9. What M14 buys the business

M14_SECURITY_PLAN.md 23 KB

История Исходник