# mTLS Incident Response Runbook

> What to do when mTLS breaks or is compromised. Read this BEFORE you
> have an incident. Covers the four most common scenarios.

**Audience:** operator on-call, super-admin.
**Scope:** source-side mTLS, internal gRPC mTLS, cert rotation, CA
compromise.
**Last reviewed:** 2026-06-16 (initial draft, pre-M14-backend ship).

---

## Quick reference

| Symptom | Likely cause | Jump to |
|---|---|---|
| Source can't connect, 401 from ingestd | cert expired | [Cert expired](#cert-expired-and-source-is-down) |
| 401 from ingestd after a recent change | cert revoked, or wrong CN/SAN | [Cert revoked](#cert-revoked-but-source-still-trying-to-use-it) |
| `mTLSHandshakeErrorsHigh` alert firing | clock skew, missing CA, or wrong issuer | [Handshake errors spiking](#handshake-errors-spiking) |
| Private key leaked (Slack DM, repo push, etc.) | key compromise | [Private key compromise](#private-key-compromise) |
| Root CA key leaked (the worst day of your life) | CA compromise | [CA compromise](#ca-compromise) |

---

## Cert expired and source is down

**Symptoms:** source operator reports 401s. PromQL `CertExpired`
alert fired. ingestd logs show `tls: failed to verify client's
certificate: x509: certificate has expired`.

**Cause:** the source's leaf cert passed its `notAfter`. cert-manager
auto-rotates serving certs (90d, renew at 30d), but **source certs
are issued on demand** — no auto-rotation. Operator must re-issue.

**Resolution:**

1. Find the source in the admin UI: Sources → `<company>` →
   `<source>` → Cert tab.
2. If `mtls_required=true` and there's an existing cert: re-issue
   via "Auto-generate" or "Upload CSR". This creates a new leaf signed
   by the current intermediate.
3. Download the new cert bundle (`.zip` with cert + chain + key).
4. Send the bundle to the source operator out-of-band (1Password,
   secure file share, NOT email).
5. Source operator installs the new cert + key on their side.
6. Verify: source sends a test alert → 200.

**If the source is one of OUR services** (ingestd, routerd,
deliverd-*): cert-manager handles this automatically. The 30-day
renewal window means the cert is re-issued well before expiry. If
you see a serving cert expire, check cert-manager logs and the
`Certificate` CR status.

---

## Cert revoked but source still trying to use it

**Symptoms:** source reports 401s after a recent revoke. PromQL
`CertRevoked` alert fired. CRL may not have propagated yet.

**Cause:** revocation is propagated via CRL, which is checked by the
verifier. The CRL refresh interval in cert-manager defaults to 600s
(10 min). Within that window, a revoked cert may still validate.

**Resolution:**

1. **Wait 10 minutes.** Most cases resolve themselves.
2. If still 401 after 10 min: check the CRL distribution point in
   the cert. `openssl x509 -in <cert> -noout -text | grep -A2
   "CRL Distribution"`.
3. If the CRL endpoint is unreachable: that's the bug. See
   [Handshake errors spiking](#handshake-errors-spiking).
4. If the cert really should be revoked but isn't: the source may
   have a cached cert in their HTTP client (some libraries cache).
   Source operator must restart their client.

**Mitigation:** the spec calls for OCSP (real-time) in v2, which
removes the 10-min lag. For now, accept the lag.

---

## Handshake errors spiking

**Symptoms:** PromQL `mTLSHandshakeErrorsHigh` > 1% for 5 min.
ingestd logs show many `tls: failed to verify client's certificate`
with various x509 errors.

**Cause (most likely first):**

1. **Clock skew** between source and broad-announce servers. x509
   validation requires `now` to be within `notBefore`–`notAfter`.
   Check NTP sync on both sides.
2. **Missing intermediate in trust store.** ingestd's CA pool must
   include the intermediate, not just the root. If the pool is
   misconfigured, all source certs fail.
3. **Wrong issuer.** Source sent a cert signed by a different CA
   (their own, or a stale intermediate from a rotation).
4. **Key usage mismatch.** Source cert doesn't have `clientAuth`
   EKU. Generator issue.

**Diagnosis:**

```bash
# From a source host, test against ingestd
openssl s_client -connect ingestd:8443 -cert source.crt -key source.key -CAfile intermediate-ca.crt

# Look for "Verify return code: 0 (ok)" or the specific error
# Common: "certificate has expired", "unable to get local issuer
# certificate", "wrong issuer"
```

**Resolution by cause:**

| Cause | Fix |
|---|---|
| Clock skew | Sync NTP. Verify `chronyc tracking` or `ntpq -p` on both sides. |
| Missing intermediate | Re-import intermediate into K8s Secret. Restart ingestd. |
| Wrong issuer | Re-issue source cert against current intermediate. |
| Key usage | Re-generate CSR with `extendedKeyUsage=clientAuth` in the extfile. |

---

## Private key compromise

**Symptoms:** a source's private key was leaked (pushed to a public
repo, stolen from a backup, exfiltrated). You may or may not have a
specific incident — sometimes you find out later.

**Severity:** high. The attacker can impersonate the source until
you revoke.

**Immediate response (within 1 hour):**

1. **Revoke the cert.** Sources → `<source>` → Cert tab → Revoke.
   Confirm with typed slug. Within ~10 min, the CRL propagates and
   the cert stops validating.
2. **Audit.** Check `audit_log` for `cert.revoke` events. Look for
   any other activity from the same actor or IP that might indicate
   broader compromise.
3. **Notify the source operator.** They need to know the cert was
   compromised so they can find the leak.
4. **Re-issue.** New CSR + new key from a clean machine. Send the
   new bundle out-of-band.
5. **Document.** Add an entry to your incident log with timeline,
   who, what, why.

**Within 24 hours:**

6. **Review access logs** for the source ID. Look for traffic from
   IPs/ASNs that don't match the source's normal pattern.
7. **Check for downstream impact.** Were any alerts forged using
   the stolen identity? Audit `alerts` table for the source_id in
   the period between compromise and revocation.
8. **Rotate the source's HMAC secret too.** If the attacker had
   access to the source machine, they may have the HMAC secret.

---

## CA compromise

**Symptoms:** the root CA private key was leaked, or you have strong
reason to believe the entire PKI is compromised. This is a
**catastrophic** event — every cert in the system is suspect.

**Severity:** catastrophic. Treat as a P0.

**Immediate response (within 1 hour):**

1. **Page the team.** This is a P0. Anyone with broad-announce
   context should be reachable.
2. **Disable mTLS at the edge.** Edit the ClusterIssuer to
   temporarily reject all source certs (this is the
   `fail-closed` behavior). Every source falls back to
   HMAC + API key (which is still secure if those weren't also
   compromised).
3. **Generate a new root CA offline** on a known-clean machine.
   - Different physical media. Air-gapped if possible.
   - New RSA 4096 (or ECDSA P-384) key.
   - New 10-year cert.
4. **Generate a new intermediate** signed by the new root.
5. **Re-import the new intermediate into K8s** (sealed-secrets).
6. **Re-issue ALL serving certs** in the cluster (delete
   `Certificate` CRs, let cert-manager re-issue).

**Within 24 hours:**

7. **Re-issue every source cert.** Every source operator must
   receive a new bundle. This is a massive coordination effort —
   plan to communicate over 1-2 weeks.
8. **Audit everything.** Treat all certs issued under the old CA
   as untrusted. Look for any cert that was issued in the
   compromise window.
9. **Post-mortem.** How did the root key leak? How can we prevent
   it next time? Update this runbook with lessons.

**This is the worst day. Plan for it, but hopefully never have it.**

---

## Preventive measures

These reduce the chance of needing the runbook:

| Measure | Where |
|---|---|
| Root CA key on encrypted USB in a safe | Physical security |
| Intermediate CA key in K8s sealed-secrets | Encrypted at rest |
| Serving certs rotate 90d, renew 30d | cert-manager config |
| Source certs rotate 90d (planned v1.1) | PromQL alert at 30/7/1 days |
| `mTLSHandshakeErrorsHigh` alert at >1% for 5m | PromQL |
| `CertExpiringSoon` alert at 30/7/1 days | PromQL |
| `CertExpired` alert at expiry | PromQL |
| Audit log on every cert issue/revoke/expire | `audit_log` table |
| Quarterly rotation drill (rotate one env, verify) | Manual |
| Annual root ceremony review | Manual |

---

**Owner:** super-admin team. Review this runbook quarterly.