Inicio Explorar Axuda
Rexistro Iniciar sesión
lrosales
/
broad-announce
1
0
Fork 0
Ficheiros Incidencias 0 Pull Requests 0 Wiki
Árbore: 8ef784cfd4
Ramas Etiquetas
broad-announce
/
docs
/
runbooks
/
mtls-incident.md

mtls-incident.md 8.4 KB
Histórico Raw

mTLS Incident Response Runbook

What to do when mTLS breaks or is compromised. Read this BEFORE you have an incident. Covers the four most common scenarios.

Audience: operator on-call, super-admin. Scope: source-side mTLS, internal gRPC mTLS, cert rotation, CA compromise. Last reviewed: 2026-06-16 (initial draft, pre-M14-backend ship).

Quick reference

Symptom Likely cause Jump to
Source can't connect, 401 from ingestd cert expired Cert expired
401 from ingestd after a recent change cert revoked, or wrong CN/SAN Cert revoked
mTLSHandshakeErrorsHigh alert firing clock skew, missing CA, or wrong issuer Handshake errors spiking
Private key leaked (Slack DM, repo push, etc.) key compromise Private key compromise
Root CA key leaked (the worst day of your life) CA compromise CA compromise

Cert expired and source is down

Symptoms: source operator reports 401s. PromQL CertExpired alert fired. ingestd logs show tls: failed to verify client's certificate: x509: certificate has expired.

Cause: the source's leaf cert passed its notAfter. cert-manager auto-rotates serving certs (90d, renew at 30d), but source certs are issued on demand — no auto-rotation. Operator must re-issue.

Resolution:

  1. Find the source in the admin UI: Sources → <company> → <source> → Cert tab.
  2. If mtls_required=true and there's an existing cert: re-issue via "Auto-generate" or "Upload CSR". This creates a new leaf signed by the current intermediate.
  3. Download the new cert bundle (.zip with cert + chain + key).
  4. Send the bundle to the source operator out-of-band (1Password, secure file share, NOT email).
  5. Source operator installs the new cert + key on their side.
  6. Verify: source sends a test alert → 200.

If the source is one of OUR services (ingestd, routerd, deliverd-*): cert-manager handles this automatically. The 30-day renewal window means the cert is re-issued well before expiry. If you see a serving cert expire, check cert-manager logs and the Certificate CR status.

Cert revoked but source still trying to use it

Symptoms: source reports 401s after a recent revoke. PromQL CertRevoked alert fired. CRL may not have propagated yet.

Cause: revocation is propagated via CRL, which is checked by the verifier. The CRL refresh interval in cert-manager defaults to 600s (10 min). Within that window, a revoked cert may still validate.

Resolution:

  1. Wait 10 minutes. Most cases resolve themselves.
  2. If still 401 after 10 min: check the CRL distribution point in the cert. openssl x509 -in <cert> -noout -text | grep -A2 "CRL Distribution".
  3. If the CRL endpoint is unreachable: that's the bug. See Handshake errors spiking.
  4. If the cert really should be revoked but isn't: the source may have a cached cert in their HTTP client (some libraries cache). Source operator must restart their client.

Mitigation: the spec calls for OCSP (real-time) in v2, which removes the 10-min lag. For now, accept the lag.

Handshake errors spiking

Symptoms: PromQL mTLSHandshakeErrorsHigh > 1% for 5 min. ingestd logs show many tls: failed to verify client's certificate with various x509 errors.

Cause (most likely first):

  1. Clock skew between source and broad-announce servers. x509 validation requires now to be within notBefore–notAfter. Check NTP sync on both sides.
  2. Missing intermediate in trust store. ingestd's CA pool must include the intermediate, not just the root. If the pool is misconfigured, all source certs fail.
  3. Wrong issuer. Source sent a cert signed by a different CA (their own, or a stale intermediate from a rotation).
  4. Key usage mismatch. Source cert doesn't have clientAuth EKU. Generator issue.

Diagnosis:

# From a source host, test against ingestd
openssl s_client -connect ingestd:8443 -cert source.crt -key source.key -CAfile intermediate-ca.crt

# Look for "Verify return code: 0 (ok)" or the specific error
# Common: "certificate has expired", "unable to get local issuer
# certificate", "wrong issuer"

Resolution by cause:

Cause Fix
Clock skew Sync NTP. Verify chronyc tracking or ntpq -p on both sides.
Missing intermediate Re-import intermediate into K8s Secret. Restart ingestd.
Wrong issuer Re-issue source cert against current intermediate.
Key usage Re-generate CSR with extendedKeyUsage=clientAuth in the extfile.

Private key compromise

Symptoms: a source's private key was leaked (pushed to a public repo, stolen from a backup, exfiltrated). You may or may not have a specific incident — sometimes you find out later.

Severity: high. The attacker can impersonate the source until you revoke.

Immediate response (within 1 hour):

  1. Revoke the cert. Sources → <source> → Cert tab → Revoke. Confirm with typed slug. Within ~10 min, the CRL propagates and the cert stops validating.
  2. Audit. Check audit_log for cert.revoke events. Look for any other activity from the same actor or IP that might indicate broader compromise.
  3. Notify the source operator. They need to know the cert was compromised so they can find the leak.
  4. Re-issue. New CSR + new key from a clean machine. Send the new bundle out-of-band.
  5. Document. Add an entry to your incident log with timeline, who, what, why.

Within 24 hours:

  1. Review access logs for the source ID. Look for traffic from IPs/ASNs that don't match the source's normal pattern.
  2. Check for downstream impact. Were any alerts forged using the stolen identity? Audit alerts table for the source_id in the period between compromise and revocation.
  3. Rotate the source's HMAC secret too. If the attacker had access to the source machine, they may have the HMAC secret.

CA compromise

Symptoms: the root CA private key was leaked, or you have strong reason to believe the entire PKI is compromised. This is a catastrophic event — every cert in the system is suspect.

Severity: catastrophic. Treat as a P0.

Immediate response (within 1 hour):

  1. Page the team. This is a P0. Anyone with broad-announce context should be reachable.
  2. Disable mTLS at the edge. Edit the ClusterIssuer to temporarily reject all source certs (this is the fail-closed behavior). Every source falls back to HMAC + API key (which is still secure if those weren't also compromised).
  3. Generate a new root CA offline on a known-clean machine.
    • Different physical media. Air-gapped if possible.
    • New RSA 4096 (or ECDSA P-384) key.
    • New 10-year cert.
  4. Generate a new intermediate signed by the new root.
  5. Re-import the new intermediate into K8s (sealed-secrets).
  6. Re-issue ALL serving certs in the cluster (delete Certificate CRs, let cert-manager re-issue).

Within 24 hours:

  1. Re-issue every source cert. Every source operator must receive a new bundle. This is a massive coordination effort — plan to communicate over 1-2 weeks.
  2. Audit everything. Treat all certs issued under the old CA as untrusted. Look for any cert that was issued in the compromise window.
  3. Post-mortem. How did the root key leak? How can we prevent it next time? Update this runbook with lessons.

This is the worst day. Plan for it, but hopefully never have it.

Preventive measures

These reduce the chance of needing the runbook:

Measure Where
Root CA key on encrypted USB in a safe Physical security
Intermediate CA key in K8s sealed-secrets Encrypted at rest
Serving certs rotate 90d, renew 30d cert-manager config
Source certs rotate 90d (planned v1.1) PromQL alert at 30/7/1 days
mTLSHandshakeErrorsHigh alert at >1% for 5m PromQL
CertExpiringSoon alert at 30/7/1 days PromQL
CertExpired alert at expiry PromQL
Audit log on every cert issue/revoke/expire audit_log table
Quarterly rotation drill (rotate one env, verify) Manual
Annual root ceremony review Manual

Owner: super-admin team. Review this runbook quarterly.