|
|
@@ -0,0 +1,231 @@
|
|
|
+# M2 Verification — Recipient Resolution
|
|
|
+
|
|
|
+This is the live smoke test for the M2 milestone
|
|
|
+(**Recipient resolution**, SPEC §6). It exercises the rules
|
|
|
+engine that replaces M1's broadcast with:
|
|
|
+
|
|
|
+1. `source.allowed_targets` (groups + individuals)
|
|
|
+2. `routing_rules` with `match_expr` overrides
|
|
|
+3. `subscriptions` with `min_severity`, `channel_mask`, and
|
|
|
+ `quiet_hours`
|
|
|
+4. `inminent_colapse` bypasses quiet hours
|
|
|
+
|
|
|
+Hard-fail (your Q2 answer): if the resolver returns zero targets,
|
|
|
+the alert is dropped and logged. No silent broadcast.
|
|
|
+
|
|
|
+## Prerequisites
|
|
|
+
|
|
|
+- Stack is up: `docker compose up -d`
|
|
|
+- Seed has been run: `docker compose up seed` (apply M2
|
|
|
+ migration + both seeds)
|
|
|
+- HMAC secret for the `acme-001:prom-prod` source is `s3cret-acme`
|
|
|
+ (matches `BA_INGESTD_SOURCES` env)
|
|
|
+
|
|
|
+## Seed summary
|
|
|
+
|
|
|
+| individual | subscription (severity, quiet hours) | notes |
|
|
|
+|---|---|---|
|
|
|
+| ind-acme-001 (Alice) | NULL, no quiet hours | gets every alert from `prom-prod` |
|
|
|
+| ind-acme-002 (Bob) | `critical`, no quiet hours | only `critical` and above |
|
|
|
+| ind-acme-003 (Carol) | NULL, **00:00–23:59 UTC** | everything except `inminent_colapse` |
|
|
|
+
|
|
|
+Groups: `sre` = Alice + Bob.
|
|
|
+
|
|
|
+Source `acme-001:prom-prod` `allowed_targets`:
|
|
|
+- `group:sre`
|
|
|
+- `individual:ind-acme-003`
|
|
|
+
|
|
|
+Routing rule: `category=storage` + `data.host=db-prod-03` → `individual:ind-acme-002` (Bob), priority 10.
|
|
|
+
|
|
|
+## Step-by-step
|
|
|
+
|
|
|
+### Step 1 — confirm M2 seed applied
|
|
|
+
|
|
|
+```bash
|
|
|
+docker compose exec postgres psql -U ba -d ba -c "
|
|
|
+SELECT i.id, i.full_name,
|
|
|
+ s.min_severity, s.quiet_hours_start, s.quiet_hours_end, s.tz
|
|
|
+FROM individuals i
|
|
|
+LEFT JOIN subscriptions s
|
|
|
+ ON s.individual_id = i.id AND s.source_id = 'prom-prod'
|
|
|
+ORDER BY i.id;
|
|
|
+"
|
|
|
+```
|
|
|
+
|
|
|
+Expected: 3 rows. Carol's `quiet_hours_start=00:00:00` and `quiet_hours_end=23:59:00`.
|
|
|
+
|
|
|
+### Step 2 — send a `warning` storage alert
|
|
|
+
|
|
|
+```bash
|
|
|
+SECRET=s3cret-acme
|
|
|
+BODY='{"company_id":"acme-001","source_id":"prom-prod","severity":"warning","category":"storage","title":"Disk 80% full","body":"db-prod-04","data":{"host":"db-prod-04"},"dedupe_key":"m2-s2"}'
|
|
|
+TS=$(date +%s)
|
|
|
+SIG=$(printf '%s.%s' "$TS" "$BODY" | openssl dgst -sha256 -hmac "$SECRET" -hex | awk '{print $2}')
|
|
|
+
|
|
|
+curl -s -X POST http://localhost:8800/v1/ingest \
|
|
|
+ -H "Content-Type: application/json" \
|
|
|
+ -H "X-BA-Signature: t=$TS,v1=$SIG" \
|
|
|
+ --data "$BODY"
|
|
|
+```
|
|
|
+
|
|
|
+Wait 2s, then check `deliveries`:
|
|
|
+
|
|
|
+```bash
|
|
|
+docker compose exec postgres psql -U ba -d ba -c "
|
|
|
+SELECT alert_id, individual_id, channel, status FROM deliveries
|
|
|
+WHERE alert_id IN (SELECT id FROM deliveries WHERE last_error IS NULL ORDER BY id DESC LIMIT 1)
|
|
|
+ORDER BY individual_id;
|
|
|
+"
|
|
|
+```
|
|
|
+
|
|
|
+**Expected recipients:** Alice only.
|
|
|
+
|
|
|
+Why:
|
|
|
+- Alice: severity=warning ≥ no min → in. (allowed via group:sre, in subscriptions)
|
|
|
+- Bob: min_severity=critical, alert is warning → out.
|
|
|
+- Carol: quiet hours cover the whole day → out (warning is not inminent_colapse).
|
|
|
+- Routing rule doesn't match (host is db-prod-04, not db-prod-03).
|
|
|
+
|
|
|
+### Step 3 — send a `critical` storage alert for `db-prod-04`
|
|
|
+
|
|
|
+```bash
|
|
|
+SECRET=s3cret-acme
|
|
|
+BODY='{"company_id":"acme-001","source_id":"prom-prod","severity":"critical","category":"storage","title":"Disk full","body":"db-prod-04","data":{"host":"db-prod-04"},"dedupe_key":"m2-s3"}'
|
|
|
+TS=$(date +%s)
|
|
|
+SIG=$(printf '%s.%s' "$TS" "$BODY" | openssl dgst -sha256 -hmac "$SECRET" -hex | awk '{print $2}')
|
|
|
+
|
|
|
+curl -s -X POST http://localhost:8800/v1/ingest \
|
|
|
+ -H "Content-Type: application/json" \
|
|
|
+ -H "X-BA-Signature: t=$TS,v1=$SIG" \
|
|
|
+ --data "$BODY"
|
|
|
+```
|
|
|
+
|
|
|
+Wait 2s, check deliveries.
|
|
|
+
|
|
|
+**Expected recipients:** Alice + Bob.
|
|
|
+
|
|
|
+Why:
|
|
|
+- Alice: in (no min).
|
|
|
+- Bob: critical meets his min_severity=critical → in.
|
|
|
+- Carol: critical, not inminent_colapse → still in quiet hours → out.
|
|
|
+
|
|
|
+### Step 4 — send a `critical` storage alert for `db-prod-03` (rule match)
|
|
|
+
|
|
|
+```bash
|
|
|
+SECRET=s3cret-acme
|
|
|
+BODY='{"company_id":"acme-001","source_id":"prom-prod","severity":"critical","category":"storage","title":"Disk full","body":"db-prod-03","data":{"host":"db-prod-03"},"dedupe_key":"m2-s4"}'
|
|
|
+TS=$(date +%s)
|
|
|
+SIG=$(printf '%s.%s' "$TS" "$BODY" | openssl dgst -sha256 -hmac "$SECRET" -hex | awk '{print $2}')
|
|
|
+
|
|
|
+curl -s -X POST http://localhost:8800/v1/ingest \
|
|
|
+ -H "Content-Type: application/json" \
|
|
|
+ -H "X-BA-Signature: t=$TS,v1=$SIG" \
|
|
|
+ --data "$BODY"
|
|
|
+```
|
|
|
+
|
|
|
+**Expected recipients:** Alice + Bob (rule doesn't add anyone new — Bob is already in via group+severity).
|
|
|
+
|
|
|
+This step proves the rule doesn't double-route. The SQL UNION just dedupes the candidate set.
|
|
|
+
|
|
|
+### Step 5 — send a `warning` alert that ONLY matches the rule (and would normally skip Bob)
|
|
|
+
|
|
|
+This is the rule-fires scenario. Add a new individual that has min_severity=warning (Alice does). But the rule points at Bob specifically, and Bob has min_severity=critical. So a `warning` alert matching the rule goes to Bob? **No** — per the algorithm, subscriptions still apply on top of rules. So a `warning` storage alert for db-prod-03 should go to Alice only (Bob's subscription filters it out).
|
|
|
+
|
|
|
+```bash
|
|
|
+SECRET=s3cret-acme
|
|
|
+BODY='{"company_id":"acme-001","source_id":"prom-prod","severity":"warning","category":"storage","title":"Disk 80% on prod-03","body":"pre-empt","data":{"host":"db-prod-03"},"dedupe_key":"m2-s5"}'
|
|
|
+TS=$(date +%s)
|
|
|
+SIG=$(printf '%s.%s' "$TS" "$BODY" | openssl dgst -sha256 -hmac "$SECRET" -hex | awk '{print $2}')
|
|
|
+
|
|
|
+curl -s -X POST http://localhost:8800/v1/ingest \
|
|
|
+ -H "Content-Type: application/json" \
|
|
|
+ -H "X-BA-Signature: t=$TS,v1=$SIG" \
|
|
|
+ --data "$BODY"
|
|
|
+```
|
|
|
+
|
|
|
+**Expected recipients:** Alice only. Rule matched Bob, but his min_severity filter (critical) drops warning.
|
|
|
+
|
|
|
+### Step 6 — send an `inminent_colapse` alert for any host
|
|
|
+
|
|
|
+This is the quiet-hours bypass test. Carol is in quiet hours; an `inminent_colapse` should reach her anyway.
|
|
|
+
|
|
|
+```bash
|
|
|
+SECRET=s3cret-acme
|
|
|
+BODY='{"company_id":"acme-001","source_id":"prom-prod","severity":"inminent_colapse","category":"power","title":"PDU overload imminent","body":"rack-B","data":{"host":"rack-b"},"dedupe_key":"m2-s6"}'
|
|
|
+TS=$(date +%s)
|
|
|
+SIG=$(printf '%s.%s' "$TS" "$BODY" | openssl dgst -sha256 -hmac "$SECRET" -hex | awk '{print $2}')
|
|
|
+
|
|
|
+curl -s -X POST http://localhost:8800/v1/ingest \
|
|
|
+ -H "Content-Type: application/json" \
|
|
|
+ -H "X-BA-Signature: t=$TS,v1=$SIG" \
|
|
|
+ --data "$BODY"
|
|
|
+```
|
|
|
+
|
|
|
+**Expected recipients:** Alice + Carol (severity bypasses Carol's quiet hours; Bob's min_severity=critical, inminent_colapse=rank 3 ≥ critical=rank 2 → in).
|
|
|
+
|
|
|
+Wait — actually Bob's min_severity=critical, and inminent_colapse rank 3 is > critical rank 2, so Bob IS in. Let me re-check:
|
|
|
+
|
|
|
+| individual | in? | why |
|
|
|
+|---|---|---|
|
|
|
+| Alice | in | no min |
|
|
|
+| Bob | in | min=critical, inminent_colapse > critical |
|
|
|
+| Carol | in | quiet hours bypassed for inminent_colapse |
|
|
|
+
|
|
|
+**Expected:** Alice + Bob + Carol. All three.
|
|
|
+
|
|
|
+### Step 7 — alert with no matching recipients (hard-fail)
|
|
|
+
|
|
|
+Send a `warning` alert via a source that has a valid HMAC secret (so
|
|
|
+ingestd doesn't 401) but no subscriptions and no routing-rule
|
|
|
+matches. The seed registers `acme-001:prom-prod` only, so the easiest
|
|
|
+is to add a second source to `BA_INGESTD_SOURCES` in compose, but
|
|
|
+that requires a restart. The cheaper path: temporarily revoke all
|
|
|
+subscriptions in psql, then send an alert, then restore.
|
|
|
+
|
|
|
+```bash
|
|
|
+# revoke
|
|
|
+docker compose exec postgres psql -U ba -d ba -c "
|
|
|
+UPDATE subscriptions SET status = 'paused' WHERE company_id = 'acme-001';
|
|
|
+"
|
|
|
+# wait a moment so the next request re-resolves
|
|
|
+sleep 1
|
|
|
+# send
|
|
|
+SECRET=s3cret-acme
|
|
|
+BODY='{"company_id":"acme-001","source_id":"prom-prod","severity":"info","category":"storage","title":"no recipients","data":{"host":"db-prod-04"},"dedupe_key":"m2-s7"}'
|
|
|
+TS=$(date +%s)
|
|
|
+SIG=$(printf '%s.%s' "$TS" "$BODY" | openssl dgst -sha256 -hmac "$SECRET" -hex | awk '{print $2}')
|
|
|
+
|
|
|
+curl -s -X POST http://localhost:8800/v1/ingest \
|
|
|
+ -H "Content-Type: application/json" \
|
|
|
+ -H "X-BA-Signature: t=$TS,v1=$SIG" \
|
|
|
+ --data "$BODY"
|
|
|
+```
|
|
|
+
|
|
|
+**Expected:** 202 from ingestd, then routerd logs
|
|
|
+`"zero recipients, dropping"`. **No** delivery row for this alert.
|
|
|
+
|
|
|
+```bash
|
|
|
+# verify
|
|
|
+docker compose logs --no-color --since 1m routerd 2>&1 | grep -E "zero recipients" | tail -3
|
|
|
+
|
|
|
+# restore
|
|
|
+docker compose exec postgres psql -U ba -d ba -c "
|
|
|
+UPDATE subscriptions SET status = 'active' WHERE company_id = 'acme-001';
|
|
|
+"
|
|
|
+```
|
|
|
+
|
|
|
+### Step 8 — summary
|
|
|
+
|
|
|
+After all steps, run:
|
|
|
+
|
|
|
+```bash
|
|
|
+docker compose exec postgres psql -U ba -d ba -c "
|
|
|
+SELECT alert_id, individual_id, status
|
|
|
+FROM deliveries
|
|
|
+WHERE alert_id LIKE '0000%'
|
|
|
+ORDER BY id DESC
|
|
|
+LIMIT 15;
|
|
|
+"
|
|
|
+```
|
|
|
+
|
|
|
+Each step's expected recipient set is in the per-step notes above. The total across steps 2–6 should be 1+2+2+1+3 = **9 deliveries** (assuming 0 dedupes within the 60s window).
|
Luis Rosales