# M2 Verification — Recipient Resolution This is the live smoke test for the M2 milestone (**Recipient resolution**, SPEC §6). It exercises the rules engine that replaces M1's broadcast with: 1. `source.allowed_targets` (groups + individuals) 2. `routing_rules` with `match_expr` overrides 3. `subscriptions` with `min_severity`, `channel_mask`, and `quiet_hours` 4. `inminent_colapse` bypasses quiet hours Hard-fail (your Q2 answer): if the resolver returns zero targets, the alert is dropped and logged. No silent broadcast. ## Prerequisites - Stack is up: `docker compose up -d` - Seed has been run: `docker compose up seed` (apply M2 migration + both seeds) - HMAC secret for the `acme-001:prom-prod` source is `s3cret-acme` (matches `BA_INGESTD_SOURCES` env) ## Seed summary | individual | subscription (severity, quiet hours) | notes | |---|---|---| | ind-acme-001 (Alice) | NULL, no quiet hours | gets every alert from `prom-prod` | | ind-acme-002 (Bob) | `critical`, no quiet hours | only `critical` and above | | ind-acme-003 (Carol) | NULL, **00:00–23:59 UTC** | everything except `inminent_colapse` | Groups: `sre` = Alice + Bob. Source `acme-001:prom-prod` `allowed_targets`: - `group:sre` - `individual:ind-acme-003` Routing rule: `category=storage` + `data.host=db-prod-03` → `individual:ind-acme-002` (Bob), priority 10. ## Step-by-step ### Step 1 — confirm M2 seed applied ```bash docker compose exec postgres psql -U ba -d ba -c " SELECT i.id, i.full_name, s.min_severity, s.quiet_hours_start, s.quiet_hours_end, s.tz FROM individuals i LEFT JOIN subscriptions s ON s.individual_id = i.id AND s.source_id = 'prom-prod' ORDER BY i.id; " ``` Expected: 3 rows. Carol's `quiet_hours_start=00:00:00` and `quiet_hours_end=23:59:00`. ### Step 2 — send a `warning` storage alert ```bash SECRET=s3cret-acme BODY='{"company_id":"acme-001","source_id":"prom-prod","severity":"warning","category":"storage","title":"Disk 80% full","body":"db-prod-04","data":{"host":"db-prod-04"},"dedupe_key":"m2-s2"}' TS=$(date +%s) SIG=$(printf '%s.%s' "$TS" "$BODY" | openssl dgst -sha256 -hmac "$SECRET" -hex | awk '{print $2}') curl -s -X POST http://localhost:8800/v1/ingest \ -H "Content-Type: application/json" \ -H "X-BA-Signature: t=$TS,v1=$SIG" \ --data "$BODY" ``` Wait 2s, then check `deliveries`: ```bash docker compose exec postgres psql -U ba -d ba -c " SELECT alert_id, individual_id, channel, status FROM deliveries WHERE alert_id IN (SELECT id FROM deliveries WHERE last_error IS NULL ORDER BY id DESC LIMIT 1) ORDER BY individual_id; " ``` **Expected recipients:** Alice only. Why: - Alice: severity=warning ≥ no min → in. (allowed via group:sre, in subscriptions) - Bob: min_severity=critical, alert is warning → out. - Carol: quiet hours cover the whole day → out (warning is not inminent_colapse). - Routing rule doesn't match (host is db-prod-04, not db-prod-03). ### Step 3 — send a `critical` storage alert for `db-prod-04` ```bash SECRET=s3cret-acme BODY='{"company_id":"acme-001","source_id":"prom-prod","severity":"critical","category":"storage","title":"Disk full","body":"db-prod-04","data":{"host":"db-prod-04"},"dedupe_key":"m2-s3"}' TS=$(date +%s) SIG=$(printf '%s.%s' "$TS" "$BODY" | openssl dgst -sha256 -hmac "$SECRET" -hex | awk '{print $2}') curl -s -X POST http://localhost:8800/v1/ingest \ -H "Content-Type: application/json" \ -H "X-BA-Signature: t=$TS,v1=$SIG" \ --data "$BODY" ``` Wait 2s, check deliveries. **Expected recipients:** Alice + Bob. Why: - Alice: in (no min). - Bob: critical meets his min_severity=critical → in. - Carol: critical, not inminent_colapse → still in quiet hours → out. ### Step 4 — send a `critical` storage alert for `db-prod-03` (rule match) ```bash SECRET=s3cret-acme BODY='{"company_id":"acme-001","source_id":"prom-prod","severity":"critical","category":"storage","title":"Disk full","body":"db-prod-03","data":{"host":"db-prod-03"},"dedupe_key":"m2-s4"}' TS=$(date +%s) SIG=$(printf '%s.%s' "$TS" "$BODY" | openssl dgst -sha256 -hmac "$SECRET" -hex | awk '{print $2}') curl -s -X POST http://localhost:8800/v1/ingest \ -H "Content-Type: application/json" \ -H "X-BA-Signature: t=$TS,v1=$SIG" \ --data "$BODY" ``` **Expected recipients:** Alice + Bob (rule doesn't add anyone new — Bob is already in via group+severity). This step proves the rule doesn't double-route. The SQL UNION just dedupes the candidate set. ### Step 5 — send a `warning` alert that ONLY matches the rule (and would normally skip Bob) This is the rule-fires scenario. Add a new individual that has min_severity=warning (Alice does). But the rule points at Bob specifically, and Bob has min_severity=critical. So a `warning` alert matching the rule goes to Bob? **No** — per the algorithm, subscriptions still apply on top of rules. So a `warning` storage alert for db-prod-03 should go to Alice only (Bob's subscription filters it out). ```bash SECRET=s3cret-acme BODY='{"company_id":"acme-001","source_id":"prom-prod","severity":"warning","category":"storage","title":"Disk 80% on prod-03","body":"pre-empt","data":{"host":"db-prod-03"},"dedupe_key":"m2-s5"}' TS=$(date +%s) SIG=$(printf '%s.%s' "$TS" "$BODY" | openssl dgst -sha256 -hmac "$SECRET" -hex | awk '{print $2}') curl -s -X POST http://localhost:8800/v1/ingest \ -H "Content-Type: application/json" \ -H "X-BA-Signature: t=$TS,v1=$SIG" \ --data "$BODY" ``` **Expected recipients:** Alice only. Rule matched Bob, but his min_severity filter (critical) drops warning. ### Step 6 — send an `inminent_colapse` alert for any host This is the quiet-hours bypass test. Carol is in quiet hours; an `inminent_colapse` should reach her anyway. ```bash SECRET=s3cret-acme BODY='{"company_id":"acme-001","source_id":"prom-prod","severity":"inminent_colapse","category":"power","title":"PDU overload imminent","body":"rack-B","data":{"host":"rack-b"},"dedupe_key":"m2-s6"}' TS=$(date +%s) SIG=$(printf '%s.%s' "$TS" "$BODY" | openssl dgst -sha256 -hmac "$SECRET" -hex | awk '{print $2}') curl -s -X POST http://localhost:8800/v1/ingest \ -H "Content-Type: application/json" \ -H "X-BA-Signature: t=$TS,v1=$SIG" \ --data "$BODY" ``` **Expected recipients:** Alice + Carol (severity bypasses Carol's quiet hours; Bob's min_severity=critical, inminent_colapse=rank 3 ≥ critical=rank 2 → in). Wait — actually Bob's min_severity=critical, and inminent_colapse rank 3 is > critical rank 2, so Bob IS in. Let me re-check: | individual | in? | why | |---|---|---| | Alice | in | no min | | Bob | in | min=critical, inminent_colapse > critical | | Carol | in | quiet hours bypassed for inminent_colapse | **Expected:** Alice + Bob + Carol. All three. ### Step 7 — alert with no matching recipients (hard-fail) Send a `warning` alert via a source that has a valid HMAC secret (so ingestd doesn't 401) but no subscriptions and no routing-rule matches. The seed registers `acme-001:prom-prod` only, so the easiest is to add a second source to `BA_INGESTD_SOURCES` in compose, but that requires a restart. The cheaper path: temporarily revoke all subscriptions in psql, then send an alert, then restore. ```bash # revoke docker compose exec postgres psql -U ba -d ba -c " UPDATE subscriptions SET status = 'paused' WHERE company_id = 'acme-001'; " # wait a moment so the next request re-resolves sleep 1 # send SECRET=s3cret-acme BODY='{"company_id":"acme-001","source_id":"prom-prod","severity":"info","category":"storage","title":"no recipients","data":{"host":"db-prod-04"},"dedupe_key":"m2-s7"}' TS=$(date +%s) SIG=$(printf '%s.%s' "$TS" "$BODY" | openssl dgst -sha256 -hmac "$SECRET" -hex | awk '{print $2}') curl -s -X POST http://localhost:8800/v1/ingest \ -H "Content-Type: application/json" \ -H "X-BA-Signature: t=$TS,v1=$SIG" \ --data "$BODY" ``` **Expected:** 202 from ingestd, then routerd logs `"zero recipients, dropping"`. **No** delivery row for this alert. ```bash # verify docker compose logs --no-color --since 1m routerd 2>&1 | grep -E "zero recipients" | tail -3 # restore docker compose exec postgres psql -U ba -d ba -c " UPDATE subscriptions SET status = 'active' WHERE company_id = 'acme-001'; " ``` ### Step 8 — summary After all steps, run: ```bash docker compose exec postgres psql -U ba -d ba -c " SELECT alert_id, individual_id, status FROM deliveries WHERE alert_id LIKE '0000%' ORDER BY id DESC LIMIT 15; " ``` Each step's expected recipient set is in the per-step notes above. The total across steps 2–6 should be 1+2+2+1+3 = **9 deliveries** (assuming 0 dedupes within the 60s window).