broad-announce/M2_VERIFICATION.md

M2 Verification — Recipient Resolution

This is the live smoke test for the M2 milestone (Recipient resolution, SPEC §6). It exercises the rules engine that replaces M1's broadcast with:

1. source.allowed_targets (groups + individuals)
2. routing_rules with match_expr overrides
3. subscriptions with min_severity, channel_mask, and quiet_hours
4. inminent_colapse bypasses quiet hours

Hard-fail (your Q2 answer): if the resolver returns zero targets, the alert is dropped and logged. No silent broadcast.

Prerequisites

• Stack is up: docker compose up -d
• Seed has been run: docker compose up seed (apply M2 migration + both seeds)
• HMAC secret for the acme-001:prom-prod source is s3cret-acme (matches BA_INGESTD_SOURCES env)

Seed summary

individual	subscription (severity, quiet hours)	notes
ind-acme-001 (Alice)	NULL, no quiet hours	gets every alert from prom-prod
ind-acme-002 (Bob)	critical, no quiet hours	only critical and above
ind-acme-003 (Carol)	NULL, 00:00–23:59 UTC	everything except inminent_colapse

Groups: sre = Alice + Bob.

Source acme-001:prom-prod allowed_targets:

• group:sre
• individual:ind-acme-003

Routing rule: category=storage + data.host=db-prod-03 → individual:ind-acme-002 (Bob), priority 10.

Step-by-step

Step 1 — confirm M2 seed applied

docker compose exec postgres psql -U ba -d ba -c "
SELECT i.id, i.full_name,
       s.min_severity, s.quiet_hours_start, s.quiet_hours_end, s.tz
FROM individuals i
LEFT JOIN subscriptions s
  ON s.individual_id = i.id AND s.source_id = 'prom-prod'
ORDER BY i.id;
"

Expected: 3 rows. Carol's quiet_hours_start=00:00:00 and quiet_hours_end=23:59:00.

Step 2 — send a warning storage alert

SECRET=s3cret-acme
BODY='{"company_id":"acme-001","source_id":"prom-prod","severity":"warning","category":"storage","title":"Disk 80% full","body":"db-prod-04","data":{"host":"db-prod-04"},"dedupe_key":"m2-s2"}'
TS=$(date +%s)
SIG=$(printf '%s.%s' "$TS" "$BODY" | openssl dgst -sha256 -hmac "$SECRET" -hex | awk '{print $2}')

curl -s -X POST http://localhost:8800/v1/ingest \
  -H "Content-Type: application/json" \
  -H "X-BA-Signature: t=$TS,v1=$SIG" \
  --data "$BODY"

Wait 2s, then check deliveries:

docker compose exec postgres psql -U ba -d ba -c "
SELECT alert_id, individual_id, channel, status FROM deliveries
WHERE alert_id IN (SELECT id FROM deliveries WHERE last_error IS NULL ORDER BY id DESC LIMIT 1)
ORDER BY individual_id;
"

Expected recipients: Alice only.

Why:

• Alice: severity=warning ≥ no min → in. (allowed via group:sre, in subscriptions)
• Bob: min_severity=critical, alert is warning → out.
• Carol: quiet hours cover the whole day → out (warning is not inminent_colapse).
• Routing rule doesn't match (host is db-prod-04, not db-prod-03).

Step 3 — send a critical storage alert for db-prod-04

SECRET=s3cret-acme
BODY='{"company_id":"acme-001","source_id":"prom-prod","severity":"critical","category":"storage","title":"Disk full","body":"db-prod-04","data":{"host":"db-prod-04"},"dedupe_key":"m2-s3"}'
TS=$(date +%s)
SIG=$(printf '%s.%s' "$TS" "$BODY" | openssl dgst -sha256 -hmac "$SECRET" -hex | awk '{print $2}')

curl -s -X POST http://localhost:8800/v1/ingest \
  -H "Content-Type: application/json" \
  -H "X-BA-Signature: t=$TS,v1=$SIG" \
  --data "$BODY"

Wait 2s, check deliveries.

Expected recipients: Alice + Bob.

Why:

• Alice: in (no min).
• Bob: critical meets his min_severity=critical → in.
• Carol: critical, not inminent_colapse → still in quiet hours → out.

Step 4 — send a critical storage alert for db-prod-03 (rule match)

SECRET=s3cret-acme
BODY='{"company_id":"acme-001","source_id":"prom-prod","severity":"critical","category":"storage","title":"Disk full","body":"db-prod-03","data":{"host":"db-prod-03"},"dedupe_key":"m2-s4"}'
TS=$(date +%s)
SIG=$(printf '%s.%s' "$TS" "$BODY" | openssl dgst -sha256 -hmac "$SECRET" -hex | awk '{print $2}')

curl -s -X POST http://localhost:8800/v1/ingest \
  -H "Content-Type: application/json" \
  -H "X-BA-Signature: t=$TS,v1=$SIG" \
  --data "$BODY"

Expected recipients: Alice + Bob (rule doesn't add anyone new — Bob is already in via group+severity).

This step proves the rule doesn't double-route. The SQL UNION just dedupes the candidate set.

Step 5 — send a warning alert that ONLY matches the rule (and would normally skip Bob)

This is the rule-fires scenario. Add a new individual that has min_severity=warning (Alice does). But the rule points at Bob specifically, and Bob has min_severity=critical. So a warning alert matching the rule goes to Bob? No — per the algorithm, subscriptions still apply on top of rules. So a warning storage alert for db-prod-03 should go to Alice only (Bob's subscription filters it out).

SECRET=s3cret-acme
BODY='{"company_id":"acme-001","source_id":"prom-prod","severity":"warning","category":"storage","title":"Disk 80% on prod-03","body":"pre-empt","data":{"host":"db-prod-03"},"dedupe_key":"m2-s5"}'
TS=$(date +%s)
SIG=$(printf '%s.%s' "$TS" "$BODY" | openssl dgst -sha256 -hmac "$SECRET" -hex | awk '{print $2}')

curl -s -X POST http://localhost:8800/v1/ingest \
  -H "Content-Type: application/json" \
  -H "X-BA-Signature: t=$TS,v1=$SIG" \
  --data "$BODY"

Expected recipients: Alice only. Rule matched Bob, but his min_severity filter (critical) drops warning.

Step 6 — send an inminent_colapse alert for any host

This is the quiet-hours bypass test. Carol is in quiet hours; an inminent_colapse should reach her anyway.

SECRET=s3cret-acme
BODY='{"company_id":"acme-001","source_id":"prom-prod","severity":"inminent_colapse","category":"power","title":"PDU overload imminent","body":"rack-B","data":{"host":"rack-b"},"dedupe_key":"m2-s6"}'
TS=$(date +%s)
SIG=$(printf '%s.%s' "$TS" "$BODY" | openssl dgst -sha256 -hmac "$SECRET" -hex | awk '{print $2}')

curl -s -X POST http://localhost:8800/v1/ingest \
  -H "Content-Type: application/json" \
  -H "X-BA-Signature: t=$TS,v1=$SIG" \
  --data "$BODY"

Expected recipients: Alice + Carol (severity bypasses Carol's quiet hours; Bob's min_severity=critical, inminent_colapse=rank 3 ≥ critical=rank 2 → in).

Wait — actually Bob's min_severity=critical, and inminent_colapse rank 3 is > critical rank 2, so Bob IS in. Let me re-check:

individual	in?	why
Alice	in	no min
Bob	in	min=critical, inminent_colapse > critical
Carol	in	quiet hours bypassed for inminent_colapse

Expected: Alice + Bob + Carol. All three.

Step 7 — alert with no matching recipients (hard-fail)

Send a warning alert via a source that has a valid HMAC secret (so ingestd doesn't 401) but no subscriptions and no routing-rule matches. The seed registers acme-001:prom-prod only, so the easiest is to add a second source to BA_INGESTD_SOURCES in compose, but that requires a restart. The cheaper path: temporarily revoke all subscriptions in psql, then send an alert, then restore.

# revoke
docker compose exec postgres psql -U ba -d ba -c "
UPDATE subscriptions SET status = 'paused' WHERE company_id = 'acme-001';
"
# wait a moment so the next request re-resolves
sleep 1
# send
SECRET=s3cret-acme
BODY='{"company_id":"acme-001","source_id":"prom-prod","severity":"info","category":"storage","title":"no recipients","data":{"host":"db-prod-04"},"dedupe_key":"m2-s7"}'
TS=$(date +%s)
SIG=$(printf '%s.%s' "$TS" "$BODY" | openssl dgst -sha256 -hmac "$SECRET" -hex | awk '{print $2}')

curl -s -X POST http://localhost:8800/v1/ingest \
  -H "Content-Type: application/json" \
  -H "X-BA-Signature: t=$TS,v1=$SIG" \
  --data "$BODY"

Expected: 202 from ingestd, then routerd logs "zero recipients, dropping". No delivery row for this alert.

# verify
docker compose logs --no-color --since 1m routerd 2>&1 | grep -E "zero recipients" | tail -3

# restore
docker compose exec postgres psql -U ba -d ba -c "
UPDATE subscriptions SET status = 'active' WHERE company_id = 'acme-001';
"

Step 8 — summary

After all steps, run:

docker compose exec postgres psql -U ba -d ba -c "
SELECT alert_id, individual_id, status
FROM deliveries
WHERE alert_id LIKE '0000%'
ORDER BY id DESC
LIMIT 15;
"

Each step's expected recipient set is in the per-step notes above. The total across steps 2–6 should be 1+2+2+1+3 = 9 deliveries (assuming 0 dedupes within the 60s window).