broad-announce/M9_SMOKE_LOG.md

M9 Smoke Log — Observability + Circuit Breaker + Quarantine

Pending live run. The smoke script scripts/m9_smoke.sh requires a running docker compose up -d stack with all services healthy.

Run command: bash scripts/m9_smoke.sh from repo root.

Pre-flight checklist

• All 7 services up: docker compose ps
• docker compose logs --tail=5 ingestd shows no crash loops
• Prometheus reachable at http://localhost:9090
• Grafana reachable at http://localhost:3001 (admin/admin)

Expected smoke steps

Step 1 — Prometheus is up and scraping all services
  ✅ ingestd
  ✅ routerd
  ✅ deliverd-fcm
  ✅ deliverd-telegram
  ✅ admind
  ✅ archiverd
  ✅ prometheus

Step 2 — ingestd metrics are present
  ✅ ba_ingestd_alerts_received_total has series

Step 3 — routerd recipient expansion latency metric
  ✅ ba_routerd_recipient_expansion_seconds has series

Step 4 — deliverd delivery attempt metrics (M9)
  ⚠️  ba_deliverd_delivery_attempts_total: 0 series (no traffic yet — OK)

Step 5 — Circuit breaker gauge (M9 ingestd)
  ✅ ba_ingestd_circuit_breaker_state has series
     (requires docker compose build --no-cache ingestd after M9 code;
      pre-rebuild smoke will warn instead of pass)

Step 6 — Grafana reachable with dashboards provisioned
  ✅ Grafana HTTP 200
  ✅ BroadAnnounce Overview dashboard found

Step 7 — Prometheus self-monitoring
  ✅ prometheus scraping itself

On the remote playground (192.168.44.94)

ssh parres 'cd /root/git/broad-announce && git pull'
ssh parres 'cd /root/git/broad-announce && docker compose build --no-cache ingestd deliverd-fcm deliverd-telegram routerd'
ssh parres 'cd /root/git/broad-announce && docker compose up -d'
# then run the smoke
ssh parres 'cd /root/git/broad-announce && bash scripts/m9_smoke.sh'

What to look for after the smoke

1. Circuit breaker gauge — query ba_ingestd_circuit_breaker_state{component="broker"} in Prometheus. Should be 0 (closed) in a healthy run. If 2 (open), NATS or the broker publish path had failures.
2. Quarantine active — query ba_ingestd_source_quarantine_active in Prometheus. Should be all 0 in a healthy smoke (no bad sources were injected).
3. Deliverd histogram — query ba_deliverd_delivery_duration_seconds_channel="fcm" histogram in Prometheus. Buckets should show the p50/p90/p99 of FCM delivery latency.
4. Router latency — query ba_routerd_recipient_expansion_seconds_bucket in Prometheus. p99 should be well under 100ms on a local docker-compose stack.
5. Grafana dashboard — log in at http://localhost:3001 (admin/admin), open "BroadAnnounce Overview". All 6 rows should have data panels populating after a few test alerts.