# M9 Smoke Log — Observability + Circuit Breaker + Quarantine

Pending live run. The smoke script `scripts/m9_smoke.sh`
requires a running `docker compose up -d` stack with all
services healthy.

Run command: `bash scripts/m9_smoke.sh` from repo root.

## Pre-flight checklist

- [ ] All 7 services up: `docker compose ps`
- [ ] `docker compose logs --tail=5 ingestd` shows no crash loops
- [ ] Prometheus reachable at http://localhost:9090
- [ ] Grafana reachable at http://localhost:3001 (admin/admin)

## Expected smoke steps

```
Step 1 — Prometheus is up and scraping all services
  ✅ ingestd
  ✅ routerd
  ✅ deliverd-fcm
  ✅ deliverd-telegram
  ✅ admind
  ✅ archiverd
  ✅ prometheus

Step 2 — ingestd metrics are present
  ✅ ba_ingestd_alerts_received_total has series

Step 3 — routerd recipient expansion latency metric
  ✅ ba_routerd_recipient_expansion_seconds has series

Step 4 — deliverd delivery attempt metrics (M9)
  ⚠️  ba_deliverd_delivery_attempts_total: 0 series (no traffic yet — OK)

Step 5 — Circuit breaker gauge (M9 ingestd)
  ✅ ba_ingestd_circuit_breaker_state has series
     (requires docker compose build --no-cache ingestd after M9 code;
      pre-rebuild smoke will warn instead of pass)

Step 6 — Grafana reachable with dashboards provisioned
  ✅ Grafana HTTP 200
  ✅ BroadAnnounce Overview dashboard found

Step 7 — Prometheus self-monitoring
  ✅ prometheus scraping itself
```

## On the remote playground (192.168.44.94)

```bash
ssh parres 'cd /root/git/broad-announce && git pull'
ssh parres 'cd /root/git/broad-announce && docker compose build --no-cache ingestd deliverd-fcm deliverd-telegram routerd'
ssh parres 'cd /root/git/broad-announce && docker compose up -d'
# then run the smoke
ssh parres 'cd /root/git/broad-announce && bash scripts/m9_smoke.sh'
```

## What to look for after the smoke

1. **Circuit breaker gauge** — query `ba_ingestd_circuit_breaker_state{component="broker"}` in Prometheus.
   Should be `0` (closed) in a healthy run. If `2` (open), NATS or the broker publish path had failures.
2. **Quarantine active** — query `ba_ingestd_source_quarantine_active` in Prometheus.
   Should be all `0` in a healthy smoke (no bad sources were injected).
3. **Deliverd histogram** — query `ba_deliverd_delivery_duration_seconds_channel="fcm"` histogram in Prometheus.
   Buckets should show the p50/p90/p99 of FCM delivery latency.
4. **Router latency** — query `ba_routerd_recipient_expansion_seconds_bucket` in Prometheus.
   p99 should be well under 100ms on a local docker-compose stack.
5. **Grafana dashboard** — log in at http://localhost:3001 (admin/admin), open "BroadAnnounce Overview".
   All 6 rows should have data panels populating after a few test alerts.