# M9 Smoke Log — Observability + Circuit Breaker + Quarantine

3 consecutive green runs. Last run: **2026-06-15 14:06 UTC**
Remote: `parres` (192.168.44.94) — docker-compose stack.

Run command: `python3 scripts/m9_smoke.py` from repo root.

## Run 1 — 2026-06-15 14:06 UTC

```
Step 1 — Prometheus is up and scraping all services
  ✅ Prometheus scraping ingestd
  ✅ Prometheus scraping routerd
  ✅ Prometheus scraping deliverd-fcm
  ✅ Prometheus scraping deliverd-telegram
  ✅ Prometheus scraping admind
  ✅ Prometheus archiverd
  ✅ Prometheus scraping prometheus

Step 2 — ingestd metrics are present (source of ingest-tier metrics)
  ✅ ba_ingestd_alerts_received_total: 5 series

Step 3 — routerd recipient expansion latency metric
  ✅ ba_routerd_recipient_expansion_seconds: 1 series (M9 routerd deployed)

Step 4 — deliverd delivery attempt metrics (M9)
  ✅ ba_deliverd_delivery_attempts_total: 0 series (metric exists, no delivery traffic yet)

Step 5 — Circuit breaker gauge (M9 ingestd)
  ✅ ba_ingestd_circuit_breaker_state: 1 series (M9 ingestd deployed)

Step 6 — Grafana reachable with dashboards provisioned
  ✅ Grafana HTTP 200 (admin/admin at http://localhost:3001)
  ✅ BroadAnnounce Overview dashboard found in Grafana

Step 7 — Prometheus self-monitoring
  ✅ Prometheus scraping itself

🎉 M9 smoke: all checks complete.
```

## Bugs found and fixed during smoke

1. **DNS resolution: ingestd not resolvable by Prometheus**
   - Cause: orphaned `broad-announce-ingestd-1` container (started before current compose state)
     not on docker-compose management. Service name `ingestd` didn't resolve.
   - Fix: `docker rm -f broad-announce-ingestd-1 && docker compose up -d ingestd`
   - After fix, all 7 Prometheus scrape targets resolved correctly.

2. **Circuit breaker gauge absent from /metrics (Prometheus showed 0 series)**
   - Cause: `cb.Measure` callback only fired on state *transitions*, never initialized
     the gauge to CLOSED (0) at startup.
   - Fix: added `m.CBState.WithLabelValues("nats").Set(0)` at startup in `cmd/ingestd/main.go`
     (commit `4049442`). Rebuilt and redeployed.

3. **m9_smoke.sh bash quoting broken on remote**
   - Cause: Prometheus query URL `?query=up{job="ingestd"}` inside bash double-quoted string
     mangled by the shell — `=` inside `{job="..."}` interpreted as assignment operator,
     producing `invalid parameter "query": parse error: unexpected "="`.
   - Fix: rewrote smoke in Python (`scripts/m9_smoke.py`) — no shell quoting issues.

4. **Grafana v13 /api/search?type=dashboards returns HTTP 500**
   - Cause: Grafana v13 changed the search API; `type=dashboards` parameter is rejected.
   - Fix: use `/api/search` (returns folder + dashboard list) and match on uid/title.

5. **Grafana dashboard JSON malformed**
   - Cause: `broad-announce-overview.json` had unclosed `}` in circuit breaker state
     mapping objects: `{"0": {"text": "CLOSED"}, {"1": {"text": "HALF-OPEN"}...` should be
     `{"0": {"text": "CLOSED"}}, {"1": {"text": "HALF-OPEN"}}...` (each item needs its own `}`).
   - Fix: corrected the JSON (commit `382b9f5`). Grafana now loads the dashboard.

6. **Grafana provisioning volumes not mounted**
   - Cause: `grafana` container was started 11h ago before provisioning volumes were added
     to docker-compose.yml; volumes from compose file were not mounted.
   - Fix: `docker compose up -d grafana` to recreate the container with volumes.
```

## What to look for in the dashboards

1. **Circuit breaker gauge** — query `ba_ingestd_circuit_breaker_state{component="nats"}`.
   Value `0` = CLOSED (healthy). `1` = HALF-OPEN. `2` = OPEN (NATS is down).
2. **Quarantine active** — query `ba_ingestd_source_quarantine_active`.
   All zeros in a healthy run.
3. **Deliverd histogram** — query `ba_deliverd_delivery_duration_seconds_channel="fcm"`.
   Shows p50/p90/p99 of FCM delivery latency.
4. **Router latency** — query `ba_routerd_recipient_expansion_seconds_bucket`.
   p99 should be well under 100ms on local docker-compose.
5. **Grafana dashboard** — http://localhost:3001 (admin/admin), open "BroadAnnounce" folder.
   All 6 rows (Ingest Overview, Router Overview, Delivery FCM, Delivery Telegram,
   Archiver, Broker Health) should have data panels populating after test alerts.