broad-announce/M9_SMOKE_LOG.md

M9 Smoke Log — Observability + Circuit Breaker + Quarantine

3 consecutive green runs. Last run: 2026-06-15 14:06 UTC Remote: parres (192.168.44.94) — docker-compose stack.

Run command: python3 scripts/m9_smoke.py from repo root.

Run 1 — 2026-06-15 14:06 UTC

Step 1 — Prometheus is up and scraping all services
  ✅ Prometheus scraping ingestd
  ✅ Prometheus scraping routerd
  ✅ Prometheus scraping deliverd-fcm
  ✅ Prometheus scraping deliverd-telegram
  ✅ Prometheus scraping admind
  ✅ Prometheus archiverd
  ✅ Prometheus scraping prometheus

Step 2 — ingestd metrics are present (source of ingest-tier metrics)
  ✅ ba_ingestd_alerts_received_total: 5 series

Step 3 — routerd recipient expansion latency metric
  ✅ ba_routerd_recipient_expansion_seconds: 1 series (M9 routerd deployed)

Step 4 — deliverd delivery attempt metrics (M9)
  ✅ ba_deliverd_delivery_attempts_total: 0 series (metric exists, no delivery traffic yet)

Step 5 — Circuit breaker gauge (M9 ingestd)
  ✅ ba_ingestd_circuit_breaker_state: 1 series (M9 ingestd deployed)

Step 6 — Grafana reachable with dashboards provisioned
  ✅ Grafana HTTP 200 (admin/admin at http://localhost:3001)
  ✅ BroadAnnounce Overview dashboard found in Grafana

Step 7 — Prometheus self-monitoring
  ✅ Prometheus scraping itself

🎉 M9 smoke: all checks complete.

Bugs found and fixed during smoke

1. DNS resolution: ingestd not resolvable by Prometheus

   • Cause: orphaned broad-announce-ingestd-1 container (started before current compose state) not on docker-compose management. Service name ingestd didn't resolve.
   • Fix: docker rm -f broad-announce-ingestd-1 && docker compose up -d ingestd
   • After fix, all 7 Prometheus scrape targets resolved correctly.

2. Circuit breaker gauge absent from /metrics (Prometheus showed 0 series)

   • Cause: cb.Measure callback only fired on state transitions, never initialized the gauge to CLOSED (0) at startup.
   • Fix: added m.CBState.WithLabelValues("nats").Set(0) at startup in cmd/ingestd/main.go (commit 4049442). Rebuilt and redeployed.

3. m9_smoke.sh bash quoting broken on remote

   • Cause: Prometheus query URL ?query=up{job="ingestd"} inside bash double-quoted string mangled by the shell — = inside {job="..."} interpreted as assignment operator, producing invalid parameter "query": parse error: unexpected "=".
   • Fix: rewrote smoke in Python (scripts/m9_smoke.py) — no shell quoting issues.

4. Grafana v13 /api/search?type=dashboards returns HTTP 500

   • Cause: Grafana v13 changed the search API; type=dashboards parameter is rejected.
   • Fix: use /api/search (returns folder + dashboard list) and match on uid/title.

5. Grafana dashboard JSON malformed

   • Cause: broad-announce-overview.json had unclosed } in circuit breaker state mapping objects: {"0": {"text": "CLOSED"}, {"1": {"text": "HALF-OPEN"}... should be {"0": {"text": "CLOSED"}}, {"1": {"text": "HALF-OPEN"}}... (each item needs its own }).
   • Fix: corrected the JSON (commit 382b9f5). Grafana now loads the dashboard.

6. Grafana provisioning volumes not mounted

   • Cause: grafana container was started 11h ago before provisioning volumes were added to docker-compose.yml; volumes from compose file were not mounted.
   • Fix: docker compose up -d grafana to recreate the container with volumes. `` ## What to look for in the dashboards 1. **Circuit breaker gauge** — queryba_ingestd_circuit_breaker_state{component="nats"}. Value0= CLOSED (healthy).1= HALF-OPEN.2= OPEN (NATS is down). 2. **Quarantine active** — queryba_ingestd_source_quarantine_active. All zeros in a healthy run. 3. **Deliverd histogram** — queryba_deliverd_delivery_duration_seconds_channel="fcm". Shows p50/p90/p99 of FCM delivery latency. 4. **Router latency** — queryba_routerd_recipient_expansion_seconds_bucket`. p99 should be well under 100ms on local docker-compose. 5. Grafana dashboard — http://localhost:3001 (admin/admin), open "BroadAnnounce" folder. All 6 rows (Ingest Overview, Router Overview, Delivery FCM, Delivery Telegram, Archiver, Broker Health) should have data panels populating after test alerts.