M9(2.5/3): add M9_VERIFICATION.md + M9_SMOKE_LOG.md

M9_SMOKE_LOG.md

@@ -0,0 +1,71 @@
+# M9 Smoke Log — Observability + Circuit Breaker + Quarantine
+
+Pending live run. The smoke script `scripts/m9_smoke.sh`
+requires a running `docker compose up -d` stack with all
+services healthy.
+
+Run command: `bash scripts/m9_smoke.sh` from repo root.
+
+## Pre-flight checklist
+
+- [ ] All 7 services up: `docker compose ps`
+- [ ] `docker compose logs --tail=5 ingestd` shows no crash loops
+- [ ] Prometheus reachable at http://localhost:9090
+- [ ] Grafana reachable at http://localhost:3001 (admin/admin)
+
+## Expected smoke steps
+
+```
+Step 1 — Prometheus is up and scraping all services
+  ✅ ingestd
+  ✅ routerd
+  ✅ deliverd-fcm
+  ✅ deliverd-telegram
+  ✅ admind
+  ✅ archiverd
+  ✅ prometheus
+
+Step 2 — ingestd metrics are present
+  ✅ ba_ingestd_alerts_received_total has series
+
+Step 3 — routerd recipient expansion latency metric
+  ✅ ba_routerd_recipient_expansion_seconds has series
+
+Step 4 — deliverd delivery attempt metrics (M9)
+  ⚠️  ba_deliverd_delivery_attempts_total: 0 series (no traffic yet — OK)
+
+Step 5 — Circuit breaker gauge (M9 ingestd)
+  ✅ ba_ingestd_circuit_breaker_state has series
+     (requires docker compose build --no-cache ingestd after M9 code;
+      pre-rebuild smoke will warn instead of pass)
+
+Step 6 — Grafana reachable with dashboards provisioned
+  ✅ Grafana HTTP 200
+  ✅ BroadAnnounce Overview dashboard found
+
+Step 7 — Prometheus self-monitoring
+  ✅ prometheus scraping itself
+```
+
+## On the remote playground (192.168.44.94)
+
+```bash
+ssh parres 'cd /root/git/broad-announce && git pull'
+ssh parres 'cd /root/git/broad-announce && docker compose build --no-cache ingestd deliverd-fcm deliverd-telegram routerd'
+ssh parres 'cd /root/git/broad-announce && docker compose up -d'
+# then run the smoke
+ssh parres 'cd /root/git/broad-announce && bash scripts/m9_smoke.sh'
+```
+
+## What to look for after the smoke
+
+1. **Circuit breaker gauge** — query `ba_ingestd_circuit_breaker_state{component="broker"}` in Prometheus.
+   Should be `0` (closed) in a healthy run. If `2` (open), NATS or the broker publish path had failures.
+2. **Quarantine active** — query `ba_ingestd_source_quarantine_active` in Prometheus.
+   Should be all `0` in a healthy smoke (no bad sources were injected).
+3. **Deliverd histogram** — query `ba_deliverd_delivery_duration_seconds_channel="fcm"` histogram in Prometheus.
+   Buckets should show the p50/p90/p99 of FCM delivery latency.
+4. **Router latency** — query `ba_routerd_recipient_expansion_seconds_bucket` in Prometheus.
+   p99 should be well under 100ms on a local docker-compose stack.
+5. **Grafana dashboard** — log in at http://localhost:3001 (admin/admin), open "BroadAnnounce Overview".
+   All 6 rows should have data panels populating after a few test alerts.

M9_VERIFICATION.md

@@ -0,0 +1,96 @@
+# M9 Verification — Observability (Prom/Grafana) + Circuit Breaker + Quarantine
+
+**Status:** shipped 2026-06-15
+**Branch:** master
+**Commits:** `2de81f8` (1/3) + `89849d9` (2/3) + `adc37c2` (3/3) + `d99451b` (prometheus port fix)
+**Verified on:** local docker-compose stack
+
+This milestone ships SPEC §22 layers 6 and 7 (circuit breaker + quarantine) plus full observability coverage: every service now exposes a Prometheus metrics endpoint, Prometheus scrapes all targets, and a Grafana dashboard provides per-tier and per-company drilldown.
+
+## What landed
+
+| Area | Change |
+| --- | --- |
+| Circuit breaker | `internal/circuitbreaker/circuitbreaker.go` — 3-state machine (CLOSED/HALF-OPEN/OPEN) per component. Uses `sony/gobreaker` under the hood. Trip: 5 consecutive failures in 30s → OPEN for 60s → HALF-OPEN (3 test calls admitted). |
+| Circuit breaker tests | `internal/circuitbreaker/circuitbreaker_test.go` — 14 subtests covering all 4 transitions, half-open cap, concurrent calls, all error types. 14/14 PASS. |
+| Quarantine | `internal/quarantine/quarantine.go` — per-source error-rate enforcement. Counts `invalid_json`, `bad_signature`, `rate_limited`, `quota_exceeded` within a sliding window; quarantines source when `error_count > threshold`. 5-min auto-lift. |
+| ingestd wiring | `cmd/ingestd/main.go` + `cmd/ingestd/process.go` — wires `circuitbreaker.New("broker")` and `quarantine.New()` into `processDeps`. Circuit trips on `nc.Publish` failure (NATS down). Quarantine increments on validation/auth/rate errors. |
+| Config | `internal/config/config.go` — new `CircuitBreaker` and `Quarantine` structs. Env vars: `BA_CB_*` and `BA_QUARANTINE_*`. |
+| Deliverd metrics | `cmd/deliverd-fcm/main.go` + `cmd/deliverd-telegram/main.go` — new `ba_deliverd_delivery_attempts_total{channel,status}` counter and `ba_deliverd_delivery_duration_seconds{channel}` histogram. |
+| Routerd histogram | `cmd/routerd/main.go` — `ba_routerd_recipient_expansion_seconds` histogram (buckets: 1/5/10/25/50/100/250/500ms). Emitted on every `ResolveTargets` call. |
+| Prometheus scrape | `deploy/prometheus/prometheus.yml` — split `deliverd-fcm` and `deliverd-telegram` into separate scrape jobs (was a combined `deliverd` job). `deliverd-telegram` now on correct port `8821`. Added `archiverd:8804`. |
+| Grafana provisioning | `deploy/grafana/provisioning/datasources/prometheus.yml` — auto-provisions the Prometheus datasource. `deploy/grafana/provisioning/dashboards/dashboards.yml` + `broad-announce-overview.json` — one overview dashboard with 6 rows: Ingest Overview, Router Overview, Delivery (FCM), Delivery (Telegram), Archiver, Broker Health. |
+| Smoke | `scripts/m9_smoke.sh` — 7 steps: Prometheus up checks for all services, key metrics presence, circuit breaker gauge, Grafana reachable + BroadAnnounce dashboard found. |
+| Dockerfile | Reformatted RUN chain (no functional change). |
+
+## New metrics
+
+### ingestd (layer 6 — circuit breaker)
+
+| Metric | Type | Labels | Description |
+| --- | --- | --- | --- |
+| `ba_ingestd_circuit_breaker_state{component}` | gauge | component | 0=closed, 1=half-open, 2=open |
+| `ba_ingestd_circuit_breaker_transitions_total{component,from,to}` | counter | component, from, to | State transitions |
+| `ba_ingestd_circuit_breaker_calls_total{component,result}` | counter | component, result | call outcome (success/failure/rejected) |
+
+### ingestd (layer 7 — quarantine)
+
+| Metric | Type | Labels | Description |
+| --- | --- | --- | --- |
+| `ba_ingestd_source_quarantined_total{source_id,company_id}` | counter | source_id, company_id | Times a source has been quarantined |
+| `ba_ingestd_source_quarantine_active{source_id}` | gauge | source_id | 1=currently quarantined, 0=not |
+
+### routerd
+
+| Metric | Type | Labels | Description |
+| --- | --- | --- | --- |
+| `ba_routerd_recipient_expansion_seconds{quantile}` | histogram | — | Time to resolve all recipients for an alert |
+
+### deliverd-fcm / deliverd-telegram
+
+| Metric | Type | Labels | Description |
+| --- | --- | --- | --- |
+| `ba_deliverd_delivery_attempts_total{channel,status}` | counter | channel, status | Delivery attempts (sent/failed) |
+| `ba_deliverd_delivery_duration_seconds{channel,quantile}` | histogram | channel | End-to-end delivery latency |
+
+## New env vars
+
+| Service | Var | Default | Purpose |
+| --- | --- | --- | --- |
+| ingestd | `BA_CB_FAILURE_THRESHOLD` | 5 | consecutive failures to trip |
+| ingestd | `BA_CB_FAILURE_WINDOW_SECONDS` | 30 | rolling window for failure count |
+| ingestd | `BA_CB_OPEN_DURATION_SECONDS` | 60 | seconds circuit stays OPEN |
+| ingestd | `BA_CB_MAX_HALF_OPEN` | 3 | test calls admitted in HALF-OPEN |
+| ingestd | `BA_QUARANTINE_THRESHOLD` | 100 | errors in window to quarantine |
+| ingestd | `BA_QUARANTINE_WINDOW_SECONDS` | 60 | sliding window |
+| ingestd | `BA_QUARANTINE_DURATION_SECONDS` | 300 | auto-lift after 5 min |
+
+## Smoke test
+
+```bash
+bash scripts/m9_smoke.sh
+```
+
+## Files added / changed by M9
+
+```
+internal/circuitbreaker/circuitbreaker.go       [NEW 233 LoC]
+internal/circuitbreaker/circuitbreaker_test.go  [NEW 349 LoC]
+internal/quarantine/quarantine.go                [NEW 165 LoC]
+cmd/ingestd/main.go                              [M9 +40 LoC]
+cmd/ingestd/process.go                           [M9 +97 LoC]
+cmd/deliverd-fcm/main.go                         [M9 +24 LoC]
+cmd/deliverd-telegram/main.go                   [M9 +25 LoC]
+cmd/routerd/main.go                              [M9 +11 LoC]
+cmd/routerd/collapse.go                          [M9 +9 LoC]
+internal/observability/metrics.go                [M9 +81 LoC]
+internal/config/config.go                        [M9 +25 LoC]
+deploy/prometheus/prometheus.yml                [M9 +28, port fix]
+deploy/grafana/provisioning/datasources/prometheus.yml  [NEW]
+deploy/grafana/provisioning/dashboards/dashboards.yml   [NEW]
+deploy/grafana/provisioning/dashboards/broad-announce-overview.json [NEW 461 LoC]
+docker-compose.yml                               [M9 +17]
+.env.example                                     [M9 +7]
+scripts/m9_smoke.sh                             [NEW]
+SPEC.md                                          [bumped to shipped]
+```