# M9 Verification — Observability (Prom/Grafana) + Circuit Breaker + Quarantine

**Status:** shipped 2026-06-15
**Branch:** master
**Commits:** `2de81f8` (1/3) + `89849d9` (2/3) + `adc37c2` (3/3) + `d99451b` (prometheus port fix)
**Verified on:** local docker-compose stack

This milestone ships SPEC §22 layers 6 and 7 (circuit breaker + quarantine) plus full observability coverage: every service now exposes a Prometheus metrics endpoint, Prometheus scrapes all targets, and a Grafana dashboard provides per-tier and per-company drilldown.

## What landed

| Area | Change |
| --- | --- |
| Circuit breaker | `internal/circuitbreaker/circuitbreaker.go` — 3-state machine (CLOSED/HALF-OPEN/OPEN) per component. Uses `sony/gobreaker` under the hood. Trip: 5 consecutive failures in 30s → OPEN for 60s → HALF-OPEN (3 test calls admitted). |
| Circuit breaker tests | `internal/circuitbreaker/circuitbreaker_test.go` — 14 subtests covering all 4 transitions, half-open cap, concurrent calls, all error types. 14/14 PASS. |
| Quarantine | `internal/quarantine/quarantine.go` — per-source error-rate enforcement. Counts `invalid_json`, `bad_signature`, `rate_limited`, `quota_exceeded` within a sliding window; quarantines source when `error_count > threshold`. 5-min auto-lift. |
| ingestd wiring | `cmd/ingestd/main.go` + `cmd/ingestd/process.go` — wires `circuitbreaker.New("broker")` and `quarantine.New()` into `processDeps`. Circuit trips on `nc.Publish` failure (NATS down). Quarantine increments on validation/auth/rate errors. |
| Config | `internal/config/config.go` — new `CircuitBreaker` and `Quarantine` structs. Env vars: `BA_CB_*` and `BA_QUARANTINE_*`. |
| Deliverd metrics | `cmd/deliverd-fcm/main.go` + `cmd/deliverd-telegram/main.go` — new `ba_deliverd_delivery_attempts_total{channel,status}` counter and `ba_deliverd_delivery_duration_seconds{channel}` histogram. |
| Routerd histogram | `cmd/routerd/main.go` — `ba_routerd_recipient_expansion_seconds` histogram (buckets: 1/5/10/25/50/100/250/500ms). Emitted on every `ResolveTargets` call. |
| Prometheus scrape | `deploy/prometheus/prometheus.yml` — split `deliverd-fcm` and `deliverd-telegram` into separate scrape jobs (was a combined `deliverd` job). `deliverd-telegram` now on correct port `8821`. Added `archiverd:8804`. |
| Grafana provisioning | `deploy/grafana/provisioning/datasources/prometheus.yml` — auto-provisions the Prometheus datasource. `deploy/grafana/provisioning/dashboards/dashboards.yml` + `broad-announce-overview.json` — one overview dashboard with 6 rows: Ingest Overview, Router Overview, Delivery (FCM), Delivery (Telegram), Archiver, Broker Health. |
| Smoke | `scripts/m9_smoke.sh` — 7 steps: Prometheus up checks for all services, key metrics presence, circuit breaker gauge, Grafana reachable + BroadAnnounce dashboard found. |
| Dockerfile | Reformatted RUN chain (no functional change). |

## New metrics

### ingestd (layer 6 — circuit breaker)

| Metric | Type | Labels | Description |
| --- | --- | --- | --- |
| `ba_ingestd_circuit_breaker_state{component}` | gauge | component | 0=closed, 1=half-open, 2=open |
| `ba_ingestd_circuit_breaker_transitions_total{component,from,to}` | counter | component, from, to | State transitions |
| `ba_ingestd_circuit_breaker_calls_total{component,result}` | counter | component, result | call outcome (success/failure/rejected) |

### ingestd (layer 7 — quarantine)

| Metric | Type | Labels | Description |
| --- | --- | --- | --- |
| `ba_ingestd_source_quarantined_total{source_id,company_id}` | counter | source_id, company_id | Times a source has been quarantined |
| `ba_ingestd_source_quarantine_active{source_id}` | gauge | source_id | 1=currently quarantined, 0=not |

### routerd

| Metric | Type | Labels | Description |
| --- | --- | --- | --- |
| `ba_routerd_recipient_expansion_seconds{quantile}` | histogram | — | Time to resolve all recipients for an alert |

### deliverd-fcm / deliverd-telegram

| Metric | Type | Labels | Description |
| --- | --- | --- | --- |
| `ba_deliverd_delivery_attempts_total{channel,status}` | counter | channel, status | Delivery attempts (sent/failed) |
| `ba_deliverd_delivery_duration_seconds{channel,quantile}` | histogram | channel | End-to-end delivery latency |

## New env vars

| Service | Var | Default | Purpose |
| --- | --- | --- | --- |
| ingestd | `BA_CB_FAILURE_THRESHOLD` | 5 | consecutive failures to trip |
| ingestd | `BA_CB_FAILURE_WINDOW_SECONDS` | 30 | rolling window for failure count |
| ingestd | `BA_CB_OPEN_DURATION_SECONDS` | 60 | seconds circuit stays OPEN |
| ingestd | `BA_CB_MAX_HALF_OPEN` | 3 | test calls admitted in HALF-OPEN |
| ingestd | `BA_QUARANTINE_THRESHOLD` | 100 | errors in window to quarantine |
| ingestd | `BA_QUARANTINE_WINDOW_SECONDS` | 60 | sliding window |
| ingestd | `BA_QUARANTINE_DURATION_SECONDS` | 300 | auto-lift after 5 min |

## Smoke test

```bash
bash scripts/m9_smoke.sh
```

## Files added / changed by M9

```
internal/circuitbreaker/circuitbreaker.go       [NEW 233 LoC]
internal/circuitbreaker/circuitbreaker_test.go  [NEW 349 LoC]
internal/quarantine/quarantine.go                [NEW 165 LoC]
cmd/ingestd/main.go                              [M9 +40 LoC]
cmd/ingestd/process.go                           [M9 +97 LoC]
cmd/deliverd-fcm/main.go                         [M9 +24 LoC]
cmd/deliverd-telegram/main.go                   [M9 +25 LoC]
cmd/routerd/main.go                              [M9 +11 LoC]
cmd/routerd/collapse.go                          [M9 +9 LoC]
internal/observability/metrics.go                [M9 +81 LoC]
internal/config/config.go                        [M9 +25 LoC]
deploy/prometheus/prometheus.yml                [M9 +28, port fix]
deploy/grafana/provisioning/datasources/prometheus.yml  [NEW]
deploy/grafana/provisioning/dashboards/dashboards.yml   [NEW]
deploy/grafana/provisioning/dashboards/broad-announce-overview.json [NEW 461 LoC]
docker-compose.yml                               [M9 +17]
.env.example                                     [M9 +7]
scripts/m9_smoke.sh                             [NEW]
SPEC.md                                          [bumped to shipped]
```