broad-announce/M9_VERIFICATION.md

M9 Verification — Observability (Prom/Grafana) + Circuit Breaker + Quarantine

Status: shipped 2026-06-15 Branch: master Commits: 2de81f8 (1/3) + 89849d9 (2/3) + adc37c2 (3/3) + d99451b (prometheus port fix) Verified on: local docker-compose stack

This milestone ships SPEC §22 layers 6 and 7 (circuit breaker + quarantine) plus full observability coverage: every service now exposes a Prometheus metrics endpoint, Prometheus scrapes all targets, and a Grafana dashboard provides per-tier and per-company drilldown.

What landed

Area	Change
Circuit breaker	internal/circuitbreaker/circuitbreaker.go — 3-state machine (CLOSED/HALF-OPEN/OPEN) per component. Uses sony/gobreaker under the hood. Trip: 5 consecutive failures in 30s → OPEN for 60s → HALF-OPEN (3 test calls admitted).
Circuit breaker tests	internal/circuitbreaker/circuitbreaker_test.go — 14 subtests covering all 4 transitions, half-open cap, concurrent calls, all error types. 14/14 PASS.
Quarantine	internal/quarantine/quarantine.go — per-source error-rate enforcement. Counts invalid_json, bad_signature, rate_limited, quota_exceeded within a sliding window; quarantines source when error_count > threshold. 5-min auto-lift.
ingestd wiring	cmd/ingestd/main.go + cmd/ingestd/process.go — wires circuitbreaker.New("broker") and quarantine.New() into processDeps. Circuit trips on nc.Publish failure (NATS down). Quarantine increments on validation/auth/rate errors.
Config	internal/config/config.go — new CircuitBreaker and Quarantine structs. Env vars: BA_CB_* and BA_QUARANTINE_*.
Deliverd metrics	cmd/deliverd-fcm/main.go + cmd/deliverd-telegram/main.go — new ba_deliverd_delivery_attempts_total{channel,status} counter and ba_deliverd_delivery_duration_seconds{channel} histogram.
Routerd histogram	cmd/routerd/main.go — ba_routerd_recipient_expansion_seconds histogram (buckets: 1/5/10/25/50/100/250/500ms). Emitted on every ResolveTargets call.
Prometheus scrape	deploy/prometheus/prometheus.yml — split deliverd-fcm and deliverd-telegram into separate scrape jobs (was a combined deliverd job). deliverd-telegram now on correct port 8821. Added archiverd:8804.
Grafana provisioning	deploy/grafana/provisioning/datasources/prometheus.yml — auto-provisions the Prometheus datasource. deploy/grafana/provisioning/dashboards/dashboards.yml + broad-announce-overview.json — one overview dashboard with 6 rows: Ingest Overview, Router Overview, Delivery (FCM), Delivery (Telegram), Archiver, Broker Health.
Smoke	scripts/m9_smoke.sh — 7 steps: Prometheus up checks for all services, key metrics presence, circuit breaker gauge, Grafana reachable + BroadAnnounce dashboard found.
Dockerfile	Reformatted RUN chain (no functional change).

New metrics

ingestd (layer 6 — circuit breaker)

Metric	Type	Labels	Description
ba_ingestd_circuit_breaker_state{component}	gauge	component	0=closed, 1=half-open, 2=open
ba_ingestd_circuit_breaker_transitions_total{component,from,to}	counter	component, from, to	State transitions
ba_ingestd_circuit_breaker_calls_total{component,result}	counter	component, result	call outcome (success/failure/rejected)

ingestd (layer 7 — quarantine)

Metric	Type	Labels	Description
ba_ingestd_source_quarantined_total{source_id,company_id}	counter	source_id, company_id	Times a source has been quarantined
ba_ingestd_source_quarantine_active{source_id}	gauge	source_id	1=currently quarantined, 0=not

routerd

Metric	Type	Labels	Description
ba_routerd_recipient_expansion_seconds{quantile}	histogram	—	Time to resolve all recipients for an alert

deliverd-fcm / deliverd-telegram

Metric	Type	Labels	Description
ba_deliverd_delivery_attempts_total{channel,status}	counter	channel, status	Delivery attempts (sent/failed)
ba_deliverd_delivery_duration_seconds{channel,quantile}	histogram	channel	End-to-end delivery latency

New env vars

Service	Var	Default	Purpose
ingestd	BA_CB_FAILURE_THRESHOLD	5	consecutive failures to trip
ingestd	BA_CB_FAILURE_WINDOW_SECONDS	30	rolling window for failure count
ingestd	BA_CB_OPEN_DURATION_SECONDS	60	seconds circuit stays OPEN
ingestd	BA_CB_MAX_HALF_OPEN	3	test calls admitted in HALF-OPEN
ingestd	BA_QUARANTINE_THRESHOLD	100	errors in window to quarantine
ingestd	BA_QUARANTINE_WINDOW_SECONDS	60	sliding window
ingestd	BA_QUARANTINE_DURATION_SECONDS	300	auto-lift after 5 min

Smoke test

bash scripts/m9_smoke.sh

Files added / changed by M9

internal/circuitbreaker/circuitbreaker.go       [NEW 233 LoC]
internal/circuitbreaker/circuitbreaker_test.go  [NEW 349 LoC]
internal/quarantine/quarantine.go                [NEW 165 LoC]
cmd/ingestd/main.go                              [M9 +40 LoC]
cmd/ingestd/process.go                           [M9 +97 LoC]
cmd/deliverd-fcm/main.go                         [M9 +24 LoC]
cmd/deliverd-telegram/main.go                   [M9 +25 LoC]
cmd/routerd/main.go                              [M9 +11 LoC]
cmd/routerd/collapse.go                          [M9 +9 LoC]
internal/observability/metrics.go                [M9 +81 LoC]
internal/config/config.go                        [M9 +25 LoC]
deploy/prometheus/prometheus.yml                [M9 +28, port fix]
deploy/grafana/provisioning/datasources/prometheus.yml  [NEW]
deploy/grafana/provisioning/dashboards/dashboards.yml   [NEW]
deploy/grafana/provisioning/dashboards/broad-announce-overview.json [NEW 461 LoC]
docker-compose.yml                               [M9 +17]
.env.example                                     [M9 +7]
scripts/m9_smoke.sh                             [NEW]
SPEC.md                                          [bumped to shipped]