|
|
@@ -110,3 +110,61 @@ The M11 dev-playground gate is shipped, but the gRPC → NATS → delivery pipel
|
|
|
- [⚠️] SPEC.md M11 row updated to **shipped conditionally 2026-06-16**
|
|
|
|
|
|
**Bottom line:** M11 is not fully shippable. The receive path works. The publish path (gRPC → NATS → delivery) is broken under sustained load due to NATS resource limits. Re-verification needed after NATS config is tuned and the smoke is updated to assert on publish success.
|
|
|
+
|
|
|
+---
|
|
|
+
|
|
|
+## F1 NATS fix verification — 2026-06-16 14:00 EDT
|
|
|
+
|
|
|
+**What was wrong:** the 2-min re-verification (2026-06-16 14:48 EDT) on the
|
|
|
+M11 conditional ship showed the rate collapsing from 9.7k/s → 0/s at the 2m
|
|
|
+mark. Root cause: NATS JetStream hit its server-level `max_storage` cap
|
|
|
+(5.46 GiB default on parres with 6.5 GiB free) because ALERTS stream's
|
|
|
+24h `MaxAge` allowed 6.1 GiB of test data to accumulate. The 10-min soak
|
|
|
+that "shipped" M11 was a false positive: `ba_ingestd_alerts_received_total`
|
|
|
+counts gRPC receive, not NATS publish success.
|
|
|
+
|
|
|
+**The fix (commits f450196, 6c82dcf, 82dbc5a, 09d5584):**
|
|
|
+
|
|
|
+| Change | File | Effect |
|
|
|
+|---|---|---|
|
|
|
+| ALERTS `MaxAge` 24h → 1h + `MaxBytes`=1 GiB + `DiscardOld` | `internal/broker/broker.go` | ALERTS self-trims; producer never blocks |
|
|
|
+| DELIVERIES `MaxBytes`=100 MiB | `internal/broker/broker.go` | Cap on delivery stream |
|
|
|
+| DLQ `MaxAge` 7d → 1h + `MaxBytes`=10 MiB | `internal/broker/broker.go` | DLQ is operational, not archival |
|
|
|
+| NATS server `max_file_store`=10 GiB via `nats.conf` | `deploy/nats/nats.conf` + `docker-compose.yml` | Explicit server-level cap; 2.10 doesn't expose these as CLI flags |
|
|
|
+| Smoke teardown: drop `-v` | `scripts/m11_smoke.py` | Was destroying pgdata/natsdata/chdata every run |
|
|
|
+
|
|
|
+**Verification (smoke run `m11-fix2-20260616-140053`):**
|
|
|
+
|
|
|
+```
|
|
|
+[0m] rate=7548/s p99=nanms dlq=0 streams=32
|
|
|
+[1m] rate=7698/s p99=nanms dlq=0 streams=32
|
|
|
+[1m] rate=7638/s p99=24.9ms dlq=0 streams=32
|
|
|
+[2m] rate=7696/s p99=nanms dlq=0 streams=32
|
|
|
+... (20/20 samples) ...
|
|
|
+[10m] rate=7208/s p99=nanms dlq=0 streams=32
|
|
|
+
|
|
|
+Step 4 — multi-stream backpressure: ✅ no crashes, 0 rate-limited acks
|
|
|
+ (per-source cap 20k/s > 9.6k/s aggregate, no backpressure expected)
|
|
|
+Step 5 — DLQ invariant: ✅ 0 over 10min
|
|
|
+Step 6 — teardown: ✅ named volumes preserved (pgdata/natsdata/chdata intact)
|
|
|
+
|
|
|
+🎉 M11 smoke: all checks complete.
|
|
|
+```
|
|
|
+
|
|
|
+**On-disk evidence of the cap working:**
|
|
|
+
|
|
|
+- ALERTS `meta.inf` has `max_age: 3600000000000` (1h) and `max_bytes: 1073741824` (1 GiB)
|
|
|
+- During the soak, ALERTS bytes stabilized at **exactly 1,024 MiB** (the cap)
|
|
|
+ — old messages trimmed as new ones arrived, no producer stall
|
|
|
+- `varz` reports `max_storage: 10 GiB`, current `storage: 0 B` after smoke
|
|
|
+- **Zero `[ERR] JetStream resource limits exceeded` log lines** during the entire 12-min smoke
|
|
|
+
|
|
|
+**Status:** M11 dev-playground gate now SHIPPED (properly, not conditionally).
|
|
|
+See commit `f450196` for the broker.go change, `82dbc5a` for the
|
|
|
+docker-compose + nats.conf, and `09d5584` for the smoke teardown fix.
|
|
|
+
|
|
|
+**What F2 still owes us:** a `ba_ingestd_nats_publish_total{result="ok"}`
|
|
|
+counter plus a smoke assertion that publish rate ≥ receive rate, so the
|
|
|
+next class of "system looks healthy but publishes are silently failing"
|
|
|
+bug gets caught at smoke time. The `M11_NATS_INVESTIGATION.md` doc
|
|
|
+proposes this in the medium-term section.
|
