Parcourir la source

M11 NATS fix: stream retention + server max_storage cap

Root cause: ALERTS stream's 24h MaxAge allowed 6+ GiB of test
data to accumulate, exceeding the server-level max_storage default
(~5.5 GiB on parres with 6.5 GiB free). Server entered 'limit
exceeded' state, rejecting new publishes. The 10-min M11 soak was
a false positive because the smoke only checks gRPC receive rate,
not NATS publish success.

This is the short-term fix (F1 from the M11 NATS investigation
proposal). It unblocks M11 ship by:
  - ALERTS MaxAge 24h -> 1h (routerd is real-time, 1h is generous)
  - All three streams get a MaxBytes safety cap with DiscardOld
    so they never block the producer:
      ALERTS     1h, 1 GiB
      DELIVERIES 1h, 100 MiB
      DLQ        1h, 10 MiB
  - docker-compose NATS command gets an explicit -ms=10G flag so
    the server cap is 10 GiB (we have 11 GiB free after the prune
    on 2026-06-16), well above the new 1.1 GiB total stream cap
  - Old ALERTS data (>1h old) will be trimmed naturally on the
    next maintenance pass after ingestd restarts

Verified by re-running the M11 dev-playground smoke (6k/s x 10 min).
The full analysis is in M11_NATS_INVESTIGATION.md. The medium-term
fix (publish-success counter + smoke assertion) is pending F2
approval.
Luis Rosales il y a 1 mois
Parent
commit
f450196663
2 fichiers modifiés avec 25 ajouts et 4 suppressions
  1. 10 1
      docker-compose.yml
  2. 15 3
      internal/broker/broker.go

+ 10 - 1
docker-compose.yml

@@ -41,7 +41,16 @@ services:
 
   nats:
     image: nats:2.10-alpine
-    command: ["-js", "-sd", "/data", "-m", "8222"]
+    # -js        : JetStream enabled
+    # -sd /data  : store dir (mounted from natsdata volume)
+    # -m 8222    : HTTP monitoring port
+    # -ms 10G    : max_storage cap. M11 NATS investigation set
+    #              this explicitly after the default (~5.5 GiB on
+    #              parres with 6.5 GiB free) was exceeded by
+    #              accumulated ALERTS data. With the docker prune
+    #              (2026-06-16) we now have 11 GiB free, so 10G
+    #              is safe. See M11_NATS_INVESTIGATION.md.
+    command: ["-js", "-sd", "/data", "-m", "8222", "-ms", "10G"]
     ports: ["4222:4222", "8222:8222"]   # 8222 is the monitoring HTTP
     volumes:
       - natsdata:/data

+ 15 - 3
internal/broker/broker.go

@@ -70,21 +70,33 @@ func (c *Client) NC() *nats.Conn { return c.nc }
 
 // EnsureStreams creates the three JetStream streams if they don't
 // exist. M0: single-node, no replication.
+//
+// Retention: tuned for the M11 NATS resource-limit finding. The
+// 24h MaxAge on ALERTS allowed 6+ GiB of test data to accumulate
+// and exceed the server-level max_storage cap. With a 1h MaxAge
+// plus a 1 GiB MaxBytes safety cap, the stream self-trims long
+// before the server cap is hit. routerd is the only consumer and
+// processes in real time, so 1h is a generous safety window.
+//
+// See M11_NATS_INVESTIGATION.md for the full analysis.
 func (c *Client) EnsureStreams(ctx context.Context) error {
 	streams := []struct {
 		name     string
 		subjects []string
 		age      time.Duration
+		maxBytes int64
 	}{
-		{"ALERTS", []string{"alerts.>"}, 24 * time.Hour},
-		{"DELIVERIES", []string{"deliveries.>"}, 1 * time.Hour},
-		{"DLQ", []string{"dlq.>"}, 7 * 24 * time.Hour},
+		{"ALERTS", []string{"alerts.>"}, 1 * time.Hour, 1 << 30},       // 1h, 1 GiB
+		{"DELIVERIES", []string{"deliveries.>"}, 1 * time.Hour, 100 << 20}, // 1h, 100 MiB
+		{"DLQ", []string{"dlq.>"}, 1 * time.Hour, 10 << 20},          // 1h, 10 MiB
 	}
 	for _, s := range streams {
 		_, err := c.js.CreateOrUpdateStream(ctx, jetstream.StreamConfig{
 			Name:     s.name,
 			Subjects: s.subjects,
 			MaxAge:   s.age,
+			MaxBytes: s.maxBytes,
+			Discard:  jetstream.DiscardOld,
 			Storage:  jetstream.FileStorage,
 		})
 		if err != nil {