Sfoglia il codice sorgente

fix: m11_smoke step 4 backpressure test — use docker compose run

The original step 4 inlined a Python subprocess.Popen that called
/app/loadgen-grpc directly on the host. That path only exists
inside the loadgen-grpc-1 image, not on the filesystem — every run
crashed with FileNotFoundError before the test even started.

Fix: spawn a one-shot container via 'docker compose --profile
loadgen-grpc run --rm loadgen-grpc-1' so it joins the compose
network (where 'ingestd' resolves correctly) and runs the binary
at /app/loadgen-grpc. Override the command with the backpressure
test's args (16 workers, --rate=16000, --dedupe-pct=0 so it
doesn't collide with the soak loadgen on the same source key,
--duration=20s, distinct --instance/--cluster-id labels for
Prometheus deduplication).

Also bumped the communicate timeout from 30s to 60s — docker
compose run has ~10s startup overhead before the binary starts,
and the loadgen needs to drain its ackCh buffer (~20s at the
~470/s per-stream cap) on shutdown.
Luis Rosales 1 mese fa
parent
commit
8da9fd24e1
1 ha cambiato i file con 23 aggiunte e 27 eliminazioni
  1. 23 27
      scripts/m11_smoke.py

+ 23 - 27
scripts/m11_smoke.py

@@ -170,37 +170,33 @@ def step4_backpressure_test() -> None:
     # The test validates that:
     #   a) No goroutine panics / connection drops under backpressure
     #   b) Rate-limited acks are received for the excess traffic
-    print("  starting 16-stream loadgen (10k/s total)...")
+    print("  starting 16-stream loadgen (16k/s total)...")
+    # Run a one-shot container that joins the compose network so the
+    # `ingestd` service name resolves. The /app/loadgen-grpc binary
+    # lives only inside the image — the original inline script tried
+    # to exec it on the host and FileNotFoundError'd.
     backpressure_proc = subprocess.Popen(
-        ["python3", "-c", f"""
-import subprocess, sys, time
-# Quick inline backpressure check
-# Use the grpc loadgen binary directly
-proc = subprocess.Popen(
-    ['/app/loadgen-grpc',
-     '--target=ingestd:9090',
-     '--api-key=acme-003:stress:s3cret-acme-003',
-     '--rate=10000',
-     '--workers=16',
-     '--duration=30s',
-     '--metrics=:8893'],
-    stdout=subprocess.DEVNULL,
-    stderr=subprocess.DEVNULL,
-)
-# Wait and check it stays up
-time.sleep(5)
-if proc.poll() is not None:
-    print('CRASHED', file=sys.stderr)
-    sys.exit(1)
-print('OK')
-proc.terminate()
-proc.wait()
-"""],
+        ["docker", "compose", "--profile", "loadgen-grpc", "run", "--rm",
+         "-e", "BA_LOG_LEVEL=info",
+         "loadgen-grpc-1",
+         "/app/loadgen-grpc",
+         "--target=ingestd:9090",
+         "--api-key=acme-001:acme-001-prom:s3cret-acme-001",
+         "--rate=16000",
+         "--workers=16",
+         "--dedupe-pct=0",
+         "--duration=20s",
+         "--metrics=:8893",
+         "--instance=loadgen-grpc-bp",
+         "--cluster-id=m11-backpressure"],
         stdout=subprocess.PIPE,
         stderr=subprocess.PIPE,
-        cwd="/root/broad-announce",
     )
-    stdout, stderr = backpressure_proc.communicate(timeout=30)
+    try:
+        stdout, stderr = backpressure_proc.communicate(timeout=60)
+    except subprocess.TimeoutExpired:
+        backpressure_proc.kill()
+        fail_("backpressure loadgen did not exit within 60s")
     if backpressure_proc.returncode != 0:
         fail_(f"backpressure loadgen exited unexpectedly: {stderr.decode().strip()}")
     pass_("16-stream backpressure loadgen ran without crashes")