Ver Fonte

fix(loadgen): main() exits when --duration expires, not just on SIGTERM

Before this fix, main()'s select only handled ctx.Done() (SIGTERM)
and the 5s progress ticker. When --duration expired, runWorker
exited cleanly (prodCtx fires -> producer closes ackCh -> consumer
range loop exits -> runWorker returns) but main sat forever in the
select waiting for a signal that never came. The smoke's step 3
backpressure test ran the loadgen with --duration=20s and saw the
loadgen hang for the full 60s smoke timeout, exiting with
'backpressure loadgen did not exit within 60s'. The smoke itself
was the only thing the loadgen was supposed to wait on, and the
smoke correctly reported the hang.

Fix: spawn a goroutine that calls wg.Wait() and closes a done
channel. Add a <-done case to the main select so workers
finishing naturally (--duration expired) triggers the same
printFinal + return path as a signal.

Verified on parres 2026-06-16: standalone backpressure loadgen
with --duration=15s now exits in 17s with exit code 0, sent
142,304 alerts at 9,487/s, p99 10-25ms, zero failures.
Luis Rosales há 1 mês atrás
pai
commit
39907d1ca3
1 ficheiros alterados com 35 adições e 8 exclusões
  1. 35 8
      loadgen/cmd/grpc/main.go

+ 35 - 8
loadgen/cmd/grpc/main.go

@@ -117,18 +117,45 @@ func main() {
 	// Status ticker.
 	ticker := time.NewTicker(5 * time.Second)
 	defer ticker.Stop()
+
+	// done is closed when every worker has returned (e.g. when
+	// --duration expires in runWorker via prodCtx.Done()). Without
+	// this, main only exits on SIGTERM and the loadgen hangs past
+	// --duration even though the workers themselves shut down cleanly
+	// (producer closes ackCh, consumer's `for range ackCh` exits,
+	// runWorker returns). Symptom: smoke step 3 times out at 60s
+	// with "backpressure loadgen did not exit within 60s".
+	done := make(chan struct{})
+	go func() {
+		wg.Wait()
+		close(done)
+	}()
+
 	for {
 		select {
 		case <-ctx.Done():
-			wg.Wait()
+			// Signal received: wait for in-flight workers to drain
+			// via the done channel, then print and return.
+			<-done
 			var tsent, tfailed, tdupes, trlHits uint64
-	for i := range results {
-		tsent += results[i].sent
-		tfailed += results[i].failed
-		tdupes += results[i].dupes
-		trlHits += results[i].rlHits
-	}
-	printFinal(logger, tsent, tfailed, tdupes, trlHits, &rttHist)
+			for i := range results {
+				tsent += results[i].sent
+				tfailed += results[i].failed
+				tdupes += results[i].dupes
+				trlHits += results[i].rlHits
+			}
+			printFinal(logger, tsent, tfailed, tdupes, trlHits, &rttHist)
+			return
+		case <-done:
+			// All workers finished naturally (e.g. --duration expired).
+			var tsent, tfailed, tdupes, trlHits uint64
+			for i := range results {
+				tsent += results[i].sent
+				tfailed += results[i].failed
+				tdupes += results[i].dupes
+				trlHits += results[i].rlHits
+			}
+			printFinal(logger, tsent, tfailed, tdupes, trlHits, &rttHist)
 			return
 		case <-ticker.C:
 			// Aggregate per-worker counters.