broad-announce/M6.5_VERIFICATION.md

M6.5 Verification — Router-level dedupe collapse

Status: shipped 2026-06-14 Branch: master Commits: see git log --oneline | grep M6.5

This is the follow-on to M6. M6 was about deduplication at ingest (sliding TTL, free-for-dupes rate limit, ×N display). M6.5 deduplicates at delivery: a burst of 100 identical alerts now produces 1 message to the recipient, not 100 (the last tagged ×100). The recipient gets the final dedupe_count on the single message.

What landed

Area	Change
Algorithm	internal/dedupe/collapser.go: per-(source, key) debounce with max-wait (2s default)
Pipeline	cmd/routerd/collapse.go: Collapser wired between alert ingest and delivery fanout; cached targets skip duplicate ResolveTargets DB calls
Config	BA_ROUTERD_DEDUPE_FLUSH_MS (default 2000ms)
Passthrough	Alerts with empty dedupe_key skip the Collapser entirely (M2 behavior preserved)
Operator visibility	Tail WS still sees every arrival (operators see the storm live, even though recipient gets 1)

New env var (routerd)

Var	Default	Purpose
BA_ROUTERD_DEDUPE_FLUSH_MS	2000	max-wait window for the in-router collapse; a continuous stream re-flushes every window

Files

internal/dedupe/collapser.go          # Collapser with Passthrough/CollapseNew/CollapseDupe
internal/dedupe/collapser_test.go     # 7 tests: passthrough, single collapse, per-source,
                                      #   re-flush, empty-key no-block, concurrent storm, FlushAll
cmd/routerd/collapse.go               # wires the Collapser; cache targets by (source, key)
cmd/routerd/main.go                   # refactored handleOne; Run goroutine; FlushAll on shutdown
testfakes/tailcount/main.go           # tail subscriber for the smoke (counts events)
internal/config/config.go             # Routerd struct + LoadRouterd + DedupeFlushMs
docker-compose.yml                    # BA_ROUTERD_DEDUPE_FLUSH_MS=2000
.env.example                          # same env var
loadgen/cmd/ws/main.go                # makeAlert injects key into title for tail filterability
M6.5_VERIFICATION.md
M6.5_SMOKE_LOG.md
scripts/m6.5_smoke.sh

Algorithm

On alert arrival (source, dedupe_key, alert):
  if dedupe_key is empty:
    resolve + publish immediately (M2 path)
  else:
    dec := Collapser.Observe(source, dedupe_key, alert)
    if dec == CollapseNew:
      resolve targets NOW and cache them
      ack the NATS message
    else (CollapseDupe):
      stored alert's dedupe_count already updated by Collapser
      ack the NATS message

On Collapser flush (every 2s):
  for each (source, key) where firstSeen + flushMs <= now:
    publishDeliveries(logger, br, a.CompanyID, a, cachedTargets)
    remove from pending + cachedTargets

Design choices

1. Router-side, not ingest-side — ingestd has to publish to NATS unconditionally (so the alert hits durable storage
   • tail subscribers). The collapse has to happen between NATS and the deliverers, which means routerd.
2. Debounce with max-wait — once the burst is quiet for 2s, flush. A continuous stream re-flushes every 2s. This gives operators the live feed (via tail) and the recipient the consolidated "×N" message.
3. Cache resolved targets — the first arrival does the DB-bound ResolveTargets call. Subsequent dupes update the dedupe_count on the stored alert and return. The flush reads the cached targets. Net: 100-alert burst produces 1 DB call, not 100.
4. Per-source isolation — the Collapser keys on (source_id, dedupe_key), not on dedupe_key alone. Two sources bursting with the same key don't collapse into one (matches M6 semantics).
5. Trust the ingestd's count — the router doesn't INCR the dedupe count itself; it takes the max of the stored and incoming values. Ingestd is the canonical counter; this avoids a second Redis round-trip per arrival.
6. Graceful shutdown — collapser.FlushAll() is called on routerd shutdown, so the last few pending collapses still get delivered.

Scenarios (spec-style)

Step 2 — single collapse (100 alerts → 1 message)

Send 100 alerts with the same --dedupe-key at 50/s. After the 2s flush window, faketgmd's /admin/sent shows count=1, with the message text containing (×100).

Asserted: /admin/sent count = 1; text contains (×100).

Step 3 — tail still shows the storm

The WS tail subscriber receives ~all 25 alerts of a smaller burst (filtered to a unique title prefix), even though the recipient only got 1 message. The tail intentionally bypasses the Collapser.

Asserted: tail count ≥20 of 25; faketgmd count = 1.

Step 4 — max-wait re-flush (continuous stream)

A 6s continuous burst at 30/s (180 alerts) re-flushes every 2s. The recipient gets 2-5 messages (typically 3), each carrying the running count.

Asserted: faketgmd count is between 2 and 5.

Step 5 — per-source isolation

Same --dedupe-key from acme (prom-prod) and globex (grafana) does NOT collapse. The Collapser's state is keyed on (source_id, dedupe_key), so each source gets its own collapse window. Result: 2 distinct chat_ids each receive 1 message.

Asserted: faketgmd count = 2; chat_ids = "1001,2001".

Step 6 — passthrough (no-key alerts)

The loadgen's normal profile gives each alert a unique key (lg-m5-0, lg-m5-1, lg-m5-shared). Each is a new (source, key) for the Collapser, so each gets a dedicated message. 3 alerts → 3 deliveries.

Asserted: faketgmd count = 3.

Run

cd /root/.openclaw/workspace/broad-announce
bash scripts/m6.5_smoke.sh

Exit code = number of failed checks (0 on success).

Out of scope for M6.5 (deferred)

• Per-source override of BA_ROUTERD_DEDUPE_FLUSH_MS — a noisy source might want a longer window. The migration added the column but the M6.5 path uses the env default for all sources. M6.5.5 or M7 (with the per-source config dashboard).
• Collapse across sources — currently per-source. A "global" collapse (across all sources with the same dedupe_key) could be a config flag. But it has bad blast-radius properties (one noisy source could shadow another), so we ship per-source-only.
• Tombstone on collapse — the collapsed delivery currently looks like a normal alert. A future M7 could add a collapsed_count: N field to the payload so deliverers can render "5 messages collapsed" in addition to the (×N) suffix.
• Active collapse metrics — we have no Prometheus metric for "how many collapses happened" or "how many alerts were collapsed". M7 dashboard work.