broad-announce/M6_VERIFICATION.md

M6 Verification — Dedupe + ×N suffix + sliding TTL + free-for-dupes

Status: shipped 2026-06-14 Branch: master Commits: see git log --oneline | grep M6

This milestone completes the dedupe work started in M0. The fixed 60s TTL is now a sliding window that refreshes on every duplicate observation. Duplicates propagate through the rate limit for free (no token burned). The dedupe_count is displayed in delivery messages as an inline (×N) suffix on the title. Per-source max-observed and a counter for "how loud is the dupe noise" are exposed as Prometheus metrics.

What landed

Area	Change
Algorithm	internal/dedupe/dedupe.go: single-Lua sliding-window script (was 2-RT SETNX+INCR)
TTL	BA_INGESTD_DEDUPE_TTL_SECONDS (default 300s, was 60s)
Rate limit	Dedupe runs BEFORE rate limit; only isNew=true burns a token in BOTH per-source and per-company buckets
Display	(×N) suffix on Telegram message title and FCM notification title when dedupe_count > 1
FCM data	New data.dedupe_count field for native clients to render
Metrics	ba_ingestd_dedupe_collapsed_total{source} (counter) and ba_ingestd_dedupe_count_max_observed{source} (gauge)
Loadgen	All three loadgens (http/mqtt/ws) gain --dedupe-key to force a specific key (used by the ×N smoke)

New env vars (ingestd)

Var	Default	Purpose
BA_INGESTD_DEDUPE_TTL_SECONDS	300	sliding-window TTL for a dedupe entry

Files

internal/dedupe/dedupe.go           # sliding-window Lua (was SETNX+INCR)
internal/dedupe/dedupe_test.go      # 6 tests: first/dup, isolation, empty-key,
                                    # window expires, sliding keeps alive, 1000-burst
internal/observability/maxseen.go   # sync.Map[source]uint32 monotonic max tracker
internal/observability/maxseen_test.go # 5 tests incl. concurrent CAS
internal/observability/metrics.go   # +2 metrics: dedupe_collapsed, dedupe_count_max
cmd/ingestd/process.go              # dedupe before rate limit, only charge new
cmd/ingestd/main.go                 # wires MaxSeen into processDeps
cmd/deliverd-fcm/main.go            # (×N) in notification title; data.dedupe_count
cmd/deliverd-telegram/main.go       # (×N) in title
loadgen/cmd/{http,mqtt,ws}/main.go  # --dedupe-key flag on all three
internal/config/config.go           # DedupeTTLSeconds
docker-compose.yml                  # BA_INGESTD_DEDUPE_TTL_SECONDS=300
.env.example                        # BA_INGESTD_DEDUPE_TTL_SECONDS=300

Algorithm (sliding window, atomic via Lua)

-- KEYS[1] = dedupe:{source_id}:{dedupe_key}
-- ARGV[1] = window in seconds (float; sub-second rounded up
--           so test windows like 500ms still work)
local ttl = tonumber(ARGV[1])
if ttl == nil or ttl <= 0 then
  return redis.error_reply("dedupe: invalid ttl")
end
local ttlInt = math.ceil(ttl)
if ttlInt < 1 then ttlInt = 1 end
local set = redis.call("SET", KEYS[1], 1, "NX", "EX", ttlInt)
if set then
  return {1, 1}                -- first arrival, isNew=1, count=1
end
local n = redis.call("INCR", KEYS[1])
redis.call("EXPIRE", KEYS[1], ttlInt)   -- <-- THE SLIDING PART
return {0, n}                          -- duplicate, isNew=0, count=n

The EXPIRE refresh on every duplicate is what makes the window slide. A steady stream of duplicates keeps the key alive indefinitely; a pause longer than the window expires the entry, and the next arrival is "new" again with count=1.

Scenarios (spec-style)

Step 2 — ×N suffix on the 5th message

Send 5 alerts with the same --dedupe-key. The faketgmd sent-log shows 5 distinct alert_ids, each with the dedupe _count climbing 1, 2, 3, 4, 5. The 5th message text contains the inline (×5) suffix on the title.

Asserted: faketgmd's last message text contains (×5).

Step 3 — dedupe metrics tick

The 5-burst from Step 2 should also tick:

• ba_ingestd_dedupe_collapsed_total{source="prom-prod"} +4 (one new + four dupes)
• ba_ingestd_dedupe_count_max_observed{source="prom-prod"} = 5

Asserted: collapsed delta ≥ 4 and max ≥ 5.

Step 4 — fresh key, all new

A new dedupe_key (timestamped) has no prior entry, so all 3 alerts are accepted as isNew=true. ws_messages_total{result="accepted"} +3.

Asserted: received=3, accepted=3.

Step 5 — free-for-dupes (200 dupes/s, cap=100/s)

The default per-source rate limit is 100/s. We send 600 alerts with the same dedupe_key at 300/s for 2s. The first alert is new and burns 1 token. The other 599 are dupes and do not burn any token. We assert that rate_limit_hits_total{scope="source"} did NOT tick, proving the rate limit gate is "free" for duplicates.

Asserted: rate_limit_hits_total{scope="source"} delta = 0; accepted delta ≥ 500.

Step 6 — per-source isolation

Same dedupe_key from acme (prom-prod) and globex (grafana) does NOT collide. After acme sends 3 and globex sends 3 with the same key, each source's max_observed gauge reads 3.

Asserted: grafana max ≥ 3, prom-prod max ≥ 1.

Run

cd /root/.openclaw/workspace/broad-announce
bash scripts/m6_smoke.sh

Exit code = number of failed checks (0 on success).

Curl sanity check (no test harness)

# Send 5 alerts with the same dedupe_key, see the (×5) suffix.
for i in 1 2 3 4 5; do
  BODY='{"company_id":"acme-001","source_id":"prom-prod","severity":"critical","title":"curl-burst","dedupe_key":"curl-burst-1"}'
  SIG="t=$(date +%s),v1=$(echo -n "$BODY" | openssl dgst -sha256 -hmac s3cret-acme | awk '{print $2}')"
  curl -sS -X POST http://localhost:8800/v1/ingest \
    -H "X-BA-Key: acme-001:prom-prod:s3cret-acme" \
    -H "X-BA-Signature: $SIG" \
    -H "Content-Type: application/json" \
    -d "$BODY"
  echo
done
# Then check the live tail or the faketgmd /admin/sent endpoint.

Out of scope for M6 (deferred)

• Router-level collapse — the recipient currently gets 5 distinct messages, each with a dedupe_count in the title. A "true single message per burst" requires the router to collapse on dedupe_key and only emit one delivery per source. That's M6.5 (the M6 spec hinted at it but it's bigger than the dedupe rewrite itself).
• Per-source overrides of max_concurrent_connections — the migration added the column but the M6 path doesn't yet read it. The default 32 from the env still applies.
• Hash-based dedupe_key — clients can opt out of supplying a key by sending a hash of the title+body and we'll derive the dedupe_key server-side. M11 (security milestone).
• Per-company dedupe_max_observed — the gauge is per-source only. A per-company rollup would be a small addition when Grafana dashboards land (M9).