# M6 Verification — Dedupe + ×N suffix + sliding TTL + free-for-dupes

**Status:** shipped 2026-06-14
**Branch:** master
**Commits:** see `git log --oneline | grep M6`

This milestone completes the dedupe work started in M0. The
fixed 60s TTL is now a sliding window that refreshes on every
duplicate observation. Duplicates propagate through the rate
limit for free (no token burned). The `dedupe_count` is
displayed in delivery messages as an inline `(×N)` suffix on
the title. Per-source max-observed and a counter for "how
loud is the dupe noise" are exposed as Prometheus metrics.

## What landed

| Area | Change |
| --- | --- |
| Algorithm | `internal/dedupe/dedupe.go`: single-Lua sliding-window script (was 2-RT SETNX+INCR) |
| TTL | `BA_INGESTD_DEDUPE_TTL_SECONDS` (default 300s, was 60s) |
| Rate limit | Dedupe runs BEFORE rate limit; only `isNew=true` burns a token in BOTH per-source and per-company buckets |
| Display | `(×N)` suffix on Telegram message title and FCM notification title when `dedupe_count > 1` |
| FCM data | New `data.dedupe_count` field for native clients to render |
| Metrics | `ba_ingestd_dedupe_collapsed_total{source}` (counter) and `ba_ingestd_dedupe_count_max_observed{source}` (gauge) |
| Loadgen | All three loadgens (http/mqtt/ws) gain `--dedupe-key` to force a specific key (used by the ×N smoke) |

## New env vars (ingestd)

| Var | Default | Purpose |
| --- | --- | --- |
| `BA_INGESTD_DEDUPE_TTL_SECONDS` | 300 | sliding-window TTL for a dedupe entry |

## Files

```
internal/dedupe/dedupe.go           # sliding-window Lua (was SETNX+INCR)
internal/dedupe/dedupe_test.go      # 6 tests: first/dup, isolation, empty-key,
                                    # window expires, sliding keeps alive, 1000-burst
internal/observability/maxseen.go   # sync.Map[source]uint32 monotonic max tracker
internal/observability/maxseen_test.go # 5 tests incl. concurrent CAS
internal/observability/metrics.go   # +2 metrics: dedupe_collapsed, dedupe_count_max
cmd/ingestd/process.go              # dedupe before rate limit, only charge new
cmd/ingestd/main.go                 # wires MaxSeen into processDeps
cmd/deliverd-fcm/main.go            # (×N) in notification title; data.dedupe_count
cmd/deliverd-telegram/main.go       # (×N) in title
loadgen/cmd/{http,mqtt,ws}/main.go  # --dedupe-key flag on all three
internal/config/config.go           # DedupeTTLSeconds
docker-compose.yml                  # BA_INGESTD_DEDUPE_TTL_SECONDS=300
.env.example                        # BA_INGESTD_DEDUPE_TTL_SECONDS=300
```

## Algorithm (sliding window, atomic via Lua)

```lua
-- KEYS[1] = dedupe:{source_id}:{dedupe_key}
-- ARGV[1] = window in seconds (float; sub-second rounded up
--           so test windows like 500ms still work)
local ttl = tonumber(ARGV[1])
if ttl == nil or ttl <= 0 then
  return redis.error_reply("dedupe: invalid ttl")
end
local ttlInt = math.ceil(ttl)
if ttlInt < 1 then ttlInt = 1 end
local set = redis.call("SET", KEYS[1], 1, "NX", "EX", ttlInt)
if set then
  return {1, 1}                -- first arrival, isNew=1, count=1
end
local n = redis.call("INCR", KEYS[1])
redis.call("EXPIRE", KEYS[1], ttlInt)   -- <-- THE SLIDING PART
return {0, n}                          -- duplicate, isNew=0, count=n
```

The `EXPIRE` refresh on every duplicate is what makes the
window slide. A steady stream of duplicates keeps the key
alive indefinitely; a pause longer than the window expires
the entry, and the next arrival is "new" again with count=1.

## Scenarios (spec-style)

### Step 2 — ×N suffix on the 5th message
Send 5 alerts with the same `--dedupe-key`. The faketgmd
sent-log shows 5 distinct alert_ids, each with the dedupe
_count climbing 1, 2, 3, 4, 5. The 5th message text contains
the inline `(×5)` suffix on the title.

**Asserted:** faketgmd's last message text contains `(×5)`.

### Step 3 — dedupe metrics tick
The 5-burst from Step 2 should also tick:
- `ba_ingestd_dedupe_collapsed_total{source="prom-prod"}` +4
  (one new + four dupes)
- `ba_ingestd_dedupe_count_max_observed{source="prom-prod"}` = 5

**Asserted:** collapsed delta ≥ 4 and max ≥ 5.

### Step 4 — fresh key, all new
A new dedupe_key (timestamped) has no prior entry, so all 3
alerts are accepted as `isNew=true`. `ws_messages_total{result="accepted"}` +3.

**Asserted:** received=3, accepted=3.

### Step 5 — free-for-dupes (200 dupes/s, cap=100/s)
The default per-source rate limit is 100/s. We send 600
alerts with the same dedupe_key at 300/s for 2s. The first
alert is new and burns 1 token. The other 599 are dupes
and **do not burn any token**. We assert that
`rate_limit_hits_total{scope="source"}` did NOT tick,
proving the rate limit gate is "free" for duplicates.

**Asserted:** `rate_limit_hits_total{scope="source"}` delta = 0; accepted delta ≥ 500.

### Step 6 — per-source isolation
Same dedupe_key from acme (`prom-prod`) and globex
(`grafana`) does NOT collide. After acme sends 3 and globex
sends 3 with the same key, each source's `max_observed`
gauge reads 3.

**Asserted:** grafana max ≥ 3, prom-prod max ≥ 1.

## Run

```
cd /root/.openclaw/workspace/broad-announce
bash scripts/m6_smoke.sh
```

Exit code = number of failed checks (0 on success).

## Curl sanity check (no test harness)

```bash
# Send 5 alerts with the same dedupe_key, see the (×5) suffix.
for i in 1 2 3 4 5; do
  BODY='{"company_id":"acme-001","source_id":"prom-prod","severity":"critical","title":"curl-burst","dedupe_key":"curl-burst-1"}'
  SIG="t=$(date +%s),v1=$(echo -n "$BODY" | openssl dgst -sha256 -hmac s3cret-acme | awk '{print $2}')"
  curl -sS -X POST http://localhost:8800/v1/ingest \
    -H "X-BA-Key: acme-001:prom-prod:s3cret-acme" \
    -H "X-BA-Signature: $SIG" \
    -H "Content-Type: application/json" \
    -d "$BODY"
  echo
done
# Then check the live tail or the faketgmd /admin/sent endpoint.
```

## Out of scope for M6 (deferred)

- **Router-level collapse** — the recipient currently gets 5
  distinct messages, each with a `dedupe_count` in the title.
  A "true single message per burst" requires the router to
  collapse on `dedupe_key` and only emit one delivery per
  source. That's M6.5 (the M6 spec hinted at it but it's
  bigger than the dedupe rewrite itself).
- **Per-source overrides of `max_concurrent_connections`** —
  the migration added the column but the M6 path doesn't yet
  read it. The default 32 from the env still applies.
- **Hash-based dedupe_key** — clients can opt out of supplying
  a key by sending a hash of the title+body and we'll derive
  the dedupe_key server-side. M11 (security milestone).
- **Per-company dedupe_max_observed** — the gauge is per-source
  only. A per-company rollup would be a small addition when
  Grafana dashboards land (M9).