|
|
@@ -596,3 +596,112 @@ Pushed: 3 commits on master mirroring M0–M4's pattern:
|
|
|
green runs, 13/13 checks each, +36 deliveries
|
|
|
cumulative)
|
|
|
3. M5(3/3): this PROMPT bump + README + SPEC §23.
|
|
|
+
|
|
|
+**2026-06-14 — M6 shipped (Dedupe + ×N suffix + sliding TTL + free-for-dupes)**
|
|
|
+
|
|
|
+What landed (~350 LoC Go + 1 SQL-friendly column + 2 new
|
|
|
+metrics across these commits):
|
|
|
+- `internal/dedupe/dedupe.go`: rewritten with a single Lua
|
|
|
+ script that does SET-NX-or-INCR-EXPIRE atomically. The
|
|
|
+ EXPIRE call on every duplicate is what makes the window
|
|
|
+ slide — a steady stream of duplicates keeps the key alive.
|
|
|
+ Sub-second windows in tests are handled by `math.ceil`
|
|
|
+ in Lua (Redis EX requires an int).
|
|
|
+- `internal/observability/maxseen.go`: a new
|
|
|
+ `MaxSeen` type that tracks per-key monotonic max. The
|
|
|
+ gauge API doesn't expose Get(), so the canonical
|
|
|
+ Prometheus pattern is to use a custom Collector; instead
|
|
|
+ we keep the max in a `sync.Map[source]uint32` and
|
|
|
+ `Set()` the gauge only on growth. CAS-safe under
|
|
|
+ concurrent writers.
|
|
|
+- `cmd/ingestd/process.go`: the order of operations is
|
|
|
+ now HMAC verify → **dedupe** → rate limit (per-source,
|
|
|
+ per-company). A duplicate (`isNew=false`) does not burn
|
|
|
+ a token in either bucket — the rate limit gate is
|
|
|
+ "free for dupes" as agreed. The duplicate still
|
|
|
+ propagates to NATS so the router can fan out the alert
|
|
|
+ with the updated `dedupe_count`. The M2 router, M3
|
|
|
+ deliverers, and the M5 tail hub all see the count climb.
|
|
|
+- `cmd/ingestd/main.go`: wires `observability.NewMaxSeen()`
|
|
|
+ into `processDeps` so HTTP, MQTT, and WS transports
|
|
|
+ share one in-process state and one max-observed gauge.
|
|
|
+- `cmd/deliverd-telegram/main.go`: `formatMessage` takes
|
|
|
+ a `dedupeCount uint32` and appends ` (×N)` to the title
|
|
|
+ when N > 1. Below 2, the title is unchanged.
|
|
|
+- `cmd/deliverd-fcm/main.go`: notification.title gets
|
|
|
+ the same suffix. The raw count also goes into
|
|
|
+ `data.dedupe_count` so a native Android client can
|
|
|
+ render it however it likes.
|
|
|
+- `loadgen/cmd/{http,mqtt,ws}/main.go`: all three
|
|
|
+ loadgens gain a `--dedupe-key` flag that forces a
|
|
|
+ specific key on every alert. This is what the ×N
|
|
|
+ smoke needs (one key, N copies, N=600 in the rate-limit
|
|
|
+ test).
|
|
|
+- `internal/config/config.go` + `docker-compose.yml`:
|
|
|
+ `BA_INGESTD_DEDUPE_TTL_SECONDS` env var, default 300s
|
|
|
+ (was 60s hard-coded).
|
|
|
+- `internal/observability/metrics.go`: 2 new metrics —
|
|
|
+ `ba_ingestd_dedupe_collapsed_total{source}` (counter
|
|
|
+ that ticks on every `isNew=false` hit) and
|
|
|
+ `ba_ingestd_dedupe_count_max_observed{source}` (gauge
|
|
|
+ that climbs to the highest dedupe_count seen for that
|
|
|
+ source since process start).
|
|
|
+
|
|
|
+Bug #1 — int-seconds truncation broke sub-second test windows
|
|
|
+The first dedupe test with a 500ms window failed: the Lua
|
|
|
+script received `0` for the TTL and returned an error
|
|
|
+("invalid ttl"). Go's `int(500ms.Seconds())` truncates to
|
|
|
+0, which is below the script's `> 0` guard. Fix: pass the
|
|
|
+TTL as a float (Go's `d.window.Seconds()` returns float64)
|
|
|
+and round up in Lua with `math.ceil`. Test sleeps were
|
|
|
+also bumped to 1.3s to account for the rounded TTL.
|
|
|
+
|
|
|
+Bug #2 — Prometheus Gauge doesn't expose Get()
|
|
|
+The "max dedupe count" gauge needs to only tick upward,
|
|
|
+but Prometheus Gauges can be Set() to any value. The
|
|
|
+canonical pattern is a custom Collector that knows the
|
|
|
+in-process state. That's a lot of boilerplate for one
|
|
|
+metric. The pragmatic alternative: keep the max in a
|
|
|
+sync.Map ourselves, and call `Set()` on the gauge only
|
|
|
+when the new value strictly exceeds the previous max.
|
|
|
+The MaxSeen type is thread-safe (CAS on conflict) and
|
|
|
+the gauge becomes a read-only view of our in-process
|
|
|
+state. 5 race-tested unit tests cover the corner cases.
|
|
|
+
|
|
|
+Decision: free-for-dupes applies to BOTH rate-limit buckets
|
|
|
+The user picked "both buckets free" over "per-source only"
|
|
|
+in the M6 design conversation. The reasoning: a duplicate
|
|
|
+is a duplicate, and the rate limit exists to backpressure
|
|
|
+*new* alert volume, not to charge *attempt* volume. A dupe
|
|
|
+storm of 600/s through a 100/s bucket now costs 1 token
|
|
|
+instead of 600. The downside (a bad source with a valid
|
|
|
+HMAC could pump dupes for free) doesn't apply because
|
|
|
+they already have full source-level access — rate limit
|
|
|
+isn't a security control, it's a backpressure mechanism.
|
|
|
+
|
|
|
+What's NOT in M6 (and not supposed to be):
|
|
|
+- Router-level collapse on `dedupe_key` — a true
|
|
|
+ "one message per burst" would require the router to
|
|
|
+ collapse before fanning out, instead of fanning out
|
|
|
+ N alerts with climbing dedupe_count. M6.5+ when the
|
|
|
+ M9 dashboard traffic pattern justifies it.
|
|
|
+- Per-source override of `max_concurrent_connections` —
|
|
|
+ the migration added the column but the M6 path still
|
|
|
+ reads the env default (32). M6.5 or M10.
|
|
|
+- Server-derived dedupe_key from a hash of (title+body+labels)
|
|
|
+ — currently the source must supply the key. M11 lets
|
|
|
+ sources opt in to a server-hash fallback for
|
|
|
+ duplicate-prone senders that don't compute their own key.
|
|
|
+- Per-company rollup of the max-observed gauge — the
|
|
|
+ gauge is per-source only. M9 dashboard work.
|
|
|
+
|
|
|
+Pushed: 3 commits on master mirroring M0–M5's pattern:
|
|
|
+1. M6(1/3): code (dedupe sliding-window Lua, MaxSeen,
|
|
|
+ metrics, process.go reorder, deliverer ×N suffix,
|
|
|
+ loadgen --dedupe-key, config + compose, 5 new
|
|
|
+ maxseen tests, 2 new dedupe tests)
|
|
|
+2. M6(2/3): verified (M6_VERIFICATION.md +
|
|
|
+ scripts/m6_smoke.sh + M6_SMOKE_LOG.md, 3 consecutive
|
|
|
+ green runs, 11/11 checks each, 600-alert dupe storm
|
|
|
+ confirms free-for-dupes, 0 rate-limit hits)
|
|
|
+3. M6(3/3): this PROMPT bump + README + SPEC §23.
|