Browse Source

M6(3/3): README + PROMPT + SPEC bump to 'shipped 2026-06-14'

README: M6 in the status banner; M6_VERIFICATION +
M6_SMOKE_LOG in the docs index; ingestd comment updated
to mention 'dedupe before rate limit (M6)'.

PROMPT: full M6 section appended. The two real bugs found
during verification (int-seconds truncation, Prometheus
Gauge has no Get()) are documented inline. The 'free-for-
dupes applies to BOTH buckets' decision is captured with
its reasoning. 'What's NOT in M6' lists the deferred
items (router-level collapse, server-derived dedupe_key,
per-company max rollup).

SPEC §23: M6 row updated to '✅ shipped 2026-06-14' with
the smoke summary inline.

Push: 3 commits ahead of origin/master, ready to push.
Luis Rosales 1 tháng trước cách đây
mục cha
commit
6f7c8174cf
3 tập tin đã thay đổi với 149 bổ sung32 xóa
  1. 109 0
      PROMPT.md
  2. 39 31
      README.md
  3. 1 1
      SPEC.md

+ 109 - 0
PROMPT.md

@@ -596,3 +596,112 @@ Pushed: 3 commits on master mirroring M0–M4's pattern:
    green runs, 13/13 checks each, +36 deliveries
    cumulative)
 3. M5(3/3): this PROMPT bump + README + SPEC §23.
+
+**2026-06-14 — M6 shipped (Dedupe + ×N suffix + sliding TTL + free-for-dupes)**
+
+What landed (~350 LoC Go + 1 SQL-friendly column + 2 new
+metrics across these commits):
+- `internal/dedupe/dedupe.go`: rewritten with a single Lua
+  script that does SET-NX-or-INCR-EXPIRE atomically. The
+  EXPIRE call on every duplicate is what makes the window
+  slide — a steady stream of duplicates keeps the key alive.
+  Sub-second windows in tests are handled by `math.ceil`
+  in Lua (Redis EX requires an int).
+- `internal/observability/maxseen.go`: a new
+  `MaxSeen` type that tracks per-key monotonic max. The
+  gauge API doesn't expose Get(), so the canonical
+  Prometheus pattern is to use a custom Collector; instead
+  we keep the max in a `sync.Map[source]uint32` and
+  `Set()` the gauge only on growth. CAS-safe under
+  concurrent writers.
+- `cmd/ingestd/process.go`: the order of operations is
+  now HMAC verify → **dedupe** → rate limit (per-source,
+  per-company). A duplicate (`isNew=false`) does not burn
+  a token in either bucket — the rate limit gate is
+  "free for dupes" as agreed. The duplicate still
+  propagates to NATS so the router can fan out the alert
+  with the updated `dedupe_count`. The M2 router, M3
+  deliverers, and the M5 tail hub all see the count climb.
+- `cmd/ingestd/main.go`: wires `observability.NewMaxSeen()`
+  into `processDeps` so HTTP, MQTT, and WS transports
+  share one in-process state and one max-observed gauge.
+- `cmd/deliverd-telegram/main.go`: `formatMessage` takes
+  a `dedupeCount uint32` and appends ` (×N)` to the title
+  when N > 1. Below 2, the title is unchanged.
+- `cmd/deliverd-fcm/main.go`: notification.title gets
+  the same suffix. The raw count also goes into
+  `data.dedupe_count` so a native Android client can
+  render it however it likes.
+- `loadgen/cmd/{http,mqtt,ws}/main.go`: all three
+  loadgens gain a `--dedupe-key` flag that forces a
+  specific key on every alert. This is what the ×N
+  smoke needs (one key, N copies, N=600 in the rate-limit
+  test).
+- `internal/config/config.go` + `docker-compose.yml`:
+  `BA_INGESTD_DEDUPE_TTL_SECONDS` env var, default 300s
+  (was 60s hard-coded).
+- `internal/observability/metrics.go`: 2 new metrics —
+  `ba_ingestd_dedupe_collapsed_total{source}` (counter
+  that ticks on every `isNew=false` hit) and
+  `ba_ingestd_dedupe_count_max_observed{source}` (gauge
+  that climbs to the highest dedupe_count seen for that
+  source since process start).
+
+Bug #1 — int-seconds truncation broke sub-second test windows
+The first dedupe test with a 500ms window failed: the Lua
+script received `0` for the TTL and returned an error
+("invalid ttl"). Go's `int(500ms.Seconds())` truncates to
+0, which is below the script's `> 0` guard. Fix: pass the
+TTL as a float (Go's `d.window.Seconds()` returns float64)
+and round up in Lua with `math.ceil`. Test sleeps were
+also bumped to 1.3s to account for the rounded TTL.
+
+Bug #2 — Prometheus Gauge doesn't expose Get()
+The "max dedupe count" gauge needs to only tick upward,
+but Prometheus Gauges can be Set() to any value. The
+canonical pattern is a custom Collector that knows the
+in-process state. That's a lot of boilerplate for one
+metric. The pragmatic alternative: keep the max in a
+sync.Map ourselves, and call `Set()` on the gauge only
+when the new value strictly exceeds the previous max.
+The MaxSeen type is thread-safe (CAS on conflict) and
+the gauge becomes a read-only view of our in-process
+state. 5 race-tested unit tests cover the corner cases.
+
+Decision: free-for-dupes applies to BOTH rate-limit buckets
+The user picked "both buckets free" over "per-source only"
+in the M6 design conversation. The reasoning: a duplicate
+is a duplicate, and the rate limit exists to backpressure
+*new* alert volume, not to charge *attempt* volume. A dupe
+storm of 600/s through a 100/s bucket now costs 1 token
+instead of 600. The downside (a bad source with a valid
+HMAC could pump dupes for free) doesn't apply because
+they already have full source-level access — rate limit
+isn't a security control, it's a backpressure mechanism.
+
+What's NOT in M6 (and not supposed to be):
+- Router-level collapse on `dedupe_key` — a true
+  "one message per burst" would require the router to
+  collapse before fanning out, instead of fanning out
+  N alerts with climbing dedupe_count. M6.5+ when the
+  M9 dashboard traffic pattern justifies it.
+- Per-source override of `max_concurrent_connections` —
+  the migration added the column but the M6 path still
+  reads the env default (32). M6.5 or M10.
+- Server-derived dedupe_key from a hash of (title+body+labels)
+  — currently the source must supply the key. M11 lets
+  sources opt in to a server-hash fallback for
+  duplicate-prone senders that don't compute their own key.
+- Per-company rollup of the max-observed gauge — the
+  gauge is per-source only. M9 dashboard work.
+
+Pushed: 3 commits on master mirroring M0–M5's pattern:
+1. M6(1/3): code (dedupe sliding-window Lua, MaxSeen,
+   metrics, process.go reorder, deliverer ×N suffix,
+   loadgen --dedupe-key, config + compose, 5 new
+   maxseen tests, 2 new dedupe tests)
+2. M6(2/3): verified (M6_VERIFICATION.md +
+   scripts/m6_smoke.sh + M6_SMOKE_LOG.md, 3 consecutive
+   green runs, 11/11 checks each, 600-alert dupe storm
+   confirms free-for-dupes, 0 rate-limit hits)
+3. M6(3/3): this PROMPT bump + README + SPEC §23.

+ 39 - 31
README.md

@@ -7,7 +7,7 @@ normalizes them, resolves recipients via `companies` → `groups` →
 Telegram, SMS, email, voice, Slack, MS Teams, and arbitrary outbound
 webhooks.
 
-> **Status**: M0 + M1 + M2 + M3 + M4 + M5 **shipped** 2026-06-14. M0 is the
+> **Status**: M0 + M1 + M2 + M3 + M4 + M5 + M6 **shipped** 2026-06-14. M0 is the
 > single-host docker-compose stack + 4 Go services + loadgen-http
 > + alert schema. M1 is the end-to-end: signed webhook → broker →
 > router → deliverd-fcm → fakefcmd (live-verified, 1530 deliveries
@@ -25,11 +25,17 @@ webhooks.
 > 12 deliveries in 5-step smoke with 0 failures). M5 is
 > WebSocket ingest (`GET /v1/ingest/ws` with `{api_key}` auth
 > frame, 35-conn load test, per-IP cap 32) + live tail
-> (`GET /v1/tail/ws?token=...&company_id=...` for operators,
+> (`GET /v1/tail/ws?token=***&company_id=...` for operators,
 > in-process pub/sub fan-out, 13/13 checks green across 3
-> consecutive smoke runs).
-> See `M0_VERIFICATION.md` … `M5_VERIFICATION.md` and
-> `M1_SMOKE_LOG.md` … `M5_SMOKE_LOG.md` for the smoke tests.
+> consecutive smoke runs). M6 is dedupe + ×N display
+> (sliding-window TTL via single Lua script, default 300s;
+> dedupe runs before rate limit so duplicates are "free";
+> recipient sees `(×N)` inline suffix on the title;
+> per-source `dedupe_collapsed_total` and
+> `dedupe_count_max_observed` metrics, 11/11 checks green
+> across 3 consecutive smoke runs).
+> See `M0_VERIFICATION.md` … `M6_VERIFICATION.md` and
+> `M1_SMOKE_LOG.md` … `M6_SMOKE_LOG.md` for the smoke tests.
 > Spec is in `SPEC.md`, diagrams in `ARCHITECTURE.md`, build log
 > in `PROMPT.md`.
 
@@ -45,8 +51,8 @@ Compose, 50k/sec design ceiling for v2 K8s.
 ## Repo layout
 
 ```
-SPEC.md              requirements, entities, severity, retention
-ARCHITECTURE.md      diagrams, sequences, SLOs, capacity model
+SPEC.md             - requirements, entities, severity, retention
+ARCHITECTURE.md     - diagrams, sequences, SLOs, capacity model
 PROMPT.md           — build log, decisions, open questions
 M0_VERIFICATION.md  — M0 smoke test (signed webhook → 202)
 M1_VERIFICATION.md  — M1 smoke test (end-to-end → fakefcmd)
@@ -54,37 +60,39 @@ M2_VERIFICATION.md  — M2 smoke test (recipient resolution)
 M3_VERIFICATION.md  — M3 smoke test (Telegram delivery + bot)
 M4_VERIFICATION.md  — M4 smoke test (MQTT ingest + EMQX ACL)
 M5_VERIFICATION.md  — M5 smoke test (WS ingest + live tail + per-IP cap)
+M6_VERIFICATION.md  — M6 smoke test (dedupe + ×N + sliding TTL + free-for-dupes)
 M1_SMOKE_LOG.md     — M1 live run results
 M2_SMOKE_LOG.md     — M2 live run results
 M3_SMOKE_LOG.md     — M3 live run results
 M4_SMOKE_LOG.md     — M4 live run results
 M5_SMOKE_LOG.md     — M5 live run results (3 consecutive green)
-docker-compose.yml  — single-host M0–M5 stack
+M6_SMOKE_LOG.md     — M6 live run results (3 consecutive green)
+docker-compose.yml  — single-host M0–M6 stack
 Dockerfile          — multi-stage build for all 7 binaries
 .env.example        — every BA_* knob documented
-cmd/ingestd/        — HTTP POST handler (M0) + MQTT subscriber (M4) + WS ingest (M5); M11 = TLS
-cmd/routerd/         M2 rules engine + M3 channel union
-cmd/deliverd-fcm/    M1 FCM HTTP v1 delivery (renamed from deliverd M3)
-cmd/deliverd-telegram/  M3 Telegram Bot API delivery
-cmd/telegramd/       M3 long-polling bot loop + command handler
-cmd/admind/          scaffold + /v1/ping (M8)
-loadgen/cmd/http/    HTTP traffic generator (M0)
-loadgen/cmd/mqtt/    MQTT traffic generator (M4)
-loadgen/cmd/ws/      WebSocket traffic generator (M5)
-internal/alert/      Alert v1 type + Validate() + Severity.Rank
-internal/broker/     NATS JetStream wrapper
-internal/config/     env-driven config
-internal/dedupe/     60s SET NX EX + INCR
-internal/ratelimit/  per-second INCR bucket
-internal/httpserver/  /health + /metrics scaffold
-internal/observability/  slog + Prometheus
-internal/postgres/   pgxpool wrapper
-internal/routing/    Resolver (M2 rules engine, M3 channel union)
-internal/telegram/   BotClient + command parser + handler
-internal/store/      Redis + (later) Postgres
-deploy/prometheus/   prometheus.yml
-migrations/         — 001–004 + seed/seed_m2/seed_m3.sql
-testfakes/           fakefcmd (M1), faketgmd (M3)
+cmd/ingestd/        — HTTP POST handler (M0) + MQTT subscriber (M4) + WS ingest (M5) + dedupe before rate limit (M6); M11 = TLS
+cmd/routerd/        - M2 rules engine + M3 channel union
+cmd/deliverd-fcm/   - M1 FCM HTTP v1 delivery (renamed from deliverd M3)
+cmd/deliverd-telegram/ - M3 Telegram Bot API delivery
+cmd/telegramd/      - M3 long-polling bot loop + command handler
+cmd/admind/         - scaffold + /v1/ping (M8)
+loadgen/cmd/http/   - HTTP traffic generator (M0)
+loadgen/cmd/mqtt/   - MQTT traffic generator (M4)
+loadgen/cmd/ws/     - WebSocket traffic generator (M5)
+internal/alert/     - Alert v1 type + Validate() + Severity.Rank
+internal/broker/    - NATS JetStream wrapper
+internal/config/    - env-driven config
+internal/dedupe/    - 60s SET NX EX + INCR
+internal/ratelimit/ - per-second INCR bucket
+internal/httpserver/ - /health + /metrics scaffold
+internal/observability/ - slog + Prometheus
+internal/postgres/  - pgxpool wrapper
+internal/routing/   - Resolver (M2 rules engine, M3 channel union)
+internal/telegram/  - BotClient + command parser + handler
+internal/store/     - Redis + (later) Postgres
+deploy/prometheus/  - prometheus.yml
+migrations/         - 001-004 + seed/seed_m2/seed_m3.sql
+testfakes/          - fakefcmd (M1), faketgmd (M3)
 ```
 
 ## License

+ 1 - 1
SPEC.md

@@ -888,7 +888,7 @@ ingestd_rejection_latency_seconds_bucket{transport,reason} histogram
 | M3 | Telegram delivery + bot commands | user can `/subscribe` and receive an alert via Telegram | **✅ shipped 2026-06-14** (live smoke test all 8 steps green; 14 deliveries, 8 sendMessage calls; see `M3_VERIFICATION.md` + `M3_SMOKE_LOG.md`) |
 | M4 | MQTT ingest | EMQX up, QoS 1, per-company topic ACLs | **✅ shipped 2026-06-14** (live smoke test all 5 steps green; 12 deliveries, 0 failures; see `M4_VERIFICATION.md` + `M4_SMOKE_LOG.md`) |
 | M5 | WebSocket ingest + live tail | admin UI (or wscat) sees alerts as they arrive; layer 2 in | **✅ shipped 2026-06-14** (live smoke test all 7 steps green; 3 consecutive green runs; 13/13 checks each; +36 deliveries cumulative; see `M5_VERIFICATION.md` + `M5_SMOKE_LOG.md`) |
-| M6 | Dedupe + dedupe_count | burst of 100 identical alerts → recipient sees "×100"; dedupe-aware rate shaping in |
+| M6 | Dedupe + dedupe_count | burst of 100 identical alerts → recipient sees "×100"; dedupe-aware rate shaping in | **✅ shipped 2026-06-14** (live smoke test all 6 steps green; 3 consecutive green runs; 11/11 checks each; sliding-window Lua + 600-alert dupe storm confirms free-for-dupes; see `M6_VERIFICATION.md` + `M6_SMOKE_LOG.md`) |
 | M7 | Timescale + ClickHouse | 7d retention + archive job |
 | M8 | DLQ + replay UI | operator can replay a failed delivery |
 | M9 | Observability (Prom/Grafana) | 1 dashboard per tier + per-company drilldown; layers 6, 7 in |