broad-announce/PROMPT.md

Broad-Announce — Build Log (PROMPT)

Decisions, lessons, blockers. Append-only. Update as we go.

2026-06-13 — kickoff

Decided

• Repo at git3.techno-world.net/lrosales/broad-announce (private).
• v1 stack: Go, PostgreSQL + Timescale, ClickHouse, NATS JetStream, Redis, EMQX (MQTT), Prometheus + Grafana, Loki. Deploy v1 = Docker Compose. v2 = K8s.
• Multi-tenancy = shared infrastructure, strict app-level isolation. No row-level security in v1; tenant filter on every query.
• FCM = single shared project for v1, schema supports per-company FCM project later (companies.fcm_shared).
• Telegram is a first-class delivery channel + a management UI (the bot is how users mute, subscribe, acknowledge).
• Severity taxonomy: info | warning | critical | inminent_colapse. Only inminent_colapse bypasses quiet hours.
• Dedupe: 60s window per (source_id, dedupe_key), attach dedupe_count so user sees "×N in 60s" not N pushes.
• v1 capacity target in docker-compose: 5k alerts/sec sustained. 50k/sec is the design ceiling; the K8s + multi-broker work is what unlocks it.

Open (was) → Resolved 2026-06-13

• Bot ↔ individual linking → admin-invites only. Flow: admin creates individuals row + generates one-time invite code; user runs /start <invite_code> in the Telegram bot; bot matches the code, links telegram_chat_id to the pre-existing individual, and burns the code. Stricter path: prevents drive-by bot self-registration, keeps individuals auditable.
• Localized titles → source-localized, pass through. v1 does NOT translate. Sources send pre-localized title/body strings (or use the locale on the subscription to key into their own lookup table before calling our API). Alert schema carries locale and title/body already-resolved. We can add a translation layer in v2 if customers ask.
• mTLS: schema supports it, but the docker-compose profile won't terminate client certs in v1. Documented as opt-in for enterprise sources.

Lessons (already)

• It's much cheaper to answer "what can this be?" with 30 questions than to refactor later. Most of the SPEC's weight is in §11 (security) and §6 (recipient resolution) — those are the parts that are expensive to change after launch.
• For multi-tenant at 10k companies, the one design choice that compounds is the subject layout in the broker. alerts.<company_id> is fine; alerts.<company_id>.<source_id> would let us scale router consumer groups per source. Locked in §6 of ARCHITECTURE.md.

2026-06-13 — added gRPC ingest (option a)

• Decision: ship gRPC bidi-streaming as the fourth ingest protocol, scoped to internal high-volume sources only. Public SaaS webhooks stay on HTTP POST, browsers stay on WebSocket, IoT stays on MQTT.
• Rationale: typed schemas, HTTP/2 + protobuf, native backpressure, no reconnect-loop code, generated Go/Java/Python/Node clients. Cost: one more protocol to operate + a buf generate build step.
• Gated on M11, after M10 (load test). M10 now caps at 5k/s on docker-compose (not 50k/s) — 50k/s is the design ceiling that requires K8s + multi-broker (M12 in v2).
• StreamAlerts(Alert) → Ack carries the same dedupe_count contract as HTTP/WS/MQTT.
• Proto lives at proto/broadannounce/v1/ingest.proto; server stub in internal/grpcserver/, reusable client in internal/grpcclient/. Auth = API key in metadata + optional mTLS. Per-stream rate limit + 256 in-flight cap = natural backpressure.

2026-06-13 — M10 split: option C

• M10 (in-cluster, real) stays at 5k/s on docker-compose.
• Added M10-bench: 50k/s against the broker + router with the delivery tier stubbed. Proves the ceiling without committing to K8s. Green here means K8s is implementable, not required.
• Implication: we need traffic generators that can hit the in-cluster path at 5k/s (M10) and the broker+router path at 50k/s (M10-bench) with the same source-protocol clients we ship. The loadgen/ tool is the new home for that.

2026-06-13 — three open questions resolved

• (Q1) Yes, build the fake-FCM / fake-Telegram / fake-SMS servers (testfakes/). M10 must not burn 50k FCM credits.
• (Q2) M10-bench delivery stub lives at the service level — a no-op deliverd binary in the bench profile, not a NATS subject drop. More realistic: we exercise the real broker + router + the message shape deliverd would consume.
• (Q3) loadgen ships as a single Docker image with all four binaries; entrypoint picks one via the image's command: field in the compose / k8s spec.

2026-06-13 — Source protection (throttling) is in

• Spec §22 (new). Seven layers, evaluated in order per request: payload-size cap → per-IP conn cap → per-source token bucket → per-company token bucket → schema validate → broker circuit breaker → per-source quarantine.
• Per-source config additive to §4 schema (rate_limit_per_sec, max_payload_bytes, max_concurrent_connections, quarantine_*).
• Rejection-code matrix is the contract across HTTP/WS/MQTT/gRPC (e.g. per-source rate → 429+Retry-After / Ack{RATE_LIMITED} / MQTT reason 0x97 / close 1013).
• Rollout tied to milestones: M0 layer 3, M1 layers 1/4/5, M5 layer 2, M6 dedupe-aware shaping, M9 layers 6+7.
• Runaway-source test is part of M10 exit (loadgen fault-injection: one source at 10× cap must not push p99 for other sources above 5s).

2026-06-13 — M0 shipped (12 commits, 2888 LoC)

What landed:

• cmd/{ingestd,routerd,deliverd,admind}/ — four Go service mains
• cmd/ingestd/http.go — HTTP POST handler implementing SPEC §22 layers 1, 3, 4, 5 + Stripe-style HMAC auth
• internal/alert — Alert v1 type + Validate() (183 LoC + 103 LoC tests)
• internal/broker — NATS JetStream wrapper, three streams (ALERTS/DELIVERIES/DLQ) auto-created
• internal/dedupe — 60s SET NX EX + INCR (Redis-required tests pass against a host-local Redis)
• internal/ratelimit — per-second INCR bucket (Redis-required tests pass)
• internal/observability — slog + Prometheus registry, the IngestdMetrics struct matches SPEC §22 metric names
• internal/httpserver — shared /health + /metrics scaffold
• internal/config — env-driven Common + Ingestd
• loadgen/cmd/http/ — loadgen-http with --mode normal, HMAC signing, 70/25/4/1 severity mix, dedupe-pct knob
• loadgen/go.mod — separate module per SPEC §21, replace directive points at the parent module
• docker-compose.yml + Dockerfile — single-host stack, all 5 binaries in one image
• M0_VERIFICATION.md — 8-step smoke test
• deploy/prometheus/prometheus.yml — scrapes all 5 services

What's NOT in M0 (and not supposed to be):

• routerd/deliverd/admind business logic (M2/M3+)
• MQTT, gRPC, WebSocket ingest (M4/M5/M11)
• per-IP concurrency cap (M5)
• circuit breaker (M9)
• quarantine (M9)
• ClickHouse archive (M7)
• source registry in DB (M2)
• Postgres migrations (M2)

Module path: git3.techno-world.net/lrosales/broad-announce. Loadgen module path: git3.techno-world.net/lrosales/broad-announce/loadgen. All pushed: 7cd922c..49b2dba on master.