broad-announce/PROMPT.md

Broad-Announce — Build Log (PROMPT)

Decisions, lessons, blockers. Append-only. Update as we go.

2026-06-13 — kickoff

Decided

• Repo at git3.techno-world.net/lrosales/broad-announce (private).
• v1 stack: Go, PostgreSQL + Timescale, ClickHouse, NATS JetStream, Redis, EMQX (MQTT), Prometheus + Grafana, Loki. Deploy v1 = Docker Compose. v2 = K8s.
• Multi-tenancy = shared infrastructure, strict app-level isolation. No row-level security in v1; tenant filter on every query.
• FCM = single shared project for v1, schema supports per-company FCM project later (companies.fcm_shared).
• Telegram is a first-class delivery channel + a management UI (the bot is how users mute, subscribe, acknowledge).
• Severity taxonomy: info | warning | critical | inminent_colapse. Only inminent_colapse bypasses quiet hours.
• Dedupe: 60s window per (source_id, dedupe_key), attach dedupe_count so user sees "×N in 60s" not N pushes.
• v1 capacity target in docker-compose: 5k alerts/sec sustained. 50k/sec is the design ceiling; the K8s + multi-broker work is what unlocks it.

Open (was) → Resolved 2026-06-13

• Bot ↔ individual linking → admin-invites only. Flow: admin creates individuals row + generates one-time invite code; user runs /start <invite_code> in the Telegram bot; bot matches the code, links telegram_chat_id to the pre-existing individual, and burns the code. Stricter path: prevents drive-by bot self-registration, keeps individuals auditable.
• Localized titles → source-localized, pass through. v1 does NOT translate. Sources send pre-localized title/body strings (or use the locale on the subscription to key into their own lookup table before calling our API). Alert schema carries locale and title/body already-resolved. We can add a translation layer in v2 if customers ask.
• mTLS: schema supports it, but the docker-compose profile won't terminate client certs in v1. Documented as opt-in for enterprise sources.

Lessons (already)

• It's much cheaper to answer "what can this be?" with 30 questions than to refactor later. Most of the SPEC's weight is in §11 (security) and §6 (recipient resolution) — those are the parts that are expensive to change after launch.
• For multi-tenant at 10k companies, the one design choice that compounds is the subject layout in the broker. alerts.<company_id> is fine; alerts.<company_id>.<source_id> would let us scale router consumer groups per source. Locked in §6 of ARCHITECTURE.md.

2026-06-13 — added gRPC ingest (option a)

• Decision: ship gRPC bidi-streaming as the fourth ingest protocol, scoped to internal high-volume sources only. Public SaaS webhooks stay on HTTP POST, browsers stay on WebSocket, IoT stays on MQTT.
• Rationale: typed schemas, HTTP/2 + protobuf, native backpressure, no reconnect-loop code, generated Go/Java/Python/Node clients. Cost: one more protocol to operate + a buf generate build step.
• Gated on M11, after M10 (load test). M10 now caps at 5k/s on docker-compose (not 50k/s) — 50k/s is the design ceiling that requires K8s + multi-broker (M12 in v2).
• StreamAlerts(Alert) → Ack carries the same dedupe_count contract as HTTP/WS/MQTT.
• Proto lives at proto/broadannounce/v1/ingest.proto; server stub in internal/grpcserver/, reusable client in internal/grpcclient/. Auth = API key in metadata + optional mTLS. Per-stream rate limit + 256 in-flight cap = natural backpressure.

2026-06-13 — M10 split: option C

• M10 (in-cluster, real) stays at 5k/s on docker-compose.
• Added M10-bench: 50k/s against the broker + router with the delivery tier stubbed. Proves the ceiling without committing to K8s. Green here means K8s is implementable, not required.
• Implication: we need traffic generators that can hit the in-cluster path at 5k/s (M10) and the broker+router path at 50k/s (M10-bench) with the same source-protocol clients we ship. The loadgen/ tool is the new home for that.

2026-06-13 — three open questions resolved

• (Q1) Yes, build the fake-FCM / fake-Telegram / fake-SMS servers (testfakes/). M10 must not burn 50k FCM credits.
• (Q2) M10-bench delivery stub lives at the service level — a no-op deliverd binary in the bench profile, not a NATS subject drop. More realistic: we exercise the real broker + router + the message shape deliverd would consume.
• (Q3) loadgen ships as a single Docker image with all four binaries; entrypoint picks one via the image's command: field in the compose / k8s spec.

2026-06-13 — Source protection (throttling) is in

• Spec §22 (new). Seven layers, evaluated in order per request: payload-size cap → per-IP conn cap → per-source token bucket → per-company token bucket → schema validate → broker circuit breaker → per-source quarantine.
• Per-source config additive to §4 schema (rate_limit_per_sec, max_payload_bytes, max_concurrent_connections, quarantine_*).
• Rejection-code matrix is the contract across HTTP/WS/MQTT/gRPC (e.g. per-source rate → 429+Retry-After / Ack{RATE_LIMITED} / MQTT reason 0x97 / close 1013).
• Rollout tied to milestones: M0 layer 3, M1 layers 1/4/5, M5 layer 2, M6 dedupe-aware shaping, M9 layers 6+7.
• Runaway-source test is part of M10 exit (loadgen fault-injection: one source at 10× cap must not push p99 for other sources above 5s).

2026-06-13 — M0 shipped (12 commits, 2888 LoC)

What landed:

• cmd/{ingestd,routerd,deliverd,admind}/ — four Go service mains
• cmd/ingestd/http.go — HTTP POST handler implementing SPEC §22 layers 1, 3, 4, 5 + Stripe-style HMAC auth
• internal/alert — Alert v1 type + Validate() (183 LoC + 103 LoC tests)
• internal/broker — NATS JetStream wrapper, three streams (ALERTS/DELIVERIES/DLQ) auto-created
• internal/dedupe — 60s SET NX EX + INCR (Redis-required tests pass against a host-local Redis)
• internal/ratelimit — per-second INCR bucket (Redis-required tests pass)
• internal/observability — slog + Prometheus registry, the IngestdMetrics struct matches SPEC §22 metric names
• internal/httpserver — shared /health + /metrics scaffold
• internal/config — env-driven Common + Ingestd
• loadgen/cmd/http/ — loadgen-http with --mode normal, HMAC signing, 70/25/4/1 severity mix, dedupe-pct knob
• loadgen/go.mod — separate module per SPEC §21, replace directive points at the parent module
• docker-compose.yml + Dockerfile — single-host stack, all 5 binaries in one image
• M0_VERIFICATION.md — 8-step smoke test
• deploy/prometheus/prometheus.yml — scrapes all 5 services

What's NOT in M0 (and not supposed to be):

• routerd/deliverd/admind business logic (M2/M3+)
• MQTT, gRPC, WebSocket ingest (M4/M5/M11)
• per-IP concurrency cap (M5)
• circuit breaker (M9)
• quarantine (M9)
• ClickHouse archive (M7)
• source registry in DB (M2)
• Postgres migrations (M2)

Module path: git3.techno-world.net/lrosales/broad-announce. Loadgen module path: git3.techno-world.net/lrosales/broad-announce/loadgen. All pushed: 7cd922c..49b2dba on master.

2026-06-13 — Port convention

Project rule: app HTTP services use 8800–8899 (ingestd 8800, routerd 8801, deliverd 8802, admind 8803, loadgen metrics 8891, fakefcmd 8820). Canonical ports stay (5432, 4222, 6379, 1883, 9090, 3000). Reason: 8080+ collides with workstation tooling. SPEC §18 now has a port-conventions sub-section. Commit: d76aa0b.

2026-06-13 — M1 code-complete (8 commits, awaiting live smoke)

What landed:

• migrations/001_init.up.sql — companies, individuals, fcm_tokens (the M1 minimum schema; full SPEC §4 lands in M2 as additional migrations)
• migrations/002_deliveries.up.sql — deliveries table (status: pending|sent|failed|dlq; payload jsonb for M8 replay)
• migrations/seed.sql — idempotent; 1 company, 1 individual, 1 token
• internal/postgres — pgxpool wrapper with retry-on-startup
• cmd/seed — applies *.up.sql in lexical order, then seed.sql
• testfakes/fakefcmd — 70 lines, /health + /v1/.../messages:send, --fail-rate knob
• internal/routing — Resolver with ResolveTokens (M1 broadcast via single SQL join; M2 swaps for the rules engine)
• cmd/routerd — M1 main: subscribes to alerts.>, resolves recipients, publishes one deliveries.fcm. per token
• cmd/deliverd — M1 main: subscribes to deliveries.fcm.>, builds FCM HTTP v1 message body (M3 swap is a no-op at this layer), posts to BA_FAKECMD_URL, writes a deliveries row
• Dockerfile builds 7 binaries
• docker-compose adds fakefcmd + seed (one-shot sidecar), wires BA_FAKECMD_URL into deliverd
• M1_VERIFICATION.md: 9-step smoke test

What we agreed to defer (per the user):

• Q1 seed: 1 company / 1 individual / 1 token — DONE
• Q2 router: M1 broadcast (no subscriptions) — DONE
• Q3 testfakes: only fakefcmd for M1 — DONE
• Q4 migrations: 3 tables in M1, expand in M2 — DONE

What's NOT in M1 (and not supposed to be):

• subscriptions, quiet hours, routing rules (M2)
• per-source allowed_targets (M2)
• real FCM (M3)
• other delivery channels (M3+)
• retry + DLQ from SPEC §9 (M3)
• ClickHouse / Timescale hypertables (M7)
• HMAC secret in DB (M2)
• per-IP cap, circuit breaker, quarantine (M5/M9)

Pushed: 264d284..9cf68a1 on master (5 commits for M1 code, plus the d76aa0b port shift). The user still has to actually run docker compose up and the M1 verification steps before M1 is fully done.