|
|
@@ -0,0 +1,433 @@
|
|
|
+# Broad-Announce — Project Spec (v1)
|
|
|
+
|
|
|
+> Multi-tenant notification router. Receives alerts from many source systems,
|
|
|
+> normalizes them, routes them to the right people (companies → groups →
|
|
|
+> individuals), and delivers via FCM, Telegram, SMS, email, voice, Slack,
|
|
|
+> Teams, and custom webhooks.
|
|
|
+
|
|
|
+## 1. Goals (v1)
|
|
|
+
|
|
|
+- Accept alerts via **HTTP POST (JSON)**, **WebSockets (client→server)**, and
|
|
|
+ **MQTT**.
|
|
|
+- Normalize every inbound alert to a single internal `Alert` shape.
|
|
|
+- Resolve recipients by querying `companies` / `groups` / `individuals` tables
|
|
|
+ + per-user `subscriptions` (opt-in per source/severity).
|
|
|
+- Deliver to recipients via **FCM** (primary), **Telegram** (primary),
|
|
|
+ **SMS**, **email**, **voice call**, **Slack**, **MS Teams**, and
|
|
|
+ **generic outbound webhook**.
|
|
|
+- Sustain **~50,000 alerts/sec** with **≤5s p99** end-to-end latency
|
|
|
+ (accept → device wake).
|
|
|
+- Multi-tenant: **~10,000 companies**, up to **1,000 devices per company**.
|
|
|
+- Strict data isolation: a company can never see another company's alerts,
|
|
|
+ recipients, or delivery logs.
|
|
|
+
|
|
|
+## 2. Non-goals (v1)
|
|
|
+
|
|
|
+- No inbound email parsing, no SMS shortcode receivers, no phone IVR trees.
|
|
|
+- No on-prem appliance mode. We deploy as Docker Compose first, K8s later.
|
|
|
+- No cross-company fan-out except an explicit `dev-team` supergroup.
|
|
|
+- No iOS APNs in v1 (FCM only). APNs comes in v2.
|
|
|
+
|
|
|
+## 3. Multi-tier architecture
|
|
|
+
|
|
|
+```
|
|
|
+[ Sources ]
|
|
|
+ │ HTTP POST (webhook) ── HMAC-SHA256, API key, optional mTLS
|
|
|
+ │ WebSocket (client push) ── JWT, TLS
|
|
|
+ │ MQTT (broker sub) ── per-company topic, QoS 1
|
|
|
+ ▼
|
|
|
+[ INGEST TIER ] (ingestd, stateless, N replicas behind LB)
|
|
|
+ │ - AuthN/Z (API key / mTLS / JWT)
|
|
|
+ │ - Rate limit (Redis token bucket per source)
|
|
|
+ │ - Schema validate (per source type)
|
|
|
+ │ - Normalize → Alert v1
|
|
|
+ │ - Dedupe check (Redis SET NX, TTL 60s, key=hash(source|dedupe_key))
|
|
|
+ │ - Stamp dedupe_count on alert if already seen
|
|
|
+ │ - Publish to broker (NATS JetStream, subject: alerts.<company_id>)
|
|
|
+ ▼
|
|
|
+[ BROKER ] NATS JetStream
|
|
|
+ │ - Persistent stream `ALERTS` (replicas=3, retention 24h)
|
|
|
+ │ - Per-company subjects for partition affinity
|
|
|
+ ▼
|
|
|
+[ ROUTER TIER ] (routerd, stateless, N replicas, JetStream consumer)
|
|
|
+ │ - Consume alert
|
|
|
+ │ - Resolve recipients:
|
|
|
+ │ companies → groups → individuals
|
|
|
+ │ ∩ subscriptions (opt-in, severity ≥ user.min_severity)
|
|
|
+ │ ∩ quiet_hours (unless severity == "inminent_colapse")
|
|
|
+ │ - For each recipient, enqueue 1..N Delivery jobs to:
|
|
|
+ │ deliveries.<channel>.<company_id> (channel-specific subjects)
|
|
|
+ ▼
|
|
|
+[ DELIVERY TIER ] (deliverd, one worker pool per channel)
|
|
|
+ │ - FCM: shared FCM project, per-token sends, FCM topic for groups>50
|
|
|
+ │ - Telegram: per-company bot token, message + inline keyboard
|
|
|
+ │ - SMS / Voice: pluggable provider (Twilio v1)
|
|
|
+ │ - Email: SMTP relay or SES
|
|
|
+ │ - Slack / Teams: incoming webhook URLs
|
|
|
+ │ - Custom outbound webhook: HMAC sign with company key
|
|
|
+ │ - Retry: exponential backoff (1s, 2s, 4s, 8s, 16s, 32s, 64s, 128s, 256s, 512s)
|
|
|
+ │ max 10 attempts, then → Dead-Letter Queue (DLQ)
|
|
|
+ ▼
|
|
|
+[ DATA TIER ]
|
|
|
+ - PostgreSQL: tenants, sources, companies, groups, individuals,
|
|
|
+ subscriptions, telegram_bots, fcm_tokens, routing_rules
|
|
|
+ - TimescaleDB: alerts (hypertable, 7d retention), deliveries
|
|
|
+ (hypertable, 7d retention)
|
|
|
+ - ClickHouse: alerts_archive, deliveries_archive, aggregates
|
|
|
+ (indefinite, columnar, cheap to query)
|
|
|
+ - Redis: dedupe set, rate-limit counters, idempotency keys
|
|
|
+```
|
|
|
+
|
|
|
+### Why 4 tiers instead of 1 monolith
|
|
|
+
|
|
|
+- **Ingest** scales with incoming HTTP/WS/MQTT concurrency; it's CPU-cheap
|
|
|
+ but connection-heavy. Independent replicas behind an L7 LB.
|
|
|
+- **Router** is CPU + DB heavy (recipient expansion can be 1 → 10,000s). It
|
|
|
+ must not be on the request path of the source system — once accepted,
|
|
|
+ the alert is in the broker and the source is acked.
|
|
|
+- **Delivery** is the slowest tier (third-party API calls). Each channel is
|
|
|
+ a separate worker pool with its own concurrency, timeouts, and circuit
|
|
|
+ breaker.
|
|
|
+- **Broker** is the buffer. If delivery falls behind, alerts queue up
|
|
|
+ instead of timing out the source.
|
|
|
+
|
|
|
+## 4. Entities (Postgres schema sketch)
|
|
|
+
|
|
|
+```sql
|
|
|
+-- Tenancy
|
|
|
+companies(id, name, slug, status, fcm_shared, telegram_bot_token_enc,
|
|
|
+ rate_limit_per_sec, created_at)
|
|
|
+
|
|
|
+-- People & devices
|
|
|
+individuals(id, company_id, full_name, email, phone_e164, locale, tz,
|
|
|
+ telegram_chat_id, telegram_user_id, status, created_at)
|
|
|
+fcm_tokens(id, individual_id, token, device_id, platform, locale,
|
|
|
+ app_version, last_seen, status, created_at)
|
|
|
+groups(id, company_id, name, description, fcm_topic, telegram_chat_id,
|
|
|
+ created_at)
|
|
|
+group_members(group_id, individual_id, added_at, added_by)
|
|
|
+
|
|
|
+-- Source allowlist
|
|
|
+sources(id, company_id, name, type, hmac_secret_enc, api_key_hash,
|
|
|
+ mtls_required, rate_limit_per_sec, allowed_ips cidr[],
|
|
|
+ topic_prefix, created_at)
|
|
|
+
|
|
|
+-- Routing
|
|
|
+subscriptions(id, individual_id, source_id, min_severity,
|
|
|
+ channel_mask, quiet_hours_start, quiet_hours_end, tz)
|
|
|
+routing_rules(id, company_id, priority, match_expr, target_type,
|
|
|
+ target_id, enabled)
|
|
|
+
|
|
|
+-- Normalized alert (Timescale hypertable)
|
|
|
+alerts(id, company_id, source_id, severity, category, title_key,
|
|
|
+ body_key, data jsonb, dedupe_key, dedupe_count, received_at)
|
|
|
+-- hypertable on received_at, 7d retention, then COPY to ClickHouse
|
|
|
+
|
|
|
+-- Delivery attempts (Timescale hypertable)
|
|
|
+deliveries(id, alert_id, individual_id, channel, target, status,
|
|
|
+ attempts, last_error, next_attempt_at, completed_at)
|
|
|
+-- hypertable on completed_at (NULL = in-flight), 7d retention
|
|
|
+```
|
|
|
+
|
|
|
+### Severity taxonomy (used everywhere)
|
|
|
+
|
|
|
+| severity | sound | bypasses_quiet_hours | fcm_priority | example |
|
|
|
+|---|---|---|---|---|
|
|
|
+| `info` | default | no | normal | "deploy finished" |
|
|
|
+| `warning` | alert | no | normal | "disk 80% full" |
|
|
|
+| `critical` | siren | no | high | "service down" |
|
|
|
+| `inminent_colapse` | klaxon | **yes** | high | "rack PDU overload imminent" |
|
|
|
+
|
|
|
+`category` is free-form per source (e.g. `network`, `power`, `security`,
|
|
|
+`hvac`, `process`). The Android app uses `category` to pick a custom sound
|
|
|
+file (`res/raw/siren_<category>.ogg`).
|
|
|
+
|
|
|
+## 5. Dedupe + dedupe_count
|
|
|
+
|
|
|
+When `ingestd` receives alert with `dedupe_key`:
|
|
|
+
|
|
|
+```
|
|
|
+key = hash(source_id || ':' || dedupe_key)
|
|
|
+SET key 1 NX EX 60 -- first arrival creates a 60s window
|
|
|
+```
|
|
|
+
|
|
|
+- If `SET` succeeded → first arrival, publish as new alert.
|
|
|
+- If `SET` failed → existing window. `INCR` the counter on the key and
|
|
|
+ attach `dedupe_count` to the published alert payload. The recipient
|
|
|
+ sees e.g. *"Disk full (×12 in 60s)"* instead of 12 separate pushes.
|
|
|
+
|
|
|
+The dedupe window default is 60s and is configurable per source.
|
|
|
+
|
|
|
+## 6. Recipient resolution (the heart of the router)
|
|
|
+
|
|
|
+For an inbound alert:
|
|
|
+
|
|
|
+```
|
|
|
+recipients = ∅
|
|
|
+for each target (group or individual) implied by source.allowed_targets:
|
|
|
+ if routing_rule matches alert.tags → override targets
|
|
|
+ individuals = expand(target) -- groups → members
|
|
|
+ for each individual:
|
|
|
+ sub = active_subscription(individual, source, alert.severity)
|
|
|
+ if not sub: skip
|
|
|
+ if in_quiet_hours(sub, now, alert.severity): skip
|
|
|
+ -- severity=inminent_colapse never skipped
|
|
|
+ for each channel in sub.channel_mask:
|
|
|
+ recipients += (individual, channel)
|
|
|
+```
|
|
|
+
|
|
|
+The router uses **prepared statement + batch IN-list** to avoid N+1.
|
|
|
+
|
|
|
+For groups > 50 members, the router uses **FCM topic messaging**
|
|
|
+(`/topics/<group_fcm_topic>`) which is one FCM API call regardless of
|
|
|
+subscriber count. For smaller groups, it uses per-token sends.
|
|
|
+
|
|
|
+## 7. FCM (shared project model)
|
|
|
+
|
|
|
+- **One FCM project** for all companies. App must be multi-tenant aware
|
|
|
+ (the Android app reads `data.company_id` and renders a "company badge").
|
|
|
+- Per-individual **device tokens** registered by the Android app on login.
|
|
|
+- Per-group **FCM topic** if group size > 50.
|
|
|
+- Quota: FCM HTTP v1 default is 600k msg/min to a project, 1k msg/min per
|
|
|
+ token recipient is the realistic soft cap. At 50k alerts/sec, even
|
|
|
+ bursty, we stay well under if we use **topics for big groups** and
|
|
|
+ **dedupe** to collapse storms. If a single company ever exceeds
|
|
|
+ per-project quota, we can **shard into a second FCM project** —
|
|
|
+ schema already supports `companies.fcm_shared=false`.
|
|
|
+
|
|
|
+### 10k companies × 1k devices = 10M tokens
|
|
|
+
|
|
|
+A single FCM project can hold this. Storage is on our side (the
|
|
|
+`fcm_tokens` table), not FCM's. We shard the table by `company_id % N`
|
|
|
+(partition by hash) for vacuum/index perf.
|
|
|
+
|
|
|
+## 8. Telegram (primary channel for ops)
|
|
|
+
|
|
|
+- Each company registers one or more **Telegram bots** (`@company_x_bot`).
|
|
|
+ Token stored encrypted at rest (`telegram_bot_token_enc`).
|
|
|
+- For 1:1 alerts we need the user's `telegram_chat_id` (resolved when
|
|
|
+ user runs `/start` in the bot).
|
|
|
+- For group alerts we use the bot in a company group chat
|
|
|
+ (`telegram_chat_id` on the `groups` table).
|
|
|
+- **Bot commands** (Telegram = management UI):
|
|
|
+ - `/start` — link telegram account to `individuals.id`
|
|
|
+ - `/mute <duration>` — e.g. `/mute 2h`, `/mute until 18:00`
|
|
|
+ - `/unmute`
|
|
|
+ - `/subscribe <source> <min_severity>`
|
|
|
+ - `/unsubscribe <source>`
|
|
|
+ - `/preferences` — show current subscriptions
|
|
|
+ - `/status` — last N alerts received
|
|
|
+- Telegram is bidirectional: users reply to acknowledge; ack updates
|
|
|
+ `deliveries.status` and (optionally) closes the alert via a webhook
|
|
|
+ callback to the source system.
|
|
|
+
|
|
|
+## 9. Delivery retry & DLQ
|
|
|
+
|
|
|
+```
|
|
|
+attempt 1, fail → wait 1s
|
|
|
+attempt 2, fail → wait 2s
|
|
|
+...
|
|
|
+attempt 10, fail → publish to dlq.<channel>.<company_id>
|
|
|
+```
|
|
|
+
|
|
|
+DLQ is a **JetStream stream** with no consumer in normal operation, plus
|
|
|
+a **DLQ table** in Postgres for the admin UI. An operator can:
|
|
|
+- View failed deliveries, with full payload + last error
|
|
|
+- Replay (re-enqueue) one or many
|
|
|
+- Discard
|
|
|
+
|
|
|
+`deliverd` exposes `/health` and Prometheus metrics
|
|
|
+(`deliverd_delivery_attempts_total{channel,status}`,
|
|
|
+`deliverd_delivery_latency_seconds_bucket{channel}`).
|
|
|
+
|
|
|
+## 10. Observability (operate from day 1)
|
|
|
+
|
|
|
+- **Prometheus** metrics on every service:
|
|
|
+ - `ingestd_alerts_received_total{source_type,result}`
|
|
|
+ - `ingestd_dedupe_hits_total`
|
|
|
+ - `routerd_recipient_expansion_seconds`
|
|
|
+ - `deliverd_delivery_attempts_total{channel,status}`
|
|
|
+ - `deliverd_dlq_total{channel}`
|
|
|
+- **Structured logs** (slog JSON) with `trace_id`, `company_id`,
|
|
|
+ `alert_id`, `individual_id`.
|
|
|
+- **OpenTelemetry traces** across ingest → broker → router → delivery
|
|
|
+ (propagate `traceparent` through NATS headers).
|
|
|
+- **Grafana dashboards**: per-company drill-down, channel health, DLQ
|
|
|
+ rate, p50/p95/p99 latency.
|
|
|
+
|
|
|
+## 11. Security
|
|
|
+
|
|
|
+- Secrets at rest: FCM service account JSON, Telegram bot tokens, HMAC
|
|
|
+ secrets → AES-256-GCM with key from env (`BA_MASTER_KEY`).
|
|
|
+- TLS 1.2+ everywhere; mTLS optional for HTTP sources.
|
|
|
+- Per-source HMAC: scheme = **Stripe-style** (header
|
|
|
+ `X-BA-Signature: t=<unix_ts>,v1=<hex_hmac>` over
|
|
|
+ `timestamp.body`, with 5-min replay window). GitHub-style is fine too
|
|
|
+ (header `X-Hub-Signature-256: sha256=...`) but Stripe's timestamp
|
|
|
+ prevents replay; we ship that one as the default and accept
|
|
|
+ GitHub-style as a per-source option.
|
|
|
+- API keys: argon2id hashed at rest.
|
|
|
+- Postgres row-level security optional (off by default; tenant filter
|
|
|
+ is enforced in app code paths).
|
|
|
+
|
|
|
+## 12. Data retention
|
|
|
+
|
|
|
+- **alerts** (Timescale): 7 days, then COPY → ClickHouse
|
|
|
+ `alerts_archive`, then drop chunk.
|
|
|
+- **deliveries** (Timescale): 7 days, then COPY → ClickHouse
|
|
|
+ `deliveries_archive`, then drop chunk.
|
|
|
+- **companies / individuals / fcm_tokens / groups / subscriptions**:
|
|
|
+ indefinite in Postgres. GDPR right-to-be-forgotten deletes the
|
|
|
+ individual + cascades fcm_tokens + subscriptions.
|
|
|
+- **ClickHouse**: indefinite. Partition by month, TTL optional later.
|
|
|
+
|
|
|
+## 13. Region / data residency (Q24)
|
|
|
+
|
|
|
+> "Don't understand this point, can you elaborate?"
|
|
|
+
|
|
|
+This is about **where the data lives** and **who has to obey whose laws**:
|
|
|
+
|
|
|
+- **Single region** (e.g. one AWS `us-east-1`): cheapest, simplest, all
|
|
|
+ customers served from one data center. Fine if your 10k companies
|
|
|
+ don't have legal requirements to keep data in a specific country.
|
|
|
+- **Multi-region active-passive** (us-east-1 primary, eu-west-1
|
|
|
+ standby): you can failover, but EU customers' data still passes
|
|
|
+ through the US at some point. Needed if you want to claim EU
|
|
|
+ residency for marketing.
|
|
|
+- **Multi-region active-active** (every region runs a full stack,
|
|
|
+ broker replicated, Postgres logical-replicated): each company pinned
|
|
|
+ to one region. Expensive, complex, but lets you say "company X's
|
|
|
+ alerts never leave Frankfurt."
|
|
|
+
|
|
|
+For v1: **single region**, but pick the FCM endpoint that matches
|
|
|
+(e.g. `fcm.googleapis.com` vs `fcm.googleapis.eu`). Document the
|
|
|
+limitation; revisit at 1k paying companies.
|
|
|
+
|
|
|
+## 14. Compliance (Q25)
|
|
|
+
|
|
|
+> "Also don't know the implications of this."
|
|
|
+
|
|
|
+Three things that matter at 10k companies:
|
|
|
+
|
|
|
+1. **GDPR / right to be forgotten**: a `DELETE /individuals/:id` must
|
|
|
+ hard-delete the row + cascade fcm_tokens + subscriptions. Keep a
|
|
|
+ tombstone (id, deleted_at) in a separate `erased_individuals` table
|
|
|
+ for 30 days so a re-created account doesn't double-deliver. ClickHouse
|
|
|
+ archives get the individual's `id` zeroed (we keep analytics counts
|
|
|
+ but no PII).
|
|
|
+2. **Data Processing Agreement** (DPA) with each customer: required by
|
|
|
+ GDPR. We are the processor; they are the controller.
|
|
|
+3. **Encryption at rest + in transit** (already in §11). SOC2 auditors
|
|
|
+ care; GDPR does too indirectly.
|
|
|
+
|
|
|
+If any customer is in healthcare/finance, add HIPAA/SOC2 controls later
|
|
|
+(audit log, BAAs). Not v1.
|
|
|
+
|
|
|
+## 15. mTLS vs HMAC vs API key (Q8)
|
|
|
+
|
|
|
+| mechanism | what it proves | who needs it | v1 |
|
|
|
+|---|---|---|---|
|
|
|
+| **API key** (`X-BA-Key: <opaque>`) | "the caller knows the secret" | any HTTP source | ✅ default, always on |
|
|
|
+| **HMAC signature** | "the caller has the secret AND the body wasn't tampered with" | higher-trust sources, public webhooks | ✅ default-on, Stripe-style |
|
|
|
+| **mTLS** | "the caller controls this exact client cert" | enterprise on-prem, very high trust | ✅ optional per source |
|
|
|
+
|
|
|
+Recommendation: **API key + HMAC for everyone, mTLS opt-in for sources
|
|
|
+that want it.** API key alone is fine for trusted internal sources. HMAC
|
|
|
+alone is the public-webhook norm. mTLS is overkill for v1 unless a
|
|
|
+customer asks.
|
|
|
+
|
|
|
+## 16. MQTT (Q9) — is it the right choice?
|
|
|
+
|
|
|
+| protocol | pros | cons | fit |
|
|
|
+|---|---|---|---|
|
|
|
+| **MQTT** | tiny, perfect for IoT, persistent sessions, QoS guarantees | broker to operate, clients must speak MQTT | great for sensor gateways, PLCs, on-prem devices |
|
|
|
+| **WebSocket ingest** | no broker, plain TLS, works from any browser/curl | we have to do the reconnect logic | great for browser dashboards, ops consoles |
|
|
|
+| **HTTP POST** | dumb simple, every device can do it | no push back, no auto-reconnect magic | great for batch sources (Prometheus, Grafana) |
|
|
|
+
|
|
|
+**Verdict**: support all three. MQTT is the **right** choice for IoT
|
|
|
+sources (cameras, sensors, PLCs) but not the only one. We'd use
|
|
|
+**EMQX** (or Mosquitto) as the broker.
|
|
|
+
|
|
|
+**Per-company topic prefix**: yes — `ba/<company_slug>/<source_id>/...`
|
|
|
+so a misbehaving publisher can be ACL'd to its prefix only.
|
|
|
+
|
|
|
+**QoS 1 vs 2**:
|
|
|
+- **QoS 1** (at-least-once): broker acks on receipt, message may be
|
|
|
+ delivered twice on reconnect. **Default for alerts.** With our dedupe
|
|
|
+ layer this is fine.
|
|
|
+- **QoS 2** (exactly-once): 4-step handshake, slower, ensures no dup.
|
|
|
+ Overkill given our 60s dedupe window. Skip.
|
|
|
+
|
|
|
+## 17. FCM topic vs per-token fan-out (Q14)
|
|
|
+
|
|
|
+| | FCM topic messaging | Per-token send |
|
|
|
+|---|---|---|
|
|
|
+| 1 push to N devices | **1 FCM API call** | N FCM API calls |
|
|
|
+| Latency at 1000 devices | ~200ms | ~30s with batching |
|
|
|
+| Subscriptions dynamic | user subs/unsubs via app | we manage the list |
|
|
|
+| Delivery report | aggregate only | per-token |
|
|
|
+
|
|
|
+**Rule**: if a group has > 50 members and is the primary target,
|
|
|
+**use FCM topic messaging**. Otherwise, per-token is fine and gives
|
|
|
+us per-device delivery reports.
|
|
|
+
|
|
|
+## 18. Stack
|
|
|
+
|
|
|
+- **Language**: Go 1.22+
|
|
|
+- **DB**: PostgreSQL 16 + TimescaleDB 2.x (active window), ClickHouse 24.x
|
|
|
+ (archive)
|
|
|
+- **Broker**: NATS JetStream (single-node for v1, 3-node later)
|
|
|
+- **Cache / dedupe / rate-limit**: Redis 7
|
|
|
+- **MQTT broker**: EMQX 5.x (Docker image)
|
|
|
+- **Observability**: Prometheus + Grafana + Loki + OTel collector
|
|
|
+- **Deployment**: Docker Compose (v1), Kubernetes manifests (v2)
|
|
|
+- **Migrations**: `golang-migrate`
|
|
|
+- **No-code admin UI** (later): React + Vite, served by `admind`
|
|
|
+
|
|
|
+## 19. Repo layout (planned)
|
|
|
+
|
|
|
+```
|
|
|
+broad-announce/
|
|
|
+├── SPEC.md ← this file
|
|
|
+├── ARCHITECTURE.md ← diagrams, sequence flows, SLOs
|
|
|
+├── PROMPT.md ← build log, decisions, lessons
|
|
|
+├── README.md
|
|
|
+├── docker-compose.yml ← one-shot local stack
|
|
|
+├── .env.example
|
|
|
+├── migrations/ ← sql migrations (golang-migrate)
|
|
|
+├── ingestd/ ← binary: HTTP/WS/MQTT ingest
|
|
|
+├── routerd/ ← binary: recipient expansion
|
|
|
+├── deliverd/ ← binary: per-channel delivery workers
|
|
|
+├── admind/ ← binary: admin HTTP API + (later) UI host
|
|
|
+├── internal/
|
|
|
+│ ├── alert/ ← Alert v1 type, validation
|
|
|
+│ ├── broker/ ← NATS wrapper
|
|
|
+│ ├── store/ ← Postgres + Timescale + ClickHouse
|
|
|
+│ ├── dedupe/ ← Redis dedupe
|
|
|
+│ ├── ratelimit/ ← Redis token bucket
|
|
|
+│ ├── fcm/ ← FCM HTTP v1 client + topic mgmt
|
|
|
+│ ├── telegram/ ← Bot client, command parser
|
|
|
+│ ├── retry/ ← exponential backoff + DLQ
|
|
|
+│ ├── observability/ ← slog + OTel + Prometheus
|
|
|
+│ └── config/ ← env-driven config
|
|
|
+└── deploy/
|
|
|
+ ├── prometheus/
|
|
|
+ ├── grafana/
|
|
|
+ └── loki/
|
|
|
+```
|
|
|
+
|
|
|
+## 20. Milestones
|
|
|
+
|
|
|
+| # | milestone | exit criterion |
|
|
|
+|---|---|---|
|
|
|
+| M0 | Skeleton + docker-compose up | Postgres + NATS + Redis up, all 4 services start, /health green |
|
|
|
+| M1 | HTTP POST ingest end-to-end | send a signed webhook → FCM test push to a fake device |
|
|
|
+| M2 | Recipient resolution | per-source `allowed_targets` honored, subscriptions applied |
|
|
|
+| M3 | Telegram delivery + bot commands | user can `/subscribe` and receive an alert via Telegram |
|
|
|
+| M4 | MQTT ingest | EMQX up, QoS 1, per-company topic ACLs |
|
|
|
+| M5 | WebSocket ingest + live tail | admin UI (or wscat) sees alerts as they arrive |
|
|
|
+| M6 | Dedupe + dedupe_count | burst of 100 identical alerts → recipient sees "×100" |
|
|
|
+| M7 | Timescale + ClickHouse | 7d retention + archive job |
|
|
|
+| M8 | DLQ + replay UI | operator can replay a failed delivery |
|
|
|
+| M9 | Observability (Prom/Grafana) | 1 dashboard per tier + per-company drilldown |
|
|
|
+| M10 | Load test 50k/s | soak 10 min, p99 ≤ 5s, zero DLQ |
|
Luis Rosales