Broad-Announce — Project Spec (v1)

Multi-tenant notification router. Receives alerts from many source systems, normalizes them, routes them to the right people (companies → groups → individuals), and delivers via FCM, Telegram, SMS, email, voice, Slack, Teams, and custom webhooks.

1. Goals (v1)

Accept alerts via HTTP POST (JSON), WebSockets (client→server), and MQTT.
Normalize every inbound alert to a single internal Alert shape.
Resolve recipients by querying companies / groups / individuals tables
- per-user subscriptions (opt-in per source/severity).
Deliver to recipients via FCM (primary), Telegram (primary), SMS, email, voice call, Slack, MS Teams, and generic outbound webhook.
Sustain ~50,000 alerts/sec with ≤5s p99 end-to-end latency (accept → device wake).
Multi-tenant: ~10,000 companies, up to 1,000 devices per company.
Strict data isolation: a company can never see another company's alerts, recipients, or delivery logs.

2. Non-goals (v1)

No inbound email parsing, no SMS shortcode receivers, no phone IVR trees.
No on-prem appliance mode. We deploy as Docker Compose first, K8s later.
No cross-company fan-out except an explicit dev-team supergroup.
No iOS APNs in v1 (FCM only). APNs comes in v2.

3. Multi-tier architecture

[ Sources ]
   │  HTTP POST   (webhook)        ── HMAC-SHA256, API key, optional mTLS
   │  WebSocket   (client push)    ── JWT, TLS
   │  MQTT        (broker sub)     ── per-company topic, QoS 1
   ▼
[ INGEST TIER ]  (ingestd, stateless, N replicas behind LB)
   │  - AuthN/Z (API key / mTLS / JWT)
   │  - Rate limit (Redis token bucket per source)
   │  - Schema validate (per source type)
   │  - Normalize → Alert v1
   │  - Dedupe check (Redis SET NX, TTL 60s, key=hash(source|dedupe_key))
   │  - Stamp dedupe_count on alert if already seen
   │  - Publish to broker (NATS JetStream, subject: alerts.<company_id>)
   ▼
[ BROKER ]  NATS JetStream
   │  - Persistent stream `ALERTS` (replicas=3, retention 24h)
   │  - Per-company subjects for partition affinity
   ▼
[ ROUTER TIER ]  (routerd, stateless, N replicas, JetStream consumer)
   │  - Consume alert
   │  - Resolve recipients:
   │       companies → groups → individuals
   │       ∩ subscriptions (opt-in, severity ≥ user.min_severity)
   │       ∩ quiet_hours (unless severity == "inminent_colapse")
   │  - For each recipient, enqueue 1..N Delivery jobs to:
   │       deliveries.<channel>.<company_id>  (channel-specific subjects)
   ▼
[ DELIVERY TIER ]  (deliverd, one worker pool per channel)
   │  - FCM: shared FCM project, per-token sends, FCM topic for groups>50
   │  - Telegram: per-company bot token, message + inline keyboard
   │  - SMS / Voice: pluggable provider (Twilio v1)
   │  - Email: SMTP relay or SES
   │  - Slack / Teams: incoming webhook URLs
   │  - Custom outbound webhook: HMAC sign with company key
   │  - Retry: exponential backoff (1s, 2s, 4s, 8s, 16s, 32s, 64s, 128s, 256s, 512s)
   │           max 10 attempts, then → Dead-Letter Queue (DLQ)
   ▼
[ DATA TIER ]
   - PostgreSQL: tenants, sources, companies, groups, individuals,
                 subscriptions, telegram_bots, fcm_tokens, routing_rules
   - TimescaleDB: alerts (hypertable, 7d retention), deliveries
                 (hypertable, 7d retention)
   - ClickHouse: alerts_archive, deliveries_archive, aggregates
                 (indefinite, columnar, cheap to query)
   - Redis: dedupe set, rate-limit counters, idempotency keys

Why 4 tiers instead of 1 monolith

Ingest scales with incoming HTTP/WS/MQTT concurrency; it's CPU-cheap but connection-heavy. Independent replicas behind an L7 LB.
Router is CPU + DB heavy (recipient expansion can be 1 → 10,000s). It must not be on the request path of the source system — once accepted, the alert is in the broker and the source is acked.
Delivery is the slowest tier (third-party API calls). Each channel is a separate worker pool with its own concurrency, timeouts, and circuit breaker.
Broker is the buffer. If delivery falls behind, alerts queue up instead of timing out the source.

4. Entities (Postgres schema sketch)

-- Tenancy
companies(id, name, slug, status, fcm_shared, telegram_bot_token_enc,
          rate_limit_per_sec, created_at)

-- People & devices
individuals(id, company_id, full_name, email, phone_e164, locale, tz,
            telegram_chat_id, telegram_user_id, status, created_at)
fcm_tokens(id, individual_id, token, device_id, platform, locale,
           app_version, last_seen, status, created_at)
groups(id, company_id, name, description, fcm_topic, telegram_chat_id,
       created_at)
group_members(group_id, individual_id, added_at, added_by)

-- Source allowlist
sources(id, company_id, name, type, hmac_secret_enc, api_key_hash,
        mtls_required, rate_limit_per_sec, allowed_ips cidr[],
        topic_prefix, created_at)

-- Routing
subscriptions(id, individual_id, source_id, min_severity,
              channel_mask, quiet_hours_start, quiet_hours_end, tz)
routing_rules(id, company_id, priority, match_expr, target_type,
              target_id, enabled)

-- Normalized alert (Timescale hypertable)
alerts(id, company_id, source_id, severity, category, title_key,
       body_key, data jsonb, dedupe_key, dedupe_count, received_at)
-- hypertable on received_at, 7d retention, then COPY to ClickHouse

-- Delivery attempts (Timescale hypertable)
deliveries(id, alert_id, individual_id, channel, target, status,
           attempts, last_error, next_attempt_at, completed_at)
-- hypertable on completed_at (NULL = in-flight), 7d retention

Severity taxonomy (used everywhere)

severity	sound	bypasses_quiet_hours	fcm_priority	example
`info`	default	no	normal	"deploy finished"
`warning`	alert	no	normal	"disk 80% full"
`critical`	siren	no	high	"service down"
`inminent_colapse`	klaxon	yes	high	"rack PDU overload imminent"

category is free-form per source (e.g. network, power, security, hvac, process). The Android app uses category to pick a custom sound file (res/raw/siren_<category>.ogg).

5. Dedupe + dedupe_count

When ingestd receives alert with dedupe_key:

key = hash(source_id || ':' || dedupe_key)
SET key 1 NX EX 60       -- first arrival creates a 60s window

If SET succeeded → first arrival, publish as new alert.
If SET failed → existing window. INCR the counter on the key and attach dedupe_count to the published alert payload. The recipient sees e.g. "Disk full (×12 in 60s)" instead of 12 separate pushes.

The dedupe window default is 60s and is configurable per source.

6. Recipient resolution (the heart of the router)

For an inbound alert:

recipients = ∅
for each target (group or individual) implied by source.allowed_targets:
    if routing_rule matches alert.tags → override targets
    individuals = expand(target)   -- groups → members
    for each individual:
        sub = active_subscription(individual, source, alert.severity)
        if not sub: skip
        if in_quiet_hours(sub, now, alert.severity): skip
                     -- severity=inminent_colapse never skipped
        for each channel in sub.channel_mask:
            recipients += (individual, channel)

The router uses prepared statement + batch IN-list to avoid N+1.

For groups > 50 members, the router uses FCM topic messaging (/topics/<group_fcm_topic>) which is one FCM API call regardless of subscriber count. For smaller groups, it uses per-token sends.

7. FCM (shared project model)

One FCM project for all companies. App must be multi-tenant aware (the Android app reads data.company_id and renders a "company badge").
Per-individual device tokens registered by the Android app on login.
Per-group FCM topic if group size > 50.
Quota: FCM HTTP v1 default is 600k msg/min to a project, 1k msg/min per token recipient is the realistic soft cap. At 50k alerts/sec, even bursty, we stay well under if we use topics for big groups and dedupe to collapse storms. If a single company ever exceeds per-project quota, we can shard into a second FCM project — schema already supports companies.fcm_shared=false.

10k companies × 1k devices = 10M tokens

A single FCM project can hold this. Storage is on our side (the fcm_tokens table), not FCM's. We shard the table by company_id % N (partition by hash) for vacuum/index perf.

8. Telegram (primary channel for ops)

Each company registers one or more Telegram bots (@company_x_bot). Token stored encrypted at rest (telegram_bot_token_enc).
For 1:1 alerts we need the user's telegram_chat_id (resolved when user runs /start in the bot).
For group alerts we use the bot in a company group chat (telegram_chat_id on the groups table).
Bot commands (Telegram = management UI):
- /start — link telegram account to individuals.id
- /mute <duration> — e.g. /mute 2h, /mute until 18:00
- /unmute
- /subscribe <source> <min_severity>
- /unsubscribe <source>
- /preferences — show current subscriptions
- /status — last N alerts received
Telegram is bidirectional: users reply to acknowledge; ack updates deliveries.status and (optionally) closes the alert via a webhook callback to the source system.

9. Delivery retry & DLQ

attempt 1, fail  → wait 1s
attempt 2, fail  → wait 2s
...
attempt 10, fail → publish to dlq.<channel>.<company_id>

DLQ is a JetStream stream with no consumer in normal operation, plus a DLQ table in Postgres for the admin UI. An operator can:

View failed deliveries, with full payload + last error
Replay (re-enqueue) one or many
Discard

deliverd exposes /health and Prometheus metrics (deliverd_delivery_attempts_total{channel,status}, deliverd_delivery_latency_seconds_bucket{channel}).

10. Observability (operate from day 1)

Prometheus metrics on every service:
- ingestd_alerts_received_total{source_type,result}
- ingestd_dedupe_hits_total
- routerd_recipient_expansion_seconds
- deliverd_delivery_attempts_total{channel,status}
- deliverd_dlq_total{channel}
Structured logs (slog JSON) with trace_id, company_id, alert_id, individual_id.
OpenTelemetry traces across ingest → broker → router → delivery (propagate traceparent through NATS headers).
Grafana dashboards: per-company drill-down, channel health, DLQ rate, p50/p95/p99 latency.

11. Security

Secrets at rest: FCM service account JSON, Telegram bot tokens, HMAC secrets → AES-256-GCM with key from env (BA_MASTER_KEY).
TLS 1.2+ everywhere; mTLS optional for HTTP sources.
Per-source HMAC: scheme = Stripe-style (header X-BA-Signature: t=<unix_ts>,v1=<hex_hmac> over timestamp.body, with 5-min replay window). GitHub-style is fine too (header X-Hub-Signature-256: sha256=...) but Stripe's timestamp prevents replay; we ship that one as the default and accept GitHub-style as a per-source option.
API keys: argon2id hashed at rest.
Postgres row-level security optional (off by default; tenant filter is enforced in app code paths).

12. Data retention

alerts (Timescale): 7 days, then COPY → ClickHouse alerts_archive, then drop chunk.
deliveries (Timescale): 7 days, then COPY → ClickHouse deliveries_archive, then drop chunk.
companies / individuals / fcm_tokens / groups / subscriptions: indefinite in Postgres. GDPR right-to-be-forgotten deletes the individual + cascades fcm_tokens + subscriptions.
ClickHouse: indefinite. Partition by month, TTL optional later.

13. Region / data residency (Q24)

"Don't understand this point, can you elaborate?"

This is about where the data lives and who has to obey whose laws:

Single region (e.g. one AWS us-east-1): cheapest, simplest, all customers served from one data center. Fine if your 10k companies don't have legal requirements to keep data in a specific country.
Multi-region active-passive (us-east-1 primary, eu-west-1 standby): you can failover, but EU customers' data still passes through the US at some point. Needed if you want to claim EU residency for marketing.
Multi-region active-active (every region runs a full stack, broker replicated, Postgres logical-replicated): each company pinned to one region. Expensive, complex, but lets you say "company X's alerts never leave Frankfurt."

For v1: single region, but pick the FCM endpoint that matches (e.g. fcm.googleapis.com vs fcm.googleapis.eu). Document the limitation; revisit at 1k paying companies.

14. Compliance (Q25)

"Also don't know the implications of this."

Three things that matter at 10k companies:

GDPR / right to be forgotten: a DELETE /individuals/:id must hard-delete the row + cascade fcm_tokens + subscriptions. Keep a tombstone (id, deleted_at) in a separate erased_individuals table for 30 days so a re-created account doesn't double-deliver. ClickHouse archives get the individual's id zeroed (we keep analytics counts but no PII).
Data Processing Agreement (DPA) with each customer: required by GDPR. We are the processor; they are the controller.
Encryption at rest + in transit (already in §11). SOC2 auditors care; GDPR does too indirectly.

If any customer is in healthcare/finance, add HIPAA/SOC2 controls later (audit log, BAAs). Not v1.

15. mTLS vs HMAC vs API key (Q8)

mechanism	what it proves	who needs it	v1
API key (`X-BA-Key: <opaque>`)	"the caller knows the secret"	any HTTP source	✅ default, always on
HMAC signature	"the caller has the secret AND the body wasn't tampered with"	higher-trust sources, public webhooks	✅ default-on, Stripe-style
mTLS	"the caller controls this exact client cert"	enterprise on-prem, very high trust	✅ optional per source

Recommendation: API key + HMAC for everyone, mTLS opt-in for sources that want it. API key alone is fine for trusted internal sources. HMAC alone is the public-webhook norm. mTLS is overkill for v1 unless a customer asks.

16. MQTT (Q9) — is it the right choice?

protocol	pros	cons	fit
MQTT	tiny, perfect for IoT, persistent sessions, QoS guarantees	broker to operate, clients must speak MQTT	great for sensor gateways, PLCs, on-prem devices
WebSocket ingest	no broker, plain TLS, works from any browser/curl	we have to do the reconnect logic	great for browser dashboards, ops consoles
HTTP POST	dumb simple, every device can do it	no push back, no auto-reconnect magic	great for batch sources (Prometheus, Grafana)

Verdict: support all three. MQTT is the right choice for IoT sources (cameras, sensors, PLCs) but not the only one. We'd use EMQX (or Mosquitto) as the broker.

Per-company topic prefix: yes — ba/<company_slug>/<source_id>/... so a misbehaving publisher can be ACL'd to its prefix only.

QoS 1 vs 2:

QoS 1 (at-least-once): broker acks on receipt, message may be delivered twice on reconnect. Default for alerts. With our dedupe layer this is fine.
QoS 2 (exactly-once): 4-step handshake, slower, ensures no dup. Overkill given our 60s dedupe window. Skip.

17. FCM topic vs per-token fan-out (Q14)

	FCM topic messaging	Per-token send
1 push to N devices	1 FCM API call	N FCM API calls
Latency at 1000 devices	~200ms	~30s with batching
Subscriptions dynamic	user subs/unsubs via app	we manage the list
Delivery report	aggregate only	per-token

Rule: if a group has > 50 members and is the primary target, use FCM topic messaging. Otherwise, per-token is fine and gives us per-device delivery reports.

18. Stack

Language: Go 1.22+
DB: PostgreSQL 16 + TimescaleDB 2.x (active window), ClickHouse 24.x (archive)
Broker: NATS JetStream (single-node for v1, 3-node later)
Cache / dedupe / rate-limit: Redis 7
MQTT broker: EMQX 5.x (Docker image)
Observability: Prometheus + Grafana + Loki + OTel collector
Deployment: Docker Compose (v1), Kubernetes manifests (v2)
Migrations: golang-migrate
No-code admin UI (later): React + Vite, served by admind

19. Repo layout (planned)

broad-announce/
├── SPEC.md                ← this file
├── ARCHITECTURE.md        ← diagrams, sequence flows, SLOs
├── PROMPT.md              ← build log, decisions, lessons
├── README.md
├── docker-compose.yml     ← one-shot local stack
├── .env.example
├── migrations/            ← sql migrations (golang-migrate)
├── ingestd/               ← binary: HTTP/WS/MQTT ingest
├── routerd/               ← binary: recipient expansion
├── deliverd/              ← binary: per-channel delivery workers
├── admind/                ← binary: admin HTTP API + (later) UI host
├── internal/
│   ├── alert/             ← Alert v1 type, validation
│   ├── broker/            ← NATS wrapper
│   ├── store/             ← Postgres + Timescale + ClickHouse
│   ├── dedupe/            ← Redis dedupe
│   ├── ratelimit/         ← Redis token bucket
│   ├── fcm/               ← FCM HTTP v1 client + topic mgmt
│   ├── telegram/          ← Bot client, command parser
│   ├── retry/             ← exponential backoff + DLQ
│   ├── observability/     ← slog + OTel + Prometheus
│   └── config/            ← env-driven config
└── deploy/
    ├── prometheus/
    ├── grafana/
    └── loki/

20. Milestones

#	milestone	exit criterion
M0	Skeleton + docker-compose up	Postgres + NATS + Redis up, all 4 services start, /health green
M1	HTTP POST ingest end-to-end	send a signed webhook → FCM test push to a fake device
M2	Recipient resolution	per-source `allowed_targets` honored, subscriptions applied
M3	Telegram delivery + bot commands	user can `/subscribe` and receive an alert via Telegram
M4	MQTT ingest	EMQX up, QoS 1, per-company topic ACLs
M5	WebSocket ingest + live tail	admin UI (or wscat) sees alerts as they arrive
M6	Dedupe + dedupe_count	burst of 100 identical alerts → recipient sees "×100"
M7	Timescale + ClickHouse	7d retention + archive job
M8	DLQ + replay UI	operator can replay a failed delivery
M9	Observability (Prom/Grafana)	1 dashboard per tier + per-company drilldown
M10	Load test 50k/s	soak 10 min, p99 ≤ 5s, zero DLQ

SPEC.md 19 KB Lịch sử Raw