# Broad-Announce — Project Spec (v1)

> Multi-tenant notification router. Receives alerts from many source systems,
> normalizes them, routes them to the right people (companies → groups →
> individuals), and delivers via FCM, Telegram, SMS, email, voice, Slack,
> Teams, and custom webhooks.

## 1. Goals (v1)

- Accept alerts via **HTTP POST (JSON)**, **WebSockets (client→server)**, and
  **MQTT**.
- Normalize every inbound alert to a single internal `Alert` shape.
- Resolve recipients by querying `companies` / `groups` / `individuals` tables
  + per-user `subscriptions` (opt-in per source/severity).
- Deliver to recipients via **FCM** (primary), **Telegram** (primary),
  **SMS**, **email**, **voice call**, **Slack**, **MS Teams**, and
  **generic outbound webhook**.
- Sustain **~50,000 alerts/sec** with **≤5s p99** end-to-end latency
  (accept → device wake).
- Multi-tenant: **~10,000 companies**, up to **1,000 devices per company**.
- Strict data isolation: a company can never see another company's alerts,
  recipients, or delivery logs.

## 2. Non-goals (v1)

- No inbound email parsing, no SMS shortcode receivers, no phone IVR trees.
- No on-prem appliance mode. We deploy as Docker Compose first, K8s later.
- No cross-company fan-out except an explicit `dev-team` supergroup.
- No iOS APNs in v1 (FCM only). APNs comes in v2.

## 3. Multi-tier architecture

```
[ Sources ]
   │  HTTP POST   (webhook)        ── HMAC-SHA256, API key, optional mTLS
   │  WebSocket   (client push)    ── JWT, TLS
   │  MQTT        (broker sub)     ── per-company topic, QoS 1
   ▼
[ INGEST TIER ]  (ingestd, stateless, N replicas behind LB)
   │  - AuthN/Z (API key / mTLS / JWT)
   │  - Rate limit (Redis token bucket per source)
   │  - Schema validate (per source type)
   │  - Normalize → Alert v1
   │  - Dedupe check (Redis SET NX, TTL 60s, key=hash(source|dedupe_key))
   │  - Stamp dedupe_count on alert if already seen
   │  - Publish to broker (NATS JetStream, subject: alerts.<company_id>)
   ▼
[ BROKER ]  NATS JetStream
   │  - Persistent stream `ALERTS` (replicas=3, retention 24h)
   │  - Per-company subjects for partition affinity
   ▼
[ ROUTER TIER ]  (routerd, stateless, N replicas, JetStream consumer)
   │  - Consume alert
   │  - Resolve recipients:
   │       companies → groups → individuals
   │       ∩ subscriptions (opt-in, severity ≥ user.min_severity)
   │       ∩ quiet_hours (unless severity == "inminent_colapse")
   │  - For each recipient, enqueue 1..N Delivery jobs to:
   │       deliveries.<channel>.<company_id>  (channel-specific subjects)
   ▼
[ DELIVERY TIER ]  (deliverd, one worker pool per channel)
   │  - FCM: shared FCM project, per-token sends, FCM topic for groups>50
   │  - Telegram: per-company bot token, message + inline keyboard
   │  - SMS / Voice: pluggable provider (Twilio v1)
   │  - Email: SMTP relay or SES
   │  - Slack / Teams: incoming webhook URLs
   │  - Custom outbound webhook: HMAC sign with company key
   │  - Retry: exponential backoff (1s, 2s, 4s, 8s, 16s, 32s, 64s, 128s, 256s, 512s)
   │           max 10 attempts, then → Dead-Letter Queue (DLQ)
   ▼
[ DATA TIER ]
   - PostgreSQL: tenants, sources, companies, groups, individuals,
                 subscriptions, telegram_bots, fcm_tokens, routing_rules
   - TimescaleDB: alerts (hypertable, 7d retention), deliveries
                 (hypertable, 7d retention)
   - ClickHouse: alerts_archive, deliveries_archive, aggregates
                 (indefinite, columnar, cheap to query)
   - Redis: dedupe set, rate-limit counters, idempotency keys
```

### Why 4 tiers instead of 1 monolith

- **Ingest** scales with incoming HTTP/WS/MQTT concurrency; it's CPU-cheap
  but connection-heavy. Independent replicas behind an L7 LB.
- **Router** is CPU + DB heavy (recipient expansion can be 1 → 10,000s). It
  must not be on the request path of the source system — once accepted,
  the alert is in the broker and the source is acked.
- **Delivery** is the slowest tier (third-party API calls). Each channel is
  a separate worker pool with its own concurrency, timeouts, and circuit
  breaker.
- **Broker** is the buffer. If delivery falls behind, alerts queue up
  instead of timing out the source.

## 4. Entities (Postgres schema sketch)

```sql
-- Tenancy
companies(id, name, slug, status, fcm_shared, telegram_bot_token_enc,
          rate_limit_per_sec, created_at)

-- People & devices
individuals(id, company_id, full_name, email, phone_e164, locale, tz,
            telegram_chat_id, telegram_user_id, status, created_at)
fcm_tokens(id, individual_id, token, device_id, platform, locale,
           app_version, last_seen, status, created_at)
groups(id, company_id, name, description, fcm_topic, telegram_chat_id,
       created_at)
group_members(group_id, individual_id, added_at, added_by)

-- Source allowlist
sources(id, company_id, name, type, hmac_secret_enc, api_key_hash,
        mtls_required, rate_limit_per_sec, allowed_ips cidr[],
        topic_prefix, created_at)

-- Routing
subscriptions(id, individual_id, source_id, min_severity,
              channel_mask, quiet_hours_start, quiet_hours_end, tz)
routing_rules(id, company_id, priority, match_expr, target_type,
              target_id, enabled)

-- Normalized alert (Timescale hypertable)
alerts(id, company_id, source_id, severity, category, title_key,
       body_key, data jsonb, dedupe_key, dedupe_count, received_at)
-- hypertable on received_at, 7d retention, then COPY to ClickHouse

-- Delivery attempts (Timescale hypertable)
deliveries(id, alert_id, individual_id, channel, target, status,
           attempts, last_error, next_attempt_at, completed_at)
-- hypertable on completed_at (NULL = in-flight), 7d retention
```

### Severity taxonomy (used everywhere)

| severity | sound | bypasses_quiet_hours | fcm_priority | example |
|---|---|---|---|---|
| `info` | default | no | normal | "deploy finished" |
| `warning` | alert | no | normal | "disk 80% full" |
| `critical` | siren | no | high | "service down" |
| `inminent_colapse` | klaxon | **yes** | high | "rack PDU overload imminent" |

`category` is free-form per source (e.g. `network`, `power`, `security`,
`hvac`, `process`). The Android app uses `category` to pick a custom sound
file (`res/raw/siren_<category>.ogg`).

## 5. Dedupe + dedupe_count

When `ingestd` receives alert with `dedupe_key`:

```
key = hash(source_id || ':' || dedupe_key)
SET key 1 NX EX 60       -- first arrival creates a 60s window
```

- If `SET` succeeded → first arrival, publish as new alert.
- If `SET` failed → existing window. `INCR` the counter on the key and
  attach `dedupe_count` to the published alert payload. The recipient
  sees e.g. *"Disk full (×12 in 60s)"* instead of 12 separate pushes.

The dedupe window default is 60s and is configurable per source.

## 6. Recipient resolution (the heart of the router)

For an inbound alert:

```
recipients = ∅
for each target (group or individual) implied by source.allowed_targets:
    if routing_rule matches alert.tags → override targets
    individuals = expand(target)   -- groups → members
    for each individual:
        sub = active_subscription(individual, source, alert.severity)
        if not sub: skip
        if in_quiet_hours(sub, now, alert.severity): skip
                     -- severity=inminent_colapse never skipped
        for each channel in sub.channel_mask:
            recipients += (individual, channel)
```

The router uses **prepared statement + batch IN-list** to avoid N+1.

For groups > 50 members, the router uses **FCM topic messaging**
(`/topics/<group_fcm_topic>`) which is one FCM API call regardless of
subscriber count. For smaller groups, it uses per-token sends.

## 7. FCM (shared project model)

- **One FCM project** for all companies. App must be multi-tenant aware
  (the Android app reads `data.company_id` and renders a "company badge").
- Per-individual **device tokens** registered by the Android app on login.
- Per-group **FCM topic** if group size > 50.
- Quota: FCM HTTP v1 default is 600k msg/min to a project, 1k msg/min per
  token recipient is the realistic soft cap. At 50k alerts/sec, even
  bursty, we stay well under if we use **topics for big groups** and
  **dedupe** to collapse storms. If a single company ever exceeds
  per-project quota, we can **shard into a second FCM project** —
  schema already supports `companies.fcm_shared=false`.

### 10k companies × 1k devices = 10M tokens

A single FCM project can hold this. Storage is on our side (the
`fcm_tokens` table), not FCM's. We shard the table by `company_id % N`
(partition by hash) for vacuum/index perf.

## 8. Telegram (primary channel for ops)

- Each company registers one or more **Telegram bots** (`@company_x_bot`).
  Token stored encrypted at rest (`telegram_bot_token_enc`).
- For 1:1 alerts we need the user's `telegram_chat_id` (resolved when
  user runs `/start` in the bot).
- For group alerts we use the bot in a company group chat
  (`telegram_chat_id` on the `groups` table).
- **Bot commands** (Telegram = management UI):
  - `/start` — link telegram account to `individuals.id`
  - `/mute <duration>` — e.g. `/mute 2h`, `/mute until 18:00`
  - `/unmute`
  - `/subscribe <source> <min_severity>`
  - `/unsubscribe <source>`
  - `/preferences` — show current subscriptions
  - `/status` — last N alerts received
- Telegram is bidirectional: users reply to acknowledge; ack updates
  `deliveries.status` and (optionally) closes the alert via a webhook
  callback to the source system.

## 9. Delivery retry & DLQ

```
attempt 1, fail  → wait 1s
attempt 2, fail  → wait 2s
...
attempt 10, fail → publish to dlq.<channel>.<company_id>
```

DLQ is a **JetStream stream** with no consumer in normal operation, plus
a **DLQ table** in Postgres for the admin UI. An operator can:
- View failed deliveries, with full payload + last error
- Replay (re-enqueue) one or many
- Discard

`deliverd` exposes `/health` and Prometheus metrics
(`deliverd_delivery_attempts_total{channel,status}`,
`deliverd_delivery_latency_seconds_bucket{channel}`).

## 10. Observability (operate from day 1)

- **Prometheus** metrics on every service:
  - `ingestd_alerts_received_total{source_type,result}`
  - `ingestd_dedupe_hits_total`
  - `routerd_recipient_expansion_seconds`
  - `deliverd_delivery_attempts_total{channel,status}`
  - `deliverd_dlq_total{channel}`
- **Structured logs** (slog JSON) with `trace_id`, `company_id`,
  `alert_id`, `individual_id`.
- **OpenTelemetry traces** across ingest → broker → router → delivery
  (propagate `traceparent` through NATS headers).
- **Grafana dashboards**: per-company drill-down, channel health, DLQ
  rate, p50/p95/p99 latency.

## 11. Security

- Secrets at rest: FCM service account JSON, Telegram bot tokens, HMAC
  secrets → AES-256-GCM with key from env (`BA_MASTER_KEY`).
- TLS 1.2+ everywhere; mTLS optional for HTTP sources.
- Per-source HMAC: scheme = **Stripe-style** (header
  `X-BA-Signature: t=<unix_ts>,v1=<hex_hmac>` over
  `timestamp.body`, with 5-min replay window). GitHub-style is fine too
  (header `X-Hub-Signature-256: sha256=...`) but Stripe's timestamp
  prevents replay; we ship that one as the default and accept
  GitHub-style as a per-source option.
- API keys: argon2id hashed at rest.
- Postgres row-level security optional (off by default; tenant filter
  is enforced in app code paths).

## 12. Data retention

- **alerts** (Timescale): 7 days, then COPY → ClickHouse
  `alerts_archive`, then drop chunk.
- **deliveries** (Timescale): 7 days, then COPY → ClickHouse
  `deliveries_archive`, then drop chunk.
- **companies / individuals / fcm_tokens / groups / subscriptions**:
  indefinite in Postgres. GDPR right-to-be-forgotten deletes the
  individual + cascades fcm_tokens + subscriptions.
- **ClickHouse**: indefinite. Partition by month, TTL optional later.

## 13. Region / data residency (Q24)

> "Don't understand this point, can you elaborate?"

This is about **where the data lives** and **who has to obey whose laws**:

- **Single region** (e.g. one AWS `us-east-1`): cheapest, simplest, all
  customers served from one data center. Fine if your 10k companies
  don't have legal requirements to keep data in a specific country.
- **Multi-region active-passive** (us-east-1 primary, eu-west-1
  standby): you can failover, but EU customers' data still passes
  through the US at some point. Needed if you want to claim EU
  residency for marketing.
- **Multi-region active-active** (every region runs a full stack,
  broker replicated, Postgres logical-replicated): each company pinned
  to one region. Expensive, complex, but lets you say "company X's
  alerts never leave Frankfurt."

For v1: **single region**, but pick the FCM endpoint that matches
(e.g. `fcm.googleapis.com` vs `fcm.googleapis.eu`). Document the
limitation; revisit at 1k paying companies.

## 14. Compliance (Q25)

> "Also don't know the implications of this."

Three things that matter at 10k companies:

1. **GDPR / right to be forgotten**: a `DELETE /individuals/:id` must
   hard-delete the row + cascade fcm_tokens + subscriptions. Keep a
   tombstone (id, deleted_at) in a separate `erased_individuals` table
   for 30 days so a re-created account doesn't double-deliver. ClickHouse
   archives get the individual's `id` zeroed (we keep analytics counts
   but no PII).
2. **Data Processing Agreement** (DPA) with each customer: required by
   GDPR. We are the processor; they are the controller.
3. **Encryption at rest + in transit** (already in §11). SOC2 auditors
   care; GDPR does too indirectly.

If any customer is in healthcare/finance, add HIPAA/SOC2 controls later
(audit log, BAAs). Not v1.

## 15. mTLS vs HMAC vs API key (Q8)

| mechanism | what it proves | who needs it | v1 |
|---|---|---|---|
| **API key** (`X-BA-Key: <opaque>`) | "the caller knows the secret" | any HTTP source | ✅ default, always on |
| **HMAC signature** | "the caller has the secret AND the body wasn't tampered with" | higher-trust sources, public webhooks | ✅ default-on, Stripe-style |
| **mTLS** | "the caller controls this exact client cert" | enterprise on-prem, very high trust | ✅ optional per source |

Recommendation: **API key + HMAC for everyone, mTLS opt-in for sources
that want it.** API key alone is fine for trusted internal sources. HMAC
alone is the public-webhook norm. mTLS is overkill for v1 unless a
customer asks.

## 16. MQTT (Q9) — is it the right choice?

| protocol | pros | cons | fit |
|---|---|---|---|
| **MQTT** | tiny, perfect for IoT, persistent sessions, QoS guarantees | broker to operate, clients must speak MQTT | great for sensor gateways, PLCs, on-prem devices |
| **WebSocket ingest** | no broker, plain TLS, works from any browser/curl | we have to do the reconnect logic | great for browser dashboards, ops consoles |
| **HTTP POST** | dumb simple, every device can do it | no push back, no auto-reconnect magic | great for batch sources (Prometheus, Grafana) |

**Verdict**: support all three. MQTT is the **right** choice for IoT
sources (cameras, sensors, PLCs) but not the only one. We'd use
**EMQX** (or Mosquitto) as the broker.

**Per-company topic prefix**: yes — `ba/<company_slug>/<source_id>/...`
so a misbehaving publisher can be ACL'd to its prefix only.

**QoS 1 vs 2**:
- **QoS 1** (at-least-once): broker acks on receipt, message may be
  delivered twice on reconnect. **Default for alerts.** With our dedupe
  layer this is fine.
- **QoS 2** (exactly-once): 4-step handshake, slower, ensures no dup.
  Overkill given our 60s dedupe window. Skip.

## 17. FCM topic vs per-token fan-out (Q14)

| | FCM topic messaging | Per-token send |
|---|---|---|
| 1 push to N devices | **1 FCM API call** | N FCM API calls |
| Latency at 1000 devices | ~200ms | ~30s with batching |
| Subscriptions dynamic | user subs/unsubs via app | we manage the list |
| Delivery report | aggregate only | per-token |

**Rule**: if a group has > 50 members and is the primary target,
**use FCM topic messaging**. Otherwise, per-token is fine and gives
us per-device delivery reports.

## 18. Stack

- **Language**: Go 1.22+
- **DB**: PostgreSQL 16 + TimescaleDB 2.x (active window), ClickHouse 24.x
  (archive)
- **Broker**: NATS JetStream (single-node for v1, 3-node later)
- **Cache / dedupe / rate-limit**: Redis 7
- **MQTT broker**: EMQX 5.x (Docker image)
- **Observability**: Prometheus + Grafana + Loki + OTel collector
- **Deployment**: Docker Compose (v1), Kubernetes manifests (v2)
- **Migrations**: `golang-migrate`
- **No-code admin UI** (later): React + Vite, served by `admind`

## 19. Repo layout (planned)

```
broad-announce/
├── SPEC.md                ← this file
├── ARCHITECTURE.md        ← diagrams, sequence flows, SLOs
├── PROMPT.md              ← build log, decisions, lessons
├── README.md
├── docker-compose.yml     ← one-shot local stack
├── .env.example
├── migrations/            ← sql migrations (golang-migrate)
├── ingestd/               ← binary: HTTP/WS/MQTT ingest
├── routerd/               ← binary: recipient expansion
├── deliverd/              ← binary: per-channel delivery workers
├── admind/                ← binary: admin HTTP API + (later) UI host
├── internal/
│   ├── alert/             ← Alert v1 type, validation
│   ├── broker/            ← NATS wrapper
│   ├── store/             ← Postgres + Timescale + ClickHouse
│   ├── dedupe/            ← Redis dedupe
│   ├── ratelimit/         ← Redis token bucket
│   ├── fcm/               ← FCM HTTP v1 client + topic mgmt
│   ├── telegram/          ← Bot client, command parser
│   ├── retry/             ← exponential backoff + DLQ
│   ├── observability/     ← slog + OTel + Prometheus
│   └── config/            ← env-driven config
└── deploy/
    ├── prometheus/
    ├── grafana/
    └── loki/
```

## 20. Milestones

| # | milestone | exit criterion |
|---|---|---|
| M0 | Skeleton + docker-compose up | Postgres + NATS + Redis up, all 4 services start, /health green |
| M1 | HTTP POST ingest end-to-end | send a signed webhook → FCM test push to a fake device |
| M2 | Recipient resolution | per-source `allowed_targets` honored, subscriptions applied |
| M3 | Telegram delivery + bot commands | user can `/subscribe` and receive an alert via Telegram |
| M4 | MQTT ingest | EMQX up, QoS 1, per-company topic ACLs |
| M5 | WebSocket ingest + live tail | admin UI (or wscat) sees alerts as they arrive |
| M6 | Dedupe + dedupe_count | burst of 100 identical alerts → recipient sees "×100" |
| M7 | Timescale + ClickHouse | 7d retention + archive job |
| M8 | DLQ + replay UI | operator can replay a failed delivery |
| M9 | Observability (Prom/Grafana) | 1 dashboard per tier + per-company drilldown |
| M10 | Load test 50k/s | soak 10 min, p99 ≤ 5s, zero DLQ |