|
|
@@ -0,0 +1,273 @@
|
|
|
+# M3 Verification — Telegram delivery + bot commands
|
|
|
+
|
|
|
+This is the live smoke test for the M3 milestone
|
|
|
+(**Telegram delivery + bot commands**, SPEC §8). It exercises
|
|
|
+the new path end-to-end:
|
|
|
+
|
|
|
+1. `telegram_bots` table (one bot per company in M3)
|
|
|
+2. `individuals.telegram_chat_id` + `telegram_user_id` +
|
|
|
+ `telegram_invite_code` + `mute_until`
|
|
|
+3. **Resolver** resolves channel='telegram' targets from the
|
|
|
+ same subscription set when `channel_mask` contains
|
|
|
+ 'telegram' AND the individual has a non-null
|
|
|
+ `telegram_chat_id`
|
|
|
+4. **`telegramd`** — long-polling bot loop that handles
|
|
|
+ `/start`, `/subscribe`, `/unsubscribe`, `/preferences`,
|
|
|
+ `/status`, `/mute`, `/unmute`
|
|
|
+5. **`deliverd-telegram`** — per-channel delivery worker that
|
|
|
+ consumes `deliveries.telegram.<company_id>` and posts to
|
|
|
+ the Bot API (`sendMessage`)
|
|
|
+6. **`faketgmd`** — fake Bot API server for the smoke test
|
|
|
+7. **Channel split** — `deliverd` was renamed to `deliverd-fcm`
|
|
|
+ (per the user's M3 Q2 answer: two binaries, not a registry)
|
|
|
+8. **Smoke flow** — bot commands drive the test; alerts
|
|
|
+ drive the delivery check; faketgmd `/admin/sent` proves
|
|
|
+ the messages actually went out
|
|
|
+
|
|
|
+Hard-fail: if the resolver returns zero targets, routerd drops
|
|
|
+the alert with a log line. Same as M2.
|
|
|
+
|
|
|
+## Prerequisites
|
|
|
+
|
|
|
+- Stack is up: `docker compose up -d`
|
|
|
+- Migrations applied: `migrations/004_telegram.up.sql` is part
|
|
|
+ of the seed sidecar's `*.up.sql` walk
|
|
|
+- Seed has been run: `docker compose up seed` (apply M1, M2,
|
|
|
+ M3 migrations + all three seeds)
|
|
|
+- HMAC secret for the `acme-001:prom-prod` source is
|
|
|
+ `s3cret-acme` (matches `BA_INGESTD_SOURCES` env)
|
|
|
+- faketgmd is reachable on `:8830` (host port mapped from
|
|
|
+ the compose)
|
|
|
+
|
|
|
+## Seed summary (M3 additions on top of M2)
|
|
|
+
|
|
|
+| individual | telegram_chat_id | telegram_user_id | telegram_invite_code | notes |
|
|
|
+|---|---|---|---|---|
|
|
|
+| ind-acme-001 (Alice) | 1001 | 900001 | (burned) | **pre-linked** in `seed_m3.sql` so the very first POST exercises both fcm and telegram delivery paths |
|
|
|
+| ind-acme-002 (Bob) | NULL | NULL | `acme-bob-002` | unlinked at start; linked via `/start <code>` in step 4 |
|
|
|
+| ind-acme-003 (Carol) | NULL | NULL | `acme-carol-003` | **never linked** in M3 — exercises "telegram skipped when no chat_id" path; quiet hours 00:00–23:59 UTC keep her out of non-inminent alerts |
|
|
|
+
|
|
|
+Subscriptions: all three individuals now have
|
|
|
+`channel_mask = ["fcm","telegram"]` (M2 had just `["fcm"]`).
|
|
|
+Bob's `min_severity` is `critical` (M2 default).
|
|
|
+
|
|
|
+One telegram bot per company: `(acme-001, "primary")` with
|
|
|
+token `fake-tg-bot-token-acme-001`.
|
|
|
+
|
|
|
+## Automated smoke
|
|
|
+
|
|
|
+```bash
|
|
|
+bash scripts/m3_smoke.sh
|
|
|
+```
|
|
|
+
|
|
|
+The script:
|
|
|
+1. **Resets state** — Bob is unlinked, his subscription is
|
|
|
+ restored to `min_severity=critical`, the `deliveries`
|
|
|
+ table is truncated, and `faketgmd` is reset. **It also
|
|
|
+ `docker compose restart telegramd`** so the bot's
|
|
|
+ in-memory long-poll offset clears (see "Known quirks"
|
|
|
+ below).
|
|
|
+2. Walks steps 2–8 below and asserts the expected row counts.
|
|
|
+3. Exits 0 on success, $fail_count on failure.
|
|
|
+
|
|
|
+## Step-by-step (also documented in `M3_SMOKE_LOG.md`)
|
|
|
+
|
|
|
+### Step 1 — confirm M3 seed applied
|
|
|
+
|
|
|
+```bash
|
|
|
+docker exec -i broad-announce-postgres-1 psql -U ba -d ba -c "
|
|
|
+SELECT id, full_name, telegram_chat_id, telegram_user_id, telegram_invite_code, mute_until
|
|
|
+ FROM individuals
|
|
|
+ ORDER BY id;
|
|
|
+"
|
|
|
+```
|
|
|
+
|
|
|
+Expected: 3 rows. Alice has `telegram_chat_id=1001`,
|
|
|
+`telegram_user_id=900001`, no invite code (she was already
|
|
|
+linked). Bob and Carol have NULL chat_id/user_id and a
|
|
|
+non-null invite code.
|
|
|
+
|
|
|
+### Step 2 — send a `warning` storage alert
|
|
|
+
|
|
|
+```bash
|
|
|
+SECRET=s3cret-acme
|
|
|
+BODY='{"company_id":"acme-001","source_id":"prom-prod","severity":"warning","category":"storage","title":"Disk 80% full","body":"db-prod-04","data":{"host":"db-prod-04"},"dedupe_key":"m3-s2"}'
|
|
|
+TS=$(date +%s)
|
|
|
+SIG=$(printf '%s.%s' "$TS" "$BODY" | openssl dgst -sha256 -hmac "$SECRET" -hex | awk '{print $2}')
|
|
|
+
|
|
|
+curl -s -X POST http://localhost:8800/v1/ingest \
|
|
|
+ -H "Content-Type: application/json" \
|
|
|
+ -H "X-BA-Signature: t=$TS,v1=$SIG" \
|
|
|
+ --data "$BODY"
|
|
|
+```
|
|
|
+
|
|
|
+Wait 4s, then check `deliveries`:
|
|
|
+
|
|
|
+```bash
|
|
|
+docker exec -i broad-announce-postgres-1 psql -U ba -d ba -c "
|
|
|
+SELECT individual_id, channel, status FROM deliveries
|
|
|
+ ORDER BY id;
|
|
|
+"
|
|
|
+```
|
|
|
+
|
|
|
+**Expected**: 2 rows — Alice fcm (sent) + Alice telegram
|
|
|
+(sent). Bob has no telegram_chat_id (resolver returns no
|
|
|
+telegram endpoint for him). Carol is in quiet hours.
|
|
|
+
|
|
|
+### Step 3 — send a `critical` storage alert (Bob unlinked)
|
|
|
+
|
|
|
+```bash
|
|
|
+BODY='{"company_id":"acme-001","source_id":"prom-prod","severity":"critical","category":"storage","title":"Disk full","body":"db-prod-04","data":{"host":"db-prod-04"},"dedupe_key":"m3-s3"}'
|
|
|
+# same HMAC dance
|
|
|
+```
|
|
|
+
|
|
|
+**Expected** (cumulative): 3 more rows this step.
|
|
|
+- Alice fcm + Alice telegram
|
|
|
+- Bob fcm (Bob's min_severity=critical passes, so he gets fcm)
|
|
|
+- Bob telegram is **not** delivered because he is unlinked.
|
|
|
+
|
|
|
+Totals after step 3: fcm=3, telegram=2.
|
|
|
+
|
|
|
+### Step 4 — Bob links his Telegram account
|
|
|
+
|
|
|
+```bash
|
|
|
+curl -s -X POST http://localhost:8830/admin/queue \
|
|
|
+ -H "Content-Type: application/json" \
|
|
|
+ --data '{
|
|
|
+ "bot_token": "fake-tg-bot-token-acme-001",
|
|
|
+ "user_id": 900002,
|
|
|
+ "chat_id": 2002,
|
|
|
+ "text": "/start acme-bob-002",
|
|
|
+ "first_name":"Bob SRE"
|
|
|
+ }'
|
|
|
+```
|
|
|
+
|
|
|
+Wait 4s. The bot (`telegramd`) long-polls faketgmd, sees the
|
|
|
+queued update, runs `/start acme-bob-002` through the
|
|
|
+handler. The handler:
|
|
|
+
|
|
|
+1. Atomically `UPDATE individuals SET telegram_user_id=900002,
|
|
|
+ telegram_chat_id='2002', telegram_invite_code=NULL
|
|
|
+ WHERE telegram_invite_code='acme-bob-002'`
|
|
|
+2. Fetches Alice's name (oh wait, Bob's name) and replies
|
|
|
+ "Linked. Welcome, Bob SRE. …"
|
|
|
+
|
|
|
+**Expected**:
|
|
|
+
|
|
|
+```bash
|
|
|
+docker exec -i broad-announce-postgres-1 psql -U ba -d ba -c "
|
|
|
+SELECT id, telegram_chat_id, telegram_user_id, telegram_invite_code
|
|
|
+ FROM individuals WHERE id='ind-acme-002';"
|
|
|
+# ind-acme-002 | 2002 | 900002 | NULL
|
|
|
+
|
|
|
+curl -sS http://localhost:8830/admin/sent | jq '.items[-1]'
|
|
|
+# should be a "Linked. Welcome, Bob SRE. …" reply
|
|
|
+```
|
|
|
+
|
|
|
+### Step 5 — send another `critical` storage alert
|
|
|
+
|
|
|
+```bash
|
|
|
+BODY='{"company_id":"acme-001","source_id":"prom-prod","severity":"critical","category":"storage","title":"Disk full","body":"db-prod-04","data":{"host":"db-prod-04"},"dedupe_key":"m3-s5"}'
|
|
|
+```
|
|
|
+
|
|
|
+**Expected** (cumulative): 4 more rows this step.
|
|
|
+- Alice fcm + Alice telegram
|
|
|
+- Bob fcm + Bob telegram (Bob is now linked!)
|
|
|
+
|
|
|
+Totals after step 5: fcm=5, telegram=4.
|
|
|
+
|
|
|
+### Step 6 — send an `inminent_colapse` alert
|
|
|
+
|
|
|
+```bash
|
|
|
+BODY='{"company_id":"acme-001","source_id":"prom-prod","severity":"inminent_colapse","category":"power","title":"PDU overload imminent","body":"rack-B","data":{"host":"rack-b"},"dedupe_key":"m3-s6"}'
|
|
|
+```
|
|
|
+
|
|
|
+**Expected** (cumulative): 5 more rows this step.
|
|
|
+- Alice fcm + Alice telegram
|
|
|
+- Bob fcm + Bob telegram (Bob's min=critical passes since
|
|
|
+ inminent_colapse > critical)
|
|
|
+- Carol fcm (Carol's quiet hours 00:00–23:59 are bypassed by
|
|
|
+ inminent_colapse, but she has no telegram_chat_id, so no
|
|
|
+ telegram)
|
|
|
+
|
|
|
+Totals after step 6: fcm=8, telegram=6.
|
|
|
+
|
|
|
+### Step 7 — Bob runs `/preferences`
|
|
|
+
|
|
|
+```bash
|
|
|
+curl -s -X POST http://localhost:8830/admin/queue \
|
|
|
+ -H "Content-Type: application/json" \
|
|
|
+ --data '{
|
|
|
+ "bot_token": "fake-tg-bot-token-acme-001",
|
|
|
+ "user_id": 900002,
|
|
|
+ "chat_id": 2002,
|
|
|
+ "text": "/preferences",
|
|
|
+ "first_name":"Bob SRE"
|
|
|
+ }'
|
|
|
+```
|
|
|
+
|
|
|
+**Expected**: faketgmd receives a bot reply whose text starts
|
|
|
+with "Your subscriptions:" and lists the prom-prod row with
|
|
|
+`min=critical, channels=["fcm","telegram"]`.
|
|
|
+
|
|
|
+### Step 8 — verify faketgmd totals
|
|
|
+
|
|
|
+```bash
|
|
|
+curl -sS http://localhost:8830/admin/sent | jq '.count'
|
|
|
+# should be 8: 4 alert alerts (alice×2, alice, alice+bob) + 1 linked-reply + 1 prefs-reply = 8
|
|
|
+```
|
|
|
+
|
|
|
+(Wait, count: 4 alert notifications: step 2 alice, step 3
|
|
|
+alice, step 5 alice+bob, step 6 alice+bob = 6 alert msgs + 1
|
|
|
+linked-reply + 1 prefs-reply = 8.)
|
|
|
+
|
|
|
+## Known quirks (and why the smoke script does what it does)
|
|
|
+
|
|
|
+1. **faketgmd's `getUpdates` is a fake.** In production,
|
|
|
+ Telegram's getUpdates tracks the offset server-side and
|
|
|
+ returns updates >= offset. Our fake uses an in-memory
|
|
|
+ `offsetByBot` map. The bot (`telegramd`) keeps its offset
|
|
|
+ in a goroutine-local `var offset int64` that advances as
|
|
|
+ updates come in. The smoke script calls
|
|
|
+ `docker compose restart telegramd` after every reset so
|
|
|
+ the bot's in-process offset goes back to 0 and aligns
|
|
|
+ with faketgmd's cleared state. In production this is a
|
|
|
+ non-issue (Telegram is the real source of truth).
|
|
|
+
|
|
|
+2. **faketgmd is in-process only.** It keeps `sent`/`queue`
|
|
|
+ in memory. Restarting the container (which the smoke
|
|
|
+ script does not do by default) clears all state. M3 does
|
|
|
+ not promise persistence for the fake.
|
|
|
+
|
|
|
+3. **`deliverd` was renamed to `deliverd-fcm`.** The old
|
|
|
+ container is gone. If you have an older compose, the
|
|
|
+ `deliverd-telegram` service will start but the old
|
|
|
+ `deliverd` service won't, leaving you with FCM disabled.
|
|
|
+ Always `docker compose up -d` with the M3 compose.
|
|
|
+
|
|
|
+## What this proves about M3
|
|
|
+
|
|
|
+1. **Multi-channel resolver works.** The single CTE
|
|
|
+ `internal/routing/routing.go` UNION-ALLs an `fcm_rows`
|
|
|
+ branch and a `tg_rows` branch. `fcm_rows` joins on
|
|
|
+ `fcm_tokens`; `tg_rows` filters to `telegram_chat_id IS
|
|
|
+ NOT NULL`. Same CTE, same cost, two channels.
|
|
|
+2. **Per-channel delivery binaries are independent.**
|
|
|
+ `deliverd-fcm` and `deliverd-telegram` each subscribe to
|
|
|
+ `deliveries.<channel>.<company_id>`. Restarting one does
|
|
|
+ not affect the other.
|
|
|
+3. **Bot long-polling works.** `telegramd` reads the active
|
|
|
+ bots from `telegram_bots`, long-polls each, dispatches
|
|
|
+ commands to the handler, and replies via `sendMessage`.
|
|
|
+4. **`/start` is the only link path.** Admin creates the
|
|
|
+ individual with an invite code; user runs `/start <code>`.
|
|
|
+ No self-registration. The handler atomically claims the
|
|
|
+ code so two simultaneous `/start`s can't race.
|
|
|
+5. **Subscriptions have channel_mask.** Bob's M2 subscription
|
|
|
+ was `["fcm"]`; M3 expanded it to `["fcm","telegram"]` via
|
|
|
+ the seed migration. The resolver honors the mask.
|
|
|
+6. **Quiet hours still apply to telegram.** Carol's 00:00–23:59
|
|
|
+ window blocked her from everything except
|
|
|
+ `inminent_colapse`, where the bypass held.
|
|
|
+7. **Same severity bypass as M2.** `inminent_colapse`
|
|
|
+ bypasses quiet hours regardless of channel.
|
Luis Rosales