broad-announce/M3_VERIFICATION.md

M3 Verification — Telegram delivery + bot commands

This is the live smoke test for the M3 milestone (Telegram delivery + bot commands, SPEC §8). It exercises the new path end-to-end:

1. telegram_bots table (one bot per company in M3)
2. individuals.telegram_chat_id + telegram_user_id + telegram_invite_code + mute_until
3. Resolver resolves channel='telegram' targets from the same subscription set when channel_mask contains 'telegram' AND the individual has a non-null telegram_chat_id
4. telegramd — long-polling bot loop that handles /start, /subscribe, /unsubscribe, /preferences, /status, /mute, /unmute
5. deliverd-telegram — per-channel delivery worker that consumes deliveries.telegram.<company_id> and posts to the Bot API (sendMessage)
6. faketgmd — fake Bot API server for the smoke test
7. Channel split — deliverd was renamed to deliverd-fcm (per the user's M3 Q2 answer: two binaries, not a registry)
8. Smoke flow — bot commands drive the test; alerts drive the delivery check; faketgmd /admin/sent proves the messages actually went out

Hard-fail: if the resolver returns zero targets, routerd drops the alert with a log line. Same as M2.

Prerequisites

• Stack is up: docker compose up -d
• Migrations applied: migrations/004_telegram.up.sql is part of the seed sidecar's *.up.sql walk
• Seed has been run: docker compose up seed (apply M1, M2, M3 migrations + all three seeds)
• HMAC secret for the acme-001:prom-prod source is s3cret-acme (matches BA_INGESTD_SOURCES env)
• faketgmd is reachable on :8830 (host port mapped from the compose)

Seed summary (M3 additions on top of M2)

individual	telegram_chat_id	telegram_user_id	telegram_invite_code	notes
ind-acme-001 (Alice)	1001	900001	(burned)	pre-linked in seed_m3.sql so the very first POST exercises both fcm and telegram delivery paths
ind-acme-002 (Bob)	NULL	NULL	acme-bob-002	unlinked at start; linked via /start <code> in step 4
ind-acme-003 (Carol)	NULL	NULL	acme-carol-003	never linked in M3 — exercises "telegram skipped when no chat_id" path; quiet hours 00:00–23:59 UTC keep her out of non-inminent alerts

Subscriptions: all three individuals now have channel_mask = ["fcm","telegram"] (M2 had just ["fcm"]). Bob's min_severity is critical (M2 default).

One telegram bot per company: (acme-001, "primary") with token fake-tg-bot-token-acme-001.

Automated smoke

bash scripts/m3_smoke.sh

The script:

1. Resets state — Bob is unlinked, his subscription is restored to min_severity=critical, the deliveries table is truncated, and faketgmd is reset. It also docker compose restart telegramd so the bot's in-memory long-poll offset clears (see "Known quirks" below).
2. Walks steps 2–8 below and asserts the expected row counts.
3. Exits 0 on success, $fail_count on failure.

Step-by-step (also documented in M3_SMOKE_LOG.md)

Step 1 — confirm M3 seed applied

docker exec -i broad-announce-postgres-1 psql -U ba -d ba -c "
SELECT id, full_name, telegram_chat_id, telegram_user_id, telegram_invite_code, mute_until
  FROM individuals
 ORDER BY id;
"

Expected: 3 rows. Alice has telegram_chat_id=1001, telegram_user_id=900001, no invite code (she was already linked). Bob and Carol have NULL chat_id/user_id and a non-null invite code.

Step 2 — send a warning storage alert

SECRET=s3cret-acme
BODY='{"company_id":"acme-001","source_id":"prom-prod","severity":"warning","category":"storage","title":"Disk 80% full","body":"db-prod-04","data":{"host":"db-prod-04"},"dedupe_key":"m3-s2"}'
TS=$(date +%s)
SIG=$(printf '%s.%s' "$TS" "$BODY" | openssl dgst -sha256 -hmac "$SECRET" -hex | awk '{print $2}')

curl -s -X POST http://localhost:8800/v1/ingest \
  -H "Content-Type: application/json" \
  -H "X-BA-Signature: t=$TS,v1=$SIG" \
  --data "$BODY"

Wait 4s, then check deliveries:

docker exec -i broad-announce-postgres-1 psql -U ba -d ba -c "
SELECT individual_id, channel, status FROM deliveries
 ORDER BY id;
"

Expected: 2 rows — Alice fcm (sent) + Alice telegram (sent). Bob has no telegram_chat_id (resolver returns no telegram endpoint for him). Carol is in quiet hours.

Step 3 — send a critical storage alert (Bob unlinked)

BODY='{"company_id":"acme-001","source_id":"prom-prod","severity":"critical","category":"storage","title":"Disk full","body":"db-prod-04","data":{"host":"db-prod-04"},"dedupe_key":"m3-s3"}'
# same HMAC dance

Expected (cumulative): 3 more rows this step.

• Alice fcm + Alice telegram
• Bob fcm (Bob's min_severity=critical passes, so he gets fcm)
• Bob telegram is not delivered because he is unlinked.

Totals after step 3: fcm=3, telegram=2.

Step 4 — Bob links his Telegram account

curl -s -X POST http://localhost:8830/admin/queue \
  -H "Content-Type: application/json" \
  --data '{
    "bot_token": "fake-tg-bot-token-acme-001",
    "user_id":   900002,
    "chat_id":   2002,
    "text":      "/start acme-bob-002",
    "first_name":"Bob SRE"
  }'

Wait 4s. The bot (telegramd) long-polls faketgmd, sees the queued update, runs /start acme-bob-002 through the handler. The handler:

1. Atomically UPDATE individuals SET telegram_user_id=900002, telegram_chat_id='2002', telegram_invite_code=NULL WHERE telegram_invite_code='acme-bob-002'
2. Fetches Alice's name (oh wait, Bob's name) and replies "Linked. Welcome, Bob SRE. …"

Expected:

docker exec -i broad-announce-postgres-1 psql -U ba -d ba -c "
SELECT id, telegram_chat_id, telegram_user_id, telegram_invite_code
  FROM individuals WHERE id='ind-acme-002';"
# ind-acme-002 | 2002 | 900002 | NULL

curl -sS http://localhost:8830/admin/sent | jq '.items[-1]'
# should be a "Linked. Welcome, Bob SRE. …" reply

Step 5 — send another critical storage alert

BODY='{"company_id":"acme-001","source_id":"prom-prod","severity":"critical","category":"storage","title":"Disk full","body":"db-prod-04","data":{"host":"db-prod-04"},"dedupe_key":"m3-s5"}'

Expected (cumulative): 4 more rows this step.

• Alice fcm + Alice telegram
• Bob fcm + Bob telegram (Bob is now linked!)

Totals after step 5: fcm=5, telegram=4.

Step 6 — send an inminent_colapse alert

BODY='{"company_id":"acme-001","source_id":"prom-prod","severity":"inminent_colapse","category":"power","title":"PDU overload imminent","body":"rack-B","data":{"host":"rack-b"},"dedupe_key":"m3-s6"}'

Expected (cumulative): 5 more rows this step.

• Alice fcm + Alice telegram
• Bob fcm + Bob telegram (Bob's min=critical passes since inminent_colapse > critical)
• Carol fcm (Carol's quiet hours 00:00–23:59 are bypassed by inminent_colapse, but she has no telegram_chat_id, so no telegram)

Totals after step 6: fcm=8, telegram=6.

Step 7 — Bob runs /preferences

curl -s -X POST http://localhost:8830/admin/queue \
  -H "Content-Type: application/json" \
  --data '{
    "bot_token": "fake-tg-bot-token-acme-001",
    "user_id":   900002,
    "chat_id":   2002,
    "text":      "/preferences",
    "first_name":"Bob SRE"
  }'

Expected: faketgmd receives a bot reply whose text starts with "Your subscriptions:" and lists the prom-prod row with min=critical, channels=["fcm","telegram"].

Step 8 — verify faketgmd totals

curl -sS http://localhost:8830/admin/sent | jq '.count'
# should be 8: 4 alert alerts (alice×2, alice, alice+bob) + 1 linked-reply + 1 prefs-reply = 8

(Wait, count: 4 alert notifications: step 2 alice, step 3 alice, step 5 alice+bob, step 6 alice+bob = 6 alert msgs + 1 linked-reply + 1 prefs-reply = 8.)

Known quirks (and why the smoke script does what it does)

1. faketgmd's getUpdates is a fake. In production, Telegram's getUpdates tracks the offset server-side and returns updates >= offset. Our fake uses an in-memory offsetByBot map. The bot (telegramd) keeps its offset in a goroutine-local var offset int64 that advances as updates come in. The smoke script calls docker compose restart telegramd after every reset so the bot's in-process offset goes back to 0 and aligns with faketgmd's cleared state. In production this is a non-issue (Telegram is the real source of truth).

2. faketgmd is in-process only. It keeps sent/queue in memory. Restarting the container (which the smoke script does not do by default) clears all state. M3 does not promise persistence for the fake.

3. deliverd was renamed to deliverd-fcm. The old container is gone. If you have an older compose, the deliverd-telegram service will start but the old deliverd service won't, leaving you with FCM disabled. Always docker compose up -d with the M3 compose.

What this proves about M3

1. Multi-channel resolver works. The single CTE internal/routing/routing.go UNION-ALLs an fcm_rows branch and a tg_rows branch. fcm_rows joins on fcm_tokens; tg_rows filters to telegram_chat_id IS NOT NULL. Same CTE, same cost, two channels.
2. Per-channel delivery binaries are independent. deliverd-fcm and deliverd-telegram each subscribe to deliveries.<channel>.<company_id>. Restarting one does not affect the other.
3. Bot long-polling works. telegramd reads the active bots from telegram_bots, long-polls each, dispatches commands to the handler, and replies via sendMessage.
4. /start is the only link path. Admin creates the individual with an invite code; user runs /start <code>. No self-registration. The handler atomically claims the code so two simultaneous /starts can't race.
5. Subscriptions have channel_mask. Bob's M2 subscription was ["fcm"]; M3 expanded it to ["fcm","telegram"] via the seed migration. The resolver honors the mask.
6. Quiet hours still apply to telegram. Carol's 00:00–23:59 window blocked her from everything except inminent_colapse, where the bypass held.
7. Same severity bypass as M2. inminent_colapse bypasses quiet hours regardless of channel.