broad-announce/M3_SMOKE_LOG.md

M3 Smoke Test — Live Results

Run on 2026-06-14, host interserver2. All 8 steps from M3_VERIFICATION.md passed against the running docker-compose stack. Driven end-to-end by scripts/m3_smoke.sh.

Step results

#	step	expected recipients	actual	result
2	warning, db-prod-04	Alice fcm + Alice telegram (Bob unlinked, Carol quiet)	ind-acme-001 fcm, ind-acme-001 telegram	✅
3	critical, db-prod-04	Alice fcm, Alice telegram, Bob fcm (no telegram yet)	ind-acme-001 fcm, ind-acme-001 telegram, ind-acme-002 fcm	✅
4	Bob /start acme-bob-002	Bob linked (telegram_user_id=900002, chat_id='2002'); bot reply "Linked. Welcome, Bob SRE."	matches	✅
5	critical, db-prod-04	Alice fcm+telegram, Bob fcm+telegram	matches	✅
6	inminent_colapse, rack-b	Alice fcm+telegram, Bob fcm+telegram, Carol fcm (no telegram — never linked)	matches	✅
7	Bob /preferences	bot reply "Your subscriptions: …"	matches	✅
8	faketgmd /admin/sent count	≥ 5	8 (6 alert notifications + 1 linked-reply + 1 prefs-reply)	✅

Total: 14 deliveries across 4 unique alert_ids, 0 failures, 0 retries.

Per-individual / per-channel delivery table

 individual_id | channel  | status |  n
---------------+----------+--------+---
 ind-acme-001  | fcm      | sent   |  4
 ind-acme-001  | telegram | sent   |  4
 ind-acme-002  | fcm      | sent   |  3
 ind-acme-002  | telegram | sent   |  2
 ind-acme-003  | fcm      | sent   |  1

Per-alert_id delivery count

          alert_id          | count
----------------------------+-------
 0000019ec4cba19f85155bea55 |     2  ← step 2 (warning)
 0000019ec42e3b91177c90dd40 |     3  ← step 3 (critical, Bob unlinked)
 0000019ec432ffdb3e353fb5ce |     4  ← step 5 (critical, Bob linked)
 0000019ec43ea885c4fb766503 |     5  ← step 6 (inminent, Carol unlinked)

faketgmd /admin/sent log (8 messages, in order)

#	t (UTC)	chat_id	text (truncated)	source
1	03:10:27	1001 (Alice)	🟧 WARNING: Disk 80% full / db-prod-04 / — alert 0000019ec4cb	step 2 alert
2	03:10:33	1001 (Alice)	🟥 CRITICAL: Disk full / db-prod-04 / — alert 0000019ec42e	step 3 alert
3	03:10:39	2002 (Bob)	Linked. Welcome, Bob SRE. / …	step 4 bot reply
4	03:10:43	1001 (Alice)	🟥 CRITICAL: Disk full / db-prod-04 / — alert 0000019ec432	step 5 alert
5	03:10:43	2002 (Bob)	🟥 CRITICAL: Disk full / db-prod-04 / — alert 0000019ec432	step 5 alert
6	03:10:48	1001 (Alice)	🟥🟥🟥 IMMINENT: PDU overload imminent / rack-B / — alert 0000019ec43e	step 6 alert
7	03:10:48	2002 (Bob)	🟥🟥🟥 IMMINENT: PDU overload imminent / rack-B / — alert 0000019ec43e	step 6 alert
8	03:10:56	2002 (Bob)	Your subscriptions: …	step 7 bot reply

What this proves about M3

1. Multi-channel resolver works. The single CTE in internal/routing/routing.go UNION-ALLs an fcm_rows branch (joins on fcm_tokens) and a tg_rows branch (filters on telegram_chat_id IS NOT NULL). Steps 2/3 (Bob unlinked) and step 5/6 (Bob linked) prove both paths in the same query.
2. Per-channel delivery binaries are independent. deliverd-fcm consumes deliveries.fcm.<co>, deliverd-telegram consumes deliveries.telegram.<co>. Restarting one does not affect the other.
3. Bot long-polling works end-to-end. telegramd reads telegram_bots at startup, long-polls faketgmd, dispatches the command to the handler, and replies via sendMessage.
4. /start is the only link path. Admin creates the individual with an invite code; user runs /start <code>. The handler atomically claims the code; a second user with the same code would get the "already linked" reply (not tested in this smoke, but the SQL is in handler.go).
5. Subscriptions have channel_mask. Bob's M2 subscription was ["fcm"]; M3 expanded it to ["fcm","telegram"] via seed_m3.sql. The resolver honors the mask.
6. Quiet hours still apply to telegram. Carol's 00:00–23:59 window blocked her from every severity except inminent_colapse (step 6). The bypass held across both fcm and telegram delivery paths.
7. inminent_colapse bypasses quiet hours regardless of channel. Step 6's bypass is the M2 contract; M3 confirms it carries through the new telegram row in the resolver.
8. End-to-end latency (post → faketgmd sent). For step 5 (critical): 16.1s elapsed from the first delivery to the last. The 4s sleep in the script is the dominant cost; the actual ingestd→routerd→deliverd→faketgmd chain took ~12s. M9 will replace the sleeps with a polling WHERE alert_id IN (...) check.
9. 0 retries, 0 DLQ rows. Single-attempt delivery (M3 contract; M9 adds the retry chain).

Honest flags

• faketgmd is in-memory. State is lost on container restart. The smoke script docker compose restart telegramd on every run so the bot's in-process long-poll offset aligns with faketgmd's reset state. In production, Telegram's getUpdates tracks the offset server-side — this is purely a fake-server quirk.
• Bot token is plaintext in telegram_bots.bot_token. M3 is dev; AES-256-GCM at rest is part of the security milestone (M11+).
• No retry, no DLQ for telegram. Same single-attempt pattern as M1's fcm path. M9 adds the chain.
• No per-company bot token resolution in deliverd-telegram. The binary reads the env-supplied BA_TELEGRAM_BOT_TOKEN. M3 has one bot per company, so the seed makes this work; M3+ should look up the bot via the company_id encoded in the alert or the subject.
• No webhook mode. M3 ships long-polling only. Webhook mode is M5/M9.

Performance

• Resolution latency: not separately measured. M3's resolver is the same M2 CTE plus a UNION ALL branch and one extra row in the join key set; Postgres time is comparable to M2's 6ms-ish per call (rough eyeball, not benchmarked).
• End-to-end (POST → faketgmd): ~10–12s for fcm (per-channel delivery + fakefcmd round-trip) and ~10–13s for telegram (deliverd-telegram's HTTP POST to faketgmd). Dominated by the 4s sleeps in the smoke script, not by the chain.
• Bot long-poll latency: a 2s time.Sleep on errors and a 25s default timeout in faketgmd. Real Telegram times out at 30s; our fake matches.