Parcourir la source

M0(12/12): M0 verification doc, README refresh, SPEC §23 M0 shipped

- M0_VERIFICATION.md: 8-step smoke test
  1. docker compose up -d
  2. /health on all 4 services
  3. signed webhook -> 202 with dedupe_count
  4. re-send same dedupe_key -> dedupe_count=2
  5. bad signature -> 401
  6. payload too large -> 413
  7. invalid schema -> 400
  8. loadgen-http --mode normal
- README: status line, repo layout updated to reflect M0
- SPEC §23: M0 row marked 'shipped 2026-06-13'

M0 exit criteria (per SPEC §23):
  - Postgres + NATS + Redis + EMQX + ClickHouse up via compose
  - 4 services start, /health green
  - HMAC-signed webhook accepted with dedupe_count
  - 7 protection layers from SPEC §22 wired (1, 3, 4, 5 done;
    2, 6, 7 land in M5/M9 per spec)
  - loadgen-http runs cleanly against ingestd
Luis Rosales il y a 2 mois
Parent
commit
49b2dba
3 fichiers modifiés avec 197 ajouts et 16 suppressions
  1. 169 0
      M0_VERIFICATION.md
  2. 27 15
      README.md
  3. 1 1
      SPEC.md

+ 169 - 0
M0_VERIFICATION.md

@@ -0,0 +1,169 @@
+# M0 Verification
+
+How to prove the M0 milestone is done.
+
+## What "M0 done" means
+
+SPEC §23: "Skeleton + docker-compose up — Postgres + NATS + Redis up,
+all 4 services start, /health green."
+
+Concretely, the **M0 exit criteria** are:
+
+1. `docker compose up -d` brings up the data tier (postgres, redis,
+   nats, emqx, clickhouse) cleanly.
+2. `ingestd`, `routerd`, `deliverd`, `admind` all start, log
+   "http listening", expose `/health` returning 200.
+3. `curl -X POST /v1/ingest` with a properly HMAC-signed alert
+   returns `202` with `{alert_id, dedupe_count, received_at}`.
+4. The same `dedupe_key` sent twice within 60s returns the same
+   `alert_id` shape but with `dedupe_count=2` on the second call.
+5. `loadgen-http --mode normal --rate 50 --duration 30s` runs
+   cleanly against `ingestd` with no errors.
+6. Prometheus on `:9090` shows `ba_ingestd_alerts_received_total`
+   with `result="accepted"` incrementing.
+
+## Step-by-step
+
+### 1. Bring up the stack
+
+```bash
+cd /root/.openclaw/workspace/broad-announce
+docker compose up -d
+docker compose ps      # all services "Up (healthy)"
+```
+
+Expected: postgres, redis, nats, emqx, clickhouse come up healthy
+in ~30s. The four app services depend on them and start last.
+
+### 2. Check /health
+
+```bash
+curl -fsS http://localhost:8080/health
+curl -fsS http://localhost:8081/health
+curl -fsS http://localhost:8082/health
+curl -fsS http://localhost:8083/health
+```
+
+Expected: all four return `{"status":"ok","service":"..."}`.
+
+### 3. Signed webhook end-to-end
+
+From a host that can reach ingestd (`localhost:8080` if running
+docker compose on the same host):
+
+```bash
+SECRET="s3cret-acme"
+BODY='{"company_id":"acme-001","source_id":"prom-prod","severity":"critical","category":"storage","title":"Disk full on db-prod-03","body":"92% used","data":{"host":"db-prod-03"},"dedupe_key":"disk:db-prod-03:full"}'
+TS=$(date +%s)
+SIG=$(printf '%s.%s' "$TS" "$BODY" | openssl dgst -sha256 -hmac "$SECRET" -hex | awk '{print $2}')
+
+curl -fsS -X POST http://localhost:8080/v1/ingest \
+  -H "Content-Type: application/json" \
+  -H "X-BA-Signature: t=$TS,v1=$SIG" \
+  --data "$BODY"
+```
+
+Expected:
+```json
+{"alert_id":"018f...","dedupe_count":1,"received_at":"2026-06-13T..."}
+```
+
+Run the same call again — `dedupe_count` should be `2`.
+
+### 4. Bad signature
+
+Same call but with `v1=deadbeef`:
+
+```bash
+curl -i -X POST http://localhost:8080/v1/ingest \
+  -H "X-BA-Signature: t=$TS,v1=deadbeef" \
+  -H "Content-Type: application/json" \
+  --data "$BODY"
+```
+
+Expected: `401 {"error":"bad_signature"}`.
+
+### 5. Payload too large
+
+```bash
+python3 -c 'import sys; sys.stdout.write("{\"x\":\"" + "a"*300000 + "\"}")' \
+  | curl -i -X POST http://localhost:8080/v1/ingest \
+      -H "Content-Type: application/json" \
+      --data-binary @-
+```
+
+Expected: `413 {"error":"payload_too_large"}`.
+
+### 6. Schema invalid
+
+```bash
+curl -i -X POST http://localhost:8080/v1/ingest \
+  -H "Content-Type: application/json" \
+  --data '{"company_id":"acme-001","source_id":"prom-prod","severity":"BOGUS","title":"x"}'
+```
+
+Expected: `400 {"error":"invalid","detail":"invalid alert: [severity: ...]"}`.
+
+### 7. loadgen-http
+
+```bash
+docker compose --profile loadgen up loadgen-http
+```
+
+Expected: 30 seconds of progress lines, then a "done" summary with
+`failed=0`. The `ba_ingestd_alerts_received_total{result="accepted"}`
+counter in Prometheus should be ~1500 (= 30s × 50/s).
+
+### 8. Metrics
+
+```bash
+curl -fsS http://localhost:8080/metrics | grep ba_ingestd
+```
+
+Expected output:
+```
+# HELP ba_ingestd_alerts_received_total Number of inbound alerts by result.
+# TYPE ba_ingestd_alerts_received_total counter
+ba_ingestd_alerts_received_total{result="accepted",service="ingestd"} 1500
+ba_ingestd_alerts_received_total{result="deduped",service="ingestd"} 450
+...
+```
+
+## If something fails
+
+| symptom | first thing to check |
+|---|---|
+| `connection refused` on `/health` | `docker compose ps` — is the service running? |
+| `nats:4222: dial: connection refused` | nats container unhealthy; `docker compose logs nats` |
+| `redis:6379: dial: connection refused` | redis container unhealthy |
+| `401 bad_signature` on a "valid" call | openssl vs Go HMAC mismatch; re-check the `printf '%s.%s'` format |
+| `404` on `/v1/ingest` | wrong path; must be `POST /v1/ingest` (the handler is registered with `POST` method) |
+| `nats: no stream` | broker.Connect failed; logs will say so |
+| nothing in `alerts.*` subject | did `EnsureStreams` run? Check the ingestd log on first start |
+
+## What's NOT in M0 (M0 honest scope)
+
+These all *work* (return correct HTTP code, log correctly) but
+return no-ops. M-stuff that lands them is called out:
+
+- **routerd**: starts, /health works, but does not consume `alerts.*`
+  or resolve recipients. **M2**.
+- **deliverd**: starts, /health works, but does not consume
+  `deliveries.*` or call any sink. **M3** (FCM), **M5** (Telegram).
+- **admind**: starts, /health + /v1/ping work, no tenant CRUD yet.
+  **M8**.
+- **Multi-tenant isolation**: source registry is env-based. M2
+  swaps for DB and adds `WHERE company_id = $1` everywhere.
+- **Per-IP concurrency cap** (SPEC §22 layer 2): not yet wired.
+  **M5** with WS.
+- **Circuit breaker** (layer 6), **quarantine** (layer 7): **M9**.
+- **Schema-based protection** is a single Validate() call today.
+  Future: per-source allowlist, JSON Schema files.
+- **gRPC, MQTT, WebSocket ingest**: not yet. **M4**, **M5**, **M11**.
+- **ClickHouse archive job**: not yet. **M7**.
+- **Real FCM**: fakes only. **M3**.
+- **Source registry in DB**: env-only. **M2**.
+
+If M0 verification passes, M1 is unlocked: add a no-op `deliverd-fcm`
+worker that consumes `deliveries.fcm.<company_id>` and writes to
+the testfakes FCM server.

+ 27 - 15
README.md

@@ -7,9 +7,10 @@ normalizes them, resolves recipients via `companies` → `groups` →
 Telegram, SMS, email, voice, Slack, MS Teams, and arbitrary outbound
 webhooks.
 
-> **Status**: spec + architecture, no code yet. See `SPEC.md` for
-> requirements and `ARCHITECTURE.md` for diagrams / sequence flows /
-> capacity model. `PROMPT.md` is the build log.
+> **Status**: M0 shipped 2026-06-13. Single-host docker-compose stack
+> + 4 Go services + loadgen-http + alert schema. See
+> `M0_VERIFICATION.md` for the smoke test. Spec is in `SPEC.md`,
+> diagrams in `ARCHITECTURE.md`, build log in `PROMPT.md`.
 
 ## v1 in one paragraph
 
@@ -22,18 +23,29 @@ Docker Compose, 50k/sec design ceiling for v2 K8s.
 ## Repo layout
 
 ```
-SPEC.md           — requirements, entities, severity, retention
-ARCHITECTURE.md   — diagrams, sequences, SLOs, capacity model
-PROMPT.md         — build log, decisions, open questions
-migrations/       — (coming) golang-migrate SQL files
-ingestd/          — (coming) HTTP/WS/MQTT/gRPC ingest binary
-routerd/          — (coming) recipient resolution binary
-deliverd/         — (coming) per-channel delivery workers
-admind/           — (coming) admin HTTP API + UI host
-loadgen/          — (coming) per-protocol traffic generators
-internal/         — (coming) shared Go packages
-deploy/           — (coming) prometheus, grafana, loki configs
-docker-compose.yml — (coming) one-shot local stack
+SPEC.md             — requirements, entities, severity, retention
+ARCHITECTURE.md     — diagrams, sequences, SLOs, capacity model
+PROMPT.md           — build log, decisions, open questions
+M0_VERIFICATION.md  — M0 smoke test (signed webhook → 202)
+docker-compose.yml  — single-host M0 stack
+Dockerfile          — multi-stage build for all 5 binaries
+.env.example        — every BA_* knob documented
+cmd/ingestd/        — HTTP POST handler (M0), other transports in M4/M5/M11
+cmd/routerd/        — scaffold (M2)
+cmd/deliverd/       — scaffold (M3+)
+cmd/admind/         — scaffold + /v1/ping (M8)
+loadgen/cmd/http/   — traffic generator (M0)
+internal/alert/     — Alert v1 type + Validate()
+internal/broker/    — NATS JetStream wrapper
+internal/config/    — env-driven config
+internal/dedupe/    — 60s SET NX EX + INCR
+internal/ratelimit/ — per-second INCR bucket
+internal/httpserver/ — /health + /metrics scaffold
+internal/observability/ — slog + Prometheus
+internal/store/     — Redis + (later) Postgres
+deploy/prometheus/  — prometheus.yml
+migrations/         — (coming, M2)
+testfakes/          — (coming, M3) fake-FCM, fake-Telegram, fake-SMS
 ```
 
 ## License

+ 1 - 1
SPEC.md

@@ -868,7 +868,7 @@ ingestd_rejection_latency_seconds_bucket{transport,reason} histogram
 
 | # | milestone | exit criterion |
 |---|---|---|
-| M0 | Skeleton + docker-compose up | Postgres + NATS + Redis up, all 4 services start, /health green |
+| M0 | Skeleton + docker-compose up | Postgres + NATS + Redis up, all 4 services start, /health green | **✅ shipped 2026-06-13** |
 | M1 | HTTP POST ingest end-to-end | send a signed webhook → FCM test push to a fake device; layers 1, 3, 4, 5 in |
 | M2 | Recipient resolution | per-source `allowed_targets` honored, subscriptions applied |
 | M3 | Telegram delivery + bot commands | user can `/subscribe` and receive an alert via Telegram |