|
|
@@ -394,3 +394,102 @@ What's NOT in M3 (and not supposed to be):
|
|
|
|
|
|
Pushed: <M3 commits land on master via 3 commits mirroring
|
|
|
M0/M1/M2's pattern — code, verified, bump.>
|
|
|
+
|
|
|
+
|
|
|
+**2026-06-14 — M4 shipped (MQTT ingest + per-source ACL)**
|
|
|
+
|
|
|
+What landed (~750 LoC Go + ~60 LoC config across these commits):
|
|
|
+- `internal/mqttclient/`: new package. Thin wrapper around
|
|
|
+ paho.MQTT that hides the token-on-publish option behind a
|
|
|
+ single error-returning `Publish()`, sets consistent
|
|
|
+ LastWill / MaxInflight / AutoReconnect defaults, and
|
|
|
+ formats `client_id` as `<service>-mqtt-<host>` so EMQX
|
|
|
+ `/admin/clients` shows them cleanly. Used by both
|
|
|
+ `cmd/ingestd` (subscriber) and `loadgen/cmd/mqtt` (publisher).
|
|
|
+- `cmd/ingestd/mqtt.go`: new file. The MQTT subscriber runs
|
|
|
+ every message through the same `processDeps.ProcessAlert`
|
|
|
+ pipeline as the HTTP POST handler — parse → validate →
|
|
|
+ HMAC verify → rate-limit → dedupe → publish to NATS. The
|
|
|
+ only MQTT-specific code is the topic parser
|
|
|
+ (`parseIncomingTopic` → `ba/<co>/<src>/incoming`, 4
|
|
|
+ segments, last = `incoming`) and the envelope sniffer
|
|
|
+ (accepts both `{alert, auth}` envelope and bare alert
|
|
|
+ bodies for future broker-native clients).
|
|
|
+- `cmd/ingestd/main.go`: wires up the MQTT path. New env vars
|
|
|
+ `BA_INGESTD_MQTT_BROKER`, `BA_INGESTD_MQTT_USERNAME`,
|
|
|
+ `BA_INGESTD_MQTT_PASSWORD`, `BA_INGESTD_MQTT_SUBSCRIBE`
|
|
|
+ (`ba/+/+/incoming` default).
|
|
|
+- `internal/observability/metrics.go`: adds the
|
|
|
+ `MQTTMessages` CounterVec (labels: `result=received |
|
|
|
+ accepted | deduped | bad_topic | bad_signature |
|
|
|
+ invalid_json | <other RejectReason>`). Exposed as
|
|
|
+ `ba_ingestd_mqtt_messages_total`.
|
|
|
+- `loadgen/cmd/mqtt/`: new binary. Same severity mix, dedupe
|
|
|
+ ratio, and burst mode as `loadgen-http`. Publishes to
|
|
|
+ `ba/<co>/<src>/incoming` with the `<src>-<co>` user and
|
|
|
+ HMAC secret for auth.
|
|
|
+- `deploy/emqx/acl.conf`: Erlang-term ACL rules. Each source
|
|
|
+ can only publish to its own `ba/<co>/<src>/incoming`;
|
|
|
+ `ingestd` can subscribe to `ba/+/+/incoming`; default
|
|
|
+ deny on `#`. Re-read on EMQX SIGHUP.
|
|
|
+- `deploy/emqx/auth-built-in-db-bootstrap.csv`: per-username
|
|
|
+ auth. `<source_id>-<company_id>` for sources, `ingestd`
|
|
|
+ for the subscriber. Password == HMAC secret so the same
|
|
|
+ secret serves both MQTT auth and per-message HMAC.
|
|
|
+- `deploy/emqx/README.md`: explains the auth model, the
|
|
|
+ env-var-vs-emqx.conf precedence (env vars win in EMQX
|
|
|
+ 5.x), and the `emqx_ctl listeners restart` command for
|
|
|
+ hot-reloading `acl.conf`.
|
|
|
+- `docker-compose.yml`: switches the `emqx` service from
|
|
|
+ volume-mounting `emqx.conf` (which EMQX 5.x rejects as a
|
|
|
+ *partial* config with `node.cookie required_field`) to
|
|
|
+ env-var config. The `EMQX_AUTHENTICATION__1__*` and
|
|
|
+ `EMQX_AUTHORIZATION__*` env vars are the only way to
|
|
|
+ inject chain-level config in 5.x. `acl.conf` and the
|
|
|
+ bootstrap CSV stay as volume mounts (their files are
|
|
|
+ partials, which is fine).
|
|
|
+- `scripts/m4_smoke.sh`: end-to-end smoke. 5 steps (1 alert,
|
|
|
+ 5 alerts with dedupe, bad sig, ACL violation, bad json).
|
|
|
+ Builds `loadgen-mqtt` and the 3 failure-path test
|
|
|
+ binaries into `/tmp/` on first run. 12 deliveries, 0
|
|
|
+ failures, 3 consecutive green runs.
|
|
|
+- `M4_VERIFICATION.md` + `M4_SMOKE_LOG.md`: the spec-style
|
|
|
+ step-by-step + the live run results.
|
|
|
+
|
|
|
+**Why MQTT in v1:** SPEC §18 calls for HTTP+MQTT as the
|
|
|
+v1 transports. M0–M3 ship HTTP; M4 ships MQTT. Sources that
|
|
|
+prefer a broker (Prometheus alertmanager webhook-bridge,
|
|
|
+Grafana, IoT) can publish to `ba/<co>/<src>/incoming`
|
|
|
+without writing a webhook client.
|
|
|
+
|
|
|
+**Why per-source ACL:** the broker is the first line of
|
|
|
+defense. A source that gets compromised can only spam its
|
|
|
+own topic — which still goes through the HMAC gate at
|
|
|
+ingestd, so a forged message without the secret is rejected
|
|
|
+in step 4. ACL stops *cross-tenant* spam; HMAC stops
|
|
|
+*forged messages*; rate-limit stops *flood*; dedupe stops
|
|
|
+*burst*; M9 adds the circuit breaker and quarantine.
|
|
|
+
|
|
|
+**Why env-var config beats emqx.conf:** EMQX 5.x's HOCON
|
|
|
+config is layered as `emqx.conf → base.hocon →
|
|
|
+cluster.hocon → env vars (highest precedence)`. The
|
|
|
+double-underscore separator in env-var names maps to nested
|
|
|
+HOCON keys. `emqx.conf` is a *full* config — partial
|
|
|
+overrides get rejected with `node.cookie required_field`.
|
|
|
+The env-var approach lets us set just the auth + authz
|
|
|
+chains without restating the entire base.hocon.
|
|
|
+
|
|
|
+What's NOT in M4 (and not supposed to be):
|
|
|
+- Per-IP concurrency cap (M5 with WS, SPEC §22 layer 2)
|
|
|
+- Circuit breaker + quarantine (M9)
|
|
|
+- TLS to EMQX (M11, security milestone)
|
|
|
+- Persistent sessions (M11)
|
|
|
+- Per-company bot token resolution for sources that share
|
|
|
+ a single EMQX user across companies (M3+ generalization)
|
|
|
+- HTTP-style signature in a header (MQTT has no headers;
|
|
|
+ the `auth` field in the JSON envelope is the equivalent)
|
|
|
+- QoS 2 (M11, when persistent sessions land)
|
|
|
+
|
|
|
+Pushed: bc907d9 on master (1 commit for the code; this PROMPT
|
|
|
+bump + README + SPEC bump + the EMQX env-var refactor
|
|
|
+consolidated into a single follow-up commit).
|
netbot