Acasă Explorează Ajutor
Înregistrare Autentificare
lrosales
Repozitorii Activitate publică

lrosales a împins spre master la lrosales/client2server

  • aea496d0c4 docs: render existing Mermaid blocks as SVG + PNG Used mermaid.ink to render every Mermaid block in every .md under this project. Output: docs/diagrams/<file>_<n>.{svg,png}.
  • aa0a605f7c Adopt workspace .dlog deployment protocol - Add client2server.dlog with the standing rule + entry template - Per DEPLOYMENT_LOG.md, every project in this workspace keeps a <project>.dlog as a resume point after any session interruption - Pre-protocol history (last 3 commits) is preserved at the bottom so the file is useful even before the first new entry
  • Vizualizați comparația pentru aceste 2 consemnări »

1 lună în urmă

lrosales a împins spre master la lrosales/broad-announce

  • 83a35bf48f docs: render existing Mermaid blocks as SVG + PNG Used mermaid.ink to render every Mermaid block in every .md under this project. Output: docs/diagrams/<file>_<n>.{svg,png}.
  • bb8f155fea M13b W4: integration smoke + verification doc; fix 2nd cross-tenant leak scripts/m13b_smoke.sh is the single-tenant end-to-end smoke covering all three M13b modules in one operator flow. 17 numbered steps + 3 sub-steps; runs against authd only; ingestd-dependent steps (7, 10) auto-skip when ingestd is unreachable so the smoke stays green in dev environments that don't have the full pipeline. Makefile: new targets m13b-smoke, w1-smoke, w2-smoke, w3-smoke, m13b-full (chains all four for release gating). M13b_VERIFICATION.md: ship doc with exit-criteria checklist, smoke-step table, cross-tenant isolation matrix, bundle breakdown (151 KB gz total — per-feature code-split still v1.1), M13a gate regression check (steps 1-5 manually verified 200/200/200/200/401), tests table, screenshot substitution matrix (no browser automation in this env; smoke covers the same surfaces programmatically). SECURITY (the reason this commit is bigger than expected): Writing the W4 smoke caught a SECOND cross-tenant data leak in the sources handler — same shape as the W3 listTelegramBots leak but in a different module: listSourcesHandler read tenantID from the URL path but never passed it to SourceFilter. ListSources' WHERE clause only added company_id = $N when CallerRole != "super_admin". So when super_admin called GET /v1/tenants/<A>/sources, the SQL had no company filter and returned sources from every tenant. Caught by step 8: "expected 1, got 2" — the second row was a leak from a different tenant in the same DB. Fix: - SourceFilter gained CompanyID field - ListSources emits company_id = $N unconditionally when CompanyID is set (was: only when CallerRole != "super_admin") - Empty CompanyID is now a hard error (rejects "list all" misuse by any future caller — defence in depth) - listSourcesHandler always sets CompanyID: tenantID - TestListSources_RequiresCompanyID pins the contract Verified: 3 consecutive smoke runs (no manual cleanup between) all 18/18 OK. Manual cross-tenant check: 6 tenants × 1 source each in DB, API returns 1 per tenant. This is the SAME PATTERN as the W3 leak fixed yesterday — two handlers, identical bug. Standing observation: the cross-tenant matrix from M13b_VERIFICATION.md §2 should be a template for auditing other list handlers (admind, routerd, archiverd, deliverd) in v1.1.
  • 8ef784cfd4 M13b W3: fix cross-tenant data leak in listTelegramBotsHandler listTelegramBotsHandler read tenantID from the URL path but never passed it to TelegramBotFilter. ListTelegramBots had no CompanyID field, so the SQL ran without a company_id scope — every caller got bots from every tenant. tenant_admin would have seen other tenants' bots if the role gate weren't super_admin only. Fix: - internal/authd/telegrambots.go: add CompanyID to TelegramBotFilter; emit 'company_id = $N' as the first WHERE condition when set. Documented contract: empty CompanyID means 'all tenants' — the HTTP handler is the gate. - cmd/authd/telegrambots.go: listTelegramBotsHandler now sets CompanyID: tenantID on the filter, so the SQL always scopes. - internal/authd/telegrambots_test.go: regression test TestListTelegramBots_CompanyID_RequiredForScoping that pins the no-DB-pool short-circuit and logs the contract. Caught by re-running the smoke multiple times: step 4 (initially empty) returned > 0 because the LIST leaked rows from previous runs. Verified 3 consecutive smoke runs (no manual cleanup between) all pass 32/32; manual cross-tenant check shows 6 tenants × 1 bot each via API = correct. W3 dlog updated with security note.
  • 4c956fd086 M13b W3: Telegram bot CRUD (super_admin only, bot_token write-only) Per-company telegram bot config: set / update / pause / activate / rotate-token. All 6 routes under /v1/tenants/{id}/telegram/bots gated to super_admin (matches canManageTelegram in web/src/lib/scope.ts). Bot token is write-only: server returns bot_token_set:bool instead of the plaintext on every read; plaintext is stored alongside a bcrypt hash so telegramd can read it for outbound calls. - Migration 012: bot_token_hash, welcome_message, default_source_id, description, last_rotated_at on public.telegram_bots; partial idx_telegram_bots_company (active-only) + idx_telegram_bots_default_source; trg_telegram_bots_touch_updated_at trigger. Reversible. - Backend: internal/authd/telegrambots.{go,_test.go} (validators, store, audit, ensurePublicCompanyRow bridge to public.companies same as W2); cmd/authd/telegrambots.go (HTTP handlers + 6 routes wired in main.go). - Web: web/src/features/telegram/{types,api,format,list,create-dialog, detail-page}; routes/telegram.tsx wired (was ComingSoon); sidebar W3 nav entry; 14 format tests (web/tests/telegram/). - Smoke: scripts/m13b_w3_smoke.sh, 32/32 OK end-to-end against running authd + Postgres. Smoke had a comparison bug (expected Python 'True' vs JSON 'true'); fixed. Bundle note: telegram ships in main chunk (index + forms ~59 KB gz). Per-feature dynamic import (< 30 KB gz) is a v1.1 follow-up; matches W1/W2's behaviour and is not blocking W3. Tests: go test -count=1 ./... 22 packages, 0 failures web vitest run 31 tests, 4 files, 0 failures web vite build clean bash scripts/m13b_w3_smoke.sh 32/32 OK
  • f618098114 M13b W2: Sources CRUD with one-time secrets + rotation Implements per-tenant source management in the admin UI. Picks option (a) from the W2 design discussion: reuse public.sources directly via authd, with a small bridge that idempotently inserts a public.companies row when a source is first created for an auth.tenants id (public.sources.company_id is TEXT and FKs public.companies; auth.tenants.id is UUID). Backend (Go): - internal/authd/sources.go: Source struct + filter + create input + update input + ErrSource*; ListSources, GetSource, CreateSource, UpdateSource, SetSourceStatus, RotateSecrets. The 'bridge' INSERT is the only place authd writes to public.companies. - internal/authd/sources_test.go: validators + secret format + generate. - cmd/authd/sources.go: 6 HTTP handlers (list, create, get, update, status, rotate-secrets). CreateSource + RotateSecrets return { source, secrets? }; the secrets field is OMITTED when no plaintext was generated so the UI knows not to render the one-time modal. - cmd/authd/main.go: 6 new routes under /v1/tenants/{id}/sources[/...] -- RequireAuth scope check via canAccessTenant. Schema (migration 011): - public.sources: hmac_secret_hash, api_key_hash (bcrypt cost 10), mtls_required (default false; M14 reads it), description (free text). All nullable so pre-existing rows still load. - The down migration drops the columns. Web: - web/src/features/sources/{types,api,format,list,create-dialog, detail-page}.{ts,tsx}: feature folder, mirrors companies/. One-time secrets modal: visible only on create/rotate success, with show/hide + copy per field and a forced 'I have saved these' confirmation. - web/src/components/ui/checkbox.tsx: new minimal native-input checkbox (no Radix dep added just for one screen). - web/src/routes/sources.tsx: now wires SourcesList + SourceDetailPage (was ComingSoon). - web/tests/sources/format.test.ts: 9 vitest format tests. Smoke: - scripts/m13b_w2_smoke.sh: 17-check E2E covering CRUD + status + rotate + duplicate + bad input + tenant_admin scope + cleanup. Not yet run E2E (no live stack here); bash -n clean. Tests: go test ./internal/authd/ ok (all incl. new validators); web tsc -b clean; web vitest 17/17; web vite build clean; bash -n smoke clean. Decision: public.sources reused as-is (no auth.sources view layer in W2; can refactor later if the auth/admin split needs to harden). The bridge INSERT is idempotent and only fires on the first source create for a given auth.tenants id.
  • Vizualizați comparația pentru aceste 9 consemnări »

1 lună în urmă

lrosales a împins spre master la lrosales/broad-announce

  • b2c43653e3 M13b W0: SPA shell + embed into admind Foundation for the React 19 admin console. Builds the SPA into cmd/admind/web-dist/* and embeds it via //go:embed, with a 503 stub if web/dist is empty (dev build before `make web-build`). ## Web - React 18 + Vite 5 + TypeScript 5 + Tailwind 3 + Radix + TanStack Query - pnpm-workspace.yaml (single `web` package, esbuild allowBuild) - Routes: /login, /forbidden, /, /companies/*, /sources/*, /telegram/*, /tail, /dlq, /audit, * - All non-auth routes guarded by <RequireAuth> and wrapped in <AppShell> (TopBar + Sidebar) - Each non-shipped route renders <ComingSoon> with the W1/W2/W3 badge so the next workstreams are obvious - Auth: AuthProvider with /v1/auth/refresh-then-/v1/auth/me boot, in-memory access token, httpOnly refresh cookie, refresh-on-401 with single-flight in-flight guard - API helpers: fetchWithAuth, apiGet, apiSend, ApiError - Theme: light/dark/system, persisted in localStorage, initTheme() runs before first paint - Role-based sidebar (super_admin / tenant_admin / viewer) - Login test (vitest + testing-library): renders form, accepts input - `/login` boot hint: scripts/bootstrap-super-admin.sh ## Admind - //go:embed web-dist (empty allowed; 503 stub on / if index.html missing) - wireSPA(): SPA history fallback for /, /login, /forbidden, /companies*, /sources*, /telegram*, /tail, /dlq, /audit - /assets/* served with immutable cache; / is no-cache so deploys pick up new bundles - /dlq keeps the M8 HTML UI (server-rendered); /v1/* keeps JWT gate ## Makefile - web-install (pnpm install), web-build, web-dev (vite :5173), web-test, web-typecheck - build-with-web alias: web-build then go build ## .gitignore - /web/node_modules, /web/dist, /web/.vite - /node_modules (root pnpm hoist) - /.pnpm-store, /pnpm-lock.yaml ## Verified - pnpm run build → clean (vite v5.4.21, 1660 modules, ~328 kB total / ~104 kB gz) - go build ./... + go vet ./... clean - go test ./cmd/admind/... clean - vitest: 2 passed (login form) - M8/M11/M12/M13a tests untouched, still green

1 lună în urmă

lrosales a împins spre master la lrosales/broad-announce

  • fa843980e2 M13a W5: JWT gate on routerd / archiverd / deliverd-fcm / deliverd-telegram Apply the same JWT middleware to the four remaining HTTP services, with a real admin route on each. W3 left these as out-of-scope (NATS-only consumers with no admin HTTP); W5 picks up the work. What ships: internal/dlq/query.go (new) List(ctx, pool, filters) + Get(ctx, pool, id). The SQL lives in one place, used by both per-channel admin endpoints. Filterable by channel, company_id, alert_id, discarded. List excludes the payload column; Get returns it inline (with a pgx.ErrNoRows -> (nil, nil) idiom for idempotent callers). cmd/routerd/admin.go (new) GET /v1/admin/dedupe/state — read-only, any auth POST /v1/admin/dedupe/flush — destructive, super_admin or tenant_admin. Calls Collapser.FlushAll() (the same path the shutdown drain uses). wireAdminRoutes(mux, collapser, state, logger) follows the same env-conditional pattern as admind. cmd/archiverd/admin.go (new) + runLoop refactor POST /v1/admin/archiver/run — non-blocking send on a chan struct{} (buffer 1). The loop now selects on {ctx.Done, tick.C, triggerCh}; the next iteration fires immediately. Coalescing: a second concurrent trigger returns 202 Accepted with triggered=false, reason=already_pending. Any authenticated user can fire (idempotent op; the archiver takes the Postgres advisory lock so two passes can't overlap). cmd/deliverd-fcm/admin.go (new) GET /v1/admin/dlq — dlq.List filtered to channel=fcm GET /v1/admin/dlq/{id} — dlq.Get, with a cross-channel safety net: a non-fcm row returns 404 instead of leaking data the fcm deliverd shouldn't see. cmd/deliverd-telegram/admin.go (new) Same shape, channel pinned to "telegram". cmd/{routerd,archiverd,deliverd-fcm,deliverd-telegram}/admin_test.go 16 new tests + 13 subtests, all passing: - TestWireAdminRoutes_NoSecret_Disabled (×4 services): BA_AUTHD_JWT_SECRET unset -> admin routes return 404 (route unregistered; LAN deploy path preserved) - TestWireAdminRoutes_WithSecret_Gated (×4): secret set + no token -> 401 - TestAdminDedupeState_Auth (routerd): super_admin can read the state JSON - TestAdminRolePolicy_ViewerReadOnly_AdminCanFlush (routerd): viewer can read (200), 403 on flush; super_admin/tenant_admin can flush (200) - TestAdminDedupeFlush_DrainsPending (routerd): end-to-end against a real dedupe.Collapser; Observe -> flush -> Pending() == 0 - TestAdminRunNow_Auth_Fires (archiverd): super_admin can fire a run; trigger channel receives - TestAdminRunNow_Coalesces (archiverd): pre-filled channel -> 202 with already_pending - TestAdminRolePolicy_ViewerCanTrigger (archiverd): viewer/tenant_admin/super_admin can all fire - TestAdminDLQ_AnyAuthenticatedUser_CanList (×2 deliverds): any authenticated user can read docker-compose.yml Passes BA_AUTHD_JWT_SECRET (and BA_AUTHD_ISSUER) to routerd, archiverd, deliverd-fcm, deliverd-telegram, using the same "${BA_AUTHD_JWT_SECRET:-}" empty-default pattern as the W3 services so the LAN deploy keeps working when the secret is unset. scripts/m13a_smoke.sh New checks 9–12 verify the W5 routes end-to-end against a running stack: 9. routerd /v1/admin/dedupe/{state,flush} (gated) 10. archiverd /v1/admin/archiver/run (gated) 11. deliverd-fcm /v1/admin/dlq (gated, channel=fcm) 12. deliverd-telegram /v1/admin/dlq (gated, channel=telegram) The summary table now has 4 new PASS rows. New env vars BA_ROUTERD_HTTP, BA_ARCHIVERD_HTTP, BA_DELIVERD_FCM_HTTP, BA_DELIVERD_TELEGRAM_HTTP let CI override the default ports. M13a_PLAN.md New W5 section with the design rationale and exit criteria. The W3 "Not changed" callout is updated to point to W5 instead. Status line + Recap section mention the new admin surface. SPEC.md M13 row updated to mention the W5 services and the per-service admin routes. M13a_W5_VERIFICATION.md (new) Runbook for proving W5 is done: build + test matrix, unit-test verification, full m13a_smoke check, and the backward-compat (no-secret -> 404) check. .gitignore Ignore the per-binary artifacts at the repo root (archiverd, deliverd-fcm, deliverd-telegram) and the Python __pycache__ from the smoke scripts. Test results: - go build ./... clean - go vet ./... clean - go test -count=1 ./... clean - 16 new tests + 13 subtests across 4 packages, all green - pre-existing tests untouched (M8 dlq, M11 grpc, M12, M13a W1-W4) all still green Backward compatibility: when BA_AUTHD_JWT_SECRET is unset, none of the new admin routes are registered, so the LAN deploy path behaves exactly as before. The M8 dlq.html and the existing M11/M12 surfaces are unchanged. Co-Authored-By: Jarvis <jarvis@techno-world.net>

1 lună în urmă

lrosales a împins spre master la lrosales/broad-announce

  • d066e01e5d M13a W4: docker-compose + bootstrap + E2E smoke (authd stack) Wire authd into the existing M0-M11 docker-compose stack, add the bootstrap scripts the operator needs to run authd in dev, and ship a one-command smoke that proves the end-to-end flow. What ships: Dockerfile Added /app/authd to the multi-stage build line so the binary is in the image. docker-compose.yml New 'authd' service on port 8804 (HTTPAddr). Port conflict with archiverd resolved by moving archiverd 8804 -> 8805. Shared env: BA_AUTHD_JWT_SECRET (required, read via ${BA_AUTHD_JWT_SECRET:?...} so compose fails fast). authd + admind + ingestd all read the same secret. New 'authd-data' Docker volume for the dev-only generated secret (authd auto-gens on first run in dev; persisted). admind now depends on authd (service_healthy). scripts/generate-jwt-secret.sh (new) Generates 48 random bytes -> 64-char base64, writes/updates .env. Idempotent: no-op if value already set, --force to overwrite, --print for CI. scripts/bootstrap-super-admin.sh (new) psql fallback for the first super_admin (per M13 decision 2.4: magic-link with psql fallback). bcrypt cost 10 (faster than prod's 12 for bootstrap). ON CONFLICT DO UPDATE so re-running rotates the password, doesn't duplicate. scripts/m13a_smoke.sh (new) 12-step E2E smoke. No Go runtime required, pure bash + curl + python3 (for JSON parsing). Steps: 1. authd /health -> 200 2. login (super_admin) -> 200 + tokens 3. /v1/users/me with Bearer -> 200 3a. /v1/users/me without Bearer -> 401 4. refresh -> 200, new JTI + new refresh 4a. confirm rotation (tokens differ) 5. re-use OLD refresh -> 401 session_killed 5a. confirm family killed (new refresh also rejected) 6. invite (super_admin) -> 200 6a. invite without Bearer -> 401 7. admind /v1/dlq without Bearer -> 401 7a. admind /v1/dlq with Bearer -> 200 8. ingestd /v1/admin/ingest without auth -> 401 8a. ingestd /v1/admin/ingest with auth -> gate passes Prints a summary table and exits non-zero on any failure. .env.example Documents all BA_AUTHD_* knobs (issuer, secret, TTLs, bcrypt cost, dev-only allow_generated_secret, secret_file). README.md Status line updated (M11 + F2 + M13a + M14-backend W1). File index updated to include cmd/authd, internal/authd, internal/auth. SPEC.md Milestones table: added M13 row (with status: M13a shipped, M13b-c pending) and M14 row (with status: M14-backend W1 prep shipped, W1 proper pending M12 W1, W2-W6 pending). M13a_PLAN.md W4 section rewritten to reflect what actually shipped vs the original sketch (UI smoke + Makefile target deferred to v1.1). Status line: 'W1+W2+W3+W4 SHIPPED 2026-06-17. M13a complete.' Backward compatibility: - archiverd moved 8804 -> 8805. Any local script using :8804 against archiverd needs updating. The smoke tests don't touch archiverd. - When BA_AUTHD_JWT_SECRET is unset (legacy LAN deploy), the gate is disabled and admind's /v1/dlq* is open as in M8. Tested by TestWireDLQRoutes_NoSecret_Unauthenticated. Verification: - bash -n on both new scripts: clean - go build ./... clean - go vet ./... clean - go test ./cmd/admind/ ./internal/authd/ ./cmd/ingestd/: all cached PASS (23 tests across the three packages) - docker compose --env-file .env config: parses, shows authd + the secret wired into all three services Co-Authored-By: Jarvis <jarvis@techno-world.net>

1 lună în urmă

lrosales a împins spre master la lrosales/broad-announce

  • d1dc1818df M13a W3: JWT gate on admind + shared NewFromEnv helper Apply the W2 middleware to admind's /v1/dlq* routes. Other services (routerd, archiverd, deliverd-*, telegramd) are NATS-only consumers with no admin HTTP and need no gate. What ships: internal/authd/jwkshared.go NewFromEnv() — construct a verifier-only *Authd from BA_AUTHD_JWT_SECRET + BA_AUTHD_ISSUER. No pool needed. MustNewFromEnv() — panic variant for main(). EnvEnabled() — boolean check; services use this to decide whether to wire the gate (backward compat: when unset, routes are open). cmd/admind/main.go New wireDLQRoutes(mux, br, pool, logger) extracted helper that decides per-env whether to gate the /v1/dlq* routes. - When BA_AUTHD_JWT_SECRET is set: GET /v1/dlq* — any authenticated user POST /v1/dlq/*/replay — super_admin or tenant_admin POST /v1/dlq/*/discard — super_admin or tenant_admin - When unset: open (M8 LAN-only behavior preserved) main() stays linear — no goto, no early returns. cmd/admind/main_test.go 4 tests + 9 subtests, all passing: - TestWireDLQRoutes_NoSecret_Unauthenticated: no secret → all routes accept requests (no 401) - TestWireDLQRoutes_WithSecret_Gated: secret set → all routes 401 without token - TestDLQGate_RolePolicy (9 subtests): viewer can list/get, 403 on replay/discard tenant_admin and super_admin can do everything - TestEnvEnabled: helper boolean works cmd/authd/README.md New section: 'Use as a JWT verifier from other services', documents the NewFromEnv + RequireAuth + RequireRole pattern with example code. M13a_PLAN.md W3 section rewritten to reflect what actually shipped (vs the original sketch, which had JWKS/asymmetric keys and /v1/companies/{id} scoping that belong to v2 / a later W). Backward compatibility: when BA_AUTHD_JWT_SECRET is unset, admind behaves exactly as before (LAN-only DLQ access). The M8 dlq.html / HTML UI at /dlq is unchanged. Test results: - cmd/admind: 4 tests + 9 subtests = 13 PASS - internal/authd: 13 unit + 6 integration = 19 PASS - cmd/ingestd: existing 6 admin-route tests still PASS - go build ./... and go vet clean Co-Authored-By: Jarvis <jarvis@techno-world.net>

1 lună în urmă

lrosales a împins spre master la lrosales/broad-announce

  • 24e6ea3e89 M13a W2: JWT auth middleware + admin ingest route in ingestd Other services (routerd, deliverd-*, archiverd) will use the same middleware; ingestd is the first adopter as the canonical 'a user sends an alert' path. What ships: internal/authd/middleware.go RequireAuth(next) — middleware that reads the Authorization header, validates the Bearer JWT, and stuffs the claims into the request context. RequireRole(...allowed) — composes on RequireAuth, returns 403 if the role doesn't match. ClaimsFromContext(ctx) — accessor for handlers downstream to read user_id / tenant_id / role. bearerFromRequest(r) — case-insensitive Bearer scheme parser, tolerant of extra whitespace. internal/authd/middleware_test.go 11 unit tests: no header → 401 bad scheme → 401 bad token → 401 expired token → 401 valid token → 200, claims in context alg=none token → 401 (CRITICAL, prevents confused-deputy) role allowed → 200 role forbidden → 403 no token + role-required → 401 ClaimsFromContext empty → nil Bearer scheme case-insensitive + whitespace tolerance cmd/authd/main.go (refactor) Removed the 3 hand-rolled bearer parsers; use ad.RequireAuth() and ad.RequireRole() from the middleware package. Handlers now read claims from context. The /v1/auth/logout endpoint stays unauthenticated on purpose (so a user with a dead access can still revoke their refresh). cmd/ingestd/http.go + main.go httpDeps gains an optional Authd *authd.Authd field. New RegisterAdminRoutes(mux, deps) registers POST /v1/admin/ingest wrapped in RequireAuth, exposing the same handleIngest handler through the gate. The handler logs user_id/tenant_id/role when present in context so every admin-originated alert is attributable. Wired in main.go behind BA_INGESTD_AUTHD_JWT_SECRET — when the env is unset, RegisterAdminRoutes is a no-op and ingestd behaves exactly as before (backward compatible). cmd/ingestd/http_test.go 6 tests for the admin route: no auth → 401 bad token → 401 valid token → 200 (asserts claims in context) no authd config → 404 (route not registered) expired token → 401 alg=none token → 401 (CRITICAL) Test results: 6/6 admin-route + 11/11 middleware + 13/13 unit + 6/6 integration (authd package) all green. Bugs found and fixed during development: 1. cmd/ingestd/main.go's main() returns void, so an authd.New error can't be returned — downgraded to logger.Error + skip the route registration. 2. json.Marshal(string) double-quotes the string. The test helper was marshaling a header as a string and producing invalid JWTs. Fixed by marshaling a map directly. 3. base64.RawURLEncoding is the right encoder for JWT (not std base64). Removed a hand-rolled base64 implementation that had a ReplaceAll typo. Backward compatibility: when BA_INGESTD_AUTHD_JWT_SECRET is unset, /v1/admin/ingest doesn't exist. The original POST /v1/ingest is unchanged — existing source integrations keep working. Co-Authored-By: Jarvis <jarvis@techno-world.net>

1 lună în urmă

lrosales a împins spre master la lrosales/broad-announce

  • 8f221518b5 M13a W1: authd IdP (in-house JWT + refresh-token store) Multi-tenant auth service for the M13 admin UI. No third-party auth, no SSO (deferred to v2). The only thing the rest of the system needs is 'send a Bearer <access_token>'. What ships: cmd/authd/main.go — HTTP server (port 8804) with 6 endpoints: /v1/auth/{login,refresh,logout,magic} /v1/users/{invite,me} /health, /metrics Env-driven config (BA_AUTHD_*). Secret auto-gen in dev only. cmd/authd/README.md — usage, env vars, smoke test, security model. internal/authd/authd.go — core service: JWT (HS256, alg-confusion rejected), bcrypt, magic-link gen/consume, sessions, invite flow, audit hooks. VerifyAccessToken is the public surface other services will use to authorize requests. internal/authd/store.go — pgx-based data access. All SQL lives here. Delegates refresh-token CRUD to the SQL functions in 009_auth.up.sql. migrations/009_auth.up.sql — schema 'auth' with: tenants, users, magic_links, refresh_tokens, sessions, audit_log. 4 SQL functions: generate_magic_link_token issue_refresh_token rotate_refresh_token (with re-use detection + family kill) revoke_refresh_token pgcrypto extension (gen_random_bytes, digest). Indexes + check constraints + updated_at triggers. migrations/009_auth.down.sql — DROP SCHEMA CASCADE. internal/authd/authd_test.go — 13 unit tests (no DB): bcrypt round-trip, JWT round-trip, bad-secret rejection, expired token, alg=none attack rejection, newJTI uniqueness/shape, defaults, magic-link input validation. internal/authd/store_test.go — 6 integration tests (build tag 'postgres'): create+get user, duplicate email rejected, magic-link issue+consume, login+refresh+re-use-kill+logout, invite+setpassword, audit log write. Test results: 19/19 pass (13 unit + 6 integration with real Postgres). Bugs found and fixed during development (worth knowing): SQL: 1. gen_random_bytes requires CREATE EXTENSION pgcrypto (not built-in on PG 13-). 2. digest(hex_string, 'sha256') != digest(decode(hex_string,'hex'), 'sha256'). Hash over raw bytes, not the hex-encoded string. 3. PL/pgSQL RETURNING id INTO v_id is ambiguous when there's a RETURNS TABLE(id ...) — qualify with schema.table.column. 4. PL/pgSQL functions need SET search_path = auth, public so gen_random_bytes and digest are findable when called externally. Go: 5. INET columns don't accept 'host:port' — strip the port from r.RemoteAddr before storing. 6. golang-jwt/v5: must explicitly reject non-HMAC signing methods in the keyfunc (alg=none + RS256 confused-deputy attacks). Co-Authored-By: Jarvis <jarvis@techno-world.net>

1 lună în urmă

lrosales ramură ștersă feature/m14-backend-prep la lrosales/broad-announce

1 lună în urmă

lrosales a împins spre master la lrosales/broad-announce

  • 57f838089e M14-backend W1 prep: CA scripts, mTLS verifier, incident runbook Sub-milestone: M14-backend W1 prep (cert-manager + CA) without the K8s pieces. The K8s/manifests/Helm work for W1 proper is blocked on M12 W1 (not started), so this PR ships the parts that don't need K8s. Scripts (scripts/cert-manager/): - ca-init.sh: generates offline root CA (RSA 4096, 10y) + intermediate (ECDSA P-256, 1y). Passphrase-protected keys. Outputs root + intermediate + chain + ca-bundle.json with fingerprints and notAfter. - ca-rotate-intermediate.sh: rotates the intermediate against the existing root. Refuses to run >60 days before expiry unless ROTATE_FORCE=1. Backs up the previous intermediate before overwriting. - test-certs.sh: generates throwaway test fixtures (valid, wrong-cn, no-san, untrusted) for the Go tests. - README.md: usage, security notes, related docs. - testdata/MANIFEST.txt + .gitignore: documents and ignores the test certs. Verifier (internal/auth/mtls.go): - Verifier struct wraps a *x509.CertPool (the intermediate CA) and a SourceIDResolver callback. - Verify() chain-checks + expiry + key usage + revocation list. - Sentinel errors: ErrCertExpired, ErrUntrustedIssuer, ErrMissingClientUsage, ErrRevoked. - SourceIDResolver interface + StaticSourceIDResolver default (parses CN of form 'source:<id>.<company_slug>'). - Revoke / Unrevoke / IsRevoked for in-memory CRL. Tests (internal/auth/mtls_test.go): - 9 table-driven tests, all passing: valid cert, expired cert, untrusted CA, wrong CN, no SAN, revoked (round-trip), nil cert, resolver unit tests, revoke idempotency. - Uses scripts/cert-manager/testdata fixtures; testdata is gitignored, regenerated by test-certs.sh. Runbook (docs/runbooks/mtls-incident.md): - 4 incident scenarios: cert expired, cert revoked, handshake errors spiking, private key compromise, CA compromise. - Resolution steps + diagnosis commands + preventive measures. - Each scenario has a 'jump to' anchor for fast on-call lookup. What is NOT in this PR (will be in M14 W1 proper, blocked on M12 W1): - cert-manager Helm chart install - ClusterIssuer + Certificate CRDs - serving certs for ingestd/routerd/deliverd-* - cert-manager rotation controller config What is NOT in this PR (will be in M14 W2): - ingestd source-side mTLS listener (the verifier is ready and tested; W2 wires it into the HTTP server). Co-Authored-By: Jarvis <jarvis@techno-world.net>

1 lună în urmă

lrosales a împins spre feature/m14-backend-prep la lrosales/broad-announce

  • 57f838089e M14-backend W1 prep: CA scripts, mTLS verifier, incident runbook Sub-milestone: M14-backend W1 prep (cert-manager + CA) without the K8s pieces. The K8s/manifests/Helm work for W1 proper is blocked on M12 W1 (not started), so this PR ships the parts that don't need K8s. Scripts (scripts/cert-manager/): - ca-init.sh: generates offline root CA (RSA 4096, 10y) + intermediate (ECDSA P-256, 1y). Passphrase-protected keys. Outputs root + intermediate + chain + ca-bundle.json with fingerprints and notAfter. - ca-rotate-intermediate.sh: rotates the intermediate against the existing root. Refuses to run >60 days before expiry unless ROTATE_FORCE=1. Backs up the previous intermediate before overwriting. - test-certs.sh: generates throwaway test fixtures (valid, wrong-cn, no-san, untrusted) for the Go tests. - README.md: usage, security notes, related docs. - testdata/MANIFEST.txt + .gitignore: documents and ignores the test certs. Verifier (internal/auth/mtls.go): - Verifier struct wraps a *x509.CertPool (the intermediate CA) and a SourceIDResolver callback. - Verify() chain-checks + expiry + key usage + revocation list. - Sentinel errors: ErrCertExpired, ErrUntrustedIssuer, ErrMissingClientUsage, ErrRevoked. - SourceIDResolver interface + StaticSourceIDResolver default (parses CN of form 'source:<id>.<company_slug>'). - Revoke / Unrevoke / IsRevoked for in-memory CRL. Tests (internal/auth/mtls_test.go): - 9 table-driven tests, all passing: valid cert, expired cert, untrusted CA, wrong CN, no SAN, revoked (round-trip), nil cert, resolver unit tests, revoke idempotency. - Uses scripts/cert-manager/testdata fixtures; testdata is gitignored, regenerated by test-certs.sh. Runbook (docs/runbooks/mtls-incident.md): - 4 incident scenarios: cert expired, cert revoked, handshake errors spiking, private key compromise, CA compromise. - Resolution steps + diagnosis commands + preventive measures. - Each scenario has a 'jump to' anchor for fast on-call lookup. What is NOT in this PR (will be in M14 W1 proper, blocked on M12 W1): - cert-manager Helm chart install - ClusterIssuer + Certificate CRDs - serving certs for ingestd/routerd/deliverd-* - cert-manager rotation controller config What is NOT in this PR (will be in M14 W2): - ingestd source-side mTLS listener (the verifier is ready and tested; W2 wires it into the HTTP server). Co-Authored-By: Jarvis <jarvis@techno-world.net>
  • 2cc3c273af M14: split into M14-backend + M14-ui, M14-backend runs before M13 User decision: finish all backend work before the frontend. 5 of 6 M14 workstreams are backend-only (cert-manager + ingestd mTLS + gRPC mTLS + rotation + alerts + smoke). W4 (cert UI) is the only frontend piece. Reorder: - M14-backend (W1, W2, W3, W5, W6) — runs after M12 W1, in parallel with M13a. 13-18d. - M14-ui (W4) — runs inside M13b W2 as a feature incremental on the Sources module. 3-5d. M14_SECURITY_PLAN.md §10: addendum documenting the split, updated sequencing diagram, and how SPEC.md rows should be split into 'M14 (backend)' and 'M14 (ui)'. M13b_PLAN.md W2: Sources CRUD scope grows to include the cert lifecycle UI (CSR upload, auto-generate, revoke, expiration banner). Estimate 2-3d → 4-6d. New exit criteria for the cert tab. M13c unchanged. M14-backend exit criteria unchanged. M14-ui smoke folded into M13b smoke. Co-Authored-By: Jarvis <jarvis@techno-world.net>
  • 4c13c3fdb4 M13 (frontend admin) + M14 (security hardening) — planning M13 — multi-tenant admin UI for broad-announce. Replaces dlq.html and adds CRUD + live-tail + audit + auth surfaces for super-admin and tenant-admin personas. Ships in 3 sub-milestones: M13a: authd (JWT IdP) + web/ skeleton + JWT gate 5-6d M13b: Companies + Sources + Telegram CRUD 7-8d M13c: Live tail + DLQ + Audit + K8s + Playwright E2E 7-9d Stack: Vite + React 19 + shadcn/ui + TanStack Query + RHF + Zod + SSE + openapi-fetch. Embedded in admind via embed.FS. Same-origin, no CORS, no separate frontend server. M14 — security hardening. Closes the M11.5 wishlist (mTLS + secrets management + network policies) with cert-manager + CA hierarchy + opt-in per-source mTLS + internal gRPC mTLS + cert lifecycle UI + rotation + PromQL alerts + runbook. 6 workstreams, 16-24d solo / 10-15d parallel with M13c. Survey decisions: hybrid super+tenant admin, in-house JWT (SSO post-v1), Vite SPA (literal SPEC), magic-link invites with psql fallback, no Next.js, no Redux, no Sentry, no i18n lib. Open questions flagged in M13_FRONTEND_SPEC.md §9 (users table location, JWT verification path, refresh token storage, audit log writer, authd port) — defaults documented, ready for review before M13a W1 starts. Co-Authored-By: Jarvis <jarvis@techno-world.net>
  • cbcb33b305 M11 + F2: SHIPPED with full self-defense F1 (NATS resource limit fix) + F2 (publish-path verification) together close the M11 NATS investigation. The system is now defended on three layers: - F1 cap: stream-level MaxAge/MaxBytes prevent storage from exceeding the server cap - F2 smoke: the next time the publish path diverges from receive (the M11 10-min false-positive class of bug), the smoke fails immediately - F2 alerts: NatsJetStreamStorageHigh/Critical and IngestdReceivePublishMismatch / IngestdNatsPublishErrorsHigh page on-call before the system goes red M11 dev-playground gate: 20/20 soak samples green at 7048-7610/s, p99=25ms, DLQ=0, rate and publish_ok within 1/s on every sample. Next: M12 (multi-broker NATS, 50k/s ceiling, K8s).
  • dc71b38625 fix(m11_smoke): include publish_ok column in summary table The F2 publish-rate assertion (assert_nats_publish_rate_near) was already wired into the per-minute soak log, but the final summary table at the end of the smoke was missing the publish_ok column. This adds it so the saved log file shows whether the NATS publish path tracked the receive path throughout the run. Without this, an operator reading a saved smoke log would see the 'rate' column green and assume the system is healthy, but would have to do PromQL gymnastics to verify the publish path was actually OK. Now it's right there in the table.

1 lună în urmă

lrosales a creat o ramură nouă feature/m14-backend-prep la lrosales/broad-announce

1 lună în urmă

lrosales a împins spre master la lrosales/broad-announce

  • 2cc3c273af M14: split into M14-backend + M14-ui, M14-backend runs before M13 User decision: finish all backend work before the frontend. 5 of 6 M14 workstreams are backend-only (cert-manager + ingestd mTLS + gRPC mTLS + rotation + alerts + smoke). W4 (cert UI) is the only frontend piece. Reorder: - M14-backend (W1, W2, W3, W5, W6) — runs after M12 W1, in parallel with M13a. 13-18d. - M14-ui (W4) — runs inside M13b W2 as a feature incremental on the Sources module. 3-5d. M14_SECURITY_PLAN.md §10: addendum documenting the split, updated sequencing diagram, and how SPEC.md rows should be split into 'M14 (backend)' and 'M14 (ui)'. M13b_PLAN.md W2: Sources CRUD scope grows to include the cert lifecycle UI (CSR upload, auto-generate, revoke, expiration banner). Estimate 2-3d → 4-6d. New exit criteria for the cert tab. M13c unchanged. M14-backend exit criteria unchanged. M14-ui smoke folded into M13b smoke. Co-Authored-By: Jarvis <jarvis@techno-world.net>

1 lună în urmă

lrosales a împins spre master la lrosales/broad-announce

  • 4c13c3fdb4 M13 (frontend admin) + M14 (security hardening) — planning M13 — multi-tenant admin UI for broad-announce. Replaces dlq.html and adds CRUD + live-tail + audit + auth surfaces for super-admin and tenant-admin personas. Ships in 3 sub-milestones: M13a: authd (JWT IdP) + web/ skeleton + JWT gate 5-6d M13b: Companies + Sources + Telegram CRUD 7-8d M13c: Live tail + DLQ + Audit + K8s + Playwright E2E 7-9d Stack: Vite + React 19 + shadcn/ui + TanStack Query + RHF + Zod + SSE + openapi-fetch. Embedded in admind via embed.FS. Same-origin, no CORS, no separate frontend server. M14 — security hardening. Closes the M11.5 wishlist (mTLS + secrets management + network policies) with cert-manager + CA hierarchy + opt-in per-source mTLS + internal gRPC mTLS + cert lifecycle UI + rotation + PromQL alerts + runbook. 6 workstreams, 16-24d solo / 10-15d parallel with M13c. Survey decisions: hybrid super+tenant admin, in-house JWT (SSO post-v1), Vite SPA (literal SPEC), magic-link invites with psql fallback, no Next.js, no Redux, no Sentry, no i18n lib. Open questions flagged in M13_FRONTEND_SPEC.md §9 (users table location, JWT verification path, refresh token storage, audit log writer, authd port) — defaults documented, ready for review before M13a W1 starts. Co-Authored-By: Jarvis <jarvis@techno-world.net>

1 lună în urmă

lrosales a împins spre master la lrosales/broad-announce

  • cbcb33b305 M11 + F2: SHIPPED with full self-defense F1 (NATS resource limit fix) + F2 (publish-path verification) together close the M11 NATS investigation. The system is now defended on three layers: - F1 cap: stream-level MaxAge/MaxBytes prevent storage from exceeding the server cap - F2 smoke: the next time the publish path diverges from receive (the M11 10-min false-positive class of bug), the smoke fails immediately - F2 alerts: NatsJetStreamStorageHigh/Critical and IngestdReceivePublishMismatch / IngestdNatsPublishErrorsHigh page on-call before the system goes red M11 dev-playground gate: 20/20 soak samples green at 7048-7610/s, p99=25ms, DLQ=0, rate and publish_ok within 1/s on every sample. Next: M12 (multi-broker NATS, 50k/s ceiling, K8s).

1 lună în urmă

lrosales a împins spre master la lrosales/broad-announce

  • dc71b38625 fix(m11_smoke): include publish_ok column in summary table The F2 publish-rate assertion (assert_nats_publish_rate_near) was already wired into the per-minute soak log, but the final summary table at the end of the smoke was missing the publish_ok column. This adds it so the saved log file shows whether the NATS publish path tracked the receive path throughout the run. Without this, an operator reading a saved smoke log would see the 'rate' column green and assume the system is healthy, but would have to do PromQL gymnastics to verify the publish path was actually OK. Now it's right there in the table.

1 lună în urmă

lrosales a împins spre master la lrosales/broad-announce

  • 549d1e3c83 fix(metrics): register NATSPublishTotal with prometheus The previous commit added the NATSPublishTotal field to the IngestdMetrics struct and the constructor, but forgot to add it to the reg.MustRegister() call. Counter vec created with prometheus.NewCounterVec is not auto-registered; it must be explicitly passed to MustRegister (or Register) on the prometheus.Registerer. Without this, the metric is created in memory but never exposed via /metrics, so neither the smoke assertion nor the new PromQL alert can see it.

1 lună în urmă

lrosales a împins spre master la lrosales/broad-announce

  • 8f4f2b2cbc F2: publish-success counter + smoke assertion + NATS resource alerts M11 NATS investigation medium-term (prevent recurrence): 1. New counter: ba_ingestd_nats_publish_total{result=ok|error} - IngestdMetrics.NATSPublishTotal in internal/observability - Incremented in pipeline.go at 3 sites: * PublishAsync submission error (CB path) * PublishAsync submission error (no-CB path) * observeAsyncAck: ok on broker ack * observeAsyncAck: error on broker rejection / timeout - This is the metric the M11 10-min soak was missing. Receive rate alone is not enough — gRPC receive and NATS publish are decoupled, so a broken publish path can hide behind a healthy receive metric. See M11_NATS_INVESTIGATION.md. 2. Smoke assertion: scripts/m11_lib.py::assert_nats_publish_rate_near - Now queried in the soak monitor loop (step3) - Receives a 'publish_ok' column in the per-minute log - Fails fast if NATS publish OK rate drops below target * (1-tol) - Catches the next class of this bug at smoke time 3. PromQL alert rules: deploy/prometheus/rules/nats.yml - NatsJetStreamStorageHigh: >80% of max_storage for 5m (warning) - NatsJetStreamStorageCritical: >95% for 1m (critical) - IngestdNatsPublishErrorsHigh: >5% publish errors for 2m - IngestdReceivePublishMismatch: receive rate > publish OK + 100/s Mounted into the prometheus container at /etc/prometheus/rules. prometheus.yml gains a rule_files directive. 4. docker-compose: mount the rules dir read-only into prometheus. This is the F2 layer from M11_NATS_INVESTIGATION.md. After this, the F1 fix becomes self-defending: any future regression on the publish path will be caught at smoke time AND alert time, not silently.

1 lună în urmă

lrosales a împins spre master la lrosales/broad-announce

  • ed9acf0103 M11 dev-playground gate: SHIPPED (was conditional) F1 NATS fix verified on parres 2026-06-16 14:00 EDT: - 10-min soak: 20/20 samples green at 7183-7706/s - p99: 24.9-25.0ms (when measured) - DLQ: 0 - 32 gRPC streams active throughout - 16-stream backpressure step: clean - ALERTS bytes stabilized at exactly 1,024 MiB (the 1 GiB cap) - Zero 'JetStream resource limits exceeded' log lines - Named volumes preserved after teardown (smoke fix 09d5584) The M11 row in SPEC.md flips from 'shipped conditionally' to 'shipped'. M11 dev-playground gate is done on parres. The M11 prod gate (10k/s) requires the M12 prod-shape cluster (3+ cores dedicated to ingestd) and remains pending M12. M11_NATS_INVESTIGATION.md still documents the original conditional ship + NATS root cause for historical reference. M11_VERIFICATION.md gains the F1 fix verification section.

1 lună în urmă

lrosales a împins spre master la lrosales/broad-announce

  • 09d5584dd4 fix(m11_smoke): teardown must not destroy named volumes The step 6 teardown was running: docker compose --profile loadgen-grpc down -v The -v flag removes ALL named volumes declared in the compose file (pgdata, natsdata, chdata) regardless of profile filter. This destroyed the postgres, nats, and clickhouse state every time the smoke ran. The smoke is a verification tool, not a reset; persistent service data must survive teardown. Fix: drop the -v flag. The teardown now only stops the loadgen containers; named volumes are preserved for subsequent runs. This was discovered after the F1 NATS fix (f450196 + 6c82dcf + 82dbc5a) was verified green, when the smoke teardown wiped natsdata / pgdata / chdata before the post-run NATS state inspection could run. The smoke verification was correct (10/10 samples green, 0 DLQ, ALERTS at exactly 1 GiB cap); the teardown afterwards destroyed the data. Also re-add the ALERTS cap insight to the smoke summary so a follow-up run can verify the cap is still working.

1 lună în urmă