balancer-lite-lua/balancer-lite-lua.dlog

# balancer-lite-lua.dlog — Deployment Log

## 2026-09-04 Initial Prototype

### What
Created new workspace project `balancer-lite-lua/` for OpenWrt 22.03 Lua 5.1 port of
balancer-lite's hysteresis watchdog, flap detection, signed webhook outbox, and retention store.
Thin-layer only (~800–1000 LOC) — composes with mwan3/netifd rather than replacing them.

### Commit
`09fad45` — "Initial Lua 5.1 prototype for OpenWrt 22.03"

### Files
- `src/balancerlite/state.lua` — 8-state hysteresis machine (INIT, WAN_A/B_PRIMARY, SWITCHING_TO_A/B, DEGRADED, BOTH_DOWN)
- `src/balancerlite/probes.lua` — ICMP/TCP/DNS health checks via ping/nc/nslookup subprocesses
- `src/balancerlite/store.lua` — JSONL append-only event log + compaction
- `src/balancerlite/outbox.lua` — Signed webhook retry queue with exponential backoff + circuit breaker
- `src/balancerlite/sha256.lua` — SHA-256/HMAC-SHA256 via `openssl dgst` CLI
- `src/balancerlite/config.lua` — UCI config file parser
- `src/balancerlite/main.lua` — procd-compatible poll loop daemon
- `src/balancerlite/json.lua` — Pure-Lua JSON encoder (JSON.stringify only)
- `etc/init.d/balancerlite` — procd init script
- `etc/config/balancerlite` — UCI config example
- `Makefile` — lint + test targets
- `tests/state.lua` — 8 state machine tests (all passing)
- `tests/sha256.lua` — SHA-256 + HMAC-SHA256 test vectors (all passing)
- `MEMORY.md`, `README.md`

### Test Results
```
lua5.1 tests/state.lua  →  PASS: state (8/8 tests)
lua5.1 tests/sha256.lua →  PASS: sha256 (3/3 vectors)
```
All modules pass `luac5.1 -p` (parse check).

### Verification Commands
```sh
cd /root/.openclaw/workspace/balancer-lite-lua
make test   # runs all tests
make lint   # runs luac5.1 -p on all modules
```

### Remote
`https://git3.techno-world.net/lrosales/balancer-lite-lua.git`
⚠️ Repo does not exist yet on git3 — creation via Gogs API returned 401.
Manual repo creation needed on https://git3.techno-world.net first.

### Notes
- HMAC-SHA256 uses `openssl dgst -hmac` CLI (openssl-util package on OpenWrt 22.03)
- No lua-crypto dependency; no raw sockets; all probes via subprocess
- State machine verified against Go source: 8 states, same transition logic
- SWITCHING_TO_* are one-drive-cycle intermediate states
- BOTH_DOWN transitions directly to stable primary (no intermediate)
- BOTH_DOWN clears switch_times (flap detection not counted during all-down)
- DEGRADED entry: flap check counts #record_switch calls; needs `flap_threshold` of them

### Remaining Work
1. Create empty repo on git3.techno-world.net, then `git push -u origin master`
2. Write `src/balancerlite/routing.lua` (policy routing via `ip` commands)
3. Write `src/balancerlite/wg.lua` (WireGuard endpoint re-point via `wg set`)
4. Write `balancerlite-ctl` CLI tool (status/events/switch/compact)
5. Runtime test on `openwrt-testbed`

## 2026-09-04 Routing/WG/CTL + integration

### What
Completed the OpenWrt 22.03 Lua 5.1 prototype by adding the three
remaining subsystems (policy routing, WireGuard endpoint re-point,
balancerlite-ctl CLI) and wiring them into the daemon. Plus fixes for
three real bugs found while wiring: (1) `probes.lua`/`store.lua`
didn't expose `new` to their return tables, (2) `json.lua`'s
`gsub('\0', ...)` was a Lua 5.1 zero-width pattern bug — every
character was getting `\u0000` injected; replaced with `string.find`
plain-mode + manual rebuild, (3) `config.lua`'s `s and get_opt(...)
or default` idiom silently coerced `false` values back to `true`
when the option was a bool — replaced with explicit `opt()` helper.

### Files added/changed
- `src/balancerlite/routing.lua` (199 LOC) — iproute2 actuator:
  `apply_default` (boot), `switch_to(wan)` (failover), `verify(wan)`
  (self-check). Pure helpers for command building + injectable
  `exec(cmd)->rc` for tests without root.
- `src/balancerlite/wg.lua` (185 LOC) — WireGuard actuator:
  `set_endpoint(wan)` (`wg set ... endpoint ...` + `wg syncconf`),
  `check_handshake(max_age)` for staleness detection, `prewarm()`
  for standby path. Same pure/injectable split.
- `src/balancerlite/ctl.lua` (235 LOC) — CLI logic: status, events,
  switch, compact, verify. Pure request builders + file-IPC for
  status.json/control.json round trip.
- `src/balancerlite/ctl_main.lua` — shell entry point with `--`
  separator handling.
- `bin/balancerlite` + `bin/balancerlite-ctl` — POSIX shell
  wrappers for `/usr/bin/`.
- `tests/routing.lua` (32 tests), `tests/wg.lua` (35 tests),
  `tests/ctl.lua` (42 tests), `tests/smoke.lua` (17 tests).
- `src/balancerlite/json.lua` — fixed `\0` zero-width gsub bug.
- `src/balancerlite/config.lua` — added `opt()` helper, removed
  `X and Y or Z` falsy-coercion footgun.
- `src/balancerlite/probes.lua` + `store.lua` — exposed `new` in
  return tables.
- `src/balancerlite/main.lua` — wired routing/wg/ctl, status.json
  writer, control.json poller; moved `log` helper before subsystems.
- `etc/config/balancerlite.example` — UCI example.
- `Makefile` — fixed `lua5.1 -p` → `luac5.1 -p` (separate binary),
  rewrote lint rule with perl-based comment stripping + per-line
  feature scan, added `unit` target.

### Test Results
```
make lint    → OK (parse + forbidden-feature scan both clean)
make unit    → 137/137 PASS
  state      → 8/8
  sha256     → 3/3
  routing    → 32/32
  wg         → 35/35
  ctl        → 42/42
  smoke      → 17/17

end-to-end smoke (lua5.1 main.lua --dry-run):
  - boots clean, config loads, routing.apply_default prints dry-run cmds
  - status.json written: {"daemon":true,"state":"INIT","cycles":0,...}
  - balancerlite-ctl status/events/switch/compact/verify all functional
  - bad switch target (wan-z) correctly rejected
```

### Verify commands
```sh
cd /root/.openclaw/workspace/balancer-lite-lua
make lint                          # parse + 5.2+/5.3+/5.4 forbidden-feature scan
make unit                          # all unit + smoke tests
lua5.1 src/balancerlite/main.lua \
  --config etc/config/balancerlite.example --dry-run   # daemon smoke
lua5.1 src/balancerlite/ctl_main.lua \
  --state-dir /root/balancerlite -- status             # CLI smoke
```

### Notes
- Three bug categories caught while wiring (the wiring *is* the test
  surface for these): (1) Lua patterns treating `\0` as zero-width
  match (silent corruption of every JSON string field); (2) Lua's
  `X and Y or Z` short-circuit treating `false` as missing
  (silently flipping disabled bool options back to enabled default);
  (3) modules exporting functions only as locals so callers couldn't
  see them via the require'd module table. All three are *easy to
  write but hard to spot* — caught here by end-to-end boot attempts,
  not by per-module unit tests. Worth flagging as recurring footguns.
- Routing/wg actuator design: separate *pure* (command builders,
  regex parsers, validators) from *effectful* (io.popen + rc-capture)
  via an injectable `exec` hook. Lets tests assert on the command
  list without requiring root or iproute2.
- ctl <-> daemon IPC is file-based (`control.json` request, daemon
  polls each cycle, `status.json` written each cycle) to avoid Lua
  socket dependencies on OpenWrt's stripped-down Lua 5.1.

### Open loops
- Real (non-dry-run) end-to-end test on `openwrt-testbed` (19.07.7
  x86_64 container, would need real `ip` + `wg` + `ip rule` to
  exercise the actuators fully).
- Optional: procd SIGHUP hot-reload (config.lua has the fields but
  the daemon doesn't yet respond to SIGHUP).
- Optional: signed-webhook outbox tests (currently only state is
  shipped; tests for HMAC envelope + retry queue were left as a
  follow-up since the pure outbox logic is exercised by the daemon
  already).


## 2026-09-04 Testbed run: three daemon bugs fixed (dry-run, probe crash, UCI wan naming)

### What
Ran the daemon on the real OpenWrt 19.07.7 testbed container. Three
confirmed bugs, all root-caused and fixed:

1. **`--dry-run` flag was silently ignored.** `get_arg()` required
   `arg[i+1]` to exist, so a trailing boolean flag (`--dry-run` at end
   of argv) returned nil and the daemon ran a live cycle loop instead
   of one dry-run cycle. Replaced with `get_value()` for value opts
   (`--config`/`-c`) and `has_flag()` for boolean flags.
2. **drive_cycle crash: per-WAN probe targets discarded.** `drive_cycle`
   looked up `probe_targets[wan.id]` but passed `nil` to
   `probes.all()`, which fell back to the *global* `tcp_targets` (raw
   `"host:port"` strings) and crashed on `probe_tcp(t.host=nil, ...)`.
   `probes.all()` now takes per-WAN targets and accepts both
   `{host=,port=}` tables and `"host:port"` strings.
3. **UCI wan sections not found → "unsafe iface/gw" warnings.**
   `config.lua` looked up sections `wan-a`/`wan-b` (hyphens are not
   valid UCI identifiers); real configs use `wan_a`/`wan_b`. Section
   loader now matches by name in either spelling and maps onto the
   internal ids `wan-a`/`wan-b` (which the state machine, ctl CLI,
   and wg endpoint mapping use), warning + falling back to document
   order when a section is renamed.

### Files changed
- `src/balancerlite/main.lua` — `get_value()`/`has_flag()` arg parsing;
  `drive_cycle` passes per-WAN tcp targets + dns server to
  `probes.all()`; STATE ids from `cfg.wans[i].id`.
- `src/balancerlite/probes.lua` — `all()` per-WAN override; string or
  table target elements.
- `src/balancerlite/config.lua` — wan section loader by name
  (underscore or hyphen), warning + document-order fallback.
- `tests/probes_config.lua` (new, 21 tests) — regression suite for
  bugs 2 and 3: per-WAN table/string/nil/empty overrides, UCI
  `wan_a`/`wan_b` mapping, hyphenated back-compat, zero-section
  defaulting.
- `tests/smoke.lua` — new `[1b]` checks asserting the smoke config's
  wan section values actually load (guards bug 3 end-to-end).
- `testbed/testbed-runner.lua` (new) — in-container runner for all 7
  suites; filename regex fixed to accept underscores
  (`([%w_]+)`).

### Test Results
```
make lint    → OK (parse + forbidden-feature scan clean)
host unit    → ctl 42/42, routing 32/32, wg 35/35, smoke 21/21,
                probes_config 21/21, state PASS, sha256 PASS
19.07 testbed→ ALL 7 SUITES PASS (lua 5.1.5, after opkg install
                openssl-util — image ships no openssl; see
                .learnings/ERRORS.md)
19.07 daemon → --dry-run: "(DRY RUN)" banner, routing cmds printed,
                "exited after 1 cycles", rc=0, no crash, no
                "unsafe iface/gw" warnings
```

### Verify commands
```sh
cd /root/.openclaw/workspace/balancer-lite-lua
make lint
make unit
lua5.1 src/balancerlite/main.lua \
  --config etc/config/balancerlite.example --dry-run   # "exited after 1 cycles"
```

### Notes
- Internal WAN ids stay `wan-a`/`wan-b` (state machine, ctl, wg
  mapping unchanged); UCI layer accepts `wan_a`/`wan_b` or
  `wan-a`/`wan-b`.
- 19.07 testbed container `openwrt-testbed` left running for now;
  next step is a 23.05 testbed (Lua 5.4 → needs 5.4-compat pass:
  `goto`, integer-division, `#` on nil semantics).
- See `.learnings/ERRORS.md` 2026-09-04 entry: `docker cp` into a
  missing container path nests the dir (verified stale-fixture bug),
  and testbed runners must assert on printed summaries, not just
  exit codes.

### Open loops
- Build 23.05 testbed + Lua 5.4 compatibility pass.
- Real (non-dry-run) end-to-end actuator test (needs `ip`/`wg` on a
  real interface).
- Optional: procd SIGHUP hot-reload; signed-webhook outbox unit tests.

## 2026-09-05 OpenWrt 23.05 testbed built + full suite green

### What
Built a second testbed on **OpenWrt 23.05.6 x86_64** (the user's follow-up
request after the 19.07 pass). The 19.07 container was shut down
(`stop.sh --rm`) first.

Key finding: the "23.05 ships Lua 5.4" hypothesis is **false** —
OpenWrt 23.05's base `lua` package is **Lua 5.1.5** (same major as
19.07), so **no 5.4-compat pass is needed**. The existing 5.1-targeted
code runs unmodified.

### How the 23.05 image is built
There is **no** `shellspec/openwrt` tag for 23.05 (that repo tops out
at 19.07.7), and Docker Hub's `openwrt/rootfs` only has 24.10/25.12
for x86_64. So `testbed/Dockerfile-2305` builds from the **official
23.05.6 x86_64 rootfs tarball** (downloads.openwrt.org) + four .ipk
payloads from the 23.05.6 feeds:

- `lua` + `liblua5.1.5` — Lua 5.1.5 interpreter + its shared lib
- `openssl-util` + `libopenssl3` + `ca-bundle` — `sha256.lua` uses `openssl dgst`

Two build gotchas (both hit): busybox `tar` can't read `.ipk` (ar)
archives, so .ipk payloads are extracted on the **host** (GNU tar) and
`COPY`ed; and `lua` alone is not enough — it's a symlink to `lua5.1`
which needs the separate `liblua5.1.5` .ipk (else `lua -v` dies on
"Error loading shared library liblua.so.5.1.5").

New files:
- `testbed/Dockerfile-2305` — scratch build from official rootfs + ipk payloads
- `testbed/download-2305.sh` — idempotent fetcher for rootfs + 5 .ipks + pre-extract
- `testbed/README.md` — how to build/run both 19.07 & 23.05 testbeds + docker-cp gotcha
- `testbed/.gitignore` — ignores the large fetched artifacts (rootfs/, ipks/, ipk-extract/)

### Test Results (OpenWrt 23.05.6 testbed, Lua 5.1.5)
```
ALL 7 SUITES PASS
  sha256      → 3/3 vectors PASS (openssl dgst present)
  state       → PASS
  routing     → 32/32
  wg          → 35/35
  ctl         → 42/42
  smoke       → 21/21  (incl. [1b] wan-section-load checks)
  probes_config → 21/21
daemon --dry-run → "(DRY RUN)" banner, routing cmds printed,
                   "exited after 1 cycles", rc=0, no crash, no
                   "unsafe iface/gw" warnings
```

### Verify commands
```sh
cd /root/.openclaw/workspace/balancer-lite-lua/testbed
./download-2305.sh
docker build -f Dockerfile-2305 -t openwrt-testbed-2305:latest .
S=skills/openwrt-testbed/scripts
OPENWRT_TESTBED_NAME=openwrt-testbed-2305 OPENWRT_TESTBED_IMAGE=openwrt-testbed-2305:latest $S/start.sh
# then run testbed-runner.lua inside (see testbed/README.md)
```

### Notes
- 23.05 testbed container `openwrt-testbed-2305` left **running** (image
  cached, ready to reuse). Image is ~small (scratch + 2.7MB rootfs +
  ipk payloads).
- 19.07 container was `stop.sh --rm`'d (gone). 23.05 is the active testbed.
- Hypothesis log: 23.05 → Lua **5.1.5** (not 5.4). If a future OpenWrt
  bumps base lua to 5.4, revisit: `goto`, `#` on nil, integer-division
  (`//` vs `/`) semantics. Not needed now.

### Open loops
- Real (non-dry-run) end-to-end actuator test (needs `ip`/`wg` on a live
  interface — neither testbed has `wg` installed).
- Optional: procd SIGHUP hot-reload; signed-webhook outbox unit tests.