mxaccessgw/docs/plans/2026-08-07-followups-windev-ldapfixtures-runners.md

# Follow-Ups: windev Redeploy, LDAP Test Fixtures, Runner Hygiene — Implementation Plan

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:subagent-driven-development
> (opus implementers; controller verifies ops evidence; final review pass).

**Goal:** Close the five items surfaced by the 2026-08-07 live-actions cycle: repair windev's
crash-looping gateway and finish the deferred SEC-36 dashboard verification (NEXT-07), fix the
DashboardLdapLiveTests fixture drift (NEXT-06), resolve the unexpected macOS instance runner,
harden runner-1's plaintext registration token, and verify the cargo Bearer fix.

**Architecture:** Three independent live streams (windev serial: 1→2→3; repo test fix: 4;
Gitea/runner hygiene: 5, 6) run concurrently; task 7 closes out docs/trackers. No contract,
gateway-logic, or client changes — one test-file edit (Task 4) plus live ops plus docs.

**Tech Stack:** SSH + PowerShell `-EncodedCommand` (windev 10.100.0.48), SSH + docker compose
(10.100.0.35), Gitea admin API (`gitea.dohertylan.com`, token via `~/.zshenv` `GITEA_TOKEN`),
NSSM, xUnit live-LDAP suite, GLAuth at `10.100.0.35:3893`.

---

## Preflight facts (verified before planning)

- Unpushed local mxaccessgw commits: `0566716`, `9760497`, `5b153da`, `41e8648` (all docs-only).
  **`origin/main` = `a346d51`** — contains all current code, so windev can build from
  `origin/main` without any push.
- windev (`10.100.0.48`): NSSM service `MxAccessGw`; deployed Server build of 2026-06-25
  (Auth 0.1.2.0, supports auth-DB schema 2) crash-loops on
  `C:\ProgramData\MxGateway\gateway-auth.db` migrated to schema 3 on 2026-07-15
  (`AuthStoreMigrationException`, ~3.8k–10k Hosting-failed events/day). The NEW LDAP secret is
  already staged as the 10th `AppEnvironmentExtra` entry (SEC-36 Task 3) — preserve it.
- `DashboardLdapLiveTests.cs` (`src/ZB.MOM.WW.MxGateway.IntegrationTests/`): uses
  `admin`/`admin123` (3 tests) and `readonly`/`readonly123`. Directory reality
  (`scadaproj/infra/glauth/config.toml`): `admin` exists, password is the standard dev test
  password (`password`, hash `5e884898…42d8` — same as `multi-role`), and IS in GwAdmin
  (othergroups `[5610, 5701]`); `readonly` does not exist; `gw-viewer` (primarygroup 5611 =
  GwReader, NOT GwAdmin) is the natural not-an-admin fixture. Test binds
  `MxGateway:Ldap` from `appsettings.json` (**`Server: localhost`**) + env overrides — so the
  live run needs `MxGateway__Ldap__Server=10.100.0.35` as well as
  `MxGateway__Ldap__ServiceAccountPassword` (from Mac user-secrets, never printed).
- Gitea instance runners (`GET /api/v1/admin/actions/runners`): id 1 `gitea-runner` (cap 4),
  id 4 `macos-local-Josephs-MacBook-Pro` (**unexpected, online, labels overlap
  ubuntu-latest**), id 5 `gitea-runner-2` (cap 2).
- `10.100.0.35:/opt/gitea/docker-compose.yml` (+ `docker-compose.yml.bak-tst30`): runner-1's
  registration token inline in plaintext env, file world-readable. runner-2 uses
  `GITEA_RUNNER_REGISTRATION_TOKEN_FILE: /run/secrets/runner_token` ← 0600
  `/opt/gitea/runner_token`. runner-1 data volume `/opt/gitea/runner:/data` (its `.runner`
  credential persists — the registration env is only needed for first registration).
- Cargo Bearer fix already applied (`~/.zshenv`, backup `~/.zshenv.bak-cli39`) and documented
  (`docs/ClientPackaging.md`, commit `5b153da`). Task 7 verifies; no further action expected.

## Secret hygiene (binding, all tasks)

- Never print the LDAP service-account password, `GITEA_TOKEN`, cargo token, runner
  registration tokens, or API keys — not in commands, logs, commits, or reports. Read the LDAP
  password from `dotnet user-secrets list` into an env var without echoing
  (e.g. `export MxGateway__Ldap__ServiceAccountPassword="$(dotnet user-secrets list --project src/ZB.MOM.WW.MxGateway.Server | awk -F' = ' '/ServiceAccountPassword/ {print $2}')"`).
- Documented dev **test users** (`multi-role`/`password`, `admin`/`password`,
  `gw-viewer`/`password`) are NOT secrets — glauth.md publishes them; fine in code/commits.
- SSH→windev PowerShell: always `powershell -NoProfile -EncodedCommand <base64-UTF16LE>`;
  never put secrets inside EncodedCommand blobs or argv.

---

### Task 1: NEXT-07 — Recon windev deployment layout + schema support

**Classification:** standard — read-only recon, but its output gates a service redeploy
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 4, Task 5, Task 6

**Files:** none edited. SSH recon on `10.100.0.48` + repo/scadaproj reads on the Mac.

Determine everything Task 2 needs, and confirm the fresh-deploy path is safe:

1. `nssm get MxAccessGw Application`, `AppDirectory`, `AppParameters`,
   `AppEnvironmentExtra` (count entries; do NOT print values of secret-bearing entries —
   names only).
2. Inventory the deployed dir: path, `ZB.MOM.WW.MxGateway.Server.exe` timestamp, whether
   `appsettings.json`/`appsettings.Production.json` in the deploy dir differ from repo
   `origin/main` (diff; windev-specific config must survive the redeploy).
3. Confirm build feasibility on windev: `dotnet --list-sdks` (need 10.x), locate an existing
   mxaccessgw checkout/worktree (CI uses `scripts/ci/windev-worker-ci.ps1` — find its
   worktree path) or pick a fresh clone location. Confirm `git fetch` reaches `origin/main`
   = `a346d514dd24e775640e5667aa7cd8e561fec68a`.
4. Confirm current code supports auth-DB schema 3: find the auth-store supported-schema
   constant (ZB.MOM.WW.Auth packages — check the package version the Server at `origin/main`
   references, and/or the migration code in the shared scadaproj libs) and state the
   evidence. **If current code does NOT support schema 3, STOP — report, do not deploy.**
5. Gateway endpoints for verification: bound URLs/ports (from deployed config/env), dashboard
   scheme (http vs https → cookie will be `MxGatewayDashboard` vs `__Host-…`).
6. Check what migrated the DB to schema 3 on 2026-07-15 (event log / file timestamps) — only
   to confirm schema 3 is the shared-lib current version, not an anomaly.

**Step: report** all findings as structured text (no secrets); no changes, no commits.

### Task 2: NEXT-07 — Build current Server on windev and redeploy the service

**Classification:** high-risk — replaces a running (crash-looping) service's binaries
**Estimated implement time:** ~10 min
**Parallelizable with:** none (needs Task 1)

**Files:** none in repo. windev filesystem + NSSM only.

Using Task 1's facts:

1. On windev, fetch/checkout `origin/main` (`a346d51…`) in the build worktree/clone.
2. `dotnet publish src/ZB.MOM.WW.MxGateway.Server -c Release` (match deployed layout/RID from
   Task 1; framework-dependent vs self-contained must match what NSSM `Application` points at).
3. Stop the service (`nssm stop MxAccessGw`), confirm process exited.
4. Backup: deployed dir → sibling `*.bak-next07` copy; copy
   `C:\ProgramData\MxGateway\gateway-auth.db` (+ `-wal`/`-shm` if present) to
   `gateway-auth.db.bak-next07`. **Never delete the live DB.**
5. Deploy publish output over the deploy dir, then restore any windev-specific config files
   identified in Task 1 (do not clobber live overrides; NSSM env entries are untouched by
   file copies but verify count unchanged after start).
6. `nssm start MxAccessGw`; verify: service state RUNNING and stable ≥60 s (no restart
   cycle), Application event log shows clean host start and **zero new
   `AuthStoreMigrationException` / `Hosting failed to start`** after the start timestamp,
   bound port answers (e.g. dashboard root or health endpoint returns HTTP).
7. Rollback if unhealthy: stop, restore `*.bak-next07` dir, start, report.

**Step: report** deployed SHA, verification evidence, backup paths. No repo commits.

### Task 3: SEC-36 deferred verification + NEXT-07/runbook closeout

**Classification:** standard
**Estimated implement time:** ~6 min
**Parallelizable with:** none (needs Task 2)

**Files:**
- Modify: `docs/runbooks/SEC-36-ldap-credential-rotation.md` (Correction 3 — mark the
  deferred dashboard check done, dated)
- Modify: `archreview/2026-07-12/remediation/90-candidate-findings-next-cycle.md` (NEXT-07 →
  resolved 2026-08-07, evidence one-liner)

1. Complete SEC-36 step 4 against the repaired windev gateway: log in to the dashboard as
   `multi-role`/`password` end-to-end. Preferred: `curl` flow — GET `/login` (capture
   antiforgery token + cookie), POST credentials, expect success redirect + auth cookie
   (name per Task 1 scheme). If the login page resists scripting (Blazor circuit), report
   exactly why and fall back to asserting a fresh `DashboardLdapLiveTests` green run
   (Task 4) plus windev log evidence of successful LDAP bind on a manual attempt.
2. Update the two docs; commit locally (`docs(sec-36,next-07): …`), do NOT push.

### Task 4: NEXT-06 — Fix DashboardLdapLiveTests fixtures to match the shared directory

**Classification:** small — one test file, but must go green against live GLAuth
**Estimated implement time:** ~6 min
**Parallelizable with:** Task 1, Task 5, Task 6

**Files:**
- Modify: `src/ZB.MOM.WW.MxGateway.IntegrationTests/DashboardLdapLiveTests.cs`
- Modify: `archreview/2026-07-12/remediation/90-candidate-findings-next-cycle.md` (NEXT-06 →
  resolved)
- Possibly modify: `docs/GatewayTesting.md` (live-LDAP opt-in row: document the
  `MxGateway__Ldap__Server` override needed when GLAuth is not localhost)

1. Read `DashboardAuthenticator` first: confirm a user who binds successfully but maps to no
   role yields `Succeeded == false` (drives the gw-viewer fixture).
2. Fix fixtures: `admin`/`admin123` → `admin`/`password` (positive + wrong-password +
   unreachable tests); `readonly`/`readonly123` → `gw-viewer`/`password` (exercises
   user-binds-but-lacks-GwAdmin; keep the no-password-leak assertion, updating the asserted
   literal). Update XML doc comments to match. Keep MXAccess-repo style rules
   (TreatWarningsAsErrors).
3. Build: `dotnet build src/ZB.MOM.WW.MxGateway.IntegrationTests` (macOS OK — net10.0).
4. Live run (env only, never echo the password):
   `MXGATEWAY_RUN_LIVE_LDAP_TESTS=1 MxGateway__Ldap__Server=10.100.0.35 MxGateway__Ldap__ServiceAccountPassword=<from user-secrets> dotnet test … --filter FullyQualifiedName~DashboardLdapLiveTests`
   → expect **5/5 passed** (this is also positive live proof of the SEC-36 service-account
   bind).
5. Update tracker row (+ GatewayTesting.md if the Server-override note is missing); commit
   locally (`test(ldap): …`), do NOT push.

### Task 5: Resolve the unexpected macOS instance runner (id 4)

**Classification:** standard — evidence-gated removal of a live runner registration
**Estimated implement time:** ~6 min
**Parallelizable with:** Task 1, Task 4, Task 6

**Files:**
- Modify: `docs/runbooks/TST-30-second-ci-runner.md` (the Correction paragraph mentions
  "id 4 — an unrelated local macOS runner" — update to final state)

1. Evidence, local: `pgrep -fl act_runner`, `launchctl list | grep -i act`,
   `brew services list | grep -i act`, look for `~/.runner`/act_runner config dirs. Evidence,
   Gitea (token from `~/.zshenv`, never printed): runner detail for id 4 (labels, last
   online), and whether any recent runs' jobs report `runner_id == 4`
   (`GET /repos/{owner}/{repo}/actions/runs?…` → `…/runs/{id}/jobs` for both `mxaccessgw`
   and `lmxopcua` recent runs).
2. Decision rule: the runner advertises ubuntu labels from a macOS host, so it can steal
   Linux container jobs → **remove it** unless evidence shows it deliberately serves jobs
   the docker runners cannot (none expected). Removal = stop the local act_runner process
   AND disable its autostart (launchd/brew), then `DELETE /api/v1/admin/actions/runners/4`.
   Keep the local config file (renamed `*.disabled-2026-08-07`) so re-registering with
   mac-specific labels stays easy; note the re-registration recipe in the runbook edit.
3. Verify: admin runner list shows only ids 1 and 5, both online; no act_runner process
   locally; a `pgrep` after 60 s still empty (nothing respawned).
4. Update the TST-30 runbook correction paragraph; commit locally, do NOT push.

### Task 6: Harden runner-1's registration token on 10.100.0.35

**Classification:** standard — touches the live CI stack's compose file
**Estimated implement time:** ~7 min
**Parallelizable with:** Task 1, Task 4, Task 5

**Files:** none in repo (host `/opt/gitea/` only; runbook note lands in Task 7 if needed).

1. Preconditions on the host: confirm runner-1's `/data/.runner` exists in its volume
   (registration credential persists → the registration env var is no longer needed);
   confirm both runners idle (no `act_runner`-spawned job containers, no in-progress runs
   via API) before recreating.
2. Edit `/opt/gitea/docker-compose.yml` (backup first → `docker-compose.yml.bak-tst30b`):
   replace runner-1's inline `GITEA_RUNNER_REGISTRATION_TOKEN: <plaintext>` with the same
   `_FILE`/secrets pattern runner-2 uses (`/opt/gitea/runner_token`, 0600). Do NOT touch the
   `gitea` service definition.
3. `docker compose up -d --no-deps` the runner-1 service only; verify it comes back online
   in the admin runner list and its `.runner` identity is unchanged (still id 1).
4. Tighten perms: `chmod 600 /opt/gitea/docker-compose.yml docker-compose.yml.bak-tst30 docker-compose.yml.bak-tst30b`
   (verify compose stack still operable by the deploy user).
5. Rotate the leaked registration token if the deployment allows:
   `docker exec … gitea actions generate-runner-token` (or admin API) — if Gitea offers no
   invalidation of the old value, say so explicitly in the report (residual risk: LAN actor
   could register a rogue runner until rotation) rather than claiming it rotated.
6. Verify CI still works: trigger nothing; just confirm both runners online and the token
   file perms; a real push lands naturally later. Report evidence.

### Task 7: Closeout — cargo Bearer verification, docs/tracker sync, commits

**Classification:** small
**Estimated implement time:** ~5 min
**Parallelizable with:** none (needs Tasks 3, 4, 5, 6)

**Files:**
- Modify: `archreview/2026-07-12/remediation/90-candidate-findings-next-cycle.md` (final
  state of NEXT-06/NEXT-07 rows if Tasks 3/4 left anything)
- Possibly modify: `docs/GatewayTesting.md` / TST-30 runbook (runner topology now ids 1+5
  only; token-hardening note)

1. Cargo Bearer verification (no printing): assert `~/.zshenv` line matches
   `CARGO_REGISTRIES_DOHERTJ2_GITEA_TOKEN="Bearer …"` via `grep -c`, confirm
   `docs/ClientPackaging.md` note present (commit `5b153da`); check no other credential
   location (CI secrets, windev profiles) publishes to cargo — expected none.
2. Sweep: every doc touched this cycle consistent (runbooks, trackers, GatewayTesting.md);
   `git grep` for stale phrases ("crash-loop… pending", "id 4", "admin123") and fix.
3. Commit remaining doc changes locally; do NOT push. List the full unpushed stack in the
   report.

---

## Out of scope

- Pushing any mxaccessgw commits (user decides; stack listed at closeout).
- The five next-cycle candidate findings other than NEXT-06/NEXT-07.
- Auth-DB restore path for windev (fresh deploy chosen — preserves schema-3 data).
- `ci.yml` changes (labels, concurrency groups).

## Dependency graph

```
{1} → 2 → 3 ┐
{4}          ├→ 7
{5}          │
{6} ─────────┘
```