From 9760497d66a9a28ad3da7efa908a28644df99886 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Fri, 7 Aug 2026 09:21:17 -0400 Subject: [PATCH] docs(sec-36): rotation executed 2026-08-07; runbook host-path/vd03/verification corrections; new findings (LDAP test fixtures, windev stale deploy) --- .../2026-07-12/remediation/00-tracking.md | 3 +- .../remediation/40-security-dashboard.md | 4 +- .../90-candidate-findings-next-cycle.md | 4 +- .../SEC-36-ldap-credential-rotation.md | 46 +++++++++++++++++++ 4 files changed, 54 insertions(+), 3 deletions(-) diff --git a/archreview/2026-07-12/remediation/00-tracking.md b/archreview/2026-07-12/remediation/00-tracking.md index 95d95c6..5fc49f9 100644 --- a/archreview/2026-07-12/remediation/00-tracking.md +++ b/archreview/2026-07-12/remediation/00-tracking.md @@ -104,7 +104,7 @@ Full design + implementation for each row lives in the linked domain doc under i | SEC-33 | Low | P1 | M | old SEC-23 (co-locate) | Done | Host-meaningful path rooting; drop Windows literals from appsettings; validate Galaxy `SnapshotCachePath` | | SEC-34 | Low | P2 | S | — | Done | Verification cache: expiry outlives TTL; `Invalidate` races in-flight repopulation | | SEC-35 | Info | — | S | — | N/A (doc note discharged 2026-08-07) | Production hard-stops key on exact `Production` environment name (doc-only) | -| SEC-36 | Low | P1 | M | cross-repo `scadaproj/infra/glauth` | Done (repo-side; live rotation is operator-pending per runbook) | Committed dev LDAP service-account password: rotate, remove, move dev channel to user-secrets | +| SEC-36 | Low | P1 | M | cross-repo `scadaproj/infra/glauth` | Done | Committed dev LDAP service-account password: rotate, remove, move dev channel to user-secrets | ### Clients — [50-clients.md](50-clients.md) @@ -184,3 +184,4 @@ Sequence these together rather than piecemeal — several are one change set spa | 2026-08-07 | Code-review follow-up on the same branch (`fix/cli-39-version-train`). **Important gap fixed:** `src/ZB.MOM.WW.MxGateway.Contracts/ZB.MOM.WW.MxGateway.Contracts.csproj` was left at `0.1.2` — the original CLI-39 pass bumped the .NET Client but missed that `Invoke-PackDotnet` in `scripts/pack-clients.ps1` packs and publishes **both** `ZB.MOM.WW.MxGateway.Contracts` and `.Client` through the same `-Publish` loop, and the new collision guard runs every `ZB.MOM.WW.MxGateway.*.nupkg` it finds through `Assert-GiteaPackageNotPublished`. Left as-is, the next real `.NET` publish would have packed Contracts at the already-published 0.1.2, the guard would have correctly refused it, and the bare `ForEach-Object` would have aborted mid-loop — with `Client` (0.2.0, alphabetically first) potentially already pushed and `Contracts` stuck at 0.1.2, the two published nupkgs out of lockstep. Fixed: bumped `Contracts.csproj` to `0.2.0` alongside the Client (they have always released together). Also updated the now-stale comment in `src/Directory.Build.props` (it previously said the repo-wide `0.1.2` default was "kept ... to match the Contracts package"; that's no longer true now that Contracts overrides it — the comment now explains the override and that Server/Worker/Tests staying at `0.1.2` is a separate, not-yet-made decision; the `Directory.Build.props` `` value itself was **not** changed, since Server/Worker are not published packages and were out of scope for CLI-39). `docs/ClientPackaging.md`'s "Versioning" section gained `Contracts.csproj` as a fifth manifest (explaining why it must move with the .NET Client) and a note on the near-miss. **Cheap hardening also applied:** the Python (`pyproject.toml`) and Rust (`Cargo.toml`) version-extraction regexes in `pack-clients.ps1` now scope to the `[project]`/`[package]` section header respectively (matching the first `version = "..."` line anywhere in the file was luck of ordering — Cargo.toml has an identical second one under `[workspace.package]`), plus a one-line comment on the nuget filename-parse regex explaining the lazy-id/anchored-version split. **Verification:** `dotnet build src/ZB.MOM.WW.MxGateway.Contracts/ZB.MOM.WW.MxGateway.Contracts.csproj` 0 warnings/0 errors; `dotnet pack` of that project confirmed the artifact is literally named `ZB.MOM.WW.MxGateway.Contracts.0.2.0.nupkg`; `dotnet build clients/dotnet/ZB.MOM.WW.MxGateway.Client.slnx` still 0 warnings/0 errors. Re-ran the pack-clients.ps1 guard function-harness live against the real Gitea registry: `ZB.MOM.WW.MxGateway.Contracts`/`.Client` **0.1.2 both still refused**, **0.2.0 both now pass** — then went one step further and packed both projects for real (`dotnet pack`, no `-Publish`) and ran the exact `Invoke-PackDotnet` filename-parse-then-guard logic against the two real `.nupkg` files, confirming both parse to `id`/`version` correctly and both clear the live collision check. The scoped Python/Rust regexes were also exercised live against the real `pyproject.toml`/`Cargo.toml` and still resolve to `0.2.0`. No publish performed. | | 2026-08-07 | **TST-30 → fully `Done`** — the operator half (register a second CI runner) executed live on `10.100.0.35` per `docs/runbooks/TST-30-second-ci-runner.md` option (a). A second **instance-level** `act_runner` container, `gitea-runner-2` (runner id 5, capacity 2, labels `ubuntu-latest`/`ubuntu-22.04`), was added to the `/opt/gitea` compose stack with the same `container.network: traefik` setting as the original, its registration token mounted from a `0600` file rather than inlined in compose. The existing `gitea-runner` (id 1, capacity 4) was **not** touched — total capacity went 4 → 6 purely by addition, so the change reverts by removing one container. **Concurrency verified live** (the finding's acceptance test): HEAD `a346d51` was pushed to two scratch branches, `scratch/tst30-a` (run 661) and `scratch/tst30-b` (run 662), while an unrelated run 660 was already in flight; at 13:07:53Z jobs from **three** runs were `in_progress` at once — run 660 `portable` plus run 662 `portable`/`java` on runner 1, and run 661 `portable`/`java` on `gitea-runner-2` — a state the pre-change single-runner topology could not produce. Runner attribution came from `GET /repos/dohertj2/mxaccessgw/actions/runs/{id}/jobs`, which exposes `runner_id`/`runner_name` per job, and was cross-checked against `docker logs gitea-runner-2` (`task 1145 …`, `task 1146 …` pickups at 13:05:22–23Z). **`gitea:3000` resolution holds on the new instance** — the property option (c) was rejected for losing: run 661's `portable` job (task 1145, scheduled on `gitea-runner-2`) logged `git remote add origin http://gitea:3000/dohertj2/mxaccessgw` then a successful `fetch … From http://gitea:3000/dohertj2/mxaccessgw`, and its job container's `/workspace/dohertj2/mxaccessgw` was confirmed checked out at `a346d514dd24e775640e5667aa7cd8e561fec68a`. **Runbook fact corrected:** its checklist expected `GET /repos/dohertj2/mxaccessgw/actions/runners` to show ≥2, but that endpoint still returns `total_count: 0` — correctly, since both runners are instance-scoped, which is exactly what this finding documented; the working check is `GET /api/v1/admin/actions/runners` (lists ids 1, 4 = an unrelated local macOS runner, and 5). A dated "Executed 2026-08-07" note recording this was added to the top of the runbook rather than rewriting its history. No source, workflow, or `docs/GatewayTesting.md` change was needed — the no-cancel reality and the `run-windev-ci.sh` bypass are unaffected by runner count, as that prose already stated. Scratch branches deleted after verification. Tracking: both registers' TST-30 rows, the pending-operator-actions list in `90-candidate-findings-next-cycle.md`, and this row. | | 2026-08-07 | **CLI-39 publish executed** — the release the version train was cut for is now live on `gitea.dohertylan.com`. Published from HEAD `a346d51` via `scripts/pack-clients.ps1`: `nuget` `ZB.MOM.WW.MxGateway.Client` **0.2.0** + `ZB.MOM.WW.MxGateway.Contracts` **0.2.0**, `pypi` `zb-mom-ww-mxaccess-gateway-client` **0.2.0**, `cargo` `zb-mom-ww-mxgateway-client` **0.2.0**, `maven` `com.zb.mom.ww.mxgateway:zb-mom-ww-mxgateway-client` **0.2.1** (the Java exception recorded in the CLI-39 row above — 0.2.0 was already taken by the pre-conformance build). All five coordinates re-queried through `GET /api/v1/packages/dohertj2/{type}/{name}/{version}` after the push and confirmed `200`. Go ships no artifact: module tag `clients/go/v0.2.0` was created and pushed at `a346d51` by `scripts/tag-go-module.ps1` (its CLI-21 version-match guard passed against `ClientVersion = "0.2.0"`), and `go list -m …@v0.2.0` resolves. Full suites ran pre-push with no failures: 128 .NET, 164 Python, 107 Rust, 79 Java. **Deviation found and fixed:** the first `cargo publish` returned `401` because `CARGO_REGISTRIES_DOHERTJ2_GITEA_TOKEN` held the raw token — cargo forwards that value as the `Authorization` header verbatim and Gitea's cargo registry requires `Bearer ` there (confirmed both ways against `/api/v1/user`: bare → 401, `Bearer`-prefixed → 200), unlike the other feeds' username/token basic-auth pairs. The publish succeeded on retry with the prefix exported shell-locally; the permanent fix was then applied to the operator's `~/.zshenv` (outside this repo; prior file backed up `0600` as `~/.zshenv.bak-cli39`) and re-verified via a `Bearer`-authed `GET /api/v1/user` → `200`. The requirement is now documented in `docs/ClientPackaging.md`'s Rust section, and that doc's "Versioning" section carries the published-coordinates record so the next bump starts from the real registry state rather than a re-check. No source, proto, script, or test change — publish execution plus docs. Tracking: `docs/ClientPackaging.md` and this row (CLI-39's register rows in both this file and `50-clients.md` were already plain `Done` with no publish-pending annotation to clear). | +| 2026-08-07 | **SEC-36 → fully `Done`** — the operator half (rotate the dev LDAP service-account credential) executed live per `docs/runbooks/SEC-36-ldap-credential-rotation.md`. A new password was generated, the `serviceaccount` `passsha256` in `scadaproj/infra/glauth/config.toml` replaced, and the shared GLAuth recreated on `10.100.0.35`, so the value this repo's git history permanently discloses (live in the directory since 2026-06-04; not reproduced here) **no longer binds** `dc=zb,dc=local` — the load-bearing half of the finding. The new value lives only in the three channels the design named: the GLAuth `passsha256` (committed in `scadaproj`, commit `aada53b`), the NSSM environment on `10.100.0.48`, and each dev box's user-secrets (set on this Mac); it is in no file of this repo. The retired plaintext was also scrubbed from the `scadaproj` glauth comments (`config.toml`, `docker-compose.yml`, `README.md`) and from the docker host's live `docker-compose.yml`; the host's `*.bak-sec36` rollback copies deliberately keep it. **Three runbook facts corrected** in a dated block at its top: (1) step 3's `cd ~/Desktop/scadaproj/infra/glauth` on the docker host is stale — the stack runs from `/home/dohertj2/zb-glauth` (container `zb-shared-glauth`), fed by the `scp` deploy in `scadaproj/infra/glauth/README.md`; (2) `wonder-app-vd03` is out of scope **on documentary evidence**, not merely unchecked — its gateway binds the ScadaBridge/ScadaLink local GLAuth under `dc=scadalink`/`dc=scadabridge`, a different directory that never held this credential (and the host is unreachable from the dev network), so no env var was staged there; (3) the "3-fail / 10-minute per-IP lockout" caution is inert for this instance — `config.toml:14` sets `LimitFailedBinds = false`. **One Done criterion is met with a caveat:** the new value is staged on windev (10th `AppEnvironmentExtra` entry on the `MxAccessGw` NSSM service) but the runbook's primary dashboard `/login` check **could not run** — windev's gateway is crash-looping on an unrelated pre-existing fault (deployed Server binary of 2026-06-25 predates the 2026-07-15 auth-DB migration, so it opens a schema-3 DB it supports only at schema 2 and aborts; ~10k Hosting-failed events/day since at least 08-06). Verified instead by the equivalent primitive: a direct `ldapsearch` bind as `cn=serviceaccount,dc=zb,dc=local` with the new value against `10.100.0.35:3893` succeeded and returned the `multi-role` entry — the same search bind the dashboard performs. The deferred check is recorded in the runbook with its completion path. **Two new next-cycle candidate findings** filed in `90-candidate-findings-next-cycle.md`: NEXT-06 (`DashboardLdapLiveTests` fixture drift — its only positive test binds `admin`/`admin123` while the directory's `admin` password is the standard dev one, and fixture user `readonly` does not exist in the GLAuth config, so the suite carries **no positive-proof coverage** of the service-account bind and could not have substituted for the dashboard check) and NEXT-07 (the windev stale deployment / crash loop above). Tracking: both registers' SEC-36 rows, the pending-operator-actions list in `90-candidate-findings-next-cycle.md`, and this row. | diff --git a/archreview/2026-07-12/remediation/40-security-dashboard.md b/archreview/2026-07-12/remediation/40-security-dashboard.md index 438caf2..506a5f9 100644 --- a/archreview/2026-07-12/remediation/40-security-dashboard.md +++ b/archreview/2026-07-12/remediation/40-security-dashboard.md @@ -15,7 +15,7 @@ Repo rules that bind every entry: docs change in the same commit as the source ( | SEC-33 | Low | P1 | M | — (co-locate SEC-23) | Done | Any-platform path-rooting acceptance re-opens SEC-01 on Unix; Galaxy `SnapshotCachePath` unvalidated | | SEC-34 | Low | P2 | S | — | Done | Verification cache: expiry outlives TTL; `Invalidate` races in-flight repopulation | | SEC-35 | Info | — | S | — | N/A (doc-only note discharged 2026-08-07) | Production hard-stops key on the exact `Production` environment name | -| SEC-36 | Low | P1 | M | cross-repo (`scadaproj/infra/glauth`) | Done (repo-side; live rotation operator-pending per runbook) | Committed dev LDAP service-account password: remove from repo and rotate | +| SEC-36 | Low | P1 | M | cross-repo (`scadaproj/infra/glauth`) | Done | Committed dev LDAP service-account password: remove from repo and rotate | --- @@ -220,3 +220,5 @@ dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj --fil (asserts the blank-password validation still fires with the updated message). Manual: with user-secrets set on the dev box, `dotnet run --project src/ZB.MOM.WW.MxGateway.Server/...` and a dashboard `/login` as `multi-role` succeeds against the rotated GLAuth; the deployed-host login re-check from step 1 counts as the production verification. Live-LDAP integration tests (`MXGATEWAY_RUN_LIVE_LDAP_TESTS=1`) only where the GLAuth instance is reachable; otherwise document skipped per the testing matrix. **Outcome (2026-08-07 — Done, repo-side; live rotation operator-pending).** Landed on `fix/sec-36-ldap-secret`. **The design's baseline had already shifted:** at HEAD `appsettings.json` no longer commits the literal — it ships `"ServiceAccountPassword": "${secret:ldap/mxgateway/bind}"`, a fail-closed encrypted-store reference (documented `GatewayConfiguration.md:252`, tested by `PreHostSecretExpansionTests`) introduced by the Secrets-store adoption after this remediation was written. **Deviation from Implementation step 2:** the `${secret:}` reference was **kept, not deleted** — deleting it regresses the shipped/documented/tested store channel and the committed-plaintext finding is already resolved for `appsettings.json`. The load-bearing residual — the literal value still present in `glauth.md`'s samples (`:33,65,103,136,245`), `docs/GatewayTesting.md`, and the historical `archreview/*` SEC-06 evidence — was scrubbed to `` placeholders, each with a pointer to the source of truth `scadaproj/infra/glauth/` and a rotation-required note. Steps 3–6 implemented as designed: `mxaccessgw-server` added (step 3); the `ValidateLdap` blank-password message now names both channels — dev `dotnet user-secrets set "MxGateway:Ldap:ServiceAccountPassword" ` and deployed `MxGateway__Ldap__ServiceAccountPassword` — plus a note on the `${secret:}` store default (step 4), asserted by the extended `Validate_Fails_WhenLdapEnabledAndServiceAccountPasswordBlank`; docs updated same commit (step 5); `git grep -i` for the old value is empty across tracked files (step 6). The cross-repo **step 1 (rotate GLAuth on `10.100.0.35`, pre-stage the NSSM env var on `10.100.0.48` and on `wonder-app-vd03` only if `Ldap.Enabled`, verify dashboard login)** is the operator's to execute, captured in the new runbook `docs/runbooks/SEC-36-ldap-credential-rotation.md`. Verification (macOS): `dotnet build …Server` 0 warnings/0 errors; `dotnet test --filter ~GatewayOptionsValidator` green. + +**Outcome (2026-08-07 — operator half executed; finding now fully `Done`).** The cross-repo step 1 left open above was executed per `docs/runbooks/SEC-36-ldap-credential-rotation.md`. A new service-account password was generated, the `serviceaccount` `passsha256` in `scadaproj/infra/glauth/config.toml` replaced, and the shared GLAuth recreated on `10.100.0.35` from its actual compose directory — the **load-bearing** half of the finding is now discharged: the value disclosed by this repo's git history (live in the directory since 2026-06-04; not reproduced here) no longer binds `dc=zb,dc=local`. The new value exists only in the three channels the design named — the GLAuth `passsha256` (committed in `scadaproj`, commit `aada53b`), the NSSM service environment on `10.100.0.48`, and each dev box's user-secrets — and in no file of this repo. The retired plaintext was additionally scrubbed from the `scadaproj` glauth comments (`config.toml`, `docker-compose.yml`, `README.md`) and from the docker host's live `docker-compose.yml`; the host's `*.bak-sec36` rollback copies deliberately retain it. **Three runbook facts were wrong and are corrected in a dated block at its top.** (1) Its step 3 said `cd ~/Desktop/scadaproj/infra/glauth` on the docker host; no such path exists there — the stack runs from `/home/dohertj2/zb-glauth` (container `zb-shared-glauth`), fed by the `scp` deploy documented in `scadaproj/infra/glauth/README.md`. (2) `wonder-app-vd03` is **out of scope on documentary evidence**, not merely unchecked: its gateway binds the ScadaBridge/ScadaLink local GLAuth under `dc=scadalink`/`dc=scadabridge`, a different directory that never held this credential (the host is also unreachable from the dev network); no env var was staged there. (3) Its "3-fail / 10-minute per-IP lockout" caution is **inert for this instance** — `config.toml:14` sets `LimitFailedBinds = false`. **One Done criterion is met with a caveat:** the new value **is** staged on windev (`10.100.0.48`, 10th `AppEnvironmentExtra` entry on the `MxAccessGw` NSSM service), but the runbook's primary check — dashboard `/login` as `multi-role` — **could not run**, because windev's gateway is crash-looping on an unrelated pre-existing fault: the deployed Server binary (2026-06-25) predates the 2026-07-15 auth-DB migration, so it opens a schema-version-3 database it supports only at version 2 and aborts at startup (~10k Hosting-failed events/day since at least 08-06). That is a stale-deployment problem, filed as a next-cycle candidate finding, not a rotation defect. **Verified instead by the equivalent primitive:** a direct `ldapsearch` bind as `cn=serviceaccount,dc=zb,dc=local` with the new value against `10.100.0.35:3893` succeeded and returned the `multi-role` entry — the same search bind the dashboard performs. Also surfaced and filed for next cycle: `DashboardLdapLiveTests` fixture drift leaves the suite with **no positive-proof coverage** of the service-account bind, so it could not have substituted for the dashboard check either. Tracking: both registers' SEC-36 rows, the pending-operator-actions list in `90-candidate-findings-next-cycle.md`, and the `00-tracking.md` progress log. diff --git a/archreview/2026-07-12/remediation/90-candidate-findings-next-cycle.md b/archreview/2026-07-12/remediation/90-candidate-findings-next-cycle.md index b1e47b6..4436481 100644 --- a/archreview/2026-07-12/remediation/90-candidate-findings-next-cycle.md +++ b/archreview/2026-07-12/remediation/90-candidate-findings-next-cycle.md @@ -9,11 +9,13 @@ These were discovered while remediating the 2026-07-12 backlog but were **out of | NEXT-03 | Gateway alarms | Low | `GatewayAlarmMonitor.ApplyReconcile` feed-repair broadcasts (the new acked-delta from GWC-26 **and** the pre-existing Raise/Clear repair) are **at-least-once, not exactly-once**: a periodic reconcile can synthesize a transition whose matching live transition is still buffered in the alarm lease, so both broadcast as indistinguishable duplicates on the alarm feed (StreamAlarms + dashboard hub). Pre-existing (the Raise/Clear repair always had it); GWC-26 documented the at-least-once contract rather than closing the race. Closing it needs reconcile/live serialization or a monotonic dedup marker. | | NEXT-04 | Worker frame writer | Low | WRK-22/WRK-25 cancellation path: a frame `Claimed` by a concurrent lock-holder just before its caller's cancellation races in is never awaited by that caller; if the write then faults, `TrySetException` lands on a `Task` nobody observes (unobserved-task-exception). By-design residual, non-crash (no `UnobservedTaskException` handler registered), pre-existing to single-frame WRK-22 and amplified per-batch by WRK-25. Hygiene fix: attach a fault-observing continuation to abandoned/tombstoned frame completions. | | NEXT-05 | Worker frame writer | Info | A batch whose remaining frames are tombstoned by cancellation leaves dead `PendingFrame` entries in `_eventFrames`/`_controlFrames` until a future `DequeueNext` pops and skips them. Same pre-existing behavior as single-frame WRK-22, amplified per-batch; in practice heartbeats purge them promptly, so not a real leak. | +| NEXT-06 | Testing / live LDAP | Medium | `DashboardLdapLiveTests` fixtures have drifted from the shared GLAuth directory, leaving the suite with **no positive-proof coverage of the service-account bind**. Its only success-path test, `AuthenticateAsync_AdminInGwAdminGroup_Succeeds`, binds `admin`/`admin123`, but the directory's `admin` user carries the standard dev password (`scadaproj/infra/glauth/config.toml`), so that assertion cannot pass. `AuthenticateAsync_ReadOnlyUserMissingGwAdminGroup_Fails` binds fixture user `readonly`, which **does not exist** in the GLAuth config at all — it passes for the wrong reason (user-not-found rather than the group-missing branch it names; the `readonly` name is in fact barred by the README's user/group case-collision rule). The three remaining tests are negative assertions that pass whether or not the service account can bind. Net effect: a green `DashboardLdapLiveTests` run proves nothing about the bind credential — surfaced during SEC-36, where the suite was considered as a substitute for the deferred dashboard-login check and rejected. Fix: realign the fixtures to real directory users (e.g. `multi-role`/`gw-viewer`) or add the missing users to the GLAuth config, and add one test that fails when the service-account credential is wrong. | +| NEXT-07 | Deployment / windev | High | The `10.100.0.48` (windev) gateway deployment is **stale and crash-looping**, and has been since at least 2026-08-06 (~10k Hosting-failed events/day). The deployed Server binary dates to 2026-06-25 and predates the auth-DB migration of 2026-07-15: it opens a schema-version-3 `gateway-auth.db` that it supports only at version 2 and aborts at startup, so the `MxAccessGw` service never reaches a listening state. Not a code defect in the current tree — a deploy-drift/operations gap — but it means the repo's only deployed host has been dark for over a day and any host-level verification (including SEC-36's dashboard-login check) is blocked until it is repaired. Fix: deploy a current Server build to windev, or restore/downgrade the auth DB to schema 2 if the old binary must stand. Worth asking separately why a service in a permanent restart loop raised no alert. Discovered during SEC-36. | ## Operator actions still pending (from this cycle's runbooks) These are **live-infrastructure actions the operator must execute** — the repo-side work is complete and merged: -- **SEC-36** — rotate the dev LDAP service-account credential per `docs/runbooks/SEC-36-ldap-credential-rotation.md` (generate new secret in `scadaproj/infra/glauth`, pre-stage the NSSM env var on deployed hosts, rotate GLAuth on `10.100.0.35`, verify dashboard login). The committed literal is gone from the working tree but remains recoverable from git history until rotation completes — **rotation is the load-bearing half.** +- ~~**SEC-36** — rotate the dev LDAP service-account credential per `docs/runbooks/SEC-36-ldap-credential-rotation.md` (generate new secret in `scadaproj/infra/glauth`, pre-stage the NSSM env var on deployed hosts, rotate GLAuth on `10.100.0.35`, verify dashboard login). The committed literal is gone from the working tree but remains recoverable from git history until rotation completes — **rotation is the load-bearing half.**~~ **Executed 2026-08-07**: the `serviceaccount` `passsha256` was replaced in `scadaproj/infra/glauth/config.toml` (commit `aada53b`) and the shared GLAuth recreated on `10.100.0.35`, so the literal recoverable from this repo's history no longer binds `dc=zb,dc=local`. The new value lives only in the GLAuth hash, windev's NSSM environment, and dev user-secrets. `wonder-app-vd03` was out of scope (it binds a different, `dc=scadalink`/`dc=scadabridge` directory). **Caveat:** windev's dashboard `/login` verification is deferred — that host's gateway is crash-looping on the unrelated stale-deployment fault filed as NEXT-07 — so the bind was verified directly by `ldapsearch` as `cn=serviceaccount,dc=zb,dc=local` instead. SEC-36 is now fully `Done`. - ~~**TST-30** — register a second Gitea `act_runner` on `10.100.0.35` per `docs/runbooks/TST-30-second-ci-runner.md` to relieve the single-shared-runner bottleneck.~~ **Executed 2026-08-07**: `gitea-runner-2` (id 5, capacity 2) is online on `10.100.0.35` via the `/opt/gitea` compose stack, same `container.network: traefik`, token from a `0600` file mount; the existing runner (id 1, capacity 4) was untouched. Concurrency verified — jobs from three runs ran simultaneously across both runners, and a `gitea-runner-2` job cloned successfully from `http://gitea:3000`. TST-30 is now fully `Done`. - **TST-25 follow-ups** — old **TST-05** (scheduled live-MXAccess smoke) is now covered by the `nightly-windev` job; old **TST-24** (client wire tests in CI) is unblocked by the working Windows tier. diff --git a/docs/runbooks/SEC-36-ldap-credential-rotation.md b/docs/runbooks/SEC-36-ldap-credential-rotation.md index cba3021..ad21e56 100644 --- a/docs/runbooks/SEC-36-ldap-credential-rotation.md +++ b/docs/runbooks/SEC-36-ldap-credential-rotation.md @@ -1,5 +1,51 @@ # SEC-36 — LDAP Service-Account Credential Rotation (Operator Runbook) +> **Executed 2026-08-07 — the rotation is done; this runbook is now history plus the four +> corrections below.** A new service-account password was generated, `scadaproj/infra/glauth/config.toml`'s +> `serviceaccount` `passsha256` was replaced and the shared GLAuth recreated, and the old value +> (the literal this repo committed, live in the directory since 2026-06-04) no longer binds. The new +> value now exists only in the +> three channels this runbook names: the GLAuth `passsha256` (committed in `scadaproj`), the NSSM +> service environment on `10.100.0.48`, and this dev Mac's user-secrets. The retired plaintext was +> also scrubbed from `scadaproj/infra/glauth/`'s `config.toml`/`docker-compose.yml`/`README.md` +> comments and from the host's live `docker-compose.yml` (the `*.bak-sec36` backups on the host still +> carry it, deliberately — they are the rollback artifacts). +> +> **Correction 1 — host paths in step 3 were stale.** The runbook says +> `cd ~/Desktop/scadaproj/infra/glauth` on `10.100.0.35`. That directory does not exist there: +> `scadaproj` is a dev-workstation checkout, and the docker host runs the stack from +> **`/home/dohertj2/zb-glauth`** (container **`zb-shared-glauth`**, project name `zb-shared-glauth`). +> The repo remains the source of truth; deployment is the `scp` of `config.toml`/`docker-compose.yml` +> into `~/zb-glauth` documented in `scadaproj/infra/glauth/README.md`, followed by +> `docker compose up -d --force-recreate` there. +> +> **Correction 2 — `wonder-app-vd03` is out of scope, on documentary evidence.** The precondition +> above says to check `MxGateway:Ldap:Enabled` on that host. It could not be checked directly (the +> host is unreachable from the dev network), but it is out of scope regardless: its gateway binds a +> **different directory** — the ScadaBridge/ScadaLink local GLAuth under `dc=scadalink`/`dc=scadabridge`, +> not `dc=zb,dc=local` — so this credential is not one it can hold. No env var was staged there and +> none is needed. +> +> **Correction 3 — step 4's dashboard verification is deferred on `10.100.0.48`; a direct bind was +> used instead.** The NEW value **is** staged on windev (added as the 10th `AppEnvironmentExtra` +> entry on the `MxAccessGw` NSSM service), but dashboard `/login` could not exercise it: windev's +> gateway is **crash-looping on a pre-existing, unrelated fault** — the deployed Server binary +> (2026-06-25) predates the auth-DB migration of 2026-07-15, so it opens a schema-version-3 database +> it only supports at version 2 and aborts at startup (~10k Hosting-failed events/day since at least +> 08-06). This is a stale-deployment problem, not a rotation problem; it is filed as a next-cycle +> finding. **Verification used instead:** a direct `ldapsearch` bind as +> `cn=serviceaccount,dc=zb,dc=local` with the new value against `10.100.0.35:3893` succeeded and +> returned the `multi-role` entry — which is precisely the search bind the dashboard performs, minus +> the HTTP shell. **Finish the deferred check when windev is repaired:** redeploy a current Server +> build (or restore a schema-2 auth DB), then browse the dashboard `/login` as `multi-role` per +> step 4. +> +> **Correction 4 — the lockout caution under "Verifying the rotation" is inert for this instance.** +> It warns that GLAuth's 3-fail / 10-minute per-IP lockout can lock the whole office when testing +> that the old value is dead. This GLAuth runs `LimitFailedBinds = false` (`config.toml:14`), so no +> failed-bind limiter is active and the caution does not apply here. Keep the caution for any +> instance that enables the limiter. + Operator steps to rotate the shared GLAuth service-account password after the repo-side removal landed (SEC-36). The repo change (removal of the committed value, the two supported secret channels, and this runbook) is already merged; the live rotation below is the