SEC-36 — LDAP Service-Account Credential Rotation (Operator Runbook)

Executed 2026-08-07 — the rotation is done; this runbook is now history plus the four corrections below. A new service-account password was generated, scadaproj/infra/glauth/config.toml's serviceaccount passsha256 was replaced and the shared GLAuth recreated, and the old value (the literal this repo committed, live in the directory since 2026-06-04) no longer binds. The new value now exists only in the three channels this runbook names: the GLAuth passsha256 (committed in scadaproj), the NSSM service environment on 10.100.0.48, and this dev Mac's user-secrets. The retired plaintext was also scrubbed from scadaproj/infra/glauth/'s config.toml/docker-compose.yml/README.md comments and from the host's live docker-compose.yml (the *.bak-sec36 backups on the host still carry it, deliberately — they are the rollback artifacts).

Correction 1 — host paths in step 3 were stale. The runbook says cd ~/Desktop/scadaproj/infra/glauth on 10.100.0.35. That directory does not exist there: scadaproj is a dev-workstation checkout, and the docker host runs the stack from /home/dohertj2/zb-glauth (container zb-shared-glauth, project name zb-shared-glauth). The repo remains the source of truth; deployment is the scp of config.toml/docker-compose.yml into ~/zb-glauth documented in scadaproj/infra/glauth/README.md, followed by docker compose up -d --force-recreate there.

Correction 2 — wonder-app-vd03 is out of scope, on documentary evidence. The precondition above says to check MxGateway:Ldap:Enabled on that host. It could not be checked directly (the host is unreachable from the dev network), but it is out of scope regardless: its gateway binds a different directory — the ScadaBridge/ScadaLink local GLAuth under dc=scadalink/dc=scadabridge, not dc=zb,dc=local — so this credential is not one it can hold. No env var was staged there and none is needed.

Correction 3 — step 4's dashboard verification is deferred on 10.100.0.48; a direct bind was used instead. The NEW value is staged on windev (added as the 10th AppEnvironmentExtra entry on the MxAccessGw NSSM service), but dashboard /login could not exercise it: windev's gateway is crash-looping on a pre-existing, unrelated fault — the deployed Server binary (2026-06-25) predates the auth-DB migration of 2026-07-15, so it opens a schema-version-3 database it only supports at version 2 and aborts at startup (~10k Hosting-failed events/day since at least 08-06). This is a stale-deployment problem, not a rotation problem; it is filed as a next-cycle finding. Verification used instead: a direct ldapsearch bind as cn=serviceaccount,dc=zb,dc=local with the new value against 10.100.0.35:3893 succeeded and returned the multi-role entry — which is precisely the search bind the dashboard performs, minus the HTTP shell. Finish the deferred check when windev is repaired: redeploy a current Server build (or restore a schema-2 auth DB), then browse the dashboard /login as multi-role per step 4.

Correction 4 — the lockout caution under "Verifying the rotation" is inert for this instance. It warns that GLAuth's 3-fail / 10-minute per-IP lockout can lock the whole office when testing that the old value is dead. This GLAuth runs LimitFailedBinds = false (config.toml:14), so no failed-bind limiter is active and the caution does not apply here. Keep the caution for any instance that enables the limiter.

Operator steps to rotate the shared GLAuth service-account password after the repo-side removal landed (SEC-36). The repo change (removal of the committed value, the two supported secret channels, and this runbook) is already merged; the live rotation below is the load-bearing half and is yours to execute.

Never put the old or new password in this repo, in a commit, in a chat, or in this file. The value lives only in the GLAuth source of truth and in each host's out-of-band channel.

Why

The dev GLAuth service-account password (cn=serviceaccount,dc=zb,dc=local) was historically committed to this repo. Removal alone is insufficient — the old value is permanently recoverable from git history — so rotation is required. Until the shared GLAuth on 10.100.0.35:3893 stops honoring the old value, the repo history discloses a live directory account with LDAP search capability over dc=zb,dc=local.

Where the credential lives now (three channels, all bind `MxGateway:Ldap:ServiceAccountPassword`)

Source of truth: scadaproj/infra/glauth/config.toml on host 10.100.0.35 (the serviceaccount user's passsha256). scadaproj is a shared monorepo — stage only the explicit glauth paths.
Encrypted secrets store (gateway default): appsettings.json ships ${secret:ldap/mxgateway/bind}, resolved from the local encrypted store (seed with secret set ldap/mxgateway/bind <value>).
Deployed hosts: env var MxGateway__Ldap__ServiceAccountPassword in the NSSM service environment.
Dev boxes: dotnet user-secrets set "MxGateway:Ldap:ServiceAccountPassword" <value> (the server carries <UserSecretsId>mxaccessgw-server</UserSecretsId>).

See docs/GatewayConfiguration.md (the ServiceAccountPassword row) and glauth.md.

Preconditions

SSH access to the GLAuth docker host 10.100.0.35 and to the deployed gateway host(s).
Write access to scadaproj/infra/glauth/.
Know which deployed hosts run LDAP-backed dashboard login:
- 10.100.0.48 (windev) — primary; verify here.
- wonder-app-vd03 — its dashboard is disabled. Check MxGateway:Ldap:Enabled there first. If LDAP is disabled (Enabled=false), it has nothing to bind and needs no env var — skip it.
A generated replacement secret (see step 1). Generate the passsha256 per glauth.md ("Generate passsha256 from a plaintext password").

Cutover order

Follow this order so no window opens where the deployed dashboard cannot bind. Do not rotate GLAuth before the deployed hosts already carry the new value.

Generate the new secret in scadaproj/infra/glauth/. Pick a new password, compute its passsha256, and stage the change to the serviceaccount user in config.toml (do not docker compose up yet — the directory must keep honoring the OLD value until the deployed hosts carry the NEW one).
Pre-stage the NEW value on every LDAP-enabled deployed host via the env-var channel, so the host is ready the instant GLAuth flips:
```
nssm get MxAccessGw AppEnvironmentExtra
nssm set MxAccessGw AppEnvironmentExtra MxGateway__Ldap__ServiceAccountPassword=<new-value>
# restart the service so the new environment is picked up
nssm restart MxAccessGw
```
Do this on 10.100.0.48, and on wonder-app-vd03 only if MxGateway:Ldap:Enabled=true there. (Alternatively seed the encrypted store with secret set ldap/mxgateway/bind <new-value>; the env var overrides the store and is the simplest per-host mechanism.) At this moment the deployed host holds the NEW value but GLAuth still honors the OLD one — binds still fail closed against the old directory, which is expected and brief; proceed immediately.

Rotate GLAuth on 10.100.0.35 to honor the new value:

ssh 10.100.0.35
cd ~/Desktop/scadaproj/infra/glauth
docker compose up -d --force-recreate
docker compose logs -f          # confirm clean startup, no TOML parse error

Verify dashboard login on the deployed host(s). Browse to the gateway dashboard on 10.100.0.48 and log in as multi-role / password (Administrator) — a successful login proves the search bind used the new service-account credential end-to-end. If wonder-app-vd03 runs LDAP, verify it too; if its dashboard/LDAP is disabled, no check is needed.
The repo change is already landed (removal of the committed value, <UserSecretsId>, the validator message naming the two channels, and doc/scrub updates). Nothing more to commit for the cutover.
Developers set user-secrets on next pull. After pulling, a dev box with no secret configured will fail startup with a validation message naming the exact command. One-time per machine:
```
dotnet user-secrets set "MxGateway:Ldap:ServiceAccountPassword" <new-value>
```
(value from scadaproj/infra/glauth/, never from a repo file).

Verifying the rotation

Primary: dashboard /login as multi-role on 10.100.0.48 succeeds (step 4).
wonder-app-vd03: only if MxGateway:Ldap:Enabled=true; otherwise no action.

Live-LDAP integration tests (opt-in, only where the GLAuth instance is reachable):

$env:MXGATEWAY_RUN_LIVE_LDAP_TESTS = "1"
$env:MxGateway__Ldap__ServiceAccountPassword = "<new-value>"   # shell env only, never committed
dotnet test src/ZB.MOM.WW.MxGateway.IntegrationTests/ZB.MOM.WW.MxGateway.IntegrationTests.csproj `
  --filter FullyQualifiedName~DashboardLdapLiveTests

A green DashboardLdapLiveTests run confirms the new credential binds and searches. Where GLAuth is unreachable, document the suite as skipped per the docs/GatewayTesting.md opt-in matrix.

Old value is dead: after step 3, a bind with the old password must fail. Do not test this from a shared-NAT box — GLAuth's 3-fail / 10-minute per-IP lockout can lock the whole office.

Rollback

If dashboard login breaks after step 3, restore the previous passsha256 in scadaproj/infra/glauth/config.toml, docker compose up -d --force-recreate, and re-point the deployed hosts' env var / store back to the previous value. Because the deployed hosts were pre-staged in step 2, the exposure window is only steps 2→4.

Done criteria

GLAuth on 10.100.0.35 honors only the new value.
Every LDAP-enabled deployed host binds with the new value (dashboard login verified).
The source of truth scadaproj/infra/glauth/config.toml carries the new passsha256.
No repo file (this one included) contains the old or new value.
The SEC-36 tracker rows are Done with this runbook cited for the operator action.

10 KiB Raw Blame History