# SEC-36 — LDAP Service-Account Credential Rotation (Operator Runbook)

> **Executed 2026-08-07 — the rotation is done; this runbook is now history plus the four
> corrections below.** A new service-account password was generated, `scadaproj/infra/glauth/config.toml`'s
> `serviceaccount` `passsha256` was replaced and the shared GLAuth recreated, and the old value
> (the literal this repo committed, live in the directory since 2026-06-04) no longer binds. The new
> value now exists only in the
> three channels this runbook names: the GLAuth `passsha256` (committed in `scadaproj`), the NSSM
> service environment on `10.100.0.48`, and this dev Mac's user-secrets. The retired plaintext was
> also scrubbed from `scadaproj/infra/glauth/`'s `config.toml`/`docker-compose.yml`/`README.md`
> comments and from the host's live `docker-compose.yml` (the `*.bak-sec36` backups on the host still
> carry it, deliberately — they are the rollback artifacts).
>
> **Correction 1 — host paths in step 3 were stale.** The runbook says
> `cd ~/Desktop/scadaproj/infra/glauth` on `10.100.0.35`. That directory does not exist there:
> `scadaproj` is a dev-workstation checkout, and the docker host runs the stack from
> **`/home/dohertj2/zb-glauth`** (container **`zb-shared-glauth`**, project name `zb-shared-glauth`).
> The repo remains the source of truth; deployment is the `scp` of `config.toml`/`docker-compose.yml`
> into `~/zb-glauth` documented in `scadaproj/infra/glauth/README.md`, followed by
> `docker compose up -d --force-recreate` there.
>
> **Correction 2 — `wonder-app-vd03` is out of scope, on documentary evidence.** The precondition
> above says to check `MxGateway:Ldap:Enabled` on that host. It could not be checked directly (the
> host is unreachable from the dev network), but it is out of scope regardless: its gateway binds a
> **different directory** — the ScadaBridge/ScadaLink local GLAuth under `dc=scadalink`/`dc=scadabridge`,
> not `dc=zb,dc=local` — so this credential is not one it can hold. No env var was staged there and
> none is needed.
>
> **Correction 3 — step 4's dashboard verification is deferred on `10.100.0.48`; a direct bind was
> used instead.** The NEW value **is** staged on windev (added as the 10th `AppEnvironmentExtra`
> entry on the `MxAccessGw` NSSM service), but dashboard `/login` could not exercise it at rotation
> time: windev's gateway was **crash-looping on a pre-existing, unrelated fault** — the deployed Server binary
> (2026-06-25) predates the auth-DB migration of 2026-07-15, so it opens a schema-version-3 database
> it only supports at version 2 and aborts at startup (~10k Hosting-failed events/day since at least
> 08-06). That was a stale-deployment problem, not a rotation problem; it was filed as next-cycle
> finding NEXT-07 and resolved by redeploy on 2026-08-07. **Verification used instead:** a direct `ldapsearch` bind as
> `cn=serviceaccount,dc=zb,dc=local` with the new value against `10.100.0.35:3893` succeeded and
> returned the `multi-role` entry — which is precisely the search bind the dashboard performs, minus
> the HTTP shell. **The deferred check was completed 2026-08-07**, once windev was repaired by the
> redeploy filed under NEXT-07. With `Dashboard:DisableLogin=false` supplied as a process-env-only
> override on a foreground run of the new build, `GET /login` returned 200 with an antiforgery token,
> `POST /auth/login` as `multi-role`/`password` returned 302 to `/` with a `MxGatewayDashboard`
> cookie, the authenticated `GET /` rendered the admin nav, and an anonymous control redirected to
> `/login?ReturnUrl=%2F` — so the rotated credential is proven through the real
> `DashboardAuthenticator` search-bind path on the deployed host. As deployed, windev keeps
> `DisableLogin=true`, so routine operation there does not exercise LDAP; the standing regression
> proof is `DashboardLdapLiveTests` (5/5 green against `10.100.0.35` since commit `de67b45`).
>
> **Correction 4 — the lockout caution under "Verifying the rotation" is inert for this instance.**
> It warns that GLAuth's 3-fail / 10-minute per-IP lockout can lock the whole office when testing
> that the old value is dead. This GLAuth runs `LimitFailedBinds = false` (`config.toml:14`), so no
> failed-bind limiter is active and the caution does not apply here. Keep the caution for any
> instance that enables the limiter.

Operator steps to rotate the shared GLAuth service-account password after the repo-side
removal landed (SEC-36). The repo change (removal of the committed value, the two supported
secret channels, and this runbook) is already merged; the live rotation below is the
load-bearing half and is yours to execute.

> **Never put the old or new password in this repo, in a commit, in a chat, or in this file.**
> The value lives only in the GLAuth source of truth and in each host's out-of-band channel.

## Why

The dev GLAuth service-account password (`cn=serviceaccount,dc=zb,dc=local`) was historically
committed to this repo. Removal alone is insufficient — the old value is permanently recoverable
from git history — so **rotation is required**. Until the shared GLAuth on `10.100.0.35:3893`
stops honoring the old value, the repo history discloses a live directory account with LDAP
search capability over `dc=zb,dc=local`.

## Where the credential lives now (three channels, all bind `MxGateway:Ldap:ServiceAccountPassword`)

- **Source of truth:** `scadaproj/infra/glauth/config.toml` on host `10.100.0.35` (the `serviceaccount`
  user's `passsha256`). `scadaproj` is a shared monorepo — stage only the explicit glauth paths.
- **Encrypted secrets store (gateway default):** `appsettings.json` ships `${secret:ldap/mxgateway/bind}`,
  resolved from the local encrypted store (seed with `secret set ldap/mxgateway/bind <value>`).
- **Deployed hosts:** env var `MxGateway__Ldap__ServiceAccountPassword` in the NSSM service environment.
- **Dev boxes:** `dotnet user-secrets set "MxGateway:Ldap:ServiceAccountPassword" <value>`
  (the server carries `<UserSecretsId>mxaccessgw-server</UserSecretsId>`).

See `docs/GatewayConfiguration.md` (the `ServiceAccountPassword` row) and `glauth.md`.

## Preconditions

- SSH access to the GLAuth docker host `10.100.0.35` and to the deployed gateway host(s).
- Write access to `scadaproj/infra/glauth/`.
- Know which deployed hosts run LDAP-backed dashboard login:
  - **`10.100.0.48`** (`windev`) — primary; verify here.
  - **`wonder-app-vd03`** — its dashboard is disabled. **Check `MxGateway:Ldap:Enabled` there first.**
    If LDAP is disabled (`Enabled=false`), it has nothing to bind and needs no env var — skip it.
- A generated replacement secret (see step 1). Generate the `passsha256` per `glauth.md`
  ("Generate `passsha256` from a plaintext password").

## Cutover order

Follow this order so no window opens where the deployed dashboard cannot bind. **Do not rotate
GLAuth before the deployed hosts already carry the new value.**

1. **Generate the new secret in `scadaproj/infra/glauth/`.** Pick a new password, compute its
   `passsha256`, and stage the change to the `serviceaccount` user in `config.toml` (do not
   `docker compose up` yet — the directory must keep honoring the OLD value until the deployed
   hosts carry the NEW one).

2. **Pre-stage the NEW value on every LDAP-enabled deployed host** via the env-var channel, so the
   host is ready the instant GLAuth flips:
   ```powershell
   nssm get MxAccessGw AppEnvironmentExtra
   nssm set MxAccessGw AppEnvironmentExtra MxGateway__Ldap__ServiceAccountPassword=<new-value>
   # restart the service so the new environment is picked up
   nssm restart MxAccessGw
   ```
   Do this on `10.100.0.48`, and on `wonder-app-vd03` **only if** `MxGateway:Ldap:Enabled=true` there.
   (Alternatively seed the encrypted store with `secret set ldap/mxgateway/bind <new-value>`; the
   env var overrides the store and is the simplest per-host mechanism.)
   At this moment the deployed host holds the NEW value but GLAuth still honors the OLD one — binds
   still fail closed against the old directory, which is expected and brief; proceed immediately.

3. **Rotate GLAuth on `10.100.0.35`** to honor the new value:
   ```bash
   ssh 10.100.0.35
   cd ~/Desktop/scadaproj/infra/glauth
   docker compose up -d --force-recreate
   docker compose logs -f          # confirm clean startup, no TOML parse error
   ```

4. **Verify dashboard login on the deployed host(s).** Browse to the gateway dashboard on
   `10.100.0.48` and log in as `multi-role` / `password` (Administrator) — a successful login proves
   the search bind used the new service-account credential end-to-end. If `wonder-app-vd03` runs
   LDAP, verify it too; if its dashboard/LDAP is disabled, no check is needed.

5. **The repo change is already landed** (removal of the committed value, `<UserSecretsId>`, the
   validator message naming the two channels, and doc/scrub updates). Nothing more to commit for
   the cutover.

6. **Developers set user-secrets on next pull.** After pulling, a dev box with no secret configured
   will fail startup with a validation message naming the exact command. One-time per machine:
   ```bash
   dotnet user-secrets set "MxGateway:Ldap:ServiceAccountPassword" <new-value>
   ```
   (value from `scadaproj/infra/glauth/`, never from a repo file).

## Verifying the rotation

- **Primary:** dashboard `/login` as `multi-role` on `10.100.0.48` succeeds (step 4).
- **`wonder-app-vd03`:** only if `MxGateway:Ldap:Enabled=true`; otherwise no action.
- **Live-LDAP integration tests** (opt-in, only where the GLAuth instance is reachable):
  ```bash
  $env:MXGATEWAY_RUN_LIVE_LDAP_TESTS = "1"
  $env:MxGateway__Ldap__ServiceAccountPassword = "<new-value>"   # shell env only, never committed
  dotnet test src/ZB.MOM.WW.MxGateway.IntegrationTests/ZB.MOM.WW.MxGateway.IntegrationTests.csproj `
    --filter FullyQualifiedName~DashboardLdapLiveTests
  ```
  A green `DashboardLdapLiveTests` run confirms the new credential binds and searches. Where GLAuth
  is unreachable, document the suite as skipped per the `docs/GatewayTesting.md` opt-in matrix.
- **Old value is dead:** after step 3, a bind with the old password must fail. Do not test this from
  a shared-NAT box — GLAuth's 3-fail / 10-minute per-IP lockout can lock the whole office.

## Rollback

If dashboard login breaks after step 3, restore the previous `passsha256` in
`scadaproj/infra/glauth/config.toml`, `docker compose up -d --force-recreate`, and re-point the
deployed hosts' env var / store back to the previous value. Because the deployed hosts were
pre-staged in step 2, the exposure window is only steps 2→4.

## Done criteria

- GLAuth on `10.100.0.35` honors only the new value.
- Every LDAP-enabled deployed host binds with the new value (dashboard login verified).
- The source of truth `scadaproj/infra/glauth/config.toml` carries the new `passsha256`.
- No repo file (this one included) contains the old or new value.
- The SEC-36 tracker rows are `Done` with this runbook cited for the operator action.