Joseph Doherty
1d6858939d
SEC-36's primary check -- dashboard /login through the real DashboardAuthenticator search bind -- was deferred because windev's gateway was crash-looping on the stale deployment filed as NEXT-07. That host was redeployed 2026-08-07, so the check ran: login as multi-role returns 302 with the dashboard cookie and the authenticated page renders the admin nav, while an anonymous control still redirects to /login. The rotated service-account credential is now proven end-to-end on the deployed host, not only by the equivalent ldapsearch primitive, and the runbook's Correction 3 is past tense throughout rather than describing a fault that no longer exists. Also record why windev runs the Staging environment name. The redeploy tripped SEC-06's Production hard-stop on Ldap:Transport=None, and windev cannot satisfy it: it binds the shared GLAuth, which offers no TLS, and runs Dashboard:DisableLogin=true. The Production label contradicted its own configuration, so the host was relabelled rather than the guard weakened -- exactly the permissive-staging-rig case the SEC-35 section already carves out.
172 lines
11 KiB
Markdown
172 lines
11 KiB
Markdown
# SEC-36 — LDAP Service-Account Credential Rotation (Operator Runbook)
|
|
|
|
> **Executed 2026-08-07 — the rotation is done; this runbook is now history plus the four
|
|
> corrections below.** A new service-account password was generated, `scadaproj/infra/glauth/config.toml`'s
|
|
> `serviceaccount` `passsha256` was replaced and the shared GLAuth recreated, and the old value
|
|
> (the literal this repo committed, live in the directory since 2026-06-04) no longer binds. The new
|
|
> value now exists only in the
|
|
> three channels this runbook names: the GLAuth `passsha256` (committed in `scadaproj`), the NSSM
|
|
> service environment on `10.100.0.48`, and this dev Mac's user-secrets. The retired plaintext was
|
|
> also scrubbed from `scadaproj/infra/glauth/`'s `config.toml`/`docker-compose.yml`/`README.md`
|
|
> comments and from the host's live `docker-compose.yml` (the `*.bak-sec36` backups on the host still
|
|
> carry it, deliberately — they are the rollback artifacts).
|
|
>
|
|
> **Correction 1 — host paths in step 3 were stale.** The runbook says
|
|
> `cd ~/Desktop/scadaproj/infra/glauth` on `10.100.0.35`. That directory does not exist there:
|
|
> `scadaproj` is a dev-workstation checkout, and the docker host runs the stack from
|
|
> **`/home/dohertj2/zb-glauth`** (container **`zb-shared-glauth`**, project name `zb-shared-glauth`).
|
|
> The repo remains the source of truth; deployment is the `scp` of `config.toml`/`docker-compose.yml`
|
|
> into `~/zb-glauth` documented in `scadaproj/infra/glauth/README.md`, followed by
|
|
> `docker compose up -d --force-recreate` there.
|
|
>
|
|
> **Correction 2 — `wonder-app-vd03` is out of scope, on documentary evidence.** The precondition
|
|
> above says to check `MxGateway:Ldap:Enabled` on that host. It could not be checked directly (the
|
|
> host is unreachable from the dev network), but it is out of scope regardless: its gateway binds a
|
|
> **different directory** — the ScadaBridge/ScadaLink local GLAuth under `dc=scadalink`/`dc=scadabridge`,
|
|
> not `dc=zb,dc=local` — so this credential is not one it can hold. No env var was staged there and
|
|
> none is needed.
|
|
>
|
|
> **Correction 3 — step 4's dashboard verification is deferred on `10.100.0.48`; a direct bind was
|
|
> used instead.** The NEW value **is** staged on windev (added as the 10th `AppEnvironmentExtra`
|
|
> entry on the `MxAccessGw` NSSM service), but dashboard `/login` could not exercise it at rotation
|
|
> time: windev's gateway was **crash-looping on a pre-existing, unrelated fault** — the deployed Server binary
|
|
> (2026-06-25) predates the auth-DB migration of 2026-07-15, so it opens a schema-version-3 database
|
|
> it only supports at version 2 and aborts at startup (~10k Hosting-failed events/day since at least
|
|
> 08-06). That was a stale-deployment problem, not a rotation problem; it was filed as next-cycle
|
|
> finding NEXT-07 and resolved by redeploy on 2026-08-07. **Verification used instead:** a direct `ldapsearch` bind as
|
|
> `cn=serviceaccount,dc=zb,dc=local` with the new value against `10.100.0.35:3893` succeeded and
|
|
> returned the `multi-role` entry — which is precisely the search bind the dashboard performs, minus
|
|
> the HTTP shell. **The deferred check was completed 2026-08-07**, once windev was repaired by the
|
|
> redeploy filed under NEXT-07. With `Dashboard:DisableLogin=false` supplied as a process-env-only
|
|
> override on a foreground run of the new build, `GET /login` returned 200 with an antiforgery token,
|
|
> `POST /auth/login` as `multi-role`/`password` returned 302 to `/` with a `MxGatewayDashboard`
|
|
> cookie, the authenticated `GET /` rendered the admin nav, and an anonymous control redirected to
|
|
> `/login?ReturnUrl=%2F` — so the rotated credential is proven through the real
|
|
> `DashboardAuthenticator` search-bind path on the deployed host. As deployed, windev keeps
|
|
> `DisableLogin=true`, so routine operation there does not exercise LDAP; the standing regression
|
|
> proof is `DashboardLdapLiveTests` (5/5 green against `10.100.0.35` since commit `de67b45`).
|
|
>
|
|
> **Correction 4 — the lockout caution under "Verifying the rotation" is inert for this instance.**
|
|
> It warns that GLAuth's 3-fail / 10-minute per-IP lockout can lock the whole office when testing
|
|
> that the old value is dead. This GLAuth runs `LimitFailedBinds = false` (`config.toml:14`), so no
|
|
> failed-bind limiter is active and the caution does not apply here. Keep the caution for any
|
|
> instance that enables the limiter.
|
|
|
|
Operator steps to rotate the shared GLAuth service-account password after the repo-side
|
|
removal landed (SEC-36). The repo change (removal of the committed value, the two supported
|
|
secret channels, and this runbook) is already merged; the live rotation below is the
|
|
load-bearing half and is yours to execute.
|
|
|
|
> **Never put the old or new password in this repo, in a commit, in a chat, or in this file.**
|
|
> The value lives only in the GLAuth source of truth and in each host's out-of-band channel.
|
|
|
|
## Why
|
|
|
|
The dev GLAuth service-account password (`cn=serviceaccount,dc=zb,dc=local`) was historically
|
|
committed to this repo. Removal alone is insufficient — the old value is permanently recoverable
|
|
from git history — so **rotation is required**. Until the shared GLAuth on `10.100.0.35:3893`
|
|
stops honoring the old value, the repo history discloses a live directory account with LDAP
|
|
search capability over `dc=zb,dc=local`.
|
|
|
|
## Where the credential lives now (three channels, all bind `MxGateway:Ldap:ServiceAccountPassword`)
|
|
|
|
- **Source of truth:** `scadaproj/infra/glauth/config.toml` on host `10.100.0.35` (the `serviceaccount`
|
|
user's `passsha256`). `scadaproj` is a shared monorepo — stage only the explicit glauth paths.
|
|
- **Encrypted secrets store (gateway default):** `appsettings.json` ships `${secret:ldap/mxgateway/bind}`,
|
|
resolved from the local encrypted store (seed with `secret set ldap/mxgateway/bind <value>`).
|
|
- **Deployed hosts:** env var `MxGateway__Ldap__ServiceAccountPassword` in the NSSM service environment.
|
|
- **Dev boxes:** `dotnet user-secrets set "MxGateway:Ldap:ServiceAccountPassword" <value>`
|
|
(the server carries `<UserSecretsId>mxaccessgw-server</UserSecretsId>`).
|
|
|
|
See `docs/GatewayConfiguration.md` (the `ServiceAccountPassword` row) and `glauth.md`.
|
|
|
|
## Preconditions
|
|
|
|
- SSH access to the GLAuth docker host `10.100.0.35` and to the deployed gateway host(s).
|
|
- Write access to `scadaproj/infra/glauth/`.
|
|
- Know which deployed hosts run LDAP-backed dashboard login:
|
|
- **`10.100.0.48`** (`windev`) — primary; verify here.
|
|
- **`wonder-app-vd03`** — its dashboard is disabled. **Check `MxGateway:Ldap:Enabled` there first.**
|
|
If LDAP is disabled (`Enabled=false`), it has nothing to bind and needs no env var — skip it.
|
|
- A generated replacement secret (see step 1). Generate the `passsha256` per `glauth.md`
|
|
("Generate `passsha256` from a plaintext password").
|
|
|
|
## Cutover order
|
|
|
|
Follow this order so no window opens where the deployed dashboard cannot bind. **Do not rotate
|
|
GLAuth before the deployed hosts already carry the new value.**
|
|
|
|
1. **Generate the new secret in `scadaproj/infra/glauth/`.** Pick a new password, compute its
|
|
`passsha256`, and stage the change to the `serviceaccount` user in `config.toml` (do not
|
|
`docker compose up` yet — the directory must keep honoring the OLD value until the deployed
|
|
hosts carry the NEW one).
|
|
|
|
2. **Pre-stage the NEW value on every LDAP-enabled deployed host** via the env-var channel, so the
|
|
host is ready the instant GLAuth flips:
|
|
```powershell
|
|
nssm get MxAccessGw AppEnvironmentExtra
|
|
nssm set MxAccessGw AppEnvironmentExtra MxGateway__Ldap__ServiceAccountPassword=<new-value>
|
|
# restart the service so the new environment is picked up
|
|
nssm restart MxAccessGw
|
|
```
|
|
Do this on `10.100.0.48`, and on `wonder-app-vd03` **only if** `MxGateway:Ldap:Enabled=true` there.
|
|
(Alternatively seed the encrypted store with `secret set ldap/mxgateway/bind <new-value>`; the
|
|
env var overrides the store and is the simplest per-host mechanism.)
|
|
At this moment the deployed host holds the NEW value but GLAuth still honors the OLD one — binds
|
|
still fail closed against the old directory, which is expected and brief; proceed immediately.
|
|
|
|
3. **Rotate GLAuth on `10.100.0.35`** to honor the new value:
|
|
```bash
|
|
ssh 10.100.0.35
|
|
cd ~/Desktop/scadaproj/infra/glauth
|
|
docker compose up -d --force-recreate
|
|
docker compose logs -f # confirm clean startup, no TOML parse error
|
|
```
|
|
|
|
4. **Verify dashboard login on the deployed host(s).** Browse to the gateway dashboard on
|
|
`10.100.0.48` and log in as `multi-role` / `password` (Administrator) — a successful login proves
|
|
the search bind used the new service-account credential end-to-end. If `wonder-app-vd03` runs
|
|
LDAP, verify it too; if its dashboard/LDAP is disabled, no check is needed.
|
|
|
|
5. **The repo change is already landed** (removal of the committed value, `<UserSecretsId>`, the
|
|
validator message naming the two channels, and doc/scrub updates). Nothing more to commit for
|
|
the cutover.
|
|
|
|
6. **Developers set user-secrets on next pull.** After pulling, a dev box with no secret configured
|
|
will fail startup with a validation message naming the exact command. One-time per machine:
|
|
```bash
|
|
dotnet user-secrets set "MxGateway:Ldap:ServiceAccountPassword" <new-value>
|
|
```
|
|
(value from `scadaproj/infra/glauth/`, never from a repo file).
|
|
|
|
## Verifying the rotation
|
|
|
|
- **Primary:** dashboard `/login` as `multi-role` on `10.100.0.48` succeeds (step 4).
|
|
- **`wonder-app-vd03`:** only if `MxGateway:Ldap:Enabled=true`; otherwise no action.
|
|
- **Live-LDAP integration tests** (opt-in, only where the GLAuth instance is reachable):
|
|
```bash
|
|
$env:MXGATEWAY_RUN_LIVE_LDAP_TESTS = "1"
|
|
$env:MxGateway__Ldap__ServiceAccountPassword = "<new-value>" # shell env only, never committed
|
|
dotnet test src/ZB.MOM.WW.MxGateway.IntegrationTests/ZB.MOM.WW.MxGateway.IntegrationTests.csproj `
|
|
--filter FullyQualifiedName~DashboardLdapLiveTests
|
|
```
|
|
A green `DashboardLdapLiveTests` run confirms the new credential binds and searches. Where GLAuth
|
|
is unreachable, document the suite as skipped per the `docs/GatewayTesting.md` opt-in matrix.
|
|
- **Old value is dead:** after step 3, a bind with the old password must fail. Do not test this from
|
|
a shared-NAT box — GLAuth's 3-fail / 10-minute per-IP lockout can lock the whole office.
|
|
|
|
## Rollback
|
|
|
|
If dashboard login breaks after step 3, restore the previous `passsha256` in
|
|
`scadaproj/infra/glauth/config.toml`, `docker compose up -d --force-recreate`, and re-point the
|
|
deployed hosts' env var / store back to the previous value. Because the deployed hosts were
|
|
pre-staged in step 2, the exposure window is only steps 2→4.
|
|
|
|
## Done criteria
|
|
|
|
- GLAuth on `10.100.0.35` honors only the new value.
|
|
- Every LDAP-enabled deployed host binds with the new value (dashboard login verified).
|
|
- The source of truth `scadaproj/infra/glauth/config.toml` carries the new `passsha256`.
|
|
- No repo file (this one included) contains the old or new value.
|
|
- The SEC-36 tracker rows are `Done` with this runbook cited for the operator action.
|