docs(deploy): vd03 API-key pepper moved into the service environment (#171)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m50s
ci / java (push) Failing after 4m55s
ci / portable (push) Failing after 11m17s

Record of the 2026-11-12 config-only change on wonder-app-vd03: MxGateway__ApiKeyPepper
added to the MxAccessGw AppEnvironmentExtra block (same value, sha256 prefix match),
removed from Server\appsettings.json; authenticated round-trip with a pre-change key,
ScadaBridge MxGateway link unchanged. Runbook and configuration doc now say where the
pepper lives on each wonder box.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
2026-11-12 19:49:42 -05:00
parent 447070a532
commit 04a40f2c1a
3 changed files with 68 additions and 2 deletions
+2 -1
View File
@@ -375,7 +375,8 @@ nssm set MxAccessGw AppEnvironmentExtra ZB_SECRETS_MASTER_KEY=<base64-32-bytes>
`AppEnvironmentExtra` is the same newline-separated `KEY=VALUE` block that `AppEnvironmentExtra` is the same newline-separated `KEY=VALUE` block that
already carries `ASPNETCORE_ENVIRONMENT`, `MxGateway__Worker__ExecutablePath`, already carries `ASPNETCORE_ENVIRONMENT`, `MxGateway__Worker__ExecutablePath`,
and the API-key pepper (see the `wonder-app-vd03`/`windev` deploy notes) — and the API-key pepper (`MxGateway__ApiKeyPepper`; on `wonder-app-vd03` only since
2026-11-12, see `docs/deploy/2026-11-12-vd03-pepper-to-service-env.md`) —
`nssm set ... AppEnvironmentExtra` **replaces the entire block**, so always `nssm set ... AppEnvironmentExtra` **replaces the entire block**, so always
`get` first and re-include every existing entry alongside the new key, then `get` first and re-include every existing entry alongside the new key, then
restart the service for it to take effect. restart the service for it to take effect.
@@ -0,0 +1,65 @@
# 2026-11-12: wonder-app-vd03 (DEV): API-key pepper moved into the service environment (mxaccessgw#171)
Config-only change on the DEV box, no binaries touched (Server/Worker stay at `2b13564e`).
## Why
The pepper (`MxGateway:ApiKeyPepper`) lived in `E:\ApiInstall\MxGateway\Server\appsettings.json`, inside the
directory every upgrade renames or replaces. Losing or changing the pepper invalidates every issued `mxgw_...`
token as completely as losing `gateway-auth.db`. The QA boxes (vq03/vq04) already keep it in the NSSM
`AppEnvironmentExtra` block; vd03 now does the same.
## What changed
| Item | Before | After |
|---|---|---|
| NSSM `AppEnvironmentExtra` (`MxAccessGw`) | 6 entries, no pepper | 7 entries: the same 6 unchanged + `MxGateway__ApiKeyPepper` |
| `Server\appsettings.json` | carried `MxGateway:ApiKeyPepper` (sha256 `035E641A…`) | that one line removed, nothing else changed (sha256 `464ACE1B…`); `Authentication:PepperSecretName` still `MxGateway:ApiKeyPepper` |
| Effective pepper | sha256 prefix `0d01abcc1fe5` | sha256 prefix `0d01abcc1fe5` (same value, compared by hash prefix only, never printed) |
The env value was written straight to the registry (`REG_MULTI_SZ`), not via `nssm set`, so the pepper never
appeared on a process command line. The JSON edit removed exactly one line; the edited file was parsed and
compared object-for-object with the original minus that property before it was written (UTF-8 BOM and CRLF kept).
The service was stopped before the edit so the running host's `reloadOnChange` could not drop the pepper
before the environment took effect.
**Not changed:** the install-root master `E:\ApiInstall\MxGateway\appsettings.json` still carries the
pepper. It sits outside the swapped `Server\` directory, and `install-service.bat` copies it into
`Server\` and **replaces** the NSSM env block (dropping `MxGateway__ApiKeyPepper`), so the master copy is
what keeps a reinstall working. Old `Server.bak.*` folders also still hold it; they are rollback targets.
## Timeline (box-local time)
- 19:47:51 backup taken, service stopped (clean, 2 s, no STOP_PENDING, 0 gateway/worker processes)
- 19:47:53 env written and read back (pepper hash prefix match, original 6 entries identical), `appsettings.json` rewritten
- 19:47:57 `/health/ready` 200 (downtime about 6 s); 30 s later still Running on the same PID, exactly one gateway process
## Verification
- `/health/ready` 200 (auth-store, worker-executable, galaxy-sql Healthy); `/health/active` 2 of 2 sessions ready.
- Authenticated round-trip from the Mac with an existing pre-change key (`scadabridgeshared`, `MXGATEWAY_API_KEY`
env, .NET client CLI): `galaxy-test-connection` `{ok:true}`, `open-session` (worker spawned) + `close-session` OK.
The same probe ran before the change. Negative control: bogus key → `MxGatewayAuthenticationException`.
Since `appsettings.json` no longer has the pepper, a pre-existing token verifying proves the environment
supplies the same value.
- ScadaBridge vd03 site health after the restart: MxGateway connection `Connected`, 810/810 tags resolved,
quality good 398 / bad 412, identical to the pre-change report (the Bad tags are pre-existing idle receivers).
`Z28061`/`Z28062` debug snapshots identical before/after.
- OtOpcUa vd03 `/health/ready` and `/health/active` 200 (it has no Galaxy driver on vd03, so it does not call the gateway).
## Backup and rollback
Backup folder `E:\psrem-stage\mxgw-171-20261112T1950\` (ACL: SYSTEM + Administrators only, inheritance off):
`backup\Server-appsettings.json` (+ `Server-appsettings.Development.json`, `root-appsettings.json`),
`backup\MxAccessGw-servicekey-20261112T1950.reg`, `backup\AppEnvironmentExtra-pre.txt`, run log `_NOTES-20261112T1950.txt`.
On-box note: `E:\ApiInstall\MxGateway\_DEPLOY-NOTES-20261112T1950-issue171-pepper-to-env.txt`.
Rollback: stop `MxAccessGw`, copy `backup\Server-appsettings.json` over `Server\appsettings.json`, restore
`AppEnvironmentExtra` from `backup\AppEnvironmentExtra-pre.txt` (or import the `.reg`), start.
## For the next upgrade
`Upgrade-MxGatewayService.ps1` already treats the env block as authoritative and copies the live
`appsettings*.json` back byte-identical, so nothing changes in the procedure. Expect an env count of **7**
on vd03 now. Never rewrite the block with `nssm set AppEnvironmentExtra` without re-including
`MxGateway__ApiKeyPepper`.
+1 -1
View File
@@ -117,7 +117,7 @@ validator therefore rejects stores inside the content root (see the 2026-08-09 i
| State | Setting | Default location (Windows) | Notes | | State | Setting | Default location (Windows) | Notes |
|---|---|---|---| |---|---|---|---|
| API-key store (SQLite) | `MxGateway:Authentication:SqlitePath` (NSSM: `MxGateway__Authentication__SqlitePath`) | `C:\ProgramData\MxGateway\gateway-auth.db` | Must be rooted; validator also rejects a path inside the content root. Back up the `-wal`/`-shm` siblings with it. | | API-key store (SQLite) | `MxGateway:Authentication:SqlitePath` (NSSM: `MxGateway__Authentication__SqlitePath`) | `C:\ProgramData\MxGateway\gateway-auth.db` | Must be rooted; validator also rejects a path inside the content root. Back up the `-wal`/`-shm` siblings with it. |
| API-key **pepper** | secret `MxGateway:ApiKeyPepper` (`PepperSecretName`); lives outside the DB, in NSSM env / Secrets store | n/a | Without the same pepper every issued `mxgw_...` token fails verification. Never in the DB (`docs/Authentication.md`). Back it up in the org vault. | | API-key **pepper** | secret `MxGateway:ApiKeyPepper` (`PepperSecretName`); lives outside the DB, in NSSM env / Secrets store | n/a | Without the same pepper every issued `mxgw_...` token fails verification. Never in the DB (`docs/Authentication.md`). Back it up in the org vault. On the wonder boxes it is the `MxGateway__ApiKeyPepper` env entry: QA vq03/vq04 since 2026-08-14, DEV vd03 since 2026-11-12 (moved out of `Server\appsettings.json`, mxaccessgw#171, `docs/deploy/2026-11-12-vd03-pepper-to-service-env.md`). |
| Secrets store (SQLite) | `Secrets:SqlitePath` (NSSM: `Secrets__SqlitePath`) | `C:\ProgramData\MxGateway\mxgateway-secrets.db` | **Must be absolute AND outside the content root**; fail-closed with no opt-out (`docs/GatewayConfiguration.md` L349; `scadaproj` Secrets notes). | | Secrets store (SQLite) | `Secrets:SqlitePath` (NSSM: `Secrets__SqlitePath`) | `C:\ProgramData\MxGateway\mxgateway-secrets.db` | **Must be absolute AND outside the content root**; fail-closed with no opt-out (`docs/GatewayConfiguration.md` L349; `scadaproj` Secrets notes). |
| Secrets master key (KEK) | `ZB_SECRETS_MASTER_KEY` in NSSM `AppEnvironmentExtra` | n/a | Losing it makes the store unrecoverable; no `RewrapAll` yet (`docs/GatewayConfiguration.md`, "Rotation / DR"). Back it up in the vault, not just on the box. | | Secrets master key (KEK) | `ZB_SECRETS_MASTER_KEY` in NSSM `AppEnvironmentExtra` | n/a | Losing it makes the store unrecoverable; no `RewrapAll` yet (`docs/GatewayConfiguration.md`, "Rotation / DR"). Back it up in the vault, not just on the box. |
| Galaxy snapshot cache | `MxGateway:Galaxy:SnapshotCachePath` | `C:\ProgramData\MxGateway\galaxy-snapshot.json` | Cache only; rejected inside content root so it survives deploys. Losing it is harmless. | | Galaxy snapshot cache | `MxGateway:Galaxy:SnapshotCachePath` | `C:\ProgramData\MxGateway\galaxy-snapshot.json` | Cache only; rejected inside content root so it survives deploys. Losing it is harmless. |