Record of the 2026-11-12 config-only change on wonder-app-vd03: MxGateway__ApiKeyPepper added to the MxAccessGw AppEnvironmentExtra block (same value, sha256 prefix match), removed from Server\appsettings.json; authenticated round-trip with a pre-change key, ScadaBridge MxGateway link unchanged. Runbook and configuration doc now say where the pepper lives on each wonder box. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
4.3 KiB
2026-11-12: wonder-app-vd03 (DEV): API-key pepper moved into the service environment (mxaccessgw#171)
Config-only change on the DEV box, no binaries touched (Server/Worker stay at 2b13564e).
Why
The pepper (MxGateway:ApiKeyPepper) lived in E:\ApiInstall\MxGateway\Server\appsettings.json, inside the
directory every upgrade renames or replaces. Losing or changing the pepper invalidates every issued mxgw_...
token as completely as losing gateway-auth.db. The QA boxes (vq03/vq04) already keep it in the NSSM
AppEnvironmentExtra block; vd03 now does the same.
What changed
| Item | Before | After |
|---|---|---|
NSSM AppEnvironmentExtra (MxAccessGw) |
6 entries, no pepper | 7 entries: the same 6 unchanged + MxGateway__ApiKeyPepper |
Server\appsettings.json |
carried MxGateway:ApiKeyPepper (sha256 035E641A…) |
that one line removed, nothing else changed (sha256 464ACE1B…); Authentication:PepperSecretName still MxGateway:ApiKeyPepper |
| Effective pepper | sha256 prefix 0d01abcc1fe5 |
sha256 prefix 0d01abcc1fe5 (same value, compared by hash prefix only, never printed) |
The env value was written straight to the registry (REG_MULTI_SZ), not via nssm set, so the pepper never
appeared on a process command line. The JSON edit removed exactly one line; the edited file was parsed and
compared object-for-object with the original minus that property before it was written (UTF-8 BOM and CRLF kept).
The service was stopped before the edit so the running host's reloadOnChange could not drop the pepper
before the environment took effect.
Not changed: the install-root master E:\ApiInstall\MxGateway\appsettings.json still carries the
pepper. It sits outside the swapped Server\ directory, and install-service.bat copies it into
Server\ and replaces the NSSM env block (dropping MxGateway__ApiKeyPepper), so the master copy is
what keeps a reinstall working. Old Server.bak.* folders also still hold it; they are rollback targets.
Timeline (box-local time)
- 19:47:51 backup taken, service stopped (clean, 2 s, no STOP_PENDING, 0 gateway/worker processes)
- 19:47:53 env written and read back (pepper hash prefix match, original 6 entries identical),
appsettings.jsonrewritten - 19:47:57
/health/ready200 (downtime about 6 s); 30 s later still Running on the same PID, exactly one gateway process
Verification
/health/ready200 (auth-store, worker-executable, galaxy-sql Healthy);/health/active2 of 2 sessions ready.- Authenticated round-trip from the Mac with an existing pre-change key (
scadabridgeshared,MXGATEWAY_API_KEYenv, .NET client CLI):galaxy-test-connection{ok:true},open-session(worker spawned) +close-sessionOK. The same probe ran before the change. Negative control: bogus key →MxGatewayAuthenticationException. Sinceappsettings.jsonno longer has the pepper, a pre-existing token verifying proves the environment supplies the same value. - ScadaBridge vd03 site health after the restart: MxGateway connection
Connected, 810/810 tags resolved, quality good 398 / bad 412, identical to the pre-change report (the Bad tags are pre-existing idle receivers).Z28061/Z28062debug snapshots identical before/after. - OtOpcUa vd03
/health/readyand/health/active200 (it has no Galaxy driver on vd03, so it does not call the gateway).
Backup and rollback
Backup folder E:\psrem-stage\mxgw-171-20261112T1950\ (ACL: SYSTEM + Administrators only, inheritance off):
backup\Server-appsettings.json (+ Server-appsettings.Development.json, root-appsettings.json),
backup\MxAccessGw-servicekey-20261112T1950.reg, backup\AppEnvironmentExtra-pre.txt, run log _NOTES-20261112T1950.txt.
On-box note: E:\ApiInstall\MxGateway\_DEPLOY-NOTES-20261112T1950-issue171-pepper-to-env.txt.
Rollback: stop MxAccessGw, copy backup\Server-appsettings.json over Server\appsettings.json, restore
AppEnvironmentExtra from backup\AppEnvironmentExtra-pre.txt (or import the .reg), start.
For the next upgrade
Upgrade-MxGatewayService.ps1 already treats the env block as authoritative and copies the live
appsettings*.json back byte-identical, so nothing changes in the procedure. Expect an env count of 7
on vd03 now. Never rewrite the block with nssm set AppEnvironmentExtra without re-including
MxGateway__ApiKeyPepper.