Files
mxaccessgw/docs/deploy/2026-11-12-vd03-pepper-to-service-env.md
T
Joseph Doherty 04a40f2c1a
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m50s
ci / java (push) Failing after 4m55s
ci / portable (push) Failing after 11m17s
docs(deploy): vd03 API-key pepper moved into the service environment (#171)
Record of the 2026-11-12 config-only change on wonder-app-vd03: MxGateway__ApiKeyPepper
added to the MxAccessGw AppEnvironmentExtra block (same value, sha256 prefix match),
removed from Server\appsettings.json; authenticated round-trip with a pre-change key,
ScadaBridge MxGateway link unchanged. Runbook and configuration doc now say where the
pepper lives on each wonder box.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 19:49:42 -05:00

4.3 KiB

2026-11-12: wonder-app-vd03 (DEV): API-key pepper moved into the service environment (mxaccessgw#171)

Config-only change on the DEV box, no binaries touched (Server/Worker stay at 2b13564e).

Why

The pepper (MxGateway:ApiKeyPepper) lived in E:\ApiInstall\MxGateway\Server\appsettings.json, inside the directory every upgrade renames or replaces. Losing or changing the pepper invalidates every issued mxgw_... token as completely as losing gateway-auth.db. The QA boxes (vq03/vq04) already keep it in the NSSM AppEnvironmentExtra block; vd03 now does the same.

What changed

Item Before After
NSSM AppEnvironmentExtra (MxAccessGw) 6 entries, no pepper 7 entries: the same 6 unchanged + MxGateway__ApiKeyPepper
Server\appsettings.json carried MxGateway:ApiKeyPepper (sha256 035E641A…) that one line removed, nothing else changed (sha256 464ACE1B…); Authentication:PepperSecretName still MxGateway:ApiKeyPepper
Effective pepper sha256 prefix 0d01abcc1fe5 sha256 prefix 0d01abcc1fe5 (same value, compared by hash prefix only, never printed)

The env value was written straight to the registry (REG_MULTI_SZ), not via nssm set, so the pepper never appeared on a process command line. The JSON edit removed exactly one line; the edited file was parsed and compared object-for-object with the original minus that property before it was written (UTF-8 BOM and CRLF kept). The service was stopped before the edit so the running host's reloadOnChange could not drop the pepper before the environment took effect.

Not changed: the install-root master E:\ApiInstall\MxGateway\appsettings.json still carries the pepper. It sits outside the swapped Server\ directory, and install-service.bat copies it into Server\ and replaces the NSSM env block (dropping MxGateway__ApiKeyPepper), so the master copy is what keeps a reinstall working. Old Server.bak.* folders also still hold it; they are rollback targets.

Timeline (box-local time)

  • 19:47:51 backup taken, service stopped (clean, 2 s, no STOP_PENDING, 0 gateway/worker processes)
  • 19:47:53 env written and read back (pepper hash prefix match, original 6 entries identical), appsettings.json rewritten
  • 19:47:57 /health/ready 200 (downtime about 6 s); 30 s later still Running on the same PID, exactly one gateway process

Verification

  • /health/ready 200 (auth-store, worker-executable, galaxy-sql Healthy); /health/active 2 of 2 sessions ready.
  • Authenticated round-trip from the Mac with an existing pre-change key (scadabridgeshared, MXGATEWAY_API_KEY env, .NET client CLI): galaxy-test-connection {ok:true}, open-session (worker spawned) + close-session OK. The same probe ran before the change. Negative control: bogus key → MxGatewayAuthenticationException. Since appsettings.json no longer has the pepper, a pre-existing token verifying proves the environment supplies the same value.
  • ScadaBridge vd03 site health after the restart: MxGateway connection Connected, 810/810 tags resolved, quality good 398 / bad 412, identical to the pre-change report (the Bad tags are pre-existing idle receivers). Z28061/Z28062 debug snapshots identical before/after.
  • OtOpcUa vd03 /health/ready and /health/active 200 (it has no Galaxy driver on vd03, so it does not call the gateway).

Backup and rollback

Backup folder E:\psrem-stage\mxgw-171-20261112T1950\ (ACL: SYSTEM + Administrators only, inheritance off): backup\Server-appsettings.json (+ Server-appsettings.Development.json, root-appsettings.json), backup\MxAccessGw-servicekey-20261112T1950.reg, backup\AppEnvironmentExtra-pre.txt, run log _NOTES-20261112T1950.txt. On-box note: E:\ApiInstall\MxGateway\_DEPLOY-NOTES-20261112T1950-issue171-pepper-to-env.txt.

Rollback: stop MxAccessGw, copy backup\Server-appsettings.json over Server\appsettings.json, restore AppEnvironmentExtra from backup\AppEnvironmentExtra-pre.txt (or import the .reg), start.

For the next upgrade

Upgrade-MxGatewayService.ps1 already treats the env block as authoritative and copies the live appsettings*.json back byte-identical, so nothing changes in the procedure. Expect an env count of 7 on vd03 now. Never rewrite the block with nssm set AppEnvironmentExtra without re-including MxGateway__ApiKeyPepper.