diff --git a/docs/GatewayConfiguration.md b/docs/GatewayConfiguration.md index f0e1d90a..c6271d1f 100644 --- a/docs/GatewayConfiguration.md +++ b/docs/GatewayConfiguration.md @@ -375,7 +375,8 @@ nssm set MxAccessGw AppEnvironmentExtra ZB_SECRETS_MASTER_KEY= `AppEnvironmentExtra` is the same newline-separated `KEY=VALUE` block that already carries `ASPNETCORE_ENVIRONMENT`, `MxGateway__Worker__ExecutablePath`, -and the API-key pepper (see the `wonder-app-vd03`/`windev` deploy notes) — +and the API-key pepper (`MxGateway__ApiKeyPepper`; on `wonder-app-vd03` only since +2026-11-12, see `docs/deploy/2026-11-12-vd03-pepper-to-service-env.md`) — `nssm set ... AppEnvironmentExtra` **replaces the entire block**, so always `get` first and re-include every existing entry alongside the new key, then restart the service for it to take effect. diff --git a/docs/deploy/2026-11-12-vd03-pepper-to-service-env.md b/docs/deploy/2026-11-12-vd03-pepper-to-service-env.md new file mode 100644 index 00000000..1af9d0a8 --- /dev/null +++ b/docs/deploy/2026-11-12-vd03-pepper-to-service-env.md @@ -0,0 +1,65 @@ +# 2026-11-12: wonder-app-vd03 (DEV): API-key pepper moved into the service environment (mxaccessgw#171) + +Config-only change on the DEV box, no binaries touched (Server/Worker stay at `2b13564e`). + +## Why + +The pepper (`MxGateway:ApiKeyPepper`) lived in `E:\ApiInstall\MxGateway\Server\appsettings.json`, inside the +directory every upgrade renames or replaces. Losing or changing the pepper invalidates every issued `mxgw_...` +token as completely as losing `gateway-auth.db`. The QA boxes (vq03/vq04) already keep it in the NSSM +`AppEnvironmentExtra` block; vd03 now does the same. + +## What changed + +| Item | Before | After | +|---|---|---| +| NSSM `AppEnvironmentExtra` (`MxAccessGw`) | 6 entries, no pepper | 7 entries: the same 6 unchanged + `MxGateway__ApiKeyPepper` | +| `Server\appsettings.json` | carried `MxGateway:ApiKeyPepper` (sha256 `035E641A…`) | that one line removed, nothing else changed (sha256 `464ACE1B…`); `Authentication:PepperSecretName` still `MxGateway:ApiKeyPepper` | +| Effective pepper | sha256 prefix `0d01abcc1fe5` | sha256 prefix `0d01abcc1fe5` (same value, compared by hash prefix only, never printed) | + +The env value was written straight to the registry (`REG_MULTI_SZ`), not via `nssm set`, so the pepper never +appeared on a process command line. The JSON edit removed exactly one line; the edited file was parsed and +compared object-for-object with the original minus that property before it was written (UTF-8 BOM and CRLF kept). +The service was stopped before the edit so the running host's `reloadOnChange` could not drop the pepper +before the environment took effect. + +**Not changed:** the install-root master `E:\ApiInstall\MxGateway\appsettings.json` still carries the +pepper. It sits outside the swapped `Server\` directory, and `install-service.bat` copies it into +`Server\` and **replaces** the NSSM env block (dropping `MxGateway__ApiKeyPepper`), so the master copy is +what keeps a reinstall working. Old `Server.bak.*` folders also still hold it; they are rollback targets. + +## Timeline (box-local time) + +- 19:47:51 backup taken, service stopped (clean, 2 s, no STOP_PENDING, 0 gateway/worker processes) +- 19:47:53 env written and read back (pepper hash prefix match, original 6 entries identical), `appsettings.json` rewritten +- 19:47:57 `/health/ready` 200 (downtime about 6 s); 30 s later still Running on the same PID, exactly one gateway process + +## Verification + +- `/health/ready` 200 (auth-store, worker-executable, galaxy-sql Healthy); `/health/active` 2 of 2 sessions ready. +- Authenticated round-trip from the Mac with an existing pre-change key (`scadabridgeshared`, `MXGATEWAY_API_KEY` + env, .NET client CLI): `galaxy-test-connection` `{ok:true}`, `open-session` (worker spawned) + `close-session` OK. + The same probe ran before the change. Negative control: bogus key → `MxGatewayAuthenticationException`. + Since `appsettings.json` no longer has the pepper, a pre-existing token verifying proves the environment + supplies the same value. +- ScadaBridge vd03 site health after the restart: MxGateway connection `Connected`, 810/810 tags resolved, + quality good 398 / bad 412, identical to the pre-change report (the Bad tags are pre-existing idle receivers). + `Z28061`/`Z28062` debug snapshots identical before/after. +- OtOpcUa vd03 `/health/ready` and `/health/active` 200 (it has no Galaxy driver on vd03, so it does not call the gateway). + +## Backup and rollback + +Backup folder `E:\psrem-stage\mxgw-171-20261112T1950\` (ACL: SYSTEM + Administrators only, inheritance off): +`backup\Server-appsettings.json` (+ `Server-appsettings.Development.json`, `root-appsettings.json`), +`backup\MxAccessGw-servicekey-20261112T1950.reg`, `backup\AppEnvironmentExtra-pre.txt`, run log `_NOTES-20261112T1950.txt`. +On-box note: `E:\ApiInstall\MxGateway\_DEPLOY-NOTES-20261112T1950-issue171-pepper-to-env.txt`. + +Rollback: stop `MxAccessGw`, copy `backup\Server-appsettings.json` over `Server\appsettings.json`, restore +`AppEnvironmentExtra` from `backup\AppEnvironmentExtra-pre.txt` (or import the `.reg`), start. + +## For the next upgrade + +`Upgrade-MxGatewayService.ps1` already treats the env block as authoritative and copies the live +`appsettings*.json` back byte-identical, so nothing changes in the procedure. Expect an env count of **7** +on vd03 now. Never rewrite the block with `nssm set AppEnvironmentExtra` without re-including +`MxGateway__ApiKeyPepper`. diff --git a/docs/runbooks/DeployUpgradeNssmService.md b/docs/runbooks/DeployUpgradeNssmService.md index 445d889e..46006c27 100644 --- a/docs/runbooks/DeployUpgradeNssmService.md +++ b/docs/runbooks/DeployUpgradeNssmService.md @@ -117,7 +117,7 @@ validator therefore rejects stores inside the content root (see the 2026-08-09 i | State | Setting | Default location (Windows) | Notes | |---|---|---|---| | API-key store (SQLite) | `MxGateway:Authentication:SqlitePath` (NSSM: `MxGateway__Authentication__SqlitePath`) | `C:\ProgramData\MxGateway\gateway-auth.db` | Must be rooted; validator also rejects a path inside the content root. Back up the `-wal`/`-shm` siblings with it. | -| API-key **pepper** | secret `MxGateway:ApiKeyPepper` (`PepperSecretName`); lives outside the DB, in NSSM env / Secrets store | n/a | Without the same pepper every issued `mxgw_...` token fails verification. Never in the DB (`docs/Authentication.md`). Back it up in the org vault. | +| API-key **pepper** | secret `MxGateway:ApiKeyPepper` (`PepperSecretName`); lives outside the DB, in NSSM env / Secrets store | n/a | Without the same pepper every issued `mxgw_...` token fails verification. Never in the DB (`docs/Authentication.md`). Back it up in the org vault. On the wonder boxes it is the `MxGateway__ApiKeyPepper` env entry: QA vq03/vq04 since 2026-08-14, DEV vd03 since 2026-11-12 (moved out of `Server\appsettings.json`, mxaccessgw#171, `docs/deploy/2026-11-12-vd03-pepper-to-service-env.md`). | | Secrets store (SQLite) | `Secrets:SqlitePath` (NSSM: `Secrets__SqlitePath`) | `C:\ProgramData\MxGateway\mxgateway-secrets.db` | **Must be absolute AND outside the content root**; fail-closed with no opt-out (`docs/GatewayConfiguration.md` L349; `scadaproj` Secrets notes). | | Secrets master key (KEK) | `ZB_SECRETS_MASTER_KEY` in NSSM `AppEnvironmentExtra` | n/a | Losing it makes the store unrecoverable; no `RewrapAll` yet (`docs/GatewayConfiguration.md`, "Rotation / DR"). Back it up in the vault, not just on the box. | | Galaxy snapshot cache | `MxGateway:Galaxy:SnapshotCachePath` | `C:\ProgramData\MxGateway\galaxy-snapshot.json` | Cache only; rejected inside content root so it survives deploys. Losing it is harmless. |