ScriptWorker token can still read other apps' appsettings and deploy-staging .reg exports on the wonder layout (Users-readable) #891

New Issue

2026-11-12T19:08:23-05:00

dohertj2 commented

2026-11-12 19:08:23 -05:00

Found on QA vq03/vq04 on 2026-11-12, during the live #794 step 5 check after the #890 re-apply. Record: docs/deploy/qa-upgrade-b8b4c18f-to-fea7de0a.md, section "Canary, live isolation check and runaway drill".

The live ScriptWorker token (restricted; restricting SIDs: the worker identity, Everyone, Users, RESTRICTED) was impersonated on the box. The #890 denies hold: everything under E:\ApiInstall\OtOpcUa and C:\ProgramData\ZB\kek is DENIED. But the token can still read these Users-readable files outside OtOpcUa's folders:

E:\psrem-stage\otopcua-fea7de0a\backup\OtOpcUaHost-pre-fea7de0a.reg, a registry export of the OtOpcUaHost service environment. It holds the ConfigDb connection string, the API-key pepper, the LocalDb replication key and the JWT signing key. Older stage folders on the box probably hold the same kind of export.
E:\ApiInstall\ScadaBridge\app\appsettings.json and E:\ApiInstall\HistorianGateway\app\appsettings.json (plus their app.bak.* copies). These inherit BUILTIN\Users:(RX) from E:\.

C:\ProgramData\ZB\qa-build\* was denied (protected ACL).

This is defence in depth. The script analyzer's System.IO bans are still the first line, and no script can name a path today. No deny was added in that run, because these folders belong to other apps (the owner asked for ScadaBridge, mxgw and HistorianGateway on QA to be left alone) or to deploy staging.

Options:

Deploy procedure: write registry exports and appsettings backups to a folder without Users access, e.g. E:\psrem-stage\…\backup created with a protected ACL (SYSTEM + Administrators). Then clean old stage folders.
Add E:\psrem-stage and/or E:\ApiInstall to the installer's -DenyPaths on the wonder boxes. The script-worker\ explicit RX already wins over an inherited deny, so this works. It does, however, change the ACLs in the other apps' trees, so it needs the owner's approval.
Remove BUILTIN\Users read from E:\ApiInstall at the box level. This is a box hardening item that affects every app.

Option 1 is recommended, with option 2 added once the owner agrees.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi

Found on QA vq03/vq04 on 2026-11-12, during the live #794 step 5 check after the #890 re-apply. Record: `docs/deploy/qa-upgrade-b8b4c18f-to-fea7de0a.md`, section "Canary, live isolation check and runaway drill". The live ScriptWorker token (restricted; restricting SIDs: the worker identity, Everyone, Users, RESTRICTED) was impersonated on the box. The #890 denies hold: everything under `E:\ApiInstall\OtOpcUa` and `C:\ProgramData\ZB\kek` is DENIED. But the token **can still read** these Users-readable files outside OtOpcUa's folders: - `E:\psrem-stage\otopcua-fea7de0a\backup\OtOpcUaHost-pre-fea7de0a.reg`, a registry export of the OtOpcUaHost service environment. It holds the ConfigDb connection string, the API-key pepper, the LocalDb replication key and the JWT signing key. Older stage folders on the box probably hold the same kind of export. - `E:\ApiInstall\ScadaBridge\app\appsettings.json` and `E:\ApiInstall\HistorianGateway\app\appsettings.json` (plus their `app.bak.*` copies). These inherit `BUILTIN\Users:(RX)` from `E:\`. `C:\ProgramData\ZB\qa-build\*` was denied (protected ACL). This is defence in depth. The script analyzer's System.IO bans are still the first line, and no script can name a path today. No deny was added in that run, because these folders belong to other apps (the owner asked for ScadaBridge, mxgw and HistorianGateway on QA to be left alone) or to deploy staging. Options: 1. Deploy procedure: write registry exports and appsettings backups to a folder without Users access, e.g. `E:\psrem-stage\…\backup` created with a protected ACL (SYSTEM + Administrators). Then clean old stage folders. 2. Add `E:\psrem-stage` and/or `E:\ApiInstall` to the installer's `-DenyPaths` on the wonder boxes. The `script-worker\` explicit RX already wins over an inherited deny, so this works. It does, however, change the ACLs in the other apps' trees, so it needs the owner's approval. 3. Remove `BUILTIN\Users` read from `E:\ApiInstall` at the box level. This is a box hardening item that affects every app. Option 1 is recommended, with option 2 added once the owner agrees. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi

dohertj2 added the priority:p3 type:followup labels 2026-11-12 19:08:42 -05:00

dohertj2 referenced this issue from a commit

2026-11-12 19:08:56 -05:00

docs(deploy): QA canary, #890 re-apply, live #794 worker check and #780 runaway drill on vq03/vq04

dohertj2 referenced this issue

2026-11-12 19:10:11 -05:00

ScriptWorker isolation: InstallRootDeny does not cover sibling pki\ and data\ on the QA layout (Users-readable) #890

dohertj2 referenced this issue

2026-11-12 19:10:11 -05:00

Deploy gate: ScriptWorker isolation installer (identity, pipe ACL, firewall, PeCacheDir ACL) on QA (lmxopcua#780) #794

dohertj2 referenced this issue

2026-11-12 19:10:12 -05:00

Implement the out-of-process script runner (design #750) — code on master; QA deploy (#793/#794) + P3 flip remain (no shadow week) #780

dohertj2 referenced this issue from a commit

2026-11-12 19:26:22 -05:00

docs(#891): QA box-wide worker denies + deploy-backup lockdown applied on vq03/vq04

dohertj2 commented

2026-11-12 19:26:34 -05:00

Done on QA vq03 + vq04 (owner-approved 2026-11-12): options 1 and 2. Record in docs/deploy/qa-upgrade-b8b4c18f-to-fea7de0a.md, section "Box-wide worker denies and backup lockdown" (f47891ff).

Registry exports locked down. On each box, all 6 deploy-backup .reg files under E:\psrem-stage (HG, mxgw, two OtOpcUa stages) now have a protected DACL allowing only SYSTEM and Administrators. Nothing was deleted, so the rollback copies remain.
Denies added. -DenyPaths now covers 11 folders. These are the 4 OtOpcUa/kek folders from #890 plus:
- E:\ApiInstall\ScadaBridge, HistorianGateway, MxAccessGateway, Overview, btdbmanager;
- E:\psrem-stage and E:\psrem.
The sequence was -WhatIf, apply, then a re-apply with everything unchanged. icacls shows the inherited (DENY)(W,D,WDAC,WO,RD,DC) ACE for NT SERVICE\OtOpcUaScriptWorker on all 11.
Installer warnings checked. They name the locked .reg files, plus historiangw.kek and mxgateway.kek on vq04. All are protected DACLs with only SYSTEM and Administrators, which a restricted worker token cannot satisfy.
No restarts. /health/ready returns 200 for OtOpcUa, Overview, ScadaBridge and mxgw on both nodes.
Procedure. docs/ServiceHosting.md now gives the box-wide list and says to keep deploy backups in a SYSTEM + Administrators-only folder. vd03 needs the same denies when its runner flip is scheduled (it has no worker isolation installed yet).

Closing.

**Done on QA vq03 + vq04 (owner-approved 2026-11-12): options 1 and 2.** Record in `docs/deploy/qa-upgrade-b8b4c18f-to-fea7de0a.md`, section "Box-wide worker denies and backup lockdown" (`f47891ff`). - **Registry exports locked down.** On each box, all 6 deploy-backup `.reg` files under `E:\psrem-stage` (HG, mxgw, two OtOpcUa stages) now have a protected DACL allowing only SYSTEM and Administrators. Nothing was deleted, so the rollback copies remain. - **Denies added.** `-DenyPaths` now covers 11 folders. These are the 4 OtOpcUa/kek folders from #890 plus: - `E:\ApiInstall\ScadaBridge`, `HistorianGateway`, `MxAccessGateway`, `Overview`, `btdbmanager`; - `E:\psrem-stage` and `E:\psrem`. The sequence was `-WhatIf`, apply, then a re-apply with everything `unchanged`. `icacls` shows the inherited `(DENY)(W,D,WDAC,WO,RD,DC)` ACE for `NT SERVICE\OtOpcUaScriptWorker` on all 11. - **Installer warnings checked.** They name the locked `.reg` files, plus `historiangw.kek` and `mxgateway.kek` on vq04. All are protected DACLs with only SYSTEM and Administrators, which a restricted worker token cannot satisfy. - **No restarts.** `/health/ready` returns 200 for OtOpcUa, Overview, ScadaBridge and mxgw on both nodes. - **Procedure.** `docs/ServiceHosting.md` now gives the box-wide list and says to keep deploy backups in a SYSTEM + Administrators-only folder. **vd03 needs the same denies when its runner flip is scheduled** (it has no worker isolation installed yet). Closing.

dohertj2 closed this issue

2026-11-12 19:26:34 -05:00

dohertj2 referenced this issue from a commit

2026-11-12 19:37:08 -05:00

docs(#891): vd03 worker isolation + box-wide ApiInstall deny + backup lockdown

dohertj2 commented

2026-11-12 19:37:17 -05:00

vd03 (DEV) done as well, owner-requested. Record: docs/deploy/vd03-upgrade-b8b4c18f-to-master.md, section "Worker isolation and box-wide denies" (aaaa55e2).

Worker isolation installed, installer from master f47891ff. Sequence: -WhatIf, apply, then a re-apply that reported every step unchanged. vd03 has secret copies throughout E:\ApiInstall (env/registry backups, kek.key files, old pki copies in about a dozen OtOpcUa.bak* folders), so the deny covers all of E:\ApiInstall, plus OtOpcUa\pki, OtOpcUa-data, C:\ProgramData\ZB\kek and E:\psrem-stage. script-worker\ keeps its explicit read+execute.
-ConfigureHostService not used. The vd03 runner is still InProcess, so there was no env change and no restart. When the vd03 runner flip happens, re-run with the same -DenyPaths plus -ConfigureHostService.
39 backup files (*.reg, *envbak*.txt, *envnames*.txt) are now restricted to SYSTEM and Administrators. None were deleted.
Installer warnings checked. The four kek.key files allow only SYSTEM and Administrators.
/health/ready returns 200 for OtOpcUa, Overview, ScadaBridge, mxgw and HG.

**vd03 (DEV) done as well**, owner-requested. Record: `docs/deploy/vd03-upgrade-b8b4c18f-to-master.md`, section "Worker isolation and box-wide denies" (`aaaa55e2`). - **Worker isolation installed**, installer from master `f47891ff`. Sequence: `-WhatIf`, apply, then a re-apply that reported every step unchanged. vd03 has secret copies throughout `E:\ApiInstall` (env/registry backups, `kek.key` files, old `pki` copies in about a dozen `OtOpcUa.bak*` folders), so the deny covers **all of `E:\ApiInstall`**, plus `OtOpcUa\pki`, `OtOpcUa-data`, `C:\ProgramData\ZB\kek` and `E:\psrem-stage`. `script-worker\` keeps its explicit read+execute. - **`-ConfigureHostService` not used.** The vd03 runner is still `InProcess`, so there was no env change and no restart. When the vd03 runner flip happens, re-run with the same `-DenyPaths` plus `-ConfigureHostService`. - **39 backup files** (`*.reg`, `*envbak*.txt`, `*envnames*.txt`) are now restricted to SYSTEM and Administrators. None were deleted. - **Installer warnings checked.** The four `kek.key` files allow only SYSTEM and Administrators. - `/health/ready` returns 200 for OtOpcUa, Overview, ScadaBridge, mxgw and HG.

dohertj2 referenced this issue

2026-11-13 02:06:37 -05:00

Upgrade folder swap drops the script-worker isolation ACLs; deploy must re-apply and verify them #901

dohertj2 referenced this issue

2026-11-13 03:16:15 -05:00

Implement the out-of-process script runner (design #750) — code on master; QA deploy (#793/#794) + P3 flip remain (no shadow week) #780

dohertj2 referenced this issue

2026-11-13 06:54:50 -05:00

Implement the out-of-process script runner (design #750) — code on master; QA deploy (#793/#794) + P3 flip remain (no shadow week) #780

Sign in to join this conversation.