dohertj2/lmxopcua
1
0
Fork 0
Code Issues 5 Pull Requests Actions Packages Projects Releases Wiki Activity

ScriptWorker isolation: InstallRootDeny does not cover sibling pki\ and data\ on the QA layout (Users-readable) #890

New Issue
Closed
opened 2026-11-12 17:02:21 -05:00 by dohertj2 · 2 comments
dohertj2 commented 2026-11-12 17:02:21 -05:00
Owner
Copy Link
Copy Source

Found during the QA SDK 2 deploy on 2026-11-12 (#794, #780). Record: docs/deploy/qa-upgrade-b8b4c18f-to-fea7de0a.md.

Install-ScriptWorkerIsolation.ps1 denies the worker identity ReadData only on -InstallRoot (the folder with OtOpcUa.Host.exe). On the QA layout, the node's secrets live in siblings of that folder, not inside it:

  • E:\ApiInstall\OtOpcUa\pki, which includes own\private\*.pfx, the application certificate's private key;
  • E:\ApiInstall\OtOpcUa\data, which holds secrets.db and the LocalDb store.

Both folders inherit BUILTIN\Users:(RX) from E:\. The worker runs under a restricted token, and an access must pass for the host account (LocalSystem) and the restricting set, which is identity + Everyone + Users + RESTRICTED. A folder that grants Users read is therefore probably readable by the worker. Only the directory ACLs were inspected; the file-level ACLs (the SDK may write the pfx with its own DACL) were not checked.

The analyzer's System.IO bans are still the first line of defence; this is about the defence-in-depth layer.

Options:

  1. The installer takes extra -DenyPaths (or derives them from OpcUa:PkiStoreRoot, LocalDb:Path and Secrets:SqlitePath) and applies the same inherited deny.
  2. Document that PKI and stores must sit under InstallRoot or be ACL'd without Users.
  3. Verify on a QA node, once a script exists there, whether a worker can actually open the pfx (a one-off check under the real token).

Not urgent: QA has no scripts today, so no worker runs.

Found during the QA SDK 2 deploy on 2026-11-12 (#794, #780). Record: `docs/deploy/qa-upgrade-b8b4c18f-to-fea7de0a.md`. `Install-ScriptWorkerIsolation.ps1` denies the worker identity ReadData only on `-InstallRoot` (the folder with `OtOpcUa.Host.exe`). On the QA layout, the node's secrets live in **siblings** of that folder, not inside it: - `E:\ApiInstall\OtOpcUa\pki`, which includes `own\private\*.pfx`, the application certificate's private key; - `E:\ApiInstall\OtOpcUa\data`, which holds `secrets.db` and the LocalDb store. Both folders inherit `BUILTIN\Users:(RX)` from `E:\`. The worker runs under a restricted token, and an access must pass for the host account (LocalSystem) **and** the restricting set, which is identity + Everyone + **Users** + RESTRICTED. A folder that grants Users read is therefore probably readable by the worker. Only the directory ACLs were inspected; the file-level ACLs (the SDK may write the pfx with its own DACL) were not checked. The analyzer's System.IO bans are still the first line of defence; this is about the defence-in-depth layer. Options: 1. The installer takes extra `-DenyPaths` (or derives them from `OpcUa:PkiStoreRoot`, `LocalDb:Path` and `Secrets:SqlitePath`) and applies the same inherited deny. 2. Document that PKI and stores must sit under InstallRoot or be ACL'd without Users. 3. Verify on a QA node, once a script exists there, whether a worker can actually open the pfx (a one-off check under the real token). Not urgent: QA has no scripts today, so no worker runs.
dohertj2 added the priority:p3type:followup labels 2026-11-12 17:02:54 -05:00
dohertj2 commented 2026-11-12 17:42:32 -05:00
Author
Owner
Copy Link
Copy Source

The fix is on master as f75ee7d0. The QA re-apply has NOT been done, so this issue stays open and blocked.

What changed

  • -DenyPaths <dir>,… on Install-ScriptWorkerIsolation.ps1.
    • Each listed folder gets the same inherited deny for NT SERVICE\OtOpcUaScriptWorker that the install root already gets (new SensitiveDirDeny steps).
    • It refuses drive and share roots, script-worker\ and the PE cache. Every listed folder must exist; this is checked before anything changes.
    • A single a,b,c string works, which is how powershell -File passes an array.
  • The deny is now read and write, mask 0xD0157. icacls shows it as (OI)(CI)(DENY)(W,D,WDAC,WO,RD,DC).
    • Traverse/Execute, ReadAttributes, ReadControl and Synchronize stay allowed, so the loader can still reach script-worker\.
    • The write half matters because a data drive lets Users create files in subfolders. Without it, a worker could plant a certificate in pki\trusted\certs.
    • This also tightens InstallRootDeny, which was 0x1.
  • The list is recorded in HKLM\SYSTEM\CurrentControlSet\Services\OtOpcUaScriptWorker\Parameters, value DenyPaths (step DenyPathRecord).
    • A re-run removes the deny from any folder dropped from the list.
    • -Uninstall removes exactly the recorded denies, so you don't need to pass -DenyPaths again.
    • -WhatIf and -PlanOnly show every step.
  • Coverage warnings after an apply:
    • A WARN for each item under a denied folder that has a protected DACL and no deny of its own. The inherited deny cannot reach these, e.g. a pfx the SDK wrote with its own ACL.
    • Without -DenyPaths, a WARN for any pki/data/logs/backup/config/secrets folder beside InstallRoot.
  • The host account is never named in a deny ACE.
  • Docs updated: docs/ServiceHosting.md, design §12.5, and the QA record docs/deploy/qa-upgrade-b8b4c18f-to-fea7de0a.md.

Tests

  • Dry-run harness (Test-ScriptWorkerIsolation.ps1, pwsh on macOS): 86 passed, up from 46.
    • Covers the QA-layout plan, the mask bits, the refusals, stale removal, rollback driven by the record, and the host account never being denied.
    • Six mutations each turn it red: no sibling deny, deny→allow, mask back to ReadData, no stale removal, rollback ignoring the record, script-worker\ not refused.
  • Test-ScriptWorkerIsolation.Windows.ps1, Windows PowerShell 5.1 on windev, run for real in a scratch folder: 34/34.
    • Apply with -DenyPaths: 11/11 changed. Re-apply: 11/11 unchanged.
    • Re-apply with a shorter list: only the stale deny and the record change.
    • Uninstall with no -DenyPaths (driven by the record): 10 changed. Second uninstall: 7/7 unchanged.
  • New InstallerLayoutIsolationTests (ScriptWorker.Tests; needs OTOPCUA_ISOLATION_INSTALLER_E2E=1 and an elevated Windows session), on windev:
    • It builds a scratch wonder layout, C:\otopcua-iso-e2e-*\{app\script-worker (the real built worker), pki\own\private\*.pfx, pki\trusted\certs, data\secrets.db, logs, backup}, with a data-drive ACL: Users RX plus create-files in subfolders.
    • Under the real restricted worker token, before the install and again after the uninstall, the token reads every file and plants a trust-list file. This is the positive control.
    • With the isolation installed (Windows PowerShell 5.1, -DenyPaths root,pki,data):
      • the token cannot read pki, data, logs, backup or app\appsettings.json, and cannot plant the certificate;
      • it can still read script-worker\;
      • a real worker launched from the layout under the identity runs a script (41+1 = 42);
      • the host still reads everything;
      • the coverage audit names a protected test file.
    • Mutation: with the -DenyPaths denies set to a no-op mask, which is the old app-only coverage, the test goes red at pki: the worker token must not read …\pki\own\private\otopcua.pfx. That reproduces this issue.
    • The test cleans up after itself. windev was left clean: no scratch service, firewall rule or folder, and MxAccessGw still running.
  • ScriptWorker suite:
    • macOS, Release: 188 passed / 10 skipped.
    • windev, Release, with the E2E gate on: 192 passed / 6 skipped / 0 failed.

QA re-apply (not done; a later deploy step, on both vq03 and vq04)

Run from the staged scripts\install at f75ee7d0 or later, in an elevated Windows PowerShell 5.1 session. The services can stay up: only folder ACLs change for the worker identity, and no worker runs on QA. First check that Secrets__SqlitePath, its key file and LocalDb__Path in the service environment all sit under E:\ApiInstall\OtOpcUa; add any folder outside it to $deny.

$deny = 'E:\ApiInstall\OtOpcUa','E:\ApiInstall\OtOpcUa\pki','E:\ApiInstall\OtOpcUa\data'
.\Install-ScriptWorkerIsolation.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -HostServiceAccount LocalSystem -ConfigureHostService -DenyPaths $deny -WhatIf
.\Install-ScriptWorkerIsolation.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -HostServiceAccount LocalSystem -ConfigureHostService -DenyPaths $deny
.\Install-ScriptWorkerIsolation.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -HostServiceAccount LocalSystem -ConfigureHostService -DenyPaths $deny

Expected:

  • First apply: 11 steps, 5 changed: DenyPathRecord, InstallRootDeny (mask 0x1 → 0xD0157) and the 3 SensitiveDirDeny steps.
  • Second apply: 11/11 unchanged.
  • State checks:
    • icacls on E:\ApiInstall\OtOpcUa, …\pki, …\data and …\app shows NT SERVICE\OtOpcUaScriptWorker:(OI)(CI)(DENY)(W,D,WDAC,WO,RD,DC).
    • reg query HKLM\SYSTEM\CurrentControlSet\Services\OtOpcUaScriptWorker\Parameters /v DenyPaths lists the 3 folders.
    • Check every coverage WARN.

Rollback is unchanged: … -ConfigureHostService -Uninstall. It now also removes the recorded denies.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi

**The fix is on master as `f75ee7d0`. The QA re-apply has NOT been done, so this issue stays open and `blocked`.** ## What changed - **`-DenyPaths <dir>,…`** on `Install-ScriptWorkerIsolation.ps1`. - Each listed folder gets the same inherited deny for `NT SERVICE\OtOpcUaScriptWorker` that the install root already gets (new `SensitiveDirDeny` steps). - It refuses drive and share roots, `script-worker\` and the PE cache. Every listed folder must exist; this is checked before anything changes. - A single `a,b,c` string works, which is how `powershell -File` passes an array. - **The deny is now read *and* write**, mask `0xD0157`. `icacls` shows it as `(OI)(CI)(DENY)(W,D,WDAC,WO,RD,DC)`. - Traverse/Execute, ReadAttributes, ReadControl and Synchronize stay allowed, so the loader can still reach `script-worker\`. - The write half matters because a data drive lets Users create files in subfolders. Without it, a worker could plant a certificate in `pki\trusted\certs`. - This also tightens `InstallRootDeny`, which was `0x1`. - **The list is recorded** in `HKLM\SYSTEM\CurrentControlSet\Services\OtOpcUaScriptWorker\Parameters`, value `DenyPaths` (step `DenyPathRecord`). - A re-run removes the deny from any folder dropped from the list. - `-Uninstall` removes exactly the recorded denies, so you don't need to pass `-DenyPaths` again. - `-WhatIf` and `-PlanOnly` show every step. - **Coverage warnings after an apply:** - A WARN for each item under a denied folder that has a protected DACL and no deny of its own. The inherited deny cannot reach these, e.g. a pfx the SDK wrote with its own ACL. - Without `-DenyPaths`, a WARN for any `pki`/`data`/`logs`/`backup`/`config`/`secrets` folder beside InstallRoot. - The host account is never named in a deny ACE. - Docs updated: `docs/ServiceHosting.md`, design §12.5, and the QA record `docs/deploy/qa-upgrade-b8b4c18f-to-fea7de0a.md`. ## Tests - **Dry-run harness** (`Test-ScriptWorkerIsolation.ps1`, pwsh on macOS): **86 passed**, up from 46. - Covers the QA-layout plan, the mask bits, the refusals, stale removal, rollback driven by the record, and the host account never being denied. - Six mutations each turn it red: no sibling deny, deny→allow, mask back to ReadData, no stale removal, rollback ignoring the record, `script-worker\` not refused. - **`Test-ScriptWorkerIsolation.Windows.ps1`**, Windows PowerShell 5.1 on windev, run for real in a scratch folder: **34/34**. - Apply with `-DenyPaths`: 11/11 changed. Re-apply: 11/11 `unchanged`. - Re-apply with a shorter list: only the stale deny and the record change. - Uninstall with no `-DenyPaths` (driven by the record): 10 changed. Second uninstall: 7/7 `unchanged`. - **New `InstallerLayoutIsolationTests`** (ScriptWorker.Tests; needs `OTOPCUA_ISOLATION_INSTALLER_E2E=1` and an elevated Windows session), on windev: - It builds a scratch wonder layout, `C:\otopcua-iso-e2e-*\{app\script-worker (the real built worker), pki\own\private\*.pfx, pki\trusted\certs, data\secrets.db, logs, backup}`, with a data-drive ACL: Users RX plus create-files in subfolders. - Under the **real restricted worker token**, before the install and again after the uninstall, the token reads every file and plants a trust-list file. This is the positive control. - With the isolation installed (Windows PowerShell 5.1, `-DenyPaths root,pki,data`): - the token **cannot** read pki, data, logs, backup or `app\appsettings.json`, and **cannot** plant the certificate; - it **can** still read `script-worker\`; - a **real worker launched from the layout under the identity runs a script** (41+1 = 42); - the host still reads everything; - the coverage audit names a protected test file. - **Mutation:** with the `-DenyPaths` denies set to a no-op mask, which is the old app-only coverage, the test goes red at `pki: the worker token must not read …\pki\own\private\otopcua.pfx`. That reproduces this issue. - The test cleans up after itself. windev was left clean: no scratch service, firewall rule or folder, and MxAccessGw still running. - **ScriptWorker suite:** - macOS, Release: 188 passed / 10 skipped. - windev, Release, with the E2E gate on: **192 passed / 6 skipped / 0 failed**. ## QA re-apply (not done; a later deploy step, on both vq03 and vq04) Run from the staged `scripts\install` at `f75ee7d0` or later, in an elevated Windows PowerShell 5.1 session. The services can stay up: only folder ACLs change for the worker identity, and no worker runs on QA. First check that `Secrets__SqlitePath`, its key file and `LocalDb__Path` in the service environment all sit under `E:\ApiInstall\OtOpcUa`; add any folder outside it to `$deny`. ```powershell $deny = 'E:\ApiInstall\OtOpcUa','E:\ApiInstall\OtOpcUa\pki','E:\ApiInstall\OtOpcUa\data' .\Install-ScriptWorkerIsolation.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -HostServiceAccount LocalSystem -ConfigureHostService -DenyPaths $deny -WhatIf .\Install-ScriptWorkerIsolation.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -HostServiceAccount LocalSystem -ConfigureHostService -DenyPaths $deny .\Install-ScriptWorkerIsolation.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -HostServiceAccount LocalSystem -ConfigureHostService -DenyPaths $deny ``` **Expected:** - **First apply:** 11 steps, 5 `changed`: `DenyPathRecord`, `InstallRootDeny` (mask 0x1 → 0xD0157) and the 3 `SensitiveDirDeny` steps. - **Second apply:** 11/11 `unchanged`. - **State checks:** - `icacls` on `E:\ApiInstall\OtOpcUa`, `…\pki`, `…\data` and `…\app` shows `NT SERVICE\OtOpcUaScriptWorker:(OI)(CI)(DENY)(W,D,WDAC,WO,RD,DC)`. - `reg query HKLM\SYSTEM\CurrentControlSet\Services\OtOpcUaScriptWorker\Parameters /v DenyPaths` lists the 3 folders. - Check every coverage WARN. Rollback is unchanged: `… -ConfigureHostService -Uninstall`. It now also removes the recorded denies. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
dohertj2 added the blocked label 2026-11-12 17:42:41 -05:00
dohertj2 commented 2026-11-12 19:10:11 -05:00
Author
Owner
Copy Link
Copy Source

Re-applied on QA vq03 and vq04, 2026-11-12, and verified against the live worker. Closing. Record: docs/deploy/qa-upgrade-b8b4c18f-to-fea7de0a.md, section "Canary, live isolation check and runaway drill" (a17ffb72).

  • Path check first. Secrets__SqlitePath and LocalDb__Path are under E:\ApiInstall\OtOpcUa\data. Secrets__MasterKey__FilePath is C:\ProgramData\ZB\kek\otopcua.kek, outside the install root, so C:\ProgramData\ZB\kek was added to the list. That folder was already SYSTEM + Administrators only.
  • Run: -DenyPaths E:\ApiInstall\OtOpcUa, …\pki, …\data, C:\ProgramData\ZB\kek. -WhatIf showed 12 operations. The first apply changed 7 of 12 steps, and the second apply reported 12/12 unchanged on both nodes.
    • The extra changed beyond what the record expected is HostServiceEnv, and it is a reorder only. The P3 flip had appended the Scripting__Runner* entries after the two isolation entries, and the step rewrites the list with its own two last. The values are the same and the env count is unchanged. This is harmless, but the step treats a reorder as a change.
  • Coverage. vq03: 0 protected items missed. vq04: WARNs for historiangw.kek and mxgateway.kek (protected DACLs). The live probe shows the worker cannot read them.
  • State on both nodes:
    • icacls shows NT SERVICE\OtOpcUaScriptWorker:(OI)(CI)(DENY)(W,D,WDAC,WO,RD,DC) on the root, pki, data, app and the KEK folder.
    • app\script-worker keeps its explicit RX.
    • The DenyPaths value lists the 4 folders.
  • Live proof (option 3 of this issue). The live worker's own token was impersonated on each node (restricted; restricting SIDs identity/Everyone/Users/RESTRICTED; SeChangeNotify only).
    • DENIED reads: pki\own\private\*.pfx, pki\trusted\certs\*.der, data\secrets.db, data\otopcua-localdb.db, app\appsettings.json, the logs, the app.bak copy, all three .kek files and the PE-cache key.
    • DENIED writes: creating a file in pki\trusted\certs, data, app and script-worker.
    • Positive controls: the worker dll and kernel32 read OK under the same token, and every denied file read OK for the admin session without impersonation.
  • /health/ready was 200 on both nodes after the run, with no restart.

Residual (new issue #891): the same token can still read Users-readable files that belong to other apps or to deploy staging: the OtOpcUaHost .reg export under E:\psrem-stage and the ScadaBridge/HistorianGateway appsettings.json. Those were left alone because they are outside OtOpcUa's folders.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi

**Re-applied on QA vq03 and vq04, 2026-11-12, and verified against the live worker. Closing.** Record: `docs/deploy/qa-upgrade-b8b4c18f-to-fea7de0a.md`, section "Canary, live isolation check and runaway drill" (`a17ffb72`). - **Path check first.** `Secrets__SqlitePath` and `LocalDb__Path` are under `E:\ApiInstall\OtOpcUa\data`. **`Secrets__MasterKey__FilePath` is `C:\ProgramData\ZB\kek\otopcua.kek`, outside the install root**, so `C:\ProgramData\ZB\kek` was added to the list. That folder was already SYSTEM + Administrators only. - **Run:** `-DenyPaths E:\ApiInstall\OtOpcUa, …\pki, …\data, C:\ProgramData\ZB\kek`. `-WhatIf` showed 12 operations. The first apply changed 7 of 12 steps, and **the second apply reported 12/12 `unchanged`** on both nodes. - The extra `changed` beyond what the record expected is `HostServiceEnv`, and it is a reorder only. The P3 flip had appended the `Scripting__Runner*` entries after the two isolation entries, and the step rewrites the list with its own two last. The values are the same and the env count is unchanged. This is harmless, but the step treats a reorder as a change. - **Coverage.** vq03: 0 protected items missed. vq04: WARNs for `historiangw.kek` and `mxgateway.kek` (protected DACLs). The live probe shows the worker cannot read them. - **State on both nodes:** - `icacls` shows `NT SERVICE\OtOpcUaScriptWorker:(OI)(CI)(DENY)(W,D,WDAC,WO,RD,DC)` on the root, `pki`, `data`, `app` and the KEK folder. - `app\script-worker` keeps its explicit RX. - The `DenyPaths` value lists the 4 folders. - **Live proof (option 3 of this issue).** The live worker's own token was impersonated on each node (restricted; restricting SIDs identity/Everyone/Users/RESTRICTED; SeChangeNotify only). - **DENIED reads:** `pki\own\private\*.pfx`, `pki\trusted\certs\*.der`, `data\secrets.db`, `data\otopcua-localdb.db`, `app\appsettings.json`, the logs, the `app.bak` copy, all three `.kek` files and the PE-cache key. - **DENIED writes:** creating a file in `pki\trusted\certs`, `data`, `app` and `script-worker`. - **Positive controls:** the worker dll and kernel32 read OK under the same token, and every denied file read OK for the admin session without impersonation. - `/health/ready` was 200 on both nodes after the run, with no restart. **Residual (new issue #891):** the same token can still read Users-readable files that belong to other apps or to deploy staging: the OtOpcUaHost `.reg` export under `E:\psrem-stage` and the ScadaBridge/HistorianGateway `appsettings.json`. Those were left alone because they are outside OtOpcUa's folders. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
dohertj2 removed the blocked label 2026-11-12 19:10:17 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
auto-managed
queue management
 blocked
Blocked by dependency or external hold
 cross-driver
queue management
 cross-repo
Needs coordinated change in another family repo (wire contract etc.)
 driver/abcip
queue management
 driver/ablegacy
queue management
 driver/focas
queue management
 driver/mqtt
MQTT / Sparkplug B broker-subscribe driver
 driver/opcuaclient
queue management
 driver/s7
queue management
 driver/twincat
queue management
 migrated
Migrated from a markdown backlog (2026-11 standardization)
 phase/1
queue management
 phase/2
queue management
 phase/3
queue management
 phase/4
queue management
 phase/5
queue management
 phase/6
queue management
 priority:p0
Must-have / blocking
 priority:p1
High priority
 priority:p2
Normal priority
 priority:p3
Low priority / nice to have
 queue/blocked
queue management
 queue/done
queue management
 queue/failed
queue management
 queue/in-progress
queue management
 queue/queued
queue management
 type:bug
Defect in shipped behaviour
 type:debt
Technical debt / hygiene
 type:deploy-gate
Rollout/deploy/migration step not yet done on an environment
 type:docs
Documentation work
 type:feature
Feature implementation
 type:followup
Follow-up deferred from a plan, PR or gate
 type:infra
Infrastructure and scaffolding
 type:review-finding
Unfixed finding from an architecture/code review
 type:test
Test work
No Label priority:p3 type:followup
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
dohertj2 (dohertj2)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dohertj2/lmxopcua#890
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?