Deploy gate: ScriptWorker isolation installer (identity, pipe ACL, firewall, PeCacheDir ACL) on QA (lmxopcua#780) #794

New Issue

2026-11-10T08:51:52-05:00

dohertj2 commented

2026-11-10 08:51:52 -05:00

Deploy gate for the lmxopcua#780 installer slice (2a8e2981). None of it has run on a real node: windev is a build box only, so no account, firewall rule or persistent ACL was created there. Run it on one QA node first (vq03), during or after the P2 shadow week, before the P3 OutOfProcess flip.

What is verified so far: the dry-run harness pwsh -File scripts/install/tests/Test-ScriptWorkerIsolation.ps1 (46 assertions), and WorkerIsolationTests on windev. Those cover the pipe ACL, the restricted token's access, and a real restricted worker running a script. The installer itself has not run on Windows. Rollback is Install-ScriptWorkerIsolation.ps1 … -ConfigureHostService -Uninstall, then restart the host (or just unset Scripting__Worker__Identity).

Steps on the QA node (an elevated session, as part of a user-requested deploy)

Deploy a Host build that contains 2a8e2981: a self-contained win-x64 publish, so that script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe exists.
Dry run: .\Install-ScriptWorkerIsolation.ps1 -InstallRoot <root> -HostServiceAccount <OtOpcUaHost account> -ConfigureHostService -WhatIf. Check that it prints 7 steps and changes nothing.
Apply without -WhatIf, then run it again. The second run must report every step unchanged (idempotency).
Verify the state:
- sc.exe qc OtOpcUaScriptWorker: start type DISABLED, SERVICE_START_NAME NT SERVICE\OtOpcUaScriptWorker. sc.exe showsid OtOpcUaScriptWorker should give S-1-5-80-1846135827-3903057317-1998105452-3938448374-3023753135.
- icacls <PeCacheDir>: protected; SYSTEM, Administrators and the host account full control; no Users. icacls <PeCacheDir>\pe adds NT SERVICE\OtOpcUaScriptWorker:(OI)(CI)(M).
- icacls <root>: NT SERVICE\OtOpcUaScriptWorker:(OI)(CI)(DENY)(RD). icacls <root>\script-worker: identity (OI)(CI)(RX).
- Get-NetFirewallRule -Name OtOpcUa-ScriptWorker-BlockOutbound | Get-NetFirewallApplicationFilter: the program is the worker exe; the rule is Outbound / Block / Any.
- The OtOpcUaHost service Environment contains Scripting__Worker__Identity and Scripting__Worker__PeCacheDir, and the other entries are intact.
With Scripting:Runner=Shadow (or OutOfProcess for Calculation only), restart the host and check:
- Workers start: otopcua_script_worker_up{shard} = 1 and script-runner is Healthy. No ScriptWorker[..] failed to start lines.
- The worker token is restricted. Process Explorer → worker → Security shows "Restricted" SIDs and only SeChangeNotifyPrivilege. Alternatively, IsTokenRestricted via a one-off script.
- The window-station grant works under a real service session, which windev (SSH) could not prove. Without it coreclr fails with ERROR_DLL_INIT_FAILED (0x80008089 worker exit).
- The PE cache fills: <PeCacheDir>\pe\*.pe appears, and on a restart otopcua_script_compile{cache="hit"} rises.
- Network block: no outbound connection from the worker exe in the firewall log (enable logging of dropped packets for the rule temporarily).
- Shadow mismatches stay at 0 with the identity on.
Rollback drill: run -Uninstall -ConfigureHostService and restart. Workers then run unrestricted, and a second uninstall reports unchanged. Re-apply afterwards if the shadow week continues.
Record the results on lmxopcua#780. Repeat on vq04, then on DEV vd03 only when the P3 flip is scheduled.

Watch-outs

A deploy that replaces the install folder (rename/swap) drops the folder ACEs, so re-run the installer. Copying files into the existing folder keeps them.
A framework-dependent worker (dotnet.exe) is not covered by the program-scoped firewall rule. The Windows installs are self-contained, so this should not apply.
If the host runs as LocalSystem, the PeCacheDir ACL collapses to SYSTEM + Administrators, which is expected.

Deploy gate for the lmxopcua#780 installer slice (`2a8e2981`). None of it has run on a real node: windev is a build box only, so no account, firewall rule or persistent ACL was created there. Run it on **one QA node first** (vq03), during or after the P2 shadow week, before the P3 `OutOfProcess` flip. What is verified so far: the dry-run harness `pwsh -File scripts/install/tests/Test-ScriptWorkerIsolation.ps1` (46 assertions), and `WorkerIsolationTests` on windev. Those cover the pipe ACL, the restricted token's access, and a real restricted worker running a script. The installer itself has not run on Windows. Rollback is `Install-ScriptWorkerIsolation.ps1 … -ConfigureHostService -Uninstall`, then restart the host (or just unset `Scripting__Worker__Identity`). ## Steps on the QA node (an elevated session, as part of a user-requested deploy) 1. Deploy a Host build that contains `2a8e2981`: a self-contained win-x64 publish, so that `script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe` exists. 2. Dry run: `.\Install-ScriptWorkerIsolation.ps1 -InstallRoot <root> -HostServiceAccount <OtOpcUaHost account> -ConfigureHostService -WhatIf`. Check that it prints 7 steps and changes nothing. 3. Apply without `-WhatIf`, then run it again. **The second run must report every step `unchanged`** (idempotency). 4. Verify the state: - `sc.exe qc OtOpcUaScriptWorker`: start type DISABLED, SERVICE_START_NAME `NT SERVICE\OtOpcUaScriptWorker`. `sc.exe showsid OtOpcUaScriptWorker` should give `S-1-5-80-1846135827-3903057317-1998105452-3938448374-3023753135`. - `icacls <PeCacheDir>`: protected; SYSTEM, Administrators and the host account full control; **no Users**. `icacls <PeCacheDir>\pe` adds `NT SERVICE\OtOpcUaScriptWorker:(OI)(CI)(M)`. - `icacls <root>`: `NT SERVICE\OtOpcUaScriptWorker:(OI)(CI)(DENY)(RD)`. `icacls <root>\script-worker`: identity `(OI)(CI)(RX)`. - `Get-NetFirewallRule -Name OtOpcUa-ScriptWorker-BlockOutbound | Get-NetFirewallApplicationFilter`: the program is the worker exe; the rule is Outbound / Block / Any. - The OtOpcUaHost service `Environment` contains `Scripting__Worker__Identity` and `Scripting__Worker__PeCacheDir`, and the other entries are intact. 5. With `Scripting:Runner=Shadow` (or `OutOfProcess` for Calculation only), restart the host and check: - Workers start: `otopcua_script_worker_up{shard}` = 1 and `script-runner` is Healthy. No `ScriptWorker[..] failed to start` lines. - The worker token is restricted. Process Explorer → worker → Security shows "Restricted" SIDs and only SeChangeNotifyPrivilege. Alternatively, `IsTokenRestricted` via a one-off script. - **The window-station grant works under a real service session**, which windev (SSH) could not prove. Without it coreclr fails with `ERROR_DLL_INIT_FAILED` (0x80008089 worker exit). - The PE cache fills: `<PeCacheDir>\pe\*.pe` appears, and on a restart `otopcua_script_compile{cache="hit"}` rises. - Network block: no outbound connection from the worker exe in the firewall log (enable logging of dropped packets for the rule temporarily). - Shadow mismatches stay at 0 with the identity on. 6. Rollback drill: run `-Uninstall -ConfigureHostService` and restart. Workers then run unrestricted, and a second uninstall reports `unchanged`. Re-apply afterwards if the shadow week continues. 7. Record the results on lmxopcua#780. Repeat on vq04, then on DEV vd03 only when the P3 flip is scheduled. ## Watch-outs - A deploy that **replaces** the install folder (rename/swap) drops the folder ACEs, so re-run the installer. Copying files into the existing folder keeps them. - A framework-dependent worker (`dotnet.exe`) is not covered by the program-scoped firewall rule. The Windows installs are self-contained, so this should not apply. - If the host runs as LocalSystem, the PeCacheDir ACL collapses to SYSTEM + Administrators, which is expected.

dohertj2 added the type:deploy-gate priority:p2 labels 2026-11-10 08:51:52 -05:00

dohertj2 referenced this issue

2026-11-10 08:52:06 -05:00

Implement the out-of-process script runner (design #750) — code on master; QA shadow week (#793, #794) + P3 flip remain #780

dohertj2 added the blocked label 2026-11-10 08:52:43 -05:00

dohertj2 referenced this issue

2026-11-10 08:59:05 -05:00

Implement the out-of-process script runner (design #750) — code on master; QA shadow week (#793, #794) + P3 flip remain #780

dohertj2 referenced this issue

2026-11-10 09:41:01 -05:00

Implement the out-of-process script runner (design #750) — code on master; QA shadow week (#793, #794) + P3 flip remain #780

dohertj2 referenced this issue