Files
lmxopcua/scripts/validation/opcua-windows-service/evidence/2026-11-12-windev/RESULTS.md
T
Joseph Doherty 1172687d4b
v2-ci / unit-tests (push) Has been cancelled
test(validation): #800 gate re-run on windev with the fixed isolation installer, both PowerShell hosts
Re-runs only the ScriptWorker step on windev (master da948e5f, SDK 2.0.1, 16 vCPUs): install as the OtOpcUaHost service,
repo Install-ScriptWorkerIsolation.ps1 applied/re-applied/uninstalled twice under pwsh 7 and Windows PowerShell 5.1,
restricted worker, pipe refuses the admin, runner Healthy, strict-peer/API-key matrix on, then cleanup. The harness drops
-PatchedIsolation, gains -PrepareFrom candidate and -ApplyHost, and suffixes the isolation evidence with -Label.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 11:24:01 -05:00

14 KiB

lmxopcua#800: Windows service gate on windev (2026-11-12)

Follow-up (2026-11-12, later): lmxopcua#883 is fixed and step 3 was re-run with the repository installer on both PowerShell hosts: see ../2026-11-12-windev-isolation-rerun/RESULTS.md. The step 3 text below is the record of the first run.

Result: steps 1, 2, 4 and 5 passed; step 3 passed only partly. The SDK 2 candidate upgraded in place under the real OtOpcUaHost Windows service:

  • it reused the existing application certificate byte-for-byte, with its file ACLs unchanged;
  • it passed the strict-peer and API-key checks on three starts;
  • it rolled back to the known-good build with the same identity.

The ScriptWorker runtime isolation works under a real service session, but the repository installer fails on Windows (lmxopcua#883). The runtime part of step 3 was proven with the #883 patch, not with the shipped script. Three pre-existing findings were filed: #879, #880 (a rollback and QA-compat risk) and #883.

Environment

  • Node: windev (DESKTOP-6JL3KKO, 10.100.0.48). Elevated SSH session, PowerShell 7.6.10; Windows PowerShell 5.1 was also used for the installer.
  • Host change during the run: the coordinator power-cycled the node between steps 1 and 2 to reconfigure it (36 → 16 vCPUs, latency sensitivity High). The installed service auto-started across that reboot (row b2reboot). Timings before and after the reboot are not like-for-like.
  • Clock: the Windows Time service is stopped. Before the reboot the clock was about 9 minutes slow, after it about 14 s. All probes ran on the same box, so #873 skew does not apply.
  • Left untouched: the existing MxAccessGw (:5120/:5130) and OtOpcUaWonderwareHistorian services were never touched.
  • Ports: the test service used only task ports: HTTP 127.0.0.1:18800, OPC UA 14841, Akka 18801.

Artifacts

Every artifact is a self-contained win-x64 publish, built on macOS (.NET SDK 10.0.204). Each manifest lists sha256 path and was matched file-for-file on windev before use.

Artifact Source Files Manifest SHA-256 Main binary SHA-256
Known-good Host master efb9197a0f48802450474413c58b9d0c461b836d (last SDK 1.5.378.182 master commit) 1376 28bdba9e…49d18d OtOpcUa.Host.exe f9a193c6c7d224006094e26ce4cd1616961bcb9048ff38a468786618a26e3a91
Candidate Host master 438484fdca4df12be6fbcae71cfbbeabb9386004 (SDK 2.0.1; Opc.Ua.Core file version 2.0.1.2) 1396 e7b6683e…090c5e34 OtOpcUa.Host.exe 6e9efdde592eb342719e9c0d47783e306bc9dd697ae80fc32d45dd1538a52000
Candidate CLI 438484fd 241 dee9a60e…2fdd1504 e2c88c071733e5807a260ee33d5064e1ec13178d2562aab8d7faff763b6974a0
Known-good CLI efb9197a 210 f7715725…0c217c66 257b5b13e980072e46f9405aeddbbddb440a17ad9d26e122ede3fddb15cb27cf
Probe (SDK 2.0.1) this directory at 438484fd 1132 abe1d8b6…ba070 Probe.dll 64597faadab6511b174c4c9aa1f5fd06054ffccbb5b77b841c318838e6cbb9fa
Seeder this directory, built in the efb9197a tree 413 ec93daf8…7106561 Seeder.dll 18690a15628f028df95bd4a578049a3997f31e41c22a5a06ee04e50739920de5

The full manifests are manifest-*.txt in this directory. The candidate script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe is 9dc0558bb62231487dafcd1b792bbe280b5bafc8849cb0347b14fb50483cfc04, one of the 275 files in script-worker\.

Install

  • Install command: scripts/install/Install-Services.ps1 -InstallRoot C:\OtOpcUa800\OtOpcUa -ServiceAccount 'NT SERVICE\OtOpcUaHost' -Roles driver, from the known-good tree. It is the repository-supported method; vd03 runs the same OtOpcUaHost sc.exe service.
  • Node configuration: set in the service Environment value, 26 entries, digest fd534f8d…. The digest was the same at every known-good and candidate start except c3, where the isolation run adds 3 entries.
    • single-node driver: FetchAndCache, with an unreachable central;
    • secrets SQLite outside the content root;
    • pepper;
    • LDAP and ACL off;
    • AllowAnonymous=false, auto-accept off;
    • EnabledSecurityProfiles=[SignAndEncrypt, Sign].
  • ACLs: C:\OtOpcUa800 was protected with SYSTEM and Administrators only. The service account got RX on the install folder and Modify on pki\, data\ and logs\.
  • First start (b0): the service created its PKI (owner = the service SID; the private key inherits service-M, SYSTEM-F, Administrators-F), its LocalDb and its logs. It reached Healthy with no deployment.
  • Seed: the service was stopped, and the efb9197a-built seeder wrote one deployment through IDeploymentArtifactCache. The deployment holds a raw folder, a UNS equipment with a timer virtual tag (return 800;), a 65-character key, a 54-character key, a disabled key and read grants.
  • Trust: the probe and CLI certificates were placed in pki\trusted\certs.

Start/stop record

Every start was through Start-Service and every stop through Stop-Service, except b2reboot. Health means /health/ready, /healthz and /health/active, all 200 and Healthy. ServiceLevel was read over OPC UA.

Start Build How Ready after ServiceLevel App cert thumbprint (SHA-1) Stop Exit Secrets in logs
b0 known-good Start-Service, empty LocalDb 13.4 s — CEF9D91F8CBC0EE78B317D3AA51EB40CD9C41ACA (created) 1.6 s 0 0
b1 known-good Start-Service 12.4 s 250 same 1.6 s 0 0
b2reboot known-good SCM auto-start after the host reboot — — same 1.6 s 0 0
b3, b4 known-good Start-Service (b4 after re-seeding the negative keys) 8.4 s, 8.3 s 250 same 1.6 s 0 0
c1 candidate Start-Service after the in-place upgrade 15.3 s 255 same 1.8 s 0 0
c2 candidate Start-Service 15.1 s 255 same 1.8 s 0 0
c3 candidate Start-Service, Scripting:Runner=OutOfProcess with a worker identity 15.1 s 255 same 1.8 s 0 0
r1 known-good (rollback) Start-Service 8.5 s 250 same 1.6 s 0 0

Certificate: the DER SHA-256 was F38E6B2E…3F845 at every start.

PKI tree: compare-b2reboot-…-r1.json digests every PKI path with its SHA-256 and SDDL. The digest was identical (c8517c1d0298…, 13 entries) at b2reboot, b3, b4, c1, c2 and r1. At c3 the only difference was the inherited (D;…;CC;;;<worker SID>) deny ACE from the isolation installer; file hashes were unchanged, and the ACE was gone again at r1. b1 had 10 entries; the 3 later ones are pki\rejected\…, the untrusted test client that the SDK filed as rejected.

Other observations:

  • The LocalDb file ACLs were the same across all starts.
  • Each stop left no leftover host process and no leftover workers.
  • The scans covered the rolling logs, the Serilog selflog and the Application event log, with redacted copies kept. They found 0 occurrences of any task secret: the API keys and their secret parts, the pepper, the fetch key and the KEK.
  • ServiceLevel is 250 on the known-good build and 255 on the candidate, which matches vd03's 255 on 5fb4b4db.
  • Readiness took about 15 s on the candidate against about 8.4 s on the known-good build after the reboot. This is an observation, not a gate criterion.

Protocol checks (same case set at every checked start; checks-*.json)

Every check used the SDK 2.0.1 probe with peer auto-accept off, Basic256Sha256. "Gating" cases must all pass.

Case Known-good (b4, r1) Candidate (c1, c2, c3)
SignAndEncrypt + Sign, 54-char valid key admitted; State 0 Good, ServiceLevel 250, VT 800 Good admitted; ServiceLevel 255, VT 800 Good
SignAndEncrypt + Sign, 65-char valid key (production form) BadIdentityTokenInvalid (#880, expected denial) admitted, Good
wrong secret (Sign + SignAndEncrypt), unknown id, disabled key, 54-char BadIdentityTokenRejected BadIdentityTokenRejected
wrong secret / unknown id, 65-char BadIdentityTokenInvalid BadIdentityTokenRejected
anonymous denied (BadIdentityTokenInvalid; not advertised) denied
untrusted client, Sign + SignAndEncrypt BadSecurityChecksFailed BadSecurityChecksFailed
untrusted server (client trust list emptied) BadCertificateUntrusted BadCertificateUntrusted
actual CLI read ServerStatus, SignAndEncrypt + API key SDK 1.5 CLI 65-char Good; SDK 2 CLI 54-char Good SDK 1.5 CLI 65-char Good; SDK 2 CLI 54- and 65-char Good
finding: SDK 2 CLI with the 65-char key BadIdentityTokenInvalid (#880) Good
finding: advertised endpoints None/None + Sign + SignAndEncrypt (#879) same
finding: session over the None endpoint, valid key admitted (#879) admitted

Step 3: ScriptWorker (isolation-*.json)

The out-of-process runner is opt-in, and no deployed node configures it; vd03 runs InProcess. On windev:

  1. Package. The candidate's script-worker\ holds the self-contained worker apphost and is 275 files.
  2. The repository installer fails (#883). This is the step #794 requires.
    • pwsh 7: -WhatIf lists 7 steps and changes nothing. The real run creates the anchor service and the PE cache, then throws DirectoryInfo … GetAccessControl.
    • Windows PowerShell 5.1: it throws property 'Count' cannot be found (StrictMode) at PeEntriesDir.
    • -Uninstall fails the same way, so the partial state could not be rolled back with the tool.
  3. With the #883 patch (isolation-installer-883.diff):
    • Install and re-apply. Apply ran under pwsh, then re-apply under 5.1 left all 7 steps unchanged.
    • Anchor service. OtOpcUaScriptWorker is DISABLED, as NT SERVICE\OtOpcUaScriptWorker, SID S-1-5-80-1846135827-…-3023753135 (matches #794).
    • Folder ACLs. The install folder has (OI)(CI)(DENY)(S,RD) for the worker and script-worker\ has (RX).
    • PE cache. It is protected with SYSTEM, Administrators and the host account at F; pe\ adds the worker at M.
    • Firewall. OtOpcUa-ScriptWorker-BlockOutbound is Outbound/Block/Any on the worker exe.
    • Service environment. It gained exactly Scripting__Worker__Identity and Scripting__Worker__PeCacheDir, plus the gate's Scripting__Runner=OutOfProcess; the other entries were kept.
  4. Runtime under the real service (c3).
    • Worker process. ScriptWorker[0] started as the child of the host, run as a restricted copy of NT SERVICE\OtOpcUaHost (IsTokenRestricted=true, 1 privilege).
    • Window-station grant. It works under a real service session: coreclr loaded, which windev over SSH could not prove for #794.
    • Pipe ACL. The otopcua-sw-* pipe refuses the elevated administrator (UnauthorizedAccessException).
    • Health. script-runner reports Healthy "Out-of-process script runner ready", with the PE cache on and 1 pe\ entry.
    • Virtual tag. It evaluated out of process to 800 Good.
    • Stop. No worker survived the stop.
    • Not exercised: dropped-packet firewall logging.
  5. Uninstall. -Uninstall under 5.1 and then again under pwsh left every step unchanged. The anchor and the firewall rule were gone, the environment digest was back to fd534f8d…, and the install-folder ACL was back to its original state.

Upgrade and rollback mechanics (upgrade.json, rollback.json)

Upgrade, with the service stopped:

  • reg export of the service key, and a robocopy /E /COPYALL /DCOPY:DAT backup (1386 files).
  • The candidate was staged in a separate folder and given the install-folder DACL.
  • The box's appsettings*.json were carried over; they are byte-identical to the candidate publisher's.
  • pki\, data\ and logs\ were carried over with /COPYALL.
  • The folders were swapped with Move-Item. The service PathName was unchanged.

Rollback, with the service stopped:

  • The candidate was moved to quarantine and the known-good folder was moved back.
  • The current pki\ and data\ were carried back with /COPYALL. No data rollback: the candidate-written LocalDb opened cleanly in the known-good build, as on vd03.

Cleanup

These were removed:

  • the service, with sc.exe delete;
  • the OtOpcUaScriptWorker anchor and its firewall rule;
  • C:\OtOpcUa800;
  • the gate scripts in the profile.

Do not use Uninstall-Services.ps1 on windev: it also stops and deletes OtOpcUaWonderwareHistorian.

After cleanup:

  • %LOCALAPPDATA%\OtOpcUaClient was restored from its backup (its settings.json and logs\ are from 2026-05-24).
  • The firewall rule count is back to 876.
  • MxAccessGw is Running with :5130/health/ready 200, and OtOpcUaWonderwareHistorian is Running.

Residual: the virtual-account profile C:\Windows\ServiceProfiles\OtOpcUaHost, with its ProfileList entry, stays loaded until the next reboot. Remove it after that with Get-CimInstance Win32_UserProfile | ? SID -eq 'S-1-5-80-3928861625-3486739760-3962757976-1980707066-3667558059' | Remove-CimInstance.

gate-run.log lists every phase invocation and its exit code. The first Prepare line reported exit 0 even though the wrapper had failed (the remote $PSStyle quoting). That attempt stopped before creating the service, and the phase was re-run.

Real-world cross-reference

wonder-app-vd03 (DEV) has run master 5fb4b4db on SDK 2.0.1 since 2026-11-12. It reused the app cert A873DF51… and reports ServiceLevel 255 with Good reads; see #782 and docs/deploy/vd03-upgrade-b8b4c18f-to-master.md. That record is consistent with this run but is not a controlled install/rollback.