# lmxopcua#800: Windows service gate on windev (2026-11-12) > **Follow-up (2026-11-12, later):** lmxopcua#883 is fixed and step 3 was re-run with the repository installer on both PowerShell hosts: see > [`../2026-11-12-windev-isolation-rerun/RESULTS.md`](../2026-11-12-windev-isolation-rerun/RESULTS.md). The step 3 text below is the record of the first run. **Result: steps 1, 2, 4 and 5 passed; step 3 passed only partly.** The SDK 2 candidate upgraded in place under the real `OtOpcUaHost` Windows service: - it reused the existing application certificate byte-for-byte, with its file ACLs unchanged; - it passed the strict-peer and API-key checks on three starts; - it rolled back to the known-good build with the same identity. The ScriptWorker runtime isolation works under a real service session, but **the repository installer fails on Windows** (lmxopcua#883). The runtime part of step 3 was proven with the #883 patch, not with the shipped script. Three pre-existing findings were filed: #879, #880 (a rollback and QA-compat risk) and #883. ## Environment - **Node:** windev (`DESKTOP-6JL3KKO`, 10.100.0.48). Elevated SSH session, PowerShell 7.6.10; Windows PowerShell 5.1 was also used for the installer. - **Host change during the run:** the coordinator power-cycled the node between steps 1 and 2 to reconfigure it (36 → 16 vCPUs, latency sensitivity High). The installed service auto-started across that reboot (row `b2reboot`). Timings before and after the reboot are not like-for-like. - **Clock:** the Windows Time service is stopped. Before the reboot the clock was about 9 minutes slow, after it about 14 s. All probes ran on the same box, so #873 skew does not apply. - **Left untouched:** the existing MxAccessGw (`:5120`/`:5130`) and OtOpcUaWonderwareHistorian services were never touched. - **Ports:** the test service used only task ports: HTTP `127.0.0.1:18800`, OPC UA `14841`, Akka `18801`. ## Artifacts Every artifact is a self-contained win-x64 publish, built on macOS (.NET SDK 10.0.204). Each manifest lists `sha256 path` and was matched file-for-file on windev before use. | Artifact | Source | Files | Manifest SHA-256 | Main binary SHA-256 | |---|---|---:|---|---| | Known-good Host | master `efb9197a0f48802450474413c58b9d0c461b836d` (last SDK 1.5.378.182 master commit) | 1376 | `28bdba9e…49d18d` | `OtOpcUa.Host.exe` `f9a193c6c7d224006094e26ce4cd1616961bcb9048ff38a468786618a26e3a91` | | Candidate Host | master `438484fdca4df12be6fbcae71cfbbeabb9386004` (SDK 2.0.1; Opc.Ua.Core file version 2.0.1.2) | 1396 | `e7b6683e…090c5e34` | `OtOpcUa.Host.exe` `6e9efdde592eb342719e9c0d47783e306bc9dd697ae80fc32d45dd1538a52000` | | Candidate CLI | `438484fd` | 241 | `dee9a60e…2fdd1504` | `e2c88c071733e5807a260ee33d5064e1ec13178d2562aab8d7faff763b6974a0` | | Known-good CLI | `efb9197a` | 210 | `f7715725…0c217c66` | `257b5b13e980072e46f9405aeddbbddb440a17ad9d26e122ede3fddb15cb27cf` | | Probe (SDK 2.0.1) | this directory at `438484fd` | 1132 | `abe1d8b6…ba070` | `Probe.dll` `64597faadab6511b174c4c9aa1f5fd06054ffccbb5b77b841c318838e6cbb9fa` | | Seeder | this directory, built in the `efb9197a` tree | 413 | `ec93daf8…7106561` | `Seeder.dll` `18690a15628f028df95bd4a578049a3997f31e41c22a5a06ee04e50739920de5` | The full manifests are `manifest-*.txt` in this directory. The candidate `script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe` is `9dc0558bb62231487dafcd1b792bbe280b5bafc8849cb0347b14fb50483cfc04`, one of the 275 files in `script-worker\`. ## Install - **Install command:** `scripts/install/Install-Services.ps1 -InstallRoot C:\OtOpcUa800\OtOpcUa -ServiceAccount 'NT SERVICE\OtOpcUaHost' -Roles driver`, from the known-good tree. It is the repository-supported method; vd03 runs the same `OtOpcUaHost` sc.exe service. - **Node configuration:** set in the service `Environment` value, 26 entries, digest `fd534f8d…`. The digest was the same at every known-good and candidate start except `c3`, where the isolation run adds 3 entries. - single-node driver: `FetchAndCache`, with an unreachable central; - secrets SQLite outside the content root; - pepper; - LDAP and ACL off; - `AllowAnonymous=false`, auto-accept off; - `EnabledSecurityProfiles=[SignAndEncrypt, Sign]`. - **ACLs:** `C:\OtOpcUa800` was protected with SYSTEM and Administrators only. The service account got RX on the install folder and Modify on `pki\`, `data\` and `logs\`. - **First start (`b0`):** the service created its PKI (owner = the service SID; the private key inherits service-M, SYSTEM-F, Administrators-F), its LocalDb and its logs. It reached Healthy with no deployment. - **Seed:** the service was stopped, and the `efb9197a`-built seeder wrote one deployment through `IDeploymentArtifactCache`. The deployment holds a raw folder, a UNS equipment with a timer virtual tag (`return 800;`), a 65-character key, a 54-character key, a disabled key and read grants. - **Trust:** the probe and CLI certificates were placed in `pki\trusted\certs`. ## Start/stop record Every start was through `Start-Service` and every stop through `Stop-Service`, except `b2reboot`. Health means `/health/ready`, `/healthz` and `/health/active`, all 200 and Healthy. ServiceLevel was read over OPC UA. | Start | Build | How | Ready after | ServiceLevel | App cert thumbprint (SHA-1) | Stop | Exit | Secrets in logs | |---|---|---|---|---|---|---|---|---| | b0 | known-good | Start-Service, empty LocalDb | 13.4 s | — | `CEF9D91F8CBC0EE78B317D3AA51EB40CD9C41ACA` (created) | 1.6 s | 0 | 0 | | b1 | known-good | Start-Service | 12.4 s | 250 | same | 1.6 s | 0 | 0 | | b2reboot | known-good | **SCM auto-start after the host reboot** | — | — | same | 1.6 s | 0 | 0 | | b3, b4 | known-good | Start-Service (b4 after re-seeding the negative keys) | 8.4 s, 8.3 s | 250 | same | 1.6 s | 0 | 0 | | **c1** | **candidate** | Start-Service after the in-place upgrade | 15.3 s | 255 | **same** | 1.8 s | 0 | 0 | | **c2** | **candidate** | Start-Service | 15.1 s | 255 | **same** | 1.8 s | 0 | 0 | | c3 | candidate | Start-Service, `Scripting:Runner=OutOfProcess` with a worker identity | 15.1 s | 255 | same | 1.8 s | 0 | 0 | | **r1** | **known-good (rollback)** | Start-Service | 8.5 s | 250 | **same** | 1.6 s | 0 | 0 | **Certificate:** the DER SHA-256 was `F38E6B2E…3F845` at every start. **PKI tree:** `compare-b2reboot-…-r1.json` digests every PKI path with its SHA-256 and SDDL. The digest was identical (`c8517c1d0298…`, 13 entries) at b2reboot, b3, b4, c1, c2 and r1. At c3 the only difference was the inherited `(D;…;CC;;;)` deny ACE from the isolation installer; file hashes were unchanged, and the ACE was gone again at r1. b1 had 10 entries; the 3 later ones are `pki\rejected\…`, the untrusted test client that the SDK filed as rejected. **Other observations:** - The LocalDb file ACLs were the same across all starts. - Each stop left no leftover host process and no leftover workers. - The scans covered the rolling logs, the Serilog selflog and the Application event log, with redacted copies kept. They found 0 occurrences of any task secret: the API keys and their secret parts, the pepper, the fetch key and the KEK. - ServiceLevel is 250 on the known-good build and 255 on the candidate, which matches vd03's 255 on `5fb4b4db`. - Readiness took about 15 s on the candidate against about 8.4 s on the known-good build after the reboot. This is an observation, not a gate criterion. ## Protocol checks (same case set at every checked start; `checks-*.json`) Every check used the SDK 2.0.1 probe with peer auto-accept off, Basic256Sha256. "Gating" cases must all pass. | Case | Known-good (b4, r1) | Candidate (c1, c2, c3) | |---|---|---| | SignAndEncrypt + Sign, 54-char valid key | admitted; State 0 Good, ServiceLevel 250, VT 800 Good | admitted; ServiceLevel 255, VT 800 Good | | SignAndEncrypt + Sign, 65-char valid key (production form) | **`BadIdentityTokenInvalid`** (#880, expected denial) | admitted, Good | | wrong secret (Sign + SignAndEncrypt), unknown id, disabled key, 54-char | `BadIdentityTokenRejected` | `BadIdentityTokenRejected` | | wrong secret / unknown id, 65-char | `BadIdentityTokenInvalid` | `BadIdentityTokenRejected` | | anonymous | denied (`BadIdentityTokenInvalid`; not advertised) | denied | | untrusted client, Sign + SignAndEncrypt | `BadSecurityChecksFailed` | `BadSecurityChecksFailed` | | untrusted server (client trust list emptied) | `BadCertificateUntrusted` | `BadCertificateUntrusted` | | actual CLI read `ServerStatus`, SignAndEncrypt + API key | SDK 1.5 CLI 65-char Good; SDK 2 CLI 54-char Good | SDK 1.5 CLI 65-char Good; SDK 2 CLI 54- and 65-char Good | | *finding:* SDK 2 CLI with the 65-char key | `BadIdentityTokenInvalid` (#880) | Good | | *finding:* advertised endpoints | **None/None + Sign + SignAndEncrypt** (#879) | same | | *finding:* session over the None endpoint, valid key | admitted (#879) | admitted | ## Step 3: ScriptWorker (`isolation-*.json`) The out-of-process runner is opt-in, and no deployed node configures it; vd03 runs InProcess. On windev: 1. **Package.** The candidate's `script-worker\` holds the self-contained worker apphost and is 275 files. 2. **The repository installer fails** (#883). This is the step #794 requires. - **pwsh 7:** `-WhatIf` lists 7 steps and changes nothing. The real run creates the anchor service and the PE cache, then throws `DirectoryInfo … GetAccessControl`. - **Windows PowerShell 5.1:** it throws `property 'Count' cannot be found` (StrictMode) at PeEntriesDir. - **`-Uninstall`** fails the same way, so the partial state could not be rolled back with the tool. 3. **With the #883 patch** (`isolation-installer-883.diff`): - **Install and re-apply.** Apply ran under pwsh, then re-apply under 5.1 left all 7 steps `unchanged`. - **Anchor service.** `OtOpcUaScriptWorker` is DISABLED, as `NT SERVICE\OtOpcUaScriptWorker`, SID `S-1-5-80-1846135827-…-3023753135` (matches #794). - **Folder ACLs.** The install folder has `(OI)(CI)(DENY)(S,RD)` for the worker and `script-worker\` has `(RX)`. - **PE cache.** It is protected with SYSTEM, Administrators and the host account at F; `pe\` adds the worker at M. - **Firewall.** `OtOpcUa-ScriptWorker-BlockOutbound` is Outbound/Block/Any on the worker exe. - **Service environment.** It gained exactly `Scripting__Worker__Identity` and `Scripting__Worker__PeCacheDir`, plus the gate's `Scripting__Runner=OutOfProcess`; the other entries were kept. 4. **Runtime under the real service (c3).** - **Worker process.** `ScriptWorker[0] started` as the child of the host, run as a restricted copy of `NT SERVICE\OtOpcUaHost` (`IsTokenRestricted=true`, 1 privilege). - **Window-station grant.** It works under a real service session: coreclr loaded, which windev over SSH could not prove for #794. - **Pipe ACL.** The `otopcua-sw-*` pipe refuses the elevated administrator (`UnauthorizedAccessException`). - **Health.** `script-runner` reports Healthy "Out-of-process script runner ready", with the PE cache on and 1 `pe\` entry. - **Virtual tag.** It evaluated out of process to 800 Good. - **Stop.** No worker survived the stop. - **Not exercised:** dropped-packet firewall logging. 5. **Uninstall.** `-Uninstall` under 5.1 and then again under pwsh left every step `unchanged`. The anchor and the firewall rule were gone, the environment digest was back to `fd534f8d…`, and the install-folder ACL was back to its original state. ## Upgrade and rollback mechanics (`upgrade.json`, `rollback.json`) **Upgrade**, with the service stopped: - `reg export` of the service key, and a `robocopy /E /COPYALL /DCOPY:DAT` backup (1386 files). - The candidate was staged in a separate folder and given the install-folder DACL. - The box's `appsettings*.json` were carried over; they are byte-identical to the candidate publisher's. - `pki\`, `data\` and `logs\` were carried over with `/COPYALL`. - The folders were swapped with `Move-Item`. The service `PathName` was unchanged. **Rollback**, with the service stopped: - The candidate was moved to quarantine and the known-good folder was moved back. - The current `pki\` and `data\` were carried back with `/COPYALL`. No data rollback: the candidate-written LocalDb opened cleanly in the known-good build, as on vd03. ## Cleanup These were removed: - the service, with `sc.exe delete`; - the `OtOpcUaScriptWorker` anchor and its firewall rule; - `C:\OtOpcUa800`; - the gate scripts in the profile. Do **not** use `Uninstall-Services.ps1` on windev: it also stops and deletes `OtOpcUaWonderwareHistorian`. After cleanup: - `%LOCALAPPDATA%\OtOpcUaClient` was restored from its backup (its `settings.json` and `logs\` are from 2026-05-24). - The firewall rule count is back to 876. - MxAccessGw is Running with `:5130/health/ready` 200, and OtOpcUaWonderwareHistorian is Running. **Residual:** the virtual-account profile `C:\Windows\ServiceProfiles\OtOpcUaHost`, with its ProfileList entry, stays loaded until the next reboot. Remove it after that with `Get-CimInstance Win32_UserProfile | ? SID -eq 'S-1-5-80-3928861625-3486739760-3962757976-1980707066-3667558059' | Remove-CimInstance`. `gate-run.log` lists every phase invocation and its exit code. The first `Prepare` line reported exit 0 even though the wrapper had failed (the remote `$PSStyle` quoting). That attempt stopped before creating the service, and the phase was re-run. ## Real-world cross-reference wonder-app-vd03 (DEV) has run master `5fb4b4db` on SDK 2.0.1 since 2026-11-12. It reused the app cert `A873DF51…` and reports ServiceLevel 255 with Good reads; see #782 and `docs/deploy/vd03-upgrade-b8b4c18f-to-master.md`. That record is consistent with this run but is not a controlled install/rollback.