1172687d4b
v2-ci / unit-tests (push) Has been cancelled
Re-runs only the ScriptWorker step on windev (master da948e5f, SDK 2.0.1, 16 vCPUs): install as the OtOpcUaHost service,
repo Install-ScriptWorkerIsolation.ps1 applied/re-applied/uninstalled twice under pwsh 7 and Windows PowerShell 5.1,
restricted worker, pipe refuses the admin, runner Healthy, strict-peer/API-key matrix on, then cleanup. The harness drops
-PatchedIsolation, gains -PrepareFrom candidate and -ApplyHost, and suffixes the isolation evidence with -Label.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
189 lines
14 KiB
Markdown
189 lines
14 KiB
Markdown
# lmxopcua#800: Windows service gate on windev (2026-11-12)
|
|
|
|
> **Follow-up (2026-11-12, later):** lmxopcua#883 is fixed and step 3 was re-run with the repository installer on both PowerShell hosts: see
|
|
> [`../2026-11-12-windev-isolation-rerun/RESULTS.md`](../2026-11-12-windev-isolation-rerun/RESULTS.md). The step 3 text below is the record of the first run.
|
|
|
|
**Result: steps 1, 2, 4 and 5 passed; step 3 passed only partly.** The SDK 2 candidate upgraded in place under the real
|
|
`OtOpcUaHost` Windows service:
|
|
- it reused the existing application certificate byte-for-byte, with its file ACLs unchanged;
|
|
- it passed the strict-peer and API-key checks on three starts;
|
|
- it rolled back to the known-good build with the same identity.
|
|
|
|
The ScriptWorker runtime isolation works under a real service session, but **the repository installer fails on Windows**
|
|
(lmxopcua#883). The runtime part of step 3 was proven with the #883 patch, not with the shipped script. Three
|
|
pre-existing findings were filed: #879, #880 (a rollback and QA-compat risk) and #883.
|
|
|
|
## Environment
|
|
|
|
- **Node:** windev (`DESKTOP-6JL3KKO`, 10.100.0.48). Elevated SSH session, PowerShell 7.6.10; Windows PowerShell 5.1 was also
|
|
used for the installer.
|
|
- **Host change during the run:** the coordinator power-cycled the node between steps 1 and 2 to reconfigure it (36 → 16 vCPUs,
|
|
latency sensitivity High). The installed service auto-started across that reboot (row `b2reboot`). Timings before and after the
|
|
reboot are not like-for-like.
|
|
- **Clock:** the Windows Time service is stopped. Before the reboot the clock was about 9 minutes slow, after it about 14 s.
|
|
All probes ran on the same box, so #873 skew does not apply.
|
|
- **Left untouched:** the existing MxAccessGw (`:5120`/`:5130`) and OtOpcUaWonderwareHistorian services were never touched.
|
|
- **Ports:** the test service used only task ports: HTTP `127.0.0.1:18800`, OPC UA `14841`, Akka `18801`.
|
|
|
|
## Artifacts
|
|
|
|
Every artifact is a self-contained win-x64 publish, built on macOS (.NET SDK 10.0.204). Each manifest lists `sha256 path` and
|
|
was matched file-for-file on windev before use.
|
|
|
|
| Artifact | Source | Files | Manifest SHA-256 | Main binary SHA-256 |
|
|
|---|---|---:|---|---|
|
|
| Known-good Host | master `efb9197a0f48802450474413c58b9d0c461b836d` (last SDK 1.5.378.182 master commit) | 1376 | `28bdba9e…49d18d` | `OtOpcUa.Host.exe` `f9a193c6c7d224006094e26ce4cd1616961bcb9048ff38a468786618a26e3a91` |
|
|
| Candidate Host | master `438484fdca4df12be6fbcae71cfbbeabb9386004` (SDK 2.0.1; Opc.Ua.Core file version 2.0.1.2) | 1396 | `e7b6683e…090c5e34` | `OtOpcUa.Host.exe` `6e9efdde592eb342719e9c0d47783e306bc9dd697ae80fc32d45dd1538a52000` |
|
|
| Candidate CLI | `438484fd` | 241 | `dee9a60e…2fdd1504` | `e2c88c071733e5807a260ee33d5064e1ec13178d2562aab8d7faff763b6974a0` |
|
|
| Known-good CLI | `efb9197a` | 210 | `f7715725…0c217c66` | `257b5b13e980072e46f9405aeddbbddb440a17ad9d26e122ede3fddb15cb27cf` |
|
|
| Probe (SDK 2.0.1) | this directory at `438484fd` | 1132 | `abe1d8b6…ba070` | `Probe.dll` `64597faadab6511b174c4c9aa1f5fd06054ffccbb5b77b841c318838e6cbb9fa` |
|
|
| Seeder | this directory, built in the `efb9197a` tree | 413 | `ec93daf8…7106561` | `Seeder.dll` `18690a15628f028df95bd4a578049a3997f31e41c22a5a06ee04e50739920de5` |
|
|
|
|
The full manifests are `manifest-*.txt` in this directory. The candidate `script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe` is
|
|
`9dc0558bb62231487dafcd1b792bbe280b5bafc8849cb0347b14fb50483cfc04`, one of the 275 files in `script-worker\`.
|
|
|
|
## Install
|
|
|
|
- **Install command:** `scripts/install/Install-Services.ps1 -InstallRoot C:\OtOpcUa800\OtOpcUa -ServiceAccount 'NT SERVICE\OtOpcUaHost' -Roles driver`,
|
|
from the known-good tree. It is the repository-supported method; vd03 runs the same `OtOpcUaHost` sc.exe service.
|
|
- **Node configuration:** set in the service `Environment` value, 26 entries, digest `fd534f8d…`. The digest was the same at every
|
|
known-good and candidate start except `c3`, where the isolation run adds 3 entries.
|
|
- single-node driver: `FetchAndCache`, with an unreachable central;
|
|
- secrets SQLite outside the content root;
|
|
- pepper;
|
|
- LDAP and ACL off;
|
|
- `AllowAnonymous=false`, auto-accept off;
|
|
- `EnabledSecurityProfiles=[SignAndEncrypt, Sign]`.
|
|
- **ACLs:** `C:\OtOpcUa800` was protected with SYSTEM and Administrators only. The service account got RX on the install folder
|
|
and Modify on `pki\`, `data\` and `logs\`.
|
|
- **First start (`b0`):** the service created its PKI (owner = the service SID; the private key inherits service-M,
|
|
SYSTEM-F, Administrators-F), its LocalDb and its logs. It reached Healthy with no deployment.
|
|
- **Seed:** the service was stopped, and the `efb9197a`-built seeder wrote one deployment through `IDeploymentArtifactCache`. The
|
|
deployment holds a raw folder, a UNS equipment with a timer virtual tag (`return 800;`), a 65-character key, a 54-character key, a
|
|
disabled key and read grants.
|
|
- **Trust:** the probe and CLI certificates were placed in `pki\trusted\certs`.
|
|
|
|
## Start/stop record
|
|
|
|
Every start was through `Start-Service` and every stop through `Stop-Service`, except `b2reboot`. Health means `/health/ready`,
|
|
`/healthz` and `/health/active`, all 200 and Healthy. ServiceLevel was read over OPC UA.
|
|
|
|
| Start | Build | How | Ready after | ServiceLevel | App cert thumbprint (SHA-1) | Stop | Exit | Secrets in logs |
|
|
|---|---|---|---|---|---|---|---|---|
|
|
| b0 | known-good | Start-Service, empty LocalDb | 13.4 s | — | `CEF9D91F8CBC0EE78B317D3AA51EB40CD9C41ACA` (created) | 1.6 s | 0 | 0 |
|
|
| b1 | known-good | Start-Service | 12.4 s | 250 | same | 1.6 s | 0 | 0 |
|
|
| b2reboot | known-good | **SCM auto-start after the host reboot** | — | — | same | 1.6 s | 0 | 0 |
|
|
| b3, b4 | known-good | Start-Service (b4 after re-seeding the negative keys) | 8.4 s, 8.3 s | 250 | same | 1.6 s | 0 | 0 |
|
|
| **c1** | **candidate** | Start-Service after the in-place upgrade | 15.3 s | 255 | **same** | 1.8 s | 0 | 0 |
|
|
| **c2** | **candidate** | Start-Service | 15.1 s | 255 | **same** | 1.8 s | 0 | 0 |
|
|
| c3 | candidate | Start-Service, `Scripting:Runner=OutOfProcess` with a worker identity | 15.1 s | 255 | same | 1.8 s | 0 | 0 |
|
|
| **r1** | **known-good (rollback)** | Start-Service | 8.5 s | 250 | **same** | 1.6 s | 0 | 0 |
|
|
|
|
**Certificate:** the DER SHA-256 was `F38E6B2E…3F845` at every start.
|
|
|
|
**PKI tree:** `compare-b2reboot-…-r1.json` digests every PKI path with its SHA-256 and SDDL. The digest was identical
|
|
(`c8517c1d0298…`, 13 entries) at b2reboot, b3, b4, c1, c2 and r1. At c3 the only difference was the inherited
|
|
`(D;…;CC;;;<worker SID>)` deny ACE from the isolation installer; file hashes were unchanged, and the ACE was gone again at r1.
|
|
b1 had 10 entries; the 3 later ones are `pki\rejected\…`, the untrusted test client that the SDK filed as rejected.
|
|
|
|
**Other observations:**
|
|
- The LocalDb file ACLs were the same across all starts.
|
|
- Each stop left no leftover host process and no leftover workers.
|
|
- The scans covered the rolling logs, the Serilog selflog and the Application event log, with redacted copies kept. They found
|
|
0 occurrences of any task secret: the API keys and their secret parts, the pepper, the fetch key and the KEK.
|
|
- ServiceLevel is 250 on the known-good build and 255 on the candidate, which matches vd03's 255 on `5fb4b4db`.
|
|
- Readiness took about 15 s on the candidate against about 8.4 s on the known-good build after the reboot. This is an observation,
|
|
not a gate criterion.
|
|
|
|
## Protocol checks (same case set at every checked start; `checks-*.json`)
|
|
|
|
Every check used the SDK 2.0.1 probe with peer auto-accept off, Basic256Sha256. "Gating" cases must all pass.
|
|
|
|
| Case | Known-good (b4, r1) | Candidate (c1, c2, c3) |
|
|
|---|---|---|
|
|
| SignAndEncrypt + Sign, 54-char valid key | admitted; State 0 Good, ServiceLevel 250, VT 800 Good | admitted; ServiceLevel 255, VT 800 Good |
|
|
| SignAndEncrypt + Sign, 65-char valid key (production form) | **`BadIdentityTokenInvalid`** (#880, expected denial) | admitted, Good |
|
|
| wrong secret (Sign + SignAndEncrypt), unknown id, disabled key, 54-char | `BadIdentityTokenRejected` | `BadIdentityTokenRejected` |
|
|
| wrong secret / unknown id, 65-char | `BadIdentityTokenInvalid` | `BadIdentityTokenRejected` |
|
|
| anonymous | denied (`BadIdentityTokenInvalid`; not advertised) | denied |
|
|
| untrusted client, Sign + SignAndEncrypt | `BadSecurityChecksFailed` | `BadSecurityChecksFailed` |
|
|
| untrusted server (client trust list emptied) | `BadCertificateUntrusted` | `BadCertificateUntrusted` |
|
|
| actual CLI read `ServerStatus`, SignAndEncrypt + API key | SDK 1.5 CLI 65-char Good; SDK 2 CLI 54-char Good | SDK 1.5 CLI 65-char Good; SDK 2 CLI 54- and 65-char Good |
|
|
| *finding:* SDK 2 CLI with the 65-char key | `BadIdentityTokenInvalid` (#880) | Good |
|
|
| *finding:* advertised endpoints | **None/None + Sign + SignAndEncrypt** (#879) | same |
|
|
| *finding:* session over the None endpoint, valid key | admitted (#879) | admitted |
|
|
|
|
## Step 3: ScriptWorker (`isolation-*.json`)
|
|
|
|
The out-of-process runner is opt-in, and no deployed node configures it; vd03 runs InProcess. On windev:
|
|
|
|
1. **Package.** The candidate's `script-worker\` holds the self-contained worker apphost and is 275 files.
|
|
2. **The repository installer fails** (#883). This is the step #794 requires.
|
|
- **pwsh 7:** `-WhatIf` lists 7 steps and changes nothing. The real run creates the anchor service and the PE cache, then
|
|
throws `DirectoryInfo … GetAccessControl`.
|
|
- **Windows PowerShell 5.1:** it throws `property 'Count' cannot be found` (StrictMode) at PeEntriesDir.
|
|
- **`-Uninstall`** fails the same way, so the partial state could not be rolled back with the tool.
|
|
3. **With the #883 patch** (`isolation-installer-883.diff`):
|
|
- **Install and re-apply.** Apply ran under pwsh, then re-apply under 5.1 left all 7 steps `unchanged`.
|
|
- **Anchor service.** `OtOpcUaScriptWorker` is DISABLED, as `NT SERVICE\OtOpcUaScriptWorker`, SID
|
|
`S-1-5-80-1846135827-…-3023753135` (matches #794).
|
|
- **Folder ACLs.** The install folder has `(OI)(CI)(DENY)(S,RD)` for the worker and `script-worker\` has `(RX)`.
|
|
- **PE cache.** It is protected with SYSTEM, Administrators and the host account at F; `pe\` adds the worker at M.
|
|
- **Firewall.** `OtOpcUa-ScriptWorker-BlockOutbound` is Outbound/Block/Any on the worker exe.
|
|
- **Service environment.** It gained exactly `Scripting__Worker__Identity` and `Scripting__Worker__PeCacheDir`, plus the
|
|
gate's `Scripting__Runner=OutOfProcess`; the other entries were kept.
|
|
4. **Runtime under the real service (c3).**
|
|
- **Worker process.** `ScriptWorker[0] started` as the child of the host, run as a restricted copy of
|
|
`NT SERVICE\OtOpcUaHost` (`IsTokenRestricted=true`, 1 privilege).
|
|
- **Window-station grant.** It works under a real service session: coreclr loaded, which windev over SSH could not prove
|
|
for #794.
|
|
- **Pipe ACL.** The `otopcua-sw-*` pipe refuses the elevated administrator (`UnauthorizedAccessException`).
|
|
- **Health.** `script-runner` reports Healthy "Out-of-process script runner ready", with the PE cache on and 1 `pe\` entry.
|
|
- **Virtual tag.** It evaluated out of process to 800 Good.
|
|
- **Stop.** No worker survived the stop.
|
|
- **Not exercised:** dropped-packet firewall logging.
|
|
5. **Uninstall.** `-Uninstall` under 5.1 and then again under pwsh left every step `unchanged`. The anchor and the firewall rule
|
|
were gone, the environment digest was back to `fd534f8d…`, and the install-folder ACL was back to its original state.
|
|
|
|
## Upgrade and rollback mechanics (`upgrade.json`, `rollback.json`)
|
|
|
|
**Upgrade**, with the service stopped:
|
|
- `reg export` of the service key, and a `robocopy /E /COPYALL /DCOPY:DAT` backup (1386 files).
|
|
- The candidate was staged in a separate folder and given the install-folder DACL.
|
|
- The box's `appsettings*.json` were carried over; they are byte-identical to the candidate publisher's.
|
|
- `pki\`, `data\` and `logs\` were carried over with `/COPYALL`.
|
|
- The folders were swapped with `Move-Item`. The service `PathName` was unchanged.
|
|
|
|
**Rollback**, with the service stopped:
|
|
- The candidate was moved to quarantine and the known-good folder was moved back.
|
|
- The current `pki\` and `data\` were carried back with `/COPYALL`. No data rollback: the candidate-written LocalDb opened
|
|
cleanly in the known-good build, as on vd03.
|
|
|
|
## Cleanup
|
|
|
|
These were removed:
|
|
- the service, with `sc.exe delete`;
|
|
- the `OtOpcUaScriptWorker` anchor and its firewall rule;
|
|
- `C:\OtOpcUa800`;
|
|
- the gate scripts in the profile.
|
|
|
|
Do **not** use `Uninstall-Services.ps1` on windev: it also stops and deletes `OtOpcUaWonderwareHistorian`.
|
|
|
|
After cleanup:
|
|
- `%LOCALAPPDATA%\OtOpcUaClient` was restored from its backup (its `settings.json` and `logs\` are from 2026-05-24).
|
|
- The firewall rule count is back to 876.
|
|
- MxAccessGw is Running with `:5130/health/ready` 200, and OtOpcUaWonderwareHistorian is Running.
|
|
|
|
**Residual:** the virtual-account profile `C:\Windows\ServiceProfiles\OtOpcUaHost`, with its ProfileList entry, stays loaded
|
|
until the next reboot. Remove it after that with
|
|
`Get-CimInstance Win32_UserProfile | ? SID -eq 'S-1-5-80-3928861625-3486739760-3962757976-1980707066-3667558059' | Remove-CimInstance`.
|
|
|
|
`gate-run.log` lists every phase invocation and its exit code. The first `Prepare` line reported exit 0 even though the wrapper
|
|
had failed (the remote `$PSStyle` quoting). That attempt stopped before creating the service, and the phase was re-run.
|
|
|
|
## Real-world cross-reference
|
|
|
|
wonder-app-vd03 (DEV) has run master `5fb4b4db` on SDK 2.0.1 since 2026-11-12. It reused the app cert `A873DF51…` and reports
|
|
ServiceLevel 255 with Good reads; see #782 and `docs/deploy/vd03-upgrade-b8b4c18f-to-master.md`. That record is consistent
|
|
with this run but is not a controlled install/rollback.
|