Files
lmxopcua/docs/deploy/2026-11-13-vd03-fea7de0a-to-6c4d2990.md
T
Joseph Doherty 2cb8a2c317
v2-ci / unit-tests (push) Has been cancelled
fix(install): worker isolation survives a folder swap: recorded settings, -Reapply, -Verify, swap script (#901)
An upgrade that swaps the install folder dropped the explicit ScriptWorker isolation ACEs on script-worker\, the
install folder and a copied-back pki\ (vd03 2026-11-13). Re-applying needed the operator to repeat the arguments.

- Install-ScriptWorkerIsolation.ps1 records its effective settings (InstallRoot, HostServiceAccount, PeCacheDir,
  HostServiceName, ConfigureHostService) beside DenyPaths under ...\Services\OtOpcUaScriptWorker\Parameters (new plan
  step SettingsRecord) and locks that key to SYSTEM + Administrators (protected DACL; no secrets recorded).
- -Reapply: re-apply from the record, no arguments; prints a summary line (changed/unchanged).
- -Verify: check only, [ok]/[MISSING] per step, exit 1 when anything is missing. -Uninstall also works from the record.
  A node isolated by a pre-#901 installer (DenyPaths only) must run the full-argument install once; both modes say so.
- Invoke-OtOpcUaInstallSwap.ps1: preflight -> stop -> move live/candidate -> carry appsettings/pki/logs ->
  -Reapply -> -Verify -> start; a failed -Verify leaves the service stopped. Also the rollback (-NoCarry).
- Tests: dry-run harness 101/101 (record + argument rules; record-key SDDL mutation turns 2 red). The Windows test adds
  swap -> Verify fails (exactly the 3 lost ACEs) -> Reapply (3 changed) -> Verify ok -> Reapply 12/12 unchanged, PE-cache
  and firewall negative controls, the swap script, and the pre-#901 refusal; 52/52 on windev under 5.1 and 7, with a
  scratch local host account and with SYSTEM. Evidence: scripts/validation/opcua-windows-service/evidence/2026-11-13-windev-901-reapply-verify.
- Docs: ServiceHosting isolation section; vd03 pack (swap step, rollback, post-deploy gate, one-time full-argument run),
  the 2026-11-13 vd03 record, and the QA pack (QA swap preceded the isolation install; ACLs checked intact over SMB on
  vq03/vq04 2026-11-13; next-deploy procedure).

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 02:06:10 -05:00

7.9 KiB
Raw Blame History

DEV wonder-app-vd03 upgrade fea7de0a → master 6c4d2990 (2026-11-13)

Status: EXECUTED 2026-11-13 06:25–06:26Z. vd03 runs 6c4d2990 (6c4d29902e18d7560ce1ff5607d524a58f674440). Owner-approved as part of the #700 hybrid decision: deploy a master build carrying the #899/#898 read-only historian binding, then run the remaining #700 checks on the deployed endpoint (record: ../plans/evidence/2026-11-13-historian-700-vd03/vd03-endpoint-gate.md, second run).

Channel: WinRM + SMB from the Mac (the winrm skill), used only for the on-box steps of this deploy (preflight/backup, swap, the worker-isolation re-apply). Health, OPC UA and AdminUI checks ran from the Mac over HTTP / opc.tcp. Layout and scripts as §(h) of vd03-upgrade-b8b4c18f-to-master.md and the DEV section of qa-upgrade-b8b4c18f-to-fea7de0a.md.

Pinned SHA. The deploy is pinned to 6c4d2990. origin/master moved to efaf3435 (#900) during the run; that commit is not deployed.

Build

  • dotnet publish src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj -c Release -r win-x64 --self-contained true on the Mac, in an isolated worktree at 6c4d2990: 1396 files, OtOpcUa.Host.dll ProductVersion 1.0.0+6c4d29902e18d7560ce1ff5607d524a58f674440, self-contained script-worker\.
  • deps.json: Telemetry + Telemetry.Serilog 0.3.1 (not 0.3.0, scadaproj#56), LocalDb 0.3.7, MxGateway.Client 0.4.0, HistorianGateway.Client 1.0.0, Auth 0.4.1, Secrets 0.6.2, Health 0.3.0, Theme 0.5.0.
  • Zip otopcua-6c4d2990.zip, SHA-256 4914D083…FC0850 (full hash in the staged .sha256), checked on the box before expanding.
  • fea7de0a..6c4d2990: 37 commits, no ConfigDb migration (last stays 20261022102459_EnableSnapshotIsolation), no new required setting. Client-visible on vd03: TimeAverage/Total advertised and served again (#694, 53978ca6); a Boolean node refuses TimeAverage/Total up front (#897); Telemetry 0.3 sends logs over OTLP too, because vd03 has OtOpcUa__Telemetry__Exporter=Otlp + OtlpEndpoint=http://127.0.0.1:4317 (scheme present, so the stricter 0.3 endpoint check passes).

Preflight (stop conditions)

Check Result
#898 upgrade hazard: historized Float/Double/Byte/SByte tags that the first start would EnsureTags None. Anonymous walk of the deployed address space (raw + UNS, probe-899 attrs): 24 variables, 0 with Historizing=true. /deployments "In sync" at dc0af731 (rev fa9a2e28…), so draft = deployed.
Continuous / alarm historization Off: ContinuousHistorization:Enabled=false, AlarmHistorian:Enabled=false in appsettings.json; no *__Enabled environment override. ServerHistorian__Enabled=true (env). ServerHistorian__ProvisionTags absent (default CreateOrUpdate); with 0 historized tags nothing can be provisioned.
Shared-lib pins on master Telemetry already 0.3.1 at 6c4d2990 (f3178c5f); no bump needed.
Migrations in range None. ConfigDb at 9 migrations, last 20261022102459_EnableSnapshotIsolation, snapshot isolation ON.
Service env 28 variables (names checked, values not printed): +2 since the fea7de0a record (OtOpcUa__Telemetry__Exporter, OtOpcUa__Telemetry__OtlpEndpoint). Scripting__Runner absent → InProcess (unchanged).
Disk E: 179 GB free.

Execution

Staged in E:\ApiInstall\_stage-otopcua-6c4d2990\ (prepare.ps1, swap.ps1, rollback.ps1, reisolate.ps1, zip + .sha256).

  • Prepare (service up), prepare.ps1: registry export of the service key to E:\ApiInstall\OtOpcUa-backups\pre-20261113-otopcua-6c4d2990\OtOpcUaHost.reg (DACL SYSTEM + Administrators only); zip hash ok, expanded to E:\ApiInstall\OtOpcUa.new-6c4d2990 (1396 files, ProductVersion checked); BACKUP DATABASE [OtOpcUaConfig] … COPY_ONLY, CHECKSUM → OtOpcUaConfig-pre-20261113-otopcua-6c4d2990.bak (instance default backup directory, 1913 pages).
  • Swap, swap.ps1 (06:25:50Z): stop (6 s), OtOpcUa-data robocopied to E:\ApiInstall\OtOpcUa-backups\pre-20261113-otopcua-6c4d2990\OtOpcUa-data, previous install moved to E:\ApiInstall\OtOpcUa.bak.20261113-pre-6c4d2990, box appsettings*, pki\, logs\ carried, start, /health/ready 200. Downtime about 15 s (stop → ready).
  • Worker isolation re-applied, reisolate.ps1 (lmxopcua#891 arguments, no -ConfigureHostService): the folder swap had dropped three ACEs, as docs/ServiceHosting.md warns — WorkerFolderAcl (script-worker\), InstallRootDeny (E:\ApiInstall\OtOpcUa) and SensitiveDirDeny (…\OtOpcUa\pki) applied as changed; the immediate re-apply reported every step unchanged. The runner stays InProcess; no restart needed. The same 6 pre-existing "protected DACL" warnings under E:\psrem-stage as before. Follow-up: lmxopcua#901. The re-apply was a manual step with recorded arguments. From #901 the installer records its settings and has -Reapply / -Verify, and scripts/install/Invoke-OtOpcUaInstallSwap.ps1 runs both after the swap (service left stopped if -Verify fails). Next vd03 deploy: run the full-argument install once to write the record, then swap with the script and gate on -Verify (vd03-upgrade-b8b4c18f-to-master.md §e/§f).

Verification

  • /healthz, /health/ready (configdb, akka Up, opcua-endpoint, script runner, deploy coordinator) and /health/active: 200.
  • Running version = target: the live E:\ApiInstall\OtOpcUa\OtOpcUa.Host.dll (SMB copy) carries 1.0.0+6c4d29902e18…; its SHA256 prefix 6d1f354f674e equals the published build's.
  • ServiceLevel 250 (unchanged).
  • FOCAS: raw subtree 22 variables, 21 Good, Program/BlockCount BadNoData — the same set as before the upgrade.
  • Boot log: roles=[admin,driver], Historian gateway probe: the gateway is reachable, 12 aggregates advertised (Average, TimeAverage, Total, …), applied plan (PureAdd, added=36 …), raw subtree materialised (… tags=22 … failed=0), the #815 preflight line, no provisioning line. Pre-existing noise only (Akka auto-down deprecation, early ClusterSingletonProxy warnings, mesh PSK absent, FQDN-vs-certificate ERR from clients dialling the FQDN).
  • HistorianGateway /metrics across the upgrade: Probe only, no EnsureTags (see the evidence file).
  • Script runner unchanged (InProcess, no Scripting__* env).

#700 checks (after the upgrade)

Boolean + Int32 typing and native TimeAverage/Total passed through the deployed endpoint with historianReadOnly: true bindings, zero historian writes, and the configuration restored byte-identical (rev fa9a2e28…, deployment 595cb909). Details: evidence, second run.

Rollback

E:\ApiInstall\_stage-otopcua-6c4d2990\rollback.ps1: stop; move the live folder to E:\ApiInstall\OtOpcUa.failed-6c4d2990; restore E:\ApiInstall\OtOpcUa.bak.20261113-pre-6c4d2990 (fea7de0a); restore OtOpcUa-data from the pre-deploy backup; start; re-apply the worker isolation. (From #901 the re-apply comes before the start, as -Reapply then -Verify: Invoke-OtOpcUaInstallSwap.ps1 -Candidate <bak> -PreviousInstall <failed> -NoCarry does it.) No schema undo (no migration in range). The ConfigDb .bak is there if the configuration itself must go back.

Observed (not deploy blockers)

  • Closing an AdminUI tab that had the script editor open logs [ERR] Unhandled exception in circuit … TaskCanceledException from MonacoEditor.DisposeAsync → SafeInvokeAsync (the JS runtime is already gone when the circuit disposes). Cosmetic.
  • /deployments showed the "In sync" chip on first render after a draft change; it switched to "Configuration drift" after the next interaction.