Executed 2026-11-13 12:27-12:48Z; target moved from the pack's 47f0b98c to the tip 76e73e7d
(HG Client 1.1.0, behaviour unchanged; note added to the pack).
- vd03: ~14 s downtime (was ~101 s; #905 window 5.7 s), SL 250, FOCAS 21/22, Mode Unset,
HG Probe 494 -> 536 with no write RPCs (added gate for #900).
- QA: pair outage 36.5 s (was ~91 s), 250/240, both failover legs, no failback, sync
connected, canary Good. #906: vq04 (sync server) shutdown -> Leaving 30.0 s on c9640f5c,
21 ms on 76e73e7d; peer at 250 in 0.47 s (leg 1: 0.36 s).
- Deviation: the first Move-Item of app\ failed "file in use" right after the stop on both
QA nodes; a retry with the service stopped succeeded (lmxopcua#908).
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Range, settings (no migration; OpcUa:Redundancy:Mode unset everywhere), read-only
ConfigDb/SMB/OPC UA preflight on both tiers, #905 scoped swap with -NoStart on both
tiers, #906 failover timing gate (stop the sync-server node as Primary), Unset
metadata gate, rollback with the vd03 #903 drift note. Not executed.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Root cause: the protected-DACL coverage scan, not the ACL writes. Both -Reapply and
-Verify walked every denied folder (all of E:\ApiInstall on vd03, nested roots again)
with Get-ChildItem -Recurse + Get-Acl per item + a pipeline per item for -Skip, about
3.4 ms an item on PS 5.1, all while the service was stopped. An unchanged deny is read,
not rewritten; the re-applied install-folder deny propagates over ~1,400 files in 0.5 s.
- Find-OtOpcUaDenyCoverageGap: .NET enumeration, -Skip folders pruned, DACL-only
GetAccessControl(Access) (about 0.15 ms an item); reparse points not followed.
- Get-OtOpcUaCoverageWalkRoots: each denied folder walked once (outermost only).
- Install-ScriptWorkerIsolation.ps1 -Scope All|InstallRoot|Outside (with -Reapply,
-Verify, -PlanOnly). InstallRoot = the folder ACEs a swap drops; Outside = the rest,
including the firewall rule (its query costs 3-5 s; its exe path does not change).
The two partition the plan. Prints the scan time.
- Invoke-OtOpcUaInstallSwap.ps1: -Reapply/-Verify -Scope Outside before the stop (a
failure stops before anything moves), -Scope InstallRoot inside the window; preflight
checks the candidate carries the worker exe; prints the stopped window and its parts.
-NoStart, -NoCarry and the pre-#901 refusal unchanged.
windev, vd03-like tree (17,851 items): stopped window 153.1 s -> 6.0 s (PS 5.1),
101.6 s -> 6.2 s (PS 7); the Outside part takes ~23 s with the old build serving.
Dry-run harness 101 -> 123 passed; Windows test 72 passed on PS 5.1 and PS 7.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
DEV vd03 6c4d2990 -> c9640f5c (~101 s), QA pair fea7de0a -> c9640f5c (~91 s pair
outage). #901 settings record written once per node, swaps gated on -Verify,
all gates pass (SL 250 / 250-240, failover no failback, LocalDb sync, canary,
no historian writes). #903 edit-and-revert leaves drift: RowVersion is in the
revision hash.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
No migrations in either range; no #834/#898/#899/#897/#896 findings on DEV or QA.
QA isolation record holds 12 deny paths (OTel collector added 2026-11-12), not 11.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
An upgrade that swaps the install folder dropped the explicit ScriptWorker isolation ACEs on script-worker\, the
install folder and a copied-back pki\ (vd03 2026-11-13). Re-applying needed the operator to repeat the arguments.
- Install-ScriptWorkerIsolation.ps1 records its effective settings (InstallRoot, HostServiceAccount, PeCacheDir,
HostServiceName, ConfigureHostService) beside DenyPaths under ...\Services\OtOpcUaScriptWorker\Parameters (new plan
step SettingsRecord) and locks that key to SYSTEM + Administrators (protected DACL; no secrets recorded).
- -Reapply: re-apply from the record, no arguments; prints a summary line (changed/unchanged).
- -Verify: check only, [ok]/[MISSING] per step, exit 1 when anything is missing. -Uninstall also works from the record.
A node isolated by a pre-#901 installer (DenyPaths only) must run the full-argument install once; both modes say so.
- Invoke-OtOpcUaInstallSwap.ps1: preflight -> stop -> move live/candidate -> carry appsettings/pki/logs ->
-Reapply -> -Verify -> start; a failed -Verify leaves the service stopped. Also the rollback (-NoCarry).
- Tests: dry-run harness 101/101 (record + argument rules; record-key SDDL mutation turns 2 red). The Windows test adds
swap -> Verify fails (exactly the 3 lost ACEs) -> Reapply (3 changed) -> Verify ok -> Reapply 12/12 unchanged, PE-cache
and firewall negative controls, the swap script, and the pre-#901 refusal; 52/52 on windev under 5.1 and 7, with a
scratch local host account and with SYSTEM. Evidence: scripts/validation/opcua-windows-service/evidence/2026-11-13-windev-901-reapply-verify.
- Docs: ServiceHosting isolation section; vd03 pack (swap step, rollback, post-deploy gate, one-time full-argument run),
the 2026-11-13 vd03 record, and the QA pack (QA swap preceded the isolation install; ACLs checked intact over SMB on
vq03/vq04 2026-11-13; next-deploy procedure).
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
vd03 runs master 6c4d2990 (#899/#898 read-only historian bindings), self-contained
win-x64, ~15 s downtime, ServiceLevel 250, FOCAS 21/22 Good as before, runner InProcess,
worker isolation re-applied after the folder swap. #898 preflight: 0 historized tags.
#700 second run on the deployed endpoint with historianReadOnly bindings: Boolean
(SysPulse) and Int32 (SysTimeSec) Raw/AtTime typed as the node DataType; native
TimeAverage/Total 9.5/29.5/49.5 and 190/590/990 (= in-process reference), WTC1
all-NULL intervals BadNoData. HistorianGateway /metrics: no EnsureTags/WriteLiveValues
or any write. Temp config + key removed, config back to rev fa9a2e28.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
- #890: installer at f75ee7d0 re-applied on both nodes with -DenyPaths root, pki, data and
C:\ProgramData\ZB\kek (the Secrets KEK sits outside the install root); second apply 12/12 unchanged.
- Script-runner canary authored in the AdminUI and deployed (Calculation tag + VirtualTag, 1 s timers):
Good and advancing on both endpoints; kept on QA as a runner monitor.
- #794 step 5: the live worker token is restricted (SeChangeNotify only; identity/Everyone/Users/RESTRICTED)
and is denied read and write on pki, data, app, logs and the KEKs; network sockets none. Finding #891.
- #780: QA runaway drill passed (killed at the 4000 ms hard deadline twice, quarantined, siblings Good,
ServiceLevel 250/240 unchanged); drill removed, redeploy back to the canary revision 82dd0598.
- #780 P4 recorded: InProcess stays the default and the rollback; StuckOrphanQuarantineFactor documented
as in-process only.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
The installer denied NT SERVICE\OtOpcUaScriptWorker only on -InstallRoot. On the wonder layout the node's keys and
secrets sit beside it (E:\ApiInstall\OtOpcUa\pki with the application certificate's private key, data\ with
secrets.db and LocalDb), inheriting BUILTIN\Users:RX. Users is a restricting SID of the worker token, so a worker
could read them.
- Install-ScriptWorkerIsolation.ps1 -DenyPaths <dir>,...: each folder gets the same inherited deny as InstallRoot
(new SensitiveDirDeny steps). Refuses drive/share roots, script-worker\ and the PE cache; folders must exist;
a comma/semicolon string works for powershell -File.
- The deny is read AND write (0xD0157: ReadData, WriteData, AppendData, WriteEA, DeleteChild, WriteAttributes,
Delete, WRITE_DAC, WRITE_OWNER). Traverse/Execute, ReadAttributes, ReadControl and Synchronize stay so the
loader still reaches script-worker\. The write half stops a worker planting a cert in pki\trusted\certs where
a data drive lets Users create files in subfolders.
- The list is recorded under the anchor service's key (Parameters\DenyPaths, step DenyPathRecord): a re-run
removes the deny from folders dropped from the list, -Uninstall removes every recorded one without repeating
-DenyPaths. -WhatIf/-PlanOnly print every step.
- After an apply, WARN for each protected-DACL item under a denied folder that the inherited deny cannot reach;
without -DenyPaths, WARN for pki/data/logs/backup/config/secrets siblings of InstallRoot.
Tests:
- Dry-run harness 46 -> 86 assertions (QA layout plan, mask bits, refusals, stale removal, record-driven
rollback, host account never denied). Mutations (no sibling deny, deny->allow, mask back to ReadData, no
stale removal, rollback ignoring the record, script-worker not refused) each turn it red.
- Test-ScriptWorkerIsolation.Windows.ps1 (Windows PowerShell 5.1, windev): 34/34, incl. -DenyPaths apply,
11/11 unchanged re-apply, shorter-list re-apply, uninstall from the record, second uninstall unchanged.
- New InstallerLayoutIsolationTests (OTOPCUA_ISOLATION_INSTALLER_E2E=1, elevated Windows): a scratch wonder
layout under the real restricted worker token. Before install and after uninstall the token reads pki/data/
logs/backup/appsettings and plants a trust-list file; installed it can do neither, still reads script-worker\,
a real worker launched from the layout runs a script, and the host still reads everything. Mutating the
-DenyPaths denies to a no-op mask turns it red on pki\own\private\otopcua.pfx (reproduces #890).
QA vq03/vq04 are NOT re-applied; the command is in docs/deploy/qa-upgrade-b8b4c18f-to-fea7de0a.md.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Read-only ConfigDb preflight (scripts/preflight/vd03-preupgrade-b8b4c18f-configdb-checks.sql)
covering every deploy gate added since the deployed build: VirtualTag/Calculation effective
triggers (#783), VT->VT reads and VirtualTagDependencyCycle (#791), newly forbidden script
APIs (#792, E2), credential-reference namespace / other-instance / reserved-name rules
(#815#835#836#848#849#850), MTConnectMultipleDevices, Sql query/write gates (#781),
Heidenhain, UNS effective-name collisions and the FOCAS renames/value changes. Verified on a
throwaway b8b4c18f-schema database (every check fires on seeded rows) and clean against the
docker-dev config.
Upgrade notes (docs/deploy/vd03-upgrade-b8b4c18f-to-master.md): the EnableSnapshotIsolation
migration and how to apply it by hand, the env-var audit (Security:ApiKeys:Pepper is now
required and absent on the box; JWT key validated at start), SDK 2.0.1 / CA KeyUsage, the
ScadaBridge-visible changes, swap/rollback and post-deploy checks.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Joseph Doherty