fix(install): cut the swap's stopped window to the swapped-in folder (#905)
v2-ci / unit-tests (push) Has been cancelled
v2-ci / unit-tests (push) Has been cancelled
Root cause: the protected-DACL coverage scan, not the ACL writes. Both -Reapply and -Verify walked every denied folder (all of E:\ApiInstall on vd03, nested roots again) with Get-ChildItem -Recurse + Get-Acl per item + a pipeline per item for -Skip, about 3.4 ms an item on PS 5.1, all while the service was stopped. An unchanged deny is read, not rewritten; the re-applied install-folder deny propagates over ~1,400 files in 0.5 s. - Find-OtOpcUaDenyCoverageGap: .NET enumeration, -Skip folders pruned, DACL-only GetAccessControl(Access) (about 0.15 ms an item); reparse points not followed. - Get-OtOpcUaCoverageWalkRoots: each denied folder walked once (outermost only). - Install-ScriptWorkerIsolation.ps1 -Scope All|InstallRoot|Outside (with -Reapply, -Verify, -PlanOnly). InstallRoot = the folder ACEs a swap drops; Outside = the rest, including the firewall rule (its query costs 3-5 s; its exe path does not change). The two partition the plan. Prints the scan time. - Invoke-OtOpcUaInstallSwap.ps1: -Reapply/-Verify -Scope Outside before the stop (a failure stops before anything moves), -Scope InstallRoot inside the window; preflight checks the candidate carries the worker exe; prints the stopped window and its parts. -NoStart, -NoCarry and the pre-#901 refusal unchanged. windev, vd03-like tree (17,851 items): stopped window 153.1 s -> 6.0 s (PS 5.1), 101.6 s -> 6.2 s (PS 7); the Outside part takes ~23 s with the old build serving. Dry-run harness 101 -> 123 passed; Windows test 72 passed on PS 5.1 and PS 7. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
+26
-5
@@ -138,25 +138,46 @@ Two argument-free modes use the record:
|
||||
|---|---|
|
||||
| `Install-ScriptWorkerIsolation.ps1 -Reapply` | Re-applies with the recorded settings; one `[changed]`/`[unchanged]` line per step and a `summary:` line. After a swap: the three folder ACEs `changed`, the rest `unchanged`; a second run all `unchanged`. |
|
||||
| `Install-ScriptWorkerIsolation.ps1 -Verify` | Check only. `[ok]`/`[MISSING]` per step (anchor service, record + key ACL, PE cache ACL, `pe\`, `script-worker\`, install-folder deny, every `-DenyPaths` deny, firewall rule and the worker exe it names, the host env entries when recorded). **Exit 1** with `VERIFY FAILED: …` when anything is missing, exit 0 with `VERIFY OK`. Protected-DACL coverage gaps are WARNed, not failed. |
|
||||
| `… -Reapply -Scope InstallRoot` / `-Verify -Scope InstallRoot` | (lmxopcua#905) Only what a swap can undo: the ACEs on the install folder, `script-worker\` and a denied folder inside it (vd03's `pki\`). The coverage scan walks the install folder only. |
|
||||
| `… -Reapply -Scope Outside` / `-Verify -Scope Outside` | Every other step (anchor service, records, PE cache, denies outside the install folder, firewall rule, host env). The firewall rule names the worker exe by a path a swap does not change, and its query costs 3–5 s; the swap checks in its preflight that the candidate carries the exe. The scan walks the denied folders and skips the install folder. `InstallRoot` + `Outside` = the full plan. |
|
||||
|
||||
Both refuse to run without a full record. A node isolated by an installer older than #901 recorded only `DenyPaths`:
|
||||
run the install once with its full arguments (the per-box deploy record lists them) to write the record.
|
||||
|
||||
`scripts/install/Invoke-OtOpcUaInstallSwap.ps1` is the swap with the gate built in: preflight (candidate carries
|
||||
`script-worker\`, `PreviousInstall` is fresh and on the same volume, isolation record present when isolation is
|
||||
installed, record `InstallRoot` equals `-InstallRoot`) → stop → move live → previous, candidate → live → copy `-Carry`
|
||||
items back (default `appsettings*.json`, `pki`, `logs`; QA: `'appsettings*.json','logs'`; rollback: `-NoCarry`) →
|
||||
`-Reapply` → `-Verify` → start (`-NoStart` for a pair window). A failed `-Verify` leaves the service **stopped**. Backups,
|
||||
installed, record `InstallRoot` equals `-InstallRoot`) → `-Reapply -Scope Outside` + `-Verify -Scope Outside` **while
|
||||
the old build still serves** (a failure stops here: nothing stopped or moved) → stop → move live → previous, candidate →
|
||||
live → copy `-Carry` items back (default `appsettings*.json`, `pki`, `logs`; QA: `'appsettings*.json','logs'`; rollback:
|
||||
`-NoCarry`) → `-Reapply -Scope InstallRoot` → `-Verify -Scope InstallRoot` → start (`-NoStart` for a pair window). The
|
||||
two scoped gates together check every install step, so the new build never starts unverified; a failed post-swap
|
||||
`-Verify` leaves the service **stopped**. The script prints the stopped window and its parts
|
||||
(`stopped window (stop issued -> start issued): 6.0 s [stop …, move 0.0 s, carry 0.1 s, isolation 5.8 s]`). Backups,
|
||||
`data\` restore, migrations and health checks stay in the deploy's own steps. Every deploy pack that swaps the folder runs
|
||||
it (or `-Reapply` + `-Verify` right after its own swap), and `-Verify` is a post-deploy gate.
|
||||
`Test-ScriptWorkerIsolation.Windows.ps1` exercises the whole sequence (swap → `-Verify` fails → `-Reapply` → `-Verify`
|
||||
passes → `-Reapply` unchanged → the swap script) on a scratch tree.
|
||||
passes → `-Reapply` unchanged → the swap script → the scoped modes, the pre-stop gate, `-NoStart`, a `-NoCarry` rollback)
|
||||
on a scratch tree; `-TimingFiles N` times the swap on a vd03-like tree (`-ScriptsDir` for a before/after).
|
||||
|
||||
**Expected stopped window (lmxopcua#905).** Before #905 the swap ran the full `-Reapply` and `-Verify` while the service
|
||||
was stopped, and each of them walked every denied folder for the protected-DACL coverage scan with `Get-Acl` per item
|
||||
(about 3.4 ms an item on Windows PowerShell 5.1). vd03 denies all of `E:\ApiInstall`, so that was most of its 101 s outage
|
||||
on 2026-11-13 (QA: 50–60 s per node). The scan now reads only the DACL through .NET (about 20× faster per item), walks
|
||||
each denied folder once (a folder nested in another denied folder is no longer walked again), and inside the stopped
|
||||
window walks only the swapped-in install folder. Setting the ACEs themselves was never the cost: an unchanged deny is
|
||||
read, not rewritten, and the re-applied install-folder deny propagates over about 1,400 files in about 0.5 s. The firewall
|
||||
query (3–5 s a run) moved to the Outside scope. Measured on windev with a vd03-like tree (17,851 items under the denied
|
||||
folder, 48 protected files): the swap's stopped window went from **153.1 s to 6.0 s** on Windows PowerShell 5.1 and from
|
||||
**101.6 s to 6.2 s** on PowerShell 7 (`Test-ScriptWorkerIsolation.Windows.ps1 -TimingFiles 17000`). Of the 6 s, about
|
||||
1.5 s per installer run is process start + script parse + module import. The Outside part (full scan, firewall,
|
||||
anchor-service query) takes about 23 s **before** the stop, with the old build serving. Add the stop, the deploy's own
|
||||
steps inside the window (a `data\` backup) and the host start-up to `/health/ready`.
|
||||
|
||||
## Install / Uninstall
|
||||
|
||||
- `scripts/install/Install-Services.ps1 -Roles admin,driver` — installs `OtOpcUaHost`.
|
||||
- `scripts/install/Install-ScriptWorkerIsolation.ps1` — the ScriptWorker identity, ACLs and firewall rule (above); only for `Scripting:Runner` ≠ `InProcess`.
|
||||
- `scripts/install/Invoke-OtOpcUaInstallSwap.ps1` — upgrade/rollback folder swap with the isolation `-Reapply` + `-Verify` gate built in (lmxopcua#901, above).
|
||||
- `scripts/install/Invoke-OtOpcUaInstallSwap.ps1` — upgrade/rollback folder swap with the isolation `-Reapply` + `-Verify` gate built in (lmxopcua#901), scoped so only the swapped-in folder is handled while the service is stopped (lmxopcua#905, above).
|
||||
- `scripts/install/Uninstall-Services.ps1` — stops + removes the host service. (The historian backend is the external HistorianGateway — not installed/removed by these scripts.)
|
||||
|
||||
## Logging
|
||||
|
||||
@@ -331,3 +331,13 @@ rollback.
|
||||
| vd03 | `fea7de0a` about **10 s** (stop 2 s); `6c4d2990` about **15 s** (stop 6 s), stop → `/health/ready` | **about 20–40 s.** The data robocopy and the in-script `-Reapply`/`-Verify` now run while the service is stopped. The step-2 record write happens with the service up. |
|
||||
| QA pair | `fea7de0a` window **about 2 min** pair outage (16:48:13 → 16:50:14), of which about 70 s was the isolation install + uninstall drill. The first node took 39 s to ready because it waits for its peer; the second took 8 s. | **about 1–1.5 min** pair outage: stops about 1–2 s each, backup + swap + `-Reapply`/`-Verify` per node, then about 40 s + 8 s to ready. One-node restarts (the OTel record) took up to 32 s to stop a Secondary with its peer up. Stopping both together avoids that graceful-leave wait. |
|
||||
| QA failover gate | 30 s graceful leave, then the new Primary at 250 on its first sample | same |
|
||||
|
||||
**After lmxopcua#905 (for the next pack that uses `Invoke-OtOpcUaInstallSwap.ps1`).** The 101 s (vd03) and 50–60 s per
|
||||
QA node were the protected-DACL coverage scan (`Get-Acl` per item over every denied folder, in both `-Reapply` and
|
||||
`-Verify`, while stopped), not the ACL writes. The swap now runs `-Reapply`/`-Verify -Scope Outside` (the full scan
|
||||
included) **before** the stop and only `-Scope InstallRoot` inside the window. windev, a vd03-like tree of 17,851 items
|
||||
under the denied folder: stopped window **153.1 s → 6.0 s** (Windows PowerShell 5.1) and **101.6 s → 6.2 s** (PowerShell 7);
|
||||
the pre-stop part takes about 23 s with the old build still serving. Estimate for the next vd03 swap: stop (2–6 s) +
|
||||
the `data\` backup + about 6 s + start-up to `/health/ready` (about 8–10 s), i.e. **about 20–30 s**. QA: about 10 s of
|
||||
swap per node inside the pair outage instead of 50–60 s, and the Secondary-first order (deviation 3) is no longer needed
|
||||
to hide it. The script prints `stopped window (…): N s [stop, move, carry, isolation]`; record that line.
|
||||
|
||||
@@ -130,6 +130,8 @@ old Primary first.
|
||||
|
||||
1. vd03 downtime about 101 s against an estimate of 20–40 s, because the coverage walk runs inside `-Reapply`/`-Verify`
|
||||
while the service is stopped. QA swaps took about 50–60 s per node for the same reason.
|
||||
→ lmxopcua#905: fixed (scan about 20× faster per item, walked once, and only the swapped-in folder inside the
|
||||
window; windev 17.9k items: stopped window 153 s → 6 s on PS 5.1, 102 s → 6 s on PS 7).
|
||||
2. #903: the edit-and-revert cannot return the chip to "In sync", because `RowVersion` is part of the revision hash (see
|
||||
above). Draft content is restored; the chip shows drift until the next deploy.
|
||||
3. QA order: the Secondary was stopped and swapped while the Primary still served, then the Primary. Both started
|
||||
|
||||
Reference in New Issue
Block a user