fix(install): swap waits for the host process to exit before moving the folder (#908)
v2-ci / unit-tests (push) Has been cancelled

On QA (76e73e7d deploy) Invoke-OtOpcUaInstallSwap.ps1's first Move-Item of
app\ failed "file in use" right after the service reported Stopped: SCM
reports Stopped while the host is still exiting (e.g. a log file under
InstallRoot still open).

- After the stop, wait (bounded, -ProcessExitTimeoutSeconds, default 60)
  until the service's host process (PID + start time captured before the
  stop) and every process imaged under InstallRoot have exited; otherwise
  throw naming them, with nothing moved.
- The renames use [IO.Directory]::Move instead of Move-Item. On windev,
  PowerShell 7's Move-Item falls back to an item-by-item move when the
  rename fails because a file under the folder is open, and leaves a partial
  PreviousInstall tree behind (a retry then nests into it).
- The first rename is retried for at most -MoveRetrySeconds (default 10)
  while it fails because something under InstallRoot is open (a backstop
  for a holder whose image is outside InstallRoot).
- -Verify/-Reapply/-Scope semantics unchanged.

Tests: dry-run harness gains fake-process-list tests for the wait and the
retry plus AST order checks (145 passed); the Windows harness gains three
real cases (holder past the budget, holder released 2 s into the wait,
holder outside InstallRoot) - 77/0 on pwsh 7 and Windows PowerShell 5.1 on
windev; with the wait removed the first two fail.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
2026-11-13 08:46:26 -05:00
parent f416b88324
commit a157b27628
6 changed files with 332 additions and 13 deletions
+11 -3
View File
@@ -147,16 +147,24 @@ run the install once with its full arguments (the per-box deploy record lists th
`scripts/install/Invoke-OtOpcUaInstallSwap.ps1` is the swap with the gate built in: preflight (candidate carries
`script-worker\`, `PreviousInstall` is fresh and on the same volume, isolation record present when isolation is
installed, record `InstallRoot` equals `-InstallRoot`) → `-Reapply -Scope Outside` + `-Verify -Scope Outside` **while
the old build still serves** (a failure stops here: nothing stopped or moved) → stop → move live → previous, candidate →
the old build still serves** (a failure stops here: nothing stopped or moved) → stop → **wait until the host process
(PID captured before the stop) and every process whose image is under `InstallRoot` have exited** (lmxopcua#908: SCM can
report `Stopped` while the host is still exiting, holding e.g. a log file open, and the folder move then fails "in use";
bounded by `-ProcessExitTimeoutSeconds`, default 60 s, after which the script throws naming the processes, with nothing
moved) → move live → previous (one `[IO.Directory]::Move` rename that fails as a whole — PowerShell 7's `Move-Item` falls
back to a partial item-by-item move when the rename fails; retried for at most `-MoveRetrySeconds`, default 10 s, while it
fails because something under `InstallRoot` is open: a backstop for a handle held by a process outside it), candidate →
live → copy `-Carry` items back (default `appsettings*.json`, `pki`, `logs`; QA: `'appsettings*.json','logs'`; rollback:
`-NoCarry`) → `-Reapply -Scope InstallRoot` → `-Verify -Scope InstallRoot` → start (`-NoStart` for a pair window). The
two scoped gates together check every install step, so the new build never starts unverified; a failed post-swap
`-Verify` leaves the service **stopped**. The script prints the stopped window and its parts
(`stopped window (stop issued -> start issued): 6.0 s [stop …, move 0.0 s, carry 0.1 s, isolation 5.8 s]`). Backups,
(`stopped window (stop issued -> start issued): 6.0 s [stop …, exit …, move 0.0 s, carry 0.1 s, isolation 5.8 s]`; `exit` is the #908 wait). Backups,
`data\` restore, migrations and health checks stay in the deploy's own steps. Every deploy pack that swaps the folder runs
it (or `-Reapply` + `-Verify` right after its own swap), and `-Verify` is a post-deploy gate.
`Test-ScriptWorkerIsolation.Windows.ps1` exercises the whole sequence (swap → `-Verify` fails → `-Reapply` → `-Verify`
passes → `-Reapply` unchanged → the swap script → the scoped modes, the pre-stop gate, `-NoStart`, a `-NoCarry` rollback)
passes → `-Reapply` unchanged → the swap script → the scoped modes, the pre-stop gate, `-NoStart`, a `-NoCarry` rollback,
and the #908 cases: a process running from `InstallRoot` past the budget, one exiting ~2 s after the wait starts, and an in-use
handle from outside `InstallRoot` that the first-move retry absorbs)
on a scratch tree; `-TimingFiles N` times the swap on a vd03-like tree (`-ScriptsDir` for a before/after).
**Expected stopped window (lmxopcua#905).** Before #905 the swap ran the full `-Reapply` and `-Verify` while the service
@@ -184,6 +184,9 @@ Do not run a mixed pair: `RedundancyStateChanged` changed shape in this range. A
process to exit; `Invoke-OtOpcUaInstallSwap.ps1` does not.
- It cost about 20 s of the vq03 pair outage (the retry re-runs `-Scope Outside`).
- Filed as **lmxopcua#908**. vd03 did not hit it.
- **Fixed (lmxopcua#908, closed):** `Invoke-OtOpcUaInstallSwap.ps1` now waits, after `Stopped`, for the host process
and every process imaged under `InstallRoot` to exit (`-ProcessExitTimeoutSeconds`, default 60) and retries the first
move on a sharing violation (`-MoveRetrySeconds`, default 10). The next deploy needs no hand retry.
3. `Stop-Service` on the new build took 5.7–6.1 s instead of the 1–3 s target, because of the ~5 s OPC UA session close
after the cluster leave (see "Observation"). The hand-over, the actual #906 criterion, was 0.36 s and 0.47 s.
4. Logs: the QA host log for the window rolled to `otopcua-20261113_001.log` on vq04 (size limit). Read it from there.