fix(install): swap waits for the host process to exit before moving the folder (#908)
v2-ci / unit-tests (push) Has been cancelled
v2-ci / unit-tests (push) Has been cancelled
On QA (76e73e7d deploy) Invoke-OtOpcUaInstallSwap.ps1's first Move-Item of
app\ failed "file in use" right after the service reported Stopped: SCM
reports Stopped while the host is still exiting (e.g. a log file under
InstallRoot still open).
- After the stop, wait (bounded, -ProcessExitTimeoutSeconds, default 60)
until the service's host process (PID + start time captured before the
stop) and every process imaged under InstallRoot have exited; otherwise
throw naming them, with nothing moved.
- The renames use [IO.Directory]::Move instead of Move-Item. On windev,
PowerShell 7's Move-Item falls back to an item-by-item move when the
rename fails because a file under the folder is open, and leaves a partial
PreviousInstall tree behind (a retry then nests into it).
- The first rename is retried for at most -MoveRetrySeconds (default 10)
while it fails because something under InstallRoot is open (a backstop
for a holder whose image is outside InstallRoot).
- -Verify/-Reapply/-Scope semantics unchanged.
Tests: dry-run harness gains fake-process-list tests for the wait and the
retry plus AST order checks (145 passed); the Windows harness gains three
real cases (holder past the budget, holder released 2 s into the wait,
holder outside InstallRoot) - 77/0 on pwsh 7 and Windows PowerShell 5.1 on
windev; with the wait removed the first two fail.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
+11
-3
@@ -147,16 +147,24 @@ run the install once with its full arguments (the per-box deploy record lists th
|
||||
`scripts/install/Invoke-OtOpcUaInstallSwap.ps1` is the swap with the gate built in: preflight (candidate carries
|
||||
`script-worker\`, `PreviousInstall` is fresh and on the same volume, isolation record present when isolation is
|
||||
installed, record `InstallRoot` equals `-InstallRoot`) → `-Reapply -Scope Outside` + `-Verify -Scope Outside` **while
|
||||
the old build still serves** (a failure stops here: nothing stopped or moved) → stop → move live → previous, candidate →
|
||||
the old build still serves** (a failure stops here: nothing stopped or moved) → stop → **wait until the host process
|
||||
(PID captured before the stop) and every process whose image is under `InstallRoot` have exited** (lmxopcua#908: SCM can
|
||||
report `Stopped` while the host is still exiting, holding e.g. a log file open, and the folder move then fails "in use";
|
||||
bounded by `-ProcessExitTimeoutSeconds`, default 60 s, after which the script throws naming the processes, with nothing
|
||||
moved) → move live → previous (one `[IO.Directory]::Move` rename that fails as a whole — PowerShell 7's `Move-Item` falls
|
||||
back to a partial item-by-item move when the rename fails; retried for at most `-MoveRetrySeconds`, default 10 s, while it
|
||||
fails because something under `InstallRoot` is open: a backstop for a handle held by a process outside it), candidate →
|
||||
live → copy `-Carry` items back (default `appsettings*.json`, `pki`, `logs`; QA: `'appsettings*.json','logs'`; rollback:
|
||||
`-NoCarry`) → `-Reapply -Scope InstallRoot` → `-Verify -Scope InstallRoot` → start (`-NoStart` for a pair window). The
|
||||
two scoped gates together check every install step, so the new build never starts unverified; a failed post-swap
|
||||
`-Verify` leaves the service **stopped**. The script prints the stopped window and its parts
|
||||
(`stopped window (stop issued -> start issued): 6.0 s [stop …, move 0.0 s, carry 0.1 s, isolation 5.8 s]`). Backups,
|
||||
(`stopped window (stop issued -> start issued): 6.0 s [stop …, exit …, move 0.0 s, carry 0.1 s, isolation 5.8 s]`; `exit` is the #908 wait). Backups,
|
||||
`data\` restore, migrations and health checks stay in the deploy's own steps. Every deploy pack that swaps the folder runs
|
||||
it (or `-Reapply` + `-Verify` right after its own swap), and `-Verify` is a post-deploy gate.
|
||||
`Test-ScriptWorkerIsolation.Windows.ps1` exercises the whole sequence (swap → `-Verify` fails → `-Reapply` → `-Verify`
|
||||
passes → `-Reapply` unchanged → the swap script → the scoped modes, the pre-stop gate, `-NoStart`, a `-NoCarry` rollback)
|
||||
passes → `-Reapply` unchanged → the swap script → the scoped modes, the pre-stop gate, `-NoStart`, a `-NoCarry` rollback,
|
||||
and the #908 cases: a process running from `InstallRoot` past the budget, one exiting ~2 s after the wait starts, and an in-use
|
||||
handle from outside `InstallRoot` that the first-move retry absorbs)
|
||||
on a scratch tree; `-TimingFiles N` times the swap on a vd03-like tree (`-ScriptsDir` for a before/after).
|
||||
|
||||
**Expected stopped window (lmxopcua#905).** Before #905 the swap ran the full `-Reapply` and `-Verify` while the service
|
||||
|
||||
@@ -184,6 +184,9 @@ Do not run a mixed pair: `RedundancyStateChanged` changed shape in this range. A
|
||||
process to exit; `Invoke-OtOpcUaInstallSwap.ps1` does not.
|
||||
- It cost about 20 s of the vq03 pair outage (the retry re-runs `-Scope Outside`).
|
||||
- Filed as **lmxopcua#908**. vd03 did not hit it.
|
||||
- **Fixed (lmxopcua#908, closed):** `Invoke-OtOpcUaInstallSwap.ps1` now waits, after `Stopped`, for the host process
|
||||
and every process imaged under `InstallRoot` to exit (`-ProcessExitTimeoutSeconds`, default 60) and retries the first
|
||||
move on a sharing violation (`-MoveRetrySeconds`, default 10). The next deploy needs no hand retry.
|
||||
3. `Stop-Service` on the new build took 5.7–6.1 s instead of the 1–3 s target, because of the ~5 s OPC UA session close
|
||||
after the cluster leave (see "Observation"). The hand-over, the actual #906 criterion, was 0.36 s and 0.47 s.
|
||||
4. Logs: the QA host log for the window rolled to `otopcua-20261113_001.log` on vq04 (size limit). Read it from there.
|
||||
|
||||
Reference in New Issue
Block a user