From 6e67250e74f2241087712905c832e97a3a6de817 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Fri, 13 Nov 2026 02:20:19 -0500 Subject: [PATCH] docs(deploy): next vd03 (6c4d2990) + QA (fea7de0a) deploy pack to c9640f5c, read-only ConfigDb preflight No migrations in either range; no #834/#898/#899/#897/#896 findings on DEV or QA. QA isolation record holds 12 deny paths (OTel collector added 2026-11-12), not 11. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi --- .../2026-11-13-next-vd03-qa-to-c9640f5c.md | 329 ++++++++++++++++++ .../deploy/qa-upgrade-b8b4c18f-to-fea7de0a.md | 4 + 2 files changed, 333 insertions(+) create mode 100644 docs/deploy/2026-11-13-next-vd03-qa-to-c9640f5c.md diff --git a/docs/deploy/2026-11-13-next-vd03-qa-to-c9640f5c.md b/docs/deploy/2026-11-13-next-vd03-qa-to-c9640f5c.md new file mode 100644 index 000000000..33beb4744 --- /dev/null +++ b/docs/deploy/2026-11-13-next-vd03-qa-to-c9640f5c.md @@ -0,0 +1,329 @@ +# Next deploy: DEV vd03 `6c4d2990` and QA vq03/vq04 `fea7de0a` → master `c9640f5c` + +> **Status: PREPARED, NOT EXECUTED (2026-11-13).** This pack covers both tiers, plus the read-only ConfigDb preflight, +> which was run over direct SQL from the Mac on 2026-11-13. Nothing was built, staged or run on any wonder box. **No +> blocker found; no issue filed.** +> +> **Target:** `origin/master` **`c9640f5c`** (`c9640f5ced383e2a722702409adbc62298731994`, #904). If master moves before the window, either pin to +> `c9640f5c` (as the 6c4d2990 run pinned) or re-run §1–§3 for the new range. +> +> Templates: [`2026-11-13-vd03-fea7de0a-to-6c4d2990.md`](2026-11-13-vd03-fea7de0a-to-6c4d2990.md) (vd03 layout and +> scripts), [`qa-upgrade-b8b4c18f-to-fea7de0a.md`](qa-upgrade-b8b4c18f-to-fea7de0a.md) (QA window, canary, #890/#891 +> denies, "Worker isolation across the next swap"), [`vd03-upgrade-b8b4c18f-to-master.md`](vd03-upgrade-b8b4c18f-to-master.md) +> §e/§f and "Worker isolation and box-wide denies". + +## 1. Commit ranges and what changes + +### vd03: `6c4d2990..c9640f5c` (9 commits) + +| Commit | Change | Effect on vd03 (one fused node, one FOCAS driver, 22 tags, runner `InProcess`) | +|---|---|---| +| `efaf3435` #900 | HistorianGateway `FailedPrecondition` + `AGGREGATE_NOT_SUPPORTED_FOR_TAG` maps to `BadAggregateNotSupported`, with no health failure. Adds `Grpc.StatusProto 2.76.0`. | **Inert today.** vd03's HG `ea9f21c` predates historiangw#135 (`a220ec0`) and still answers `Internal`, which keeps the current handling. It takes effect when HG ≥ `a220ec0` is deployed. | +| `9375c9e2` | docs (the 6c4d2990 record) | — | +| `d5288388` #895 review, `c9640f5c` #904 | Galaxy alarm reverse map: re-areaed, subtag-provider and pre-subscription references. A picker-form ack waits up to 2 s for the feed snapshot. | Not reachable (no Galaxy driver). | +| `2470f82e` #872 | Galaxy status rows Pending/Warning → **Uncertain**; detail 8001 → `BadTypeMismatch`. A Pending write reply is now "not committed". | Not reachable. | +| `b85e004b` #902 | AdminUI: `MonacoEditor.DisposeAsync` on a gone circuit logs at Debug, not `[ERR]`. | Fixes the "Observed" `[ERR]` from the 6c4d2990 record. | +| `35576485` #903 | AdminUI `/deployments`: the drift chip is never a cached "In sync". The first render shows "Checking for changes...". A failed check shows "Drift check failed". | Fixes the stale "In sync" chip from the 6c4d2990 record. | +| `2cb8a2c3` #901 | Installer records its settings and gains `-Reapply`/`-Verify`. New `scripts/install/Invoke-OtOpcUaInstallSwap.ps1`. | Changes the deploy procedure (§4). No runtime change. | +| `f0c6cf96` #834 | Running Galaxy/OpcUaClient drivers **refuse** out-of-namespace credential refs at resolve time (the warn-only path is gone). The preflight logs one **Error** per finding, and the `/raw` banner is `alert-danger`. | FOCAS has no credential fields. Preflight: 0 findings (§3). Only the boot line wording changes (§5). | + +### QA: `fea7de0a..c9640f5c` (46 commits: the 37 above `6c4d2990` plus the 9 vd03 commits) + +The vd03 rows above, plus everything vd03 already got on 2026-11-13 (`fea7de0a..6c4d2990`, 37 commits): + +- **Historian.** + - `ServerHistorian:Gateway:EnableNativeTimeWeighted` now **defaults to `true`** (#694, `53978ca6`). TimeAverage/Total + are advertised and served natively (12 aggregates). QA HG `fa9cf31` includes `9f711c1`, so it supports this. + - Alarm history goes out in `SendEvents` batches of ≤200, with a per-event fallback (#746). + - A Boolean node refuses TimeAverage/Total up front with `BadAggregateNotSupported` (#897). A String/DateTime node + refuses the numeric-only aggregates. + - `historianReadOnly: true` TagConfig plus the **new `ServerHistorian:ProvisionTags` setting** (#899). + - Float/Double/Byte/SByte historized tags are now provisioned (#898). See the upgrade hazard in §3. +- **Address space (#896):** a UNS reference to an alarm-bearing raw tag no longer gets a UNS value Variable. The + alarm surfaces only as the condition on the equipment folder. +- **Galaxy (#895):** picker-authored native alarms resolve and ack. Galaxy spans and `galaxy.events.*` counters are + exported (#892). +- **Telemetry 0.1.0 → 0.3.1** (scadaproj#55/#56). + - The OTel Resource carries `service.version` (the git SHA), `site.id` (the Cluster id) and `node.role`. + - **With `Exporter=Otlp`, Serilog logs go over OTLP too.** QA has `OtOpcUa__Telemetry__Exporter=Otlp` and + `OtlpEndpoint=http://127.0.0.1:4317` since 2026-11-12, so logs will start to reach the local `ZbOtelCollector`. + - 0.3 refuses a scheme-less `host:port` when `Exporter=Otlp`. The recorded value has `http://`, so it passes; + re-check it in the on-box preflight. +- **AdminUI #761 (`fa8c3d1a`):** a bodyless `POST /api/deployments` returned 404 on fused admin+driver nodes. It now + deploys. +- **Drivers.** + - AB Legacy MicroLogix function files are refused locally with `BadNotSupported`, not reported as a comms fault (#592). + - Heidenhain gains the `Controller/Family`, `Status/*StateName` and `Tool/AxisName` leaves, PLC + `INPUT_DWORD`/`OUTPUT_DWORD`, and new refusal-code mappings (#829/#853). Not reachable on QA. +- **Installer #890 (`f75ee7d0`):** `-DenyPaths`. Already applied on QA from staged scripts on 2026-11-12. +- Other commits are docs, tests or docker-dev only (#893). + +### What a ScadaBridge consumer would notice + +**On DEV and QA: nothing.** ScadaBridge on both tiers reads no OtOpcUa data; its only data connection is `MxGateway` +(QA pack §verification, vd03 fea7de0a section). For an OtOpcUa-bound ScadaBridge elsewhere: + +- **Galaxy.** + - Pending/Warning rows now read **Uncertain**, not Good/Bad (#872). + - A Galaxy write that MXAccess answers Pending is reported **not committed**. The node reverts and the later commit + arrives as a data change. + - 8001 now reads `BadTypeMismatch`. +- **Alarm-bearing references (#896):** a UNS binding to a reference on an alarm-bearing tag now gets + `BadNodeIdUnknown`. Before, it read `BadWaitingForInitialData` forever. Bind the alarm as an event subscription. +- **HistoryRead-Processed:** TimeAverage/Total on a Boolean node now answers `BadAggregateNotSupported` (#897). With + HG ≥ `a220ec0`, the same holds for a numeric node backed by a discrete AVEVA tag (#900). TimeAverage/Total are + advertised again (QA range). +- **Credential refs (#834):** a Galaxy or OpcUaClient driver with an out-of-namespace credential reference no longer + connects. It is not a ScadaBridge-visible wire change, but its nodes go Bad. +- No proto, NodeId-scheme or ServiceLevel-semantics change. Contracts are unchanged. + +## 2. Migrations and settings + +- **ConfigDb migrations: none in either range.** The only file touched under `src/Core/ZB.MOM.WW.OtOpcUa.Configuration` + is `Validation/DraftValidator.cs` (#899 `HistorianReadOnlyInvalid` deploy rule, already on vd03). The repo head stays + `20261022102459_EnableSnapshotIsolation` (9 migrations). Both DBs are already there (§3). +- **New optional setting:** `ServerHistorian:ProvisionTags` = `CreateOrUpdate` (default, the previous behaviour) | + `CreateOnly` (Gateway backend only) | `Off`. An **unknown value, or `CreateOnly` on a SQL backend, fails host start**. + Neither tier sets it today (vd03 record). **Leave it unset:** there are 0 historized tags on either tier. +- **Changed default:** `ServerHistorian:Gateway:EnableNativeTimeWeighted` `false` → `true` (QA only; vd03 already has it). + The kill switch is `ServerHistorian__Gateway__EnableNativeTimeWeighted=false`. It is not needed: QA HG `fa9cf31` supports + native time-weighted reads. +- **Telemetry (QA):** no new keys. 0.3 reads the same `OtOpcUa:Telemetry` section and adds optional + `SiteId`/`NodeRole`/`ServiceVersion`/`ResourceAttributes:*` overrides. +- **Required env:** no new required variable. The existing required ones stay: `Security__ApiKeys__Pepper` + (≥16 chars) and the JWT key on both tiers, and `LocalDb__Replication__ApiKey`, identical on both QA nodes. +- **Packages:** `Grpc.StatusProto 2.76.0` (new). Telemetry(.Serilog) is 0.3.1 (already on vd03). LocalDb 0.3.7, + MxGateway.Client 0.4.0 and HistorianGateway.Client 1.0.0 are unchanged. +- **Peer services:** none to upgrade first. HG `ea9f21c` (vd03) and `fa9cf31` (QA) serve every RPC this build uses; + #900 is capability-tolerant. mxgw `76a71bc8` is unaffected, because neither tier has a Galaxy driver. + +## 3. Read-only preflight (run 2026-11-13 from the Mac, direct SQL) + +Run as `wwadmin` with `sqlcmd -C -I`, the password passed in `SQLCMDPASSWORD`, against `OtOpcUaConfig` on +`wonder-app-vd03.zmr.zimmer.com,1433` (DEV) and `wonder-sql-vq03.zmr.zimmer.com,1433` (QA). The queries were SELECTs only, +plus the existing script's local table variables. Two passes: + +1. `scripts/preflight/vd03-preupgrade-b8b4c18f-configdb-checks.sql` unchanged (15 checks). It already reproduces + `DriverCredentialPolicy` / `CredentialReferenceDeployChecks` on the draft rows (checks 6/7). +2. Ad-hoc SELECTs for what the script does not cover: + - the **deployed** side of `CredentialReferencePreflight.ScanAsync`: the latest `Status=2` (Sealed) `ArtifactBlob`, + `DECOMPRESS`ed and searched; + - #898/#899/#897/#896 over `dbo.Tag.TagConfig` (`isHistorized`, `historianTagname`, `historianReadOnly`, `alarm`) and + `dbo.VirtualTag.Historize`. + +| Check | DEV vd03 | QA (shared ConfigDb on wonder-sql-vq03) | +|---|---|---| +| `__EFMigrationsHistory` | 9 rows, head `20261022102459_EnableSnapshotIsolation` = repo head. Snapshot isolation ON, compat 160. | Same: 9 rows, same head, snapshot ON, compat 160. | +| Inventory | 1 driver (`FOCAS` `focas-z-34184`), 1 device, 22 tags (8 Double, 8 Int32, 6 String), 0 VT, 0 scripted alarms, 2 UNS refs, 10 deployments | 1 driver (`Calculation` `script-runner-canary`), 1 device, 1 tag (Int32), 1 VT (Int32), 0 scripted alarms, 0 UNS refs, 3 deployments | +| Latest Sealed deployment | `595cb909`, rev `fa9a2e28`, 2026-11-13 06:34Z (gzip, 13,655 B JSON) | `5905003e`, rev `82dd0598`, 2026-11-13 00:05Z (gzip, 4,146 B JSON) | +| **#834 draft**: script checks 6 (#815/#836 OutOfScope) and 7 (#835/#849 OtherInstance) | **ok, 0 rows** | **ok, 0 rows** | +| **#834 draft**: credential-ruled driver types (Galaxy / OpcUaClient / Heidenhain) and any `secret:`/`env:` text in DriverConfig/DeviceConfig | none. FOCAS has no credential fields, and neither config holds a reference. | none. Calculation has no credential fields; no reference. | +| **#834 deployed**: latest Sealed artifact contains `"Galaxy"` / `"OpcUaClient"` / `"Heidenhain"` / `secret:` / `env:` | 0 / 0 / 0 / 0 / 0 | 0 / 0 / 0 / 0 / 0 | +| → #834 verdict | **No finding.** The boot preflight will log "every stored and deployed reference is namespaced". | **No finding.** | +| **#898** historized Float/Double/Byte/SByte tags (first start would `EnsureTags`) | 0 (0 tags with `isHistorized: true` at all; no `historianTagname`) | 0 (no historized tag; the VT has `Historize=0`) | +| **#899** tags with a `historianReadOnly` key / a non-boolean value (`HistorianReadOnlyInvalid`) | 0 / 0 | 0 / 0 | +| **#899** `ProvisionTags` default | Nothing to provision, so the default `CreateOrUpdate` is harmless. | Same. | +| **#897** historized Boolean tags/VTs (TimeAverage/Total now refused up front) | 0 (no Boolean tags at all). Aggregates are chosen per client request, not stored in config; no historized Boolean node can receive one. | 0 | +| **#896** UNS references to alarm-bearing tags | 0 (no tag has an `alarm` object) | 0 | +| Other script checks (#783, Calc trigger, #791 ×2, #792, #848, MTConnect, #781 Sql, #803 Heidenhain, UNS collisions) | all **ok** | all **ok** | +| Script check 0 "Environment" | `attention`: a stale expectation. The script was written for `b8b4c18f`, and the migration it asks for is already applied. Not an issue. | Same. | +| Script check 13 (FOCAS renames, info) | 9 info rows. These are the b8b4c18f→master FOCAS address notes, already live since the 2026-11-12 deploy. | ok | + +**Verdict: no blocker on either tier; no issue filed.** Re-run both passes just before the window if the configuration +has changed since 2026-11-13. On QA the canary is the only configuration. + +### Pack correction found during preflight: QA records 12 deny paths, not 11 + +The 2026-11-12 OTel collector install (scadaproj `infra/otel-collector/wonder/2026-11-12-wonder-otel-collector.md`, +21:46–21:47) re-ran the installer on vq03/vq04 and added **`E:\ApiInstall\OtelCollector`** (`DenyPathRecord` + +`SensitiveDirDeny` changed). The QA pack's 11-path list is therefore one short. If the one-time full-argument run +(§4) used the 11-path list, it would **remove** the collector deny. §4 uses the 12-path list. As before, the rule is to +use exactly what `reg query` shows. + +The same record also shows that the OtOpcUa Primary failed over to **vq04** during the restarts at 21:49–21:50. Read +ServiceLevel before the window to know which node is Primary now. + +## 4. Build, stage, one-time isolation record, swap + +### Build (Mac, isolated worktree at `c9640f5c`; not done here) + +```bash +git worktree add ../OtOpcUa-wt-c9640f5c c9640f5c && cd ../OtOpcUa-wt-c9640f5c +dotnet publish src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj -c Release -r win-x64 --self-contained true -o ../publish-c9640f5c +# expect ~1396+ files, OtOpcUa.Host.dll ProductVersion 1.0.0+c9640f5c…, script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe present; +# deps.json: Telemetry(.Serilog) 0.3.1, LocalDb 0.3.7, MxGateway.Client 0.4.0, HistorianGateway.Client 1.0.0, Grpc.StatusProto 2.76.0 +cd ../publish-c9640f5c && zip -qr ../otopcua-c9640f5c.zip . && shasum -a 256 ../otopcua-c9640f5c.zip > ../otopcua-c9640f5c.zip.sha256 +``` + +Copy the zip and `.sha256` over SMB into the staging folders below, and check the hash on the box before expanding. +Stage the three install scripts from `c9640f5c` next to them. Their SHA-256 values at `c9640f5c`: + +| File | SHA-256 | +|---|---| +| `scripts/install/Install-ScriptWorkerIsolation.ps1` | `7d4078cb93a42b362ab50ba5b79bc293bf92001d5444f58d5c6733e0d51c4914` | +| `scripts/install/ScriptWorkerIsolation.psm1` | `ae9d84fd0077a730ecaaf01b8d476ee83755c2d3fb163e752379c276f227d902` | +| `scripts/install/Invoke-OtOpcUaInstallSwap.ps1` | `9498fd8092c3b2ea95f7c7f8275b704d6bed99dfeea0dcfadfbab1045d79e88d` | + +All remote steps below are on-box deploy steps (`winrm` skill, deploy scope). Run them in an elevated Windows +PowerShell, from the staged `install\` folder. **Never run `Install-Services.ps1`, `Uninstall-Services.ps1` or +`Refresh-Services.ps1`** (vd03 pack §e step 5). + +### DEV vd03 (`E:\ApiInstall\OtOpcUa`, data in `E:\ApiInstall\OtOpcUa-data`, runner `InProcess`) + +Staging: `E:\ApiInstall\_stage-otopcua-c9640f5c\` (zip, `.sha256`, `install\`). Backups go to +`E:\ApiInstall\OtOpcUa-backups\pre-20261113-otopcua-c9640f5c\`. Create that folder with a protected DACL (SYSTEM + +Administrators only, #891). + +1. **Prepare (service up).** + - `reg export HKLM\SYSTEM\CurrentControlSet\Services\OtOpcUaHost` into the backup folder. + - Check the hash, then expand to `E:\ApiInstall\OtOpcUa.new-c9640f5c` and check `ProductVersion`. + - `BACKUP DATABASE [OtOpcUaConfig] TO DISK = N'OtOpcUaConfig-pre-20261113-otopcua-c9640f5c.bak' WITH COPY_ONLY, CHECKSUM`. + There is no migration, so this is only a configuration fallback. + - Read the service env (names and count only; expect 28). Check that `OtOpcUa__Telemetry__OtlpEndpoint` has a scheme. +2. **One-time #901 settings record** (service can stay up; vd03 was isolated by the pre-#901 installer, so + `Invoke-OtOpcUaInstallSwap.ps1` refuses until the record exists): + ```powershell + reg query HKLM\SYSTEM\CurrentControlSet\Services\OtOpcUaScriptWorker\Parameters /v DenyPaths + # expect exactly: E:\ApiInstall, E:\ApiInstall\OtOpcUa\pki, E:\ApiInstall\OtOpcUa-data, C:\ProgramData\ZB\kek, E:\psrem-stage + .\Install-ScriptWorkerIsolation.ps1 -InstallRoot E:\ApiInstall\OtOpcUa -HostServiceAccount LocalSystem ` + -DenyPaths E:\ApiInstall,E:\ApiInstall\OtOpcUa\pki,E:\ApiInstall\OtOpcUa-data,C:\ProgramData\ZB\kek,E:\psrem-stage -WhatIf + .\Install-ScriptWorkerIsolation.ps1 -InstallRoot E:\ApiInstall\OtOpcUa -HostServiceAccount LocalSystem ` + -DenyPaths E:\ApiInstall,E:\ApiInstall\OtOpcUa\pki,E:\ApiInstall\OtOpcUa-data,C:\ProgramData\ZB\kek,E:\psrem-stage + .\Install-ScriptWorkerIsolation.ps1 -Verify # VERIFY OK, exit 0 + reg query HKLM\SYSTEM\CurrentControlSet\Services\OtOpcUaScriptWorker\Parameters /v DenyPaths # same 5 paths + ``` + - **No `-ConfigureHostService`:** the vd03 runner stays `InProcess` (vd03 pack §"Worker isolation and box-wide denies"). + - Expect **`SettingsRecord` changed**, `DenyPathRecord` unchanged, and every other step unchanged. The `reisolate.ps1` + run after the 6c4d2990 swap restored them. + - The "protected DACL" WARNs under `E:\psrem-stage` and the locked backups are pre-existing and expected. + - If `reg query` lists anything other than the 5 paths, stop and use the recorded list. +3. **Swap.** + - `Stop-Service OtOpcUaHost`. + - `robocopy E:\ApiInstall\OtOpcUa-data \OtOpcUa-data /E /COPY:DAT` (stopped, so the SQLite triplets are consistent). + - Then run: + ```powershell + .\Invoke-OtOpcUaInstallSwap.ps1 -InstallRoot E:\ApiInstall\OtOpcUa -Candidate E:\ApiInstall\OtOpcUa.new-c9640f5c ` + -PreviousInstall E:\ApiInstall\OtOpcUa.bak.20261113-pre-c9640f5c + ``` + - The default `-Carry` (`appsettings*.json`, `pki`, `logs`) is right for vd03. + - The script tolerates an already-stopped service. It moves the folders, carries the box files, runs `-Reapply` + (expect `WorkerFolderAcl`, `InstallRootDeny` and the `pki` `SensitiveDirDeny` **changed**), then `-Verify`, then starts. + - **Gate:** the script must finish without throwing. A failed `-Verify` leaves the service **stopped**: fix the + problem, re-run `-Reapply`/`-Verify`, then start. + - After the start, run `.\Install-ScriptWorkerIsolation.ps1 -Verify` once more and record its output (exit 0). + +### QA vq03 + vq04 (stop both, swap both, start both) + +Layout: `E:\ApiInstall\OtOpcUa\{app,data,pki}`, service `OtOpcUaHost` (LocalSystem), runner `OutOfProcess` (Calculation + +VirtualTag). Staging: `E:\psrem-stage\otopcua-c9640f5c\` (zip, `.sha256`, `install\`). Backups go to +`E:\psrem-stage\otopcua-c9640f5c\backup\`, created with a protected DACL (#891). + +1. **Prepare (both nodes, services up).** + - Expand to `E:\ApiInstall\OtOpcUa\app.new-c9640f5c` (check the hash and ProductVersion). + - `reg export …\OtOpcUaHost backup\OtOpcUaHost-pre-c9640f5c.reg`. + - Read the env names and count: expect 48 on vq03 and 47 on vq04, including `Scripting__Runner*`, + `Scripting__Worker__*`, the pepper, `LocalDb__Replication__ApiKey` and `OtOpcUa__Telemetry__*`. Check that + `OtlpEndpoint` has a scheme. + - Once: `BACKUP DATABASE OtOpcUaConfig … COPY_ONLY, CHECKSUM, COMPRESSION` on wonder-sql-vq03 → + `OtOpcUaConfig_pre-c9640f5c_20261113.bak`, then `RESTORE VERIFYONLY`. + - Read ServiceLevel on both nodes to learn which one is Primary (250). It may be **vq04** since the 2026-11-12 OTel restarts. +2. **One-time #901 settings record (both nodes, services up).** QA was isolated by the pre-#901 installer + (DenyPaths only): + ```powershell + reg query HKLM\SYSTEM\CurrentControlSet\Services\OtOpcUaScriptWorker\Parameters /v DenyPaths + # expect these 12 (the 11 #891 paths + the OTel collector folder added 2026-11-12): + $deny = 'E:\ApiInstall\OtOpcUa','E:\ApiInstall\OtOpcUa\pki','E:\ApiInstall\OtOpcUa\data','C:\ProgramData\ZB\kek', + 'E:\ApiInstall\ScadaBridge','E:\ApiInstall\HistorianGateway','E:\ApiInstall\MxAccessGateway','E:\ApiInstall\Overview', + 'E:\ApiInstall\btdbmanager','E:\psrem-stage','E:\psrem','E:\ApiInstall\OtelCollector' + .\Install-ScriptWorkerIsolation.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -HostServiceAccount LocalSystem -ConfigureHostService -DenyPaths $deny -WhatIf + .\Install-ScriptWorkerIsolation.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -HostServiceAccount LocalSystem -ConfigureHostService -DenyPaths $deny + .\Install-ScriptWorkerIsolation.ps1 -Verify # VERIFY OK, exit 0 + reg query HKLM\SYSTEM\CurrentControlSet\Services\OtOpcUaScriptWorker\Parameters /v DenyPaths # same 12 paths + ``` + - If `reg query` differs from the 12 above, use **exactly the recorded list**. A missing entry in `$deny` removes that + deny. + - `-ConfigureHostService` stays, because QA's runner flip used it. + - Expect **`SettingsRecord` changed** and `DenyPathRecord` unchanged. + - `HostServiceEnv` may report **changed** as a reorder only, as in the #890 run, if the OTel entries were appended + after the `Scripting__Worker__*` pair. Confirm the env count is unchanged (48/47) and that no value changed + (compare SHA-256 prefixes, never print values). Every other step should be unchanged. + - The vq04 `kek\historiangw.kek`/`mxgateway.kek` and locked `.reg` protected-DACL WARNs are expected. +3. **Stop both.** Stop the Secondary (240) first, then the Primary (250). On each node, while stopped, run + `robocopy data backup\data-pre-c9640f5c /E /COPY:DAT` and the same for `pki`. +4. **Swap each node** (services stopped): + ```powershell + .\Invoke-OtOpcUaInstallSwap.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -Candidate E:\ApiInstall\OtOpcUa\app.new-c9640f5c ` + -PreviousInstall E:\ApiInstall\OtOpcUa\app.bak.20261113-pre-c9640f5c -Carry 'appsettings*.json','logs' -NoStart + ``` + - `pki\` and `data\` sit beside `app\` and are not moved. + - Expect the re-apply to report `WorkerFolderAcl` and `InstallRootDeny` changed, then `-Verify` OK. + - **Gate: the script returns without throwing on both nodes.** If `-Verify` fails on either node, do not start + either; fix it, then run `-Reapply` and `-Verify`. +5. **Start both.** Start the node that was Primary first; it waits for its peer, about 40 s last time. Then start the + other node. +6. On each node, after ready: `.\Install-ScriptWorkerIsolation.ps1 -Verify` → `VERIFY OK`, exit 0. Record the output. + +## 5. Post-deploy gates + +Run these from the Mac over HTTP / opc.tcp. Read logs over SMB (`logs\otopcua-YYYYMMDD.log`). The Windows-listed size of +an open log reads 0; read its contents. + +| Gate | DEV vd03 | QA vq03 + vq04 | +|---|---|---| +| Isolation | `-Verify` exit 0 after the swap and after the start (§4) | same, both nodes | +| Version | live `OtOpcUa.Host.dll` (SMB copy) ProductVersion `1.0.0+c9640f5c…`; SHA-256 equals the published build | same, both nodes | +| Health | `/healthz`, `/health/ready`, `/health/active` all 200 on `:9000` | `/health/ready` 200 on both, with 5/5 components including `Out-of-process script runner ready.`; `/health/active` 200 on the Primary, 503 on the Secondary | +| ServiceLevel | **250** | **250 / 240**, stable over repeated fresh-session reads | +| Failover (QA) | — | Stop the Primary: the other node reaches **250** and `/health/active` 200. Restart the old Primary: **240**, never above, **no failback**. | +| LocalDb sync (QA) | — | `localdb-replication` connected, oplog backlog 0 on both, before and after the failover | +| Canary (QA) | — | `ns=2;s=script-runner-canary/Device1/canary-calc-seconds` and `ns=3;s=EQ-ec3979550d90/canary-vt-second` are **Good and advancing about once a second** on both endpoints. `otopcua_script_worker_up` = 1 per shard. The #780 runaway drill (`while(true)` VT, expect 2 kills + quarantine, siblings Good, 250/240 unchanged, then delete and redeploy back to rev `82dd0598…`) is **optional**. | +| Driver data | FOCAS raw: 22 variables, 21 Good, `Program/BlockCount` `BadNoData` (as before) | the canary above | +| Boot log | `roles=[admin,driver]`; **`Credential-reference preflight (lmxopcua#815/#834): every stored and deployed reference is namespaced.`** (the new wording; any `Error` line from this preflight is a stop); `Historian gateway probe: the gateway is reachable`; no `FTL` | same, both nodes; also no `Telemetry` endpoint FATAL (0.3 check) | +| **#903 drift chip** | On `/deployments` (login disabled on DEV): the first render shows **"Checking for changes..."**, then "In sync". Make a harmless draft change, for example a tag description on `/raw`, with no deploy. Reload `/deployments`: it must show **"Configuration drift"** at once. Revert the change: the chip returns to "In sync", and the revision is still `fa9a2e28…`. | Optional. The QA AdminUI needs the `localhost` forwarder plus a temporary role-mapping row (QA pack §2). If you do it, make the change on the canary and revert it. Delete the temporary row afterwards. | +| **#902 Monaco** | Open `/scripts/new` (the Monaco editor) and close the browser tab without saving. Then search the log after that time for `MonacoEditor` / `Unhandled exception in circuit`: expect **no `[ERR]`** (it is Debug now). | Same if the AdminUI is opened; otherwise n/a | +| **Historian writes** | HG `http://wonder-app-vd03…:5220/metrics` `historian_gateway_grpc_requests_total`: snapshot before the stop and about 2 min after ready. Only `HistorianStatus/Probe` should grow. **No `EnsureTags`, `AddTagExtendedProperties`, `WriteLiveValues`, `AddHistoricalValues`, `SendEvent(s)`.** | Same on vq03's HG `:5220/metrics`; both QA nodes point at `10.220.158.2:5221` | +| OTLP logs (QA) | — (vd03 already sends logs) | New: the collector's file exporter on each box starts receiving OtOpcUa log records. Check over SMB, read-only. Not a gate. | +| Overview | `:5320` shows OtOpcUa Up | vq03/vq04 `:5320`: OtOpcUa Up, 2 members / 0 unreachable, 5/5 checks | +| ScadaBridge | `/health/ready` 200 (no OtOpcUa link) | `/health/ready` 200 on both (no OtOpcUa link) | + +## 6. Rollback + +There is no schema undo in either range: no migration. The ConfigDb `.bak` exists only for a configuration restore. + +**DEV vd03 → `6c4d2990`:** +```powershell +.\Invoke-OtOpcUaInstallSwap.ps1 -InstallRoot E:\ApiInstall\OtOpcUa -Candidate E:\ApiInstall\OtOpcUa.bak.20261113-pre-c9640f5c ` + -PreviousInstall E:\ApiInstall\OtOpcUa.failed-c9640f5c -NoCarry -NoStart +``` +- The script stops the service, swaps the complete previous folder back (its own `appsettings*`, `pki`, `logs`), then + runs `-Reapply` + `-Verify`. The service stays stopped. +- Then mirror `\OtOpcUa-data` back over `E:\ApiInstall\OtOpcUa-data` and `Start-Service OtOpcUaHost`. +- Run `-Verify` again, then check health and ServiceLevel 250. +- The registry is unchanged by this deploy, so no import is needed. +- Fallback: `E:\ApiInstall\_stage-otopcua-6c4d2990\rollback.ps1` is one generation older and targets `fea7de0a`. Do not + use it for this rollback. + +**QA → `fea7de0a`:** stop both, then on each node: +```powershell +.\Invoke-OtOpcUaInstallSwap.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -Candidate E:\ApiInstall\OtOpcUa\app.bak.20261113-pre-c9640f5c ` + -PreviousInstall E:\ApiInstall\OtOpcUa\app.failed-c9640f5c -NoCarry -NoStart +``` +- After the swap, mirror `backup\data-pre-c9640f5c` back over `data\`. +- Import `backup\OtOpcUaHost-pre-c9640f5c.reg` only if step 2's `-ConfigureHostService` changed the env. A reorder + is harmless. +- Run `-Verify` on each node (exit 0), then start both, the old Primary first. +- The isolation record and denies stay valid for `fea7de0a`. The installer works independently of the host build. +- DB fallback: `OtOpcUaConfig_pre-c9640f5c_20261113.bak`. +- SDK tooling caveat: not applicable. Both builds are on SDK 2.0.1. + +**Isolation step only (§4 step 2):** the settings record is additive. Re-running the installer with the same arguments +is idempotent. `-Uninstall` removes everything, including the recorded denies; do not run it as part of a binary +rollback. + +## 7. Downtime estimates (from the previous records) + +| Tier | Previous | Estimate for this pack | +|---|---|---| +| vd03 | `fea7de0a` about **10 s** (stop 2 s); `6c4d2990` about **15 s** (stop 6 s), stop → `/health/ready` | **about 20–40 s.** The data robocopy and the in-script `-Reapply`/`-Verify` now run while the service is stopped. The step-2 record write happens with the service up. | +| QA pair | `fea7de0a` window **about 2 min** pair outage (16:48:13 → 16:50:14), of which about 70 s was the isolation install + uninstall drill. The first node took 39 s to ready because it waits for its peer; the second took 8 s. | **about 1–1.5 min** pair outage: stops about 1–2 s each, backup + swap + `-Reapply`/`-Verify` per node, then about 40 s + 8 s to ready. One-node restarts (the OTel record) took up to 32 s to stop a Secondary with its peer up. Stopping both together avoids that graceful-leave wait. | +| QA failover gate | 30 s graceful leave, then the new Primary at 250 on its first sample | same | diff --git a/docs/deploy/qa-upgrade-b8b4c18f-to-fea7de0a.md b/docs/deploy/qa-upgrade-b8b4c18f-to-fea7de0a.md index 87f0d387b..82d50f728 100644 --- a/docs/deploy/qa-upgrade-b8b4c18f-to-fea7de0a.md +++ b/docs/deploy/qa-upgrade-b8b4c18f-to-fea7de0a.md @@ -11,6 +11,10 @@ > > **DEV vd03 moved on 2026-11-13:** `fea7de0a` → `6c4d2990` (#899 read-only historian bindings; #700 checks). Record: > [`2026-11-13-vd03-fea7de0a-to-6c4d2990.md`](2026-11-13-vd03-fea7de0a-to-6c4d2990.md). QA is unchanged. +> +> **Next deploy (prepared, not executed):** [`2026-11-13-next-vd03-qa-to-c9640f5c.md`](2026-11-13-next-vd03-qa-to-c9640f5c.md). +> Note: QA now records **12** deny paths (the 2026-11-12 OTel collector install added `E:\ApiInstall\OtelCollector`); +> the 11-path list below is one short — use the recorded list. Channel: WinRM + SMB from the Mac (the `winrm` skill), used only for the on-box steps of this deploy, which the owner requested on 2026-11-12. Health, OPC UA, ScadaBridge and Overview checks ran from the Mac over HTTP / opc.tcp.