docs(deploy): next vd03 (6c4d2990) + QA (fea7de0a) deploy pack to c9640f5c, read-only ConfigDb preflight
v2-ci / unit-tests (push) Has been cancelled
v2-ci / unit-tests (push) Has been cancelled
No migrations in either range; no #834/#898/#899/#897/#896 findings on DEV or QA. QA isolation record holds 12 deny paths (OTel collector added 2026-11-12), not 11. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
@@ -0,0 +1,329 @@
|
||||
# Next deploy: DEV vd03 `6c4d2990` and QA vq03/vq04 `fea7de0a` → master `c9640f5c`
|
||||
|
||||
> **Status: PREPARED, NOT EXECUTED (2026-11-13).** This pack covers both tiers, plus the read-only ConfigDb preflight,
|
||||
> which was run over direct SQL from the Mac on 2026-11-13. Nothing was built, staged or run on any wonder box. **No
|
||||
> blocker found; no issue filed.**
|
||||
>
|
||||
> **Target:** `origin/master` **`c9640f5c`** (`c9640f5ced383e2a722702409adbc62298731994`, #904). If master moves before the window, either pin to
|
||||
> `c9640f5c` (as the 6c4d2990 run pinned) or re-run §1–§3 for the new range.
|
||||
>
|
||||
> Templates: [`2026-11-13-vd03-fea7de0a-to-6c4d2990.md`](2026-11-13-vd03-fea7de0a-to-6c4d2990.md) (vd03 layout and
|
||||
> scripts), [`qa-upgrade-b8b4c18f-to-fea7de0a.md`](qa-upgrade-b8b4c18f-to-fea7de0a.md) (QA window, canary, #890/#891
|
||||
> denies, "Worker isolation across the next swap"), [`vd03-upgrade-b8b4c18f-to-master.md`](vd03-upgrade-b8b4c18f-to-master.md)
|
||||
> §e/§f and "Worker isolation and box-wide denies".
|
||||
|
||||
## 1. Commit ranges and what changes
|
||||
|
||||
### vd03: `6c4d2990..c9640f5c` (9 commits)
|
||||
|
||||
| Commit | Change | Effect on vd03 (one fused node, one FOCAS driver, 22 tags, runner `InProcess`) |
|
||||
|---|---|---|
|
||||
| `efaf3435` #900 | HistorianGateway `FailedPrecondition` + `AGGREGATE_NOT_SUPPORTED_FOR_TAG` maps to `BadAggregateNotSupported`, with no health failure. Adds `Grpc.StatusProto 2.76.0`. | **Inert today.** vd03's HG `ea9f21c` predates historiangw#135 (`a220ec0`) and still answers `Internal`, which keeps the current handling. It takes effect when HG ≥ `a220ec0` is deployed. |
|
||||
| `9375c9e2` | docs (the 6c4d2990 record) | — |
|
||||
| `d5288388` #895 review, `c9640f5c` #904 | Galaxy alarm reverse map: re-areaed, subtag-provider and pre-subscription references. A picker-form ack waits up to 2 s for the feed snapshot. | Not reachable (no Galaxy driver). |
|
||||
| `2470f82e` #872 | Galaxy status rows Pending/Warning → **Uncertain**; detail 8001 → `BadTypeMismatch`. A Pending write reply is now "not committed". | Not reachable. |
|
||||
| `b85e004b` #902 | AdminUI: `MonacoEditor.DisposeAsync` on a gone circuit logs at Debug, not `[ERR]`. | Fixes the "Observed" `[ERR]` from the 6c4d2990 record. |
|
||||
| `35576485` #903 | AdminUI `/deployments`: the drift chip is never a cached "In sync". The first render shows "Checking for changes...". A failed check shows "Drift check failed". | Fixes the stale "In sync" chip from the 6c4d2990 record. |
|
||||
| `2cb8a2c3` #901 | Installer records its settings and gains `-Reapply`/`-Verify`. New `scripts/install/Invoke-OtOpcUaInstallSwap.ps1`. | Changes the deploy procedure (§4). No runtime change. |
|
||||
| `f0c6cf96` #834 | Running Galaxy/OpcUaClient drivers **refuse** out-of-namespace credential refs at resolve time (the warn-only path is gone). The preflight logs one **Error** per finding, and the `/raw` banner is `alert-danger`. | FOCAS has no credential fields. Preflight: 0 findings (§3). Only the boot line wording changes (§5). |
|
||||
|
||||
### QA: `fea7de0a..c9640f5c` (46 commits: the 37 above `6c4d2990` plus the 9 vd03 commits)
|
||||
|
||||
The vd03 rows above, plus everything vd03 already got on 2026-11-13 (`fea7de0a..6c4d2990`, 37 commits):
|
||||
|
||||
- **Historian.**
|
||||
- `ServerHistorian:Gateway:EnableNativeTimeWeighted` now **defaults to `true`** (#694, `53978ca6`). TimeAverage/Total
|
||||
are advertised and served natively (12 aggregates). QA HG `fa9cf31` includes `9f711c1`, so it supports this.
|
||||
- Alarm history goes out in `SendEvents` batches of ≤200, with a per-event fallback (#746).
|
||||
- A Boolean node refuses TimeAverage/Total up front with `BadAggregateNotSupported` (#897). A String/DateTime node
|
||||
refuses the numeric-only aggregates.
|
||||
- `historianReadOnly: true` TagConfig plus the **new `ServerHistorian:ProvisionTags` setting** (#899).
|
||||
- Float/Double/Byte/SByte historized tags are now provisioned (#898). See the upgrade hazard in §3.
|
||||
- **Address space (#896):** a UNS reference to an alarm-bearing raw tag no longer gets a UNS value Variable. The
|
||||
alarm surfaces only as the condition on the equipment folder.
|
||||
- **Galaxy (#895):** picker-authored native alarms resolve and ack. Galaxy spans and `galaxy.events.*` counters are
|
||||
exported (#892).
|
||||
- **Telemetry 0.1.0 → 0.3.1** (scadaproj#55/#56).
|
||||
- The OTel Resource carries `service.version` (the git SHA), `site.id` (the Cluster id) and `node.role`.
|
||||
- **With `Exporter=Otlp`, Serilog logs go over OTLP too.** QA has `OtOpcUa__Telemetry__Exporter=Otlp` and
|
||||
`OtlpEndpoint=http://127.0.0.1:4317` since 2026-11-12, so logs will start to reach the local `ZbOtelCollector`.
|
||||
- 0.3 refuses a scheme-less `host:port` when `Exporter=Otlp`. The recorded value has `http://`, so it passes;
|
||||
re-check it in the on-box preflight.
|
||||
- **AdminUI #761 (`fa8c3d1a`):** a bodyless `POST /api/deployments` returned 404 on fused admin+driver nodes. It now
|
||||
deploys.
|
||||
- **Drivers.**
|
||||
- AB Legacy MicroLogix function files are refused locally with `BadNotSupported`, not reported as a comms fault (#592).
|
||||
- Heidenhain gains the `Controller/Family`, `Status/*StateName` and `Tool/AxisName` leaves, PLC
|
||||
`INPUT_DWORD`/`OUTPUT_DWORD`, and new refusal-code mappings (#829/#853). Not reachable on QA.
|
||||
- **Installer #890 (`f75ee7d0`):** `-DenyPaths`. Already applied on QA from staged scripts on 2026-11-12.
|
||||
- Other commits are docs, tests or docker-dev only (#893).
|
||||
|
||||
### What a ScadaBridge consumer would notice
|
||||
|
||||
**On DEV and QA: nothing.** ScadaBridge on both tiers reads no OtOpcUa data; its only data connection is `MxGateway`
|
||||
(QA pack §verification, vd03 fea7de0a section). For an OtOpcUa-bound ScadaBridge elsewhere:
|
||||
|
||||
- **Galaxy.**
|
||||
- Pending/Warning rows now read **Uncertain**, not Good/Bad (#872).
|
||||
- A Galaxy write that MXAccess answers Pending is reported **not committed**. The node reverts and the later commit
|
||||
arrives as a data change.
|
||||
- 8001 now reads `BadTypeMismatch`.
|
||||
- **Alarm-bearing references (#896):** a UNS binding to a reference on an alarm-bearing tag now gets
|
||||
`BadNodeIdUnknown`. Before, it read `BadWaitingForInitialData` forever. Bind the alarm as an event subscription.
|
||||
- **HistoryRead-Processed:** TimeAverage/Total on a Boolean node now answers `BadAggregateNotSupported` (#897). With
|
||||
HG ≥ `a220ec0`, the same holds for a numeric node backed by a discrete AVEVA tag (#900). TimeAverage/Total are
|
||||
advertised again (QA range).
|
||||
- **Credential refs (#834):** a Galaxy or OpcUaClient driver with an out-of-namespace credential reference no longer
|
||||
connects. It is not a ScadaBridge-visible wire change, but its nodes go Bad.
|
||||
- No proto, NodeId-scheme or ServiceLevel-semantics change. Contracts are unchanged.
|
||||
|
||||
## 2. Migrations and settings
|
||||
|
||||
- **ConfigDb migrations: none in either range.** The only file touched under `src/Core/ZB.MOM.WW.OtOpcUa.Configuration`
|
||||
is `Validation/DraftValidator.cs` (#899 `HistorianReadOnlyInvalid` deploy rule, already on vd03). The repo head stays
|
||||
`20261022102459_EnableSnapshotIsolation` (9 migrations). Both DBs are already there (§3).
|
||||
- **New optional setting:** `ServerHistorian:ProvisionTags` = `CreateOrUpdate` (default, the previous behaviour) |
|
||||
`CreateOnly` (Gateway backend only) | `Off`. An **unknown value, or `CreateOnly` on a SQL backend, fails host start**.
|
||||
Neither tier sets it today (vd03 record). **Leave it unset:** there are 0 historized tags on either tier.
|
||||
- **Changed default:** `ServerHistorian:Gateway:EnableNativeTimeWeighted` `false` → `true` (QA only; vd03 already has it).
|
||||
The kill switch is `ServerHistorian__Gateway__EnableNativeTimeWeighted=false`. It is not needed: QA HG `fa9cf31` supports
|
||||
native time-weighted reads.
|
||||
- **Telemetry (QA):** no new keys. 0.3 reads the same `OtOpcUa:Telemetry` section and adds optional
|
||||
`SiteId`/`NodeRole`/`ServiceVersion`/`ResourceAttributes:*` overrides.
|
||||
- **Required env:** no new required variable. The existing required ones stay: `Security__ApiKeys__Pepper`
|
||||
(≥16 chars) and the JWT key on both tiers, and `LocalDb__Replication__ApiKey`, identical on both QA nodes.
|
||||
- **Packages:** `Grpc.StatusProto 2.76.0` (new). Telemetry(.Serilog) is 0.3.1 (already on vd03). LocalDb 0.3.7,
|
||||
MxGateway.Client 0.4.0 and HistorianGateway.Client 1.0.0 are unchanged.
|
||||
- **Peer services:** none to upgrade first. HG `ea9f21c` (vd03) and `fa9cf31` (QA) serve every RPC this build uses;
|
||||
#900 is capability-tolerant. mxgw `76a71bc8` is unaffected, because neither tier has a Galaxy driver.
|
||||
|
||||
## 3. Read-only preflight (run 2026-11-13 from the Mac, direct SQL)
|
||||
|
||||
Run as `wwadmin` with `sqlcmd -C -I`, the password passed in `SQLCMDPASSWORD`, against `OtOpcUaConfig` on
|
||||
`wonder-app-vd03.zmr.zimmer.com,1433` (DEV) and `wonder-sql-vq03.zmr.zimmer.com,1433` (QA). The queries were SELECTs only,
|
||||
plus the existing script's local table variables. Two passes:
|
||||
|
||||
1. `scripts/preflight/vd03-preupgrade-b8b4c18f-configdb-checks.sql` unchanged (15 checks). It already reproduces
|
||||
`DriverCredentialPolicy` / `CredentialReferenceDeployChecks` on the draft rows (checks 6/7).
|
||||
2. Ad-hoc SELECTs for what the script does not cover:
|
||||
- the **deployed** side of `CredentialReferencePreflight.ScanAsync`: the latest `Status=2` (Sealed) `ArtifactBlob`,
|
||||
`DECOMPRESS`ed and searched;
|
||||
- #898/#899/#897/#896 over `dbo.Tag.TagConfig` (`isHistorized`, `historianTagname`, `historianReadOnly`, `alarm`) and
|
||||
`dbo.VirtualTag.Historize`.
|
||||
|
||||
| Check | DEV vd03 | QA (shared ConfigDb on wonder-sql-vq03) |
|
||||
|---|---|---|
|
||||
| `__EFMigrationsHistory` | 9 rows, head `20261022102459_EnableSnapshotIsolation` = repo head. Snapshot isolation ON, compat 160. | Same: 9 rows, same head, snapshot ON, compat 160. |
|
||||
| Inventory | 1 driver (`FOCAS` `focas-z-34184`), 1 device, 22 tags (8 Double, 8 Int32, 6 String), 0 VT, 0 scripted alarms, 2 UNS refs, 10 deployments | 1 driver (`Calculation` `script-runner-canary`), 1 device, 1 tag (Int32), 1 VT (Int32), 0 scripted alarms, 0 UNS refs, 3 deployments |
|
||||
| Latest Sealed deployment | `595cb909`, rev `fa9a2e28`, 2026-11-13 06:34Z (gzip, 13,655 B JSON) | `5905003e`, rev `82dd0598`, 2026-11-13 00:05Z (gzip, 4,146 B JSON) |
|
||||
| **#834 draft**: script checks 6 (#815/#836 OutOfScope) and 7 (#835/#849 OtherInstance) | **ok, 0 rows** | **ok, 0 rows** |
|
||||
| **#834 draft**: credential-ruled driver types (Galaxy / OpcUaClient / Heidenhain) and any `secret:`/`env:` text in DriverConfig/DeviceConfig | none. FOCAS has no credential fields, and neither config holds a reference. | none. Calculation has no credential fields; no reference. |
|
||||
| **#834 deployed**: latest Sealed artifact contains `"Galaxy"` / `"OpcUaClient"` / `"Heidenhain"` / `secret:` / `env:` | 0 / 0 / 0 / 0 / 0 | 0 / 0 / 0 / 0 / 0 |
|
||||
| → #834 verdict | **No finding.** The boot preflight will log "every stored and deployed reference is namespaced". | **No finding.** |
|
||||
| **#898** historized Float/Double/Byte/SByte tags (first start would `EnsureTags`) | 0 (0 tags with `isHistorized: true` at all; no `historianTagname`) | 0 (no historized tag; the VT has `Historize=0`) |
|
||||
| **#899** tags with a `historianReadOnly` key / a non-boolean value (`HistorianReadOnlyInvalid`) | 0 / 0 | 0 / 0 |
|
||||
| **#899** `ProvisionTags` default | Nothing to provision, so the default `CreateOrUpdate` is harmless. | Same. |
|
||||
| **#897** historized Boolean tags/VTs (TimeAverage/Total now refused up front) | 0 (no Boolean tags at all). Aggregates are chosen per client request, not stored in config; no historized Boolean node can receive one. | 0 |
|
||||
| **#896** UNS references to alarm-bearing tags | 0 (no tag has an `alarm` object) | 0 |
|
||||
| Other script checks (#783, Calc trigger, #791 ×2, #792, #848, MTConnect, #781 Sql, #803 Heidenhain, UNS collisions) | all **ok** | all **ok** |
|
||||
| Script check 0 "Environment" | `attention`: a stale expectation. The script was written for `b8b4c18f`, and the migration it asks for is already applied. Not an issue. | Same. |
|
||||
| Script check 13 (FOCAS renames, info) | 9 info rows. These are the b8b4c18f→master FOCAS address notes, already live since the 2026-11-12 deploy. | ok |
|
||||
|
||||
**Verdict: no blocker on either tier; no issue filed.** Re-run both passes just before the window if the configuration
|
||||
has changed since 2026-11-13. On QA the canary is the only configuration.
|
||||
|
||||
### Pack correction found during preflight: QA records 12 deny paths, not 11
|
||||
|
||||
The 2026-11-12 OTel collector install (scadaproj `infra/otel-collector/wonder/2026-11-12-wonder-otel-collector.md`,
|
||||
21:46–21:47) re-ran the installer on vq03/vq04 and added **`E:\ApiInstall\OtelCollector`** (`DenyPathRecord` +
|
||||
`SensitiveDirDeny` changed). The QA pack's 11-path list is therefore one short. If the one-time full-argument run
|
||||
(§4) used the 11-path list, it would **remove** the collector deny. §4 uses the 12-path list. As before, the rule is to
|
||||
use exactly what `reg query` shows.
|
||||
|
||||
The same record also shows that the OtOpcUa Primary failed over to **vq04** during the restarts at 21:49–21:50. Read
|
||||
ServiceLevel before the window to know which node is Primary now.
|
||||
|
||||
## 4. Build, stage, one-time isolation record, swap
|
||||
|
||||
### Build (Mac, isolated worktree at `c9640f5c`; not done here)
|
||||
|
||||
```bash
|
||||
git worktree add ../OtOpcUa-wt-c9640f5c c9640f5c && cd ../OtOpcUa-wt-c9640f5c
|
||||
dotnet publish src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj -c Release -r win-x64 --self-contained true -o ../publish-c9640f5c
|
||||
# expect ~1396+ files, OtOpcUa.Host.dll ProductVersion 1.0.0+c9640f5c…, script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe present;
|
||||
# deps.json: Telemetry(.Serilog) 0.3.1, LocalDb 0.3.7, MxGateway.Client 0.4.0, HistorianGateway.Client 1.0.0, Grpc.StatusProto 2.76.0
|
||||
cd ../publish-c9640f5c && zip -qr ../otopcua-c9640f5c.zip . && shasum -a 256 ../otopcua-c9640f5c.zip > ../otopcua-c9640f5c.zip.sha256
|
||||
```
|
||||
|
||||
Copy the zip and `.sha256` over SMB into the staging folders below, and check the hash on the box before expanding.
|
||||
Stage the three install scripts from `c9640f5c` next to them. Their SHA-256 values at `c9640f5c`:
|
||||
|
||||
| File | SHA-256 |
|
||||
|---|---|
|
||||
| `scripts/install/Install-ScriptWorkerIsolation.ps1` | `7d4078cb93a42b362ab50ba5b79bc293bf92001d5444f58d5c6733e0d51c4914` |
|
||||
| `scripts/install/ScriptWorkerIsolation.psm1` | `ae9d84fd0077a730ecaaf01b8d476ee83755c2d3fb163e752379c276f227d902` |
|
||||
| `scripts/install/Invoke-OtOpcUaInstallSwap.ps1` | `9498fd8092c3b2ea95f7c7f8275b704d6bed99dfeea0dcfadfbab1045d79e88d` |
|
||||
|
||||
All remote steps below are on-box deploy steps (`winrm` skill, deploy scope). Run them in an elevated Windows
|
||||
PowerShell, from the staged `install\` folder. **Never run `Install-Services.ps1`, `Uninstall-Services.ps1` or
|
||||
`Refresh-Services.ps1`** (vd03 pack §e step 5).
|
||||
|
||||
### DEV vd03 (`E:\ApiInstall\OtOpcUa`, data in `E:\ApiInstall\OtOpcUa-data`, runner `InProcess`)
|
||||
|
||||
Staging: `E:\ApiInstall\_stage-otopcua-c9640f5c\` (zip, `.sha256`, `install\`). Backups go to
|
||||
`E:\ApiInstall\OtOpcUa-backups\pre-20261113-otopcua-c9640f5c\`. Create that folder with a protected DACL (SYSTEM +
|
||||
Administrators only, #891).
|
||||
|
||||
1. **Prepare (service up).**
|
||||
- `reg export HKLM\SYSTEM\CurrentControlSet\Services\OtOpcUaHost` into the backup folder.
|
||||
- Check the hash, then expand to `E:\ApiInstall\OtOpcUa.new-c9640f5c` and check `ProductVersion`.
|
||||
- `BACKUP DATABASE [OtOpcUaConfig] TO DISK = N'OtOpcUaConfig-pre-20261113-otopcua-c9640f5c.bak' WITH COPY_ONLY, CHECKSUM`.
|
||||
There is no migration, so this is only a configuration fallback.
|
||||
- Read the service env (names and count only; expect 28). Check that `OtOpcUa__Telemetry__OtlpEndpoint` has a scheme.
|
||||
2. **One-time #901 settings record** (service can stay up; vd03 was isolated by the pre-#901 installer, so
|
||||
`Invoke-OtOpcUaInstallSwap.ps1` refuses until the record exists):
|
||||
```powershell
|
||||
reg query HKLM\SYSTEM\CurrentControlSet\Services\OtOpcUaScriptWorker\Parameters /v DenyPaths
|
||||
# expect exactly: E:\ApiInstall, E:\ApiInstall\OtOpcUa\pki, E:\ApiInstall\OtOpcUa-data, C:\ProgramData\ZB\kek, E:\psrem-stage
|
||||
.\Install-ScriptWorkerIsolation.ps1 -InstallRoot E:\ApiInstall\OtOpcUa -HostServiceAccount LocalSystem `
|
||||
-DenyPaths E:\ApiInstall,E:\ApiInstall\OtOpcUa\pki,E:\ApiInstall\OtOpcUa-data,C:\ProgramData\ZB\kek,E:\psrem-stage -WhatIf
|
||||
.\Install-ScriptWorkerIsolation.ps1 -InstallRoot E:\ApiInstall\OtOpcUa -HostServiceAccount LocalSystem `
|
||||
-DenyPaths E:\ApiInstall,E:\ApiInstall\OtOpcUa\pki,E:\ApiInstall\OtOpcUa-data,C:\ProgramData\ZB\kek,E:\psrem-stage
|
||||
.\Install-ScriptWorkerIsolation.ps1 -Verify # VERIFY OK, exit 0
|
||||
reg query HKLM\SYSTEM\CurrentControlSet\Services\OtOpcUaScriptWorker\Parameters /v DenyPaths # same 5 paths
|
||||
```
|
||||
- **No `-ConfigureHostService`:** the vd03 runner stays `InProcess` (vd03 pack §"Worker isolation and box-wide denies").
|
||||
- Expect **`SettingsRecord` changed**, `DenyPathRecord` unchanged, and every other step unchanged. The `reisolate.ps1`
|
||||
run after the 6c4d2990 swap restored them.
|
||||
- The "protected DACL" WARNs under `E:\psrem-stage` and the locked backups are pre-existing and expected.
|
||||
- If `reg query` lists anything other than the 5 paths, stop and use the recorded list.
|
||||
3. **Swap.**
|
||||
- `Stop-Service OtOpcUaHost`.
|
||||
- `robocopy E:\ApiInstall\OtOpcUa-data <backup>\OtOpcUa-data /E /COPY:DAT` (stopped, so the SQLite triplets are consistent).
|
||||
- Then run:
|
||||
```powershell
|
||||
.\Invoke-OtOpcUaInstallSwap.ps1 -InstallRoot E:\ApiInstall\OtOpcUa -Candidate E:\ApiInstall\OtOpcUa.new-c9640f5c `
|
||||
-PreviousInstall E:\ApiInstall\OtOpcUa.bak.20261113-pre-c9640f5c
|
||||
```
|
||||
- The default `-Carry` (`appsettings*.json`, `pki`, `logs`) is right for vd03.
|
||||
- The script tolerates an already-stopped service. It moves the folders, carries the box files, runs `-Reapply`
|
||||
(expect `WorkerFolderAcl`, `InstallRootDeny` and the `pki` `SensitiveDirDeny` **changed**), then `-Verify`, then starts.
|
||||
- **Gate:** the script must finish without throwing. A failed `-Verify` leaves the service **stopped**: fix the
|
||||
problem, re-run `-Reapply`/`-Verify`, then start.
|
||||
- After the start, run `.\Install-ScriptWorkerIsolation.ps1 -Verify` once more and record its output (exit 0).
|
||||
|
||||
### QA vq03 + vq04 (stop both, swap both, start both)
|
||||
|
||||
Layout: `E:\ApiInstall\OtOpcUa\{app,data,pki}`, service `OtOpcUaHost` (LocalSystem), runner `OutOfProcess` (Calculation +
|
||||
VirtualTag). Staging: `E:\psrem-stage\otopcua-c9640f5c\` (zip, `.sha256`, `install\`). Backups go to
|
||||
`E:\psrem-stage\otopcua-c9640f5c\backup\`, created with a protected DACL (#891).
|
||||
|
||||
1. **Prepare (both nodes, services up).**
|
||||
- Expand to `E:\ApiInstall\OtOpcUa\app.new-c9640f5c` (check the hash and ProductVersion).
|
||||
- `reg export …\OtOpcUaHost backup\OtOpcUaHost-pre-c9640f5c.reg`.
|
||||
- Read the env names and count: expect 48 on vq03 and 47 on vq04, including `Scripting__Runner*`,
|
||||
`Scripting__Worker__*`, the pepper, `LocalDb__Replication__ApiKey` and `OtOpcUa__Telemetry__*`. Check that
|
||||
`OtlpEndpoint` has a scheme.
|
||||
- Once: `BACKUP DATABASE OtOpcUaConfig … COPY_ONLY, CHECKSUM, COMPRESSION` on wonder-sql-vq03 →
|
||||
`OtOpcUaConfig_pre-c9640f5c_20261113.bak`, then `RESTORE VERIFYONLY`.
|
||||
- Read ServiceLevel on both nodes to learn which one is Primary (250). It may be **vq04** since the 2026-11-12 OTel restarts.
|
||||
2. **One-time #901 settings record (both nodes, services up).** QA was isolated by the pre-#901 installer
|
||||
(DenyPaths only):
|
||||
```powershell
|
||||
reg query HKLM\SYSTEM\CurrentControlSet\Services\OtOpcUaScriptWorker\Parameters /v DenyPaths
|
||||
# expect these 12 (the 11 #891 paths + the OTel collector folder added 2026-11-12):
|
||||
$deny = 'E:\ApiInstall\OtOpcUa','E:\ApiInstall\OtOpcUa\pki','E:\ApiInstall\OtOpcUa\data','C:\ProgramData\ZB\kek',
|
||||
'E:\ApiInstall\ScadaBridge','E:\ApiInstall\HistorianGateway','E:\ApiInstall\MxAccessGateway','E:\ApiInstall\Overview',
|
||||
'E:\ApiInstall\btdbmanager','E:\psrem-stage','E:\psrem','E:\ApiInstall\OtelCollector'
|
||||
.\Install-ScriptWorkerIsolation.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -HostServiceAccount LocalSystem -ConfigureHostService -DenyPaths $deny -WhatIf
|
||||
.\Install-ScriptWorkerIsolation.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -HostServiceAccount LocalSystem -ConfigureHostService -DenyPaths $deny
|
||||
.\Install-ScriptWorkerIsolation.ps1 -Verify # VERIFY OK, exit 0
|
||||
reg query HKLM\SYSTEM\CurrentControlSet\Services\OtOpcUaScriptWorker\Parameters /v DenyPaths # same 12 paths
|
||||
```
|
||||
- If `reg query` differs from the 12 above, use **exactly the recorded list**. A missing entry in `$deny` removes that
|
||||
deny.
|
||||
- `-ConfigureHostService` stays, because QA's runner flip used it.
|
||||
- Expect **`SettingsRecord` changed** and `DenyPathRecord` unchanged.
|
||||
- `HostServiceEnv` may report **changed** as a reorder only, as in the #890 run, if the OTel entries were appended
|
||||
after the `Scripting__Worker__*` pair. Confirm the env count is unchanged (48/47) and that no value changed
|
||||
(compare SHA-256 prefixes, never print values). Every other step should be unchanged.
|
||||
- The vq04 `kek\historiangw.kek`/`mxgateway.kek` and locked `.reg` protected-DACL WARNs are expected.
|
||||
3. **Stop both.** Stop the Secondary (240) first, then the Primary (250). On each node, while stopped, run
|
||||
`robocopy data backup\data-pre-c9640f5c /E /COPY:DAT` and the same for `pki`.
|
||||
4. **Swap each node** (services stopped):
|
||||
```powershell
|
||||
.\Invoke-OtOpcUaInstallSwap.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -Candidate E:\ApiInstall\OtOpcUa\app.new-c9640f5c `
|
||||
-PreviousInstall E:\ApiInstall\OtOpcUa\app.bak.20261113-pre-c9640f5c -Carry 'appsettings*.json','logs' -NoStart
|
||||
```
|
||||
- `pki\` and `data\` sit beside `app\` and are not moved.
|
||||
- Expect the re-apply to report `WorkerFolderAcl` and `InstallRootDeny` changed, then `-Verify` OK.
|
||||
- **Gate: the script returns without throwing on both nodes.** If `-Verify` fails on either node, do not start
|
||||
either; fix it, then run `-Reapply` and `-Verify`.
|
||||
5. **Start both.** Start the node that was Primary first; it waits for its peer, about 40 s last time. Then start the
|
||||
other node.
|
||||
6. On each node, after ready: `.\Install-ScriptWorkerIsolation.ps1 -Verify` → `VERIFY OK`, exit 0. Record the output.
|
||||
|
||||
## 5. Post-deploy gates
|
||||
|
||||
Run these from the Mac over HTTP / opc.tcp. Read logs over SMB (`logs\otopcua-YYYYMMDD.log`). The Windows-listed size of
|
||||
an open log reads 0; read its contents.
|
||||
|
||||
| Gate | DEV vd03 | QA vq03 + vq04 |
|
||||
|---|---|---|
|
||||
| Isolation | `-Verify` exit 0 after the swap and after the start (§4) | same, both nodes |
|
||||
| Version | live `OtOpcUa.Host.dll` (SMB copy) ProductVersion `1.0.0+c9640f5c…`; SHA-256 equals the published build | same, both nodes |
|
||||
| Health | `/healthz`, `/health/ready`, `/health/active` all 200 on `:9000` | `/health/ready` 200 on both, with 5/5 components including `Out-of-process script runner ready.`; `/health/active` 200 on the Primary, 503 on the Secondary |
|
||||
| ServiceLevel | **250** | **250 / 240**, stable over repeated fresh-session reads |
|
||||
| Failover (QA) | — | Stop the Primary: the other node reaches **250** and `/health/active` 200. Restart the old Primary: **240**, never above, **no failback**. |
|
||||
| LocalDb sync (QA) | — | `localdb-replication` connected, oplog backlog 0 on both, before and after the failover |
|
||||
| Canary (QA) | — | `ns=2;s=script-runner-canary/Device1/canary-calc-seconds` and `ns=3;s=EQ-ec3979550d90/canary-vt-second` are **Good and advancing about once a second** on both endpoints. `otopcua_script_worker_up` = 1 per shard. The #780 runaway drill (`while(true)` VT, expect 2 kills + quarantine, siblings Good, 250/240 unchanged, then delete and redeploy back to rev `82dd0598…`) is **optional**. |
|
||||
| Driver data | FOCAS raw: 22 variables, 21 Good, `Program/BlockCount` `BadNoData` (as before) | the canary above |
|
||||
| Boot log | `roles=[admin,driver]`; **`Credential-reference preflight (lmxopcua#815/#834): every stored and deployed reference is namespaced.`** (the new wording; any `Error` line from this preflight is a stop); `Historian gateway probe: the gateway is reachable`; no `FTL` | same, both nodes; also no `Telemetry` endpoint FATAL (0.3 check) |
|
||||
| **#903 drift chip** | On `/deployments` (login disabled on DEV): the first render shows **"Checking for changes..."**, then "In sync". Make a harmless draft change, for example a tag description on `/raw`, with no deploy. Reload `/deployments`: it must show **"Configuration drift"** at once. Revert the change: the chip returns to "In sync", and the revision is still `fa9a2e28…`. | Optional. The QA AdminUI needs the `localhost` forwarder plus a temporary role-mapping row (QA pack §2). If you do it, make the change on the canary and revert it. Delete the temporary row afterwards. |
|
||||
| **#902 Monaco** | Open `/scripts/new` (the Monaco editor) and close the browser tab without saving. Then search the log after that time for `MonacoEditor` / `Unhandled exception in circuit`: expect **no `[ERR]`** (it is Debug now). | Same if the AdminUI is opened; otherwise n/a |
|
||||
| **Historian writes** | HG `http://wonder-app-vd03…:5220/metrics` `historian_gateway_grpc_requests_total`: snapshot before the stop and about 2 min after ready. Only `HistorianStatus/Probe` should grow. **No `EnsureTags`, `AddTagExtendedProperties`, `WriteLiveValues`, `AddHistoricalValues`, `SendEvent(s)`.** | Same on vq03's HG `:5220/metrics`; both QA nodes point at `10.220.158.2:5221` |
|
||||
| OTLP logs (QA) | — (vd03 already sends logs) | New: the collector's file exporter on each box starts receiving OtOpcUa log records. Check over SMB, read-only. Not a gate. |
|
||||
| Overview | `:5320` shows OtOpcUa Up | vq03/vq04 `:5320`: OtOpcUa Up, 2 members / 0 unreachable, 5/5 checks |
|
||||
| ScadaBridge | `/health/ready` 200 (no OtOpcUa link) | `/health/ready` 200 on both (no OtOpcUa link) |
|
||||
|
||||
## 6. Rollback
|
||||
|
||||
There is no schema undo in either range: no migration. The ConfigDb `.bak` exists only for a configuration restore.
|
||||
|
||||
**DEV vd03 → `6c4d2990`:**
|
||||
```powershell
|
||||
.\Invoke-OtOpcUaInstallSwap.ps1 -InstallRoot E:\ApiInstall\OtOpcUa -Candidate E:\ApiInstall\OtOpcUa.bak.20261113-pre-c9640f5c `
|
||||
-PreviousInstall E:\ApiInstall\OtOpcUa.failed-c9640f5c -NoCarry -NoStart
|
||||
```
|
||||
- The script stops the service, swaps the complete previous folder back (its own `appsettings*`, `pki`, `logs`), then
|
||||
runs `-Reapply` + `-Verify`. The service stays stopped.
|
||||
- Then mirror `<backup>\OtOpcUa-data` back over `E:\ApiInstall\OtOpcUa-data` and `Start-Service OtOpcUaHost`.
|
||||
- Run `-Verify` again, then check health and ServiceLevel 250.
|
||||
- The registry is unchanged by this deploy, so no import is needed.
|
||||
- Fallback: `E:\ApiInstall\_stage-otopcua-6c4d2990\rollback.ps1` is one generation older and targets `fea7de0a`. Do not
|
||||
use it for this rollback.
|
||||
|
||||
**QA → `fea7de0a`:** stop both, then on each node:
|
||||
```powershell
|
||||
.\Invoke-OtOpcUaInstallSwap.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -Candidate E:\ApiInstall\OtOpcUa\app.bak.20261113-pre-c9640f5c `
|
||||
-PreviousInstall E:\ApiInstall\OtOpcUa\app.failed-c9640f5c -NoCarry -NoStart
|
||||
```
|
||||
- After the swap, mirror `backup\data-pre-c9640f5c` back over `data\`.
|
||||
- Import `backup\OtOpcUaHost-pre-c9640f5c.reg` only if step 2's `-ConfigureHostService` changed the env. A reorder
|
||||
is harmless.
|
||||
- Run `-Verify` on each node (exit 0), then start both, the old Primary first.
|
||||
- The isolation record and denies stay valid for `fea7de0a`. The installer works independently of the host build.
|
||||
- DB fallback: `OtOpcUaConfig_pre-c9640f5c_20261113.bak`.
|
||||
- SDK tooling caveat: not applicable. Both builds are on SDK 2.0.1.
|
||||
|
||||
**Isolation step only (§4 step 2):** the settings record is additive. Re-running the installer with the same arguments
|
||||
is idempotent. `-Uninstall` removes everything, including the recorded denies; do not run it as part of a binary
|
||||
rollback.
|
||||
|
||||
## 7. Downtime estimates (from the previous records)
|
||||
|
||||
| Tier | Previous | Estimate for this pack |
|
||||
|---|---|---|
|
||||
| vd03 | `fea7de0a` about **10 s** (stop 2 s); `6c4d2990` about **15 s** (stop 6 s), stop → `/health/ready` | **about 20–40 s.** The data robocopy and the in-script `-Reapply`/`-Verify` now run while the service is stopped. The step-2 record write happens with the service up. |
|
||||
| QA pair | `fea7de0a` window **about 2 min** pair outage (16:48:13 → 16:50:14), of which about 70 s was the isolation install + uninstall drill. The first node took 39 s to ready because it waits for its peer; the second took 8 s. | **about 1–1.5 min** pair outage: stops about 1–2 s each, backup + swap + `-Reapply`/`-Verify` per node, then about 40 s + 8 s to ready. One-node restarts (the OTel record) took up to 32 s to stop a Secondary with its peer up. Stopping both together avoids that graceful-leave wait. |
|
||||
| QA failover gate | 30 s graceful leave, then the new Primary at 250 on its first sample | same |
|
||||
@@ -11,6 +11,10 @@
|
||||
>
|
||||
> **DEV vd03 moved on 2026-11-13:** `fea7de0a` → `6c4d2990` (#899 read-only historian bindings; #700 checks). Record:
|
||||
> [`2026-11-13-vd03-fea7de0a-to-6c4d2990.md`](2026-11-13-vd03-fea7de0a-to-6c4d2990.md). QA is unchanged.
|
||||
>
|
||||
> **Next deploy (prepared, not executed):** [`2026-11-13-next-vd03-qa-to-c9640f5c.md`](2026-11-13-next-vd03-qa-to-c9640f5c.md).
|
||||
> Note: QA now records **12** deny paths (the 2026-11-12 OTel collector install added `E:\ApiInstall\OtelCollector`);
|
||||
> the 11-path list below is one short — use the recorded list.
|
||||
|
||||
Channel: WinRM + SMB from the Mac (the `winrm` skill), used only for the on-box steps of this deploy, which the owner
|
||||
requested on 2026-11-12. Health, OPC UA, ScadaBridge and Overview checks ran from the Mac over HTTP / opc.tcp.
|
||||
|
||||
Reference in New Issue
Block a user