lmxopcua/docs/deploy/vd03-upgrade-b8b4c18f-to-master.md

# wonder-app-vd03 upgrade: `b8b4c18f` → master `5fb4b4db`

> **Status: EXECUTED 2026-11-12** — vd03 runs `5fb4b4db`. The readiness notes (§a–§g) were written against master
> `363f0d81`; nothing between that and `5fb4b4db` changed the schema, configuration or preflight. The execution
> record is §(h); the owner-action table and "Not determined" list below carry the outcomes. **All six owner actions
> are closed and nothing is left undetermined (close-out 2026-11-12, §h last bullet).**
>
> **Superseded on vd03 the same day:** vd03 was upgraded again, `5fb4b4db` → `fea7de0a` (ServiceLevel 255 → 250), as the
> DEV rehearsal for the QA pair. Record: [`qa-upgrade-b8b4c18f-to-fea7de0a.md`](qa-upgrade-b8b4c18f-to-fea7de0a.md).

Upgrade-readiness notes for the DEV box wonder-app-vd03. The box runs the `OtOpcUaHost` service from
`E:\ApiInstall\OtOpcUa`: one fused admin+driver node with one FOCAS driver (Makino Z-34184). The ConfigDb
is `OtOpcUaConfig` on the box's SQL instance (`docs/deployments/wonder-app-vd03-makino-z-34184.md`). Tracking
issue: Gitea #793.

**Baseline.** The deployed build is `b8b4c18f` (2026-09-03), not the `e82881c1` recorded in the 2026-08-11
notes. The evidence:
- All 409 source documents in six deployed PDBs match `b8b4c18f`.
- `deps.json` pins `LocalDb 0.3.4` and `MxGateway.Client 0.2.1`, which only `b8b4c18f` used.

Everything below is the `b8b4c18f..master` delta. The ConfigDb schema is the same at both commits. The last
migration at both is `20260731162342_AddApiKeyTable`.

> **Executed 2026-11-12 (master `5fb4b4db`), owner-authorised WinRM for this upgrade.** See §(h) for the record.
> The general wonder-box remote-exec hold in the root `CLAUDE.md` is unchanged.

## Owner actions, in order

| # | Action | Blocks | Source | Outcome (2026-11-12) |
|---|---|---|---|---|
| 1 | Run `scripts/preflight/vd03-preupgrade-b8b4c18f-configdb-checks.sql` against `OtOpcUaConfig`, then fix every `attention` + `blocking = YES` row. | The first deploy on master | §g | **Done.** Every blocking check `ok` (run with `sqlcmd -I`, see §g). |
| 2 | **Add `Security__ApiKeys__Pepper`** to the service environment (at least 16 characters, or a `${secret:}` reference). The box does not have this setting. | **Host start**: the host exits with code 1 | §b | **Done.** Generated on the box; service environment 25 → 26 variables. |
| 3 | Check that `Security__Jwt__SigningKey` is at least 32 UTF-8 bytes. | Host start | §b | **Done.** 59 bytes; Issuer/Audience set. |
| 4 | Apply the `EnableSnapshotIsolation` migration to `OtOpcUaConfig`. | Every deploy (SQL error 3952) | §a | **Done.** `snapshot=ON`; post-deploy deploy had no 3952. |
| 5 | Check the 22 FOCAS tags for the `MaxRpm` → `MotorSpeedPct` and `AA_1/AA_2` → `AA6/AA7` renames, and for `String` on MACRO/PMC/PARAM addresses. Tell ScadaBridge about the FOCAS value changes. | Client-visible | §d, check 13 | **Done.** The one rename (`Spindle/S1/MaxRpm`, TAG-000b5c2fdf3f) was re-pointed and renamed `MotorSpeedPct` (address `FIXED:Spindle/S1/MotorSpeedPct`, Float64 / Double) and shipped in deployment `9adc5590` (11:06 UTC, Applied); it reads Good. Nine value-change rows. **ScadaBridge: not applicable on vd03** — its only data connection there is `MxGateway` (`localhost:5120`), with no OPC UA connection to OtOpcUa; the consumer-impact list lives in scadaproj `sister-project-history/otopcua.md` for when ScadaBridge binds OtOpcUa (QA is still on `b8b4c18f`). |
| 6 | Make the #782 go/hold decision for SDK 2.0.1. The rollout runbook still marks it **HOLD**. | Policy | §c | **Done.** Owner go for vd03, recorded in #782 (2026-11-12). The runbook's general HOLD for other hosts is unchanged. |

## (a) ConfigDb migration

- **One migration is in range:** `20261022102459_EnableSnapshotIsolation`, from commit `73e74e27`.
  - It runs `ALTER DATABASE CURRENT SET ALLOW_SNAPSHOT_ISOLATION ON` with `suppressTransaction: true`.
  - It changes no tables. `Down` is deliberately a no-op.
  - Source: `src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Migrations/20261022102459_EnableSnapshotIsolation.cs`.
- **Why it is required.** `DraftSnapshotFactory.FromConfigDbAsync` now reads the whole draft inside one
  `IsolationLevel.Snapshot` transaction. Without the option, the first read fails with SQL error 3952 and the
  deploy is rejected. Source: the class remarks in `Configuration/Validation/DraftSnapshotFactory.cs`.
- **The migration is not applied automatically.** The host never migrates the ConfigDb: `Program.cs:129` only
  migrates the secrets store. docker-dev runs a one-shot `migrator` container because "production owns schema
  changes" (`docker-dev/README.md`, `docker-dev/docker-compose.yml`). The owner applies it by hand, using either
  of these:
  - **Option 1:** `dotnet ef database update 20261022102459_EnableSnapshotIsolation` from
    `src/Core/ZB.MOM.WW.OtOpcUa.Configuration`, with `OTOPCUA_CONFIG_CONNECTION` pointing at `OtOpcUaConfig`
    (`DesignTimeDbContextFactory`, `docs/Configuration.md`).
  - **Option 2:** run this idempotent script in SSMS. It is the output of
    `dotnet ef migrations script 20260731162342_AddApiKeyTable 20261022102459_EnableSnapshotIsolation --idempotent`:
    ```sql
    IF NOT EXISTS (SELECT * FROM [__EFMigrationsHistory] WHERE [MigrationId] = N'20261022102459_EnableSnapshotIsolation')
    BEGIN ALTER DATABASE CURRENT SET ALLOW_SNAPSHOT_ISOLATION ON; END;
    GO
    IF NOT EXISTS (SELECT * FROM [__EFMigrationsHistory] WHERE [MigrationId] = N'20261022102459_EnableSnapshotIsolation')
    BEGIN INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) VALUES (N'20261022102459_EnableSnapshotIsolation', N'10.0.7'); END;
    GO
    ```
- **Login and timing.** The login needs `ALTER` on the database. Turning the option on waits for in-flight
  transactions but does not need exclusive access, so it can run before the binary swap while the old build is
  still up. **The old build is unaffected by it:** it uses no `IsolationLevel` and `ALLOW_SNAPSHOT_ISOLATION`
  does not change default reads. A rollback needs no schema undo.

## (b) Configuration: environment variables and appsettings

**The 23 service environment variables against master** (names only; values were not inspected):

| Variable | On master |
|---|---|
| `ASPNETCORE_URLS`, `Cluster__Hostname`, `Cluster__Port`, `Cluster__PublicHostname`, `Cluster__SeedNodes__0`, `ConnectionStrings__ConfigDb`, `DOTNET_ENVIRONMENT`, `LocalDb__Path`, `Security__Auth__DisableLogin`, `Security__DeployApiKey` | Unchanged. `Security:DeployApiKey` now also gates the new `GET /api/deployments/{id}` (`AdminUI/Api/DeployApiEndpoints.cs`). |
| `Cluster__Roles__0/1` + `OTOPCUA_ROLES` | Unchanged (`EffectiveRoles` is already in the baseline). `Cluster:Roles` wins. If the two sets disagree, master logs a Warning. If a role were a `cluster-…` role, #678.6 (`f292cf3b`) now still loads `appsettings.admin-driver.json` (`docs/Configuration.md`). |
| `Secrets__SqlitePath` | Unchanged. The Secrets 0.6.2 absolute, outside-content-root path rule (`b037cc7a`) is **already in the baseline**, and the box boots with it. |
| `Security__Jwt__SigningKey/Issuer/Audience` | **Re-interpreted: now validated at start** (`f72af76e`, `3ddb6876`; `Security/Jwt/JwtOptions.Validate`). When a key is set, it must be at least 32 UTF-8 bytes, and Issuer/Audience must be non-blank, or the host does not start. Bearer validation is now the first-party `AddJwtBearer`, pinned to HS256. |
| `ServerHistorian__Enabled/Endpoint/ApiKey/UseTls` | Same keys. The HistorianGateway client goes 0.3.0 → 1.0.0 (`230cef4e`, historiangw#131). On `UseTls=false`, the cert options are ignored, not refused. A partially valid `CaCertificatePath` bundle now fails client creation. **Not determined:** the oldest gateway build that master works with. If `Enabled=true` on the box, check the box's gateway version first. |
| `GALAXY_MXGW_API_KEY` | Not read by any code directly. A Galaxy config that references `env:GALAXY_MXGW_API_KEY` is **in scope** for #815: `GALAXY_MXGW_` is an allowed Galaxy prefix and `API_KEY` matches `[A-Z0-9_]`. It is also **allowed** under #835/#849 as a shared name, so the deploy gate does not refuse it. It would only collide with a Galaxy driver whose id or name has the env form `API` or `API_KEY` (`DriverCredentialPolicy.cs`, the `GalaxyScope` definition and the comment at line 374). The box has no Galaxy driver. The 2026-08-11 notes say this key was already rejected by the gateway. |
| `Authentication__Ldap__DevStubMode` | **Dead at both commits.** No code reads an `Authentication:` section; the live key is `Security:Ldap:DevStubMode`, which the deployed `appsettings.admin-driver.json` sets to `false`, with `Transport=Ldaps`. Harmless. |

**New settings master needs that are not in that list:**

- **`Security__ApiKeys__Pepper`: REQUIRED. It is missing on the box.**
  - `ApiKeyOptionsValidator` (`f72af76e`, `Host/Configuration/ApiKeyOptionsValidator.cs`) fails host start on
    any admin or driver node when the pepper is blank or shorter than 16 characters.
  - The box has no pepper in its environment and none in the deployed `appsettings*.json`. Without it the
    upgraded service exits during start-up. Because that happens before Serilog starts, the reason only reaches
    stderr and the Event Log.
  - On `b8b4c18f` an empty pepper already made minting refuse and verification reject every key
    (`ApiKeySecret.cs`, `ApiKeyOpcUaUserAuthenticator.cs`). So no working data-plane API keys exist today, and
    choosing a fresh pepper invalidates nothing. Check 14 counts the `ApiKey` rows.
  - Supply it as a `${secret:}` reference or an environment variable. Never commit it.
- Everything else is optional and has defaults:
  - `OpcUa:CredentialLockout:*` (#833/#844/#845; UserName/API-key only, never Anonymous).
  - `Security:ApiKeys` miss budgets, `StoreTimeoutMs` and `StoreCacheSeconds`.
  - `DriverWrites:Timeout` (10 s, #777).
  - `ServerHistorian:EventReadHorizon` (`00:00:00` = off, #694).
  - `ServerHistorian:Gateway:*`.
  - `Scripting:Runner`: the default stays `InProcess`. #793 says not to enable `Shadow` or `OutOfProcess` on the
    wonder boxes yet.
  - Out-of-range values for these log a Warning or fail start only when they are set (`docs/Configuration.md`,
    `docs/security.md`).

**Deployed appsettings vs master.** The deployed files were diffed against the `b8b4c18f` and `master`
copies, with secrets redacted.

- `appsettings.json`:
  - `ContinuousHistorization` still carries the pre-b8b4c18f values (`Capacity 0`, `DrainBatchSize 64`; no
    `DrainConcurrency`, `MaxWriteAttempts` or `CaptureWhilePrimaryUnreachable`). Master's in-code defaults fill
    the absent keys, and the box's explicit values win. This only matters if continuous historization is
    enabled.
  - `DriverWrites` and `EventReadHorizon` are absent, so the defaults apply.
  - The box uses `Secrets:MasterKey` `FilePath` instead of the shipped `EnvVarName`. That is a site choice and is
    unchanged.
- `appsettings.driver.json`: master lowers `Opc.Ua` from Debug to Information. The fused node loads the
  `admin-driver` overlay instead, which is unchanged.
- **Keep the box's own `appsettings*.json` across the swap.** Do not take the publisher's copies.

**Other host-level changes:**
- The publish output now contains `script-worker/` (about 97 MB). It is unused while the runner is `InProcess`
  (`docs/ServiceHosting.md` §Script worker processes).
- `/health/ready` gains three checks:
  - OPC UA endpoint started;
  - script runner (at worst Degraded);
  - deploy coordinator (at worst Degraded).

  Source: `Host/Health/HealthEndpoints.cs`.
- No new ports are opened.

## (c) PKI and OPC UA SDK 2.0.1 (#782)

- **SDK version.** It moves from 1.5.378.106 to 2.0.1 (`Directory.Packages.props`), passing through
  1.5.378.182 at `2da58d03`. Namespace URIs and NodeIds are unchanged.
- **Certificates and policies.** The PKI store is still the Directory store under `OpcUa:PkiStoreRoot` (default
  `pki`, beside the exe). The existing application certificate is reused with the same thumbprint and private
  key. Security profiles are unchanged: `None`, `Basic256Sha256Sign` and `Basic256Sha256SignAndEncrypt`.
  Sources: `docs/security.md`, `docs/plans/2026-11-10-opcua-sdk-2-upgrade.md`.
- **CA KeyUsage.** #793 says CA certificates must carry the `keyCertSign`/`cRLSign` KeyUsage, or SDK
  1.5.378.182+ rejects them. The upgrade plan (line 159) lists it as a preflight check. **Before the swap, check
  every CA in `pki\issuer` and `pki\trusted`, and ScadaBridge's trust of this server's chain.** A self-signed
  application certificate is not a CA and is not affected. The repo states no further KeyUsage rule.
- **Release status.** `docs/plans/2026-11-10-opcua-sdk-2-rollout.md` still marks the release **HOLD**. Its open
  gates are the memory soak (#799), Windows service validation (#800) and live consumer testing (#801). vd03
  would be the first Windows host to run SDK 2. Record the go/hold decision in #782.
- **ScadaBridge.** The adapter passed its 18 tests on `None` + API key. SignAndEncrypt from ScadaBridge is still
  untested because ScadaBridge has no client certificate yet (ScadaBridge#121; the issue predates this upgrade).

## (d) What ScadaBridge will see (b8b4c18f..master)

The detail is in `../scadaproj/.claude/skills/sister-project-history/otopcua.md`; the commits are cited below.

**On this FOCAS box:**
- **Macros.** Macros are now decoded at the right offset (`665ecd55`), so `parts-count`/`parts-required`
  (`MACRO:3901`/`3902`) change magnitude. A vacant macro reads `BadNoData`. A fractional value on an integer tag
  reads `BadTypeMismatch`.
- **FixedTree.**
  - Positions are scaled by their envelope (`69e0a622`; for example 2801574 → 280.1574).
  - Servo/spindle `Load` is now a real percent, and spindle Load is Float64 (`1dba46bd`).
  - `Spindle/*/MaxRpm` is renamed `MotorSpeedPct` (`1dba46bd`).
  - `Axes/AA_1`/`AA_2` are renamed `AA6`/`AA7` (`9460da51`).
  - Unreported program fields read `BadNoData` instead of a Good 0 (`26b512ad`).
  - A CNC refusal makes only that tag Bad and no longer Degrades the driver (`2724d7da`, #740).
  - `String` on a PMC/PARAM/MACRO address is refused, and the tag is skipped (`cc140593`).
  - New nodes: a `Status/` subtree (#257) and optional `ALARM:` conditions (#647).
- **ServiceLevel** drops to 0 if the OPC UA endpoint fails to start (#762).
- **Virtual tags.** A disabled virtual tag reads `BadOutOfService` (#787). VT→VT reads now produce values (#791).
- **Logins and API keys.**
  - Credential lockout now covers UserName/API-key logins only, never Anonymous (#833/#844/#845).
  - Unknown API keys are rate-limited per client (#751).
  - Expired or disabled keys are refused for writes and acks (#563).
- **ACL.** Event and subscription filtering by ACL changes only if `Security:Acl:Enabled` is on (#586/#588,
  #797/#808, #826). Whether it is on for vd03 was not determined.
- **Address-space changes.** Surgically added raw nodes are now announced with model-change events (#599).
  Native condition state survives a rebuild (#641), and the native alarm feed survives a deploy (#662).
- **Alarms and history.** Each method transition emits exactly one event (#587). Overload deferral is per
  condition (#838/#854/#855). Event history now carries the event type and condition name (#746). Processed
  reads with no historian return `BadAggregateNotSupported` (#682).

**Not reachable here (no such driver):**
- Galaxy: OPC DA quality grading of status rows (#866, `363f0d81`), first-value race (#859), acks wait for
  confirmation (#869).
- MTConnect: blank string reads Good `""` (#824).
- OpcUaClient: host-name check (#816).
- Heidenhain: new driver (#803).
- AbCip: the native transport is the default.

## (e) Install, swap and rollback

The source is `docs/plans/2026-11-10-opcua-sdk-2-rollout.md` (§Sequence, §Rollback). `docs/ServiceHosting.md`
§Install/Uninstall describes a fresh install only.

1. **Publish** a fresh directory:
   `dotnet publish src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj -c Release -r win-x64 --self-contained true -o <dir>`.
   Hash it into a manifest stored outside the directory, then copy it to the box over SMB.
2. **Back up** these:
   - the whole install, with the service stopped: `robocopy E:\ApiInstall\OtOpcUa <backup> /E /COPYALL /DCOPY:DAT`.
     This covers `pki\`, `appsettings*.json`, `data\` (LocalDb) and `logs\`;
   - the secrets SQLite file at `Secrets__SqlitePath` (it is outside the install), and its master-key file;
   - `reg export HKLM\SYSTEM\CurrentControlSet\Services\OtOpcUaHost`. **It holds secrets, so protect the file;**
   - `BACKUP DATABASE OtOpcUaConfig`.
3. **Add the pepper** (action 2) and check the JWT key length (action 3) on the service key. Apply the migration
   (§a).
4. **Swap the directories:**
   - `Stop-Service OtOpcUaHost` and wait for Stopped.
   - `Move-Item` the live install to a fresh sibling `$PreviousInstall`, then `Move-Item` the candidate into
     `E:\ApiInstall\OtOpcUa`.
   - Copy back the box's `appsettings*.json`, `pki\` and `data\`.
   - `Start-Service` and wait for Running.
5. **Never run `Install-Services.ps1`, `Uninstall-Services.ps1` or `Refresh-Services.ps1` here.**
   - `sc delete` wipes the 23 environment variables.
   - `Install-Services.ps1` rewrites the `Environment` value with only 2–3 entries.
   - `Refresh-Services.ps1` assumes NSSM and `C:\publish`.
   - The folder swap leaves the registry key untouched.
6. **Rollback.**
   - Stop the service and move the candidate to a quarantine path.
   - Restore `$PreviousInstall` (or the robocopy backup), including its `appsettings*.json`, `pki\` and `data\`.
   - Check the certificate thumbprint, then start the service.
   - Clients reconnect. Do not replay writes or acks.

   No schema undo is needed (§a). Remove the pepper only if it got in the way; `b8b4c18f` ignores it. The
   baseline already reads gzip artifacts (`9dbd5e96` predates `b8b4c18f`), so artifacts written by master stay
   readable after a rollback.

   **Not determined:** whether the `b8b4c18f` build reads a LocalDb or alarm-outbox file after master has
   written to it. Restore `data\` from the backup.

## (f) Post-deploy checks

- **Boot log** `logs\otopcua-YYYYMMDD.log`. Read its contents; the listed size shows 0 while the service holds
  it open.
  - Expect `OtOpcUa.Host starting with roles=[admin,driver]`, with no roles-disagreement warning.
  - About 15 s later expect `Credential-reference preflight (lmxopcua#815): every stored and deployed reference is namespaced.`
    This comes from `CredentialReferencePreflightService` in `AdminUI/Deployments/CredentialReferencePreflight.cs`.
  - If the service stops immediately, check the Windows Event Log and stderr for the pepper/JWT `FATAL:` message
    (`Program.cs`, #618).
- **Health**, at `http://wonder-app-vd03.zmr.zimmer.com:9000`:
  - `/healthz`: process up.
  - `/health/ready`: ConfigDb, Akka Up, OPC UA endpoint started, script runner, deploy coordinator.
  - `/health/active`.

  All three should return 200 on this single node (`Host/Health/HealthEndpoints.cs`, `docs/ServiceHosting.md`
  §Health endpoints).
- **`/raw` credential banner (#834, warn-only by owner decision).** It should be absent. FOCAS has no credential
  fields (`docs/security.md` §"Credential references are namespaced per driver"). A banner listing rows means a
  stored, deployed or running out-of-namespace reference that the next deploy will refuse.
- **`/hosts`.** The FOCAS driver row should be Healthy, with the per-node chips (#847/#851). Galaxy host statuses
  do not apply here (no Galaxy driver).
- **Deploy once from the AdminUI**, then check `/fleet` and `/deployments`.
  - Expect `Applied`, not `Applied (degraded)` (#601, `docs/Redundancy.md`). Node rows show `Applying` until the
    address-space outcome is reported (#868).
  - A 3952 error means §a was skipped.
  - A `VirtualTagNoEffectiveTrigger`, `CredentialReferenceOutOfScope` or similar rejection means a preflight row
    was missed.
- **OPC UA** at `opc.tcp://wonder-app-vd03.zmr.zimmer.com:4840/OtOpcUa`:
  - The server certificate thumbprint is unchanged (`A873DF51…`) and ServiceLevel is non-zero.
  - Browse the raw and UNS trees and read `parts-count` with the Client CLI (`docs/Client.CLI.md`). Expect the new
    magnitude.
  - From ScadaBridge: it reconnects, values are Good, and any `MaxRpm`/`AA_x`/`Load` bindings have been
    re-pointed.

## (g) Preflight SQL: `scripts/preflight/vd03-preupgrade-b8b4c18f-configdb-checks.sql`

**What it is.** The script is read-only: it uses SELECT plus table variables. It is written against the
`b8b4c18f` schema and needs SQL Server 2016+ at compatibility level 130 or higher.

**How to run it:**
`sqlcmd -S <instance> -d OtOpcUaConfig -E -I -i vd03-preupgrade-b8b4c18f-configdb-checks.sql -W -s "|" -o preflight.txt`

**`-I` is required.** `sqlcmd` starts with `QUOTED_IDENTIFIER OFF`, and the script's XML-method inserts then fail
with Msg 1934 after check 2 (found on the 2026-11-12 run).

**Output.**
- Each check produces a header row (`ok` / `attention` / `info`, the row count, whether it blocks the deploy,
  and its source), followed by its detail rows.
- Credential values are never printed: reference names are shown, and literals are redacted.

| # | Check | Blocks deploy? | Enforced on master by | Source |
|---|---|---|---|---|
| 0 | Environment: database, compatibility level, last migration, snapshot-isolation state | — | — | — |
| 1 | `VirtualTagNoEffectiveTrigger` | yes | `DraftValidator.ValidateVirtualTagTriggers` | #783 `d9b95dcf` |
| 2 | `CalculationNoEffectiveTrigger` | yes | `DraftValidator.ValidateCalculationTags` rule 3 | `4d3fc9c8` |
| 3 | Scripts that read a VirtualTag: they now resolve and start producing values | no (behaviour) | `VirtualTagHostActor` via dependency mux | #791 `000d7f55` |
| 4 | `VirtualTagDependencyCycle` (approximate: bounded search, 25 hops) | yes | `DraftValidator.ValidateVirtualTagDependencyCycles` | #791 `000d7f55` |
| 5 | Newly forbidden script APIs (#793 list plus E2's `System.Threading`, `Type` reflection, `RuntimeHelpers`) | no (compile failure at runtime → node Bad) | `Core.Scripting/ForbiddenTypeAnalyzer.cs` | #792 `0e96dd08`, E2 `2e916031` |
| 6 | `CredentialReferenceOutOfScope`, plus the #836 `UserCertificatePath` rule | yes | `CredentialReferenceDeployChecks`, `DriverCredentialPolicy` | #815 `a224fd63`, #836 `63331b04`, #850 `3dbfe6ec` |
| 7 | `CredentialReferenceOtherInstance` (`secret:` segment / `env:` part names another same-type driver) | yes | `DriverCredentialPolicy.DescribeOtherInstanceProblems` | #835 `63331b04`, #849 `4a6229e4`/`d68dff09` |
| 8 | Reserved driver names (`drv`, `DRV-<12 hex>`) | no (save/rename refused) | `DriverCredentialPolicy.IsReservedInstanceName` | #848/#849 `d08e59ee`/`d68dff09` |
| 9 | `MTConnectMultipleDevices` | yes | `DraftValidator.ValidateMTConnectSingleDevice` | #628 `dcc827ca` |
| 10 | Sql: `SqlQueryNotDeclared` (10a), the `write` block (10b, review), writable mismatch (10c, warning) | 10a/10b yes | `SqlTagDeployChecks` | #781 `241318e7`, `f5c30ed0` |
| 11 | Heidenhain drivers present (expected none; the type is new) | yes | `HeidenhainDeployChecks` | #803 `2b45694f` |
| 12 | UNS effective-name collisions (an unchanged rule, re-checked) | yes | `DraftValidator.ValidateUnsEffectiveLeafUniqueness` | — |
| 13 | FOCAS renames, `String` refusals and value changes | no (deploy warning; client-visible) | `FocasTagDefinitionFactory`, FOCAS wire client | `1dba46bd`, `9460da51`, `cc140593`, `69e0a622`, `665ecd55`, `26b512ad` |
| 14 | Inventory: drivers, tags, VTs, scripts, API keys | — | — | — |

**Approximations.** T-SQL cannot reproduce the C# regexes or `System.Text.Json` exactly; the script header lists
every approximation. In short:
- `ctx.GetTag("…")` literals are extracted after whitespace normalisation.
- Cycles are found by a bounded path search.
- `ISJSON` is stricter than `System.Text.Json` (comments, trailing commas).
- The #850 merged driver+device path check is shown as `review`.
- Check 5 is a text search, so its `review` patterns can be false positives (for example
  `JsonElement.GetProperty`).

Some branches cannot occur under the database's own CHECK constraints but are kept for completeness. For
example, `CK` on `VirtualTag` makes `ChangeTriggered = 0` without a timer impossible, so only the "no literal
read" branch of check 1 can fire.

**Not covered.** `EquipmentTagConfigInspector` findings are warnings only (`Deployment:TagConfigValidationMode=Warn`
by default, `AdminOperationsActor`). Cluster topology, ACL scope and `EquipReferenceUnresolved` are unchanged
since the baseline.

**How it was verified (2026-11-11):**
- A throwaway database `otopcua_vd03preflight_<hex>` was built on the local docker-dev SQL Server 2022
  (`localhost,14330`) at compatibility level 130. It was migrated with `dotnet ef database update
  20260731162342_AddApiKeyTable`, which is the `b8b4c18f` schema.
- It was seeded with one or more rows that trigger each check. Every check fired as expected, and check 11 was
  the expected `ok`. The database was then dropped.
- The script also ran clean against the docker-dev `OtOpcUa` database (master schema, 53 drivers). Every gate
  check was `ok` there, which matches master's own gate accepting that config.

## Not determined

Resolved during the 2026-11-12 run (§h):
- ~~`ServerHistorian__Enabled`~~: enabled, pointing at `http://localhost:5221` — HistorianGateway `ea9f21c`
  (Contracts/Client 1.0.0, deployed 2026-11-11).
- ~~JWT signing key length~~: 59 bytes.
- ~~#782 go/hold~~: owner go for vd03, recorded in #782.

- ~~`Security:Acl:Enabled`~~: **off.** Neither `appsettings.json` nor `appsettings.admin-driver.json` has a
  `Security:Acl` section (default `false`), and the 05:56 boot log has no `node ACL enforcement is ENABLED`
  warning, which `OtOpcUaServerHostedService` logs whenever it is on.
- ~~ScadaBridge's security mode against vd03~~: **not applicable.** ScadaBridge's management API on vd03 lists one
  data connection, `MxGateway` (`http://localhost:5120`); it has no OPC UA connection to this server.
- ~~Rollback reopen of `data\`~~: **compatible.** LocalDb stays at `0.3.4` at both commits; master's only LocalDb
  schema change is the additive `alarm_condition_comment` table (plus a move of the scripted-alarm comment trail
  out of `alarm_condition_state.comments_json`). A copy of vd03's `otopcua-localdb.db` taken over SMB after the
  upgrade has 0 rows in `alarm_condition_state` and `alarm_condition_comment`, so a rollback loses nothing. The
  FasterLog outbox is not in use (`ContinuousHistorization:Enabled=false`), and neither is the alarm
  store-and-forward (`AlarmHistorian:Enabled=false`). The rollback still restores `data\` from the backup.

## (h) Execution record, 2026-11-12 (`b8b4c18f` → `5fb4b4db`)

Run over WinRM (owner-authorised for this upgrade) with staged scripts; the build was a fresh
`dotnet publish -r win-x64 --self-contained`, zip SHA-256 `D9B88959…4DD6`, 1396 files, staged at
`E:\ApiInstall\_stage-otopcua-5fb4b4db`.

- **Preflight.** The ConfigDb is on the app box's own instance (`WONDER-APP-VD03`, SQL 16.0.4245, compat 160),
  not on wonder-sql-vd03. Every blocking check was `ok`. Check 13: one rename (`Spindle/S1/MaxRpm`, TAG-000b5c2fdf3f)
  and nine value-change rows. The tag was left as authored; on master it reads `BadNodeIdUnknown` until it is
  re-pointed to `MotorSpeedPct`. Inventory: 1 FOCAS driver, 22 raw tags, 2 UNS references, 0 API keys.
- **Box facts.** JWT signing key 59 bytes, Issuer/Audience set. `ServerHistorian` enabled → `http://localhost:5221`
  (HistorianGateway `ea9f21c`, deployed 2026-11-11). Box clock agrees with the Mac within seconds (w32tm source
  USWAP-VDC-ZMR1); ScadaBridge runs on the same box, so #873 skew does not apply to it. PKI: only the self-signed
  `A873DF51…` (not a CA); no `issuer\certs`.
- **Phase 1 (service up).** Registry export and `BACKUP DATABASE … COPY_ONLY` (default backup directory,
  `OtOpcUaConfig-pre-20261112-otopcua-5fb4b4db.bak`); snapshot-isolation migration applied (`snapshot=ON`, last
  migration `20261022102459_EnableSnapshotIsolation`); `Security__ApiKeys__Pepper` generated on the box (48 random
  bytes, base64, never left the box) and added to the service environment (25 → 26 variables).
- **Phase 2.** Stop (1 s), robocopy of `E:\ApiInstall\OtOpcUa-data` to `E:\ApiInstall\OtOpcUa-backups\pre-20261112-otopcua-5fb4b4db`,
  folder swap (previous install at `E:\ApiInstall\OtOpcUa.bak.20261112-pre-5fb4b4db`), box `appsettings*`, `pki\`,
  `logs\` carried, start. **Downtime about 11 s.** Rollback script: `_stage-otopcua-5fb4b4db\rollback.ps1`.
- **Verified.** `/healthz`, `/health/ready` (opcua-endpoint, script-runner, configdb, config-publish, akka) and
  `/health/active` all 200. Boot log: `roles=[admin,driver]` with no disagreement warning, and the #815 preflight
  line "every stored and deployed reference is namespaced". OPC UA: ServiceLevel 255; raw and UNS `parts-count`
  Good; `Program/Name` carries the live program path.
- **New log lines.** SDK 2 retries LDS `RegisterServer`/`RegisterServer2` against its own endpoint
  (`BadServiceUnsupported`, backing off); this is visible now because #874 routes SDK warnings to Serilog (#876). The
  mesh-PSK refresh warns that `ot-mesh-psk-dev` is absent; the box runs the default DPS mesh transport, so the
  PSK is not used.
- **Post-deploy deploy (owner-authorised, 11:02 UTC).** `/deployments` "Deploy current configuration" →
  deployment `e4b4824a` (rev `475db34b…`) Sealed; chip back to "In sync"; `/fleet` 1 node **Applied** (not
  degraded), no failure reason; no 3952. The drift had been master's composer re-serialising the 2026-08-05
  configuration (0 `ConfigEdit` rows since that seal). Afterwards: ServiceLevel 255, `parts-count` and
  `Program/Name` Good, `/health/ready` 200.
- **Close-out (2026-11-12, after the deploy).** Done without WinRM: SMB reads of the box's appsettings, boot log
  and a LocalDb copy; ScadaBridge's management API; the AdminUI; and the Client CLI. The `MaxRpm` tag had already
  been re-pointed in the AdminUI and deployed as `9adc5590` (11:06:43 UTC, `/fleet` Applied, chip "In sync").
  Client CLI reads: `…/Spindle/S1/MotorSpeedPct` Good (0, spindle idle), `…/Spindle/S1/Load` Good,
  `…/Spindle/S1/MaxRpm` `BadNodeIdUnknown` as expected, and the UNS `parts-count`/`parts-required` still present.

## Worker isolation and box-wide denies (lmxopcua#891, 2026-11-12, owner-requested)

The vd03 layout differs from QA. The install root is `E:\ApiInstall\OtOpcUa`, with no `app\` folder, and data lives in `E:\ApiInstall\OtOpcUa-data`. Copies containing secrets are spread across `E:\ApiInstall`: env/registry backups, `kek.key` files, and old `pki\own\private` copies in about a dozen `OtOpcUa.bak*` folders. So the deny covers **all of `E:\ApiInstall`** instead of listing each app folder.

**Installer:** `E:\psrem-stage\otopcua-891-f47891ff\install`, master `f47891ff`. SHA256: `.ps1` `e0c11aea…41a3`, `.psm1` `f9425e4c…dfde`.

```powershell
.\Install-ScriptWorkerIsolation.ps1 -InstallRoot E:\ApiInstall\OtOpcUa -HostServiceAccount LocalSystem `
  -DenyPaths E:\ApiInstall,E:\ApiInstall\OtOpcUa\pki,E:\ApiInstall\OtOpcUa-data,C:\ProgramData\ZB\kek,E:\psrem-stage
```

- **Order:** `-WhatIf`, then apply (12 steps changed), then a re-apply (12 steps unchanged).
- **What the record holds:** the folders in `-DenyPaths` (the installer drops the install root, because `InstallRootDeny` already covers it), the PE cache, worker read+execute on `script-worker\`, and the outbound firewall block.
- **`script-worker\` stays usable:** its explicit `(RX)` comes before the deny inherited from `E:\ApiInstall`.
- **`-ConfigureHostService` deliberately left out.** The vd03 runner is still `InProcess` (the default), so the host service environment is unchanged and no restart was needed. When the vd03 runner flip is scheduled, re-run with the same `-DenyPaths` **plus** `-ConfigureHostService`, then set `Scripting__Runner` and restart.

**Backups locked down:** 39 files under `E:\ApiInstall` and `E:\psrem-stage` now have a protected DACL that allows only SYSTEM and Administrators: `*.reg`, `*envbak*.txt` and `*envnames*.txt`. That covers OtOpcUa, HG, mxgw and the ScadaBridge deploy backups. None were deleted.

**Installer warnings checked:**
- The locked backups have protected DACLs by design.
- `kek.key` in `HistorianGateway\data`, `MxGateway\data`, `OtOpcUa-data` and `ScadaBridge` allows only SYSTEM (Read) and Administrators.

None of them grants anything to Everyone, Users or RESTRICTED.

**After:** `/health/ready` returns 200 for OtOpcUa `:9000`, Overview `:5320`, ScadaBridge `:8085`, mxgw `:5130` and HG `:5220`.

**Rollback:** `-Uninstall`, which removes the recorded denies, the anchor service, the firewall rule and the PE cache ACLs. To undo the lockdown on a backup file, run `icacls <file> /reset`.