dohertj2/lmxopcua
1
0
Fork 0
Code Issues 5 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
00a5be261cc29022c2b88c6cebc0a2148c77156f
lmxopcua/docs/deploy/vd03-upgrade-b8b4c18f-to-master.md
T
Joseph Doherty aaaa55e211
v2-ci / unit-tests (push) Has been cancelled
Details
docs(#891): vd03 worker isolation + box-wide ApiInstall deny + backup lockdown 
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 19:37:04 -05:00

32 KiB
Raw Blame History

wonder-app-vd03 upgrade: b8b4c18f → master 5fb4b4db

Status: EXECUTED 2026-11-12 — vd03 runs 5fb4b4db. The readiness notes (§a–§g) were written against master 363f0d81; nothing between that and 5fb4b4db changed the schema, configuration or preflight. The execution record is §(h); the owner-action table and "Not determined" list below carry the outcomes. All six owner actions are closed and nothing is left undetermined (close-out 2026-11-12, §h last bullet).

Superseded on vd03 the same day: vd03 was upgraded again, 5fb4b4db → fea7de0a (ServiceLevel 255 → 250), as the DEV rehearsal for the QA pair. Record: qa-upgrade-b8b4c18f-to-fea7de0a.md.

Upgrade-readiness notes for the DEV box wonder-app-vd03. The box runs the OtOpcUaHost service from E:\ApiInstall\OtOpcUa: one fused admin+driver node with one FOCAS driver (Makino Z-34184). The ConfigDb is OtOpcUaConfig on the box's SQL instance (docs/deployments/wonder-app-vd03-makino-z-34184.md). Tracking issue: Gitea #793.

Baseline. The deployed build is b8b4c18f (2026-09-03), not the e82881c1 recorded in the 2026-08-11 notes. The evidence:

  • All 409 source documents in six deployed PDBs match b8b4c18f.
  • deps.json pins LocalDb 0.3.4 and MxGateway.Client 0.2.1, which only b8b4c18f used.

Everything below is the b8b4c18f..master delta. The ConfigDb schema is the same at both commits. The last migration at both is 20260731162342_AddApiKeyTable.

Executed 2026-11-12 (master 5fb4b4db), owner-authorised WinRM for this upgrade. See §(h) for the record. The general wonder-box remote-exec hold in the root CLAUDE.md is unchanged.

Owner actions, in order

# Action Blocks Source Outcome (2026-11-12)
1 Run scripts/preflight/vd03-preupgrade-b8b4c18f-configdb-checks.sql against OtOpcUaConfig, then fix every attention + blocking = YES row. The first deploy on master §g Done. Every blocking check ok (run with sqlcmd -I, see §g).
2 Add Security__ApiKeys__Pepper to the service environment (at least 16 characters, or a ${secret:} reference). The box does not have this setting. Host start: the host exits with code 1 §b Done. Generated on the box; service environment 25 → 26 variables.
3 Check that Security__Jwt__SigningKey is at least 32 UTF-8 bytes. Host start §b Done. 59 bytes; Issuer/Audience set.
4 Apply the EnableSnapshotIsolation migration to OtOpcUaConfig. Every deploy (SQL error 3952) §a Done. snapshot=ON; post-deploy deploy had no 3952.
5 Check the 22 FOCAS tags for the MaxRpm → MotorSpeedPct and AA_1/AA_2 → AA6/AA7 renames, and for String on MACRO/PMC/PARAM addresses. Tell ScadaBridge about the FOCAS value changes. Client-visible §d, check 13 Done. The one rename (Spindle/S1/MaxRpm, TAG-000b5c2fdf3f) was re-pointed and renamed MotorSpeedPct (address FIXED:Spindle/S1/MotorSpeedPct, Float64 / Double) and shipped in deployment 9adc5590 (11:06 UTC, Applied); it reads Good. Nine value-change rows. ScadaBridge: not applicable on vd03 — its only data connection there is MxGateway (localhost:5120), with no OPC UA connection to OtOpcUa; the consumer-impact list lives in scadaproj sister-project-history/otopcua.md for when ScadaBridge binds OtOpcUa (QA is still on b8b4c18f).
6 Make the #782 go/hold decision for SDK 2.0.1. The rollout runbook still marks it HOLD. Policy §c Done. Owner go for vd03, recorded in #782 (2026-11-12). The runbook's general HOLD for other hosts is unchanged.

(a) ConfigDb migration

  • One migration is in range: 20261022102459_EnableSnapshotIsolation, from commit 73e74e27.
    • It runs ALTER DATABASE CURRENT SET ALLOW_SNAPSHOT_ISOLATION ON with suppressTransaction: true.
    • It changes no tables. Down is deliberately a no-op.
    • Source: src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Migrations/20261022102459_EnableSnapshotIsolation.cs.
  • Why it is required. DraftSnapshotFactory.FromConfigDbAsync now reads the whole draft inside one IsolationLevel.Snapshot transaction. Without the option, the first read fails with SQL error 3952 and the deploy is rejected. Source: the class remarks in Configuration/Validation/DraftSnapshotFactory.cs.
  • The migration is not applied automatically. The host never migrates the ConfigDb: Program.cs:129 only migrates the secrets store. docker-dev runs a one-shot migrator container because "production owns schema changes" (docker-dev/README.md, docker-dev/docker-compose.yml). The owner applies it by hand, using either of these:
    • Option 1: dotnet ef database update 20261022102459_EnableSnapshotIsolation from src/Core/ZB.MOM.WW.OtOpcUa.Configuration, with OTOPCUA_CONFIG_CONNECTION pointing at OtOpcUaConfig (DesignTimeDbContextFactory, docs/Configuration.md).
    • Option 2: run this idempotent script in SSMS. It is the output of dotnet ef migrations script 20260731162342_AddApiKeyTable 20261022102459_EnableSnapshotIsolation --idempotent: 
      IF NOT EXISTS (SELECT * FROM [__EFMigrationsHistory] WHERE [MigrationId] = N'20261022102459_EnableSnapshotIsolation')
BEGIN ALTER DATABASE CURRENT SET ALLOW_SNAPSHOT_ISOLATION ON; END;
GO
IF NOT EXISTS (SELECT * FROM [__EFMigrationsHistory] WHERE [MigrationId] = N'20261022102459_EnableSnapshotIsolation')
BEGIN INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) VALUES (N'20261022102459_EnableSnapshotIsolation', N'10.0.7'); END;
GO
  • Login and timing. The login needs ALTER on the database. Turning the option on waits for in-flight transactions but does not need exclusive access, so it can run before the binary swap while the old build is still up. The old build is unaffected by it: it uses no IsolationLevel and ALLOW_SNAPSHOT_ISOLATION does not change default reads. A rollback needs no schema undo.

(b) Configuration: environment variables and appsettings

The 23 service environment variables against master (names only; values were not inspected):

Variable On master
ASPNETCORE_URLS, Cluster__Hostname, Cluster__Port, Cluster__PublicHostname, Cluster__SeedNodes__0, ConnectionStrings__ConfigDb, DOTNET_ENVIRONMENT, LocalDb__Path, Security__Auth__DisableLogin, Security__DeployApiKey Unchanged. Security:DeployApiKey now also gates the new GET /api/deployments/{id} (AdminUI/Api/DeployApiEndpoints.cs).
Cluster__Roles__0/1 + OTOPCUA_ROLES Unchanged (EffectiveRoles is already in the baseline). Cluster:Roles wins. If the two sets disagree, master logs a Warning. If a role were a cluster-… role, #678.6 (f292cf3b) now still loads appsettings.admin-driver.json (docs/Configuration.md).
Secrets__SqlitePath Unchanged. The Secrets 0.6.2 absolute, outside-content-root path rule (b037cc7a) is already in the baseline, and the box boots with it.
Security__Jwt__SigningKey/Issuer/Audience Re-interpreted: now validated at start (f72af76e, 3ddb6876; Security/Jwt/JwtOptions.Validate). When a key is set, it must be at least 32 UTF-8 bytes, and Issuer/Audience must be non-blank, or the host does not start. Bearer validation is now the first-party AddJwtBearer, pinned to HS256.
ServerHistorian__Enabled/Endpoint/ApiKey/UseTls Same keys. The HistorianGateway client goes 0.3.0 → 1.0.0 (230cef4e, historiangw#131). On UseTls=false, the cert options are ignored, not refused. A partially valid CaCertificatePath bundle now fails client creation. Not determined: the oldest gateway build that master works with. If Enabled=true on the box, check the box's gateway version first.
GALAXY_MXGW_API_KEY Not read by any code directly. A Galaxy config that references env:GALAXY_MXGW_API_KEY is in scope for #815: GALAXY_MXGW_ is an allowed Galaxy prefix and API_KEY matches [A-Z0-9_]. It is also allowed under #835/#849 as a shared name, so the deploy gate does not refuse it. It would only collide with a Galaxy driver whose id or name has the env form API or API_KEY (DriverCredentialPolicy.cs, the GalaxyScope definition and the comment at line 374). The box has no Galaxy driver. The 2026-08-11 notes say this key was already rejected by the gateway.
Authentication__Ldap__DevStubMode Dead at both commits. No code reads an Authentication: section; the live key is Security:Ldap:DevStubMode, which the deployed appsettings.admin-driver.json sets to false, with Transport=Ldaps. Harmless.

New settings master needs that are not in that list:

  • Security__ApiKeys__Pepper: REQUIRED. It is missing on the box.
    • ApiKeyOptionsValidator (f72af76e, Host/Configuration/ApiKeyOptionsValidator.cs) fails host start on any admin or driver node when the pepper is blank or shorter than 16 characters.
    • The box has no pepper in its environment and none in the deployed appsettings*.json. Without it the upgraded service exits during start-up. Because that happens before Serilog starts, the reason only reaches stderr and the Event Log.
    • On b8b4c18f an empty pepper already made minting refuse and verification reject every key (ApiKeySecret.cs, ApiKeyOpcUaUserAuthenticator.cs). So no working data-plane API keys exist today, and choosing a fresh pepper invalidates nothing. Check 14 counts the ApiKey rows.
    • Supply it as a ${secret:} reference or an environment variable. Never commit it.
  • Everything else is optional and has defaults:
    • OpcUa:CredentialLockout:* (#833/#844/#845; UserName/API-key only, never Anonymous).
    • Security:ApiKeys miss budgets, StoreTimeoutMs and StoreCacheSeconds.
    • DriverWrites:Timeout (10 s, #777).
    • ServerHistorian:EventReadHorizon (00:00:00 = off, #694).
    • ServerHistorian:Gateway:*.
    • Scripting:Runner: the default stays InProcess. #793 says not to enable Shadow or OutOfProcess on the wonder boxes yet.
    • Out-of-range values for these log a Warning or fail start only when they are set (docs/Configuration.md, docs/security.md).

Deployed appsettings vs master. The deployed files were diffed against the b8b4c18f and master copies, with secrets redacted.

  • appsettings.json:
    • ContinuousHistorization still carries the pre-b8b4c18f values (Capacity 0, DrainBatchSize 64; no DrainConcurrency, MaxWriteAttempts or CaptureWhilePrimaryUnreachable). Master's in-code defaults fill the absent keys, and the box's explicit values win. This only matters if continuous historization is enabled.
    • DriverWrites and EventReadHorizon are absent, so the defaults apply.
    • The box uses Secrets:MasterKey FilePath instead of the shipped EnvVarName. That is a site choice and is unchanged.
  • appsettings.driver.json: master lowers Opc.Ua from Debug to Information. The fused node loads the admin-driver overlay instead, which is unchanged.
  • Keep the box's own appsettings*.json across the swap. Do not take the publisher's copies.

Other host-level changes:

  • The publish output now contains script-worker/ (about 97 MB). It is unused while the runner is InProcess (docs/ServiceHosting.md §Script worker processes).

  • /health/ready gains three checks:

    • OPC UA endpoint started;
    • script runner (at worst Degraded);
    • deploy coordinator (at worst Degraded).

    Source: Host/Health/HealthEndpoints.cs.

  • No new ports are opened.

(c) PKI and OPC UA SDK 2.0.1 (#782)

  • SDK version. It moves from 1.5.378.106 to 2.0.1 (Directory.Packages.props), passing through 1.5.378.182 at 2da58d03. Namespace URIs and NodeIds are unchanged.
  • Certificates and policies. The PKI store is still the Directory store under OpcUa:PkiStoreRoot (default pki, beside the exe). The existing application certificate is reused with the same thumbprint and private key. Security profiles are unchanged: None, Basic256Sha256Sign and Basic256Sha256SignAndEncrypt. Sources: docs/security.md, docs/plans/2026-11-10-opcua-sdk-2-upgrade.md.
  • CA KeyUsage. #793 says CA certificates must carry the keyCertSign/cRLSign KeyUsage, or SDK 1.5.378.182+ rejects them. The upgrade plan (line 159) lists it as a preflight check. Before the swap, check every CA in pki\issuer and pki\trusted, and ScadaBridge's trust of this server's chain. A self-signed application certificate is not a CA and is not affected. The repo states no further KeyUsage rule.
  • Release status. docs/plans/2026-11-10-opcua-sdk-2-rollout.md still marks the release HOLD. Its open gates are the memory soak (#799), Windows service validation (#800) and live consumer testing (#801). vd03 would be the first Windows host to run SDK 2. Record the go/hold decision in #782.
  • ScadaBridge. The adapter passed its 18 tests on None + API key. SignAndEncrypt from ScadaBridge is still untested because ScadaBridge has no client certificate yet (ScadaBridge#121; the issue predates this upgrade).

(d) What ScadaBridge will see (b8b4c18f..master)

The detail is in ../scadaproj/.claude/skills/sister-project-history/otopcua.md; the commits are cited below.

On this FOCAS box:

  • Macros. Macros are now decoded at the right offset (665ecd55), so parts-count/parts-required (MACRO:3901/3902) change magnitude. A vacant macro reads BadNoData. A fractional value on an integer tag reads BadTypeMismatch.
  • FixedTree.
    • Positions are scaled by their envelope (69e0a622; for example 2801574 → 280.1574).
    • Servo/spindle Load is now a real percent, and spindle Load is Float64 (1dba46bd).
    • Spindle/*/MaxRpm is renamed MotorSpeedPct (1dba46bd).
    • Axes/AA_1/AA_2 are renamed AA6/AA7 (9460da51).
    • Unreported program fields read BadNoData instead of a Good 0 (26b512ad).
    • A CNC refusal makes only that tag Bad and no longer Degrades the driver (2724d7da, #740).
    • String on a PMC/PARAM/MACRO address is refused, and the tag is skipped (cc140593).
    • New nodes: a Status/ subtree (#257) and optional ALARM: conditions (#647).
  • ServiceLevel drops to 0 if the OPC UA endpoint fails to start (#762).
  • Virtual tags. A disabled virtual tag reads BadOutOfService (#787). VT→VT reads now produce values (#791).
  • Logins and API keys.
    • Credential lockout now covers UserName/API-key logins only, never Anonymous (#833/#844/#845).
    • Unknown API keys are rate-limited per client (#751).
    • Expired or disabled keys are refused for writes and acks (#563).
  • ACL. Event and subscription filtering by ACL changes only if Security:Acl:Enabled is on (#586/#588, #797/#808, #826). Whether it is on for vd03 was not determined.
  • Address-space changes. Surgically added raw nodes are now announced with model-change events (#599). Native condition state survives a rebuild (#641), and the native alarm feed survives a deploy (#662).
  • Alarms and history. Each method transition emits exactly one event (#587). Overload deferral is per condition (#838/#854/#855). Event history now carries the event type and condition name (#746). Processed reads with no historian return BadAggregateNotSupported (#682).

Not reachable here (no such driver):

  • Galaxy: OPC DA quality grading of status rows (#866, 363f0d81), first-value race (#859), acks wait for confirmation (#869).
  • MTConnect: blank string reads Good "" (#824).
  • OpcUaClient: host-name check (#816).
  • Heidenhain: new driver (#803).
  • AbCip: the native transport is the default.

(e) Install, swap and rollback

The source is docs/plans/2026-11-10-opcua-sdk-2-rollout.md (§Sequence, §Rollback). docs/ServiceHosting.md §Install/Uninstall describes a fresh install only.

  1. Publish a fresh directory: dotnet publish src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj -c Release -r win-x64 --self-contained true -o <dir>. Hash it into a manifest stored outside the directory, then copy it to the box over SMB.

  2. Back up these:

    • the whole install, with the service stopped: robocopy E:\ApiInstall\OtOpcUa <backup> /E /COPYALL /DCOPY:DAT. This covers pki\, appsettings*.json, data\ (LocalDb) and logs\;
    • the secrets SQLite file at Secrets__SqlitePath (it is outside the install), and its master-key file;
    • reg export HKLM\SYSTEM\CurrentControlSet\Services\OtOpcUaHost. It holds secrets, so protect the file;
    • BACKUP DATABASE OtOpcUaConfig.

  3. Add the pepper (action 2) and check the JWT key length (action 3) on the service key. Apply the migration (§a).

  4. Swap the directories:

    • Stop-Service OtOpcUaHost and wait for Stopped.
    • Move-Item the live install to a fresh sibling $PreviousInstall, then Move-Item the candidate into E:\ApiInstall\OtOpcUa.
    • Copy back the box's appsettings*.json, pki\ and data\.
    • Start-Service and wait for Running.

  5. Never run Install-Services.ps1, Uninstall-Services.ps1 or Refresh-Services.ps1 here.

    • sc delete wipes the 23 environment variables.
    • Install-Services.ps1 rewrites the Environment value with only 2–3 entries.
    • Refresh-Services.ps1 assumes NSSM and C:\publish.
    • The folder swap leaves the registry key untouched.

  6. Rollback.

    • Stop the service and move the candidate to a quarantine path.
    • Restore $PreviousInstall (or the robocopy backup), including its appsettings*.json, pki\ and data\.
    • Check the certificate thumbprint, then start the service.
    • Clients reconnect. Do not replay writes or acks.

    No schema undo is needed (§a). Remove the pepper only if it got in the way; b8b4c18f ignores it. The baseline already reads gzip artifacts (9dbd5e96 predates b8b4c18f), so artifacts written by master stay readable after a rollback.

    Not determined: whether the b8b4c18f build reads a LocalDb or alarm-outbox file after master has written to it. Restore data\ from the backup.

(f) Post-deploy checks

  • Boot log logs\otopcua-YYYYMMDD.log. Read its contents; the listed size shows 0 while the service holds it open.

    • Expect OtOpcUa.Host starting with roles=[admin,driver], with no roles-disagreement warning.
    • About 15 s later expect Credential-reference preflight (lmxopcua#815): every stored and deployed reference is namespaced. This comes from CredentialReferencePreflightService in AdminUI/Deployments/CredentialReferencePreflight.cs.
    • If the service stops immediately, check the Windows Event Log and stderr for the pepper/JWT FATAL: message (Program.cs, #618).

  • Health, at http://wonder-app-vd03.zmr.zimmer.com:9000:

    • /healthz: process up.
    • /health/ready: ConfigDb, Akka Up, OPC UA endpoint started, script runner, deploy coordinator.
    • /health/active.

    All three should return 200 on this single node (Host/Health/HealthEndpoints.cs, docs/ServiceHosting.md §Health endpoints).

  • /raw credential banner (#834, warn-only by owner decision). It should be absent. FOCAS has no credential fields (docs/security.md §"Credential references are namespaced per driver"). A banner listing rows means a stored, deployed or running out-of-namespace reference that the next deploy will refuse.

  • /hosts. The FOCAS driver row should be Healthy, with the per-node chips (#847/#851). Galaxy host statuses do not apply here (no Galaxy driver).

  • Deploy once from the AdminUI, then check /fleet and /deployments.

    • Expect Applied, not Applied (degraded) (#601, docs/Redundancy.md). Node rows show Applying until the address-space outcome is reported (#868).
    • A 3952 error means §a was skipped.
    • A VirtualTagNoEffectiveTrigger, CredentialReferenceOutOfScope or similar rejection means a preflight row was missed.

  • OPC UA at opc.tcp://wonder-app-vd03.zmr.zimmer.com:4840/OtOpcUa:

    • The server certificate thumbprint is unchanged (A873DF51…) and ServiceLevel is non-zero.
    • Browse the raw and UNS trees and read parts-count with the Client CLI (docs/Client.CLI.md). Expect the new magnitude.
    • From ScadaBridge: it reconnects, values are Good, and any MaxRpm/AA_x/Load bindings have been re-pointed.

(g) Preflight SQL: scripts/preflight/vd03-preupgrade-b8b4c18f-configdb-checks.sql

What it is. The script is read-only: it uses SELECT plus table variables. It is written against the b8b4c18f schema and needs SQL Server 2016+ at compatibility level 130 or higher.

How to run it: sqlcmd -S <instance> -d OtOpcUaConfig -E -I -i vd03-preupgrade-b8b4c18f-configdb-checks.sql -W -s "|" -o preflight.txt

-I is required. sqlcmd starts with QUOTED_IDENTIFIER OFF, and the script's XML-method inserts then fail with Msg 1934 after check 2 (found on the 2026-11-12 run).

Output.

  • Each check produces a header row (ok / attention / info, the row count, whether it blocks the deploy, and its source), followed by its detail rows.
  • Credential values are never printed: reference names are shown, and literals are redacted.
# Check Blocks deploy? Enforced on master by Source
0 Environment: database, compatibility level, last migration, snapshot-isolation state — — —
1 VirtualTagNoEffectiveTrigger yes DraftValidator.ValidateVirtualTagTriggers #783 d9b95dcf
2 CalculationNoEffectiveTrigger yes DraftValidator.ValidateCalculationTags rule 3 4d3fc9c8
3 Scripts that read a VirtualTag: they now resolve and start producing values no (behaviour) VirtualTagHostActor via dependency mux #791 000d7f55
4 VirtualTagDependencyCycle (approximate: bounded search, 25 hops) yes DraftValidator.ValidateVirtualTagDependencyCycles #791 000d7f55
5 Newly forbidden script APIs (#793 list plus E2's System.Threading, Type reflection, RuntimeHelpers) no (compile failure at runtime → node Bad) Core.Scripting/ForbiddenTypeAnalyzer.cs #792 0e96dd08, E2 2e916031
6 CredentialReferenceOutOfScope, plus the #836 UserCertificatePath rule yes CredentialReferenceDeployChecks, DriverCredentialPolicy #815 a224fd63, #836 63331b04, #850 3dbfe6ec
7 CredentialReferenceOtherInstance (secret: segment / env: part names another same-type driver) yes DriverCredentialPolicy.DescribeOtherInstanceProblems #835 63331b04, #849 4a6229e4/d68dff09
8 Reserved driver names (drv, DRV-<12 hex>) no (save/rename refused) DriverCredentialPolicy.IsReservedInstanceName #848/#849 d08e59ee/d68dff09
9 MTConnectMultipleDevices yes DraftValidator.ValidateMTConnectSingleDevice #628 dcc827ca
10 Sql: SqlQueryNotDeclared (10a), the write block (10b, review), writable mismatch (10c, warning) 10a/10b yes SqlTagDeployChecks #781 241318e7, f5c30ed0
11 Heidenhain drivers present (expected none; the type is new) yes HeidenhainDeployChecks #803 2b45694f
12 UNS effective-name collisions (an unchanged rule, re-checked) yes DraftValidator.ValidateUnsEffectiveLeafUniqueness —
13 FOCAS renames, String refusals and value changes no (deploy warning; client-visible) FocasTagDefinitionFactory, FOCAS wire client 1dba46bd, 9460da51, cc140593, 69e0a622, 665ecd55, 26b512ad
14 Inventory: drivers, tags, VTs, scripts, API keys — — —

Approximations. T-SQL cannot reproduce the C# regexes or System.Text.Json exactly; the script header lists every approximation. In short:

  • ctx.GetTag("…") literals are extracted after whitespace normalisation.
  • Cycles are found by a bounded path search.
  • ISJSON is stricter than System.Text.Json (comments, trailing commas).
  • The #850 merged driver+device path check is shown as review.
  • Check 5 is a text search, so its review patterns can be false positives (for example JsonElement.GetProperty).

Some branches cannot occur under the database's own CHECK constraints but are kept for completeness. For example, CK on VirtualTag makes ChangeTriggered = 0 without a timer impossible, so only the "no literal read" branch of check 1 can fire.

Not covered. EquipmentTagConfigInspector findings are warnings only (Deployment:TagConfigValidationMode=Warn by default, AdminOperationsActor). Cluster topology, ACL scope and EquipReferenceUnresolved are unchanged since the baseline.

How it was verified (2026-11-11):

  • A throwaway database otopcua_vd03preflight_<hex> was built on the local docker-dev SQL Server 2022 (localhost,14330) at compatibility level 130. It was migrated with dotnet ef database update 20260731162342_AddApiKeyTable, which is the b8b4c18f schema.
  • It was seeded with one or more rows that trigger each check. Every check fired as expected, and check 11 was the expected ok. The database was then dropped.
  • The script also ran clean against the docker-dev OtOpcUa database (master schema, 53 drivers). Every gate check was ok there, which matches master's own gate accepting that config.

Not determined

Resolved during the 2026-11-12 run (§h):

  • ServerHistorian__Enabled: enabled, pointing at http://localhost:5221 — HistorianGateway ea9f21c (Contracts/Client 1.0.0, deployed 2026-11-11).

  • JWT signing key length: 59 bytes.

  • #782 go/hold: owner go for vd03, recorded in #782.

  • Security:Acl:Enabled: off. Neither appsettings.json nor appsettings.admin-driver.json has a Security:Acl section (default false), and the 05:56 boot log has no node ACL enforcement is ENABLED warning, which OtOpcUaServerHostedService logs whenever it is on.

  • ScadaBridge's security mode against vd03: not applicable. ScadaBridge's management API on vd03 lists one data connection, MxGateway (http://localhost:5120); it has no OPC UA connection to this server.

  • Rollback reopen of data\: compatible. LocalDb stays at 0.3.4 at both commits; master's only LocalDb schema change is the additive alarm_condition_comment table (plus a move of the scripted-alarm comment trail out of alarm_condition_state.comments_json). A copy of vd03's otopcua-localdb.db taken over SMB after the upgrade has 0 rows in alarm_condition_state and alarm_condition_comment, so a rollback loses nothing. The FasterLog outbox is not in use (ContinuousHistorization:Enabled=false), and neither is the alarm store-and-forward (AlarmHistorian:Enabled=false). The rollback still restores data\ from the backup.

(h) Execution record, 2026-11-12 (b8b4c18f → 5fb4b4db)

Run over WinRM (owner-authorised for this upgrade) with staged scripts; the build was a fresh dotnet publish -r win-x64 --self-contained, zip SHA-256 D9B88959…4DD6, 1396 files, staged at E:\ApiInstall\_stage-otopcua-5fb4b4db.

  • Preflight. The ConfigDb is on the app box's own instance (WONDER-APP-VD03, SQL 16.0.4245, compat 160), not on wonder-sql-vd03. Every blocking check was ok. Check 13: one rename (Spindle/S1/MaxRpm, TAG-000b5c2fdf3f) and nine value-change rows. The tag was left as authored; on master it reads BadNodeIdUnknown until it is re-pointed to MotorSpeedPct. Inventory: 1 FOCAS driver, 22 raw tags, 2 UNS references, 0 API keys.
  • Box facts. JWT signing key 59 bytes, Issuer/Audience set. ServerHistorian enabled → http://localhost:5221 (HistorianGateway ea9f21c, deployed 2026-11-11). Box clock agrees with the Mac within seconds (w32tm source USWAP-VDC-ZMR1); ScadaBridge runs on the same box, so #873 skew does not apply to it. PKI: only the self-signed A873DF51… (not a CA); no issuer\certs.
  • Phase 1 (service up). Registry export and BACKUP DATABASE … COPY_ONLY (default backup directory, OtOpcUaConfig-pre-20261112-otopcua-5fb4b4db.bak); snapshot-isolation migration applied (snapshot=ON, last migration 20261022102459_EnableSnapshotIsolation); Security__ApiKeys__Pepper generated on the box (48 random bytes, base64, never left the box) and added to the service environment (25 → 26 variables).
  • Phase 2. Stop (1 s), robocopy of E:\ApiInstall\OtOpcUa-data to E:\ApiInstall\OtOpcUa-backups\pre-20261112-otopcua-5fb4b4db, folder swap (previous install at E:\ApiInstall\OtOpcUa.bak.20261112-pre-5fb4b4db), box appsettings*, pki\, logs\ carried, start. Downtime about 11 s. Rollback script: _stage-otopcua-5fb4b4db\rollback.ps1.
  • Verified. /healthz, /health/ready (opcua-endpoint, script-runner, configdb, config-publish, akka) and /health/active all 200. Boot log: roles=[admin,driver] with no disagreement warning, and the #815 preflight line "every stored and deployed reference is namespaced". OPC UA: ServiceLevel 255; raw and UNS parts-count Good; Program/Name carries the live program path.
  • New log lines. SDK 2 retries LDS RegisterServer/RegisterServer2 against its own endpoint (BadServiceUnsupported, backing off); this is visible now because #874 routes SDK warnings to Serilog (#876). The mesh-PSK refresh warns that ot-mesh-psk-dev is absent; the box runs the default DPS mesh transport, so the PSK is not used.
  • Post-deploy deploy (owner-authorised, 11:02 UTC). /deployments "Deploy current configuration" → deployment e4b4824a (rev 475db34b…) Sealed; chip back to "In sync"; /fleet 1 node Applied (not degraded), no failure reason; no 3952. The drift had been master's composer re-serialising the 2026-08-05 configuration (0 ConfigEdit rows since that seal). Afterwards: ServiceLevel 255, parts-count and Program/Name Good, /health/ready 200.
  • Close-out (2026-11-12, after the deploy). Done without WinRM: SMB reads of the box's appsettings, boot log and a LocalDb copy; ScadaBridge's management API; the AdminUI; and the Client CLI. The MaxRpm tag had already been re-pointed in the AdminUI and deployed as 9adc5590 (11:06:43 UTC, /fleet Applied, chip "In sync"). Client CLI reads: …/Spindle/S1/MotorSpeedPct Good (0, spindle idle), …/Spindle/S1/Load Good, …/Spindle/S1/MaxRpm BadNodeIdUnknown as expected, and the UNS parts-count/parts-required still present.

Worker isolation and box-wide denies (lmxopcua#891, 2026-11-12, owner-requested)

The vd03 layout differs from QA. The install root is E:\ApiInstall\OtOpcUa, with no app\ folder, and data lives in E:\ApiInstall\OtOpcUa-data. Copies containing secrets are spread across E:\ApiInstall: env/registry backups, kek.key files, and old pki\own\private copies in about a dozen OtOpcUa.bak* folders. So the deny covers all of E:\ApiInstall instead of listing each app folder.

Installer: E:\psrem-stage\otopcua-891-f47891ff\install, master f47891ff. SHA256: .ps1 e0c11aea…41a3, .psm1 f9425e4c…dfde.

.\Install-ScriptWorkerIsolation.ps1 -InstallRoot E:\ApiInstall\OtOpcUa -HostServiceAccount LocalSystem `
  -DenyPaths E:\ApiInstall,E:\ApiInstall\OtOpcUa\pki,E:\ApiInstall\OtOpcUa-data,C:\ProgramData\ZB\kek,E:\psrem-stage
  • Order: -WhatIf, then apply (12 steps changed), then a re-apply (12 steps unchanged).
  • What the record holds: the folders in -DenyPaths (the installer drops the install root, because InstallRootDeny already covers it), the PE cache, worker read+execute on script-worker\, and the outbound firewall block.
  • script-worker\ stays usable: its explicit (RX) comes before the deny inherited from E:\ApiInstall.
  • -ConfigureHostService deliberately left out. The vd03 runner is still InProcess (the default), so the host service environment is unchanged and no restart was needed. When the vd03 runner flip is scheduled, re-run with the same -DenyPaths plus -ConfigureHostService, then set Scripting__Runner and restart.

Backups locked down: 39 files under E:\ApiInstall and E:\psrem-stage now have a protected DACL that allows only SYSTEM and Administrators: *.reg, *envbak*.txt and *envnames*.txt. That covers OtOpcUa, HG, mxgw and the ScadaBridge deploy backups. None were deleted.

Installer warnings checked:

  • The locked backups have protected DACLs by design.
  • kek.key in HistorianGateway\data, MxGateway\data, OtOpcUa-data and ScadaBridge allows only SYSTEM (Read) and Administrators.

None of them grants anything to Everyone, Users or RESTRICTED.

After: /health/ready returns 200 for OtOpcUa :9000, Overview :5320, ScadaBridge :8085, mxgw :5130 and HG :5220.

Rollback: -Uninstall, which removes the recorded denies, the anchor service, the firewall rule and the PE cache ACLs. To undo the lockdown on a backup file, run icacls <file> /reset.

Reference in New Issue View Git Blame Copy Permalink