ScadaBridge/docs/deployment/deployment-records/2026-10-22-wonder-app-vq03-vq04-3808b49f.md

Deployment record — wonder-app-vq03 + wonder-app-vq04 (QA) — 2026-10-22 — b3021e7c → 3808b49f

This record covers the QA two-host topology:

Box	Central node	Site-a node
vq03 10.220.158.2	central-a	node-a
vq04 10.220.158.3	central-b	node-b

• Centrals on :8085; site gRPC on :8083, health on :8084.
• LocalDb replication is on.
• The shared config DB ScadaBridge is on wonder-sql-vq03.

The package is the one DEV received the same day. See 2026-10-22-wonder-app-vd03-3808b49f.md for the build delta: 191 commits, including site secrets via Deploy Artifacts, the 2026-10-21 review remediation, and Topology #38/#39/#40.

The deploy ran from the Mac with the winrm CLI. The zip and SQL went over SMB, followed by one prep script per box and seven window calls, all from one mode-switched script.

Pre-flight (breaking-change check)

Area	Change in delta	QA state	Verdict
Schema	57 → 58 ReviewRemediationIndexes (5 additive indexes)	57; AuditLog 2 132 rows, AuditLogEntries 515; ps_AuditLog_Month present; Enterprise	OK
Secrets:Replication:Enabled ≠ false → startup failure	—	false in appsettings.json on both boxes; not in the service env	OK (stale Mode/GrpcHub keys log one ignore-Warning)
JwtSigningKey ≥ 32 bytes; must match on both centrals now that the CLI uses bearer tokens	—	64 chars on both; identical hash	OK
MachineDataDb, Transport:SchemaVersionMajor, JwtRefreshThresholdMinutes removed	—	MachineDataDb still in the Central env and appsettings; the others are unset	Ignored, harmless
Actor-name deploy gate	—	33 instance, script, alarm, native-source and connection names	All valid
LocalDb replicated set 10 → 12 tables (deployed_secrets, secret_mirror_status)	Registered-table digests are compared fail-closed	Both site nodes upgraded in one window	OK; the pair reconnected, backlog 0
Data-connection secret: references	Artifact deploy resolves them centrally	The artifact hash is unchanged after the deploy, so there are no references	OK
Service env	—	Re-exported during prep; identical to the 2026-09-28 export on all four services	OK

No appsettings change. The three files were kept and restored on each box, with matching hashes:

Box	appsettings.json	appsettings.Central.json	appsettings.Site.json
vq03	F94000B1	D1320FA8	4CBAF66C
vq04	F94000B1	2FD3EC4C	D0C1AF05

Upgrade

Item	Value
Build	main @ 3808b49f, ProductVersion 1.0.0+3808b49fc7558a177219d00eac97747b28850a80, all four nodes
Artifact	E:\psrem-stage\3808b49f\scadabridge.zip on both boxes; 125 543 156 B; SHA-256 770BB992…5FB1098E47, re-verified on each box; app 1028 files, cli 223
Migration	ReviewRemediationIndexes.sql (idempotent) applied with sqlcmd -C -b -I from vq03 while both centrals were stopped; 57 → 58. SQL Server warned that IX_AuditLog_Kind_Occurred has a maximum key of 8 008 bytes against the 1 700-byte limit. Kind holds short enum names, so this is benign
DB backup	F:\MSSQL\BACKUP\ScadaBridge_pre-3808b49f_20261022T152457.bak on wonder-sql-vq03 (COPY_ONLY, CHECKSUM, COMPRESSION; VERIFYONLY valid)
Binary backups	app.bak.20261022T152457 / cli.bak.20261022T152457 plus _appsettings-keep-20261022T152457 on both boxes
Notes on box	E:\ApiInstall\ScadaBridge\_DEPLOY-NOTES-3808b49f.txt

Step	Local time
Stop Site vq04 (window opens)	15:29:58 → 15:30:28
Stop Site vq03	15:30:37 → 15:30:47
Stop Central vq04, swap vq04	15:30:55 → 15:31:12
Stop Central vq03, migrate, swap vq03, start Central vq03 → ready	15:31:19 → 15:31:49
Start Central vq04 → ready	15:32:00 → 15:32:05
Start Site vq03 → ready	15:32:12 → 15:32:21
Start Site vq04 → ready (window closes)	15:32:30 → 15:32:33

Downtime: 2 min 35 s.

Verification (from the Mac)

• Centrals: /health/ready 200 on both. database, akka-cluster (memberCount 2, unreachable 0) and required-singletons are all Healthy. /health/active is 200 on vq03 and 503 on vq04, the same active node as before.
• Sites: /health/ready 200 on both. localdb shows replicationConnected:true, backlog 0. /health/active is 200 on vq03 and 503 on vq04.
• Auth:
  • POST /auth/token returns 200; the CLI now runs on bearer tokens against QA.
  • /management without credentials returns 401, unchanged.
• Snapshots:
  • Z28061–Z28069: 242 Good / 2 Bad on both sides of the window. Z28063 showed 114 Uncertain on the first read while its subscriptions settled, then matched.
  • Alarm counts are unchanged (14, or 8 on Z28062).
  • ZIPSEN is not deployed, as before.
• Deploy Artifacts: deployment d7505cd8…, site-a success. The hash is unchanged (sha256:d79ba1a3…); ArtifactsDeployedAt is 19:32:56Z.
• Logs (pulled over SMB):
  • The usual first-probe akka-cluster … Removed ERR lines.
  • vq03 site logged four Replication session … faulted Warnings between 15:32:16 and 15:32:29, while vq04's site was still down. The session has been connected since.
  • New startup Warnings: Secrets per-node SQLite on a clustered central, and the stale Secrets:Replication/GrpcHub keys.
  • No other ERR or FTL lines.
• Smoke: flow-debug run-method of SimpleAlarmStatusRequest (id 9) with SAPID=000000 returned InboundRequest → BTDB DbCall (1 row) → Response Ok.

Rollback

1. Stop Site vq04 → vq03, then Central vq04 → vq03.
2. On each box rename app → app.failed and app.bak.20261022T152457 → app; do the same for cli.
3. Start Central vq03 → vq04, then Site vq03 → vq04.

The indexes are additive, so b3021e7c runs against 58 migrations. For an exact DB, RESTORE DATABASE [ScadaBridge] FROM DISK = 'F:\MSSQL\BACKUP\ScadaBridge_pre-3808b49f_20261022T152457.bak' WITH REPLACE with everything stopped.

Outstanding

These pre-existing items now carry startup Warnings:

• Shared stores. Set both, then recreate:

  • ScadaBridge:InboundApi:ApiKeyStore:Provider=SqlServer for inbound keys.
  • Secrets:Store:Provider=SqlServer plus Secrets:SqlServer:ConnectionString for secrets.

  Neither is needed for current QA operation: the site holds no deployed secrets.

• Stale keys. Remove the Secrets:Replication Mode/GrpcHub keys from both boxes' appsettings.json.

Housekeeping after soak: older app.bak.*, cli.bak.*, _appsettings-keep-* and E:\psrem-stage\* folders on both boxes.