ScadaBridge/docs/deployment/deploy-prep/2026-11-13-qa-vq03-vq04

Deploy prep: QA vq03/vq04, coordinated central-pair migration window (ScadaBridge#107 item 4) + #116 leader-auto-down

Prepared: 2026-11-13. Not executed. Running it is a deploy, and it waits for the owner's separate go (#107, 2026-11-13).

Target build: main @ 3a59c3f6. That is the tip this pack was analysed at. If main moves before the window, re-run §1.4 and §2 over 3a59c3f6..<new tip>: look for new migrations under src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/ and option changes in *Options.cs / *OptionsValidator.cs. If anything is new, regenerate the script in §2.2.

Sources:

• The read-only SQL preflight on wonder-sql-vq03 (2026-11-13, §3).
• SMB reads of the live app\ folders and appsettings on both boxes, and HTTP health probes. Nothing was written on any box.
• Deployment records 2026-10-22-wonder-app-vq03-vq04-3808b49f.md and 2026-10-22-wonder-app-vd03-vq03-vq04-0e540291.md.
• The DEV vd03 424db155 upgrade of 2026-11-11 (migrations 58→64, flip of 2,394 rows / 8 MB in about 1.5 s).
• Issues #107, #116, #126, #131 and #52.

Files

File	What
README.md	This pack.
preflight.sql	Read-only preflight (SELECTs and sp_spaceused only). Re-run it right before the window.
migrate-58-to-64.idempotent.sql	dotnet ef migrations script 20261022133907_ReviewRemediationIndexes --idempotent, generated at 3a59c3f6 (EF tools 10.0.10, runtime 10.0.13). It covers six migrations, 59→64. SHA-256 488da93db0f464ee17687a32a1dabbbe810553c109db5b65d5b3ecbd5c34ecc0. It is unedited and starts with a UTF-8 BOM at byte 0, which sqlcmd accepts; #126's problem was a BOM after a prepended header.

---

0. Correction: QA is on 0e540291 with 58 migrations, not b3021e7c with 57

Evidence	Value
ZB.MOM.WW.ScadaBridge.Host.dll in E:\ApiInstall\ScadaBridge\app\ on both boxes (SMB read)	1.0.0+0e540291
__EFMigrationsHistory on wonder-sql-vq03	58 rows; the last is 20261022133907_ReviewRemediationIndexes
Records	2026-10-22: b3021e7c→3808b49f (pair window, migration 58), then 3808b49f→0e540291 (rolling, no migration). Also on 2026-10-22: shared SQL key and secret stores, and the standby relay

As a result, these items from the brief are already on QA and are not part of this window:

• bearer-token CLI auth;
• MachineDataDb removed (the Central file still carries a dead MachineDataDb key, which is ignored);
• site secrets via Deploy Artifacts, with the gRPC hub retired;
• ReviewRemediationIndexes;
• the shared zbauth/zbsecrets SQL stores on the central pair.

The effective delta is 0e540291..3a59c3f6: 128 commits and 6 migrations.

1. What changes: 0e540291..3a59c3f6

1.1 Schema (central DB): 6 migrations

See §2.

1.2 Behaviour changes worth knowing on QA

Area	Change	QA impact
Audit (#41, #107, #109, #113–#115, #42)	Hash chain, head export, maintenance login, Parquet archive and archive retention.	All default OFF and config-only. Nothing is enabled in this window (§6).
AuditLog clustered key (#111)	(OccurredAtUtc, EventId) replaces EventId. Binaries from 5e7d5389 on purge either shape.	Old binaries cannot partition-switch the flipped table. That is the reason for the both-centrals-stopped window and for restore-only rollback.
AuditLog ingest (#124)	Set-based ingest uses FORCESEEK, so concurrent writers no longer deadlock (1205).	Positive.
Cluster downing (#106, #132)	Opt-in leader-auto-down with shut-out restart. The Host now exits 75 after any unplanned ActorSystem termination (it used to exit 0), under any strategy.	Windows SCM recovery has to restart on a non-zero exit. See §5.3, which checks it in the window.
Host stop (#76)	A routine service stop logs as Information, not [FTL] host terminated unexpectedly.	Drop the old "known FTL on stop" exclusion from log review.
Alarms (#129)	Severity is banded with higher = more urgent; the raw value is on Priority. The MES 900–999 band keys off Priority.	The Alarm Summary Sev column and its sort flip visibly. #131: QA has nothing to change: CvdReactor.AlarmStatus (39) is a stub, and there is no TestMesMachine.
MES-reportable (#59, #60, #102, #104)	Per-binding IsMesReportable on native alarm sources, plus UI badges and CSV columns.	Additive, delivered by migration AddNativeAlarmSourceMesReportable.
Security (#46, Auth 0.5.0)	Active sessions re-query LDAP group membership: ScadaBridge:Security:LdapGroupRefresh, Enabled true, every 15 min, 60 s cache.	QA's centrals bind to their box-local GLAuth (127.0.0.1:3893) with a service account, which this supports. A user removed from a group loses the role within about 15 min.
Central HTTP (#108)	An https:// entry in ASPNETCORE_URLS now needs ScadaBridge:Node:Tls (a certificate) or the node refuses to boot.	QA serves plain http://…:8085. The prep step confirms there is no https:// in either Central's service env (§5.1).
Akka.Remote TLS (#64)	ScadaBridge:Cluster:RemotingTls, default off.	None.
DCL OPC UA (#117, #121, #112, #58; SDK 1.5.378.106 → 2.0.1, #125)	Good subcodes now map to Good. Secure sessions present a real app cert (OpcUaGlobal:*). BadResourceUnavailable is treated as transient. Scalar writes are coerced to the node type.	QA has no OPC UA connection. Its only data connection is MxGateway, so there is no exposure.
DCL MxGateway (#128)	The resume cursor ignores worker_sequence 0 sentinels.	Positive for QA's single connection.
Site runtime (#135, #136, #36)	Post-boot settle sweep: replicated instances start within seconds. Site commands fail over within the call. The legacy WriteBatchAndWaitAsync waits for its own reset echo.	Faster site start after the swap.
Inbound API (#103, #75, #80)	Recompile and invalidate reach the peer central. Compile errors are visible through api-method validate/recompile.	Positive on a pair.
Pending updates (#62, #63)	A DB rowversion watermark short-circuits an unchanged fleet (this is what the 18 RowVersion columns are for). The counts appear on the Health dashboard and CLI.	Needs migration AddConfigGraphRowVersions.
Transport (#45, #110, #118, #93–#95, #100)	Optional detached bundle signing (default off). Derived templates round-trip. Name lookups are case-insensitive.	None unless bundles are used.
Management (#105)	Case-variant instance override names are refused with a did-you-mean hint.	Behaviour change for scripts that send wrong-case names.
Logging (#56, #120, #94)	Akka logging goes through Serilog. ExecutionId, InstanceName and ScriptName are added to the LogContext.	More (and better) lines in the file logs.
Telemetry 0.1.0 → 0.3.1 (scadaproj#55/#56)	service.version = git SHA. Logs now go over OTLP when Exporter=Otlp. The endpoint check is stricter (a scheme-less host:port fails).	QA's service env (2026-11-12) points at http://127.0.0.1:4317, which has a scheme, so it boots. The box-local ZbOtelCollector starts receiving logs as well. Watch E:\ApiInstall\OtelCollector\data\logs.jsonl (about 1.8 GB cap per box).
CLI (#137)	Secret-bearing option values can be read from a file or stdin.	Optional.
Health (#97)	Site /health/active uses the shared role-scoped check.	The Overview site chips now show active and standby correctly.

1.3 Shared-library and package pins

Package	QA today (0e540291)	Tip	Note
ZB.MOM.WW.Auth.*	0.3.0	0.5.0	BindMode is still ServiceAccount. Adds the passwordless group lookup (#46).
ZB.MOM.WW.LocalDb.*	0.3.4	0.3.7	Wire-compatible, but the pair stops and starts together. The library now enforces LocalDb:Replication:ApiKey itself, which must be byte-identical on both site nodes (it already is: QA's own fail-closed interceptor has been connected since 2026-09-10). The tombstone ledger grows while a peer is down (#16). MaxBatchSize: 16 in both Site files is a retired pin; it is harmless.
ZB.MOM.WW.Telemetry(.Serilog)	0.1.0	0.3.1	See above.
ZB.MOM.WW.Theme	0.4.1	0.5.3	ScrollNav rail (#61).
ZB.MOM.WW.Secrets	0.6.2	0.6.2	Unchanged.
Microsoft.Extensions/EF/ASP.NET	10.0.7	10.0.13	Sqlite pool fix (#98). SqlClient 6.1.7.
OPCFoundation Opc.Ua.Client	1.5.378.106	2.0.1	No OPC UA connections on QA.
Parquet.Net	—	6.1.0	Archive and export (off).

1.4 Settings and env

No setting has to change for this build to boot on QA. No shipped appsettings changed in the range, and every new option has a safe default. New keys, all optional:

• ScadaBridge:Cluster:{LeaderAutoDown:{RestartWhenShutOut,ShutOutGrace},RemotingTls:*}
• ScadaBridge:Node:Tls:*
• ScadaBridge:Communication:GrpcClientIdleTtl
• ScadaBridge:DeploymentManager:PendingUpdatesFullEvaluationEvery
• ScadaBridge:SiteRuntime:InstanceSweepSettle{Interval,Window}Seconds
• ScadaBridge:Security:LdapGroupRefresh:*
• ScadaBridge:Transport:Signing:*
• OpcUaGlobal:{ApplicationUri,ApplicationCertificateSubject,OwnCertificateStorePath,AutoCreateApplicationCertificate,ApplicationCertificateLifetimeMonths}
• AuditLog:{MaintenanceConnectionString,HashChain:*,HashChain:HeadExport:*,Archive:*}
• LocalDb:Replication:{MaxSnapshotDeferredRows,TombstoneHardCap}

Changes this window does make: only the #116 strategy flip (§5.4).

Drift found, not fixed here: the vq03 and vq04 appsettings.Central.json differ in JwtExpiryMinutes (25 vs 15) and IdleTimeoutMinutes (60 vs 30). The CLI bearer token is signed with the shared key, so the pair still interoperates, but session lifetime depends on which central answers. Align them (pick one pair of values) as a separate change, or here if the owner says so.

2. Migrations 58 → 64

2.1 Ordered list (EF timestamp order = apply order)

#	Migration	Issue	What it does	Cost on QA	Old-binary compatible?
59	20261109200211_AddNativeAlarmSourceMesReportable	#102/#59	Nullable IsMesReportable on native-source bindings/overrides	Metadata only	Yes (additive)
60	20261109203753_PrefixSearchIndexes	#50	Prefix-search indexes for Deployments and config-audit	Tiny tables	Yes
61	20261109211317_AddAuditLogHashChain	#41	ChainSeq/RowHash on AuditLog. AuditLogChainHead and AuditLogChainTombstone. Two filtered indexes, wrapped in EXEC(N'…') (the #126 fix). Roles scadabridge_audit_chainer (new) and _purger/_writer (exist), grants, and a DENY to the writer on the chain tables	3,441 rows; seconds	No: old binaries can't partition-purge once the columns exist
62	20261110032839_AddConfigGraphRowVersions	#62	Non-concurrency rowversion + index on 18 configuration tables. SQL Server fills each table as it adds the column	All config tables are under 2,300 rows; seconds	Yes (ignored)
63	20261110033732_FlipAuditLogClusteredKey	#111	Offline whole-table rebuild: drop the 10 NCIs + PK, re-add the PK clustered on (OccurredAtUtc, EventId) on ps_AuditLog_Month, add UX_AuditLog_EventId_Occurred, recreate 12 NCIs. Guarded by "clustered key ordinal 1 = EventId"	18.3 MB reserved / 3,441 rows → about 2–4 s (DEV: 8 MB / 2,394 rows in about 1.5 s). Log about 2× = about 40 MB, against 142 MB free in a 200 MB log (64 MB autogrowth, 240 GB free on G:)	No: older binaries can't partition-switch a flipped table
64	20261110120000_AuditMaintenanceGrants	#113	scadabridge_audit_purger: ALTER ANY DATASPACE + UPDATE (SourceNode) on dbo.AuditLog	Instant	Yes

Owner's §item-4 list vs EF order. AddNativeAlarmSourceMesReportable sorts first, by timestamp. The EF script applies migrations in timestamp order, which is also the order DEV vd03 used. The relative order of the five audit/config migrations is the one the owner listed.

2.2 How they are applied

• Tool: the generated idempotent script migrate-58-to-64.idempotent.sql in this folder. Per the #126 deploy note, QA can run it as is, with no GO insertion. It is 6 batches, each BEGIN TRANSACTION … COMMIT; GO.
  • IdempotentMigrationScriptTests (MSSQL fixture) proves that from-57 → head runs clean and a re-run is a no-op.
  • The hand scripts in docs/plans/sql/ remain the fallback. They were BOM-fixed in c7dfc21b. There is no hand script for migrations 59 and 64, so the hand route needs EF fragments as well. That makes it two tools for one job; prefer the single EF script.
• When: with both centrals and both sites stopped (§5). This is required by migrations 61 and 63.
• Where from: the Mac, straight to SQL (direct SQL is always allowed, so no WinRM is needed for this step):

  cd docs/deployment/deploy-prep/2026-11-13-qa-vq03-vq04
  shasum -a 256 migrate-58-to-64.idempotent.sql   # must be 488da93d…c34ecc0
  printf 'SET XACT_ABORT ON;\n' > /tmp/xact.sql   # session-scoped; same connection as the script
  SQLCMDPASSWORD=… sqlcmd -S wonder-sql-vq03.zmr.zimmer.com,1433 -U <login> -C -d ScadaBridge -I -b \
      -i /tmp/xact.sql,migrate-58-to-64.idempotent.sql
  

  The fallback is the same command with Windows sqlcmd -C -b -I on vq03, as in the 2026-10-22 window. vq04 has no sqlcmd.
• Idempotency:
  • Every statement is guarded by IF NOT EXISTS (… __EFMigrationsHistory …).
  • Role creation is additionally guarded by IF DATABASE_PRINCIPAL_ID(…) IS NULL, and the flip by the key-shape check.
  • Re-running after a partial failure therefore completes only the missing migrations. A failed batch rolls back whole, because of XACT_ABORT + -b.
• Expected after:
  • 64 rows, with the last one 20261110120000_AuditMaintenanceGrants.
  • Clustered key ordinal 1 = OccurredAtUtc.
  • AuditLogChainHead/AuditLogChainTombstone exist.
  • 3 scadabridge_audit_* roles.
  • Re-run preflight.sql sections 1, 4 and 7.

3. Read-only SQL preflight: results (2026-11-13 ~02:45 EST)

Server wonder-sql-vq03.zmr.zimmer.com,1433 (WONDER-SQL-VQ03, SQL Server 2022 RTM-CU24 16.0.4245.2, Enterprise), DB ScadaBridge. Login wwadmin, password via SQLCMDPASSWORD only, -C. SELECTs only. The script is in preflight.sql.

Check	Result
__EFMigrationsHistory	58; last 20261022133907_ReviewRemediationIndexes (ProductVersion 10.0.7)
Recovery model	FULL; log_reuse_wait NOTHING; compatibility 160
Log backups	Regular: full at 22:00 daily, log at about 12:00 / 24:00. A COPY_ONLY backup does not disturb the chain
dbo.AuditLog	3,441 rows; reserved 18.3 MB (data 10.7 MB, index 3.2 MB, unused 4.4 MB); 2026-05-29 → 2026-11-12
Clustered key now	PK_AuditLog = (EventId, OccurredAtUtc) on ps_AuditLog_Month, so the flip has not been applied yet
pf_AuditLog_Month	24 boundaries, the last 2027-12-01
Log file	200 MB, 57 MB used (28.6 %), 64 MB autogrowth, unlimited; G: 245,855 MB free
Data file	904 MB, 64 MB autogrowth; F: 249,267 MB free. The largest table is KpiSample (4.9 M rows, 804 MB), which the window does not touch
SourceNode backfill	Not run. 843 rows have SourceNode IS NULL: InboundRequest 793 (newest 2026-07-07) and InboundAuthFailure 50 (newest 2026-06-27), all from before M5.6. Every row since 2026-07-07 is stamped (central-a 1,742, central-b 777, node-a 59, node-b 20)
Hash-chain objects	None yet. Roles scadabridge_audit_writer and _purger exist; _chainer does not. DeploymentRecords.RowVersion already exists, and AddConfigGraphRowVersions handles that table (covered by the from-57 test)
Topology	1 data connection (MxGateway, protocol MxGateway, site 2), 1 site (site-a, artifacts sha256:d79ba1a3…, deployed 2026-10-27), 10 instances
Live health (HTTP, 02:45 EST)	All four /health/ready return 200. vq03 is active for central and site; vq04 answers 503. Central: memberCount 2, unreachable 0, 5/5 singletons

Window estimate for the flip: about 2–4 s on 18 MB, scaled from DEV's 1.5 s for 8 MB. All six migrations take well under 30 s.

4. #116 leader-auto-down: bundled

Recommendation: bundle it. It fits the same window with no extra restart:

• The 3a59c3f6 binary already contains #106 and #132 (shut-out restart, exit 75).
• The change is an appsettings flip only.
• #116's one rollout rule is "each pair stops and starts together; never roll across a downing-provider change", and this window already stops all four services and starts each pair together.
• The owner approved the QA rollout (rig and env2 gates passed, and env2 is soaking).
• If it went in a separate window, QA would take a second full outage.

The order inside the window is: binaries → migrations → config (including the #116 flip) → start. §5.4 has the edit, §6 G9 the drill, and §8 the rollback, which is to flip the value back.

5. Procedure

All times are local (EST). Roles: one operator on the Mac; WinRM for the on-box steps only, under the deploy-only scope; SMB for files; direct SQL for the DB. Layout on each box: E:\ApiInstall\ScadaBridge\{app,cli,central,site,…}. app\ is shared by the ScadaBridge-Central and ScadaBridge-Site services on that box, so a swap needs both services on that box stopped. The sites stop anyway, because LocalDb 0.3.7 pairs stop and start together.

5.1 Prep (before the window, no downtime)

1. Build: publish 3a59c3f6 self-contained win-x64, app and cli, with the same flags as the 2026-10-22 builds. Zip it, record the SHA-256, and confirm ProductVersion 1.0.0+3a59c3f6…. Stage it over SMB to E:\psrem-stage\3a59c3f6\ on both boxes and re-verify the hash on each box.
2. Re-run preflight.sql and compare it with §3. Abort if the migration count is not 58 or the key is not EventId-first.
3. On-box prep (WinRM, deploy scope; read-only except the exports):
   • reg export both services' env;
   • confirm that ASPNETCORE_URLS on both Centrals has no https:// (#108 refuse-to-boot);
   • confirm that LocalDb__Replication__ApiKey is present and identical on both Site services (compare hashes, never values);
   • record sc.exe qfailure and sc.exe qfailureflag for all four services (input to §5.3);
   • check E: free space.
4. Hash the three appsettings files on each box (they are kept and restored across the swap).
5. Agree the slot. Tell the team that uses Z28061 about the restart. The window restarts every instance actor, but deploys and writes nothing to Z28061.

5.2 Window

Step	Action	Notes
W0	Backup from the Mac (direct SQL): BACKUP DATABASE [ScadaBridge] TO DISK = N'F:\MSSQL\BACKUP\ScadaBridge_pre-3a59c3f6_<ts>.bak' WITH COPY_ONLY, CHECKSUM, COMPRESSION, STATS = 10; then RESTORE VERIFYONLY FROM DISK = … WITH CHECKSUM;	About 900 MB of data, compressed. The previous run took a few seconds. Taken while everything is still up; nothing writes config during the window
W1	Window opens. Stop ScadaBridge-Site on the standby box (vq04), then on the active box (vq03)	The site data path stops here
W2	Stop ScadaBridge-Central on vq04, then on vq03. Wait until no ZB.MOM.WW.ScadaBridge.Host.exe process remains on either box	Both centrals are now down, which migrations 61 and 63 require
W3	Migrate from the Mac (§2.2). Check: 64 migrations, flipped key	About 10–30 s
W4	Swap on each box: keep the three appsettings, rename app / cli → *.bak.<ts>, unpack, restore the appsettings (hash match)	As in 2026-10-22
W5	Config edits (§5.4) on both boxes, pushed over SMB while stopped. Record new hashes	#116
W6	Service recovery (§5.3) if the prep found it missing
W7	Start both Centrals together (vq03 and vq04 within a few seconds of each other), then wait for /health/ready 200 and memberCount 2 on both	BootstrapGuard is on. Don't start one central and wait (memory: a cold single-node start can self-down)
W8	Start both Sites together, then wait for ready 200 on both, localdb replicationConnected:true and backlog 0. Window closes

5.3 Service recovery for exit 75

The Host now exits 75 when it shuts itself down (#132) or its ActorSystem terminates unexpectedly. Windows recovery acts on a clean-stop-with-non-zero-exit only when failureflag is 1. If §5.1 step 3 shows that any of the four services lacks restart actions or has FailureActionsOnNonCrashFailures = FALSE, set it while the services are stopped:

sc.exe failure ScadaBridge-Central reset= 86400 actions= restart/10000/restart/30000/restart/60000
sc.exe failureflag ScadaBridge-Central 1

Do the same for ScadaBridge-Site. This is a change to the service definition, so the owner should confirm it when giving the go. Without it, the #132 self-heal leaves a node stopped, not restarted.

5.4 Config edits (both boxes, while stopped)

File	Edit	Why
app\appsettings.Central.json	"SplitBrainResolverStrategy": "auto-down" → "leader-auto-down"	#116
app\appsettings.Site.json	the same (in this PowerShell-written file the line reads "SplitBrainResolverStrategy": "auto-down", with two spaces)	#116

• LeaderAutoDown stays at its defaults: RestartWhenShutOut true, ShutOutGrace 00:01:00.
• StableAfter 15 s, heartbeat 2 s and failure detection 10 s are unchanged, which are the rig and env2 values.
• Preserve each file's encoding: the Site file has a BOM.
• Back up the originals as _appsettings-keep-<ts>.

Not changed:

• AuditLog:*: the chain, head export, archive and maintenance login all stay off (§7).
• MachineDataDb: dead, so leave it.
• The JWT and idle drift (§1.4): only if the owner asks.

6. Post-deploy gates

Run them from the Mac over HTTP, SQL, SMB and the CLI. No ad-hoc WinRM, except the §5 exit-code read that is part of the deploy. Nothing deploys to, writes to or flow-debugs Z28061. Snapshots and smokes use Z28062–Z28069 and SAPID 000000.

Gate	Check	Pass
G1 Build	Host.dll ProductVersion on both boxes (SMB, strings)	1.0.0+3a59c3f6 ×2
G2 Schema	preflight.sql §1/§4/§7	64; OccurredAtUtc first; chain tables + 3 roles
G3 Health	/health/ready on :8085 and :8084 on both boxes	200 ×4; database, akka-cluster, required-singletons Healthy
G4 Cluster	Central akka-cluster data on both nodes; site /health/ready	memberCount 2, unreachable 0, one leader. Site localdb.replicationConnected:true, backlog 0
G5 Active node	/health/active on both pairs, sampled 10×	Exactly one 200 per pair at every sample
G6 Boot logs	SMB-pull central\logs and site\logs for today	#116: [leader-auto-down] started on … on all 4 and no auto-down-unreachable-after … keep-majority deprecation line. No new ERR/FTL beyond the first-probe akka-cluster … Removed lines. Service stops log at Information, with no [FTL] (#76)
G7 CLI	POST /auth/token 200; scadabridge health / cluster status; security api-key list on both centrals	The same 3 keys (MESTestKey, MES_Key, delmia-notifier-qa) on both
G8 Inbound API	A bogus-key call to each node → 401 UNAUTHORIZED (the standby relays). flow-debug run-method of SimpleAlarmStatusRequest (id 9) with SAPID=000000	InboundRequest → BTDB DbCall (1 row) → Response Ok
G9 MxGateway	The site health report shows connection MxGateway Connected. Read-only snapshots of Z28062–Z28069	Good/Bad counts match the pre-window snapshot (2026-10-22 baseline: 242 Good / 2 Bad across Z28061–69). Take the before-snapshot in prep
G10 AuditLog	SQL: SELECT COUNT(*), MAX(OccurredAtUtc) FROM dbo.AuditLog WHERE OccurredAtUtc > <window close> after G8	New rows, SourceNode set, ChainSeq NULL (chain off); no 1205 in the logs
G11 Overview	http://wonder-app-vq03:5320 and vq04:5320	The ScadaBridge Central and Site clusters each show one Active, the leader chip, and no "Split brain"
G12 Artifacts / staleness	instance diff on every instance; the Sites page	No stale instances; artifact hash unchanged sha256:d79ba1a3…. No redeploy needed
G13 OTLP	SMB: E:\ApiInstall\OtelCollector\data\logs.jsonl on each box	Lines with service.name scadabridge and service.version 3a59c3f6 (logs are new with Telemetry 0.3.x)
G14 #116 drill (bundled)	Kill the active central's Host (Stop-Process -Force, deploy scope), then repeat on the standby. Watch /health/active and the logs	Survivor [leader-auto-down] … DOWNING about +27 s, takes over in about 30 s; the victim is restarted by SCM and rejoins as standby; then 10/10 single-active. The site pair drill re-hosts every instance, including Z28061 (about 30 s data gap), so run it only if the owner OKs that in the go. Otherwise record "central only" in #116

7. Hash chain, head export, maintenance login (the #107 owner decisions)

• In this window: the chain stays OFF on QA (AuditLog:HashChain:Enabled is unset, so false). The migration only adds the columns, tables and roles.
• After the window, as a separate step:
  1. SourceNode backfill first, because a chained row's SourceNode is frozen. Run scadabridge audit backfill-source-node --before 2026-07-08T00:00:00Z --sentinel unknown, which covers the 843 pre-M5.6 rows (Admin; idempotent). Then confirm that preflight.sql §6 shows 0 NULLs.
  2. Then set AuditLog:HashChain:Enabled=true on both centrals and restart them together.
  3. "No backlog chaining" (only rows written after enablement): ⚠ BLOCKED, #139. On main the chaining job links every unchained row, oldest first (AuditLogHashChain.cs:102), and has no cutoff option. Enabling it on QA as the code stands would chain all 3,441+ historical rows, which is the opposite of the decision. Wait for #139 (a cutoff option, or a revised decision) before step 2.
• Head export: File sink to a UNC share that the DB login cannot write. The service account gets create + append-data only. To add once the share is named, with HashChain:Enabled:

  "AuditLog": { "HashChain": { "HeadExport": {
      "Enabled": true, "IntervalMinutes": 60, "Sink": "File",
      "File": { "Directory": "\\\\<UNC-SHARE-TBD>\\scadabridge-qa\\audit-heads" } } } }
  

  <UNC-SHARE-TBD> is a placeholder: the owner hasn't named the share. Verify with scadabridge audit verify-heads --path <dir>. A failing export shows as /health/ready entry audit-chain-head-export Degraded (#115).
• Maintenance login: later. Keep the single login. AuditLog:MaintenanceConnectionString stays unset, and migration 64's grants are harmless without a second login.

8. Rollback

Rollback is a DB restore plus the old binaries. Schema 64 cannot run under 0e540291: old binaries cannot partition-switch the flipped AuditLog and cannot partition-purge once the chain columns exist.

1. Stop both Sites, then both Centrals (and wait for the Host processes to exit).
2. Restore from the Mac:

   ALTER DATABASE [ScadaBridge] SET SINGLE_USER WITH ROLLBACK IMMEDIATE;
   RESTORE DATABASE [ScadaBridge] FROM DISK = N'F:\MSSQL\BACKUP\ScadaBridge_pre-3a59c3f6_<ts>.bak' WITH REPLACE, CHECKSUM;
   ALTER DATABASE [ScadaBridge] SET MULTI_USER;
   

   Any audit or config rows written after W0 are lost.
3. On each box, rename app → app.failed, app.bak.<ts> → app, and the same for cli. Restore the pre-window appsettings from _appsettings-keep-<ts>. That puts the strategy back to auto-down.
4. Start both Centrals together, then both Sites together. Confirm 0e540291, 58 migrations, health.

Protocol config is encrypted at rest (since 2b9a7c3c, migration EnforceGlobalInstanceNamesAndProtectProtocolConfiguration). The format did not change in this range, so 0e540291 reads what 3a59c3f6 writes. However, any binary older than 2b9a7c3c cannot read protected rows at all, and a binary-only rollback against schema 64 is unsupported. So restore the DB together with the binaries, and never roll back past 0e540291.

#116-only rollback (keep the new build): set SplitBrainResolverStrategy back to auto-down in both files on both boxes, then stop and start each pair together.

9. Downtime estimate

Phase	Estimate	Basis
W1–W2 stop 4 services	60–90 s	2026-10-22: Site stops 10–30 s each, Central 10–17 s
W3 migrations	10–30 s	Flip about 2–4 s on 18 MB; the rest is small
W4–W6 swap + config	30–60 s	2026-10-22 swap took about 17 s per box
W7–W8 start both pairs	40–60 s	Central ready about 30 s, site about 10 s
Full outage (central API + site data path)	about 3–4 min	The 2026-10-22 coordinated window was 2 min 35 s
Book	45 min	Backup and verify, the gates, and the #116 kill drills (each adds about 30 s of single-pair failover, but no full outage)

10. Related gates in or near this window

• #131: QA has nothing to change (deploy-prep 2026-11-12-gates-52-131).
• #52: CvdReactor.ProcessRecipeDownload on QA still addresses Children["DelmiaRecv"]. The fix (2026-11-12-gates-52-131/apply.sh apply 52) is binary-independent, but it redeploys Z28061–Z28069, including Z28061. Not bundled by default. If the owner agrees a Z28061 slot in the same go, run it after G12, so there is still only one redeploy.
• #139 (new): hash-chain "no backlog" gate, §7.