ScadaBridge/docs/deployment/deploy-prep/2026-11-13-qa-vq03-vq04/README.md

# Deploy prep: QA vq03/vq04 (target `8e32f353`), coordinated central-pair migration window (ScadaBridge#107 item 4) + #116 leader-auto-down

**Prepared:** 2026-11-13. **Refreshed:** 2026-11-13 ~07:10 EST for the new tip. **Not executed.**

**Owner's go (2026-11-13, "just deploy"):** the window itself is approved. Scope:
- **in:** the upgrade, the bundled #116 `leader-auto-down` config flip (§5.4), and the exit-code-75 service recovery setting (§5.3, `sc.exe failure` + `failureflag 1` on all four services);
- **out:** **no #116 drills** (neither the central kill drill nor the site-pair kill drill; G14 is not run), **hash chain OFF** on QA, **head export OFF** (both stay unset; §7 is a later, separate step), **#52 not bundled** (§10).

**Target build:** `main` @ **`8e32f353`**. The first version of this pack (`16209831`) targeted `3a59c3f6`; this refresh re-ran §1.4, §2 and §3 over `3a59c3f6..8e32f353` (§1.5). **No new migration in that range**: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/` is untouched, and a fresh `dotnet ef migrations script` at `8e32f353` is byte-identical to the script in this folder (same SHA-256). If `main` moves again before the window, repeat that check over `8e32f353..<new tip>`.

**Sources:**
- The read-only SQL preflight on `wonder-sql-vq03` (2026-11-13 ~02:45 EST, re-run ~07:05 EST for the refresh, §3).
- SMB reads of the live `app\` folders and appsettings on both boxes, and HTTP health probes. Nothing was written on any box.
- Deployment records `2026-10-22-wonder-app-vq03-vq04-3808b49f.md` and `2026-10-22-wonder-app-vd03-vq03-vq04-0e540291.md`.
- The DEV vd03 `424db155` upgrade of 2026-11-11 (migrations 58→64, flip of 2,394 rows / 8 MB in about 1.5 s).
- Issues #107, #116, #126, #131, #52, #138, #139 and #140.

## Files

| File | What |
|---|---|
| `README.md` | This pack. |
| `preflight.sql` | Read-only preflight (SELECTs and `sp_spaceused` only). Re-run it right before the window. |
| `migrate-58-to-64.idempotent.sql` | `dotnet ef migrations script 20261022133907_ReviewRemediationIndexes --idempotent`, generated at `3a59c3f6` and **re-generated at `8e32f353` with an identical result** (EF tools 10.0.10, runtime 10.0.13). It covers six migrations, 59→64. SHA-256 `488da93db0f464ee17687a32a1dabbbe810553c109db5b65d5b3ecbd5c34ecc0`. It is unedited and starts with a UTF-8 BOM at byte 0, which sqlcmd accepts; #126's problem was a BOM *after* a prepended header. |

---

## 0. Correction: QA is on `0e540291` with 58 migrations, not `b3021e7c` with 57

| Evidence | Value |
|---|---|
| `ZB.MOM.WW.ScadaBridge.Host.dll` in `E:\ApiInstall\ScadaBridge\app\` on **both** boxes (SMB read) | `1.0.0+0e540291` |
| `__EFMigrationsHistory` on `wonder-sql-vq03` | **58** rows; the last is `20261022133907_ReviewRemediationIndexes` |
| Records | 2026-10-22: `b3021e7c`→`3808b49f` (pair window, migration 58), then `3808b49f`→`0e540291` (rolling, no migration). Also on 2026-10-22: shared SQL key and secret stores, and the standby relay |

As a result, these items from the brief are **already on QA** and are not part of this window:
- bearer-token CLI auth;
- `MachineDataDb` removed (the Central file still carries a dead `MachineDataDb` key, which is ignored);
- site secrets via Deploy Artifacts, with the gRPC hub retired;
- `ReviewRemediationIndexes`;
- the shared `zbauth`/`zbsecrets` SQL stores on the central pair.

The effective delta is **`0e540291..8e32f353`: 134 commits and 6 migrations** (128 of them up to `3a59c3f6`, the rest in §1.5).

## 1. What changes: `0e540291..8e32f353`

### 1.1 Schema (central DB): 6 migrations

See §2.

### 1.2 Behaviour changes worth knowing on QA

| Area | Change | QA impact |
|---|---|---|
| Audit (#41, #107, #109, #113–#115, #42) | Hash chain, head export, maintenance login, Parquet archive and archive retention. | **All default OFF and config-only.** Nothing is enabled in this window (§6). |
| AuditLog clustered key (#111) | `(OccurredAtUtc, EventId)` replaces `EventId`. Binaries from `5e7d5389` on purge either shape. | Old binaries cannot partition-switch the flipped table. That is the reason for the both-centrals-stopped window and for restore-only rollback. |
| AuditLog ingest (#124) | Set-based ingest uses FORCESEEK, so concurrent writers no longer deadlock (1205). | Positive. |
| Cluster downing (#106, #132) | Opt-in `leader-auto-down` with shut-out restart. **The Host now exits 75 after any unplanned ActorSystem termination** (it used to exit 0), under any strategy. | Windows SCM recovery has to restart on a non-zero exit. See §5.3, which checks it in the window. |
| Host stop (#76) | A routine service stop logs as Information, not `[FTL] host terminated unexpectedly`. | Drop the old "known FTL on stop" exclusion from log review. |
| Host stop (#140, new in the refresh) | A graceful stop now leaves the Akka cluster **before** Kestrel drains (`AkkaHostedService` is an `IHostedLifecycleService`; CoordinatedShutdown runs in `StoppingAsync`), and every open gRPC stream (the peer's LocalDb sync stream above all) ends on `ApplicationStopping`; new streams are refused `Unavailable` while stopping. Exit codes unchanged: planned stop 0, self-down/shut-out 75. | **Only the new binaries' stops benefit.** W1–W2 stop the *old* `0e540291` binaries, so the window's own stops still take the old time (a Site stop can wait out the 30 s drain on the sync stream). Stops after the window (and the rollback in §8, for the new build) are faster. New log lines on stop: `Host stopping: leaving the Akka cluster before the web server drains…` and `Akka CoordinatedShutdown completed in … ms`. |
| Alarms (#129) | `Severity` is banded with higher = more urgent; the raw value is on `Priority`. The MES 900–999 band keys off `Priority`. | The Alarm Summary Sev column and its sort flip visibly. **#131: QA has nothing to change**: `CvdReactor.AlarmStatus` (39) is a stub, and there is no TestMesMachine. |
| MES-reportable (#59, #60, #102, #104) | Per-binding `IsMesReportable` on native alarm sources, plus UI badges and CSV columns. | Additive, delivered by migration `AddNativeAlarmSourceMesReportable`. |
| Security (#46, Auth 0.5.0) | Active sessions re-query LDAP group membership: `ScadaBridge:Security:LdapGroupRefresh`, `Enabled` true, every 15 min, 60 s cache. | QA's centrals bind to their box-local GLAuth (`127.0.0.1:3893`) with a service account, which this supports. A user removed from a group loses the role within about 15 min. |
| Central HTTP (#108) | An `https://` entry in `ASPNETCORE_URLS` now needs `ScadaBridge:Node:Tls` (a certificate) or the node **refuses to boot**. | QA serves plain `http://…:8085`. **The prep step confirms there is no `https://` in either Central's service env** (§5.1). |
| Akka.Remote TLS (#64) | `ScadaBridge:Cluster:RemotingTls`, default off. | None. |
| DCL OPC UA (#117, #121, #112, #58; SDK 1.5.378.106 → **2.0.1**, #125) | Good subcodes now map to Good. Secure sessions present a real app cert (`OpcUaGlobal:*`). `BadResourceUnavailable` is treated as transient. Scalar writes are coerced to the node type. | **QA has no OPC UA connection.** Its only data connection is `MxGateway`, so there is no exposure. |
| DCL OPC UA failover (#138, new in the refresh) | ServiceLevel-aware failover for an OPC UA connection with a primary **and** a backup endpoint. New per-connection setting `ServiceLevelFailover {Enabled, DegradedWindowSeconds, MinDwellSeconds}` on the **primary** endpoint, **default on** (`true`, 10 s, 60 s; a stored connection without it gets the defaults). It lives inside the encrypted protocol JSON, so **no migration**. Global knobs `ScadaBridge:DataConnection:ServiceLevelPollInterval` (2 s) and `ServiceLevelProbeTimeout` (15 s). | **None on QA** (no OPC UA connection, re-checked §3). Where it does apply, today's OtOpcUa pair (250/240, both in the Healthy band ≥ 200) never starts a degraded streak, so it is safe with 250/240. It matters once OtOpcUa Warm redundancy (lmxopcua#717) is switched on. |
| Audit hash-chain cutoff (#139, new in the refresh) | First enablement anchors the chain at the DB's `SYSUTCDATETIME()` and chains only rows from then on; `AuditLog:HashChain:ChainBacklog` (default false) restores chain-everything. **No migration.** | None in this window: the chain stays off. It is what makes §7 safe later: this build is the one §7 step 0 asks for. |
| DCL MxGateway (#128) | The resume cursor ignores `worker_sequence` 0 sentinels. | Positive for QA's single connection. |
| Site runtime (#135, #136, #36) | Post-boot settle sweep: replicated instances start within seconds. Site commands fail over within the call. The legacy `WriteBatchAndWaitAsync` waits for its own reset echo. | Faster site start after the swap. |
| Inbound API (#103, #75, #80) | Recompile and invalidate reach the peer central. Compile errors are visible through `api-method validate/recompile`. | Positive on a pair. |
| Pending updates (#62, #63) | A DB rowversion watermark short-circuits an unchanged fleet (this is what the 18 `RowVersion` columns are for). The counts appear on the Health dashboard and CLI. | Needs migration `AddConfigGraphRowVersions`. |
| Transport (#45, #110, #118, #93–#95, #100) | Optional detached bundle signing (default off). Derived templates round-trip. Name lookups are case-insensitive. | None unless bundles are used. |
| Management (#105) | Case-variant instance override names are refused with a did-you-mean hint. | Behaviour change for scripts that send wrong-case names. |
| Logging (#56, #120, #94) | Akka logging goes through Serilog. `ExecutionId`, `InstanceName` and `ScriptName` are added to the LogContext. | More (and better) lines in the file logs. |
| Telemetry 0.1.0 → **0.3.1** (scadaproj#55/#56) | `service.version` = git SHA. **Logs now go over OTLP when `Exporter=Otlp`.** The endpoint check is stricter (a scheme-less `host:port` fails). | QA's service env (2026-11-12) points at `http://127.0.0.1:4317`, which has a scheme, so it boots. The box-local `ZbOtelCollector` starts receiving **logs** as well. Watch `E:\ApiInstall\OtelCollector\data\logs.jsonl` (about 1.8 GB cap per box). |
| CLI (#137) | Secret-bearing option values can be read from a file or stdin. | Optional. |
| Health (#97) | Site `/health/active` uses the shared role-scoped check. | The Overview site chips now show active and standby correctly. |

### 1.3 Shared-library and package pins

| Package | QA today (`0e540291`) | Tip | Note |
|---|---|---|---|
| ZB.MOM.WW.Auth.* | 0.3.0 | **0.5.0** | `BindMode` is still `ServiceAccount`. Adds the passwordless group lookup (#46). |
| ZB.MOM.WW.LocalDb.* | 0.3.4 | **0.3.8** | 0.3.8 (scadaproj#57, `8e32f353`) adds only the graceful-stop fix: open sync streams end on `ApplicationStopping` (lmxopcua#906). Wire-compatible, but **the pair stops and starts together**. The library now enforces `LocalDb:Replication:ApiKey` itself, which must be byte-identical on both site nodes (it already is: QA's own fail-closed interceptor has been connected since 2026-09-10). The tombstone ledger grows while a peer is down (#16). `MaxBatchSize: 16` in both Site files is a retired pin; it is harmless. |
| ZB.MOM.WW.Telemetry(.Serilog) | 0.1.0 | **0.3.1** | See above. |
| ZB.MOM.WW.Theme | 0.4.1 | **0.5.3** | ScrollNav rail (#61). |
| ZB.MOM.WW.Secrets | 0.6.2 | 0.6.2 | Unchanged. |
| Microsoft.Extensions/EF/ASP.NET | 10.0.7 | 10.0.13 | Sqlite pool fix (#98). SqlClient 6.1.7. |
| OPCFoundation Opc.Ua.Client | 1.5.378.106 | 2.0.1 | No OPC UA connections on QA. |
| Parquet.Net | — | 6.1.0 | Archive and export (off). |

### 1.4 Settings and env

**No setting has to change for this build to boot on QA.** No shipped appsettings changed in the range, and every new option has a safe default. New keys, all optional:
- `ScadaBridge:Cluster:{LeaderAutoDown:{RestartWhenShutOut,ShutOutGrace},RemotingTls:*}`
- `ScadaBridge:Node:Tls:*`
- `ScadaBridge:Communication:GrpcClientIdleTtl`
- `ScadaBridge:DeploymentManager:PendingUpdatesFullEvaluationEvery`
- `ScadaBridge:SiteRuntime:InstanceSweepSettle{Interval,Window}Seconds`
- `ScadaBridge:Security:LdapGroupRefresh:*`
- `ScadaBridge:Transport:Signing:*`
- `OpcUaGlobal:{ApplicationUri,ApplicationCertificateSubject,OwnCertificateStorePath,AutoCreateApplicationCertificate,ApplicationCertificateLifetimeMonths}`
- `AuditLog:{MaintenanceConnectionString,HashChain:*,HashChain:HeadExport:*,Archive:*}`
- `LocalDb:Replication:{MaxSnapshotDeferredRows,TombstoneHardCap}`
- new in the refresh: `ScadaBridge:DataConnection:{ServiceLevelPollInterval,ServiceLevelProbeTimeout}` (#138; validated positive) and `AuditLog:HashChain:ChainBacklog` (#139; default false). The per-connection `ServiceLevelFailover` object is data, not appsettings.

**Changes this window does make:** the #116 strategy flip (§5.4) and the exit-75 service recovery setting (§5.3). Both are approved in the owner's go.

**Drift found, not fixed here:** the vq03 and vq04 `appsettings.Central.json` differ in `JwtExpiryMinutes` (25 vs 15) and `IdleTimeoutMinutes` (60 vs 30). The CLI bearer token is signed with the shared key, so the pair still interoperates, but session lifetime depends on which central answers. Align them (pick one pair of values) as a separate change, or here if the owner says so.

### 1.5 Refresh delta: `3a59c3f6..8e32f353`

Six commits (the two docs commits are this pack's own):

| Commit | Change | Migration | QA impact |
|---|---|---|---|
| `16209831`, `3badd1f9` | This pack (first version, then the §7 #139 update) | — | — |
| `712c5117` | #138 ServiceLevel-aware OPC UA failover (§1.2) | No | None: no OPC UA connection. New per-connection `ServiceLevelFailover`, **default on** (10 s window, 60 s dwell); safe with OtOpcUa's 250/240 |
| `52da02a7` | #139 hash-chain cutoff (§1.2, §7) | No | None in the window (chain off); makes the later §7 enablement backlog-free |
| `1a6108c4` | #140 graceful stop: leave the cluster before Kestrel drains; end open gRPC streams on stop (§1.2) | No | Faster stops of the *new* binary only |
| `8e32f353` | LocalDb **0.3.8** pin (scadaproj#57): the library itself ends open sync streams on `ApplicationStopping` and refuses new ones `Unavailable` while stopping (lmxopcua#906). The #140 host interceptor stays as the backstop | No | Same as #140; still wire-compatible with 0.3.4/0.3.7, pair still stops and starts together |

Checks over this range:
- **Migrations:** none. `git diff 3a59c3f6 8e32f353 -- src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/` is empty (the ConfigurationDatabase changes are in `Maintenance/` and `Repositories/`, for #139). The regenerated idempotent script has the same SHA-256 as the one in this folder.
- **Options:** only the three keys listed in §1.4 ("new in the refresh"), all with safe defaults. No shipped appsettings changed. **Nothing has to change on QA for this build to boot.**
- **Rollback (§8):** unchanged. None of the new commits changes the schema or the protocol-config encryption, so `0e540291` + the restored DB is still the rollback.

## 2. Migrations 58 → 64

### 2.1 Ordered list (EF timestamp order = apply order)

| # | Migration | Issue | What it does | Cost on QA | Old-binary compatible? |
|---|---|---|---|---|---|
| 59 | `20261109200211_AddNativeAlarmSourceMesReportable` | #102/#59 | Nullable `IsMesReportable` on native-source bindings/overrides | Metadata only | Yes (additive) |
| 60 | `20261109203753_PrefixSearchIndexes` | #50 | Prefix-search indexes for Deployments and config-audit | Tiny tables | Yes |
| 61 | `20261109211317_AddAuditLogHashChain` | #41 | `ChainSeq`/`RowHash` on `AuditLog`. `AuditLogChainHead` and `AuditLogChainTombstone`. Two filtered indexes, wrapped in `EXEC(N'…')` (the #126 fix). Roles `scadabridge_audit_chainer` (new) and `_purger`/`_writer` (exist), grants, and a DENY to the writer on the chain tables | 3,441 rows; seconds | **No**: old binaries can't partition-purge once the columns exist |
| 62 | `20261110032839_AddConfigGraphRowVersions` | #62 | Non-concurrency `rowversion` + index on 18 configuration tables. SQL Server fills each table as it adds the column | All config tables are under 2,300 rows; seconds | Yes (ignored) |
| 63 | `20261110033732_FlipAuditLogClusteredKey` | #111 | **Offline whole-table rebuild**: drop the 10 NCIs + PK, re-add the PK clustered on `(OccurredAtUtc, EventId)` on `ps_AuditLog_Month`, add `UX_AuditLog_EventId_Occurred`, recreate 12 NCIs. Guarded by "clustered key ordinal 1 = `EventId`" | **18.3 MB reserved / 3,441 rows → about 2–4 s** (DEV: 8 MB / 2,394 rows in about 1.5 s). Log about 2× = about 40 MB, against 136 MB free in a 200 MB log at the 07:05 re-run (64 MB autogrowth, 240 GB free on G:) | **No**: older binaries can't partition-switch a flipped table |
| 64 | `20261110120000_AuditMaintenanceGrants` | #113 | `scadabridge_audit_purger`: `ALTER ANY DATASPACE` + `UPDATE (SourceNode)` on `dbo.AuditLog` | Instant | Yes |

**Owner's §item-4 list vs EF order.** `AddNativeAlarmSourceMesReportable` sorts **first**, by timestamp. The EF script applies migrations in timestamp order, which is also the order DEV vd03 used. The relative order of the five audit/config migrations is the one the owner listed.

### 2.2 How they are applied

- **Tool:** the generated idempotent script `migrate-58-to-64.idempotent.sql` in this folder. Per the #126 deploy note, QA can run it **as is**, with no GO insertion. It is 6 batches, each `BEGIN TRANSACTION … COMMIT; GO`.
  - `IdempotentMigrationScriptTests` (MSSQL fixture) proves that from-57 → head runs clean and a re-run is a no-op.
  - The hand scripts in `docs/plans/sql/` remain the fallback. They were BOM-fixed in `c7dfc21b`. There is no hand script for migrations 59 and 64, so the hand route needs EF fragments as well. That makes it two tools for one job; prefer the single EF script.
- **When:** with **both centrals and both sites stopped** (§5). This is required by migrations 61 and 63.
- **Where from:** the Mac, straight to SQL (direct SQL is always allowed, so no WinRM is needed for this step):
  ```bash
  cd docs/deployment/deploy-prep/2026-11-13-qa-vq03-vq04
  shasum -a 256 migrate-58-to-64.idempotent.sql   # must be 488da93d…c34ecc0
  printf 'SET XACT_ABORT ON;\n' > /tmp/xact.sql   # session-scoped; same connection as the script
  SQLCMDPASSWORD=… sqlcmd -S wonder-sql-vq03.zmr.zimmer.com,1433 -U <login> -C -d ScadaBridge -I -b \
      -i /tmp/xact.sql,migrate-58-to-64.idempotent.sql
  ```
  The fallback is the same command with Windows `sqlcmd -C -b -I` on vq03, as in the 2026-10-22 window. **vq04 has no `sqlcmd`.**
- **Idempotency:**
  - Every statement is guarded by `IF NOT EXISTS (… __EFMigrationsHistory …)`.
  - Role creation is additionally guarded by `IF DATABASE_PRINCIPAL_ID(…) IS NULL`, and the flip by the key-shape check.
  - Re-running after a partial failure therefore completes only the missing migrations. A failed batch rolls back whole, because of `XACT_ABORT` + `-b`.
- **Expected after:**
  - 64 rows, with the last one `20261110120000_AuditMaintenanceGrants`.
  - Clustered key ordinal 1 = `OccurredAtUtc`.
  - `AuditLogChainHead`/`AuditLogChainTombstone` exist.
  - 3 `scadabridge_audit_*` roles.
  - Re-run `preflight.sql` sections 1, 4 and 7.

## 3. Read-only SQL preflight: results (2026-11-13 ~02:45 EST, re-run ~07:05 EST)

The re-run for the refresh changed nothing that matters: still 58 migrations, the same 3,441 AuditLog rows and the same `EventId`-first key. Only the log usage moved (57 → 63 MB). The table shows the re-run values.

Server **`wonder-sql-vq03.zmr.zimmer.com,1433`** (`WONDER-SQL-VQ03`, SQL Server 2022 RTM-CU24 16.0.4245.2, **Enterprise**), DB `ScadaBridge`. Login `wwadmin`, password via `SQLCMDPASSWORD` only, `-C`. SELECTs only. The script is in `preflight.sql`.

| Check | Result |
|---|---|
| `__EFMigrationsHistory` | **58**; last `20261022133907_ReviewRemediationIndexes` (ProductVersion 10.0.7). Unchanged at 07:05 |
| Recovery model | **FULL**; `log_reuse_wait` NOTHING; compatibility 160 |
| Log backups | Regular: full at 22:00 daily (last 2026-11-12 22:00), log at about 12:00 / 24:00 (last 2026-11-12 23:57). A `COPY_ONLY` backup does not disturb the chain |
| `dbo.AuditLog` | **3,441 rows**; reserved **18.3 MB** (18,768 KB: data 10.7 MB, index 3.2 MB, unused 4.4 MB); 2026-05-29 → 2026-11-12 22:13 UTC. Unchanged at 07:05 |
| Clustered key now | `PK_AuditLog` = (`EventId`, `OccurredAtUtc`) on `ps_AuditLog_Month`, so the flip has not been applied yet |
| `pf_AuditLog_Month` | 24 boundaries, the last 2027-12-01 |
| Log file | 200 MB, **63 MB used (31.6 %)** at 07:05 (57 MB / 28.6 % at 02:45), 64 MB autogrowth, unlimited; G: 245,855 MB free. The flip's ~40 MB still fits in the free 136 MB |
| Data file | 904 MB, 64 MB autogrowth; F: 249,323 MB free. The largest table is `KpiSample` (4.9 M rows, 804 MB), which the window does not touch |
| SourceNode backfill | **Not run.** **843 rows** have `SourceNode IS NULL` (unchanged at 07:05): InboundRequest 793 (newest 2026-07-07) and InboundAuthFailure 50 (newest 2026-06-27), all from before M5.6. Irrelevant to this window (chain off); with the #139 cutoff it no longer gates §7 either. Every row since 2026-07-07 is stamped (central-a 1,742, central-b 777, node-a 59, node-b 20) |
| Hash-chain objects | None yet (re-checked 07:05). Roles `scadabridge_audit_writer` and `_purger` exist; `_chainer` does not. Columns present: only `DeploymentRecords.RowVersion` and `TemplateAlarms.IsMesReportable` (both pre-existing). `DeploymentRecords.RowVersion` already exists, and `AddConfigGraphRowVersions` handles that table (covered by the from-57 test) |
| Topology | 1 data connection (`MxGateway`, protocol MxGateway, site 2), **no OPC UA connection** (so #138 has nothing to act on), 1 site (`site-a`, artifacts `sha256:d79ba1a3…`, deployed 2026-10-27), 10 instances. Unchanged at 07:05 |
| #131 / #52 scripts (re-checked 07:05) | No `TestMesMachine` template. `CvdReactor.AlarmStatus` (script 39) is a 110-char stub that reads no `.Severity`; no shared scripts; none of the 8 API methods reads `Alarms.`/`.Severity`. So **#131 needs no QA script change**, confirming the 2026-11-12 finding. `CvdReactor.ProcessRecipeDownload` (script 4) still has `Children["DelmiaRecv"]` (#52 still open on QA, not bundled) |
| Live health (HTTP, 02:45 and again 07:05 EST) | All four `/health/ready` return 200. **vq03 is active** for central and site; vq04 answers 503 on `/health/active`. Central: memberCount 2, unreachable 0, leader vq03 (`10.220.158.2`), 5/5 singletons |

**Window estimate for the flip:** about 2–4 s on 18 MB, scaled from DEV's 1.5 s for 8 MB. All six migrations take well under 30 s.

## 4. #116 leader-auto-down: bundled

**Recommendation: bundle it.** It fits the same window with no extra restart:
- The `8e32f353` binary already contains #106 and #132 (shut-out restart, exit 75).
- The change is an **appsettings flip only**.
- #116's one rollout rule is "each pair stops and starts together; never roll across a downing-provider change", and this window already stops all four services and starts each pair together.
- The owner approved the QA rollout (rig and env2 gates passed, and env2 is soaking).
- If it went in a separate window, QA would take a second full outage.

The order inside the window is: binaries → migrations → config (including the #116 flip) → start. §5.4 has the edit and §8 the rollback, which is to flip the value back. **The owner's go excludes the #116 drills** (central and site pair): G14 is not run, and #116 records "deployed on QA, not drilled". The config, the boot-log check (G6) and the single-active check (G5) still apply.

## 5. Procedure

All times are local (EST). Roles: one operator on the Mac; WinRM for the on-box steps only, under the deploy-only scope; SMB for files; direct SQL for the DB. Layout on each box: `E:\ApiInstall\ScadaBridge\{app,cli,central,site,…}`. **`app\` is shared by the `ScadaBridge-Central` and `ScadaBridge-Site` services on that box**, so a swap needs both services on that box stopped. The sites stop anyway, because LocalDb pairs stop and start together.

### 5.1 Prep (before the window, no downtime)

1. Build: publish `8e32f353` self-contained win-x64, app and cli, with the same flags as the 2026-10-22 builds. Zip it, record the SHA-256, and confirm ProductVersion `1.0.0+8e32f353…`. Stage it over SMB to `E:\psrem-stage\8e32f353\` on both boxes and re-verify the hash on each box.
2. Re-run `preflight.sql` and compare it with §3. Abort if the migration count is not 58 or the key is not `EventId`-first.
3. On-box prep (WinRM, deploy scope; read-only except the exports):
   - `reg export` both services' env;
   - **confirm that `ASPNETCORE_URLS` on both Centrals has no `https://`** (#108 refuse-to-boot);
   - confirm that `LocalDb__Replication__ApiKey` is present and identical on both Site services (compare hashes, never values);
   - record `sc.exe qfailure` and `sc.exe qfailureflag` for all four services (input to §5.3);
   - check E: free space.
4. Hash the three appsettings files on each box (they are kept and restored across the swap).
5. Agree the slot. Tell the team that uses **Z28061** about the restart. The window restarts every instance actor, but deploys and writes nothing to Z28061.

### 5.2 Window

| Step | Action | Notes |
|---|---|---|
| W0 | **Backup** from the Mac (direct SQL): `BACKUP DATABASE [ScadaBridge] TO DISK = N'F:\MSSQL\BACKUP\ScadaBridge_pre-8e32f353_<ts>.bak' WITH COPY_ONLY, CHECKSUM, COMPRESSION, STATS = 10;` then `RESTORE VERIFYONLY FROM DISK = … WITH CHECKSUM;` | About 900 MB of data, compressed. The previous run took a few seconds. Taken while everything is still up; nothing writes config during the window |
| W1 | **Window opens.** Stop `ScadaBridge-Site` on the standby box (vq04), then on the active box (vq03) | The site data path stops here |
| W2 | Stop `ScadaBridge-Central` on vq04, then on vq03. **Wait until no `ZB.MOM.WW.ScadaBridge.Host.exe` process remains on either box** | Both centrals are now down, which migrations 61 and 63 require |
| W3 | Migrate from the Mac (§2.2). Check: 64 migrations, flipped key | About 10–30 s |
| W4 | Swap on each box: keep the three appsettings, rename `app` / `cli` → `*.bak.<ts>`, unpack, restore the appsettings (hash match) | As in 2026-10-22 |
| W5 | Config edits (§5.4) on both boxes, pushed over SMB while stopped. Record new hashes | #116 |
| W6 | Service recovery (§5.3) on all four services (approved in the go) | Record `sc.exe qfailure`/`qfailureflag` after |
| W7 | **Start both Centrals together** (vq03 and vq04 within a few seconds of each other), then wait for `/health/ready` 200 and memberCount 2 on both | BootstrapGuard is on. Don't start one central and wait (memory: a cold single-node start can self-down) |
| W8 | **Start both Sites together**, then wait for ready 200 on both, `localdb replicationConnected:true` and backlog 0. **Window closes** | |

### 5.3 Service recovery for exit 75

The Host now exits **75** when it shuts itself down (#132) or its ActorSystem terminates unexpectedly. Windows recovery acts on a clean-stop-with-non-zero-exit only when `failureflag` is 1. If §5.1 step 3 shows that any of the four services lacks restart actions or has `FailureActionsOnNonCrashFailures = FALSE`, set it while the services are stopped:

```
sc.exe failure ScadaBridge-Central reset= 86400 actions= restart/10000/restart/30000/restart/60000
sc.exe failureflag ScadaBridge-Central 1
```

Do the same for `ScadaBridge-Site`. This is a change to the service definition; **the owner approved it in the 2026-11-13 go**, so set it in W6 whatever the prep finds missing (it is idempotent). Without it, the #132 self-heal leaves a node **stopped**, not restarted.

### 5.4 Config edits (both boxes, while stopped)

| File | Edit | Why |
|---|---|---|
| `app\appsettings.Central.json` | `"SplitBrainResolverStrategy": "auto-down"` → `"leader-auto-down"` | #116 |
| `app\appsettings.Site.json` | the same (in this PowerShell-written file the line reads `"SplitBrainResolverStrategy":  "auto-down"`, with two spaces) | #116 |

- `LeaderAutoDown` stays at its defaults: `RestartWhenShutOut` true, `ShutOutGrace` 00:01:00.
- `StableAfter` 15 s, heartbeat 2 s and failure detection 10 s are unchanged, which are the rig and env2 values.
- Preserve each file's encoding: the Site file has a BOM.
- Back up the originals as `_appsettings-keep-<ts>`.

**Not changed:**
- `AuditLog:*`: the chain, head export, archive and maintenance login all stay off (§7).
- `MachineDataDb`: dead, so leave it.
- The JWT and idle drift (§1.4): only if the owner asks.

## 6. Post-deploy gates

Run them from the Mac over HTTP, SQL, SMB and the CLI. No ad-hoc WinRM, except the §5 exit-code read that is part of the deploy. **Nothing deploys to, writes to or flow-debugs Z28061.** Snapshots and smokes use Z28062–Z28069 and SAPID `000000`.

| Gate | Check | Pass |
|---|---|---|
| G1 Build | `Host.dll` ProductVersion on both boxes (SMB, `strings`) | `1.0.0+8e32f353` ×2 |
| G2 Schema | `preflight.sql` §1/§4/§7 | 64; `OccurredAtUtc` first; chain tables + 3 roles |
| G3 Health | `/health/ready` on :8085 and :8084 on both boxes | 200 ×4; `database`, `akka-cluster`, `required-singletons` Healthy |
| G4 Cluster | Central `akka-cluster` data on both nodes; site `/health/ready` | memberCount 2, unreachable 0, one leader. Site `localdb.replicationConnected:true`, backlog 0 |
| G5 Active node | `/health/active` on both pairs, sampled 10× | Exactly one 200 per pair at every sample |
| G6 Boot logs | SMB-pull `central\logs` and `site\logs` for today | **#116:** `[leader-auto-down] started on …` on all 4 and **no** `auto-down-unreachable-after … keep-majority` deprecation line. No new ERR/FTL beyond the first-probe `akka-cluster … Removed` lines. Service stops log at Information, with no `[FTL]` (#76) |
| G7 CLI | `POST /auth/token` 200; `scadabridge health` / `cluster` status; `security api-key list` on **both** centrals | The same 3 keys (`MESTestKey`, `MES_Key`, `delmia-notifier-qa`) on both |
| G8 Inbound API | A bogus-key call to each node → 401 `UNAUTHORIZED` (the standby relays). `flow-debug run-method` of `SimpleAlarmStatusRequest` (id 9) with `SAPID=000000` | InboundRequest → BTDB DbCall (1 row) → Response Ok |
| G9 MxGateway | The site health report shows connection `MxGateway` Connected. Read-only snapshots of Z28062–Z28069 | Good/Bad counts match the pre-window snapshot (2026-10-22 baseline: 242 Good / 2 Bad across Z28061–69). Take the before-snapshot in prep |
| G10 AuditLog | SQL: `SELECT COUNT(*), MAX(OccurredAtUtc) FROM dbo.AuditLog WHERE OccurredAtUtc > <window close>` after G8 | New rows, `SourceNode` set, `ChainSeq` NULL (chain off); no 1205 in the logs |
| G11 Overview | `http://wonder-app-vq03:5320` and `vq04:5320` | The ScadaBridge Central and Site clusters each show one Active, the leader chip, and **no** "Split brain" |
| G12 Artifacts / staleness | `instance diff` on every instance; the Sites page | No stale instances; artifact hash unchanged `sha256:d79ba1a3…`. No redeploy needed |
| G13 OTLP | SMB: `E:\ApiInstall\OtelCollector\data\logs.jsonl` on each box | Lines with `service.name` scadabridge and `service.version` 8e32f353 (logs are new with Telemetry 0.3.x) |
| ~~G14 #116 drill~~ | **Not run: excluded by the owner's go (2026-11-13).** Neither the central kill drill nor the site-pair kill drill. For a later window, the drill was: kill the active central's Host (`Stop-Process -Force`), then the standby; survivor `[leader-auto-down] … DOWNING` about +27 s, takeover about 30 s, the victim restarted by SCM (§5.3) and rejoins as standby. The site-pair drill re-hosts every instance including Z28061 | Record in #116: "QA on leader-auto-down, not drilled" |
| G15 Recovery setting | `sc.exe qfailure` / `qfailureflag` on all four services (read back in the window, deploy scope) | restart/10000/restart/30000/restart/60000, reset 86400, failure flag 1, ×4 |

## 7. Hash chain, head export, maintenance login (the #107 owner decisions)

- **In this window: the chain stays OFF on QA, and the head export stays OFF** (owner's go, 2026-11-13: `AuditLog:HashChain:Enabled` and `AuditLog:HashChain:HeadExport:Enabled` stay unset, so false). The migration only adds the columns, tables and roles. G10 checks `ChainSeq` stays NULL.
- **After the window, as a separate step:**
  0. **Build: needs the #139 cutoff.** "No backlog chaining" is implemented on `main` from `52da02a7` (#139): the first chaining pass records a chain-start anchor at the database's `SYSUTCDATETIME()` and only rows dated from then on are chained. **This window's build `8e32f353` contains it** (the refresh moved the target past `52da02a7`; no migration, §1.5), so no further build is needed before step 2. Confirm with `Host.dll` ProductVersion `1.0.0+8e32f353` (G1). Never enable the chain on a build older than `52da02a7`: it would chain all 3,441+ historical rows.
  1. SourceNode backfill first, because a chained row's `SourceNode` is frozen. Run `scadabridge audit backfill-source-node --before 2026-07-08T00:00:00Z --sentinel unknown`, which covers the 843 pre-M5.6 rows (Admin; idempotent). Then confirm that `preflight.sql` §6 shows 0 NULLs. (With the cutoff the backlog is never chained, so its `SourceNode` is never frozen: the order no longer matters, but running it first keeps the documented #107 sequence.)
  2. Then set `AuditLog:HashChain:Enabled=true` on both centrals and restart them **together**. **Do not set `AuditLog:HashChain:ChainBacklog`** (default `false` = no backlog chaining, the owner decision). It is only read on the very first enablement; the first pass fixes the cutoff in the database and both nodes use it from then on.
  3. **Verify the cutoff** (direct SQL, read-only, a minute after start):
     ```sql
     SELECT ChainMonth, PurgedAtUtc AS ChainFromUtc FROM dbo.AuditLogChainTombstone WHERE ChainSeq = 0;      -- exactly 1 row, at about the restart time
     SELECT COUNT_BIG(*) FROM dbo.AuditLog WHERE ChainSeq IS NOT NULL
        AND OccurredAtUtc < (SELECT PurgedAtUtc FROM dbo.AuditLogChainTombstone WHERE ChainSeq = 0);       -- 0: no backlog row chained
     SELECT COUNT_BIG(*) FROM dbo.AuditLog WHERE ChainSeq IS NULL;                                        -- about the 3,441+ backlog, unchanged
     ```
     Then `scadabridge audit verify-chain --month <current YYYY-MM> --format table`: `Intact`, `Chain from` = the anchor time, `Pre-cutoff rows` = this month's backlog, `Unchained rows` 0 (or a minute's worth). An earlier month shows `NotChained` with its rows as `Pre-cutoff rows`. Exit code 0 throughout.
  4. **Rollback of the enablement only:** unset `Enabled` and restart. The anchor and any chained rows stay (harmless; chaining resumes from them if re-enabled). Do not delete the anchor row: that is exactly what `verify-chain` reports as tampering.
- **Head export: File sink to a UNC share** that the DB login cannot write. The service account gets create + append-data only. To add once the share is named, with `HashChain:Enabled`:
  ```jsonc
  "AuditLog": { "HashChain": { "HeadExport": {
      "Enabled": true, "IntervalMinutes": 60, "Sink": "File",
      "File": { "Directory": "\\\\<UNC-SHARE-TBD>\\scadabridge-qa\\audit-heads" } } } }
  ```
  `<UNC-SHARE-TBD>` is a placeholder: the owner hasn't named the share. Verify with `scadabridge audit verify-heads --path <dir>`. A failing export shows as `/health/ready` entry `audit-chain-head-export` Degraded (#115).
- **Maintenance login:** later. Keep the single login. `AuditLog:MaintenanceConnectionString` stays unset, and migration 64's grants are harmless without a second login.

## 8. Rollback

**Rollback is a DB restore plus the old binaries**. Schema 64 cannot run under `0e540291`: old binaries cannot partition-switch the flipped `AuditLog` and cannot partition-purge once the chain columns exist.

1. Stop both Sites, then both Centrals (and wait for the Host processes to exit).
2. Restore from the Mac:
   ```sql
   ALTER DATABASE [ScadaBridge] SET SINGLE_USER WITH ROLLBACK IMMEDIATE;
   RESTORE DATABASE [ScadaBridge] FROM DISK = N'F:\MSSQL\BACKUP\ScadaBridge_pre-8e32f353_<ts>.bak' WITH REPLACE, CHECKSUM;
   ALTER DATABASE [ScadaBridge] SET MULTI_USER;
   ```
   Any audit or config rows written after W0 are lost.
3. On each box, rename `app` → `app.failed`, `app.bak.<ts>` → `app`, and the same for `cli`. Restore the pre-window appsettings from `_appsettings-keep-<ts>`. That puts the strategy back to `auto-down`.
4. Start both Centrals together, then both Sites together. Confirm `0e540291`, 58 migrations, health.

**Protocol config is encrypted at rest** (since `2b9a7c3c`, migration `EnforceGlobalInstanceNamesAndProtectProtocolConfiguration`). The format did not change in this range, so `0e540291` reads what `8e32f353` writes. However, **any binary older than `2b9a7c3c` cannot read protected rows at all**, and a binary-only rollback against schema 64 is unsupported. So restore the DB together with the binaries, and never roll back past `0e540291`.

**#116-only rollback** (keep the new build): set `SplitBrainResolverStrategy` back to `auto-down` in both files on both boxes, then stop and start **each pair together**.

## 9. Downtime estimate

| Phase | Estimate | Basis |
|---|---|---|
| W1–W2 stop 4 services | 60–90 s | 2026-10-22: Site stops 10–30 s each, Central 10–17 s |
| W3 migrations | 10–30 s | Flip about 2–4 s on 18 MB; the rest is small |
| W4–W6 swap + config | 30–60 s | 2026-10-22 swap took about 17 s per box |
| W7–W8 start both pairs | 40–60 s | Central ready about 30 s, site about 10 s |
| W6 service recovery | 10–20 s | Four `sc.exe` pairs, while stopped |
| **Full outage (central API + site data path)** | **about 3–4 min** | The 2026-10-22 coordinated window was **2 min 35 s**. #140 does not shorten it: the window's stops run the old binaries |
| Book | **40 min** | Backup and verify plus the gates. No #116 drills (excluded by the go), so no single-pair failovers after the window |

## 10. Related gates in or near this window

- **#131 is DEV-only:** QA has **nothing** to change (deploy-prep `2026-11-12-gates-52-131`), re-confirmed against the QA config DB on 2026-11-13 07:05 (§3: no `TestMesMachine`, `CvdReactor.AlarmStatus` is a stub with no `.Severity` read, no shared scripts, no API method reads alarms). No script step in this window.
- **#52:** `CvdReactor.ProcessRecipeDownload` on QA still addresses `Children["DelmiaRecv"]`. The fix (`2026-11-12-gates-52-131/apply.sh apply 52`) is binary-independent, but it **redeploys Z28061–Z28069**, including **Z28061**. **Stays excluded:** the owner's 2026-11-13 go covers the binary window only, not a Z28061 redeploy. Still open on QA after the window (re-checked §3); it needs its own slot agreed with the Z28061 team.
- **#139:** hash-chain "no backlog" cutoff, `52da02a7` (no migration), **included in `8e32f353`**. The chain stays off in this window; §7 can run later on this build.
- **#138:** ServiceLevel failover, `712c5117` (no migration), included; no effect on QA (no OPC UA connection).
- **#140:** graceful stop, `1a6108c4`, included (§1.2).
- **#116 drills:** excluded by the go (§6, G14).