docs(deploy): QA vq03/vq04 upgrade pack 0e540291 -> 3a59c3f6, migrations 58 -> 64, #116 bundled (not executed) (#107)

Coordinated central-pair window for #107 item 4: range analysis, ordered
idempotent migration script (SHA-256 488da93d...), read-only SQL preflight
results from wonder-sql-vq03, procedure, gates, rollback, downtime estimate.
QA is on 0e540291 / 58 migrations (not b3021e7c / 57). Hash-chain enablement
after the window is gated on #139.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi

docs/deployment/deploy-prep/2026-11-13-qa-vq03-vq04/README.md

@@ -0,0 +1,307 @@
+# Deploy prep: QA vq03/vq04, coordinated central-pair migration window (ScadaBridge#107 item 4) + #116 leader-auto-down
+
+**Prepared:** 2026-11-13. **Not executed.** Running it is a deploy, and it waits for the owner's separate go (#107, 2026-11-13).
+
+**Target build:** `main` @ **`3a59c3f6`**. That is the tip this pack was analysed at. If `main` moves before the window, re-run §1.4 and §2 over `3a59c3f6..<new tip>`: look for new migrations under `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/` and option changes in `*Options.cs` / `*OptionsValidator.cs`. If anything is new, regenerate the script in §2.2.
+
+**Sources:**
+- The read-only SQL preflight on `wonder-sql-vq03` (2026-11-13, §3).
+- SMB reads of the live `app\` folders and appsettings on both boxes, and HTTP health probes. Nothing was written on any box.
+- Deployment records `2026-10-22-wonder-app-vq03-vq04-3808b49f.md` and `2026-10-22-wonder-app-vd03-vq03-vq04-0e540291.md`.
+- The DEV vd03 `424db155` upgrade of 2026-11-11 (migrations 58→64, flip of 2,394 rows / 8 MB in about 1.5 s).
+- Issues #107, #116, #126, #131 and #52.
+
+## Files
+
+| File | What |
+|---|---|
+| `README.md` | This pack. |
+| `preflight.sql` | Read-only preflight (SELECTs and `sp_spaceused` only). Re-run it right before the window. |
+| `migrate-58-to-64.idempotent.sql` | `dotnet ef migrations script 20261022133907_ReviewRemediationIndexes --idempotent`, generated at `3a59c3f6` (EF tools 10.0.10, runtime 10.0.13). It covers six migrations, 59→64. SHA-256 `488da93db0f464ee17687a32a1dabbbe810553c109db5b65d5b3ecbd5c34ecc0`. It is unedited and starts with a UTF-8 BOM at byte 0, which sqlcmd accepts; #126's problem was a BOM *after* a prepended header. |
+
+---
+
+## 0. Correction: QA is on `0e540291` with 58 migrations, not `b3021e7c` with 57
+
+| Evidence | Value |
+|---|---|
+| `ZB.MOM.WW.ScadaBridge.Host.dll` in `E:\ApiInstall\ScadaBridge\app\` on **both** boxes (SMB read) | `1.0.0+0e540291` |
+| `__EFMigrationsHistory` on `wonder-sql-vq03` | **58** rows; the last is `20261022133907_ReviewRemediationIndexes` |
+| Records | 2026-10-22: `b3021e7c`→`3808b49f` (pair window, migration 58), then `3808b49f`→`0e540291` (rolling, no migration). Also on 2026-10-22: shared SQL key and secret stores, and the standby relay |
+
+As a result, these items from the brief are **already on QA** and are not part of this window:
+- bearer-token CLI auth;
+- `MachineDataDb` removed (the Central file still carries a dead `MachineDataDb` key, which is ignored);
+- site secrets via Deploy Artifacts, with the gRPC hub retired;
+- `ReviewRemediationIndexes`;
+- the shared `zbauth`/`zbsecrets` SQL stores on the central pair.
+
+The effective delta is **`0e540291..3a59c3f6`: 128 commits and 6 migrations**.
+
+## 1. What changes: `0e540291..3a59c3f6`
+
+### 1.1 Schema (central DB): 6 migrations
+
+See §2.
+
+### 1.2 Behaviour changes worth knowing on QA
+
+| Area | Change | QA impact |
+|---|---|---|
+| Audit (#41, #107, #109, #113–#115, #42) | Hash chain, head export, maintenance login, Parquet archive and archive retention. | **All default OFF and config-only.** Nothing is enabled in this window (§6). |
+| AuditLog clustered key (#111) | `(OccurredAtUtc, EventId)` replaces `EventId`. Binaries from `5e7d5389` on purge either shape. | Old binaries cannot partition-switch the flipped table. That is the reason for the both-centrals-stopped window and for restore-only rollback. |
+| AuditLog ingest (#124) | Set-based ingest uses FORCESEEK, so concurrent writers no longer deadlock (1205). | Positive. |
+| Cluster downing (#106, #132) | Opt-in `leader-auto-down` with shut-out restart. **The Host now exits 75 after any unplanned ActorSystem termination** (it used to exit 0), under any strategy. | Windows SCM recovery has to restart on a non-zero exit. See §5.3, which checks it in the window. |
+| Host stop (#76) | A routine service stop logs as Information, not `[FTL] host terminated unexpectedly`. | Drop the old "known FTL on stop" exclusion from log review. |
+| Alarms (#129) | `Severity` is banded with higher = more urgent; the raw value is on `Priority`. The MES 900–999 band keys off `Priority`. | The Alarm Summary Sev column and its sort flip visibly. **#131: QA has nothing to change**: `CvdReactor.AlarmStatus` (39) is a stub, and there is no TestMesMachine. |
+| MES-reportable (#59, #60, #102, #104) | Per-binding `IsMesReportable` on native alarm sources, plus UI badges and CSV columns. | Additive, delivered by migration `AddNativeAlarmSourceMesReportable`. |
+| Security (#46, Auth 0.5.0) | Active sessions re-query LDAP group membership: `ScadaBridge:Security:LdapGroupRefresh`, `Enabled` true, every 15 min, 60 s cache. | QA's centrals bind to their box-local GLAuth (`127.0.0.1:3893`) with a service account, which this supports. A user removed from a group loses the role within about 15 min. |
+| Central HTTP (#108) | An `https://` entry in `ASPNETCORE_URLS` now needs `ScadaBridge:Node:Tls` (a certificate) or the node **refuses to boot**. | QA serves plain `http://…:8085`. **The prep step confirms there is no `https://` in either Central's service env** (§5.1). |
+| Akka.Remote TLS (#64) | `ScadaBridge:Cluster:RemotingTls`, default off. | None. |
+| DCL OPC UA (#117, #121, #112, #58; SDK 1.5.378.106 → **2.0.1**, #125) | Good subcodes now map to Good. Secure sessions present a real app cert (`OpcUaGlobal:*`). `BadResourceUnavailable` is treated as transient. Scalar writes are coerced to the node type. | **QA has no OPC UA connection.** Its only data connection is `MxGateway`, so there is no exposure. |
+| DCL MxGateway (#128) | The resume cursor ignores `worker_sequence` 0 sentinels. | Positive for QA's single connection. |
+| Site runtime (#135, #136, #36) | Post-boot settle sweep: replicated instances start within seconds. Site commands fail over within the call. The legacy `WriteBatchAndWaitAsync` waits for its own reset echo. | Faster site start after the swap. |
+| Inbound API (#103, #75, #80) | Recompile and invalidate reach the peer central. Compile errors are visible through `api-method validate/recompile`. | Positive on a pair. |
+| Pending updates (#62, #63) | A DB rowversion watermark short-circuits an unchanged fleet (this is what the 18 `RowVersion` columns are for). The counts appear on the Health dashboard and CLI. | Needs migration `AddConfigGraphRowVersions`. |
+| Transport (#45, #110, #118, #93–#95, #100) | Optional detached bundle signing (default off). Derived templates round-trip. Name lookups are case-insensitive. | None unless bundles are used. |
+| Management (#105) | Case-variant instance override names are refused with a did-you-mean hint. | Behaviour change for scripts that send wrong-case names. |
+| Logging (#56, #120, #94) | Akka logging goes through Serilog. `ExecutionId`, `InstanceName` and `ScriptName` are added to the LogContext. | More (and better) lines in the file logs. |
+| Telemetry 0.1.0 → **0.3.1** (scadaproj#55/#56) | `service.version` = git SHA. **Logs now go over OTLP when `Exporter=Otlp`.** The endpoint check is stricter (a scheme-less `host:port` fails). | QA's service env (2026-11-12) points at `http://127.0.0.1:4317`, which has a scheme, so it boots. The box-local `ZbOtelCollector` starts receiving **logs** as well. Watch `E:\ApiInstall\OtelCollector\data\logs.jsonl` (about 1.8 GB cap per box). |
+| CLI (#137) | Secret-bearing option values can be read from a file or stdin. | Optional. |
+| Health (#97) | Site `/health/active` uses the shared role-scoped check. | The Overview site chips now show active and standby correctly. |
+
+### 1.3 Shared-library and package pins
+
+| Package | QA today (`0e540291`) | Tip | Note |
+|---|---|---|---|
+| ZB.MOM.WW.Auth.* | 0.3.0 | **0.5.0** | `BindMode` is still `ServiceAccount`. Adds the passwordless group lookup (#46). |
+| ZB.MOM.WW.LocalDb.* | 0.3.4 | **0.3.7** | Wire-compatible, but **the pair stops and starts together**. The library now enforces `LocalDb:Replication:ApiKey` itself, which must be byte-identical on both site nodes (it already is: QA's own fail-closed interceptor has been connected since 2026-09-10). The tombstone ledger grows while a peer is down (#16). `MaxBatchSize: 16` in both Site files is a retired pin; it is harmless. |
+| ZB.MOM.WW.Telemetry(.Serilog) | 0.1.0 | **0.3.1** | See above. |
+| ZB.MOM.WW.Theme | 0.4.1 | **0.5.3** | ScrollNav rail (#61). |
+| ZB.MOM.WW.Secrets | 0.6.2 | 0.6.2 | Unchanged. |
+| Microsoft.Extensions/EF/ASP.NET | 10.0.7 | 10.0.13 | Sqlite pool fix (#98). SqlClient 6.1.7. |
+| OPCFoundation Opc.Ua.Client | 1.5.378.106 | 2.0.1 | No OPC UA connections on QA. |
+| Parquet.Net | — | 6.1.0 | Archive and export (off). |
+
+### 1.4 Settings and env
+
+**No setting has to change for this build to boot on QA.** No shipped appsettings changed in the range, and every new option has a safe default. New keys, all optional:
+- `ScadaBridge:Cluster:{LeaderAutoDown:{RestartWhenShutOut,ShutOutGrace},RemotingTls:*}`
+- `ScadaBridge:Node:Tls:*`
+- `ScadaBridge:Communication:GrpcClientIdleTtl`
+- `ScadaBridge:DeploymentManager:PendingUpdatesFullEvaluationEvery`
+- `ScadaBridge:SiteRuntime:InstanceSweepSettle{Interval,Window}Seconds`
+- `ScadaBridge:Security:LdapGroupRefresh:*`
+- `ScadaBridge:Transport:Signing:*`
+- `OpcUaGlobal:{ApplicationUri,ApplicationCertificateSubject,OwnCertificateStorePath,AutoCreateApplicationCertificate,ApplicationCertificateLifetimeMonths}`
+- `AuditLog:{MaintenanceConnectionString,HashChain:*,HashChain:HeadExport:*,Archive:*}`
+- `LocalDb:Replication:{MaxSnapshotDeferredRows,TombstoneHardCap}`
+
+**Changes this window does make:** only the #116 strategy flip (§5.4).
+
+**Drift found, not fixed here:** the vq03 and vq04 `appsettings.Central.json` differ in `JwtExpiryMinutes` (25 vs 15) and `IdleTimeoutMinutes` (60 vs 30). The CLI bearer token is signed with the shared key, so the pair still interoperates, but session lifetime depends on which central answers. Align them (pick one pair of values) as a separate change, or here if the owner says so.
+
+## 2. Migrations 58 → 64
+
+### 2.1 Ordered list (EF timestamp order = apply order)
+
+| # | Migration | Issue | What it does | Cost on QA | Old-binary compatible? |
+|---|---|---|---|---|---|
+| 59 | `20261109200211_AddNativeAlarmSourceMesReportable` | #102/#59 | Nullable `IsMesReportable` on native-source bindings/overrides | Metadata only | Yes (additive) |
+| 60 | `20261109203753_PrefixSearchIndexes` | #50 | Prefix-search indexes for Deployments and config-audit | Tiny tables | Yes |
+| 61 | `20261109211317_AddAuditLogHashChain` | #41 | `ChainSeq`/`RowHash` on `AuditLog`. `AuditLogChainHead` and `AuditLogChainTombstone`. Two filtered indexes, wrapped in `EXEC(N'…')` (the #126 fix). Roles `scadabridge_audit_chainer` (new) and `_purger`/`_writer` (exist), grants, and a DENY to the writer on the chain tables | 3,441 rows; seconds | **No**: old binaries can't partition-purge once the columns exist |
+| 62 | `20261110032839_AddConfigGraphRowVersions` | #62 | Non-concurrency `rowversion` + index on 18 configuration tables. SQL Server fills each table as it adds the column | All config tables are under 2,300 rows; seconds | Yes (ignored) |
+| 63 | `20261110033732_FlipAuditLogClusteredKey` | #111 | **Offline whole-table rebuild**: drop the 10 NCIs + PK, re-add the PK clustered on `(OccurredAtUtc, EventId)` on `ps_AuditLog_Month`, add `UX_AuditLog_EventId_Occurred`, recreate 12 NCIs. Guarded by "clustered key ordinal 1 = `EventId`" | **18.3 MB reserved / 3,441 rows → about 2–4 s** (DEV: 8 MB / 2,394 rows in about 1.5 s). Log about 2× = about 40 MB, against 142 MB free in a 200 MB log (64 MB autogrowth, 240 GB free on G:) | **No**: older binaries can't partition-switch a flipped table |
+| 64 | `20261110120000_AuditMaintenanceGrants` | #113 | `scadabridge_audit_purger`: `ALTER ANY DATASPACE` + `UPDATE (SourceNode)` on `dbo.AuditLog` | Instant | Yes |
+
+**Owner's §item-4 list vs EF order.** `AddNativeAlarmSourceMesReportable` sorts **first**, by timestamp. The EF script applies migrations in timestamp order, which is also the order DEV vd03 used. The relative order of the five audit/config migrations is the one the owner listed.
+
+### 2.2 How they are applied
+
+- **Tool:** the generated idempotent script `migrate-58-to-64.idempotent.sql` in this folder. Per the #126 deploy note, QA can run it **as is**, with no GO insertion. It is 6 batches, each `BEGIN TRANSACTION … COMMIT; GO`.
+  - `IdempotentMigrationScriptTests` (MSSQL fixture) proves that from-57 → head runs clean and a re-run is a no-op.
+  - The hand scripts in `docs/plans/sql/` remain the fallback. They were BOM-fixed in `c7dfc21b`. There is no hand script for migrations 59 and 64, so the hand route needs EF fragments as well. That makes it two tools for one job; prefer the single EF script.
+- **When:** with **both centrals and both sites stopped** (§5). This is required by migrations 61 and 63.
+- **Where from:** the Mac, straight to SQL (direct SQL is always allowed, so no WinRM is needed for this step):
+  ```bash
+  cd docs/deployment/deploy-prep/2026-11-13-qa-vq03-vq04
+  shasum -a 256 migrate-58-to-64.idempotent.sql   # must be 488da93d…c34ecc0
+  printf 'SET XACT_ABORT ON;\n' > /tmp/xact.sql   # session-scoped; same connection as the script
+  SQLCMDPASSWORD=… sqlcmd -S wonder-sql-vq03.zmr.zimmer.com,1433 -U <login> -C -d ScadaBridge -I -b \
+      -i /tmp/xact.sql,migrate-58-to-64.idempotent.sql
+  ```
+  The fallback is the same command with Windows `sqlcmd -C -b -I` on vq03, as in the 2026-10-22 window. **vq04 has no `sqlcmd`.**
+- **Idempotency:**
+  - Every statement is guarded by `IF NOT EXISTS (… __EFMigrationsHistory …)`.
+  - Role creation is additionally guarded by `IF DATABASE_PRINCIPAL_ID(…) IS NULL`, and the flip by the key-shape check.
+  - Re-running after a partial failure therefore completes only the missing migrations. A failed batch rolls back whole, because of `XACT_ABORT` + `-b`.
+- **Expected after:**
+  - 64 rows, with the last one `20261110120000_AuditMaintenanceGrants`.
+  - Clustered key ordinal 1 = `OccurredAtUtc`.
+  - `AuditLogChainHead`/`AuditLogChainTombstone` exist.
+  - 3 `scadabridge_audit_*` roles.
+  - Re-run `preflight.sql` sections 1, 4 and 7.
+
+## 3. Read-only SQL preflight: results (2026-11-13 ~02:45 EST)
+
+Server **`wonder-sql-vq03.zmr.zimmer.com,1433`** (`WONDER-SQL-VQ03`, SQL Server 2022 RTM-CU24 16.0.4245.2, **Enterprise**), DB `ScadaBridge`. Login `wwadmin`, password via `SQLCMDPASSWORD` only, `-C`. SELECTs only. The script is in `preflight.sql`.
+
+| Check | Result |
+|---|---|
+| `__EFMigrationsHistory` | **58**; last `20261022133907_ReviewRemediationIndexes` (ProductVersion 10.0.7) |
+| Recovery model | **FULL**; `log_reuse_wait` NOTHING; compatibility 160 |
+| Log backups | Regular: full at 22:00 daily, log at about 12:00 / 24:00. A `COPY_ONLY` backup does not disturb the chain |
+| `dbo.AuditLog` | **3,441 rows**; reserved **18.3 MB** (data 10.7 MB, index 3.2 MB, unused 4.4 MB); 2026-05-29 → 2026-11-12 |
+| Clustered key now | `PK_AuditLog` = (`EventId`, `OccurredAtUtc`) on `ps_AuditLog_Month`, so the flip has not been applied yet |
+| `pf_AuditLog_Month` | 24 boundaries, the last 2027-12-01 |
+| Log file | 200 MB, **57 MB used (28.6 %)**, 64 MB autogrowth, unlimited; G: 245,855 MB free |
+| Data file | 904 MB, 64 MB autogrowth; F: 249,267 MB free. The largest table is `KpiSample` (4.9 M rows, 804 MB), which the window does not touch |
+| SourceNode backfill | **Not run.** **843 rows** have `SourceNode IS NULL`: InboundRequest 793 (newest 2026-07-07) and InboundAuthFailure 50 (newest 2026-06-27), all from before M5.6. Every row since 2026-07-07 is stamped (central-a 1,742, central-b 777, node-a 59, node-b 20) |
+| Hash-chain objects | None yet. Roles `scadabridge_audit_writer` and `_purger` exist; `_chainer` does not. `DeploymentRecords.RowVersion` already exists, and `AddConfigGraphRowVersions` handles that table (covered by the from-57 test) |
+| Topology | 1 data connection (`MxGateway`, protocol MxGateway, site 2), 1 site (`site-a`, artifacts `sha256:d79ba1a3…`, deployed 2026-10-27), 10 instances |
+| Live health (HTTP, 02:45 EST) | All four `/health/ready` return 200. **vq03 is active** for central and site; vq04 answers 503. Central: memberCount 2, unreachable 0, 5/5 singletons |
+
+**Window estimate for the flip:** about 2–4 s on 18 MB, scaled from DEV's 1.5 s for 8 MB. All six migrations take well under 30 s.
+
+## 4. #116 leader-auto-down: bundled
+
+**Recommendation: bundle it.** It fits the same window with no extra restart:
+- The `3a59c3f6` binary already contains #106 and #132 (shut-out restart, exit 75).
+- The change is an **appsettings flip only**.
+- #116's one rollout rule is "each pair stops and starts together; never roll across a downing-provider change", and this window already stops all four services and starts each pair together.
+- The owner approved the QA rollout (rig and env2 gates passed, and env2 is soaking).
+- If it went in a separate window, QA would take a second full outage.
+
+The order inside the window is: binaries → migrations → config (including the #116 flip) → start. §5.4 has the edit, §6 G9 the drill, and §8 the rollback, which is to flip the value back.
+
+## 5. Procedure
+
+All times are local (EST). Roles: one operator on the Mac; WinRM for the on-box steps only, under the deploy-only scope; SMB for files; direct SQL for the DB. Layout on each box: `E:\ApiInstall\ScadaBridge\{app,cli,central,site,…}`. **`app\` is shared by the `ScadaBridge-Central` and `ScadaBridge-Site` services on that box**, so a swap needs both services on that box stopped. The sites stop anyway, because LocalDb 0.3.7 pairs stop and start together.
+
+### 5.1 Prep (before the window, no downtime)
+
+1. Build: publish `3a59c3f6` self-contained win-x64, app and cli, with the same flags as the 2026-10-22 builds. Zip it, record the SHA-256, and confirm ProductVersion `1.0.0+3a59c3f6…`. Stage it over SMB to `E:\psrem-stage\3a59c3f6\` on both boxes and re-verify the hash on each box.
+2. Re-run `preflight.sql` and compare it with §3. Abort if the migration count is not 58 or the key is not `EventId`-first.
+3. On-box prep (WinRM, deploy scope; read-only except the exports):
+   - `reg export` both services' env;
+   - **confirm that `ASPNETCORE_URLS` on both Centrals has no `https://`** (#108 refuse-to-boot);
+   - confirm that `LocalDb__Replication__ApiKey` is present and identical on both Site services (compare hashes, never values);
+   - record `sc.exe qfailure` and `sc.exe qfailureflag` for all four services (input to §5.3);
+   - check E: free space.
+4. Hash the three appsettings files on each box (they are kept and restored across the swap).
+5. Agree the slot. Tell the team that uses **Z28061** about the restart. The window restarts every instance actor, but deploys and writes nothing to Z28061.
+
+### 5.2 Window
+
+| Step | Action | Notes |
+|---|---|---|
+| W0 | **Backup** from the Mac (direct SQL): `BACKUP DATABASE [ScadaBridge] TO DISK = N'F:\MSSQL\BACKUP\ScadaBridge_pre-3a59c3f6_<ts>.bak' WITH COPY_ONLY, CHECKSUM, COMPRESSION, STATS = 10;` then `RESTORE VERIFYONLY FROM DISK = … WITH CHECKSUM;` | About 900 MB of data, compressed. The previous run took a few seconds. Taken while everything is still up; nothing writes config during the window |
+| W1 | **Window opens.** Stop `ScadaBridge-Site` on the standby box (vq04), then on the active box (vq03) | The site data path stops here |
+| W2 | Stop `ScadaBridge-Central` on vq04, then on vq03. **Wait until no `ZB.MOM.WW.ScadaBridge.Host.exe` process remains on either box** | Both centrals are now down, which migrations 61 and 63 require |
+| W3 | Migrate from the Mac (§2.2). Check: 64 migrations, flipped key | About 10–30 s |
+| W4 | Swap on each box: keep the three appsettings, rename `app` / `cli` → `*.bak.<ts>`, unpack, restore the appsettings (hash match) | As in 2026-10-22 |
+| W5 | Config edits (§5.4) on both boxes, pushed over SMB while stopped. Record new hashes | #116 |
+| W6 | Service recovery (§5.3) if the prep found it missing | |
+| W7 | **Start both Centrals together** (vq03 and vq04 within a few seconds of each other), then wait for `/health/ready` 200 and memberCount 2 on both | BootstrapGuard is on. Don't start one central and wait (memory: a cold single-node start can self-down) |
+| W8 | **Start both Sites together**, then wait for ready 200 on both, `localdb replicationConnected:true` and backlog 0. **Window closes** | |
+
+### 5.3 Service recovery for exit 75
+
+The Host now exits **75** when it shuts itself down (#132) or its ActorSystem terminates unexpectedly. Windows recovery acts on a clean-stop-with-non-zero-exit only when `failureflag` is 1. If §5.1 step 3 shows that any of the four services lacks restart actions or has `FailureActionsOnNonCrashFailures = FALSE`, set it while the services are stopped:
+
+```
+sc.exe failure ScadaBridge-Central reset= 86400 actions= restart/10000/restart/30000/restart/60000
+sc.exe failureflag ScadaBridge-Central 1
+```
+
+Do the same for `ScadaBridge-Site`. This is a change to the service definition, so the owner should confirm it when giving the go. Without it, the #132 self-heal leaves a node **stopped**, not restarted.
+
+### 5.4 Config edits (both boxes, while stopped)
+
+| File | Edit | Why |
+|---|---|---|
+| `app\appsettings.Central.json` | `"SplitBrainResolverStrategy": "auto-down"` → `"leader-auto-down"` | #116 |
+| `app\appsettings.Site.json` | the same (in this PowerShell-written file the line reads `"SplitBrainResolverStrategy":  "auto-down"`, with two spaces) | #116 |
+
+- `LeaderAutoDown` stays at its defaults: `RestartWhenShutOut` true, `ShutOutGrace` 00:01:00.
+- `StableAfter` 15 s, heartbeat 2 s and failure detection 10 s are unchanged, which are the rig and env2 values.
+- Preserve each file's encoding: the Site file has a BOM.
+- Back up the originals as `_appsettings-keep-<ts>`.
+
+**Not changed:**
+- `AuditLog:*`: the chain, head export, archive and maintenance login all stay off (§7).
+- `MachineDataDb`: dead, so leave it.
+- The JWT and idle drift (§1.4): only if the owner asks.
+
+## 6. Post-deploy gates
+
+Run them from the Mac over HTTP, SQL, SMB and the CLI. No ad-hoc WinRM, except the §5 exit-code read that is part of the deploy. **Nothing deploys to, writes to or flow-debugs Z28061.** Snapshots and smokes use Z28062–Z28069 and SAPID `000000`.
+
+| Gate | Check | Pass |
+|---|---|---|
+| G1 Build | `Host.dll` ProductVersion on both boxes (SMB, `strings`) | `1.0.0+3a59c3f6` ×2 |
+| G2 Schema | `preflight.sql` §1/§4/§7 | 64; `OccurredAtUtc` first; chain tables + 3 roles |
+| G3 Health | `/health/ready` on :8085 and :8084 on both boxes | 200 ×4; `database`, `akka-cluster`, `required-singletons` Healthy |
+| G4 Cluster | Central `akka-cluster` data on both nodes; site `/health/ready` | memberCount 2, unreachable 0, one leader. Site `localdb.replicationConnected:true`, backlog 0 |
+| G5 Active node | `/health/active` on both pairs, sampled 10× | Exactly one 200 per pair at every sample |
+| G6 Boot logs | SMB-pull `central\logs` and `site\logs` for today | **#116:** `[leader-auto-down] started on …` on all 4 and **no** `auto-down-unreachable-after … keep-majority` deprecation line. No new ERR/FTL beyond the first-probe `akka-cluster … Removed` lines. Service stops log at Information, with no `[FTL]` (#76) |
+| G7 CLI | `POST /auth/token` 200; `scadabridge health` / `cluster` status; `security api-key list` on **both** centrals | The same 3 keys (`MESTestKey`, `MES_Key`, `delmia-notifier-qa`) on both |
+| G8 Inbound API | A bogus-key call to each node → 401 `UNAUTHORIZED` (the standby relays). `flow-debug run-method` of `SimpleAlarmStatusRequest` (id 9) with `SAPID=000000` | InboundRequest → BTDB DbCall (1 row) → Response Ok |
+| G9 MxGateway | The site health report shows connection `MxGateway` Connected. Read-only snapshots of Z28062–Z28069 | Good/Bad counts match the pre-window snapshot (2026-10-22 baseline: 242 Good / 2 Bad across Z28061–69). Take the before-snapshot in prep |
+| G10 AuditLog | SQL: `SELECT COUNT(*), MAX(OccurredAtUtc) FROM dbo.AuditLog WHERE OccurredAtUtc > <window close>` after G8 | New rows, `SourceNode` set, `ChainSeq` NULL (chain off); no 1205 in the logs |
+| G11 Overview | `http://wonder-app-vq03:5320` and `vq04:5320` | The ScadaBridge Central and Site clusters each show one Active, the leader chip, and **no** "Split brain" |
+| G12 Artifacts / staleness | `instance diff` on every instance; the Sites page | No stale instances; artifact hash unchanged `sha256:d79ba1a3…`. No redeploy needed |
+| G13 OTLP | SMB: `E:\ApiInstall\OtelCollector\data\logs.jsonl` on each box | Lines with `service.name` scadabridge and `service.version` 3a59c3f6 (logs are new with Telemetry 0.3.x) |
+| G14 #116 drill (bundled) | Kill the **active** central's Host (`Stop-Process -Force`, deploy scope), then repeat on the standby. Watch `/health/active` and the logs | Survivor `[leader-auto-down] … DOWNING` about +27 s, takes over in about 30 s; the victim is restarted by SCM and rejoins as standby; then 10/10 single-active. **The site pair drill re-hosts every instance, including Z28061 (about 30 s data gap)**, so run it only if the owner OKs that in the go. Otherwise record "central only" in #116 |
+
+## 7. Hash chain, head export, maintenance login (the #107 owner decisions)
+
+- **In this window: the chain stays OFF on QA** (`AuditLog:HashChain:Enabled` is unset, so false). The migration only adds the columns, tables and roles.
+- **After the window, as a separate step:**
+  1. SourceNode backfill first, because a chained row's `SourceNode` is frozen. Run `scadabridge audit backfill-source-node --before 2026-07-08T00:00:00Z --sentinel unknown`, which covers the 843 pre-M5.6 rows (Admin; idempotent). Then confirm that `preflight.sql` §6 shows 0 NULLs.
+  2. Then set `AuditLog:HashChain:Enabled=true` on both centrals and restart them **together**.
+  3. **"No backlog chaining" (only rows written after enablement): ⚠ BLOCKED, #139.** On `main` the chaining job links *every* unchained row, oldest first (`AuditLogHashChain.cs:102`), and has no cutoff option. Enabling it on QA as the code stands would chain all 3,441+ historical rows, which is the opposite of the decision. Wait for #139 (a cutoff option, or a revised decision) before step 2.
+- **Head export: File sink to a UNC share** that the DB login cannot write. The service account gets create + append-data only. To add once the share is named, with `HashChain:Enabled`:
+  ```jsonc
+  "AuditLog": { "HashChain": { "HeadExport": {
+      "Enabled": true, "IntervalMinutes": 60, "Sink": "File",
+      "File": { "Directory": "\\\\<UNC-SHARE-TBD>\\scadabridge-qa\\audit-heads" } } } }
+  ```
+  `<UNC-SHARE-TBD>` is a placeholder: the owner hasn't named the share. Verify with `scadabridge audit verify-heads --path <dir>`. A failing export shows as `/health/ready` entry `audit-chain-head-export` Degraded (#115).
+- **Maintenance login:** later. Keep the single login. `AuditLog:MaintenanceConnectionString` stays unset, and migration 64's grants are harmless without a second login.
+
+## 8. Rollback
+
+**Rollback is a DB restore plus the old binaries**. Schema 64 cannot run under `0e540291`: old binaries cannot partition-switch the flipped `AuditLog` and cannot partition-purge once the chain columns exist.
+
+1. Stop both Sites, then both Centrals (and wait for the Host processes to exit).
+2. Restore from the Mac:
+   ```sql
+   ALTER DATABASE [ScadaBridge] SET SINGLE_USER WITH ROLLBACK IMMEDIATE;
+   RESTORE DATABASE [ScadaBridge] FROM DISK = N'F:\MSSQL\BACKUP\ScadaBridge_pre-3a59c3f6_<ts>.bak' WITH REPLACE, CHECKSUM;
+   ALTER DATABASE [ScadaBridge] SET MULTI_USER;
+   ```
+   Any audit or config rows written after W0 are lost.
+3. On each box, rename `app` → `app.failed`, `app.bak.<ts>` → `app`, and the same for `cli`. Restore the pre-window appsettings from `_appsettings-keep-<ts>`. That puts the strategy back to `auto-down`.
+4. Start both Centrals together, then both Sites together. Confirm `0e540291`, 58 migrations, health.
+
+**Protocol config is encrypted at rest** (since `2b9a7c3c`, migration `EnforceGlobalInstanceNamesAndProtectProtocolConfiguration`). The format did not change in this range, so `0e540291` reads what `3a59c3f6` writes. However, **any binary older than `2b9a7c3c` cannot read protected rows at all**, and a binary-only rollback against schema 64 is unsupported. So restore the DB together with the binaries, and never roll back past `0e540291`.
+
+**#116-only rollback** (keep the new build): set `SplitBrainResolverStrategy` back to `auto-down` in both files on both boxes, then stop and start **each pair together**.
+
+## 9. Downtime estimate
+
+| Phase | Estimate | Basis |
+|---|---|---|
+| W1–W2 stop 4 services | 60–90 s | 2026-10-22: Site stops 10–30 s each, Central 10–17 s |
+| W3 migrations | 10–30 s | Flip about 2–4 s on 18 MB; the rest is small |
+| W4–W6 swap + config | 30–60 s | 2026-10-22 swap took about 17 s per box |
+| W7–W8 start both pairs | 40–60 s | Central ready about 30 s, site about 10 s |
+| **Full outage (central API + site data path)** | **about 3–4 min** | The 2026-10-22 coordinated window was **2 min 35 s** |
+| Book | **45 min** | Backup and verify, the gates, and the #116 kill drills (each adds about 30 s of single-pair failover, but no full outage) |
+
+## 10. Related gates in or near this window
+
+- **#131:** QA has **nothing** to change (deploy-prep `2026-11-12-gates-52-131`).
+- **#52:** `CvdReactor.ProcessRecipeDownload` on QA still addresses `Children["DelmiaRecv"]`. The fix (`2026-11-12-gates-52-131/apply.sh apply 52`) is binary-independent, but it **redeploys Z28061–Z28069**, including **Z28061**. **Not bundled by default.** If the owner agrees a Z28061 slot in the same go, run it after G12, so there is still only one redeploy.
+- **#139 (new):** hash-chain "no backlog" gate, §7.

docs/deployment/deploy-prep/2026-11-13-qa-vq03-vq04/migrate-58-to-64.idempotent.sql

@@ -0,0 +1,560 @@
+BEGIN TRANSACTION;
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261109200211_AddNativeAlarmSourceMesReportable'
+)
+BEGIN
+    ALTER TABLE [TemplateNativeAlarmSources] ADD [IsMesReportable] bit NOT NULL DEFAULT CAST(0 AS bit);
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261109200211_AddNativeAlarmSourceMesReportable'
+)
+BEGIN
+    ALTER TABLE [InstanceNativeAlarmSourceOverrides] ADD [IsMesReportableOverride] bit NULL;
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261109200211_AddNativeAlarmSourceMesReportable'
+)
+BEGIN
+    INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion])
+    VALUES (N'20261109200211_AddNativeAlarmSourceMesReportable', N'10.0.13');
+END;
+
+COMMIT;
+GO
+
+BEGIN TRANSACTION;
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261109203753_PrefixSearchIndexes'
+)
+BEGIN
+    CREATE INDEX [IX_DeploymentRecords_DeployedBy] ON [DeploymentRecords] ([DeployedBy]);
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261109203753_PrefixSearchIndexes'
+)
+BEGIN
+    CREATE INDEX [IX_DeploymentRecords_RevisionHash] ON [DeploymentRecords] ([RevisionHash]);
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261109203753_PrefixSearchIndexes'
+)
+BEGIN
+    CREATE INDEX [IX_AuditLogEntries_EntityName] ON [AuditLogEntries] ([EntityName]);
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261109203753_PrefixSearchIndexes'
+)
+BEGIN
+    INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion])
+    VALUES (N'20261109203753_PrefixSearchIndexes', N'10.0.13');
+END;
+
+COMMIT;
+GO
+
+BEGIN TRANSACTION;
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261109211317_AddAuditLogHashChain'
+)
+BEGIN
+    ALTER TABLE [AuditLog] ADD [ChainSeq] bigint NULL;
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261109211317_AddAuditLogHashChain'
+)
+BEGIN
+    ALTER TABLE [AuditLog] ADD [RowHash] binary(32) NULL;
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261109211317_AddAuditLogHashChain'
+)
+BEGIN
+    CREATE TABLE [AuditLogChainHead] (
+        [ChainMonth] date NOT NULL,
+        [LastSeq] bigint NOT NULL,
+        [LastHash] binary(32) NOT NULL,
+        [PurgedThroughSeq] bigint NULL,
+        [PurgedThroughHash] binary(32) NULL,
+        [UpdatedAtUtc] datetime2 NOT NULL,
+        CONSTRAINT [PK_AuditLogChainHead] PRIMARY KEY ([ChainMonth])
+    );
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261109211317_AddAuditLogHashChain'
+)
+BEGIN
+    CREATE TABLE [AuditLogChainTombstone] (
+        [ChainMonth] date NOT NULL,
+        [ChainSeq] bigint NOT NULL,
+        [RowHash] binary(32) NOT NULL,
+        [EventId] uniqueidentifier NOT NULL,
+        [PurgedAtUtc] datetime2 NOT NULL,
+        CONSTRAINT [PK_AuditLogChainTombstone] PRIMARY KEY ([ChainMonth], [ChainSeq])
+    );
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261109211317_AddAuditLogHashChain'
+)
+BEGIN
+
+    EXEC(N'CREATE NONCLUSTERED INDEX IX_AuditLog_Unchained
+    ON dbo.AuditLog (OccurredAtUtc, EventId)
+    WHERE ChainSeq IS NULL
+    ON ps_AuditLog_Month(OccurredAtUtc);');
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261109211317_AddAuditLogHashChain'
+)
+BEGIN
+
+    EXEC(N'CREATE NONCLUSTERED INDEX IX_AuditLog_ChainSeq
+    ON dbo.AuditLog (ChainSeq)
+    WHERE ChainSeq IS NOT NULL
+    ON ps_AuditLog_Month(OccurredAtUtc);');
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261109211317_AddAuditLogHashChain'
+)
+BEGIN
+
+    IF DATABASE_PRINCIPAL_ID('scadabridge_audit_chainer') IS NULL
+        EXEC sp_executesql N'CREATE ROLE scadabridge_audit_chainer';
+    -- The chaining job reads unchained rows and writes ONLY the two link columns.
+    GRANT SELECT ON dbo.AuditLog TO scadabridge_audit_chainer;
+    GRANT UPDATE (ChainSeq, RowHash) ON dbo.AuditLog TO scadabridge_audit_chainer;
+    GRANT SELECT, INSERT, UPDATE ON dbo.AuditLogChainHead TO scadabridge_audit_chainer;
+    GRANT SELECT ON dbo.AuditLogChainTombstone TO scadabridge_audit_chainer;
+
+    IF DATABASE_PRINCIPAL_ID('scadabridge_audit_purger') IS NULL
+        EXEC sp_executesql N'CREATE ROLE scadabridge_audit_purger';
+    -- The per-channel purge writes a tombstone per chained row it removes; the partition
+    -- switch moves the month's head to its checkpoint anchor and drops its tombstones.
+    GRANT SELECT, UPDATE ON dbo.AuditLogChainHead TO scadabridge_audit_purger;
+    GRANT SELECT, INSERT, DELETE ON dbo.AuditLogChainTombstone TO scadabridge_audit_purger;
+
+    IF DATABASE_PRINCIPAL_ID('scadabridge_audit_writer') IS NULL
+        EXEC sp_executesql N'CREATE ROLE scadabridge_audit_writer';
+    -- The ingest principal must never be able to forge chain state.
+    DENY INSERT, UPDATE, DELETE ON dbo.AuditLogChainHead TO scadabridge_audit_writer;
+    DENY INSERT, UPDATE, DELETE ON dbo.AuditLogChainTombstone TO scadabridge_audit_writer;
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261109211317_AddAuditLogHashChain'
+)
+BEGIN
+    INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion])
+    VALUES (N'20261109211317_AddAuditLogHashChain', N'10.0.13');
+END;
+
+COMMIT;
+GO
+
+BEGIN TRANSACTION;
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261110032839_AddConfigGraphRowVersions'
+)
+BEGIN
+    ALTER TABLE [TemplateScripts] ADD [RowVersion] rowversion NOT NULL;
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261110032839_AddConfigGraphRowVersions'
+)
+BEGIN
+    ALTER TABLE [Templates] ADD [RowVersion] rowversion NOT NULL;
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261110032839_AddConfigGraphRowVersions'
+)
+BEGIN
+    ALTER TABLE [TemplateNativeAlarmSources] ADD [RowVersion] rowversion NOT NULL;
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261110032839_AddConfigGraphRowVersions'
+)
+BEGIN
+    ALTER TABLE [TemplateCompositions] ADD [RowVersion] rowversion NOT NULL;
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261110032839_AddConfigGraphRowVersions'
+)
+BEGIN
+    ALTER TABLE [TemplateAttributes] ADD [RowVersion] rowversion NOT NULL;
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261110032839_AddConfigGraphRowVersions'
+)
+BEGIN
+    ALTER TABLE [TemplateAlarms] ADD [RowVersion] rowversion NOT NULL;
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261110032839_AddConfigGraphRowVersions'
+)
+BEGIN
+    ALTER TABLE [Sites] ADD [RowVersion] rowversion NOT NULL;
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261110032839_AddConfigGraphRowVersions'
+)
+BEGIN
+    ALTER TABLE [SharedScripts] ADD [RowVersion] rowversion NOT NULL;
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261110032839_AddConfigGraphRowVersions'
+)
+BEGIN
+    ALTER TABLE [Instances] ADD [RowVersion] rowversion NOT NULL;
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261110032839_AddConfigGraphRowVersions'
+)
+BEGIN
+    ALTER TABLE [InstanceNativeAlarmSourceOverrides] ADD [RowVersion] rowversion NOT NULL;
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261110032839_AddConfigGraphRowVersions'
+)
+BEGIN
+    ALTER TABLE [InstanceConnectionBindings] ADD [RowVersion] rowversion NOT NULL;
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261110032839_AddConfigGraphRowVersions'
+)
+BEGIN
+    ALTER TABLE [InstanceAttributeOverrides] ADD [RowVersion] rowversion NOT NULL;
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261110032839_AddConfigGraphRowVersions'
+)
+BEGIN
+    ALTER TABLE [InstanceAlarmOverrides] ADD [RowVersion] rowversion NOT NULL;
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261110032839_AddConfigGraphRowVersions'
+)
+BEGIN
+    ALTER TABLE [ExternalSystemMethods] ADD [RowVersion] rowversion NOT NULL;
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261110032839_AddConfigGraphRowVersions'
+)
+BEGIN
+    ALTER TABLE [ExternalSystemDefinitions] ADD [RowVersion] rowversion NOT NULL;
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261110032839_AddConfigGraphRowVersions'
+)
+BEGIN
+    ALTER TABLE [DeployedConfigSnapshots] ADD [RowVersion] rowversion NOT NULL;
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261110032839_AddConfigGraphRowVersions'
+)
+BEGIN
+    ALTER TABLE [DataConnections] ADD [RowVersion] rowversion NOT NULL;
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261110032839_AddConfigGraphRowVersions'
+)
+BEGIN
+    ALTER TABLE [DatabaseConnectionDefinitions] ADD [RowVersion] rowversion NOT NULL;
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261110032839_AddConfigGraphRowVersions'
+)
+BEGIN
+    CREATE INDEX [IX_TemplateScripts_RowVersion] ON [TemplateScripts] ([RowVersion]);
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261110032839_AddConfigGraphRowVersions'
+)
+BEGIN
+    CREATE INDEX [IX_Templates_RowVersion] ON [Templates] ([RowVersion]);
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261110032839_AddConfigGraphRowVersions'
+)
+BEGIN
+    CREATE INDEX [IX_TemplateNativeAlarmSources_RowVersion] ON [TemplateNativeAlarmSources] ([RowVersion]);
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261110032839_AddConfigGraphRowVersions'
+)
+BEGIN
+    CREATE INDEX [IX_TemplateCompositions_RowVersion] ON [TemplateCompositions] ([RowVersion]);
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261110032839_AddConfigGraphRowVersions'
+)
+BEGIN
+    CREATE INDEX [IX_TemplateAttributes_RowVersion] ON [TemplateAttributes] ([RowVersion]);
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261110032839_AddConfigGraphRowVersions'
+)
+BEGIN
+    CREATE INDEX [IX_TemplateAlarms_RowVersion] ON [TemplateAlarms] ([RowVersion]);
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261110032839_AddConfigGraphRowVersions'
+)
+BEGIN
+    CREATE INDEX [IX_Sites_RowVersion] ON [Sites] ([RowVersion]);
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261110032839_AddConfigGraphRowVersions'
+)
+BEGIN
+    CREATE INDEX [IX_SharedScripts_RowVersion] ON [SharedScripts] ([RowVersion]);
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261110032839_AddConfigGraphRowVersions'
+)
+BEGIN
+    CREATE INDEX [IX_Instances_RowVersion] ON [Instances] ([RowVersion]);
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261110032839_AddConfigGraphRowVersions'
+)
+BEGIN
+    CREATE INDEX [IX_InstanceNativeAlarmSourceOverrides_RowVersion] ON [InstanceNativeAlarmSourceOverrides] ([RowVersion]);
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261110032839_AddConfigGraphRowVersions'
+)
+BEGIN
+    CREATE INDEX [IX_InstanceConnectionBindings_RowVersion] ON [InstanceConnectionBindings] ([RowVersion]);
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261110032839_AddConfigGraphRowVersions'
+)
+BEGIN
+    CREATE INDEX [IX_InstanceAttributeOverrides_RowVersion] ON [InstanceAttributeOverrides] ([RowVersion]);
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261110032839_AddConfigGraphRowVersions'
+)
+BEGIN
+    CREATE INDEX [IX_InstanceAlarmOverrides_RowVersion] ON [InstanceAlarmOverrides] ([RowVersion]);
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261110032839_AddConfigGraphRowVersions'
+)
+BEGIN
+    CREATE INDEX [IX_ExternalSystemMethods_RowVersion] ON [ExternalSystemMethods] ([RowVersion]);
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261110032839_AddConfigGraphRowVersions'
+)
+BEGIN
+    CREATE INDEX [IX_ExternalSystemDefinitions_RowVersion] ON [ExternalSystemDefinitions] ([RowVersion]);
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261110032839_AddConfigGraphRowVersions'
+)
+BEGIN
+    CREATE INDEX [IX_DeployedConfigSnapshots_RowVersion] ON [DeployedConfigSnapshots] ([RowVersion]);
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261110032839_AddConfigGraphRowVersions'
+)
+BEGIN
+    CREATE INDEX [IX_DataConnections_RowVersion] ON [DataConnections] ([RowVersion]);
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261110032839_AddConfigGraphRowVersions'
+)
+BEGIN
+    CREATE INDEX [IX_DatabaseConnectionDefinitions_RowVersion] ON [DatabaseConnectionDefinitions] ([RowVersion]);
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261110032839_AddConfigGraphRowVersions'
+)
+BEGIN
+    INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion])
+    VALUES (N'20261110032839_AddConfigGraphRowVersions', N'10.0.13');
+END;
+
+COMMIT;
+GO
+
+BEGIN TRANSACTION;
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261110033732_FlipAuditLogClusteredKey'
+)
+BEGIN
+
+    IF 
+    (SELECT c.name FROM sys.index_columns ic
+       JOIN sys.columns c ON c.object_id = ic.object_id AND c.column_id = ic.column_id
+      WHERE ic.object_id = OBJECT_ID('dbo.AuditLog') AND ic.index_id = 1 AND ic.key_ordinal = 1) = N'EventId'
+    BEGIN
+
+    DECLARE @dropSql nvarchar(max) = N'';
+    SELECT @dropSql = @dropSql + N'DROP INDEX ' + QUOTENAME(i.name) + N' ON dbo.AuditLog;' + CHAR(10)
+    FROM sys.indexes i
+    WHERE i.object_id = OBJECT_ID('dbo.AuditLog') AND i.type = 2 AND i.name IN (
+        N'IX_AuditLog_OccurredAtUtc', N'IX_AuditLog_Site_Occurred', N'IX_AuditLog_CorrelationId',
+        N'IX_AuditLog_Channel_Status_Occurred', N'IX_AuditLog_Target_Occurred', N'IX_AuditLog_Execution',
+        N'IX_AuditLog_ParentExecution', N'IX_AuditLog_Node_Occurred', N'IX_AuditLog_Kind_Occurred',
+        N'IX_AuditLog_Actor_Occurred', N'IX_AuditLog_Unchained', N'IX_AuditLog_ChainSeq',
+        N'UX_AuditLog_EventId_Occurred');
+    EXEC sp_executesql @dropSql;
+        ALTER TABLE dbo.AuditLog DROP CONSTRAINT PK_AuditLog;
+        ALTER TABLE dbo.AuditLog ADD CONSTRAINT PK_AuditLog PRIMARY KEY CLUSTERED (OccurredAtUtc, EventId)
+            ON ps_AuditLog_Month(OccurredAtUtc);
+        CREATE UNIQUE NONCLUSTERED INDEX UX_AuditLog_EventId_Occurred ON dbo.AuditLog (EventId, OccurredAtUtc)
+            ON ps_AuditLog_Month(OccurredAtUtc);
+
+    CREATE NONCLUSTERED INDEX IX_AuditLog_OccurredAtUtc ON dbo.AuditLog (OccurredAtUtc DESC) INCLUDE (Status) ON ps_AuditLog_Month(OccurredAtUtc);
+    CREATE NONCLUSTERED INDEX IX_AuditLog_Site_Occurred ON dbo.AuditLog (SourceSiteId ASC, OccurredAtUtc DESC) ON ps_AuditLog_Month(OccurredAtUtc);
+    CREATE NONCLUSTERED INDEX IX_AuditLog_CorrelationId ON dbo.AuditLog (CorrelationId) WHERE CorrelationId IS NOT NULL ON ps_AuditLog_Month(OccurredAtUtc);
+    CREATE NONCLUSTERED INDEX IX_AuditLog_Channel_Status_Occurred ON dbo.AuditLog (Category ASC, Status ASC, OccurredAtUtc DESC) ON ps_AuditLog_Month(OccurredAtUtc);
+    CREATE NONCLUSTERED INDEX IX_AuditLog_Target_Occurred ON dbo.AuditLog (Target ASC, OccurredAtUtc DESC) WHERE Target IS NOT NULL ON ps_AuditLog_Month(OccurredAtUtc);
+    CREATE NONCLUSTERED INDEX IX_AuditLog_Execution ON dbo.AuditLog (ExecutionId) ON ps_AuditLog_Month(OccurredAtUtc);
+    CREATE NONCLUSTERED INDEX IX_AuditLog_ParentExecution ON dbo.AuditLog (ParentExecutionId) ON ps_AuditLog_Month(OccurredAtUtc);
+    CREATE NONCLUSTERED INDEX IX_AuditLog_Node_Occurred ON dbo.AuditLog (SourceNode, OccurredAtUtc) ON ps_AuditLog_Month(OccurredAtUtc);
+    CREATE NONCLUSTERED INDEX IX_AuditLog_Kind_Occurred ON dbo.AuditLog (Kind ASC, OccurredAtUtc DESC) ON ps_AuditLog_Month(OccurredAtUtc);
+    CREATE NONCLUSTERED INDEX IX_AuditLog_Actor_Occurred ON dbo.AuditLog (Actor ASC, OccurredAtUtc DESC) ON ps_AuditLog_Month(OccurredAtUtc);
+    CREATE NONCLUSTERED INDEX IX_AuditLog_Unchained ON dbo.AuditLog (OccurredAtUtc, EventId) WHERE ChainSeq IS NULL ON ps_AuditLog_Month(OccurredAtUtc);
+    CREATE NONCLUSTERED INDEX IX_AuditLog_ChainSeq ON dbo.AuditLog (ChainSeq) WHERE ChainSeq IS NOT NULL ON ps_AuditLog_Month(OccurredAtUtc);
+    END
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261110033732_FlipAuditLogClusteredKey'
+)
+BEGIN
+    INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion])
+    VALUES (N'20261110033732_FlipAuditLogClusteredKey', N'10.0.13');
+END;
+
+COMMIT;
+GO
+
+BEGIN TRANSACTION;
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261110120000_AuditMaintenanceGrants'
+)
+BEGIN
+
+    IF DATABASE_PRINCIPAL_ID('scadabridge_audit_purger') IS NULL
+        EXEC sp_executesql N'CREATE ROLE scadabridge_audit_purger';
+    GRANT ALTER ANY DATASPACE TO scadabridge_audit_purger;
+    GRANT UPDATE (SourceNode) ON dbo.AuditLog TO scadabridge_audit_purger;
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20261110120000_AuditMaintenanceGrants'
+)
+BEGIN
+    INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion])
+    VALUES (N'20261110120000_AuditMaintenanceGrants', N'10.0.13');
+END;
+
+COMMIT;
+GO
+

docs/deployment/deploy-prep/2026-11-13-qa-vq03-vq04/preflight.sql

@@ -0,0 +1,56 @@
+-- Read-only preflight for the QA vq03/vq04 window (ScadaBridge#107 item 4). SELECTs only.
+-- Run against the QA config DB:
+--   SQLCMDPASSWORD=... sqlcmd -S wonder-sql-vq03.zmr.zimmer.com,1433 -U <login> -C -d ScadaBridge -W -s '|' -i preflight.sql
+-- Re-run it right before the window and compare with README.md section 3.
+SET NOCOUNT ON;
+
+-- 1. Migration level: expect 58 rows, last = 20261022133907_ReviewRemediationIndexes.
+SELECT COUNT(*) AS migrations FROM dbo.__EFMigrationsHistory;
+SELECT TOP 3 MigrationId, ProductVersion FROM dbo.__EFMigrationsHistory ORDER BY MigrationId DESC;
+
+-- 2. Recovery model and log reuse.
+SELECT name, recovery_model_desc, log_reuse_wait_desc, compatibility_level
+FROM sys.databases WHERE name = DB_NAME();
+
+-- 3. AuditLog size (sizes the FlipAuditLogClusteredKey rebuild; the log needs about 2x reserved).
+EXEC sp_spaceused 'dbo.AuditLog';
+SELECT MIN(OccurredAtUtc) AS oldest, MAX(OccurredAtUtc) AS newest FROM dbo.AuditLog;
+
+-- 4. Clustered key shape: expect key_ordinal 1 = EventId before the flip, OccurredAtUtc after.
+SELECT ic.key_ordinal, c.name
+FROM sys.index_columns ic
+JOIN sys.columns c ON c.object_id = ic.object_id AND c.column_id = ic.column_id
+WHERE ic.object_id = OBJECT_ID('dbo.AuditLog') AND ic.index_id = 1
+ORDER BY ic.key_ordinal;
+
+-- 5. Log space, files and volumes.
+SELECT total_log_size_in_bytes / 1048576 AS log_mb,
+       used_log_space_in_bytes / 1048576 AS log_used_mb,
+       CAST(used_log_space_in_percent AS decimal(5,1)) AS log_used_pct
+FROM sys.dm_db_log_space_usage;
+SELECT name, type_desc, CAST(size AS bigint) * 8 / 1024 AS size_mb,
+       CAST(growth AS bigint) * 8 / 1024 AS growth_mb, is_percent_growth, physical_name
+FROM sys.database_files;
+SELECT DISTINCT vs.volume_mount_point, vs.available_bytes / 1048576 AS free_mb
+FROM sys.database_files f CROSS APPLY sys.dm_os_volume_stats(DB_ID(), f.file_id) vs;
+
+-- 6. SourceNode backfill state (run `audit backfill-source-node` before the hash chain is enabled).
+SELECT Kind, COUNT(*) AS n, SUM(CASE WHEN SourceNode IS NULL THEN 1 ELSE 0 END) AS null_sourcenode,
+       MAX(CASE WHEN SourceNode IS NULL THEN OccurredAtUtc END) AS newest_null
+FROM dbo.AuditLog GROUP BY Kind ORDER BY null_sourcenode DESC;
+
+-- 7. Objects the window creates: expect none of these before, all after.
+SELECT OBJECT_NAME(object_id) AS tbl, name AS col FROM sys.columns
+WHERE name IN ('ChainSeq', 'RowHash', 'RowVersion', 'IsMesReportable') ORDER BY 1, 2;
+SELECT name FROM sys.tables WHERE name IN ('AuditLogChainHead', 'AuditLogChainTombstone');
+SELECT name FROM sys.database_principals WHERE name LIKE 'scadabridge[_]audit[_]%';
+
+-- 8. Partition function headroom (the flip rebuilds onto ps_AuditLog_Month).
+SELECT pf.name, COUNT(prv.boundary_id) AS boundaries, MAX(CAST(prv.value AS datetime2)) AS last_boundary
+FROM sys.partition_functions pf
+LEFT JOIN sys.partition_range_values prv ON prv.function_id = pf.function_id
+GROUP BY pf.name;
+
+-- 9. Recent backups.
+SELECT TOP 5 type, backup_finish_date, is_copy_only
+FROM msdb.dbo.backupset WHERE database_name = DB_NAME() ORDER BY backup_finish_date DESC;