docs(deploy): refresh QA vq03/vq04 pack to main 8e32f353 - no new migration, owner's go scope (#107, #116)

- Target 3a59c3f6 -> 8e32f353 (adds #138 ServiceLevel failover, #139 chain
  cutoff, #140 graceful stop, LocalDb 0.3.8). No migration in the range;
  the idempotent 58->64 script regenerated at 8e32f353 is byte-identical
  (SHA-256 488da93d...c34ecc0).
- Owner's go 2026-11-13 ("just deploy"): #116 leader-auto-down flip and the
  exit-75 service recovery setting in; no #116 drills; hash chain and head
  export off; #52 stays excluded.
- #131 re-confirmed DEV-only against the QA config DB.
- Preflight re-run 07:05 EST: 58 migrations, 3,441 AuditLog rows, EventId
  key, log 63 MB used of 200.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi

docs/deployment/deploy-prep/2026-11-13-qa-vq03-vq04/README.md

@@ -1,15 +1,19 @@
-# Deploy prep: QA vq03/vq04, coordinated central-pair migration window (ScadaBridge#107 item 4) + #116 leader-auto-down
+# Deploy prep: QA vq03/vq04 (target `8e32f353`), coordinated central-pair migration window (ScadaBridge#107 item 4) + #116 leader-auto-down
 
-**Prepared:** 2026-11-13. **Not executed.** Running it is a deploy, and it waits for the owner's separate go (#107, 2026-11-13).
+**Prepared:** 2026-11-13. **Refreshed:** 2026-11-13 ~07:10 EST for the new tip. **Not executed.**
 
-**Target build:** `main` @ **`3a59c3f6`**. That is the tip this pack was analysed at. If `main` moves before the window, re-run §1.4 and §2 over `3a59c3f6..<new tip>`: look for new migrations under `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/` and option changes in `*Options.cs` / `*OptionsValidator.cs`. If anything is new, regenerate the script in §2.2.
+**Owner's go (2026-11-13, "just deploy"):** the window itself is approved. Scope:
+- **in:** the upgrade, the bundled #116 `leader-auto-down` config flip (§5.4), and the exit-code-75 service recovery setting (§5.3, `sc.exe failure` + `failureflag 1` on all four services);
+- **out:** **no #116 drills** (neither the central kill drill nor the site-pair kill drill; G14 is not run), **hash chain OFF** on QA, **head export OFF** (both stay unset; §7 is a later, separate step), **#52 not bundled** (§10).
+
+**Target build:** `main` @ **`8e32f353`**. The first version of this pack (`16209831`) targeted `3a59c3f6`; this refresh re-ran §1.4, §2 and §3 over `3a59c3f6..8e32f353` (§1.5). **No new migration in that range**: `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/` is untouched, and a fresh `dotnet ef migrations script` at `8e32f353` is byte-identical to the script in this folder (same SHA-256). If `main` moves again before the window, repeat that check over `8e32f353..<new tip>`.
 
 **Sources:**
-- The read-only SQL preflight on `wonder-sql-vq03` (2026-11-13, §3).
+- The read-only SQL preflight on `wonder-sql-vq03` (2026-11-13 ~02:45 EST, re-run ~07:05 EST for the refresh, §3).
 - SMB reads of the live `app\` folders and appsettings on both boxes, and HTTP health probes. Nothing was written on any box.
 - Deployment records `2026-10-22-wonder-app-vq03-vq04-3808b49f.md` and `2026-10-22-wonder-app-vd03-vq03-vq04-0e540291.md`.
 - The DEV vd03 `424db155` upgrade of 2026-11-11 (migrations 58→64, flip of 2,394 rows / 8 MB in about 1.5 s).
-- Issues #107, #116, #126, #131 and #52.
+- Issues #107, #116, #126, #131, #52, #138, #139 and #140.
 
 ## Files
 
@@ -17,7 +21,7 @@
 |---|---|
 | `README.md` | This pack. |
 | `preflight.sql` | Read-only preflight (SELECTs and `sp_spaceused` only). Re-run it right before the window. |
-| `migrate-58-to-64.idempotent.sql` | `dotnet ef migrations script 20261022133907_ReviewRemediationIndexes --idempotent`, generated at `3a59c3f6` (EF tools 10.0.10, runtime 10.0.13). It covers six migrations, 59→64. SHA-256 `488da93db0f464ee17687a32a1dabbbe810553c109db5b65d5b3ecbd5c34ecc0`. It is unedited and starts with a UTF-8 BOM at byte 0, which sqlcmd accepts; #126's problem was a BOM *after* a prepended header. |
+| `migrate-58-to-64.idempotent.sql` | `dotnet ef migrations script 20261022133907_ReviewRemediationIndexes --idempotent`, generated at `3a59c3f6` and **re-generated at `8e32f353` with an identical result** (EF tools 10.0.10, runtime 10.0.13). It covers six migrations, 59→64. SHA-256 `488da93db0f464ee17687a32a1dabbbe810553c109db5b65d5b3ecbd5c34ecc0`. It is unedited and starts with a UTF-8 BOM at byte 0, which sqlcmd accepts; #126's problem was a BOM *after* a prepended header. |
 
 ---
 
@@ -36,9 +40,9 @@ As a result, these items from the brief are **already on QA** and are not part o
 - `ReviewRemediationIndexes`;
 - the shared `zbauth`/`zbsecrets` SQL stores on the central pair.
 
-The effective delta is **`0e540291..3a59c3f6`: 128 commits and 6 migrations**.
+The effective delta is **`0e540291..8e32f353`: 134 commits and 6 migrations** (128 of them up to `3a59c3f6`, the rest in §1.5).
 
-## 1. What changes: `0e540291..3a59c3f6`
+## 1. What changes: `0e540291..8e32f353`
 
 ### 1.1 Schema (central DB): 6 migrations
 
@@ -53,12 +57,15 @@ See §2.
 | AuditLog ingest (#124) | Set-based ingest uses FORCESEEK, so concurrent writers no longer deadlock (1205). | Positive. |
 | Cluster downing (#106, #132) | Opt-in `leader-auto-down` with shut-out restart. **The Host now exits 75 after any unplanned ActorSystem termination** (it used to exit 0), under any strategy. | Windows SCM recovery has to restart on a non-zero exit. See §5.3, which checks it in the window. |
 | Host stop (#76) | A routine service stop logs as Information, not `[FTL] host terminated unexpectedly`. | Drop the old "known FTL on stop" exclusion from log review. |
+| Host stop (#140, new in the refresh) | A graceful stop now leaves the Akka cluster **before** Kestrel drains (`AkkaHostedService` is an `IHostedLifecycleService`; CoordinatedShutdown runs in `StoppingAsync`), and every open gRPC stream (the peer's LocalDb sync stream above all) ends on `ApplicationStopping`; new streams are refused `Unavailable` while stopping. Exit codes unchanged: planned stop 0, self-down/shut-out 75. | **Only the new binaries' stops benefit.** W1–W2 stop the *old* `0e540291` binaries, so the window's own stops still take the old time (a Site stop can wait out the 30 s drain on the sync stream). Stops after the window (and the rollback in §8, for the new build) are faster. New log lines on stop: `Host stopping: leaving the Akka cluster before the web server drains…` and `Akka CoordinatedShutdown completed in … ms`. |
 | Alarms (#129) | `Severity` is banded with higher = more urgent; the raw value is on `Priority`. The MES 900–999 band keys off `Priority`. | The Alarm Summary Sev column and its sort flip visibly. **#131: QA has nothing to change**: `CvdReactor.AlarmStatus` (39) is a stub, and there is no TestMesMachine. |
 | MES-reportable (#59, #60, #102, #104) | Per-binding `IsMesReportable` on native alarm sources, plus UI badges and CSV columns. | Additive, delivered by migration `AddNativeAlarmSourceMesReportable`. |
 | Security (#46, Auth 0.5.0) | Active sessions re-query LDAP group membership: `ScadaBridge:Security:LdapGroupRefresh`, `Enabled` true, every 15 min, 60 s cache. | QA's centrals bind to their box-local GLAuth (`127.0.0.1:3893`) with a service account, which this supports. A user removed from a group loses the role within about 15 min. |
 | Central HTTP (#108) | An `https://` entry in `ASPNETCORE_URLS` now needs `ScadaBridge:Node:Tls` (a certificate) or the node **refuses to boot**. | QA serves plain `http://…:8085`. **The prep step confirms there is no `https://` in either Central's service env** (§5.1). |
 | Akka.Remote TLS (#64) | `ScadaBridge:Cluster:RemotingTls`, default off. | None. |
 | DCL OPC UA (#117, #121, #112, #58; SDK 1.5.378.106 → **2.0.1**, #125) | Good subcodes now map to Good. Secure sessions present a real app cert (`OpcUaGlobal:*`). `BadResourceUnavailable` is treated as transient. Scalar writes are coerced to the node type. | **QA has no OPC UA connection.** Its only data connection is `MxGateway`, so there is no exposure. |
+| DCL OPC UA failover (#138, new in the refresh) | ServiceLevel-aware failover for an OPC UA connection with a primary **and** a backup endpoint. New per-connection setting `ServiceLevelFailover {Enabled, DegradedWindowSeconds, MinDwellSeconds}` on the **primary** endpoint, **default on** (`true`, 10 s, 60 s; a stored connection without it gets the defaults). It lives inside the encrypted protocol JSON, so **no migration**. Global knobs `ScadaBridge:DataConnection:ServiceLevelPollInterval` (2 s) and `ServiceLevelProbeTimeout` (15 s). | **None on QA** (no OPC UA connection, re-checked §3). Where it does apply, today's OtOpcUa pair (250/240, both in the Healthy band ≥ 200) never starts a degraded streak, so it is safe with 250/240. It matters once OtOpcUa Warm redundancy (lmxopcua#717) is switched on. |
+| Audit hash-chain cutoff (#139, new in the refresh) | First enablement anchors the chain at the DB's `SYSUTCDATETIME()` and chains only rows from then on; `AuditLog:HashChain:ChainBacklog` (default false) restores chain-everything. **No migration.** | None in this window: the chain stays off. It is what makes §7 safe later: this build is the one §7 step 0 asks for. |
 | DCL MxGateway (#128) | The resume cursor ignores `worker_sequence` 0 sentinels. | Positive for QA's single connection. |
 | Site runtime (#135, #136, #36) | Post-boot settle sweep: replicated instances start within seconds. Site commands fail over within the call. The legacy `WriteBatchAndWaitAsync` waits for its own reset echo. | Faster site start after the swap. |
 | Inbound API (#103, #75, #80) | Recompile and invalidate reach the peer central. Compile errors are visible through `api-method validate/recompile`. | Positive on a pair. |
@@ -75,7 +82,7 @@ See §2.
 | Package | QA today (`0e540291`) | Tip | Note |
 |---|---|---|---|
 | ZB.MOM.WW.Auth.* | 0.3.0 | **0.5.0** | `BindMode` is still `ServiceAccount`. Adds the passwordless group lookup (#46). |
-| ZB.MOM.WW.LocalDb.* | 0.3.4 | **0.3.7** | Wire-compatible, but **the pair stops and starts together**. The library now enforces `LocalDb:Replication:ApiKey` itself, which must be byte-identical on both site nodes (it already is: QA's own fail-closed interceptor has been connected since 2026-09-10). The tombstone ledger grows while a peer is down (#16). `MaxBatchSize: 16` in both Site files is a retired pin; it is harmless. |
+| ZB.MOM.WW.LocalDb.* | 0.3.4 | **0.3.8** | 0.3.8 (scadaproj#57, `8e32f353`) adds only the graceful-stop fix: open sync streams end on `ApplicationStopping` (lmxopcua#906). Wire-compatible, but **the pair stops and starts together**. The library now enforces `LocalDb:Replication:ApiKey` itself, which must be byte-identical on both site nodes (it already is: QA's own fail-closed interceptor has been connected since 2026-09-10). The tombstone ledger grows while a peer is down (#16). `MaxBatchSize: 16` in both Site files is a retired pin; it is harmless. |
 | ZB.MOM.WW.Telemetry(.Serilog) | 0.1.0 | **0.3.1** | See above. |
 | ZB.MOM.WW.Theme | 0.4.1 | **0.5.3** | ScrollNav rail (#61). |
 | ZB.MOM.WW.Secrets | 0.6.2 | 0.6.2 | Unchanged. |
@@ -96,11 +103,29 @@ See §2.
 - `OpcUaGlobal:{ApplicationUri,ApplicationCertificateSubject,OwnCertificateStorePath,AutoCreateApplicationCertificate,ApplicationCertificateLifetimeMonths}`
 - `AuditLog:{MaintenanceConnectionString,HashChain:*,HashChain:HeadExport:*,Archive:*}`
 - `LocalDb:Replication:{MaxSnapshotDeferredRows,TombstoneHardCap}`
+- new in the refresh: `ScadaBridge:DataConnection:{ServiceLevelPollInterval,ServiceLevelProbeTimeout}` (#138; validated positive) and `AuditLog:HashChain:ChainBacklog` (#139; default false). The per-connection `ServiceLevelFailover` object is data, not appsettings.
 
-**Changes this window does make:** only the #116 strategy flip (§5.4).
+**Changes this window does make:** the #116 strategy flip (§5.4) and the exit-75 service recovery setting (§5.3). Both are approved in the owner's go.
 
 **Drift found, not fixed here:** the vq03 and vq04 `appsettings.Central.json` differ in `JwtExpiryMinutes` (25 vs 15) and `IdleTimeoutMinutes` (60 vs 30). The CLI bearer token is signed with the shared key, so the pair still interoperates, but session lifetime depends on which central answers. Align them (pick one pair of values) as a separate change, or here if the owner says so.
 
+### 1.5 Refresh delta: `3a59c3f6..8e32f353`
+
+Six commits (the two docs commits are this pack's own):
+
+| Commit | Change | Migration | QA impact |
+|---|---|---|---|
+| `16209831`, `3badd1f9` | This pack (first version, then the §7 #139 update) | — | — |
+| `712c5117` | #138 ServiceLevel-aware OPC UA failover (§1.2) | No | None: no OPC UA connection. New per-connection `ServiceLevelFailover`, **default on** (10 s window, 60 s dwell); safe with OtOpcUa's 250/240 |
+| `52da02a7` | #139 hash-chain cutoff (§1.2, §7) | No | None in the window (chain off); makes the later §7 enablement backlog-free |
+| `1a6108c4` | #140 graceful stop: leave the cluster before Kestrel drains; end open gRPC streams on stop (§1.2) | No | Faster stops of the *new* binary only |
+| `8e32f353` | LocalDb **0.3.8** pin (scadaproj#57): the library itself ends open sync streams on `ApplicationStopping` and refuses new ones `Unavailable` while stopping (lmxopcua#906). The #140 host interceptor stays as the backstop | No | Same as #140; still wire-compatible with 0.3.4/0.3.7, pair still stops and starts together |
+
+Checks over this range:
+- **Migrations:** none. `git diff 3a59c3f6 8e32f353 -- src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/` is empty (the ConfigurationDatabase changes are in `Maintenance/` and `Repositories/`, for #139). The regenerated idempotent script has the same SHA-256 as the one in this folder.
+- **Options:** only the three keys listed in §1.4 ("new in the refresh"), all with safe defaults. No shipped appsettings changed. **Nothing has to change on QA for this build to boot.**
+- **Rollback (§8):** unchanged. None of the new commits changes the schema or the protocol-config encryption, so `0e540291` + the restored DB is still the rollback.
+
 ## 2. Migrations 58 → 64
 
 ### 2.1 Ordered list (EF timestamp order = apply order)
@@ -111,7 +136,7 @@ See §2.
 | 60 | `20261109203753_PrefixSearchIndexes` | #50 | Prefix-search indexes for Deployments and config-audit | Tiny tables | Yes |
 | 61 | `20261109211317_AddAuditLogHashChain` | #41 | `ChainSeq`/`RowHash` on `AuditLog`. `AuditLogChainHead` and `AuditLogChainTombstone`. Two filtered indexes, wrapped in `EXEC(N'…')` (the #126 fix). Roles `scadabridge_audit_chainer` (new) and `_purger`/`_writer` (exist), grants, and a DENY to the writer on the chain tables | 3,441 rows; seconds | **No**: old binaries can't partition-purge once the columns exist |
 | 62 | `20261110032839_AddConfigGraphRowVersions` | #62 | Non-concurrency `rowversion` + index on 18 configuration tables. SQL Server fills each table as it adds the column | All config tables are under 2,300 rows; seconds | Yes (ignored) |
-| 63 | `20261110033732_FlipAuditLogClusteredKey` | #111 | **Offline whole-table rebuild**: drop the 10 NCIs + PK, re-add the PK clustered on `(OccurredAtUtc, EventId)` on `ps_AuditLog_Month`, add `UX_AuditLog_EventId_Occurred`, recreate 12 NCIs. Guarded by "clustered key ordinal 1 = `EventId`" | **18.3 MB reserved / 3,441 rows → about 2–4 s** (DEV: 8 MB / 2,394 rows in about 1.5 s). Log about 2× = about 40 MB, against 142 MB free in a 200 MB log (64 MB autogrowth, 240 GB free on G:) | **No**: older binaries can't partition-switch a flipped table |
+| 63 | `20261110033732_FlipAuditLogClusteredKey` | #111 | **Offline whole-table rebuild**: drop the 10 NCIs + PK, re-add the PK clustered on `(OccurredAtUtc, EventId)` on `ps_AuditLog_Month`, add `UX_AuditLog_EventId_Occurred`, recreate 12 NCIs. Guarded by "clustered key ordinal 1 = `EventId`" | **18.3 MB reserved / 3,441 rows → about 2–4 s** (DEV: 8 MB / 2,394 rows in about 1.5 s). Log about 2× = about 40 MB, against 136 MB free in a 200 MB log at the 07:05 re-run (64 MB autogrowth, 240 GB free on G:) | **No**: older binaries can't partition-switch a flipped table |
 | 64 | `20261110120000_AuditMaintenanceGrants` | #113 | `scadabridge_audit_purger`: `ALTER ANY DATASPACE` + `UPDATE (SourceNode)` on `dbo.AuditLog` | Instant | Yes |
 
 **Owner's §item-4 list vs EF order.** `AddNativeAlarmSourceMesReportable` sorts **first**, by timestamp. The EF script applies migrations in timestamp order, which is also the order DEV vd03 used. The relative order of the five audit/config migrations is the one the owner listed.
@@ -142,45 +167,48 @@ See §2.
   - 3 `scadabridge_audit_*` roles.
   - Re-run `preflight.sql` sections 1, 4 and 7.
 
-## 3. Read-only SQL preflight: results (2026-11-13 ~02:45 EST)
+## 3. Read-only SQL preflight: results (2026-11-13 ~02:45 EST, re-run ~07:05 EST)
+
+The re-run for the refresh changed nothing that matters: still 58 migrations, the same 3,441 AuditLog rows and the same `EventId`-first key. Only the log usage moved (57 → 63 MB). The table shows the re-run values.
 
 Server **`wonder-sql-vq03.zmr.zimmer.com,1433`** (`WONDER-SQL-VQ03`, SQL Server 2022 RTM-CU24 16.0.4245.2, **Enterprise**), DB `ScadaBridge`. Login `wwadmin`, password via `SQLCMDPASSWORD` only, `-C`. SELECTs only. The script is in `preflight.sql`.
 
 | Check | Result |
 |---|---|
-| `__EFMigrationsHistory` | **58**; last `20261022133907_ReviewRemediationIndexes` (ProductVersion 10.0.7) |
+| `__EFMigrationsHistory` | **58**; last `20261022133907_ReviewRemediationIndexes` (ProductVersion 10.0.7). Unchanged at 07:05 |
 | Recovery model | **FULL**; `log_reuse_wait` NOTHING; compatibility 160 |
-| Log backups | Regular: full at 22:00 daily, log at about 12:00 / 24:00. A `COPY_ONLY` backup does not disturb the chain |
+| Log backups | Regular: full at 22:00 daily (last 2026-11-12 22:00), log at about 12:00 / 24:00 (last 2026-11-12 23:57). A `COPY_ONLY` backup does not disturb the chain |
-| `dbo.AuditLog` | **3,441 rows**; reserved **18.3 MB** (data 10.7 MB, index 3.2 MB, unused 4.4 MB); 2026-05-29 → 2026-11-12 |
+| `dbo.AuditLog` | **3,441 rows**; reserved **18.3 MB** (18,768 KB: data 10.7 MB, index 3.2 MB, unused 4.4 MB); 2026-05-29 → 2026-11-12 22:13 UTC. Unchanged at 07:05 |
 | Clustered key now | `PK_AuditLog` = (`EventId`, `OccurredAtUtc`) on `ps_AuditLog_Month`, so the flip has not been applied yet |
 | `pf_AuditLog_Month` | 24 boundaries, the last 2027-12-01 |
-| Log file | 200 MB, **57 MB used (28.6 %)**, 64 MB autogrowth, unlimited; G: 245,855 MB free |
+| Log file | 200 MB, **63 MB used (31.6 %)** at 07:05 (57 MB / 28.6 % at 02:45), 64 MB autogrowth, unlimited; G: 245,855 MB free. The flip's ~40 MB still fits in the free 136 MB |
-| Data file | 904 MB, 64 MB autogrowth; F: 249,267 MB free. The largest table is `KpiSample` (4.9 M rows, 804 MB), which the window does not touch |
+| Data file | 904 MB, 64 MB autogrowth; F: 249,323 MB free. The largest table is `KpiSample` (4.9 M rows, 804 MB), which the window does not touch |
-| SourceNode backfill | **Not run.** **843 rows** have `SourceNode IS NULL`: InboundRequest 793 (newest 2026-07-07) and InboundAuthFailure 50 (newest 2026-06-27), all from before M5.6. Every row since 2026-07-07 is stamped (central-a 1,742, central-b 777, node-a 59, node-b 20) |
+| SourceNode backfill | **Not run.** **843 rows** have `SourceNode IS NULL` (unchanged at 07:05): InboundRequest 793 (newest 2026-07-07) and InboundAuthFailure 50 (newest 2026-06-27), all from before M5.6. Irrelevant to this window (chain off); with the #139 cutoff it no longer gates §7 either. Every row since 2026-07-07 is stamped (central-a 1,742, central-b 777, node-a 59, node-b 20) |
-| Hash-chain objects | None yet. Roles `scadabridge_audit_writer` and `_purger` exist; `_chainer` does not. `DeploymentRecords.RowVersion` already exists, and `AddConfigGraphRowVersions` handles that table (covered by the from-57 test) |
+| Hash-chain objects | None yet (re-checked 07:05). Roles `scadabridge_audit_writer` and `_purger` exist; `_chainer` does not. Columns present: only `DeploymentRecords.RowVersion` and `TemplateAlarms.IsMesReportable` (both pre-existing). `DeploymentRecords.RowVersion` already exists, and `AddConfigGraphRowVersions` handles that table (covered by the from-57 test) |
-| Topology | 1 data connection (`MxGateway`, protocol MxGateway, site 2), 1 site (`site-a`, artifacts `sha256:d79ba1a3…`, deployed 2026-10-27), 10 instances |
+| Topology | 1 data connection (`MxGateway`, protocol MxGateway, site 2), **no OPC UA connection** (so #138 has nothing to act on), 1 site (`site-a`, artifacts `sha256:d79ba1a3…`, deployed 2026-10-27), 10 instances. Unchanged at 07:05 |
-| Live health (HTTP, 02:45 EST) | All four `/health/ready` return 200. **vq03 is active** for central and site; vq04 answers 503. Central: memberCount 2, unreachable 0, 5/5 singletons |
+| #131 / #52 scripts (re-checked 07:05) | No `TestMesMachine` template. `CvdReactor.AlarmStatus` (script 39) is a 110-char stub that reads no `.Severity`; no shared scripts; none of the 8 API methods reads `Alarms.`/`.Severity`. So **#131 needs no QA script change**, confirming the 2026-11-12 finding. `CvdReactor.ProcessRecipeDownload` (script 4) still has `Children["DelmiaRecv"]` (#52 still open on QA, not bundled) |
+| Live health (HTTP, 02:45 and again 07:05 EST) | All four `/health/ready` return 200. **vq03 is active** for central and site; vq04 answers 503 on `/health/active`. Central: memberCount 2, unreachable 0, leader vq03 (`10.220.158.2`), 5/5 singletons |
 
 **Window estimate for the flip:** about 2–4 s on 18 MB, scaled from DEV's 1.5 s for 8 MB. All six migrations take well under 30 s.
 
 ## 4. #116 leader-auto-down: bundled
 
 **Recommendation: bundle it.** It fits the same window with no extra restart:
-- The `3a59c3f6` binary already contains #106 and #132 (shut-out restart, exit 75).
+- The `8e32f353` binary already contains #106 and #132 (shut-out restart, exit 75).
 - The change is an **appsettings flip only**.
 - #116's one rollout rule is "each pair stops and starts together; never roll across a downing-provider change", and this window already stops all four services and starts each pair together.
 - The owner approved the QA rollout (rig and env2 gates passed, and env2 is soaking).
 - If it went in a separate window, QA would take a second full outage.
 
-The order inside the window is: binaries → migrations → config (including the #116 flip) → start. §5.4 has the edit, §6 G9 the drill, and §8 the rollback, which is to flip the value back.
+The order inside the window is: binaries → migrations → config (including the #116 flip) → start. §5.4 has the edit and §8 the rollback, which is to flip the value back. **The owner's go excludes the #116 drills** (central and site pair): G14 is not run, and #116 records "deployed on QA, not drilled". The config, the boot-log check (G6) and the single-active check (G5) still apply.
 
 ## 5. Procedure
 
-All times are local (EST). Roles: one operator on the Mac; WinRM for the on-box steps only, under the deploy-only scope; SMB for files; direct SQL for the DB. Layout on each box: `E:\ApiInstall\ScadaBridge\{app,cli,central,site,…}`. **`app\` is shared by the `ScadaBridge-Central` and `ScadaBridge-Site` services on that box**, so a swap needs both services on that box stopped. The sites stop anyway, because LocalDb 0.3.7 pairs stop and start together.
+All times are local (EST). Roles: one operator on the Mac; WinRM for the on-box steps only, under the deploy-only scope; SMB for files; direct SQL for the DB. Layout on each box: `E:\ApiInstall\ScadaBridge\{app,cli,central,site,…}`. **`app\` is shared by the `ScadaBridge-Central` and `ScadaBridge-Site` services on that box**, so a swap needs both services on that box stopped. The sites stop anyway, because LocalDb pairs stop and start together.
 
 ### 5.1 Prep (before the window, no downtime)
 
-1. Build: publish `3a59c3f6` self-contained win-x64, app and cli, with the same flags as the 2026-10-22 builds. Zip it, record the SHA-256, and confirm ProductVersion `1.0.0+3a59c3f6…`. Stage it over SMB to `E:\psrem-stage\3a59c3f6\` on both boxes and re-verify the hash on each box.
+1. Build: publish `8e32f353` self-contained win-x64, app and cli, with the same flags as the 2026-10-22 builds. Zip it, record the SHA-256, and confirm ProductVersion `1.0.0+8e32f353…`. Stage it over SMB to `E:\psrem-stage\8e32f353\` on both boxes and re-verify the hash on each box.
 2. Re-run `preflight.sql` and compare it with §3. Abort if the migration count is not 58 or the key is not `EventId`-first.
 3. On-box prep (WinRM, deploy scope; read-only except the exports):
    - `reg export` both services' env;
@@ -195,13 +223,13 @@ All times are local (EST). Roles: one operator on the Mac; WinRM for the on-box
 
 | Step | Action | Notes |
 |---|---|---|
-| W0 | **Backup** from the Mac (direct SQL): `BACKUP DATABASE [ScadaBridge] TO DISK = N'F:\MSSQL\BACKUP\ScadaBridge_pre-3a59c3f6_<ts>.bak' WITH COPY_ONLY, CHECKSUM, COMPRESSION, STATS = 10;` then `RESTORE VERIFYONLY FROM DISK = … WITH CHECKSUM;` | About 900 MB of data, compressed. The previous run took a few seconds. Taken while everything is still up; nothing writes config during the window |
+| W0 | **Backup** from the Mac (direct SQL): `BACKUP DATABASE [ScadaBridge] TO DISK = N'F:\MSSQL\BACKUP\ScadaBridge_pre-8e32f353_<ts>.bak' WITH COPY_ONLY, CHECKSUM, COMPRESSION, STATS = 10;` then `RESTORE VERIFYONLY FROM DISK = … WITH CHECKSUM;` | About 900 MB of data, compressed. The previous run took a few seconds. Taken while everything is still up; nothing writes config during the window |
 | W1 | **Window opens.** Stop `ScadaBridge-Site` on the standby box (vq04), then on the active box (vq03) | The site data path stops here |
 | W2 | Stop `ScadaBridge-Central` on vq04, then on vq03. **Wait until no `ZB.MOM.WW.ScadaBridge.Host.exe` process remains on either box** | Both centrals are now down, which migrations 61 and 63 require |
 | W3 | Migrate from the Mac (§2.2). Check: 64 migrations, flipped key | About 10–30 s |
 | W4 | Swap on each box: keep the three appsettings, rename `app` / `cli` → `*.bak.<ts>`, unpack, restore the appsettings (hash match) | As in 2026-10-22 |
 | W5 | Config edits (§5.4) on both boxes, pushed over SMB while stopped. Record new hashes | #116 |
-| W6 | Service recovery (§5.3) if the prep found it missing | |
+| W6 | Service recovery (§5.3) on all four services (approved in the go) | Record `sc.exe qfailure`/`qfailureflag` after |
 | W7 | **Start both Centrals together** (vq03 and vq04 within a few seconds of each other), then wait for `/health/ready` 200 and memberCount 2 on both | BootstrapGuard is on. Don't start one central and wait (memory: a cold single-node start can self-down) |
 | W8 | **Start both Sites together**, then wait for ready 200 on both, `localdb replicationConnected:true` and backlog 0. **Window closes** | |
 
@@ -214,7 +242,7 @@ sc.exe failure ScadaBridge-Central reset= 86400 actions= restart/10000/restart/3
 sc.exe failureflag ScadaBridge-Central 1
 ```
 
-Do the same for `ScadaBridge-Site`. This is a change to the service definition, so the owner should confirm it when giving the go. Without it, the #132 self-heal leaves a node **stopped**, not restarted.
+Do the same for `ScadaBridge-Site`. This is a change to the service definition; **the owner approved it in the 2026-11-13 go**, so set it in W6 whatever the prep finds missing (it is idempotent). Without it, the #132 self-heal leaves a node **stopped**, not restarted.
 
 ### 5.4 Config edits (both boxes, while stopped)
 
@@ -239,7 +267,7 @@ Run them from the Mac over HTTP, SQL, SMB and the CLI. No ad-hoc WinRM, except t
 
 | Gate | Check | Pass |
 |---|---|---|
-| G1 Build | `Host.dll` ProductVersion on both boxes (SMB, `strings`) | `1.0.0+3a59c3f6` ×2 |
+| G1 Build | `Host.dll` ProductVersion on both boxes (SMB, `strings`) | `1.0.0+8e32f353` ×2 |
 | G2 Schema | `preflight.sql` §1/§4/§7 | 64; `OccurredAtUtc` first; chain tables + 3 roles |
 | G3 Health | `/health/ready` on :8085 and :8084 on both boxes | 200 ×4; `database`, `akka-cluster`, `required-singletons` Healthy |
 | G4 Cluster | Central `akka-cluster` data on both nodes; site `/health/ready` | memberCount 2, unreachable 0, one leader. Site `localdb.replicationConnected:true`, backlog 0 |
@@ -251,14 +279,15 @@ Run them from the Mac over HTTP, SQL, SMB and the CLI. No ad-hoc WinRM, except t
 | G10 AuditLog | SQL: `SELECT COUNT(*), MAX(OccurredAtUtc) FROM dbo.AuditLog WHERE OccurredAtUtc > <window close>` after G8 | New rows, `SourceNode` set, `ChainSeq` NULL (chain off); no 1205 in the logs |
 | G11 Overview | `http://wonder-app-vq03:5320` and `vq04:5320` | The ScadaBridge Central and Site clusters each show one Active, the leader chip, and **no** "Split brain" |
 | G12 Artifacts / staleness | `instance diff` on every instance; the Sites page | No stale instances; artifact hash unchanged `sha256:d79ba1a3…`. No redeploy needed |
-| G13 OTLP | SMB: `E:\ApiInstall\OtelCollector\data\logs.jsonl` on each box | Lines with `service.name` scadabridge and `service.version` 3a59c3f6 (logs are new with Telemetry 0.3.x) |
+| G13 OTLP | SMB: `E:\ApiInstall\OtelCollector\data\logs.jsonl` on each box | Lines with `service.name` scadabridge and `service.version` 8e32f353 (logs are new with Telemetry 0.3.x) |
-| G14 #116 drill (bundled) | Kill the **active** central's Host (`Stop-Process -Force`, deploy scope), then repeat on the standby. Watch `/health/active` and the logs | Survivor `[leader-auto-down] … DOWNING` about +27 s, takes over in about 30 s; the victim is restarted by SCM and rejoins as standby; then 10/10 single-active. **The site pair drill re-hosts every instance, including Z28061 (about 30 s data gap)**, so run it only if the owner OKs that in the go. Otherwise record "central only" in #116 |
+| ~~G14 #116 drill~~ | **Not run: excluded by the owner's go (2026-11-13).** Neither the central kill drill nor the site-pair kill drill. For a later window, the drill was: kill the active central's Host (`Stop-Process -Force`), then the standby; survivor `[leader-auto-down] … DOWNING` about +27 s, takeover about 30 s, the victim restarted by SCM (§5.3) and rejoins as standby. The site-pair drill re-hosts every instance including Z28061 | Record in #116: "QA on leader-auto-down, not drilled" |
+| G15 Recovery setting | `sc.exe qfailure` / `qfailureflag` on all four services (read back in the window, deploy scope) | restart/10000/restart/30000/restart/60000, reset 86400, failure flag 1, ×4 |
 
 ## 7. Hash chain, head export, maintenance login (the #107 owner decisions)
 
-- **In this window: the chain stays OFF on QA** (`AuditLog:HashChain:Enabled` is unset, so false). The migration only adds the columns, tables and roles.
+- **In this window: the chain stays OFF on QA, and the head export stays OFF** (owner's go, 2026-11-13: `AuditLog:HashChain:Enabled` and `AuditLog:HashChain:HeadExport:Enabled` stay unset, so false). The migration only adds the columns, tables and roles. G10 checks `ChainSeq` stays NULL.
 - **After the window, as a separate step:**
-  0. **Build: needs the #139 cutoff.** "No backlog chaining" is implemented on `main` from `52da02a7` (#139): the first chaining pass records a chain-start anchor at the database's `SYSUTCDATETIME()` and only rows dated from then on are chained. **`3a59c3f6` does not have it**: enabling the chain on `3a59c3f6` would chain all 3,441+ historical rows. So either run this window at a tip that contains `52da02a7` (it adds **no migration**: the script in §2.2 is unchanged, re-check only the option delta), or ship such a build before step 2. Confirm with `Host.dll` ProductVersion.
+  0. **Build: needs the #139 cutoff.** "No backlog chaining" is implemented on `main` from `52da02a7` (#139): the first chaining pass records a chain-start anchor at the database's `SYSUTCDATETIME()` and only rows dated from then on are chained. **This window's build `8e32f353` contains it** (the refresh moved the target past `52da02a7`; no migration, §1.5), so no further build is needed before step 2. Confirm with `Host.dll` ProductVersion `1.0.0+8e32f353` (G1). Never enable the chain on a build older than `52da02a7`: it would chain all 3,441+ historical rows.
   1. SourceNode backfill first, because a chained row's `SourceNode` is frozen. Run `scadabridge audit backfill-source-node --before 2026-07-08T00:00:00Z --sentinel unknown`, which covers the 843 pre-M5.6 rows (Admin; idempotent). Then confirm that `preflight.sql` §6 shows 0 NULLs. (With the cutoff the backlog is never chained, so its `SourceNode` is never frozen: the order no longer matters, but running it first keeps the documented #107 sequence.)
   2. Then set `AuditLog:HashChain:Enabled=true` on both centrals and restart them **together**. **Do not set `AuditLog:HashChain:ChainBacklog`** (default `false` = no backlog chaining, the owner decision). It is only read on the very first enablement; the first pass fixes the cutoff in the database and both nodes use it from then on.
   3. **Verify the cutoff** (direct SQL, read-only, a minute after start):
@@ -287,14 +316,14 @@ Run them from the Mac over HTTP, SQL, SMB and the CLI. No ad-hoc WinRM, except t
 2. Restore from the Mac:
    ```sql
    ALTER DATABASE [ScadaBridge] SET SINGLE_USER WITH ROLLBACK IMMEDIATE;
-   RESTORE DATABASE [ScadaBridge] FROM DISK = N'F:\MSSQL\BACKUP\ScadaBridge_pre-3a59c3f6_<ts>.bak' WITH REPLACE, CHECKSUM;
+   RESTORE DATABASE [ScadaBridge] FROM DISK = N'F:\MSSQL\BACKUP\ScadaBridge_pre-8e32f353_<ts>.bak' WITH REPLACE, CHECKSUM;
    ALTER DATABASE [ScadaBridge] SET MULTI_USER;
    ```
    Any audit or config rows written after W0 are lost.
 3. On each box, rename `app` → `app.failed`, `app.bak.<ts>` → `app`, and the same for `cli`. Restore the pre-window appsettings from `_appsettings-keep-<ts>`. That puts the strategy back to `auto-down`.
 4. Start both Centrals together, then both Sites together. Confirm `0e540291`, 58 migrations, health.
 
-**Protocol config is encrypted at rest** (since `2b9a7c3c`, migration `EnforceGlobalInstanceNamesAndProtectProtocolConfiguration`). The format did not change in this range, so `0e540291` reads what `3a59c3f6` writes. However, **any binary older than `2b9a7c3c` cannot read protected rows at all**, and a binary-only rollback against schema 64 is unsupported. So restore the DB together with the binaries, and never roll back past `0e540291`.
+**Protocol config is encrypted at rest** (since `2b9a7c3c`, migration `EnforceGlobalInstanceNamesAndProtectProtocolConfiguration`). The format did not change in this range, so `0e540291` reads what `8e32f353` writes. However, **any binary older than `2b9a7c3c` cannot read protected rows at all**, and a binary-only rollback against schema 64 is unsupported. So restore the DB together with the binaries, and never roll back past `0e540291`.
 
 **#116-only rollback** (keep the new build): set `SplitBrainResolverStrategy` back to `auto-down` in both files on both boxes, then stop and start **each pair together**.
 
@@ -306,11 +335,15 @@ Run them from the Mac over HTTP, SQL, SMB and the CLI. No ad-hoc WinRM, except t
 | W3 migrations | 10–30 s | Flip about 2–4 s on 18 MB; the rest is small |
 | W4–W6 swap + config | 30–60 s | 2026-10-22 swap took about 17 s per box |
 | W7–W8 start both pairs | 40–60 s | Central ready about 30 s, site about 10 s |
-| **Full outage (central API + site data path)** | **about 3–4 min** | The 2026-10-22 coordinated window was **2 min 35 s** |
+| W6 service recovery | 10–20 s | Four `sc.exe` pairs, while stopped |
-| Book | **45 min** | Backup and verify, the gates, and the #116 kill drills (each adds about 30 s of single-pair failover, but no full outage) |
+| **Full outage (central API + site data path)** | **about 3–4 min** | The 2026-10-22 coordinated window was **2 min 35 s**. #140 does not shorten it: the window's stops run the old binaries |
+| Book | **40 min** | Backup and verify plus the gates. No #116 drills (excluded by the go), so no single-pair failovers after the window |
 
 ## 10. Related gates in or near this window
 
-- **#131:** QA has **nothing** to change (deploy-prep `2026-11-12-gates-52-131`).
+- **#131 is DEV-only:** QA has **nothing** to change (deploy-prep `2026-11-12-gates-52-131`), re-confirmed against the QA config DB on 2026-11-13 07:05 (§3: no `TestMesMachine`, `CvdReactor.AlarmStatus` is a stub with no `.Severity` read, no shared scripts, no API method reads alarms). No script step in this window.
-- **#52:** `CvdReactor.ProcessRecipeDownload` on QA still addresses `Children["DelmiaRecv"]`. The fix (`2026-11-12-gates-52-131/apply.sh apply 52`) is binary-independent, but it **redeploys Z28061–Z28069**, including **Z28061**. **Not bundled by default.** If the owner agrees a Z28061 slot in the same go, run it after G12, so there is still only one redeploy.
+- **#52:** `CvdReactor.ProcessRecipeDownload` on QA still addresses `Children["DelmiaRecv"]`. The fix (`2026-11-12-gates-52-131/apply.sh apply 52`) is binary-independent, but it **redeploys Z28061–Z28069**, including **Z28061**. **Stays excluded:** the owner's 2026-11-13 go covers the binary window only, not a Z28061 redeploy. Still open on QA after the window (re-checked §3); it needs its own slot agreed with the Z28061 team.
-- **#139:** hash-chain "no backlog" cutoff, fixed on `main` at `52da02a7` (no migration). The enablement steps in §7 need a build that contains it.
+- **#139:** hash-chain "no backlog" cutoff, `52da02a7` (no migration), **included in `8e32f353`**. The chain stays off in this window; §7 can run later on this build.
+- **#138:** ServiceLevel failover, `712c5117` (no migration), included; no effect on QA (no OPC UA connection).
+- **#140:** graceful stop, `1a6108c4`, included (§1.2).
+- **#116 drills:** excluded by the go (§6, G14).