fix(localdb): disable SQLite pooling on legacy reads - a Windows site node could not boot

SiteLocalDbLegacyMigrator opens each pre-Phase-2 file read-only, copies its rows into
the consolidated site database, then renames the file so a later boot skips it.
Microsoft.Data.Sqlite POOLS connections, so disposing one returns it to the pool and
leaves the underlying sqlite3 handle - and the OS file handle - open. On Windows the
subsequent File.Move throws IOException ("being used by another process"), the
migration faults out of AddZbLocalDb's factory, and the site node does not start at
all. There is no partial-migration path: it is a hard boot failure, once, on the first
upgrade past LocalDb Phase 2.

Why it shipped: POSIX rename ignores open handles, so this cannot reproduce on Linux or
macOS. Every existing rename assertion in SiteLocalDbLegacyMigratorTests - including
LegacyTrackingRows_AreCopiedAndTheFileIsRenamed - passes with the bug fully present, and
the docker rig migrates cleanly. It was found by pre-flighting the wonder-app-vd03
upgrade against COPIES of that box's real site databases; an earlier pre-flight pass
with empty data directories had nothing to drain and passed clean.

Fix: both legacy read-only connection strings now go through one
LegacyReadOnlyConnectionString helper carrying Pooling=False.

The accompanying test is deliberately white-box. A behavioural assertion cannot
discriminate here on the platform this suite runs on, so it pins the connection string
instead; it is red without the fix.

Verified: Host.Tests 440/440, 0 warnings. The build deployed to wonder-app-vd03 carries
this change (as the then-uncommitted fix) and has been running since 2026-08-05.

CLAUDE.md

@@ -177,7 +177,7 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
 - Automatic dual-node recovery from persistent storage.
 - **Active/standby is decided by `ActiveNodeEvaluator.SelfIsOldestUp`, never by cluster leadership** — see the Architecture note above. `/health/active` is **central-only** (site nodes map no `/health/*` at all) and backs both Traefik's active-node routing and `IActiveNodeGate`, so the proxy and the Inbound API always agree on which node is active. Central reaches a site by dialling `GrpcNodeAAddress`/`GrpcNodeBAddress` explicitly and flipping on error; whichever node answers, the site-internal `ClusterSingletonProxy` lands the work on the active node for free, so central still never needs to track which *site* node is active.
 - **Seed-node ordering: every node lists ITSELF first (decision 2026-07-22) — the boot-alone gap is CLOSED.** Only `seed-nodes[0]` may self-join to form a new cluster (Akka runs `FirstSeedNodeProcess` for it, `JoinSeedNodeProcess` — which can never form one — for everyone else). All 14 shipped node appsettings now lead with the node's own address, so any node can cold-start alone and become operational unattended (~5s, `seed-node-timeout`); `StartupValidator` fails the boot if the ordering is broken (compares host AND port; Akka does no DNS canonicalisation). Two nodes cold-starting together while mutually reachable converge on ONE cluster via the `InitJoin` handshake — they split only under a genuine boot-time partition, the same class auto-down accepts. **An external self-form timer (`Cluster.Join(SelfAddress)` after a window) was implemented and REJECTED:** it sits outside the join handshake, so on a routine standby restart — where the peer is alive but the join is stalled behind removal of the node's own stale incarnation — it fires mid-join and permanently splits the pair (measured: still split after 90s). Regression tests: `SelfFirstSeedBootstrapTests`. The keep-oldest active-crash total outage was separately closed by the auto-down decision. See `docs/requirements/Component-ClusterInfrastructure.md` → Seed Node Ordering.
-- **Simultaneous-cold-start split-brain guard (opt-in, Gitea #33) — the residual self-first cost, closed.** Self-first-on-both means a *truly simultaneous* cold start (shared power/hypervisor event) races `FirstSeedNodeProcess` on BOTH nodes → two 1-node clusters that never merge (in-process loopback tests converge and hide it; real parallel VM starts don't — OtOpcUa reproduced it live). Ported from OtOpcUa (`lmxopcua` `d1dac87f`): a **dark switch `ScadaBridge:Cluster:BootstrapGuard:Enabled` (default OFF, guard-off byte-identical)**. On: `BuildHocon` emits an EMPTY seed list (Akka does not auto-join) and `ClusterBootstrapCoordinator` (`IHostedService`, registered in BOTH the Central and Site composition roots) issues ONE reachability-gated `JoinSeedNodes` — the pure `ClusterBootstrapGuard` core makes the lower canonical `host:port` the **founder** (self-first, forms immediately), the higher node TCP-probes the founder up to `PartnerProbeSeconds` (25s) and joins **peer-first** if reachable else **self-first** (cold-start-alone preserved). Decided BEFORE a single join, **never re-forms mid-handshake** (that was the rejected self-form-timer's flaw); case-insensitive founder tie-break; probe timings validated `>0` when enabled. Residual accepted trade: founder dies in the probe→join window → higher node hangs unjoined, coordinator WARNs, a restart recovers it. Tests: `ClusterBootstrapGuardTests` (pure) + `ClusterBootstrapCoordinatorTests` (real-ActorSystem, incl. both-cold-start-together-form-one-cluster). See `docs/requirements/Component-ClusterInfrastructure.md` → Simultaneous cold start.
+- **Simultaneous-cold-start split-brain guard (opt-in, Gitea #33) — the residual self-first cost, closed.** Self-first-on-both means a *truly simultaneous* cold start (shared power/hypervisor event) races `FirstSeedNodeProcess` on BOTH nodes → two 1-node clusters that never merge (in-process loopback tests converge and hide it; real parallel VM starts don't — OtOpcUa reproduced it live). Ported from OtOpcUa (`lmxopcua` `d1dac87f`): a **dark switch `ScadaBridge:Cluster:BootstrapGuard:Enabled` (default OFF, guard-off byte-identical)**. On: `BuildHocon` emits an EMPTY seed list (Akka does not auto-join) and `ClusterBootstrapCoordinator` (`IHostedService`, registered in BOTH the Central and Site composition roots) issues ONE reachability-gated `JoinSeedNodes` — the pure `ClusterBootstrapGuard` core makes the lower canonical `host:port` the **founder** (self-first, forms immediately), the higher node TCP-probes the founder up to `PartnerProbeSeconds` (25s) and joins **peer-first** if reachable else **self-first** (cold-start-alone preserved). Decided BEFORE a single join, **never re-forms mid-handshake** (that was the rejected self-form-timer's flaw); case-insensitive founder tie-break; probe timings validated `>0` when enabled. Residual accepted trade: founder dies in the probe→join window → higher node hangs unjoined, coordinator WARNs, a restart recovers it. Tests: `ClusterBootstrapGuardTests` (pure) + `ClusterBootstrapCoordinatorTests` (real-ActorSystem, incl. both-cold-start-together-form-one-cluster). **ENABLED on the docker rig (all 8 nodes) 2026-08-02** after `deploy.sh`'s simultaneous recreate split site pairs twice on 2026-08-01 with the guard off; live gate PASS — two consecutive simultaneous-start trials (a full redeploy + a full-topology `docker compose restart`), all four pairs converged (every lower-address node founded self-first, every higher node probed-then-joined peer-first), zero splits. Default remains OFF elsewhere (wonder-app-vd03, docker-env2, Host defaults). See `docs/requirements/Component-ClusterInfrastructure.md` → Simultaneous cold start.
 
 ### UI & Monitoring
 - Central UI: Blazor Server (ASP.NET Core + SignalR) with Bootstrap CSS. No third-party component frameworks (no Blazorise, MudBlazor, Radzen, etc.). Build custom Blazor components for tables, grids, forms, etc.

docker/central-node-a/appsettings.Central.json

@@ -15,7 +15,14 @@
       "StableAfter": "00:00:15",
       "HeartbeatInterval": "00:00:02",
       "FailureDetectionThreshold": "00:00:10",
-      "MinNrOfMembers": 1
+      "MinNrOfMembers": 1,
+      "_bootstrapGuard": "Gitea #33 guard ENABLED on the docker rig (2026-08-02): deploy.sh recreates all containers simultaneously, which twice split site pairs into two 1-node clusters on 2026-08-01. Lower host:port founds self-first; the higher node TCP-probes then joins peer-first.",
+      "BootstrapGuard": {
+        "Enabled": true,
+        "PartnerProbeSeconds": 25,
+        "PartnerProbeIntervalMs": 500,
+        "ProbeConnectTimeoutMs": 1000
+      }
     },
     "Database": {
       "_comment": "ConfigurationDb/MachineDataDb (which carry the dev SQL password) are supplied as ScadaBridge__Database__* whole-key env overrides in docker/docker-compose.yml (dev-only-insecure, mirroring the ApiKeyPepper convention). The same dev password already ships committed for the sibling scadabridge-mssql container (infra/docker-compose.yml MSSQL_SA_PASSWORD) — this is consolidation, not new exposure. Env overrides layer over JSON before the ${secret:} expander runs, so the dev cluster boots with no KEK/secret store. Real/prod config uses ${secret:} tokens in src/.../appsettings.Central.json (T4)."

docker/central-node-b/appsettings.Central.json

@@ -15,7 +15,14 @@
       "StableAfter": "00:00:15",
       "HeartbeatInterval": "00:00:02",
       "FailureDetectionThreshold": "00:00:10",
-      "MinNrOfMembers": 1
+      "MinNrOfMembers": 1,
+      "_bootstrapGuard": "Gitea #33 guard ENABLED on the docker rig (2026-08-02): deploy.sh recreates all containers simultaneously, which twice split site pairs into two 1-node clusters on 2026-08-01. Lower host:port founds self-first; the higher node TCP-probes then joins peer-first.",
+      "BootstrapGuard": {
+        "Enabled": true,
+        "PartnerProbeSeconds": 25,
+        "PartnerProbeIntervalMs": 500,
+        "ProbeConnectTimeoutMs": 1000
+      }
     },
     "Database": {
       "_comment": "ConfigurationDb/MachineDataDb (which carry the dev SQL password) are supplied as ScadaBridge__Database__* whole-key env overrides in docker/docker-compose.yml (dev-only-insecure, mirroring the ApiKeyPepper convention). The same dev password already ships committed for the sibling scadabridge-mssql container (infra/docker-compose.yml MSSQL_SA_PASSWORD) — this is consolidation, not new exposure. Env overrides layer over JSON before the ${secret:} expander runs, so the dev cluster boots with no KEK/secret store. Real/prod config uses ${secret:} tokens in src/.../appsettings.Central.json (T4)."

docker/site-a-node-a/appsettings.Site.json

@@ -18,7 +18,14 @@
       "StableAfter": "00:00:15",
       "HeartbeatInterval": "00:00:02",
       "FailureDetectionThreshold": "00:00:10",
-      "MinNrOfMembers": 1
+      "MinNrOfMembers": 1,
+      "_bootstrapGuard": "Gitea #33 guard ENABLED on the docker rig (2026-08-02): deploy.sh recreates all containers simultaneously, which twice split site pairs into two 1-node clusters on 2026-08-01. Lower host:port founds self-first; the higher node TCP-probes then joins peer-first.",
+      "BootstrapGuard": {
+        "Enabled": true,
+        "PartnerProbeSeconds": 25,
+        "PartnerProbeIntervalMs": 500,
+        "ProbeConnectTimeoutMs": 1000
+      }
     },
     "Database": {
       // Migration-only as of LocalDb Phase 2. The site config tables now live in the

docker/site-a-node-b/appsettings.Site.json

@@ -18,7 +18,14 @@
       "StableAfter": "00:00:15",
       "HeartbeatInterval": "00:00:02",
       "FailureDetectionThreshold": "00:00:10",
-      "MinNrOfMembers": 1
+      "MinNrOfMembers": 1,
+      "_bootstrapGuard": "Gitea #33 guard ENABLED on the docker rig (2026-08-02): deploy.sh recreates all containers simultaneously, which twice split site pairs into two 1-node clusters on 2026-08-01. Lower host:port founds self-first; the higher node TCP-probes then joins peer-first.",
+      "BootstrapGuard": {
+        "Enabled": true,
+        "PartnerProbeSeconds": 25,
+        "PartnerProbeIntervalMs": 500,
+        "ProbeConnectTimeoutMs": 1000
+      }
     },
     "Database": {
       // Migration-only as of LocalDb Phase 2. The site config tables now live in the

docker/site-b-node-a/appsettings.Site.json

@@ -18,7 +18,14 @@
       "StableAfter": "00:00:15",
       "HeartbeatInterval": "00:00:02",
       "FailureDetectionThreshold": "00:00:10",
-      "MinNrOfMembers": 1
+      "MinNrOfMembers": 1,
+      "_bootstrapGuard": "Gitea #33 guard ENABLED on the docker rig (2026-08-02): deploy.sh recreates all containers simultaneously, which twice split site pairs into two 1-node clusters on 2026-08-01. Lower host:port founds self-first; the higher node TCP-probes then joins peer-first.",
+      "BootstrapGuard": {
+        "Enabled": true,
+        "PartnerProbeSeconds": 25,
+        "PartnerProbeIntervalMs": 500,
+        "ProbeConnectTimeoutMs": 1000
+      }
     },
     "Database": {
       // Migration-only as of LocalDb Phase 2. The site config tables now live in the

docker/site-b-node-b/appsettings.Site.json

@@ -18,7 +18,14 @@
       "StableAfter": "00:00:15",
       "HeartbeatInterval": "00:00:02",
       "FailureDetectionThreshold": "00:00:10",
-      "MinNrOfMembers": 1
+      "MinNrOfMembers": 1,
+      "_bootstrapGuard": "Gitea #33 guard ENABLED on the docker rig (2026-08-02): deploy.sh recreates all containers simultaneously, which twice split site pairs into two 1-node clusters on 2026-08-01. Lower host:port founds self-first; the higher node TCP-probes then joins peer-first.",
+      "BootstrapGuard": {
+        "Enabled": true,
+        "PartnerProbeSeconds": 25,
+        "PartnerProbeIntervalMs": 500,
+        "ProbeConnectTimeoutMs": 1000
+      }
     },
     "Database": {
       // Migration-only as of LocalDb Phase 2. The site config tables now live in the

docker/site-c-node-a/appsettings.Site.json

@@ -18,7 +18,14 @@
       "StableAfter": "00:00:15",
       "HeartbeatInterval": "00:00:02",
       "FailureDetectionThreshold": "00:00:10",
-      "MinNrOfMembers": 1
+      "MinNrOfMembers": 1,
+      "_bootstrapGuard": "Gitea #33 guard ENABLED on the docker rig (2026-08-02): deploy.sh recreates all containers simultaneously, which twice split site pairs into two 1-node clusters on 2026-08-01. Lower host:port founds self-first; the higher node TCP-probes then joins peer-first.",
+      "BootstrapGuard": {
+        "Enabled": true,
+        "PartnerProbeSeconds": 25,
+        "PartnerProbeIntervalMs": 500,
+        "ProbeConnectTimeoutMs": 1000
+      }
     },
     "Database": {
       // Migration-only as of LocalDb Phase 2. The site config tables now live in the

docker/site-c-node-b/appsettings.Site.json

@@ -18,7 +18,14 @@
       "StableAfter": "00:00:15",
       "HeartbeatInterval": "00:00:02",
       "FailureDetectionThreshold": "00:00:10",
-      "MinNrOfMembers": 1
+      "MinNrOfMembers": 1,
+      "_bootstrapGuard": "Gitea #33 guard ENABLED on the docker rig (2026-08-02): deploy.sh recreates all containers simultaneously, which twice split site pairs into two 1-node clusters on 2026-08-01. Lower host:port founds self-first; the higher node TCP-probes then joins peer-first.",
+      "BootstrapGuard": {
+        "Enabled": true,
+        "PartnerProbeSeconds": 25,
+        "PartnerProbeIntervalMs": 500,
+        "ProbeConnectTimeoutMs": 1000
+      }
     },
     "Database": {
       // Migration-only as of LocalDb Phase 2. The site config tables now live in the

docs/requirements/Component-ClusterInfrastructure.md

@@ -156,7 +156,9 @@ The **bootstrap guard** eliminates the split without giving up cold-start-alone.
 | `BootstrapGuard:PartnerProbeIntervalMs` | `500` | Interval between probes. Validated `> 0` when enabled. |
 | `BootstrapGuard:ProbeConnectTimeoutMs` | `1000` | Per-probe TCP connect timeout. Validated `> 0` when enabled. |
 
-Decision core (`ClusterBootstrapGuard`) is a pure, fully unit-tested function; the probe + `JoinSeedNodes` runtime (`ClusterBootstrapCoordinator`) is covered by real-ActorSystem tests including the load-bearing higher-node-cold-start-alone case and the headline both-cold-start-together-form-one-cluster case (`ClusterBootstrapCoordinatorTests`). The alternative is purely operational (stagger the two VMs' service-manager start / start the founder first); the docker rig's compose `depends_on` does that today, but that does not exist on the real co-located VMs — the guard is the production-faithful fix. Ported from OtOpcUa (`lmxopcua` commit `d1dac87f`); implementing it in both products keeps their failure/recovery model identical, since a shared power event hits both pairs at once.
+Decision core (`ClusterBootstrapGuard`) is a pure, fully unit-tested function; the probe + `JoinSeedNodes` runtime (`ClusterBootstrapCoordinator`) is covered by real-ActorSystem tests including the load-bearing higher-node-cold-start-alone case and the headline both-cold-start-together-form-one-cluster case (`ClusterBootstrapCoordinatorTests`). The alternative is purely operational (stagger the two VMs' service-manager start / start the founder first); compose `depends_on` was assumed to do that on the docker rig, but it demonstrably does not — see below. Ported from OtOpcUa (`lmxopcua` commit `d1dac87f`); implementing it in both products keeps their failure/recovery model identical, since a shared power event hits both pairs at once.
+
+**Live status (2026-08-02): ENABLED on the primary docker rig (all 8 nodes), live gate PASS.** With the guard off, `docker/deploy.sh`'s simultaneous container recreate reproduced the split naturally TWICE on 2026-08-01 — first run split both site-b and site-c (mutual `InitJoinNack`, each node "JOINING itself and forming a new cluster"), second run split site-c while the other pairs converged: the race is a per-pair coin flip, and compose `depends_on` does not serialize far enough to prevent it. With the guard enabled on every rig node, two consecutive simultaneous-start trials (a full redeploy plus a full-topology `docker compose restart`) converged all four pairs deterministically — every lower-address node logged founder/self-first, every higher node logged partner-REACHABLE/peer-first and received `Welcome from` the founder; zero splits, `/health/ready` green. This is the issue-acceptance live gate that was deferred at implementation time. The switch stays default-off in the Host appsettings and everywhere else (wonder-app-vd03, docker-env2).
 
 ### Manual Failover (admin-triggered)
 

src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbLegacyMigrator.cs

@@ -415,7 +415,7 @@ public static class SiteLocalDbLegacyMigrator
     /// </summary>
     private static List<object?[]>? ReadAll(string legacyPath, string sql, int expectedColumns)
     {
-        using var connection = new SqliteConnection($"Data Source={legacyPath};Mode=ReadOnly");
+        using var connection = new SqliteConnection(LegacyReadOnlyConnectionString(legacyPath));
         connection.Open();
 
         using var cmd = connection.CreateCommand();
@@ -450,11 +450,34 @@ public static class SiteLocalDbLegacyMigrator
 
     private static SqliteConnection OpenLegacyReadOnly(string legacyPath)
     {
-        var connection = new SqliteConnection($"Data Source={legacyPath};Mode=ReadOnly");
+        var connection = new SqliteConnection(LegacyReadOnlyConnectionString(legacyPath));
         connection.Open();
         return connection;
     }
 
+    /// <summary>
+    /// Read-only connection string for a legacy file, with connection pooling DISABLED.
+    /// </summary>
+    /// <remarks>
+    /// <para>
+    /// <b>Pooling=False is load-bearing, not a tuning choice.</b> Microsoft.Data.Sqlite pools
+    /// connections by default, so disposing one returns it to the pool and leaves the
+    /// underlying sqlite3 handle — and therefore the OS file handle — open. Every migrate step
+    /// finishes by renaming its legacy file (<see cref="MarkMigrated"/>), and on Windows
+    /// <see cref="File.Move(string, string)"/> against a file someone still holds open throws
+    /// <see cref="IOException"/> ("being used by another process"). The whole migration then
+    /// faults out of <c>AddZbLocalDb</c>'s factory and the node cannot boot at all.
+    /// </para>
+    /// <para>
+    /// This never shows up on Linux or macOS: POSIX rename does not care about open handles,
+    /// so the docker rig and developer machines migrate cleanly. It reproduces on the first
+    /// Windows site node that boots with a pre-Phase-2 database beside it — which is every
+    /// Windows install being upgraded, exactly once.
+    /// </para>
+    /// </remarks>
+    internal static string LegacyReadOnlyConnectionString(string legacyPath)
+        => $"Data Source={legacyPath};Mode=ReadOnly;Pooling=False";
+
     /// <summary>
     /// Returns the subset of <paramref name="wanted"/> that the legacy table actually has,
     /// in the caller's order. An absent table yields an empty list rather than throwing.

tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbLegacyMigratorTests.cs

@@ -228,6 +228,29 @@ public class SiteLocalDbLegacyMigratorTests : IDisposable
         Assert.True(File.Exists(trackingPath + ".migrated"));
     }
 
+    [Fact]
+    public void LegacyReadsDisablePooling_OrTheRenameFailsOnWindows()
+    {
+        // Deliberately white-box, and deliberately NOT a behavioural test.
+        //
+        // Microsoft.Data.Sqlite pools connections, so disposing one returns it to the pool
+        // and leaves the underlying sqlite3 handle — and the OS file handle — open. Every
+        // migrate step then renames its legacy file. On Windows, File.Move against a file
+        // someone still holds open throws IOException, the migration faults out of
+        // AddZbLocalDb's factory, and the site node cannot boot at all.
+        //
+        // It cannot be reproduced from a behavioural assertion off Windows: POSIX rename does
+        // not care about open handles, so every rename test above — including
+        // LegacyTrackingRows_AreCopiedAndTheFileIsRenamed — passes on Linux and macOS with the
+        // pooling bug fully present. That is exactly how it shipped and reached a production
+        // Windows box. Asserting on the connection string is the only guard that holds on the
+        // platform this suite actually runs on.
+        var cs = SiteLocalDbLegacyMigrator.LegacyReadOnlyConnectionString(Path_("whatever.db"));
+
+        Assert.Contains("Pooling=False", cs, StringComparison.OrdinalIgnoreCase);
+        Assert.Contains("Mode=ReadOnly", cs, StringComparison.OrdinalIgnoreCase);
+    }
+
     [Fact]
     public void LegacyEvents_GetDeterministicIds_NotFreshGuids()
     {