test(harness): seed ClusterNode rows in in-memory mode; retarget the failover deploy test

The full-suite run surfaced two load-dependent failures that were green in
isolation. Root cause: SeedDefaultClusterAsync no-opped unless
OTOPCUA_HARNESS_USE_SQL=1, on the reasoning that "the in-memory provider ignores
FK constraints, so deploy E2E tests pass without it". Phase 1 invalidated that.

With no ClusterNode rows the coordinator's expected-ack set is empty, so it
seals IMMEDIATELY. "Wait for Sealed" therefore stopped implying "every node has
applied" — and because DriverHostActor.UpsertNodeDeploymentState writes each
node's row on its own schedule, every test counting those rows after a seal
became a race. Reproducibly green alone, red under suite load, and it surfaced
in a different test each run, which is what made it look like flakiness rather
than a contract change.

Seeding in both modes restores the invariant those tests were written against
and is closer to production either way: a real fleet always has these rows,
because the FK requires them.

Deployment_started_with_node_b_down_seals_with_one_node_state asserted the
behaviour Phase 1 deliberately removed — its own comment documented membership
snapshotting, i.e. sealing green while a configured node never received the
deployment. Replaced by two tests covering the new contract: a stopped node is
still expected (both state rows exist, B still Applying, deployment does not
seal), and MaintenanceMode is what makes it seal with one.

The new "does not seal" test asserts both rows exist rather than only the
absence of a seal, so it cannot pass against a coordinator that simply died.

Host.IntegrationTests 195/201, sole remaining failure AbCip_Green_AgainstSim —
verified failing on master in a clean worktree, so pre-existing fixture
baseline, not a regression.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW

tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/FailoverDuringDeployTests.cs

@@ -50,13 +50,22 @@ public sealed class FailoverDuringDeployTests
             .Count(m => m.Status == MemberStatus.Up).ShouldBe(2);
     }
 
-    /// <summary>Verifies that a deployment started with node B down seals with one-node state.</summary>
+    /// <summary>
+    ///     A deployment started with node B down no longer seals without it — B's <c>ClusterNode</c>
+    ///     row is enabled, so it is expected to ack and the deployment waits.
+    /// </summary>
+    /// <remarks>
+    ///     <b>This test asserted the opposite until per-cluster mesh Phase 1.</b> It was named
+    ///     <c>Deployment_started_with_node_b_down_seals_with_one_node_state</c> and documented that
+    ///     "<c>DiscoverDriverNodes</c> snapshots membership at dispatch time — when only node A is Up,
+    ///     only one ApplyAck is expected and the deployment seals without B ever participating". That
+    ///     is exactly the behaviour Phase 1 removed: it told the operator the fleet was deployed while
+    ///     a configured node had not received it. The expected-ack set now comes from enabled
+    ///     <c>ClusterNode</c> rows, so a node that is merely switched off is still expected.
+    /// </remarks>
     [Fact]
-    public async Task Deployment_started_with_node_b_down_seals_with_one_node_state()
+    public async Task Deployment_started_with_node_b_down_does_not_seal_without_it()
     {
-        // Establishes that ConfigPublishCoordinator.DiscoverDriverNodes snapshots membership at
-        // dispatch time — when only node A is Up, only one ApplyAck is expected and the
-        // deployment seals without B ever participating.
         await using var harness = await TwoNodeClusterHarness.StartAsync();
         await harness.SeedDefaultClusterAsync();
 
@@ -70,10 +79,68 @@ public sealed class FailoverDuringDeployTests
         result.Outcome.ShouldBe(StartDeploymentOutcome.Accepted);
         var deploymentId = result.DeploymentId!.Value.Value;
 
+        // Positive evidence that B was EXPECTED, not merely slow: the coordinator seeds a row per
+        // expected node at dispatch, so both rows must exist with B still Applying. Asserting only
+        // "it didn't seal" would pass just as well against a coordinator that had died.
         await WaitForAsync(async () =>
         {
+            await using var pollDb = await CreateDbAsync(harness);
+            return await pollDb.NodeDeploymentStates.AsNoTracking()
+                .CountAsync(s => s.DeploymentId == deploymentId, Ct) == 2;
+        }, TimeSpan.FromSeconds(15));
+
         await using var db = await CreateDbAsync(harness);
-            var d = await db.Deployments.AsNoTracking()
+        var nodeStates = await db.NodeDeploymentStates.AsNoTracking()
+            .Where(s => s.DeploymentId == deploymentId)
+            .ToListAsync(Ct);
+        nodeStates.Count.ShouldBe(2, "both configured nodes are expected to ack");
+        nodeStates.Count(s => s.Status == NodeDeploymentStatus.Applied)
+            .ShouldBe(1, "only the running node applied");
+        nodeStates.ShouldContain(s => s.Status == NodeDeploymentStatus.Applying,
+            "the stopped node's ack is still outstanding");
+
+        var deployment = await db.Deployments.AsNoTracking()
+            .FirstAsync(d => d.DeploymentId == deploymentId, Ct);
+        deployment.Status.ShouldNotBe(DeploymentStatus.Sealed,
+            "a deployment must not seal green while a configured node has not received it");
+    }
+
+    /// <summary>
+    ///     The maintenance hatch, end-to-end: a node down <i>and</i> flagged
+    ///     <c>MaintenanceMode</c> is not expected, so the deployment seals with one node state — the
+    ///     behaviour the test above used to assert unconditionally, now something an operator has to
+    ///     ask for.
+    /// </summary>
+    [Fact]
+    public async Task Deployment_seals_without_a_node_flagged_for_maintenance()
+    {
+        await using var harness = await TwoNodeClusterHarness.StartAsync();
+        await harness.SeedDefaultClusterAsync();
+
+        await harness.StopNodeBAsync();
+        await harness.WaitForClusterSizeAsync(1, TimeSpan.FromSeconds(20));
+
+        await using (var setup = await CreateDbAsync(harness))
+        {
+            var nodeB = await setup.ClusterNodes.FirstAsync(n => n.NodeId == harness.NodeBNodeId, Ct);
+            nodeB.MaintenanceMode = true;
+            await setup.SaveChangesAsync(Ct);
+            // Still Enabled — DraftValidator.ValidateClusterTopology requires the enabled-node count
+            // to equal ServerCluster.NodeCount, which is why MaintenanceMode exists as its own flag.
+            nodeB.Enabled.ShouldBeTrue();
+        }
+
+        await using var scope = harness.NodeA.Services.CreateAsyncScope();
+        var client = scope.ServiceProvider.GetRequiredService<IAdminOperationsClient>();
+
+        var result = await client.StartDeploymentAsync(createdBy: "alice@test", Ct);
+        result.Outcome.ShouldBe(StartDeploymentOutcome.Accepted, $"Deploy not accepted: {result.Message}");
+        var deploymentId = result.DeploymentId!.Value.Value;
+
+        await WaitForAsync(async () =>
+        {
+            await using var pollDb = await CreateDbAsync(harness);
+            var d = await pollDb.Deployments.AsNoTracking()
                 .FirstOrDefaultAsync(d => d.DeploymentId == deploymentId, Ct);
             return d?.Status == DeploymentStatus.Sealed;
         }, TimeSpan.FromSeconds(15));
@@ -82,7 +149,8 @@ public sealed class FailoverDuringDeployTests
         var nodeStates = await db.NodeDeploymentStates.AsNoTracking()
             .Where(s => s.DeploymentId == deploymentId)
             .ToListAsync(Ct);
-        nodeStates.Count.ShouldBe(1);
+        nodeStates.Count.ShouldBe(1, "the maintenance node is not expected to ack");
+        nodeStates[0].NodeId.ShouldBe(harness.NodeANodeId);
         nodeStates[0].Status.ShouldBe(NodeDeploymentStatus.Applied);
     }
 

tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/TwoNodeClusterHarness.cs

@@ -122,16 +122,26 @@ public sealed class TwoNodeClusterHarness : IAsyncDisposable
     /// Seeds a default <see cref="ServerCluster"/> plus a <see cref="ClusterNode"/> row for BOTH
     /// harness nodes (<see cref="NodeANodeId"/> / <see cref="NodeBNodeId"/>) so the real-SQL FK
     /// constraint <c>FK_NodeDeploymentState_ClusterNode_NodeId</c> is satisfied when a deployment
-    /// records per-node state. The EF in-memory provider ignores FK constraints, so deploy E2E
+    /// records per-node state. Against SQL Server each node's <c>NodeDeploymentState</c> INSERT
-    /// tests pass without this; against SQL Server each node's <c>NodeDeploymentState</c> INSERT
     /// fails without its parent <see cref="ClusterNode"/> row and the deployment never seals.
-    /// No-op unless <c>OTOPCUA_HARNESS_USE_SQL=1</c> (in-memory needs no seeding). Call once, before
+    /// Call once, before <c>StartDeploymentAsync</c>, in tests that don't already seed their own
-    /// <c>StartDeploymentAsync</c>, in tests that don't already seed their own cluster + both nodes.
+    /// cluster + both nodes.
     /// </summary>
+    /// <remarks>
+    ///     <b>This used to no-op in in-memory mode</b> ("the in-memory provider ignores FK
+    ///     constraints, so deploy E2E tests pass without it"). That stopped being true in per-cluster
+    ///     mesh Phase 1: <c>ConfigPublishCoordinator</c> now derives its expected-ack set from these
+    ///     rows rather than from cluster membership, so with none seeded it seals <i>immediately</i>
+    ///     with an empty set. "Wait for Sealed" then no longer implies "every node has applied", and
+    ///     since <c>DriverHostActor.UpsertNodeDeploymentState</c> writes each node's row on its own
+    ///     schedule, any test counting those rows after a seal became a race — reproducibly green
+    ///     alone and red under suite load. Seeding in both modes restores the invariant the tests
+    ///     were written against, and is closer to production either way: a real fleet always has
+    ///     these rows, because the FK requires them.
+    /// </remarks>
     /// <param name="clusterId">Cluster id for the seeded rows. Defaults to <c>MAIN</c>.</param>
     public async Task SeedDefaultClusterAsync(string clusterId = "MAIN")
     {
-        if (!Mode.UseSqlServer) return;
         await using var db = await CreateConfigDbContextAsync();
         db.ServerClusters.Add(new ServerCluster
         {