fix(harness): deploy E2E tests pass against real SQL (#452)

archreview/plans/INTEGRATION-SWEEP-STATUS.md

@@ -37,12 +37,12 @@ correctly bound — the sweep's non-standard "failures" were the stale-container
 and all driver **unit** suites + the live `/run` gates pass. These integration suites were DEFERRED in round 2
 and never run, so they'd bit-rotted in several distinct ways.
 
-**Follow-up progress (2026-07-15):** #1 AbCip (PR #446), #5 OpcUaServer (PR #447), #2 Modbus (PR #448 — false
-alarm), #4 AbLegacy (PR #449 — fixture gap), #3 FOCAS (PR #450 — topology + mock-timer), #7 (fixture deploys),
-and the #6 **LDAP-image** leg (PR #451 — swapped OpenLDAP→GLAuth) all closed. **The only thing left is the #6
-amd64-emulated-SQL timing leg** (7 heavy E2E tests time out under emulated `mssql:2022` on arm64 — run against
-real SQL or raise deadlines; a local-emulation artifact, not a product defect). Every driver + server suite is
-verified GREEN.
+**Follow-up progress (2026-07-15): ALL 7 CLOSED.** #1 AbCip (PR #446), #5 OpcUaServer (PR #447), #2 Modbus
+(PR #448 — false alarm), #4 AbLegacy (PR #449 — fixture gap), #3 FOCAS (PR #450 — topology + mock-timer), #7
+(fixture deploys), #6 LDAP-image (PR #451 — OpenLDAP→GLAuth), and #6 SQL-deploy-E2E (PR #452 — misdiagnosed
+"emulation timing"; really missing FK-parent seeds + a stale EquipmentId, hidden by the in-memory provider).
+**Every driver + server integration suite is now verified GREEN. Zero OtOpcUa production regressions across the
+entire sweep** — every failure was fixture/harness/test rot.
 
 ---
 
@@ -119,12 +119,28 @@ verified GREEN.
     password → `Invalid credentials (49)`). NB: the real-LDAP mode is **opt-in** — the default Host run uses
     `StubLdapAuthService`, so this never blocked the default suite (the sweep's "5 LDAP skips" were actually the
     driver-fixture-gated E2E tests, not LDAP).
-  - **amd64-emulated-SQL timing (still open).** 7 heavy 2-node deploy/failover/reconnect E2E tests
-    (`DeployHappyPathTests`, `FailoverDuringDeployTests`, `DriverReconnectE2eTests`,
-    `FleetDiagnosticsRoundTripTests`, `EquipmentNamespaceMaterializationTests`, `RoslynVirtualTagEvaluatorTests`)
-    take 20–45 s and time out under the amd64-emulated `mssql:2022` (no arm64 image) on the arm64 Mac. **Fix:**
-    run Host E2E against real (non-emulated) SQL — e.g. the shared SQL on the Linux Docker host — or raise the
-    per-test deadlines. Not a product defect; a local-emulation timing artifact.
+  - **SQL-backed deploy E2E — ✅ FIXED 2026-07-15 (PR #452). The sweep's "amd64-emulated-SQL timing" diagnosis
+    was WRONG.** Running the 7 heavy E2E against real (native-amd64) SQL on the Docker host proved they don't
+    time out for speed — they had **never actually run against real SQL**, only the EF in-memory provider (which
+    ignores FK constraints). Two distinct root causes, both hidden by in-memory:
+    - *Missing FK-parent seed (5 tests):* the deploy writes a `NodeDeploymentState` row per node, which FKs to
+      `ClusterNode` (`FK_NodeDeploymentState_ClusterNode_NodeId`). The tests never seeded `ServerCluster` +
+      `ClusterNode` rows, so on SQL each node-state INSERT throws → the deploy never seals (waits the full
+      deadline regardless of how high — proven at 4× it still hung). Added `TwoNodeClusterHarness.SeedDefaultClusterAsync`
+      (seeds a `ServerCluster` + a `ClusterNode` per node; `Warm`/NodeCount=2 to satisfy both the SQL CHECK
+      constraint `CK_ServerCluster_RedundancyMode_NodeCount` and the `ClusterEnabledNodeCountMismatch` validator)
+      and called it in `DeployHappyPathTests`×2 / `FailoverDuringDeployTests` / `FleetDiagnosticsRoundTripTests` /
+      `EquipmentNamespaceMaterializationTests`; fixed `DriverReconnectE2eTests` to seed node B + declare NodeCount=2.
+    - *Stale `EquipmentId` (1 test, pre-existing on BOTH providers):* `EquipmentNamespaceMaterializationTests`
+      seeded `EquipmentId="eq-1"`, but the `EquipmentIdNotDerived` validator (added later) requires
+      `EquipmentId == DeriveEquipmentId(EquipmentUuid)` = `EQ-` + first 12 hex of the UUID. Fixed the seed to a
+      canonical id + matching UUID. (Was failing on in-memory too; only surfaced once the FK fix let the deploy
+      reach equipment validation.)
+    - Added `OTOPCUA_HARNESS_SQL_HOST` / `OTOPCUA_HARNESS_LDAP_HOST` env overrides so the fixtures can live on the
+      native-amd64 Docker host (`docker compose up -d sql` there; run with `OTOPCUA_HARNESS_SQL_HOST=10.100.0.35,14331`).
+    - Result: the 5 SQL-backed classes go **11/11 green vs native SQL**, and the in-memory default is unregressed
+      (12/12). `RoslynVirtualTagEvaluatorTests.Evaluate_racing_ClearCompiledScripts…` is a **pre-existing flaky
+      race test** (not SQL-backed; passes in isolation, flaked once under heavy parallel load) — left as-is.
 
 - **TwinCAT** — 13 skip cleanly (no TC3 XAR fixture exists; unit-only per the driver's design). Expected.
 
@@ -159,8 +175,11 @@ verified GREEN.
 6. **Host** — ⬐ split into two:
    - ~~replace `bitnami/openldap:2.6` (image gone)~~ ✅ **DONE 2026-07-15 (PR #451)** — swapped the harness LDAP
      to **GLAuth** (unifies on the LDAP the rest of the project uses); live-verified the binds on `:3894`.
-   - **[OPEN] amd64-emulated-SQL timing** — the 7 heavy E2E tests time out under emulated `mssql:2022` on arm64.
-     Run them against real (non-emulated) SQL or raise deadlines. Local-emulation artifact, not a product defect.
+   - ~~**amd64-emulated-SQL timing**~~ ✅ **DONE 2026-07-15 (PR #452) — misdiagnosed.** Not a timing/emulation
+     issue: the deploy E2E tests had never run against real SQL (in-memory ignores FKs) and lacked the
+     `ServerCluster`/`ClusterNode` seed the `NodeDeploymentState` FK requires → deploy never seals. Added a
+     harness seeding helper + `OTOPCUA_HARNESS_SQL_HOST` override; also fixed a stale `EquipmentId` in one test.
+     11/11 green vs native SQL, in-memory unregressed. (Roslyn racing test = pre-existing flaky, left as-is.)
 7. ~~**Fixture deployment gap** — FOCAS + AbLegacy fixtures aren't on the host.~~
    ✅ **RESOLVED 2026-07-15.** Both now deployed under the home dir: `~/otopcua-ablegacy` (PR #449) +
    `~/otopcua-focas` (PR #450). (`/opt` still needs sudo; the home-dir deploys work fine.)

tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/DeployHappyPathTests.cs

@@ -23,6 +23,7 @@ public sealed class DeployHappyPathTests
     public async Task StartDeployment_seals_after_both_nodes_apply()
     {
         await using var harness = await TwoNodeClusterHarness.StartAsync();
+        await harness.SeedDefaultClusterAsync();
 
         await using var scope = harness.NodeA.Services.CreateAsyncScope();
         var client = scope.ServiceProvider.GetRequiredService<IAdminOperationsClient>();
@@ -59,6 +60,7 @@ public sealed class DeployHappyPathTests
     public async Task Replaying_dispatch_to_same_revision_is_idempotent_no_op()
     {
         await using var harness = await TwoNodeClusterHarness.StartAsync();
+        await harness.SeedDefaultClusterAsync();
 
         await using var scope = harness.NodeA.Services.CreateAsyncScope();
         var client = scope.ServiceProvider.GetRequiredService<IAdminOperationsClient>();

tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/DriverReconnectE2eTests.cs

@@ -218,8 +218,10 @@ public sealed class DriverReconnectE2eTests
             Name = "Reconnect E2E Cluster",
             Enterprise = "zb",
             Site = "central",
-            NodeCount = 1,
-            RedundancyMode = RedundancyMode.None,
+            // Both harness nodes are Enabled driver members, so NodeCount must be 2 (Warm) to
+            // satisfy the ClusterEnabledNodeCountMismatch validator + the SQL CHECK constraint.
+            NodeCount = 2,
+            RedundancyMode = RedundancyMode.Warm,
             CreatedBy = "test",
         });
 
@@ -240,6 +242,18 @@ public sealed class DriverReconnectE2eTests
             CreatedBy = "test",
         });
 
+        // The harness runs BOTH nodes as driver members, so the deployment records a
+        // NodeDeploymentState for node B too — seed its ClusterNode row so the real-SQL FK
+        // (FK_NodeDeploymentState_ClusterNode_NodeId) resolves. (In-memory ignores FKs.)
+        db.ClusterNodes.Add(new ClusterNode
+        {
+            NodeId = harness.NodeBNodeId,
+            ClusterId = ClusterId,
+            Host = TwoNodeClusterHarness.LoopbackHost,
+            ApplicationUri = "urn:zb:reconnect-e2e:node-b",
+            CreatedBy = "test",
+        });
+
         db.DriverInstances.Add(new DriverInstance
         {
             DriverInstanceId = DriverId,

tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/EquipmentNamespaceMaterializationTests.cs

@@ -7,6 +7,7 @@ using ZB.MOM.WW.OtOpcUa.Commons.Messages.Admin;
 using ZB.MOM.WW.OtOpcUa.Configuration;
 using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
 using ZB.MOM.WW.OtOpcUa.Configuration.Enums;
+using ZB.MOM.WW.OtOpcUa.Configuration.Validation;
 using ZB.MOM.WW.OtOpcUa.Runtime.Drivers;
 
 namespace ZB.MOM.WW.OtOpcUa.Host.IntegrationTests;
@@ -28,19 +29,28 @@ public sealed class EquipmentNamespaceMaterializationTests
 {
     private static CancellationToken Ct => TestContext.Current.CancellationToken;
 
+    // Equipment.EquipmentId must equal DraftValidator.DeriveEquipmentId(EquipmentUuid) — the
+    // 'EquipmentIdNotDerived' rule (EquipmentId = 'EQ-' + first 12 hex of the UUID). A fixed UUID
+    // keeps the derived id stable for the assertions below.
+    private static readonly Guid EquipmentUuid = Guid.Parse("11111111-1111-1111-1111-111111111111");
+    private static readonly string EquipmentId = DraftValidator.DeriveEquipmentId(EquipmentUuid); // EQ-111111111111
+
     /// <summary>Verifies a deployed Equipment namespace carries its signal into the composed artifact.</summary>
     [Fact]
     public async Task Deploying_an_equipment_namespace_carries_the_signal_into_the_artifact()
     {
         await using var harness = await TwoNodeClusterHarness.StartAsync();
 
+        // Seed the parent ServerCluster + ClusterNode rows the SQL FKs require (the c1 config below
+        // FKs to ServerCluster; on the in-memory provider this is unnecessary — FKs aren't enforced).
+        await harness.SeedDefaultClusterAsync("c1");
         await SeedEquipmentNamespaceAsync(harness);
 
         await using var scope = harness.NodeA.Services.CreateAsyncScope();
         var client = scope.ServiceProvider.GetRequiredService<IAdminOperationsClient>();
 
         var result = await client.StartDeploymentAsync(createdBy: "alice@test", Ct);
-        result.Outcome.ShouldBe(StartDeploymentOutcome.Accepted);
+        result.Outcome.ShouldBe(StartDeploymentOutcome.Accepted, $"Deploy not accepted: {result.Message}");
         var deploymentId = result.DeploymentId!.Value.Value;
 
         // The artifact is composed + persisted at dispatch time; assert on it without waiting for
@@ -65,13 +75,13 @@ public sealed class EquipmentNamespaceMaterializationTests
         // ParseComposition extracted it as an EquipmentTag with FullName pulled from TagConfig.
         var tag = composition.EquipmentTags.ShouldHaveSingleItem();
         tag.TagId.ShouldBe("tag-speed");
-        tag.EquipmentId.ShouldBe("eq-1");
+        tag.EquipmentId.ShouldBe(EquipmentId);
         tag.Name.ShouldBe("Speed");
         tag.DataType.ShouldBe("Float");
         tag.FullName.ShouldBe("40001");
 
         // The equipment folder browses by its friendly UNS Name, not the MachineCode.
-        composition.EquipmentNodes.ShouldContain(e => e.EquipmentId == "eq-1" && e.DisplayName == "station-1");
+        composition.EquipmentNodes.ShouldContain(e => e.EquipmentId == EquipmentId && e.DisplayName == "station-1");
         composition.UnsAreas.ShouldContain(a => a.UnsAreaId == "nw-area-filling" && a.DisplayName == "filling");
         composition.UnsLines.ShouldContain(l => l.UnsLineId == "nw-line-1" && l.DisplayName == "line-1");
     }
@@ -95,12 +105,13 @@ public sealed class EquipmentNamespaceMaterializationTests
         db.UnsLines.Add(new UnsLine { UnsLineId = "nw-line-1", UnsAreaId = "nw-area-filling", Name = "line-1" });
         db.Equipment.Add(new Equipment
         {
-            EquipmentId = "eq-1", DriverInstanceId = "drv-modbus", UnsLineId = "nw-line-1",
+            EquipmentId = EquipmentId, EquipmentUuid = EquipmentUuid,
+            DriverInstanceId = "drv-modbus", UnsLineId = "nw-line-1",
             Name = "station-1", MachineCode = "STATION_001",
         });
         db.Tags.Add(new Tag
         {
-            TagId = "tag-speed", DriverInstanceId = "drv-modbus", EquipmentId = "eq-1",
+            TagId = "tag-speed", DriverInstanceId = "drv-modbus", EquipmentId = EquipmentId,
             Name = "Speed", DataType = "Float", AccessLevel = TagAccessLevel.Read,
             TagConfig = "{\"FullName\":\"40001\"}",
         });

tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/FailoverDuringDeployTests.cs

@@ -58,6 +58,7 @@ public sealed class FailoverDuringDeployTests
         // dispatch time — when only node A is Up, only one ApplyAck is expected and the
         // deployment seals without B ever participating.
         await using var harness = await TwoNodeClusterHarness.StartAsync();
+        await harness.SeedDefaultClusterAsync();
 
         await harness.StopNodeBAsync();
         await harness.WaitForClusterSizeAsync(1, TimeSpan.FromSeconds(20));

tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/FleetDiagnosticsRoundTripTests.cs

@@ -45,6 +45,7 @@ public sealed class FleetDiagnosticsRoundTripTests
     public async Task GetDiagnostics_after_deploy_reports_current_revision()
     {
         await using var harness = await TwoNodeClusterHarness.StartAsync();
+        await harness.SeedDefaultClusterAsync();
 
         await using var scope = harness.NodeA.Services.CreateAsyncScope();
         var adminOps = scope.ServiceProvider.GetRequiredService<IAdminOperationsClient>();

tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/TwoNodeClusterHarness.cs

@@ -14,6 +14,8 @@ using ZB.MOM.WW.OtOpcUa.AdminUI.Clients;
 using ZB.MOM.WW.OtOpcUa.AdminUI.Hubs;
 using ZB.MOM.WW.OtOpcUa.Cluster;
 using ZB.MOM.WW.OtOpcUa.Configuration;
+using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
+using ZB.MOM.WW.OtOpcUa.Configuration.Enums;
 using ZB.MOM.WW.OtOpcUa.ControlPlane;
 using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
 using ZB.MOM.WW.OtOpcUa.Host.Health;
@@ -36,7 +38,9 @@ namespace ZB.MOM.WW.OtOpcUa.Host.IntegrationTests;
 ///   <item><c>OTOPCUA_HARNESS_USE_SQL=1</c> → swap the in-memory DB for SQL Server on
 ///         <c>localhost:14331</c> (see <c>docker-compose.yml</c>). Each harness gets a unique
 ///         database name (<c>OtOpcUa_Harness_{guid}</c>) created via <c>EnsureCreated</c>
-///         and dropped via <c>EnsureDeleted</c> on dispose.</item>
+///         and dropped via <c>EnsureDeleted</c> on dispose. Override the SQL / LDAP host with
+///         <c>OTOPCUA_HARNESS_SQL_HOST</c> / <c>OTOPCUA_HARNESS_LDAP_HOST</c> to point at native
+///         fixtures on the shared Docker host (avoids the arm64-Mac emulated-mssql slowdown).</item>
 ///   <item><c>OTOPCUA_HARNESS_USE_LDAP=1</c> → drop the stub and point <c>LdapAuthService</c>
 ///         at the GLAuth container on <c>localhost:3894</c> (see Docker/glauth/config.toml).</item>
 /// </list>
@@ -114,6 +118,53 @@ public sealed class TwoNodeClusterHarness : IAsyncDisposable
     public Task<OtOpcUaConfigDbContext> CreateConfigDbContextAsync()
         => NodeA.Services.GetRequiredService<IDbContextFactory<OtOpcUaConfigDbContext>>().CreateDbContextAsync();
 
+    /// <summary>
+    /// Seeds a default <see cref="ServerCluster"/> plus a <see cref="ClusterNode"/> row for BOTH
+    /// harness nodes (<see cref="NodeANodeId"/> / <see cref="NodeBNodeId"/>) so the real-SQL FK
+    /// constraint <c>FK_NodeDeploymentState_ClusterNode_NodeId</c> is satisfied when a deployment
+    /// records per-node state. The EF in-memory provider ignores FK constraints, so deploy E2E
+    /// tests pass without this; against SQL Server each node's <c>NodeDeploymentState</c> INSERT
+    /// fails without its parent <see cref="ClusterNode"/> row and the deployment never seals.
+    /// No-op unless <c>OTOPCUA_HARNESS_USE_SQL=1</c> (in-memory needs no seeding). Call once, before
+    /// <c>StartDeploymentAsync</c>, in tests that don't already seed their own cluster + both nodes.
+    /// </summary>
+    /// <param name="clusterId">Cluster id for the seeded rows. Defaults to <c>MAIN</c>.</param>
+    public async Task SeedDefaultClusterAsync(string clusterId = "MAIN")
+    {
+        if (!Mode.UseSqlServer) return;
+        await using var db = await CreateConfigDbContextAsync();
+        db.ServerClusters.Add(new ServerCluster
+        {
+            ClusterId = clusterId,
+            Name = "Harness Default Cluster",
+            Enterprise = "zb",
+            Site = "central",
+            // 2-node cluster → Warm/Hot per CK_ServerCluster_RedundancyMode_NodeCount
+            // (NodeCount=1↔None, NodeCount=2↔Warm|Hot). The harness runs two nodes.
+            NodeCount = 2,
+            RedundancyMode = RedundancyMode.Warm,
+            CreatedBy = "harness",
+        });
+        db.ClusterNodes.AddRange(
+            new ClusterNode
+            {
+                NodeId = NodeANodeId,
+                ClusterId = clusterId,
+                Host = LoopbackHost,
+                ApplicationUri = "urn:zb:harness:node-a",
+                CreatedBy = "harness",
+            },
+            new ClusterNode
+            {
+                NodeId = NodeBNodeId,
+                ClusterId = clusterId,
+                Host = LoopbackHost,
+                ApplicationUri = "urn:zb:harness:node-b",
+                CreatedBy = "harness",
+            });
+        await db.SaveChangesAsync();
+    }
+
     /// <summary>Boots both nodes and waits up to <paramref name="formationTimeout"/> for cluster convergence.</summary>
     /// <param name="formationTimeout">Maximum time to wait for cluster formation; defaults to 20 seconds if not provided.</param>
     /// <param name="driverFactory">Optional opt-in <see cref="IDriverFactory"/> both nodes register so deployed
@@ -131,7 +182,12 @@ public sealed class TwoNodeClusterHarness : IAsyncDisposable
         if (harness.Mode.UseSqlServer)
         {
             harness._sqlDbName = $"OtOpcUa_Harness_{Guid.NewGuid():N}";
-            harness._sqlConnString = $"Server=localhost,14331;Database={harness._sqlDbName};User Id=sa;Password=OtOpcUa!Harness123;TrustServerCertificate=True;";
+            // SQL host is overridable so the heavy 2-node E2E tests can target NATIVE-amd64 SQL on the
+            // shared Docker host instead of the arm64 Mac's emulated mssql:2022 (no arm64 image → ~5-10×
+            // slower → the deploy/failover E2E tests time out). Set OTOPCUA_HARNESS_SQL_HOST=10.100.0.35,14331
+            // with the harness sql container brought up there. Defaults to localhost,14331.
+            var sqlHost = Environment.GetEnvironmentVariable("OTOPCUA_HARNESS_SQL_HOST") ?? "localhost,14331";
+            harness._sqlConnString = $"Server={sqlHost};Database={harness._sqlDbName};User Id=sa;Password=OtOpcUa!Harness123;TrustServerCertificate=True;";
             await EnsureSqlSchemaCreatedAsync(harness._sqlConnString);
         }
 
@@ -295,7 +351,8 @@ public sealed class TwoNodeClusterHarness : IAsyncDisposable
             // Bound section is Security:Ldap (see LdapOptions.SectionName); Transport replaces the
             // old UseTls bool and AllowInsecure replaces AllowInsecureLdap (Task 1.4).
             configOverrides["Security:Ldap:Enabled"] = "true";
-            configOverrides["Security:Ldap:Server"] = "localhost";
+            // Overridable alongside the SQL host so the GLAuth container can live on the Docker host too.
+            configOverrides["Security:Ldap:Server"] = Environment.GetEnvironmentVariable("OTOPCUA_HARNESS_LDAP_HOST") ?? "localhost";
             configOverrides["Security:Ldap:Port"] = "3894";
             configOverrides["Security:Ldap:Transport"] = "None";
             configOverrides["Security:Ldap:AllowInsecure"] = "true";