fix(tests): three real failures the integration suite had stopped catching

None of these were fixture absence. Each is a guard that went quietly wrong and
then reported the wrong thing.

PrimaryGateFailoverTests — per-cluster mesh Phase 6 (a2c5d57f) moved
RedundancyStateActor out of WithOtOpcUaControlPlaneSingletons into its own
cluster-role-scoped registration, made from Program.cs's DRIVER branch. The
harness models a fused admin,driver node but was never updated, so from that
commit onward NO node it built spawned the singleton: nothing published
redundancy-state, both nodes sat at "role unknown" forever, and the test polled
45s and timed out. The test's own summary calls this its negative control for a
broken redundancy-topic subscribe — it fired correctly, at the harness rather
than at production. Second time this shape has bitten the harness; the
AddOtOpcUaRuntime note a few lines up records the first.

DriverProbeRegistrationTests — the expected driver set was a hand-written array
kept in sync by comment, and it drifted when Sql (9b5a9687) and Mqtt (f79d13e2)
shipped. It failed as "12 where 10 expected", which names neither driver and
reads as if the probes were wrong. It also could not catch the direction that
matters: a driver type registered with no probe passed silently, because a
hand-written list only grows when someone remembers, and this test exists for
the case where nobody did. Now derived from DriverTypeNames.All, both
directions, plus an explicit case-drift test for the two types whose AdminUI key
differs in case from the constant (TwinCat/TwinCAT, Focas/FOCAS) — which the
derived tests cannot cover, since they compare the constants to themselves.

DriverProbeHandshakeE2eTests.AbCip_Green_AgainstSim — the probe config still used
the retired v2 `Tags: [{DeviceHostAddress, TagPath}]` shape, so
AbCipDriverOptions.RawTags bound empty, the probe fell through to its
@raw_cpu_type system-tag fallback, and ab_server answered ErrorBadParam. The test
has therefore spent the whole v3 migration exercising the one path its own
comment says it does not exercise. Rewritten to the v3 RawTags shape, and the
assertion tightened from ShouldContain("CIP session OK") to an exact match:
the probe's two green messages both contain that substring, so the loose form
would go green again the moment the config merely reached the controller,
whether or not the authored tag resolved.

Falsified each: drop the MTConnect probe (2 red, by name), remove the harness
singleton registration (back to the 48s "role unknown" timeout), break the
RawTags key (back to ErrorBadParam).

Host.IntegrationTests 211 passed / 4 skipped / 0 failed, against live fixtures on
10.100.0.35.

Still red and NOT addressed here: the AbLegacy integration suite (4 tests). Its
fixture is genuinely down, but it hard-fails instead of skipping because the
AbCip ControlLogix container holds :44818 — the reachability guard cannot tell
"my ab_server" from "someone else's", so SkipReason stays null and SLC500-mode
PCCC runs against a ControlLogix. Needs either distinct fixture ports or a
family-aware guard; both are shared-infrastructure changes.

Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
This commit is contained in:
Joseph Doherty
2026-08-01 02:09:15 -04:00
parent 06c56d7719
commit df340b8bba
3 changed files with 88 additions and 25 deletions
@@ -178,11 +178,26 @@ public sealed class DriverProbeHandshakeE2eTests
// ControlLogix instead returns ErrorNotFound, which the probe classifies as
// reachable). Whether the `@raw_cpu_type` system-tag fallback is valid on a real
// ControlLogix is a hardware-gated follow-up (AbCipDriverOptions.cs flags it deferred).
// ⚠️ The config must use the v3 `RawTags` shape — `[{RawPath, TagConfig, WriteIdempotent,
// DeviceName}]`, where the Logix tag path lives inside the TagConfig blob as `tagPath`. This
// test passed the retired v2 `Tags: [{DeviceHostAddress, TagPath}]` shape until 2026-08-01, so
// AbCipDriverOptions.RawTags bound EMPTY, the probe fell through to its `@raw_cpu_type` system-tag
// fallback, and ab_server answered ErrorBadParam. Which is to say it spent that whole time
// exercising the one path the comment above says it does not exercise — and reporting it as a
// failure of the fixture rather than of the config shape.
var result = await new AbCipDriverProbe().ProbeAsync(
$"{{\"Devices\":[{{\"HostAddress\":\"ab://{DockerHost}:{AbCipPort}/1,0\"}}]," +
$"\"Tags\":[{{\"DeviceHostAddress\":\"ab://{DockerHost}:{AbCipPort}/1,0\",\"TagPath\":\"TestDINT\"}}]}}",
$"\"RawTags\":[{{\"RawPath\":\"Plant/AbCip/dev1/TestDINT\"," +
$"\"TagConfig\":\"{{\\\"tagPath\\\":\\\"TestDINT\\\",\\\"dataType\\\":\\\"DInt\\\"}}\"," +
$"\"WriteIdempotent\":false,\"DeviceName\":\"ab://{DockerHost}:{AbCipPort}/1,0\"}}]}}",
Timeout, Ct);
result.Ok.ShouldBeTrue($"Probe message: {result.Message}");
result.Message!.ShouldContain("CIP session OK");
// EXACT, not ShouldContain. The probe's two green messages are "CIP session OK" (the tag was
// read) and "CIP session OK (controller reachable; probe tag not found)" (the fallback), and
// `ShouldContain` matches both — so it would have gone green again the moment the config shape
// was merely well-formed enough to reach the controller, whether or not the authored tag was
// ever resolved. Reading the authored tag is the entire point of this test.
result.Message.ShouldBe("CIP session OK");
}
}
@@ -14,29 +14,20 @@ namespace ZB.MOM.WW.OtOpcUa.Host.IntegrationTests;
/// admin path — not only the driver path — or every Test Connect button returns
/// "No probe registered for driver type X".
/// </summary>
/// <remarks>
/// ⚠️ <b>The expected set is derived from <see cref="DriverTypeNames.All"/>, not hand-written.</b>
/// It used to be a literal array kept "in sync" by comment, and it drifted the moment `Sql`
/// (<c>9b5a9687</c>) and `Mqtt` (<c>f79d13e2</c>) shipped: the count assertion below went red
/// reporting <i>12 where 10 expected</i> — a true statement that named neither driver and read like
/// the probes were wrong rather than the list. Worse is the direction it could NOT catch: a new
/// driver type registered with <b>no</b> probe passed silently, because a hand-written list grows
/// only when someone remembers to grow it, and this test exists precisely for the case where nobody
/// did. Deriving from the constants makes the guard bidirectional and self-maintaining.
/// </remarks>
public sealed class DriverProbeRegistrationTests
{
// The canonical "all drivers" set — one entry per AdminUI typed driver page's DriverTypeKey.
// Keep in sync with the IDriverProbe registrations in
// src/Server/.../Host/Drivers/DriverFactoryBootstrap.AddOtOpcUaDriverProbes (the routed
// *DriverPage.razor flow was retired in v3 Batch 2 — the driver-type keys now live in the /raw
// modals, but the probe set is the authority for what the AdminUI can Test-connect).
private static readonly string[] AdminUiDriverTypeKeys =
[
"Modbus",
"AbCip",
"AbLegacy",
"S7",
"TwinCat", // page key; probe reports "TwinCAT" — must resolve case-insensitively
"Focas", // page key; probe reports "FOCAS" — must resolve case-insensitively
"OpcUaClient",
"GalaxyMxGateway",
"Calculation", // v3 Batch 2 pseudo-driver; CalculationProbe registered in DriverFactoryBootstrap
"MTConnect", // MTConnectProbe — read-only Agent driver; probe issues a bare GET /probe
];
[Fact]
public void AddOtOpcUaDriverProbes_registers_a_probe_for_every_AdminUI_driver_type()
public void AddOtOpcUaDriverProbes_registers_a_probe_for_every_driver_type()
{
var services = new ServiceCollection();
services.AddOtOpcUaDriverProbes();
@@ -48,8 +39,50 @@ public sealed class DriverProbeRegistrationTests
// (case-insensitive) and would throw on a duplicate key, crashing the singleton.
var byType = probes.ToDictionary(p => p.DriverType, StringComparer.OrdinalIgnoreCase);
foreach (var key in AdminUiDriverTypeKeys)
byType.ContainsKey(key).ShouldBeTrue($"No IDriverProbe registered for AdminUI driver type '{key}'.");
var missing = DriverTypeNames.All.Where(t => !byType.ContainsKey(t)).ToList();
missing.ShouldBeEmpty(
"Driver types with no IDriverProbe — Test Connect returns 'No probe registered for driver "
+ $"type X' for each: {string.Join(", ", missing)}");
}
[Fact]
public void No_probe_is_registered_for_a_driver_type_that_does_not_exist()
{
// The other direction. A probe whose DriverType matches no DriverTypeNames constant can never
// be reached — AdminOperationsActor looks probes up by the authored DriverType — so it is a
// typo or a leftover from a renamed driver, and it reads as coverage while providing none.
var services = new ServiceCollection();
services.AddOtOpcUaDriverProbes();
using var sp = services.BuildServiceProvider();
var orphans = sp.GetServices<IDriverProbe>()
.Select(p => p.DriverType)
.Where(t => !DriverTypeNames.All.Contains(t, StringComparer.OrdinalIgnoreCase))
.ToList();
orphans.ShouldBeEmpty($"Probes for unknown driver types: {string.Join(", ", orphans)}");
}
/// <summary>
/// Two driver types are spelled differently by the constant and by the AdminUI's own key
/// (<c>TwinCAT</c>/<c>TwinCat</c>, <c>FOCAS</c>/<c>Focas</c>), so the lookup must fold case. The
/// derived tests above cannot cover this — they compare the constants against themselves.
/// </summary>
[Theory]
[InlineData("TwinCat", DriverTypeNames.TwinCAT)]
[InlineData("Focas", DriverTypeNames.FOCAS)]
[InlineData("modbus", DriverTypeNames.Modbus)]
public void A_probe_resolves_by_a_case_drifted_driver_type_key(string authoredKey, string canonical)
{
var services = new ServiceCollection();
services.AddOtOpcUaDriverProbes();
using var sp = services.BuildServiceProvider();
var byType = sp.GetServices<IDriverProbe>().ToDictionary(p => p.DriverType, StringComparer.OrdinalIgnoreCase);
byType.ShouldContainKey(authoredKey);
byType[authoredKey].DriverType.ShouldBe(canonical);
}
[Fact]
@@ -67,6 +100,6 @@ public sealed class DriverProbeRegistrationTests
var distinctTypes = probes.Select(p => p.DriverType).Distinct(StringComparer.OrdinalIgnoreCase).Count();
probes.Count.ShouldBe(distinctTypes, "Duplicate IDriverProbe registrations — TryAddEnumerable should de-dup.");
distinctTypes.ShouldBe(AdminUiDriverTypeKeys.Length);
distinctTypes.ShouldBe(DriverTypeNames.All.Count);
}
}
@@ -423,6 +423,21 @@ public sealed class TwoNodeClusterHarness : IAsyncDisposable
ab.WithOtOpcUaClusterBootstrap(sp);
ab.WithOtOpcUaControlPlaneSingletons();
ab.WithOtOpcUaRuntimeActors();
// Production fidelity, per-cluster mesh Phase 6 (`a2c5d57f`). RedundancyStateActor moved OUT
// of WithOtOpcUaControlPlaneSingletons and into its own cluster-role-scoped registration,
// which Program.cs makes from its DRIVER branch — a driver-only site pair has no admin node
// to elect its Primary. The harness models a fused admin,driver node but was never updated,
// so from that commit until now NO node here spawned the singleton: nothing published
// `redundancy-state`, every node sat at "role unknown" forever, and PrimaryGateFailoverTests
// polled for 45s and timed out.
//
// This is the second time this exact shape has bitten the harness — see the AddOtOpcUaRuntime
// note above, where calling only the Akka-spawn half left IDriverHealthPublisher on its Null
// default. When Program.cs grows an actor registration, this list has to grow with it; the
// harness's whole value is that it is wired the way a real node is.
ab.WithOtOpcUaClusterRedundancySingleton(
sp.GetRequiredService<Microsoft.Extensions.Options.IOptions<AkkaClusterOptions>>().Value);
});
builder.Services.AddOtOpcUaAuth(builder.Configuration);