06c56d7719
Two independent defects surfaced in review of #520 stage 3.
#532 — Cluster:Roles had no OTOPCUA_ROLES fallback, despite two doc sites
promising one. A node configured only through the env var — which drives ALL of
Program.cs's conditional wiring, so it reads as the authoritative knob — bound an
empty Roles array and joined the cluster carrying no Akka roles at all. It ran,
served, and logged nothing wrong, while central reported "no driver-role nodes
are Up" and every deployment to it timed out. Not hypothetical: the field fix on
wonder-app-vd03 was to hand-add Cluster__Roles__0/1 to a Windows service's
registry environment. Anyone reading the code to diagnose that box would have
concluded from the XML doc that the fallback was already applied.
AddOtOpcUaCluster now post-configures AkkaClusterOptions through the shared
RoleParser.ResolveEffectiveRoles, so BuildClusterOptions (Akka's own member
roles), ClusterRoleInfo, the ControlPlane singleton scope and
ConfigDbApiKeyStore's ClusterId all see one answer. Two sibling validators
(ConfigSource, Telemetry) read Cluster:Roles directly and were exempting the
exact node they exist for — an env-only driver-only node has no ConfigDb and
could sit on ConfigSource:Mode=Direct unchallenged. They resolve the same way now.
Also collapsed the four open-coded `Roles.FirstOrDefault(IsClusterRole)` +
ClusterIdFromRole copies into RoleParser.ClusterIdFromRoles. They had drifted:
three classified the cluster role ordinally, so a verbatim-bound `Cluster-SITE-A`
yielded a null ClusterId — and a null ClusterId applies NO cluster filter, which
made #528's cross-cluster over-grant guard fail OPEN. IsClusterRole is now
case-insensitive; the ClusterId suffix keeps its authored spelling, because
normalizing it would silently rename the Akka role of a running deployment.
#533 — three string-identity rules in one authorization path. AclScopeRef used
default (ordinal) equality while the trie builder filtered ClusterId and hashed
principals OrdinalIgnoreCase; the deploy gate's cross-cluster check was ordinal
too, which was the worse half — it could reject a whole deploy over a ClusterId
case difference the runtime would have accepted. It only worked because resolver
and builder were handed the same NodeAcl instances.
The rule is now stated once, on AclScopeRef, and applied throughout:
identifiers (ScopeId, ClusterId, Principal) compare OrdinalIgnoreCase; path
segments compare Ordinal. AclScopePathResolver re-keys its own maps rather than
documenting a comparer contract for callers, so the two independent producers —
EF at the gate, artifact JSON on the node — cannot diverge from the outside.
PermissionTrieNode.Children stays Ordinal: those are browse names, and Speed /
speed are two nodes that can coexist under one device.
Split PermissionTrieBuildDiagnosticReason.RetiredScopeKind out of
UnresolvedScope. A legacy FolderSegment row and a typo'd id had the same message
and different remedies.
Falsified: five mutations, each reddening only its own guard — remove the
PostConfigure (3 tests), revert IsClusterRole to ordinal (3), revert AclScopeRef
equality (1), revert the resolver's map comparer (6), revert the gate's
comparisons (2), remove the FolderSegment short-circuit (1).
Test isolation: OTOPCUA_ROLES is process-global and xUnit runs classes in
parallel, so the three validator suites that now resolve effective roles share a
non-parallel collection. Without it, cases passing no roles would intermittently
observe another class's env var.
Full solution build clean; full suite green apart from two pre-existing failures
confirmed identical on 3a6b2a1d (PrimaryGateFailoverTests, and
DriverProbeRegistrationTests expecting 10 probe types where 12 now register)
plus the AbLegacy integration suite, which needs a docker fixture.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
253 lines
9.8 KiB
C#
253 lines
9.8 KiB
C#
using Microsoft.Extensions.Configuration;
|
|
using Microsoft.Extensions.Options;
|
|
using Shouldly;
|
|
using Xunit;
|
|
|
|
namespace ZB.MOM.WW.OtOpcUa.Cluster.Tests;
|
|
|
|
/// <summary>
|
|
/// A split-topology node (one carrying a <c>cluster-{ClusterId}</c> role) left on a DPS transport
|
|
/// mode does not error — the transport simply delivers nothing across the mesh boundary, and the
|
|
/// symptom is a deployment that never applies or telemetry that never arrives. These tests make
|
|
/// that misconfiguration a loud host-start failure instead. A node with no cluster-role is exempt
|
|
/// (legacy / single-mesh / test), and the validator is a no-op for it.
|
|
/// </summary>
|
|
[Collection(RoleEnvironmentCollection.Name)]
|
|
public class SplitTopologyTransportValidatorTests
|
|
{
|
|
private static ValidateOptionsResult Validate(
|
|
string? meshMode,
|
|
string? telemetryMode,
|
|
string? telemetryDialMode,
|
|
params string[] roles)
|
|
{
|
|
var pairs = new Dictionary<string, string?>();
|
|
for (var i = 0; i < roles.Length; i++)
|
|
{
|
|
pairs[$"Cluster:Roles:{i}"] = roles[i];
|
|
}
|
|
|
|
if (meshMode is not null) pairs["MeshTransport:Mode"] = meshMode;
|
|
if (telemetryMode is not null) pairs["Telemetry:Mode"] = telemetryMode;
|
|
if (telemetryDialMode is not null) pairs["TelemetryDial:Mode"] = telemetryDialMode;
|
|
|
|
var configuration = new ConfigurationBuilder().AddInMemoryCollection(pairs).Build();
|
|
|
|
// The validated options instance is never read (all four values come from IConfiguration), so a
|
|
// default MeshTransportOptions is a fine stand-in.
|
|
return new SplitTopologyTransportValidator(configuration)
|
|
.Validate(MeshTransportOptions.SectionName, new MeshTransportOptions());
|
|
}
|
|
|
|
[Fact]
|
|
public void Cluster_role_driver_node_on_Dps_mesh_transport_fails()
|
|
{
|
|
// Red-before-green anchor: a split-topology node on DistributedPubSub cannot deliver commands.
|
|
var result = Validate(
|
|
meshMode: MeshTransportOptions.ModeDps,
|
|
telemetryMode: TelemetryOptions.ModeGrpc,
|
|
telemetryDialMode: null,
|
|
"driver", "cluster-SITE-A");
|
|
|
|
result.Failed.ShouldBeTrue();
|
|
result.FailureMessage.ShouldContain("MeshTransport:Mode");
|
|
result.FailureMessage.ShouldContain(MeshTransportOptions.ModeClusterClient);
|
|
result.FailureMessage.ShouldContain("SITE-A");
|
|
}
|
|
|
|
[Fact]
|
|
public void Cluster_role_driver_node_fully_configured_passes()
|
|
{
|
|
Validate(
|
|
meshMode: MeshTransportOptions.ModeClusterClient,
|
|
telemetryMode: TelemetryOptions.ModeGrpc,
|
|
telemetryDialMode: null,
|
|
"driver", "cluster-SITE-A").Succeeded.ShouldBeTrue();
|
|
}
|
|
|
|
[Fact]
|
|
public void Cluster_role_admin_node_fully_configured_passes()
|
|
{
|
|
Validate(
|
|
meshMode: MeshTransportOptions.ModeClusterClient,
|
|
telemetryMode: null,
|
|
telemetryDialMode: TelemetryDialOptions.ModeGrpc,
|
|
"admin", "cluster-SITE-A").Succeeded.ShouldBeTrue();
|
|
}
|
|
|
|
[Fact]
|
|
public void Cluster_role_driver_node_on_Dps_telemetry_fails()
|
|
{
|
|
var result = Validate(
|
|
meshMode: MeshTransportOptions.ModeClusterClient,
|
|
telemetryMode: TelemetryOptions.ModeDps,
|
|
telemetryDialMode: null,
|
|
"driver", "cluster-SITE-A");
|
|
|
|
result.Failed.ShouldBeTrue();
|
|
result.FailureMessage.ShouldContain("Telemetry:Mode");
|
|
result.FailureMessage.ShouldContain(TelemetryOptions.ModeGrpc);
|
|
}
|
|
|
|
[Fact]
|
|
public void Cluster_role_admin_node_on_Dps_telemetry_dial_fails()
|
|
{
|
|
var result = Validate(
|
|
meshMode: MeshTransportOptions.ModeClusterClient,
|
|
telemetryMode: null,
|
|
telemetryDialMode: TelemetryDialOptions.ModeDps,
|
|
"admin", "cluster-SITE-A");
|
|
|
|
result.Failed.ShouldBeTrue();
|
|
result.FailureMessage.ShouldContain("TelemetryDial:Mode");
|
|
result.FailureMessage.ShouldContain(TelemetryDialOptions.ModeGrpc);
|
|
}
|
|
|
|
[Fact]
|
|
public void Fused_cluster_role_node_needs_all_three_to_pass()
|
|
{
|
|
Validate(
|
|
meshMode: MeshTransportOptions.ModeClusterClient,
|
|
telemetryMode: TelemetryOptions.ModeGrpc,
|
|
telemetryDialMode: TelemetryDialOptions.ModeGrpc,
|
|
"admin", "driver", "cluster-MAIN").Succeeded.ShouldBeTrue();
|
|
}
|
|
|
|
[Fact]
|
|
public void Fused_cluster_role_node_missing_telemetry_grpc_fails()
|
|
{
|
|
var result = Validate(
|
|
meshMode: MeshTransportOptions.ModeClusterClient,
|
|
telemetryMode: TelemetryOptions.ModeDps,
|
|
telemetryDialMode: TelemetryDialOptions.ModeGrpc,
|
|
"admin", "driver", "cluster-MAIN");
|
|
|
|
result.Failed.ShouldBeTrue();
|
|
result.FailureMessage.ShouldContain("Telemetry:Mode");
|
|
}
|
|
|
|
[Fact]
|
|
public void Fused_cluster_role_node_missing_telemetry_dial_grpc_fails()
|
|
{
|
|
var result = Validate(
|
|
meshMode: MeshTransportOptions.ModeClusterClient,
|
|
telemetryMode: TelemetryOptions.ModeGrpc,
|
|
telemetryDialMode: TelemetryDialOptions.ModeDps,
|
|
"admin", "driver", "cluster-MAIN");
|
|
|
|
result.Failed.ShouldBeTrue();
|
|
result.FailureMessage.ShouldContain("TelemetryDial:Mode");
|
|
}
|
|
|
|
[Fact]
|
|
public void Non_cluster_role_node_on_all_Dps_passes()
|
|
{
|
|
// The exemption that keeps the validator a no-op for every legacy / single-mesh / test node:
|
|
// no cluster-{ClusterId} role means the split-topology rule does not apply at all.
|
|
Validate(
|
|
meshMode: MeshTransportOptions.ModeDps,
|
|
telemetryMode: TelemetryOptions.ModeDps,
|
|
telemetryDialMode: TelemetryDialOptions.ModeDps,
|
|
"admin", "driver").Succeeded.ShouldBeTrue();
|
|
}
|
|
|
|
[Fact]
|
|
public void Node_with_no_roles_on_all_Dps_passes()
|
|
{
|
|
// Absolute default: nothing declared, nothing to enforce.
|
|
Validate(
|
|
meshMode: MeshTransportOptions.ModeDps,
|
|
telemetryMode: TelemetryOptions.ModeDps,
|
|
telemetryDialMode: TelemetryDialOptions.ModeDps).Succeeded.ShouldBeTrue();
|
|
}
|
|
|
|
[Fact]
|
|
public void Unset_modes_on_a_cluster_role_node_fail_as_the_Dps_default()
|
|
{
|
|
// A split node that never sets the mode keys inherits the Dps defaults — which the split
|
|
// topology cannot use. Leaving them unset must fail exactly as setting them to Dps would.
|
|
var result = Validate(
|
|
meshMode: null,
|
|
telemetryMode: null,
|
|
telemetryDialMode: null,
|
|
"admin", "driver", "cluster-MAIN");
|
|
|
|
result.Failed.ShouldBeTrue();
|
|
result.FailureMessage.ShouldContain("MeshTransport:Mode");
|
|
}
|
|
|
|
[Fact]
|
|
public void Mode_matching_is_case_insensitive()
|
|
{
|
|
Validate(
|
|
meshMode: "clusterclient",
|
|
telemetryMode: "grpc",
|
|
telemetryDialMode: "grpc",
|
|
"admin", "driver", "cluster-MAIN").Succeeded.ShouldBeTrue();
|
|
}
|
|
|
|
[Fact]
|
|
public void Cluster_role_from_OTOPCUA_ROLES_only_on_Dps_fails()
|
|
{
|
|
// Roles-source divergence: Cluster:Roles is empty, so the node's effective Akka roles come from
|
|
// OTOPCUA_ROLES (via RoleParser.Parse) exactly as Program.cs resolves them. A validator that
|
|
// read only Cluster:Roles would see no cluster role and pass silently on Dps — the documented
|
|
// production-incident shape.
|
|
using (new EnvVarScope("OTOPCUA_ROLES", "driver,cluster-SITE-A"))
|
|
{
|
|
var pairs = new Dictionary<string, string?> { ["MeshTransport:Mode"] = MeshTransportOptions.ModeDps };
|
|
var configuration = new ConfigurationBuilder().AddInMemoryCollection(pairs).Build();
|
|
|
|
var result = new SplitTopologyTransportValidator(configuration)
|
|
.Validate(MeshTransportOptions.SectionName, new MeshTransportOptions());
|
|
|
|
result.Failed.ShouldBeTrue();
|
|
result.FailureMessage.ShouldContain("MeshTransport:Mode");
|
|
}
|
|
}
|
|
|
|
[Fact]
|
|
public void Cluster_role_from_OTOPCUA_ROLES_is_ignored_when_Cluster_Roles_is_set()
|
|
{
|
|
// When Cluster:Roles is populated it wins outright — the OTOPCUA_ROLES fallback is not consulted,
|
|
// mirroring AkkaClusterOptions.Roles. A non-cluster Cluster:Roles therefore stays exempt even if
|
|
// a stray OTOPCUA_ROLES carries a cluster role.
|
|
using (new EnvVarScope("OTOPCUA_ROLES", "driver,cluster-SITE-A"))
|
|
{
|
|
Validate(
|
|
meshMode: MeshTransportOptions.ModeDps,
|
|
telemetryMode: MeshTransportOptions.ModeDps,
|
|
telemetryDialMode: MeshTransportOptions.ModeDps,
|
|
"admin", "driver").Succeeded.ShouldBeTrue();
|
|
}
|
|
}
|
|
|
|
[Fact]
|
|
public void Mixed_case_cluster_and_role_names_on_Dps_fail()
|
|
{
|
|
// Case-sensitivity: the Cluster:Roles bind path is verbatim, so mixed-case entries must be
|
|
// normalized here or they slip past the ordinal IsClusterRole / driver checks and pass silently.
|
|
var result = Validate(
|
|
meshMode: MeshTransportOptions.ModeDps,
|
|
telemetryMode: TelemetryOptions.ModeGrpc,
|
|
telemetryDialMode: null,
|
|
"Driver", "Cluster-SITE-A");
|
|
|
|
result.Failed.ShouldBeTrue();
|
|
result.FailureMessage.ShouldContain("MeshTransport:Mode");
|
|
}
|
|
|
|
[Fact]
|
|
public void Cluster_role_node_with_neither_admin_nor_driver_only_needs_ClusterClient()
|
|
{
|
|
// A cluster-role node that is neither admin nor driver has no telemetry surface to constrain —
|
|
// only the always-on MeshTransport=ClusterClient rule applies. Telemetry/TelemetryDial left on
|
|
// their Dps defaults must not fail it.
|
|
Validate(
|
|
meshMode: MeshTransportOptions.ModeClusterClient,
|
|
telemetryMode: null,
|
|
telemetryDialMode: null,
|
|
"cluster-SITE-A").Succeeded.ShouldBeTrue();
|
|
}
|
|
}
|