fix(mesh): also scope the reconciler's observed membership to the own cluster

Task 3 review: scoping only the rows/enabled DB queries by ClusterId left
the observed set (from Cluster.State.Members) filtered by the driver role
alone. In the mixed-mesh window — DB scoping live via a cluster role, but
the physical Akka mesh not yet split so central still sees foreign members
via gossip — a foreign-cluster driver member stays in observed while its row
is filtered out, hitting the RunningNodeHasNoRow branch, which logs at ERROR.
That traded the Warning-level false positive for a worse Error-level one.

Filter observed by the node's own cluster ROLE too. Extracted the projection
into a static FilterOwnClusterDriverMembers(members, ownClusterRole) so the
membership filter is unit-testable without seeding a live cluster (the actor
harness joins as admin only, so observed is always empty). Sourced at
registration from IClusterRoleInfo.ClusterRole alongside ClusterId. Null role
(legacy, no cluster role) keeps every driver member — reconcile-all, unchanged.

Also clarified the ownClusterId doc wording (non-null-and-non-empty).

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
2026-07-24 02:07:02 -04:00
parent a4d3ab4005
commit 1ce2042e14
4 changed files with 165 additions and 23 deletions
@@ -1,3 +1,8 @@
using System.Collections.Immutable;
using System.Reflection;
using Akka.Actor;
using Akka.Cluster;
using Akka.Util;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.ControlPlane.Fleet;
@@ -123,4 +128,90 @@ public sealed class ClusterNodeAddressReconcilerTests
[("SITE-A-1", 4053)], [Row("site-a-1", 4053)], ["site-a-1:4053"])
.ShouldBeEmpty();
}
private static int _memberUid;
// Akka's Member.Create factories are internal, so reflect the Up-status overload
// Create(UniqueAddress, int upNumber, MemberStatus, ImmutableHashSet<string> roles, AppVersion).
private static readonly MethodInfo MemberCreate = typeof(Member).GetMethod(
"Create",
BindingFlags.Static | BindingFlags.NonPublic | BindingFlags.Public,
binder: null,
types:
[
typeof(UniqueAddress), typeof(int), typeof(MemberStatus),
typeof(ImmutableHashSet<string>), typeof(AppVersion),
],
modifiers: null)
?? throw new InvalidOperationException("Akka.Cluster.Member.Create(5-arg) not found — Akka API changed");
/// <summary>
/// Builds an Up cluster <see cref="Member"/> at <paramref name="host"/>:<paramref name="port"/>
/// carrying <paramref name="roles"/>, for exercising
/// <see cref="ClusterNodeAddressReconcilerActor.FilterOwnClusterDriverMembers"/> directly.
/// </summary>
private static Member UpMember(string host, int port, params string[] roles) =>
(Member)MemberCreate.Invoke(null,
[
new UniqueAddress(new Address("akka.tcp", "otopcua", host, port), Interlocked.Increment(ref _memberUid)),
1,
MemberStatus.Up,
roles.ToImmutableHashSet(),
AppVersion.Zero,
])!;
/// <summary>
/// Per-cluster mesh Phase 6: scoped to <c>cluster-MAIN</c>, a driver member carrying a FOREIGN
/// cluster role (<c>cluster-SITE-A</c>) — the mixed-mesh window, where central still sees it via
/// gossip while its row is already filtered out by ClusterId — is excluded from observed. Left
/// in, it would hit the Error-level <see cref="AddressMismatchKind.RunningNodeHasNoRow"/> branch.
/// </summary>
[Fact]
public void Foreign_cluster_role_member_is_excluded_when_scoped()
{
var observed = ClusterNodeAddressReconcilerActor.FilterOwnClusterDriverMembers(
[UpMember("site-a-1", 4053, "driver", "cluster-SITE-A")],
ownClusterRole: "cluster-MAIN");
observed.ShouldBeEmpty();
}
/// <summary>Scoped to <c>cluster-MAIN</c>, a member carrying that same role is kept.</summary>
[Fact]
public void Own_cluster_role_member_is_kept_when_scoped()
{
var observed = ClusterNodeAddressReconcilerActor.FilterOwnClusterDriverMembers(
[UpMember("main-1", 4053, "driver", "cluster-MAIN")],
ownClusterRole: "cluster-MAIN");
observed.ShouldHaveSingleItem().ShouldBe(("main-1", 4053));
}
/// <summary>
/// Legacy: a null cluster role keeps every Up driver member regardless of cluster — an admin
/// node with no cluster role genuinely sees the whole fleet via gossip (reconcile-all).
/// </summary>
[Fact]
public void Null_cluster_role_keeps_all_driver_members()
{
var observed = ClusterNodeAddressReconcilerActor.FilterOwnClusterDriverMembers(
[
UpMember("main-1", 4053, "driver", "cluster-MAIN"),
UpMember("site-a-1", 4053, "driver", "cluster-SITE-A"),
],
ownClusterRole: null);
observed.Count.ShouldBe(2);
observed.ShouldContain(("main-1", 4053));
observed.ShouldContain(("site-a-1", 4053));
}
/// <summary>Non-driver members (e.g. admin-only) are excluded whether or not a scope is set.</summary>
[Fact]
public void Non_driver_members_are_always_excluded()
{
ClusterNodeAddressReconcilerActor.FilterOwnClusterDriverMembers(
[UpMember("central-1", 4053, "admin", "cluster-MAIN")], ownClusterRole: "cluster-MAIN")
.ShouldBeEmpty();
}
}