feat(fleet): reconcile ClusterNode dial targets against live membership

ClusterNode.AkkaPort and the node's own Cluster:Port are the same fact in two
places and nothing made them agree. Phase 2 dials the row instead of gossiping,
so a node binding 4054 while its row says 4053 becomes unreachable from central
— and the symptom is a silent absence of acks rather than an error. That is the
shape of the Modbus/ModbusTcp and TwinCat/Focas drifts already in this repo.
Since Phase 1 also made the rows the deploy path's expected-ack set, drift
already costs a failed deployment today.

Implemented as the plan's preferred option: an admin-role singleton comparing
rows against the membership an admin node can already see, rather than each
driver node asserting its own row — Phase 4 removes the driver nodes' ConfigDb
connection, so a self-assertion written there would have to be deleted again.

Three shapes, split by severity: a row whose dial target disagrees with its own
NodeId and a running node with no row are Errors; an enabled row with no
matching member is a Warning, because a node down for maintenance is a
legitimate state. Findings are logged only when the set changes — a check that
reprints the same warning every sweep trains operators to filter it out.

Documented limitation: Phase 2 must revisit this. Once the fleet splits into
one mesh per cluster an admin node cannot see site members, and every site row
would report EnabledRowNotInCluster forever.

Sabotage: removing the change-detection guard turns the repeat test red.
ControlPlane.Tests 100/100.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
2026-07-22 08:36:44 -04:00
parent d88e245503
commit 90c1302f71
6 changed files with 511 additions and 3 deletions
@@ -23,6 +23,7 @@ public static class ServiceCollectionExtensions
public const string AuditWriterSingletonName = "audit-writer";
public const string FleetStatusSingletonName = "fleet-status";
public const string RedundancyStateSingletonName = "redundancy-state";
public const string ClusterNodeAddressReconcilerSingletonName = "cluster-node-address-reconciler";
/// <summary>
/// Registers all five admin-role cluster singletons + their proxies on the AkkaConfigurationBuilder.
@@ -90,6 +91,18 @@ public static class ServiceCollectionExtensions
(system, registry, resolver) => RedundancyStateActor.Props(),
singletonOptions);
// Per-cluster mesh Phase 1: ClusterNode.AkkaPort duplicates the node's own Cluster:Port and
// nothing makes them agree. Runs here rather than on each driver node because Phase 4 removes
// the driver nodes' ConfigDb connection entirely.
builder.WithSingleton<ClusterNodeAddressReconcilerKey>(
ClusterNodeAddressReconcilerSingletonName,
(system, registry, resolver) =>
{
var dbFactory = resolver.GetService<IDbContextFactory<OtOpcUaConfigDbContext>>();
return ClusterNodeAddressReconcilerActor.Props(dbFactory);
},
singletonOptions);
return builder;
}
}
@@ -100,3 +113,4 @@ public sealed class AdminOperationsActorKey { }
public sealed class AuditWriterActorKey { }
public sealed class FleetStatusBroadcasterKey { }
public sealed class RedundancyStateActorKey { }
public sealed class ClusterNodeAddressReconcilerKey { }