dohertj2/lmxopcua
1
0
Fork 0
Code Issues 9 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(deploy): coordinator sources its expected-ack set from ClusterNode rows

Browse Source 
Cuts ConfigPublishCoordinator's last genuinely mesh-bound dependency: central
must be able to name the nodes a deployment is for without sharing a gossip
ring with them, because Phase 2 splits the fleet into one mesh per Cluster.

Behaviour change, confirmed before implementing: a configured node that is
switched off is now expected, so deploying while it is down fails at the apply
deadline instead of sealing green without it. Under the membership rule the
operator was told the fleet was deployed when it was not. ClusterNode.Enabled
= 0 is the maintenance hatch, and the deadline log now names the silent nodes
and points at that hatch — "4/5 acks landed" would leave an operator reading
logs on five machines.

Two assumptions are now documented on the class rather than left implicit:
every ClusterNode row is a driver node (the DB has no role column and
deliberately does not gain one — it would drift from Cluster:Roles), and
ServerCluster.Enabled is not consulted because nothing else consults it.

Dropped from the plan: cluster-scope filtering of the expected set. There is
no cluster-scoped deployment to filter on — Deployment has no ClusterId,
ConfigComposer always snapshots the whole DB, and ResolveClusterScope is
node-side self-scoping of a fleet-wide artifact.

Positive control: reverting DiscoverDriverNodes to the membership scan turns
three of the four new tests red. The fourth pins the seal-empty branch and is
documented as derivation-insensitive rather than left to look like coverage.

ControlPlane.Tests 90/90.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
2026-07-22 08:30:46 -04:00
parent da74ebd696
commit d88e245503
2 changed files with 290 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Coordinators/ConfigPublishCoordinator.cs
+71 -25
View File
@@ -1,5 +1,4 @@
using Akka.Actor;
using Akka.Cluster;
using Akka.Cluster.Tools.PublishSubscribe;
using Akka.Event;
using Microsoft.EntityFrameworkCore;
@@ -18,8 +17,32 @@ namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Coordinators;
/// has acked Applied. Per-node ACKs are persisted in <c>NodeDeploymentState</c> so a failover of
/// this singleton can recover in-flight state from the DB.
///
/// Discovery of the "expected ACK set" comes from <c>Akka.Cluster.State.Members</c> filtered by
/// the <c>driver</c> role — the DB does not own per-node role assignment.
/// Discovery of the "expected ACK set" comes from the <b>enabled <c>ClusterNode</c> rows</b>, not
/// from <c>Akka.Cluster.State.Members</c> (per-cluster mesh Phase 1). Central must be able to name
/// the nodes a deployment is for without sharing a gossip ring with them — Phase 2 splits the fleet
/// into one mesh per <c>Cluster</c>, after which central can no longer see a site node's membership
/// at all. This was the coordinator's last genuinely mesh-bound dependency.
///
/// <para>
/// <b>Behaviour change:</b> a configured node that is switched off is now <i>expected</i>, so a
/// deployment dispatched while it is down fails at the apply deadline instead of sealing green
/// without it. That is deliberate — under the membership rule the operator was told the fleet was
/// deployed when it was not. <c>ClusterNode.Enabled = 0</c> is the escape hatch for a node taken
/// down for maintenance.
/// </para>
/// <para>
/// <b>Every <c>ClusterNode</c> row is assumed to be a driver node.</b> The membership rule filtered
/// on the <c>driver</c> role; the DB has no per-node role column, and deliberately does not gain one
/// here (a second declaration of node roles would drift from <c>Cluster:Roles</c> in the node's own
/// appsettings). The assumption holds because the entity's whole shape — <c>Host</c>,
/// <c>OpcUaPort</c>, <c>ApplicationUri</c>, <c>ServiceLevelBase</c> — describes an OPC UA server
/// node. An admin-only node must not be given a <c>ClusterNode</c> row: it would be expected to ack
/// a deployment it has no <c>DriverHostActor</c> to apply, and every deploy would time out.
/// </para>
/// <para>
/// <c>ServerCluster.Enabled</c> is <b>not</b> consulted — nothing else in the codebase consults it,
/// so honouring it here would invent semantics. Disable the nodes, not the cluster row.
/// </para>
/// </summary>
public sealed class ConfigPublishCoordinator : ReceiveActor, IWithTimers
{
@@ -119,11 +142,15 @@ public sealed class ConfigPublishCoordinator : ReceiveActor, IWithTimers
{
_current = msg.DeploymentId;
_acks.Clear();
_expectedAcks = DiscoverDriverNodes();
// Seed NodeDeploymentState rows so a failover knows which nodes were expected to ack.
using (var db = _dbFactory.CreateDbContext())
{
// Discovery reuses this context rather than opening a second one — it reads the same
// ClusterNode table the NodeDeploymentState rows below are FK-bound to, so a single
// context keeps the expected set and the seeded rows consistent by construction.
_expectedAcks = DiscoverDriverNodes(db);
foreach (var node in _expectedAcks)
{
db.NodeDeploymentStates.Add(new NodeDeploymentState
@@ -142,9 +169,14 @@ public sealed class ConfigPublishCoordinator : ReceiveActor, IWithTimers
if (_expectedAcks.Count == 0)
{
// No driver-role members. Seal immediately — the alternative is hanging forever
// waiting for ACKs that will never come.
_log.Warning("DispatchDeployment {Id}: no driver-role members in cluster; sealing empty",
// No enabled ClusterNode rows. Seal immediately — the alternative is hanging forever
// waiting for ACKs that will never come. Note this is now a *configuration* error
// rather than a topology observation: under the old membership rule an empty set meant
// "no driver nodes have joined yet", which could resolve itself; an empty set here means
// the fleet has no enabled nodes registered, which cannot.
_log.Warning(
"DispatchDeployment {Id}: no enabled ClusterNode rows registered; sealing empty. " +
"Nothing will receive this deployment — check the fleet's node configuration",
msg.DeploymentId);
SealDeployment();
}
@@ -233,8 +265,15 @@ public sealed class ConfigPublishCoordinator : ReceiveActor, IWithTimers
using var db = _dbFactory.CreateDbContext();
UpdateDeploymentStatus(db, _current.Value, DeploymentStatus.TimedOut);
db.SaveChanges();
_log.Warning("Deployment {Id} timed out after {Deadline} ({Acked}/{Total} acks landed)",
_current.Value, _applyDeadline, _acks.Count, _expectedAcks.Count);
// Name the silent nodes, not just the counts. Since the expected set became DB-derived this
// is the failure an operator meets after deploying while a configured node is switched off,
// so it has to say which node — "4/5 acks landed" leaves them reading logs on five machines.
var missing = _expectedAcks.Where(n => !_acks.ContainsKey(n)).Select(n => n.Value).Order().ToList();
_log.Warning(
"Deployment {Id} timed out after {Deadline} ({Acked}/{Total} acks landed). " +
"No ack from: {MissingNodes}. Each is an enabled ClusterNode row — start the node, or " +
"set ClusterNode.Enabled = 0 if it is down for maintenance, then redeploy",
_current.Value, _applyDeadline, _acks.Count, _expectedAcks.Count, string.Join(", ", missing));
ResetForNext();
}
@@ -259,22 +298,29 @@ public sealed class ConfigPublishCoordinator : ReceiveActor, IWithTimers
if (sealNow) d.SealedAtUtc = DateTime.UtcNow;
}
private HashSet<NodeId> DiscoverDriverNodes()
{
var cluster = Akka.Cluster.Cluster.Get(Context.System);
var nodes = new HashSet<NodeId>(NodeIdComparer);
foreach (var member in cluster.State.Members)
{
if (member.Status is not (MemberStatus.Up or MemberStatus.Joining)) continue;
if (!member.Roles.Contains("driver")) continue;
var host = member.Address.Host;
if (string.IsNullOrWhiteSpace(host)) continue;
// Match ClusterRoleInfo's NodeId derivation (host:port) so DriverHostActor's
// self-identification and the coordinator's expected-ack set agree.
nodes.Add(NodeId.Parse($"{host}:{member.Address.Port ?? 0}"));
}
return nodes;
}
/// <summary>
/// The set of nodes expected to ack this deployment: every <b>enabled</b> <c>ClusterNode</c>
/// row. See the class remarks for why this is DB-derived rather than membership-derived, and
/// for the two assumptions it rests on (every row is a driver node; <c>Enabled</c> is the
/// maintenance hatch).
/// </summary>
/// <remarks>
/// <c>ClusterNode.NodeId</c> is already in the <c>host:port</c> identifier space the
/// membership rule derived — the rig seeds <c>central-1:4053</c> and so on, and
/// <c>NodeDeploymentState.NodeId</c> is FK-bound to this column, so the fact that the
/// membership-derived rows have always satisfied that FK is standing proof the two sets
/// agree. It also has to stay that way: <c>DriverHostActor</c> self-identifies from
/// <c>ClusterRoleInfo</c>'s <c>host:port</c>, so a row whose NodeId is anything else is a
/// node whose acks will never match its expected-ack entry.
/// </remarks>
private static HashSet<NodeId> DiscoverDriverNodes(OtOpcUaConfigDbContext db) =>
db.ClusterNodes
.AsNoTracking()
.Where(n => n.Enabled)
.Select(n => n.NodeId)
.ToList()
.Select(NodeId.Parse)
.ToHashSet(NodeIdComparer);
/// <summary>Case-insensitive <see cref="NodeId"/> equality (by <see cref="NodeId.Value"/>),
/// matching the case-insensitive scoping in <c>DeploymentArtifact.ResolveClusterScope</c> so the