feat(mqtt): Sparkplug primary-host STATE publishing, Primary-gated (#511)

`ActAsPrimaryHost` was accepted by config and by the AdminUI form and did nothing but
log a warning. It now delivers the whole three-part contract or none of it: a retained
`ONLINE` on `spBv1.0/STATE/{hostId}`, a matching retained `OFFLINE` armed as the MQTT
Last Will, and an explicit `OFFLINE` on a clean stand-down.

**The claim is a dedicated connection, not a message.** A Will can only be registered in
the CONNECT packet — it cannot be added to a live session, changed, or withdrawn. So a
claim that is acquired and released at runtime cannot ride the long-lived ingest session.
`SparkplugPrimaryHost` opens its own connection (client id `{clientId}-spbhost`, forced
clean-session) when the node gains Primary and closes it when the node loses Primary; the
connection's lifetime *is* the claim, so every leg follows from its lifecycle. Ingest is
untouched by a role change — no dropped subscriptions, no lost birth cache, no data gap at
the Secondary->Primary transition.

**Primary-gated** (owner decision, recorded on the issue). Both pair nodes subscribe
independently, so an ungated claim means both claim the same host id — and the sharp
failure is not the two fighting over the retained STATE, it is that the SECONDARY's death
fires its Will and publishes OFFLINE while the Primary is healthy, stopping every edge node
in the group. Registering the Will only on whichever node is Primary at connect time fails
the other way: a node that later gains Primary would publish ONLINE with no Will behind it.

New seam: `IDriverPrimaryRoleSource` (`Core.Abstractions`) carrying the *data-plane* verdict
(`PrimaryGatePolicy.ShouldServiceAsPrimary`), not the alarm-drain verdict — an unknown role
with a driver peer present must deny. Registered unconditionally by `AddOtOpcUaRuntime`,
deliberately NOT alongside `IRedundancyRoleView`, which `AddAlarmHistorian` registers only
when that section is enabled: piggybacking would leave a pair with no alarm historian
falling back to "claim unconditionally". Polled on a 1s timer rather than evented — a
process singleton raising events at per-instance drivers outlives every driver it holds.

Accepted trade: a brief edge-visible OFFLINE window during failover. That is a genuine
state transition, not a glitch to hide.

Live-gated 5/5 against the Mosquitto fixture, including the Will leg — `AbandonForTests`
drops the socket with no DISCONNECT, because a clean disconnect makes the broker discard
the Will by protocol rule, so a test driven through `StopAsync` would observe the
stand-down OFFLINE and pass against an implementation that never armed a Will. Falsified:
removing the Will turns that leg and only that leg red.

Also corrects two stale claims found en route: the AdminUI form still said MQTT tags are
read-only (#508 shipped), and the driver doc still said nothing consumes `IRediscoverable`.

Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
This commit is contained in:
Joseph Doherty
2026-07-31 07:17:17 -04:00
parent 2a7d11674f
commit 7f2108dcf2
21 changed files with 1881 additions and 87 deletions
@@ -273,6 +273,19 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
/// today, the alarm store-and-forward drain. Null on nodes that do not wire one.</summary>
private readonly IRedundancyRoleView? _redundancyRoleView;
/// <summary>
/// Where the Primary <b>data-plane</b> verdict is published for drivers that must know it
/// directly (Gitea #511). Null on nodes that wire no redundancy manager.
/// </summary>
/// <remarks>
/// Note the different verdict from <see cref="_redundancyRoleView"/>: that one carries
/// <see cref="PrimaryGatePolicy.ShouldDrainAlarmHistory"/>, which OPENS on an unknown role. A
/// Sparkplug host-id claim is an exclusive assertion on a shared broker, so it takes the
/// data-plane verdict, which closes. Publishing the drain verdict here would let both nodes of
/// a pair claim the host id during every boot window.
/// </remarks>
private readonly IDriverPrimaryRoleSource? _primaryRoleSource;
/// <summary>
/// Host of this node's LocalDb replication partner — the only node that holds a copy of this
/// node's alarm queue, and therefore the only node it may ever stand down in favour of.
@@ -408,7 +421,8 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
bool fetchAndCacheMode = false,
IDeploymentArtifactFetcher? artifactFetcher = null,
IAlarmStateStore? alarmStateStore = null,
ITelemetryLocalHub? telemetryHub = null) =>
ITelemetryLocalHub? telemetryHub = null,
IDriverPrimaryRoleSource? primaryRoleSource = null) =>
// WARNING: this forwarding list is POSITIONAL, and Props.Create compiles it into an
// expression tree. Six IActorRef? parameters and several interface-typed ones mean a
// mis-ordered argument is usually type-compatible and therefore compiles clean, then binds
@@ -419,7 +433,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
healthPublisher, virtualTagEvaluator, historyWriter, virtualTagHostOverride,
loggerFactory, scriptRootLogger, scriptedAlarmHostOverride, invokerFactory, driverMemberCountProvider,
deploymentArtifactCache, redundancyRoleView, replicationPeerHost, fetchAndCacheMode, artifactFetcher,
alarmStateStore, telemetryHub));
alarmStateStore, telemetryHub, primaryRoleSource));
/// <summary>Initializes a new DriverHostActor with the specified dependencies.</summary>
/// <param name="dbFactory">Database context factory for configuration database access.</param>
@@ -460,6 +474,12 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
/// <param name="telemetryHub">Per-cluster mesh Phase 5: optional node-local live-telemetry hub each
/// Primary-gated native-alarm transition is also emitted into, and which is threaded into the spawned
/// VirtualTag + ScriptedAlarm hosts. Null (admin-only nodes / tests) skips the emit and passes null down.</param>
/// <param name="primaryRoleSource">Gitea #511: where the Primary <i>data-plane</i> verdict is published
/// for drivers that assert a standing exclusive identity on a shared external system — today the MQTT
/// driver's Sparkplug primary-host STATE claim. Deliberately a separate parameter from
/// <paramref name="redundancyRoleView"/> even though the production object implements both: a test that
/// supplies only a role-view double must not silently also satisfy this, or its primary-role assertions
/// pass without the publish ever happening. Null on nodes that wire no redundancy manager.</param>
public DriverHostActor(
IDbContextFactory<OtOpcUaConfigDbContext>? dbFactory,
CommonsNodeId localNode,
@@ -483,11 +503,13 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
bool fetchAndCacheMode = false,
IDeploymentArtifactFetcher? artifactFetcher = null,
IAlarmStateStore? alarmStateStore = null,
ITelemetryLocalHub? telemetryHub = null)
ITelemetryLocalHub? telemetryHub = null,
IDriverPrimaryRoleSource? primaryRoleSource = null)
{
_telemetryHub = telemetryHub;
_deploymentArtifactCache = deploymentArtifactCache;
_redundancyRoleView = redundancyRoleView;
_primaryRoleSource = primaryRoleSource;
_replicationPeerHost = replicationPeerHost;
_fetchAndCacheMode = fetchAndCacheMode;
_artifactFetcher = artifactFetcher;
@@ -1331,6 +1353,12 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
_redundancyRoleView?.Publish(
PrimaryGatePolicy.ShouldDrainAlarmHistory(_localRole, queueSharingPeerIsPrimary: peerIsPrimary));
// The DATA-PLANE verdict, deliberately — the same one HandleRouteNodeWrite gates on, not the
// drain verdict published immediately above. A driver reading this claims an exclusive identity
// on a shared external system (Gitea #511: the Sparkplug host id), where two claimants is the
// harm, so an unknown role with a driver peer present must deny.
_primaryRoleSource?.PublishPrimaryRole(ShouldServiceAsPrimary());
}
private void Stale()
@@ -1,3 +1,4 @@
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Runtime.Drivers;
namespace ZB.MOM.WW.OtOpcUa.Runtime.Redundancy;
@@ -39,8 +40,22 @@ public interface IRedundancyRoleView
void Publish(bool shouldDrainAlarmHistory);
}
/// <summary>Thread-safe <see cref="IRedundancyRoleView"/> backed by a volatile field.</summary>
public sealed class RedundancyRoleView : IRedundancyRoleView
/// <summary>
/// Thread-safe <see cref="IRedundancyRoleView"/> + <see cref="IDriverPrimaryRoleSource"/>, each
/// backed by its own volatile field.
/// </summary>
/// <remarks>
/// <para>
/// <b>Two verdicts, one object, on purpose.</b> Both are published from the same
/// <c>DriverHostActor</c> snapshot handler and both are read off-actor, so a second class
/// would duplicate the publish plumbing to hold a second bool. They are kept as distinct
/// <i>properties</i> rather than collapsed into one because they are genuinely different
/// decisions from different policies — see <see cref="IRedundancyRoleView"/>'s remarks on why
/// the drain gate must open on an unknown role where the data-plane gate closes. Collapsing
/// them is the bug that live gate already found once.
/// </para>
/// </remarks>
public sealed class RedundancyRoleView : IRedundancyRoleView, IDriverPrimaryRoleSource
{
// Seeded with the decision for "role unknown" rather than a bare false. An unpublished view and a
// node whose role nothing ever reports are genuinely the same situation, and they must behave the
@@ -49,9 +64,22 @@ public sealed class RedundancyRoleView : IRedundancyRoleView
private volatile bool _shouldDrainAlarmHistory =
PrimaryGatePolicy.ShouldDrainAlarmHistory(localRole: null, queueSharingPeerIsPrimary: false);
// Seeded from the same policy that will later publish here, for the unknown-role/no-peer case:
// a node nothing has reported a role for, on a cluster with no driver peer, is a single-node
// deployment and may claim. The moment a snapshot arrives this is replaced by the real verdict,
// and a multi-driver cluster's unknown role denies.
private volatile bool _shouldActAsPrimary =
PrimaryGatePolicy.ShouldServiceAsPrimary(localRole: null, driverMemberCount: 1);
/// <inheritdoc/>
public bool ShouldDrainAlarmHistory => _shouldDrainAlarmHistory;
/// <inheritdoc/>
public bool ShouldActAsPrimary => _shouldActAsPrimary;
/// <inheritdoc/>
public void Publish(bool shouldDrainAlarmHistory) => _shouldDrainAlarmHistory = shouldDrainAlarmHistory;
/// <inheritdoc/>
public void PublishPrimaryRole(bool shouldActAsPrimary) => _shouldActAsPrimary = shouldActAsPrimary;
}
@@ -75,6 +75,22 @@ public static class ServiceCollectionExtensions
// telemetry (never DPS) to the Phase-5 gRPC streaming service central dials in on. AddOtOpcUaRuntime
// runs only inside Program.cs's hasDriver block, so this lands on driver-role nodes only.
services.TryAddSingleton<ITelemetryLocalHub, TelemetryLocalHub>();
// Gitea #511: the Primary data-plane verdict, published for drivers that assert a standing
// exclusive identity on a shared external system (today the MQTT driver's Sparkplug host id).
//
// Registered HERE, unconditionally on every driver-role node, and NOT alongside the alarm
// drain's IRedundancyRoleView — which AddAlarmHistorian registers only when that section is
// enabled. Piggybacking on it would leave a redundant pair with no alarm historian configured
// with no primary-role source at all, and the driver's documented no-source fallback is
// "claim" — so both nodes of that pair would claim the same host id and their Wills would
// fight over the retained STATE. Exactly the failure #511 exists to prevent, reached by a
// registration accident in an unrelated section.
//
// Both interfaces are forwarded to ONE RedundancyRoleView instance, so a node that has both
// publishes two verdicts into the same object rather than into two that disagree.
services.TryAddSingleton<RedundancyRoleView>();
services.TryAddSingleton<IDriverPrimaryRoleSource>(sp => sp.GetRequiredService<RedundancyRoleView>());
return services;
}
@@ -117,8 +133,10 @@ public static class ServiceCollectionExtensions
// The view is a plain singleton so it exists even on nodes whose DriverHostActor never
// publishes to it — an unpublished view reads as "no peer, drain", which is the correct
// posture for a deployment that runs no redundancy at all.
services.TryAddSingleton<IRedundancyRoleView, RedundancyRoleView>();
// posture for a deployment that runs no redundancy at all. Forwarded to the same instance
// AddOtOpcUaRuntime registered for IDriverPrimaryRoleSource: one object, two verdicts.
services.TryAddSingleton<RedundancyRoleView>();
services.TryAddSingleton<IRedundancyRoleView>(sp => sp.GetRequiredService<RedundancyRoleView>());
// THE GATE ONLY APPLIES TO A REPLICATED QUEUE. Standing down is only ever safe because some
// other node holds the same rows and will send them instead — and the only node that does is
@@ -499,7 +517,11 @@ public static class ServiceCollectionExtensions
// Phase 5: the node-local live-telemetry hub (registered in AddOtOpcUaRuntime). The host
// emits its Primary-gated native-alarm transitions into it and threads it into the
// VirtualTag + ScriptedAlarm hosts it spawns.
telemetryHub: resolver.GetService<ITelemetryLocalHub>()),
telemetryHub: resolver.GetService<ITelemetryLocalHub>(),
// Gitea #511: where this actor publishes its Primary DATA-PLANE verdict for drivers
// that claim an exclusive external identity. Registered unconditionally by
// AddOtOpcUaRuntime, unlike redundancyRoleView above — see the comment there.
primaryRoleSource: resolver.GetService<IDriverPrimaryRoleSource>()),
DriverHostActorName);
registry.Register<DriverHostActorKey>(driverHost);