feat(mqtt): Sparkplug primary-host STATE publishing, Primary-gated (#511)

`ActAsPrimaryHost` was accepted by config and by the AdminUI form and did nothing but
log a warning. It now delivers the whole three-part contract or none of it: a retained
`ONLINE` on `spBv1.0/STATE/{hostId}`, a matching retained `OFFLINE` armed as the MQTT
Last Will, and an explicit `OFFLINE` on a clean stand-down.

**The claim is a dedicated connection, not a message.** A Will can only be registered in
the CONNECT packet — it cannot be added to a live session, changed, or withdrawn. So a
claim that is acquired and released at runtime cannot ride the long-lived ingest session.
`SparkplugPrimaryHost` opens its own connection (client id `{clientId}-spbhost`, forced
clean-session) when the node gains Primary and closes it when the node loses Primary; the
connection's lifetime *is* the claim, so every leg follows from its lifecycle. Ingest is
untouched by a role change — no dropped subscriptions, no lost birth cache, no data gap at
the Secondary->Primary transition.

**Primary-gated** (owner decision, recorded on the issue). Both pair nodes subscribe
independently, so an ungated claim means both claim the same host id — and the sharp
failure is not the two fighting over the retained STATE, it is that the SECONDARY's death
fires its Will and publishes OFFLINE while the Primary is healthy, stopping every edge node
in the group. Registering the Will only on whichever node is Primary at connect time fails
the other way: a node that later gains Primary would publish ONLINE with no Will behind it.

New seam: `IDriverPrimaryRoleSource` (`Core.Abstractions`) carrying the *data-plane* verdict
(`PrimaryGatePolicy.ShouldServiceAsPrimary`), not the alarm-drain verdict — an unknown role
with a driver peer present must deny. Registered unconditionally by `AddOtOpcUaRuntime`,
deliberately NOT alongside `IRedundancyRoleView`, which `AddAlarmHistorian` registers only
when that section is enabled: piggybacking would leave a pair with no alarm historian
falling back to "claim unconditionally". Polled on a 1s timer rather than evented — a
process singleton raising events at per-instance drivers outlives every driver it holds.

Accepted trade: a brief edge-visible OFFLINE window during failover. That is a genuine
state transition, not a glitch to hide.

Live-gated 5/5 against the Mosquitto fixture, including the Will leg — `AbandonForTests`
drops the socket with no DISCONNECT, because a clean disconnect makes the broker discard
the Will by protocol rule, so a test driven through `StopAsync` would observe the
stand-down OFFLINE and pass against an implementation that never armed a Will. Falsified:
removing the Will turns that leg and only that leg red.

Also corrects two stale claims found en route: the AdminUI form still said MQTT tags are
read-only (#508 shipped), and the driver doc still said nothing consumes `IRediscoverable`.

Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
This commit is contained in:
Joseph Doherty
2026-07-31 07:17:17 -04:00
parent 2a7d11674f
commit 7f2108dcf2
21 changed files with 1881 additions and 87 deletions
@@ -0,0 +1,90 @@
namespace ZB.MOM.WW.OtOpcUa.Core.Abstractions;
/// <summary>
/// A driver-facing snapshot of "should this node be the one asserting an exclusive identity on a
/// shared external system right now?".
/// </summary>
/// <remarks>
/// <para>
/// <b>Why drivers need this at all.</b> The driver layer has no notion of redundancy, by
/// design — a driver talks to a device and the Primary gate is applied above it, in
/// <c>DriverHostActor</c>, on the way in. That works because every gated operation is
/// <i>inbound</i>: a write, an alarm ack, an alerts publish. A Sparkplug primary-host claim is
/// not inbound. It is a standing assertion the driver makes to the broker, on its own
/// connection, for as long as it holds one — there is no message for the host actor to
/// intercept. So the decision has to reach the driver instead (Gitea #511).
/// </para>
/// <para>
/// <b>The decision, not the role.</b> Same discipline as the alarm-drain view in
/// <c>Runtime</c>: this publishes the verdict a single policy produced, so the policy stays the
/// one place that decides and a driver can never re-derive a subtly different rule from a raw
/// role. The verdict fed here is the <i>data-plane</i> one (an unknown role on a cluster that
/// has a driver peer denies), because two nodes claiming one Sparkplug host id is the harm the
/// gate exists to prevent — not a duplicate row, but a Secondary's Last Will publishing
/// <c>OFFLINE</c> over a live Primary's <c>ONLINE</c> and stopping every edge node in the
/// group.
/// </para>
/// <para>
/// <b>Polled, not evented, and that is deliberate.</b> A process-wide singleton raising an
/// event onto per-instance drivers outlives every driver it hands a reference to — the exact
/// missing-<c>-=</c> leak shape the rediscovery signal had to be careful about. Consumers
/// sample this instead, so a driver that is torn down simply stops asking. The cost is up to
/// one poll interval of latency on a failover, which is inside the edge-visible
/// <c>OFFLINE</c> window a failover already implies.
/// </para>
/// </remarks>
public interface IDriverPrimaryRoleSource
{
/// <summary>
/// Whether this node currently holds the Primary data-plane role — and may therefore claim an
/// exclusive external identity such as a Sparkplug host id.
/// </summary>
bool ShouldActAsPrimary { get; }
/// <summary>
/// Records a fresh decision. Called by <c>DriverHostActor</c> on every redundancy snapshot.
/// </summary>
/// <param name="shouldActAsPrimary">
/// The verdict <c>PrimaryGatePolicy.ShouldServiceAsPrimary</c> produced.
/// </param>
void PublishPrimaryRole(bool shouldActAsPrimary);
}
/// <summary>
/// The <see cref="IDriverPrimaryRoleSource"/> a driver uses when nothing publishes one — a
/// standalone host, a unit test, or any deployment with no redundancy manager wired.
/// </summary>
/// <remarks>
/// <para>
/// <b>It answers <see langword="true"/>, and the alternative is worse.</b> Defaulting closed
/// would be the reflexive "fail-safe" choice, but there is no safety to gain: with nothing
/// publishing a role there is by definition no peer to collide with, and a closed default
/// would make an exclusive-identity claim silently never happen on the single-node deployment
/// that is the common case — an inert feature with no error, which is precisely the failure
/// mode Gitea #511 was filed about.
/// </para>
/// <para>
/// The risk this shifts onto the wiring is real and is guarded there instead: a redundant
/// deployment whose composition root forgets to supply the live source would fall back to this
/// and let both nodes claim. <c>DriverFactoryBootstrapPrimaryRoleSourceTests</c> pins that the
/// Host resolves a real one.
/// </para>
/// </remarks>
public sealed class UnmanagedDriverPrimaryRoleSource : IDriverPrimaryRoleSource
{
/// <summary>The shared instance — it holds no state.</summary>
public static readonly UnmanagedDriverPrimaryRoleSource Instance = new();
private UnmanagedDriverPrimaryRoleSource()
{
}
/// <inheritdoc/>
public bool ShouldActAsPrimary => true;
/// <inheritdoc/>
/// <remarks>No-op: there is no redundancy manager to publish anything here.</remarks>
public void PublishPrimaryRole(bool shouldActAsPrimary)
{
}
}
@@ -190,6 +190,12 @@ public sealed record MqttSparkplugOptions
/// When <c>true</c>, this driver instance acts as the Sparkplug Primary Host
/// Application. At most one primary host may exist per host-id per broker.
/// </summary>
/// <remarks>
/// Implemented by <c>SparkplugPrimaryHost</c> (Gitea #511) on a dedicated broker connection
/// whose lifetime <i>is</i> the claim, and gated on the redundancy Primary role so the two nodes
/// of a pair never both claim. Requires <see cref="HostId"/>; set without one, the driver warns
/// and declines rather than publishing to a malformed topic.
/// </remarks>
public bool ActAsPrimaryHost { get; init; } = false;
/// <summary>
@@ -0,0 +1,37 @@
namespace ZB.MOM.WW.OtOpcUa.Driver.Mqtt;
/// <summary>
/// An MQTT Last Will and Testament, as the client options factory wants it (Gitea #511).
/// </summary>
/// <remarks>
/// <para>
/// <b>A Will can only be registered at CONNECT, and that single fact drives the whole design
/// of the Sparkplug primary-host claim.</b> It rides in the CONNECT packet, so it cannot be
/// added, changed or withdrawn on a live session. A node that gains the Primary role after
/// connecting therefore cannot simply start publishing <c>ONLINE</c> — it would have no Will,
/// and a crash would leave a retained <c>ONLINE</c> asserting forever that a dead host is
/// alive. Equally, a node that registers a Will and later loses the role cannot withdraw it,
/// so its eventual disconnect would publish <c>OFFLINE</c> over the new Primary's live claim
/// and stop every edge node in the group.
/// </para>
/// <para>
/// The consequence is that <b>the claim must be bound to the lifetime of a connection</b>:
/// <c>SparkplugPrimaryHost</c> opens a dedicated connection when the node gains Primary and
/// closes it when the node loses Primary, rather than trying to mutate the ingest session it
/// shares with the rest of the driver.
/// </para>
/// <para>
/// <b>Clean session matters too.</b> A broker that resumes an existing session for the same
/// client id may discard the Will registered by the new CONNECT. The primary-host connection
/// forces clean-session on for that reason, regardless of the driver's configured value.
/// </para>
/// </remarks>
/// <param name="Topic">The topic the broker publishes the Will to.</param>
/// <param name="Payload">The Will body.</param>
/// <param name="Qos">Requested QoS, 0–2.</param>
/// <param name="Retain">
/// Whether the broker retains the Will. For a Sparkplug host death certificate this is
/// <see langword="true"/>: an edge node that connects <i>after</i> the host died must still learn
/// the host is offline, and only a retained message reaches it.
/// </param>
public sealed record MqttLastWill(string Topic, byte[] Payload, int Qos, bool Retain);
@@ -0,0 +1,141 @@
using System.Text;
using System.Text.Json;
namespace ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Sparkplug;
/// <summary>
/// Builds the Sparkplug v3.0 primary-host <c>STATE</c> bodies — the retained <c>ONLINE</c> birth
/// and the matching <c>OFFLINE</c> death that is registered as the MQTT Last Will (Gitea #511).
/// </summary>
/// <remarks>
/// <para>
/// <b>Encode only.</b> The receive side lives in <c>SparkplugIngestor.TryReadStateOnline</c>,
/// which additionally tolerates the pre-3.0 bare <c>ONLINE</c>/<c>OFFLINE</c> text form. This
/// type only ever <i>produces</i> the v3.0 JSON shape, matching
/// <see cref="SparkplugTopic.FormatState"/>'s refusal to produce the legacy topic.
/// </para>
/// <para>
/// <b>The two bodies share one timestamp, and that is a spec requirement rather than a
/// convenience.</b> Sparkplug 3.0 pairs a host's birth and death certificates by their
/// <c>timestamp</c> field: an edge node that receives an <c>OFFLINE</c> whose timestamp is
/// older than the <c>ONLINE</c> it already holds is entitled to ignore it as a stale death
/// from a previous session. That rule is what makes the mechanism safe across a reconnect —
/// and it means the death body has to be built <i>before</i> CONNECT (a Will is registered in
/// the CONNECT packet) while the birth is published <i>after</i> CONNACK. Deriving each from
/// its own <c>UtcNow</c> would give them different timestamps and quietly break the pairing,
/// so <see cref="ForSession"/> mints both at once and there is no single-body overload to
/// reach for by accident.
/// </para>
/// </remarks>
public static class SparkplugStatePayload
{
/// <summary>
/// QoS for both STATE bodies. Sparkplug 3.0 requires QoS 1 for the host birth and death —
/// unlike the QoS 0 that carries NBIRTH/DBIRTH/DATA. A host claim delivered at most once is a
/// claim that may simply never arrive.
/// </summary>
public const int StateQos = 1;
/// <summary>
/// Mints the matched <c>ONLINE</c>/<c>OFFLINE</c> pair for one host session, both carrying
/// <paramref name="timestampMs"/>.
/// </summary>
/// <param name="hostId">The Sparkplug host id; used to build the topic.</param>
/// <param name="timestampMs">
/// Sparkplug epoch milliseconds for this session. Supplied by the caller rather than read from
/// the clock here so the value is visible at the call site and testable.
/// </param>
/// <returns>The topic plus both bodies.</returns>
/// <exception cref="ArgumentException"><paramref name="hostId"/> is blank.</exception>
public static SparkplugStateSession ForSession(string hostId, long timestampMs)
{
ArgumentException.ThrowIfNullOrWhiteSpace(hostId);
return new SparkplugStateSession(
Topic: SparkplugTopic.FormatState(hostId),
TimestampMs: timestampMs,
OnlineBody: Encode(online: true, timestampMs),
OfflineBody: Encode(online: false, timestampMs));
}
/// <summary>Encodes one STATE body: <c>{"online":&lt;bool&gt;,"timestamp":&lt;ms&gt;}</c>.</summary>
/// <param name="online">The online flag.</param>
/// <param name="timestampMs">Sparkplug epoch milliseconds.</param>
/// <returns>The UTF-8 body bytes.</returns>
/// <remarks>
/// Hand-written rather than serialized from a DTO. The body is two fields, and the wire form
/// is fixed by the spec — a serializer would put naming policy, casing options and a shared
/// <c>JsonSerializerOptions</c> between this repo and a wire contract it does not control.
/// <c>timestamp</c> is emitted as a bare JSON number, never a string: a quoted timestamp
/// parses as a valid JSON object and is silently wrong to every conformant host.
/// </remarks>
public static byte[] Encode(bool online, long timestampMs) =>
Encoding.UTF8.GetBytes(
$"{{\"online\":{(online ? "true" : "false")},\"timestamp\":{timestampMs.ToString(System.Globalization.CultureInfo.InvariantCulture)}}}");
/// <summary>Reads the <c>online</c> flag back out of a v3.0 STATE body, for tests and diagnostics.</summary>
/// <param name="body">The STATE body bytes.</param>
/// <param name="online">The decoded flag on success.</param>
/// <param name="timestampMs">The decoded timestamp on success.</param>
/// <returns><see langword="true"/> when the body is a well-formed v3.0 STATE object.</returns>
public static bool TryDecode(ReadOnlySpan<byte> body, out bool online, out long timestampMs)
{
online = false;
timestampMs = 0;
if (body.IsEmpty)
{
return false;
}
try
{
var reader = new Utf8JsonReader(body);
if (!JsonDocument.TryParseValue(ref reader, out var doc))
{
return false;
}
using (doc)
{
if (doc.RootElement.ValueKind != JsonValueKind.Object
|| !doc.RootElement.TryGetProperty("online", out var flag)
|| flag.ValueKind is not (JsonValueKind.True or JsonValueKind.False))
{
return false;
}
online = flag.GetBoolean();
timestampMs = doc.RootElement.TryGetProperty("timestamp", out var ts)
&& ts.ValueKind == JsonValueKind.Number
&& ts.TryGetInt64(out var parsed)
? parsed
: 0;
return true;
}
}
catch (JsonException)
{
return false;
}
}
}
/// <summary>
/// The STATE topic plus the matched pair of bodies for one primary-host session (Gitea #511).
/// </summary>
/// <param name="Topic">The v3.0 STATE topic, <c>spBv1.0/STATE/{hostId}</c>.</param>
/// <param name="TimestampMs">The session timestamp carried by BOTH bodies.</param>
/// <param name="OnlineBody">
/// The birth certificate — published retained, at QoS 1, immediately after CONNACK.
/// </param>
/// <param name="OfflineBody">
/// The death certificate — registered as the MQTT Last Will in the CONNECT packet, and also
/// published explicitly on a clean stand-down so the broker holds it without waiting for a
/// keep-alive timeout.
/// </param>
public sealed record SparkplugStateSession(
string Topic,
long TimestampMs,
byte[] OnlineBody,
byte[] OfflineBody);
@@ -41,20 +41,51 @@ public static class MqttClientOptionsFactory
/// empty MQTTnet generates a random client id.
/// </param>
/// <param name="logger">Optional logger for the deferred CA-pin load; never receives credentials.</param>
/// <param name="lastWill">
/// Optional MQTT Last Will registered in the CONNECT packet (Gitea #511 — the Sparkplug
/// primary-host death certificate). See <see cref="MqttLastWill"/> for why this can only be
/// supplied here.
/// </param>
/// <param name="cleanSessionOverride">
/// Forces clean-session on or off, ignoring <see cref="MqttDriverOptions.CleanSession"/>. Used
/// by the primary-host connection, which must connect clean so the broker cannot resume a
/// previous session and discard the Will — see <see cref="MqttLastWill"/>.
/// </param>
public static MqttClientOptions BuildClientOptions(
MqttDriverOptions options,
string? clientIdSuffix,
ILogger? logger = null)
ILogger? logger = null,
MqttLastWill? lastWill = null,
bool? cleanSessionOverride = null)
{
ArgumentNullException.ThrowIfNull(options);
var builder = new MqttClientOptionsBuilder()
.WithTcpServer(options.Host, options.Port)
.WithProtocolVersion(MapProtocolVersion(options.ProtocolVersion))
.WithCleanSession(options.CleanSession)
.WithCleanSession(cleanSessionOverride ?? options.CleanSession)
.WithKeepAlivePeriod(TimeSpan.FromSeconds(options.KeepAliveSeconds))
.WithTimeout(TimeSpan.FromSeconds(options.ConnectTimeoutSeconds));
if (lastWill is { } will)
{
builder = builder
.WithWillTopic(will.Topic)
.WithWillPayload(will.Payload)
.WithWillQualityOfServiceLevel((MQTTnet.Protocol.MqttQualityOfServiceLevel)will.Qos)
.WithWillRetain(will.Retain);
// MQTT 5.0 only. Left at its default the broker MAY hold the Will back for up to the
// session-expiry interval after an ungraceful disconnect, which for a host death
// certificate is precisely the wrong behaviour: the whole point is that edge nodes learn
// promptly. Zero means "publish it the moment the network connection is lost".
// The setter is harmless on 3.1.1 — the property simply has no wire representation there.
if (options.ProtocolVersion == MqttProtocolVersion.V500)
{
builder = builder.WithWillDelayInterval(0);
}
}
var clientId = string.Concat(options.ClientId ?? string.Empty, clientIdSuffix ?? string.Empty);
if (!string.IsNullOrWhiteSpace(clientId))
{
@@ -286,11 +286,40 @@ public sealed class MqttConnection : IAsyncDisposable, IMqttSubscribeTransport,
private Task? _reconnectWorker;
private int _state = (int)MqttConnectionState.Disconnected;
/// <summary>Client-id suffix distinguishing this connection from the driver's other one; see the ctor.</summary>
private readonly string? _clientIdSuffix;
/// <summary>The Will to register on every CONNECT, or null. See <see cref="MqttLastWill"/>.</summary>
private readonly MqttLastWill? _lastWill;
/// <summary>Clean-session override, or null to use the configured value.</summary>
private readonly bool? _cleanSessionOverride;
/// <summary>Stores configuration only — no network, no client instantiation, no supervisor.</summary>
/// <param name="options">Broker connection settings.</param>
/// <param name="driverId">Driver instance id, used only for log/diagnostic correlation.</param>
/// <param name="logger">Optional logger; never receives credentials.</param>
public MqttConnection(MqttDriverOptions options, string driverId, ILogger? logger = null)
/// <param name="clientIdSuffix">
/// Appended to the configured client id so a second connection from the same driver — the
/// Sparkplug primary-host claim (Gitea #511) — does not collide with the ingest session. Two
/// MQTT clients presenting one client id is not an error the broker reports: it silently
/// disconnects the older one, so the two would take turns evicting each other forever.
/// </param>
/// <param name="lastWill">
/// Optional Last Will registered in the CONNECT packet, and therefore rebuilt on every
/// reconnect attempt along with the rest of the options. See <see cref="MqttLastWill"/>.
/// </param>
/// <param name="cleanSessionOverride">
/// Forces clean-session on or off regardless of <see cref="MqttDriverOptions.CleanSession"/>.
/// Null keeps the configured value.
/// </param>
public MqttConnection(
MqttDriverOptions options,
string driverId,
ILogger? logger = null,
string? clientIdSuffix = null,
MqttLastWill? lastWill = null,
bool? cleanSessionOverride = null)
{
ArgumentNullException.ThrowIfNull(options);
ArgumentException.ThrowIfNullOrWhiteSpace(driverId);
@@ -298,6 +327,9 @@ public sealed class MqttConnection : IAsyncDisposable, IMqttSubscribeTransport,
_options = options;
_driverId = driverId;
_logger = logger;
_clientIdSuffix = clientIdSuffix;
_lastWill = lastWill;
_cleanSessionOverride = cleanSessionOverride;
}
/// <summary>
@@ -370,6 +402,47 @@ public sealed class MqttConnection : IAsyncDisposable, IMqttSubscribeTransport,
/// </summary>
internal Func<Task>? AfterConnectHookForTests { get; set; }
/// <summary>
/// Test seam: drops the session the way a crashed process does — no MQTT <c>DISCONNECT</c>, so
/// the broker treats the closed socket as an abnormal termination and publishes the Last Will.
/// </summary>
/// <returns>A task completing once the client is gone and the supervisor has stopped.</returns>
/// <remarks>
/// <para>
/// Exists because the Sparkplug primary-host contract's most important leg is the one a
/// normal teardown cannot reach (Gitea #511): <c>DisposeAsync</c> sends a clean
/// <c>DISCONNECT</c>, which by MQTT rule means the broker <b>discards</b> the Will. So an
/// in-suite test driven through the ordinary lifecycle can only ever observe the explicit
/// stand-down <c>OFFLINE</c>, and would pass identically against an implementation whose
/// Will was never armed at all.
/// </para>
/// <para>
/// Not a full substitute for the <c>kill -9</c> the live gate runs by hand — that also
/// proves the process dies without any of this code running — but it is the closest an
/// in-process test can get, and it is what makes the Will's <i>content</i> observable.
/// </para>
/// </remarks>
internal async Task AbandonForTests()
{
// Mark disposed first, so the supervisor cannot reconnect into the gap and publish a fresh
// birth on top of the Will the broker is about to deliver.
Interlocked.Exchange(ref _disposed, 1);
SetState(MqttConnectionState.Disposed);
try
{
await _lifetimeCts.CancelAsync().ConfigureAwait(false);
}
catch (ObjectDisposedException)
{
// Nothing to cancel.
}
// Dispose WITHOUT DisconnectAsync: MQTTnet closes the channel, the broker sees a dead socket
// rather than a graceful goodbye, and the Will fires. This is the whole point of the seam.
Interlocked.Exchange(ref _client, null)?.Dispose();
}
private bool Disposed => Volatile.Read(ref _disposed) != 0;
/// <inheritdoc/>
@@ -568,7 +641,8 @@ public sealed class MqttConnection : IAsyncDisposable, IMqttSubscribeTransport,
MqttClientOptions clientOptions;
try
{
clientOptions = MqttClientOptionsFactory.BuildClientOptions(_options, clientIdSuffix: null, _logger);
clientOptions = MqttClientOptionsFactory.BuildClientOptions(
_options, _clientIdSuffix, _logger, _lastWill, _cleanSessionOverride);
}
catch (Exception ex)
{
@@ -185,6 +185,16 @@ public sealed class MqttDriver
private MqttConnection? _connection;
private CancellationTokenSource? _hostProbeCts;
/// <summary>The Primary data-plane verdict the Sparkplug host claim is gated on (Gitea #511).</summary>
private readonly IDriverPrimaryRoleSource _primaryRoleSource;
/// <summary>
/// The Sparkplug primary-host claim, or <see langword="null"/> when this configuration does not
/// ask for one. Owns its own broker connection — see <see cref="SparkplugPrimaryHost"/> for why
/// it cannot share the ingest session.
/// </summary>
private SparkplugPrimaryHost? _primaryHost;
private DriverHealth _health = new(DriverState.Unknown, null, null);
private HostState _hostState = HostState.Unknown;
private DateTime _hostStateChangedUtc = DateTime.UtcNow;
@@ -198,7 +208,18 @@ public sealed class MqttDriver
/// </param>
/// <param name="driverInstanceId">Stable logical id of this driver instance.</param>
/// <param name="logger">Optional logger; never receives credentials or payload bodies.</param>
public MqttDriver(MqttDriverOptions options, string driverInstanceId, ILogger<MqttDriver>? logger = null)
/// <param name="primaryRoleSource">
/// Gitea #511: the Primary data-plane verdict the Sparkplug primary-host claim is gated on.
/// <see langword="null"/> falls back to <see cref="UnmanagedDriverPrimaryRoleSource"/> — which
/// always claims — for standalone hosts and tests. On a redundant pair that fallback is the
/// dual-claim this gate exists to prevent, so the Host resolves a real one and
/// <c>DriverFactoryBootstrap</c> does so with <c>GetRequiredService</c>.
/// </param>
public MqttDriver(
MqttDriverOptions options,
string driverInstanceId,
ILogger<MqttDriver>? logger = null,
IDriverPrimaryRoleSource? primaryRoleSource = null)
{
ArgumentNullException.ThrowIfNull(options);
ArgumentException.ThrowIfNullOrWhiteSpace(driverInstanceId);
@@ -206,6 +227,7 @@ public sealed class MqttDriver
_options = options;
_driverInstanceId = driverInstanceId;
_logger = logger;
_primaryRoleSource = primaryRoleSource ?? UnmanagedDriverPrimaryRoleSource.Instance;
_subscriptions = CreateSubscriptionManager(options);
_sparkplug = CreateSparkplugIngestor(options);
RegisterAuthoredTags(options);
@@ -1001,6 +1023,7 @@ public sealed class MqttDriver
WriteHealth(new DriverHealth(DriverState.Healthy, connection.LastMessageUtc, null));
TransitionHost(HostState.Running);
StartHostProbe();
await StartPrimaryHostAsync(cancellationToken).ConfigureAwait(false);
}
catch (Exception ex)
{
@@ -1094,21 +1117,15 @@ public sealed class MqttDriver
return null;
}
if (options.Sparkplug is { ActAsPrimaryHost: true })
// The flag no longer reads as if it does something it does not (Gitea #511) — but a request it
// CANNOT honour still warns loudly rather than being dropped, because the operator's plant is
// waiting on a STATE message that will never arrive.
if (SparkplugPrimaryHost.DescribeUnusable(options) is { } unusable)
{
// Loud, because the flag reads as if it does something. Being a Sparkplug Primary Host is a
// three-part contract — a retained ONLINE after CONNECT, a matching retained OFFLINE
// registered as the MQTT Last Will AT CONNECT TIME, and an explicit OFFLINE on clean
// shutdown — and edge nodes stop publishing while their host is offline. Shipping the
// ONLINE half without the Will is strictly worse than shipping neither: a crashed node
// would leave a retained ONLINE asserting forever that a dead host is alive, which is the
// exact state the mechanism exists to prevent. The Will belongs in
// MqttClientOptionsFactory.BuildClientOptions; until it is there this flag stays inert and said so.
_logger?.LogWarning(
"MQTT driver '{DriverId}': Sparkplug actAsPrimaryHost is set but primary-host STATE "
+ "publishing is not implemented — this driver observes STATE and never publishes one. "
+ "Edge nodes gated on this host id will not see it come online.",
_driverInstanceId);
"MQTT driver '{DriverId}': {Reason} Edge nodes gated on this host id will not see it come online.",
_driverInstanceId,
unusable);
}
var ingestor = new SparkplugIngestor(
@@ -1263,9 +1280,55 @@ public sealed class MqttDriver
o.Sparkplug,
};
/// <summary>
/// Opens the Sparkplug primary-host claim when this configuration asks for one (Gitea #511).
/// </summary>
/// <remarks>
/// <para>
/// <b>Runs last, and can never fail the initialize.</b> The claim is an advisory assertion
/// to other applications on the broker; the ingest path is this driver's data plane. A
/// broker that accepts the ingest session but rejects a publish on the STATE topic — an ACL
/// that does not grant <c>spBv1.0/STATE/#</c> is the ordinary way this happens — must cost
/// the operator a warning and a plant-visible missing host, not every tag's value.
/// <see cref="SparkplugPrimaryHost.StartAsync"/> already swallows and retries; the catch
/// here covers the rest so no future edit can turn this into a fault path by accident.
/// </para>
/// </remarks>
private async Task StartPrimaryHostAsync(CancellationToken cancellationToken)
{
if (!SparkplugPrimaryHost.CanClaim(_options))
{
return;
}
try
{
var host = new SparkplugPrimaryHost(_options, _driverInstanceId, _primaryRoleSource, _logger);
_primaryHost = host;
await host.StartAsync(cancellationToken).ConfigureAwait(false);
}
catch (Exception ex)
{
_logger?.LogWarning(
ex,
"MQTT driver '{DriverId}': the Sparkplug primary-host claim could not be started; ingest is unaffected.",
_driverInstanceId);
}
}
/// <summary>Shared teardown for <see cref="ShutdownAsync"/>, <see cref="DisposeAsync"/> and the session rebuild.</summary>
private async Task TeardownAsync()
{
// Stand the host claim down FIRST, while the broker is still reachable and before the ingest
// session goes. Its explicit retained OFFLINE is what makes the transition immediate for edge
// nodes; leaving it to the Will costs them up to 1.5 keep-alive periods of believing a host
// that has already stopped is still online.
var primaryHost = Interlocked.Exchange(ref _primaryHost, null);
if (primaryHost is not null)
{
await primaryHost.DisposeAsync().ConfigureAwait(false);
}
var probe = Interlocked.Exchange(ref _hostProbeCts, null);
if (probe is not null)
{
@@ -49,16 +49,30 @@ public static class MqttDriverFactoryExtensions
/// </summary>
/// <param name="registry">The driver factory registry to register with.</param>
/// <param name="loggerFactory">Optional logger factory used to create per-instance loggers.</param>
public static void Register(DriverFactoryRegistry registry, ILoggerFactory? loggerFactory = null)
/// <param name="primaryRoleSource">
/// Gitea #511: the Primary data-plane verdict the Sparkplug primary-host claim is gated on,
/// captured at registration time like <paramref name="loggerFactory"/>. Null falls back to
/// <see cref="UnmanagedDriverPrimaryRoleSource"/> (always claims) — correct for a standalone
/// host or a test, and the reason <c>DriverFactoryBootstrap</c> resolves it with
/// <c>GetRequiredService</c> rather than letting a redundant deployment reach that fallback.
/// </param>
public static void Register(
DriverFactoryRegistry registry,
ILoggerFactory? loggerFactory = null,
IDriverPrimaryRoleSource? primaryRoleSource = null)
{
ArgumentNullException.ThrowIfNull(registry);
registry.Register(DriverTypeName, (id, json) => CreateInstance(id, json, loggerFactory));
registry.Register(DriverTypeName, (id, json) => CreateInstance(id, json, loggerFactory, primaryRoleSource));
}
/// <summary>Public for the Server-side bootstrapper + test consumers.</summary>
/// <param name="driverInstanceId">Stable logical id of the driver instance.</param>
/// <param name="driverConfigJson">The <c>DriverConfig</c> JSON blob for the instance.</param>
/// <param name="loggerFactory">Optional logger factory for the per-instance logger.</param>
/// <param name="primaryRoleSource">
/// Gitea #511: the Primary data-plane verdict gating the Sparkplug primary-host claim; null
/// falls back to <see cref="UnmanagedDriverPrimaryRoleSource"/>.
/// </param>
/// <returns>A configured, not-yet-connected <see cref="MqttDriver"/>.</returns>
/// <exception cref="ArgumentException">
/// <paramref name="driverInstanceId"/> or <paramref name="driverConfigJson"/> is blank.
@@ -68,7 +82,8 @@ public static class MqttDriverFactoryExtensions
public static MqttDriver CreateInstance(
string driverInstanceId,
string driverConfigJson,
ILoggerFactory? loggerFactory = null)
ILoggerFactory? loggerFactory = null,
IDriverPrimaryRoleSource? primaryRoleSource = null)
{
ArgumentException.ThrowIfNullOrWhiteSpace(driverInstanceId);
ArgumentException.ThrowIfNullOrWhiteSpace(driverConfigJson);
@@ -79,6 +94,7 @@ public static class MqttDriverFactoryExtensions
?? throw new InvalidOperationException(
$"MQTT driver config for '{driverInstanceId}' deserialised to null");
return new MqttDriver(options, driverInstanceId, loggerFactory?.CreateLogger<MqttDriver>());
return new MqttDriver(
options, driverInstanceId, loggerFactory?.CreateLogger<MqttDriver>(), primaryRoleSource);
}
}
@@ -73,9 +73,11 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Sparkplug;
/// subscriber is contained, and the rebirth publish is dispatched off-thread.
/// </para>
/// <para>
/// <b>Primary-host STATE publishing is deliberately NOT implemented</b> — see
/// <see cref="MqttSparkplugOptions.ActAsPrimaryHost"/> and the remarks on
/// <see cref="HandleState"/>. This type <i>observes</i> STATE; it never publishes one.
/// <b>This type only ever <i>observes</i> STATE.</b> Primary-host STATE publishing ships
/// (Gitea #511) but deliberately does not live here: the claim is bound to the lifetime of its
/// own dedicated connection, because an MQTT Will can only be registered at CONNECT. See
/// <see cref="SparkplugPrimaryHost"/>, and the remarks on <see cref="HandleState"/> for what
/// this side does with an observed one.
/// </para>
/// </remarks>
public sealed class SparkplugIngestor
@@ -1029,18 +1031,17 @@ public sealed class SparkplugIngestor
/// </summary>
/// <remarks>
/// <para>
/// <b>Receive only. This driver never publishes STATE, and
/// <see cref="MqttSparkplugOptions.ActAsPrimaryHost"/> is therefore inert — deliberately, and
/// loudly.</b> Being a Sparkplug Primary Host is not "publish an ONLINE message": it is a
/// three-part contract — a retained <c>ONLINE</c> STATE published after CONNECT, a matching
/// retained <c>OFFLINE</c> registered as the client's <b>MQTT Last Will at CONNECT time</b>,
/// and an explicit OFFLINE on clean shutdown. Edge nodes subscribe to it and stop publishing
/// while their host is offline. Shipping the ONLINE half without the Will is strictly worse
/// than shipping neither: an OtOpcUa node that crashes would leave a retained ONLINE asserting
/// forever that a dead host is alive, which is precisely the state the mechanism exists to
/// prevent. The Will has to be set inside <c>MqttClientOptionsFactory.BuildClientOptions</c> — a
/// different file, a different task — so this task implements the observation half and
/// <see cref="MqttDriver"/> warns when the flag is set. See the design doc §7/§8.
/// <b>Receive only, on this connection.</b> The publish half ships as
/// <see cref="SparkplugPrimaryHost"/> (Gitea #511), on a connection of its own — an MQTT Will
/// can only be registered in the CONNECT packet, so a claim that is acquired and released at
/// runtime cannot ride the long-lived ingest session this type is attached to.
/// </para>
/// <para>
/// <b>A host claimed by THIS node is observed here too, and that is correct rather than an
/// oversight.</b> The ingestor subscribes the configured host id whether or not this node
/// publishes it, so a node acting as primary host sees its own retained <c>ONLINE</c> come
/// back. Nothing here gates on it — <see cref="HostStateObserved"/> is informational — so the
/// echo costs one log line at the transition and cannot feed back into the claim.
/// </para>
/// <para>
/// Both payload forms are accepted on receive: the v3.0 JSON <c>{"online":bool,...}</c> and
@@ -0,0 +1,470 @@
using Microsoft.Extensions.Logging;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Sparkplug;
/// <summary>
/// Claims the Sparkplug Primary Host Application identity for this node — the three-part contract
/// <see cref="MqttSparkplugOptions.ActAsPrimaryHost"/> promises and, until Gitea #511, did not
/// deliver: a retained <c>ONLINE</c> while the claim is held, a matching retained <c>OFFLINE</c>
/// registered as the MQTT Last Will, and an explicit <c>OFFLINE</c> on a clean stand-down.
/// </summary>
/// <remarks>
/// <para>
/// <b>The claim is a whole dedicated connection, not a message.</b> A Will can only be
/// registered in the CONNECT packet (see <see cref="MqttLastWill"/>), so a claim that must be
/// acquired and released at runtime cannot live on the driver's shared ingest session — that
/// session connects once and stays up, and its Will could be neither added on gaining the role
/// nor withdrawn on losing it. This type therefore opens its <i>own</i> connection when the
/// node gains Primary and closes it when the node loses Primary. The connection's lifetime
/// <i>is</i> the claim, which is what makes every leg of the contract fall out for free:
/// <list type="bullet">
/// <item>the Will is registered because the connection was opened to hold the claim;</item>
/// <item>a <c>kill -9</c> drops the connection, so the broker publishes the Will;</item>
/// <item>a clean stand-down publishes <c>OFFLINE</c> and then disconnects, so the retained
/// message is correct before the Will could ever fire.</item>
/// </list>
/// </para>
/// <para>
/// <b>Why not rebuild the ingest connection instead.</b> That was the shape first sketched on
/// the issue, and it works, but it drops every subscription and the whole birth cache on each
/// role change — producing a data gap on the Secondary→Primary transition, which is precisely
/// the moment a plant most needs values flowing. A second connection costs one extra client id
/// and leaves ingest untouched.
/// </para>
/// <para>
/// <b>Primary-gated, by owner decision (2026-07-31).</b> Both nodes of a redundant pair
/// subscribe independently — MQTT fan-out is per-connection and the Primary gate applies
/// downstream of ingest — so without gating, both would claim the same host id. The sharp
/// failure is not the two fighting over the retained <c>STATE</c>: it is that the
/// <b>Secondary's</b> death would publish <c>OFFLINE</c>, edge nodes would stop publishing,
/// and the surviving Primary would be perfectly healthy throughout. Redundancy would cause the
/// outage the mechanism exists to prevent. The gate is
/// <see cref="IDriverPrimaryRoleSource.ShouldActAsPrimary"/>, sampled on a timer.
/// </para>
/// <para>
/// <b>Accepted trade: a brief edge-visible <c>OFFLINE</c> window during failover</b>, while the
/// old Primary stands down and the new one connects and republishes. That is a genuine state
/// transition — for a window there really is no host — not a glitch to paper over.
/// </para>
/// <para>
/// <b>Reconnect re-asserts the claim.</b> The dedicated connection carries the same supervisor
/// as the ingest one, and <c>Reconnected</c> republishes the birth. Without that, a broker
/// restart would leave the host connected but its retained <c>ONLINE</c> gone — a live host
/// invisible to every edge node, with nothing in any log to say so.
/// </para>
/// </remarks>
public sealed class SparkplugPrimaryHost : IAsyncDisposable
{
/// <summary>
/// Appended to the driver's configured client id for the STATE connection. Two MQTT clients
/// presenting one client id is not reported as an error — the broker silently evicts the older
/// one — so ingest and the host claim would take turns killing each other forever.
/// </summary>
internal const string ClientIdSuffix = "-spbhost";
/// <summary>How often the role gate is sampled. Bounds the failover republish latency.</summary>
private static readonly TimeSpan RoleSampleInterval = TimeSpan.FromSeconds(1);
private readonly MqttDriverOptions _options;
private readonly string _hostId;
private readonly string _driverId;
private readonly IDriverPrimaryRoleSource _roleSource;
private readonly ILogger? _logger;
private readonly TimeProvider _timeProvider;
/// <summary>Serializes claim/release against each other and against <see cref="DisposeAsync"/>.</summary>
private readonly SemaphoreSlim _gate = new(1, 1);
private CancellationTokenSource? _loopCts;
/// <summary>The live claim, or null when this node is not currently the primary host.</summary>
private MqttConnection? _connection;
/// <summary>The session bodies backing <see cref="_connection"/>; non-null exactly when it is.</summary>
private SparkplugStateSession? _session;
private int _disposed;
/// <summary>Creates a primary-host claim. Opens nothing — <see cref="StartAsync"/> does that.</summary>
/// <param name="options">Broker connection settings; the host id comes from its Sparkplug section.</param>
/// <param name="driverId">Driver instance id, for log correlation.</param>
/// <param name="roleSource">The Primary data-plane verdict this claim is gated on.</param>
/// <param name="logger">Optional logger.</param>
/// <param name="timeProvider">Clock seam; defaults to <see cref="TimeProvider.System"/>.</param>
/// <exception cref="ArgumentException">
/// The options carry no Sparkplug host id — <see cref="CanClaim"/> exists so a caller never has
/// to hit this.
/// </exception>
public SparkplugPrimaryHost(
MqttDriverOptions options,
string driverId,
IDriverPrimaryRoleSource roleSource,
ILogger? logger = null,
TimeProvider? timeProvider = null)
{
ArgumentNullException.ThrowIfNull(options);
ArgumentException.ThrowIfNullOrWhiteSpace(driverId);
ArgumentNullException.ThrowIfNull(roleSource);
var hostId = options.Sparkplug?.HostId;
if (string.IsNullOrWhiteSpace(hostId))
{
throw new ArgumentException(
"A Sparkplug primary-host claim needs a hostId; check CanClaim first.", nameof(options));
}
_options = options;
_hostId = hostId.Trim();
_driverId = driverId;
_roleSource = roleSource;
_logger = logger;
_timeProvider = timeProvider ?? TimeProvider.System;
}
/// <summary>
/// Whether <paramref name="options"/> asks for — and can support — a primary-host claim:
/// Sparkplug mode, <c>actAsPrimaryHost</c> set, and a non-blank <c>hostId</c>.
/// </summary>
/// <param name="options">The driver options to test.</param>
/// <returns><see langword="true"/> when a claim should be made.</returns>
/// <remarks>
/// The blank-host-id case is a configuration error the AdminUI form already refuses, but the
/// driver must not fault on it: the config could have been authored before that validation
/// existed. It is reported by <see cref="DescribeUnusable"/> and the flag is treated as off.
/// </remarks>
public static bool CanClaim(MqttDriverOptions options) =>
options is { Mode: MqttMode.SparkplugB, Sparkplug: { ActAsPrimaryHost: true } spb }
&& !string.IsNullOrWhiteSpace(spb.HostId);
/// <summary>
/// Describes why an <c>actAsPrimaryHost</c> request cannot be honoured, or null when it can be
/// (or was never made).
/// </summary>
/// <param name="options">The driver options to test.</param>
/// <returns>An operator-facing reason, or <see langword="null"/>.</returns>
public static string? DescribeUnusable(MqttDriverOptions options)
{
if (options?.Sparkplug is not { ActAsPrimaryHost: true } spb)
{
return null;
}
if (options.Mode != MqttMode.SparkplugB)
{
return "actAsPrimaryHost is set but the driver is not in SparkplugB mode; there is no host state to publish.";
}
return string.IsNullOrWhiteSpace(spb.HostId)
? "actAsPrimaryHost is set but hostId is blank; a host claim has no topic to publish on."
: null;
}
/// <summary>The STATE topic this claim publishes on.</summary>
public string StateTopic => SparkplugTopic.FormatState(_hostId);
/// <summary>
/// Whether the claim is currently held — i.e. a retained <c>ONLINE</c> has been published and a
/// Will is armed behind it.
/// </summary>
public bool IsClaimed => Volatile.Read(ref _connection) is not null;
/// <summary>
/// Test seam: how many times a claim connect has been <i>attempted</i>, successful or not.
/// Always ignored in production.
/// </summary>
/// <remarks>
/// <b>Without this, "a Secondary does not claim" is a vacuous assertion.</b> The obvious test —
/// start a Secondary and assert <see cref="IsClaimed"/> is false — passes identically whether
/// the gate works or whether the gate is deleted and the connect simply fails against a broker
/// the test cannot reach. Counting attempts separates "did not try" from "tried and failed",
/// which is the whole distinction the gate is.
/// </remarks>
internal int ClaimAttemptsForTests => Volatile.Read(ref _claimAttempts);
private int _claimAttempts;
/// <summary>
/// Test seam: kills the claim connection the way a crashed process does — no clean
/// <c>DISCONNECT</c> and no explicit stand-down publish — so the broker delivers the Last Will.
/// See <see cref="MqttConnection.AbandonForTests"/> for why this cannot be reached through the
/// ordinary lifecycle.
/// </summary>
/// <returns>A task completing once the connection is gone.</returns>
internal async Task AbandonClaimForTests()
{
// Stop the reconcile loop first, or the next tick sees no connection, decides the claim is
// lost, and re-claims — republishing ONLINE over the Will this is meant to observe.
var loop = Interlocked.Exchange(ref _loopCts, null);
if (loop is not null)
{
await loop.CancelAsync().ConfigureAwait(false);
loop.Dispose();
}
var connection = Interlocked.Exchange(ref _connection, null);
_session = null;
if (connection is not null)
{
await connection.AbandonForTests().ConfigureAwait(false);
}
}
/// <summary>
/// Starts the role-reconcile loop. Returns once the first reconcile has run, so a caller that
/// starts as Primary has published its <c>ONLINE</c> before this completes.
/// </summary>
/// <param name="cancellationToken">Cancels the initial reconcile.</param>
/// <returns>A task completing after the first reconcile.</returns>
public async Task StartAsync(CancellationToken cancellationToken)
{
ObjectDisposedException.ThrowIf(Volatile.Read(ref _disposed) != 0, this);
var cts = new CancellationTokenSource();
var previous = Interlocked.Exchange(ref _loopCts, cts);
if (previous is not null)
{
await previous.CancelAsync().ConfigureAwait(false);
previous.Dispose();
}
// Reconcile once inline. A failure here is logged and swallowed, never rethrown: the driver's
// ingest path is the data plane, and refusing to start it because a broker would not accept a
// host claim would trade every tag's value for an advisory message. The loop retries.
await ReconcileAsync(cancellationToken).ConfigureAwait(false);
_ = Task.Run(() => ReconcileLoopAsync(cts.Token), CancellationToken.None);
}
/// <summary>
/// Stands the claim down cleanly: publishes an explicit retained <c>OFFLINE</c>, then
/// disconnects. Idempotent, and safe to call when no claim is held.
/// </summary>
/// <param name="cancellationToken">Bounds the stand-down publish.</param>
/// <returns>A task completing when the claim is released.</returns>
public async Task StopAsync(CancellationToken cancellationToken)
{
var loop = Interlocked.Exchange(ref _loopCts, null);
if (loop is not null)
{
await loop.CancelAsync().ConfigureAwait(false);
loop.Dispose();
}
await ReleaseAsync("shutdown", cancellationToken).ConfigureAwait(false);
}
/// <inheritdoc/>
public async ValueTask DisposeAsync()
{
if (Interlocked.Exchange(ref _disposed, 1) != 0)
{
return;
}
// Bounded: dispose must not hang on an unreachable broker. A stand-down we cannot publish is
// covered by the Will the moment the socket closes, which is the whole point of having one.
using var deadline = new CancellationTokenSource(TimeSpan.FromSeconds(_options.ConnectTimeoutSeconds));
try
{
await StopAsync(deadline.Token).ConfigureAwait(false);
}
catch (Exception ex)
{
_logger?.LogDebug(
ex, "MQTT driver '{DriverId}': Sparkplug primary-host stand-down failed during dispose.", _driverId);
}
_gate.Dispose();
}
private async Task ReconcileLoopAsync(CancellationToken ct)
{
try
{
while (!ct.IsCancellationRequested)
{
await Task.Delay(RoleSampleInterval, _timeProvider, ct).ConfigureAwait(false);
await ReconcileAsync(ct).ConfigureAwait(false);
}
}
catch (OperationCanceledException)
{
// Shutdown.
}
catch (Exception ex)
{
// The loop must not die silently — a dead loop means the claim freezes in whatever state
// it was last in, including "claimed by a node that is no longer Primary".
_logger?.LogError(
ex, "MQTT driver '{DriverId}': Sparkplug primary-host reconcile loop stopped.", _driverId);
}
}
/// <summary>
/// Brings the held claim into line with the current role verdict. Never throws — a transient
/// broker failure leaves the claim as it was and the next tick tries again.
/// </summary>
private async Task ReconcileAsync(CancellationToken cancellationToken)
{
try
{
if (_roleSource.ShouldActAsPrimary)
{
await ClaimAsync(cancellationToken).ConfigureAwait(false);
}
else
{
await ReleaseAsync("this node is not the redundancy Primary", cancellationToken).ConfigureAwait(false);
}
}
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
{
throw;
}
catch (Exception ex)
{
_logger?.LogWarning(
ex,
"MQTT driver '{DriverId}': could not reconcile the Sparkplug primary-host claim for '{HostId}'; will retry.",
_driverId,
_hostId);
}
}
private async Task ClaimAsync(CancellationToken cancellationToken)
{
if (Volatile.Read(ref _connection) is not null)
{
return; // Already held; the connection's own supervisor keeps it that way.
}
await _gate.WaitAsync(cancellationToken).ConfigureAwait(false);
try
{
if (_connection is not null || Volatile.Read(ref _disposed) != 0)
{
return;
}
Interlocked.Increment(ref _claimAttempts);
// ONE timestamp for both bodies — the Will is built here, before CONNECT, and the birth
// published after CONNACK. See SparkplugStatePayload's remarks on why they must match.
var session = SparkplugStatePayload.ForSession(
_hostId, _timeProvider.GetUtcNow().ToUnixTimeMilliseconds());
var connection = new MqttConnection(
_options,
_driverId,
_logger,
clientIdSuffix: ClientIdSuffix,
lastWill: new MqttLastWill(
session.Topic, session.OfflineBody, SparkplugStatePayload.StateQos, Retain: true),
// Forced clean, ignoring the driver's configured value: a broker that resumes a prior
// session for this client id may keep the OLD session's Will and discard the one this
// CONNECT carries — leaving a claim with no death certificate behind it, which is the
// single state this whole mechanism exists to make impossible.
cleanSessionOverride: true);
// Re-assert on every reconnect. A broker restart drops retained messages; without this the
// host would be connected and believed offline by every edge node, silently.
connection.Reconnected += async ct =>
{
await connection.PublishAsync(
session.Topic, session.OnlineBody, SparkplugStatePayload.StateQos, retain: true, ct)
.ConfigureAwait(false);
_logger?.LogInformation(
"MQTT driver '{DriverId}': re-published Sparkplug primary-host ONLINE for '{HostId}' after reconnect.",
_driverId,
_hostId);
};
await connection.ConnectAsync(cancellationToken).ConfigureAwait(false);
try
{
await connection.PublishAsync(
session.Topic, session.OnlineBody, SparkplugStatePayload.StateQos, retain: true, cancellationToken)
.ConfigureAwait(false);
}
catch
{
// A connection whose birth never landed is worse than no connection: it holds a Will
// that will publish OFFLINE for a claim that was never asserted, and it looks claimed
// to IsClaimed. Tear it down and let the next tick retry from scratch.
await connection.DisposeAsync().ConfigureAwait(false);
throw;
}
_session = session;
Volatile.Write(ref _connection, connection);
_logger?.LogInformation(
"MQTT driver '{DriverId}': claimed Sparkplug primary host '{HostId}' — retained ONLINE published on '{Topic}', "
+ "OFFLINE armed as the Last Will.",
_driverId,
_hostId,
session.Topic);
}
finally
{
_gate.Release();
}
}
private async Task ReleaseAsync(string reason, CancellationToken cancellationToken)
{
if (Volatile.Read(ref _connection) is null)
{
return;
}
await _gate.WaitAsync(CancellationToken.None).ConfigureAwait(false);
try
{
var connection = Interlocked.Exchange(ref _connection, null);
var session = _session;
_session = null;
if (connection is null || session is null)
{
return;
}
try
{
// Explicit OFFLINE before disconnecting. The Will would eventually say the same thing,
// but only after the broker's keep-alive timeout notices the socket is gone — up to
// 1.5 keep-alive periods of edge nodes believing a host that has already stood down is
// still online. Publishing it makes the transition immediate.
await connection.PublishAsync(
session.Topic, session.OfflineBody, SparkplugStatePayload.StateQos, retain: true, cancellationToken)
.ConfigureAwait(false);
_logger?.LogInformation(
"MQTT driver '{DriverId}': released Sparkplug primary host '{HostId}' ({Reason}) — retained OFFLINE published.",
_driverId,
_hostId,
reason);
}
catch (Exception ex)
{
// Not fatal, and not silent. The Will still covers this: dropping the connection below
// makes the broker publish the identical OFFLINE body. The only thing lost is promptness.
_logger?.LogWarning(
ex,
"MQTT driver '{DriverId}': could not publish the Sparkplug primary-host OFFLINE for '{HostId}'; "
+ "the Last Will will deliver it when the session drops.",
_driverId,
_hostId);
}
await connection.DisposeAsync().ConfigureAwait(false);
}
finally
{
_gate.Release();
}
}
}
@@ -25,6 +25,10 @@
<ItemGroup>
<InternalsVisibleTo Include="ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Tests"/>
<!-- The Sparkplug primary-host live gate (Gitea #511) drives AbandonForTests: the Will leg is
unreachable through the ordinary lifecycle, because a clean DISCONNECT makes the broker
discard the Will by protocol rule. -->
<InternalsVisibleTo Include="ZB.MOM.WW.OtOpcUa.Driver.Mqtt.IntegrationTests"/>
</ItemGroup>
</Project>
@@ -225,7 +225,7 @@
<div class="col-md-3">
<label class="form-label" for="mqtt-spb-host">Host ID (optional)</label>
<InputText id="mqtt-spb-host" @bind-Value="_form.SparkplugHostId" @bind-Value:after="EmitAsync" class="form-control form-control-sm mono" placeholder="scada-primary" />
<div class="form-text">Sparkplug Host Application identity. Receive-only in this version.</div>
<div class="form-text">Sparkplug Host Application identity — the <code class="mono">spBv1.0/STATE/&#123;hostId&#125;</code> topic. Required to act as Primary Host.</div>
</div>
<div class="col-md-3 d-flex flex-column justify-content-center">
<div class="form-check form-switch">
@@ -237,25 +237,42 @@
<InputCheckbox id="mqtt-spb-primary" @bind-Value="_form.SparkplugActAsPrimaryHost" @bind-Value:after="EmitAsync" class="form-check-input" />
<label class="form-check-label" for="mqtt-spb-primary">
Act as Primary Host
<span class="badge text-bg-warning">not implemented</span>
</label>
</div>
</div>
@if (_form.SparkplugActAsPrimaryHost)
{
@* Shown rather than the flag being hidden: an operator who needs a primary host must
learn it is absent HERE, not from a plant that never sees a STATE message. The
driver logs the same warning at startup so the two cannot drift. *@
<div class="col-12">
<div class="alert alert-warning py-2 px-3 mb-0 small" role="alert">
<strong>Primary-host STATE publishing is not implemented.</strong> This driver is
receive-only: it will not publish <code class="mono">spBv1.0/STATE/@(string.IsNullOrWhiteSpace(_form.SparkplugHostId) ? "{HostId}" : _form.SparkplugHostId)</code>,
so edge nodes will not treat it as their primary host. The driver logs a warning at
startup rather than being silently inert. Leave this off unless you are staging
configuration ahead of that feature.
@* Two distinct messages, and which one shows is decided by SparkplugPrimaryHostAdvice —
a plain static, not markup logic, because this repo has no bUnit and a Razor-only rule
is untestable. A blank host id is a REFUSAL (there is no topic to publish on) and must
not read like the ordinary case; the driver logs the same distinction at startup so the
two surfaces cannot drift. *@
@if (SparkplugPrimaryHostAdvice.DescribeBlockingProblem(_form.SparkplugHostId) is { } problem)
{
<div class="col-12">
<div class="alert alert-warning py-2 px-3 mb-0 small" role="alert">
<strong>This host claim cannot be made.</strong> @problem
</div>
</div>
</div>
}
else
{
<div class="col-12">
<div class="alert alert-info py-2 px-3 mb-0 small" role="alert">
This node will publish a retained <code class="mono">ONLINE</code> on
<code class="mono">spBv1.0/STATE/@_form.SparkplugHostId.Trim()</code> and arm a matching
<code class="mono">OFFLINE</code> as its MQTT Last Will, so edge nodes learn promptly
whether the host is alive — including after a crash.
<strong>Only the node holding the redundancy Primary role claims it.</strong>
Two nodes claiming one host id would let a <em>Secondary's</em> death publish
<code class="mono">OFFLINE</code> over a healthy Primary and stop the whole group, so a
failover leaves a brief window in which edge nodes correctly see no host.
At most one primary host may exist per host id per broker — do not set this on two
separate deployments pointing at the same broker.
</div>
</div>
}
}
<div class="col-12">
@@ -263,7 +280,6 @@
Sparkplug tags bind by the <code class="mono">group / edge node / device / metric</code>
tuple, not by topic — author them with <strong>Browse device…</strong> on a device under
this driver, which builds its tree from observed birth certificates.
<strong>MQTT tags are read-only</strong> (write-through is not implemented).
</div>
</div>
}
@@ -142,17 +142,23 @@ public sealed class MqttDriverFormModel
public string SparkplugGroupId { get; set; } = "";
/// <summary>
/// Sparkplug mode: the Host Application identity. Optional, and <b>receive-only</b> in this
/// version — see <see cref="SparkplugActAsPrimaryHost"/>.
/// Sparkplug mode: the Host Application identity — the <c>spBv1.0/STATE/{hostId}</c> topic the
/// driver observes, and publishes on when <see cref="SparkplugActAsPrimaryHost"/> is set.
/// Optional for observation; <b>required</b> to claim.
/// </summary>
public string SparkplugHostId { get; set; } = "";
/// <summary>
/// Sparkplug mode: claim the Primary Host Application role. <b>Not implemented</b> — STATE
/// publishing is out of scope, and the driver logs a warning rather than being silently inert
/// when this is set. Surfaced (rather than hidden) so an operator who needs it learns that from
/// the form instead of from a quiet plant.
/// Sparkplug mode: claim the Primary Host Application role (Gitea #511) — a retained
/// <c>ONLINE</c>, a matching <c>OFFLINE</c> armed as the MQTT Last Will, and an explicit
/// <c>OFFLINE</c> on clean shutdown.
/// </summary>
/// <remarks>
/// <b>Primary-gated:</b> only the node holding the redundancy Primary role claims it, so the two
/// nodes of a pair cannot fight over one host id. At most one primary host may exist per host id
/// per broker, which the form states — it is not enforceable from here, since a second
/// deployment pointing at the same broker is invisible to this one.
/// </remarks>
public bool SparkplugActAsPrimaryHost { get; set; }
/// <summary>Sparkplug mode: answer a detected sequence-number gap with a rebirth request (NCMD).</summary>
@@ -0,0 +1,35 @@
namespace ZB.MOM.WW.OtOpcUa.AdminUI.Components.Shared.Drivers.Forms;
/// <summary>
/// The rule deciding which primary-host advisory <c>MqttDriverForm</c> shows when
/// <c>Act as Primary Host</c> is on (Gitea #511).
/// </summary>
/// <remarks>
/// <para>
/// <b>A plain static rather than markup logic, because this repo has no bUnit.</b> A condition
/// written inline in a <c>.razor</c> file is unreachable from any test, and the failure mode
/// here is specific: an operator who ticks the box with a blank host id must see a
/// <i>refusal</i>, not the ordinary "here is what will happen" note. Getting that backwards
/// tells them a claim is being made when the driver is in fact declining to make one.
/// </para>
/// <para>
/// Deliberately mirrors <c>SparkplugPrimaryHost.DescribeUnusable</c>'s blank-host-id arm. The
/// mode arm is not duplicated: this panel only renders inside the SparkplugB branch of the
/// form, so mode cannot be wrong here.
/// </para>
/// </remarks>
public static class SparkplugPrimaryHostAdvice
{
/// <summary>
/// Describes why a primary-host claim cannot be made for <paramref name="hostId"/>, or
/// <see langword="null"/> when it can.
/// </summary>
/// <param name="hostId">The Sparkplug host id as typed into the form.</param>
/// <returns>An operator-facing reason, or <see langword="null"/>.</returns>
public static string? DescribeBlockingProblem(string? hostId) =>
string.IsNullOrWhiteSpace(hostId)
? "A Sparkplug host claim is published on spBv1.0/STATE/{hostId}, so it needs a Host ID. "
+ "Set one above, or turn this off — the driver will otherwise log a warning at startup "
+ "and publish nothing."
: null;
}
@@ -55,7 +55,16 @@ public static class DriverFactoryBootstrap
// The Galaxy driver resolves its Gateway.ApiKeySecretRef secret: arm through the
// shared ISecretResolver (registered unconditionally by AddZbSecrets on every node).
var secretResolver = sp.GetRequiredService<ISecretResolver>();
Register(registry, loggerFactory, secretResolver, scriptRoot);
// Gitea #511: the MQTT driver's Sparkplug primary-host claim is Primary-gated, and unlike
// every other gated operation it is a STANDING assertion on the driver's own connection —
// there is no inbound message for DriverHostActor to intercept — so the verdict has to
// reach the driver itself. Registered unconditionally on driver-role nodes by
// AddOtOpcUaRuntime. GetRequiredService, not GetService: a null here silently falls the
// driver back to "claim unconditionally", which on a redundant pair is the dual-claim this
// whole mechanism exists to prevent, and it would fail as a plant outage rather than as a
// startup error.
var primaryRoleSource = sp.GetRequiredService<IDriverPrimaryRoleSource>();
Register(registry, loggerFactory, secretResolver, primaryRoleSource, scriptRoot);
return registry;
});
services.AddSingleton<IDriverFactory>(sp =>
@@ -142,6 +151,7 @@ public static class DriverFactoryBootstrap
DriverFactoryRegistry registry,
ILoggerFactory? loggerFactory,
ISecretResolver secretResolver,
IDriverPrimaryRoleSource primaryRoleSource,
ZB.MOM.WW.OtOpcUa.Core.Scripting.ScriptRootLogger? scriptRoot = null)
{
Driver.AbCip.AbCipDriverFactoryExtensions.Register(registry);
@@ -150,7 +160,7 @@ public static class DriverFactoryBootstrap
Driver.FOCAS.FocasDriverFactoryExtensions.Register(registry);
Driver.Galaxy.GalaxyDriverFactoryExtensions.Register(registry, secretResolver, loggerFactory);
Driver.Modbus.ModbusDriverFactoryExtensions.Register(registry, loggerFactory);
Driver.Mqtt.MqttDriverFactoryExtensions.Register(registry, loggerFactory);
Driver.Mqtt.MqttDriverFactoryExtensions.Register(registry, loggerFactory, primaryRoleSource);
// Tier A (the Register default): fully managed — HttpClient + System.Xml.Linq, no native SDK,
// no COM. Tier C is the only tier that arms process-level recycle (MemoryRecycle hard-breach /
// ScheduledRecycleScheduler), which would be wrong here: the driver is in-process and killing
@@ -273,6 +273,19 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
/// today, the alarm store-and-forward drain. Null on nodes that do not wire one.</summary>
private readonly IRedundancyRoleView? _redundancyRoleView;
/// <summary>
/// Where the Primary <b>data-plane</b> verdict is published for drivers that must know it
/// directly (Gitea #511). Null on nodes that wire no redundancy manager.
/// </summary>
/// <remarks>
/// Note the different verdict from <see cref="_redundancyRoleView"/>: that one carries
/// <see cref="PrimaryGatePolicy.ShouldDrainAlarmHistory"/>, which OPENS on an unknown role. A
/// Sparkplug host-id claim is an exclusive assertion on a shared broker, so it takes the
/// data-plane verdict, which closes. Publishing the drain verdict here would let both nodes of
/// a pair claim the host id during every boot window.
/// </remarks>
private readonly IDriverPrimaryRoleSource? _primaryRoleSource;
/// <summary>
/// Host of this node's LocalDb replication partner — the only node that holds a copy of this
/// node's alarm queue, and therefore the only node it may ever stand down in favour of.
@@ -408,7 +421,8 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
bool fetchAndCacheMode = false,
IDeploymentArtifactFetcher? artifactFetcher = null,
IAlarmStateStore? alarmStateStore = null,
ITelemetryLocalHub? telemetryHub = null) =>
ITelemetryLocalHub? telemetryHub = null,
IDriverPrimaryRoleSource? primaryRoleSource = null) =>
// WARNING: this forwarding list is POSITIONAL, and Props.Create compiles it into an
// expression tree. Six IActorRef? parameters and several interface-typed ones mean a
// mis-ordered argument is usually type-compatible and therefore compiles clean, then binds
@@ -419,7 +433,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
healthPublisher, virtualTagEvaluator, historyWriter, virtualTagHostOverride,
loggerFactory, scriptRootLogger, scriptedAlarmHostOverride, invokerFactory, driverMemberCountProvider,
deploymentArtifactCache, redundancyRoleView, replicationPeerHost, fetchAndCacheMode, artifactFetcher,
alarmStateStore, telemetryHub));
alarmStateStore, telemetryHub, primaryRoleSource));
/// <summary>Initializes a new DriverHostActor with the specified dependencies.</summary>
/// <param name="dbFactory">Database context factory for configuration database access.</param>
@@ -460,6 +474,12 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
/// <param name="telemetryHub">Per-cluster mesh Phase 5: optional node-local live-telemetry hub each
/// Primary-gated native-alarm transition is also emitted into, and which is threaded into the spawned
/// VirtualTag + ScriptedAlarm hosts. Null (admin-only nodes / tests) skips the emit and passes null down.</param>
/// <param name="primaryRoleSource">Gitea #511: where the Primary <i>data-plane</i> verdict is published
/// for drivers that assert a standing exclusive identity on a shared external system — today the MQTT
/// driver's Sparkplug primary-host STATE claim. Deliberately a separate parameter from
/// <paramref name="redundancyRoleView"/> even though the production object implements both: a test that
/// supplies only a role-view double must not silently also satisfy this, or its primary-role assertions
/// pass without the publish ever happening. Null on nodes that wire no redundancy manager.</param>
public DriverHostActor(
IDbContextFactory<OtOpcUaConfigDbContext>? dbFactory,
CommonsNodeId localNode,
@@ -483,11 +503,13 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
bool fetchAndCacheMode = false,
IDeploymentArtifactFetcher? artifactFetcher = null,
IAlarmStateStore? alarmStateStore = null,
ITelemetryLocalHub? telemetryHub = null)
ITelemetryLocalHub? telemetryHub = null,
IDriverPrimaryRoleSource? primaryRoleSource = null)
{
_telemetryHub = telemetryHub;
_deploymentArtifactCache = deploymentArtifactCache;
_redundancyRoleView = redundancyRoleView;
_primaryRoleSource = primaryRoleSource;
_replicationPeerHost = replicationPeerHost;
_fetchAndCacheMode = fetchAndCacheMode;
_artifactFetcher = artifactFetcher;
@@ -1331,6 +1353,12 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
_redundancyRoleView?.Publish(
PrimaryGatePolicy.ShouldDrainAlarmHistory(_localRole, queueSharingPeerIsPrimary: peerIsPrimary));
// The DATA-PLANE verdict, deliberately — the same one HandleRouteNodeWrite gates on, not the
// drain verdict published immediately above. A driver reading this claims an exclusive identity
// on a shared external system (Gitea #511: the Sparkplug host id), where two claimants is the
// harm, so an unknown role with a driver peer present must deny.
_primaryRoleSource?.PublishPrimaryRole(ShouldServiceAsPrimary());
}
private void Stale()
@@ -1,3 +1,4 @@
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Runtime.Drivers;
namespace ZB.MOM.WW.OtOpcUa.Runtime.Redundancy;
@@ -39,8 +40,22 @@ public interface IRedundancyRoleView
void Publish(bool shouldDrainAlarmHistory);
}
/// <summary>Thread-safe <see cref="IRedundancyRoleView"/> backed by a volatile field.</summary>
public sealed class RedundancyRoleView : IRedundancyRoleView
/// <summary>
/// Thread-safe <see cref="IRedundancyRoleView"/> + <see cref="IDriverPrimaryRoleSource"/>, each
/// backed by its own volatile field.
/// </summary>
/// <remarks>
/// <para>
/// <b>Two verdicts, one object, on purpose.</b> Both are published from the same
/// <c>DriverHostActor</c> snapshot handler and both are read off-actor, so a second class
/// would duplicate the publish plumbing to hold a second bool. They are kept as distinct
/// <i>properties</i> rather than collapsed into one because they are genuinely different
/// decisions from different policies — see <see cref="IRedundancyRoleView"/>'s remarks on why
/// the drain gate must open on an unknown role where the data-plane gate closes. Collapsing
/// them is the bug that live gate already found once.
/// </para>
/// </remarks>
public sealed class RedundancyRoleView : IRedundancyRoleView, IDriverPrimaryRoleSource
{
// Seeded with the decision for "role unknown" rather than a bare false. An unpublished view and a
// node whose role nothing ever reports are genuinely the same situation, and they must behave the
@@ -49,9 +64,22 @@ public sealed class RedundancyRoleView : IRedundancyRoleView
private volatile bool _shouldDrainAlarmHistory =
PrimaryGatePolicy.ShouldDrainAlarmHistory(localRole: null, queueSharingPeerIsPrimary: false);
// Seeded from the same policy that will later publish here, for the unknown-role/no-peer case:
// a node nothing has reported a role for, on a cluster with no driver peer, is a single-node
// deployment and may claim. The moment a snapshot arrives this is replaced by the real verdict,
// and a multi-driver cluster's unknown role denies.
private volatile bool _shouldActAsPrimary =
PrimaryGatePolicy.ShouldServiceAsPrimary(localRole: null, driverMemberCount: 1);
/// <inheritdoc/>
public bool ShouldDrainAlarmHistory => _shouldDrainAlarmHistory;
/// <inheritdoc/>
public bool ShouldActAsPrimary => _shouldActAsPrimary;
/// <inheritdoc/>
public void Publish(bool shouldDrainAlarmHistory) => _shouldDrainAlarmHistory = shouldDrainAlarmHistory;
/// <inheritdoc/>
public void PublishPrimaryRole(bool shouldActAsPrimary) => _shouldActAsPrimary = shouldActAsPrimary;
}
@@ -75,6 +75,22 @@ public static class ServiceCollectionExtensions
// telemetry (never DPS) to the Phase-5 gRPC streaming service central dials in on. AddOtOpcUaRuntime
// runs only inside Program.cs's hasDriver block, so this lands on driver-role nodes only.
services.TryAddSingleton<ITelemetryLocalHub, TelemetryLocalHub>();
// Gitea #511: the Primary data-plane verdict, published for drivers that assert a standing
// exclusive identity on a shared external system (today the MQTT driver's Sparkplug host id).
//
// Registered HERE, unconditionally on every driver-role node, and NOT alongside the alarm
// drain's IRedundancyRoleView — which AddAlarmHistorian registers only when that section is
// enabled. Piggybacking on it would leave a redundant pair with no alarm historian configured
// with no primary-role source at all, and the driver's documented no-source fallback is
// "claim" — so both nodes of that pair would claim the same host id and their Wills would
// fight over the retained STATE. Exactly the failure #511 exists to prevent, reached by a
// registration accident in an unrelated section.
//
// Both interfaces are forwarded to ONE RedundancyRoleView instance, so a node that has both
// publishes two verdicts into the same object rather than into two that disagree.
services.TryAddSingleton<RedundancyRoleView>();
services.TryAddSingleton<IDriverPrimaryRoleSource>(sp => sp.GetRequiredService<RedundancyRoleView>());
return services;
}
@@ -117,8 +133,10 @@ public static class ServiceCollectionExtensions
// The view is a plain singleton so it exists even on nodes whose DriverHostActor never
// publishes to it — an unpublished view reads as "no peer, drain", which is the correct
// posture for a deployment that runs no redundancy at all.
services.TryAddSingleton<IRedundancyRoleView, RedundancyRoleView>();
// posture for a deployment that runs no redundancy at all. Forwarded to the same instance
// AddOtOpcUaRuntime registered for IDriverPrimaryRoleSource: one object, two verdicts.
services.TryAddSingleton<RedundancyRoleView>();
services.TryAddSingleton<IRedundancyRoleView>(sp => sp.GetRequiredService<RedundancyRoleView>());
// THE GATE ONLY APPLIES TO A REPLICATED QUEUE. Standing down is only ever safe because some
// other node holds the same rows and will send them instead — and the only node that does is
@@ -499,7 +517,11 @@ public static class ServiceCollectionExtensions
// Phase 5: the node-local live-telemetry hub (registered in AddOtOpcUaRuntime). The host
// emits its Primary-gated native-alarm transitions into it and threads it into the
// VirtualTag + ScriptedAlarm hosts it spawns.
telemetryHub: resolver.GetService<ITelemetryLocalHub>()),
telemetryHub: resolver.GetService<ITelemetryLocalHub>(),
// Gitea #511: where this actor publishes its Primary DATA-PLANE verdict for drivers
// that claim an exclusive external identity. Registered unconditionally by
// AddOtOpcUaRuntime, unlike redundancyRoleView above — see the comment there.
primaryRoleSource: resolver.GetService<IDriverPrimaryRoleSource>()),
DriverHostActorName);
registry.Register<DriverHostActorKey>(driverHost);