dohertj2/lmxopcua
1
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(mqtt): Sparkplug primary-host STATE publishing (ActAsPrimaryHost) #511

New Issue
Closed
opened 2026-07-27 13:50:30 -04:00 by dohertj2 · 2 comments
dohertj2 commented 2026-07-27 13:50:30 -04:00
Owner
Copy Link
Copy Source

MqttSparkplugOptions.ActAsPrimaryHost (Driver.Mqtt.Contracts/MqttDriverOptions.cs:193) is accepted by
config and by the AdminUI driver form, but does nothing. Setting it logs a startup warning
(MqttDriver.cs:933) rather than sitting silently inert.

Why it shipped unimplemented rather than half-implemented. Being a Sparkplug primary host is a three-part
contract:

  1. a retained STATE message asserting ONLINE,
  2. a matching OFFLINE registered as the MQTT Last Will, at connect time,
  3. an explicit OFFLINE published on clean shutdown.

Shipping (1) without (2) is strictly worse than shipping neither: a crashed node leaves a retained ONLINE
on the broker asserting forever that a dead host is alive, and edge nodes trusting that assertion will not
take their own offline actions. The warning is the deliberate middle ground.

Extra consideration for this server: both redundancy pair nodes subscribe independently (MQTT fan-out is
per-connection, and the Primary gate applies downstream of ingest, not to it). Two nodes both claiming primary
host on the same host id would fight over the retained STATE. An implementation must decide whether
primary-host duty is Primary-gated — and if it is, whether a failover republishes ONLINE fast enough to
avoid an edge-visible flap.

Acceptance. With ActAsPrimaryHost: true, the broker holds a retained ONLINE while the node runs; a
kill -9 of the node leaves OFFLINE (the Will fires); a clean shutdown leaves OFFLINE. Verified live
against the Mosquitto fixture, and the pair-node question answered explicitly rather than left to chance.

`MqttSparkplugOptions.ActAsPrimaryHost` (`Driver.Mqtt.Contracts/MqttDriverOptions.cs:193`) is accepted by config and by the AdminUI driver form, but does nothing. Setting it **logs a startup warning** (`MqttDriver.cs:933`) rather than sitting silently inert. **Why it shipped unimplemented rather than half-implemented.** Being a Sparkplug primary host is a three-part contract: 1. a **retained** `STATE` message asserting `ONLINE`, 2. a matching `OFFLINE` registered as the MQTT **Last Will**, at *connect time*, 3. an explicit `OFFLINE` published on clean shutdown. Shipping (1) without (2) is strictly **worse than shipping neither**: a crashed node leaves a retained `ONLINE` on the broker asserting forever that a dead host is alive, and edge nodes trusting that assertion will not take their own offline actions. The warning is the deliberate middle ground. **Extra consideration for this server:** both redundancy pair nodes subscribe independently (MQTT fan-out is per-connection, and the Primary gate applies downstream of ingest, not to it). Two nodes both claiming primary host on the same host id would fight over the retained `STATE`. An implementation must decide whether primary-host duty is **Primary-gated** — and if it is, whether a failover republishes `ONLINE` fast enough to avoid an edge-visible flap. **Acceptance.** With `ActAsPrimaryHost: true`, the broker holds a retained `ONLINE` while the node runs; a `kill -9` of the node leaves `OFFLINE` (the Will fires); a clean shutdown leaves `OFFLINE`. Verified live against the Mosquitto fixture, and the pair-node question answered explicitly rather than left to chance.
dohertj2 added the driver/mqtt label 2026-07-27 13:50:30 -04:00
dohertj2 commented 2026-07-31 06:46:01 -04:00
Author
Owner
Copy Link
Copy Source

Decision (owner, 2026-07-31): primary-host duty is Primary-gated. Not yet implemented — recording
the decision and a constraint the issue does not cover, found while scoping.

The pair-node question, answered

Gating is not optional. The issue notes two nodes would "fight over the retained STATE", but the sharper
failure is this: if both nodes claim the host id and the Secondary dies, its Will publishes
OFFLINE, edge nodes stop publishing, and the surviving Primary is still perfectly healthy. Redundancy
would cause the plant outage the mechanism exists to prevent.

⚠️ Constraint the issue misses: the Will cannot simply be registered on both nodes

The Will is registered at CONNECT time, but the Primary role changes after connect. The two
obvious shortcuts are both wrong:

  • Register the Will on every node — a Secondary's disconnect then fires OFFLINE and clobbers the
    Primary's retained ONLINE. This is the failure above.
  • Register it only on the node that is Primary at connect, and leave it alone afterwards — a node
    that later gains Primary and publishes ONLINE has no Will, so if it crashes it leaves a retained
    ONLINE asserting a dead host is alive. That is precisely the state the three-part contract exists to
    prevent, and it is what makes a partial implementation worse than none.

So the STATE claim must be bound to the connection, and the connection rebuilt on role change: on
gaining Primary, reconnect with the Will registered and publish retained ONLINE; on losing it,
reconnect without a Will so a later disconnect cannot clobber the new Primary's ONLINE. The driver
already has a session-rebuild path (ReinitializeAsync rebuilds when the broker identity changes).

Accepted trade, explicitly: a brief edge-visible OFFLINE window during failover while the new Primary
reconnects and republishes. That is a genuine state transition, not a glitch to hide.

Seam required

The driver layer has no notion of redundancy, by design. IRedundancyRoleView — the existing precedent,
used for the alarm store-and-forward drain gate — lives in Runtime, which drivers cannot reference. So
this needs a small driver-facing role source in Core.Abstractions, implemented in Runtime off the
snapshot DriverHostActor already publishes, and injected through MqttDriverFactoryExtensions.

Do not ship any leg of this without all three. The retained ONLINE, the Will, and the clean-shutdown
OFFLINE are one contract.

**Decision (owner, 2026-07-31): primary-host duty is Primary-gated.** Not yet implemented — recording the decision and a constraint the issue does not cover, found while scoping. ### The pair-node question, answered Gating is not optional. The issue notes two nodes would "fight over the retained STATE", but the sharper failure is this: if both nodes claim the host id and the **Secondary** dies, its Will publishes `OFFLINE`, edge nodes stop publishing, and the surviving Primary is still perfectly healthy. Redundancy would *cause* the plant outage the mechanism exists to prevent. ### ⚠️ Constraint the issue misses: the Will cannot simply be registered on both nodes The Will is registered **at CONNECT time**, but the Primary role changes **after** connect. The two obvious shortcuts are both wrong: - **Register the Will on every node** — a Secondary's disconnect then fires `OFFLINE` and clobbers the Primary's retained `ONLINE`. This is the failure above. - **Register it only on the node that is Primary at connect, and leave it alone afterwards** — a node that later *gains* Primary and publishes `ONLINE` has no Will, so if it crashes it leaves a retained `ONLINE` asserting a dead host is alive. That is precisely the state the three-part contract exists to prevent, and it is what makes a partial implementation worse than none. So **the STATE claim must be bound to the connection, and the connection rebuilt on role change**: on gaining Primary, reconnect with the Will registered and publish retained `ONLINE`; on losing it, reconnect *without* a Will so a later disconnect cannot clobber the new Primary's `ONLINE`. The driver already has a session-rebuild path (`ReinitializeAsync` rebuilds when the broker identity changes). Accepted trade, explicitly: a brief edge-visible `OFFLINE` window during failover while the new Primary reconnects and republishes. That is a genuine state transition, not a glitch to hide. ### Seam required The driver layer has no notion of redundancy, by design. `IRedundancyRoleView` — the existing precedent, used for the alarm store-and-forward drain gate — lives in `Runtime`, which drivers cannot reference. So this needs a small driver-facing role source in `Core.Abstractions`, implemented in `Runtime` off the snapshot `DriverHostActor` already publishes, and injected through `MqttDriverFactoryExtensions`. **Do not ship any leg of this without all three.** The retained `ONLINE`, the Will, and the clean-shutdown `OFFLINE` are one contract.
dohertj2 commented 2026-07-31 08:14:37 -04:00
Author
Owner
Copy Link
Copy Source

Implemented and live-gated. master 857a10b4 (feature commit 7f2108dc).

All three legs, or none — as the issue required. SparkplugPrimaryHost publishes a retained
{"online":true,...} on spBv1.0/STATE/{hostId} at QoS 1, arms the matching OFFLINE as the MQTT
Last Will, and publishes an explicit OFFLINE on a clean stand-down.

The design the constraint forced

A Will can only be registered in the CONNECT packet — it cannot be added to a live session, changed,
or withdrawn. So the claim gets its own connection (client id {clientId}-spbhost, forced
clean-session) whose lifetime is the claim: opened on gaining Primary, closed on losing it. Every
leg then follows from the connection's lifecycle rather than from bookkeeping that could drift.

That also turned out better than the "rebuild the ingest connection on role change" shape recorded
earlier: rebuilding ingest drops every subscription and the whole birth cache on each role change,
producing a data gap at the Secondary→Primary transition — the moment a plant most needs values
flowing. A second connection costs one client id and leaves ingest untouched.

Primary-gated

New seam IDriverPrimaryRoleSource in Core.Abstractions, carrying the data-plane verdict
(PrimaryGatePolicy.ShouldServiceAsPrimary) rather than the alarm-drain one — an unknown role with a
driver peer present must deny, because two claimants is the harm. Polled on a 1s timer rather than
evented: a process singleton raising events at per-instance drivers outlives every driver it holds a
reference to.

⚠️ One wiring trap worth recording. It is registered by AddOtOpcUaRuntime, deliberately not
alongside the alarm drain's IRedundancyRoleView — which AddAlarmHistorian registers only when that
section is enabled. Piggybacking would have left a redundant pair with no alarm historian configured
falling back to the documented no-source default, which is claim. Both nodes would then claim the
same host id, reached purely by a registration accident in an unrelated section.

Live gate — 5/5 against the Mosquitto fixture

  1. retained ONLINE visible to a subscriber that joins after the birth;
  2. abrupt death leaves OFFLINE — the Will fires, and its timestamp matches the birth's;
  3. clean shutdown leaves OFFLINE promptly, without waiting out a keep-alive;
  4. a node that is not Primary never opens a claim connection at all (asserted on connect
    attempts, not on IsClaimed — see below);
  5. failover publishes OFFLINE then re-publishes ONLINE on failback.

Leg 2 needed a seam: DisposeAsync sends a clean DISCONNECT, which by protocol rule makes the
broker discard the Will — so a test driven through the ordinary lifecycle can only ever observe
the explicit stand-down OFFLINE, and would pass against an implementation that never armed a Will.
AbandonForTests drops the socket with no DISCONNECT. Falsified: removing the Will turns that leg,
and only that leg, red.

Leg 4 had the same problem in miniature. "Start a Secondary, assert IsClaimed is false" passes
identically whether the gate works or the gate is deleted and the connect merely fails against an
unreachable broker — so it asserts on attempt count, with the Primary case as the falsifying
control on the same broker.

Accepted trade

A brief edge-visible OFFLINE window during failover, while the old Primary stands down and the new
one connects and republishes. That is a genuine state transition — for that window there really is no
host — not a glitch to hide. Stated on the AdminUI form, which also replaces the "not implemented"
badge.

Docs: docs/drivers/Mqtt.md §"Primary host". Two stale claims corrected en route: the form still said
MQTT tags are read-only (#508 shipped), and the driver doc still said nothing consumes
IRediscoverable.

**Implemented and live-gated.** `master` `857a10b4` (feature commit `7f2108dc`). All three legs, or none — as the issue required. `SparkplugPrimaryHost` publishes a retained `{"online":true,...}` on `spBv1.0/STATE/{hostId}` at QoS 1, arms the matching `OFFLINE` as the MQTT Last Will, and publishes an explicit `OFFLINE` on a clean stand-down. ### The design the constraint forced A Will can only be registered in the CONNECT packet — it cannot be added to a live session, changed, or withdrawn. So the claim gets **its own connection** (client id `{clientId}-spbhost`, forced clean-session) whose lifetime *is* the claim: opened on gaining Primary, closed on losing it. Every leg then follows from the connection's lifecycle rather than from bookkeeping that could drift. That also turned out better than the "rebuild the ingest connection on role change" shape recorded earlier: rebuilding ingest drops every subscription and the whole birth cache on each role change, producing a data gap at the Secondary→Primary transition — the moment a plant most needs values flowing. A second connection costs one client id and leaves ingest untouched. ### Primary-gated New seam `IDriverPrimaryRoleSource` in `Core.Abstractions`, carrying the **data-plane** verdict (`PrimaryGatePolicy.ShouldServiceAsPrimary`) rather than the alarm-drain one — an unknown role with a driver peer present must deny, because two claimants is the harm. Polled on a 1s timer rather than evented: a process singleton raising events at per-instance drivers outlives every driver it holds a reference to. ⚠️ **One wiring trap worth recording.** It is registered by `AddOtOpcUaRuntime`, deliberately *not* alongside the alarm drain's `IRedundancyRoleView` — which `AddAlarmHistorian` registers only when that section is enabled. Piggybacking would have left a redundant pair with no alarm historian configured falling back to the documented no-source default, which is *claim*. Both nodes would then claim the same host id, reached purely by a registration accident in an unrelated section. ### Live gate — 5/5 against the Mosquitto fixture 1. retained `ONLINE` visible to a subscriber that joins *after* the birth; 2. **abrupt death leaves `OFFLINE`** — the Will fires, and its timestamp matches the birth's; 3. clean shutdown leaves `OFFLINE` promptly, without waiting out a keep-alive; 4. a node that is not Primary **never opens a claim connection at all** (asserted on connect *attempts*, not on `IsClaimed` — see below); 5. failover publishes `OFFLINE` then re-publishes `ONLINE` on failback. Leg 2 needed a seam: `DisposeAsync` sends a clean `DISCONNECT`, which by protocol rule makes the broker **discard** the Will — so a test driven through the ordinary lifecycle can only ever observe the explicit stand-down `OFFLINE`, and would pass against an implementation that never armed a Will. `AbandonForTests` drops the socket with no DISCONNECT. Falsified: removing the Will turns that leg, and only that leg, red. Leg 4 had the same problem in miniature. "Start a Secondary, assert `IsClaimed` is false" passes identically whether the gate works or the gate is deleted and the connect merely fails against an unreachable broker — so it asserts on attempt *count*, with the Primary case as the falsifying control on the same broker. ### Accepted trade A brief edge-visible `OFFLINE` window during failover, while the old Primary stands down and the new one connects and republishes. That is a genuine state transition — for that window there really is no host — not a glitch to hide. Stated on the AdminUI form, which also replaces the "not implemented" badge. Docs: `docs/drivers/Mqtt.md` §"Primary host". Two stale claims corrected en route: the form still said MQTT tags are read-only (#508 shipped), and the driver doc still said nothing consumes `IRediscoverable`.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
auto-managed
queue management
 cross-driver
queue management
 driver/abcip
queue management
 driver/ablegacy
queue management
 driver/focas
queue management
 driver/mqtt
MQTT / Sparkplug B broker-subscribe driver
 driver/opcuaclient
queue management
 driver/s7
queue management
 driver/twincat
queue management
 phase/1
queue management
 phase/2
queue management
 phase/3
queue management
 phase/4
queue management
 phase/5
queue management
 phase/6
queue management
 queue/blocked
queue management
 queue/done
queue management
 queue/failed
queue management
 queue/in-progress
queue management
 queue/queued
queue management
No Label driver/mqtt
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
dohertj2 (dohertj2)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dohertj2/lmxopcua#511
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?