Files
lmxopcua/src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/PrimaryGatePolicy.cs
T
Joseph Doherty 1b2c36eb87 fix(historian): capture while the recording Primary is unreachable (#672)
A SIGKILLed Primary stays an unreachable-but-not-downed member until
auto-down, and the redundancy singleton usually dies with it, so the
standby's cached snapshot kept naming it Primary and nobody captured for
~30 s. DriverHostActor now holds its own ReachabilityEvent (+MemberUp)
subscription, caches the last snapshot, and excludes unreachable peers
from the historization peer-is-Primary predicate only, republishing just
the capture verdict on a reachability change. Alarm drain and the Primary
data-plane verdict are unchanged. Gated by
ContinuousHistorization:CaptureWhilePrimaryUnreachable (default true).

Tests: message-driven cases with real ReachabilityEvents (incl. a
falsification control on the drain/data-plane verdicts) plus a real
two-node cluster crash test with downing disabled.
2026-10-11 08:26:39 -04:00

134 lines
8.7 KiB
C#

using ZB.MOM.WW.OtOpcUa.Commons.Messages.Redundancy;
namespace ZB.MOM.WW.OtOpcUa.Runtime.Drivers;
/// <summary>
/// Single decision point for the Primary data-plane gates (inbound device writes, native alarm acks,
/// fleet-wide alerts publishes). A KNOWN role wins outright; an UNKNOWN role (no redundancy snapshot
/// yet, or the snapshot never mentioned this node — the 03/S5 identity-mismatch shape) is resolved by
/// cluster membership: a single-driver cluster stays default-ALLOW (no peer exists — the
/// boot-window / single-node posture), a multi-driver cluster is default-DENY (a real Primary peer
/// exists; do not touch the shared field device until the redundancy snapshot proves this node is it).
/// Fixes archreview 03/S4 (primary gate defaulted to allow while role unknown → dual-primary window).
/// </summary>
public static class PrimaryGatePolicy
{
/// <summary>Decide whether this node should service a Primary-only data-plane operation.</summary>
/// <param name="localRole">This node's last-known redundancy role, or <c>null</c> when unknown
/// (no snapshot received yet, or the snapshot omitted this node).</param>
/// <param name="driverMemberCount">Count of Up cluster members carrying the <c>driver</c> role.</param>
/// <returns><c>true</c> to service as Primary; <c>false</c> to deny.</returns>
public static bool ShouldServiceAsPrimary(RedundancyRole? localRole, int driverMemberCount) =>
localRole switch
{
RedundancyRole.Primary => true,
RedundancyRole.Secondary or RedundancyRole.Detached => false,
_ => driverMemberCount <= 1, // unknown: allow only when no driver peer exists
};
/// <summary>
/// Decide whether this node should drain the replicated alarm store-and-forward queue.
/// </summary>
/// <remarks>
/// <para>
/// <b>Deliberately not <see cref="ShouldServiceAsPrimary"/>.</b> That gate protects a shared
/// field device, where two nodes acting at once is dangerous and irreversible, so an unknown
/// role with a peer present must deny. This gate protects an append-only historian, and the
/// delivery contract is already <i>at-least-once across a failover, by design</i> — identical
/// events even collapse to a single row, because ids hash the payload.
/// </para>
/// <para>
/// The two error costs are therefore asymmetric: a false allow costs a duplicate history row;
/// a false deny silently stops the alarm audit trail and eventually evicts it at the capacity
/// wall. So this decision keys on the role <b>alone</b> and opens when the role is unknown —
/// no membership tie-break, because cluster-wide driver count says nothing about whether
/// <i>this</i> node has a redundant partner.
/// </para>
/// <para>
/// Found by the LocalDb Phase 2 live gate: on a rig of four driver nodes in one cluster with
/// no <c>Redundancy</c> section, borrowing the write gate's verdict suspended the drain on
/// every node — including unpaired ones — so nothing drained anywhere.
/// </para>
/// </remarks>
/// <param name="localRole">This node's last-known redundancy role, or <c>null</c> when unknown.</param>
/// <param name="queueSharingPeerIsPrimary">
/// Whether the node holding <see cref="RedundancyRole.Primary"/> in the same snapshot is a node
/// that <b>shares this node's queue</b> — i.e. its LocalDb replication partner.
/// <para>
/// Merely knowing that <i>a</i> Primary exists is not enough, because the redundancy role is
/// a CLUSTER-WIDE election while the alarm queue is PAIR-LOCAL. A fleet runs many pairs in
/// one cluster, so the elected driver Primary is usually in some other pair — and on the
/// docker-dev rig it is a central node, which carries the driver Akka role, replicates
/// nobody's LocalDb and does not even run the alarm historian. Deferring to it means this
/// node's events are delivered by no one, ever.
/// </para>
/// </param>
/// <returns><c>true</c> to drain; <c>false</c> only when a node holding these same rows is doing it.</returns>
public static bool ShouldDrainAlarmHistory(RedundancyRole? localRole, bool queueSharingPeerIsPrimary) =>
localRole switch
{
RedundancyRole.Primary => true,
RedundancyRole.Secondary or RedundancyRole.Detached => !queueSharingPeerIsPrimary,
_ => true, // unknown role ⇒ drain; a duplicate row beats a silent gap
};
/// <summary>
/// Decide whether this node should <b>capture</b> continuous-historization values into its
/// durable outbox (Gitea #537).
/// </summary>
/// <remarks>
/// <para>
/// <b>Same rule as <see cref="ShouldDrainAlarmHistory"/>, deliberately — and for the same
/// reason.</b> The historian is append-only and the error costs are asymmetric: a false
/// allow costs a duplicate sample, a false deny costs a permanent hole in the trend. So an
/// unknown role captures, and only an explicit Secondary/Detached stands down.
/// </para>
/// <para>
/// <b>But it gates the CAPTURE, not the drain — the opposite of the alarm sink.</b> That
/// choice follows from where the queue lives. The alarm queue is in LocalDb and
/// <i>replicates</i>, so the Secondary already holds the Primary's rows and gating the
/// drain loses nothing. The historization outbox is a node-local FasterLog that replicates
/// to nobody: gating only its drain would leave the standby appending forever to a queue no
/// one empties (unbounded disk, and a flood of stale duplicates the moment it is promoted),
/// while gating the capture leaves it holding nothing. Values already captured are still
/// drained by the node that captured them, so a demotion loses nothing that was recorded.
/// </para>
/// <para>
/// The residual cost is a capture gap across a failover, bounded by how long the newly
/// promoted node takes to see its first snapshot. That is inherent to any non-duplicating
/// design; the alternative — both nodes capturing — is what #537 was filed about.
/// </para>
/// <para>
/// Gitea #672: on a CRASH the newly promoted node may not see a snapshot until auto-down (the
/// redundancy singleton died with the old Primary), so the caller excludes a Primary the cluster
/// failure detector reports unreachable from <paramref name="addressSpaceSharingPeerIsPrimary"/> —
/// shrinking the gap to the failure-detector window. This method is unchanged by that; it only
/// sees the narrower input.
/// </para>
/// </remarks>
/// <param name="localRole">This node's last-known redundancy role, or <c>null</c> when unknown.</param>
/// <param name="addressSpaceSharingPeerIsPrimary">
/// Whether the node holding <see cref="RedundancyRole.Primary"/> in the same snapshot is one
/// that serves <b>this node's address space</b>, and therefore captures the very same historized
/// refs — its per-cluster mesh partner.
/// <para>
/// The scoping matters for the same reason it does on the alarm queue, and the mistake is
/// easier to make here because it looks safe: the redundancy election reaches only as far as
/// the caller can prove, whereas historization is fleet-wide by tag name. A node that stands
/// down in favour of a Primary running some <i>other</i> cluster's configuration hands its
/// tags to a node that never records them, and nothing says so — the drain simply reports
/// success on an empty outbox forever. Callers that cannot establish the sharing pass
/// <c>false</c>, which never stands down; duplicates are recoverable, a silent gap is not.
/// </para>
/// </param>
/// <returns><c>true</c> to capture; <c>false</c> only when a node recording these same tags is doing it.</returns>
public static bool ShouldRecordHistorizedValues(
RedundancyRole? localRole, bool addressSpaceSharingPeerIsPrimary) =>
localRole switch
{
RedundancyRole.Primary => true,
RedundancyRole.Secondary or RedundancyRole.Detached => !addressSpaceSharingPeerIsPrimary,
_ => true, // unknown role ⇒ capture; a duplicate sample beats a hole in the trend
};
}