1b2c36eb87
A SIGKILLed Primary stays an unreachable-but-not-downed member until auto-down, and the redundancy singleton usually dies with it, so the standby's cached snapshot kept naming it Primary and nobody captured for ~30 s. DriverHostActor now holds its own ReachabilityEvent (+MemberUp) subscription, caches the last snapshot, and excludes unreachable peers from the historization peer-is-Primary predicate only, republishing just the capture verdict on a reachability change. Alarm drain and the Primary data-plane verdict are unchanged. Gated by ContinuousHistorization:CaptureWhilePrimaryUnreachable (default true). Tests: message-driven cases with real ReachabilityEvents (incl. a falsification control on the drain/data-plane verdicts) plus a real two-node cluster crash test with downing disabled.
134 lines
8.7 KiB
C#
134 lines
8.7 KiB
C#
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Redundancy;
|
|
|
|
namespace ZB.MOM.WW.OtOpcUa.Runtime.Drivers;
|
|
|
|
/// <summary>
|
|
/// Single decision point for the Primary data-plane gates (inbound device writes, native alarm acks,
|
|
/// fleet-wide alerts publishes). A KNOWN role wins outright; an UNKNOWN role (no redundancy snapshot
|
|
/// yet, or the snapshot never mentioned this node — the 03/S5 identity-mismatch shape) is resolved by
|
|
/// cluster membership: a single-driver cluster stays default-ALLOW (no peer exists — the
|
|
/// boot-window / single-node posture), a multi-driver cluster is default-DENY (a real Primary peer
|
|
/// exists; do not touch the shared field device until the redundancy snapshot proves this node is it).
|
|
/// Fixes archreview 03/S4 (primary gate defaulted to allow while role unknown → dual-primary window).
|
|
/// </summary>
|
|
public static class PrimaryGatePolicy
|
|
{
|
|
/// <summary>Decide whether this node should service a Primary-only data-plane operation.</summary>
|
|
/// <param name="localRole">This node's last-known redundancy role, or <c>null</c> when unknown
|
|
/// (no snapshot received yet, or the snapshot omitted this node).</param>
|
|
/// <param name="driverMemberCount">Count of Up cluster members carrying the <c>driver</c> role.</param>
|
|
/// <returns><c>true</c> to service as Primary; <c>false</c> to deny.</returns>
|
|
public static bool ShouldServiceAsPrimary(RedundancyRole? localRole, int driverMemberCount) =>
|
|
localRole switch
|
|
{
|
|
RedundancyRole.Primary => true,
|
|
RedundancyRole.Secondary or RedundancyRole.Detached => false,
|
|
_ => driverMemberCount <= 1, // unknown: allow only when no driver peer exists
|
|
};
|
|
|
|
/// <summary>
|
|
/// Decide whether this node should drain the replicated alarm store-and-forward queue.
|
|
/// </summary>
|
|
/// <remarks>
|
|
/// <para>
|
|
/// <b>Deliberately not <see cref="ShouldServiceAsPrimary"/>.</b> That gate protects a shared
|
|
/// field device, where two nodes acting at once is dangerous and irreversible, so an unknown
|
|
/// role with a peer present must deny. This gate protects an append-only historian, and the
|
|
/// delivery contract is already <i>at-least-once across a failover, by design</i> — identical
|
|
/// events even collapse to a single row, because ids hash the payload.
|
|
/// </para>
|
|
/// <para>
|
|
/// The two error costs are therefore asymmetric: a false allow costs a duplicate history row;
|
|
/// a false deny silently stops the alarm audit trail and eventually evicts it at the capacity
|
|
/// wall. So this decision keys on the role <b>alone</b> and opens when the role is unknown —
|
|
/// no membership tie-break, because cluster-wide driver count says nothing about whether
|
|
/// <i>this</i> node has a redundant partner.
|
|
/// </para>
|
|
/// <para>
|
|
/// Found by the LocalDb Phase 2 live gate: on a rig of four driver nodes in one cluster with
|
|
/// no <c>Redundancy</c> section, borrowing the write gate's verdict suspended the drain on
|
|
/// every node — including unpaired ones — so nothing drained anywhere.
|
|
/// </para>
|
|
/// </remarks>
|
|
/// <param name="localRole">This node's last-known redundancy role, or <c>null</c> when unknown.</param>
|
|
/// <param name="queueSharingPeerIsPrimary">
|
|
/// Whether the node holding <see cref="RedundancyRole.Primary"/> in the same snapshot is a node
|
|
/// that <b>shares this node's queue</b> — i.e. its LocalDb replication partner.
|
|
/// <para>
|
|
/// Merely knowing that <i>a</i> Primary exists is not enough, because the redundancy role is
|
|
/// a CLUSTER-WIDE election while the alarm queue is PAIR-LOCAL. A fleet runs many pairs in
|
|
/// one cluster, so the elected driver Primary is usually in some other pair — and on the
|
|
/// docker-dev rig it is a central node, which carries the driver Akka role, replicates
|
|
/// nobody's LocalDb and does not even run the alarm historian. Deferring to it means this
|
|
/// node's events are delivered by no one, ever.
|
|
/// </para>
|
|
/// </param>
|
|
/// <returns><c>true</c> to drain; <c>false</c> only when a node holding these same rows is doing it.</returns>
|
|
public static bool ShouldDrainAlarmHistory(RedundancyRole? localRole, bool queueSharingPeerIsPrimary) =>
|
|
localRole switch
|
|
{
|
|
RedundancyRole.Primary => true,
|
|
RedundancyRole.Secondary or RedundancyRole.Detached => !queueSharingPeerIsPrimary,
|
|
_ => true, // unknown role ⇒ drain; a duplicate row beats a silent gap
|
|
};
|
|
|
|
/// <summary>
|
|
/// Decide whether this node should <b>capture</b> continuous-historization values into its
|
|
/// durable outbox (Gitea #537).
|
|
/// </summary>
|
|
/// <remarks>
|
|
/// <para>
|
|
/// <b>Same rule as <see cref="ShouldDrainAlarmHistory"/>, deliberately — and for the same
|
|
/// reason.</b> The historian is append-only and the error costs are asymmetric: a false
|
|
/// allow costs a duplicate sample, a false deny costs a permanent hole in the trend. So an
|
|
/// unknown role captures, and only an explicit Secondary/Detached stands down.
|
|
/// </para>
|
|
/// <para>
|
|
/// <b>But it gates the CAPTURE, not the drain — the opposite of the alarm sink.</b> That
|
|
/// choice follows from where the queue lives. The alarm queue is in LocalDb and
|
|
/// <i>replicates</i>, so the Secondary already holds the Primary's rows and gating the
|
|
/// drain loses nothing. The historization outbox is a node-local FasterLog that replicates
|
|
/// to nobody: gating only its drain would leave the standby appending forever to a queue no
|
|
/// one empties (unbounded disk, and a flood of stale duplicates the moment it is promoted),
|
|
/// while gating the capture leaves it holding nothing. Values already captured are still
|
|
/// drained by the node that captured them, so a demotion loses nothing that was recorded.
|
|
/// </para>
|
|
/// <para>
|
|
/// The residual cost is a capture gap across a failover, bounded by how long the newly
|
|
/// promoted node takes to see its first snapshot. That is inherent to any non-duplicating
|
|
/// design; the alternative — both nodes capturing — is what #537 was filed about.
|
|
/// </para>
|
|
/// <para>
|
|
/// Gitea #672: on a CRASH the newly promoted node may not see a snapshot until auto-down (the
|
|
/// redundancy singleton died with the old Primary), so the caller excludes a Primary the cluster
|
|
/// failure detector reports unreachable from <paramref name="addressSpaceSharingPeerIsPrimary"/> —
|
|
/// shrinking the gap to the failure-detector window. This method is unchanged by that; it only
|
|
/// sees the narrower input.
|
|
/// </para>
|
|
/// </remarks>
|
|
/// <param name="localRole">This node's last-known redundancy role, or <c>null</c> when unknown.</param>
|
|
/// <param name="addressSpaceSharingPeerIsPrimary">
|
|
/// Whether the node holding <see cref="RedundancyRole.Primary"/> in the same snapshot is one
|
|
/// that serves <b>this node's address space</b>, and therefore captures the very same historized
|
|
/// refs — its per-cluster mesh partner.
|
|
/// <para>
|
|
/// The scoping matters for the same reason it does on the alarm queue, and the mistake is
|
|
/// easier to make here because it looks safe: the redundancy election reaches only as far as
|
|
/// the caller can prove, whereas historization is fleet-wide by tag name. A node that stands
|
|
/// down in favour of a Primary running some <i>other</i> cluster's configuration hands its
|
|
/// tags to a node that never records them, and nothing says so — the drain simply reports
|
|
/// success on an empty outbox forever. Callers that cannot establish the sharing pass
|
|
/// <c>false</c>, which never stands down; duplicates are recoverable, a silent gap is not.
|
|
/// </para>
|
|
/// </param>
|
|
/// <returns><c>true</c> to capture; <c>false</c> only when a node recording these same tags is doing it.</returns>
|
|
public static bool ShouldRecordHistorizedValues(
|
|
RedundancyRole? localRole, bool addressSpaceSharingPeerIsPrimary) =>
|
|
localRole switch
|
|
{
|
|
RedundancyRole.Primary => true,
|
|
RedundancyRole.Secondary or RedundancyRole.Detached => !addressSpaceSharingPeerIsPrimary,
|
|
_ => true, // unknown role ⇒ capture; a duplicate sample beats a hole in the trend
|
|
};
|
|
}
|