fix(historian): capture while the recording Primary is unreachable (#672)
A SIGKILLed Primary stays an unreachable-but-not-downed member until auto-down, and the redundancy singleton usually dies with it, so the standby's cached snapshot kept naming it Primary and nobody captured for ~30 s. DriverHostActor now holds its own ReachabilityEvent (+MemberUp) subscription, caches the last snapshot, and excludes unreachable peers from the historization peer-is-Primary predicate only, republishing just the capture verdict on a reachability change. Alarm drain and the Primary data-plane verdict are unchanged. Gated by ContinuousHistorization:CaptureWhilePrimaryUnreachable (default true). Tests: message-driven cases with real ReachabilityEvents (incl. a falsification control on the drain/data-plane verdicts) plus a real two-node cluster crash test with downing disabled.
This commit is contained in:
@@ -1408,6 +1408,7 @@ only the recorder + outbox knobs:
|
||||
| `MaxWriteAttempts` | `5` | #538: consecutive failed drain passes one tag may accumulate — counted **only across passes where other tags wrote successfully** — before its queued entries are abandoned so the FIFO head can advance |
|
||||
| `MinBackoffSeconds` | `1` | Initial retry backoff after a failed drain pass |
|
||||
| `MaxBackoffSeconds` | `30` | Cap on the exponential retry backoff |
|
||||
| `CaptureWhilePrimaryUnreachable` | `true` | #672: a standby stood down for its mesh Primary (#537) captures as soon as that Primary is reported UNREACHABLE, instead of after auto-down — crash-failover gap ~30 s ⇒ ~10–12 s, at the cost of duplicate windows (see #537/#672 below). `false` = pre-#672 |
|
||||
|
||||
### Tag auto-provisioning (`IHistorianProvisioning` EnsureTags hook)
|
||||
|
||||
@@ -1547,6 +1548,21 @@ tag); evidence in the same plan doc. Two things to know before touching either.*
|
||||
`RedundancyRoleView` behind **`IHistorizationRoleView`**, registered unconditionally on driver nodes
|
||||
(NOT alongside the alarm `IRedundancyRoleView`, which only exists when that section is enabled — the
|
||||
#511 registration accident).
|
||||
- **#672 — an UNREACHABLE recording Primary does not count (crash-failover gap).** A SIGKILLed Primary
|
||||
stays a member (unreachable, not downed) for 10 s failure detection + 15 s auto-down, and the
|
||||
redundancy singleton died with it, so the standby's cached snapshot kept naming it Primary and
|
||||
*nobody* captured for ~30 s. `DriverHostActor` now holds its **own** `ReachabilityEvent` (+`MemberUp`)
|
||||
subscription — not inside `StartPrimaryVerdictPublishing`, which early-returns with no primary-role
|
||||
source — handled in **all three** behaviours, caches the last snapshot, and on a reachability change
|
||||
re-derives and republishes **only** the historization verdict with unreachable peers excluded. Gap ⇒ the
|
||||
failure-detector window (~10–12 s). Gated by `ContinuousHistorization:CaptureWhilePrimaryUnreachable`
|
||||
(default `true`). ⚠️ **The alarm drain and `ShouldServiceAsPrimary`/`PublishPrimaryVerdict` must NOT
|
||||
adopt this** (a false allow there is a dual claimant, not a duplicate) — the tests assert both stay
|
||||
put. ⚠️ A **removed** member deliberately stays marked unreachable (only `MemberUp` at that address
|
||||
clears it): un-marking it would re-derive from a cached snapshot still naming it Primary and close the
|
||||
gate on a dead node. Cost: duplicates while a live Primary is merely unreachable and under a real
|
||||
partition — idempotent on SqlServer/QuestDB only for source-timestamped drivers (Modbus stamps each
|
||||
node's own poll time ⇒ duplicate rows on every backend); Gateway stores duplicates.
|
||||
|
||||
Re-confirmed live in 2026-06 and **since fixed (#600.1)**: `EnsureTags` provisioning used to fire only for
|
||||
newly-ADDED historized tags, so flipping an existing tag's `historianTagname` (or historization, or DataType)
|
||||
|
||||
@@ -447,6 +447,7 @@ Only the knobs; the backend connection comes from `ServerHistorian`. Outbox reco
|
||||
| `ContinuousHistorization:DrainConcurrency` | `8` | Historian tags one drain pass writes **concurrently** — a batch is grouped per tag and each group is its own gateway round-trip, so this turns a sequential `tags × latency` pass into `⌈tags / concurrency⌉ × latency` while bounding outstanding round-trips. `1` = fully sequential; `< 1` is clamped to 1 with a `Validate()` warning. The ack is unchanged (all-or-nothing after every group reports), as is the #538 quarantine rule. |
|
||||
| `ContinuousHistorization:Capacity` | `500000` | Max un-acked entries before drop-oldest; `0` = unbounded. The #538 disk backstop for a historian refusing **every** write — the one case the per-tag quarantine cannot resolve. |
|
||||
| `ContinuousHistorization:MaxWriteAttempts` | `5` | #538 poison-tag budget: consecutive failed passes one tag may accumulate — **counted only across passes where *other* tags wrote successfully** — before its entries are abandoned so the FIFO head can advance. |
|
||||
| `ContinuousHistorization:CaptureWhilePrimaryUnreachable` | `true` | #672: a standby stood down for its mesh-partner Primary (#537) resumes capture as soon as the failure detector reports that Primary **unreachable**, rather than after auto-down + a fresh snapshot — a crash-failover gap of ~25–30 s becomes the failure-detector window (~10–12 s). Cost: duplicate capture while a live Primary is merely unreachable and under a real partition (absorbed idempotently on SqlServer/QuestDB only for source-timestamped drivers; Modbus-style poll-time stamps and the Gateway backend store duplicates). Affects only the capture verdict. `false` = pre-#672 behaviour. |
|
||||
| `ContinuousHistorization:MinBackoffSeconds` / `MaxBackoffSeconds` | `1` / `30` | Initial and capped exponential retry backoff after a failed drain pass. |
|
||||
|
||||
### `LocalDb` (pair-local config cache — driver role only)
|
||||
|
||||
@@ -242,6 +242,7 @@ and is registered the same way for every backend.
|
||||
| `DrainConcurrency` | int | `8` | Historian tags a single drain pass writes **concurrently**. One peeked batch is grouped per historian tag and each group is an independent gateway round-trip, so a sequential pass cost `tags × latency`; this window makes it `⌈tags / concurrency⌉ × latency` while bounding how many round-trips can be outstanding against the gateway at once. `1` restores the fully sequential drain; a value `< 1` is clamped to 1 (with a `Validate()` warning). **The ack is unchanged** — still all-or-nothing after every group has reported, and the #538 per-tag quarantine still keys on "another group wrote this pass". |
|
||||
| `Capacity` | int | `500000` | Max un-acked outbox entries before the drop-oldest policy sheds the front of the queue; `0` = unbounded. Roughly 50 MB of FasterLog. The #538 **disk backstop** for the one case the per-tag quarantine deliberately cannot resolve — a historian unreachable or refusing *every* write, where nothing proves any particular tag is at fault so the recorder correctly retains. |
|
||||
| `MaxWriteAttempts` | int | `5` | #538 poison-tag budget: consecutive failed drain passes one historian tag may accumulate before its queued entries are abandoned and the FIFO head is allowed to advance. **Counted only across passes where *other* tags wrote successfully**, so an ordinary gateway outage never advances it. Clamped to at least 1; each abandonment logs a Warning naming the tag. |
|
||||
| `CaptureWhilePrimaryUnreachable` | bool | `true` | #672: a standby whose capture is stood down for its mesh-partner Primary (#537) resumes capturing as soon as the cluster failure detector reports that Primary **unreachable**, instead of waiting for auto-down + a fresh redundancy snapshot. Shrinks a crash-failover capture gap from ~25 s to the failure-detector window (~10–12 s), at the cost of short duplicate windows — see [the crash-failover note](#a-crashed-primary-capture-gap-672). `false` restores the pre-#672 behaviour. Never affects the alarm drain or the Primary data-plane gate. |
|
||||
| `MinBackoffSeconds` | double | `1` | Initial retry backoff after a failed drain pass. |
|
||||
| `MaxBackoffSeconds` | double | `30` | Cap on the exponential retry backoff. |
|
||||
|
||||
@@ -381,6 +382,37 @@ trend is not. The residual cost is a capture gap across a failover, bounded by h
|
||||
node takes to see its first redundancy snapshot — inherent to any non-duplicating design. Suppressed
|
||||
values are counted in `RecorderStatus.SuppressedNotPrimary`, and each gate transition logs once.
|
||||
|
||||
### A crashed Primary: capture gap (#672)
|
||||
|
||||
On a **crash** (SIGKILL / `docker kill`) that "first snapshot" can be a long way off. The dead Primary
|
||||
stays a cluster member — *unreachable, not downed* — for the failure-detector window plus
|
||||
`auto-down-unreachable-after` (`acceptable-heartbeat-pause` 10 s + 15 s in `akka.conf`), and the
|
||||
redundancy singleton almost always lived on it (it is the oldest member), so no snapshot is published
|
||||
until auto-down re-homes the singleton. The standby's cached snapshot keeps naming the dead node Primary
|
||||
and **nobody captures** for ~30 s (measured on the QuestDB live gate).
|
||||
|
||||
`DriverHostActor` therefore follows cluster reachability (its own `ClusterEvent.ReachabilityEvent`
|
||||
subscription) and, with `ContinuousHistorization:CaptureWhilePrimaryUnreachable` on (the default),
|
||||
**excludes an unreachable peer from the historization "a mesh-partner Primary is recording" question**,
|
||||
re-deriving and republishing only the capture verdict from the last snapshot. A node at the same address
|
||||
coming Up again clears the mark; a reachable-again Primary closes the gate again. It logs
|
||||
`capturing historized values while recording Primary <id> is unreachable` once on entry and once on
|
||||
revert. The alarm drain and the Primary data-plane verdict (writes, native acks, the Sparkplug host claim)
|
||||
are **not** touched — a false allow there is a dual claimant, not a duplicate sample.
|
||||
|
||||
Trade-offs, all in the duplicate direction #537 already accepts:
|
||||
|
||||
- **The gap shrinks to the failure-detector window** (~10–12 s), not to zero — detection is the floor.
|
||||
- **Duplicates during the overlap.** A Primary that is only *briefly* unreachable (GC pause, network blip)
|
||||
overlaps the standby until it is reachable again, and under a genuine **partition** both sides capture
|
||||
from detection onward (as they already did after auto-down, now ~15 s earlier). SqlServer and QuestDB
|
||||
absorb a duplicate of a **source-timestamped** sample idempotently (`(TagId, SourceTimeUtc)` key /
|
||||
dedup), but a driver with **no source timestamp** (e.g. Modbus: `SourceTimestampUtc` is null, so each
|
||||
node stamps its own poll time) writes a duplicate row on **every** backend, and the **Gateway** backend
|
||||
stores duplicates of everything.
|
||||
- Set `CaptureWhilePrimaryUnreachable=false` to keep the pre-#672 behaviour (no duplicates from this
|
||||
path, the full ~30 s hole on a crash).
|
||||
|
||||
---
|
||||
|
||||
## Tag auto-provisioning (EnsureTags)
|
||||
|
||||
+10
-1
@@ -546,7 +546,16 @@ Under warm/hot redundancy both cluster nodes run `ScriptedAlarmHostActor` and ev
|
||||
node. ⚠️ **The peer it stands down for must record the same refs** — a Primary in its own per-cluster
|
||||
mesh — not merely hold the same rows; a node with no cluster-scoped role never stands down, because a
|
||||
fleet-wide election's Primary usually runs a different cluster's configuration and deferring to it would
|
||||
end this cluster's historization silently. See [Historian.md](Historian.md).
|
||||
end this cluster's historization silently. **Crash failover (Gitea #672):** a SIGKILLed Primary stays an
|
||||
unreachable-but-not-downed member until auto-down (10 s failure detector + 15 s), and the redundancy
|
||||
singleton usually died with it, so the standby's cached snapshot kept naming it Primary and nobody
|
||||
captured for ~30 s. `DriverHostActor` now follows `ClusterEvent.ReachabilityEvent` and excludes an
|
||||
**unreachable** peer from the historization peer-is-Primary question only (re-deriving the capture verdict
|
||||
from the cached snapshot), shrinking the gap to the failure-detector window; gated by
|
||||
`ContinuousHistorization:CaptureWhilePrimaryUnreachable` (default `true`). The alarm drain and the Primary
|
||||
data-plane verdict are **unchanged** — they still wait for a snapshot. The cost is duplicate capture while a
|
||||
live Primary is merely unreachable, and from detection onward under a real partition. See
|
||||
[Historian.md](Historian.md) §"A crashed Primary: capture gap (#672)".
|
||||
|
||||
Net effect: each alarm transition appears **once** on `/alerts` and would historize once, not once per node.
|
||||
|
||||
|
||||
@@ -55,7 +55,9 @@
|
||||
"MaxWriteAttempts": 5,
|
||||
"_MaxWriteAttemptsComment": "Gitea #538: consecutive failed drain passes one tag may accumulate — counted only across passes where OTHER tags wrote successfully — before its queued entries are abandoned so the FIFO head can advance.",
|
||||
"MinBackoffSeconds": 1,
|
||||
"MaxBackoffSeconds": 30
|
||||
"MaxBackoffSeconds": 30,
|
||||
"CaptureWhilePrimaryUnreachable": true,
|
||||
"_CaptureWhilePrimaryUnreachableComment": "Gitea #672: a standby stood down for its mesh Primary resumes capture as soon as that Primary is reported UNREACHABLE (not after auto-down) — shrinks a crash-failover capture gap to the failure-detector window, at the cost of short duplicate windows. false = pre-#672."
|
||||
},
|
||||
"AlarmHistorian": {
|
||||
"_comment": "Durable store-and-forward alarm sink. Buffers into the node's consolidated LocalDb (see the LocalDb section) so the queue replicates to the redundant pair peer, and drains alarm events to the ServerHistorian gateway's SendEvent path from whichever node holds the Primary role. The downstream connection (endpoint/key/TLS) is sourced from the ServerHistorian section.",
|
||||
|
||||
@@ -461,6 +461,30 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
/// </summary>
|
||||
private readonly Func<CommonsNodeId, bool>? _sharesHistorizationScopeWithLocal;
|
||||
|
||||
/// <summary>Gitea #672: whether an UNREACHABLE recording Primary still stands this node's historization
|
||||
/// capture down (<c>false</c>) or not (<c>true</c>, the default). See <see cref="ComputeHistorizationVerdict"/>.</summary>
|
||||
private readonly bool _captureWhilePrimaryUnreachable;
|
||||
|
||||
/// <summary>Gitea #672: the last redundancy snapshot's nodes, so a reachability change can re-derive the
|
||||
/// historization capture verdict without waiting for the next snapshot — which, when the recording Primary
|
||||
/// crashed, may not come until auto-down (the redundancy singleton usually lived on that Primary).</summary>
|
||||
private IReadOnlyList<NodeRedundancyState>? _lastRedundancySnapshotNodes;
|
||||
|
||||
/// <summary>Gitea #672: peers the cluster failure detector currently reports unreachable (never this node).
|
||||
/// Fed by <see cref="Akka.Cluster.ClusterEvent.ReachabilityEvent"/>; cleared for an address that comes Up again.</summary>
|
||||
private readonly HashSet<CommonsNodeId> _unreachableNodes = new();
|
||||
|
||||
/// <summary>Gitea #672: the unreachable recording Primary this node is currently capturing in place of, or
|
||||
/// null — drives the once-per-transition Information log.</summary>
|
||||
private CommonsNodeId? _capturingForUnreachablePrimary;
|
||||
|
||||
/// <summary>The last historization capture verdict published (null before the first), so the reachability
|
||||
/// path republishes only a CHANGED verdict.</summary>
|
||||
private bool? _lastPublishedHistorizationVerdict;
|
||||
|
||||
/// <summary>Whether this actor holds its own (#672) cluster reachability subscription.</summary>
|
||||
private bool _subscribedToReachabilityEvents;
|
||||
|
||||
/// <summary>Gitea #601 — how long an Applied ack waits for the address-space outcome (see
|
||||
/// <see cref="DefaultApplyOutcomeTimeout"/>).</summary>
|
||||
private readonly TimeSpan _applyOutcomeTimeout;
|
||||
@@ -664,7 +688,8 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
TimeSpan? reprovisionBackoff = null,
|
||||
TimeSpan? commsRecoveryAuditDelay = null,
|
||||
Func<CommonsNodeId, bool>? sharesAlarmQueueWithLocal = null,
|
||||
TimeSpan? applyOutcomeTimeout = null) =>
|
||||
TimeSpan? applyOutcomeTimeout = null,
|
||||
bool captureWhilePrimaryUnreachable = true) =>
|
||||
// WARNING: this forwarding list is POSITIONAL, and Props.Create compiles it into an
|
||||
// expression tree. Six IActorRef? parameters and several interface-typed ones mean a
|
||||
// mis-ordered argument is usually type-compatible and therefore compiles clean, then binds
|
||||
@@ -677,7 +702,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
deploymentArtifactCache, redundancyRoleView, replicationPeerHost, fetchAndCacheMode, artifactFetcher,
|
||||
alarmStateStore, telemetryHub, primaryRoleSource, historizationRoleView,
|
||||
sharesHistorizationScopeWithLocal, applyWedgeTimeout, reprovisionBackoff, commsRecoveryAuditDelay,
|
||||
sharesAlarmQueueWithLocal, applyOutcomeTimeout));
|
||||
sharesAlarmQueueWithLocal, applyOutcomeTimeout, captureWhilePrimaryUnreachable));
|
||||
|
||||
/// <summary>Initializes a new DriverHostActor with the specified dependencies.</summary>
|
||||
/// <param name="dbFactory">Database context factory for configuration database access.</param>
|
||||
@@ -745,6 +770,11 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
/// PeerAddress), whose partner is its per-cluster mesh partner. Ignored when
|
||||
/// <paramref name="replicationPeerHost"/> is set. Null ⇒ no such peer is known and this node never stands its
|
||||
/// drain down.</param>
|
||||
/// <param name="captureWhilePrimaryUnreachable">Gitea #672 (<c>ContinuousHistorization:CaptureWhilePrimaryUnreachable</c>,
|
||||
/// default <c>true</c>): when a mesh-partner Primary this node would stand its historization capture down for is
|
||||
/// reported UNREACHABLE by the cluster failure detector, stop deferring to it and capture — rather than waiting
|
||||
/// for auto-down + a fresh redundancy snapshot. Affects ONLY the historization capture verdict; the alarm drain
|
||||
/// and the Primary data-plane verdict are unchanged. <c>false</c> restores the pre-#672 behaviour.</param>
|
||||
public DriverHostActor(
|
||||
IDbContextFactory<OtOpcUaConfigDbContext>? dbFactory,
|
||||
CommonsNodeId localNode,
|
||||
@@ -776,8 +806,10 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
TimeSpan? reprovisionBackoff = null,
|
||||
TimeSpan? commsRecoveryAuditDelay = null,
|
||||
Func<CommonsNodeId, bool>? sharesAlarmQueueWithLocal = null,
|
||||
TimeSpan? applyOutcomeTimeout = null)
|
||||
TimeSpan? applyOutcomeTimeout = null,
|
||||
bool captureWhilePrimaryUnreachable = true)
|
||||
{
|
||||
_captureWhilePrimaryUnreachable = captureWhilePrimaryUnreachable;
|
||||
_applyOutcomeTimeout = applyOutcomeTimeout ?? DefaultApplyOutcomeTimeout;
|
||||
_sharesAlarmQueueWithLocal = sharesAlarmQueueWithLocal;
|
||||
_applyWedgeTimeout = applyWedgeTimeout ?? DefaultApplyWedgeTimeout;
|
||||
@@ -847,6 +879,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
// ApplyScriptedAlarms through the same PushDesiredSubscriptions pass.
|
||||
SpawnScriptedAlarmHost();
|
||||
StartPrimaryVerdictPublishing();
|
||||
StartReachabilityTracking();
|
||||
Bootstrap();
|
||||
}
|
||||
|
||||
@@ -894,8 +927,18 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
/// <summary>A cluster member changed status: re-publish the Primary verdict now and once more shortly after.
|
||||
/// The deferred pass covers the cluster read-view (what <see cref="DriverMemberCount"/> reads) updating on its
|
||||
/// own subscriber AFTER this event was delivered here — the two are not ordered.</summary>
|
||||
private void OnDriverMembershipChanged(Akka.Cluster.ClusterEvent.IMemberEvent _)
|
||||
private void OnDriverMembershipChanged(Akka.Cluster.ClusterEvent.IMemberEvent evt)
|
||||
{
|
||||
// Gitea #672: a member that comes Up at an address is reachable — the usual case being a restarted
|
||||
// node's new incarnation. A REMOVED member deliberately stays in the set: it never answers again, and
|
||||
// dropping it would re-derive the verdict from a cached snapshot that may still name it Primary,
|
||||
// closing the gate on a dead node until the next snapshot (the very hole #672 closes).
|
||||
if (evt is Akka.Cluster.ClusterEvent.MemberUp up
|
||||
&& _unreachableNodes.Remove(ToNodeId(up.Member.Address)))
|
||||
{
|
||||
RepublishHistorizationVerdictIfChanged();
|
||||
}
|
||||
|
||||
PublishPrimaryVerdictIfChanged();
|
||||
// One keyed timer: a burst of member events collapses into a single deferred pass.
|
||||
Timers.StartSingleTimer("republish-primary-verdict", RepublishPrimaryVerdict.Instance, MembershipRepublishDelay);
|
||||
@@ -919,10 +962,62 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
PublishPrimaryVerdict();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Gitea #672: follow cluster reachability so the historization capture gate stops deferring to a recording
|
||||
/// Primary the failure detector already considers dead. Its OWN subscription, deliberately not folded into
|
||||
/// <see cref="StartPrimaryVerdictPublishing"/>, which early-returns on nodes with no primary-role source.
|
||||
/// <see cref="Akka.Cluster.ClusterEvent.MemberUp"/> rides along so a re-joined incarnation leaves the set.
|
||||
/// </summary>
|
||||
private void StartReachabilityTracking()
|
||||
{
|
||||
if (_historizationRoleView is null || !_captureWhilePrimaryUnreachable) return;
|
||||
try
|
||||
{
|
||||
Akka.Cluster.Cluster.Get(Context.System).Subscribe(
|
||||
Self, Akka.Cluster.ClusterEvent.SubscriptionInitialStateMode.InitialStateAsEvents,
|
||||
typeof(Akka.Cluster.ClusterEvent.ReachabilityEvent),
|
||||
typeof(Akka.Cluster.ClusterEvent.MemberUp));
|
||||
_subscribedToReachabilityEvents = true;
|
||||
}
|
||||
catch (Exception)
|
||||
{
|
||||
// Non-cluster ActorRefProvider — no failure detector to follow; the snapshot path alone decides.
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Gitea #672: a peer became unreachable or reachable again. Records it and re-derives ONLY the
|
||||
/// historization capture verdict from the cached snapshot — the alarm drain and the Primary data-plane verdict
|
||||
/// are deliberately untouched (a false allow there is a dual claimant, not a duplicate sample).</summary>
|
||||
private void OnReachabilityChanged(Akka.Cluster.ClusterEvent.ReachabilityEvent evt)
|
||||
{
|
||||
var node = ToNodeId(evt.Member.Address);
|
||||
if (node == _localNode) return; // this node's own reachability as seen by others is not a peer question
|
||||
var changed = evt is Akka.Cluster.ClusterEvent.UnreachableMember
|
||||
? _unreachableNodes.Add(node)
|
||||
: _unreachableNodes.Remove(node);
|
||||
if (changed) RepublishHistorizationVerdictIfChanged();
|
||||
}
|
||||
|
||||
/// <summary>Re-derives the historization capture verdict from the cached snapshot and publishes it only when it
|
||||
/// differs from the last one published. No-op before the first snapshot.</summary>
|
||||
private void RepublishHistorizationVerdictIfChanged()
|
||||
{
|
||||
if (_historizationRoleView is null || _lastRedundancySnapshotNodes is null) return;
|
||||
var verdict = ComputeHistorizationVerdict(_lastRedundancySnapshotNodes);
|
||||
if (verdict == _lastPublishedHistorizationVerdict) return;
|
||||
_lastPublishedHistorizationVerdict = verdict;
|
||||
_historizationRoleView.PublishHistorizationRole(verdict);
|
||||
}
|
||||
|
||||
/// <summary>Same host:port projection as <c>ClusterRoleInfo</c> / <c>RedundancyStateActor.ToNodeId</c>, so a
|
||||
/// reachability event's member compares equal to the NodeId a redundancy snapshot names.</summary>
|
||||
private static CommonsNodeId ToNodeId(Akka.Actor.Address address) =>
|
||||
CommonsNodeId.Parse($"{address.Host ?? string.Empty}:{address.Port ?? 0}");
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override void PostStop()
|
||||
{
|
||||
if (_subscribedToMemberEvents)
|
||||
if (_subscribedToMemberEvents || _subscribedToReachabilityEvents)
|
||||
{
|
||||
try { Akka.Cluster.Cluster.Get(Context.System).Unsubscribe(Self); }
|
||||
catch (Exception) { /* system shutting down */ }
|
||||
@@ -1349,6 +1444,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
Receive<RouteNativeAlarmAck>(HandleRouteNativeAlarmAck);
|
||||
Receive<RedundancyStateChanged>(OnRedundancyStateChanged);
|
||||
Receive<Akka.Cluster.ClusterEvent.IMemberEvent>(OnDriverMembershipChanged);
|
||||
Receive<Akka.Cluster.ClusterEvent.ReachabilityEvent>(OnReachabilityChanged);
|
||||
Receive<RepublishPrimaryVerdict>(_ => PublishPrimaryVerdictIfChanged());
|
||||
Receive<Akka.Cluster.ClusterEvent.CurrentClusterState>(_ => PublishPrimaryVerdictIfChanged());
|
||||
// The redundancy-state topic also carries OpcUaProbeResult (OpcUaPublishActor peer-probes).
|
||||
@@ -1410,6 +1506,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
Receive<RouteNativeAlarmAck>(HandleRouteNativeAlarmAck);
|
||||
Receive<RedundancyStateChanged>(OnRedundancyStateChanged);
|
||||
Receive<Akka.Cluster.ClusterEvent.IMemberEvent>(OnDriverMembershipChanged);
|
||||
Receive<Akka.Cluster.ClusterEvent.ReachabilityEvent>(OnReachabilityChanged);
|
||||
Receive<RepublishPrimaryVerdict>(_ => PublishPrimaryVerdictIfChanged());
|
||||
Receive<Akka.Cluster.ClusterEvent.CurrentClusterState>(_ => PublishPrimaryVerdictIfChanged());
|
||||
// The redundancy-state topic also carries OpcUaProbeResult (OpcUaPublishActor peer-probes).
|
||||
@@ -2208,20 +2305,15 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
PrimaryGatePolicy.ShouldDrainAlarmHistory(_localRole, queueSharingPeerIsPrimary: peerIsPrimary));
|
||||
|
||||
// Gitea #537: the continuous-historization CAPTURE verdict. Same role rule as the drain gate
|
||||
// above, but a different peer question — historization is not pair-local, it is fleet-wide by
|
||||
// historian tag name, so the only node worth standing down for is one recording the SAME refs:
|
||||
// a mesh partner serving this node's address space. With no way to establish that (a legacy
|
||||
// node carrying no cluster-scoped role, or a test harness), the predicate is null and this
|
||||
// node keeps capturing. Duplicate samples are recoverable; a hole in the trend is not.
|
||||
var historizationPeerIsPrimary =
|
||||
_sharesHistorizationScopeWithLocal is { } sharesScope
|
||||
&& msg.Nodes.Any(n => n.Role == RedundancyRole.Primary
|
||||
&& n.NodeId != _localNode
|
||||
&& sharesScope(n.NodeId));
|
||||
|
||||
_historizationRoleView?.PublishHistorizationRole(
|
||||
PrimaryGatePolicy.ShouldRecordHistorizedValues(
|
||||
_localRole, addressSpaceSharingPeerIsPrimary: historizationPeerIsPrimary));
|
||||
// above, but a different peer question — see ComputeHistorizationVerdict. Cached (#672) so a
|
||||
// reachability change can re-derive it before the next snapshot arrives.
|
||||
_lastRedundancySnapshotNodes = msg.Nodes;
|
||||
if (_historizationRoleView is not null)
|
||||
{
|
||||
var historizationVerdict = ComputeHistorizationVerdict(msg.Nodes);
|
||||
_lastPublishedHistorizationVerdict = historizationVerdict;
|
||||
_historizationRoleView.PublishHistorizationRole(historizationVerdict);
|
||||
}
|
||||
|
||||
// The DATA-PLANE verdict, deliberately — the same one HandleRouteNodeWrite gates on, not the
|
||||
// drain verdict published immediately above. A driver reading this claims an exclusive identity
|
||||
@@ -2230,6 +2322,66 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
PublishPrimaryVerdict();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The continuous-historization capture verdict for a snapshot (Gitea #537, #672).
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>Historization is not pair-local, it is fleet-wide by historian tag name, so the only node worth
|
||||
/// standing down for is one recording the SAME refs: a mesh partner serving this node's address space. With
|
||||
/// no way to establish that (a legacy node carrying no cluster-scoped role, or a test harness), the
|
||||
/// predicate is null and this node keeps capturing. Duplicate samples are recoverable; a hole in the trend
|
||||
/// is not.</para>
|
||||
/// <para><b>#672 — an UNREACHABLE recording Primary does not count.</b> A crashed (SIGKILLed) Primary stays a
|
||||
/// cluster member until auto-down, and the redundancy singleton usually died with it, so the cached snapshot
|
||||
/// keeps naming it Primary for the whole failure-detector + auto-down window (~25 s) while nobody captures.
|
||||
/// Excluding unreachable peers shrinks that hole to the failure-detector window alone. The cost — a short
|
||||
/// duplicate window, and both nodes capturing for the duration of a genuine partition — is the same
|
||||
/// asymmetric trade #537 already makes. Gated by <c>ContinuousHistorization:CaptureWhilePrimaryUnreachable</c>.</para>
|
||||
/// </remarks>
|
||||
private bool ComputeHistorizationVerdict(IReadOnlyList<NodeRedundancyState> nodes)
|
||||
{
|
||||
var peerIsPrimary = false;
|
||||
CommonsNodeId? unreachablePrimary = null;
|
||||
if (_sharesHistorizationScopeWithLocal is { } sharesScope)
|
||||
{
|
||||
foreach (var n in nodes)
|
||||
{
|
||||
if (n.Role != RedundancyRole.Primary || n.NodeId == _localNode || !sharesScope(n.NodeId)) continue;
|
||||
if (_captureWhilePrimaryUnreachable && _unreachableNodes.Contains(n.NodeId))
|
||||
{
|
||||
unreachablePrimary ??= n.NodeId;
|
||||
continue;
|
||||
}
|
||||
|
||||
peerIsPrimary = true;
|
||||
}
|
||||
}
|
||||
|
||||
var verdict = PrimaryGatePolicy.ShouldRecordHistorizedValues(
|
||||
_localRole, addressSpaceSharingPeerIsPrimary: peerIsPrimary);
|
||||
|
||||
// Log only when the unreachable exclusion is what OPENED the gate — i.e. the verdict would have been
|
||||
// "stand down" had that Primary been reachable.
|
||||
var overriding = unreachablePrimary is not null && verdict
|
||||
&& !PrimaryGatePolicy.ShouldRecordHistorizedValues(_localRole, addressSpaceSharingPeerIsPrimary: true);
|
||||
if (overriding && _capturingForUnreachablePrimary is null)
|
||||
{
|
||||
_capturingForUnreachablePrimary = unreachablePrimary;
|
||||
_log.Info(
|
||||
"DriverHost {Node}: capturing historized values while recording Primary {Primary} is unreachable (Gitea #672)",
|
||||
_localNode, unreachablePrimary);
|
||||
}
|
||||
else if (!overriding && _capturingForUnreachablePrimary is { } previous)
|
||||
{
|
||||
_capturingForUnreachablePrimary = null;
|
||||
_log.Info(
|
||||
"DriverHost {Node}: no longer capturing in place of unreachable recording Primary {Primary}; capture verdict now {Verdict}",
|
||||
_localNode, previous, verdict ? "capture" : "stand down");
|
||||
}
|
||||
|
||||
return verdict;
|
||||
}
|
||||
|
||||
private void Stale()
|
||||
{
|
||||
Receive<DispatchDeployment>(_ =>
|
||||
@@ -2251,6 +2403,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
Receive<ReconnectDriver>(HandleReconnectDriver);
|
||||
Receive<RedundancyStateChanged>(OnRedundancyStateChanged);
|
||||
Receive<Akka.Cluster.ClusterEvent.IMemberEvent>(OnDriverMembershipChanged);
|
||||
Receive<Akka.Cluster.ClusterEvent.ReachabilityEvent>(OnReachabilityChanged);
|
||||
Receive<RepublishPrimaryVerdict>(_ => PublishPrimaryVerdictIfChanged());
|
||||
Receive<Akka.Cluster.ClusterEvent.CurrentClusterState>(_ => PublishPrimaryVerdictIfChanged());
|
||||
// The redundancy-state topic also carries OpcUaProbeResult (OpcUaPublishActor peer-probes).
|
||||
|
||||
@@ -98,6 +98,13 @@ public static class PrimaryGatePolicy
|
||||
/// promoted node takes to see its first snapshot. That is inherent to any non-duplicating
|
||||
/// design; the alternative — both nodes capturing — is what #537 was filed about.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Gitea #672: on a CRASH the newly promoted node may not see a snapshot until auto-down (the
|
||||
/// redundancy singleton died with the old Primary), so the caller excludes a Primary the cluster
|
||||
/// failure detector reports unreachable from <paramref name="addressSpaceSharingPeerIsPrimary"/> —
|
||||
/// shrinking the gap to the failure-detector window. This method is unchanged by that; it only
|
||||
/// sees the narrower input.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
/// <param name="localRole">This node's last-known redundancy role, or <c>null</c> when unknown.</param>
|
||||
/// <param name="addressSpaceSharingPeerIsPrimary">
|
||||
|
||||
@@ -129,6 +129,25 @@ public sealed class ContinuousHistorizationOptions
|
||||
/// <summary>Cap (seconds) on the exponential retry backoff after repeated drain failures. Defaults to 30.</summary>
|
||||
public double MaxBackoffSeconds { get; init; } = 30;
|
||||
|
||||
/// <summary>
|
||||
/// Gitea #672: when <c>true</c> (the default), a redundant node whose capture is stood down for its
|
||||
/// mesh-partner Primary (Gitea #537) resumes capturing as soon as the cluster failure detector reports
|
||||
/// that Primary <b>unreachable</b>, instead of waiting for auto-down and a fresh redundancy snapshot.
|
||||
/// <para>
|
||||
/// A crashed Primary otherwise leaves a capture hole of failure-detection + auto-down (~25 s on the
|
||||
/// shipped <c>akka.conf</c>): the redundancy singleton usually died with it, so the standby's last
|
||||
/// snapshot keeps naming the dead node Primary. With this on, the hole shrinks to the failure-detector
|
||||
/// window (~10–12 s). The cost is duplicates: during a genuine partition both nodes capture from
|
||||
/// detection onward, and a briefly-unreachable-but-alive Primary overlaps the standby. SqlServer and
|
||||
/// QuestDB absorb a duplicate of a source-timestamped sample idempotently; a driver with no source
|
||||
/// timestamp (e.g. Modbus, stamped with each node's own poll time) writes a duplicate row on every
|
||||
/// backend, and the Gateway backend stores duplicates. <c>false</c> restores the pre-#672 behaviour.
|
||||
/// Only the historization capture verdict is affected — never the alarm drain or the Primary data-plane
|
||||
/// gate.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public bool CaptureWhilePrimaryUnreachable { get; init; } = true;
|
||||
|
||||
/// <summary>Returns operator-facing misconfiguration warnings for an <c>Enabled</c> recorder
|
||||
/// (empty when disabled or correctly configured). Pure — the registration logs each entry.</summary>
|
||||
/// <returns>Zero or more human-readable warning messages (never carrying secret values).</returns>
|
||||
|
||||
@@ -625,6 +625,9 @@ public static class ServiceCollectionExtensions
|
||||
// verdict, and how it decides which peer is worth standing down for.
|
||||
historizationRoleView: resolver.GetService<IHistorizationRoleView>(),
|
||||
sharesHistorizationScopeWithLocal: historizationScopePeer,
|
||||
// Gitea #672: stop deferring the capture to a recording Primary the failure detector
|
||||
// already reports unreachable. Unbound section ⇒ the option's default (true).
|
||||
captureWhilePrimaryUnreachable: continuousOptions?.CaptureWhilePrimaryUnreachable ?? true,
|
||||
// Review R6 MINOR-4: the passive half of a LocalDb pair has no PeerAddress, so it is told
|
||||
// its queue-sharing peer as its mesh partner instead — otherwise it never stands down.
|
||||
sharesAlarmQueueWithLocal: BuildAlarmQueuePeerPredicate(
|
||||
|
||||
+309
@@ -1,8 +1,13 @@
|
||||
using System.Collections.Immutable;
|
||||
using System.Reflection;
|
||||
using Akka.Actor;
|
||||
using Akka.Cluster;
|
||||
using Akka.Util;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Redundancy;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Types;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Drivers;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Redundancy;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Tests.Harness;
|
||||
@@ -134,8 +139,312 @@ public sealed class DriverHostActorHistorizationRoleViewTests : RuntimeActorTest
|
||||
AwaitAssert(() => view.Published.ShouldBe([false, true]), duration: Timeout);
|
||||
}
|
||||
|
||||
// ---------------- Gitea #672: an unreachable recording Primary ----------------
|
||||
//
|
||||
// A SIGKILLed Primary stays a member (unreachable, not downed) for the failure-detector + auto-down
|
||||
// window, and the redundancy singleton usually died with it — so the survivor's cached snapshot keeps
|
||||
// naming the dead node Primary and nobody captures. These drive REAL Akka ReachabilityEvents (built by
|
||||
// reflection, as the ControlPlane tests do) into the host, so the host:port projection is exercised too.
|
||||
|
||||
private static readonly NodeId Self672 = NodeId.Parse("self-node:4053");
|
||||
private static readonly NodeId Peer672 = NodeId.Parse("mesh-peer:4053");
|
||||
private static readonly NodeId Foreign672 = NodeId.Parse("other-cluster:4053");
|
||||
|
||||
/// <summary>
|
||||
/// An unreachable mesh-partner Primary opens the capture gate and its return closes it again — and
|
||||
/// neither moves the alarm-drain verdict nor the Primary data-plane verdict.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The drain/data-plane half is the falsification control: a lazier fix that excluded unreachable
|
||||
/// nodes from every peer question (or published the whole verdict set on a reachability change)
|
||||
/// would let a Secondary claim the Sparkplug host id / drain the replicated alarm queue during a
|
||||
/// mere failure-detector blip.
|
||||
/// </remarks>
|
||||
[Fact]
|
||||
public void An_unreachable_mesh_primary_opens_capture_and_its_return_closes_it_without_touching_other_verdicts()
|
||||
{
|
||||
var hist = new RecordingHistorizationView();
|
||||
var others = new RecordingOtherViews();
|
||||
IActorRef host = SpawnHost672(hist, others, captureWhilePrimaryUnreachable: true);
|
||||
|
||||
host.Tell(Snapshot((Self672, RedundancyRole.Secondary), (Peer672, RedundancyRole.Primary)));
|
||||
AwaitAssert(() => hist.Published.ShouldBe([false]), duration: Timeout);
|
||||
AwaitAssert(() => others.Drained.ShouldNotBeEmpty(), duration: Timeout);
|
||||
var drainedBefore = others.Drained;
|
||||
var primaryBefore = others.Primary;
|
||||
drainedBefore[^1].ShouldBeFalse(); // queue-sharing mesh peer is Primary ⇒ drain stands down
|
||||
primaryBefore[^1].ShouldBeFalse(); // Secondary ⇒ not the data-plane Primary
|
||||
|
||||
host.Tell(new ClusterEvent.UnreachableMember(MemberAt(Peer672)));
|
||||
AwaitAssert(() => hist.Published.ShouldBe([false, true]), duration: Timeout);
|
||||
|
||||
// Same handler run that published `true` above — anything it published elsewhere is already recorded.
|
||||
others.Drained.ShouldBe(drainedBefore);
|
||||
others.Primary.ShouldBe(primaryBefore);
|
||||
others.Current.ShouldDrainAlarmHistory.ShouldBeFalse();
|
||||
others.Current.ShouldActAsPrimary.ShouldBeFalse();
|
||||
|
||||
host.Tell(new ClusterEvent.ReachableMember(MemberAt(Peer672)));
|
||||
AwaitAssert(() => hist.Published.ShouldBe([false, true, false]), duration: Timeout);
|
||||
others.Drained.ShouldBe(drainedBefore);
|
||||
others.Primary.ShouldBe(primaryBefore);
|
||||
}
|
||||
|
||||
/// <summary>A peer outside this node's mesh going unreachable changes nothing.</summary>
|
||||
[Fact]
|
||||
public void An_unreachable_out_of_scope_primary_changes_nothing()
|
||||
{
|
||||
var hist = new RecordingHistorizationView();
|
||||
IActorRef host = SpawnHost672(hist, others: null, captureWhilePrimaryUnreachable: true);
|
||||
|
||||
// Both a foreign-cluster Primary (never stood down for) and the mesh Primary (stood down for).
|
||||
host.Tell(Snapshot(
|
||||
(Self672, RedundancyRole.Secondary), (Peer672, RedundancyRole.Primary), (Foreign672, RedundancyRole.Primary)));
|
||||
AwaitAssert(() => hist.Published.ShouldBe([false]), duration: Timeout);
|
||||
|
||||
host.Tell(new ClusterEvent.UnreachableMember(MemberAt(Foreign672)));
|
||||
// Positive control: the mesh peer's loss, sent AFTER, must be the only change — had the foreign
|
||||
// one published anything it would sit between the two.
|
||||
host.Tell(new ClusterEvent.UnreachableMember(MemberAt(Peer672)));
|
||||
AwaitAssert(() => hist.Published.ShouldBe([false, true]), duration: Timeout);
|
||||
}
|
||||
|
||||
/// <summary>This node's own reachability (as gossiped by others) is not a peer question.</summary>
|
||||
[Fact]
|
||||
public void Self_unreachable_is_ignored()
|
||||
{
|
||||
var hist = new RecordingHistorizationView();
|
||||
IActorRef host = SpawnHost672(hist, others: null, captureWhilePrimaryUnreachable: true);
|
||||
|
||||
host.Tell(Snapshot((Self672, RedundancyRole.Secondary), (Peer672, RedundancyRole.Primary)));
|
||||
AwaitAssert(() => hist.Published.ShouldBe([false]), duration: Timeout);
|
||||
|
||||
host.Tell(new ClusterEvent.UnreachableMember(MemberAt(Self672)));
|
||||
host.Tell(new ClusterEvent.UnreachableMember(MemberAt(Peer672)));
|
||||
AwaitAssert(() => hist.Published.ShouldBe([false, true]), duration: Timeout);
|
||||
|
||||
// And self "returning" does not close what the peer's loss opened.
|
||||
host.Tell(new ClusterEvent.ReachableMember(MemberAt(Self672)));
|
||||
host.Tell(Snapshot((Self672, RedundancyRole.Secondary), (Peer672, RedundancyRole.Primary)));
|
||||
AwaitAssert(() => hist.Published.ShouldBe([false, true, true]), duration: Timeout);
|
||||
}
|
||||
|
||||
/// <summary><c>CaptureWhilePrimaryUnreachable = false</c> restores the pre-#672 behaviour.</summary>
|
||||
[Fact]
|
||||
public void With_the_option_off_an_unreachable_primary_keeps_the_gate_closed()
|
||||
{
|
||||
var hist = new RecordingHistorizationView();
|
||||
IActorRef host = SpawnHost672(hist, others: null, captureWhilePrimaryUnreachable: false);
|
||||
|
||||
host.Tell(Snapshot((Self672, RedundancyRole.Secondary), (Peer672, RedundancyRole.Primary)));
|
||||
AwaitAssert(() => hist.Published.ShouldBe([false]), duration: Timeout);
|
||||
|
||||
host.Tell(new ClusterEvent.UnreachableMember(MemberAt(Peer672)));
|
||||
// Positive control: a re-sent snapshot publishes on every arrival, so a `true` from the
|
||||
// reachability event would show up between the two `false`s.
|
||||
host.Tell(Snapshot((Self672, RedundancyRole.Secondary), (Peer672, RedundancyRole.Primary)));
|
||||
AwaitAssert(() => hist.Published.ShouldBe([false, false]), duration: Timeout);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A fresh snapshot naming this node Primary — the singleton re-homed after auto-down removed the
|
||||
/// dead peer — keeps the gate open; the removal itself must not flap it closed in between.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void A_snapshot_naming_me_primary_after_the_peer_is_removed_keeps_capturing()
|
||||
{
|
||||
var hist = new RecordingHistorizationView();
|
||||
IActorRef host = SpawnHost672(hist, others: null, captureWhilePrimaryUnreachable: true);
|
||||
|
||||
host.Tell(Snapshot((Self672, RedundancyRole.Secondary), (Peer672, RedundancyRole.Primary)));
|
||||
AwaitAssert(() => hist.Published.ShouldBe([false]), duration: Timeout);
|
||||
|
||||
host.Tell(new ClusterEvent.UnreachableMember(MemberAt(Peer672)));
|
||||
AwaitAssert(() => hist.Published.ShouldBe([false, true]), duration: Timeout);
|
||||
|
||||
host.Tell(new ClusterEvent.MemberRemoved(MemberAt(Peer672, MemberStatus.Removed), MemberStatus.Down));
|
||||
host.Tell(Snapshot((Self672, RedundancyRole.Primary)));
|
||||
AwaitAssert(() => hist.Published.ShouldBe([false, true, true]), duration: Timeout);
|
||||
}
|
||||
|
||||
/// <summary>A re-joined incarnation at the dead peer's address is reachable again.</summary>
|
||||
[Fact]
|
||||
public void A_rejoined_peer_is_reachable_again()
|
||||
{
|
||||
var hist = new RecordingHistorizationView();
|
||||
IActorRef host = SpawnHost672(hist, others: null, captureWhilePrimaryUnreachable: true);
|
||||
|
||||
host.Tell(Snapshot((Self672, RedundancyRole.Secondary), (Peer672, RedundancyRole.Primary)));
|
||||
host.Tell(new ClusterEvent.UnreachableMember(MemberAt(Peer672)));
|
||||
AwaitAssert(() => hist.Published.ShouldBe([false, true]), duration: Timeout);
|
||||
|
||||
host.Tell(new ClusterEvent.MemberUp(MemberAt(Peer672)));
|
||||
AwaitAssert(() => hist.Published.ShouldBe([false, true, false]), duration: Timeout);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// End to end on a REAL two-node cluster: crash the recording Primary and the survivor's capture gate
|
||||
/// opens on failure detection alone — while the dead node is still a member, i.e. well before any
|
||||
/// auto-down could have removed it and re-homed the redundancy singleton.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Downing is disabled outright here, so the dead node stays a member forever: the only signal that
|
||||
/// can open the gate is the failure detector's <c>UnreachableMember</c>. The crash is
|
||||
/// <c>Terminate()</c> with <c>run-by-actor-system-terminate = off</c>, which gossips no Leave — a
|
||||
/// process death, not a graceful drain (pattern from <c>SelfFirstSeedBootstrapTests</c>).
|
||||
/// </remarks>
|
||||
[Fact]
|
||||
public async Task A_crashed_primary_opens_the_survivors_capture_gate_before_it_is_downed()
|
||||
{
|
||||
var portA = FreePort();
|
||||
var portB = FreePort();
|
||||
const string systemName = "otopcua-672";
|
||||
var sysA = ActorSystem.Create(systemName, TwoNodeHocon(portA, portA));
|
||||
var sysB = ActorSystem.Create(systemName, TwoNodeHocon(portB, portA));
|
||||
try
|
||||
{
|
||||
var clusterB = Akka.Cluster.Cluster.Get(sysB);
|
||||
AwaitCondition(
|
||||
() => clusterB.State.Members.Count(m => m.Status == MemberStatus.Up) == 2,
|
||||
TimeSpan.FromSeconds(30));
|
||||
|
||||
var nodeA = NodeId.Parse($"127.0.0.1:{portA}");
|
||||
var nodeB = NodeId.Parse($"127.0.0.1:{portB}");
|
||||
var hist = new RecordingHistorizationView();
|
||||
IActorRef host = sysB.ActorOf(DriverHostActor.Props(
|
||||
dbFactory: null,
|
||||
nodeB,
|
||||
ackRouter: null,
|
||||
driverMemberCountProvider: () => 2,
|
||||
historizationRoleView: hist,
|
||||
sharesHistorizationScopeWithLocal: node => node == nodeA));
|
||||
|
||||
host.Tell(Snapshot((nodeB, RedundancyRole.Secondary), (nodeA, RedundancyRole.Primary)));
|
||||
AwaitAssert(() => hist.Published.ShouldBe([false]), duration: Timeout);
|
||||
|
||||
await sysA.Terminate();
|
||||
|
||||
AwaitAssert(() => hist.Published.ShouldBe([false, true]), duration: TimeSpan.FromSeconds(30));
|
||||
// Still a member: the gate opened on reachability, not on a removal + fresh snapshot.
|
||||
clusterB.State.Members.ShouldContain(m => m.Address.Port == portA);
|
||||
clusterB.State.Unreachable.ShouldContain(m => m.Address.Port == portA);
|
||||
}
|
||||
finally
|
||||
{
|
||||
await sysB.Terminate();
|
||||
if (!sysA.WhenTerminated.IsCompleted) await sysA.Terminate();
|
||||
}
|
||||
}
|
||||
|
||||
private static int FreePort()
|
||||
{
|
||||
var listener = new System.Net.Sockets.TcpListener(System.Net.IPAddress.Loopback, 0);
|
||||
listener.Start();
|
||||
var port = ((System.Net.IPEndPoint)listener.LocalEndpoint).Port;
|
||||
listener.Stop();
|
||||
return port;
|
||||
}
|
||||
|
||||
private static Akka.Configuration.Config TwoNodeHocon(int selfPort, int seedPort) =>
|
||||
Akka.Configuration.ConfigurationFactory.ParseString($@"
|
||||
akka {{
|
||||
loglevel = ""WARNING""
|
||||
extensions = [""Akka.Cluster.Tools.PublishSubscribe.DistributedPubSubExtensionProvider, Akka.Cluster.Tools""]
|
||||
actor.provider = ""Akka.Cluster.ClusterActorRefProvider, Akka.Cluster""
|
||||
remote.dot-netty.tcp {{ hostname = ""127.0.0.1"", port = {selfPort} }}
|
||||
coordinated-shutdown.run-by-actor-system-terminate = off
|
||||
cluster {{
|
||||
seed-nodes = [""akka.tcp://otopcua-672@127.0.0.1:{seedPort}""]
|
||||
roles = [""driver""]
|
||||
downing-provider-class = """"
|
||||
run-coordinated-shutdown-when-down = off
|
||||
failure-detector {{ heartbeat-interval = 1s, acceptable-heartbeat-pause = 3s }}
|
||||
}}
|
||||
}}");
|
||||
|
||||
// ---------------- helpers ----------------
|
||||
|
||||
/// <summary>Records the drain and data-plane verdicts the host publishes.</summary>
|
||||
private sealed class RecordingOtherViews : IRedundancyRoleView, IDriverPrimaryRoleSource
|
||||
{
|
||||
private readonly List<bool> _drained = [];
|
||||
private readonly List<bool> _primary = [];
|
||||
private readonly Lock _gate = new();
|
||||
|
||||
public RedundancyRoleView Current { get; } = new();
|
||||
|
||||
public IReadOnlyList<bool> Drained
|
||||
{
|
||||
get { lock (_gate) return _drained.ToArray(); }
|
||||
}
|
||||
|
||||
public IReadOnlyList<bool> Primary
|
||||
{
|
||||
get { lock (_gate) return _primary.ToArray(); }
|
||||
}
|
||||
|
||||
public bool ShouldDrainAlarmHistory => Current.ShouldDrainAlarmHistory;
|
||||
|
||||
public bool ShouldActAsPrimary => Current.ShouldActAsPrimary;
|
||||
|
||||
public void Publish(bool shouldDrainAlarmHistory)
|
||||
{
|
||||
lock (_gate) _drained.Add(shouldDrainAlarmHistory);
|
||||
Current.Publish(shouldDrainAlarmHistory);
|
||||
}
|
||||
|
||||
public void PublishPrimaryRole(bool shouldActAsPrimary)
|
||||
{
|
||||
lock (_gate) _primary.Add(shouldActAsPrimary);
|
||||
Current.PublishPrimaryRole(shouldActAsPrimary);
|
||||
}
|
||||
}
|
||||
|
||||
private IActorRef SpawnHost672(
|
||||
IHistorizationRoleView view, RecordingOtherViews? others, bool captureWhilePrimaryUnreachable) =>
|
||||
Sys.ActorOf(DriverHostActor.Props(
|
||||
NewInMemoryDbFactory(),
|
||||
Self672,
|
||||
ackRouter: null,
|
||||
driverMemberCountProvider: () => 2,
|
||||
redundancyRoleView: others,
|
||||
primaryRoleSource: others,
|
||||
historizationRoleView: view,
|
||||
sharesHistorizationScopeWithLocal: node => node == Peer672 || node == Self672,
|
||||
sharesAlarmQueueWithLocal: node => node == Peer672,
|
||||
captureWhilePrimaryUnreachable: captureWhilePrimaryUnreachable));
|
||||
|
||||
private static int _memberUid;
|
||||
|
||||
// Akka's Member.Create factories are internal — same reflection idiom as the ControlPlane tests.
|
||||
private static readonly MethodInfo MemberCreate = typeof(Member).GetMethod(
|
||||
"Create",
|
||||
BindingFlags.Static | BindingFlags.NonPublic | BindingFlags.Public,
|
||||
binder: null,
|
||||
types:
|
||||
[
|
||||
typeof(UniqueAddress), typeof(int), typeof(MemberStatus),
|
||||
typeof(ImmutableHashSet<string>), typeof(AppVersion),
|
||||
],
|
||||
modifiers: null)
|
||||
?? throw new InvalidOperationException("Akka.Cluster.Member.Create(5-arg) not found — Akka API changed");
|
||||
|
||||
private static Member MemberAt(NodeId node, MemberStatus status = MemberStatus.Up)
|
||||
{
|
||||
var text = node.ToString();
|
||||
var colon = text.LastIndexOf(':');
|
||||
return (Member)MemberCreate.Invoke(null,
|
||||
[
|
||||
new UniqueAddress(
|
||||
new Address("akka.tcp", "otopcua", text[..colon], int.Parse(text[(colon + 1)..])),
|
||||
Interlocked.Increment(ref _memberUid)),
|
||||
1,
|
||||
status,
|
||||
ImmutableHashSet.Create("driver"),
|
||||
AppVersion.Zero,
|
||||
])!;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A view that remembers every published decision, so a test asserts on the actor's ACTION
|
||||
/// rather than on a state its open seed already satisfies.
|
||||
|
||||
+1
@@ -54,6 +54,7 @@ public sealed class ContinuousHistorizationOptionsTests
|
||||
Assert.Equal(8, options.DrainConcurrency);
|
||||
Assert.Equal(500_000, options.Capacity);
|
||||
Assert.Equal(5, options.MaxWriteAttempts);
|
||||
Assert.True(options.CaptureWhilePrimaryUnreachable); // Gitea #672
|
||||
}
|
||||
|
||||
/// <summary>A fully-configured enabled recorder produces no warnings.</summary>
|
||||
|
||||
Reference in New Issue
Block a user