fix(runtime): pin the Primary-verdict member-event subscription with a test; document the pre-join allow window (review R6 MINOR-2/3)
MINOR-3: new DriverHostActorPrimaryRoleSourceTests case drives the deny flip with a real ClusterEvent.MemberUp on the EventStream (how the cluster's domain-event publisher reaches a Cluster.Subscribe subscriber) instead of a direct RepublishPrimaryVerdict tell; it fails when the Cluster.Subscribe(IMemberEvent) call is deleted. MINOR-2: the published verdict MIRRORS the live gate; before this node has joined its member count is 0 and both allow (seconds; up to the bootstrap guard's 25 s probe), so the Sparkplug primary-host claim can be held on both nodes in that window (native ack is unaffected - view and route agree). Behaviour deliberately unchanged: denying until self is Up would change the live gate too and delay writes/acks on a single-node cluster's own join. Comments + AlarmTracking.md / Redundancy.md / Mqtt.md corrected. Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
This commit is contained in:
@@ -357,6 +357,11 @@ is client-visible.
|
||||
> pair (role still unknown, a driver peer Up) the gate already denies, exactly
|
||||
> as the route does (review R1 m2 — previously the view sat on its single-node
|
||||
> "allow" seed until the first snapshot, reopening the flap for that window).
|
||||
> It mirrors, it does not deny from start: until this node has **joined** and
|
||||
> sees its peer Up, its member count is 0 and both the published verdict and the
|
||||
> live route **allow** (seconds normally; up to the bootstrap guard's 25 s
|
||||
> partner probe). The two agree there, so the flap cannot occur, but a native
|
||||
> ack in that pre-join window is serviced (review R6 MINOR-2).
|
||||
|
||||
> **Best-effort during reconnect.** If a native ack arrives while the owning
|
||||
> driver is `Connecting`/`Reconnecting`, it is dropped with a debug log and **not**
|
||||
|
||||
+1
-1
@@ -526,7 +526,7 @@ Three data-plane surfaces are **Primary-gated** so a warm standby never touches
|
||||
|
||||
Before archreview 03/S4 the unknown-role case defaulted to *allow* unconditionally, opening a dual-primary window: a freshly-booted secondary on a multi-node cluster serviced shared-device writes and emitted duplicate fleet-wide alerts for up to the ~10 s `RedundancyStateActor` heartbeat (and indefinitely if the snapshot's NodeId never matched this node). Membership is authoritative much earlier than DPS delivery, so an unknown role on a **multi-driver** cluster now default-**denies**. The driver member count is read live from `Cluster.State.Members` (Up members with the `driver` role); a non-cluster ActorRefProvider yields 0 ⇒ single-node posture. If a snapshot arrives that never mentions this node while a driver peer exists (the 03/S5 identity-mismatch shape), `DriverHostActor` logs a one-time Warning so the silent mismatch is diagnosable.
|
||||
|
||||
**What a client sees during the boot window (multi-node, role unknown):** an inbound write is **rejected** (not queued — OT actuation seconds late is a surprise). The reject rides the sanctioned optimistic-write self-correction surface: the node reverts to its prior value with a transient Bad-quality blip and a Part 8 `AuditWriteUpdateEvent`; the client's synchronous write call still returned Good (exactly as any failed device write behaves since the #5 write-outcome self-correction). The rejection reason distinguishes the boot window (`"not primary (role unknown)"`) from a steady-state Secondary (`"not primary"`). Acks are dropped (Warning-logged when role-unknown) — and since review R1 m2 a native OPC UA Acknowledge is refused up front (`BadInvalidState`) in the same window: `DriverHostActor` publishes its Primary verdict to `IDriverPrimaryRoleSource` (the native-ack gate, and the MQTT Sparkplug primary-host claim) once at start and again on every cluster membership change, not only on redundancy snapshots, so the published verdict denies as soon as a driver peer is Up rather than sitting on the single-node "allow" seed until the first snapshot. The alerts emit is skipped (the Primary peer publishes the single fleet copy). The window closes on the first delivered snapshot.
|
||||
**What a client sees during the boot window (multi-node, role unknown):** an inbound write is **rejected** (not queued — OT actuation seconds late is a surprise). The reject rides the sanctioned optimistic-write self-correction surface: the node reverts to its prior value with a transient Bad-quality blip and a Part 8 `AuditWriteUpdateEvent`; the client's synchronous write call still returned Good (exactly as any failed device write behaves since the #5 write-outcome self-correction). The rejection reason distinguishes the boot window (`"not primary (role unknown)"`) from a steady-state Secondary (`"not primary"`). Acks are dropped (Warning-logged when role-unknown) — and since review R1 m2 a native OPC UA Acknowledge is refused up front (`BadInvalidState`) in the same window: `DriverHostActor` publishes its Primary verdict to `IDriverPrimaryRoleSource` (the native-ack gate, and the MQTT Sparkplug primary-host claim) once at start and again on every cluster membership change, not only on redundancy snapshots, so the published verdict denies as soon as a driver peer is Up rather than sitting on the single-node "allow" seed until the first snapshot. ⚠️ "As soon as a driver peer is Up" means **in this node's own cluster view**, i.e. after it has joined: the host is spawned before the join, when the view holds no members, and a member count of 0 takes the single-node *allow* posture — for the published verdict and the live gate alike. So a node cold-starting into a pair whose peer is already Primary allows until its own join completes (seconds normally; up to `Cluster:BootstrapGuard:PartnerProbeSeconds`, 25 s, under the bootstrap guard), and in that window the MQTT Sparkplug primary-host claim can be held on both nodes. Native ack is not exposed (the view and the route agree, so there is no acked → unacked revert). Denying until self is Up was not adopted: it would also deny writes/acks during a single-node cluster's own join (review R6 MINOR-2). The alerts emit is skipped (the Primary peer publishes the single fleet copy). The window closes on the first delivered snapshot.
|
||||
|
||||
Under warm/hot redundancy both cluster nodes run `ScriptedAlarmHostActor` and evaluate scripted alarms, keeping each node's address space and engine state warm for instant failover. To avoid duplicate rows on `/alerts` and duplicate historian writes, only the Primary node publishes externally:
|
||||
|
||||
|
||||
@@ -363,9 +363,14 @@ Hence: bind the claim to a connection, and rebuild the connection on role change
|
||||
| Role unknown, no driver peer (single node) | open | `ONLINE` |
|
||||
|
||||
The verdict comes from `PrimaryGatePolicy.ShouldServiceAsPrimary` via `IDriverPrimaryRoleSource`
|
||||
(`Core.Abstractions`), published by `DriverHostActor` on every redundancy snapshot and **sampled on a
|
||||
1-second timer** rather than evented — a process-singleton raising events at per-instance drivers
|
||||
outlives every driver it holds a reference to.
|
||||
(`Core.Abstractions`), published by `DriverHostActor` at host start, on every cluster membership change and on
|
||||
every redundancy snapshot, and **sampled on a 1-second timer** rather than evented — a process-singleton
|
||||
raising events at per-instance drivers outlives every driver it holds a reference to.
|
||||
|
||||
> ⚠️ **"Driver peer present" is this node's own cluster view.** Before the node has joined, it sees no
|
||||
> members, which is the single-node row: a node cold-starting into a pair whose peer is already Primary
|
||||
> can open the claim until its join completes (seconds; up to the bootstrap guard's 25 s partner probe),
|
||||
> so both nodes may briefly hold it (review R6 MINOR-2).
|
||||
|
||||
> **Accepted trade: a brief edge-visible `OFFLINE` during failover**, while the old Primary stands down
|
||||
> and the new one connects and republishes. That is a genuine state transition — for that window there
|
||||
|
||||
@@ -802,6 +802,19 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
/// membership changes (a driver peer coming Up turns an unknown role from allow into deny before any
|
||||
/// snapshot names this node). A non-cluster ActorRefProvider (legacy/test harnesses) just keeps the
|
||||
/// initial publish.
|
||||
/// <para>
|
||||
/// ⚠️ <b>It MIRRORS the live gate; it does not deny from host start.</b> The host is spawned at
|
||||
/// ActorSystem start, before this node has joined, when its view of <c>Cluster.State.Members</c> is
|
||||
/// empty — a member count of 0, which the single-node posture ALLOWS. So the verdict published here is
|
||||
/// allow, and it flips to deny only once this node's own cluster state shows a second Up driver member:
|
||||
/// normally seconds, up to <c>Cluster:BootstrapGuard:PartnerProbeSeconds</c> (25 s) under the
|
||||
/// bootstrap guard. In that pre-join window the Sparkplug primary-host claim can be held on both nodes of
|
||||
/// a pair. Native ack is NOT exposed by it: the live route gate reads the same count and also allows, so
|
||||
/// the view and the route still agree and the acked → unacked revert cannot occur (review R6 MINOR-2).
|
||||
/// Denying until self is Up was considered and NOT done: it would change the live data-plane gate (the
|
||||
/// two must stay one decision) and so delay writes and acks on a genuinely single-node cluster during its
|
||||
/// own join, and this actor has no reliable way to know that a partner is configured.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
private void StartPrimaryVerdictPublishing()
|
||||
{
|
||||
|
||||
+25
@@ -66,6 +66,31 @@ public sealed class DriverHostActorPrimaryRoleSourceTests : RuntimeActorTestBase
|
||||
AwaitAssert(() => source.Published.ShouldBe([true, false]), duration: Timeout);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Review R6 MINOR-3 — the same flip driven by a real cluster MEMBER EVENT rather than a direct
|
||||
/// <see cref="DriverHostActor.RepublishPrimaryVerdict"/> tell, so it fails if the host's
|
||||
/// <c>Cluster.Subscribe(… IMemberEvent)</c> or its <c>IMemberEvent</c> handler is deleted. The event is published
|
||||
/// on the ActorSystem EventStream, which is exactly how the cluster's domain-event publisher delivers it to a
|
||||
/// <c>Cluster.Subscribe</c> subscriber — and to nothing else. Re-published inside the wait because the
|
||||
/// subscription is registered asynchronously by the publisher actor after <c>PreStart</c>.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void A_cluster_member_event_flips_the_published_verdict_without_a_direct_republish()
|
||||
{
|
||||
var source = new RecordingPrimaryRoleSource();
|
||||
var members = 1;
|
||||
SpawnHost(source, () => Volatile.Read(ref members));
|
||||
AwaitAssert(() => source.Published.ShouldBe([true]), duration: Timeout);
|
||||
|
||||
Volatile.Write(ref members, 2);
|
||||
var memberUp = new Akka.Cluster.ClusterEvent.MemberUp(Akka.Cluster.Cluster.Get(Sys).SelfMember);
|
||||
AwaitAssert(() =>
|
||||
{
|
||||
Sys.EventStream.Publish(memberUp);
|
||||
source.Published.ShouldBe([true, false]);
|
||||
}, duration: Timeout, interval: TimeSpan.FromMilliseconds(100));
|
||||
}
|
||||
|
||||
private IActorRef SpawnHost(IDriverPrimaryRoleSource source, Func<int> driverMemberCount) =>
|
||||
Sys.ActorOf(DriverHostActor.Props(
|
||||
NewInMemoryDbFactory(),
|
||||
|
||||
Reference in New Issue
Block a user