fix(runtime): pin the Primary-verdict member-event subscription with a test; document the pre-join allow window (review R6 MINOR-2/3)
MINOR-3: new DriverHostActorPrimaryRoleSourceTests case drives the deny flip with a real ClusterEvent.MemberUp on the EventStream (how the cluster's domain-event publisher reaches a Cluster.Subscribe subscriber) instead of a direct RepublishPrimaryVerdict tell; it fails when the Cluster.Subscribe(IMemberEvent) call is deleted. MINOR-2: the published verdict MIRRORS the live gate; before this node has joined its member count is 0 and both allow (seconds; up to the bootstrap guard's 25 s probe), so the Sparkplug primary-host claim can be held on both nodes in that window (native ack is unaffected - view and route agree). Behaviour deliberately unchanged: denying until self is Up would change the live gate too and delay writes/acks on a single-node cluster's own join. Comments + AlarmTracking.md / Redundancy.md / Mqtt.md corrected. Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
This commit is contained in:
@@ -357,6 +357,11 @@ is client-visible.
|
|||||||
> pair (role still unknown, a driver peer Up) the gate already denies, exactly
|
> pair (role still unknown, a driver peer Up) the gate already denies, exactly
|
||||||
> as the route does (review R1 m2 — previously the view sat on its single-node
|
> as the route does (review R1 m2 — previously the view sat on its single-node
|
||||||
> "allow" seed until the first snapshot, reopening the flap for that window).
|
> "allow" seed until the first snapshot, reopening the flap for that window).
|
||||||
|
> It mirrors, it does not deny from start: until this node has **joined** and
|
||||||
|
> sees its peer Up, its member count is 0 and both the published verdict and the
|
||||||
|
> live route **allow** (seconds normally; up to the bootstrap guard's 25 s
|
||||||
|
> partner probe). The two agree there, so the flap cannot occur, but a native
|
||||||
|
> ack in that pre-join window is serviced (review R6 MINOR-2).
|
||||||
|
|
||||||
> **Best-effort during reconnect.** If a native ack arrives while the owning
|
> **Best-effort during reconnect.** If a native ack arrives while the owning
|
||||||
> driver is `Connecting`/`Reconnecting`, it is dropped with a debug log and **not**
|
> driver is `Connecting`/`Reconnecting`, it is dropped with a debug log and **not**
|
||||||
|
|||||||
+1
-1
@@ -526,7 +526,7 @@ Three data-plane surfaces are **Primary-gated** so a warm standby never touches
|
|||||||
|
|
||||||
Before archreview 03/S4 the unknown-role case defaulted to *allow* unconditionally, opening a dual-primary window: a freshly-booted secondary on a multi-node cluster serviced shared-device writes and emitted duplicate fleet-wide alerts for up to the ~10 s `RedundancyStateActor` heartbeat (and indefinitely if the snapshot's NodeId never matched this node). Membership is authoritative much earlier than DPS delivery, so an unknown role on a **multi-driver** cluster now default-**denies**. The driver member count is read live from `Cluster.State.Members` (Up members with the `driver` role); a non-cluster ActorRefProvider yields 0 ⇒ single-node posture. If a snapshot arrives that never mentions this node while a driver peer exists (the 03/S5 identity-mismatch shape), `DriverHostActor` logs a one-time Warning so the silent mismatch is diagnosable.
|
Before archreview 03/S4 the unknown-role case defaulted to *allow* unconditionally, opening a dual-primary window: a freshly-booted secondary on a multi-node cluster serviced shared-device writes and emitted duplicate fleet-wide alerts for up to the ~10 s `RedundancyStateActor` heartbeat (and indefinitely if the snapshot's NodeId never matched this node). Membership is authoritative much earlier than DPS delivery, so an unknown role on a **multi-driver** cluster now default-**denies**. The driver member count is read live from `Cluster.State.Members` (Up members with the `driver` role); a non-cluster ActorRefProvider yields 0 ⇒ single-node posture. If a snapshot arrives that never mentions this node while a driver peer exists (the 03/S5 identity-mismatch shape), `DriverHostActor` logs a one-time Warning so the silent mismatch is diagnosable.
|
||||||
|
|
||||||
**What a client sees during the boot window (multi-node, role unknown):** an inbound write is **rejected** (not queued — OT actuation seconds late is a surprise). The reject rides the sanctioned optimistic-write self-correction surface: the node reverts to its prior value with a transient Bad-quality blip and a Part 8 `AuditWriteUpdateEvent`; the client's synchronous write call still returned Good (exactly as any failed device write behaves since the #5 write-outcome self-correction). The rejection reason distinguishes the boot window (`"not primary (role unknown)"`) from a steady-state Secondary (`"not primary"`). Acks are dropped (Warning-logged when role-unknown) — and since review R1 m2 a native OPC UA Acknowledge is refused up front (`BadInvalidState`) in the same window: `DriverHostActor` publishes its Primary verdict to `IDriverPrimaryRoleSource` (the native-ack gate, and the MQTT Sparkplug primary-host claim) once at start and again on every cluster membership change, not only on redundancy snapshots, so the published verdict denies as soon as a driver peer is Up rather than sitting on the single-node "allow" seed until the first snapshot. The alerts emit is skipped (the Primary peer publishes the single fleet copy). The window closes on the first delivered snapshot.
|
**What a client sees during the boot window (multi-node, role unknown):** an inbound write is **rejected** (not queued — OT actuation seconds late is a surprise). The reject rides the sanctioned optimistic-write self-correction surface: the node reverts to its prior value with a transient Bad-quality blip and a Part 8 `AuditWriteUpdateEvent`; the client's synchronous write call still returned Good (exactly as any failed device write behaves since the #5 write-outcome self-correction). The rejection reason distinguishes the boot window (`"not primary (role unknown)"`) from a steady-state Secondary (`"not primary"`). Acks are dropped (Warning-logged when role-unknown) — and since review R1 m2 a native OPC UA Acknowledge is refused up front (`BadInvalidState`) in the same window: `DriverHostActor` publishes its Primary verdict to `IDriverPrimaryRoleSource` (the native-ack gate, and the MQTT Sparkplug primary-host claim) once at start and again on every cluster membership change, not only on redundancy snapshots, so the published verdict denies as soon as a driver peer is Up rather than sitting on the single-node "allow" seed until the first snapshot. ⚠️ "As soon as a driver peer is Up" means **in this node's own cluster view**, i.e. after it has joined: the host is spawned before the join, when the view holds no members, and a member count of 0 takes the single-node *allow* posture — for the published verdict and the live gate alike. So a node cold-starting into a pair whose peer is already Primary allows until its own join completes (seconds normally; up to `Cluster:BootstrapGuard:PartnerProbeSeconds`, 25 s, under the bootstrap guard), and in that window the MQTT Sparkplug primary-host claim can be held on both nodes. Native ack is not exposed (the view and the route agree, so there is no acked → unacked revert). Denying until self is Up was not adopted: it would also deny writes/acks during a single-node cluster's own join (review R6 MINOR-2). The alerts emit is skipped (the Primary peer publishes the single fleet copy). The window closes on the first delivered snapshot.
|
||||||
|
|
||||||
Under warm/hot redundancy both cluster nodes run `ScriptedAlarmHostActor` and evaluate scripted alarms, keeping each node's address space and engine state warm for instant failover. To avoid duplicate rows on `/alerts` and duplicate historian writes, only the Primary node publishes externally:
|
Under warm/hot redundancy both cluster nodes run `ScriptedAlarmHostActor` and evaluate scripted alarms, keeping each node's address space and engine state warm for instant failover. To avoid duplicate rows on `/alerts` and duplicate historian writes, only the Primary node publishes externally:
|
||||||
|
|
||||||
|
|||||||
@@ -363,9 +363,14 @@ Hence: bind the claim to a connection, and rebuild the connection on role change
|
|||||||
| Role unknown, no driver peer (single node) | open | `ONLINE` |
|
| Role unknown, no driver peer (single node) | open | `ONLINE` |
|
||||||
|
|
||||||
The verdict comes from `PrimaryGatePolicy.ShouldServiceAsPrimary` via `IDriverPrimaryRoleSource`
|
The verdict comes from `PrimaryGatePolicy.ShouldServiceAsPrimary` via `IDriverPrimaryRoleSource`
|
||||||
(`Core.Abstractions`), published by `DriverHostActor` on every redundancy snapshot and **sampled on a
|
(`Core.Abstractions`), published by `DriverHostActor` at host start, on every cluster membership change and on
|
||||||
1-second timer** rather than evented — a process-singleton raising events at per-instance drivers
|
every redundancy snapshot, and **sampled on a 1-second timer** rather than evented — a process-singleton
|
||||||
outlives every driver it holds a reference to.
|
raising events at per-instance drivers outlives every driver it holds a reference to.
|
||||||
|
|
||||||
|
> ⚠️ **"Driver peer present" is this node's own cluster view.** Before the node has joined, it sees no
|
||||||
|
> members, which is the single-node row: a node cold-starting into a pair whose peer is already Primary
|
||||||
|
> can open the claim until its join completes (seconds; up to the bootstrap guard's 25 s partner probe),
|
||||||
|
> so both nodes may briefly hold it (review R6 MINOR-2).
|
||||||
|
|
||||||
> **Accepted trade: a brief edge-visible `OFFLINE` during failover**, while the old Primary stands down
|
> **Accepted trade: a brief edge-visible `OFFLINE` during failover**, while the old Primary stands down
|
||||||
> and the new one connects and republishes. That is a genuine state transition — for that window there
|
> and the new one connects and republishes. That is a genuine state transition — for that window there
|
||||||
|
|||||||
@@ -802,6 +802,19 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
|||||||
/// membership changes (a driver peer coming Up turns an unknown role from allow into deny before any
|
/// membership changes (a driver peer coming Up turns an unknown role from allow into deny before any
|
||||||
/// snapshot names this node). A non-cluster ActorRefProvider (legacy/test harnesses) just keeps the
|
/// snapshot names this node). A non-cluster ActorRefProvider (legacy/test harnesses) just keeps the
|
||||||
/// initial publish.
|
/// initial publish.
|
||||||
|
/// <para>
|
||||||
|
/// ⚠️ <b>It MIRRORS the live gate; it does not deny from host start.</b> The host is spawned at
|
||||||
|
/// ActorSystem start, before this node has joined, when its view of <c>Cluster.State.Members</c> is
|
||||||
|
/// empty — a member count of 0, which the single-node posture ALLOWS. So the verdict published here is
|
||||||
|
/// allow, and it flips to deny only once this node's own cluster state shows a second Up driver member:
|
||||||
|
/// normally seconds, up to <c>Cluster:BootstrapGuard:PartnerProbeSeconds</c> (25 s) under the
|
||||||
|
/// bootstrap guard. In that pre-join window the Sparkplug primary-host claim can be held on both nodes of
|
||||||
|
/// a pair. Native ack is NOT exposed by it: the live route gate reads the same count and also allows, so
|
||||||
|
/// the view and the route still agree and the acked → unacked revert cannot occur (review R6 MINOR-2).
|
||||||
|
/// Denying until self is Up was considered and NOT done: it would change the live data-plane gate (the
|
||||||
|
/// two must stay one decision) and so delay writes and acks on a genuinely single-node cluster during its
|
||||||
|
/// own join, and this actor has no reliable way to know that a partner is configured.
|
||||||
|
/// </para>
|
||||||
/// </summary>
|
/// </summary>
|
||||||
private void StartPrimaryVerdictPublishing()
|
private void StartPrimaryVerdictPublishing()
|
||||||
{
|
{
|
||||||
|
|||||||
+25
@@ -66,6 +66,31 @@ public sealed class DriverHostActorPrimaryRoleSourceTests : RuntimeActorTestBase
|
|||||||
AwaitAssert(() => source.Published.ShouldBe([true, false]), duration: Timeout);
|
AwaitAssert(() => source.Published.ShouldBe([true, false]), duration: Timeout);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Review R6 MINOR-3 — the same flip driven by a real cluster MEMBER EVENT rather than a direct
|
||||||
|
/// <see cref="DriverHostActor.RepublishPrimaryVerdict"/> tell, so it fails if the host's
|
||||||
|
/// <c>Cluster.Subscribe(… IMemberEvent)</c> or its <c>IMemberEvent</c> handler is deleted. The event is published
|
||||||
|
/// on the ActorSystem EventStream, which is exactly how the cluster's domain-event publisher delivers it to a
|
||||||
|
/// <c>Cluster.Subscribe</c> subscriber — and to nothing else. Re-published inside the wait because the
|
||||||
|
/// subscription is registered asynchronously by the publisher actor after <c>PreStart</c>.
|
||||||
|
/// </summary>
|
||||||
|
[Fact]
|
||||||
|
public void A_cluster_member_event_flips_the_published_verdict_without_a_direct_republish()
|
||||||
|
{
|
||||||
|
var source = new RecordingPrimaryRoleSource();
|
||||||
|
var members = 1;
|
||||||
|
SpawnHost(source, () => Volatile.Read(ref members));
|
||||||
|
AwaitAssert(() => source.Published.ShouldBe([true]), duration: Timeout);
|
||||||
|
|
||||||
|
Volatile.Write(ref members, 2);
|
||||||
|
var memberUp = new Akka.Cluster.ClusterEvent.MemberUp(Akka.Cluster.Cluster.Get(Sys).SelfMember);
|
||||||
|
AwaitAssert(() =>
|
||||||
|
{
|
||||||
|
Sys.EventStream.Publish(memberUp);
|
||||||
|
source.Published.ShouldBe([true, false]);
|
||||||
|
}, duration: Timeout, interval: TimeSpan.FromMilliseconds(100));
|
||||||
|
}
|
||||||
|
|
||||||
private IActorRef SpawnHost(IDriverPrimaryRoleSource source, Func<int> driverMemberCount) =>
|
private IActorRef SpawnHost(IDriverPrimaryRoleSource source, Func<int> driverMemberCount) =>
|
||||||
Sys.ActorOf(DriverHostActor.Props(
|
Sys.ActorOf(DriverHostActor.Props(
|
||||||
NewInMemoryDbFactory(),
|
NewInMemoryDbFactory(),
|
||||||
|
|||||||
Reference in New Issue
Block a user