dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 5 Pull Requests Actions Packages Projects Releases Wiki Activity

Frozen node returns as a permanent second active after being downed (ThisActorSystemQuarantinedEvent ignored) — auto-down and leader-auto-down #132

New Issue
Closed
opened 2026-11-12 20:13:22 -05:00 by dohertj2 · 1 comment
dohertj2 commented 2026-11-12 20:13:22 -05:00
Owner
Copy Link
Copy Source

Found during the #116 rig drill on 2026-11-12. Evidence: docs/plans/2026-11-12-leader-auto-down-rig-drill.md §5.

What happens. A node that is frozen (docker pause) for longer than failure detection plus the down window (about 25 s) comes back as a second active node, and it stays that way:

  1. The survivor downs and removes the frozen node and takes over in about 29 s. This part is correct.
  2. When the frozen node resumes, every association attempt gets The remote system has quarantined this system. It marks the survivor unreachable, becomes leader of its own one-node view, downs the survivor 15 s later, and reports /health/active 200.
  3. Both nodes then stay 200 until an operator restarts one of them. On the rig this lasted 120–180 s, until the drill restarted it.

Reproduced on central and on site-a. It also happens on today's default auto-down, so this is not a regression in LeaderAutoDowningProvider, which is a faithful port. On a site pair, both nodes keep their data connections open and run scripts while in this state, so data is collected twice.

Why. Akka.Remote 1.5.62 publishes ThisActorSystemQuarantinedEvent on the event stream (EndpointManager, on an InvalidAssociation with DisassociateInfo.Quarantined). Akka.Cluster 1.5.62 does not subscribe to it: ClusterCoreDaemon handles only the opposite QuarantinedEvent. So nothing tells the quarantined node that it was removed.

Option (recommended, needs an owner decision). In LeaderAutoDowningProvider, subscribe to ThisActorSystemQuarantinedEvent. When the quarantining address is a peer in this cluster, log it and down this node (Cluster.Down(SelfAddress)). The existing WhenTerminated → StopApplication path then exits the process. The restart policy (docker unless-stopped, or NSSM on the boxes) brings it back as a new incarnation, and it rejoins as standby. This would apply to leader-auto-down only. Fit: this is the same "a removed node must not keep acting" rule that Akka JVM and Akka's own SBR follow. It suits the availability-first posture, because it trades an indefinite dual-active for a restart of about 10–30 s.

Trade-off to accept. After a real partition heals, both sides have quarantined each other, so both would down themselves and restart. That is a short full-pair outage that heals itself, instead of today's dual-active that waits for an operator. It changes the contract in #116 gate 4 ("both stay 200 until an operator restarts one side") and in the in-process test PartitionLongerThanWindow_EachSideDownsTheOther_BothStayUp. Alternative: leave the behaviour as it is and document freeze longer than ~25 s as part of the accepted dual-active class, with Overview's existing Split brain banner as the detection.

Not blocking the #116 rig gate, because it is parity with the current default.

Found during the #116 rig drill on 2026-11-12. Evidence: `docs/plans/2026-11-12-leader-auto-down-rig-drill.md` §5. **What happens.** A node that is frozen (`docker pause`) for longer than failure detection plus the down window (about 25 s) comes back as a second active node, and it stays that way: 1. The survivor downs and removes the frozen node and takes over in about 29 s. This part is correct. 2. When the frozen node resumes, every association attempt gets `The remote system has quarantined this system`. It marks the survivor unreachable, becomes leader of its own one-node view, downs the survivor 15 s later, and reports `/health/active` 200. 3. Both nodes then stay 200 until an operator restarts one of them. On the rig this lasted 120–180 s, until the drill restarted it. Reproduced on central and on site-a. **It also happens on today's default `auto-down`**, so this is not a regression in `LeaderAutoDowningProvider`, which is a faithful port. On a site pair, both nodes keep their data connections open and run scripts while in this state, so data is collected twice. **Why.** Akka.Remote 1.5.62 publishes `ThisActorSystemQuarantinedEvent` on the event stream (`EndpointManager`, on an `InvalidAssociation` with `DisassociateInfo.Quarantined`). Akka.Cluster 1.5.62 does not subscribe to it: `ClusterCoreDaemon` handles only the opposite `QuarantinedEvent`. So nothing tells the quarantined node that it was removed. **Option (recommended, needs an owner decision).** In `LeaderAutoDowningProvider`, subscribe to `ThisActorSystemQuarantinedEvent`. When the quarantining address is a peer in this cluster, log it and down this node (`Cluster.Down(SelfAddress)`). The existing `WhenTerminated → StopApplication` path then exits the process. The restart policy (docker `unless-stopped`, or NSSM on the boxes) brings it back as a new incarnation, and it rejoins as standby. This would apply to `leader-auto-down` only. Fit: this is the same "a removed node must not keep acting" rule that Akka JVM and Akka's own SBR follow. It suits the availability-first posture, because it trades an indefinite dual-active for a restart of about 10–30 s. **Trade-off to accept.** After a *real* partition heals, both sides have quarantined each other, so **both** would down themselves and restart. That is a short full-pair outage that heals itself, instead of today's dual-active that waits for an operator. It changes the contract in #116 gate 4 ("both stay 200 until an operator restarts one side") and in the in-process test `PartitionLongerThanWindow_EachSideDownsTheOther_BothStayUp`. Alternative: leave the behaviour as it is and document freeze longer than ~25 s as part of the accepted dual-active class, with Overview's existing **Split brain** banner as the detection. Not blocking the #116 rig gate, because it is parity with the current default.
dohertj2 added the type:followuppriority:p2 labels 2026-11-12 20:13:22 -05:00
dohertj2 commented 2026-11-12 21:38:48 -05:00
Author
Owner
Copy Link
Copy Source

Fixed: main @ 952db02a (code) + 482ac665 (docs and rig evidence)

The owner-approved approach is implemented under leader-auto-down only. The shipped default stays auto-down, and no wonder box was touched.

What fires (Akka.NET 1.5.62, from the decompiled assemblies, confirmed in-process and on the rig)

Two different signals, depending on who still counts whom as a member:

  • S1 ThisActorSystemQuarantinedEvent: "the peer has quarantined this node". It is published by EndpointManager on an InvalidAssociation with DisassociateInfo.Quarantined. The frozen node gets it at unpause (rig log: The remote system has quarantined this system). ClusterCoreDaemon subscribes only to QuarantinedEvent, which is the root cause.
  • S2 AssociationErrorEvent whose cause reads "The remote system has a UID that has been quarantined" (ProtocolStateActor, ForbiddenUidReason): "the peer still runs the incarnation this node removed". After a healed partition this is the only signal, and both sides get it. Each side aborts the other's handshake on its own refused UID before any quarantine disassociate arrives. So S1 alone would not have fixed the partition case.
  • After both sides have removed each other, nothing ever contacts the peer again. The new actor therefore probes every peer it saw removed after a Down (an Identify to /system/cluster/core/daemon, with a 5 s → 60 s back-off) until that address is a member again. Mutation check: with the probe removed, even the in-process freeze test goes red.

Design (ShutOutRestart, a child of the LeaderAutoDown downing actor, so LeaderAutoDownBase stays a line-for-line port)

Signal Decision
S1 while the peer is still a live member in this node's view restart now, no tiebreak. This node has not removed the peer, so the peer cannot be shut out by it. Only this side is.
S1 after this node has removed the peer too tiebreak now
S2 wait ShutOutGrace (60 s), because a frozen peer restarts itself on S1 in the meantime; if still split at the end, tiebreak
Tiebreak higher address restarts (Akka Member.AddressOrdering: host, then port). The lower node stays and restarts only if the peer has neither rejoined nor gone quiet one more grace later, which covers the single-sided case where the stale side is the lower address.

Why the address tiebreak, not age. Both nodes hold both addresses, in every signal and in their seed lists, so they compute the same answer without talking. Up-number is not safe: by the time a signal fires, each side has removed the other, so the peer's up-number exists only as remembered history. That history can differ after an unconverged join or a restart, and a disagreement means both nodes restart or neither does. The lower address is also the Akka leader of the re-formed pair and the bootstrap guard's preferred founder.

Restart reuses the existing self-down path: Cluster.Down(self) → run-coordinated-shutdown-when-down → the ActorSystem terminates → the WhenTerminated watchdog runs StopApplication, with a direct CoordinatedShutdown fallback after 20 s.

Supervisor check.

  • Docker restart: unless-stopped restarts on any exit code.
  • The wonder boxes run ScadaBridge as native .NET Windows services, not NSSM. NSSM wraps only the LDAP directory. Recovery is sc.exe failure … restart/5000/restart/5000/restart/10000 plus sc.exe failureflag 1.
  • With failureflag 1 the SCM restarts a service that stops on its own only if the reported exit code is non-zero. The watchdog used to exit 0. I did not test this on a box, because no wonder-box remote execution is allowed.
  • Fixed: after an unplanned ActorSystem termination the Host now exits 75 (AkkaHostedService.SelfDownExitCode). Program sets it after the host stops, and the watchdog sets it on the Windows service (ServiceBase.ExitCode) before the service reports SERVICE_STOPPED. This also covers the existing SBR / auto-down self-down path.

Config: ScadaBridge:Cluster:LeaderAutoDown:RestartWhenShutOut (default true, takes effect only with leader-auto-down) and ShutOutGrace (default 60s, validated > 0). Logs use the [leader-auto-down] prefix:

  • shut out by <peer> … tiebreak not needed — this node RESTARTS
  • split from <peer> … Giving it 00:01:00
  • Tiebreak says RESTART|STAY
  • rejoined (Up) within the shut-out grace; this node stays

Tests (all in Release)

  • ShutOutRestartTests: 18 unit tests (TestKit). They cover the single-side shut-out in both address orders, the tiebreak in both orders, the both-sides case with exactly one restart (S1 in both delivery orders, and S2 as a real healed partition produces it), the grace (stays if the peer goes quiet or rejoins, restarts if still split), duplicate and stranger events, probe back-off, and settings.
  • ShutOutRestartScenarioTests: 4 tests on real two-node clusters with the throttler transport:
    • the freeze shape in both address orders: the shut-out node terminates and the other stays Up;
    • a healed partition: only the higher port terminates, a fresh incarnation rejoins, and the survivor is still Up after its grace;
    • the same partition with the switch off: both stay up (positive control).
  • Also UnexpectedTerminationTests (exit flag, code ≠ 0), HoconBuilderTests, ClusterOptionsValidatorTests.
  • Mutation-checked, all red:
    • inverting the tiebreak: 7 unit tests plus the integration heal test;
    • removing the S1 handler: 11 unit tests plus both integration freeze tests;
    • removing the S2 handler: 4 unit tests;
    • removing the live-member fast path: 2 unit tests;
    • skipping the S2 grace: 3 unit tests;
    • removing the probe: 3 integration tests.
  • Full solution: every assembly passed except the following.
    • The 185 Playwright UI tests did not run, because there was no browser server.
    • TemplateEngine ScriptCompilerTests.GetOrAdd_SameCodeDifferentSurface_ComputesSeparateVerdicts failed once in the parallel run and passed 3/3 alone. TemplateEngine is untouched, so this is a shared-cache flake.
    • CoordinatedShutdownTests.HostSource_DoesNotContainEnvironmentExit caught an Environment.ExitCode mention in a comment; the comment was reworded, and Host.Tests is now 717/717.

Rig gates (local docker rig, image built from the fix; record in docs/plans/2026-11-12-leader-auto-down-rig-drill.md §7, logs sor-*.log)

Gate Central Site-a
Freeze (DRILL_ACTION=pause, active) PASS. Takeover 27 s. At unpause the victim logged shut out … RESTARTS within 1 s, exited 75 and rejoined as standby 34 s after unpause (30 s of that was the central host's own shutdown). 10/10 single-active. PASS. Takeover 27 s; rejoined 6 s after unpause; 10/10
Split > window, then heal PASS. Dual-active during the cut, as expected. Healed at +51 s; S2 on both sides 12 s later; tiebreak 60 s after that. Only central-b (higher) restarted; one 2-member cluster with central-a active at +76 s after heal PASS. Only site-a-b restarted; one active node at +111 s after heal (30 s of that was its host shutdown)
Kill regression (active) PASS, unchanged. Down +26 s, Traefik 28 s, rejoin 2 s, no shut-out lines PASS, unchanged. Down +28 s, rejoin 1 s

The rig is back on the default auto-down, all 8 nodes recreated on the fix image. Every node is ready, each pair has one active node and memberCount 2 / unreachableCount 0, and the active nodes are central-a, site-a-a, site-b-a and site-c-a.

Contract change (#116 gate 4). The dual-active still happens during a partition longer than the window. After the heal it now clears itself in roughly one probe interval (up to 60 s), plus the 60 s grace, plus one restart. Before, it waited for an operator. LeaderAutoDownScenarioTests.PartitionLongerThanWindow_EachSideDownsTheOther_BothStayUp still holds, because that test never heals.

Residual risks (documented):

  • If a frozen node takes longer than ShutOutGrace to shut down after S1, the survivor may also restart. That is a short double restart that still converges.
  • A probe of a dead peer logs one Akka association warning, backing off to one per minute.

Docs updated: Component-ClusterInfrastructure.md (Shut-out restart, and the downed-node recovery exit code), docs/components/ClusterInfrastructure.md, the #106 plan (§7 addendum) and CLAUDE.md. The untracked wonder deploy/…/RUNBOOK.md was not changed.

Closing: both rig gates pass. The rollout stays on #116.

## Fixed: `main` @ `952db02a` (code) + `482ac665` (docs and rig evidence) The owner-approved approach is implemented under `leader-auto-down` only. The shipped default stays `auto-down`, and no wonder box was touched. ### What fires (Akka.NET 1.5.62, from the decompiled assemblies, confirmed in-process and on the rig) Two different signals, depending on who still counts whom as a member: - **S1 `ThisActorSystemQuarantinedEvent`**: "the peer has quarantined *this* node". It is published by `EndpointManager` on an `InvalidAssociation` with `DisassociateInfo.Quarantined`. The **frozen node** gets it at unpause (rig log: `The remote system has quarantined this system`). `ClusterCoreDaemon` subscribes only to `QuarantinedEvent`, which is the root cause. - **S2 `AssociationErrorEvent`** whose cause reads "The remote system has a UID that has been quarantined" (`ProtocolStateActor`, `ForbiddenUidReason`): "the peer still runs the incarnation *this* node removed". **After a healed partition this is the only signal, and both sides get it.** Each side aborts the other's handshake on its own refused UID before any quarantine disassociate arrives. So S1 alone would not have fixed the partition case. - After both sides have removed each other, nothing ever contacts the peer again. The new actor therefore **probes** every peer it saw removed after a Down (an `Identify` to `/system/cluster/core/daemon`, with a 5 s → 60 s back-off) until that address is a member again. Mutation check: with the probe removed, even the in-process freeze test goes red. ### Design (`ShutOutRestart`, a child of the `LeaderAutoDown` downing actor, so `LeaderAutoDownBase` stays a line-for-line port) | Signal | Decision | |---|---| | S1 while the peer is still a live member in this node's view | **restart now**, no tiebreak. This node has not removed the peer, so the peer cannot be shut out by it. Only this side is. | | S1 after this node has removed the peer too | **tiebreak now** | | S2 | wait `ShutOutGrace` (60 s), because a frozen peer restarts itself on S1 in the meantime; if still split at the end, **tiebreak** | | Tiebreak | **higher address restarts** (Akka `Member.AddressOrdering`: host, then port). The lower node stays and restarts only if the peer has neither rejoined nor gone quiet one more grace later, which covers the single-sided case where the stale side is the lower address. | **Why the address tiebreak, not age.** Both nodes hold both addresses, in every signal and in their seed lists, so they compute the same answer without talking. Up-number is not safe: by the time a signal fires, each side has removed the other, so the peer's up-number exists only as remembered history. That history can differ after an unconverged join or a restart, and a disagreement means both nodes restart or neither does. The lower address is also the Akka leader of the re-formed pair and the bootstrap guard's preferred founder. **Restart** reuses the existing self-down path: `Cluster.Down(self)` → `run-coordinated-shutdown-when-down` → the `ActorSystem` terminates → the WhenTerminated watchdog runs `StopApplication`, with a direct `CoordinatedShutdown` fallback after 20 s. **Supervisor check.** - Docker `restart: unless-stopped` restarts on any exit code. - The wonder boxes run ScadaBridge as **native .NET Windows services, not NSSM**. NSSM wraps only the LDAP directory. Recovery is `sc.exe failure … restart/5000/restart/5000/restart/10000` plus `sc.exe failureflag 1`. - With `failureflag 1` the SCM restarts a service that stops on its own only if the reported exit code is **non-zero**. The watchdog used to exit **0**. I did not test this on a box, because no wonder-box remote execution is allowed. - **Fixed:** after an unplanned ActorSystem termination the Host now exits **75** (`AkkaHostedService.SelfDownExitCode`). `Program` sets it after the host stops, and the watchdog sets it on the Windows service (`ServiceBase.ExitCode`) before the service reports `SERVICE_STOPPED`. This also covers the existing SBR / auto-down self-down path. **Config:** `ScadaBridge:Cluster:LeaderAutoDown:RestartWhenShutOut` (default `true`, takes effect only with `leader-auto-down`) and `ShutOutGrace` (default `60s`, validated > 0). Logs use the `[leader-auto-down]` prefix: - `shut out by <peer> … tiebreak not needed — this node RESTARTS` - `split from <peer> … Giving it 00:01:00` - `Tiebreak says RESTART|STAY` - `rejoined (Up) within the shut-out grace; this node stays` ### Tests (all in Release) - `ShutOutRestartTests`: 18 unit tests (TestKit). They cover the single-side shut-out in both address orders, the tiebreak in both orders, the both-sides case with exactly one restart (S1 in both delivery orders, and S2 as a real healed partition produces it), the grace (stays if the peer goes quiet or rejoins, restarts if still split), duplicate and stranger events, probe back-off, and settings. - `ShutOutRestartScenarioTests`: 4 tests on real two-node clusters with the throttler transport: - the freeze shape in both address orders: the shut-out node terminates and the other stays Up; - a healed partition: only the higher port terminates, a fresh incarnation rejoins, and the survivor is still Up after its grace; - the same partition with the switch off: both stay up (positive control). - Also `UnexpectedTerminationTests` (exit flag, code ≠ 0), `HoconBuilderTests`, `ClusterOptionsValidatorTests`. - **Mutation-checked, all red:** - inverting the tiebreak: 7 unit tests plus the integration heal test; - removing the S1 handler: 11 unit tests plus both integration freeze tests; - removing the S2 handler: 4 unit tests; - removing the live-member fast path: 2 unit tests; - skipping the S2 grace: 3 unit tests; - removing the probe: 3 integration tests. - **Full solution:** every assembly passed except the following. - The 185 Playwright UI tests did not run, because there was no browser server. - `TemplateEngine ScriptCompilerTests.GetOrAdd_SameCodeDifferentSurface_ComputesSeparateVerdicts` failed once in the parallel run and passed 3/3 alone. TemplateEngine is untouched, so this is a shared-cache flake. - `CoordinatedShutdownTests.HostSource_DoesNotContainEnvironmentExit` caught an `Environment.ExitCode` mention in a comment; the comment was reworded, and Host.Tests is now 717/717. ### Rig gates (local docker rig, image built from the fix; record in `docs/plans/2026-11-12-leader-auto-down-rig-drill.md` §7, logs `sor-*.log`) | Gate | Central | Site-a | |---|---|---| | Freeze (`DRILL_ACTION=pause`, active) | **PASS.** Takeover 27 s. At unpause the victim logged `shut out … RESTARTS` within 1 s, exited 75 and rejoined as standby 34 s after unpause (30 s of that was the central host's own shutdown). 10/10 single-active. | **PASS.** Takeover 27 s; rejoined 6 s after unpause; 10/10 | | Split > window, then heal | **PASS.** Dual-active during the cut, as expected. Healed at +51 s; S2 on both sides 12 s later; tiebreak 60 s after that. **Only central-b (higher) restarted**; one 2-member cluster with central-a active at **+76 s after heal** | **PASS.** Only site-a-b restarted; one active node at **+111 s after heal** (30 s of that was its host shutdown) | | Kill regression (active) | **PASS, unchanged.** Down +26 s, Traefik 28 s, rejoin 2 s, no shut-out lines | **PASS, unchanged.** Down +28 s, rejoin 1 s | The rig is back on the default `auto-down`, all 8 nodes recreated on the fix image. Every node is ready, each pair has one active node and `memberCount 2 / unreachableCount 0`, and the active nodes are central-a, site-a-a, site-b-a and site-c-a. **Contract change (#116 gate 4).** The dual-active still happens *during* a partition longer than the window. After the heal it now clears itself in roughly one probe interval (up to 60 s), plus the 60 s grace, plus one restart. Before, it waited for an operator. `LeaderAutoDownScenarioTests.PartitionLongerThanWindow_EachSideDownsTheOther_BothStayUp` still holds, because that test never heals. **Residual risks (documented):** - If a frozen node takes longer than `ShutOutGrace` to shut down after S1, the survivor may also restart. That is a short double restart that still converges. - A probe of a dead peer logs one Akka association warning, backing off to one per minute. Docs updated: `Component-ClusterInfrastructure.md` (*Shut-out restart*, and the downed-node recovery exit code), `docs/components/ClusterInfrastructure.md`, the #106 plan (§7 addendum) and `CLAUDE.md`. The untracked wonder `deploy/…/RUNBOOK.md` was not changed. Closing: both rig gates pass. The rollout stays on #116.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
blocked
Blocked by dependency or external hold
 cross-repo
Needs coordinated change in another family repo (wire contract etc.)
 migrated
Migrated from a markdown backlog (2026-11 standardization)
 priority:p0
Must-have / blocking
 priority:p1
High priority
 priority:p2
Normal priority
 priority:p3
Low priority / nice to have
 type:bug
Defect in shipped behaviour
 type:debt
Technical debt / hygiene
 type:deploy-gate
Rollout/deploy/migration step not yet done on an environment
 type:docs
Documentation work
 type:feature
Feature implementation
 type:followup
Follow-up deferred from a plan, PR or gate
 type:infra
Infrastructure and scaffolding
 type:review-finding
Unfixed finding from an architecture/code review
 type:test
Test work
No Label priority:p2 type:followup
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
dohertj2 (dohertj2)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dohertj2/ScadaBridge#132
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?