fix(cluster): gossiped-role predicates compare case-insensitively — a miscased peer no longer un-elects the Primary

This commit is contained in:
Joseph Doherty
2026-08-20 10:04:16 -04:00
parent bcec88ef70
commit e46c1db5e5
12 changed files with 370 additions and 11 deletions
@@ -456,3 +456,49 @@ T11, T12 (rig config) before T13; T13 last, after everything is merge-ready
```
Overall effort: **~1 dev-day code+unit, plus the rig session.** The three HIGH-RISK slices (T1 identity type, T3 composition root, T8 the guard) each land behind their full project suites before anything stacks on them; T13 is merge-blocking and is only meaningful because T11/T12 gave the rig a node that can actually exhibit the defect.
---
## Execution notes (recorded during T1–T12 implementation, 2026-08-20)
**T5 — `SelectOldestUpMemberOfRole`/`SelectDriverPrimary` cannot be reached from this repo; the plan
was wrong about this one call site.** `RedundancyStateActor.SelectOldestUpMemberOfRole`
(`RedundancyStateActor.cs:172-178`) is a thin wrapper around `ClusterActiveNode.OldestUpMember`
(`ZB.MOM.WW.Health.Akka`, a separately-versioned NuGet package out of `scadaproj`), which itself
matches via **Akka's own `Member.HasRole`** — an ordinal comparison this repo does not own. This is
structurally identical to the "not ours to change" carve-out the plan's own choice 2 already makes
for `ClusterSingletonOptions.Role` (rows 18/19); the plan simply did not notice the same constraint
applies to `SelectOldestUpMemberOfRole` before naming it as a `GossipedRolesHave` adoption site.
Consequence: the plan's stated T5 test — "a `RedundancyStateActor` selection case where the member
gossips 'Driver' and a Primary IS elected" — is **not achievable**. What shipped instead
(`tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/GossipedRoleCaseInsensitivityTests.cs`):
- `GossipedRolesHave` was implemented and adopted at every site that genuinely reads gossiped roles
directly in this repo: `RedundancyStateActor.cs` (`BuildSnapshot`'s own role classification, NOT
the Primary-selection call), `ManualFailoverService.cs` (`OrderedDrivers`),
`ClusterNodeAddressReconcilerActor.cs` (`FilterOwnClusterDriverMembers`), `DriverHostActor.cs` and
`ScriptedAlarmHostActor.cs` (`DriverMemberCount`), and `Fleet.razor` (the bare `"driver"` literals,
now `RoleParser.Driver` — the actual case-insensitivity there comes from `ClusterRoleInfo`'s
`_roleLeaders`/`_membersByRole` dictionaries moving to `OrdinalIgnoreCase`, which
`IClusterRoleInfo.MembersWithRole`/`RoleLeader` read through).
- `A_member_gossiping_a_miscased_driver_role_classifies_Secondary_not_Detached` pins the fix that
**is** achievable: a single member gossiping `"Driver"` now classifies `Secondary` in the
redundancy snapshot (was `Detached` pre-fix) — real value, because `/hosts` and `/fleet` render
this list, even though that member can still never be *named* Primary by itself.
- `Primary_election_itself_remains_ordinal_and_still_elects_no_one_for_an_all_miscased_pair` pins the
**true** (negative) outcome instead of asserting a false positive: a node (or pair) gossiping only
`"Driver"` still elects no Primary, because that decision runs through the vendored ordinal match.
This is a genuine residual — **not closed by R3-04** — worth a follow-up issue against
`ZB.MOM.WW.Health.Akka`'s `ClusterActiveNode.OldestUpMember` (e.g. an optional case-insensitive
role-match parameter) if the rolling-upgrade window this defends against is judged worth closing
fully. In practice T1/T2 already close the *headline* outage the census describes for a
FIXED-build node (a node can no longer gossip a miscased plain role itself — normalization happens
before the gossip handoff), so this residual only bites a peer still running a pre-R3-04 build.
- `ManualFailoverService_recognises_a_miscased_driver_node_as_a_candidate` uses
`GetSnapshot()`/`PrimaryAddress` rather than `FailOverCore(dryRun: true)` as originally sketched —
`FailOverCore` refuses below 2 driver members ("nothing to fail over TO"), which a single-node
fixture can never satisfy; `GetSnapshot()` shares the same `OrderedDrivers` candidate selection
this test is actually about, without needing a second live node.
No other T1–T5 deviations from the plan.
@@ -35,7 +35,7 @@
{
"id": "T5",
"subject": "RoleParser.GossipedRolesHave adopted at the six ordinal gossip predicates + ClusterRoleInfo index (Primary election, R2-04 member count, /fleet)",
"status": "pending",
"status": "completed",
"blockedBy": [
"T1"
]