The dashboard derived every leader chip from AkkaClusterHealthCheck's `leader`
data — ClusterState.Leader, the LOWEST-ADDRESSED Up member. That is not the node
in charge. The node holding the singletons, gated for writes, and answering 200
on /health/active is the OLDEST Up member. The two agree only until some node
restarts, after which the restarted node rejoins youngest but keeps its address.
Observed live on the rebuilt OtOpcUa rig during the Health 0.3.0 work: leader
central-1, active node central-2. The dashboard would have put the Active chip on
central-1 while /health/active said central-2 — the chip contradicting the tier
it exists to visualise. The Phase 4 acceptance run missed it because the rig had
just been formed, the one state in which the two orderings agree.
Health 0.3.0's active tier now publishes `activeNode` on every member, including
standbys, so the fix is to prefer it and keep `leader` as the fallback for an
instance on an older Health or whose active tier was not probed.
The instance card now shows `active` and `leader` side by side rather than
swapping one for the other: they are legitimately different nodes, and showing
both turns a silent contradiction into something readable at 3am.
Tests: 177 (+3) — precedence when the two disagree, a standby naming the active
node rather than itself, and the fallback still working with no activeNode.
Found by the live rig, and invisible to every unit test that existed.
LeaderResolver and InstanceCard.ClusterDataLine both looked the cluster view up
under the check name "akka-cluster". ScadaBridge registers the shared
AkkaClusterHealthCheck under that name; OtOpcUa registers the SAME check as
"akka". So once the OtOpcUa rig was rebuilt on Health 0.2.0 and started
publishing data.leader, its groups still rendered with no leader chip and no
evidence line - and a group with no votes is a legitimate state, so nothing
looked wrong. ScadaBridge's four groups were fully populated the whole time,
which is what made it look like the feature worked.
Both now find the entry by the DATA KEY ("leader"), which is the part of the
contract the shared check actually owns rather than the part each consumer names
for itself.
Regression tests: the leader is found under akka-cluster, akka, and an arbitrary
future name; and an entry that publishes data WITHOUT a leader (ScadaBridge's
localdb check, whose data carries replication counters) does not get picked in
preference to the one that answers the question. Every prior fixture used
ScadaBridge's name, which is precisely why this got through.
After the fix all seven cluster groups render a leader chip with zero
split-brain flags - design section 9 check 2 now passes for BOTH products.
174 tests, 0 warnings.
Tasks 3.4-3.6 of the overview-dashboard plan.
Polling (3.4/3.5):
- OverviewPollerService: one timer at the fastest configured cadence, each
target polled when its own interval is due, fan-out per sweep, no retries
(the next tick is the retry, so damping stays the only place that decides
whether a failure is worth showing).
- OverviewSnapshotStore: atomic swap of a complete immutable graph plus a
change event. Pages read it and never probe — the cache-first requirement.
- LeaderResolver: per-group leader from akka-cluster data, with the
split-brain flag scoped to members that are actually answering, so an
ordinary failover is not misread as a disagreement.
UI (3.6): ThemeShell + StatusPill/TechCard composition against the mockup;
instance cards carry the status stripe (dashed for Unreachable), the check
detail list and the raw ready/active signal line.
Two defects found by verification rather than by review:
- The active tier answers with a status code and an EMPTY body, but the client
demanded parseable JSON on every probe — so a healthy pair rendered as
"role unknown" instead of Active/Standby. Split into ProbeAsync (ready tier,
body is the payload) and ProbeStatusAsync (active tier, code is the answer).
Caught by a live smoke run against two fake health endpoints; the poller
tests now serve an empty body so they hold the fix.
- SweepAsync published even when cancellation was already requested, leaving a
half-probed registry as the store's final state on shutdown. It now checks
the token itself rather than relying on the transport to throw.
Also: app.UseAntiforgery() is required despite the anonymous-by-design
pipeline — AddRazorComponents stamps antiforgery metadata unconditionally and
the endpoint middleware hard-fails without it (every page was 500). Chosen
over DisableAntiforgery() so a future form is protected by default.
147 tests, 0 warnings. Live-verified end to end against fake endpoints:
Up/Unreachable/Active/Standby, leader chip, cluster-data line, KPI counts.
Joseph Doherty