#542 filed FleetStatusBroadcaster as blind to site nodes since the Phase 6 per-cluster mesh
split: it derives "the fleet" from central's own Akka gossip ring, which after the split is
just central's own pair. Scoping the cross-mesh rewrite found the deeper truth — the whole
chain is dead end to end, so this deletes it rather than reviving it.
- Zero consumers. FleetStatusBroadcaster published FleetStatusChanged on the DPS
"fleet-status" topic; FleetStatusSignalRBridge forwarded it to FleetStatusHub
("/hubs/fleet-status"); nothing anywhere connected. The AdminUI bans self-connecting
HubConnections, no browser JS subscribes, and /fleet (Fleet.razor) reads
NodeDeploymentState from the ConfigDb plus IClusterRoleInfo on a 5 s poll instead.
- Its second input never existed. DriverHostStatusHeartbeat had no production sender —
only the broadcaster's own test told it — so CurrentRevision was always null and the
30 s staleness -> Degraded flip could never fire.
- Post-Phase-6 DPS is pair-local, so even the central-only view it did produce was an
accident of topology rather than a design.
The fleet visibility that actually works is unchanged and already cross-mesh-correct:
/fleet over NodeDeploymentState, and the four Phase 5 per-node telemetry dials.
Precedent for deleting inert machinery rather than activating it: #522 (driver stability
tiers), #507 (discovered-node injection), #525 (GetMemoryFootprint).
Deleted: FleetStatusBroadcaster (+ its nested DriverHostStatusHeartbeat), its admin
singleton registration and FleetStatusBroadcasterKey, FleetStatusSignalRBridge and its
key, FleetStatusHub and its route mapping, and the FleetStatusChanged / FleetNodeStatus /
FleetNodeHealth messages. Both Fleet/ directories survive — ControlPlane keeps
ClusterNodeAddressReconciler, Commons keeps GetDiagnostics. Docs swept where they claimed
a fleet-status channel exists (Telemetry.md's Phase 5 non-migrated channels,
Configuration.md, CLAUDE.md); completed-program plan docs under docs/plans and docs/v2 are
left as the historical record they are.
Build 0 errors, 868 warnings — byte-identical to master's rebuild baseline.
ControlPlane.Tests 231/231, AdminUI.Tests 1109/1109.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Phase 6 wired the re-homed redundancy singleton in Program.cs as
`WithOtOpcUaClusterRedundancySingleton(sp.GetRequiredService<IClusterRoleInfo>())`
inside the AddAkka configurator lambda. That lambda runs WHILE the ActorSystem is
being built, but ClusterRoleInfo depends on the ActorSystem (it is a live
Cluster.State view) — so resolving it there recurses into ActorSystem construction
and every node dies at boot with StackOverflowException. It compiles cleanly (a
runtime DI cycle) and RedundancyStateSingletonRehomeTests passed a hand-built
FakeClusterRoleInfo straight into the extension, mocking around the composition-root
line that overflows. The docker-dev live gate caught it on first boot.
Derive the singleton's cluster-role scope from AkkaClusterOptions (pure config, no
ActorSystem) instead: BuildClusterRedundancySingletonOptions / the extension now take
AkkaClusterOptions and compute Role = Roles.FirstOrDefault(IsClusterRole) ?? driver —
identical to ClusterRoleInfo.ClusterRole, which derives from the same config. Program.cs
passes IOptions<AkkaClusterOptions>.Value, which has no ActorSystem dependency.
Live-verified: rebuilt image boots with zero stack-overflow lines, cluster singletons
form per mesh. 5/5 rehome tests pass; Host builds clean.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Review follow-ups on the redundancy re-home (Phase 6, Task 2):
- Program.cs: the hasDriver comment claimed IClusterRoleInfo "throws at host
start if this driver node carries no cluster role" — false since the fallback
landed. Rewrite it to describe the cluster-{ClusterId} scope with the driver-role
fallback for legacy single-mesh / not-yet-migrated nodes.
- ClusterRoleInfo.cs: the SubscriberActor comment described RedundancyStateActor as
the "admin-role singleton" — stale. Note it is now a cluster-{ClusterId}-scoped
singleton spawned on every driver node (LeaderChanged stays a no-op here).
- Add a host/registry-level test for the driver-role FALLBACK path (roles ["driver"],
no cluster role) closing the coverage asymmetry — it was proven only in the pure
helper. Asserts the node boots and registers RedundancyStateActorKey, i.e. the boot
the earlier throw-based version would have aborted.
The boot helper now supplies an in-memory IDbContextFactory and a fake IClusterRoleInfo:
Akka.Hosting invokes singleton props factories eagerly at StartAsync, and the admin
ClusterNodeAddressReconciler factory reads both (the latter for its per-cluster reconcile
scope, added by a sibling Phase 6 task) — without them the admin boot NREs.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 6 splits the single fleet-wide Akka mesh into one 2-node mesh per
application Cluster. RedundancyStateActor was an admin-scoped cluster singleton
that elected ONE fleet-wide driver Primary — but after the split a driver-only
site pair has no admin node in its mesh to host it, so its Primary would never
be elected.
Re-home it: remove the redundancy WithSingleton block from the admin
WithOtOpcUaControlPlaneSingletons set, and add WithOtOpcUaClusterRedundancySingleton,
a singleton spawned from Program.cs's hasDriver branch on EVERY driver node (the
fused central included). It scopes to the node's own cluster-{ClusterId} role when
present, and falls back to the fixed `driver` role otherwise. Each mesh then has
exactly one, electing its own pair-local Primary and publishing redundancy-state
on its own mesh's DistributedPubSub. The election logic (oldest Up driver member)
and the DPS publish are unchanged — they become pair-local after the split.
The driver-role fallback (Decision 2) deliberately does NOT throw when a node has
no cluster role: that is the pre-Phase-6 fleet-wide behavior on a legacy single
mesh, so legacy/harness (TwoNodeClusterHarness: admin,driver) and not-yet-migrated
deployments keep booting unchanged. On a genuinely split 2-node mesh the `driver`
role is already pair-local (the mesh IS the pair), so the fallback is correct in
both worlds. Cluster-scope, when present, additionally survives an accidental
two-mesh merge by keeping one singleton per cluster.
The role scope + driver-role fallback are pinned by unit tests against the
extracted BuildClusterRedundancySingletonOptions helper (the BuildDowningHocon
pattern); admin-removal + driver-registration are pinned behaviorally against a
booted node's ActorRegistry.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Joseph Doherty