FleetStatusBroadcaster is blind to site nodes post-Phase-6 mesh split (fleet-wide gossip assumption) #542

New Issue

2026-08-06T04:33:10-04:00

dohertj2 commented

2026-08-06 04:33:10 -04:00

Summary

FleetStatusBroadcaster derives "fleet" status from central's own Akka gossip ring. Since the
per-cluster mesh split (Phase 6, 2026-07-24 — one independent 2-node Akka mesh per application
Cluster instead of one fleet-wide mesh), central no longer shares gossip with the site meshes
at all, so this component is silently blind to every site node: it can only ever see the central
pair it lives on.

Detail

This is pre-existing and transport-independent — it does not depend on MeshTransport:Mode
(Dps / ClusterClient / Grpc) and is not introduced or worsened by the ClusterClient→gRPC
migration. It was surfaced during that migration's design pass as a code-verified finding, and is
filed here rather than folded into the migration because fixing it is a different kind of work
(giving FleetStatusBroadcaster a cross-mesh data source) than swapping the command-plane
transport.

See docs/plans/2026-08-06-clusterclient-to-grpc-design.md §3.8 ("Deliberately out of scope")
for the code-verified writeup, and §4 finding 5 for the one-line cross-reference.

Suggested scope

Give FleetStatusBroadcaster a cross-mesh-aware source for site-node status — e.g. central's
existing ClusterNode DB rows (already the source of truth Phase 1 gives the deploy-ack set and
Phase 2 gives the ClusterClient/gRPC contact-point set) plus per-node health already flowing over
the Phase 5 telemetry stream, rather than reading the Akka gossip ring it happens to sit on.

## Summary `FleetStatusBroadcaster` derives "fleet" status from central's own Akka gossip ring. Since the per-cluster mesh split (Phase 6, 2026-07-24 — one independent 2-node Akka mesh per application `Cluster` instead of one fleet-wide mesh), central no longer shares gossip with the site meshes at all, so this component is silently blind to every site node: it can only ever see the central pair it lives on. ## Detail This is **pre-existing and transport-independent** — it does not depend on `MeshTransport:Mode` (`Dps` / `ClusterClient` / `Grpc`) and is not introduced or worsened by the ClusterClient→gRPC migration. It was surfaced during that migration's design pass as a code-verified finding, and is filed here rather than folded into the migration because fixing it is a different kind of work (giving `FleetStatusBroadcaster` a cross-mesh data source) than swapping the command-plane transport. See `docs/plans/2026-08-06-clusterclient-to-grpc-design.md` §3.8 ("Deliberately out of scope") for the code-verified writeup, and §4 finding 5 for the one-line cross-reference. ## Suggested scope Give `FleetStatusBroadcaster` a cross-mesh-aware source for site-node status — e.g. central's existing `ClusterNode` DB rows (already the source of truth Phase 1 gives the deploy-ack set and Phase 2 gives the ClusterClient/gRPC contact-point set) plus per-node health already flowing over the Phase 5 telemetry stream, rather than reading the Akka gossip ring it happens to sit on.

dohertj2 referenced this issue from a commit

2026-08-06 19:36:00 -04:00

refactor(fleet): #542 delete the dead fleet-status chain (broadcaster, SignalR bridge, hub, messages)

dohertj2 referenced this issue from a commit

2026-08-06 19:36:00 -04:00

refactor(fleet): #542 review residue — drop dead proxyOptions local, retire stale compliance defer line

dohertj2 referenced this issue from a commit

2026-08-06 19:36:00 -04:00

Merge fix/issue-542-fleetstatus: delete the dead fleet-status chain (#542)

dohertj2 commented

2026-08-06 19:36:59 -04:00

Fixed by deletion, merged to master @ d01b717e and pushed (branch fix/issue-542-fleetstatus, commits f1c18811 + b5607d35).

Why deletion rather than the suggested cross-mesh rewrite: scoping found the defect is wider than filed — the chain was dead end-to-end, not just blind:

FleetStatusBroadcaster published FleetStatusChanged on DPS topic fleet-status → FleetStatusSignalRBridge → FleetStatusHub (/hubs/fleet-status) — and no client of that hub exists anywhere (the AdminUI deliberately opens no self-HubConnections; /fleet reads NodeDeploymentState + IClusterRoleInfo on a 5 s poll and never referenced these types).
Its second input, DriverHostStatusHeartbeat, had no production sender (only its own test told it), so CurrentRevision was always null and the 30 s staleness→Degraded flip could never fire.
Post-Phase-6, the gossip blindness filed here made even the central-pair view accidental.

Giving a zero-consumer pipeline a cross-mesh data source would have activated inert machinery — the same call this repo made in #522, #507 and #525: when the premise is gone, delete. The real, cross-mesh-correct fleet visibility already exists: /fleet (deploy acks ride the gRPC mesh transport into NodeDeploymentState) and the Phase 5 per-node telemetry dialers feeding /hosts.

Deleted: broadcaster (+ DriverHostStatusHeartbeat), SignalR bridge, hub, FleetStatusChanged/FleetNodeStatus/FleetNodeHealth, registrations, tests. Kept: everything else under the Fleet namespaces (ClusterNodeAddressReconciler*, GetDiagnostics/IFleetDiagnosticsClient — live, unrelated). Docs swept (Telemetry.md, Configuration.md, CLAUDE.md; plans/v2 kept as history; the phase-6-1 compliance script's stale Assert-Deferred retired). Grep-gate: the only surviving mention in src/tests is the tombstone comment at the old registration site.

Review chain: spec ✅ + code ✅ (one residue find swept in b5607d35; the reviewer's "build-breaking CS0219" claim was disproven — the dead proxyOptions local predated this change on master and object-creation initializers don't raise CS0219, but the dead line was removed anyway). ControlPlane.Tests 231/0, AdminUI.Tests green, build warnings byte-identical to master baseline. Full suite on the merged SHA 9420f8ef: exit 0, 50/50 suites, 9,333 passed / 0 failed.

Fixed by **deletion**, merged to `master` @ `d01b717e` and pushed (branch `fix/issue-542-fleetstatus`, commits `f1c18811` + `b5607d35`). **Why deletion rather than the suggested cross-mesh rewrite:** scoping found the defect is wider than filed — the chain was dead end-to-end, not just blind: - `FleetStatusBroadcaster` published `FleetStatusChanged` on DPS topic `fleet-status` → `FleetStatusSignalRBridge` → `FleetStatusHub` (`/hubs/fleet-status`) — and **no client of that hub exists anywhere** (the AdminUI deliberately opens no self-`HubConnection`s; `/fleet` reads `NodeDeploymentState` + `IClusterRoleInfo` on a 5 s poll and never referenced these types). - Its second input, `DriverHostStatusHeartbeat`, had **no production sender** (only its own test told it), so `CurrentRevision` was always null and the 30 s staleness→Degraded flip could never fire. - Post-Phase-6, the gossip blindness filed here made even the central-pair view accidental. Giving a zero-consumer pipeline a cross-mesh data source would have activated inert machinery — the same call this repo made in #522, #507 and #525: when the premise is gone, delete. The real, cross-mesh-correct fleet visibility already exists: `/fleet` (deploy acks ride the gRPC mesh transport into `NodeDeploymentState`) and the Phase 5 per-node telemetry dialers feeding `/hosts`. Deleted: broadcaster (+ `DriverHostStatusHeartbeat`), SignalR bridge, hub, `FleetStatusChanged`/`FleetNodeStatus`/`FleetNodeHealth`, registrations, tests. Kept: everything else under the `Fleet` namespaces (`ClusterNodeAddressReconciler*`, `GetDiagnostics`/`IFleetDiagnosticsClient` — live, unrelated). Docs swept (Telemetry.md, Configuration.md, CLAUDE.md; plans/v2 kept as history; the phase-6-1 compliance script's stale `Assert-Deferred` retired). Grep-gate: the only surviving mention in src/tests is the tombstone comment at the old registration site. Review chain: spec ✅ + code ✅ (one residue find swept in `b5607d35`; the reviewer's "build-breaking CS0219" claim was disproven — the dead `proxyOptions` local predated this change on master and object-creation initializers don't raise CS0219, but the dead line was removed anyway). ControlPlane.Tests 231/0, AdminUI.Tests green, build warnings byte-identical to master baseline. Full suite on the merged SHA `9420f8ef`: exit 0, 50/50 suites, 9,333 passed / 0 failed.

dohertj2 closed this issue

2026-08-06 19:37:04 -04:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dohertj2/lmxopcua#542