Compare commits

...

11 Commits

Author SHA1 Message Date
Joseph Doherty 9420f8efff Merge fix/issue-546-panel: wire DriverStatusPanel into /hosts driver rows (#546)
v2-ci / build (push) Successful in 5m22s
v2-ci / unit-tests (push) Failing after 17m32s
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
2026-08-06 17:28:48 -04:00
Joseph Doherty d01b717eb9 Merge fix/issue-542-fleetstatus: delete the dead fleet-status chain (#542)
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
2026-08-06 17:28:48 -04:00
Joseph Doherty 1df8470356 Merge fix/issues-543-544-545: whitespace-key fail-closed backport (#543), de-flaked deadline test (#544), h2c accepted-posture record (#545)
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
2026-08-06 17:28:48 -04:00
Joseph Doherty 4870785209 docs(auth): #543 add the one-public-ctor remarks paragraph to the two newly-pinned interceptors
ConfigServeAuthInterceptor and LocalDbSyncAuthInterceptor gained HasExactlyOnePublicConstructor
pin tests in this branch but lacked the sibling-standard <remarks> paragraph naming them
(TelemetryStream/MeshCommand/CentralCommand all carry it). Cosmetic consistency item from the
#543 code review.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
2026-08-06 17:28:31 -04:00
Joseph Doherty b5607d355a refactor(fleet): #542 review residue — drop dead proxyOptions local, retire stale compliance defer line
The proxyOptions local in WithOtOpcUaControlPlaneSingletons was already unused on master
(pre-existing dead code, not orphaned by the deletion commit — every registration passes
singletonOptions); removed while the file is open. The phase-6-1 compliance script still
Assert-Deferred'd the FleetStatusHub /hosts follow-up — an item that is now deleted, not
deferred — replaced with a retirement comment.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
2026-08-06 17:28:11 -04:00
Joseph Doherty e669fb349a fix(adminui): #546 recycle DriverStatusPanel on Enabled flip via @key
DriverStatusPanel decides whether to subscribe exactly once, in
OnInitializedAsync (`if (!Enabled) return;`), and exposes no OnParametersSet
hook. The detail row was keyed only by DriverInstanceId, so a live config edit
that flipped a row's Enabled true->false — surfaced on the page by the 5 s
LoadConfigAsync refresh — reused the same panel instance: the markup correctly
switched to "Disabled - not deployed" while the panel's two store SnapshotChanged
subscriptions and its 5 s staleness timer stayed wired underneath, until the
operator happened to collapse the row.

Including the flag in the detail row's key recycles the component instead. Both
directions verified against the panel's own lifecycle: true->false disposes the
old instance (unsubscribing both stores, draining both timers) and mounts one
that skips subscription setup entirely; false->true disposes an instance that
never subscribed (the unsubscribes are no-ops, both timers null) and mounts one
that subscribes and primes from the stores.

The data row's key is unchanged. No behaviour change when Enabled is stable.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
2026-08-06 17:25:41 -04:00
Joseph Doherty f1c188114b refactor(fleet): #542 delete the dead fleet-status chain (broadcaster, SignalR bridge, hub, messages)
#542 filed FleetStatusBroadcaster as blind to site nodes since the Phase 6 per-cluster mesh
split: it derives "the fleet" from central's own Akka gossip ring, which after the split is
just central's own pair. Scoping the cross-mesh rewrite found the deeper truth — the whole
chain is dead end to end, so this deletes it rather than reviving it.

  - Zero consumers. FleetStatusBroadcaster published FleetStatusChanged on the DPS
    "fleet-status" topic; FleetStatusSignalRBridge forwarded it to FleetStatusHub
    ("/hubs/fleet-status"); nothing anywhere connected. The AdminUI bans self-connecting
    HubConnections, no browser JS subscribes, and /fleet (Fleet.razor) reads
    NodeDeploymentState from the ConfigDb plus IClusterRoleInfo on a 5 s poll instead.
  - Its second input never existed. DriverHostStatusHeartbeat had no production sender —
    only the broadcaster's own test told it — so CurrentRevision was always null and the
    30 s staleness -> Degraded flip could never fire.
  - Post-Phase-6 DPS is pair-local, so even the central-only view it did produce was an
    accident of topology rather than a design.

The fleet visibility that actually works is unchanged and already cross-mesh-correct:
/fleet over NodeDeploymentState, and the four Phase 5 per-node telemetry dials.

Precedent for deleting inert machinery rather than activating it: #522 (driver stability
tiers), #507 (discovered-node injection), #525 (GetMemoryFootprint).

Deleted: FleetStatusBroadcaster (+ its nested DriverHostStatusHeartbeat), its admin
singleton registration and FleetStatusBroadcasterKey, FleetStatusSignalRBridge and its
key, FleetStatusHub and its route mapping, and the FleetStatusChanged / FleetNodeStatus /
FleetNodeHealth messages. Both Fleet/ directories survive — ControlPlane keeps
ClusterNodeAddressReconciler, Commons keeps GetDiagnostics. Docs swept where they claimed
a fleet-status channel exists (Telemetry.md's Phase 5 non-migrated channels,
Configuration.md, CLAUDE.md); completed-program plan docs under docs/plans and docs/v2 are
left as the historical record they are.

Build 0 errors, 868 warnings — byte-identical to master's rebuild baseline.
ControlPlane.Tests 231/231, AdminUI.Tests 1109/1109.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
2026-08-06 17:21:30 -04:00
Joseph Doherty d33e4e20b5 feat(adminui): #546 wire DriverStatusPanel into /hosts driver rows (collapsed expander)
DriverStatusPanel was orphaned — zero references repo-wide — so the Restart and
Reconnect driver commands had NO reachable UI at all, despite working end to end
over the new gRPC mesh transport. Nothing failed to build, test or deploy while
that was true, which is why it went unnoticed.

The panel is now reachable per driver row on /hosts, behind a per-row "Controls"
expander that is collapsed by default: the table stays scannable, and only the
opened panels hold snapshot-store subscriptions. Expansion state is a HashSet
keyed by DriverInstanceId held on the page, so it survives the 5 s refresh that
replaces every row record. Both rows are @key'd by instance id so the panel —
which binds its DriverInstanceId once in OnInitializedAsync — can never be diffed
onto a different driver.

The panel's Enabled parameter is fed from real config rather than defaulted:
HostsDriverInstanceInfo/HostsDriverRow now carry DriverInstance.Enabled, which
LoadConfigAsync already had in hand. A snapshot with no matching configured
instance is treated as enabled — the row exists only because live health arrived,
so "Disabled — not deployed" would contradict the same row.

The expander is visible to every AuthenticatedRead viewer; the panel self-gates
only its Reconnect/Restart buttons on DriverOperator.

A source-scan pin (HostsPanelWiringTests) asserts /hosts still renders the panel
and still binds it to the row's own driver, cluster and enabled state — removing
the tag would silently restore the orphaned state.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
2026-08-06 17:18:37 -04:00
Joseph Doherty 17a822b7dc docs(security): #545 record the accepted h2c posture for the five internal gRPC planes
OtOpcUa runs five internal gRPC planes between its own processes, all of
them cleartext HTTP/2 (h2c) with a shared-bearer interceptor: the live
telemetry stream, config-serve artifact fetch, LocalDb pair sync, mesh
commands and mesh deploy acks. The bearer therefore crosses the wire in
the clear on every one of them. That was true and undocumented; the two
mesh planes are command surfaces, so it is worth saying out loud rather
than leaving to be rediscovered per plane.

docs/security.md gains "Internal gRPC planes (h2c) — accepted cleartext
posture" under Transport Security (whose existing content is entirely
about the OPC UA endpoint). It carries a five-row table — service name,
listener config key, credential config key, interceptor — the honest
read-vs-write breakdown per plane, and the decision itself.

Decision (issue #545, option 1): risk ACCEPTED 2026-08-06, once, for all
five planes together, on the trusted-link assumption every plane already
makes (VPN / dedicated WAN). Per-cluster PSKs already bound blast radius
on the two command planes — a captured SITE-A key cannot drive SITE-B —
and production keys arrive as ${secret:...} references that must never be
reused across clusters. Recorded future work: a UseTls/scheme option,
which if built must cover all five planes together, since a mixed posture
is harder to reason about than either uniform answer.

docs/Configuration.md's MeshTransport section gains one cross-reference
paragraph pointing at the new subsection.

Docs only — no code, no behaviour change. Every service name, config key,
interceptor name and shared-listener claim was verified against the
source; ports 4056/4055 are phrased as the docker-dev rig's values of
Telemetry:GrpcListenPort / ConfigServe:GrpcListenPort, not as hardcoded
ports.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
2026-08-06 17:17:51 -04:00
Joseph Doherty 722a222db5 test(controlplane): #544 de-flake deadline-timeout test — seed acks before spawn, un-hittable wall deadline
DeadlineElapsed_for_current_deployment_marks_TimedOut carried two races,
both test-side. ConfigPublishCoordinator itself is untouched.

(a) The NodeDeploymentState row was inserted AFTER Sys.ActorOf. The
coordinator's PreStart reads NodeDeploymentStates to recover the
expected-ack set, and it runs as soon as the actor is scheduled — so if
PreStart won, the expected-ack set was empty and the coordinator could
short-circuit to Sealed instead of ever reaching TimedOut. The row is now
seeded before the spawn.

(b) The applyDeadline was 150 ms of real wall clock, and it arms a real
Akka timer. Under load that timer could fire on its own and race the
test's explicit DeadlineElapsed tell — so the test was intermittently
exercising the timer rather than the message handler it names. The
deadline is now TimeSpan.FromMinutes(5), matching the other two tests in
this class, which makes it un-hittable inside the test's own 3 s
AwaitAssert bound and leaves the explicit tell as the only deadline
signal the coordinator ever sees. That tell, and the AwaitAssert bound,
are unchanged.

Verified: the single test 5/5 green in a row, then the whole class 3/3.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
2026-08-06 17:15:42 -04:00
Joseph Doherty f10f4c3754 fix(auth): #543 whitespace-only configured key fails closed in the three pre-existing gRPC auth interceptors
TelemetryStreamAuthInterceptor, ConfigServeAuthInterceptor and
LocalDbSyncAuthInterceptor all decided "is a key configured?" with
string.IsNullOrEmpty. A whitespace-only configured value therefore passed
that guard and fell through to the constant-time comparison — where it is
satisfied by a whitespace-only bearer token. The gate would be open to
anyone who guessed a blank credential.

This backports the check the two NEW mesh interceptors already use:
MeshCommandAuthInterceptor (and its central-side clone) validate with
string.IsNullOrWhiteSpace, so the four/five planes now answer the same way
to the same misconfiguration. Nothing else changes — the denial status
codes (PermissionDenied / Unauthenticated / PermissionDenied respectively)
and every denial message are untouched, and no key material is logged.

No deployment configures a whitespace key: the docker-dev rig and
production both set real keys, so this is a fail-closed hardening of an
unreachable-today misconfiguration rather than a live exposure.

Tests: each of the three test files gains a ..._WithBlankKeyConfigured_IsDenied
fact mirroring MeshCommandAuthInterceptorTests.ExecuteMethod_WithBlankPskConfigured_IsDenied
(configured key "   ", presented "Bearer    "), verified red before the fix.
ConfigServeAuthInterceptor and LocalDbSyncAuthInterceptor also gain the
HasExactlyOnePublicConstructor pin their two siblings already had —
Grpc.AspNetCore silently stops invoking an interceptor with more than one
public constructor, which would disable auth without failing anything.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
2026-08-06 17:14:36 -04:00
28 changed files with 360 additions and 377 deletions
+6 -4
View File
@@ -509,11 +509,13 @@ touching it:
- **Deferred, with reasons (do not read "seven observability topics" from the program sketch as the
delivered scope):** `redundancy-state` stays on DPS in every mode (pair-local, built from
`Cluster.State`, drives ServiceLevel + the Primary gate — genuinely mesh-bound by design, not an
observability panel); `fleet-status` is central-internal (`FleetStatusBroadcaster` builds it from
the admin node's own membership events, `Fleet.razor` polls the Config DB and ignores the feed —
never a node→central stream); `deployment-acks` already rides the Phase 2 gRPC command transport
observability panel); `deployment-acks` already rides the Phase 2 gRPC command transport
(`MeshTransport:Mode=Grpc`, `CentralCommandService.ReportApply`) as a command-plane reply, not
telemetry.
telemetry. A third, `fleet-status`, was **DELETED** rather than deferred (2026-08-06, Gitea #542):
`FleetStatusBroadcaster` → DPS → SignalR bridge → `FleetStatusHub` had **zero clients end to end**
(no browser JS, no `HubConnection` — `/fleet` polls `NodeDeploymentState` in the Config DB), its
`DriverHostStatusHeartbeat` input had no production sender, and Phase 6 left its "fleet" as just
central's own pair. Fleet visibility is `/fleet` + the four per-node telemetry dials above.
- **The node ALWAYS hosts and ALWAYS double-publishes, in both modes.** Every publish seam emits
into a node-local `ITelemetryLocalHub` **and** still publishes DPS, unconditionally; the gRPC
server binds whenever `Telemetry:GrpcListenPort > 0`, independent of `Telemetry:Mode`. Only
+7 -4
View File
@@ -184,6 +184,8 @@ The checked-in `appsettings*.json` files are deliberately thin: they carry only
**Both command-plane gRPC services ride an existing listener — neither claims a port of its own.** `NodeCommandService` is mapped alongside `TelemetryStreamGrpcService` on the driver-role node's `Telemetry:GrpcListenPort` listener (the docker-dev rig's `:4056`) whenever that port is bound, deliberately NOT gated on `MeshTransport:Mode` — mapping it under `Dps` costs nothing (nothing dials it), whereas gating it would mean a running fleet needs a central restart before any node could ack after flipping the mode. `CentralCommandService` (the node→central ack ingress) is mapped alongside `DeploymentArtifactService` on the admin-role node's `ConfigServe:GrpcListenPort` listener (the rig's `:4055`), for the same reason — see [`docs/Telemetry.md`](Telemetry.md) for the shared driver-role listener and [§ `ConfigSource`/`ConfigServe`](#configsource--configserve-config-fetch-and-cache) for the shared admin-role one.
**The transport underneath is cleartext h2c, deliberately.** Both command-plane listeners are HTTP/2-only on a plaintext port and every dial is a prior-knowledge `http://host:port` address, so the PSK bearer crosses the wire in the clear on a link assumed private. That posture is **recorded and accepted** for these two planes and the three others that share their shape — see [`security.md` § Internal gRPC planes (h2c)](security.md#internal-grpc-planes-h2c--accepted-cleartext-posture) for the full five-plane table, what each one exposes, and why a TLS option would have to cover all five at once.
On the docker-dev rig every node carries these keys already; the rig's compose default is `Grpc` (`OTOPCUA_MESH_MODE:-Grpc`, cutover commit `caf2eec6`) — the compiled class default in `MeshTransportOptions` stays `Dps`, so an unconfigured deployment outside the rig still needs `Mode=Grpc` set explicitly. Every rig node carries a `cluster-{ClusterId}` role (Phase 6), so `OTOPCUA_MESH_MODE=Dps` is not a usable override on this rig as shipped — `SplitTopologyTransportValidator` refuses to boot any of them on `Dps`.
See the [ClusterClient→gRPC migration design](plans/2026-08-06-clusterclient-to-grpc-design.md), [implementation plan](plans/2026-08-06-clusterclient-to-grpc.md), [tracking doc](plans/2026-08-06-clusterclient-to-grpc-tracking.md) and [live gate](plans/2026-08-06-clusterclient-to-grpc-live-gate.md) for the migration's phase-by-phase history — those are historical records of the cutover and are not updated further by this sweep.
@@ -273,11 +275,12 @@ discipline as `MeshTransport:Mode` and `ConfigSource:Mode`.
otherwise) — enforced by `TelemetryOptionsValidator`. **`Grpc` mode on either side requires a
non-empty `ApiKey`** — fail-closed, same as `ConfigServe`/`LocalDb:Replication`.
**Three DPS observability topics stay out of scope for this transport, on purpose:**
**Two DPS observability topics stay out of scope for this transport, on purpose:**
`redundancy-state` (pair-local, built from `Cluster.State`, stays on DPS in both modes — see
[`Redundancy.md`](Redundancy.md#command-transport-centralnode)), `fleet-status` (central-internal;
`Fleet.razor` polls the Config DB, not any stream), and `deployment-acks` (already rides the Phase 2
gRPC command transport, `CentralCommandService.ReportApply`, as a command-plane reply). See
[`Redundancy.md`](Redundancy.md#command-transport-centralnode)) and `deployment-acks` (already rides
the Phase 2 gRPC command transport, `CentralCommandService.ReportApply`, as a command-plane reply).
A third, `fleet-status`, was **deleted** rather than deferred (2026-08-06, Gitea #542 — it had no
client end to end; `Fleet.razor` polls the Config DB). See
[`docs/Telemetry.md`](Telemetry.md) for the full architecture, the reconnect/generation-stamped
dialer story, and the rationale for the node-hosts/central-dials inversion.
+13 -8
View File
@@ -78,7 +78,6 @@ What actually changes per mode, precisely:
`IDriverStatusSnapshotStore`, `IDriverResilienceStatusStore`).
- **The AdminUI panels themselves are untouched.** `/alerts`, `/script-log`, and the `/hosts`
driver table all read from the same sinks in both modes — only the sinks' upstream feed swaps.
The `fleet-status` bridge (out of Phase 5 scope — see below) stays on DPS in both modes.
Because hosting is unconditional and both ingest paths are always compiled in, a fleet can be
flipped node-by-node and central-by-central with no coordination window where telemetry is lost —
@@ -130,10 +129,10 @@ Three things to know before touching it:
and no reader — the only reference in the repo is its DbSet declaration — while the live data reaches
the AdminUI over the `driver-resilience-status` channel above. Keep-or-delete is Gitea **#524**.
## The three deferred channels (NOT migrated in Phase 5 — do not read this as "seven done")
## The three non-migrated channels (NOT migrated in Phase 5 — do not read this as "seven done")
The program sketch originally named seven observability topics for Phase 5. Three were scoped out,
each for a distinct, settled reason:
each for a distinct, settled reason — and one of those three has since been deleted outright:
- **`redundancy-state`** — bidirectional, built directly from `Cluster.State`, and **pair-local**:
it drives ServiceLevel and the Primary data-plane gate, consumed in-process by
@@ -143,11 +142,17 @@ each for a distinct, settled reason:
in-mesh. Central's *display* of each pair's redundancy state is a Phase 6 cross-mesh concern
(a future observability channel, once central no longer shares gossip with any site pair), not a
Phase 5 telemetry-panel migration.
- **`fleet-status`** — **central-internal**, not a node→central stream at all. The admin singleton
`FleetStatusBroadcaster` builds it from the admin node's own cluster membership/reachability/
leader events, and `Fleet.razor` **polls the Config DB** and ignores the feed entirely. There is
nothing here for a per-node gRPC stream to carry. Revisit in Phase 6, once central genuinely
loses gossip visibility of site nodes and needs another way to know a pair's membership.
- **`fleet-status`** — **DELETED, 2026-08-06, Gitea #542.** There is no `fleet-status` channel any
more, on DPS or anywhere else. It was central-internal rather than a node→central stream: the
admin singleton `FleetStatusBroadcaster` built it from the admin node's own cluster membership/
reachability/leader events and published to a SignalR hub that **no client anywhere ever
connected to** — `Fleet.razor` polls `NodeDeploymentState` in the Config DB and always ignored
the feed. Its second input, `DriverHostStatusHeartbeat`, had no production sender either. Phase 6
then made the central-only view wrong as well (central's gossip ring is its own pair). The whole
chain — broadcaster, singleton registration, DPS topic, SignalR bridge, `FleetStatusHub`, and the
`FleetStatusChanged`/`FleetNodeStatus` messages — was removed rather than rewritten cross-mesh.
Fleet visibility that does work today: `/fleet` over `NodeDeploymentState`, and the four Phase 5
telemetry channels above, which are per-node dials and therefore already mesh-independent.
- **`deployment-acks`** — already migrated, but onto a different transport: it rides the Phase 2
gRPC command transport (`MeshTransport:Mode=Grpc`, `CentralCommandService.ReportApply`) as a
command-plane reply, not an observability broadcast. It was never a Phase 5 candidate.
+57
View File
@@ -104,6 +104,63 @@ The Admin UI `Certificates.razor` page (`src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Co
- Promote trusted client certs with the Admin UI Certificates page's **Trust** action (FleetAdmin) — or by moving the `.der` from `rejected/` to `trusted/certs/` — rather than relying on the auto-accept fallback.
- Periodically audit the `rejected/` directory; an unexpected entry is often a misconfigured client or a probe attempt.
### Internal gRPC planes (h2c) — accepted cleartext posture
Everything above concerns the **OPC UA** endpoint — the surface external clients bind to. OtOpcUa
also runs **five internal gRPC planes** between its own processes (central ↔ driver node, and node ↔
pair partner). Those are a separate transport story, recorded here so it is written down once rather
than rediscovered per plane.
| # | Plane | gRPC service | Listener (config key) | Credential (config key) | Interceptor |
|---|---|---|---|---|---|
| 1 | Live telemetry stream (central dials the node) | `telemetry.v1.TelemetryStreamService/Subscribe` | driver node, `Telemetry:GrpcListenPort` | `Telemetry:ApiKey` (dialer presents `TelemetryDial:ApiKey`) | `TelemetryStreamAuthInterceptor` |
| 2 | Config-serve / artifact fetch (node fetches from central) | `deployment_artifact.v1.DeploymentArtifactService/Fetch` | central (admin role), `ConfigServe:GrpcListenPort` | `ConfigServe:ApiKey` (fetcher presents `ConfigSource:ApiKey`) | `ConfigServeAuthInterceptor` |
| 3 | LocalDb pair sync (one node dials its partner) | `localdb_sync.v1.LocalDbSync/Sync` | driver node, `LocalDb:SyncListenPort` | `LocalDb:Replication:ApiKey` (byte-identical on both nodes of the pair) | `LocalDbSyncAuthInterceptor` |
| 4 | Mesh commands (central drives the node) | `mesh_transport.v1.NodeCommandService/Execute` | **shares** the node's `Telemetry:GrpcListenPort` listener | `MeshTransport:GrpcPsk` — **per application `Cluster`**, not fleet-wide | `MeshCommandAuthInterceptor` |
| 5 | Mesh deploy acks (node reports to central) | `mesh_transport.v1.CentralCommandService/ReportApply` | **shares** central's `ConfigServe:GrpcListenPort` listener | per-cluster key resolved by `IMeshPskProvider` — `MeshTransport:ClusterPsks:<clusterId>` override, else the `OT-MESH-PSK-<clusterId>` secret | `CentralCommandAuthInterceptor` |
Planes 4 and 5 claim no port of their own; they ride the listener their sibling already binds. On the
docker-dev rig those two config keys resolve to `:4056` (driver nodes) and `:4055` (central) — rig
values of the keys above, **not** hardcoded ports.
**The posture.** All five listeners are bound `HttpProtocols.Http2` on a **cleartext** port, and every
client dials a prior-knowledge `http://host:port` address — h2c, no TLS, no ALPN. (Cleartext is
structural, not incidental: a cleartext `Http1AndHttp2` endpoint cannot negotiate HTTP/2 without ALPN,
which is why these are dedicated HTTP/2-only ports rather than a multiplex onto the main one.)
Authentication on all five is a **shared bearer token in an `Authorization` header**, compared in
constant time and fail-closed when unconfigured. Because the transport is cleartext, **the bearer
itself crosses the wire in the clear** — anyone who can observe or interpose on the link can read a
key and then replay it.
What that buys an attacker is not uniform:
- Planes 1–2 are **read** surfaces: the live telemetry feed, and the full deployed configuration.
- Plane 3 **writes** rows into a node's deployment-artifact cache — which is what that node boots
from when central SQL is unreachable.
- Planes 4–5 are the **command** surfaces, and are the sharpest: plane 4 carries `RestartDriver`,
`AlarmCommand` and deployment dispatch; plane 5 carries the apply acks the deploy coordinator
counts, so forged acks can seal a deployment nodes never applied.
**Decision: risk accepted, 2026-08-06** (Gitea [`lmxopcua#545`](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/545),
option 1). Accepted once, for all five planes together, on the **trusted-link assumption every one of
them already makes**: the central↔site and pair-local links are private (VPN or dedicated WAN), not
routed over anything an untrusted party can observe. Two things bound the consequences if that
assumption is wrong:
- **Per-cluster PSKs bound blast radius on the two command planes.** A captured `SITE-A` key drives
`SITE-A` only; it is refused by `SITE-B`'s node and by central when it claims to be `SITE-B`.
A single fleet-wide key would have made one observed link equal to fleet-wide command authority.
- **Production keys are delivered as `${secret:...}` references**, never committed literals, and
**MUST NOT be reused across clusters** — reusing one key collapses the isolation above back into a
fleet-wide key with extra steps. The three literal `dev-mesh-psk-*` keys exist only on the
docker-dev rig.
**Future work.** The recorded alternative is a `UseTls`/scheme option on these planes (dial `https://`,
bind the listener with a certificate). If it is ever built it **must cover all five planes together**
— a mixed posture, where some internal links are encrypted and others are not, is harder to reason
about than either uniform answer, and invites exactly the "which one was it again?" mistake this note
exists to prevent. See issue #545.
---
## OPC UA Authentication
+2 -1
View File
@@ -109,7 +109,8 @@ Write-Host ""
Write-Host "Stream E - Admin /hosts (data layer)"
Assert-FileExists "DriverInstanceResilienceStatus entity" "src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Entities/DriverInstanceResilienceStatus.cs"
Assert-FileExists "DriverResilienceStatusTracker present" "src/Core/ZB.MOM.WW.OtOpcUa.Core/Resilience/DriverResilienceStatusTracker.cs"
Assert-Deferred "FleetStatusHub SignalR push + Blazor /hosts column refresh" "Phase 6.1 Stream E.2/E.3 visual-compliance follow-up"
# Gitea #542 (2026-08-06): the FleetStatusHub chain was DELETED (dead end-to-end), so the
# Stream E.2/E.3 "SignalR push + /hosts column refresh" follow-up is retired, not deferred.
Write-Host ""
Write-Host "Cross-cutting"
@@ -13,7 +13,7 @@ public enum StartDeploymentOutcome
/// <summary>
/// Reply from the <c>AdminOperationsActor</c> singleton. <c>Accepted</c> means the snapshot
/// was sealed and a <c>Deployment</c> row was created; the in-flight deployment can be
/// tracked through fleet-status broadcasts.
/// tracked through the per-node <c>NodeDeploymentState</c> rows the AdminUI's /fleet page reads.
/// </summary>
public sealed record StartDeploymentResult(
StartDeploymentOutcome Outcome,
@@ -1,21 +0,0 @@
using ZB.MOM.WW.OtOpcUa.Commons.Types;
namespace ZB.MOM.WW.OtOpcUa.Commons.Messages.Fleet;
public enum FleetNodeHealth { Healthy, Degraded, Unreachable }
public sealed record FleetNodeStatus(
NodeId NodeId,
FleetNodeHealth Health,
RevisionHash? CurrentRevision,
DateTime LastSeenUtc);
/// <summary>
/// Periodic fleet-wide status broadcast pushed by <c>FleetStatusBroadcaster</c> to admin UI
/// subscribers via SignalR.
/// </summary>
public sealed record FleetStatusChanged(
IReadOnlyList<FleetNodeStatus> Nodes,
DeploymentId? CurrentDeployment,
DateTime AsOfUtc,
CorrelationId CorrelationId);
@@ -11,6 +11,7 @@
@using Microsoft.EntityFrameworkCore
@using Microsoft.Extensions.Logging
@using ZB.MOM.WW.OtOpcUa.Configuration
@using ZB.MOM.WW.OtOpcUa.AdminUI.Components.Shared.Drivers
@using ZB.MOM.WW.OtOpcUa.AdminUI.Hosts
@using ZB.MOM.WW.OtOpcUa.AdminUI.Hubs
@using ZB.MOM.WW.OtOpcUa.Commons.Messages.Drivers
@@ -173,20 +174,25 @@ else
<th>Last read</th>
<th>Errors/5 min</th>
<th>Last error</th>
<th></th>
</tr>
</thead>
<tbody>
@if (g.Drivers.Count == 0)
{
<tr>
<td colspan="8"><span class="text-muted">No drivers.</span></td>
<td colspan="9"><span class="text-muted">No drivers.</span></td>
</tr>
}
else
{
@foreach (var d in g.Drivers)
{
<tr>
@* Keyed by instance id so the detail row's DriverStatusPanel is never
diffed onto a DIFFERENT driver: the panel binds DriverInstanceId
once in OnInitializedAsync (store subscription + resilience prime), so
a reused instance would keep reporting the driver it first bound. *@
<tr @key="@($"row-{d.DriverInstanceId}")">
<td>
@(d.Name ?? d.DriverInstanceId)
@if (d.Name is not null)
@@ -258,7 +264,42 @@ else
<td>@(d.LastSuccessfulReadUtc?.ToString("HH:mm:ss 'UTC'") ?? "—")</td>
<td class="numeric">@d.ErrorCount5Min</td>
<td><span class="text-muted small">@(d.LastError ?? "—")</span></td>
<td>
@* Per-row expander for the driver's status panel (Gitea #546). Shown to
every AuthenticatedRead viewer — the panel carries read-only detail
(resilience/breaker state, last error) as well as the operator actions,
and self-gates only the Reconnect/Restart buttons on DriverOperator. *@
<button type="button"
class="btn btn-sm btn-outline-secondary"
aria-expanded="@(IsExpanded(d.DriverInstanceId) ? "true" : "false")"
@onclick="() => ToggleExpanded(d.DriverInstanceId)"
title="Show live status, resilience and driver controls">
@(IsExpanded(d.DriverInstanceId) ? "▾" : "▸") Controls
</button>
</td>
</tr>
@if (IsExpanded(d.DriverInstanceId))
{
@* Full-width detail row beneath its driver. Rendered only while expanded so
the table stays scannable and only the opened panels hold store
subscriptions — collapsing disposes the panel and unsubscribes.
Enabled participates in the key on purpose. The panel decides whether to
subscribe ONCE, in OnInitializedAsync (`if (!Enabled) return;`), and has no
OnParametersSet hook — so a live config edit that flips this row's Enabled
(surfaced by the 5 s LoadConfigAsync refresh) would re-render the panel's
"Disabled — not deployed" markup while its two store subscriptions and its
staleness timer stayed wired underneath. Keying on the flag recycles the
component instead: the old instance is disposed (unsubscribing both stores,
draining both timers) and a fresh one mounts against the new value. *@
<tr @key="@($"detail-{d.DriverInstanceId}-{d.Enabled}")">
<td colspan="9">
<DriverStatusPanel DriverInstanceId="@d.DriverInstanceId"
ClusterId="@g.ClusterId"
Enabled="@d.Enabled" />
</td>
</tr>
}
}
}
</tbody>
@@ -281,6 +322,11 @@ else
private List<HostsNodeInfo> _nodes = new();
private List<HostsDriverInstanceInfo> _instances = new();
// Driver instances whose DriverStatusPanel is expanded (Gitea #546). Held on the component, NOT
// on the row records, so it survives the 5 s timer's RebuildDriverGroups — which replaces every
// HostsDriverRow instance and would otherwise collapse a panel an operator just opened.
private readonly HashSet<string> _expandedDrivers = new(StringComparer.OrdinalIgnoreCase);
protected override async Task OnInitializedAsync()
{
Refresh();
@@ -336,7 +382,7 @@ else
.Select(n => new HostsNodeInfo(n.ClusterId, n.NodeId, n.Host, n.OpcUaPort))
.ToListAsync();
_instances = await db.DriverInstances.AsNoTracking()
.Select(d => new HostsDriverInstanceInfo(d.DriverInstanceId, d.ClusterId, d.Name, d.DriverType))
.Select(d => new HostsDriverInstanceInfo(d.DriverInstanceId, d.ClusterId, d.Name, d.DriverType, d.Enabled))
.ToListAsync();
}
catch (Exception ex)
@@ -351,6 +397,17 @@ else
private void RebuildDriverGroups()
=> _driverGroups = HostsDriverView.Build(DriverStore.GetAll(), _nodes, _instances);
// Whether this driver's DriverStatusPanel is currently expanded.
private bool IsExpanded(string driverInstanceId) => _expandedDrivers.Contains(driverInstanceId);
// Open/close one driver's panel. Ids are globally unique, so no cluster qualifier is needed; a
// removed driver simply leaves a stale id in the set, which matches nothing and costs nothing.
private void ToggleExpanded(string driverInstanceId)
{
if (!_expandedDrivers.Remove(driverInstanceId))
_expandedDrivers.Add(driverInstanceId);
}
// Raised by the snapshot store on the bridge actor's thread for every driver instance. Rebuild
// from the cached config (cheap — no DB hit) and marshal onto the render sync context. A
// brand-new driver shows by id until the next config reload (timer/manual Refresh) enriches it.
@@ -395,7 +452,9 @@ else
_ => "chip-idle",
};
// Map DriverState string → chip CSS class (mirrors DriverStatusPanel.ChipClass).
// Map DriverState string → chip CSS class (mirrors DriverStatusPanel.ChipClass). Since #546 that
// panel renders inside this page's expanded rows, so the two mappings are visible side by side —
// they must stay in step or one driver shows two different colours for one state.
private static string DriverChipClass(string? state) => state switch
{
"Healthy" => "chip-ok",
@@ -22,7 +22,11 @@ public sealed record HostsNodeInfo(string ClusterId, string NodeId, string Host,
/// <param name="ClusterId">Cluster the instance is configured under.</param>
/// <param name="Name">Authored display name.</param>
/// <param name="DriverType">Driver type string (Modbus, S7, Galaxy, ...).</param>
public sealed record HostsDriverInstanceInfo(string DriverInstanceId, string ClusterId, string Name, string DriverType);
/// <param name="Enabled">Whether the instance is authored enabled — forwarded to the row's
/// <c>DriverStatusPanel</c>, which renders a "Disabled — not deployed" notice and opens no
/// subscription when false.</param>
public sealed record HostsDriverInstanceInfo(
string DriverInstanceId, string ClusterId, string Name, string DriverType, bool Enabled = true);
/// <summary>
/// One row in the <c>/hosts</c> "Driver Instances" table: a live health snapshot enriched with
@@ -48,12 +52,17 @@ public sealed record HostsDriverInstanceInfo(string DriverInstanceId, string Clu
/// is no answer — the driver cannot report skips, or it faulted before building a tag table. Lets
/// "ignoring 20 of 22 tags" show even while every other column on the row — state, last read, error
/// count, last error — reads perfectly healthy (Gitea #541).</param>
/// <param name="Enabled">Whether the matching configured instance is authored enabled; <see langword="true"/>
/// when no instance matched. Fed to the row's expandable <c>DriverStatusPanel</c> (Gitea #546). An
/// unmatched row defaults to enabled on purpose: the row only exists because a live snapshot arrived,
/// so telling the operator it is "not deployed" would contradict the evidence in front of them.</param>
public sealed record HostsDriverRow(
string DriverInstanceId, string? Name, string? DriverType, string State,
DateTime? LastSuccessfulReadUtc, string? LastError, int ErrorCount5Min, DateTime PublishedUtc,
DateTime? RediscoveryNeededUtc = null, string? RediscoveryReason = null,
IReadOnlyList<HostConnectivityStatus>? HostStatuses = null,
SkippedTagReport? SkippedTags = null)
SkippedTagReport? SkippedTags = null,
bool Enabled = true)
{
/// <summary>Whether this driver is ignoring at least one authored tag — the condition the whole
/// column exists to make visible.</summary>
@@ -114,7 +123,7 @@ public static class HostsDriverView
/// seen across <paramref name="snapshots"/> and <paramref name="nodes"/>, ordered
/// case-insensitively. Each group carries its nodes (ordered by node id) and its driver rows
/// (snapshots enriched by joining <paramref name="instances"/> on driver instance id;
/// unmatched ids yield null name/type), ordered by name-then-instance-id.
/// unmatched ids yield null name/type and an enabled row), ordered by name-then-instance-id.
/// </summary>
/// <param name="snapshots">Live health snapshots; null treated as empty.</param>
/// <param name="nodes">Configured cluster nodes; null treated as empty.</param>
@@ -167,7 +176,8 @@ public static class HostsDriverView
s.RediscoveryNeededUtc,
s.RediscoveryReason,
s.HostStatuses,
s.SkippedTags);
s.SkippedTags,
instance?.Enabled ?? true);
})
.OrderBy(d => d.Name ?? d.DriverInstanceId, StringComparer.OrdinalIgnoreCase)
.ThenBy(d => d.DriverInstanceId, StringComparer.OrdinalIgnoreCase)
@@ -9,8 +9,7 @@ namespace ZB.MOM.WW.OtOpcUa.AdminUI.Hubs;
/// <summary>
/// Akka actor that subscribes to the <c>alerts</c> DistributedPubSub topic and forwards each
/// <see cref="AlarmTransitionEvent"/> to every SignalR client connected to <see cref="AlertHub"/>.
/// Mirrors <c>FleetStatusSignalRBridge</c>'s design — one bridge per admin node, hub fan-out is
/// per-node, no cluster-singleton needed.
/// One bridge per admin node — hub fan-out is per-node, so no cluster-singleton is needed.
/// </summary>
public sealed class AlertSignalRBridge : ReceiveActor
{
@@ -1,19 +0,0 @@
using Microsoft.AspNetCore.Authorization;
using Microsoft.AspNetCore.SignalR;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Fleet;
namespace ZB.MOM.WW.OtOpcUa.AdminUI.Hubs;
/// <summary>
/// Browser-facing fleet-status push channel. Subscribers receive <see cref="FleetStatusChanged"/>
/// snapshots whenever the admin-role <c>FleetStatusBroadcaster</c> publishes a diff.
///
/// Server pushes fleet-status updates to connected clients via <c>FleetStatusSignalRBridge</c>
/// (DistributedPubSub 'fleet-status' → <c>IHubContext&lt;FleetStatusHub&gt;</c>).
/// </summary>
[Authorize]
public sealed class FleetStatusHub : Hub
{
public const string Endpoint = "/hubs/fleet-status";
public const string MethodName = "fleetStatusChanged";
}
@@ -1,58 +0,0 @@
using Akka.Actor;
using Akka.Cluster.Tools.PublishSubscribe;
using Akka.Event;
using Microsoft.AspNetCore.SignalR;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Fleet;
namespace ZB.MOM.WW.OtOpcUa.AdminUI.Hubs;
/// <summary>
/// Akka actor that subscribes to the <c>fleet-status</c> DistributedPubSub topic and forwards
/// every <see cref="FleetStatusChanged"/> snapshot to all SignalR clients connected to
/// <see cref="FleetStatusHub"/>. Spawned on admin-role nodes by
/// <c>AddOtOpcUaSignalRBridges</c>.
///
/// The bridge runs locally on each admin node — every node that hosts the hub also forwards
/// snapshots to its own connected clients. That keeps the hub-to-actor wiring simple (no
/// cluster-singleton coordination needed) at the cost of duplicated DPS subscriptions on
/// multi-admin deployments. Acceptable since the messages are small + the SignalR fan-out
/// is per-node anyway.
/// </summary>
public sealed class FleetStatusSignalRBridge : ReceiveActor
{
public const string TopicName = "fleet-status";
private readonly IHubContext<FleetStatusHub> _hub;
private readonly ILoggingAdapter _log = Context.GetLogger();
/// <summary>Creates actor props for a FleetStatusSignalRBridge with the specified hub context.</summary>
/// <param name="hub">The SignalR hub context.</param>
/// <returns>Props for creating a new FleetStatusSignalRBridge actor.</returns>
public static Props Props(IHubContext<FleetStatusHub> hub) =>
Akka.Actor.Props.Create(() => new FleetStatusSignalRBridge(hub));
/// <summary>Initializes a new instance of the FleetStatusSignalRBridge class.</summary>
/// <param name="hub">The SignalR hub context for sending messages to clients.</param>
public FleetStatusSignalRBridge(IHubContext<FleetStatusHub> hub)
{
_hub = hub;
ReceiveAsync<FleetStatusChanged>(ForwardAsync);
Receive<SubscribeAck>(_ => { /* DPS confirmation */ });
}
/// <inheritdoc />
protected override void PreStart() =>
DistributedPubSub.Get(Context.System).Mediator.Tell(new Subscribe(TopicName, Self));
private async Task ForwardAsync(FleetStatusChanged msg)
{
try
{
await _hub.Clients.All.SendAsync(FleetStatusHub.MethodName, msg);
}
catch (Exception ex)
{
_log.Warning(ex, "FleetStatusSignalRBridge: SignalR push failed (count={Count})", msg.Nodes.Count);
}
}
}
@@ -13,7 +13,6 @@ public static class HubRouteBuilderExtensions
/// <returns>The same endpoint route builder, for chaining.</returns>
public static IEndpointRouteBuilder MapOtOpcUaHubs(this IEndpointRouteBuilder app)
{
app.MapHub<FleetStatusHub>(FleetStatusHub.Endpoint);
app.MapHub<AlertHub>(AlertHub.Endpoint);
app.MapHub<ScriptLogHub>(ScriptLogHub.Endpoint);
app.MapHub<DriverStatusHub>(DriverStatusHub.Endpoint);
@@ -14,7 +14,6 @@ namespace ZB.MOM.WW.OtOpcUa.AdminUI.Hubs;
public static class HubServiceCollectionExtensions
{
public const string FleetStatusSignalRBridgeName = "fleet-status-signalr-bridge";
public const string AlertSignalRBridgeName = "alert-signalr-bridge";
public const string ScriptLogSignalRBridgeName = "script-log-signalr-bridge";
public const string DriverStatusSignalRBridgeName = "driver-status-signalr-bridge";
@@ -44,8 +43,8 @@ public static class HubServiceCollectionExtensions
/// <summary>
/// Spawns the SignalR bridge actors that forward DPS messages to browser-facing SignalR
/// hubs: <c>fleet-status</c> → <see cref="FleetStatusHub"/>, <c>alerts</c> →
/// <see cref="AlertHub"/>, <c>script-logs</c> → <see cref="ScriptLogHub"/>.
/// hubs: <c>alerts</c> → <see cref="AlertHub"/>, <c>script-logs</c> →
/// <see cref="ScriptLogHub"/>, <c>driver-health</c> → <see cref="DriverStatusHub"/>.
///
/// Call inside the admin-role configurator on the shared <see cref="AkkaConfigurationBuilder"/>:
/// <code>
@@ -62,12 +61,6 @@ public static class HubServiceCollectionExtensions
{
builder.WithActors((system, registry, resolver) =>
{
// Fleet-status always stays on DPS in both modes (deferred / out of Phase 5 scope) — it is
// never gated by the telemetry dark switch.
var fleetHub = resolver.GetService<IHubContext<FleetStatusHub>>();
var fleetBridge = system.ActorOf(FleetStatusSignalRBridge.Props(fleetHub), FleetStatusSignalRBridgeName);
registry.Register<FleetStatusSignalRBridgeKey>(fleetBridge);
// The four telemetry sinks are registered identically in both modes; only the upstream that
// feeds them swaps (DPS bridges vs. the gRPC dial supervisor). Resolve them once, above the
// branch, so both paths feed the SAME singletons the Blazor panels read.
@@ -131,7 +124,6 @@ public static class HubServiceCollectionExtensions
}
/// <summary>Marker keys for <see cref="ActorRegistry"/> lookup of the SignalR bridge actors.</summary>
public sealed class FleetStatusSignalRBridgeKey { }
public sealed class AlertSignalRBridgeKey { }
public sealed class ScriptLogSignalRBridgeKey { }
public sealed class DriverStatusSignalRBridgeKey { }
@@ -1,168 +0,0 @@
using Akka.Actor;
using Akka.Cluster;
using Akka.Cluster.Tools.PublishSubscribe;
using Akka.Event;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Fleet;
using ZB.MOM.WW.OtOpcUa.Commons.Types;
namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Fleet;
/// <summary>
/// Admin-role cluster singleton that maintains an in-memory <see cref="FleetStatusChanged"/>
/// snapshot from cluster membership + reachability events plus per-node heartbeats, and pushes
/// diffs on the <c>fleet-status</c> DistributedPubSub topic. The SignalR hub layer
/// subscribes to that topic and forwards to browser clients.
///
/// Heartbeat staleness: a node we haven't heard from in <see cref="HeartbeatTimeout"/> flips
/// from <see cref="FleetNodeHealth.Healthy"/> to <see cref="FleetNodeHealth.Degraded"/>. A
/// cluster <c>Unreachable</c> event flips it straight to <see cref="FleetNodeHealth.Unreachable"/>.
/// </summary>
public sealed class FleetStatusBroadcaster : ReceiveActor, IWithTimers
{
public const string Topic = "fleet-status";
public static readonly TimeSpan BroadcastInterval = TimeSpan.FromSeconds(5);
public static readonly TimeSpan HeartbeatTimeout = TimeSpan.FromSeconds(30);
private readonly Akka.Cluster.Cluster _cluster;
private readonly Action<object> _broadcast;
private readonly ILoggingAdapter _log = Context.GetLogger();
private readonly Dictionary<NodeId, NodeRecord> _nodes = new();
/// <summary>Gets or sets the timer scheduler for this actor.</summary>
public ITimerScheduler Timers { get; set; } = null!;
/// <summary>
/// Per-node heartbeat carrying the node's currently-applied revision hash, sent by
/// <c>DriverHostActor</c> periodically.
/// </summary>
public sealed record DriverHostStatusHeartbeat(NodeId NodeId, RevisionHash? CurrentRevision);
/// <summary>Creates actor props for the fleet status broadcaster.</summary>
/// <param name="broadcast">Optional custom broadcast delegate; defaults to distributed pub-sub mediator.</param>
/// <returns>Props configured to create a FleetStatusBroadcaster.</returns>
public static Props Props(Action<object>? broadcast = null) =>
Akka.Actor.Props.Create(() => new FleetStatusBroadcaster(broadcast));
/// <summary>Initializes a new instance of the FleetStatusBroadcaster actor.</summary>
/// <param name="broadcast">Optional custom broadcast delegate; if null, uses distributed pub-sub mediator.</param>
public FleetStatusBroadcaster(Action<object>? broadcast = null)
{
_cluster = Akka.Cluster.Cluster.Get(Context.System);
_broadcast = broadcast ?? (msg =>
DistributedPubSub.Get(Context.System).Mediator.Tell(new Publish(Topic, msg)));
Receive<ClusterEvent.MemberUp>(e => OnMemberUp(e.Member));
Receive<ClusterEvent.MemberRemoved>(e => OnMemberRemoved(e.Member));
Receive<ClusterEvent.UnreachableMember>(e => OnUnreachable(e.Member));
Receive<ClusterEvent.ReachableMember>(e => OnReachable(e.Member));
Receive<ClusterEvent.LeaderChanged>(_ => PublishSnapshot());
Receive<ClusterEvent.RoleLeaderChanged>(_ => PublishSnapshot());
Receive<DriverHostStatusHeartbeat>(OnHeartbeat);
Receive<Tick>(_ => OnTick());
}
/// <inheritdoc />
protected override void PreStart()
{
_cluster.Subscribe(
Self,
ClusterEvent.InitialStateAsEvents,
typeof(ClusterEvent.IMemberEvent),
typeof(ClusterEvent.LeaderChanged),
typeof(ClusterEvent.RoleLeaderChanged),
typeof(ClusterEvent.ReachabilityEvent));
Timers.StartPeriodicTimer("tick", Tick.Instance, BroadcastInterval);
}
/// <inheritdoc />
protected override void PostStop() => _cluster.Unsubscribe(Self);
private void OnMemberUp(Member m)
{
if (!TryNode(m, out var nodeId)) return;
_nodes[nodeId] = _nodes.TryGetValue(nodeId, out var prev)
? prev with { Health = FleetNodeHealth.Healthy, LastSeenUtc = DateTime.UtcNow }
: new NodeRecord(nodeId, FleetNodeHealth.Healthy, CurrentRevision: null, DateTime.UtcNow);
PublishSnapshot();
}
private void OnMemberRemoved(Member m)
{
if (!TryNode(m, out var nodeId)) return;
_nodes.Remove(nodeId);
PublishSnapshot();
}
private void OnUnreachable(Member m)
{
if (!TryNode(m, out var nodeId) || !_nodes.TryGetValue(nodeId, out var rec)) return;
_nodes[nodeId] = rec with { Health = FleetNodeHealth.Unreachable };
PublishSnapshot();
}
private void OnReachable(Member m)
{
if (!TryNode(m, out var nodeId) || !_nodes.TryGetValue(nodeId, out var rec)) return;
_nodes[nodeId] = rec with { Health = FleetNodeHealth.Healthy, LastSeenUtc = DateTime.UtcNow };
PublishSnapshot();
}
private void OnHeartbeat(DriverHostStatusHeartbeat hb)
{
var rec = _nodes.TryGetValue(hb.NodeId, out var prev)
? prev with
{
CurrentRevision = hb.CurrentRevision,
LastSeenUtc = DateTime.UtcNow,
Health = prev.Health == FleetNodeHealth.Unreachable ? prev.Health : FleetNodeHealth.Healthy,
}
: new NodeRecord(hb.NodeId, FleetNodeHealth.Healthy, hb.CurrentRevision, DateTime.UtcNow);
_nodes[hb.NodeId] = rec;
}
/// <summary>
/// Periodic tick — flips heartbeat-stale nodes to <see cref="FleetNodeHealth.Degraded"/> and
/// rebroadcasts the snapshot regardless so SignalR subscribers always have a recent baseline.
/// </summary>
private void OnTick()
{
var stale = DateTime.UtcNow - HeartbeatTimeout;
var flippedCount = 0;
foreach (var kv in _nodes.ToList())
{
if (kv.Value.LastSeenUtc < stale && kv.Value.Health == FleetNodeHealth.Healthy)
{
_nodes[kv.Key] = kv.Value with { Health = FleetNodeHealth.Degraded };
flippedCount++;
}
}
if (flippedCount > 0) _log.Debug("FleetStatusBroadcaster flipped {Count} nodes to Degraded", flippedCount);
PublishSnapshot();
}
private void PublishSnapshot()
{
var statuses = _nodes.Values
.Select(r => new FleetNodeStatus(r.NodeId, r.Health, r.CurrentRevision, r.LastSeenUtc))
.ToList();
var msg = new FleetStatusChanged(statuses, CurrentDeployment: null, DateTime.UtcNow, CorrelationId.NewId());
_broadcast(msg);
}
private static bool TryNode(Member m, out NodeId nodeId)
{
nodeId = default;
var host = m.Address.Host;
if (string.IsNullOrWhiteSpace(host)) return false;
nodeId = NodeId.Parse(host);
return true;
}
private sealed record NodeRecord(NodeId NodeId, FleetNodeHealth Health, RevisionHash? CurrentRevision, DateTime LastSeenUtc);
public sealed class Tick
{
public static readonly Tick Instance = new();
private Tick() { }
}
}
@@ -26,12 +26,11 @@ public static class ServiceCollectionExtensions
public const string ConfigPublishSingletonName = "config-publish";
public const string AdminOperationsSingletonName = "admin-operations";
public const string AuditWriterSingletonName = "audit-writer";
public const string FleetStatusSingletonName = "fleet-status";
public const string RedundancyStateSingletonName = "redundancy-state";
public const string ClusterNodeAddressReconcilerSingletonName = "cluster-node-address-reconciler";
/// <summary>
/// Registers the admin-role cluster singletons on the AkkaConfigurationBuilder — five with
/// Registers the admin-role cluster singletons on the AkkaConfigurationBuilder — three with
/// registry proxies, plus the address reconciler, which is proxy-less because nothing addresses it.
/// Must be called against the same builder used by <c>AkkaHostedService</c> so the singletons
/// share the host's ActorSystem.
@@ -51,7 +50,6 @@ public static class ServiceCollectionExtensions
public static AkkaConfigurationBuilder WithOtOpcUaControlPlaneSingletons(this AkkaConfigurationBuilder builder)
{
var singletonOptions = new ClusterSingletonOptions { Role = AdminRole };
var proxyOptions = new ClusterSingletonOptions { Role = AdminRole };
// Per-cluster mesh Phase 2: the central end of the central↔node command boundary.
//
@@ -139,10 +137,11 @@ public static class ServiceCollectionExtensions
},
singletonOptions);
builder.WithSingleton<FleetStatusBroadcasterKey>(
FleetStatusSingletonName,
(system, registry, resolver) => FleetStatusBroadcaster.Props(),
singletonOptions);
// Gitea #542 (2026-08-06): the fleet-status singleton is NO LONGER registered here — the
// whole FleetStatusBroadcaster → DPS → SignalR bridge → FleetStatusHub chain was DELETED.
// It had zero clients end-to-end (no browser JS, no HubConnection — /fleet reads
// NodeDeploymentState from the ConfigDb on a poll), its DriverHostStatusHeartbeat input had
// no production sender, and after mesh Phase 6 its "fleet" was only central's own pair.
// Per-cluster mesh Phase 6: the redundancy-state singleton is NO LONGER registered here.
// It was an admin-scoped, fleet-wide singleton — one Primary elected across the whole mesh.
@@ -253,7 +252,6 @@ public static class ServiceCollectionExtensions
public sealed class ConfigPublishCoordinatorKey { }
public sealed class AdminOperationsActorKey { }
public sealed class AuditWriterActorKey { }
public sealed class FleetStatusBroadcasterKey { }
public sealed class RedundancyStateActorKey { }
public sealed class ClusterNodeAddressReconcilerKey { }
public sealed class CentralCommunicationActorKey { }
@@ -33,6 +33,11 @@ namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
/// wrong key cannot be recovered byte-by-byte from response timing. Length differences are
/// unavoidably observable and are not sensitive.
/// </para>
/// <para>
/// <b>Exactly one public constructor.</b> <c>Grpc.AspNetCore</c> silently stops invoking an
/// interceptor that has more than one public ctor — see
/// <c>ConfigServeAuthInterceptorTests.HasExactlyOnePublicConstructor</c>.
/// </para>
/// </remarks>
public sealed class ConfigServeAuthInterceptor : Interceptor
{
@@ -110,7 +115,9 @@ public sealed class ConfigServeAuthInterceptor : Interceptor
return;
var expected = _options.Value.ApiKey;
if (string.IsNullOrEmpty(expected))
// Whitespace-only counts as unconfigured: an all-blank expected key would otherwise be
// satisfied by an all-blank bearer token, which is no gate at all.
if (string.IsNullOrWhiteSpace(expected))
{
_logger.LogWarning(
"Rejected a config-serve call to {Method}: no ConfigServe:ApiKey is configured, so the " +
@@ -38,6 +38,11 @@ namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
/// a wrong key cannot be recovered byte-by-byte from response timing. Length differences are
/// unavoidably observable and are not sensitive.
/// </para>
/// <para>
/// <b>Exactly one public constructor.</b> <c>Grpc.AspNetCore</c> silently stops invoking an
/// interceptor that has more than one public ctor — see
/// <c>LocalDbSyncAuthInterceptorTests.HasExactlyOnePublicConstructor</c>.
/// </para>
/// </remarks>
public sealed class LocalDbSyncAuthInterceptor : Interceptor
{
@@ -115,7 +120,9 @@ public sealed class LocalDbSyncAuthInterceptor : Interceptor
return;
var expected = _options.Value.ApiKey;
if (string.IsNullOrEmpty(expected))
// Whitespace-only counts as unconfigured: an all-blank expected key would otherwise be
// satisfied by an all-blank bearer token, which is no gate at all.
if (string.IsNullOrWhiteSpace(expected))
{
_logger.LogWarning(
"Rejected a LocalDb sync call to {Method}: no LocalDb:Replication:ApiKey is configured, " +
@@ -117,7 +117,9 @@ public sealed class TelemetryStreamAuthInterceptor : Interceptor
return;
var expected = _options.Value.ApiKey;
if (string.IsNullOrEmpty(expected))
// Whitespace-only counts as unconfigured: an all-blank expected key would otherwise be
// satisfied by an all-blank bearer token, which is no gate at all.
if (string.IsNullOrWhiteSpace(expected))
{
_logger.LogWarning(
"Rejected a telemetry stream call to {Method}: no Telemetry:ApiKey is configured, so " +
@@ -22,8 +22,9 @@ public sealed class HostsDriverViewTests
private static HostsNodeInfo Node(string cluster, string node, string host = "h", int port = 4840) =>
new(cluster, node, host, port);
private static HostsDriverInstanceInfo Instance(string instance, string cluster, string name, string type) =>
new(instance, cluster, name, type);
private static HostsDriverInstanceInfo Instance(
string instance, string cluster, string name, string type, bool enabled = true) =>
new(instance, cluster, name, type, enabled);
[Fact]
public void Build_groups_two_clusters_with_their_nodes_and_snapshots()
@@ -214,6 +215,38 @@ public sealed class HostsDriverViewTests
row.DegradedHosts.ShouldBeEmpty();
}
// ---- authored enabled-state (Gitea #546) ----
/// <summary>
/// The authored enabled flag reaches the row, which forwards it to the row's
/// <c>DriverStatusPanel</c>. Defaulting it to true would have the panel open a live
/// subscription for an instance that was deliberately taken out of the deployment.
/// </summary>
[Fact]
public void Build_carries_the_authored_enabled_flag_onto_the_row()
{
var groups = HostsDriverView.Build(
[Snap("MAIN", "drv-a")],
nodes: null,
[Instance("drv-a", "MAIN", "Pump House Modbus", "Modbus", enabled: false)]);
groups.Single().Drivers.Single().Enabled.ShouldBeFalse();
}
/// <summary>
/// A snapshot with no matching configured instance is treated as enabled. The row only exists
/// because live health arrived for it, so rendering the panel's "Disabled — not deployed"
/// notice would contradict the evidence on the same row.
/// </summary>
[Fact]
public void An_unmatched_driver_row_is_treated_as_enabled()
{
var row = HostsDriverView.Build([Snap("MAIN", "orphan")], nodes: null, instances: null)
.Single().Drivers.Single();
row.Enabled.ShouldBeTrue();
}
// ---- skipped tags (Gitea #541) ----
/// <summary>
@@ -0,0 +1,59 @@
using Shouldly;
using Xunit;
namespace ZB.MOM.WW.OtOpcUa.AdminUI.Tests;
/// <summary>
/// Pins <c>DriverStatusPanel</c> to the <c>/hosts</c> page (Gitea #546). The panel is the only
/// component carrying the Restart/Reconnect driver controls, and it had ZERO references
/// repo-wide — the commands worked end to end over the mesh transport while no user could reach
/// them, and nothing failed to build, test or deploy. Deleting the last <c>&lt;DriverStatusPanel</c>
/// tag would silently restore exactly that state, so it is pinned here.
/// </summary>
/// <remarks>
/// This is a source scan rather than a rendering assertion because the repo has no bUnit (same
/// technique as <c>MeshRegistrationRoleScopingTests</c>; panel markup is otherwise live-verified).
/// A scan proves reachability of the component, not that it renders correctly.
/// </remarks>
[Trait("Category", "Unit")]
public sealed class HostsPanelWiringTests
{
private static string ReadHostsPageSource()
{
var dir = new DirectoryInfo(AppContext.BaseDirectory);
while (dir is not null && !File.Exists(Path.Combine(dir.FullName, "ZB.MOM.WW.OtOpcUa.slnx")))
dir = dir.Parent;
dir.ShouldNotBeNull("could not locate the repository root (no ZB.MOM.WW.OtOpcUa.slnx above the test binary)");
return File.ReadAllText(Path.Combine(
dir!.FullName, "src", "Server", "ZB.MOM.WW.OtOpcUa.AdminUI",
"Components", "Pages", "Hosts.razor"));
}
/// <summary>THE #546 regression pin: the panel must remain reachable from a routable page.</summary>
[Fact]
public void Hosts_page_renders_the_driver_status_panel()
{
ReadHostsPageSource().ShouldContain(
"<DriverStatusPanel",
Case.Sensitive,
customMessage:
"/hosts no longer renders DriverStatusPanel — the Restart/Reconnect driver controls " +
"are orphaned again with no reachable UI, which is Gitea #546");
}
/// <summary>
/// The panel must be fed the row's real identity and enabled-state, not literals. Hard-coding
/// any of the three would make every expanded row drive the same driver, or claim a disabled
/// instance is live — both of which a source scan for the tag alone would pass.
/// </summary>
[Fact]
public void The_panel_is_bound_to_the_rows_own_driver_cluster_and_enabled_state()
{
var source = ReadHostsPageSource();
source.ShouldContain("DriverInstanceId=\"@d.DriverInstanceId\"");
source.ShouldContain("ClusterId=\"@g.ClusterId\"");
source.ShouldContain("Enabled=\"@d.Enabled\"");
}
}
@@ -16,12 +16,11 @@ namespace ZB.MOM.WW.OtOpcUa.AdminUI.Tests.Hubs;
/// Verifies the Phase 5 dark switch in <see cref="HubServiceCollectionExtensions.WithOtOpcUaSignalRBridges"/>:
/// <c>TelemetryDial:Mode = Dps</c> (default) spawns the four telemetry DPS bridge actors and NOT the
/// gRPC dial supervisor; <c>Grpc</c> spawns the dial supervisor and NONE of the four telemetry
/// bridges. The fleet-status bridge (deferred / out of Phase 5 scope) is spawned in BOTH modes.
/// The gate is proven on a real <see cref="ActorRegistry"/> from a started Akka host.
/// bridges. The gate is proven on a real <see cref="ActorRegistry"/> from a started Akka host.
/// </summary>
public sealed class TelemetryModeWiringTests
{
/// <summary>Dps mode: the four telemetry bridges + fleet bridge are registered; the dial supervisor is not.</summary>
/// <summary>Dps mode: the four telemetry bridges are registered; the dial supervisor is not.</summary>
[Fact]
public async Task Dps_mode_spawns_the_four_telemetry_bridges_and_not_the_dial_supervisor()
{
@@ -31,7 +30,6 @@ public sealed class TelemetryModeWiringTests
{
var registry = host.Services.GetRequiredService<ActorRegistry>();
registry.TryGet<FleetStatusSignalRBridgeKey>(out _).ShouldBeTrue();
registry.TryGet<AlertSignalRBridgeKey>(out _).ShouldBeTrue();
registry.TryGet<ScriptLogSignalRBridgeKey>(out _).ShouldBeTrue();
registry.TryGet<DriverStatusSignalRBridgeKey>(out _).ShouldBeTrue();
@@ -45,7 +43,7 @@ public sealed class TelemetryModeWiringTests
}
}
/// <summary>Grpc mode: the dial supervisor + fleet bridge are registered; none of the four telemetry bridges are.</summary>
/// <summary>Grpc mode: the dial supervisor is registered; none of the four telemetry bridges are.</summary>
[Fact]
public async Task Grpc_mode_spawns_the_dial_supervisor_and_none_of_the_four_telemetry_bridges()
{
@@ -55,7 +53,6 @@ public sealed class TelemetryModeWiringTests
{
var registry = host.Services.GetRequiredService<ActorRegistry>();
registry.TryGet<FleetStatusSignalRBridgeKey>(out _).ShouldBeTrue();
registry.TryGet<TelemetryDialSupervisorKey>(out _).ShouldBeTrue();
registry.TryGet<AlertSignalRBridgeKey>(out _).ShouldBeFalse();
@@ -22,11 +22,11 @@ public sealed class ConfigPublishCoordinatorTimeoutTests : ControlPlaneActorTest
var dbFactory = NewInMemoryDbFactory();
var deploymentId = SeedDispatchingDeployment(dbFactory);
// Short deadline so we can drive it deterministically in the test.
var actor = Sys.ActorOf(ConfigPublishCoordinator.Props(dbFactory, TimeSpan.FromMilliseconds(150)));
// Seed a NodeDeploymentState row so the coordinator doesn't see "zero expected acks"
// and short-circuit to Sealed. We pretend a driver node exists in the cluster.
// Seed the NodeDeploymentState row BEFORE the actor is spawned. PreStart reads
// NodeDeploymentStates to recover the expected-ack set, and it runs as soon as Sys.ActorOf
// schedules the actor — so seeding afterwards is a race: if PreStart wins, the coordinator
// sees zero expected acks and can short-circuit straight to Sealed, and the assertion below
// never sees TimedOut. We pretend a driver node exists in the cluster.
using (var db = dbFactory.CreateDbContext())
{
db.NodeDeploymentStates.Add(new Configuration.Entities.NodeDeploymentState
@@ -38,6 +38,13 @@ public sealed class ConfigPublishCoordinatorTimeoutTests : ControlPlaneActorTest
db.SaveChanges();
}
// Deliberately un-hittable wall-clock deadline (matching the other tests in this class).
// The apply deadline is a real Akka timer: a short one can self-fire under load and race
// the explicit DeadlineElapsed below, so the test would sometimes be measuring the timer
// rather than the message handler. Five minutes guarantees the ONLY deadline signal the
// coordinator ever receives is the one this test sends.
var actor = Sys.ActorOf(ConfigPublishCoordinator.Props(dbFactory, TimeSpan.FromMinutes(5)));
// Drive the deadline ourselves rather than waiting for the cluster's empty driver-set
// bypass. Tell the actor the deadline elapsed for this id.
actor.Tell(new ConfigPublishCoordinator.DeadlineElapsed(deploymentId));
@@ -1,47 +0,0 @@
using Akka.Actor;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Fleet;
using ZB.MOM.WW.OtOpcUa.Commons.Types;
using ZB.MOM.WW.OtOpcUa.ControlPlane.Fleet;
using ZB.MOM.WW.OtOpcUa.ControlPlane.Tests.Harness;
namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Tests;
public sealed class FleetStatusBroadcasterTests : ControlPlaneActorTestBase
{
/// <summary>Verifies that the self member's join appears in the initial fleet status snapshot.</summary>
[Fact]
public void Self_member_up_lands_in_snapshot()
{
var probe = CreateTestProbe();
Sys.ActorOf(FleetStatusBroadcaster.Props(broadcast: msg => probe.Ref.Tell(msg)));
// The cluster self-joined in the harness — InitialStateAsEvents replays MemberUp for self.
var snapshot = probe.ExpectMsg<FleetStatusChanged>(TimeSpan.FromSeconds(5));
snapshot.Nodes.Count.ShouldBeGreaterThanOrEqualTo(1);
snapshot.Nodes.Any(n => n.Health == FleetNodeHealth.Healthy).ShouldBeTrue();
}
/// <summary>Verifies that heartbeat messages update the node revision in the next broadcast.</summary>
[Fact]
public void Heartbeat_updates_revision_in_next_snapshot()
{
var probe = CreateTestProbe();
var actor = Sys.ActorOf(FleetStatusBroadcaster.Props(broadcast: msg => probe.Ref.Tell(msg)));
// Drain the initial MemberUp snapshot.
probe.ExpectMsg<FleetStatusChanged>(TimeSpan.FromSeconds(5));
// Send a heartbeat for a node we know about (the self node from the cluster).
var selfHost = Akka.Cluster.Cluster.Get(Sys).SelfAddress.Host!;
var rev = RevisionHash.Parse(new string('c', 64));
actor.Tell(new FleetStatusBroadcaster.DriverHostStatusHeartbeat(NodeId.Parse(selfHost), rev));
// Wait for next periodic broadcast and assert the revision propagated.
var next = probe.FishForMessage<FleetStatusChanged>(
isMessage: m => m.Nodes.Any(n => n.CurrentRevision == rev),
max: TimeSpan.FromSeconds(10));
next.Nodes.Single(n => n.NodeId.Value == selfHost).CurrentRevision.ShouldBe(rev);
}
}
@@ -69,7 +69,7 @@ public sealed class RedundancyStateSingletonRehomeTests
registry.TryGet<RedundancyStateActorKey>(out _).ShouldBeFalse(
"the redundancy singleton was re-homed off the admin node onto every driver node (Phase 6)");
registry.TryGet<FleetStatusBroadcasterKey>(out _).ShouldBeTrue(
registry.TryGet<AuditWriterActorKey>(out _).ShouldBeTrue(
"the remaining admin singletons must still register");
}
@@ -65,6 +65,19 @@ public sealed class ConfigServeAuthInterceptorTests
ex.StatusCode.ShouldBe(StatusCode.Unauthenticated);
}
[Fact]
public async Task FetchMethod_WithBlankKeyConfigured_IsDenied()
{
// Blank is distinct from unset in configuration (a stray-whitespace appsettings value) but
// must be treated identically here: an all-blank expected key would be satisfied by an
// all-blank bearer. Pinned identically on MeshCommandAuthInterceptor.
var interceptor = CreateInterceptor(" ");
var context = CreateContext(FetchMethod, "Bearer ");
var ex = await Should.ThrowAsync<RpcException>(() => Invoke(interceptor, context));
ex.StatusCode.ShouldBe(StatusCode.Unauthenticated);
}
[Fact]
public async Task FetchMethod_WithNoBearerToken_IsDenied()
{
@@ -121,6 +134,16 @@ public sealed class ConfigServeAuthInterceptorTests
ex.StatusCode.ShouldBe(StatusCode.Unauthenticated);
}
[Fact]
public void HasExactlyOnePublicConstructor()
{
// Grpc.AspNetCore silently stops invoking an interceptor with more than one public ctor —
// this pins the shape so a well-meaning overload doesn't quietly disable auth in production.
var publicCtors = typeof(ConfigServeAuthInterceptor).GetConstructors();
publicCtors.Length.ShouldBe(1);
}
/// <summary>
/// Minimal <see cref="ServerCallContext"/> carrying just a method name and request headers —
/// the only two things the interceptor reads. Hand-rolled because
@@ -99,6 +99,19 @@ public sealed class LocalDbSyncAuthInterceptorTests
ex.StatusCode.ShouldBe(StatusCode.PermissionDenied);
}
[Fact]
public async Task SyncMethod_WithBlankKeyConfigured_IsDenied()
{
// Blank is distinct from unset in configuration (a stray-whitespace appsettings value) but
// must be treated identically here: an all-blank expected key would be satisfied by an
// all-blank bearer. Pinned identically on MeshCommandAuthInterceptor.
var interceptor = CreateInterceptor(" ");
var context = CreateContext(SyncMethod, "Bearer ");
var ex = await Should.ThrowAsync<RpcException>(() => Invoke(interceptor, context));
ex.StatusCode.ShouldBe(StatusCode.PermissionDenied);
}
[Fact]
public async Task SyncMethod_WithNoBearerToken_IsDenied()
{
@@ -152,6 +165,16 @@ public sealed class LocalDbSyncAuthInterceptorTests
ex.StatusCode.ShouldBe(StatusCode.PermissionDenied);
}
[Fact]
public void HasExactlyOnePublicConstructor()
{
// Grpc.AspNetCore silently stops invoking an interceptor with more than one public ctor —
// this pins the shape so a well-meaning overload doesn't quietly disable auth in production.
var publicCtors = typeof(LocalDbSyncAuthInterceptor).GetConstructors();
publicCtors.Length.ShouldBe(1);
}
[Fact]
public async Task DuplexStreaming_IsGated_BecauseThatIsHowSyncActuallyRuns()
{
@@ -65,6 +65,19 @@ public sealed class TelemetryStreamAuthInterceptorTests
ex.StatusCode.ShouldBe(StatusCode.PermissionDenied);
}
[Fact]
public async Task SubscribeMethod_WithBlankKeyConfigured_IsDenied()
{
// Blank is distinct from unset in configuration (a stray-whitespace appsettings value) but
// must be treated identically here: an all-blank expected key would be satisfied by an
// all-blank bearer. Pinned identically on MeshCommandAuthInterceptor.
var interceptor = CreateInterceptor(" ");
var context = CreateContext(SubscribeMethod, "Bearer ");
var ex = await Should.ThrowAsync<RpcException>(() => Invoke(interceptor, context));
ex.StatusCode.ShouldBe(StatusCode.PermissionDenied);
}
[Fact]
public async Task SubscribeMethod_WithNoBearerToken_IsDenied()
{