Compare commits

...

16 Commits

Author SHA1 Message Date
Joseph Doherty 521ee75355 docs: redundancy primary-only alarm emission/historization, redundancy-state NodeId+heartbeat fixes, Galaxy reconnect, live-pill
v2-ci / build (push) Failing after 48s
v2-ci / unit-tests (tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests) (push) Has been skipped
v2-ci / unit-tests (tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests) (push) Has been skipped
v2-ci / unit-tests (tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests) (push) Has been skipped
v2-ci / unit-tests (tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests) (push) Has been skipped
v2-ci / unit-tests (tests/Server/ZB.MOM.WW.OtOpcUa.Security.Tests) (push) Has been skipped
v2-ci / integration (tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests) (push) Has been skipped
v2-ci / integration (tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests) (push) Has been skipped
2026-06-11 10:18:25 -04:00
Joseph Doherty 7891e28b52 fix(redundancy): periodic heartbeat re-publish so late subscribers learn their role 2026-06-11 10:06:46 -04:00
Joseph Doherty e241332a24 fix(redundancy): key redundancy-state snapshot by canonical host:port NodeId (was host-only — broke ServiceLevel + scripted-alarm emit gate) 2026-06-11 09:56:17 -04:00
Joseph Doherty 23a4a0093b fix(adminui): guard Alerts chip auto-clear against stale-timer race (review) 2026-06-11 09:38:58 -04:00
Joseph Doherty 19ec656cdc feat(adminui): ScriptLog live-pill bound to broadcaster health
Subscribe ConnectionStateChanged before reading IsConnected (subscribe-then-read
idiom, matches DriverStatusPanel) so no transition is missed. Add
OnConnectionStateChanged handler that marshals to the circuit sync context via
InvokeAsync. Dispose unsubscribes both events.

Reconnect-overlay: App.razor loads _framework/blazor.web.js and contains no
custom #components-reconnect-modal element; .NET 10 Blazor's default reconnect
overlay is active automatically — no custom markup needed.

No unit tests; live-verify follows.
2026-06-11 09:37:19 -04:00
Joseph Doherty d29c933499 feat(adminui): Alerts timed-shelve picker + chip auto-clear + live-pill health 2026-06-11 09:34:33 -04:00
Joseph Doherty f9932f2d8e refactor(admin): use CorrelationId wrapper for alarm ack/shelve commands 2026-06-11 09:27:24 -04:00
Joseph Doherty 3a0e0907e4 feat(adminui): add connection-health signal to in-process broadcaster + bridges 2026-06-11 09:20:36 -04:00
Joseph Doherty 565b77e6cf fix(galaxy): unify IsConnected with _connected guard; AttachForTests marks connected (review) 2026-06-11 09:16:51 -04:00
Joseph Doherty 43b96441a5 fix(galaxy): reconnect recreates a faulted session instead of no-op'ing 2026-06-11 09:10:52 -04:00
Joseph Doherty 4f291ed09c test(redundancy): cover absent-node default-historize for HistorianAdapter (A2) 2026-06-11 09:02:02 -04:00
Joseph Doherty 0742946108 feat(redundancy): gate alarm historization on Primary (A2, defensive — actor currently unfed)
HistorianAdapterActor now subscribes to the redundancy-state DPS topic,
caches the local node's RedundancyRole, and SKIPS the durable-sink enqueue
when the local node is Secondary or Detached. Unknown/null role default-writes
so single-node deploys and the boot window never silently drop historization.
GetStatus stays ungated.

PREMISE: verified the actor is registered but FED BY NOTHING in production —
there is no AlarmHistorianEvent producer and nothing resolves its registry key
to Tell it. This is a FORWARD-LOOKING / DEFENSIVE guard, not a fix for a live
double-write: the moment a per-node feeder lands (engine -> historian, expected
as a per-node cluster broadcast like the alerts topic), only the Primary will
write to the durable sink (exactly-once across all alarm sources).

Mirrors the sibling A1 treatment of ScriptedAlarmHostActor (06c4155) and
OpcUaPublishActor's redundancy-state handler. localNode threaded through
HistorianAdapterActor.Props from ServiceCollectionExtensions (roleInfo.LocalNode).
2026-06-11 08:57:41 -04:00
Joseph Doherty 9ac9f0b7a9 test(redundancy): cover Detached suppression + absent-node default-emit (A1) 2026-06-11 08:50:59 -04:00
Joseph Doherty 06c415598c feat(redundancy): gate scripted-alarm alerts publish on Primary (A1) 2026-06-11 08:44:44 -04:00
Joseph Doherty 535787bd85 docs(plan): alarm follow-ups implementation plan (T0-T10) 2026-06-11 08:28:37 -04:00
Joseph Doherty bcb9f45cb3 docs(design): alarm ack/shelve follow-ups (redundancy double-emit, Timed-shelve UI, T21 minors, rig cleanup, Galaxy reconnect, live-pill) 2026-06-11 08:23:58 -04:00
28 changed files with 1522 additions and 57 deletions
+4
View File
@@ -153,6 +153,10 @@ The AdminUI `/alerts` Shelve flow was live-verified on docker-dev
2026-06-11: singleton → topic → host actor → engine → "Shelved" status
reflected on `/alerts` with the operator identity threaded through.
## Redundancy deduplication
Under warm/hot redundancy, both cluster nodes run `ScriptedAlarmHostActor` and the scripted-alarm engine. To prevent duplicate `/alerts` rows and duplicate historian writes, alarm transition publication to the `alerts` topic and `HistorianAdapterActor` historization are **Primary-gated**: only the node whose `RedundancyRole` is `Primary` publishes externally. OPC UA condition-node writes and inbound ack/shelve processing remain ungated on both nodes so the secondary stays warm for failover. See [Redundancy.md §Primary-gated alarm emission and historization](Redundancy.md#primary-gated-alarm-emission-and-historization).
## Historian write-back (non-Galaxy alarms)
Scripted alarms (and any future non-Galaxy `IAlarmSource` like
+13
View File
@@ -111,6 +111,8 @@ Both nodes share the same `ConfigDb` connection string; `Cluster.PublicHostname`
There is no longer a `Node:NodeId` setting and no `ClusterNode.RedundancyRole` column (the V2 migration dropped it — primary/secondary is now derived from cluster role-leadership). NodeId is derived as `host:port` of the cluster `PublicHostname` (see `ClusterRoleInfo.LocalNode` for the formula).
> **`RedundancyStateActor` NodeId consistency (fixed).** `RedundancyStateActor` now keys each node's `NodeRedundancyState` entry by the canonical `host:port` node id (via a `ToNodeId(Address)` helper mirroring `ClusterRoleInfo.ToNodeId`). Previously it keyed by `member.Address.Host` (host-only, e.g. `central-2`); since every subscriber matches by the canonical `host:port` form, the mismatch silently meant no node ever matched its own entry — all nodes stayed at the default ServiceLevel 255 and never learned their role. This fix makes `RedundancyStateActor` consistent with the stated contract above. Additionally, `RedundancyStateActor` now **re-publishes the current snapshot on a periodic heartbeat (default 10 s)** so any node that subscribes after the last topology-change publish converges within the interval (DistributedPubSub does not replay to late subscribers).
The `ClusterNode.ServiceLevelBase` column still exists and is editable in the Admin UI (NodeEdit / Cluster Redundancy pages), but it no longer drives the runtime ServiceLevel — that value is computed from cluster role/health and published per the mapping above, independent of this stored preference.
### Peer URI advertising
@@ -133,6 +135,17 @@ Node A lists Node B's `ApplicationUri` and vice-versa. Validated by `DualEndpoin
There is no operator-driven role swap during a partition. Failover is what the cluster does automatically.
## Primary-gated alarm emission and historization
Under warm/hot redundancy both cluster nodes run `ScriptedAlarmHostActor` and evaluate scripted alarms, keeping each node's address space and engine state warm for instant failover. However, to avoid duplicate rows on `/alerts` and duplicate historian writes, only the Primary node publishes externally:
- **`alerts` topic emission** — `ScriptedAlarmHostActor` subscribes to the `redundancy-state` DPS topic and caches the local node's `RedundancyRole`. Each alarm transition is published to the cluster `alerts` topic **only when the node's role is `Primary`**. The default behaviour before any `redundancy-state` message arrives is to emit, so single-node deployments and the boot window never drop transitions. The OPC UA condition-node write and inbound ack/shelve command processing remain **ungated** on both nodes so the secondary is always ready to serve clients after a failover.
- **`HistorianAdapterActor` historization** — likewise Primary-gated so alarm historization is exactly-once across all alarm sources. (This actor currently has no production feeder; the gate is a forward-looking guard.)
Net effect: each alarm transition appears **once** on `/alerts` and would historize once, not once per node.
See [ScriptedAlarms.md](ScriptedAlarms.md) and [AlarmTracking.md](AlarmTracking.md) for the scripted-alarm engine internals.
## Client-side failover
The OtOpcUa Client CLI at `src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI` supports `-F` / `--failover-urls` for automatic client-side failover; for long-running subscriptions the CLI monitors session KeepAlive and reconnects to the next available server, recreating the subscription on the new endpoint. See [`Client.CLI.md`](Client.CLI.md).
+3 -1
View File
@@ -125,7 +125,9 @@ On allow, the handler publishes a `Commons.OpcUa.AlarmCommand` (containing comma
### AdminUI path
The AdminUI `/alerts` page (`Alerts.razor`) shows per-row **Acknowledge / Shelve / Unshelve** buttons. These are gated by the `DriverOperator` AdminUI policy and routed through the `AdminOperationsActor` cluster singleton (`AcknowledgeAlarmCommand` / `ShelveAlarmCommand`), which publishes onto the same `alarm-commands` topic. Cross-node routing is handled by the cluster singleton — the command always reaches the driver-role node hosting the engine that owns the alarm regardless of which AdminUI instance the operator is connected to.
The AdminUI `/alerts` page (`Alerts.razor`) shows per-row **Acknowledge / Shelve / Unshelve** buttons. These are gated by the `DriverOperator` AdminUI policy and routed through the `AdminOperationsActor` cluster singleton (`AcknowledgeAlarmCommand` / `ShelveAlarmCommand`), which publishes onto the same `alarm-commands` topic. Cross-node routing is handled by the cluster singleton — the command always reaches the driver-role node hosting the engine that owns the alarm regardless of which AdminUI instance the operator is connected to. The Shelve button accepts a duration (minutes) and passes it to `ShelveAlarmAsync(Timed, unshelveAtUtc)`; result chips auto-clear after ~8 s. The `/alerts` and `/script-log` live-status pills reflect real feed health driven by the SignalR bridge actors' DPS subscription state.
Under redundancy, the `alerts` topic rows are deduplicated to the Primary node — see [Redundancy.md §Primary-gated alarm emission and historization](Redundancy.md#primary-gated-alarm-emission-and-historization).
### Client.CLI path
+1 -1
View File
@@ -90,7 +90,7 @@ Full per-field descriptions live in `Config/GalaxyDriverOptions.cs`. The full JS
## Reconnect + Replay
`ReconnectSupervisor` owns an exponential-backoff loop bounded by `Reconnect.InitialBackoffMs` / `MaxBackoffMs`. On session loss it tears down the gRPC channel, redials, and — when `ReplayOnSessionLost = true` — calls the gateway's `ReplaySubscriptions` RPC with the cached subscription set from `SubscriptionRegistry` instead of re-subscribing tag-by-tag. The gateway's worker then re-issues `AdviseSupervisory` server-side under the apartment lock.
`ReconnectSupervisor` owns an exponential-backoff loop bounded by `Reconnect.InitialBackoffMs` / `MaxBackoffMs`. On session loss it calls `GalaxyDriver.ReopenAsync`, which invokes `GalaxyMxSession.RecreateAsync` to dispose the stale/faulted session and client before rebuilding (`OpenSessionAsync` + `RegisterAsync`). Previously `ConnectAsync` was a no-op when a stale session handle was still present, so the reopen supervisor looped forever without recovering. After a successful reopen — when `ReplayOnSessionLost = true` — the supervisor calls the gateway's `ReplaySubscriptions` RPC with the cached subscription set from `SubscriptionRegistry` instead of re-subscribing tag-by-tag. The gateway's worker then re-issues `AdviseSupervisory` server-side under the apartment lock.
## Testing
@@ -0,0 +1,167 @@
# Alarm Ack/Shelve Follow-ups — Design
**Date:** 2026-06-11
**Status:** Approved (brainstorming) — ready for implementation plan.
## Goal
Resolve the six notes/follow-ups left open by the T17–T24 inbound-alarm-ack work
(merged to master `bc9843d2`): the redundancy **double-emit**, the deferred Timed-shelve
UI, two T21 minors, the docker-dev rig cleanup, and two pre-existing Layer-1 gaps (Galaxy
reconnect, the live-pill). Scope decided in brainstorming: **everything**, including the
pre-existing L1 gaps. Double-emit approach decided: **primary-only at the source**.
---
## 1 — Redundancy double-emit fix (core; high-risk)
### Problem (verified)
Both central nodes run roles `admin,driver` in the same MAIN cluster. Each spawns its own
`ScriptedAlarmHostActor` + `ScriptedAlarmEngine` (`DriverHostActor.SpawnScriptedAlarmHost`,
per-node, no singleton) and, for a single-cluster artifact, loads the **same** alarms
(`DeploymentArtifact.ParseComposition` returns `ClusterFilterMode.None` → unfiltered). Both
independently evaluate and both hit `ScriptedAlarmHostActor.OnEngineEmission`:
- line 261 `_publishActor.Tell(new OpcUaPublishActor.AlarmStateUpdate(...))` — writes the OPC
UA node (directed, local).
- line 278 `_mediator.Tell(new Publish(AlertsTopic /* "alerts" */, evt))` — **cluster-wide,
ungated** → every transition appears **twice** on `/alerts`. Inbound ack/shelve commands
are likewise processed by both nodes' engines.
`HistorianAdapterActor` (`Runtime/Historian/`, registered **per-node** in
`Runtime/ServiceCollectionExtensions.cs:146`, name `historian-adapter`) also consumes the
`alerts` topic, so a single publish is historized **once per node** → double DB writes in
production (invisible in docker-dev: `NullAlarmHistorianSink`).
### Existing signal to mirror
`RedundancyStateActor` (admin singleton, `ControlPlane/Redundancy/RedundancyStateActor.cs:90-114`)
publishes `RedundancyStateChanged` to the `redundancy-state` DPS topic. Per-node
`NodeRedundancyState` carries `RedundancyRole` (`Primary`/`Secondary`/`Detached`) +
`IsRoleLeaderForDriver`; **Primary = the driver-role cluster leader**. `OpcUaPublishActor`
already subscribes to this topic (`OpcUaPublishActor.cs:30,147,156,335`) to drive its
ServiceLevel — the exact pattern to copy (`Subscribe(RedundancyStateTopic, Self)` in
PreStart; `Receive<RedundancyStateChanged>`; read `msg.Nodes.FirstOrDefault(n => n.NodeId ==
localNode)?.Role`).
### Decision — gate **only the cluster-wide emission**, on `Primary`
- **Emission gate (task A1):** `ScriptedAlarmHostActor` subscribes to `redundancy-state`,
caches the local role, and **skips the `alerts` publish (line 278) when not Primary**.
**Default = emit until a `RedundancyStateChanged` says this node is Secondary/Detached** —
so single-node deploys (sole node is always the driver leader = Primary) and the boot
window never drop transitions. The OPC UA node write (line 261) and **inbound command
processing stay UNGATED** — the secondary must keep its address space warm and its engine
state consistent for failover; clients only ever see the Primary via ServiceLevel, so the
secondary's node writes + its (subscriber-less) condition events are harmless.
- **Historian gate (task A2):** `HistorianAdapterActor` subscribes to `redundancy-state`,
caches the local role, and **skips the sink write when not Primary**. Gives exactly-once
historization for **all** alarm sources (native Galaxy/AB-CIP too), not just scripted.
- **Edge:** a brief failover window (old Primary gone, new not yet elected) may drop a
transition/historization — acceptable, and identical to the existing ServiceLevel handoff
behaviour.
### Tests
TestKit: with a Secondary `RedundancyStateChanged`, `OnEngineEmission` does NOT publish to
`alerts` but DOES still `Tell` the OPC UA `AlarmStateUpdate`; with Primary (or before any
state) it publishes. Inbound `AlarmCommand` is still processed regardless of role. Historian:
a Secondary skips the sink write, a Primary writes (fake sink + role injection).
---
## 2 — Timed-shelve picker UI (small)
`Alerts.razor` currently exposes Ack / Shelve(OneShot) / Unshelve. The `Timed` backend is
fully wired + tested (`ShelveAlarmCommand{ShelveKind.Timed, UnshelveAtUtc}` →
`AdminOperationsActor` → `engine.TimedShelveAsync`). Add a small duration input (a minutes/
seconds number box, or a `datetime-local`) to the shelve control on each row and call
`ShelveAlarmAsync(kind: Timed, unshelveAtUtc: now + duration)`. Razor-only; **no backend
change**; proven by docker-dev live-verify (no bUnit). Keep the existing OneShot/Unshelve
buttons.
## 3 — T21 minors (small)
- **Chip auto-clear:** the `_opResult*` chip persists until the next action. Add the ~8s
auto-clear timer pattern `DriverStatusPanel.razor` already uses (a `Task.Delay` →
`InvokeAsync(StateHasChanged)` that clears the per-row result).
- **CorrelationId consistency:** `AcknowledgeAlarmCommand`/`ShelveAlarmCommand` use a bare
`Guid CorrelationId`; the project's other control-plane commands use the `CorrelationId`
wrapper type. Switch both records + `AdminOperationsClient` (`CorrelationId.NewId()`) +
the actor reply contract + tests to the wrapper for uniform correlation tracing.
## 4 — Rig cleanup (operational, last)
Delete the docker-dev seed artifacts left for live-verify: the `t12-overheat` scripted alarm,
the `SC-ba675b168a85` predicate script, the `layer0-logcheck` vtag + script; revert
filler-02's inert `cycle-time-s` logger line to `return ctx.GetTag("TestMachine_002.TestDuration").Value;`.
Redeploy (`POST /api/deployments`, `X-Api-Key: docker-dev-deploy-key`). DB/redeploy, **not a
code change** — done last so the rig stays available for verifying tasks 1–3/6.
## 5 — Galaxy reconnect recreate (high-risk)
### Problem (verified)
`GalaxyMxSession.ConnectAsync` (`Driver.Galaxy/Runtime/GalaxyMxSession.cs:58-69`) is
idempotent: `if (_session is not null) return;`. When the gateway restarts and the session
goes Faulted/NotFound, `_session` is still a non-null (dead) handle, so `ConnectAsync` is a
silent no-op. `GalaxyDriver.ReopenAsync` (`GalaxyDriver.cs:289`) calls it expecting a
reconnect → no-op; `ReconnectSupervisor.RecoveryLoopAsync`
(`Driver.Galaxy/Runtime/ReconnectSupervisor.cs:158-186`) sees reopen "succeed", proceeds to
replay (which fails on the dead session), and **loops forever** (backoff capped 30s).
### Decision — recreate on reopen
Add a recreate path to `GalaxyMxSession` (e.g. `RecreateAsync`/`DisposeSessionForRecreationAsync`)
that disposes + nulls `_session` and `_ownedClient`, and have `ReopenAsync` call it **before**
`ConnectAsync` so a reopen always routes through the happy-path create (`OpenSessionAsync` +
`RegisterAsync`). Confirm what status/exception marks Faulted/NotFound and that the dispose is
safe (gRPC channel teardown). Keep the supervisor's backoff loop; it now actually recovers.
### Tests
A reconnect test asserting that after a faulted session, the reopen path **creates a new
session** (new handle / `OpenSessionAsync` called again) rather than no-op'ing. Mirror any
existing `ReconnectSupervisor`/session tests.
## 6 — Live-pill circuit health (standard)
### Problem (verified)
`ScriptLog.razor:122` and `Alerts.razor:132` set `_connected = true` once in
`OnInitialized` and never update it; the pill markup binds to that set-once bool.
`IInProcessBroadcaster<T>` (`AdminUI/Hubs/IInProcessBroadcaster.cs`) exposes only
`Received` + `Publish` — no health signal.
### Decision
- Extend `IInProcessBroadcaster<T>` with a connection-health signal (`bool IsConnected` +
`event … ConnectionStateChanged`). The bridge actors (`ScriptLogSignalRBridge` /
`AlertSignalRBridge`) set it from their DPS-subscription health (SubscribeAck up / failure
down). The razors bind the pill to it and subscribe/unsubscribe like
`DriverStatusPanel.razor` (`OnConnectionStateChanged` → `InvokeAsync(StateHasChanged)`).
- **Dead-circuit case** (node recreate kills the server-side circuit — the component is dead
and cannot self-update its pill): this is Blazor Server's built-in reconnection concern.
**Verify the default reconnect overlay is present/visible** (it is what actually signals a
dropped circuit) rather than trying to fake liveness from a dead component. If absent, add
the standard Blazor reconnect UI.
### Tests
Broadcaster unit test for the new health signal + `SetConnected` propagation. Razor proven by
docker-dev live-verify (no bUnit).
---
## Sequencing & risk
| Item | Risk | Notes |
|---|---|---|
| 1 redundancy double-emit (A1 emit gate, A2 historian gate) | high-risk | independent subsystem; A1∥A2 (different files) |
| 5 Galaxy reconnect | high-risk | independent subsystem (Galaxy driver) |
| 2 Timed-shelve picker | small | razor-only, live-verify |
| 3a chip auto-clear / 3b CorrelationId | small | razor / mechanical refactor |
| 6 live-pill | standard | interface + bridge + 2 razors |
| 4 rig cleanup | trivial/operational | last |
The two high-risk items (1, 5) are in different subsystems and can run in parallel with each
other and with the UI items (2, 3, 6). Rig cleanup (4) is last. TDD where there's logic; UI
proven by docker-dev `/run` live-verify (agent drives — login disabled on docker-dev).
## Hard rules
Stage by explicit path (never `git add .`); never stage `sql_login.txt` /
`src/Server/.../Host/pki/`; never echo the gateway API key into a **new** tracked file; no
force-push, no `--no-verify`; **no Configuration entity / EF migration change** (none of these
items needs one). Build on a feature branch off `master`.
+291
View File
@@ -0,0 +1,291 @@
# Alarm Ack/Shelve Follow-ups Implementation Plan
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans (or subagent-driven-development) to implement this plan task-by-task.
**Goal:** Resolve the six follow-ups left by the T17–T24 inbound-alarm-ack work: the redundancy double-emit, the Timed-shelve UI, two T21 minors, the docker-dev rig cleanup, and two pre-existing Layer-1 gaps (Galaxy reconnect, the live-pill).
**Architecture:** The double-emit is fixed by **primary-only emission at the source** — both `ScriptedAlarmHostActor` (the `alerts` publish) and the per-node `HistorianAdapterActor` (the sink write) subscribe to the existing `redundancy-state` DPS topic, cache the local `RedundancyRole`, and act only when `Primary` (mirroring `OpcUaPublishActor`). OPC UA node writes + inbound-command processing stay ungated for warm-standby. The other five items are localized fixes (one Galaxy-driver lifecycle bug, one broadcaster-health interface extension + 2 razors, two small Alerts.razor additions, one mechanical refactor, one operational cleanup).
**Tech Stack:** .NET 10, Akka.NET (cluster, DistributedPubSub, TestKit/xunit2), Blazor Server (InteractiveServer, NO bUnit), xUnit + Shouldly, OPC Foundation UA .NET Standard, Serilog.
**Design of record:** `docs/plans/2026-06-11-alarm-followups-design.md` (committed master `bcb9f45c`).
**Hard rules:** stage by explicit path (never `git add .`); never stage `sql_login.txt` / `src/Server/.../Host/pki/`; never echo the gateway API key into a **new** tracked file; no force-push, no `--no-verify`; **no Configuration entity / EF migration change**. Build on a feature branch off master.
---
### Task 0: Branch + baseline
**Classification:** trivial
**Estimated implement time:** ~1 min
**Parallelizable with:** none
**Files:** (none — git only)
**Steps:**
1. `git checkout master && git switch -c feat/alarm-followups` (off `bcb9f45c`).
2. Confirm clean tree + green baseline build: `dotnet build ZB.MOM.WW.OtOpcUa.slnx` → 0 errors.
3. No commit (branch only).
---
### Task 1: Redundancy emit-gate in `ScriptedAlarmHostActor` (A1)
**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 2, Task 3, Task 4, Task 5
**Files:**
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ScriptedAlarms/ScriptedAlarmHostActor.cs`
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/ScriptedAlarms/ScriptedAlarmHostActorTests.cs`
**Context:** `OnEngineEmission` (≈ line 247–279) does two things per emission: `_publishActor.Tell(AlarmStateUpdate(...))` (line 261, OPC UA node write) and `_mediator.Tell(new Publish(AlertsTopic, evt))` (line 278, cluster-wide `alerts`). Both central nodes run this → the `alerts` publish doubles. Gate **only line 278** on `Primary`.
**Reference pattern to mirror** (`src/Server/ZB.MOM.WW.OtOpcUa.Runtime/OpcUa/OpcUaPublishActor.cs`): `RedundancyStateTopic = "redundancy-state"` (line 30); `Subscribe(RedundancyStateTopic, Self)` in PreStart (line 156); `Receive<RedundancyStateChanged>(HandleRedundancyStateChanged)` (line 147); `HandleRedundancyStateChanged` reads `msg.Nodes.FirstOrDefault(n => n.NodeId == localNode)?.Role` (lines 335–351). `RedundancyRole` lives in `ZB.MOM.WW.OtOpcUa.Commons.Messages.Redundancy` (`Primary`/`Secondary`/`Detached`). The host already knows its local node id (it's used elsewhere; if not, derive it the same way `OpcUaPublishActor` resolves `_localNode`).
**Step 1: Failing TestKit tests** (extend the existing harness; Runtime.Tests = xunit v2 + Akka.TestKit.Xunit2):
- `Emission_is_published_to_alerts_by_default_before_any_redundancy_state` — a fresh host (no `RedundancyStateChanged` yet) DOES publish the `AlarmTransitionEvent` to `alerts` (subscribe a probe to `AlertsTopic`).
- `Secondary_node_suppresses_alerts_publish_but_still_writes_opcua` — after a `RedundancyStateChanged` marking the local node `Secondary`, an emission does NOT publish to `alerts` but DOES still `Tell` the `OpcUaPublishActor.AlarmStateUpdate` (probe the publish-actor seam the test already uses for T9/T19).
- `Primary_node_publishes_alerts` — after a `Primary` `RedundancyStateChanged`, the emission publishes to `alerts`.
- `Inbound_AlarmCommand_is_processed_regardless_of_role` — a `Secondary` host still drives the engine for an inbound `AlarmCommand` (the existing T19 ack test, asserted under a Secondary role).
**Step 2:** Run them — expect FAIL (no gate yet; Secondary still publishes).
**Step 3: Implement.** Add a cached `RedundancyRole? _localRole = null;` (null = unknown ⇒ treat as Primary/emit). In PreStart, `_mediator.Tell(new Subscribe(OpcUaPublishActor.RedundancyStateTopic, Self))` (reuse the const; if cross-project reference is awkward, introduce a shared `Commons` const for `"redundancy-state"` and point both at it — surface that deviation). Add `Receive<RedundancyStateChanged>` + a `SubscribeAck` no-op. In the handler, set `_localRole` from the snapshot for the local node. In `OnEngineEmission`, before line 278:
```csharp
// Warm-standby dedup: only the Primary (driver-role leader) publishes the cluster-wide
// transition + drives historization. Default-emit until told we are Secondary/Detached so
// single-node deploys + the boot window never drop transitions. The OPC UA node write
// above (warm address space) and inbound command processing stay ungated.
if (_localRole is RedundancyRole.Secondary or RedundancyRole.Detached)
return;
_mediator.Tell(new Publish(AlertsTopic, evt));
```
Leave line 261 (the `_publishActor.Tell`) and `OnAlarmCommand` untouched.
**Step 4:** Run `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests --filter ScriptedAlarmHostActor` → all green.
**Step 5: Commit** by explicit path (`ScriptedAlarmHostActor.cs` + the test).
> High-risk: concurrency (role cached on the actor thread, read in the emission path which is also marshalled onto the actor thread — confirm `OnEngineEmission` runs on the actor thread, it does via `Self.Tell(EngineEmission)`) + redundancy semantics. Do NOT gate the OPC UA write or commands.
---
### Task 2: Redundancy historize-gate in `HistorianAdapterActor` (A2)
**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 1, Task 3, Task 4, Task 5
**Files:**
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/HistorianAdapterActor.cs`
- Modify (only if the actor needs the local node id / mediator wired): `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs` (≈ line 146, where it's spawned `historian-adapter`)
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Historian/HistorianAdapterActorTests.cs` (create if absent; else extend the existing historian test)
**Context:** `HistorianAdapterActor` is **per-node** and consumes the `alerts` topic, so one publish historizes once per node → double DB writes. Gate the sink write on `Primary`, same pattern as Task 1.
**Step 1: Failing tests** (TestKit + a fake `IAlarmHistorianSink` that records writes):
- `Default_before_redundancy_state_historizes` — a fresh adapter writes to the sink (treat unknown as Primary).
- `Secondary_node_does_not_historize` — after a `Secondary` `RedundancyStateChanged`, an incoming alarm event is NOT written to the sink.
- `Primary_node_historizes` — after `Primary`, it IS written.
**Step 2:** Run — FAIL.
**Step 3: Implement** the same `_localRole` cache + `redundancy-state` subscription + `Receive<RedundancyStateChanged>` as Task 1, and guard the sink write: `if (_localRole is RedundancyRole.Secondary or RedundancyRole.Detached) return;` before `_sink.Write...`. The actor will need its local node id; resolve it the way `OpcUaPublishActor`/`ServiceCollectionExtensions` already does (pass it into `Props` if not already available — check how `OpcUaPublishActor` gets `_localNode` and mirror; thread it through `HistorianAdapterActor.Props` + the registration at `ServiceCollectionExtensions.cs:146`).
**Step 4:** `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests --filter Historian` → green. Also confirm the full Runtime.Tests still pass.
**Step 5: Commit** by explicit path.
> High-risk: data-historization correctness + concurrency. If wiring the node id into `Props` ripples beyond the two files, surface it before expanding.
---
### Task 3: Galaxy reconnect recreates a faulted session
**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 1, Task 2, Task 4, Task 5
**Files:**
- Modify: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/Runtime/GalaxyMxSession.cs` (≈ line 58–69)
- Modify: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/GalaxyDriver.cs` (≈ line 289, `ReopenAsync`)
- Test: the Galaxy driver test project (find `*ReconnectSupervisor*` / `*Session*` tests under `tests/Drivers/...Driver.Galaxy.Tests/`; mirror their seam)
**Context:** `GalaxyMxSession.ConnectAsync` has `if (_session is not null) return;` — so when the gRPC session is Faulted/NotFound the field is still a non-null dead handle and `ConnectAsync` is a silent no-op. `GalaxyDriver.ReopenAsync` calls `ConnectAsync` expecting a reconnect → no-op → `ReconnectSupervisor.RecoveryLoopAsync` (`Runtime/ReconnectSupervisor.cs:158-186`) sees reopen "succeed", replay fails, loops forever.
**Step 1: Failing test.** Assert the reconnect path **creates a new session** after a faulted one: drive `GalaxyMxSession` (or a fake `MxGatewayClient`/session seam) to a connected state, simulate fault/dispose, call the reopen/recreate path, and assert `OpenSessionAsync` + `RegisterAsync` were invoked **again** (a second create), not a no-op. Mirror the existing session/reconnect test construction (read how they fake `MxGatewayClient`/`OpenSessionAsync`).
**Step 2:** Run — FAIL (current code no-ops, second create never happens).
**Step 3: Implement.** Add a recreate path to `GalaxyMxSession`, e.g.:
```csharp
/// <summary>Disposes the current (faulted/stale) session + owned client so the next
/// <see cref="ConnectAsync"/> rebuilds a fresh session instead of no-op'ing on the dead handle.</summary>
public async Task RecreateAsync(MxGatewayClientOptions clientOptions, CancellationToken ct)
{
ObjectDisposedException.ThrowIf(_disposed, this);
await DisposeSessionAsync().ConfigureAwait(false); // dispose+null _session, _serverHandle, _ownedClient
await ConnectAsync(clientOptions, ct).ConfigureAwait(false);
}
```
(Factor a private `DisposeSessionAsync()` that safely tears down the gRPC session/channel + nulls the fields — reuse whatever `DisposeAsync` already does for teardown, minus marking `_disposed`.) Then in `GalaxyDriver.ReopenAsync` call `RecreateAsync` instead of `ConnectAsync` so every reopen rebuilds. Keep the supervisor's backoff loop unchanged — it now actually recovers.
**Step 4:** Run the Galaxy reconnect/session tests → green. `dotnet build` the Galaxy driver clean.
**Step 5: Commit** by explicit path.
> High-risk: driver session lifecycle + gRPC channel teardown + concurrency with the supervisor loop. Confirm `DisposeSessionAsync` is safe to call repeatedly and from the supervisor thread.
---
### Task 4: Broadcaster connection-health signal + bridges
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 1, Task 2, Task 3, Task 5
**Files:**
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Hubs/IInProcessBroadcaster.cs`
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Hubs/InProcessBroadcaster.cs` (the impl)
- Modify: the two bridge actors that publish into the broadcaster — `ScriptLogSignalRBridge` + `AlertSignalRBridge` (find under `AdminUI/` or `Runtime/`; they `Tell`/`Subscribe` the DPS topics and push to the broadcaster)
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Client.UI.Tests/` (or wherever `InProcessBroadcaster`/broadcaster tests live — find `InProcessBroadcasterTests`)
**Step 1: Failing unit test** for the new health signal: a new `InProcessBroadcaster<T>` reports `IsConnected == false` (or a sensible default); calling `SetConnected(true)` raises `ConnectionStateChanged` with `true` and flips `IsConnected`; `SetConnected(false)` flips back + raises.
**Step 2:** Run — FAIL (members don't exist).
**Step 3: Implement.** Extend the interface:
```csharp
bool IsConnected { get; }
event Action<bool>? ConnectionStateChanged;
void SetConnected(bool connected);
```
Implement in `InProcessBroadcaster<T>` (raise only on change). Then in each bridge actor, call `_broadcaster.SetConnected(true)` when its DPS `SubscribeAck` lands (subscription live) and `SetConnected(false)` on `PostStop`/failure. (Read how the bridge subscribes; mirror the `SubscribeAck` handling already there.) Default `IsConnected`: choose `true` once subscribed; before the first ack, `false` — the razor will reconcile on the `ConnectionStateChanged` event.
**Step 4:** Run the broadcaster tests → green; AdminUI builds clean (`TreatWarningsAsErrors`).
**Step 5: Commit** by explicit path.
---
### Task 5: `CorrelationId` wrapper for the alarm commands (3b)
**Classification:** small
**Estimated implement time:** ~4 min
**Parallelizable with:** Task 1, Task 2, Task 3, Task 4
**Files:**
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Commons/Messages/Admin/AcknowledgeAlarmCommand.cs` + `ShelveAlarmCommand.cs` (+ their `*Result` records)
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/AdminOperations/AdminOperationsActor.cs` (the alarm handlers)
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Clients/AdminOperationsClient.cs`
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/AdminOperationsActorTests.cs` (the 5 alarm tests)
**Context:** the alarm commands use a bare `Guid CorrelationId`; the project's other control-plane commands (`StartDeployment`/`RestartDriver`/`ReconnectDriver`) use the `CorrelationId` wrapper type. Make them consistent.
**Steps:**
1. Find the `CorrelationId` wrapper type + how `RestartDriver` uses it (`CorrelationId.NewId()`, the record field type, the reply echo).
2. Change `AcknowledgeAlarmCommand`/`ShelveAlarmCommand` (+ `*Result`) `CorrelationId` field type `Guid` → `CorrelationId`. Update `AdminOperationsClient.AcknowledgeAlarmAsync`/`ShelveAlarmAsync` to mint `CorrelationId.NewId()`. Update the `AdminOperationsActor` handlers' reply construction.
3. Update the 5 `AdminOperationsActorTests` to the wrapper type.
4. `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests --filter AdminOperations` → green; build clean.
5. **Commit** by explicit path.
---
### Task 6: `Alerts.razor` — Timed-shelve picker + chip auto-clear + live-pill
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 7
**Blocked by:** Task 4 (needs the broadcaster `ConnectionStateChanged`/`IsConnected`)
**Files:**
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Pages/Alerts.razor`
**Context:** all three changes are in one file → bundled so they don't contend. NO bUnit; proven by docker-dev live-verify (Task 8).
**Steps (no failing test — razor; verify by build + Task 8):**
1. **Timed-shelve picker:** add a small duration input to the row's shelve control (a minutes number box is simplest, default e.g. 5). Add a "Shelve (timed)" action that calls `IAdminOperationsClient.ShelveAlarmAsync(alarmId, user, ShelveKind.Timed, unshelveAtUtc: <computed now+duration>, comment: null, ct)`. Keep the existing OneShot Shelve + Unshelve buttons. Dispose the CTS (`using var cts = …`) like the existing handlers.
2. **Chip auto-clear:** after `ShowOpResult`, start a `~8s` auto-clear (mirror `DriverStatusPanel.razor`'s timer: a `Task.Delay(8000)` continuation that, if the chip is still the same one, clears `_opResult*` + `InvokeAsync(StateHasChanged)`). Read `DriverStatusPanel.razor` for the exact pattern (cancellation on a newer action).
3. **Live-pill:** replace the set-once `_connected = true` (≈ line 132) — in `OnInitializedAsync` read `Alarms.IsConnected` and subscribe `Alarms.ConnectionStateChanged += OnConnChanged;` (handler sets `_connected` + `InvokeAsync(StateHasChanged)`); unsubscribe in `Dispose`. (`Alarms` = the injected `IInProcessBroadcaster<AlarmTransitionEvent>`.)
4. `dotnet build src/Server/ZB.MOM.WW.OtOpcUa.AdminUI` → 0 warnings/errors.
5. **Commit** by explicit path.
---
### Task 7: `ScriptLog.razor` live-pill + reconnect-overlay check
**Classification:** small
**Estimated implement time:** ~3 min
**Parallelizable with:** Task 6
**Blocked by:** Task 4
**Files:**
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Pages/ScriptLog.razor` (≈ line 122)
- Read/verify (no edit unless missing): the Blazor host shell (`App.razor` / `_Host.cshtml` / `MainLayout`) for the default reconnect overlay (`components-reconnect-modal` / `<div id="components-reconnect-modal">`)
**Steps:**
1. Same live-pill fix as Task 6 step 3, against `ScriptLogs` (the injected `IInProcessBroadcaster<ScriptLogEntry>`): read `IsConnected`, subscribe `ConnectionStateChanged`, update + `StateHasChanged`, unsubscribe in `Dispose`.
2. **Verify the dead-circuit overlay exists:** grep the host shell for the Blazor reconnect modal markup. If present, note it in the commit message (the dead-circuit case is covered by Blazor's built-in reconnection UI — the component can't self-update a dead circuit). If ABSENT, add the standard `components-reconnect-modal` markup. Do NOT fake liveness from a dead component.
3. `dotnet build src/Server/ZB.MOM.WW.OtOpcUa.AdminUI` → clean.
4. **Commit** by explicit path.
---
### Task 8: Live-verify on docker-dev
**Classification:** verification
**Estimated implement time:** ~ (manual)
**Parallelizable with:** none
**Blocked by:** Task 1, Task 2, Task 3, Task 6, Task 7 (Task 5 is internal; include it in the build)
**Steps:** rebuild docker-dev central nodes on the new image (`docker compose -f docker-dev/docker-compose.yml up -d --build central-1 central-2`), then on `/alerts` (login disabled — agent drives):
1. **Double-emit fixed:** confirm each t12-overheat transition now appears **ONCE** (not twice). Cross-check both nodes' logs: only the **driver-leader (Primary)** node logs the alerts publish; the Secondary suppresses it but still writes its OPC UA node (its address space stays current). (Find the Primary via the `redundancy-state`/ServiceLevel — or just observe single rows.)
2. **Timed-shelve:** drive the new timed-shelve control → a "Shelved" transition appears (operator = `multi-role-test`); after the duration the SDK auto-unshelves (or verify the `UnshelveAtUtc` was set).
3. **Chip auto-clear:** the Ack/Shelve result chip clears after ~8s.
4. **Live-pill:** the pill shows "live" while connected; (optional) kill the bridge / restart a node and confirm the pill flips / the Blazor reconnect overlay shows.
5. **Galaxy reconnect** (best-effort, may need the gateway): if feasible, restart the Galaxy gateway and confirm the driver recreates the session + recovers without a container restart (watch the logs for a fresh `OpenSessionAsync`). If not feasible live, rely on the Task 3 unit test + note it.
Defects → new fix tasks. **Agent drives; no sign-in needed (docker-dev login disabled).**
---
### Task 9: docker-dev rig cleanup
**Classification:** trivial (operational)
**Estimated implement time:** ~ (manual)
**Parallelizable with:** none
**Blocked by:** Task 8 (the rig artifacts are the live-verify vehicle)
**Steps:** remove the seed artifacts left for verification, then redeploy:
1. In the docker-dev AdminUI (or via DB), delete the `t12-overheat` scripted alarm, the `SC-ba675b168a85` predicate script, and the `layer0-logcheck` vtag + script. Revert filler-02's `cycle-time-s` script to `return ctx.GetTag("TestMachine_002.TestDuration").Value;`.
2. Redeploy: `POST http://localhost:9200/api/deployments` with `X-Api-Key: docker-dev-deploy-key` → `202`.
3. Confirm `/alerts` + `/scripted-alarms` + `/scripts` no longer list the removed artifacts. No code commit (DB/deploy only). Note completion in Task 10's commit message.
> Operational only — no source change. If the user wants the rig kept, skip this task and record it as deferred.
---
### Task 10: Docs + finish branch
**Classification:** small
**Estimated implement time:** ~4 min
**Parallelizable with:** none
**Blocked by:** Task 9
**Files:** `docs/Redundancy.md` (alarm-emission + historization are now Primary-gated), `docs/ScriptedAlarms.md` / `docs/AlarmTracking.md` (the redundancy dedup note), `docs/Client.CLI.md` only if affected, and a one-line note wherever the live-pill/Galaxy-reconnect behaviour is documented. Keep terse.
**Steps:**
1. Update the docs above to reflect: Primary-only alarm emission + historization under redundancy; the Galaxy reconnect now recreates a faulted session; the live-pill reflects feed health.
2. Run the FULL suite: `dotnet test ZB.MOM.WW.OtOpcUa.slnx` — confirm all affected unit suites green; the only failures should be the known pre-existing macOS/integration ones (OpcUaServer.IntegrationTests PKI, Host.IntegrationTests deploy-Rejected, AbLegacy/AbCip fixtures). Verify the **Galaxy** suite is green (Task 3).
3. **Commit** docs by explicit path.
4. Run **superpowers-extended-cc:finishing-a-development-branch** (verify tests → present the 4 options → merge).
---
## Execution notes
- **Parallel dispatch:** Tasks **1, 2, 3, 4, 5** are mutually parallelizable (disjoint files across Runtime/ScriptedAlarms, Runtime/Historian, Driver.Galaxy, AdminUI/Hubs+bridges, Commons+ControlPlane). Dispatch their implementers concurrently. **6 and 7** wait on **4** (broadcaster health) and are ∥ each other (Alerts.razor vs ScriptLog.razor). **8** waits on 1/2/3/6/7; **9** on 8; **10** on 9.
- **One writer per file:** `Alerts.razor` is touched ONLY by Task 6 (its three changes are bundled for exactly this reason). `ScriptLog.razor` only by Task 7.
- **High-risk tasks (1, 2, 3):** serial spec→code review each. **Standard (4, 6):** parallel spec+code review. **Small (5, 7):** code review only.
- **Live-verify (8) is the integration checkpoint** — the single-alerts-row proof is the whole point of item 1.
- TDD where there's logic (1, 2, 3, 4, 5); razor (6, 7) proven by docker-dev `/run` (login disabled, agent drives).
@@ -0,0 +1,23 @@
{
"planPath": "docs/plans/2026-06-11-alarm-followups.md",
"designPath": "docs/plans/2026-06-11-alarm-followups-design.md",
"branch": "feat/alarm-followups",
"baseBranch": "master",
"baseSha": "bcb9f45c",
"status": "pending",
"note": "Resolves the 6 T17-T24 follow-ups. Double-emit = primary-only emission (T1 ScriptedAlarmHostActor alerts-gate + T2 HistorianAdapterActor historize-gate, both on Primary via redundancy-state; OPC UA write + commands stay ungated). T3 Galaxy reconnect recreate. T4 broadcaster health signal. T5 CorrelationId wrapper. T6 Alerts.razor (Timed picker + chip auto-clear + live-pill). T7 ScriptLog.razor live-pill. T8 live-verify. T9 rig cleanup. T10 docs+finish. T1-T5 mutually parallel; T6/T7 after T4 (∥ each other); T8 after 1/2/3/6/7; T9 after 8; T10 after 9. NO bUnit (razor proven by docker-dev /run). NO Configuration/EF change.",
"tasks": [
{"id": 235, "planTask": 0, "subject": "T0: Branch + baseline", "classification": "trivial", "status": "pending", "blockedBy": []},
{"id": 236, "planTask": 1, "subject": "T1: Redundancy emit-gate in ScriptedAlarmHostActor (A1)", "classification": "high-risk", "status": "pending", "blockedBy": [235], "parallelizableWith": [237, 238, 239, 240]},
{"id": 237, "planTask": 2, "subject": "T2: Redundancy historize-gate in HistorianAdapterActor (A2)", "classification": "high-risk", "status": "pending", "blockedBy": [235], "parallelizableWith": [236, 238, 239, 240]},
{"id": 238, "planTask": 3, "subject": "T3: Galaxy reconnect recreates a faulted session", "classification": "high-risk", "status": "pending", "blockedBy": [235], "parallelizableWith": [236, 237, 239, 240]},
{"id": 239, "planTask": 4, "subject": "T4: Broadcaster connection-health signal + bridges", "classification": "standard", "status": "pending", "blockedBy": [235], "parallelizableWith": [236, 237, 238, 240]},
{"id": 240, "planTask": 5, "subject": "T5: CorrelationId wrapper for the alarm commands (3b)", "classification": "small", "status": "pending", "blockedBy": [235], "parallelizableWith": [236, 237, 238, 239]},
{"id": 241, "planTask": 6, "subject": "T6: Alerts.razor — Timed-shelve picker + chip auto-clear + live-pill", "classification": "standard", "status": "pending", "blockedBy": [239], "parallelizableWith": [242]},
{"id": 242, "planTask": 7, "subject": "T7: ScriptLog.razor live-pill + reconnect-overlay check", "classification": "small", "status": "pending", "blockedBy": [239], "parallelizableWith": [241]},
{"id": 243, "planTask": 8, "subject": "T8: Live-verify on docker-dev", "classification": "verification", "status": "pending", "blockedBy": [236, 237, 238, 241, 242]},
{"id": 244, "planTask": 9, "subject": "T9: docker-dev rig cleanup", "classification": "trivial", "status": "pending", "blockedBy": [243]},
{"id": 245, "planTask": 10, "subject": "T10: Docs + finish branch", "classification": "small", "status": "pending", "blockedBy": [244]}
],
"lastUpdated": "2026-06-11"
}
@@ -1,3 +1,5 @@
using ZB.MOM.WW.OtOpcUa.Commons.Types;
namespace ZB.MOM.WW.OtOpcUa.Commons.Messages.Admin;
/// <summary>
@@ -16,7 +18,7 @@ public sealed record AcknowledgeAlarmCommand(
string AlarmId,
string User,
string? Comment,
Guid CorrelationId);
CorrelationId CorrelationId);
/// <summary>Reply for <see cref="AcknowledgeAlarmCommand"/>.</summary>
/// <param name="Ok">True iff the command was published without error.</param>
@@ -25,4 +27,4 @@ public sealed record AcknowledgeAlarmCommand(
public sealed record AcknowledgeAlarmResult(
bool Ok,
string? Message,
Guid CorrelationId);
CorrelationId CorrelationId);
@@ -1,3 +1,5 @@
using ZB.MOM.WW.OtOpcUa.Commons.Types;
namespace ZB.MOM.WW.OtOpcUa.Commons.Messages.Admin;
/// <summary>
@@ -40,7 +42,7 @@ public sealed record ShelveAlarmCommand(
ShelveKind Kind,
DateTime? UnshelveAtUtc,
string? Comment,
Guid CorrelationId);
CorrelationId CorrelationId);
/// <summary>Reply for <see cref="ShelveAlarmCommand"/>.</summary>
/// <param name="Ok">True iff the command was published without error.</param>
@@ -49,4 +51,4 @@ public sealed record ShelveAlarmCommand(
public sealed record ShelveAlarmResult(
bool Ok,
string? Message,
Guid CorrelationId);
CorrelationId CorrelationId);
@@ -282,15 +282,17 @@ public sealed class GalaxyDriver
}
/// <summary>
/// Reopen callback for <see cref="ReconnectSupervisor"/>: re-Register the gw session.
/// If the session never connected, this is a fresh ConnectAsync; otherwise it's a
/// reconnect against the existing client.
/// Reopen callback for <see cref="ReconnectSupervisor"/>: recreate the gw session. After a
/// gateway restart the existing session handle is faulted / stale but non-null, so a plain
/// <c>ConnectAsync</c> would no-op and the supervisor would loop forever replaying against a
/// dead session. <see cref="GalaxyMxSession.RecreateAsync"/> disposes the stale session +
/// owned client and rebuilds, so a never-connected session still opens fresh.
/// </summary>
private async Task ReopenAsync(CancellationToken cancellationToken)
{
if (_ownedMxSession is null) return;
var clientOptions = BuildClientOptions(_options.Gateway);
await _ownedMxSession.ConnectAsync(clientOptions, cancellationToken).ConfigureAwait(false);
await _ownedMxSession.RecreateAsync(clientOptions, cancellationToken).ConfigureAwait(false);
}
/// <summary>
@@ -28,6 +28,7 @@ public sealed class GalaxyMxSession : IAsyncDisposable
private MxGatewayClient? _ownedClient;
private MxGatewaySession? _session;
private int _serverHandle;
private bool _connected;
private bool _disposed;
/// <summary>Initializes a new instance of the GalaxyMxSession class.</summary>
@@ -39,8 +40,12 @@ public sealed class GalaxyMxSession : IAsyncDisposable
_logger = logger ?? NullLogger.Instance;
}
/// <summary>Gets a value indicating whether the session is connected.</summary>
public bool IsConnected => _session is not null;
/// <summary>
/// Gets a value indicating whether a session is currently established. Set <c>true</c> on a
/// successful connect/attach and <c>false</c> after teardown/dispose — tracks the same
/// <c>_connected</c> flag that <see cref="ConnectAsync"/> guards on, so the two never disagree.
/// </summary>
public bool IsConnected => _connected;
/// <summary>
/// Server-side handle returned by MXAccess <c>Register</c>. Zero before
@@ -48,24 +53,65 @@ public sealed class GalaxyMxSession : IAsyncDisposable
/// </summary>
public int ServerHandle => _serverHandle;
/// <summary>
/// Test seam — when set, replaces the real "open session + register" work so unit tests can
/// exercise the connect / recreate orchestration without a live gateway (the SDK session/client
/// types are sealed + internal-ctor and cannot be faked). Null in production = real gateway path.
/// </summary>
internal Func<CancellationToken, Task>? OpenAndRegisterOverrideForTests { get; set; }
/// <summary>
/// Connect the underlying gateway client + open an MXAccess session + register the
/// configured client name. Idempotent — second calls are no-ops while
/// <see cref="IsConnected"/> is true.
/// configured client name. Idempotent — second calls are no-ops while a connect has
/// already succeeded. Use <see cref="RecreateAsync"/> to force a rebuild after the
/// gateway restarts and leaves a faulted / stale session handle.
/// </summary>
/// <param name="clientOptions">The MX gateway client options.</param>
/// <param name="cancellationToken">The cancellation token.</param>
public async Task ConnectAsync(MxGatewayClientOptions clientOptions, CancellationToken cancellationToken)
{
ObjectDisposedException.ThrowIf(_disposed, this);
if (_session is not null) return;
if (_connected) return;
_ownedClient = MxGatewayClient.Create(clientOptions);
_session = await _ownedClient.OpenSessionAsync(cancellationToken: cancellationToken).ConfigureAwait(false);
_serverHandle = await _session.RegisterAsync(_options.ClientName, cancellationToken).ConfigureAwait(false);
_logger.LogInformation(
"GalaxyMxSession connected — clientName={ClientName} serverHandle={Handle}",
_options.ClientName, _serverHandle);
try
{
if (OpenAndRegisterOverrideForTests is not null)
{
await OpenAndRegisterOverrideForTests(cancellationToken).ConfigureAwait(false);
}
else
{
_ownedClient = MxGatewayClient.Create(clientOptions);
_session = await _ownedClient.OpenSessionAsync(cancellationToken: cancellationToken).ConfigureAwait(false);
_serverHandle = await _session.RegisterAsync(_options.ClientName, cancellationToken).ConfigureAwait(false);
}
_connected = true;
_logger.LogInformation(
"GalaxyMxSession connected — clientName={ClientName} serverHandle={Handle}",
_options.ClientName, _serverHandle);
}
catch
{
// A partial open (e.g. OpenSession succeeded but Register threw) must not leave a half-open
// handle that blocks the next reconnect. Tear the partial state down so a retry rebuilds cleanly.
await TeardownSessionAsync().ConfigureAwait(false);
throw;
}
}
/// <summary>
/// Disposes the current (faulted / stale) session + owned client and rebuilds. Use this on
/// reconnect: <see cref="ConnectAsync"/> alone no-ops while a (possibly dead) session handle is
/// still present, so the supervisor's reopen must route through here to actually re-establish.
/// </summary>
/// <param name="clientOptions">The MX gateway client options.</param>
/// <param name="cancellationToken">The cancellation token.</param>
public async Task RecreateAsync(MxGatewayClientOptions clientOptions, CancellationToken cancellationToken)
{
ObjectDisposedException.ThrowIf(_disposed, this);
await TeardownSessionAsync().ConfigureAwait(false);
await ConnectAsync(clientOptions, cancellationToken).ConfigureAwait(false);
}
/// <summary>
@@ -80,6 +126,7 @@ public sealed class GalaxyMxSession : IAsyncDisposable
ObjectDisposedException.ThrowIf(_disposed, this);
_session = session ?? throw new ArgumentNullException(nameof(session));
_serverHandle = serverHandle;
_connected = true; // an attached session means connected — keeps IsConnected + the ConnectAsync no-op guard consistent.
}
/// <summary>
@@ -93,19 +140,31 @@ public sealed class GalaxyMxSession : IAsyncDisposable
{
if (_disposed) return;
_disposed = true;
await TeardownSessionAsync().ConfigureAwait(false);
}
/// <summary>
/// Resets the connect flag and best-effort disposes + nulls the owned session and client.
/// Shared by <see cref="DisposeAsync"/> (final teardown) and <see cref="RecreateAsync"/> /
/// the partial-open recovery path (rebuild teardown) — it does NOT set <c>_disposed</c>, so
/// a torn-down session can be re-opened.
/// </summary>
private async Task TeardownSessionAsync()
{
_connected = false;
if (_session is not null)
{
try { await _session.DisposeAsync().ConfigureAwait(false); }
catch (Exception ex) { _logger.LogWarning(ex, "GalaxyMxSession session dispose failed (best-effort)"); }
_session = null;
}
_session = null;
if (_ownedClient is not null)
{
try { await _ownedClient.DisposeAsync().ConfigureAwait(false); }
catch (Exception ex) { _logger.LogWarning(ex, "GalaxyMxSession client dispose failed (best-effort)"); }
_ownedClient = null;
}
_ownedClient = null;
}
}
@@ -46,7 +46,7 @@ public sealed class AdminOperationsClient : IAdminOperationsClient
public async Task<AcknowledgeAlarmResult> AcknowledgeAlarmAsync(
string alarmId, string user, string? comment, CancellationToken ct)
{
var msg = new AcknowledgeAlarmCommand(alarmId, user, comment, Guid.NewGuid());
var msg = new AcknowledgeAlarmCommand(alarmId, user, comment, CorrelationId.NewId());
using var linked = CancellationTokenSource.CreateLinkedTokenSource(ct);
linked.CancelAfter(AskTimeout);
return await _proxy.Ask<AcknowledgeAlarmResult>(msg, AskTimeout, linked.Token);
@@ -63,7 +63,7 @@ public sealed class AdminOperationsClient : IAdminOperationsClient
public async Task<ShelveAlarmResult> ShelveAlarmAsync(
string alarmId, string user, ShelveKind kind, DateTime? unshelveAtUtc, string? comment, CancellationToken ct)
{
var msg = new ShelveAlarmCommand(alarmId, user, kind, unshelveAtUtc, comment, Guid.NewGuid());
var msg = new ShelveAlarmCommand(alarmId, user, kind, unshelveAtUtc, comment, CorrelationId.NewId());
using var linked = CancellationTokenSource.CreateLinkedTokenSource(ct);
linked.CancelAfter(AskTimeout);
return await _proxy.Ask<ShelveAlarmResult>(msg, AskTimeout, linked.Token);
@@ -13,7 +13,7 @@
@inject AuthenticationStateProvider AuthState
@inject IAuthorizationService AuthorizationService
@inject IAdminOperationsClient AdminOps
@implements IDisposable
@implements IAsyncDisposable
<div class="d-flex justify-content-between align-items-center mb-3">
<h4 class="mb-0">Alerts</h4>
@@ -87,6 +87,15 @@ else
disabled="@_busyAlarmId.Equals(e.AlarmId)"
@onclick="() => ShelveAsync(e.AlarmId, ShelveKind.OneShot)"
title="Shelve this alarm until it next clears">Shelve</button>
<input type="number" min="1" class="form-control form-control-sm" style="width:4.5rem"
disabled="@_busyAlarmId.Equals(e.AlarmId)"
@bind:get="GetShelveMinutes(e.AlarmId)" @bind:set="m => SetShelveMinutes(e.AlarmId, m)"
title="Timed-shelve duration in minutes" />
<button type="button"
class="btn btn-sm btn-outline-secondary"
disabled="@_busyAlarmId.Equals(e.AlarmId)"
@onclick="() => ShelveTimedAsync(e.AlarmId)"
title="Shelve this alarm for the chosen number of minutes">Shelve (timed)</button>
<button type="button"
class="btn btn-sm btn-outline-secondary"
disabled="@_busyAlarmId.Equals(e.AlarmId)"
@@ -123,13 +132,31 @@ else
private string? _opResultMessage;
private bool _opResultOk;
// Auto-clear timer for the per-row result chip (mirrors DriverStatusPanel): the chip is set in
// ShowOpResult and cleared 8 s later so it doesn't persist until the next action.
private System.Threading.Timer? _opResultClearTimer;
private object? _opResultClearToken;
// Per-row timed-shelve duration (minutes). Keyed by AlarmId so each row's number input is
// independent — binding every row to one shared field would couple all the inputs together.
private const int DefaultShelveMinutes = 5;
private readonly Dictionary<string, int> _shelveMinutes = new();
private int GetShelveMinutes(string alarmId) =>
_shelveMinutes.TryGetValue(alarmId, out var m) ? m : DefaultShelveMinutes;
private void SetShelveMinutes(string alarmId, int minutes) =>
_shelveMinutes[alarmId] = minutes < 1 ? 1 : minutes;
protected override async Task OnInitializedAsync()
{
// Live alarm tail straight from the in-process broadcaster (fed by AlertSignalRBridge off the
// 'alerts' DPS topic). A Blazor Server component can't self-connect a SignalR HubConnection
// behind a reverse proxy — see IInProcessBroadcaster — so we subscribe in-process instead.
// Subscribe to the health signal BEFORE reading IsConnected so a state change between the
// read and the subscribe can't be missed (TOCTOU) — mirrors DriverStatusPanel's
// SnapshotChanged-then-TryGet ordering.
Alarms.ConnectionStateChanged += OnConnectionStateChanged;
Alarms.Received += OnAlarm;
_connected = true;
_connected = Alarms.IsConnected;
// Check DriverOperator authorization so the per-row action controls only render for
// permitted users. The username is re-read at click time (GetCurrentUserNameAsync) so a
@@ -163,7 +190,12 @@ else
}
}
private async Task ShelveAsync(string alarmId, ShelveKind kind)
// Thin wrapper for the per-row "Shelve (timed)" button: shelves until now + the row's chosen
// minutes. OneShot/Unshelve call ShelveAsync directly with the default null unshelve-at.
private Task ShelveTimedAsync(string alarmId) =>
ShelveAsync(alarmId, ShelveKind.Timed, DateTime.UtcNow.AddMinutes(GetShelveMinutes(alarmId)));
private async Task ShelveAsync(string alarmId, ShelveKind kind, DateTime? unshelveAtUtc = null)
{
_busyAlarmId = alarmId;
_opResultMessage = null;
@@ -171,12 +203,9 @@ else
try
{
var user = await GetCurrentUserNameAsync();
// Timed shelve (with an unshelve-at datetime picker) is deferred — only OneShot + Unshelve
// are surfaced here, so unshelveAtUtc is always null. TimedShelve is fully wired through the
// singleton + AlarmCommand if a UI is added later.
using var cts = new System.Threading.CancellationTokenSource(TimeSpan.FromSeconds(15));
var result = await AdminOps.ShelveAlarmAsync(
alarmId, user, kind, unshelveAtUtc: null, comment: null, cts.Token);
alarmId, user, kind, unshelveAtUtc: unshelveAtUtc, comment: null, cts.Token);
var verb = kind == ShelveKind.Unshelve ? "Unshelve" : "Shelve";
ShowOpResult(alarmId, result.Ok, result.Ok ? $"{verb} sent" : (result.Message ?? "Failed"));
}
@@ -210,6 +239,22 @@ else
_opResultOk = ok;
_opResultMessage = message;
StateHasChanged();
// Auto-clear the result chip after 8 s (mirrors DriverStatusPanel). System.Threading.Timer
// is used (not System.Timers.Timer) so DisposeAsync can drain any in-flight callback.
var token = _opResultClearToken = new object();
_opResultClearTimer?.Dispose();
_opResultClearTimer = new System.Threading.Timer(_ =>
InvokeAsync(() =>
{
// Ignore a stale fire: a newer action superseded this timer's chip. (Timer.Dispose does
// not drain an already-queued callback, so without this a stale timer could clear a
// freshly-set result from a different row.)
if (!ReferenceEquals(_opResultClearToken, token)) return;
_opResultMessage = null;
_opResultAlarmId = "";
StateHasChanged();
}),
null, TimeSpan.FromSeconds(8), Timeout.InfiniteTimeSpan);
}
private void OnAlarm(AlarmTransitionEvent evt) =>
@@ -222,6 +267,12 @@ else
StateHasChanged();
});
// Raised by the broadcaster (on the bridge actor's thread) whenever the upstream DPS
// subscription's health flips. Marshal onto the circuit sync context like OnAlarm so the live
// pill re-renders safely.
private void OnConnectionStateChanged(bool connected) =>
InvokeAsync(() => { _connected = connected; StateHasChanged(); });
private async Task ClearAsync()
{
_rows.Clear();
@@ -237,5 +288,13 @@ else
_ => "chip-idle",
};
public void Dispose() => Alarms.Received -= OnAlarm;
public async ValueTask DisposeAsync()
{
// Unsubscribe BOTH broadcaster events first so the singleton can't invoke a handler on a
// disposed component, then drain the auto-clear timer (its async dispose awaits any in-flight
// callback so it can't call StateHasChanged on a gone component).
Alarms.ConnectionStateChanged -= OnConnectionStateChanged;
Alarms.Received -= OnAlarm;
if (_opResultClearTimer is not null) await _opResultClearTimer.DisposeAsync();
}
}
@@ -118,10 +118,15 @@ else
// Live tail straight from the in-process broadcaster (fed by ScriptLogSignalRBridge off the
// 'script-logs' DPS topic). Blazor Server can't self-connect a SignalR HubConnection behind
// a reverse proxy — see IInProcessBroadcaster — so we subscribe in-process instead.
// Subscribe the state-change event BEFORE reading IsConnected so no transition is missed.
ScriptLogs.ConnectionStateChanged += OnConnectionStateChanged;
ScriptLogs.Received += OnEntry;
_connected = true;
_connected = ScriptLogs.IsConnected;
}
private void OnConnectionStateChanged(bool connected) =>
InvokeAsync(() => { _connected = connected; StateHasChanged(); });
private void OnEntry(ScriptLogEntry entry) =>
// Marshal both the mutation and the re-render onto the circuit sync context so this can't
// race ClearAsync (which runs there) over the shared _rows list.
@@ -146,5 +151,9 @@ else
_ => "chip-idle",
};
public void Dispose() => ScriptLogs.Received -= OnEntry;
public void Dispose()
{
ScriptLogs.ConnectionStateChanged -= OnConnectionStateChanged;
ScriptLogs.Received -= OnEntry;
}
}
@@ -38,13 +38,22 @@ public sealed class AlertSignalRBridge : ReceiveActor
_hub = hub;
_broadcaster = broadcaster;
ReceiveAsync<AlarmTransitionEvent>(ForwardAsync);
Receive<SubscribeAck>(_ => { /* DPS confirmation */ });
// DPS subscription is now live — mark the feed connected so the Blazor "live" pill lights up.
Receive<SubscribeAck>(_ => _broadcaster.SetConnected(true));
}
/// <inheritdoc />
protected override void PreStart() =>
DistributedPubSub.Get(Context.System).Mediator.Tell(new Subscribe(TopicName, Self));
/// <inheritdoc />
protected override void PostStop()
{
// Bridge stopping — the feed is no longer live, drop the "live" pill.
_broadcaster.SetConnected(false);
base.PostStop();
}
private async Task ForwardAsync(AlarmTransitionEvent msg)
{
// In-process fan-out first — this is what the Blazor Server Alerts page reads. The hub push
@@ -26,16 +26,78 @@ public interface IInProcessBroadcaster<T>
/// <summary>Fan the item out to all current <see cref="Received"/> subscribers.</summary>
void Publish(T item);
/// <summary>
/// Whether the upstream feed (the per-node SignalR bridge's DPS subscription) is currently
/// live. Drives the "live" pill on the Blazor pages. False until the bridge's first
/// <c>SubscribeAck</c>; flips false again when the bridge stops.
/// </summary>
bool IsConnected { get; }
/// <summary>
/// Raised whenever <see cref="IsConnected"/> changes (and only on change), with the new value.
/// Handlers run on the caller's thread (the bridge actor), so Blazor subscribers must marshal
/// via <c>InvokeAsync</c>.
/// </summary>
event Action<bool>? ConnectionStateChanged;
/// <summary>
/// Set by the bridge actor from its DPS-subscription health: <c>true</c> on <c>SubscribeAck</c>
/// (subscription live), <c>false</c> on <c>PostStop</c>/failure. Raises
/// <see cref="ConnectionStateChanged"/> only when the value actually changes.
/// </summary>
/// <param name="connected">The new connection state.</param>
void SetConnected(bool connected);
}
/// <summary>Thread-safe singleton implementation of <see cref="IInProcessBroadcaster{T}"/>.</summary>
/// <typeparam name="T">The event payload type.</typeparam>
public sealed class InProcessBroadcaster<T> : IInProcessBroadcaster<T>
{
// Guards _isConnected: the bridge actor sets it on the actor thread; Blazor reads it on the
// render thread, so access must be serialised.
private readonly object _connectionLock = new();
private bool _isConnected;
/// <inheritdoc />
public event Action<T>? Received;
/// <inheritdoc />
public event Action<bool>? ConnectionStateChanged;
/// <inheritdoc />
// Capture-then-invoke (via ?.) so a concurrent unsubscribe can't null the delegate mid-raise.
public void Publish(T item) => Received?.Invoke(item);
/// <inheritdoc />
public bool IsConnected
{
get
{
lock (_connectionLock)
{
return _isConnected;
}
}
}
/// <inheritdoc />
public void SetConnected(bool connected)
{
Action<bool>? handler;
lock (_connectionLock)
{
if (_isConnected == connected)
{
return;
}
_isConnected = connected;
// Capture inside the lock, invoke outside (mirrors Publish) so a concurrent
// unsubscribe can't null the delegate mid-raise and we never hold the lock during a callback.
handler = ConnectionStateChanged;
}
handler?.Invoke(connected);
}
}
@@ -32,13 +32,22 @@ public sealed class ScriptLogSignalRBridge : ReceiveActor
_hub = hub;
_broadcaster = broadcaster;
ReceiveAsync<ScriptLogEntry>(ForwardAsync);
Receive<SubscribeAck>(_ => { /* DPS confirmation */ });
// DPS subscription is now live — mark the feed connected so the Blazor "live" pill lights up.
Receive<SubscribeAck>(_ => _broadcaster.SetConnected(true));
}
/// <inheritdoc />
protected override void PreStart() =>
DistributedPubSub.Get(Context.System).Mediator.Tell(new Subscribe(TopicName, Self));
/// <inheritdoc />
protected override void PostStop()
{
// Bridge stopping — the feed is no longer live, drop the "live" pill.
_broadcaster.SetConnected(false);
base.PostStop();
}
private async Task ForwardAsync(ScriptLogEntry msg)
{
// In-process fan-out first — this is what the Blazor Server Script log page reads. The hub
@@ -22,9 +22,18 @@ public sealed class RedundancyStateActor : ReceiveActor, IWithTimers
public const string Topic = "redundancy-state";
public static readonly TimeSpan DebounceWindow = TimeSpan.FromMilliseconds(250);
/// <summary>
/// Default interval for the periodic heartbeat re-publish. DistributedPubSub does not replay
/// to late subscribers, so a node that subscribes just after a change-driven publish would
/// otherwise never learn its role until the next membership change. The heartbeat re-publishes
/// the current snapshot so any missed subscriber converges within this interval.
/// </summary>
public static readonly TimeSpan DefaultHeartbeatInterval = TimeSpan.FromSeconds(10);
private readonly ILoggingAdapter _log = Context.GetLogger();
private readonly Akka.Cluster.Cluster _cluster;
private readonly Action<object>? _broadcastOverride;
private readonly TimeSpan _heartbeatInterval;
private bool _dirty;
/// <summary>Gets the timer scheduler for this actor.</summary>
@@ -32,19 +41,28 @@ public sealed class RedundancyStateActor : ReceiveActor, IWithTimers
/// <summary>Creates a Props for creating a RedundancyStateActor.</summary>
/// <param name="broadcast">Optional broadcast override for testing.</param>
/// <param name="heartbeatInterval">
/// Optional periodic heartbeat re-publish interval; defaults to
/// <see cref="DefaultHeartbeatInterval"/>. Injectable so tests can use a short interval.
/// </param>
/// <returns>Props for actor creation.</returns>
public static Props Props(Action<object>? broadcast = null) =>
Akka.Actor.Props.Create(() => new RedundancyStateActor(broadcast));
public static Props Props(Action<object>? broadcast = null, TimeSpan? heartbeatInterval = null) =>
Akka.Actor.Props.Create(() => new RedundancyStateActor(broadcast, heartbeatInterval));
/// <summary>Initializes a new instance of RedundancyStateActor with no broadcast override.</summary>
public RedundancyStateActor() : this(broadcast: null) { }
public RedundancyStateActor() : this(null, null) { }
/// <summary>Initializes a new instance of RedundancyStateActor.</summary>
/// <param name="broadcast">Optional broadcast override for testing.</param>
public RedundancyStateActor(Action<object>? broadcast)
/// <param name="heartbeatInterval">
/// Optional periodic heartbeat re-publish interval; defaults to
/// <see cref="DefaultHeartbeatInterval"/>.
/// </param>
public RedundancyStateActor(Action<object>? broadcast, TimeSpan? heartbeatInterval = null)
{
_cluster = Akka.Cluster.Cluster.Get(Context.System);
_broadcastOverride = broadcast;
_heartbeatInterval = heartbeatInterval ?? DefaultHeartbeatInterval;
Receive<ClusterEvent.IMemberEvent>(_ => MarkDirty());
Receive<ClusterEvent.LeaderChanged>(_ => MarkDirty());
@@ -52,6 +70,7 @@ public sealed class RedundancyStateActor : ReceiveActor, IWithTimers
Receive<ClusterEvent.CurrentClusterState>(_ => MarkDirty());
Receive<ClusterEvent.ReachabilityEvent>(_ => MarkDirty());
Receive<RecomputeNow>(_ => PublishIfDirty());
Receive<Heartbeat>(_ => { _dirty = true; PublishIfDirty(); });
}
/// <inheritdoc />
@@ -64,6 +83,11 @@ public sealed class RedundancyStateActor : ReceiveActor, IWithTimers
typeof(ClusterEvent.LeaderChanged),
typeof(ClusterEvent.RoleLeaderChanged),
typeof(ClusterEvent.ReachabilityEvent));
// DistributedPubSub does not replay to late subscribers, so re-publish the current
// snapshot on a recurring heartbeat — any node that missed the change-driven publish
// converges within _heartbeatInterval. Idempotent: consumers re-cache the same role.
Timers.StartPeriodicTimer("heartbeat", Heartbeat.Instance, _heartbeatInterval);
}
/// <inheritdoc />
@@ -104,7 +128,7 @@ public sealed class RedundancyStateActor : ReceiveActor, IWithTimers
: CommonsRedundancyRole.Detached;
list.Add(new NodeRedundancyState(
NodeId.Parse(host),
ToNodeId(member.Address),
role,
IsClusterLeader: clusterLeader == member.Address,
IsRoleLeaderForDriver: driverLeader == member.Address,
@@ -113,9 +137,30 @@ public sealed class RedundancyStateActor : ReceiveActor, IWithTimers
return list;
}
/// <summary>
/// Builds the canonical cluster node id (<c>host:port</c>) for an Akka member address — the SAME
/// format <c>ClusterRoleInfo.LocalNode</c>/<c>ToNodeId</c> use, so redundancy-state consumers
/// (OPC UA ServiceLevel, scripted-alarm emit gate, historian gate) can match their local node.
/// </summary>
/// <param name="address">The cluster member's address.</param>
/// <returns>The canonical <c>host:port</c> node id.</returns>
public static NodeId ToNodeId(Akka.Actor.Address address) =>
NodeId.Parse($"{address.Host ?? string.Empty}:{address.Port ?? 0}");
public sealed class RecomputeNow
{
public static readonly RecomputeNow Instance = new();
private RecomputeNow() { }
}
/// <summary>
/// Marker message fired by the periodic heartbeat timer. Forces a re-publish of the current
/// snapshot regardless of the change-driven <c>_dirty</c> flag, so late subscribers converge.
/// </summary>
public sealed class Heartbeat
{
/// <summary>The singleton heartbeat message instance.</summary>
public static readonly Heartbeat Instance = new();
private Heartbeat() { }
}
}
@@ -280,7 +280,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
var engine = new ScriptedAlarmEngine(
upstream, store, new ScriptLoggerFactory(_scriptRootLogger.Logger), _scriptRootLogger.Logger);
_scriptedAlarmHost = Context.ActorOf(
ScriptedAlarmHostActor.Props(_opcUaPublishActor, _dependencyMux, upstream, engine),
ScriptedAlarmHostActor.Props(_opcUaPublishActor, _dependencyMux, upstream, engine, _localNode),
"scripted-alarm-host");
}
@@ -1,6 +1,10 @@
using Akka.Actor;
using Akka.Cluster.Tools.PublishSubscribe;
using Akka.Event;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Redundancy;
using ZB.MOM.WW.OtOpcUa.Commons.Types;
using ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian;
using ZB.MOM.WW.OtOpcUa.Runtime.OpcUa;
namespace ZB.MOM.WW.OtOpcUa.Runtime.Historian;
@@ -23,22 +27,47 @@ public sealed class HistorianAdapterActor : ReceiveActor
}
private readonly IAlarmHistorianSink _sink;
private readonly NodeId? _localNode;
private readonly ILoggingAdapter _log = Context.GetLogger();
private IActorRef? _mediator;
/// <summary>Cached local <see cref="RedundancyRole"/> from the latest <see cref="RedundancyStateChanged"/>
/// snapshot (null = unknown until the first snapshot arrives, or no <see cref="_localNode"/> wired). The
/// durable sink enqueue in the <see cref="AlarmHistorianEvent"/> handler is gated on this: only the
/// Primary historizes (default-write while unknown so single-node deploys + the boot window never drop
/// historization).</summary>
private RedundancyRole? _localRole;
/// <summary>Creates the props for a HistorianAdapterActor instance.</summary>
/// <param name="sink">The alarm historian sink implementation, or null to use a null sink.</param>
/// <param name="localNode">The local cluster node id, used to read this node's <see cref="RedundancyRole"/>
/// from the <c>redundancy-state</c> topic so only the Primary historizes to the durable sink. Null (the
/// default) leaves the role unknown ⇒ default-write (single-node deploys + tests).</param>
/// <returns>Props configured for creating a HistorianAdapterActor.</returns>
public static Props Props(IAlarmHistorianSink? sink = null) =>
Akka.Actor.Props.Create(() => new HistorianAdapterActor(sink ?? NullAlarmHistorianSink.Instance));
public static Props Props(IAlarmHistorianSink? sink = null, NodeId? localNode = null) =>
Akka.Actor.Props.Create(() => new HistorianAdapterActor(sink ?? NullAlarmHistorianSink.Instance, localNode));
/// <summary>Initializes a new instance of the HistorianAdapterActor class.</summary>
/// <param name="sink">The alarm historian sink to forward enqueued events to.</param>
public HistorianAdapterActor(IAlarmHistorianSink sink)
/// <param name="localNode">The local cluster node id, used to read this node's <see cref="RedundancyRole"/>
/// from the <c>redundancy-state</c> topic so only the Primary historizes to the durable sink. Null leaves
/// the role unknown ⇒ default-write (single-node deploys + tests).</param>
public HistorianAdapterActor(IAlarmHistorianSink sink, NodeId? localNode = null)
{
_sink = sink;
_localNode = localNode;
Receive<AlarmHistorianEvent>(evt =>
{
// Warm-standby dedup (forward-looking): only the Primary historizes to the durable sink so a
// future per-node feeder writes exactly once. Default-write until told Secondary/Detached so
// single-node deploys + the boot window never drop historization. (Currently the actor has no
// production feeder — this is a defensive guard for when engine→historian wiring lands.)
if (_localRole is RedundancyRole.Secondary or RedundancyRole.Detached)
{
return;
}
// Fire-and-forget: SqliteStoreAndForwardSink persists to local SQLite synchronously
// inside EnqueueAsync (it returns once the row is committed), so we don't block on
// network/pipe latency. Failures are surfaced via GetStatus's LastError + drain state.
@@ -46,6 +75,41 @@ public sealed class HistorianAdapterActor : ReceiveActor
});
Receive<GetStatus>(_ => Sender.Tell(_sink.GetStatus()));
// Cluster redundancy snapshots (published on the `redundancy-state` topic, subscribed in PreStart)
// cache this node's role so the AlarmHistorianEvent handler can gate the durable sink enqueue to the
// Primary. The PubSub Subscribe is acked back to Self (no-op below).
Receive<RedundancyStateChanged>(OnRedundancyStateChanged);
Receive<SubscribeAck>(_ => { });
}
/// <summary>Subscribes to the <c>redundancy-state</c> topic so cluster role changes land as
/// <see cref="RedundancyStateChanged"/> and cache this node's role — the historian enqueue is gated to
/// the Primary so a future per-node feeder doesn't double-write across the warm-redundant pair.</summary>
protected override void PreStart()
{
_mediator = DistributedPubSub.Get(Context.System).Mediator;
_mediator.Tell(new Subscribe(OpcUaPublishActor.RedundancyStateTopic, Self));
base.PreStart();
}
/// <summary>Caches this node's <see cref="RedundancyRole"/> from a cluster redundancy snapshot so the
/// <see cref="AlarmHistorianEvent"/> handler can gate the durable sink enqueue to the Primary. A snapshot
/// that doesn't mention <see cref="_localNode"/> (or no local node wired) leaves the cached role unchanged
/// ⇒ default-write. Mirrors <see cref="OpcUaPublishActor"/>'s handler.</summary>
/// <param name="msg">The cluster redundancy snapshot.</param>
private void OnRedundancyStateChanged(RedundancyStateChanged msg)
{
if (_localNode is null)
{
return;
}
var local = msg.Nodes.FirstOrDefault(n => n.NodeId == _localNode.Value);
if (local is not null)
{
_localRole = local.Role;
}
}
private async Task EnqueueAsync(AlarmHistorianEvent evt)
@@ -2,7 +2,9 @@ using Akka.Actor;
using Akka.Cluster.Tools.PublishSubscribe;
using Akka.Event;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Alerts;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Redundancy;
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
using ZB.MOM.WW.OtOpcUa.Commons.Types;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms;
using ZB.MOM.WW.OtOpcUa.OpcUaServer;
@@ -95,10 +97,17 @@ public sealed class ScriptedAlarmHostActor : ReceiveActor
private readonly IActorRef? _mux;
private readonly DependencyMuxTagUpstreamSource _upstream;
private readonly ScriptedAlarmEngine _engine;
private readonly NodeId? _localNode;
private readonly ILoggingAdapter _log = Context.GetLogger();
private readonly CancellationTokenSource _cts = new();
private readonly EventHandler<ScriptedAlarmEvent> _onEngineEvent;
/// <summary>Cached local <see cref="RedundancyRole"/> from the latest <see cref="RedundancyStateChanged"/>
/// snapshot (null = unknown until the first snapshot arrives, or no <see cref="_localNode"/> wired). The
/// cluster-wide <c>alerts</c> publish in <see cref="OnEngineEmission"/> is gated on this: only the Primary
/// publishes (default-emit while unknown so single-node deploys + the boot window never drop transitions).</summary>
private RedundancyRole? _localRole;
/// <summary>Monotonic load generation, bumped on every <see cref="OnApply"/>. The continuation that
/// pipes back an <see cref="AlarmsLoaded"/> captures the generation it was started under; a stale
/// completion (an earlier generation arriving after a newer apply) is discarded in
@@ -118,23 +127,31 @@ public sealed class ScriptedAlarmHostActor : ReceiveActor
/// <param name="upstream">The mux-fed upstream the engine reads + subscribes from. MUST be the
/// same instance the <paramref name="engine"/> was constructed around.</param>
/// <param name="engine">The scripted-alarm engine this host owns + disposes.</param>
/// <param name="localNode">The local cluster node id, used to read this node's <see cref="RedundancyRole"/>
/// from the <c>redundancy-state</c> topic so only the Primary publishes the cluster-wide <c>alerts</c>
/// transition. Null (the default) leaves the role unknown ⇒ default-emit (single-node deploys + tests).</param>
public static Props Props(
IActorRef publishActor,
IActorRef? mux,
DependencyMuxTagUpstreamSource upstream,
ScriptedAlarmEngine engine) =>
Akka.Actor.Props.Create(() => new ScriptedAlarmHostActor(publishActor, mux, upstream, engine));
ScriptedAlarmEngine engine,
NodeId? localNode = null) =>
Akka.Actor.Props.Create(() => new ScriptedAlarmHostActor(publishActor, mux, upstream, engine, localNode));
/// <summary>Initializes a new instance of the <see cref="ScriptedAlarmHostActor"/> class.</summary>
/// <param name="publishActor">The OPC UA publish actor emissions are bridged to.</param>
/// <param name="mux">Optional dependency multiplexer the host registers dependency interest with.</param>
/// <param name="upstream">The mux-fed upstream the engine reads + subscribes from (same instance the engine wraps).</param>
/// <param name="engine">The scripted-alarm engine this host owns + disposes.</param>
/// <param name="localNode">The local cluster node id, used to read this node's <see cref="RedundancyRole"/>
/// from the <c>redundancy-state</c> topic so only the Primary publishes the cluster-wide <c>alerts</c>
/// transition. Null leaves the role unknown ⇒ default-emit (single-node deploys + tests).</param>
public ScriptedAlarmHostActor(
IActorRef publishActor,
IActorRef? mux,
DependencyMuxTagUpstreamSource upstream,
ScriptedAlarmEngine engine)
ScriptedAlarmEngine engine,
NodeId? localNode = null)
{
ArgumentNullException.ThrowIfNull(publishActor);
ArgumentNullException.ThrowIfNull(upstream);
@@ -143,6 +160,7 @@ public sealed class ScriptedAlarmHostActor : ReceiveActor
_mux = mux;
_upstream = upstream;
_engine = engine;
_localNode = localNode;
// OnEvent fires on the engine's worker thread. NEVER touch Context / actor state here —
// marshal onto the actor thread via the thread-safe Self.Tell. Keep the handler in a field
@@ -155,6 +173,9 @@ public sealed class ScriptedAlarmHostActor : ReceiveActor
Receive<AlarmsLoaded>(OnAlarmsLoaded);
Receive<VirtualTagActor.DependencyValueChanged>(OnDependencyChanged);
Receive<EngineEmission>(OnEngineEmission);
// Cluster redundancy snapshots (published on the `redundancy-state` topic, subscribed in PreStart)
// cache this node's role so OnEngineEmission can gate the cluster-wide alerts publish to the Primary.
Receive<RedundancyStateChanged>(OnRedundancyStateChanged);
// Inbound OPC UA Part 9 alarm method calls arrive as AlarmCommands on the cluster
// `alarm-commands` DPS topic (T18 publishes them after the AlarmAck role gate). The topic is a
// cluster-wide broadcast — every host node receives every command — so OnAlarmCommand filters to
@@ -275,9 +296,37 @@ public sealed class ScriptedAlarmHostActor : ReceiveActor
User: TransitionUser(e),
TimestampUtc: e.TimestampUtc);
// Warm-standby dedup: only the Primary (driver-role leader) publishes the cluster-wide
// transition. Default-emit until told we are Secondary/Detached so single-node deploys + the
// boot window never drop transitions. The OPC UA node write above + inbound command processing
// stay ungated (the secondary keeps its address space + engine state warm for failover).
if (_localRole is RedundancyRole.Secondary or RedundancyRole.Detached)
{
return;
}
_mediator.Tell(new Publish(AlertsTopic, evt));
}
/// <summary>Caches this node's <see cref="RedundancyRole"/> from a cluster redundancy snapshot so
/// <see cref="OnEngineEmission"/> can gate the cluster-wide <c>alerts</c> publish to the Primary. A
/// snapshot that doesn't mention <see cref="_localNode"/> (or no local node wired) leaves the cached
/// role unchanged ⇒ default-emit. Mirrors <see cref="OpcUaPublishActor"/>'s handler.</summary>
/// <param name="msg">The cluster redundancy snapshot.</param>
private void OnRedundancyStateChanged(RedundancyStateChanged msg)
{
if (_localNode is null)
{
return;
}
var local = msg.Nodes.FirstOrDefault(n => n.NodeId == _localNode.Value);
if (local is not null)
{
_localRole = local.Role;
}
}
/// <summary>
/// Drives an inbound OPC UA Part 9 alarm method call (delivered as an <see cref="AlarmCommand"/>
/// on the cluster <c>alarm-commands</c> topic) onto the matching <see cref="ScriptedAlarmEngine"/>
@@ -386,6 +435,10 @@ public sealed class ScriptedAlarmHostActor : ReceiveActor
// the node manager's condition handlers, T18) land here as AlarmCommands. The Subscribe is sent
// from Self so the SubscribeAck returns to this actor (handled as a no-op in the ctor wiring).
_mediator.Tell(new Subscribe(AlarmCommandsTopic, Self));
// Also subscribe to the `redundancy-state` topic so cluster role changes land as
// RedundancyStateChanged and cache this node's role — OnEngineEmission gates the cluster-wide
// alerts publish to the Primary so a 2-node single-cluster deploy doesn't double-emit transitions.
_mediator.Tell(new Subscribe(OpcUaPublishActor.RedundancyStateTopic, Self));
base.PreStart();
}
@@ -143,7 +143,7 @@ public static class ServiceCollectionExtensions
registry.Register<DriverHostActorKey>(driverHost);
var historian = system.ActorOf(
HistorianAdapterActor.Props(historianSink),
HistorianAdapterActor.Props(historianSink, roleInfo.LocalNode),
HistorianAdapterActorName);
registry.Register<HistorianAdapterActorKey>(historian);
});
@@ -0,0 +1,135 @@
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Config;
using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Runtime;
namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Tests.Runtime;
/// <summary>
/// Connect / recreate orchestration tests for <see cref="GalaxyMxSession"/>. The SDK
/// session/client types are sealed with internal ctors and cannot be faked, so these
/// drive the open body through the <c>OpenAndRegisterOverrideForTests</c> seam. The
/// core regression guard is <see cref="Recreate_after_connect_opens_a_fresh_session"/>:
/// a stale-session reconnect must rebuild rather than no-op (the gateway-restart bug).
/// </summary>
public sealed class GalaxyMxSessionReconnectTests
{
private static GalaxyMxAccessOptions MinimalOptions() => new(ClientName: "OtOpcUaTest");
private static GalaxyMxSession NewSession() => new(MinimalOptions());
/// <summary>A second <c>ConnectAsync</c> while connected must be a no-op (idempotent guard).</summary>
[Fact]
public async Task Connect_then_connect_again_is_a_noop()
{
var session = NewSession();
var openCount = 0;
session.OpenAndRegisterOverrideForTests = _ =>
{
openCount++;
return Task.CompletedTask;
};
await session.ConnectAsync(null!, CancellationToken.None);
openCount.ShouldBe(1);
await session.ConnectAsync(null!, CancellationToken.None);
openCount.ShouldBe(1);
}
/// <summary>
/// The regression guard for the gateway-restart bug: <c>RecreateAsync</c> must bypass
/// the no-op guard and re-run the open body even though a (stale) session is present.
/// </summary>
[Fact]
public async Task Recreate_after_connect_opens_a_fresh_session()
{
var session = NewSession();
var openCount = 0;
session.OpenAndRegisterOverrideForTests = _ =>
{
openCount++;
return Task.CompletedTask;
};
await session.ConnectAsync(null!, CancellationToken.None);
openCount.ShouldBe(1);
await session.RecreateAsync(null!, CancellationToken.None);
openCount.ShouldBe(2);
}
/// <summary><c>RecreateAsync</c> on a never-connected session still opens (teardown is a no-op first).</summary>
[Fact]
public async Task Recreate_when_never_connected_still_opens()
{
var session = NewSession();
var openCount = 0;
session.OpenAndRegisterOverrideForTests = _ =>
{
openCount++;
return Task.CompletedTask;
};
await session.RecreateAsync(null!, CancellationToken.None);
openCount.ShouldBe(1);
}
/// <summary>
/// A failed first connect must not leave <c>_connected</c> set — the next attempt has
/// to reach the open body again (partial-open teardown).
/// </summary>
[Fact]
public async Task Connect_failure_is_not_left_half_open()
{
var session = NewSession();
var openCount = 0;
session.OpenAndRegisterOverrideForTests = _ =>
{
// Throw on the first attempt, succeed on the second.
if (openCount++ == 0)
{
throw new InvalidOperationException("simulated open failure");
}
return Task.CompletedTask;
};
await Should.ThrowAsync<InvalidOperationException>(
async () => await session.ConnectAsync(null!, CancellationToken.None));
openCount.ShouldBe(1);
session.IsConnected.ShouldBeFalse(); // _connected must NOT be latched by the failed attempt.
// The failed first attempt must not have latched _connected — the retry reaches the body.
await session.ConnectAsync(null!, CancellationToken.None);
openCount.ShouldBe(2);
}
/// <summary>
/// <see cref="GalaxyMxSession.IsConnected"/> tracks the <c>_connected</c> guard across the
/// full lifecycle: false when fresh, true after connect, still true after a recreate, and
/// false again after dispose.
/// </summary>
[Fact]
public async Task IsConnected_reflects_connect_recreate_and_dispose()
{
var session = NewSession();
var openCount = 0;
session.OpenAndRegisterOverrideForTests = _ =>
{
openCount++;
return Task.CompletedTask;
};
session.IsConnected.ShouldBeFalse();
await session.ConnectAsync(null!, CancellationToken.None);
session.IsConnected.ShouldBeTrue();
await session.RecreateAsync(null!, CancellationToken.None);
session.IsConnected.ShouldBeTrue();
await session.DisposeAsync();
session.IsConnected.ShouldBeFalse();
}
}
@@ -48,4 +48,52 @@ public sealed class InProcessBroadcasterTests
var broadcaster = new InProcessBroadcaster<int>();
Should.NotThrow(() => broadcaster.Publish(42));
}
[Fact]
public void New_broadcaster_is_not_connected()
{
var broadcaster = new InProcessBroadcaster<string>();
broadcaster.IsConnected.ShouldBeFalse();
}
[Fact]
public void SetConnected_true_flips_state_and_raises_once()
{
var broadcaster = new InProcessBroadcaster<string>();
var raised = new List<bool>();
broadcaster.ConnectionStateChanged += raised.Add;
broadcaster.SetConnected(true);
broadcaster.IsConnected.ShouldBeTrue();
raised.ShouldBe([true]);
}
[Fact]
public void SetConnected_same_value_does_not_raise()
{
var broadcaster = new InProcessBroadcaster<string>();
var raised = new List<bool>();
broadcaster.ConnectionStateChanged += raised.Add;
broadcaster.SetConnected(true);
broadcaster.SetConnected(true);
raised.ShouldBe([true]);
}
[Fact]
public void SetConnected_false_after_true_raises_false()
{
var broadcaster = new InProcessBroadcaster<string>();
var raised = new List<bool>();
broadcaster.ConnectionStateChanged += raised.Add;
broadcaster.SetConnected(true);
broadcaster.SetConnected(false);
broadcaster.IsConnected.ShouldBeFalse();
raised[^1].ShouldBeFalse();
}
}
@@ -43,7 +43,7 @@ public sealed class AdminOperationsActorTests : ControlPlaneActorTestBase
var actor = Sys.ActorOf(AdminOperationsActor.Props(dbFactory, coordinator.Ref, Enumerable.Empty<IDriverProbe>()));
var topicProbe = SubscribeAlarmCommandsProbe();
var correlationId = Guid.NewGuid();
var correlationId = CorrelationId.NewId();
actor.Tell(new AcknowledgeAlarmCommand("alarm-42", "operator-jo", "looking into it", correlationId));
var published = topicProbe.ExpectMsg<AlarmCommand>(TimeSpan.FromSeconds(3));
@@ -69,7 +69,7 @@ public sealed class AdminOperationsActorTests : ControlPlaneActorTestBase
var actor = Sys.ActorOf(AdminOperationsActor.Props(dbFactory, coordinator.Ref, Enumerable.Empty<IDriverProbe>()));
var topicProbe = SubscribeAlarmCommandsProbe();
var correlationId = Guid.NewGuid();
var correlationId = CorrelationId.NewId();
actor.Tell(new ShelveAlarmCommand("alarm-7", "op-kim", ShelveKind.OneShot, UnshelveAtUtc: null, Comment: null, correlationId));
var published = topicProbe.ExpectMsg<AlarmCommand>(TimeSpan.FromSeconds(3));
@@ -94,7 +94,7 @@ public sealed class AdminOperationsActorTests : ControlPlaneActorTestBase
var topicProbe = SubscribeAlarmCommandsProbe();
var unshelveAt = DateTime.UtcNow.AddMinutes(15);
actor.Tell(new ShelveAlarmCommand("alarm-9", "op-lee", ShelveKind.Timed, unshelveAt, Comment: "maint window", Guid.NewGuid()));
actor.Tell(new ShelveAlarmCommand("alarm-9", "op-lee", ShelveKind.Timed, unshelveAt, Comment: "maint window", CorrelationId.NewId()));
var published = topicProbe.ExpectMsg<AlarmCommand>(TimeSpan.FromSeconds(3));
published.AlarmId.ShouldBe("alarm-9");
@@ -116,7 +116,7 @@ public sealed class AdminOperationsActorTests : ControlPlaneActorTestBase
var actor = Sys.ActorOf(AdminOperationsActor.Props(dbFactory, coordinator.Ref, Enumerable.Empty<IDriverProbe>()));
var topicProbe = SubscribeAlarmCommandsProbe();
actor.Tell(new ShelveAlarmCommand("alarm-3", "op-sam", ShelveKind.Unshelve, UnshelveAtUtc: null, Comment: null, Guid.NewGuid()));
actor.Tell(new ShelveAlarmCommand("alarm-3", "op-sam", ShelveKind.Unshelve, UnshelveAtUtc: null, Comment: null, CorrelationId.NewId()));
var published = topicProbe.ExpectMsg<AlarmCommand>(TimeSpan.FromSeconds(3));
published.Operation.ShouldBe("Unshelve");
@@ -135,7 +135,7 @@ public sealed class AdminOperationsActorTests : ControlPlaneActorTestBase
var actor = Sys.ActorOf(AdminOperationsActor.Props(dbFactory, coordinator.Ref, Enumerable.Empty<IDriverProbe>()));
var topicProbe = SubscribeAlarmCommandsProbe();
actor.Tell(new ShelveAlarmCommand("alarm-1", "op-zoe", ShelveKind.Timed, UnshelveAtUtc: null, Comment: null, Guid.NewGuid()));
actor.Tell(new ShelveAlarmCommand("alarm-1", "op-zoe", ShelveKind.Timed, UnshelveAtUtc: null, Comment: null, CorrelationId.NewId()));
var reply = ExpectMsg<ShelveAlarmResult>(TimeSpan.FromSeconds(3));
reply.Ok.ShouldBeFalse();
@@ -42,4 +42,53 @@ public sealed class RedundancyStateActorTests : ControlPlaneActorTestBase
// After debounce settles, no more events are fired by a quiescent cluster.
probe.ExpectNoMsg(TimeSpan.FromMilliseconds(500));
}
/// <summary>
/// Verifies the periodic heartbeat re-publishes the CURRENT snapshot even when no new cluster
/// events arrive — so a late subscriber that missed the change-driven publish converges within
/// the heartbeat interval. The first message is the self-join publish; the second proves the
/// recurring heartbeat fires with no intervening cluster event.
/// </summary>
[Fact]
public void Heartbeat_republishes_current_snapshot_without_cluster_events()
{
var probe = CreateTestProbe("heartbeat-listener");
Sys.ActorOf(
RedundancyStateActor.Props(
broadcast: msg => probe.Ref.Tell(msg),
heartbeatInterval: TimeSpan.FromMilliseconds(300)),
"redundancy-heartbeat");
// First publish: the change-driven self-join snapshot.
probe.ExpectMsg<RedundancyStateChanged>(TimeSpan.FromSeconds(2));
// Second publish: the periodic heartbeat re-publish, with NO new cluster event sent.
probe.ExpectMsg<RedundancyStateChanged>(TimeSpan.FromSeconds(2));
}
/// <summary>
/// Regression guard: the snapshot node id MUST be the canonical <c>host:port</c> form (matching
/// ClusterRoleInfo.LocalNode/ToNodeId), or every consumer's
/// <c>n.NodeId == _localNode.Value</c> match fails and no node ever learns its role.
/// </summary>
[Fact]
public void ToNodeId_uses_canonical_host_and_port()
{
var nodeId = RedundancyStateActor.ToNodeId(
new Address("akka.tcp", "otopcua", "central-2", 4053));
nodeId.Value.ShouldBe("central-2:4053");
}
/// <summary>
/// Documents the host-less/port-less fallback (<c>:0</c>). Such members are skipped by
/// BuildSnapshot's guard, but the helper must still format deterministically.
/// </summary>
[Fact]
public void ToNodeId_handles_missing_port()
{
var nodeId = RedundancyStateActor.ToNodeId(new Address("akka.tcp", "otopcua"));
nodeId.Value.ShouldBe(":0");
}
}
@@ -0,0 +1,167 @@
using Akka.Actor;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Redundancy;
using ZB.MOM.WW.OtOpcUa.Commons.Types;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian;
using ZB.MOM.WW.OtOpcUa.Runtime.Historian;
using ZB.MOM.WW.OtOpcUa.Runtime.Tests.Harness;
namespace ZB.MOM.WW.OtOpcUa.Runtime.Tests.Historian;
/// <summary>
/// TestKit coverage for <see cref="HistorianAdapterActor"/>'s Primary-only historization gate.
/// The actor caches this node's <see cref="RedundancyRole"/> from the <c>redundancy-state</c>
/// topic and SKIPS the sink enqueue when the local node is Secondary/Detached so a future
/// per-node feeder writes exactly once across the warm-redundant pair. Unknown/null role
/// default-writes (single-node deploys + the boot window must never silently drop historization).
/// </summary>
public sealed class HistorianAdapterActorTests : RuntimeActorTestBase
{
/// <summary>The local node id the gating tests construct the adapter with.</summary>
private static readonly NodeId LocalNode = new("node-A");
/// <summary>A short window we allow the fire-and-forget enqueue to land within.</summary>
private static readonly TimeSpan Settle = TimeSpan.FromMilliseconds(500);
/// <summary>Thread-safe fake sink that records every <see cref="EnqueueAsync"/> call.</summary>
private sealed class RecordingSink : IAlarmHistorianSink
{
private int _count;
/// <summary>The number of <see cref="EnqueueAsync"/> calls observed so far.</summary>
public int EnqueueCount => Volatile.Read(ref _count);
/// <inheritdoc />
public Task EnqueueAsync(AlarmHistorianEvent evt, CancellationToken cancellationToken)
{
Interlocked.Increment(ref _count);
return Task.CompletedTask;
}
/// <inheritdoc />
public HistorianSinkStatus GetStatus() => new(
QueueDepth: 0,
DeadLetterDepth: 0,
LastDrainUtc: null,
LastSuccessUtc: null,
LastError: null,
DrainState: HistorianDrainState.Idle);
}
/// <summary>Builds a minimal <see cref="AlarmHistorianEvent"/> for the gate tests.</summary>
private static AlarmHistorianEvent SampleEvent() => new(
AlarmId: "alm-1",
EquipmentPath: "Area/Line/Equip",
AlarmName: "HiHi",
AlarmTypeName: "LimitAlarm",
Severity: AlarmSeverity.High,
EventKind: "Activated",
Message: "level high",
User: "system",
Comment: null,
TimestampUtc: DateTime.UtcNow);
/// <summary>Tell <paramref name="actor"/> a <see cref="RedundancyStateChanged"/> snapshot marking
/// <see cref="LocalNode"/> with <paramref name="role"/> so the gate observes the local role.</summary>
private static void TellRedundancyRole(IActorRef actor, RedundancyRole role) =>
actor.Tell(new RedundancyStateChanged(
new[]
{
new NodeRedundancyState(
NodeId: LocalNode,
Role: role,
IsClusterLeader: role == RedundancyRole.Primary,
IsRoleLeaderForDriver: role == RedundancyRole.Primary,
AsOfUtc: DateTime.UtcNow),
},
CorrelationId.NewId()));
/// <summary>Default-write (T1): before any redundancy snapshot — the boot window and the steady
/// state for single-node deploys — the adapter MUST historize. Constructed WITH a localNode but
/// no snapshot sent, so the cached role is unknown ⇒ default-write.</summary>
[Fact]
public void Default_before_redundancy_state_historizes()
{
var sink = new RecordingSink();
var actor = Sys.ActorOf(HistorianAdapterActor.Props(sink, LocalNode));
actor.Tell(SampleEvent());
AwaitAssert(() => sink.EnqueueCount.ShouldBe(1), Settle);
}
/// <summary>Secondary suppression (T2): when the cached local role is Secondary, the adapter MUST
/// NOT enqueue to the durable sink (the Primary writes the single copy).</summary>
[Fact]
public void Secondary_node_does_not_historize()
{
var sink = new RecordingSink();
var actor = Sys.ActorOf(HistorianAdapterActor.Props(sink, LocalNode));
TellRedundancyRole(actor, RedundancyRole.Secondary);
actor.Tell(SampleEvent());
// Give the (suppressed) fire-and-forget a stable window, then assert nothing landed.
ExpectNoMsg(Settle);
sink.EnqueueCount.ShouldBe(0);
}
/// <summary>Detached suppression (T3): a Detached node likewise MUST NOT historize.</summary>
[Fact]
public void Detached_node_does_not_historize()
{
var sink = new RecordingSink();
var actor = Sys.ActorOf(HistorianAdapterActor.Props(sink, LocalNode));
TellRedundancyRole(actor, RedundancyRole.Detached);
actor.Tell(SampleEvent());
ExpectNoMsg(Settle);
sink.EnqueueCount.ShouldBe(0);
}
/// <summary>Primary writes (T4): when the cached local role is Primary, the adapter historizes as
/// normal (this is the single copy the durable sink sees).</summary>
[Fact]
public void Primary_node_historizes()
{
var sink = new RecordingSink();
var actor = Sys.ActorOf(HistorianAdapterActor.Props(sink, LocalNode));
TellRedundancyRole(actor, RedundancyRole.Primary);
actor.Tell(SampleEvent());
AwaitAssert(() => sink.EnqueueCount.ShouldBe(1), Settle);
}
/// <summary>Absent-node default-historize (T5): a snapshot that mentions only a DIFFERENT node
/// must NOT update the local cached role — the actor's own node is absent, so the role stays
/// null/unknown and the default-historize path must fire. Partial/stale snapshots MUST NOT
/// silently suppress historization for nodes not yet observed.</summary>
[Fact]
public void Redundancy_snapshot_without_local_node_leaves_role_unknown_and_historizes()
{
var sink = new RecordingSink();
var actor = Sys.ActorOf(HistorianAdapterActor.Props(sink, LocalNode));
// Send a snapshot that only describes a peer node — the local node is absent.
actor.Tell(new RedundancyStateChanged(
new[]
{
new NodeRedundancyState(
NodeId: new NodeId("some-other-node"),
Role: RedundancyRole.Secondary,
IsClusterLeader: false,
IsRoleLeaderForDriver: false,
AsOfUtc: DateTime.UtcNow),
},
CorrelationId.NewId()));
actor.Tell(SampleEvent());
// Local role is still unknown ⇒ default-historize path: sink must record exactly one enqueue.
AwaitAssert(() => sink.EnqueueCount.ShouldBe(1), Settle);
}
}
@@ -5,7 +5,9 @@ using Serilog;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Alerts;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Redundancy;
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
using ZB.MOM.WW.OtOpcUa.Commons.Types;
using ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms;
using ZB.MOM.WW.OtOpcUa.Core.Scripting;
using ZB.MOM.WW.OtOpcUa.OpcUaServer;
@@ -74,15 +76,38 @@ public sealed class ScriptedAlarmHostActorTests : RuntimeActorTestBase
return new ScriptedAlarmEngine(upstream, new InMemoryAlarmStateStore(), new ScriptLoggerFactory(logger), logger);
}
/// <summary>The local node id used by the redundancy-gating tests.</summary>
private static readonly NodeId LocalNode = new("node-A");
private (IActorRef Host, DependencyMuxTagUpstreamSource Upstream) Spawn(
TestProbe publish, TestProbe mux)
TestProbe publish, TestProbe mux, NodeId? localNode = null)
{
var upstream = new DependencyMuxTagUpstreamSource();
var engine = BuildEngine(upstream);
var host = Sys.ActorOf(ScriptedAlarmHostActor.Props(publish.Ref, mux.Ref, upstream, engine));
var host = Sys.ActorOf(ScriptedAlarmHostActor.Props(publish.Ref, mux.Ref, upstream, engine, localNode));
return (host, upstream);
}
/// <summary>Tell the host a <see cref="RedundancyStateChanged"/> snapshot marking
/// <paramref name="nodeId"/> (defaults to <see cref="LocalNode"/>) with <paramref name="role"/>
/// so the gate observes the local role. Pass a different <paramref name="nodeId"/> to produce a
/// snapshot that does NOT contain the host's own node.</summary>
private static void TellRedundancyRole(IActorRef host, RedundancyRole role, NodeId? nodeId = null)
{
var id = nodeId ?? LocalNode;
host.Tell(new RedundancyStateChanged(
new[]
{
new NodeRedundancyState(
NodeId: id,
Role: role,
IsClusterLeader: role == RedundancyRole.Primary,
IsRoleLeaderForDriver: role == RedundancyRole.Primary,
AsOfUtc: DateTime.UtcNow),
},
CorrelationId.NewId()));
}
/// <summary>Subscribe <paramref name="probe"/> to the <c>alerts</c> DPS topic and wait for the ack.
/// The Subscribe is sent FROM the probe so the SubscribeAck returns to it.</summary>
private void SubscribeToAlerts(TestProbe probe)
@@ -381,4 +406,170 @@ public sealed class ScriptedAlarmHostActorTests : RuntimeActorTestBase
var state = publish.FishForMessage<OpcUaPublishActor.AlarmStateUpdate>(m => m.State.Active, Timeout);
state.AlarmNodeId.ShouldBe("alm-1");
}
/// <summary>Default-emit (T1): before ANY RedundancyStateChanged snapshot arrives — the boot window,
/// and the steady state for single-node deploys (the sole node is always Primary) — the host MUST
/// publish the cluster-wide alerts transition. Constructed WITH a localNode but no snapshot sent, so
/// the cached local role is unknown ⇒ treated as Primary/emit.</summary>
[Fact]
public void Emission_is_published_to_alerts_by_default_before_any_redundancy_state()
{
var publish = CreateTestProbe();
var mux = CreateTestProbe();
var alerts = CreateTestProbe();
SubscribeToAlerts(alerts);
var (host, _) = Spawn(publish, mux, LocalNode);
host.Tell(new ScriptedAlarmHostActor.ApplyScriptedAlarms(new[] { Plan(severity: 800) }));
mux.ExpectMsg<DependencyMuxActor.RegisterInterest>(Timeout); // load completed
// No RedundancyStateChanged sent — local role unknown ⇒ default-emit.
host.Tell(new VirtualTagActor.DependencyValueChanged("M.T", 99, DateTime.UtcNow));
// The OPC UA node write happens AND the alerts transition is published.
publish.FishForMessage<OpcUaPublishActor.AlarmStateUpdate>(m => m.State.Active, Timeout);
var evt = alerts.ExpectMsg<AlarmTransitionEvent>(Timeout);
evt.AlarmId.ShouldBe("alm-1");
evt.TransitionKind.ShouldBe("Activated");
}
/// <summary>Secondary suppression (T1): when the cached local role is Secondary, the host MUST NOT
/// publish the cluster-wide alerts transition (the Primary publishes the single copy) — but it MUST
/// still write the local OPC UA condition node so the secondary's address space stays warm for failover.</summary>
[Fact]
public void Secondary_node_suppresses_alerts_publish_but_still_writes_opcua()
{
var publish = CreateTestProbe();
var mux = CreateTestProbe();
var alerts = CreateTestProbe();
SubscribeToAlerts(alerts);
var (host, _) = Spawn(publish, mux, LocalNode);
host.Tell(new ScriptedAlarmHostActor.ApplyScriptedAlarms(new[] { Plan(severity: 800) }));
mux.ExpectMsg<DependencyMuxActor.RegisterInterest>(Timeout); // load completed
// Mark this node Secondary, then activate.
TellRedundancyRole(host, RedundancyRole.Secondary);
host.Tell(new VirtualTagActor.DependencyValueChanged("M.T", 99, DateTime.UtcNow));
// The local OPC UA node write is UNGATED — it must still arrive.
var state = publish.FishForMessage<OpcUaPublishActor.AlarmStateUpdate>(m => m.State.Active, Timeout);
state.AlarmNodeId.ShouldBe("alm-1");
// The cluster-wide alerts publish is gated off on the secondary.
alerts.ExpectNoMsg(TimeSpan.FromMilliseconds(500));
}
/// <summary>Primary publishes (T1): when the cached local role is Primary, the host publishes the
/// cluster-wide alerts transition as normal (this is the single copy the fleet sees).</summary>
[Fact]
public void Primary_node_publishes_alerts()
{
var publish = CreateTestProbe();
var mux = CreateTestProbe();
var alerts = CreateTestProbe();
SubscribeToAlerts(alerts);
var (host, _) = Spawn(publish, mux, LocalNode);
host.Tell(new ScriptedAlarmHostActor.ApplyScriptedAlarms(new[] { Plan(severity: 800) }));
mux.ExpectMsg<DependencyMuxActor.RegisterInterest>(Timeout); // load completed
// Mark this node Primary, then activate.
TellRedundancyRole(host, RedundancyRole.Primary);
host.Tell(new VirtualTagActor.DependencyValueChanged("M.T", 99, DateTime.UtcNow));
publish.FishForMessage<OpcUaPublishActor.AlarmStateUpdate>(m => m.State.Active, Timeout);
var evt = alerts.ExpectMsg<AlarmTransitionEvent>(Timeout);
evt.AlarmId.ShouldBe("alm-1");
evt.TransitionKind.ShouldBe("Activated");
}
/// <summary>Inbound command ungated by role (T1): the alerts-publish gate must NOT affect inbound
/// command processing. Under a Secondary role, an AlarmCommand("Acknowledge") for an owned, active
/// alarm still drives the engine — observed via the resulting AlarmStateUpdate(Acknowledged=true)
/// (the OPC UA node write is ungated so the secondary's engine state + address space stay consistent).</summary>
[Fact]
public void Inbound_AlarmCommand_is_processed_regardless_of_role()
{
var publish = CreateTestProbe();
var mux = CreateTestProbe();
var (host, _) = Spawn(publish, mux, LocalNode);
host.Tell(new ScriptedAlarmHostActor.ApplyScriptedAlarms(new[] { Plan(id: "alm-1", depRef: "M.T") }));
mux.ExpectMsg<DependencyMuxActor.RegisterInterest>(Timeout); // load completed
// Mark this node Secondary — the alerts publish is gated, but command processing is NOT.
TellRedundancyRole(host, RedundancyRole.Secondary);
// Activate so there is something to acknowledge.
host.Tell(new VirtualTagActor.DependencyValueChanged("M.T", 99, DateTime.UtcNow));
publish.FishForMessage<OpcUaPublishActor.AlarmStateUpdate>(m => m.State.Active && !m.State.Acknowledged, Timeout);
// Acknowledge via the command topic — the engine must process it even on the secondary.
host.Tell(new AlarmCommand(
AlarmId: "alm-1", Operation: "Acknowledge", User: "alice", Comment: "ack-note", UnshelveAtUtc: null));
var acked = publish.FishForMessage<OpcUaPublishActor.AlarmStateUpdate>(m => m.State.Acknowledged, Timeout);
acked.AlarmNodeId.ShouldBe("alm-1");
acked.State.Acknowledged.ShouldBeTrue();
}
/// <summary>Detached suppression (A1): when the cached local role is Detached the host MUST NOT
/// publish the cluster-wide alerts transition (identical gating as Secondary) — but it MUST still
/// write the local OPC UA condition node so the detached node's address space stays warm.</summary>
[Fact]
public void Detached_node_suppresses_alerts_publish_but_still_writes_opcua()
{
var publish = CreateTestProbe();
var mux = CreateTestProbe();
var alerts = CreateTestProbe();
SubscribeToAlerts(alerts);
var (host, _) = Spawn(publish, mux, LocalNode);
host.Tell(new ScriptedAlarmHostActor.ApplyScriptedAlarms(new[] { Plan(severity: 800) }));
mux.ExpectMsg<DependencyMuxActor.RegisterInterest>(Timeout); // load completed
// Mark this node Detached, then activate.
TellRedundancyRole(host, RedundancyRole.Detached);
host.Tell(new VirtualTagActor.DependencyValueChanged("M.T", 99, DateTime.UtcNow));
// The local OPC UA node write is UNGATED — it must still arrive.
var state = publish.FishForMessage<OpcUaPublishActor.AlarmStateUpdate>(m => m.State.Active, Timeout);
state.AlarmNodeId.ShouldBe("alm-1");
// The cluster-wide alerts publish is gated off on a detached node.
alerts.ExpectNoMsg(TimeSpan.FromMilliseconds(500));
}
/// <summary>Absent-node default-emit (A1): a <see cref="RedundancyStateChanged"/> snapshot that
/// contains ONLY other nodes (the host's own <see cref="LocalNode"/> is absent) must leave the
/// cached local role unchanged (null/unknown) — the host therefore defaults to emit, publishing
/// the alerts transition AND writing the OPC UA node, exactly as in the boot-window case.</summary>
[Fact]
public void Redundancy_snapshot_without_local_node_leaves_role_unknown_and_emits()
{
var publish = CreateTestProbe();
var mux = CreateTestProbe();
var alerts = CreateTestProbe();
SubscribeToAlerts(alerts);
var (host, _) = Spawn(publish, mux, LocalNode);
host.Tell(new ScriptedAlarmHostActor.ApplyScriptedAlarms(new[] { Plan(severity: 800) }));
mux.ExpectMsg<DependencyMuxActor.RegisterInterest>(Timeout); // load completed
// Send a snapshot that mentions only a DIFFERENT node — LocalNode is absent.
// The host cannot determine its own role from this snapshot, so the cached role
// stays null (unknown) ⇒ treated as Primary ⇒ default-emit path.
TellRedundancyRole(host, RedundancyRole.Primary, nodeId: new NodeId("some-other-node"));
host.Tell(new VirtualTagActor.DependencyValueChanged("M.T", 99, DateTime.UtcNow));
// Both the OPC UA write AND the alerts publish must arrive (default-emit, role unknown).
var state = publish.FishForMessage<OpcUaPublishActor.AlarmStateUpdate>(m => m.State.Active, Timeout);
state.AlarmNodeId.ShouldBe("alm-1");
var evt = alerts.ExpectMsg<AlarmTransitionEvent>(Timeout);
evt.AlarmId.ShouldBe("alm-1");
evt.TransitionKind.ShouldBe("Activated");
}
}