Compare commits
16 Commits
bc9843d2bd
...
521ee75355
| Author | SHA1 | Date | |
|---|---|---|---|
| 521ee75355 | |||
| 7891e28b52 | |||
| e241332a24 | |||
| 23a4a0093b | |||
| 19ec656cdc | |||
| d29c933499 | |||
| f9932f2d8e | |||
| 3a0e0907e4 | |||
| 565b77e6cf | |||
| 43b96441a5 | |||
| 4f291ed09c | |||
| 0742946108 | |||
| 9ac9f0b7a9 | |||
| 06c415598c | |||
| 535787bd85 | |||
| bcb9f45cb3 |
@@ -153,6 +153,10 @@ The AdminUI `/alerts` Shelve flow was live-verified on docker-dev
|
||||
2026-06-11: singleton → topic → host actor → engine → "Shelved" status
|
||||
reflected on `/alerts` with the operator identity threaded through.
|
||||
|
||||
## Redundancy deduplication
|
||||
|
||||
Under warm/hot redundancy, both cluster nodes run `ScriptedAlarmHostActor` and the scripted-alarm engine. To prevent duplicate `/alerts` rows and duplicate historian writes, alarm transition publication to the `alerts` topic and `HistorianAdapterActor` historization are **Primary-gated**: only the node whose `RedundancyRole` is `Primary` publishes externally. OPC UA condition-node writes and inbound ack/shelve processing remain ungated on both nodes so the secondary stays warm for failover. See [Redundancy.md §Primary-gated alarm emission and historization](Redundancy.md#primary-gated-alarm-emission-and-historization).
|
||||
|
||||
## Historian write-back (non-Galaxy alarms)
|
||||
|
||||
Scripted alarms (and any future non-Galaxy `IAlarmSource` like
|
||||
|
||||
@@ -111,6 +111,8 @@ Both nodes share the same `ConfigDb` connection string; `Cluster.PublicHostname`
|
||||
|
||||
There is no longer a `Node:NodeId` setting and no `ClusterNode.RedundancyRole` column (the V2 migration dropped it — primary/secondary is now derived from cluster role-leadership). NodeId is derived as `host:port` of the cluster `PublicHostname` (see `ClusterRoleInfo.LocalNode` for the formula).
|
||||
|
||||
> **`RedundancyStateActor` NodeId consistency (fixed).** `RedundancyStateActor` now keys each node's `NodeRedundancyState` entry by the canonical `host:port` node id (via a `ToNodeId(Address)` helper mirroring `ClusterRoleInfo.ToNodeId`). Previously it keyed by `member.Address.Host` (host-only, e.g. `central-2`); since every subscriber matches by the canonical `host:port` form, the mismatch silently meant no node ever matched its own entry — all nodes stayed at the default ServiceLevel 255 and never learned their role. This fix makes `RedundancyStateActor` consistent with the stated contract above. Additionally, `RedundancyStateActor` now **re-publishes the current snapshot on a periodic heartbeat (default 10 s)** so any node that subscribes after the last topology-change publish converges within the interval (DistributedPubSub does not replay to late subscribers).
|
||||
|
||||
The `ClusterNode.ServiceLevelBase` column still exists and is editable in the Admin UI (NodeEdit / Cluster Redundancy pages), but it no longer drives the runtime ServiceLevel — that value is computed from cluster role/health and published per the mapping above, independent of this stored preference.
|
||||
|
||||
### Peer URI advertising
|
||||
@@ -133,6 +135,17 @@ Node A lists Node B's `ApplicationUri` and vice-versa. Validated by `DualEndpoin
|
||||
|
||||
There is no operator-driven role swap during a partition. Failover is what the cluster does automatically.
|
||||
|
||||
## Primary-gated alarm emission and historization
|
||||
|
||||
Under warm/hot redundancy both cluster nodes run `ScriptedAlarmHostActor` and evaluate scripted alarms, keeping each node's address space and engine state warm for instant failover. However, to avoid duplicate rows on `/alerts` and duplicate historian writes, only the Primary node publishes externally:
|
||||
|
||||
- **`alerts` topic emission** — `ScriptedAlarmHostActor` subscribes to the `redundancy-state` DPS topic and caches the local node's `RedundancyRole`. Each alarm transition is published to the cluster `alerts` topic **only when the node's role is `Primary`**. The default behaviour before any `redundancy-state` message arrives is to emit, so single-node deployments and the boot window never drop transitions. The OPC UA condition-node write and inbound ack/shelve command processing remain **ungated** on both nodes so the secondary is always ready to serve clients after a failover.
|
||||
- **`HistorianAdapterActor` historization** — likewise Primary-gated so alarm historization is exactly-once across all alarm sources. (This actor currently has no production feeder; the gate is a forward-looking guard.)
|
||||
|
||||
Net effect: each alarm transition appears **once** on `/alerts` and would historize once, not once per node.
|
||||
|
||||
See [ScriptedAlarms.md](ScriptedAlarms.md) and [AlarmTracking.md](AlarmTracking.md) for the scripted-alarm engine internals.
|
||||
|
||||
## Client-side failover
|
||||
|
||||
The OtOpcUa Client CLI at `src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI` supports `-F` / `--failover-urls` for automatic client-side failover; for long-running subscriptions the CLI monitors session KeepAlive and reconnects to the next available server, recreating the subscription on the new endpoint. See [`Client.CLI.md`](Client.CLI.md).
|
||||
|
||||
@@ -125,7 +125,9 @@ On allow, the handler publishes a `Commons.OpcUa.AlarmCommand` (containing comma
|
||||
|
||||
### AdminUI path
|
||||
|
||||
The AdminUI `/alerts` page (`Alerts.razor`) shows per-row **Acknowledge / Shelve / Unshelve** buttons. These are gated by the `DriverOperator` AdminUI policy and routed through the `AdminOperationsActor` cluster singleton (`AcknowledgeAlarmCommand` / `ShelveAlarmCommand`), which publishes onto the same `alarm-commands` topic. Cross-node routing is handled by the cluster singleton — the command always reaches the driver-role node hosting the engine that owns the alarm regardless of which AdminUI instance the operator is connected to.
|
||||
The AdminUI `/alerts` page (`Alerts.razor`) shows per-row **Acknowledge / Shelve / Unshelve** buttons. These are gated by the `DriverOperator` AdminUI policy and routed through the `AdminOperationsActor` cluster singleton (`AcknowledgeAlarmCommand` / `ShelveAlarmCommand`), which publishes onto the same `alarm-commands` topic. Cross-node routing is handled by the cluster singleton — the command always reaches the driver-role node hosting the engine that owns the alarm regardless of which AdminUI instance the operator is connected to. The Shelve button accepts a duration (minutes) and passes it to `ShelveAlarmAsync(Timed, unshelveAtUtc)`; result chips auto-clear after ~8 s. The `/alerts` and `/script-log` live-status pills reflect real feed health driven by the SignalR bridge actors' DPS subscription state.
|
||||
|
||||
Under redundancy, the `alerts` topic rows are deduplicated to the Primary node — see [Redundancy.md §Primary-gated alarm emission and historization](Redundancy.md#primary-gated-alarm-emission-and-historization).
|
||||
|
||||
### Client.CLI path
|
||||
|
||||
|
||||
@@ -90,7 +90,7 @@ Full per-field descriptions live in `Config/GalaxyDriverOptions.cs`. The full JS
|
||||
|
||||
## Reconnect + Replay
|
||||
|
||||
`ReconnectSupervisor` owns an exponential-backoff loop bounded by `Reconnect.InitialBackoffMs` / `MaxBackoffMs`. On session loss it tears down the gRPC channel, redials, and — when `ReplayOnSessionLost = true` — calls the gateway's `ReplaySubscriptions` RPC with the cached subscription set from `SubscriptionRegistry` instead of re-subscribing tag-by-tag. The gateway's worker then re-issues `AdviseSupervisory` server-side under the apartment lock.
|
||||
`ReconnectSupervisor` owns an exponential-backoff loop bounded by `Reconnect.InitialBackoffMs` / `MaxBackoffMs`. On session loss it calls `GalaxyDriver.ReopenAsync`, which invokes `GalaxyMxSession.RecreateAsync` to dispose the stale/faulted session and client before rebuilding (`OpenSessionAsync` + `RegisterAsync`). Previously `ConnectAsync` was a no-op when a stale session handle was still present, so the reopen supervisor looped forever without recovering. After a successful reopen — when `ReplayOnSessionLost = true` — the supervisor calls the gateway's `ReplaySubscriptions` RPC with the cached subscription set from `SubscriptionRegistry` instead of re-subscribing tag-by-tag. The gateway's worker then re-issues `AdviseSupervisory` server-side under the apartment lock.
|
||||
|
||||
## Testing
|
||||
|
||||
|
||||
@@ -0,0 +1,167 @@
|
||||
# Alarm Ack/Shelve Follow-ups — Design
|
||||
|
||||
**Date:** 2026-06-11
|
||||
**Status:** Approved (brainstorming) — ready for implementation plan.
|
||||
|
||||
## Goal
|
||||
|
||||
Resolve the six notes/follow-ups left open by the T17–T24 inbound-alarm-ack work
|
||||
(merged to master `bc9843d2`): the redundancy **double-emit**, the deferred Timed-shelve
|
||||
UI, two T21 minors, the docker-dev rig cleanup, and two pre-existing Layer-1 gaps (Galaxy
|
||||
reconnect, the live-pill). Scope decided in brainstorming: **everything**, including the
|
||||
pre-existing L1 gaps. Double-emit approach decided: **primary-only at the source**.
|
||||
|
||||
---
|
||||
|
||||
## 1 — Redundancy double-emit fix (core; high-risk)
|
||||
|
||||
### Problem (verified)
|
||||
Both central nodes run roles `admin,driver` in the same MAIN cluster. Each spawns its own
|
||||
`ScriptedAlarmHostActor` + `ScriptedAlarmEngine` (`DriverHostActor.SpawnScriptedAlarmHost`,
|
||||
per-node, no singleton) and, for a single-cluster artifact, loads the **same** alarms
|
||||
(`DeploymentArtifact.ParseComposition` returns `ClusterFilterMode.None` → unfiltered). Both
|
||||
independently evaluate and both hit `ScriptedAlarmHostActor.OnEngineEmission`:
|
||||
|
||||
- line 261 `_publishActor.Tell(new OpcUaPublishActor.AlarmStateUpdate(...))` — writes the OPC
|
||||
UA node (directed, local).
|
||||
- line 278 `_mediator.Tell(new Publish(AlertsTopic /* "alerts" */, evt))` — **cluster-wide,
|
||||
ungated** → every transition appears **twice** on `/alerts`. Inbound ack/shelve commands
|
||||
are likewise processed by both nodes' engines.
|
||||
|
||||
`HistorianAdapterActor` (`Runtime/Historian/`, registered **per-node** in
|
||||
`Runtime/ServiceCollectionExtensions.cs:146`, name `historian-adapter`) also consumes the
|
||||
`alerts` topic, so a single publish is historized **once per node** → double DB writes in
|
||||
production (invisible in docker-dev: `NullAlarmHistorianSink`).
|
||||
|
||||
### Existing signal to mirror
|
||||
`RedundancyStateActor` (admin singleton, `ControlPlane/Redundancy/RedundancyStateActor.cs:90-114`)
|
||||
publishes `RedundancyStateChanged` to the `redundancy-state` DPS topic. Per-node
|
||||
`NodeRedundancyState` carries `RedundancyRole` (`Primary`/`Secondary`/`Detached`) +
|
||||
`IsRoleLeaderForDriver`; **Primary = the driver-role cluster leader**. `OpcUaPublishActor`
|
||||
already subscribes to this topic (`OpcUaPublishActor.cs:30,147,156,335`) to drive its
|
||||
ServiceLevel — the exact pattern to copy (`Subscribe(RedundancyStateTopic, Self)` in
|
||||
PreStart; `Receive<RedundancyStateChanged>`; read `msg.Nodes.FirstOrDefault(n => n.NodeId ==
|
||||
localNode)?.Role`).
|
||||
|
||||
### Decision — gate **only the cluster-wide emission**, on `Primary`
|
||||
- **Emission gate (task A1):** `ScriptedAlarmHostActor` subscribes to `redundancy-state`,
|
||||
caches the local role, and **skips the `alerts` publish (line 278) when not Primary**.
|
||||
**Default = emit until a `RedundancyStateChanged` says this node is Secondary/Detached** —
|
||||
so single-node deploys (sole node is always the driver leader = Primary) and the boot
|
||||
window never drop transitions. The OPC UA node write (line 261) and **inbound command
|
||||
processing stay UNGATED** — the secondary must keep its address space warm and its engine
|
||||
state consistent for failover; clients only ever see the Primary via ServiceLevel, so the
|
||||
secondary's node writes + its (subscriber-less) condition events are harmless.
|
||||
- **Historian gate (task A2):** `HistorianAdapterActor` subscribes to `redundancy-state`,
|
||||
caches the local role, and **skips the sink write when not Primary**. Gives exactly-once
|
||||
historization for **all** alarm sources (native Galaxy/AB-CIP too), not just scripted.
|
||||
- **Edge:** a brief failover window (old Primary gone, new not yet elected) may drop a
|
||||
transition/historization — acceptable, and identical to the existing ServiceLevel handoff
|
||||
behaviour.
|
||||
|
||||
### Tests
|
||||
TestKit: with a Secondary `RedundancyStateChanged`, `OnEngineEmission` does NOT publish to
|
||||
`alerts` but DOES still `Tell` the OPC UA `AlarmStateUpdate`; with Primary (or before any
|
||||
state) it publishes. Inbound `AlarmCommand` is still processed regardless of role. Historian:
|
||||
a Secondary skips the sink write, a Primary writes (fake sink + role injection).
|
||||
|
||||
---
|
||||
|
||||
## 2 — Timed-shelve picker UI (small)
|
||||
|
||||
`Alerts.razor` currently exposes Ack / Shelve(OneShot) / Unshelve. The `Timed` backend is
|
||||
fully wired + tested (`ShelveAlarmCommand{ShelveKind.Timed, UnshelveAtUtc}` →
|
||||
`AdminOperationsActor` → `engine.TimedShelveAsync`). Add a small duration input (a minutes/
|
||||
seconds number box, or a `datetime-local`) to the shelve control on each row and call
|
||||
`ShelveAlarmAsync(kind: Timed, unshelveAtUtc: now + duration)`. Razor-only; **no backend
|
||||
change**; proven by docker-dev live-verify (no bUnit). Keep the existing OneShot/Unshelve
|
||||
buttons.
|
||||
|
||||
## 3 — T21 minors (small)
|
||||
|
||||
- **Chip auto-clear:** the `_opResult*` chip persists until the next action. Add the ~8s
|
||||
auto-clear timer pattern `DriverStatusPanel.razor` already uses (a `Task.Delay` →
|
||||
`InvokeAsync(StateHasChanged)` that clears the per-row result).
|
||||
- **CorrelationId consistency:** `AcknowledgeAlarmCommand`/`ShelveAlarmCommand` use a bare
|
||||
`Guid CorrelationId`; the project's other control-plane commands use the `CorrelationId`
|
||||
wrapper type. Switch both records + `AdminOperationsClient` (`CorrelationId.NewId()`) +
|
||||
the actor reply contract + tests to the wrapper for uniform correlation tracing.
|
||||
|
||||
## 4 — Rig cleanup (operational, last)
|
||||
|
||||
Delete the docker-dev seed artifacts left for live-verify: the `t12-overheat` scripted alarm,
|
||||
the `SC-ba675b168a85` predicate script, the `layer0-logcheck` vtag + script; revert
|
||||
filler-02's inert `cycle-time-s` logger line to `return ctx.GetTag("TestMachine_002.TestDuration").Value;`.
|
||||
Redeploy (`POST /api/deployments`, `X-Api-Key: docker-dev-deploy-key`). DB/redeploy, **not a
|
||||
code change** — done last so the rig stays available for verifying tasks 1–3/6.
|
||||
|
||||
## 5 — Galaxy reconnect recreate (high-risk)
|
||||
|
||||
### Problem (verified)
|
||||
`GalaxyMxSession.ConnectAsync` (`Driver.Galaxy/Runtime/GalaxyMxSession.cs:58-69`) is
|
||||
idempotent: `if (_session is not null) return;`. When the gateway restarts and the session
|
||||
goes Faulted/NotFound, `_session` is still a non-null (dead) handle, so `ConnectAsync` is a
|
||||
silent no-op. `GalaxyDriver.ReopenAsync` (`GalaxyDriver.cs:289`) calls it expecting a
|
||||
reconnect → no-op; `ReconnectSupervisor.RecoveryLoopAsync`
|
||||
(`Driver.Galaxy/Runtime/ReconnectSupervisor.cs:158-186`) sees reopen "succeed", proceeds to
|
||||
replay (which fails on the dead session), and **loops forever** (backoff capped 30s).
|
||||
|
||||
### Decision — recreate on reopen
|
||||
Add a recreate path to `GalaxyMxSession` (e.g. `RecreateAsync`/`DisposeSessionForRecreationAsync`)
|
||||
that disposes + nulls `_session` and `_ownedClient`, and have `ReopenAsync` call it **before**
|
||||
`ConnectAsync` so a reopen always routes through the happy-path create (`OpenSessionAsync` +
|
||||
`RegisterAsync`). Confirm what status/exception marks Faulted/NotFound and that the dispose is
|
||||
safe (gRPC channel teardown). Keep the supervisor's backoff loop; it now actually recovers.
|
||||
|
||||
### Tests
|
||||
A reconnect test asserting that after a faulted session, the reopen path **creates a new
|
||||
session** (new handle / `OpenSessionAsync` called again) rather than no-op'ing. Mirror any
|
||||
existing `ReconnectSupervisor`/session tests.
|
||||
|
||||
## 6 — Live-pill circuit health (standard)
|
||||
|
||||
### Problem (verified)
|
||||
`ScriptLog.razor:122` and `Alerts.razor:132` set `_connected = true` once in
|
||||
`OnInitialized` and never update it; the pill markup binds to that set-once bool.
|
||||
`IInProcessBroadcaster<T>` (`AdminUI/Hubs/IInProcessBroadcaster.cs`) exposes only
|
||||
`Received` + `Publish` — no health signal.
|
||||
|
||||
### Decision
|
||||
- Extend `IInProcessBroadcaster<T>` with a connection-health signal (`bool IsConnected` +
|
||||
`event … ConnectionStateChanged`). The bridge actors (`ScriptLogSignalRBridge` /
|
||||
`AlertSignalRBridge`) set it from their DPS-subscription health (SubscribeAck up / failure
|
||||
down). The razors bind the pill to it and subscribe/unsubscribe like
|
||||
`DriverStatusPanel.razor` (`OnConnectionStateChanged` → `InvokeAsync(StateHasChanged)`).
|
||||
- **Dead-circuit case** (node recreate kills the server-side circuit — the component is dead
|
||||
and cannot self-update its pill): this is Blazor Server's built-in reconnection concern.
|
||||
**Verify the default reconnect overlay is present/visible** (it is what actually signals a
|
||||
dropped circuit) rather than trying to fake liveness from a dead component. If absent, add
|
||||
the standard Blazor reconnect UI.
|
||||
|
||||
### Tests
|
||||
Broadcaster unit test for the new health signal + `SetConnected` propagation. Razor proven by
|
||||
docker-dev live-verify (no bUnit).
|
||||
|
||||
---
|
||||
|
||||
## Sequencing & risk
|
||||
|
||||
| Item | Risk | Notes |
|
||||
|---|---|---|
|
||||
| 1 redundancy double-emit (A1 emit gate, A2 historian gate) | high-risk | independent subsystem; A1∥A2 (different files) |
|
||||
| 5 Galaxy reconnect | high-risk | independent subsystem (Galaxy driver) |
|
||||
| 2 Timed-shelve picker | small | razor-only, live-verify |
|
||||
| 3a chip auto-clear / 3b CorrelationId | small | razor / mechanical refactor |
|
||||
| 6 live-pill | standard | interface + bridge + 2 razors |
|
||||
| 4 rig cleanup | trivial/operational | last |
|
||||
|
||||
The two high-risk items (1, 5) are in different subsystems and can run in parallel with each
|
||||
other and with the UI items (2, 3, 6). Rig cleanup (4) is last. TDD where there's logic; UI
|
||||
proven by docker-dev `/run` live-verify (agent drives — login disabled on docker-dev).
|
||||
|
||||
## Hard rules
|
||||
|
||||
Stage by explicit path (never `git add .`); never stage `sql_login.txt` /
|
||||
`src/Server/.../Host/pki/`; never echo the gateway API key into a **new** tracked file; no
|
||||
force-push, no `--no-verify`; **no Configuration entity / EF migration change** (none of these
|
||||
items needs one). Build on a feature branch off `master`.
|
||||
@@ -0,0 +1,291 @@
|
||||
# Alarm Ack/Shelve Follow-ups Implementation Plan
|
||||
|
||||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans (or subagent-driven-development) to implement this plan task-by-task.
|
||||
|
||||
**Goal:** Resolve the six follow-ups left by the T17–T24 inbound-alarm-ack work: the redundancy double-emit, the Timed-shelve UI, two T21 minors, the docker-dev rig cleanup, and two pre-existing Layer-1 gaps (Galaxy reconnect, the live-pill).
|
||||
|
||||
**Architecture:** The double-emit is fixed by **primary-only emission at the source** — both `ScriptedAlarmHostActor` (the `alerts` publish) and the per-node `HistorianAdapterActor` (the sink write) subscribe to the existing `redundancy-state` DPS topic, cache the local `RedundancyRole`, and act only when `Primary` (mirroring `OpcUaPublishActor`). OPC UA node writes + inbound-command processing stay ungated for warm-standby. The other five items are localized fixes (one Galaxy-driver lifecycle bug, one broadcaster-health interface extension + 2 razors, two small Alerts.razor additions, one mechanical refactor, one operational cleanup).
|
||||
|
||||
**Tech Stack:** .NET 10, Akka.NET (cluster, DistributedPubSub, TestKit/xunit2), Blazor Server (InteractiveServer, NO bUnit), xUnit + Shouldly, OPC Foundation UA .NET Standard, Serilog.
|
||||
|
||||
**Design of record:** `docs/plans/2026-06-11-alarm-followups-design.md` (committed master `bcb9f45c`).
|
||||
|
||||
**Hard rules:** stage by explicit path (never `git add .`); never stage `sql_login.txt` / `src/Server/.../Host/pki/`; never echo the gateway API key into a **new** tracked file; no force-push, no `--no-verify`; **no Configuration entity / EF migration change**. Build on a feature branch off master.
|
||||
|
||||
---
|
||||
|
||||
### Task 0: Branch + baseline
|
||||
|
||||
**Classification:** trivial
|
||||
**Estimated implement time:** ~1 min
|
||||
**Parallelizable with:** none
|
||||
|
||||
**Files:** (none — git only)
|
||||
|
||||
**Steps:**
|
||||
1. `git checkout master && git switch -c feat/alarm-followups` (off `bcb9f45c`).
|
||||
2. Confirm clean tree + green baseline build: `dotnet build ZB.MOM.WW.OtOpcUa.slnx` → 0 errors.
|
||||
3. No commit (branch only).
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Redundancy emit-gate in `ScriptedAlarmHostActor` (A1)
|
||||
|
||||
**Classification:** high-risk
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** Task 2, Task 3, Task 4, Task 5
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ScriptedAlarms/ScriptedAlarmHostActor.cs`
|
||||
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/ScriptedAlarms/ScriptedAlarmHostActorTests.cs`
|
||||
|
||||
**Context:** `OnEngineEmission` (≈ line 247–279) does two things per emission: `_publishActor.Tell(AlarmStateUpdate(...))` (line 261, OPC UA node write) and `_mediator.Tell(new Publish(AlertsTopic, evt))` (line 278, cluster-wide `alerts`). Both central nodes run this → the `alerts` publish doubles. Gate **only line 278** on `Primary`.
|
||||
|
||||
**Reference pattern to mirror** (`src/Server/ZB.MOM.WW.OtOpcUa.Runtime/OpcUa/OpcUaPublishActor.cs`): `RedundancyStateTopic = "redundancy-state"` (line 30); `Subscribe(RedundancyStateTopic, Self)` in PreStart (line 156); `Receive<RedundancyStateChanged>(HandleRedundancyStateChanged)` (line 147); `HandleRedundancyStateChanged` reads `msg.Nodes.FirstOrDefault(n => n.NodeId == localNode)?.Role` (lines 335–351). `RedundancyRole` lives in `ZB.MOM.WW.OtOpcUa.Commons.Messages.Redundancy` (`Primary`/`Secondary`/`Detached`). The host already knows its local node id (it's used elsewhere; if not, derive it the same way `OpcUaPublishActor` resolves `_localNode`).
|
||||
|
||||
**Step 1: Failing TestKit tests** (extend the existing harness; Runtime.Tests = xunit v2 + Akka.TestKit.Xunit2):
|
||||
- `Emission_is_published_to_alerts_by_default_before_any_redundancy_state` — a fresh host (no `RedundancyStateChanged` yet) DOES publish the `AlarmTransitionEvent` to `alerts` (subscribe a probe to `AlertsTopic`).
|
||||
- `Secondary_node_suppresses_alerts_publish_but_still_writes_opcua` — after a `RedundancyStateChanged` marking the local node `Secondary`, an emission does NOT publish to `alerts` but DOES still `Tell` the `OpcUaPublishActor.AlarmStateUpdate` (probe the publish-actor seam the test already uses for T9/T19).
|
||||
- `Primary_node_publishes_alerts` — after a `Primary` `RedundancyStateChanged`, the emission publishes to `alerts`.
|
||||
- `Inbound_AlarmCommand_is_processed_regardless_of_role` — a `Secondary` host still drives the engine for an inbound `AlarmCommand` (the existing T19 ack test, asserted under a Secondary role).
|
||||
|
||||
**Step 2:** Run them — expect FAIL (no gate yet; Secondary still publishes).
|
||||
|
||||
**Step 3: Implement.** Add a cached `RedundancyRole? _localRole = null;` (null = unknown ⇒ treat as Primary/emit). In PreStart, `_mediator.Tell(new Subscribe(OpcUaPublishActor.RedundancyStateTopic, Self))` (reuse the const; if cross-project reference is awkward, introduce a shared `Commons` const for `"redundancy-state"` and point both at it — surface that deviation). Add `Receive<RedundancyStateChanged>` + a `SubscribeAck` no-op. In the handler, set `_localRole` from the snapshot for the local node. In `OnEngineEmission`, before line 278:
|
||||
```csharp
|
||||
// Warm-standby dedup: only the Primary (driver-role leader) publishes the cluster-wide
|
||||
// transition + drives historization. Default-emit until told we are Secondary/Detached so
|
||||
// single-node deploys + the boot window never drop transitions. The OPC UA node write
|
||||
// above (warm address space) and inbound command processing stay ungated.
|
||||
if (_localRole is RedundancyRole.Secondary or RedundancyRole.Detached)
|
||||
return;
|
||||
_mediator.Tell(new Publish(AlertsTopic, evt));
|
||||
```
|
||||
Leave line 261 (the `_publishActor.Tell`) and `OnAlarmCommand` untouched.
|
||||
|
||||
**Step 4:** Run `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests --filter ScriptedAlarmHostActor` → all green.
|
||||
|
||||
**Step 5: Commit** by explicit path (`ScriptedAlarmHostActor.cs` + the test).
|
||||
|
||||
> High-risk: concurrency (role cached on the actor thread, read in the emission path which is also marshalled onto the actor thread — confirm `OnEngineEmission` runs on the actor thread, it does via `Self.Tell(EngineEmission)`) + redundancy semantics. Do NOT gate the OPC UA write or commands.
|
||||
|
||||
---
|
||||
|
||||
### Task 2: Redundancy historize-gate in `HistorianAdapterActor` (A2)
|
||||
|
||||
**Classification:** high-risk
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** Task 1, Task 3, Task 4, Task 5
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/HistorianAdapterActor.cs`
|
||||
- Modify (only if the actor needs the local node id / mediator wired): `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs` (≈ line 146, where it's spawned `historian-adapter`)
|
||||
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Historian/HistorianAdapterActorTests.cs` (create if absent; else extend the existing historian test)
|
||||
|
||||
**Context:** `HistorianAdapterActor` is **per-node** and consumes the `alerts` topic, so one publish historizes once per node → double DB writes. Gate the sink write on `Primary`, same pattern as Task 1.
|
||||
|
||||
**Step 1: Failing tests** (TestKit + a fake `IAlarmHistorianSink` that records writes):
|
||||
- `Default_before_redundancy_state_historizes` — a fresh adapter writes to the sink (treat unknown as Primary).
|
||||
- `Secondary_node_does_not_historize` — after a `Secondary` `RedundancyStateChanged`, an incoming alarm event is NOT written to the sink.
|
||||
- `Primary_node_historizes` — after `Primary`, it IS written.
|
||||
|
||||
**Step 2:** Run — FAIL.
|
||||
|
||||
**Step 3: Implement** the same `_localRole` cache + `redundancy-state` subscription + `Receive<RedundancyStateChanged>` as Task 1, and guard the sink write: `if (_localRole is RedundancyRole.Secondary or RedundancyRole.Detached) return;` before `_sink.Write...`. The actor will need its local node id; resolve it the way `OpcUaPublishActor`/`ServiceCollectionExtensions` already does (pass it into `Props` if not already available — check how `OpcUaPublishActor` gets `_localNode` and mirror; thread it through `HistorianAdapterActor.Props` + the registration at `ServiceCollectionExtensions.cs:146`).
|
||||
|
||||
**Step 4:** `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests --filter Historian` → green. Also confirm the full Runtime.Tests still pass.
|
||||
|
||||
**Step 5: Commit** by explicit path.
|
||||
|
||||
> High-risk: data-historization correctness + concurrency. If wiring the node id into `Props` ripples beyond the two files, surface it before expanding.
|
||||
|
||||
---
|
||||
|
||||
### Task 3: Galaxy reconnect recreates a faulted session
|
||||
|
||||
**Classification:** high-risk
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** Task 1, Task 2, Task 4, Task 5
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/Runtime/GalaxyMxSession.cs` (≈ line 58–69)
|
||||
- Modify: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/GalaxyDriver.cs` (≈ line 289, `ReopenAsync`)
|
||||
- Test: the Galaxy driver test project (find `*ReconnectSupervisor*` / `*Session*` tests under `tests/Drivers/...Driver.Galaxy.Tests/`; mirror their seam)
|
||||
|
||||
**Context:** `GalaxyMxSession.ConnectAsync` has `if (_session is not null) return;` — so when the gRPC session is Faulted/NotFound the field is still a non-null dead handle and `ConnectAsync` is a silent no-op. `GalaxyDriver.ReopenAsync` calls `ConnectAsync` expecting a reconnect → no-op → `ReconnectSupervisor.RecoveryLoopAsync` (`Runtime/ReconnectSupervisor.cs:158-186`) sees reopen "succeed", replay fails, loops forever.
|
||||
|
||||
**Step 1: Failing test.** Assert the reconnect path **creates a new session** after a faulted one: drive `GalaxyMxSession` (or a fake `MxGatewayClient`/session seam) to a connected state, simulate fault/dispose, call the reopen/recreate path, and assert `OpenSessionAsync` + `RegisterAsync` were invoked **again** (a second create), not a no-op. Mirror the existing session/reconnect test construction (read how they fake `MxGatewayClient`/`OpenSessionAsync`).
|
||||
|
||||
**Step 2:** Run — FAIL (current code no-ops, second create never happens).
|
||||
|
||||
**Step 3: Implement.** Add a recreate path to `GalaxyMxSession`, e.g.:
|
||||
```csharp
|
||||
/// <summary>Disposes the current (faulted/stale) session + owned client so the next
|
||||
/// <see cref="ConnectAsync"/> rebuilds a fresh session instead of no-op'ing on the dead handle.</summary>
|
||||
public async Task RecreateAsync(MxGatewayClientOptions clientOptions, CancellationToken ct)
|
||||
{
|
||||
ObjectDisposedException.ThrowIf(_disposed, this);
|
||||
await DisposeSessionAsync().ConfigureAwait(false); // dispose+null _session, _serverHandle, _ownedClient
|
||||
await ConnectAsync(clientOptions, ct).ConfigureAwait(false);
|
||||
}
|
||||
```
|
||||
(Factor a private `DisposeSessionAsync()` that safely tears down the gRPC session/channel + nulls the fields — reuse whatever `DisposeAsync` already does for teardown, minus marking `_disposed`.) Then in `GalaxyDriver.ReopenAsync` call `RecreateAsync` instead of `ConnectAsync` so every reopen rebuilds. Keep the supervisor's backoff loop unchanged — it now actually recovers.
|
||||
|
||||
**Step 4:** Run the Galaxy reconnect/session tests → green. `dotnet build` the Galaxy driver clean.
|
||||
|
||||
**Step 5: Commit** by explicit path.
|
||||
|
||||
> High-risk: driver session lifecycle + gRPC channel teardown + concurrency with the supervisor loop. Confirm `DisposeSessionAsync` is safe to call repeatedly and from the supervisor thread.
|
||||
|
||||
---
|
||||
|
||||
### Task 4: Broadcaster connection-health signal + bridges
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** Task 1, Task 2, Task 3, Task 5
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Hubs/IInProcessBroadcaster.cs`
|
||||
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Hubs/InProcessBroadcaster.cs` (the impl)
|
||||
- Modify: the two bridge actors that publish into the broadcaster — `ScriptLogSignalRBridge` + `AlertSignalRBridge` (find under `AdminUI/` or `Runtime/`; they `Tell`/`Subscribe` the DPS topics and push to the broadcaster)
|
||||
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Client.UI.Tests/` (or wherever `InProcessBroadcaster`/broadcaster tests live — find `InProcessBroadcasterTests`)
|
||||
|
||||
**Step 1: Failing unit test** for the new health signal: a new `InProcessBroadcaster<T>` reports `IsConnected == false` (or a sensible default); calling `SetConnected(true)` raises `ConnectionStateChanged` with `true` and flips `IsConnected`; `SetConnected(false)` flips back + raises.
|
||||
|
||||
**Step 2:** Run — FAIL (members don't exist).
|
||||
|
||||
**Step 3: Implement.** Extend the interface:
|
||||
```csharp
|
||||
bool IsConnected { get; }
|
||||
event Action<bool>? ConnectionStateChanged;
|
||||
void SetConnected(bool connected);
|
||||
```
|
||||
Implement in `InProcessBroadcaster<T>` (raise only on change). Then in each bridge actor, call `_broadcaster.SetConnected(true)` when its DPS `SubscribeAck` lands (subscription live) and `SetConnected(false)` on `PostStop`/failure. (Read how the bridge subscribes; mirror the `SubscribeAck` handling already there.) Default `IsConnected`: choose `true` once subscribed; before the first ack, `false` — the razor will reconcile on the `ConnectionStateChanged` event.
|
||||
|
||||
**Step 4:** Run the broadcaster tests → green; AdminUI builds clean (`TreatWarningsAsErrors`).
|
||||
|
||||
**Step 5: Commit** by explicit path.
|
||||
|
||||
---
|
||||
|
||||
### Task 5: `CorrelationId` wrapper for the alarm commands (3b)
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** Task 1, Task 2, Task 3, Task 4
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Commons/Messages/Admin/AcknowledgeAlarmCommand.cs` + `ShelveAlarmCommand.cs` (+ their `*Result` records)
|
||||
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/AdminOperations/AdminOperationsActor.cs` (the alarm handlers)
|
||||
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Clients/AdminOperationsClient.cs`
|
||||
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/AdminOperationsActorTests.cs` (the 5 alarm tests)
|
||||
|
||||
**Context:** the alarm commands use a bare `Guid CorrelationId`; the project's other control-plane commands (`StartDeployment`/`RestartDriver`/`ReconnectDriver`) use the `CorrelationId` wrapper type. Make them consistent.
|
||||
|
||||
**Steps:**
|
||||
1. Find the `CorrelationId` wrapper type + how `RestartDriver` uses it (`CorrelationId.NewId()`, the record field type, the reply echo).
|
||||
2. Change `AcknowledgeAlarmCommand`/`ShelveAlarmCommand` (+ `*Result`) `CorrelationId` field type `Guid` → `CorrelationId`. Update `AdminOperationsClient.AcknowledgeAlarmAsync`/`ShelveAlarmAsync` to mint `CorrelationId.NewId()`. Update the `AdminOperationsActor` handlers' reply construction.
|
||||
3. Update the 5 `AdminOperationsActorTests` to the wrapper type.
|
||||
4. `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests --filter AdminOperations` → green; build clean.
|
||||
5. **Commit** by explicit path.
|
||||
|
||||
---
|
||||
|
||||
### Task 6: `Alerts.razor` — Timed-shelve picker + chip auto-clear + live-pill
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** Task 7
|
||||
**Blocked by:** Task 4 (needs the broadcaster `ConnectionStateChanged`/`IsConnected`)
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Pages/Alerts.razor`
|
||||
|
||||
**Context:** all three changes are in one file → bundled so they don't contend. NO bUnit; proven by docker-dev live-verify (Task 8).
|
||||
|
||||
**Steps (no failing test — razor; verify by build + Task 8):**
|
||||
1. **Timed-shelve picker:** add a small duration input to the row's shelve control (a minutes number box is simplest, default e.g. 5). Add a "Shelve (timed)" action that calls `IAdminOperationsClient.ShelveAlarmAsync(alarmId, user, ShelveKind.Timed, unshelveAtUtc: <computed now+duration>, comment: null, ct)`. Keep the existing OneShot Shelve + Unshelve buttons. Dispose the CTS (`using var cts = …`) like the existing handlers.
|
||||
2. **Chip auto-clear:** after `ShowOpResult`, start a `~8s` auto-clear (mirror `DriverStatusPanel.razor`'s timer: a `Task.Delay(8000)` continuation that, if the chip is still the same one, clears `_opResult*` + `InvokeAsync(StateHasChanged)`). Read `DriverStatusPanel.razor` for the exact pattern (cancellation on a newer action).
|
||||
3. **Live-pill:** replace the set-once `_connected = true` (≈ line 132) — in `OnInitializedAsync` read `Alarms.IsConnected` and subscribe `Alarms.ConnectionStateChanged += OnConnChanged;` (handler sets `_connected` + `InvokeAsync(StateHasChanged)`); unsubscribe in `Dispose`. (`Alarms` = the injected `IInProcessBroadcaster<AlarmTransitionEvent>`.)
|
||||
4. `dotnet build src/Server/ZB.MOM.WW.OtOpcUa.AdminUI` → 0 warnings/errors.
|
||||
5. **Commit** by explicit path.
|
||||
|
||||
---
|
||||
|
||||
### Task 7: `ScriptLog.razor` live-pill + reconnect-overlay check
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~3 min
|
||||
**Parallelizable with:** Task 6
|
||||
**Blocked by:** Task 4
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Pages/ScriptLog.razor` (≈ line 122)
|
||||
- Read/verify (no edit unless missing): the Blazor host shell (`App.razor` / `_Host.cshtml` / `MainLayout`) for the default reconnect overlay (`components-reconnect-modal` / `<div id="components-reconnect-modal">`)
|
||||
|
||||
**Steps:**
|
||||
1. Same live-pill fix as Task 6 step 3, against `ScriptLogs` (the injected `IInProcessBroadcaster<ScriptLogEntry>`): read `IsConnected`, subscribe `ConnectionStateChanged`, update + `StateHasChanged`, unsubscribe in `Dispose`.
|
||||
2. **Verify the dead-circuit overlay exists:** grep the host shell for the Blazor reconnect modal markup. If present, note it in the commit message (the dead-circuit case is covered by Blazor's built-in reconnection UI — the component can't self-update a dead circuit). If ABSENT, add the standard `components-reconnect-modal` markup. Do NOT fake liveness from a dead component.
|
||||
3. `dotnet build src/Server/ZB.MOM.WW.OtOpcUa.AdminUI` → clean.
|
||||
4. **Commit** by explicit path.
|
||||
|
||||
---
|
||||
|
||||
### Task 8: Live-verify on docker-dev
|
||||
|
||||
**Classification:** verification
|
||||
**Estimated implement time:** ~ (manual)
|
||||
**Parallelizable with:** none
|
||||
**Blocked by:** Task 1, Task 2, Task 3, Task 6, Task 7 (Task 5 is internal; include it in the build)
|
||||
|
||||
**Steps:** rebuild docker-dev central nodes on the new image (`docker compose -f docker-dev/docker-compose.yml up -d --build central-1 central-2`), then on `/alerts` (login disabled — agent drives):
|
||||
1. **Double-emit fixed:** confirm each t12-overheat transition now appears **ONCE** (not twice). Cross-check both nodes' logs: only the **driver-leader (Primary)** node logs the alerts publish; the Secondary suppresses it but still writes its OPC UA node (its address space stays current). (Find the Primary via the `redundancy-state`/ServiceLevel — or just observe single rows.)
|
||||
2. **Timed-shelve:** drive the new timed-shelve control → a "Shelved" transition appears (operator = `multi-role-test`); after the duration the SDK auto-unshelves (or verify the `UnshelveAtUtc` was set).
|
||||
3. **Chip auto-clear:** the Ack/Shelve result chip clears after ~8s.
|
||||
4. **Live-pill:** the pill shows "live" while connected; (optional) kill the bridge / restart a node and confirm the pill flips / the Blazor reconnect overlay shows.
|
||||
5. **Galaxy reconnect** (best-effort, may need the gateway): if feasible, restart the Galaxy gateway and confirm the driver recreates the session + recovers without a container restart (watch the logs for a fresh `OpenSessionAsync`). If not feasible live, rely on the Task 3 unit test + note it.
|
||||
Defects → new fix tasks. **Agent drives; no sign-in needed (docker-dev login disabled).**
|
||||
|
||||
---
|
||||
|
||||
### Task 9: docker-dev rig cleanup
|
||||
|
||||
**Classification:** trivial (operational)
|
||||
**Estimated implement time:** ~ (manual)
|
||||
**Parallelizable with:** none
|
||||
**Blocked by:** Task 8 (the rig artifacts are the live-verify vehicle)
|
||||
|
||||
**Steps:** remove the seed artifacts left for verification, then redeploy:
|
||||
1. In the docker-dev AdminUI (or via DB), delete the `t12-overheat` scripted alarm, the `SC-ba675b168a85` predicate script, and the `layer0-logcheck` vtag + script. Revert filler-02's `cycle-time-s` script to `return ctx.GetTag("TestMachine_002.TestDuration").Value;`.
|
||||
2. Redeploy: `POST http://localhost:9200/api/deployments` with `X-Api-Key: docker-dev-deploy-key` → `202`.
|
||||
3. Confirm `/alerts` + `/scripted-alarms` + `/scripts` no longer list the removed artifacts. No code commit (DB/deploy only). Note completion in Task 10's commit message.
|
||||
|
||||
> Operational only — no source change. If the user wants the rig kept, skip this task and record it as deferred.
|
||||
|
||||
---
|
||||
|
||||
### Task 10: Docs + finish branch
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** none
|
||||
**Blocked by:** Task 9
|
||||
|
||||
**Files:** `docs/Redundancy.md` (alarm-emission + historization are now Primary-gated), `docs/ScriptedAlarms.md` / `docs/AlarmTracking.md` (the redundancy dedup note), `docs/Client.CLI.md` only if affected, and a one-line note wherever the live-pill/Galaxy-reconnect behaviour is documented. Keep terse.
|
||||
|
||||
**Steps:**
|
||||
1. Update the docs above to reflect: Primary-only alarm emission + historization under redundancy; the Galaxy reconnect now recreates a faulted session; the live-pill reflects feed health.
|
||||
2. Run the FULL suite: `dotnet test ZB.MOM.WW.OtOpcUa.slnx` — confirm all affected unit suites green; the only failures should be the known pre-existing macOS/integration ones (OpcUaServer.IntegrationTests PKI, Host.IntegrationTests deploy-Rejected, AbLegacy/AbCip fixtures). Verify the **Galaxy** suite is green (Task 3).
|
||||
3. **Commit** docs by explicit path.
|
||||
4. Run **superpowers-extended-cc:finishing-a-development-branch** (verify tests → present the 4 options → merge).
|
||||
|
||||
---
|
||||
|
||||
## Execution notes
|
||||
|
||||
- **Parallel dispatch:** Tasks **1, 2, 3, 4, 5** are mutually parallelizable (disjoint files across Runtime/ScriptedAlarms, Runtime/Historian, Driver.Galaxy, AdminUI/Hubs+bridges, Commons+ControlPlane). Dispatch their implementers concurrently. **6 and 7** wait on **4** (broadcaster health) and are ∥ each other (Alerts.razor vs ScriptLog.razor). **8** waits on 1/2/3/6/7; **9** on 8; **10** on 9.
|
||||
- **One writer per file:** `Alerts.razor` is touched ONLY by Task 6 (its three changes are bundled for exactly this reason). `ScriptLog.razor` only by Task 7.
|
||||
- **High-risk tasks (1, 2, 3):** serial spec→code review each. **Standard (4, 6):** parallel spec+code review. **Small (5, 7):** code review only.
|
||||
- **Live-verify (8) is the integration checkpoint** — the single-alerts-row proof is the whole point of item 1.
|
||||
- TDD where there's logic (1, 2, 3, 4, 5); razor (6, 7) proven by docker-dev `/run` (login disabled, agent drives).
|
||||
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"planPath": "docs/plans/2026-06-11-alarm-followups.md",
|
||||
"designPath": "docs/plans/2026-06-11-alarm-followups-design.md",
|
||||
"branch": "feat/alarm-followups",
|
||||
"baseBranch": "master",
|
||||
"baseSha": "bcb9f45c",
|
||||
"status": "pending",
|
||||
"note": "Resolves the 6 T17-T24 follow-ups. Double-emit = primary-only emission (T1 ScriptedAlarmHostActor alerts-gate + T2 HistorianAdapterActor historize-gate, both on Primary via redundancy-state; OPC UA write + commands stay ungated). T3 Galaxy reconnect recreate. T4 broadcaster health signal. T5 CorrelationId wrapper. T6 Alerts.razor (Timed picker + chip auto-clear + live-pill). T7 ScriptLog.razor live-pill. T8 live-verify. T9 rig cleanup. T10 docs+finish. T1-T5 mutually parallel; T6/T7 after T4 (∥ each other); T8 after 1/2/3/6/7; T9 after 8; T10 after 9. NO bUnit (razor proven by docker-dev /run). NO Configuration/EF change.",
|
||||
"tasks": [
|
||||
{"id": 235, "planTask": 0, "subject": "T0: Branch + baseline", "classification": "trivial", "status": "pending", "blockedBy": []},
|
||||
{"id": 236, "planTask": 1, "subject": "T1: Redundancy emit-gate in ScriptedAlarmHostActor (A1)", "classification": "high-risk", "status": "pending", "blockedBy": [235], "parallelizableWith": [237, 238, 239, 240]},
|
||||
{"id": 237, "planTask": 2, "subject": "T2: Redundancy historize-gate in HistorianAdapterActor (A2)", "classification": "high-risk", "status": "pending", "blockedBy": [235], "parallelizableWith": [236, 238, 239, 240]},
|
||||
{"id": 238, "planTask": 3, "subject": "T3: Galaxy reconnect recreates a faulted session", "classification": "high-risk", "status": "pending", "blockedBy": [235], "parallelizableWith": [236, 237, 239, 240]},
|
||||
{"id": 239, "planTask": 4, "subject": "T4: Broadcaster connection-health signal + bridges", "classification": "standard", "status": "pending", "blockedBy": [235], "parallelizableWith": [236, 237, 238, 240]},
|
||||
{"id": 240, "planTask": 5, "subject": "T5: CorrelationId wrapper for the alarm commands (3b)", "classification": "small", "status": "pending", "blockedBy": [235], "parallelizableWith": [236, 237, 238, 239]},
|
||||
{"id": 241, "planTask": 6, "subject": "T6: Alerts.razor — Timed-shelve picker + chip auto-clear + live-pill", "classification": "standard", "status": "pending", "blockedBy": [239], "parallelizableWith": [242]},
|
||||
{"id": 242, "planTask": 7, "subject": "T7: ScriptLog.razor live-pill + reconnect-overlay check", "classification": "small", "status": "pending", "blockedBy": [239], "parallelizableWith": [241]},
|
||||
{"id": 243, "planTask": 8, "subject": "T8: Live-verify on docker-dev", "classification": "verification", "status": "pending", "blockedBy": [236, 237, 238, 241, 242]},
|
||||
{"id": 244, "planTask": 9, "subject": "T9: docker-dev rig cleanup", "classification": "trivial", "status": "pending", "blockedBy": [243]},
|
||||
{"id": 245, "planTask": 10, "subject": "T10: Docs + finish branch", "classification": "small", "status": "pending", "blockedBy": [244]}
|
||||
],
|
||||
"lastUpdated": "2026-06-11"
|
||||
}
|
||||
@@ -1,3 +1,5 @@
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Types;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Commons.Messages.Admin;
|
||||
|
||||
/// <summary>
|
||||
@@ -16,7 +18,7 @@ public sealed record AcknowledgeAlarmCommand(
|
||||
string AlarmId,
|
||||
string User,
|
||||
string? Comment,
|
||||
Guid CorrelationId);
|
||||
CorrelationId CorrelationId);
|
||||
|
||||
/// <summary>Reply for <see cref="AcknowledgeAlarmCommand"/>.</summary>
|
||||
/// <param name="Ok">True iff the command was published without error.</param>
|
||||
@@ -25,4 +27,4 @@ public sealed record AcknowledgeAlarmCommand(
|
||||
public sealed record AcknowledgeAlarmResult(
|
||||
bool Ok,
|
||||
string? Message,
|
||||
Guid CorrelationId);
|
||||
CorrelationId CorrelationId);
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Types;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Commons.Messages.Admin;
|
||||
|
||||
/// <summary>
|
||||
@@ -40,7 +42,7 @@ public sealed record ShelveAlarmCommand(
|
||||
ShelveKind Kind,
|
||||
DateTime? UnshelveAtUtc,
|
||||
string? Comment,
|
||||
Guid CorrelationId);
|
||||
CorrelationId CorrelationId);
|
||||
|
||||
/// <summary>Reply for <see cref="ShelveAlarmCommand"/>.</summary>
|
||||
/// <param name="Ok">True iff the command was published without error.</param>
|
||||
@@ -49,4 +51,4 @@ public sealed record ShelveAlarmCommand(
|
||||
public sealed record ShelveAlarmResult(
|
||||
bool Ok,
|
||||
string? Message,
|
||||
Guid CorrelationId);
|
||||
CorrelationId CorrelationId);
|
||||
|
||||
@@ -282,15 +282,17 @@ public sealed class GalaxyDriver
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Reopen callback for <see cref="ReconnectSupervisor"/>: re-Register the gw session.
|
||||
/// If the session never connected, this is a fresh ConnectAsync; otherwise it's a
|
||||
/// reconnect against the existing client.
|
||||
/// Reopen callback for <see cref="ReconnectSupervisor"/>: recreate the gw session. After a
|
||||
/// gateway restart the existing session handle is faulted / stale but non-null, so a plain
|
||||
/// <c>ConnectAsync</c> would no-op and the supervisor would loop forever replaying against a
|
||||
/// dead session. <see cref="GalaxyMxSession.RecreateAsync"/> disposes the stale session +
|
||||
/// owned client and rebuilds, so a never-connected session still opens fresh.
|
||||
/// </summary>
|
||||
private async Task ReopenAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
if (_ownedMxSession is null) return;
|
||||
var clientOptions = BuildClientOptions(_options.Gateway);
|
||||
await _ownedMxSession.ConnectAsync(clientOptions, cancellationToken).ConfigureAwait(false);
|
||||
await _ownedMxSession.RecreateAsync(clientOptions, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
|
||||
@@ -28,6 +28,7 @@ public sealed class GalaxyMxSession : IAsyncDisposable
|
||||
private MxGatewayClient? _ownedClient;
|
||||
private MxGatewaySession? _session;
|
||||
private int _serverHandle;
|
||||
private bool _connected;
|
||||
private bool _disposed;
|
||||
|
||||
/// <summary>Initializes a new instance of the GalaxyMxSession class.</summary>
|
||||
@@ -39,8 +40,12 @@ public sealed class GalaxyMxSession : IAsyncDisposable
|
||||
_logger = logger ?? NullLogger.Instance;
|
||||
}
|
||||
|
||||
/// <summary>Gets a value indicating whether the session is connected.</summary>
|
||||
public bool IsConnected => _session is not null;
|
||||
/// <summary>
|
||||
/// Gets a value indicating whether a session is currently established. Set <c>true</c> on a
|
||||
/// successful connect/attach and <c>false</c> after teardown/dispose — tracks the same
|
||||
/// <c>_connected</c> flag that <see cref="ConnectAsync"/> guards on, so the two never disagree.
|
||||
/// </summary>
|
||||
public bool IsConnected => _connected;
|
||||
|
||||
/// <summary>
|
||||
/// Server-side handle returned by MXAccess <c>Register</c>. Zero before
|
||||
@@ -48,24 +53,65 @@ public sealed class GalaxyMxSession : IAsyncDisposable
|
||||
/// </summary>
|
||||
public int ServerHandle => _serverHandle;
|
||||
|
||||
/// <summary>
|
||||
/// Test seam — when set, replaces the real "open session + register" work so unit tests can
|
||||
/// exercise the connect / recreate orchestration without a live gateway (the SDK session/client
|
||||
/// types are sealed + internal-ctor and cannot be faked). Null in production = real gateway path.
|
||||
/// </summary>
|
||||
internal Func<CancellationToken, Task>? OpenAndRegisterOverrideForTests { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// Connect the underlying gateway client + open an MXAccess session + register the
|
||||
/// configured client name. Idempotent — second calls are no-ops while
|
||||
/// <see cref="IsConnected"/> is true.
|
||||
/// configured client name. Idempotent — second calls are no-ops while a connect has
|
||||
/// already succeeded. Use <see cref="RecreateAsync"/> to force a rebuild after the
|
||||
/// gateway restarts and leaves a faulted / stale session handle.
|
||||
/// </summary>
|
||||
/// <param name="clientOptions">The MX gateway client options.</param>
|
||||
/// <param name="cancellationToken">The cancellation token.</param>
|
||||
public async Task ConnectAsync(MxGatewayClientOptions clientOptions, CancellationToken cancellationToken)
|
||||
{
|
||||
ObjectDisposedException.ThrowIf(_disposed, this);
|
||||
if (_session is not null) return;
|
||||
if (_connected) return;
|
||||
|
||||
_ownedClient = MxGatewayClient.Create(clientOptions);
|
||||
_session = await _ownedClient.OpenSessionAsync(cancellationToken: cancellationToken).ConfigureAwait(false);
|
||||
_serverHandle = await _session.RegisterAsync(_options.ClientName, cancellationToken).ConfigureAwait(false);
|
||||
_logger.LogInformation(
|
||||
"GalaxyMxSession connected — clientName={ClientName} serverHandle={Handle}",
|
||||
_options.ClientName, _serverHandle);
|
||||
try
|
||||
{
|
||||
if (OpenAndRegisterOverrideForTests is not null)
|
||||
{
|
||||
await OpenAndRegisterOverrideForTests(cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
else
|
||||
{
|
||||
_ownedClient = MxGatewayClient.Create(clientOptions);
|
||||
_session = await _ownedClient.OpenSessionAsync(cancellationToken: cancellationToken).ConfigureAwait(false);
|
||||
_serverHandle = await _session.RegisterAsync(_options.ClientName, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
_connected = true;
|
||||
_logger.LogInformation(
|
||||
"GalaxyMxSession connected — clientName={ClientName} serverHandle={Handle}",
|
||||
_options.ClientName, _serverHandle);
|
||||
}
|
||||
catch
|
||||
{
|
||||
// A partial open (e.g. OpenSession succeeded but Register threw) must not leave a half-open
|
||||
// handle that blocks the next reconnect. Tear the partial state down so a retry rebuilds cleanly.
|
||||
await TeardownSessionAsync().ConfigureAwait(false);
|
||||
throw;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Disposes the current (faulted / stale) session + owned client and rebuilds. Use this on
|
||||
/// reconnect: <see cref="ConnectAsync"/> alone no-ops while a (possibly dead) session handle is
|
||||
/// still present, so the supervisor's reopen must route through here to actually re-establish.
|
||||
/// </summary>
|
||||
/// <param name="clientOptions">The MX gateway client options.</param>
|
||||
/// <param name="cancellationToken">The cancellation token.</param>
|
||||
public async Task RecreateAsync(MxGatewayClientOptions clientOptions, CancellationToken cancellationToken)
|
||||
{
|
||||
ObjectDisposedException.ThrowIf(_disposed, this);
|
||||
await TeardownSessionAsync().ConfigureAwait(false);
|
||||
await ConnectAsync(clientOptions, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
@@ -80,6 +126,7 @@ public sealed class GalaxyMxSession : IAsyncDisposable
|
||||
ObjectDisposedException.ThrowIf(_disposed, this);
|
||||
_session = session ?? throw new ArgumentNullException(nameof(session));
|
||||
_serverHandle = serverHandle;
|
||||
_connected = true; // an attached session means connected — keeps IsConnected + the ConnectAsync no-op guard consistent.
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
@@ -93,19 +140,31 @@ public sealed class GalaxyMxSession : IAsyncDisposable
|
||||
{
|
||||
if (_disposed) return;
|
||||
_disposed = true;
|
||||
await TeardownSessionAsync().ConfigureAwait(false);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Resets the connect flag and best-effort disposes + nulls the owned session and client.
|
||||
/// Shared by <see cref="DisposeAsync"/> (final teardown) and <see cref="RecreateAsync"/> /
|
||||
/// the partial-open recovery path (rebuild teardown) — it does NOT set <c>_disposed</c>, so
|
||||
/// a torn-down session can be re-opened.
|
||||
/// </summary>
|
||||
private async Task TeardownSessionAsync()
|
||||
{
|
||||
_connected = false;
|
||||
|
||||
if (_session is not null)
|
||||
{
|
||||
try { await _session.DisposeAsync().ConfigureAwait(false); }
|
||||
catch (Exception ex) { _logger.LogWarning(ex, "GalaxyMxSession session dispose failed (best-effort)"); }
|
||||
_session = null;
|
||||
}
|
||||
_session = null;
|
||||
|
||||
if (_ownedClient is not null)
|
||||
{
|
||||
try { await _ownedClient.DisposeAsync().ConfigureAwait(false); }
|
||||
catch (Exception ex) { _logger.LogWarning(ex, "GalaxyMxSession client dispose failed (best-effort)"); }
|
||||
_ownedClient = null;
|
||||
}
|
||||
_ownedClient = null;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -46,7 +46,7 @@ public sealed class AdminOperationsClient : IAdminOperationsClient
|
||||
public async Task<AcknowledgeAlarmResult> AcknowledgeAlarmAsync(
|
||||
string alarmId, string user, string? comment, CancellationToken ct)
|
||||
{
|
||||
var msg = new AcknowledgeAlarmCommand(alarmId, user, comment, Guid.NewGuid());
|
||||
var msg = new AcknowledgeAlarmCommand(alarmId, user, comment, CorrelationId.NewId());
|
||||
using var linked = CancellationTokenSource.CreateLinkedTokenSource(ct);
|
||||
linked.CancelAfter(AskTimeout);
|
||||
return await _proxy.Ask<AcknowledgeAlarmResult>(msg, AskTimeout, linked.Token);
|
||||
@@ -63,7 +63,7 @@ public sealed class AdminOperationsClient : IAdminOperationsClient
|
||||
public async Task<ShelveAlarmResult> ShelveAlarmAsync(
|
||||
string alarmId, string user, ShelveKind kind, DateTime? unshelveAtUtc, string? comment, CancellationToken ct)
|
||||
{
|
||||
var msg = new ShelveAlarmCommand(alarmId, user, kind, unshelveAtUtc, comment, Guid.NewGuid());
|
||||
var msg = new ShelveAlarmCommand(alarmId, user, kind, unshelveAtUtc, comment, CorrelationId.NewId());
|
||||
using var linked = CancellationTokenSource.CreateLinkedTokenSource(ct);
|
||||
linked.CancelAfter(AskTimeout);
|
||||
return await _proxy.Ask<ShelveAlarmResult>(msg, AskTimeout, linked.Token);
|
||||
|
||||
@@ -13,7 +13,7 @@
|
||||
@inject AuthenticationStateProvider AuthState
|
||||
@inject IAuthorizationService AuthorizationService
|
||||
@inject IAdminOperationsClient AdminOps
|
||||
@implements IDisposable
|
||||
@implements IAsyncDisposable
|
||||
|
||||
<div class="d-flex justify-content-between align-items-center mb-3">
|
||||
<h4 class="mb-0">Alerts</h4>
|
||||
@@ -87,6 +87,15 @@ else
|
||||
disabled="@_busyAlarmId.Equals(e.AlarmId)"
|
||||
@onclick="() => ShelveAsync(e.AlarmId, ShelveKind.OneShot)"
|
||||
title="Shelve this alarm until it next clears">Shelve</button>
|
||||
<input type="number" min="1" class="form-control form-control-sm" style="width:4.5rem"
|
||||
disabled="@_busyAlarmId.Equals(e.AlarmId)"
|
||||
@bind:get="GetShelveMinutes(e.AlarmId)" @bind:set="m => SetShelveMinutes(e.AlarmId, m)"
|
||||
title="Timed-shelve duration in minutes" />
|
||||
<button type="button"
|
||||
class="btn btn-sm btn-outline-secondary"
|
||||
disabled="@_busyAlarmId.Equals(e.AlarmId)"
|
||||
@onclick="() => ShelveTimedAsync(e.AlarmId)"
|
||||
title="Shelve this alarm for the chosen number of minutes">Shelve (timed)</button>
|
||||
<button type="button"
|
||||
class="btn btn-sm btn-outline-secondary"
|
||||
disabled="@_busyAlarmId.Equals(e.AlarmId)"
|
||||
@@ -123,13 +132,31 @@ else
|
||||
private string? _opResultMessage;
|
||||
private bool _opResultOk;
|
||||
|
||||
// Auto-clear timer for the per-row result chip (mirrors DriverStatusPanel): the chip is set in
|
||||
// ShowOpResult and cleared 8 s later so it doesn't persist until the next action.
|
||||
private System.Threading.Timer? _opResultClearTimer;
|
||||
private object? _opResultClearToken;
|
||||
|
||||
// Per-row timed-shelve duration (minutes). Keyed by AlarmId so each row's number input is
|
||||
// independent — binding every row to one shared field would couple all the inputs together.
|
||||
private const int DefaultShelveMinutes = 5;
|
||||
private readonly Dictionary<string, int> _shelveMinutes = new();
|
||||
private int GetShelveMinutes(string alarmId) =>
|
||||
_shelveMinutes.TryGetValue(alarmId, out var m) ? m : DefaultShelveMinutes;
|
||||
private void SetShelveMinutes(string alarmId, int minutes) =>
|
||||
_shelveMinutes[alarmId] = minutes < 1 ? 1 : minutes;
|
||||
|
||||
protected override async Task OnInitializedAsync()
|
||||
{
|
||||
// Live alarm tail straight from the in-process broadcaster (fed by AlertSignalRBridge off the
|
||||
// 'alerts' DPS topic). A Blazor Server component can't self-connect a SignalR HubConnection
|
||||
// behind a reverse proxy — see IInProcessBroadcaster — so we subscribe in-process instead.
|
||||
// Subscribe to the health signal BEFORE reading IsConnected so a state change between the
|
||||
// read and the subscribe can't be missed (TOCTOU) — mirrors DriverStatusPanel's
|
||||
// SnapshotChanged-then-TryGet ordering.
|
||||
Alarms.ConnectionStateChanged += OnConnectionStateChanged;
|
||||
Alarms.Received += OnAlarm;
|
||||
_connected = true;
|
||||
_connected = Alarms.IsConnected;
|
||||
|
||||
// Check DriverOperator authorization so the per-row action controls only render for
|
||||
// permitted users. The username is re-read at click time (GetCurrentUserNameAsync) so a
|
||||
@@ -163,7 +190,12 @@ else
|
||||
}
|
||||
}
|
||||
|
||||
private async Task ShelveAsync(string alarmId, ShelveKind kind)
|
||||
// Thin wrapper for the per-row "Shelve (timed)" button: shelves until now + the row's chosen
|
||||
// minutes. OneShot/Unshelve call ShelveAsync directly with the default null unshelve-at.
|
||||
private Task ShelveTimedAsync(string alarmId) =>
|
||||
ShelveAsync(alarmId, ShelveKind.Timed, DateTime.UtcNow.AddMinutes(GetShelveMinutes(alarmId)));
|
||||
|
||||
private async Task ShelveAsync(string alarmId, ShelveKind kind, DateTime? unshelveAtUtc = null)
|
||||
{
|
||||
_busyAlarmId = alarmId;
|
||||
_opResultMessage = null;
|
||||
@@ -171,12 +203,9 @@ else
|
||||
try
|
||||
{
|
||||
var user = await GetCurrentUserNameAsync();
|
||||
// Timed shelve (with an unshelve-at datetime picker) is deferred — only OneShot + Unshelve
|
||||
// are surfaced here, so unshelveAtUtc is always null. TimedShelve is fully wired through the
|
||||
// singleton + AlarmCommand if a UI is added later.
|
||||
using var cts = new System.Threading.CancellationTokenSource(TimeSpan.FromSeconds(15));
|
||||
var result = await AdminOps.ShelveAlarmAsync(
|
||||
alarmId, user, kind, unshelveAtUtc: null, comment: null, cts.Token);
|
||||
alarmId, user, kind, unshelveAtUtc: unshelveAtUtc, comment: null, cts.Token);
|
||||
var verb = kind == ShelveKind.Unshelve ? "Unshelve" : "Shelve";
|
||||
ShowOpResult(alarmId, result.Ok, result.Ok ? $"{verb} sent" : (result.Message ?? "Failed"));
|
||||
}
|
||||
@@ -210,6 +239,22 @@ else
|
||||
_opResultOk = ok;
|
||||
_opResultMessage = message;
|
||||
StateHasChanged();
|
||||
// Auto-clear the result chip after 8 s (mirrors DriverStatusPanel). System.Threading.Timer
|
||||
// is used (not System.Timers.Timer) so DisposeAsync can drain any in-flight callback.
|
||||
var token = _opResultClearToken = new object();
|
||||
_opResultClearTimer?.Dispose();
|
||||
_opResultClearTimer = new System.Threading.Timer(_ =>
|
||||
InvokeAsync(() =>
|
||||
{
|
||||
// Ignore a stale fire: a newer action superseded this timer's chip. (Timer.Dispose does
|
||||
// not drain an already-queued callback, so without this a stale timer could clear a
|
||||
// freshly-set result from a different row.)
|
||||
if (!ReferenceEquals(_opResultClearToken, token)) return;
|
||||
_opResultMessage = null;
|
||||
_opResultAlarmId = "";
|
||||
StateHasChanged();
|
||||
}),
|
||||
null, TimeSpan.FromSeconds(8), Timeout.InfiniteTimeSpan);
|
||||
}
|
||||
|
||||
private void OnAlarm(AlarmTransitionEvent evt) =>
|
||||
@@ -222,6 +267,12 @@ else
|
||||
StateHasChanged();
|
||||
});
|
||||
|
||||
// Raised by the broadcaster (on the bridge actor's thread) whenever the upstream DPS
|
||||
// subscription's health flips. Marshal onto the circuit sync context like OnAlarm so the live
|
||||
// pill re-renders safely.
|
||||
private void OnConnectionStateChanged(bool connected) =>
|
||||
InvokeAsync(() => { _connected = connected; StateHasChanged(); });
|
||||
|
||||
private async Task ClearAsync()
|
||||
{
|
||||
_rows.Clear();
|
||||
@@ -237,5 +288,13 @@ else
|
||||
_ => "chip-idle",
|
||||
};
|
||||
|
||||
public void Dispose() => Alarms.Received -= OnAlarm;
|
||||
public async ValueTask DisposeAsync()
|
||||
{
|
||||
// Unsubscribe BOTH broadcaster events first so the singleton can't invoke a handler on a
|
||||
// disposed component, then drain the auto-clear timer (its async dispose awaits any in-flight
|
||||
// callback so it can't call StateHasChanged on a gone component).
|
||||
Alarms.ConnectionStateChanged -= OnConnectionStateChanged;
|
||||
Alarms.Received -= OnAlarm;
|
||||
if (_opResultClearTimer is not null) await _opResultClearTimer.DisposeAsync();
|
||||
}
|
||||
}
|
||||
|
||||
@@ -118,10 +118,15 @@ else
|
||||
// Live tail straight from the in-process broadcaster (fed by ScriptLogSignalRBridge off the
|
||||
// 'script-logs' DPS topic). Blazor Server can't self-connect a SignalR HubConnection behind
|
||||
// a reverse proxy — see IInProcessBroadcaster — so we subscribe in-process instead.
|
||||
// Subscribe the state-change event BEFORE reading IsConnected so no transition is missed.
|
||||
ScriptLogs.ConnectionStateChanged += OnConnectionStateChanged;
|
||||
ScriptLogs.Received += OnEntry;
|
||||
_connected = true;
|
||||
_connected = ScriptLogs.IsConnected;
|
||||
}
|
||||
|
||||
private void OnConnectionStateChanged(bool connected) =>
|
||||
InvokeAsync(() => { _connected = connected; StateHasChanged(); });
|
||||
|
||||
private void OnEntry(ScriptLogEntry entry) =>
|
||||
// Marshal both the mutation and the re-render onto the circuit sync context so this can't
|
||||
// race ClearAsync (which runs there) over the shared _rows list.
|
||||
@@ -146,5 +151,9 @@ else
|
||||
_ => "chip-idle",
|
||||
};
|
||||
|
||||
public void Dispose() => ScriptLogs.Received -= OnEntry;
|
||||
public void Dispose()
|
||||
{
|
||||
ScriptLogs.ConnectionStateChanged -= OnConnectionStateChanged;
|
||||
ScriptLogs.Received -= OnEntry;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -38,13 +38,22 @@ public sealed class AlertSignalRBridge : ReceiveActor
|
||||
_hub = hub;
|
||||
_broadcaster = broadcaster;
|
||||
ReceiveAsync<AlarmTransitionEvent>(ForwardAsync);
|
||||
Receive<SubscribeAck>(_ => { /* DPS confirmation */ });
|
||||
// DPS subscription is now live — mark the feed connected so the Blazor "live" pill lights up.
|
||||
Receive<SubscribeAck>(_ => _broadcaster.SetConnected(true));
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override void PreStart() =>
|
||||
DistributedPubSub.Get(Context.System).Mediator.Tell(new Subscribe(TopicName, Self));
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override void PostStop()
|
||||
{
|
||||
// Bridge stopping — the feed is no longer live, drop the "live" pill.
|
||||
_broadcaster.SetConnected(false);
|
||||
base.PostStop();
|
||||
}
|
||||
|
||||
private async Task ForwardAsync(AlarmTransitionEvent msg)
|
||||
{
|
||||
// In-process fan-out first — this is what the Blazor Server Alerts page reads. The hub push
|
||||
|
||||
@@ -26,16 +26,78 @@ public interface IInProcessBroadcaster<T>
|
||||
|
||||
/// <summary>Fan the item out to all current <see cref="Received"/> subscribers.</summary>
|
||||
void Publish(T item);
|
||||
|
||||
/// <summary>
|
||||
/// Whether the upstream feed (the per-node SignalR bridge's DPS subscription) is currently
|
||||
/// live. Drives the "live" pill on the Blazor pages. False until the bridge's first
|
||||
/// <c>SubscribeAck</c>; flips false again when the bridge stops.
|
||||
/// </summary>
|
||||
bool IsConnected { get; }
|
||||
|
||||
/// <summary>
|
||||
/// Raised whenever <see cref="IsConnected"/> changes (and only on change), with the new value.
|
||||
/// Handlers run on the caller's thread (the bridge actor), so Blazor subscribers must marshal
|
||||
/// via <c>InvokeAsync</c>.
|
||||
/// </summary>
|
||||
event Action<bool>? ConnectionStateChanged;
|
||||
|
||||
/// <summary>
|
||||
/// Set by the bridge actor from its DPS-subscription health: <c>true</c> on <c>SubscribeAck</c>
|
||||
/// (subscription live), <c>false</c> on <c>PostStop</c>/failure. Raises
|
||||
/// <see cref="ConnectionStateChanged"/> only when the value actually changes.
|
||||
/// </summary>
|
||||
/// <param name="connected">The new connection state.</param>
|
||||
void SetConnected(bool connected);
|
||||
}
|
||||
|
||||
/// <summary>Thread-safe singleton implementation of <see cref="IInProcessBroadcaster{T}"/>.</summary>
|
||||
/// <typeparam name="T">The event payload type.</typeparam>
|
||||
public sealed class InProcessBroadcaster<T> : IInProcessBroadcaster<T>
|
||||
{
|
||||
// Guards _isConnected: the bridge actor sets it on the actor thread; Blazor reads it on the
|
||||
// render thread, so access must be serialised.
|
||||
private readonly object _connectionLock = new();
|
||||
private bool _isConnected;
|
||||
|
||||
/// <inheritdoc />
|
||||
public event Action<T>? Received;
|
||||
|
||||
/// <inheritdoc />
|
||||
public event Action<bool>? ConnectionStateChanged;
|
||||
|
||||
/// <inheritdoc />
|
||||
// Capture-then-invoke (via ?.) so a concurrent unsubscribe can't null the delegate mid-raise.
|
||||
public void Publish(T item) => Received?.Invoke(item);
|
||||
|
||||
/// <inheritdoc />
|
||||
public bool IsConnected
|
||||
{
|
||||
get
|
||||
{
|
||||
lock (_connectionLock)
|
||||
{
|
||||
return _isConnected;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public void SetConnected(bool connected)
|
||||
{
|
||||
Action<bool>? handler;
|
||||
lock (_connectionLock)
|
||||
{
|
||||
if (_isConnected == connected)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
_isConnected = connected;
|
||||
// Capture inside the lock, invoke outside (mirrors Publish) so a concurrent
|
||||
// unsubscribe can't null the delegate mid-raise and we never hold the lock during a callback.
|
||||
handler = ConnectionStateChanged;
|
||||
}
|
||||
|
||||
handler?.Invoke(connected);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -32,13 +32,22 @@ public sealed class ScriptLogSignalRBridge : ReceiveActor
|
||||
_hub = hub;
|
||||
_broadcaster = broadcaster;
|
||||
ReceiveAsync<ScriptLogEntry>(ForwardAsync);
|
||||
Receive<SubscribeAck>(_ => { /* DPS confirmation */ });
|
||||
// DPS subscription is now live — mark the feed connected so the Blazor "live" pill lights up.
|
||||
Receive<SubscribeAck>(_ => _broadcaster.SetConnected(true));
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override void PreStart() =>
|
||||
DistributedPubSub.Get(Context.System).Mediator.Tell(new Subscribe(TopicName, Self));
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override void PostStop()
|
||||
{
|
||||
// Bridge stopping — the feed is no longer live, drop the "live" pill.
|
||||
_broadcaster.SetConnected(false);
|
||||
base.PostStop();
|
||||
}
|
||||
|
||||
private async Task ForwardAsync(ScriptLogEntry msg)
|
||||
{
|
||||
// In-process fan-out first — this is what the Blazor Server Script log page reads. The hub
|
||||
|
||||
@@ -22,9 +22,18 @@ public sealed class RedundancyStateActor : ReceiveActor, IWithTimers
|
||||
public const string Topic = "redundancy-state";
|
||||
public static readonly TimeSpan DebounceWindow = TimeSpan.FromMilliseconds(250);
|
||||
|
||||
/// <summary>
|
||||
/// Default interval for the periodic heartbeat re-publish. DistributedPubSub does not replay
|
||||
/// to late subscribers, so a node that subscribes just after a change-driven publish would
|
||||
/// otherwise never learn its role until the next membership change. The heartbeat re-publishes
|
||||
/// the current snapshot so any missed subscriber converges within this interval.
|
||||
/// </summary>
|
||||
public static readonly TimeSpan DefaultHeartbeatInterval = TimeSpan.FromSeconds(10);
|
||||
|
||||
private readonly ILoggingAdapter _log = Context.GetLogger();
|
||||
private readonly Akka.Cluster.Cluster _cluster;
|
||||
private readonly Action<object>? _broadcastOverride;
|
||||
private readonly TimeSpan _heartbeatInterval;
|
||||
private bool _dirty;
|
||||
|
||||
/// <summary>Gets the timer scheduler for this actor.</summary>
|
||||
@@ -32,19 +41,28 @@ public sealed class RedundancyStateActor : ReceiveActor, IWithTimers
|
||||
|
||||
/// <summary>Creates a Props for creating a RedundancyStateActor.</summary>
|
||||
/// <param name="broadcast">Optional broadcast override for testing.</param>
|
||||
/// <param name="heartbeatInterval">
|
||||
/// Optional periodic heartbeat re-publish interval; defaults to
|
||||
/// <see cref="DefaultHeartbeatInterval"/>. Injectable so tests can use a short interval.
|
||||
/// </param>
|
||||
/// <returns>Props for actor creation.</returns>
|
||||
public static Props Props(Action<object>? broadcast = null) =>
|
||||
Akka.Actor.Props.Create(() => new RedundancyStateActor(broadcast));
|
||||
public static Props Props(Action<object>? broadcast = null, TimeSpan? heartbeatInterval = null) =>
|
||||
Akka.Actor.Props.Create(() => new RedundancyStateActor(broadcast, heartbeatInterval));
|
||||
|
||||
/// <summary>Initializes a new instance of RedundancyStateActor with no broadcast override.</summary>
|
||||
public RedundancyStateActor() : this(broadcast: null) { }
|
||||
public RedundancyStateActor() : this(null, null) { }
|
||||
|
||||
/// <summary>Initializes a new instance of RedundancyStateActor.</summary>
|
||||
/// <param name="broadcast">Optional broadcast override for testing.</param>
|
||||
public RedundancyStateActor(Action<object>? broadcast)
|
||||
/// <param name="heartbeatInterval">
|
||||
/// Optional periodic heartbeat re-publish interval; defaults to
|
||||
/// <see cref="DefaultHeartbeatInterval"/>.
|
||||
/// </param>
|
||||
public RedundancyStateActor(Action<object>? broadcast, TimeSpan? heartbeatInterval = null)
|
||||
{
|
||||
_cluster = Akka.Cluster.Cluster.Get(Context.System);
|
||||
_broadcastOverride = broadcast;
|
||||
_heartbeatInterval = heartbeatInterval ?? DefaultHeartbeatInterval;
|
||||
|
||||
Receive<ClusterEvent.IMemberEvent>(_ => MarkDirty());
|
||||
Receive<ClusterEvent.LeaderChanged>(_ => MarkDirty());
|
||||
@@ -52,6 +70,7 @@ public sealed class RedundancyStateActor : ReceiveActor, IWithTimers
|
||||
Receive<ClusterEvent.CurrentClusterState>(_ => MarkDirty());
|
||||
Receive<ClusterEvent.ReachabilityEvent>(_ => MarkDirty());
|
||||
Receive<RecomputeNow>(_ => PublishIfDirty());
|
||||
Receive<Heartbeat>(_ => { _dirty = true; PublishIfDirty(); });
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
@@ -64,6 +83,11 @@ public sealed class RedundancyStateActor : ReceiveActor, IWithTimers
|
||||
typeof(ClusterEvent.LeaderChanged),
|
||||
typeof(ClusterEvent.RoleLeaderChanged),
|
||||
typeof(ClusterEvent.ReachabilityEvent));
|
||||
|
||||
// DistributedPubSub does not replay to late subscribers, so re-publish the current
|
||||
// snapshot on a recurring heartbeat — any node that missed the change-driven publish
|
||||
// converges within _heartbeatInterval. Idempotent: consumers re-cache the same role.
|
||||
Timers.StartPeriodicTimer("heartbeat", Heartbeat.Instance, _heartbeatInterval);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
@@ -104,7 +128,7 @@ public sealed class RedundancyStateActor : ReceiveActor, IWithTimers
|
||||
: CommonsRedundancyRole.Detached;
|
||||
|
||||
list.Add(new NodeRedundancyState(
|
||||
NodeId.Parse(host),
|
||||
ToNodeId(member.Address),
|
||||
role,
|
||||
IsClusterLeader: clusterLeader == member.Address,
|
||||
IsRoleLeaderForDriver: driverLeader == member.Address,
|
||||
@@ -113,9 +137,30 @@ public sealed class RedundancyStateActor : ReceiveActor, IWithTimers
|
||||
return list;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Builds the canonical cluster node id (<c>host:port</c>) for an Akka member address — the SAME
|
||||
/// format <c>ClusterRoleInfo.LocalNode</c>/<c>ToNodeId</c> use, so redundancy-state consumers
|
||||
/// (OPC UA ServiceLevel, scripted-alarm emit gate, historian gate) can match their local node.
|
||||
/// </summary>
|
||||
/// <param name="address">The cluster member's address.</param>
|
||||
/// <returns>The canonical <c>host:port</c> node id.</returns>
|
||||
public static NodeId ToNodeId(Akka.Actor.Address address) =>
|
||||
NodeId.Parse($"{address.Host ?? string.Empty}:{address.Port ?? 0}");
|
||||
|
||||
public sealed class RecomputeNow
|
||||
{
|
||||
public static readonly RecomputeNow Instance = new();
|
||||
private RecomputeNow() { }
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Marker message fired by the periodic heartbeat timer. Forces a re-publish of the current
|
||||
/// snapshot regardless of the change-driven <c>_dirty</c> flag, so late subscribers converge.
|
||||
/// </summary>
|
||||
public sealed class Heartbeat
|
||||
{
|
||||
/// <summary>The singleton heartbeat message instance.</summary>
|
||||
public static readonly Heartbeat Instance = new();
|
||||
private Heartbeat() { }
|
||||
}
|
||||
}
|
||||
|
||||
@@ -280,7 +280,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
|
||||
var engine = new ScriptedAlarmEngine(
|
||||
upstream, store, new ScriptLoggerFactory(_scriptRootLogger.Logger), _scriptRootLogger.Logger);
|
||||
_scriptedAlarmHost = Context.ActorOf(
|
||||
ScriptedAlarmHostActor.Props(_opcUaPublishActor, _dependencyMux, upstream, engine),
|
||||
ScriptedAlarmHostActor.Props(_opcUaPublishActor, _dependencyMux, upstream, engine, _localNode),
|
||||
"scripted-alarm-host");
|
||||
}
|
||||
|
||||
|
||||
@@ -1,6 +1,10 @@
|
||||
using Akka.Actor;
|
||||
using Akka.Cluster.Tools.PublishSubscribe;
|
||||
using Akka.Event;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Redundancy;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Types;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.OpcUa;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Runtime.Historian;
|
||||
|
||||
@@ -23,22 +27,47 @@ public sealed class HistorianAdapterActor : ReceiveActor
|
||||
}
|
||||
|
||||
private readonly IAlarmHistorianSink _sink;
|
||||
private readonly NodeId? _localNode;
|
||||
private readonly ILoggingAdapter _log = Context.GetLogger();
|
||||
private IActorRef? _mediator;
|
||||
|
||||
/// <summary>Cached local <see cref="RedundancyRole"/> from the latest <see cref="RedundancyStateChanged"/>
|
||||
/// snapshot (null = unknown until the first snapshot arrives, or no <see cref="_localNode"/> wired). The
|
||||
/// durable sink enqueue in the <see cref="AlarmHistorianEvent"/> handler is gated on this: only the
|
||||
/// Primary historizes (default-write while unknown so single-node deploys + the boot window never drop
|
||||
/// historization).</summary>
|
||||
private RedundancyRole? _localRole;
|
||||
|
||||
/// <summary>Creates the props for a HistorianAdapterActor instance.</summary>
|
||||
/// <param name="sink">The alarm historian sink implementation, or null to use a null sink.</param>
|
||||
/// <param name="localNode">The local cluster node id, used to read this node's <see cref="RedundancyRole"/>
|
||||
/// from the <c>redundancy-state</c> topic so only the Primary historizes to the durable sink. Null (the
|
||||
/// default) leaves the role unknown ⇒ default-write (single-node deploys + tests).</param>
|
||||
/// <returns>Props configured for creating a HistorianAdapterActor.</returns>
|
||||
public static Props Props(IAlarmHistorianSink? sink = null) =>
|
||||
Akka.Actor.Props.Create(() => new HistorianAdapterActor(sink ?? NullAlarmHistorianSink.Instance));
|
||||
public static Props Props(IAlarmHistorianSink? sink = null, NodeId? localNode = null) =>
|
||||
Akka.Actor.Props.Create(() => new HistorianAdapterActor(sink ?? NullAlarmHistorianSink.Instance, localNode));
|
||||
|
||||
/// <summary>Initializes a new instance of the HistorianAdapterActor class.</summary>
|
||||
/// <param name="sink">The alarm historian sink to forward enqueued events to.</param>
|
||||
public HistorianAdapterActor(IAlarmHistorianSink sink)
|
||||
/// <param name="localNode">The local cluster node id, used to read this node's <see cref="RedundancyRole"/>
|
||||
/// from the <c>redundancy-state</c> topic so only the Primary historizes to the durable sink. Null leaves
|
||||
/// the role unknown ⇒ default-write (single-node deploys + tests).</param>
|
||||
public HistorianAdapterActor(IAlarmHistorianSink sink, NodeId? localNode = null)
|
||||
{
|
||||
_sink = sink;
|
||||
_localNode = localNode;
|
||||
|
||||
Receive<AlarmHistorianEvent>(evt =>
|
||||
{
|
||||
// Warm-standby dedup (forward-looking): only the Primary historizes to the durable sink so a
|
||||
// future per-node feeder writes exactly once. Default-write until told Secondary/Detached so
|
||||
// single-node deploys + the boot window never drop historization. (Currently the actor has no
|
||||
// production feeder — this is a defensive guard for when engine→historian wiring lands.)
|
||||
if (_localRole is RedundancyRole.Secondary or RedundancyRole.Detached)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
// Fire-and-forget: SqliteStoreAndForwardSink persists to local SQLite synchronously
|
||||
// inside EnqueueAsync (it returns once the row is committed), so we don't block on
|
||||
// network/pipe latency. Failures are surfaced via GetStatus's LastError + drain state.
|
||||
@@ -46,6 +75,41 @@ public sealed class HistorianAdapterActor : ReceiveActor
|
||||
});
|
||||
|
||||
Receive<GetStatus>(_ => Sender.Tell(_sink.GetStatus()));
|
||||
|
||||
// Cluster redundancy snapshots (published on the `redundancy-state` topic, subscribed in PreStart)
|
||||
// cache this node's role so the AlarmHistorianEvent handler can gate the durable sink enqueue to the
|
||||
// Primary. The PubSub Subscribe is acked back to Self (no-op below).
|
||||
Receive<RedundancyStateChanged>(OnRedundancyStateChanged);
|
||||
Receive<SubscribeAck>(_ => { });
|
||||
}
|
||||
|
||||
/// <summary>Subscribes to the <c>redundancy-state</c> topic so cluster role changes land as
|
||||
/// <see cref="RedundancyStateChanged"/> and cache this node's role — the historian enqueue is gated to
|
||||
/// the Primary so a future per-node feeder doesn't double-write across the warm-redundant pair.</summary>
|
||||
protected override void PreStart()
|
||||
{
|
||||
_mediator = DistributedPubSub.Get(Context.System).Mediator;
|
||||
_mediator.Tell(new Subscribe(OpcUaPublishActor.RedundancyStateTopic, Self));
|
||||
base.PreStart();
|
||||
}
|
||||
|
||||
/// <summary>Caches this node's <see cref="RedundancyRole"/> from a cluster redundancy snapshot so the
|
||||
/// <see cref="AlarmHistorianEvent"/> handler can gate the durable sink enqueue to the Primary. A snapshot
|
||||
/// that doesn't mention <see cref="_localNode"/> (or no local node wired) leaves the cached role unchanged
|
||||
/// ⇒ default-write. Mirrors <see cref="OpcUaPublishActor"/>'s handler.</summary>
|
||||
/// <param name="msg">The cluster redundancy snapshot.</param>
|
||||
private void OnRedundancyStateChanged(RedundancyStateChanged msg)
|
||||
{
|
||||
if (_localNode is null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
var local = msg.Nodes.FirstOrDefault(n => n.NodeId == _localNode.Value);
|
||||
if (local is not null)
|
||||
{
|
||||
_localRole = local.Role;
|
||||
}
|
||||
}
|
||||
|
||||
private async Task EnqueueAsync(AlarmHistorianEvent evt)
|
||||
|
||||
@@ -2,7 +2,9 @@ using Akka.Actor;
|
||||
using Akka.Cluster.Tools.PublishSubscribe;
|
||||
using Akka.Event;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Alerts;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Redundancy;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Types;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms;
|
||||
using ZB.MOM.WW.OtOpcUa.OpcUaServer;
|
||||
@@ -95,10 +97,17 @@ public sealed class ScriptedAlarmHostActor : ReceiveActor
|
||||
private readonly IActorRef? _mux;
|
||||
private readonly DependencyMuxTagUpstreamSource _upstream;
|
||||
private readonly ScriptedAlarmEngine _engine;
|
||||
private readonly NodeId? _localNode;
|
||||
private readonly ILoggingAdapter _log = Context.GetLogger();
|
||||
private readonly CancellationTokenSource _cts = new();
|
||||
private readonly EventHandler<ScriptedAlarmEvent> _onEngineEvent;
|
||||
|
||||
/// <summary>Cached local <see cref="RedundancyRole"/> from the latest <see cref="RedundancyStateChanged"/>
|
||||
/// snapshot (null = unknown until the first snapshot arrives, or no <see cref="_localNode"/> wired). The
|
||||
/// cluster-wide <c>alerts</c> publish in <see cref="OnEngineEmission"/> is gated on this: only the Primary
|
||||
/// publishes (default-emit while unknown so single-node deploys + the boot window never drop transitions).</summary>
|
||||
private RedundancyRole? _localRole;
|
||||
|
||||
/// <summary>Monotonic load generation, bumped on every <see cref="OnApply"/>. The continuation that
|
||||
/// pipes back an <see cref="AlarmsLoaded"/> captures the generation it was started under; a stale
|
||||
/// completion (an earlier generation arriving after a newer apply) is discarded in
|
||||
@@ -118,23 +127,31 @@ public sealed class ScriptedAlarmHostActor : ReceiveActor
|
||||
/// <param name="upstream">The mux-fed upstream the engine reads + subscribes from. MUST be the
|
||||
/// same instance the <paramref name="engine"/> was constructed around.</param>
|
||||
/// <param name="engine">The scripted-alarm engine this host owns + disposes.</param>
|
||||
/// <param name="localNode">The local cluster node id, used to read this node's <see cref="RedundancyRole"/>
|
||||
/// from the <c>redundancy-state</c> topic so only the Primary publishes the cluster-wide <c>alerts</c>
|
||||
/// transition. Null (the default) leaves the role unknown ⇒ default-emit (single-node deploys + tests).</param>
|
||||
public static Props Props(
|
||||
IActorRef publishActor,
|
||||
IActorRef? mux,
|
||||
DependencyMuxTagUpstreamSource upstream,
|
||||
ScriptedAlarmEngine engine) =>
|
||||
Akka.Actor.Props.Create(() => new ScriptedAlarmHostActor(publishActor, mux, upstream, engine));
|
||||
ScriptedAlarmEngine engine,
|
||||
NodeId? localNode = null) =>
|
||||
Akka.Actor.Props.Create(() => new ScriptedAlarmHostActor(publishActor, mux, upstream, engine, localNode));
|
||||
|
||||
/// <summary>Initializes a new instance of the <see cref="ScriptedAlarmHostActor"/> class.</summary>
|
||||
/// <param name="publishActor">The OPC UA publish actor emissions are bridged to.</param>
|
||||
/// <param name="mux">Optional dependency multiplexer the host registers dependency interest with.</param>
|
||||
/// <param name="upstream">The mux-fed upstream the engine reads + subscribes from (same instance the engine wraps).</param>
|
||||
/// <param name="engine">The scripted-alarm engine this host owns + disposes.</param>
|
||||
/// <param name="localNode">The local cluster node id, used to read this node's <see cref="RedundancyRole"/>
|
||||
/// from the <c>redundancy-state</c> topic so only the Primary publishes the cluster-wide <c>alerts</c>
|
||||
/// transition. Null leaves the role unknown ⇒ default-emit (single-node deploys + tests).</param>
|
||||
public ScriptedAlarmHostActor(
|
||||
IActorRef publishActor,
|
||||
IActorRef? mux,
|
||||
DependencyMuxTagUpstreamSource upstream,
|
||||
ScriptedAlarmEngine engine)
|
||||
ScriptedAlarmEngine engine,
|
||||
NodeId? localNode = null)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(publishActor);
|
||||
ArgumentNullException.ThrowIfNull(upstream);
|
||||
@@ -143,6 +160,7 @@ public sealed class ScriptedAlarmHostActor : ReceiveActor
|
||||
_mux = mux;
|
||||
_upstream = upstream;
|
||||
_engine = engine;
|
||||
_localNode = localNode;
|
||||
|
||||
// OnEvent fires on the engine's worker thread. NEVER touch Context / actor state here —
|
||||
// marshal onto the actor thread via the thread-safe Self.Tell. Keep the handler in a field
|
||||
@@ -155,6 +173,9 @@ public sealed class ScriptedAlarmHostActor : ReceiveActor
|
||||
Receive<AlarmsLoaded>(OnAlarmsLoaded);
|
||||
Receive<VirtualTagActor.DependencyValueChanged>(OnDependencyChanged);
|
||||
Receive<EngineEmission>(OnEngineEmission);
|
||||
// Cluster redundancy snapshots (published on the `redundancy-state` topic, subscribed in PreStart)
|
||||
// cache this node's role so OnEngineEmission can gate the cluster-wide alerts publish to the Primary.
|
||||
Receive<RedundancyStateChanged>(OnRedundancyStateChanged);
|
||||
// Inbound OPC UA Part 9 alarm method calls arrive as AlarmCommands on the cluster
|
||||
// `alarm-commands` DPS topic (T18 publishes them after the AlarmAck role gate). The topic is a
|
||||
// cluster-wide broadcast — every host node receives every command — so OnAlarmCommand filters to
|
||||
@@ -275,9 +296,37 @@ public sealed class ScriptedAlarmHostActor : ReceiveActor
|
||||
User: TransitionUser(e),
|
||||
TimestampUtc: e.TimestampUtc);
|
||||
|
||||
// Warm-standby dedup: only the Primary (driver-role leader) publishes the cluster-wide
|
||||
// transition. Default-emit until told we are Secondary/Detached so single-node deploys + the
|
||||
// boot window never drop transitions. The OPC UA node write above + inbound command processing
|
||||
// stay ungated (the secondary keeps its address space + engine state warm for failover).
|
||||
if (_localRole is RedundancyRole.Secondary or RedundancyRole.Detached)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
_mediator.Tell(new Publish(AlertsTopic, evt));
|
||||
}
|
||||
|
||||
/// <summary>Caches this node's <see cref="RedundancyRole"/> from a cluster redundancy snapshot so
|
||||
/// <see cref="OnEngineEmission"/> can gate the cluster-wide <c>alerts</c> publish to the Primary. A
|
||||
/// snapshot that doesn't mention <see cref="_localNode"/> (or no local node wired) leaves the cached
|
||||
/// role unchanged ⇒ default-emit. Mirrors <see cref="OpcUaPublishActor"/>'s handler.</summary>
|
||||
/// <param name="msg">The cluster redundancy snapshot.</param>
|
||||
private void OnRedundancyStateChanged(RedundancyStateChanged msg)
|
||||
{
|
||||
if (_localNode is null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
var local = msg.Nodes.FirstOrDefault(n => n.NodeId == _localNode.Value);
|
||||
if (local is not null)
|
||||
{
|
||||
_localRole = local.Role;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Drives an inbound OPC UA Part 9 alarm method call (delivered as an <see cref="AlarmCommand"/>
|
||||
/// on the cluster <c>alarm-commands</c> topic) onto the matching <see cref="ScriptedAlarmEngine"/>
|
||||
@@ -386,6 +435,10 @@ public sealed class ScriptedAlarmHostActor : ReceiveActor
|
||||
// the node manager's condition handlers, T18) land here as AlarmCommands. The Subscribe is sent
|
||||
// from Self so the SubscribeAck returns to this actor (handled as a no-op in the ctor wiring).
|
||||
_mediator.Tell(new Subscribe(AlarmCommandsTopic, Self));
|
||||
// Also subscribe to the `redundancy-state` topic so cluster role changes land as
|
||||
// RedundancyStateChanged and cache this node's role — OnEngineEmission gates the cluster-wide
|
||||
// alerts publish to the Primary so a 2-node single-cluster deploy doesn't double-emit transitions.
|
||||
_mediator.Tell(new Subscribe(OpcUaPublishActor.RedundancyStateTopic, Self));
|
||||
base.PreStart();
|
||||
}
|
||||
|
||||
|
||||
@@ -143,7 +143,7 @@ public static class ServiceCollectionExtensions
|
||||
registry.Register<DriverHostActorKey>(driverHost);
|
||||
|
||||
var historian = system.ActorOf(
|
||||
HistorianAdapterActor.Props(historianSink),
|
||||
HistorianAdapterActor.Props(historianSink, roleInfo.LocalNode),
|
||||
HistorianAdapterActorName);
|
||||
registry.Register<HistorianAdapterActorKey>(historian);
|
||||
});
|
||||
|
||||
+135
@@ -0,0 +1,135 @@
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Config;
|
||||
using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Runtime;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Tests.Runtime;
|
||||
|
||||
/// <summary>
|
||||
/// Connect / recreate orchestration tests for <see cref="GalaxyMxSession"/>. The SDK
|
||||
/// session/client types are sealed with internal ctors and cannot be faked, so these
|
||||
/// drive the open body through the <c>OpenAndRegisterOverrideForTests</c> seam. The
|
||||
/// core regression guard is <see cref="Recreate_after_connect_opens_a_fresh_session"/>:
|
||||
/// a stale-session reconnect must rebuild rather than no-op (the gateway-restart bug).
|
||||
/// </summary>
|
||||
public sealed class GalaxyMxSessionReconnectTests
|
||||
{
|
||||
private static GalaxyMxAccessOptions MinimalOptions() => new(ClientName: "OtOpcUaTest");
|
||||
|
||||
private static GalaxyMxSession NewSession() => new(MinimalOptions());
|
||||
|
||||
/// <summary>A second <c>ConnectAsync</c> while connected must be a no-op (idempotent guard).</summary>
|
||||
[Fact]
|
||||
public async Task Connect_then_connect_again_is_a_noop()
|
||||
{
|
||||
var session = NewSession();
|
||||
var openCount = 0;
|
||||
session.OpenAndRegisterOverrideForTests = _ =>
|
||||
{
|
||||
openCount++;
|
||||
return Task.CompletedTask;
|
||||
};
|
||||
|
||||
await session.ConnectAsync(null!, CancellationToken.None);
|
||||
openCount.ShouldBe(1);
|
||||
|
||||
await session.ConnectAsync(null!, CancellationToken.None);
|
||||
openCount.ShouldBe(1);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The regression guard for the gateway-restart bug: <c>RecreateAsync</c> must bypass
|
||||
/// the no-op guard and re-run the open body even though a (stale) session is present.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task Recreate_after_connect_opens_a_fresh_session()
|
||||
{
|
||||
var session = NewSession();
|
||||
var openCount = 0;
|
||||
session.OpenAndRegisterOverrideForTests = _ =>
|
||||
{
|
||||
openCount++;
|
||||
return Task.CompletedTask;
|
||||
};
|
||||
|
||||
await session.ConnectAsync(null!, CancellationToken.None);
|
||||
openCount.ShouldBe(1);
|
||||
|
||||
await session.RecreateAsync(null!, CancellationToken.None);
|
||||
openCount.ShouldBe(2);
|
||||
}
|
||||
|
||||
/// <summary><c>RecreateAsync</c> on a never-connected session still opens (teardown is a no-op first).</summary>
|
||||
[Fact]
|
||||
public async Task Recreate_when_never_connected_still_opens()
|
||||
{
|
||||
var session = NewSession();
|
||||
var openCount = 0;
|
||||
session.OpenAndRegisterOverrideForTests = _ =>
|
||||
{
|
||||
openCount++;
|
||||
return Task.CompletedTask;
|
||||
};
|
||||
|
||||
await session.RecreateAsync(null!, CancellationToken.None);
|
||||
openCount.ShouldBe(1);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A failed first connect must not leave <c>_connected</c> set — the next attempt has
|
||||
/// to reach the open body again (partial-open teardown).
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task Connect_failure_is_not_left_half_open()
|
||||
{
|
||||
var session = NewSession();
|
||||
var openCount = 0;
|
||||
session.OpenAndRegisterOverrideForTests = _ =>
|
||||
{
|
||||
// Throw on the first attempt, succeed on the second.
|
||||
if (openCount++ == 0)
|
||||
{
|
||||
throw new InvalidOperationException("simulated open failure");
|
||||
}
|
||||
|
||||
return Task.CompletedTask;
|
||||
};
|
||||
|
||||
await Should.ThrowAsync<InvalidOperationException>(
|
||||
async () => await session.ConnectAsync(null!, CancellationToken.None));
|
||||
openCount.ShouldBe(1);
|
||||
session.IsConnected.ShouldBeFalse(); // _connected must NOT be latched by the failed attempt.
|
||||
|
||||
// The failed first attempt must not have latched _connected — the retry reaches the body.
|
||||
await session.ConnectAsync(null!, CancellationToken.None);
|
||||
openCount.ShouldBe(2);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// <see cref="GalaxyMxSession.IsConnected"/> tracks the <c>_connected</c> guard across the
|
||||
/// full lifecycle: false when fresh, true after connect, still true after a recreate, and
|
||||
/// false again after dispose.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task IsConnected_reflects_connect_recreate_and_dispose()
|
||||
{
|
||||
var session = NewSession();
|
||||
var openCount = 0;
|
||||
session.OpenAndRegisterOverrideForTests = _ =>
|
||||
{
|
||||
openCount++;
|
||||
return Task.CompletedTask;
|
||||
};
|
||||
|
||||
session.IsConnected.ShouldBeFalse();
|
||||
|
||||
await session.ConnectAsync(null!, CancellationToken.None);
|
||||
session.IsConnected.ShouldBeTrue();
|
||||
|
||||
await session.RecreateAsync(null!, CancellationToken.None);
|
||||
session.IsConnected.ShouldBeTrue();
|
||||
|
||||
await session.DisposeAsync();
|
||||
session.IsConnected.ShouldBeFalse();
|
||||
}
|
||||
}
|
||||
@@ -48,4 +48,52 @@ public sealed class InProcessBroadcasterTests
|
||||
var broadcaster = new InProcessBroadcaster<int>();
|
||||
Should.NotThrow(() => broadcaster.Publish(42));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void New_broadcaster_is_not_connected()
|
||||
{
|
||||
var broadcaster = new InProcessBroadcaster<string>();
|
||||
|
||||
broadcaster.IsConnected.ShouldBeFalse();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void SetConnected_true_flips_state_and_raises_once()
|
||||
{
|
||||
var broadcaster = new InProcessBroadcaster<string>();
|
||||
var raised = new List<bool>();
|
||||
broadcaster.ConnectionStateChanged += raised.Add;
|
||||
|
||||
broadcaster.SetConnected(true);
|
||||
|
||||
broadcaster.IsConnected.ShouldBeTrue();
|
||||
raised.ShouldBe([true]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void SetConnected_same_value_does_not_raise()
|
||||
{
|
||||
var broadcaster = new InProcessBroadcaster<string>();
|
||||
var raised = new List<bool>();
|
||||
broadcaster.ConnectionStateChanged += raised.Add;
|
||||
|
||||
broadcaster.SetConnected(true);
|
||||
broadcaster.SetConnected(true);
|
||||
|
||||
raised.ShouldBe([true]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void SetConnected_false_after_true_raises_false()
|
||||
{
|
||||
var broadcaster = new InProcessBroadcaster<string>();
|
||||
var raised = new List<bool>();
|
||||
broadcaster.ConnectionStateChanged += raised.Add;
|
||||
|
||||
broadcaster.SetConnected(true);
|
||||
broadcaster.SetConnected(false);
|
||||
|
||||
broadcaster.IsConnected.ShouldBeFalse();
|
||||
raised[^1].ShouldBeFalse();
|
||||
}
|
||||
}
|
||||
|
||||
@@ -43,7 +43,7 @@ public sealed class AdminOperationsActorTests : ControlPlaneActorTestBase
|
||||
var actor = Sys.ActorOf(AdminOperationsActor.Props(dbFactory, coordinator.Ref, Enumerable.Empty<IDriverProbe>()));
|
||||
var topicProbe = SubscribeAlarmCommandsProbe();
|
||||
|
||||
var correlationId = Guid.NewGuid();
|
||||
var correlationId = CorrelationId.NewId();
|
||||
actor.Tell(new AcknowledgeAlarmCommand("alarm-42", "operator-jo", "looking into it", correlationId));
|
||||
|
||||
var published = topicProbe.ExpectMsg<AlarmCommand>(TimeSpan.FromSeconds(3));
|
||||
@@ -69,7 +69,7 @@ public sealed class AdminOperationsActorTests : ControlPlaneActorTestBase
|
||||
var actor = Sys.ActorOf(AdminOperationsActor.Props(dbFactory, coordinator.Ref, Enumerable.Empty<IDriverProbe>()));
|
||||
var topicProbe = SubscribeAlarmCommandsProbe();
|
||||
|
||||
var correlationId = Guid.NewGuid();
|
||||
var correlationId = CorrelationId.NewId();
|
||||
actor.Tell(new ShelveAlarmCommand("alarm-7", "op-kim", ShelveKind.OneShot, UnshelveAtUtc: null, Comment: null, correlationId));
|
||||
|
||||
var published = topicProbe.ExpectMsg<AlarmCommand>(TimeSpan.FromSeconds(3));
|
||||
@@ -94,7 +94,7 @@ public sealed class AdminOperationsActorTests : ControlPlaneActorTestBase
|
||||
var topicProbe = SubscribeAlarmCommandsProbe();
|
||||
|
||||
var unshelveAt = DateTime.UtcNow.AddMinutes(15);
|
||||
actor.Tell(new ShelveAlarmCommand("alarm-9", "op-lee", ShelveKind.Timed, unshelveAt, Comment: "maint window", Guid.NewGuid()));
|
||||
actor.Tell(new ShelveAlarmCommand("alarm-9", "op-lee", ShelveKind.Timed, unshelveAt, Comment: "maint window", CorrelationId.NewId()));
|
||||
|
||||
var published = topicProbe.ExpectMsg<AlarmCommand>(TimeSpan.FromSeconds(3));
|
||||
published.AlarmId.ShouldBe("alarm-9");
|
||||
@@ -116,7 +116,7 @@ public sealed class AdminOperationsActorTests : ControlPlaneActorTestBase
|
||||
var actor = Sys.ActorOf(AdminOperationsActor.Props(dbFactory, coordinator.Ref, Enumerable.Empty<IDriverProbe>()));
|
||||
var topicProbe = SubscribeAlarmCommandsProbe();
|
||||
|
||||
actor.Tell(new ShelveAlarmCommand("alarm-3", "op-sam", ShelveKind.Unshelve, UnshelveAtUtc: null, Comment: null, Guid.NewGuid()));
|
||||
actor.Tell(new ShelveAlarmCommand("alarm-3", "op-sam", ShelveKind.Unshelve, UnshelveAtUtc: null, Comment: null, CorrelationId.NewId()));
|
||||
|
||||
var published = topicProbe.ExpectMsg<AlarmCommand>(TimeSpan.FromSeconds(3));
|
||||
published.Operation.ShouldBe("Unshelve");
|
||||
@@ -135,7 +135,7 @@ public sealed class AdminOperationsActorTests : ControlPlaneActorTestBase
|
||||
var actor = Sys.ActorOf(AdminOperationsActor.Props(dbFactory, coordinator.Ref, Enumerable.Empty<IDriverProbe>()));
|
||||
var topicProbe = SubscribeAlarmCommandsProbe();
|
||||
|
||||
actor.Tell(new ShelveAlarmCommand("alarm-1", "op-zoe", ShelveKind.Timed, UnshelveAtUtc: null, Comment: null, Guid.NewGuid()));
|
||||
actor.Tell(new ShelveAlarmCommand("alarm-1", "op-zoe", ShelveKind.Timed, UnshelveAtUtc: null, Comment: null, CorrelationId.NewId()));
|
||||
|
||||
var reply = ExpectMsg<ShelveAlarmResult>(TimeSpan.FromSeconds(3));
|
||||
reply.Ok.ShouldBeFalse();
|
||||
|
||||
@@ -42,4 +42,53 @@ public sealed class RedundancyStateActorTests : ControlPlaneActorTestBase
|
||||
// After debounce settles, no more events are fired by a quiescent cluster.
|
||||
probe.ExpectNoMsg(TimeSpan.FromMilliseconds(500));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Verifies the periodic heartbeat re-publishes the CURRENT snapshot even when no new cluster
|
||||
/// events arrive — so a late subscriber that missed the change-driven publish converges within
|
||||
/// the heartbeat interval. The first message is the self-join publish; the second proves the
|
||||
/// recurring heartbeat fires with no intervening cluster event.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void Heartbeat_republishes_current_snapshot_without_cluster_events()
|
||||
{
|
||||
var probe = CreateTestProbe("heartbeat-listener");
|
||||
Sys.ActorOf(
|
||||
RedundancyStateActor.Props(
|
||||
broadcast: msg => probe.Ref.Tell(msg),
|
||||
heartbeatInterval: TimeSpan.FromMilliseconds(300)),
|
||||
"redundancy-heartbeat");
|
||||
|
||||
// First publish: the change-driven self-join snapshot.
|
||||
probe.ExpectMsg<RedundancyStateChanged>(TimeSpan.FromSeconds(2));
|
||||
|
||||
// Second publish: the periodic heartbeat re-publish, with NO new cluster event sent.
|
||||
probe.ExpectMsg<RedundancyStateChanged>(TimeSpan.FromSeconds(2));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Regression guard: the snapshot node id MUST be the canonical <c>host:port</c> form (matching
|
||||
/// ClusterRoleInfo.LocalNode/ToNodeId), or every consumer's
|
||||
/// <c>n.NodeId == _localNode.Value</c> match fails and no node ever learns its role.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void ToNodeId_uses_canonical_host_and_port()
|
||||
{
|
||||
var nodeId = RedundancyStateActor.ToNodeId(
|
||||
new Address("akka.tcp", "otopcua", "central-2", 4053));
|
||||
|
||||
nodeId.Value.ShouldBe("central-2:4053");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Documents the host-less/port-less fallback (<c>:0</c>). Such members are skipped by
|
||||
/// BuildSnapshot's guard, but the helper must still format deterministically.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void ToNodeId_handles_missing_port()
|
||||
{
|
||||
var nodeId = RedundancyStateActor.ToNodeId(new Address("akka.tcp", "otopcua"));
|
||||
|
||||
nodeId.Value.ShouldBe(":0");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,167 @@
|
||||
using Akka.Actor;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Redundancy;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Types;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Historian;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Tests.Harness;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Runtime.Tests.Historian;
|
||||
|
||||
/// <summary>
|
||||
/// TestKit coverage for <see cref="HistorianAdapterActor"/>'s Primary-only historization gate.
|
||||
/// The actor caches this node's <see cref="RedundancyRole"/> from the <c>redundancy-state</c>
|
||||
/// topic and SKIPS the sink enqueue when the local node is Secondary/Detached so a future
|
||||
/// per-node feeder writes exactly once across the warm-redundant pair. Unknown/null role
|
||||
/// default-writes (single-node deploys + the boot window must never silently drop historization).
|
||||
/// </summary>
|
||||
public sealed class HistorianAdapterActorTests : RuntimeActorTestBase
|
||||
{
|
||||
/// <summary>The local node id the gating tests construct the adapter with.</summary>
|
||||
private static readonly NodeId LocalNode = new("node-A");
|
||||
|
||||
/// <summary>A short window we allow the fire-and-forget enqueue to land within.</summary>
|
||||
private static readonly TimeSpan Settle = TimeSpan.FromMilliseconds(500);
|
||||
|
||||
/// <summary>Thread-safe fake sink that records every <see cref="EnqueueAsync"/> call.</summary>
|
||||
private sealed class RecordingSink : IAlarmHistorianSink
|
||||
{
|
||||
private int _count;
|
||||
|
||||
/// <summary>The number of <see cref="EnqueueAsync"/> calls observed so far.</summary>
|
||||
public int EnqueueCount => Volatile.Read(ref _count);
|
||||
|
||||
/// <inheritdoc />
|
||||
public Task EnqueueAsync(AlarmHistorianEvent evt, CancellationToken cancellationToken)
|
||||
{
|
||||
Interlocked.Increment(ref _count);
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public HistorianSinkStatus GetStatus() => new(
|
||||
QueueDepth: 0,
|
||||
DeadLetterDepth: 0,
|
||||
LastDrainUtc: null,
|
||||
LastSuccessUtc: null,
|
||||
LastError: null,
|
||||
DrainState: HistorianDrainState.Idle);
|
||||
}
|
||||
|
||||
/// <summary>Builds a minimal <see cref="AlarmHistorianEvent"/> for the gate tests.</summary>
|
||||
private static AlarmHistorianEvent SampleEvent() => new(
|
||||
AlarmId: "alm-1",
|
||||
EquipmentPath: "Area/Line/Equip",
|
||||
AlarmName: "HiHi",
|
||||
AlarmTypeName: "LimitAlarm",
|
||||
Severity: AlarmSeverity.High,
|
||||
EventKind: "Activated",
|
||||
Message: "level high",
|
||||
User: "system",
|
||||
Comment: null,
|
||||
TimestampUtc: DateTime.UtcNow);
|
||||
|
||||
/// <summary>Tell <paramref name="actor"/> a <see cref="RedundancyStateChanged"/> snapshot marking
|
||||
/// <see cref="LocalNode"/> with <paramref name="role"/> so the gate observes the local role.</summary>
|
||||
private static void TellRedundancyRole(IActorRef actor, RedundancyRole role) =>
|
||||
actor.Tell(new RedundancyStateChanged(
|
||||
new[]
|
||||
{
|
||||
new NodeRedundancyState(
|
||||
NodeId: LocalNode,
|
||||
Role: role,
|
||||
IsClusterLeader: role == RedundancyRole.Primary,
|
||||
IsRoleLeaderForDriver: role == RedundancyRole.Primary,
|
||||
AsOfUtc: DateTime.UtcNow),
|
||||
},
|
||||
CorrelationId.NewId()));
|
||||
|
||||
/// <summary>Default-write (T1): before any redundancy snapshot — the boot window and the steady
|
||||
/// state for single-node deploys — the adapter MUST historize. Constructed WITH a localNode but
|
||||
/// no snapshot sent, so the cached role is unknown ⇒ default-write.</summary>
|
||||
[Fact]
|
||||
public void Default_before_redundancy_state_historizes()
|
||||
{
|
||||
var sink = new RecordingSink();
|
||||
var actor = Sys.ActorOf(HistorianAdapterActor.Props(sink, LocalNode));
|
||||
|
||||
actor.Tell(SampleEvent());
|
||||
|
||||
AwaitAssert(() => sink.EnqueueCount.ShouldBe(1), Settle);
|
||||
}
|
||||
|
||||
/// <summary>Secondary suppression (T2): when the cached local role is Secondary, the adapter MUST
|
||||
/// NOT enqueue to the durable sink (the Primary writes the single copy).</summary>
|
||||
[Fact]
|
||||
public void Secondary_node_does_not_historize()
|
||||
{
|
||||
var sink = new RecordingSink();
|
||||
var actor = Sys.ActorOf(HistorianAdapterActor.Props(sink, LocalNode));
|
||||
|
||||
TellRedundancyRole(actor, RedundancyRole.Secondary);
|
||||
actor.Tell(SampleEvent());
|
||||
|
||||
// Give the (suppressed) fire-and-forget a stable window, then assert nothing landed.
|
||||
ExpectNoMsg(Settle);
|
||||
sink.EnqueueCount.ShouldBe(0);
|
||||
}
|
||||
|
||||
/// <summary>Detached suppression (T3): a Detached node likewise MUST NOT historize.</summary>
|
||||
[Fact]
|
||||
public void Detached_node_does_not_historize()
|
||||
{
|
||||
var sink = new RecordingSink();
|
||||
var actor = Sys.ActorOf(HistorianAdapterActor.Props(sink, LocalNode));
|
||||
|
||||
TellRedundancyRole(actor, RedundancyRole.Detached);
|
||||
actor.Tell(SampleEvent());
|
||||
|
||||
ExpectNoMsg(Settle);
|
||||
sink.EnqueueCount.ShouldBe(0);
|
||||
}
|
||||
|
||||
/// <summary>Primary writes (T4): when the cached local role is Primary, the adapter historizes as
|
||||
/// normal (this is the single copy the durable sink sees).</summary>
|
||||
[Fact]
|
||||
public void Primary_node_historizes()
|
||||
{
|
||||
var sink = new RecordingSink();
|
||||
var actor = Sys.ActorOf(HistorianAdapterActor.Props(sink, LocalNode));
|
||||
|
||||
TellRedundancyRole(actor, RedundancyRole.Primary);
|
||||
actor.Tell(SampleEvent());
|
||||
|
||||
AwaitAssert(() => sink.EnqueueCount.ShouldBe(1), Settle);
|
||||
}
|
||||
|
||||
/// <summary>Absent-node default-historize (T5): a snapshot that mentions only a DIFFERENT node
|
||||
/// must NOT update the local cached role — the actor's own node is absent, so the role stays
|
||||
/// null/unknown and the default-historize path must fire. Partial/stale snapshots MUST NOT
|
||||
/// silently suppress historization for nodes not yet observed.</summary>
|
||||
[Fact]
|
||||
public void Redundancy_snapshot_without_local_node_leaves_role_unknown_and_historizes()
|
||||
{
|
||||
var sink = new RecordingSink();
|
||||
var actor = Sys.ActorOf(HistorianAdapterActor.Props(sink, LocalNode));
|
||||
|
||||
// Send a snapshot that only describes a peer node — the local node is absent.
|
||||
actor.Tell(new RedundancyStateChanged(
|
||||
new[]
|
||||
{
|
||||
new NodeRedundancyState(
|
||||
NodeId: new NodeId("some-other-node"),
|
||||
Role: RedundancyRole.Secondary,
|
||||
IsClusterLeader: false,
|
||||
IsRoleLeaderForDriver: false,
|
||||
AsOfUtc: DateTime.UtcNow),
|
||||
},
|
||||
CorrelationId.NewId()));
|
||||
|
||||
actor.Tell(SampleEvent());
|
||||
|
||||
// Local role is still unknown ⇒ default-historize path: sink must record exactly one enqueue.
|
||||
AwaitAssert(() => sink.EnqueueCount.ShouldBe(1), Settle);
|
||||
}
|
||||
}
|
||||
+193
-2
@@ -5,7 +5,9 @@ using Serilog;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Alerts;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Redundancy;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Types;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Scripting;
|
||||
using ZB.MOM.WW.OtOpcUa.OpcUaServer;
|
||||
@@ -74,15 +76,38 @@ public sealed class ScriptedAlarmHostActorTests : RuntimeActorTestBase
|
||||
return new ScriptedAlarmEngine(upstream, new InMemoryAlarmStateStore(), new ScriptLoggerFactory(logger), logger);
|
||||
}
|
||||
|
||||
/// <summary>The local node id used by the redundancy-gating tests.</summary>
|
||||
private static readonly NodeId LocalNode = new("node-A");
|
||||
|
||||
private (IActorRef Host, DependencyMuxTagUpstreamSource Upstream) Spawn(
|
||||
TestProbe publish, TestProbe mux)
|
||||
TestProbe publish, TestProbe mux, NodeId? localNode = null)
|
||||
{
|
||||
var upstream = new DependencyMuxTagUpstreamSource();
|
||||
var engine = BuildEngine(upstream);
|
||||
var host = Sys.ActorOf(ScriptedAlarmHostActor.Props(publish.Ref, mux.Ref, upstream, engine));
|
||||
var host = Sys.ActorOf(ScriptedAlarmHostActor.Props(publish.Ref, mux.Ref, upstream, engine, localNode));
|
||||
return (host, upstream);
|
||||
}
|
||||
|
||||
/// <summary>Tell the host a <see cref="RedundancyStateChanged"/> snapshot marking
|
||||
/// <paramref name="nodeId"/> (defaults to <see cref="LocalNode"/>) with <paramref name="role"/>
|
||||
/// so the gate observes the local role. Pass a different <paramref name="nodeId"/> to produce a
|
||||
/// snapshot that does NOT contain the host's own node.</summary>
|
||||
private static void TellRedundancyRole(IActorRef host, RedundancyRole role, NodeId? nodeId = null)
|
||||
{
|
||||
var id = nodeId ?? LocalNode;
|
||||
host.Tell(new RedundancyStateChanged(
|
||||
new[]
|
||||
{
|
||||
new NodeRedundancyState(
|
||||
NodeId: id,
|
||||
Role: role,
|
||||
IsClusterLeader: role == RedundancyRole.Primary,
|
||||
IsRoleLeaderForDriver: role == RedundancyRole.Primary,
|
||||
AsOfUtc: DateTime.UtcNow),
|
||||
},
|
||||
CorrelationId.NewId()));
|
||||
}
|
||||
|
||||
/// <summary>Subscribe <paramref name="probe"/> to the <c>alerts</c> DPS topic and wait for the ack.
|
||||
/// The Subscribe is sent FROM the probe so the SubscribeAck returns to it.</summary>
|
||||
private void SubscribeToAlerts(TestProbe probe)
|
||||
@@ -381,4 +406,170 @@ public sealed class ScriptedAlarmHostActorTests : RuntimeActorTestBase
|
||||
var state = publish.FishForMessage<OpcUaPublishActor.AlarmStateUpdate>(m => m.State.Active, Timeout);
|
||||
state.AlarmNodeId.ShouldBe("alm-1");
|
||||
}
|
||||
|
||||
/// <summary>Default-emit (T1): before ANY RedundancyStateChanged snapshot arrives — the boot window,
|
||||
/// and the steady state for single-node deploys (the sole node is always Primary) — the host MUST
|
||||
/// publish the cluster-wide alerts transition. Constructed WITH a localNode but no snapshot sent, so
|
||||
/// the cached local role is unknown ⇒ treated as Primary/emit.</summary>
|
||||
[Fact]
|
||||
public void Emission_is_published_to_alerts_by_default_before_any_redundancy_state()
|
||||
{
|
||||
var publish = CreateTestProbe();
|
||||
var mux = CreateTestProbe();
|
||||
var alerts = CreateTestProbe();
|
||||
SubscribeToAlerts(alerts);
|
||||
|
||||
var (host, _) = Spawn(publish, mux, LocalNode);
|
||||
host.Tell(new ScriptedAlarmHostActor.ApplyScriptedAlarms(new[] { Plan(severity: 800) }));
|
||||
mux.ExpectMsg<DependencyMuxActor.RegisterInterest>(Timeout); // load completed
|
||||
|
||||
// No RedundancyStateChanged sent — local role unknown ⇒ default-emit.
|
||||
host.Tell(new VirtualTagActor.DependencyValueChanged("M.T", 99, DateTime.UtcNow));
|
||||
|
||||
// The OPC UA node write happens AND the alerts transition is published.
|
||||
publish.FishForMessage<OpcUaPublishActor.AlarmStateUpdate>(m => m.State.Active, Timeout);
|
||||
var evt = alerts.ExpectMsg<AlarmTransitionEvent>(Timeout);
|
||||
evt.AlarmId.ShouldBe("alm-1");
|
||||
evt.TransitionKind.ShouldBe("Activated");
|
||||
}
|
||||
|
||||
/// <summary>Secondary suppression (T1): when the cached local role is Secondary, the host MUST NOT
|
||||
/// publish the cluster-wide alerts transition (the Primary publishes the single copy) — but it MUST
|
||||
/// still write the local OPC UA condition node so the secondary's address space stays warm for failover.</summary>
|
||||
[Fact]
|
||||
public void Secondary_node_suppresses_alerts_publish_but_still_writes_opcua()
|
||||
{
|
||||
var publish = CreateTestProbe();
|
||||
var mux = CreateTestProbe();
|
||||
var alerts = CreateTestProbe();
|
||||
SubscribeToAlerts(alerts);
|
||||
|
||||
var (host, _) = Spawn(publish, mux, LocalNode);
|
||||
host.Tell(new ScriptedAlarmHostActor.ApplyScriptedAlarms(new[] { Plan(severity: 800) }));
|
||||
mux.ExpectMsg<DependencyMuxActor.RegisterInterest>(Timeout); // load completed
|
||||
|
||||
// Mark this node Secondary, then activate.
|
||||
TellRedundancyRole(host, RedundancyRole.Secondary);
|
||||
host.Tell(new VirtualTagActor.DependencyValueChanged("M.T", 99, DateTime.UtcNow));
|
||||
|
||||
// The local OPC UA node write is UNGATED — it must still arrive.
|
||||
var state = publish.FishForMessage<OpcUaPublishActor.AlarmStateUpdate>(m => m.State.Active, Timeout);
|
||||
state.AlarmNodeId.ShouldBe("alm-1");
|
||||
|
||||
// The cluster-wide alerts publish is gated off on the secondary.
|
||||
alerts.ExpectNoMsg(TimeSpan.FromMilliseconds(500));
|
||||
}
|
||||
|
||||
/// <summary>Primary publishes (T1): when the cached local role is Primary, the host publishes the
|
||||
/// cluster-wide alerts transition as normal (this is the single copy the fleet sees).</summary>
|
||||
[Fact]
|
||||
public void Primary_node_publishes_alerts()
|
||||
{
|
||||
var publish = CreateTestProbe();
|
||||
var mux = CreateTestProbe();
|
||||
var alerts = CreateTestProbe();
|
||||
SubscribeToAlerts(alerts);
|
||||
|
||||
var (host, _) = Spawn(publish, mux, LocalNode);
|
||||
host.Tell(new ScriptedAlarmHostActor.ApplyScriptedAlarms(new[] { Plan(severity: 800) }));
|
||||
mux.ExpectMsg<DependencyMuxActor.RegisterInterest>(Timeout); // load completed
|
||||
|
||||
// Mark this node Primary, then activate.
|
||||
TellRedundancyRole(host, RedundancyRole.Primary);
|
||||
host.Tell(new VirtualTagActor.DependencyValueChanged("M.T", 99, DateTime.UtcNow));
|
||||
|
||||
publish.FishForMessage<OpcUaPublishActor.AlarmStateUpdate>(m => m.State.Active, Timeout);
|
||||
var evt = alerts.ExpectMsg<AlarmTransitionEvent>(Timeout);
|
||||
evt.AlarmId.ShouldBe("alm-1");
|
||||
evt.TransitionKind.ShouldBe("Activated");
|
||||
}
|
||||
|
||||
/// <summary>Inbound command ungated by role (T1): the alerts-publish gate must NOT affect inbound
|
||||
/// command processing. Under a Secondary role, an AlarmCommand("Acknowledge") for an owned, active
|
||||
/// alarm still drives the engine — observed via the resulting AlarmStateUpdate(Acknowledged=true)
|
||||
/// (the OPC UA node write is ungated so the secondary's engine state + address space stay consistent).</summary>
|
||||
[Fact]
|
||||
public void Inbound_AlarmCommand_is_processed_regardless_of_role()
|
||||
{
|
||||
var publish = CreateTestProbe();
|
||||
var mux = CreateTestProbe();
|
||||
var (host, _) = Spawn(publish, mux, LocalNode);
|
||||
|
||||
host.Tell(new ScriptedAlarmHostActor.ApplyScriptedAlarms(new[] { Plan(id: "alm-1", depRef: "M.T") }));
|
||||
mux.ExpectMsg<DependencyMuxActor.RegisterInterest>(Timeout); // load completed
|
||||
|
||||
// Mark this node Secondary — the alerts publish is gated, but command processing is NOT.
|
||||
TellRedundancyRole(host, RedundancyRole.Secondary);
|
||||
|
||||
// Activate so there is something to acknowledge.
|
||||
host.Tell(new VirtualTagActor.DependencyValueChanged("M.T", 99, DateTime.UtcNow));
|
||||
publish.FishForMessage<OpcUaPublishActor.AlarmStateUpdate>(m => m.State.Active && !m.State.Acknowledged, Timeout);
|
||||
|
||||
// Acknowledge via the command topic — the engine must process it even on the secondary.
|
||||
host.Tell(new AlarmCommand(
|
||||
AlarmId: "alm-1", Operation: "Acknowledge", User: "alice", Comment: "ack-note", UnshelveAtUtc: null));
|
||||
|
||||
var acked = publish.FishForMessage<OpcUaPublishActor.AlarmStateUpdate>(m => m.State.Acknowledged, Timeout);
|
||||
acked.AlarmNodeId.ShouldBe("alm-1");
|
||||
acked.State.Acknowledged.ShouldBeTrue();
|
||||
}
|
||||
|
||||
/// <summary>Detached suppression (A1): when the cached local role is Detached the host MUST NOT
|
||||
/// publish the cluster-wide alerts transition (identical gating as Secondary) — but it MUST still
|
||||
/// write the local OPC UA condition node so the detached node's address space stays warm.</summary>
|
||||
[Fact]
|
||||
public void Detached_node_suppresses_alerts_publish_but_still_writes_opcua()
|
||||
{
|
||||
var publish = CreateTestProbe();
|
||||
var mux = CreateTestProbe();
|
||||
var alerts = CreateTestProbe();
|
||||
SubscribeToAlerts(alerts);
|
||||
|
||||
var (host, _) = Spawn(publish, mux, LocalNode);
|
||||
host.Tell(new ScriptedAlarmHostActor.ApplyScriptedAlarms(new[] { Plan(severity: 800) }));
|
||||
mux.ExpectMsg<DependencyMuxActor.RegisterInterest>(Timeout); // load completed
|
||||
|
||||
// Mark this node Detached, then activate.
|
||||
TellRedundancyRole(host, RedundancyRole.Detached);
|
||||
host.Tell(new VirtualTagActor.DependencyValueChanged("M.T", 99, DateTime.UtcNow));
|
||||
|
||||
// The local OPC UA node write is UNGATED — it must still arrive.
|
||||
var state = publish.FishForMessage<OpcUaPublishActor.AlarmStateUpdate>(m => m.State.Active, Timeout);
|
||||
state.AlarmNodeId.ShouldBe("alm-1");
|
||||
|
||||
// The cluster-wide alerts publish is gated off on a detached node.
|
||||
alerts.ExpectNoMsg(TimeSpan.FromMilliseconds(500));
|
||||
}
|
||||
|
||||
/// <summary>Absent-node default-emit (A1): a <see cref="RedundancyStateChanged"/> snapshot that
|
||||
/// contains ONLY other nodes (the host's own <see cref="LocalNode"/> is absent) must leave the
|
||||
/// cached local role unchanged (null/unknown) — the host therefore defaults to emit, publishing
|
||||
/// the alerts transition AND writing the OPC UA node, exactly as in the boot-window case.</summary>
|
||||
[Fact]
|
||||
public void Redundancy_snapshot_without_local_node_leaves_role_unknown_and_emits()
|
||||
{
|
||||
var publish = CreateTestProbe();
|
||||
var mux = CreateTestProbe();
|
||||
var alerts = CreateTestProbe();
|
||||
SubscribeToAlerts(alerts);
|
||||
|
||||
var (host, _) = Spawn(publish, mux, LocalNode);
|
||||
host.Tell(new ScriptedAlarmHostActor.ApplyScriptedAlarms(new[] { Plan(severity: 800) }));
|
||||
mux.ExpectMsg<DependencyMuxActor.RegisterInterest>(Timeout); // load completed
|
||||
|
||||
// Send a snapshot that mentions only a DIFFERENT node — LocalNode is absent.
|
||||
// The host cannot determine its own role from this snapshot, so the cached role
|
||||
// stays null (unknown) ⇒ treated as Primary ⇒ default-emit path.
|
||||
TellRedundancyRole(host, RedundancyRole.Primary, nodeId: new NodeId("some-other-node"));
|
||||
|
||||
host.Tell(new VirtualTagActor.DependencyValueChanged("M.T", 99, DateTime.UtcNow));
|
||||
|
||||
// Both the OPC UA write AND the alerts publish must arrive (default-emit, role unknown).
|
||||
var state = publish.FishForMessage<OpcUaPublishActor.AlarmStateUpdate>(m => m.State.Active, Timeout);
|
||||
state.AlarmNodeId.ShouldBe("alm-1");
|
||||
|
||||
var evt = alerts.ExpectMsg<AlarmTransitionEvent>(Timeout);
|
||||
evt.AlarmId.ShouldBe("alm-1");
|
||||
evt.TransitionKind.ShouldBe("Activated");
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user