docs(design): alarm ack/shelve follow-ups (redundancy double-emit, Timed-shelve UI, T21 minors, rig cleanup, Galaxy reconnect, live-pill)
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,167 @@
|
|||||||
|
# Alarm Ack/Shelve Follow-ups — Design
|
||||||
|
|
||||||
|
**Date:** 2026-06-11
|
||||||
|
**Status:** Approved (brainstorming) — ready for implementation plan.
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
Resolve the six notes/follow-ups left open by the T17–T24 inbound-alarm-ack work
|
||||||
|
(merged to master `bc9843d2`): the redundancy **double-emit**, the deferred Timed-shelve
|
||||||
|
UI, two T21 minors, the docker-dev rig cleanup, and two pre-existing Layer-1 gaps (Galaxy
|
||||||
|
reconnect, the live-pill). Scope decided in brainstorming: **everything**, including the
|
||||||
|
pre-existing L1 gaps. Double-emit approach decided: **primary-only at the source**.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1 — Redundancy double-emit fix (core; high-risk)
|
||||||
|
|
||||||
|
### Problem (verified)
|
||||||
|
Both central nodes run roles `admin,driver` in the same MAIN cluster. Each spawns its own
|
||||||
|
`ScriptedAlarmHostActor` + `ScriptedAlarmEngine` (`DriverHostActor.SpawnScriptedAlarmHost`,
|
||||||
|
per-node, no singleton) and, for a single-cluster artifact, loads the **same** alarms
|
||||||
|
(`DeploymentArtifact.ParseComposition` returns `ClusterFilterMode.None` → unfiltered). Both
|
||||||
|
independently evaluate and both hit `ScriptedAlarmHostActor.OnEngineEmission`:
|
||||||
|
|
||||||
|
- line 261 `_publishActor.Tell(new OpcUaPublishActor.AlarmStateUpdate(...))` — writes the OPC
|
||||||
|
UA node (directed, local).
|
||||||
|
- line 278 `_mediator.Tell(new Publish(AlertsTopic /* "alerts" */, evt))` — **cluster-wide,
|
||||||
|
ungated** → every transition appears **twice** on `/alerts`. Inbound ack/shelve commands
|
||||||
|
are likewise processed by both nodes' engines.
|
||||||
|
|
||||||
|
`HistorianAdapterActor` (`Runtime/Historian/`, registered **per-node** in
|
||||||
|
`Runtime/ServiceCollectionExtensions.cs:146`, name `historian-adapter`) also consumes the
|
||||||
|
`alerts` topic, so a single publish is historized **once per node** → double DB writes in
|
||||||
|
production (invisible in docker-dev: `NullAlarmHistorianSink`).
|
||||||
|
|
||||||
|
### Existing signal to mirror
|
||||||
|
`RedundancyStateActor` (admin singleton, `ControlPlane/Redundancy/RedundancyStateActor.cs:90-114`)
|
||||||
|
publishes `RedundancyStateChanged` to the `redundancy-state` DPS topic. Per-node
|
||||||
|
`NodeRedundancyState` carries `RedundancyRole` (`Primary`/`Secondary`/`Detached`) +
|
||||||
|
`IsRoleLeaderForDriver`; **Primary = the driver-role cluster leader**. `OpcUaPublishActor`
|
||||||
|
already subscribes to this topic (`OpcUaPublishActor.cs:30,147,156,335`) to drive its
|
||||||
|
ServiceLevel — the exact pattern to copy (`Subscribe(RedundancyStateTopic, Self)` in
|
||||||
|
PreStart; `Receive<RedundancyStateChanged>`; read `msg.Nodes.FirstOrDefault(n => n.NodeId ==
|
||||||
|
localNode)?.Role`).
|
||||||
|
|
||||||
|
### Decision — gate **only the cluster-wide emission**, on `Primary`
|
||||||
|
- **Emission gate (task A1):** `ScriptedAlarmHostActor` subscribes to `redundancy-state`,
|
||||||
|
caches the local role, and **skips the `alerts` publish (line 278) when not Primary**.
|
||||||
|
**Default = emit until a `RedundancyStateChanged` says this node is Secondary/Detached** —
|
||||||
|
so single-node deploys (sole node is always the driver leader = Primary) and the boot
|
||||||
|
window never drop transitions. The OPC UA node write (line 261) and **inbound command
|
||||||
|
processing stay UNGATED** — the secondary must keep its address space warm and its engine
|
||||||
|
state consistent for failover; clients only ever see the Primary via ServiceLevel, so the
|
||||||
|
secondary's node writes + its (subscriber-less) condition events are harmless.
|
||||||
|
- **Historian gate (task A2):** `HistorianAdapterActor` subscribes to `redundancy-state`,
|
||||||
|
caches the local role, and **skips the sink write when not Primary**. Gives exactly-once
|
||||||
|
historization for **all** alarm sources (native Galaxy/AB-CIP too), not just scripted.
|
||||||
|
- **Edge:** a brief failover window (old Primary gone, new not yet elected) may drop a
|
||||||
|
transition/historization — acceptable, and identical to the existing ServiceLevel handoff
|
||||||
|
behaviour.
|
||||||
|
|
||||||
|
### Tests
|
||||||
|
TestKit: with a Secondary `RedundancyStateChanged`, `OnEngineEmission` does NOT publish to
|
||||||
|
`alerts` but DOES still `Tell` the OPC UA `AlarmStateUpdate`; with Primary (or before any
|
||||||
|
state) it publishes. Inbound `AlarmCommand` is still processed regardless of role. Historian:
|
||||||
|
a Secondary skips the sink write, a Primary writes (fake sink + role injection).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2 — Timed-shelve picker UI (small)
|
||||||
|
|
||||||
|
`Alerts.razor` currently exposes Ack / Shelve(OneShot) / Unshelve. The `Timed` backend is
|
||||||
|
fully wired + tested (`ShelveAlarmCommand{ShelveKind.Timed, UnshelveAtUtc}` →
|
||||||
|
`AdminOperationsActor` → `engine.TimedShelveAsync`). Add a small duration input (a minutes/
|
||||||
|
seconds number box, or a `datetime-local`) to the shelve control on each row and call
|
||||||
|
`ShelveAlarmAsync(kind: Timed, unshelveAtUtc: now + duration)`. Razor-only; **no backend
|
||||||
|
change**; proven by docker-dev live-verify (no bUnit). Keep the existing OneShot/Unshelve
|
||||||
|
buttons.
|
||||||
|
|
||||||
|
## 3 — T21 minors (small)
|
||||||
|
|
||||||
|
- **Chip auto-clear:** the `_opResult*` chip persists until the next action. Add the ~8s
|
||||||
|
auto-clear timer pattern `DriverStatusPanel.razor` already uses (a `Task.Delay` →
|
||||||
|
`InvokeAsync(StateHasChanged)` that clears the per-row result).
|
||||||
|
- **CorrelationId consistency:** `AcknowledgeAlarmCommand`/`ShelveAlarmCommand` use a bare
|
||||||
|
`Guid CorrelationId`; the project's other control-plane commands use the `CorrelationId`
|
||||||
|
wrapper type. Switch both records + `AdminOperationsClient` (`CorrelationId.NewId()`) +
|
||||||
|
the actor reply contract + tests to the wrapper for uniform correlation tracing.
|
||||||
|
|
||||||
|
## 4 — Rig cleanup (operational, last)
|
||||||
|
|
||||||
|
Delete the docker-dev seed artifacts left for live-verify: the `t12-overheat` scripted alarm,
|
||||||
|
the `SC-ba675b168a85` predicate script, the `layer0-logcheck` vtag + script; revert
|
||||||
|
filler-02's inert `cycle-time-s` logger line to `return ctx.GetTag("TestMachine_002.TestDuration").Value;`.
|
||||||
|
Redeploy (`POST /api/deployments`, `X-Api-Key: docker-dev-deploy-key`). DB/redeploy, **not a
|
||||||
|
code change** — done last so the rig stays available for verifying tasks 1–3/6.
|
||||||
|
|
||||||
|
## 5 — Galaxy reconnect recreate (high-risk)
|
||||||
|
|
||||||
|
### Problem (verified)
|
||||||
|
`GalaxyMxSession.ConnectAsync` (`Driver.Galaxy/Runtime/GalaxyMxSession.cs:58-69`) is
|
||||||
|
idempotent: `if (_session is not null) return;`. When the gateway restarts and the session
|
||||||
|
goes Faulted/NotFound, `_session` is still a non-null (dead) handle, so `ConnectAsync` is a
|
||||||
|
silent no-op. `GalaxyDriver.ReopenAsync` (`GalaxyDriver.cs:289`) calls it expecting a
|
||||||
|
reconnect → no-op; `ReconnectSupervisor.RecoveryLoopAsync`
|
||||||
|
(`Driver.Galaxy/Runtime/ReconnectSupervisor.cs:158-186`) sees reopen "succeed", proceeds to
|
||||||
|
replay (which fails on the dead session), and **loops forever** (backoff capped 30s).
|
||||||
|
|
||||||
|
### Decision — recreate on reopen
|
||||||
|
Add a recreate path to `GalaxyMxSession` (e.g. `RecreateAsync`/`DisposeSessionForRecreationAsync`)
|
||||||
|
that disposes + nulls `_session` and `_ownedClient`, and have `ReopenAsync` call it **before**
|
||||||
|
`ConnectAsync` so a reopen always routes through the happy-path create (`OpenSessionAsync` +
|
||||||
|
`RegisterAsync`). Confirm what status/exception marks Faulted/NotFound and that the dispose is
|
||||||
|
safe (gRPC channel teardown). Keep the supervisor's backoff loop; it now actually recovers.
|
||||||
|
|
||||||
|
### Tests
|
||||||
|
A reconnect test asserting that after a faulted session, the reopen path **creates a new
|
||||||
|
session** (new handle / `OpenSessionAsync` called again) rather than no-op'ing. Mirror any
|
||||||
|
existing `ReconnectSupervisor`/session tests.
|
||||||
|
|
||||||
|
## 6 — Live-pill circuit health (standard)
|
||||||
|
|
||||||
|
### Problem (verified)
|
||||||
|
`ScriptLog.razor:122` and `Alerts.razor:132` set `_connected = true` once in
|
||||||
|
`OnInitialized` and never update it; the pill markup binds to that set-once bool.
|
||||||
|
`IInProcessBroadcaster<T>` (`AdminUI/Hubs/IInProcessBroadcaster.cs`) exposes only
|
||||||
|
`Received` + `Publish` — no health signal.
|
||||||
|
|
||||||
|
### Decision
|
||||||
|
- Extend `IInProcessBroadcaster<T>` with a connection-health signal (`bool IsConnected` +
|
||||||
|
`event … ConnectionStateChanged`). The bridge actors (`ScriptLogSignalRBridge` /
|
||||||
|
`AlertSignalRBridge`) set it from their DPS-subscription health (SubscribeAck up / failure
|
||||||
|
down). The razors bind the pill to it and subscribe/unsubscribe like
|
||||||
|
`DriverStatusPanel.razor` (`OnConnectionStateChanged` → `InvokeAsync(StateHasChanged)`).
|
||||||
|
- **Dead-circuit case** (node recreate kills the server-side circuit — the component is dead
|
||||||
|
and cannot self-update its pill): this is Blazor Server's built-in reconnection concern.
|
||||||
|
**Verify the default reconnect overlay is present/visible** (it is what actually signals a
|
||||||
|
dropped circuit) rather than trying to fake liveness from a dead component. If absent, add
|
||||||
|
the standard Blazor reconnect UI.
|
||||||
|
|
||||||
|
### Tests
|
||||||
|
Broadcaster unit test for the new health signal + `SetConnected` propagation. Razor proven by
|
||||||
|
docker-dev live-verify (no bUnit).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Sequencing & risk
|
||||||
|
|
||||||
|
| Item | Risk | Notes |
|
||||||
|
|---|---|---|
|
||||||
|
| 1 redundancy double-emit (A1 emit gate, A2 historian gate) | high-risk | independent subsystem; A1∥A2 (different files) |
|
||||||
|
| 5 Galaxy reconnect | high-risk | independent subsystem (Galaxy driver) |
|
||||||
|
| 2 Timed-shelve picker | small | razor-only, live-verify |
|
||||||
|
| 3a chip auto-clear / 3b CorrelationId | small | razor / mechanical refactor |
|
||||||
|
| 6 live-pill | standard | interface + bridge + 2 razors |
|
||||||
|
| 4 rig cleanup | trivial/operational | last |
|
||||||
|
|
||||||
|
The two high-risk items (1, 5) are in different subsystems and can run in parallel with each
|
||||||
|
other and with the UI items (2, 3, 6). Rig cleanup (4) is last. TDD where there's logic; UI
|
||||||
|
proven by docker-dev `/run` live-verify (agent drives — login disabled on docker-dev).
|
||||||
|
|
||||||
|
## Hard rules
|
||||||
|
|
||||||
|
Stage by explicit path (never `git add .`); never stage `sql_login.txt` /
|
||||||
|
`src/Server/.../Host/pki/`; never echo the gateway API key into a **new** tracked file; no
|
||||||
|
force-push, no `--no-verify`; **no Configuration entity / EF migration change** (none of these
|
||||||
|
items needs one). Build on a feature branch off `master`.
|
||||||
Reference in New Issue
Block a user