# Alarm Ack/Shelve Follow-ups — Design

**Date:** 2026-06-11
**Status:** Approved (brainstorming) — ready for implementation plan.

## Goal

Resolve the six notes/follow-ups left open by the T17–T24 inbound-alarm-ack work
(merged to master `bc9843d2`): the redundancy **double-emit**, the deferred Timed-shelve
UI, two T21 minors, the docker-dev rig cleanup, and two pre-existing Layer-1 gaps (Galaxy
reconnect, the live-pill). Scope decided in brainstorming: **everything**, including the
pre-existing L1 gaps. Double-emit approach decided: **primary-only at the source**.

---

## 1 — Redundancy double-emit fix (core; high-risk)

### Problem (verified)
Both central nodes run roles `admin,driver` in the same MAIN cluster. Each spawns its own
`ScriptedAlarmHostActor` + `ScriptedAlarmEngine` (`DriverHostActor.SpawnScriptedAlarmHost`,
per-node, no singleton) and, for a single-cluster artifact, loads the **same** alarms
(`DeploymentArtifact.ParseComposition` returns `ClusterFilterMode.None` → unfiltered). Both
independently evaluate and both hit `ScriptedAlarmHostActor.OnEngineEmission`:

- line 261 `_publishActor.Tell(new OpcUaPublishActor.AlarmStateUpdate(...))` — writes the OPC
  UA node (directed, local).
- line 278 `_mediator.Tell(new Publish(AlertsTopic /* "alerts" */, evt))` — **cluster-wide,
  ungated** → every transition appears **twice** on `/alerts`. Inbound ack/shelve commands
  are likewise processed by both nodes' engines.

`HistorianAdapterActor` (`Runtime/Historian/`, registered **per-node** in
`Runtime/ServiceCollectionExtensions.cs:146`, name `historian-adapter`) also consumes the
`alerts` topic, so a single publish is historized **once per node** → double DB writes in
production (invisible in docker-dev: `NullAlarmHistorianSink`).

### Existing signal to mirror
`RedundancyStateActor` (admin singleton, `ControlPlane/Redundancy/RedundancyStateActor.cs:90-114`)
publishes `RedundancyStateChanged` to the `redundancy-state` DPS topic. Per-node
`NodeRedundancyState` carries `RedundancyRole` (`Primary`/`Secondary`/`Detached`) +
`IsRoleLeaderForDriver`; **Primary = the driver-role cluster leader**. `OpcUaPublishActor`
already subscribes to this topic (`OpcUaPublishActor.cs:30,147,156,335`) to drive its
ServiceLevel — the exact pattern to copy (`Subscribe(RedundancyStateTopic, Self)` in
PreStart; `Receive<RedundancyStateChanged>`; read `msg.Nodes.FirstOrDefault(n => n.NodeId ==
localNode)?.Role`).

### Decision — gate **only the cluster-wide emission**, on `Primary`
- **Emission gate (task A1):** `ScriptedAlarmHostActor` subscribes to `redundancy-state`,
  caches the local role, and **skips the `alerts` publish (line 278) when not Primary**.
  **Default = emit until a `RedundancyStateChanged` says this node is Secondary/Detached** —
  so single-node deploys (sole node is always the driver leader = Primary) and the boot
  window never drop transitions. The OPC UA node write (line 261) and **inbound command
  processing stay UNGATED** — the secondary must keep its address space warm and its engine
  state consistent for failover; clients only ever see the Primary via ServiceLevel, so the
  secondary's node writes + its (subscriber-less) condition events are harmless.
- **Historian gate (task A2):** `HistorianAdapterActor` subscribes to `redundancy-state`,
  caches the local role, and **skips the sink write when not Primary**. Gives exactly-once
  historization for **all** alarm sources (native Galaxy/AB-CIP too), not just scripted.
- **Edge:** a brief failover window (old Primary gone, new not yet elected) may drop a
  transition/historization — acceptable, and identical to the existing ServiceLevel handoff
  behaviour.

### Tests
TestKit: with a Secondary `RedundancyStateChanged`, `OnEngineEmission` does NOT publish to
`alerts` but DOES still `Tell` the OPC UA `AlarmStateUpdate`; with Primary (or before any
state) it publishes. Inbound `AlarmCommand` is still processed regardless of role. Historian:
a Secondary skips the sink write, a Primary writes (fake sink + role injection).

---

## 2 — Timed-shelve picker UI (small)

`Alerts.razor` currently exposes Ack / Shelve(OneShot) / Unshelve. The `Timed` backend is
fully wired + tested (`ShelveAlarmCommand{ShelveKind.Timed, UnshelveAtUtc}` →
`AdminOperationsActor` → `engine.TimedShelveAsync`). Add a small duration input (a minutes/
seconds number box, or a `datetime-local`) to the shelve control on each row and call
`ShelveAlarmAsync(kind: Timed, unshelveAtUtc: now + duration)`. Razor-only; **no backend
change**; proven by docker-dev live-verify (no bUnit). Keep the existing OneShot/Unshelve
buttons.

## 3 — T21 minors (small)

- **Chip auto-clear:** the `_opResult*` chip persists until the next action. Add the ~8s
  auto-clear timer pattern `DriverStatusPanel.razor` already uses (a `Task.Delay` →
  `InvokeAsync(StateHasChanged)` that clears the per-row result).
- **CorrelationId consistency:** `AcknowledgeAlarmCommand`/`ShelveAlarmCommand` use a bare
  `Guid CorrelationId`; the project's other control-plane commands use the `CorrelationId`
  wrapper type. Switch both records + `AdminOperationsClient` (`CorrelationId.NewId()`) +
  the actor reply contract + tests to the wrapper for uniform correlation tracing.

## 4 — Rig cleanup (operational, last)

Delete the docker-dev seed artifacts left for live-verify: the `t12-overheat` scripted alarm,
the `SC-ba675b168a85` predicate script, the `layer0-logcheck` vtag + script; revert
filler-02's inert `cycle-time-s` logger line to `return ctx.GetTag("TestMachine_002.TestDuration").Value;`.
Redeploy (`POST /api/deployments`, `X-Api-Key: docker-dev-deploy-key`). DB/redeploy, **not a
code change** — done last so the rig stays available for verifying tasks 1–3/6.

## 5 — Galaxy reconnect recreate (high-risk)

### Problem (verified)
`GalaxyMxSession.ConnectAsync` (`Driver.Galaxy/Runtime/GalaxyMxSession.cs:58-69`) is
idempotent: `if (_session is not null) return;`. When the gateway restarts and the session
goes Faulted/NotFound, `_session` is still a non-null (dead) handle, so `ConnectAsync` is a
silent no-op. `GalaxyDriver.ReopenAsync` (`GalaxyDriver.cs:289`) calls it expecting a
reconnect → no-op; `ReconnectSupervisor.RecoveryLoopAsync`
(`Driver.Galaxy/Runtime/ReconnectSupervisor.cs:158-186`) sees reopen "succeed", proceeds to
replay (which fails on the dead session), and **loops forever** (backoff capped 30s).

### Decision — recreate on reopen
Add a recreate path to `GalaxyMxSession` (e.g. `RecreateAsync`/`DisposeSessionForRecreationAsync`)
that disposes + nulls `_session` and `_ownedClient`, and have `ReopenAsync` call it **before**
`ConnectAsync` so a reopen always routes through the happy-path create (`OpenSessionAsync` +
`RegisterAsync`). Confirm what status/exception marks Faulted/NotFound and that the dispose is
safe (gRPC channel teardown). Keep the supervisor's backoff loop; it now actually recovers.

### Tests
A reconnect test asserting that after a faulted session, the reopen path **creates a new
session** (new handle / `OpenSessionAsync` called again) rather than no-op'ing. Mirror any
existing `ReconnectSupervisor`/session tests.

## 6 — Live-pill circuit health (standard)

### Problem (verified)
`ScriptLog.razor:122` and `Alerts.razor:132` set `_connected = true` once in
`OnInitialized` and never update it; the pill markup binds to that set-once bool.
`IInProcessBroadcaster<T>` (`AdminUI/Hubs/IInProcessBroadcaster.cs`) exposes only
`Received` + `Publish` — no health signal.

### Decision
- Extend `IInProcessBroadcaster<T>` with a connection-health signal (`bool IsConnected` +
  `event … ConnectionStateChanged`). The bridge actors (`ScriptLogSignalRBridge` /
  `AlertSignalRBridge`) set it from their DPS-subscription health (SubscribeAck up / failure
  down). The razors bind the pill to it and subscribe/unsubscribe like
  `DriverStatusPanel.razor` (`OnConnectionStateChanged` → `InvokeAsync(StateHasChanged)`).
- **Dead-circuit case** (node recreate kills the server-side circuit — the component is dead
  and cannot self-update its pill): this is Blazor Server's built-in reconnection concern.
  **Verify the default reconnect overlay is present/visible** (it is what actually signals a
  dropped circuit) rather than trying to fake liveness from a dead component. If absent, add
  the standard Blazor reconnect UI.

### Tests
Broadcaster unit test for the new health signal + `SetConnected` propagation. Razor proven by
docker-dev live-verify (no bUnit).

---

## Sequencing & risk

| Item | Risk | Notes |
|---|---|---|
| 1 redundancy double-emit (A1 emit gate, A2 historian gate) | high-risk | independent subsystem; A1∥A2 (different files) |
| 5 Galaxy reconnect | high-risk | independent subsystem (Galaxy driver) |
| 2 Timed-shelve picker | small | razor-only, live-verify |
| 3a chip auto-clear / 3b CorrelationId | small | razor / mechanical refactor |
| 6 live-pill | standard | interface + bridge + 2 razors |
| 4 rig cleanup | trivial/operational | last |

The two high-risk items (1, 5) are in different subsystems and can run in parallel with each
other and with the UI items (2, 3, 6). Rig cleanup (4) is last. TDD where there's logic; UI
proven by docker-dev `/run` live-verify (agent drives — login disabled on docker-dev).

## Hard rules

Stage by explicit path (never `git add .`); never stage `sql_login.txt` /
`src/Server/.../Host/pki/`; never echo the gateway API key into a **new** tracked file; no
force-push, no `--no-verify`; **no Configuration entity / EF migration change** (none of these
items needs one). Build on a feature branch off `master`.
