Files
lmxopcua/docs/plans/alarms-over-gateway.md
Joseph Doherty 8d0e13e69e docs: alarms-over-gateway plan — add Track D deployment refresh
After A/B/C all merge, the running services on C:\publish need to be
refreshed before the Galaxy alarm-event family flows end-to-end. Add
PR D.1: a Refresh-Services.ps1 script + runbook for stopping in
reverse-dependency order, restaging binaries from the build outputs,
restarting in forward-dependency order, and capturing a smoke-run
artifact.

D.1 gates B.5 (docs sweep) — the documentation records the
as-deployed shape, so the deployment has to be live first.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-30 15:11:23 -04:00

838 lines
40 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Plan — alarms over the mxaccessgw gateway
Coordinated epic across two repos:
- **`lmxopcua`** (this repo) — `c:\Users\dohertj2\Desktop\lmxopcua\`
- **`mxaccessgw`** — `c:\Users\dohertj2\Desktop\mxaccessgw\`
## Why
PR 7.2 (2026-04-30, commit `ae7106d`) retired the in-process v1 Galaxy stack
(`Driver.Galaxy.Host` / `.Proxy` / `.Shared` + `OtOpcUaGalaxyHost` Windows
service) and migrated Galaxy access to the in-process `GalaxyDriver` over
mxaccessgw's gRPC. In doing so, three v1 capabilities regressed:
1. **Native MxAccess alarm-event metadata** — v1's `GalaxyAlarmTracker`
surfaced rich alarm transitions (operator comment, original raise time,
ack time, alarm category, native severity). The current architecture
reconstructs Part 9 transitions by subscribing to four sub-attribute
value updates (`InAlarm`, `Acked`, `Priority`, `Description`) — fine for
raise/clear but loses everything else.
2. **Native MxAccess Acknowledge semantics** — v1 called the MxAccess ack
API directly from `GalaxyAlarmTracker`. Today, OPC UA acks are written
into the `AckMsgWriteRef` sub-attribute — semantically valid but a
round-trip through the value path that loses operator-comment fidelity.
3. **Alarm-historian write-back path for non-Galaxy alarm sources.**
v1's `GalaxyHistorianWriter` implemented `IAlarmHistorianWriter` and
forwarded *scripted-alarm* transitions (and any future non-Galaxy
alarm source — AB CIP ALMD, OpcUaClient A&E, etc.) back to AVEVA
Historian via `aahClientManaged`. PR 7.2 deleted it.
`Phase7Composer.ResolveHistorianSink` now finds no writer and falls
back to `NullAlarmHistorianSink`, so **scripted-alarm transitions
queue locally and silently discard.** Galaxy-native alarms (with
`$Alarm*` extensions) reach AVEVA Historian via System Platform's
own `HistorizeToAveva` toggle on the Galaxy template — that path
was never broken and is not in scope for this epic.
`gateway.md` (mxaccessgw, line 8) explicitly commits the gateway to "full
MXAccess parity… preserve MXAccess behavior first… **native MXAccess event
families**." Today's gateway proto exposes only data-change families. Closing
the alarm regression and fulfilling that parity statement are the same task.
## Goals
- Restore all three regressed capabilities to feature parity with v1.
- Keep the v2 architectural split — gateway owns MxAccess transport;
lmxopcua owns OPC UA Part 9 semantics, ACL/role enforcement, and
multi-source aggregation (driver-native + scripted + sub-attribute).
- Preserve the value-driven sub-attribute path as a fallback for Galaxy
templates that don't carry `$Alarm*` extensions.
- Land the work as a sequence of small, independently-reviewable PRs that
alternate between repos in dependency order.
## Non-goals
- Reimplementing the Part 9 state machine inside mxaccessgw. The gateway
stays UA-agnostic.
- Reworking the LDAP role-grant or OPC UA AlarmAck ACL surface — those
already exist and route through `Server/Alarms/IAlarmAcknowledger`.
- Adding alarm support to non-Galaxy drivers (AbCip / FOCAS / OpcUaClient
already have their own `IAlarmSource` implementations; Modbus / S7 /
AbLegacy / TwinCAT don't have a native alarm bus and are out of scope).
- Altering Galaxy template conventions or `$Alarm*` extensions in the
customer's Galaxy.
## Before → after
**Today (post-PR 7.2):**
```
MxAccess COM (gateway worker)
│ data-change events only on the MxEvent stream
GalaxyDriver (no IAlarmSource)
│ IWritable / ISubscribable / ITagDiscovery only
DriverNodeManager
├─ subscribes to four $Alarm* sub-attributes per condition
├─ AlarmConditionService rebuilds Part 9 transitions from value updates
└─ DriverWritableAcknowledger writes AckMsgWriteRef on ack
Phase7Composer.ResolveHistorianSink → NullAlarmHistorianSink
(scripted-alarm transitions queue → silently discarded)
```
**After this epic:**
```
MxAccess COM (gateway worker)
│ data-change ──┐
│ alarm-transition │
│ write-complete ├─► single MxEvent stream (new family added)
▼ ▼
GalaxyDriver : ITagDiscovery, IReadable, IWritable, ISubscribable, IRediscoverable,
IHostConnectivityProbe, IAlarmSource ← restored
├─ EventPump dispatches OnAlarmTransition family → IAlarmSource.OnAlarmEvent
├─ AcknowledgeAsync → gateway RPC AcknowledgeAlarm
└─ QueryActiveAlarmsAsync → gateway RPC QueryActiveAlarms (ConditionRefresh)
DriverNodeManager
├─ rich alarm events from IAlarmSource.OnAlarmEvent → AlarmConditionService
├─ value-driven sub-attribute path STILL WORKS for templates without $Alarm
├─ DriverWritableAcknowledger preserved as fallback for the value path
└─ ScriptedAlarmEngine output continues to feed AlarmConditionService
Phase7Composer.ResolveHistorianSink → GatewayAlarmHistorianWriter
├─ scripted-alarm transitions → SqliteStoreAndForwardSink
└─ drain worker → gateway RPC WriteHistorianEvent → AVEVA Historian
```
## Architecture decisions
**D1 — Where the Part 9 state machine runs.** Stays in lmxopcua's
`AlarmConditionService`. Gateway is UA-agnostic. ScriptedAlarmEngine produces
Part 9 transitions with no MxAccess origin; the aggregator must live where all
sources converge.
**D2 — Where authz on Acknowledge runs.** Stays in lmxopcua. The OPC UA
`AlarmConditionState.OnAcknowledge` delegate already checks the session's
roles for `AlarmAck` against the LDAP/role-grant ACL. The gateway should
never be reachable in a way that bypasses that check.
**D3 — How rich alarm events reach OPC UA clients.** New `MxEventFamily`
on the existing `StreamEvents` RPC (no second stream). Adds latency
parity with data-change events, reuses the bounded-channel + worker-side
delivery semantics already documented in `gateway.md`.
**D4 — Sub-attribute fallback path stays.** Some Galaxy templates won't
have `$Alarm*` extensions yet; the existing value-driven path remains the
only way to surface alarms for those templates. Both paths feed
`AlarmConditionService`. Driver-native events take precedence when both
are present (more authoritative, lower latency).
**D5 — Where the historian writer lives.** In the **Wonderware historian
sidecar**, not in the gateway. The sidecar already owns `aahClientManaged`,
already has a `WriteAlarmEvents` IPC slot defined in `Ipc/Contracts.cs`, and
already dispatches to an `IAlarmEventWriter` interface — it's just unwired
in `Program.cs:57`. The gateway is for MxAccess (live data + Galaxy
hierarchy); the historian sidecar is for `aahClientManaged` (time-series +
alarms historian). Two different SDKs, two different concerns; keep the
split. Bonus: completing the sidecar's write path also gives it a clearer
long-term role — once the REST-API migration in `histsdk\instructions.md`
takes over reads, write-back keeps the sidecar relevant rather than
retiring it as a read-only relic. **Galaxy-native alarms bypass this
entirely** — System Platform's own `HistorizeToAveva` toggle on the
Galaxy template publishes them directly. The sidecar write path is
exclusively for non-Galaxy producers (today: scripted alarms; future: AB
CIP ALMD or any other lmxopcua-side alarm source the customer wants
unified into AVEVA Historian).
## Track A — mxaccessgw changes
All five PRs land in `c:\Users\dohertj2\Desktop\mxaccessgw\`.
### PR A.1 — proto: add alarm-transition event family + ack/query RPCs
**Files** (`src\MxGateway.Contracts\Protos\mxaccess_gateway.proto`):
1. Extend `MxEventFamily` (line 403):
```
MX_EVENT_FAMILY_ON_ALARM_TRANSITION = 5;
```
2. Extend `MxEvent.body` oneof (line 395) with:
```
OnAlarmTransitionEvent on_alarm_transition = 24;
```
3. New message `OnAlarmTransitionEvent` after the existing event-family
bodies (line 425+). Carry the full MxAccess alarm payload — alarm name,
source object reference, alarm-type-name (e.g. "AnalogLimitAlarm.HiHi"),
transition kind enum (`Raise` / `Acknowledge` / `Clear`), severity (raw
numeric — keep MxAccess scale; mapping to OPC UA 0-1000 happens
server-side in lmxopcua), `original_raise_timestamp`,
`transition_timestamp`, optional `operator_user`, optional
`operator_comment`, alarm `category` string, alarm `description`. Mirror
the field set documented in v1's `GalaxyAlarmTracker`.
4. New RPC on `MxAccessGateway` service (line 11):
```
rpc AcknowledgeAlarm(AcknowledgeAlarmRequest) returns (AcknowledgeAlarmReply);
rpc QueryActiveAlarms(QueryActiveAlarmsRequest) returns (stream ActiveAlarmSnapshot);
```
`AcknowledgeAlarmRequest` carries `session_id`, `alarm_full_reference`,
`comment`, `user_principal`. Reply carries `MxStatusProxy`.
`QueryActiveAlarmsRequest` carries `session_id`, optional
`alarm_filter_prefix` (for ConditionRefresh on a sub-tree).
`ActiveAlarmSnapshot` carries the same fields as
`OnAlarmTransitionEvent` plus `current_state` enum (`Active` /
`ActiveAcked` / `Inactive`).
**Tests** (`MxGateway.Tests` — proto/codegen sanity):
- Round-trip Serialize→Deserialize for the new messages with all-fields
populated and empty-optional-fields cases.
- `MxEvent.body` oneof selection guard — supplying multiple bodies
rejected.
**Out of scope:** worker-side wiring (PR A.2), gateway-side dispatch (PR A.3).
PR A.1 is a pure contract-surface change; nothing functional yet.
### PR A.2 — worker: subscribe to MxAccess alarm event source
**Files** (`src\MxGateway.Worker\` — net48/x86):
The MxAccess Toolkit exposes alarm subscription separately from data
subscription. Per AVEVA's MXAccess C++ Toolkit reference (canonical doc
referenced from `gateway.md`), alarm events arrive through the
`IAlarmEventSink` interface registered against the MxAccess `Alarms`
collection of an open session, OR via the MxAccess "alarm provider"
subscription pattern (depends on Toolkit version on the worker host —
verify against the version actually deployed in the worker bin during
PR A.2).
1. Worker subscribes to MxAccess alarms once per session, with a single
sink that fans out into the same bounded channel the data-change pump
uses (`MxGateway.Worker\Eventing\EventChannel.cs` or whatever the worker
currently calls its sink — verify name during the PR).
2. Sink translates each MxAccess alarm event into a `WorkerEvent` proto
(defined in `mxaccess_worker.proto`) carrying the new
`OnAlarmTransitionEvent` body. Reuses the existing `worker_sequence`
counter so ordering is preserved across families.
3. Worker honours the same backpressure rules as data-change events —
newest-dropped on full channel, single dropped-counter metric per
family.
**Tests** (`MxGateway.Worker.Tests`):
- Fake `IAlarmEventSink` source emits canned transitions; assert the
worker forwards each as the right `WorkerEvent` shape.
- Cancellation test — closing the session unsubscribes from MxAccess
alarms cleanly (no leaked sinks if the worker is recycled mid-session).
**Out of scope:** any gateway-side dispatch, any RPC handler — PR A.2
is worker-internal.
### PR A.3 — gateway: dispatch OnAlarmTransition + implement AcknowledgeAlarm
**Files** (`src\MxGateway.Server\`):
1. The session-level event multiplexer (`Sessions\SessionEventStream.cs`
or equivalent — verify name during PR) recognizes the new
`WorkerEvent` body and forwards as an `MxEvent` with family
`MX_EVENT_FAMILY_ON_ALARM_TRANSITION` to the gRPC
`StreamEvents` consumer.
2. New RPC handler `AcknowledgeAlarm` builds an MxAccess `WorkerCommand`
carrying an `AlarmAcknowledgeCommand` (new in `mxaccess_worker.proto`
under PR A.1). Forwarded to the worker; reply mapped to
`AcknowledgeAlarmReply` with the MxAccess `MxStatus` proxy populated.
3. AuthN — same API-key + scope check as existing RPCs. Add a new scope
`invoke:alarm-ack` (mirrors `invoke:write` granularity); existing keys
without it return `PERMISSION_DENIED`.
**Tests** (`MxGateway.Tests`, `MxGateway.IntegrationTests`):
- Unit: dispatch test — fake worker emits an `AlarmTransition` event;
assert the gateway forwards it on the live `StreamEvents` channel of
every subscribed session.
- Integration: end-to-end against the real worker (requires the parity
rig setup — see `docs\v2\Galaxy.ParityRig.md` in lmxopcua for the
MxAccess-installed dev box prerequisites). Trigger a real Galaxy
alarm, assert the gateway emits `OnAlarmTransition`. Acknowledge via
the new RPC, assert the alarm transitions to `ActiveAcked` and an
`Acknowledge` transition event is emitted back.
- AuthN: existing key without `invoke:alarm-ack` scope rejected.
### PR A.4 — gateway: ConditionRefresh snapshot via QueryActiveAlarms
**Files** (`src\MxGateway.Server\`, `src\MxGateway.Worker\`):
1. Worker exposes a `QueryActiveAlarmsCommand` that walks the session's
active-alarm collection and streams snapshots back through the
existing command-reply channel. The MxAccess Toolkit's
`Alarms.GetActive()` (verify exact API name during PR) is the
underlying call.
2. Gateway RPC `QueryActiveAlarms` opens a server-streaming reply,
batches snapshots through.
3. AuthN — new scope `invoke:alarm-query` (separate from ack so a
read-only client can refresh without ack rights).
**Tests:**
- Worker-test: synthetic active set of 0 / 1 / 100 alarms; assert
pagination respects worker channel capacity.
- Integration: against the parity rig, assert a ConditionRefresh after
reconnect returns every alarm currently `Active` or `ActiveAcked` in
the Galaxy.
**Sequencing within Track A:** A.1 → A.2 → A.3 → A.4. A.1 is
mechanical; A.2 + A.3 are the load-bearing changes that unlock lmxopcua
side. A.4 can ship after lmxopcua starts consuming A.3 output. The
historian-write capability moved to **Track C** below — the gateway
intentionally stays out of `aahClientManaged`.
## Track B — lmxopcua changes
All five PRs land in `c:\Users\dohertj2\Desktop\lmxopcua\`. Each B-PR
depends on a specific A-PR — see the sequencing matrix below.
### PR B.1 — EventPump: dispatch OnAlarmTransition family
**Depends on:** A.1 (proto), A.3 (gateway dispatching the new family).
**Files:**
- `src\ZB.MOM.WW.OtOpcUa.Driver.Galaxy\Runtime\EventPump.cs:160`
current `Dispatch(MxEvent ev)` returns early for any non-`OnDataChange`
family. Add a branch:
```csharp
switch (ev.Family) {
case MxEventFamily.OnDataChange: DispatchDataChange(ev); break;
case MxEventFamily.OnAlarmTransition: DispatchAlarmTransition(ev); break;
default: return;
}
```
- New `DispatchAlarmTransition` translates the proto event into an
`AlarmEventArgs` (existing type from `Core.Abstractions`) and raises an
internal event the driver subscribes to.
- New `MxAccessSeverityMapper` in `Driver.Galaxy\Runtime\` — maps the
MxAccess raw severity into the `AlarmSeverity` enum + the OPC UA
numeric severity (250 / 500 / 700 / 900 ladder per v1's
`AlarmTracking.md`).
**Tests** (`tests\ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Tests\Runtime\`):
- `EventPumpAlarmTests` — feed three synthetic MxEvents (raise / ack /
clear); assert each fires `OnAlarmEvent` on the driver with correct
payload.
- Severity-mapping table tests — every documented MxAccess severity
level → expected (`AlarmSeverity`, OPC UA numeric) tuple.
### PR B.2 — GalaxyDriver re-implements IAlarmSource
**Depends on:** A.3 (`AcknowledgeAlarm` RPC available), B.1 (event
dispatch).
**Files:**
- `src\ZB.MOM.WW.OtOpcUa.Driver.Galaxy\GalaxyDriver.cs:28` — extend the
class declaration:
```csharp
public sealed class GalaxyDriver
: IDriver, ITagDiscovery, IReadable, IWritable, ISubscribable,
IRediscoverable, IHostConnectivityProbe, IAlarmSource, IDisposable
```
- Implement the four `IAlarmSource` members:
- `SubscribeAlarmsAsync` — no-op returning a sentinel handle. The
driver is already subscribed for data; alarm events arrive on the
same event stream once the gateway emits the new family. (Same
pattern AbCip uses today — see `Driver.AbCip\AbCipDriver.cs:208`.)
- `UnsubscribeAlarmsAsync` — no-op.
- `OnAlarmEvent` — wired to the EventPump branch added in B.1.
- `AcknowledgeAsync` — calls the new gateway RPC via the
`IGalaxyAlarmAcknowledger` abstraction (new file, mirrors the
`IGalaxyDataWriter` pattern), with `GatewayGalaxyAlarmAcknowledger`
as the production implementation in `Runtime\`. Resilience wrapping
via `AlarmSurfaceInvoker` per existing pattern.
- `DriverInstanceFactory` for Galaxy registers
`IGalaxyAlarmAcknowledger` alongside the existing data writer.
**Tests:**
- Subscribe-noop returns a non-null handle; unsubscribe accepts it.
- Acknowledge — fake `IGalaxyAlarmAcknowledger` records the call; assert
the request shape and resilience-pipeline routing.
- End-to-end test in `Driver.Galaxy.Tests` — fake gateway emits a
raise-then-ack event sequence; assert the driver fires `OnAlarmEvent`
twice with matching alarm-id correlation.
### PR B.3 — DriverNodeManager: route to driver-native when present
**Depends on:** B.2.
**Files:**
- `src\ZB.MOM.WW.OtOpcUa.Server\OpcUa\DriverNodeManager.cs` — when
registering an `AlarmConditionState` for a Galaxy variable, check
whether the driver is `IAlarmSource`. If yes, prefer the
`OnAlarmEvent`-driven path; the value-driven sub-attribute path
becomes the secondary path that handles transitions the driver-native
stream missed (network blip, gateway restart, gw missing the
`$Alarm*` extension on this template).
- `Server\Alarms\AlarmConditionService` — already accepts events from
multiple sources; only addition is a `DriverEventOrigin` enum on
internal transitions so the dedup logic prefers the richer
driver-native record over a stale sub-attribute synthesis.
- `IAlarmAcknowledger` resolution in `DriverNodeManager`
prefer the driver's `IAlarmSource.AcknowledgeAsync` over
`DriverWritableAcknowledger` when both are available. Keep
`DriverWritableAcknowledger` as the fallback for templates without
`$Alarm*` extensions.
**Tests:**
- Two-source-fan-in test: same alarm condition receives both a
driver-native ack event and a sub-attribute value update for the same
transition; assert no duplicate Part 9 transition fires.
- Acknowledger routing — driver implements `IAlarmSource`
ack-via-RPC; driver implements only `IWritable` → ack-via-write
(existing path).
### PR B.4 — IAlarmHistorianWriter via the historian sidecar IPC
**Depends on:** C.2 (sidecar wires its `IAlarmEventWriter`). See Track C
for the sidecar-side work; B.4 is the lmxopcua-side consumer.
**Files:**
- New `src\ZB.MOM.WW.OtOpcUa.Driver.Historian.Wonderware.Client\SidecarAlarmHistorianWriter.cs`
implementing `IAlarmHistorianWriter`. Sends batches over the existing
named-pipe IPC using the **already-defined**
`WriteAlarmEventsRequest` / `WriteAlarmEventsReply` contracts at
`Ipc\Contracts.cs:153`. No protocol changes — the slot is wired today
on the contract side; only the production behaviour and the consumer
on this side need to land.
- `Server\Phase7\Phase7Composer.ResolveHistorianSink` — already scans
for registered `IAlarmHistorianWriter` instances. Register the new
sidecar-backed writer at server bootstrap when the historian sidecar
is enabled (`appsettings.json` `Historian:Wonderware:Enabled = true`).
`SqliteStoreAndForwardSink` then boots with a real writer attached
and the `NullAlarmHistorianSink` fallback no longer applies on
installs that have the sidecar deployed.
**Tests:**
- `SidecarAlarmHistorianWriter` against a fake `PipeServer`
single record, batch, per-row failure modes (Ack / RetryPlease /
PermanentFail) mapped from the sidecar's `PerEventOk[]` reply.
- `Phase7Composer` end-to-end — start the server with the historian
sidecar enabled; assert `ResolveHistorianSink` picks
`SqliteStoreAndForwardSink` with the new sidecar writer attached.
**Note on producer scope:** This path historizes **non-Galaxy alarms
only.** Galaxy-native alarms (with `$Alarm*` extensions) reach AVEVA
Historian directly via System Platform's `HistorizeToAveva` toggle on
the alarm primitive, with no involvement from us. Today the only live
producer feeding `SqliteStoreAndForwardSink` is
`Phase7EngineComposer.RouteToHistorianAsync` for scripted alarms; future
producers (AB CIP ALMD, FOCAS CNC alarms if a customer wants unified
storage) plug into the same path.
### PR B.5 — docs + memory housekeeping
**Depends on:** B.1 / B.2 / B.3 / B.4 all green on the parity rig + D.1
(deployment refresh) verified on the dev rig.
**Files:**
- `docs\drivers\Galaxy.md` — current text says the driver implements
five capability interfaces; update to seven (`IAlarmSource`,
`IAlarmHistorianWriter`-via-companion).
- `docs\AlarmTracking.md` — promote a fresh top-level doc that
describes the v2-final architecture (driver-native primary path +
sub-attribute fallback + scripted-alarm aggregation). Cross-link from
`docs\README.md`. The v1 archive stays as historical record.
- `docs\v1\AlarmTracking.md` — extend the existing historical banner
with "Restored to functional parity in this epic — see
`docs\AlarmTracking.md` for current state."
- Memory entries (`C:\Users\dohertj2\.claude\projects\…\memory\`):
- Update `project_galaxy_via_mxgateway.md` — add the alarm path
restoration.
- Update `project_server_history_alarm_subsystems.md` — note that
`Phase7Composer.ResolveHistorianSink` now finds a writer on
Galaxy installs.
- `docs\plans\alarms-over-gateway.md` (this file) — banner the doc
`✅ Completed YYYY-MM-DD — historical record.` matching the existing
v2-mxgw plan retirement convention.
## Track C — historian sidecar wires the dormant write path
The Wonderware historian sidecar at
`src\ZB.MOM.WW.OtOpcUa.Driver.Historian.Wonderware\` is a separately
deployable Windows service (NSSM-wrapped) that already loads
`aahClientManaged` x64 and serves a named-pipe IPC for read operations.
The `WriteAlarmEvents` IPC slot is defined but unwired (`Program.cs:57`
constructs `HistorianFrameHandler` without an `alarmWriter`). Track C
completes that slot. Two PRs in the sidecar + one consumer-side PR
(B.4) in lmxopcua finishes the path.
### PR C.1 — sidecar: AahClientManagedAlarmEventWriter
**Files** (`src\ZB.MOM.WW.OtOpcUa.Driver.Historian.Wonderware\Backend\`):
1. New `AahClientManagedAlarmEventWriter.cs` implementing the existing
`IAlarmEventWriter` interface (defined in `Ipc\HistorianFrameHandler.cs:242`).
2. Implementation calls `aahClientManaged`'s alarm-event write API —
the same path v1's `GalaxyHistorianWriter` used. Use the existing
`HistorianClusterEndpointPicker` for multi-node routing so write
failures fail over the same way reads do.
3. Batch size + retry behaviour mirrors v1's `GalaxyHistorianWriter`
per-row outcome reporting (`HistorianWriteOutcome` enum: Ack /
PermanentFail / RetryPlease). Map MxStatus codes onto outcomes.
4. Reuses `HistorianDataSource`'s existing connection-pool / health
gating — no new TCP work needed; the same session that serves
reads can issue writes too.
**Tests** (`tests\ZB.MOM.WW.OtOpcUa.Driver.Historian.Wonderware.Tests\`):
- Outcome-mapping table: every documented MxStatus on alarm-write →
expected `HistorianWriteOutcome`.
- Batching: 1 / 100 / 1000 events through a fake `aahClientManaged`
writer; assert per-row outcome list parallel to input order.
- Cluster failover: primary node returns `BadCommunicationError`;
picker rotates to secondary; assert eventual success.
### PR C.2 — sidecar: wire IAlarmEventWriter into Program.cs
**Files** (`src\ZB.MOM.WW.OtOpcUa.Driver.Historian.Wonderware\Program.cs`):
1. Build an `AahClientManagedAlarmEventWriter` next to the existing
`BuildHistorian()` call.
2. Pass it to `HistorianFrameHandler` (currently constructed at line 57
without an `alarmWriter`). The dispatcher already routes
`WriteAlarmEventsRequest` through `_alarmWriter` when non-null
(`HistorianFrameHandler.cs:158-172`); supplying it makes the slot
functional.
3. Gate behind a new env var `OTOPCUA_HISTORIAN_ALARM_WRITE_ENABLED`
(default `true` when `OTOPCUA_HISTORIAN_ENABLED=true`). Lets a
read-only deployment skip the writer registration if needed.
4. Update `Install-Services.ps1` install-time env block in
lmxopcua's `scripts\install\` to include the new toggle.
**Tests:**
- `Program.cs` unit-test seam: assert handler is constructed with
alarm writer when enabled and without when disabled.
- Live integration (parity rig): write a synthetic alarm event
through the IPC; query it back via `ReadEvents`; assert
round-trip fidelity.
### Sequencing within Track C: C.1 → C.2.
C.2's lmxopcua-side consumer is **PR B.4 in Track B**, which depends
on C.2 being deployed.
## Track D — deployment refresh
The dev box at `DESKTOP-6JL3KKO` runs three live services from
`C:\publish\` (installed in the session that produced commit
`ea04547`'s install scripts). Once Tracks A / B / C are merged, the
deployed binaries need to be refreshed so the running services pick
up the new alarm path. Track D is one PR — pure ops, no code change.
### PR D.1 — refresh C:\publish + restart services
**Depends on:** A.4 + B.4 + C.2 merged (every code-change PR landed).
**Order matters** — services must stop in reverse-dependency order
(`OtOpcUa``OtOpcUaWonderwareHistorian``MxAccessGw`) and start in
forward-dependency order (`MxAccessGw``OtOpcUaWonderwareHistorian`
`OtOpcUa`). Touching binaries while a dependent service holds them
locked produces the publish-time `MSB3027` file-lock error caught
during the original install (see commit `80104ca`).
**Steps (run as a single PowerShell session on the deploy host):**
1. **Stop in reverse order**:
```powershell
nssm stop OtOpcUa
nssm stop OtOpcUaWonderwareHistorian
nssm stop MxAccessGw
Start-Sleep -Seconds 3
Get-Process MxGateway.Server, MxGateway.Worker, OtOpcUa.Server, `
OtOpcUa.Driver.Historian.Wonderware -ErrorAction SilentlyContinue |
Stop-Process -Force
```
2. **Refresh mxaccessgw binaries** (Track A output):
```powershell
$gwSrc = "C:\Users\dohertj2\Desktop\mxaccessgw"
dotnet build "$gwSrc\src\MxGateway.Worker" -c Release
dotnet build "$gwSrc\src\MxGateway.Server" -c Release
Copy-Item -Recurse -Force `
"$gwSrc\src\MxGateway.Server\bin\Release\net10.0\*" `
"C:\publish\mxaccessgw\Server\"
Copy-Item -Recurse -Force `
"$gwSrc\src\MxGateway.Worker\bin\x86\Release\net48\*" `
"C:\publish\mxaccessgw\Worker\"
```
3. **Refresh OtOpcUa + historian sidecar binaries** (Tracks B + C
output):
```powershell
$repo = "C:\Users\dohertj2\Desktop\lmxopcua"
dotnet publish "$repo\src\ZB.MOM.WW.OtOpcUa.Server" `
-c Release -o "C:\publish\lmxopcua"
dotnet publish "$repo\src\ZB.MOM.WW.OtOpcUa.Driver.Historian.Wonderware" `
-c Release -o "C:\publish\lmxopcua\WonderwareHistorian"
```
4. **Update service env block if Track C added the new toggle**:
```powershell
# Pull existing env, append OTOPCUA_HISTORIAN_ALARM_WRITE_ENABLED=true
# (default-on per C.2 design, but explicit assignment lets us flip false
# for read-only deployments without re-installing)
nssm set OtOpcUaWonderwareHistorian AppEnvironmentExtra `
(((nssm get OtOpcUaWonderwareHistorian AppEnvironmentExtra) `
+ "`r`nOTOPCUA_HISTORIAN_ALARM_WRITE_ENABLED=true"))
```
5. **Start in forward order**:
```powershell
nssm start MxAccessGw
Start-Sleep -Seconds 4
nssm start OtOpcUaWonderwareHistorian
Start-Sleep -Seconds 4
nssm start OtOpcUa
Start-Sleep -Seconds 8
```
6. **Smoke verification:**
```powershell
foreach ($s in 'MxAccessGw','OtOpcUaWonderwareHistorian','OtOpcUa') {
(Get-Service $s).Status
}
foreach ($p in 5120, 4840, 4841) {
Get-NetTCPConnection -LocalPort $p -State Listen `
-ErrorAction SilentlyContinue
}
Get-Content "C:\publish\lmxopcua\logs\otopcua-*.log" -Tail 20
Get-Content "C:\publish\mxaccessgw\stdout.log" -Tail 20
Get-Content "C:\ProgramData\OtOpcUa\historian-wonderware-*.log" -Tail 10
```
Pass criterion: all three services `Running`; ports 5120 + 4840
listening; sidecar log shows `Wonderware historian sidecar
serving — pipe=OtOpcUaWonderwareHistorian`; OtOpcUa log shows
`OPC UA server started — endpoint=opc.tcp://0.0.0.0:4840/OtOpcUa`
and a new line `IAlarmHistorianWriter resolved: Sidecar` (added
in B.4).
7. **Functional verification — fire one alarm of each kind and assert
it propagates:**
- **Galaxy-native** — raise the `OtOpcUaParityTest_001.Counter`
`$Alarm*` extension via Galaxy's alarm-fire mechanism; assert an
OPC UA Part 9 transition reaches a connected `otopcua-cli alarms`
subscriber with rich payload (operator-comment field non-null,
original-raise-timestamp present). This validates Track A + B.1
+ B.2 + B.3.
- **Scripted** — author a one-line scripted alarm in the Admin UI
against any always-true predicate; assert the transition lands in
AVEVA Historian via `aaHistClientTrend` query (or
`Driver.Historian.Wonderware.IntegrationTests` with a query for
the alarm event). Validates Track C + B.4.
- **Sub-attribute fallback** — disable `IAlarmSource` on the
GalaxyDriver via the test seam (B.3 will introduce one); fire an
alarm; assert Part 9 transition still raised by the value-driven
path. Validates the fallback wasn't broken.
**Files:**
- `scripts\install\Refresh-Services.ps1` *(new — automates the above)*
- `docs\v2\dev-environment.md` — add the refresh script to the dev
workflow section.
**Tests:** smoke run on the dev rig (`DESKTOP-6JL3KKO`) producing
`docs\plans\artifacts\d1-rollout-YYYY-MM-DD.md` with the captured log
tails + smoke-test assertions. Captured artifact lands as part of the
PR.
**Rollback:** the refresh script keeps a timestamped backup of the
existing `C:\publish\mxaccessgw\` and `C:\publish\lmxopcua\` trees
before overwriting (mirrored to `C:\publish\.backup-YYYY-MM-DD\`).
Rollback is a stop / restore-from-backup / start sequence; no service
re-install needed since the NSSM service definitions don't change.
**Production deploy:** out of scope for D.1 — the dev rig is the only
deployment in scope at this point. A separate PR-or-runbook lands the
production refresh once the dev rig has soaked for the documented
duration (parity-rig validation gate; see "Test gates" above).
## Sequencing matrix
```
Track A (mxaccessgw) Track B (lmxopcua) Track C (sidecar)
───────────────────────── ───────────────────────── ─────────────────────────
A.1 proto (waits) C.1 AahClientManagedAlarmEventWriter
│ │ no cross-repo dep
├──────────────────────────► B.1 EventPump branch │
A.2 worker subscription │ uses proto types only │
│ │ unit-testable │
│ C.2 Program.cs wires writer
A.3 gateway dispatch + ack RPC ──►B.2 GalaxyDriver : IAlarmSource │
│ │ │
│ ──►B.3 DriverNodeManager routing │
│ │
A.4 ConditionRefresh │ │
│ │
B.4 SidecarAlarmHistorianWriter
(depends on C.2 deployed)
Track D (deployment)
─────────────────────────
D.1 Refresh C:\publish + restart services
(depends on A.4 + B.4 + C.2 merged)
──►B.5 docs + memory + completion banner
```
A.1 + B.1 + C.1 can all land in parallel — none have cross-repo runtime
dependencies. B.1's tests use proto types without needing a running
gateway. C.1 is purely sidecar-internal. The gateway-side dispatch (A.3)
gates B.2; the sidecar-side wiring (C.2) gates B.4. D.1 (deployment
refresh) gates B.5 (docs) — the docs sweep records the as-deployed
state, so the deploy must be live first.
## Test gates
Per PR: unit tests pass + build green + analyzer clean (Roslyn
OTOPCUA0001 still wraps every alarm-capability call through
`AlarmSurfaceInvoker`).
End-of-epic gate: re-run the parity rig (`docs\v2\Galaxy.ParityRig.md`)
with these scenarios added:
1. **Native alarm raise** — Galaxy `$Alarm*` raise with operator-time
metadata appears as an OPC UA Part 9 transition with full payload
(no longer reconstructed from sub-attribute writes).
2. **Native ack** — OPC UA client acks; assert the gateway records the
ack against MxAccess directly (not via sub-attribute write); operator
comment present in the resulting `Acknowledged` transition.
3. **ConditionRefresh after reconnect** — disconnect the GalaxyDriver,
raise three alarms in Galaxy, reconnect; assert all three appear in
the next ConditionRefresh.
4. **Historian write-back** — fire a scripted alarm; assert it arrives in
AVEVA Historian via the gateway path (use the existing Historian
sidecar's read API to query it back).
5. **Sub-attribute fallback still works** — disable `IAlarmSource` on
the GalaxyDriver via test seam, fire a sub-attribute value change;
assert Part 9 transition still raised.
Soak target: 24h × 1k tags (light) — same parity-rig harness but
extended to also subscribe to alarms. Pass criterion: zero dropped
alarm transitions, zero state-machine inversions, zero unhandled
exceptions in the AlarmSurfaceInvoker pipeline.
## Risks and mitigations
| Risk | Mitigation |
|---|---|
| MxAccess Toolkit alarm subscription API differs across installed AVEVA versions | PR A.2 verifies against the worker-host's installed Toolkit version; documents the exact API used. Pin the worker DLL set per major MxAccess version if needed. |
| Worker-side alarm subscription leaks between sessions if cleanup is wrong | PR A.2 includes a session-recycle test that asserts no `IAlarmEventSink` instances remain registered after Close. |
| Gateway adds a new auth scope (`invoke:alarm-ack`); existing keys lack it | PR A.3 + A.5 ship with a one-time bootstrap migration: keys with `invoke:write` get the new scope auto-granted on the dev rig and parity rig. Production keys are reissued via `apikey rotate-key` (existing CLI). |
| Two simultaneous alarm sources (driver-native + sub-attribute) double-fire transitions | PR B.3 dedup is the load-bearing design. End-to-end test #1 covers it explicitly. |
| Historian write-back batch fails mid-batch — partial success | The existing `SqliteStoreAndForwardSink.HistorianWriteOutcome` per-row enum + dead-letter retention already handles this; PR A.5 just exposes the same outcome shape over gRPC. |
| Sidecar starts honouring the `WriteAlarmEvents` slot — old lmxopcua-side consumers can now reach a previously inert path | The slot returns `Success=false, Error="not configured"` today; flipping to live writes means a build that *speculatively* sent the frame would suddenly start producing real historian rows. Inventory of any such caller is empty — `WriteAlarmEvents` was never invoked from the lmxopcua side; `Phase7EngineComposer.RouteToHistorianAsync` queues into `SqliteStoreAndForwardSink` and the drain worker is gated on `IAlarmHistorianWriter` registration which only the new B.4 path provides. So enabling C.2 without B.4 is safe. |
## Roll-out
Track A lands first onto `mxaccessgw/main`, deployed to the parity rig.
Track B lands onto `lmxopcua/master` once A.3 is live on the rig — earlier
Track B PRs can target a feature branch (`feat/alarms-over-gateway`) and
merge to master after the rig is fully green.
## Back-out
Each PR is individually revertable. The cleanest back-out point is at
the gateway-side enum extension: removing `MX_EVENT_FAMILY_ON_ALARM_TRANSITION`
from the proto means EventPump silently drops alarm events again and
GalaxyDriver's `OnAlarmEvent` never fires — but the sub-attribute fallback
path still produces functional alarms, so the OPC UA surface degrades to
v2-current behaviour without breaking. PR B.4 is the only one with a
non-trivial back-out (re-add the deleted sidecar IPC slot if revert
needed); land B.4 last and only after end-of-epic gate is green.
## Out of scope (explicit)
- **Other alarm sources beyond Galaxy.** AbCip / FOCAS / OpcUaClient
drivers already implement `IAlarmSource`; they're untouched.
- **Modbus / S7 / AbLegacy / TwinCAT alarms.** None of those protocols
has a native alarm bus. Alarms on those drivers, if needed, ship via
the scripted-alarm path.
- **Multi-Galaxy ack routing.** Today's gateway model is one Galaxy per
session; if a deployment splits across galaxies, each gets its own
GalaxyDriver and they don't cross-talk. No change.
- **OPC UA Part 9 advanced features** beyond the current scope —
shelving, subscribed-to-events-only, branch-state for re-trigger
semantics. Future epic if a customer asks.
- **Insight / cloud Historian write-back path.** Track A.5 targets the
on-prem AVEVA Historian via aahClientManaged. The cloud variant
would mirror the same gateway RPC over the REST API discussed in
`docs/histsdk` — separate epic.
## File inventory (touched)
**mxaccessgw (Track A):**
- `src\MxGateway.Contracts\Protos\mxaccess_gateway.proto` (A.1)
- `src\MxGateway.Contracts\Protos\mxaccess_worker.proto` (A.2, A.4)
- `src\MxGateway.Worker\…\Eventing\` (A.2, A.3, A.4)
- `src\MxGateway.Worker\…\Commands\` (A.3, A.4)
- `src\MxGateway.Server\Sessions\SessionEventStream.cs` (A.3)
- `src\MxGateway.Server\Rpc\` (A.3, A.4)
- `src\MxGateway.Server\Auth\Scopes.cs` (A.3, A.4)
- `MxGateway.Tests`, `MxGateway.Worker.Tests`, `MxGateway.IntegrationTests`
**lmxopcua — Galaxy driver + server (Track B):**
- `src\ZB.MOM.WW.OtOpcUa.Driver.Galaxy\Runtime\EventPump.cs` (B.1)
- `src\ZB.MOM.WW.OtOpcUa.Driver.Galaxy\Runtime\MxAccessSeverityMapper.cs` *(new — B.1)*
- `src\ZB.MOM.WW.OtOpcUa.Driver.Galaxy\Runtime\IGalaxyAlarmAcknowledger.cs` *(new — B.2)*
- `src\ZB.MOM.WW.OtOpcUa.Driver.Galaxy\Runtime\GatewayGalaxyAlarmAcknowledger.cs` *(new — B.2)*
- `src\ZB.MOM.WW.OtOpcUa.Driver.Galaxy\GalaxyDriver.cs` (B.2)
- `src\ZB.MOM.WW.OtOpcUa.Driver.Galaxy\GalaxyDriverFactory.cs` (B.2)
- `src\ZB.MOM.WW.OtOpcUa.Server\OpcUa\DriverNodeManager.cs` (B.3)
- `src\ZB.MOM.WW.OtOpcUa.Server\Alarms\AlarmConditionService.cs` (B.3)
- `src\ZB.MOM.WW.OtOpcUa.Server\Phase7\Phase7Composer.cs` (B.4)
- `src\ZB.MOM.WW.OtOpcUa.Driver.Historian.Wonderware.Client\SidecarAlarmHistorianWriter.cs` *(new — B.4)*
- `tests\ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Tests\Runtime\` (B.1, B.2)
- `tests\ZB.MOM.WW.OtOpcUa.Server.Tests\Alarms\` (B.3)
- `tests\ZB.MOM.WW.OtOpcUa.Driver.Historian.Wonderware.Client.Tests\` (B.4 — new tests)
- `docs\drivers\Galaxy.md` (B.5)
- `docs\AlarmTracking.md` *(new — B.5)*
- `docs\v1\AlarmTracking.md` (B.5 — banner update)
- `docs\plans\alarms-over-gateway.md` (B.5 — completion banner)
**lmxopcua — Wonderware historian sidecar (Track C):**
- `src\ZB.MOM.WW.OtOpcUa.Driver.Historian.Wonderware\Backend\AahClientManagedAlarmEventWriter.cs` *(new — C.1)*
- `src\ZB.MOM.WW.OtOpcUa.Driver.Historian.Wonderware\Program.cs` (C.2 — wire writer)
- `scripts\install\Install-Services.ps1` (C.2 — env-var toggle for write-enable)
- `tests\ZB.MOM.WW.OtOpcUa.Driver.Historian.Wonderware.Tests\` (C.1 — outcome mapping + batch + cluster failover)
**lmxopcua — deployment refresh (Track D):**
- `scripts\install\Refresh-Services.ps1` *(new — D.1)*
- `docs\v2\dev-environment.md` (D.1 — document the refresh workflow)
- `docs\plans\artifacts\d1-rollout-YYYY-MM-DD.md` *(new — D.1 captured smoke run)*
Total: ~10 source files added/modified in mxaccessgw; ~14 in lmxopcua
proper; ~3 in the historian sidecar; ~2 deployment scripts; ~12 test
files across all repos. Should land in 4-6 weeks of focused work given
the parity-rig dependency for end-to-end validation, plus a short
final-week ops slot for D.1.