lmxopcua/docs/plans/alarms-over-gateway.md

Plan — alarms over the mxaccessgw gateway

Coordinated epic across two repos:

• lmxopcua (this repo) — c:\Users\dohertj2\Desktop\lmxopcua\
• mxaccessgw — c:\Users\dohertj2\Desktop\mxaccessgw\

Why

PR 7.2 (2026-04-30, commit ae7106d) retired the in-process v1 Galaxy stack (Driver.Galaxy.Host / .Proxy / .Shared + OtOpcUaGalaxyHost Windows service) and migrated Galaxy access to the in-process GalaxyDriver over mxaccessgw's gRPC. In doing so, three v1 capabilities regressed:

1. Native MxAccess alarm-event metadata — v1's GalaxyAlarmTracker surfaced rich alarm transitions (operator comment, original raise time, ack time, alarm category, native severity). The current architecture reconstructs Part 9 transitions by subscribing to four sub-attribute value updates (InAlarm, Acked, Priority, Description) — fine for raise/clear but loses everything else.
2. Native MxAccess Acknowledge semantics — v1 called the MxAccess ack API directly from GalaxyAlarmTracker. Today, OPC UA acks are written into the AckMsgWriteRef sub-attribute — semantically valid but a round-trip through the value path that loses operator-comment fidelity.
3. Alarm-historian write-back path for non-Galaxy alarm sources. v1's GalaxyHistorianWriter implemented IAlarmHistorianWriter and forwarded scripted-alarm transitions (and any future non-Galaxy alarm source — AB CIP ALMD, OpcUaClient A&E, etc.) back to AVEVA Historian via aahClientManaged. PR 7.2 deleted it. Phase7Composer.ResolveHistorianSink now finds no writer and falls back to NullAlarmHistorianSink, so scripted-alarm transitions queue locally and silently discard. Galaxy-native alarms (with $Alarm* extensions) reach AVEVA Historian via System Platform's own HistorizeToAveva toggle on the Galaxy template — that path was never broken and is not in scope for this epic.

gateway.md (mxaccessgw, line 8) explicitly commits the gateway to "full MXAccess parity… preserve MXAccess behavior first… native MXAccess event families." Today's gateway proto exposes only data-change families. Closing the alarm regression and fulfilling that parity statement are the same task.

Goals

• Restore all three regressed capabilities to feature parity with v1.
• Keep the v2 architectural split — gateway owns MxAccess transport; lmxopcua owns OPC UA Part 9 semantics, ACL/role enforcement, and multi-source aggregation (driver-native + scripted + sub-attribute).
• Preserve the value-driven sub-attribute path as a fallback for Galaxy templates that don't carry $Alarm* extensions.
• Land the work as a sequence of small, independently-reviewable PRs that alternate between repos in dependency order.

Non-goals

• Reimplementing the Part 9 state machine inside mxaccessgw. The gateway stays UA-agnostic.
• Reworking the LDAP role-grant or OPC UA AlarmAck ACL surface — those already exist and route through Server/Alarms/IAlarmAcknowledger.
• Adding alarm support to non-Galaxy drivers (AbCip / FOCAS / OpcUaClient already have their own IAlarmSource implementations; Modbus / S7 / AbLegacy / TwinCAT don't have a native alarm bus and are out of scope).
• Altering Galaxy template conventions or $Alarm* extensions in the customer's Galaxy.

Before → after

Today (post-PR 7.2):

MxAccess COM (gateway worker)
   │ data-change events only on the MxEvent stream
   ▼
GalaxyDriver (no IAlarmSource)
   │ IWritable / ISubscribable / ITagDiscovery only
   ▼
DriverNodeManager
   ├─ subscribes to four $Alarm* sub-attributes per condition
   ├─ AlarmConditionService rebuilds Part 9 transitions from value updates
   └─ DriverWritableAcknowledger writes AckMsgWriteRef on ack

Phase7Composer.ResolveHistorianSink → NullAlarmHistorianSink
   (scripted-alarm transitions queue → silently discarded)

After this epic:

MxAccess COM (gateway worker)
   │ data-change   ──┐
   │ alarm-transition │
   │ write-complete   ├─► single MxEvent stream (new family added)
   ▼                  ▼
GalaxyDriver : ITagDiscovery, IReadable, IWritable, ISubscribable, IRediscoverable,
               IHostConnectivityProbe, IAlarmSource             ← restored
   ├─ EventPump dispatches OnAlarmTransition family → IAlarmSource.OnAlarmEvent
   ├─ AcknowledgeAsync → gateway RPC AcknowledgeAlarm
   └─ QueryActiveAlarmsAsync → gateway RPC QueryActiveAlarms (ConditionRefresh)

DriverNodeManager
   ├─ rich alarm events from IAlarmSource.OnAlarmEvent → AlarmConditionService
   ├─ value-driven sub-attribute path STILL WORKS for templates without $Alarm
   ├─ DriverWritableAcknowledger preserved as fallback for the value path
   └─ ScriptedAlarmEngine output continues to feed AlarmConditionService

Phase7Composer.ResolveHistorianSink → GatewayAlarmHistorianWriter
   ├─ scripted-alarm transitions → SqliteStoreAndForwardSink
   └─ drain worker → gateway RPC WriteHistorianEvent → AVEVA Historian

Architecture decisions

D1 — Where the Part 9 state machine runs. Stays in lmxopcua's AlarmConditionService. Gateway is UA-agnostic. ScriptedAlarmEngine produces Part 9 transitions with no MxAccess origin; the aggregator must live where all sources converge.

D2 — Where authz on Acknowledge runs. Stays in lmxopcua. The OPC UA AlarmConditionState.OnAcknowledge delegate already checks the session's roles for AlarmAck against the LDAP/role-grant ACL. The gateway should never be reachable in a way that bypasses that check.

D3 — How rich alarm events reach OPC UA clients. New MxEventFamily on the existing StreamEvents RPC (no second stream). Adds latency parity with data-change events, reuses the bounded-channel + worker-side delivery semantics already documented in gateway.md.

D4 — Sub-attribute fallback path stays. Some Galaxy templates won't have $Alarm* extensions yet; the existing value-driven path remains the only way to surface alarms for those templates. Both paths feed AlarmConditionService. Driver-native events take precedence when both are present (more authoritative, lower latency).

D5 — Where the historian writer lives. In the Wonderware historian sidecar, not in the gateway. The sidecar already owns aahClientManaged, already has a WriteAlarmEvents IPC slot defined in Ipc/Contracts.cs, and already dispatches to an IAlarmEventWriter interface — it's just unwired in Program.cs:57. The gateway is for MxAccess (live data + Galaxy hierarchy); the historian sidecar is for aahClientManaged (time-series + alarms historian). Two different SDKs, two different concerns; keep the split. Bonus: completing the sidecar's write path also gives it a clearer long-term role — once the REST-API migration in histsdk\instructions.md takes over reads, write-back keeps the sidecar relevant rather than retiring it as a read-only relic. Galaxy-native alarms bypass this entirely — System Platform's own HistorizeToAveva toggle on the Galaxy template publishes them directly. The sidecar write path is exclusively for non-Galaxy producers (today: scripted alarms; future: AB CIP ALMD or any other lmxopcua-side alarm source the customer wants unified into AVEVA Historian).

Track A — mxaccessgw changes

All five PRs land in c:\Users\dohertj2\Desktop\mxaccessgw\.

PR A.1 — proto: add alarm-transition event family + ack/query RPCs

Files (src\MxGateway.Contracts\Protos\mxaccess_gateway.proto):

1. Extend MxEventFamily (line 403):

   MX_EVENT_FAMILY_ON_ALARM_TRANSITION = 5;
   

2. Extend MxEvent.body oneof (line 395) with:

   OnAlarmTransitionEvent on_alarm_transition = 24;
   

3. New message OnAlarmTransitionEvent after the existing event-family bodies (line 425+). Carry the full MxAccess alarm payload — alarm name, source object reference, alarm-type-name (e.g. "AnalogLimitAlarm.HiHi"), transition kind enum (Raise / Acknowledge / Clear), severity (raw numeric — keep MxAccess scale; mapping to OPC UA 0-1000 happens server-side in lmxopcua), original_raise_timestamp, transition_timestamp, optional operator_user, optional operator_comment, alarm category string, alarm description. Mirror the field set documented in v1's GalaxyAlarmTracker.

4. New RPC on MxAccessGateway service (line 11):

   rpc AcknowledgeAlarm(AcknowledgeAlarmRequest) returns (AcknowledgeAlarmReply);
   rpc QueryActiveAlarms(QueryActiveAlarmsRequest) returns (stream ActiveAlarmSnapshot);
   

   AcknowledgeAlarmRequest carries session_id, alarm_full_reference, comment, user_principal. Reply carries MxStatusProxy.

   QueryActiveAlarmsRequest carries session_id, optional alarm_filter_prefix (for ConditionRefresh on a sub-tree). ActiveAlarmSnapshot carries the same fields as OnAlarmTransitionEvent plus current_state enum (Active / ActiveAcked / Inactive).

Tests (MxGateway.Tests — proto/codegen sanity):

• Round-trip Serialize→Deserialize for the new messages with all-fields populated and empty-optional-fields cases.
• MxEvent.body oneof selection guard — supplying multiple bodies rejected.

Out of scope: worker-side wiring (PR A.2), gateway-side dispatch (PR A.3). PR A.1 is a pure contract-surface change; nothing functional yet.

PR A.2 — worker: subscribe to MxAccess alarm event source

Files (src\MxGateway.Worker\ — net48/x86):

The MxAccess Toolkit exposes alarm subscription separately from data subscription. Per AVEVA's MXAccess C++ Toolkit reference (canonical doc referenced from gateway.md), alarm events arrive through the IAlarmEventSink interface registered against the MxAccess Alarms collection of an open session, OR via the MxAccess "alarm provider" subscription pattern (depends on Toolkit version on the worker host — verify against the version actually deployed in the worker bin during PR A.2).

1. Worker subscribes to MxAccess alarms once per session, with a single sink that fans out into the same bounded channel the data-change pump uses (MxGateway.Worker\Eventing\EventChannel.cs or whatever the worker currently calls its sink — verify name during the PR).
2. Sink translates each MxAccess alarm event into a WorkerEvent proto (defined in mxaccess_worker.proto) carrying the new OnAlarmTransitionEvent body. Reuses the existing worker_sequence counter so ordering is preserved across families.
3. Worker honours the same backpressure rules as data-change events — newest-dropped on full channel, single dropped-counter metric per family.

Tests (MxGateway.Worker.Tests):

• Fake IAlarmEventSink source emits canned transitions; assert the worker forwards each as the right WorkerEvent shape.
• Cancellation test — closing the session unsubscribes from MxAccess alarms cleanly (no leaked sinks if the worker is recycled mid-session).

Out of scope: any gateway-side dispatch, any RPC handler — PR A.2 is worker-internal.

PR A.3 — gateway: dispatch OnAlarmTransition + implement AcknowledgeAlarm

Files (src\MxGateway.Server\):

1. The session-level event multiplexer (Sessions\SessionEventStream.cs or equivalent — verify name during PR) recognizes the new WorkerEvent body and forwards as an MxEvent with family MX_EVENT_FAMILY_ON_ALARM_TRANSITION to the gRPC StreamEvents consumer.
2. New RPC handler AcknowledgeAlarm builds an MxAccess WorkerCommand carrying an AlarmAcknowledgeCommand (new in mxaccess_worker.proto under PR A.1). Forwarded to the worker; reply mapped to AcknowledgeAlarmReply with the MxAccess MxStatus proxy populated.
3. AuthN — same API-key + scope check as existing RPCs. Add a new scope invoke:alarm-ack (mirrors invoke:write granularity); existing keys without it return PERMISSION_DENIED.

Tests (MxGateway.Tests, MxGateway.IntegrationTests):

• Unit: dispatch test — fake worker emits an AlarmTransition event; assert the gateway forwards it on the live StreamEvents channel of every subscribed session.
• Integration: end-to-end against the real worker (requires the parity rig setup — see docs\v2\Galaxy.ParityRig.md in lmxopcua for the MxAccess-installed dev box prerequisites). Trigger a real Galaxy alarm, assert the gateway emits OnAlarmTransition. Acknowledge via the new RPC, assert the alarm transitions to ActiveAcked and an Acknowledge transition event is emitted back.
• AuthN: existing key without invoke:alarm-ack scope rejected.

PR A.4 — gateway: ConditionRefresh snapshot via QueryActiveAlarms

Files (src\MxGateway.Server\, src\MxGateway.Worker\):

1. Worker exposes a QueryActiveAlarmsCommand that walks the session's active-alarm collection and streams snapshots back through the existing command-reply channel. The MxAccess Toolkit's Alarms.GetActive() (verify exact API name during PR) is the underlying call.
2. Gateway RPC QueryActiveAlarms opens a server-streaming reply, batches snapshots through.
3. AuthN — new scope invoke:alarm-query (separate from ack so a read-only client can refresh without ack rights).

Tests:

• Worker-test: synthetic active set of 0 / 1 / 100 alarms; assert pagination respects worker channel capacity.
• Integration: against the parity rig, assert a ConditionRefresh after reconnect returns every alarm currently Active or ActiveAcked in the Galaxy.

Sequencing within Track A: A.1 → A.2 → A.3 → A.4. A.1 is mechanical; A.2 + A.3 are the load-bearing changes that unlock lmxopcua side. A.4 can ship after lmxopcua starts consuming A.3 output. The historian-write capability moved to Track C below — the gateway intentionally stays out of aahClientManaged.

Track B — lmxopcua changes

All five PRs land in c:\Users\dohertj2\Desktop\lmxopcua\. Each B-PR depends on a specific A-PR — see the sequencing matrix below.

PR B.1 — EventPump: dispatch OnAlarmTransition family

Depends on: A.1 (proto), A.3 (gateway dispatching the new family).

Files:

• src\ZB.MOM.WW.OtOpcUa.Driver.Galaxy\Runtime\EventPump.cs:160 — current Dispatch(MxEvent ev) returns early for any non-OnDataChange family. Add a branch:

  switch (ev.Family) {
    case MxEventFamily.OnDataChange: DispatchDataChange(ev); break;
    case MxEventFamily.OnAlarmTransition: DispatchAlarmTransition(ev); break;
    default: return;
  }
  

• New DispatchAlarmTransition translates the proto event into an AlarmEventArgs (existing type from Core.Abstractions) and raises an internal event the driver subscribes to.
• New MxAccessSeverityMapper in Driver.Galaxy\Runtime\ — maps the MxAccess raw severity into the AlarmSeverity enum + the OPC UA numeric severity (250 / 500 / 700 / 900 ladder per v1's AlarmTracking.md).

Tests (tests\ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Tests\Runtime\):

• EventPumpAlarmTests — feed three synthetic MxEvents (raise / ack / clear); assert each fires OnAlarmEvent on the driver with correct payload.
• Severity-mapping table tests — every documented MxAccess severity level → expected (AlarmSeverity, OPC UA numeric) tuple.

PR B.2 — GalaxyDriver re-implements IAlarmSource

Depends on: A.3 (AcknowledgeAlarm RPC available), B.1 (event dispatch).

Files:

• src\ZB.MOM.WW.OtOpcUa.Driver.Galaxy\GalaxyDriver.cs:28 — extend the class declaration:

  public sealed class GalaxyDriver
      : IDriver, ITagDiscovery, IReadable, IWritable, ISubscribable,
        IRediscoverable, IHostConnectivityProbe, IAlarmSource, IDisposable
  

• Implement the four IAlarmSource members:
  • SubscribeAlarmsAsync — no-op returning a sentinel handle. The driver is already subscribed for data; alarm events arrive on the same event stream once the gateway emits the new family. (Same pattern AbCip uses today — see Driver.AbCip\AbCipDriver.cs:208.)
  • UnsubscribeAlarmsAsync — no-op.
  • OnAlarmEvent — wired to the EventPump branch added in B.1.
  • AcknowledgeAsync — calls the new gateway RPC via the IGalaxyAlarmAcknowledger abstraction (new file, mirrors the IGalaxyDataWriter pattern), with GatewayGalaxyAlarmAcknowledger as the production implementation in Runtime\. Resilience wrapping via AlarmSurfaceInvoker per existing pattern.
• DriverInstanceFactory for Galaxy registers IGalaxyAlarmAcknowledger alongside the existing data writer.

Tests:

• Subscribe-noop returns a non-null handle; unsubscribe accepts it.
• Acknowledge — fake IGalaxyAlarmAcknowledger records the call; assert the request shape and resilience-pipeline routing.
• End-to-end test in Driver.Galaxy.Tests — fake gateway emits a raise-then-ack event sequence; assert the driver fires OnAlarmEvent twice with matching alarm-id correlation.

PR B.3 — DriverNodeManager: route to driver-native when present

Depends on: B.2.

Files:

• src\ZB.MOM.WW.OtOpcUa.Server\OpcUa\DriverNodeManager.cs — when registering an AlarmConditionState for a Galaxy variable, check whether the driver is IAlarmSource. If yes, prefer the OnAlarmEvent-driven path; the value-driven sub-attribute path becomes the secondary path that handles transitions the driver-native stream missed (network blip, gateway restart, gw missing the $Alarm* extension on this template).
• Server\Alarms\AlarmConditionService — already accepts events from multiple sources; only addition is a DriverEventOrigin enum on internal transitions so the dedup logic prefers the richer driver-native record over a stale sub-attribute synthesis.
• IAlarmAcknowledger resolution in DriverNodeManager — prefer the driver's IAlarmSource.AcknowledgeAsync over DriverWritableAcknowledger when both are available. Keep DriverWritableAcknowledger as the fallback for templates without $Alarm* extensions.

Tests:

• Two-source-fan-in test: same alarm condition receives both a driver-native ack event and a sub-attribute value update for the same transition; assert no duplicate Part 9 transition fires.
• Acknowledger routing — driver implements IAlarmSource → ack-via-RPC; driver implements only IWritable → ack-via-write (existing path).

PR B.4 — IAlarmHistorianWriter via the historian sidecar IPC

Depends on: C.2 (sidecar wires its IAlarmEventWriter). See Track C for the sidecar-side work; B.4 is the lmxopcua-side consumer.

Files:

• New src\ZB.MOM.WW.OtOpcUa.Driver.Historian.Wonderware.Client\SidecarAlarmHistorianWriter.cs implementing IAlarmHistorianWriter. Sends batches over the existing named-pipe IPC using the already-defined WriteAlarmEventsRequest / WriteAlarmEventsReply contracts at Ipc\Contracts.cs:153. No protocol changes — the slot is wired today on the contract side; only the production behaviour and the consumer on this side need to land.
• Server\Phase7\Phase7Composer.ResolveHistorianSink — already scans for registered IAlarmHistorianWriter instances. Register the new sidecar-backed writer at server bootstrap when the historian sidecar is enabled (appsettings.json Historian:Wonderware:Enabled = true). SqliteStoreAndForwardSink then boots with a real writer attached and the NullAlarmHistorianSink fallback no longer applies on installs that have the sidecar deployed.

Tests:

• SidecarAlarmHistorianWriter against a fake PipeServer — single record, batch, per-row failure modes (Ack / RetryPlease / PermanentFail) mapped from the sidecar's PerEventOk[] reply.
• Phase7Composer end-to-end — start the server with the historian sidecar enabled; assert ResolveHistorianSink picks SqliteStoreAndForwardSink with the new sidecar writer attached.

Note on producer scope: This path historizes non-Galaxy alarms only. Galaxy-native alarms (with $Alarm* extensions) reach AVEVA Historian directly via System Platform's HistorizeToAveva toggle on the alarm primitive, with no involvement from us. Today the only live producer feeding SqliteStoreAndForwardSink is Phase7EngineComposer.RouteToHistorianAsync for scripted alarms; future producers (AB CIP ALMD, FOCAS CNC alarms if a customer wants unified storage) plug into the same path.

PR B.5 — docs + memory housekeeping

Depends on: B.1 / B.2 / B.3 / B.4 all green on the parity rig + D.1 (deployment refresh) verified on the dev rig.

Files:

• docs\drivers\Galaxy.md — current text says the driver implements five capability interfaces; update to seven (IAlarmSource, IAlarmHistorianWriter-via-companion).
• docs\AlarmTracking.md — promote a fresh top-level doc that describes the v2-final architecture (driver-native primary path + sub-attribute fallback + scripted-alarm aggregation). Cross-link from docs\README.md. The v1 archive stays as historical record.
• docs\v1\AlarmTracking.md — extend the existing historical banner with "Restored to functional parity in this epic — see docs\AlarmTracking.md for current state."
• Memory entries (C:\Users\dohertj2\.claude\projects\…\memory\):
  • Update project_galaxy_via_mxgateway.md — add the alarm path restoration.
  • Update project_server_history_alarm_subsystems.md — note that Phase7Composer.ResolveHistorianSink now finds a writer on Galaxy installs.
• docs\plans\alarms-over-gateway.md (this file) — banner the doc ✅ Completed YYYY-MM-DD — historical record. matching the existing v2-mxgw plan retirement convention.

Track C — historian sidecar wires the dormant write path

The Wonderware historian sidecar at src\ZB.MOM.WW.OtOpcUa.Driver.Historian.Wonderware\ is a separately deployable Windows service (NSSM-wrapped) that already loads aahClientManaged x64 and serves a named-pipe IPC for read operations. The WriteAlarmEvents IPC slot is defined but unwired (Program.cs:57 constructs HistorianFrameHandler without an alarmWriter). Track C completes that slot. Two PRs in the sidecar + one consumer-side PR (B.4) in lmxopcua finishes the path.

PR C.1 — sidecar: AahClientManagedAlarmEventWriter

Files (src\ZB.MOM.WW.OtOpcUa.Driver.Historian.Wonderware\Backend\):

1. New AahClientManagedAlarmEventWriter.cs implementing the existing IAlarmEventWriter interface (defined in Ipc\HistorianFrameHandler.cs:242).
2. Implementation calls aahClientManaged's alarm-event write API — the same path v1's GalaxyHistorianWriter used. Use the existing HistorianClusterEndpointPicker for multi-node routing so write failures fail over the same way reads do.
3. Batch size + retry behaviour mirrors v1's GalaxyHistorianWriter per-row outcome reporting (HistorianWriteOutcome enum: Ack / PermanentFail / RetryPlease). Map MxStatus codes onto outcomes.
4. Reuses HistorianDataSource's existing connection-pool / health gating — no new TCP work needed; the same session that serves reads can issue writes too.

Tests (tests\ZB.MOM.WW.OtOpcUa.Driver.Historian.Wonderware.Tests\):

• Outcome-mapping table: every documented MxStatus on alarm-write → expected HistorianWriteOutcome.
• Batching: 1 / 100 / 1000 events through a fake aahClientManaged writer; assert per-row outcome list parallel to input order.
• Cluster failover: primary node returns BadCommunicationError; picker rotates to secondary; assert eventual success.

PR C.2 — sidecar: wire IAlarmEventWriter into Program.cs

Files (src\ZB.MOM.WW.OtOpcUa.Driver.Historian.Wonderware\Program.cs):

1. Build an AahClientManagedAlarmEventWriter next to the existing BuildHistorian() call.
2. Pass it to HistorianFrameHandler (currently constructed at line 57 without an alarmWriter). The dispatcher already routes WriteAlarmEventsRequest through _alarmWriter when non-null (HistorianFrameHandler.cs:158-172); supplying it makes the slot functional.
3. Gate behind a new env var OTOPCUA_HISTORIAN_ALARM_WRITE_ENABLED (default true when OTOPCUA_HISTORIAN_ENABLED=true). Lets a read-only deployment skip the writer registration if needed.
4. Update Install-Services.ps1 install-time env block in lmxopcua's scripts\install\ to include the new toggle.

Tests:

• Program.cs unit-test seam: assert handler is constructed with alarm writer when enabled and without when disabled.
• Live integration (parity rig): write a synthetic alarm event through the IPC; query it back via ReadEvents; assert round-trip fidelity.

Sequencing within Track C: C.1 → C.2.

C.2's lmxopcua-side consumer is PR B.4 in Track B, which depends on C.2 being deployed.

Track D — deployment refresh

The dev box at DESKTOP-6JL3KKO runs three live services from C:\publish\ (installed in the session that produced commit ea04547's install scripts). Once Tracks A / B / C are merged, the deployed binaries need to be refreshed so the running services pick up the new alarm path. Track D is one PR — pure ops, no code change.

PR D.1 — refresh C:\publish + restart services

Depends on: A.4 + B.4 + C.2 merged (every code-change PR landed).

Order matters — services must stop in reverse-dependency order (OtOpcUa → OtOpcUaWonderwareHistorian → MxAccessGw) and start in forward-dependency order (MxAccessGw → OtOpcUaWonderwareHistorian → OtOpcUa). Touching binaries while a dependent service holds them locked produces the publish-time MSB3027 file-lock error caught during the original install (see commit 80104ca).

Steps (run as a single PowerShell session on the deploy host):

1. Stop in reverse order:

   nssm stop OtOpcUa
   nssm stop OtOpcUaWonderwareHistorian
   nssm stop MxAccessGw
   Start-Sleep -Seconds 3
   Get-Process MxGateway.Server, MxGateway.Worker, OtOpcUa.Server, `
               OtOpcUa.Driver.Historian.Wonderware -ErrorAction SilentlyContinue |
       Stop-Process -Force
   

2. Refresh mxaccessgw binaries (Track A output):

   $gwSrc = "C:\Users\dohertj2\Desktop\mxaccessgw"
   dotnet build "$gwSrc\src\MxGateway.Worker" -c Release
   dotnet build "$gwSrc\src\MxGateway.Server" -c Release
   
   Copy-Item -Recurse -Force `
       "$gwSrc\src\MxGateway.Server\bin\Release\net10.0\*" `
       "C:\publish\mxaccessgw\Server\"
   Copy-Item -Recurse -Force `
       "$gwSrc\src\MxGateway.Worker\bin\x86\Release\net48\*" `
       "C:\publish\mxaccessgw\Worker\"
   

3. Refresh OtOpcUa + historian sidecar binaries (Tracks B + C output):

   $repo = "C:\Users\dohertj2\Desktop\lmxopcua"
   dotnet publish "$repo\src\ZB.MOM.WW.OtOpcUa.Server" `
       -c Release -o "C:\publish\lmxopcua"
   dotnet publish "$repo\src\ZB.MOM.WW.OtOpcUa.Driver.Historian.Wonderware" `
       -c Release -o "C:\publish\lmxopcua\WonderwareHistorian"
   

4. Update service env block if Track C added the new toggle:

   # Pull existing env, append OTOPCUA_HISTORIAN_ALARM_WRITE_ENABLED=true
   # (default-on per C.2 design, but explicit assignment lets us flip false
   # for read-only deployments without re-installing)
   nssm set OtOpcUaWonderwareHistorian AppEnvironmentExtra `
       (((nssm get OtOpcUaWonderwareHistorian AppEnvironmentExtra) `
         + "`r`nOTOPCUA_HISTORIAN_ALARM_WRITE_ENABLED=true"))
   

5. Start in forward order:

   nssm start MxAccessGw
   Start-Sleep -Seconds 4
   nssm start OtOpcUaWonderwareHistorian
   Start-Sleep -Seconds 4
   nssm start OtOpcUa
   Start-Sleep -Seconds 8
   

6. Smoke verification:

   foreach ($s in 'MxAccessGw','OtOpcUaWonderwareHistorian','OtOpcUa') {
       (Get-Service $s).Status
   }
   foreach ($p in 5120, 4840, 4841) {
       Get-NetTCPConnection -LocalPort $p -State Listen `
           -ErrorAction SilentlyContinue
   }
   Get-Content "C:\publish\lmxopcua\logs\otopcua-*.log" -Tail 20
   Get-Content "C:\publish\mxaccessgw\stdout.log" -Tail 20
   Get-Content "C:\ProgramData\OtOpcUa\historian-wonderware-*.log" -Tail 10
   

   Pass criterion: all three services Running; ports 5120 + 4840 listening; sidecar log shows Wonderware historian sidecar serving — pipe=OtOpcUaWonderwareHistorian; OtOpcUa log shows OPC UA server started — endpoint=opc.tcp://0.0.0.0:4840/OtOpcUa and a new line IAlarmHistorianWriter resolved: Sidecar (added in B.4).

7. Functional verification — fire one alarm of each kind and assert it propagates:

   • Galaxy-native — raise the OtOpcUaParityTest_001.Counter $Alarm* extension via Galaxy's alarm-fire mechanism; assert an OPC UA Part 9 transition reaches a connected otopcua-cli alarms subscriber with rich payload (operator-comment field non-null, original-raise-timestamp present). This validates Track A + B.1
     • B.2 + B.3.
   • Scripted — author a one-line scripted alarm in the Admin UI against any always-true predicate; assert the transition lands in AVEVA Historian via aaHistClientTrend query (or Driver.Historian.Wonderware.IntegrationTests with a query for the alarm event). Validates Track C + B.4.
   • Sub-attribute fallback — disable IAlarmSource on the GalaxyDriver via the test seam (B.3 will introduce one); fire an alarm; assert Part 9 transition still raised by the value-driven path. Validates the fallback wasn't broken.

Files:

• scripts\install\Refresh-Services.ps1 (new — automates the above)
• docs\v2\dev-environment.md — add the refresh script to the dev workflow section.

Tests: smoke run on the dev rig (DESKTOP-6JL3KKO) producing docs\plans\artifacts\d1-rollout-YYYY-MM-DD.md with the captured log tails + smoke-test assertions. Captured artifact lands as part of the PR.

Rollback: the refresh script keeps a timestamped backup of the existing C:\publish\mxaccessgw\ and C:\publish\lmxopcua\ trees before overwriting (mirrored to C:\publish\.backup-YYYY-MM-DD\). Rollback is a stop / restore-from-backup / start sequence; no service re-install needed since the NSSM service definitions don't change.

Production deploy: out of scope for D.1 — the dev rig is the only deployment in scope at this point. A separate PR-or-runbook lands the production refresh once the dev rig has soaked for the documented duration (parity-rig validation gate; see "Test gates" above).

Sequencing matrix

Track A (mxaccessgw)              Track B (lmxopcua)              Track C (sidecar)
─────────────────────────         ─────────────────────────       ─────────────────────────
A.1 proto                         (waits)                         C.1 AahClientManagedAlarmEventWriter
   │                                                                 │ no cross-repo dep
   ├──────────────────────────►   B.1 EventPump branch                │
A.2 worker subscription                  │ uses proto types only      │
   │                                     │ unit-testable               │
   │                                                                 C.2 Program.cs wires writer
A.3 gateway dispatch + ack RPC ──►B.2 GalaxyDriver : IAlarmSource     │
   │                                     │                            │
   │                              ──►B.3 DriverNodeManager routing    │
   │                                                                  │
A.4 ConditionRefresh                     │                            │
                                         │                            │
                                         B.4 SidecarAlarmHistorianWriter
                                            (depends on C.2 deployed)

                                            ▼
                              Track D (deployment)
                              ─────────────────────────
                              D.1 Refresh C:\publish + restart services
                                  (depends on A.4 + B.4 + C.2 merged)
                                            ▼
                                  ──►B.5 docs + memory + completion banner

A.1 + B.1 + C.1 can all land in parallel — none have cross-repo runtime dependencies. B.1's tests use proto types without needing a running gateway. C.1 is purely sidecar-internal. The gateway-side dispatch (A.3) gates B.2; the sidecar-side wiring (C.2) gates B.4. D.1 (deployment refresh) gates B.5 (docs) — the docs sweep records the as-deployed state, so the deploy must be live first.

Test gates

Per PR: unit tests pass + build green + analyzer clean (Roslyn OTOPCUA0001 still wraps every alarm-capability call through AlarmSurfaceInvoker).

End-of-epic gate: re-run the parity rig (docs\v2\Galaxy.ParityRig.md) with these scenarios added:

1. Native alarm raise — Galaxy $Alarm* raise with operator-time metadata appears as an OPC UA Part 9 transition with full payload (no longer reconstructed from sub-attribute writes).
2. Native ack — OPC UA client acks; assert the gateway records the ack against MxAccess directly (not via sub-attribute write); operator comment present in the resulting Acknowledged transition.
3. ConditionRefresh after reconnect — disconnect the GalaxyDriver, raise three alarms in Galaxy, reconnect; assert all three appear in the next ConditionRefresh.
4. Historian write-back — fire a scripted alarm; assert it arrives in AVEVA Historian via the gateway path (use the existing Historian sidecar's read API to query it back).
5. Sub-attribute fallback still works — disable IAlarmSource on the GalaxyDriver via test seam, fire a sub-attribute value change; assert Part 9 transition still raised.

Soak target: 24h × 1k tags (light) — same parity-rig harness but extended to also subscribe to alarms. Pass criterion: zero dropped alarm transitions, zero state-machine inversions, zero unhandled exceptions in the AlarmSurfaceInvoker pipeline.

Risks and mitigations

Risk	Mitigation
MxAccess Toolkit alarm subscription API differs across installed AVEVA versions	PR A.2 verifies against the worker-host's installed Toolkit version; documents the exact API used. Pin the worker DLL set per major MxAccess version if needed.
Worker-side alarm subscription leaks between sessions if cleanup is wrong	PR A.2 includes a session-recycle test that asserts no IAlarmEventSink instances remain registered after Close.
Gateway adds a new auth scope (invoke:alarm-ack); existing keys lack it	PR A.3 + A.5 ship with a one-time bootstrap migration: keys with invoke:write get the new scope auto-granted on the dev rig and parity rig. Production keys are reissued via apikey rotate-key (existing CLI).
Two simultaneous alarm sources (driver-native + sub-attribute) double-fire transitions	PR B.3 dedup is the load-bearing design. End-to-end test #1 covers it explicitly.
Historian write-back batch fails mid-batch — partial success	The existing SqliteStoreAndForwardSink.HistorianWriteOutcome per-row enum + dead-letter retention already handles this; PR A.5 just exposes the same outcome shape over gRPC.
Sidecar starts honouring the WriteAlarmEvents slot — old lmxopcua-side consumers can now reach a previously inert path	The slot returns Success=false, Error="not configured" today; flipping to live writes means a build that speculatively sent the frame would suddenly start producing real historian rows. Inventory of any such caller is empty — WriteAlarmEvents was never invoked from the lmxopcua side; Phase7EngineComposer.RouteToHistorianAsync queues into SqliteStoreAndForwardSink and the drain worker is gated on IAlarmHistorianWriter registration which only the new B.4 path provides. So enabling C.2 without B.4 is safe.

Roll-out

Track A lands first onto mxaccessgw/main, deployed to the parity rig. Track B lands onto lmxopcua/master once A.3 is live on the rig — earlier Track B PRs can target a feature branch (feat/alarms-over-gateway) and merge to master after the rig is fully green.

Back-out

Each PR is individually revertable. The cleanest back-out point is at the gateway-side enum extension: removing MX_EVENT_FAMILY_ON_ALARM_TRANSITION from the proto means EventPump silently drops alarm events again and GalaxyDriver's OnAlarmEvent never fires — but the sub-attribute fallback path still produces functional alarms, so the OPC UA surface degrades to v2-current behaviour without breaking. PR B.4 is the only one with a non-trivial back-out (re-add the deleted sidecar IPC slot if revert needed); land B.4 last and only after end-of-epic gate is green.

Out of scope (explicit)

• Other alarm sources beyond Galaxy. AbCip / FOCAS / OpcUaClient drivers already implement IAlarmSource; they're untouched.
• Modbus / S7 / AbLegacy / TwinCAT alarms. None of those protocols has a native alarm bus. Alarms on those drivers, if needed, ship via the scripted-alarm path.
• Multi-Galaxy ack routing. Today's gateway model is one Galaxy per session; if a deployment splits across galaxies, each gets its own GalaxyDriver and they don't cross-talk. No change.
• OPC UA Part 9 advanced features beyond the current scope — shelving, subscribed-to-events-only, branch-state for re-trigger semantics. Future epic if a customer asks.
• Insight / cloud Historian write-back path. Track A.5 targets the on-prem AVEVA Historian via aahClientManaged. The cloud variant would mirror the same gateway RPC over the REST API discussed in docs/histsdk — separate epic.

File inventory (touched)

mxaccessgw (Track A):

• src\MxGateway.Contracts\Protos\mxaccess_gateway.proto (A.1)
• src\MxGateway.Contracts\Protos\mxaccess_worker.proto (A.2, A.4)
• src\MxGateway.Worker\…\Eventing\ (A.2, A.3, A.4)
• src\MxGateway.Worker\…\Commands\ (A.3, A.4)
• src\MxGateway.Server\Sessions\SessionEventStream.cs (A.3)
• src\MxGateway.Server\Rpc\ (A.3, A.4)
• src\MxGateway.Server\Auth\Scopes.cs (A.3, A.4)
• MxGateway.Tests, MxGateway.Worker.Tests, MxGateway.IntegrationTests

lmxopcua — Galaxy driver + server (Track B):

• src\ZB.MOM.WW.OtOpcUa.Driver.Galaxy\Runtime\EventPump.cs (B.1)
• src\ZB.MOM.WW.OtOpcUa.Driver.Galaxy\Runtime\MxAccessSeverityMapper.cs (new — B.1)
• src\ZB.MOM.WW.OtOpcUa.Driver.Galaxy\Runtime\IGalaxyAlarmAcknowledger.cs (new — B.2)
• src\ZB.MOM.WW.OtOpcUa.Driver.Galaxy\Runtime\GatewayGalaxyAlarmAcknowledger.cs (new — B.2)
• src\ZB.MOM.WW.OtOpcUa.Driver.Galaxy\GalaxyDriver.cs (B.2)
• src\ZB.MOM.WW.OtOpcUa.Driver.Galaxy\GalaxyDriverFactory.cs (B.2)
• src\ZB.MOM.WW.OtOpcUa.Server\OpcUa\DriverNodeManager.cs (B.3)
• src\ZB.MOM.WW.OtOpcUa.Server\Alarms\AlarmConditionService.cs (B.3)
• src\ZB.MOM.WW.OtOpcUa.Server\Phase7\Phase7Composer.cs (B.4)
• src\ZB.MOM.WW.OtOpcUa.Driver.Historian.Wonderware.Client\SidecarAlarmHistorianWriter.cs (new — B.4)
• tests\ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Tests\Runtime\ (B.1, B.2)
• tests\ZB.MOM.WW.OtOpcUa.Server.Tests\Alarms\ (B.3)
• tests\ZB.MOM.WW.OtOpcUa.Driver.Historian.Wonderware.Client.Tests\ (B.4 — new tests)
• docs\drivers\Galaxy.md (B.5)
• docs\AlarmTracking.md (new — B.5)
• docs\v1\AlarmTracking.md (B.5 — banner update)
• docs\plans\alarms-over-gateway.md (B.5 — completion banner)

lmxopcua — Wonderware historian sidecar (Track C):

• src\ZB.MOM.WW.OtOpcUa.Driver.Historian.Wonderware\Backend\AahClientManagedAlarmEventWriter.cs (new — C.1)
• src\ZB.MOM.WW.OtOpcUa.Driver.Historian.Wonderware\Program.cs (C.2 — wire writer)
• scripts\install\Install-Services.ps1 (C.2 — env-var toggle for write-enable)
• tests\ZB.MOM.WW.OtOpcUa.Driver.Historian.Wonderware.Tests\ (C.1 — outcome mapping + batch + cluster failover)

lmxopcua — deployment refresh (Track D):

• scripts\install\Refresh-Services.ps1 (new — D.1)
• docs\v2\dev-environment.md (D.1 — document the refresh workflow)
• docs\plans\artifacts\d1-rollout-YYYY-MM-DD.md (new — D.1 captured smoke run)

Total: ~10 source files added/modified in mxaccessgw; ~14 in lmxopcua proper; ~3 in the historian sidecar; ~2 deployment scripts; ~12 test files across all repos. Should land in 4-6 weeks of focused work given the parity-rig dependency for end-to-end validation, plus a short final-week ops slot for D.1.