dohertj2/lmxopcua
1
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases Wiki Activity

OpcUaClient native-alarm feed not restored by reconcile / lost on a non-respawning deploy #662

New Issue
Closed
opened 2026-10-09 07:12:42 -04:00 by dohertj2 · 1 comment
dohertj2 commented 2026-10-09 07:12:42 -04:00
Owner
Copy Link
Copy Source

Found during the #657 live gate on the docker-dev rig (2026-10-09). This is not fixed by #657.

Symptom

An OpcUaClient driver's native-alarm feed silently stops. Nothing marks it as stopped: the condition freezes at its last state while upstream keeps emitting events. Only a driver respawn brings it back (e.g. toggling the DriverConfig so the next deploy recreates the driver).

Case 1: a deploy that doesn't respawn the driver

  • 10:12: the first deploy added the alarm tag. The log said "native-alarm subscription established", but no events were routed.
  • 10:42: a deploy that only disabled another driver re-subscribed MAIN-opcua's data (opcua-sub-2) and logged no alarm re-subscribe. Central's condition then stayed frozen at its 10:42:11 state for 25+ minutes while opc-plc kept emitting.

Case 2: after an outage

  • The driver reconnected but logged "native-alarm subscription failed — feed stays gated until reconnect".
  • The periodic reconcile then re-subscribed data only (opcua-sub-20). The alarm subscription was never retried.

Evidence

central-1 container logs, 10:12–10:42 and 10:39:04–10:40:18 (2026-10-09).

Likely area

The resubscribe/reconcile path that DriverInstanceActor runs after init or reconnect (#630: "the actor re-subscribes after every successful init") covers ISubscribable but apparently not IAlarmSource.SubscribeAlarmsAsync. The OpcUaClient alarm subscription also has a "gated until reconnect" state that nothing re-drives.

Expected

Every path that restores the data subscription also restores the alarm subscription: a non-respawning deploy, a reconnect, and the periodic reconcile. A failed alarm subscribe is retried, or surfaces in driver health/diagnostics instead of silently gating the feed.

Test

An end-to-end test through DriverHostActor/DriverInstanceActor:

  1. Bring up the alarm feed.
  2. Trigger a non-respawning deploy (and, separately, a reconnect).
  3. Assert that a subsequent upstream alarm event still reaches the condition.

Related: #657, #641, #630.

Found during the #657 live gate on the docker-dev rig (2026-10-09). This is not fixed by #657. ## Symptom An OpcUaClient driver's native-alarm feed silently stops. Nothing marks it as stopped: the condition freezes at its last state while upstream keeps emitting events. Only a driver **respawn** brings it back (e.g. toggling the DriverConfig so the next deploy recreates the driver). ## Case 1: a deploy that doesn't respawn the driver - 10:12: the first deploy added the alarm tag. The log said "native-alarm subscription established", but no events were routed. - 10:42: a deploy that only disabled *another* driver re-subscribed MAIN-opcua's **data** (`opcua-sub-2`) and logged no alarm re-subscribe. Central's condition then stayed frozen at its 10:42:11 state for 25+ minutes while opc-plc kept emitting. ## Case 2: after an outage - The driver reconnected but logged "native-alarm subscription failed — feed stays gated until reconnect". - The periodic reconcile then re-subscribed **data only** (`opcua-sub-20`). The alarm subscription was never retried. ## Evidence central-1 container logs, 10:12–10:42 and 10:39:04–10:40:18 (2026-10-09). ## Likely area The resubscribe/reconcile path that `DriverInstanceActor` runs after init or reconnect (#630: "the actor re-subscribes after every successful init") covers `ISubscribable` but apparently not `IAlarmSource.SubscribeAlarmsAsync`. The OpcUaClient alarm subscription also has a "gated until reconnect" state that nothing re-drives. ## Expected Every path that restores the data subscription also restores the alarm subscription: a non-respawning deploy, a reconnect, and the periodic reconcile. A failed alarm subscribe is retried, or surfaces in driver health/diagnostics instead of silently gating the feed. ## Test An end-to-end test through `DriverHostActor`/`DriverInstanceActor`: 1. Bring up the alarm feed. 2. Trigger a non-respawning deploy (and, separately, a reconnect). 3. Assert that a subsequent upstream alarm event still reaches the condition. Related: #657, #641, #630.
dohertj2 commented 2026-10-09 07:41:09 -04:00
Author
Owner
Copy Link
Copy Source

Fixed on master 83d129d1 (5067509d + review follow-up 28ca04e4).

Root cause

In DriverInstanceActor, the data re-subscribe path (HandleSubscribeAsync, then UnsubscribeAsync, then DetachSubscription) also ran DetachAlarmSource. That unhooked OnAlarmEvent and forgot the alarm handle.

Every deploy's bulk SetDesiredSubscriptions push re-subscribes the data of each driver it keeps. So after any deploy that did not respawn a driver, the alarm subscription still logged "established" but its events reached nobody. This explains both 10:12 and 10:42. The bug is in generic actor code, so it hit every native-alarm driver (OpcUaClient, Galaxy, FOCAS, MTConnect, AbCip), not just OpcUaClient.

The #488 health-poll reconcile only checked data. That is why a failed alarm subscribe after the outage was never retried (case 2).

Fix

  • Data-only teardown: data paths now use DetachDataSubscription. Only DisconnectObserved, ForceReconnect and PostStop detach both data and alarms.
  • Alarm reconcile: ReconcileAlarmSubscription runs on each health-poll tick.
    • It re-attaches a missing handler and re-subscribes when the alarm handle is missing or doesn't match the desired refs.
    • A persistently failing subscribe backs off to at most every 7 ticks, which is about 4 minutes in production.
    • The stack trace is logged once per failure streak.
  • OpcUaClient cleanup: a failed SubscribeAlarmsAsync detaches its handler and removes the half-built subscription from the session, with a 5 s bound. Events from an unregistered subscription id are dropped.

Tests

  • Actor level: a data re-subscribe keeps the feed attached. A failed alarm subscribe is retried by the reconcile, tested with 1 failure and with 3 failures, which crosses the backoff.
  • End to end, through the real DriverHostActor and DriverInstanceActor (NativeAlarmFeedSurvivesDeployTests): a deploy that only adds another driver re-subscribes DI-1's data without re-initialising it, and an upstream Raise still reaches the condition.
  • All three new tests fail against the unfixed actor.
  • Full suite on the merged result: 13214 passed, 0 failed.

Not re-gated live on the docker-dev rig. The end-to-end test reproduces the exact actor path the rig logs showed.

Fixed on master `83d129d1` (`5067509d` + review follow-up `28ca04e4`). ## Root cause In `DriverInstanceActor`, the data re-subscribe path (`HandleSubscribeAsync`, then `UnsubscribeAsync`, then `DetachSubscription`) also ran `DetachAlarmSource`. That unhooked `OnAlarmEvent` and forgot the alarm handle. Every deploy's bulk `SetDesiredSubscriptions` push re-subscribes the data of each driver it keeps. So after any deploy that did not respawn a driver, the alarm subscription still logged "established" but its events reached nobody. This explains both 10:12 and 10:42. The bug is in generic actor code, so it hit **every native-alarm driver** (OpcUaClient, Galaxy, FOCAS, MTConnect, AbCip), not just OpcUaClient. The #488 health-poll reconcile only checked data. That is why a failed alarm subscribe after the outage was never retried (case 2). ## Fix - **Data-only teardown:** data paths now use `DetachDataSubscription`. Only `DisconnectObserved`, `ForceReconnect` and `PostStop` detach both data and alarms. - **Alarm reconcile:** `ReconcileAlarmSubscription` runs on each health-poll tick. - It re-attaches a missing handler and re-subscribes when the alarm handle is missing or doesn't match the desired refs. - A persistently failing subscribe backs off to at most every 7 ticks, which is about 4 minutes in production. - The stack trace is logged once per failure streak. - **OpcUaClient cleanup:** a failed `SubscribeAlarmsAsync` detaches its handler and removes the half-built subscription from the session, with a 5 s bound. Events from an unregistered subscription id are dropped. ## Tests - Actor level: a data re-subscribe keeps the feed attached. A failed alarm subscribe is retried by the reconcile, tested with 1 failure and with 3 failures, which crosses the backoff. - End to end, through the real `DriverHostActor` and `DriverInstanceActor` (`NativeAlarmFeedSurvivesDeployTests`): a deploy that only adds another driver re-subscribes DI-1's data without re-initialising it, and an upstream Raise still reaches the condition. - All three new tests fail against the unfixed actor. - Full suite on the merged result: 13214 passed, 0 failed. Not re-gated live on the docker-dev rig. The end-to-end test reproduces the exact actor path the rig logs showed.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
auto-managed
queue management
 cross-driver
queue management
 driver/abcip
queue management
 driver/ablegacy
queue management
 driver/focas
queue management
 driver/mqtt
MQTT / Sparkplug B broker-subscribe driver
 driver/opcuaclient
queue management
 driver/s7
queue management
 driver/twincat
queue management
 phase/1
queue management
 phase/2
queue management
 phase/3
queue management
 phase/4
queue management
 phase/5
queue management
 phase/6
queue management
 queue/blocked
queue management
 queue/done
queue management
 queue/failed
queue management
 queue/in-progress
queue management
 queue/queued
queue management
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
dohertj2 (dohertj2)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dohertj2/lmxopcua#662
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?