dohertj2/lmxopcua
1
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(alarms): bridge driver comms-loss into scripted-alarm input quality (Layer 4 of #477) #481

New Issue
Closed
opened 2026-07-17 16:07:34 -04:00 by dohertj2 · 1 comment
dohertj2 commented 2026-07-17 16:07:34 -04:00
Owner
Copy Link
Copy Source

Summary

Layer 4 of the condition-Quality chain (#477 → #478). #478 shipped scripted-alarm Quality derived from the worst of its input tags' qualities, but only for inputs whose driver publishes a data change carrying a Bad/Uncertain StatusCode (e.g. an OpcUaClient input forwarding a server's per-item Bad).

It does not cover a driver comms loss: a poll driver (Modbus/S7) whose device goes unreachable emits only DriverInstanceActor.ConnectivityChanged(false) and goes silent on the value feed — it does not republish its tags as Bad. So the scripted engine keeps the last-known Good value, WorstInputStatus stays Good, and:

  • the scripted condition's Quality stays Good (opposite of the "can I trust this condition's state?" intent), and
  • AreInputsReady stays true (last value is Good), so the predicate keeps evaluating a stale value after the source died.

Native alarms already solve this: DriverHostActor.OnDriverConnectivityChanged fans an AlarmQualityUpdate(Bad) to every native condition the driver owns (_alarmNodeIdByDriverRef). Scripted alarms need the symmetric bridge into their inputs.

The ask

Bridge driver connectivity into scripted-alarm input quality:

  • On ConnectivityChanged(false), mark every mux ref the driver publishes as Bad for the scripted engine (restored to real quality on reconnect, which self-heals via ResubscribeDesired republishing values).
  • The DriverHostActor currently tracks driver → native alarm nodes (_alarmNodeIdByDriverRef) but not driver → all tag refs; that per-driver ref set needs to exist to fan connectivity into the mux.
  • Resolve the null-value / cold-start asymmetry (#478 Finding 3): WorstInputStatus skips a null-value input as "no data yet" (contributes Good) to avoid a deploy-time Bad flash, but that makes a genuine runtime Bad with a null value indistinguishable from cold start. Distinguish "never pushed" (cold → Good) from "pushed Bad" (→ Bad), e.g. by tracking pushed-ness in DependencyMuxTagUpstreamSource.

Notes / considerations

  • Pushing a synthetic Bad into the mux also makes virtual tags computed from that input go Bad — arguably correct, but a behavior change to weigh (VT already has its own Bad-propagation path; confirm no double-handling).
  • Verify on the docker-dev rig with a Modbus/S7-fed scripted alarm: pause the fixture container (comms loss) → assert the scripted condition reports Bad on the wire; unpause → Good. (This is the same live-gate style the #477 native path used.)

References

  • #477 (Layers 1+2, native connectivity-driven quality) and its OnDriverConnectivityChanged bridge.
  • #478 (Layer 3, mux-delivered input quality) — this issue is its comms-loss extension. See docs/AlarmTracking.md §"Scripted alarms (Layer 3, #478)" "Coverage boundary" and docs/plans/2026-07-17-alarm-condition-quality-477-design.md.
## Summary **Layer 4** of the condition-Quality chain (#477 → #478). #478 shipped scripted-alarm `Quality` derived from the **worst of its input tags' qualities**, but only for inputs whose driver **publishes a data change carrying a Bad/Uncertain `StatusCode`** (e.g. an OpcUaClient input forwarding a server's per-item Bad). It does **not** cover a driver **comms loss**: a poll driver (Modbus/S7) whose device goes unreachable emits only `DriverInstanceActor.ConnectivityChanged(false)` and goes *silent* on the value feed — it does not republish its tags as Bad. So the scripted engine keeps the last-known **Good** value, `WorstInputStatus` stays Good, and: - the scripted condition's `Quality` stays **Good** (opposite of the "can I trust this condition's state?" intent), and - `AreInputsReady` stays true (last value is Good), so the predicate **keeps evaluating a stale value** after the source died. Native alarms already solve this: `DriverHostActor.OnDriverConnectivityChanged` fans an `AlarmQualityUpdate(Bad)` to every **native** condition the driver owns (`_alarmNodeIdByDriverRef`). Scripted alarms need the symmetric bridge into their **inputs**. ## The ask Bridge driver connectivity into scripted-alarm input quality: - On `ConnectivityChanged(false)`, mark every mux ref the driver publishes as `Bad` for the scripted engine (restored to real quality on reconnect, which self-heals via `ResubscribeDesired` republishing values). - The `DriverHostActor` currently tracks driver → **native alarm nodes** (`_alarmNodeIdByDriverRef`) but not driver → **all tag refs**; that per-driver ref set needs to exist to fan connectivity into the mux. - Resolve the **null-value / cold-start asymmetry** (#478 Finding 3): `WorstInputStatus` skips a null-value input as "no data yet" (contributes Good) to avoid a deploy-time Bad flash, but that makes a genuine runtime `Bad` with a null value indistinguishable from cold start. Distinguish "never pushed" (cold → Good) from "pushed Bad" (→ Bad), e.g. by tracking pushed-ness in `DependencyMuxTagUpstreamSource`. ## Notes / considerations - Pushing a synthetic Bad into the mux also makes **virtual tags** computed from that input go `Bad` — arguably correct, but a behavior change to weigh (VT already has its own Bad-propagation path; confirm no double-handling). - Verify on the docker-dev rig with a Modbus/S7-fed scripted alarm: pause the fixture container (comms loss) → assert the scripted condition reports `Bad` on the wire; unpause → `Good`. (This is the same live-gate style the #477 native path used.) ## References - #477 (Layers 1+2, native connectivity-driven quality) and its `OnDriverConnectivityChanged` bridge. - #478 (Layer 3, mux-delivered input quality) — this issue is its comms-loss extension. See `docs/AlarmTracking.md` §"Scripted alarms (Layer 3, #478)" "Coverage boundary" and `docs/plans/2026-07-17-alarm-condition-quality-477-design.md`.
dohertj2 commented 2026-07-31 10:04:09 -04:00
Author
Owner
Copy Link
Copy Source

Implemented — master c81cef39 (merged 9fa701b1). Closing the code work; the live gate is carried by #527, filed from this gate — read the last section before relying on this path.

DriverHostActor.FanCommsLossToDependencyMux pushes a synthetic BadNoCommunication with a null value into the dependency mux for every value ref a just-disconnected driver owns.

The ref set already existed. This issue asked for a driver → all-tag-refs map to be built; _nodeIdByDriverRef is exactly that, keyed (DriverInstanceId, RawPath) and rebuilt from the composition on every apply. Alarm-bearing tags live in the separate _alarmNodeIdByDriverRef and are deliberately excluded — a native condition is not a mux value and takes the Part 9 quality path instead.

Nothing is pushed on reconnect, deliberately — publishing Good would assert a value quality the host has not observed. Recovery rides the driver's own republish, and I verified the mechanism rather than assuming it: the Connected → Reconnecting edge calls DetachSubscription, which nulls the subscription handle, so the reconnect's ResubscribeDesired builds a fresh subscription whose PollGroupEngine initial push runs with forceRaise: true. Without that detach the poll engine would compare against its retained LastValues, see an unchanged value, raise nothing, and strand these refs Bad forever — the value-dedup trap that bit the v3 Batch 4 virtual-tag reassert.

Finding 3 is closed, and the fan-out depends on it. WorstInputStatus skipped every input whose value was null as "no data yet" — and Bad-with-no-value is exactly the shape a comms-lost driver produces, so the skip was hiding the case that matters most. Without this half the fan-out above would have been silently inert. The upstream source now stamps a never-pushed path BadWaitingForInitialData (the honest Part 4 code) and the engine skips that specific code rather than every null.

Deleting the null-skip outright is the obvious fix and is wrong in the other direction: it makes every not-yet-published input drag its condition Bad at deploy, which is what the skip existed to prevent. Both mistakes are pinned by a test pair that goes red under the opposite error. Falsified by reverting the production code:

old null-skip restored     -> A_pushed_Bad_carrying_no_value   RED
null-skip deleted outright -> An_input_that_was_never_pushed   RED
fan-out call removed       -> the three DriverHostActor tests  RED

The never-pushed test was vacuous when first written and is worth recording: the quality bucket only emits on change, so the load-time evaluation is the one that flashes Bad, and subscribing to OnEvent after LoadAsync made the assertion unfalsifiable — it passed against the broken behaviour. It now subscribes first. Relatedly, FakeUpstream.ReadTag returned 0x80340000 for an unset path, a different Bad code from production's placeholder; harmless while any Bad behaved alike, load-bearing now, so it is aligned and pinned by a reflection guard asserting the engine's private constant agrees with what the production source stamps.

Behaviour change you flagged, confirmed: virtual tags computed from a comms-lost input now go Bad too. That follows from using the shared mux and reads as correct, but it is broader than scripted alarms. No double-handling was found.

Live gate: could not close on the headline scenario — #527

The docker-dev gate this issue specifies (Modbus-fed scripted alarm, pause the fixture, assert Bad on the wire) did not close, for a reason worth knowing: this issue's premise is wrong.

a poll driver (Modbus/S7) whose device goes unreachable emits only DriverInstanceActor.ConnectivityChanged(false)

It emits no such thing. DisconnectObserved has no production sender anywhere in src/ — the only paths into Reconnecting are a failed initial connect and an operator Force Reconnect. Live on the rig: with the Modbus fixture stopped, the driver stayed Connected for over ten minutes publishing per-tag Bad, and did not recover when the fixture came back until the node was restarted. Filed with the full evidence as #527.

So this fan-out is correctly wired to a signal that a dying device never raises. It does fire on the two paths that exist today (failed initial connect, admin Force Reconnect), and it is the right wiring for when #527 lands — whose acceptance criteria include exactly the Quality-on-the-wire check this gate wanted.

What the gate did establish: the change deploys and runs (scripted alarm materialised on central-1 against the seeded Modbus tag through a real POST /api/deployments), and #478 Layer 3 does carry Bad into scripted input quality for Modbus, because that driver publishes Bad per tag rather than going silent. That is a per-driver accident, not a guarantee.

One incidental gap: Client.CLI cannot observe condition Quality — its alarm event filter has no select clause for it, and read throws on the node's Bad DataValue status before showing the value. Three consecutive issues (#477, #478, this one) have turned on Quality, so adding it to the CLI filter would pay for itself; noted rather than done here.

Implemented — master `c81cef39` (merged `9fa701b1`). **Closing the code work; the live gate is carried by #527, filed from this gate — read the last section before relying on this path.** `DriverHostActor.FanCommsLossToDependencyMux` pushes a synthetic `BadNoCommunication` with a **null value** into the dependency mux for every value ref a just-disconnected driver owns. **The ref set already existed.** This issue asked for a driver → all-tag-refs map to be built; `_nodeIdByDriverRef` is exactly that, keyed `(DriverInstanceId, RawPath)` and rebuilt from the composition on every apply. Alarm-bearing tags live in the separate `_alarmNodeIdByDriverRef` and are deliberately excluded — a native condition is not a mux value and takes the Part 9 quality path instead. **Nothing is pushed on reconnect, deliberately** — publishing Good would assert a value quality the host has not observed. Recovery rides the driver's own republish, and I verified the mechanism rather than assuming it: the `Connected → Reconnecting` edge calls `DetachSubscription`, which nulls the subscription handle, so the reconnect's `ResubscribeDesired` builds a **fresh** subscription whose `PollGroupEngine` initial push runs with `forceRaise: true`. Without that detach the poll engine would compare against its retained `LastValues`, see an unchanged value, raise nothing, and strand these refs Bad forever — the value-dedup trap that bit the v3 Batch 4 virtual-tag reassert. **Finding 3 is closed, and the fan-out depends on it.** `WorstInputStatus` skipped every input whose *value* was null as "no data yet" — and Bad-with-no-value is exactly the shape a comms-lost driver produces, so the skip was hiding the case that matters most. Without this half the fan-out above would have been silently inert. The upstream source now stamps a never-pushed path `BadWaitingForInitialData` (the honest Part 4 code) and the engine skips *that specific code* rather than every null. Deleting the null-skip outright is the obvious fix and is wrong in the other direction: it makes every not-yet-published input drag its condition Bad at deploy, which is what the skip existed to prevent. Both mistakes are pinned by a test pair that goes red under the opposite error. Falsified by reverting the production code: ``` old null-skip restored -> A_pushed_Bad_carrying_no_value RED null-skip deleted outright -> An_input_that_was_never_pushed RED fan-out call removed -> the three DriverHostActor tests RED ``` The never-pushed test was **vacuous when first written** and is worth recording: the quality bucket only emits on *change*, so the load-time evaluation is the one that flashes Bad, and subscribing to `OnEvent` after `LoadAsync` made the assertion unfalsifiable — it passed against the broken behaviour. It now subscribes first. Relatedly, `FakeUpstream.ReadTag` returned `0x80340000` for an unset path, a *different* Bad code from production's placeholder; harmless while any Bad behaved alike, load-bearing now, so it is aligned and pinned by a reflection guard asserting the engine's private constant agrees with what the production source stamps. **Behaviour change you flagged, confirmed:** virtual tags computed from a comms-lost input now go Bad too. That follows from using the shared mux and reads as correct, but it is broader than scripted alarms. No double-handling was found. --- ### Live gate: could not close on the headline scenario — #527 The docker-dev gate this issue specifies (Modbus-fed scripted alarm, pause the fixture, assert Bad on the wire) **did not close**, for a reason worth knowing: this issue's premise is wrong. > a poll driver (Modbus/S7) whose device goes unreachable emits only `DriverInstanceActor.ConnectivityChanged(false)` It emits **no such thing**. `DisconnectObserved` has no production sender anywhere in `src/` — the only paths into `Reconnecting` are a failed *initial* connect and an operator Force Reconnect. Live on the rig: with the Modbus fixture stopped, the driver stayed `Connected` for over ten minutes publishing per-tag Bad, and **did not recover when the fixture came back** until the node was restarted. Filed with the full evidence as **#527**. So this fan-out is correctly wired to a signal that a dying device never raises. It does fire on the two paths that exist today (failed initial connect, admin Force Reconnect), and it is the right wiring for when #527 lands — whose acceptance criteria include exactly the Quality-on-the-wire check this gate wanted. What the gate **did** establish: the change deploys and runs (scripted alarm materialised on central-1 against the seeded Modbus tag through a real `POST /api/deployments`), and #478 Layer 3 does carry Bad into scripted input quality for Modbus, because that driver publishes Bad per tag rather than going silent. That is a per-driver accident, not a guarantee. One incidental gap: **Client.CLI cannot observe condition `Quality`** — its alarm event filter has no select clause for it, and `read` throws on the node's Bad DataValue status before showing the value. Three consecutive issues (#477, #478, this one) have turned on Quality, so adding it to the CLI filter would pay for itself; noted rather than done here.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
auto-managed
queue management
 cross-driver
queue management
 driver/abcip
queue management
 driver/ablegacy
queue management
 driver/focas
queue management
 driver/mqtt
MQTT / Sparkplug B broker-subscribe driver
 driver/opcuaclient
queue management
 driver/s7
queue management
 driver/twincat
queue management
 phase/1
queue management
 phase/2
queue management
 phase/3
queue management
 phase/4
queue management
 phase/5
queue management
 phase/6
queue management
 queue/blocked
queue management
 queue/done
queue management
 queue/failed
queue management
 queue/in-progress
queue management
 queue/queued
queue management
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
dohertj2 (dohertj2)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dohertj2/lmxopcua#481
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?