Merge branch 'fix/r3-02-stale-good-family' — R3-02: one quality contract at four seams — comms loss marks raw+UNS value variables BadNoCommunication (WriteStatuses sink member, recovery audit), Calculation + VT hold on Bad / worst-of sub-code / (value,status) dedup, MQTT initial push + ConnectionLost stale, Sparkplug birth replay (archreview 03/S15, 05/STAB-19, 05/STAB-18, 02/S14)
This commit is contained in:
Joseph Doherty
+64
-2
@@ -306,6 +306,37 @@ So a driver whose `InitializeAsync` disposes anything on its failure path must b
|
||||
`MTConnectDriver` does today, because it is the only one that tears down inside a `catch`. The
|
||||
shutdown paths deliberately keep the single trailing write.
|
||||
|
||||
### The quality contract
|
||||
|
||||
> **A value consumer never reports a quality better than its source's.** A node derived from a driver
|
||||
> value — the raw node, every UNS mirror, a Calculation output, a virtual tag, a scripted condition's
|
||||
> quality — carries, at worst, the worst status among its inputs: a Bad input makes the derived node Bad
|
||||
> **with that input's sub-code and the last-known value**; an Uncertain input makes it Uncertain; a source
|
||||
> that has stopped communicating is a Bad input (`BadNoCommunication`) for everything downstream of it,
|
||||
> **including the external OPC UA data plane**. Recovery is a *publish*, never an assumption: the first
|
||||
> post-recovery value must land even when it equals the value held before the outage, so every dedup on
|
||||
> the path — driver caches, the node's own status, a consumer's last-published memory — must be keyed on
|
||||
> **(value, status)**, not value alone.
|
||||
|
||||
This is one rule with one implementation of its comparison,
|
||||
`ZB.MOM.WW.OtOpcUa.Core.Abstractions.InputQuality` (`WorstOf` / `IsUsable`), used by every derived-value
|
||||
seam. It was written down because the same lie had been fixed four times in four places and re-opened in
|
||||
a fifth (archreview round 3, theme "the stale-Good family"). The seams that implement it:
|
||||
|
||||
| Seam | What it does | Where |
|
||||
|---|---|---|
|
||||
| Driver comms loss → **value nodes** | one status-only batch stamps `BadNoCommunication` on the driver's raw NodeIds and every referencing UNS NodeId, **value retained** | `DriverHostActor.FanCommsLossToValueNodes` (below) |
|
||||
| Driver comms loss → dependency mux | synthetic Bad per ref, so scripts and alarms see the source die | `DriverHostActor.FanCommsLossToDependencyMux` |
|
||||
| Driver comms loss → Part 9 conditions | condition quality annotated out of band | `DriverHostActor.OnDriverConnectivityChanged` |
|
||||
| MQTT / Sparkplug | connection loss stales observed refs; every (re)subscribe pushes the cached snapshot | [`docs/drivers/Mqtt.md`](drivers/Mqtt.md#quality-on-connection-loss-and-re-subscribe) |
|
||||
| Calculation driver | holds on Bad, propagates the worst input status | [`docs/drivers/Calculation.md`](drivers/Calculation.md#input-quality) |
|
||||
| Virtual tags | holds on Bad, inherits the worst input status, reassert carries it | [`docs/ScriptEditor.md`](ScriptEditor.md#input-quality-what-a-script-sees-and-when-it-runs-at-all) |
|
||||
| Scripted alarms | worst-of-input quality annotated out of band; Bad inputs hold the condition | [`docs/ScriptedAlarms.md`](ScriptedAlarms.md) |
|
||||
|
||||
⚠️ **Consumer-visible:** value nodes of a comms-lost driver now go **Bad**, where they used to serve
|
||||
their last value at Good. A consumer that treats a Bad value as "drop the tag" (ScadaBridge's data
|
||||
connection layer, say) will see more Bads than before — that is the fix working, not a regression.
|
||||
|
||||
### The comms-loss watchdog (Gitea #527) — `GetHealth()` is now load-bearing
|
||||
|
||||
`GetHealth()` used to be **reporting-only**: `DriverInstanceActor.PublishHealthSnapshot` read it every 30 s
|
||||
@@ -391,8 +422,8 @@ on it still gets its initialise attempt.
|
||||
#### The fan-out that watchdog triggers: indexed per driver, and one batch
|
||||
|
||||
`DisconnectObserved` raises `ConnectivityChanged`, and `DriverHostActor` answers it by pushing
|
||||
`BadNoCommunication` at every ref the flapping driver owns — the value fan to the dependency mux, and the
|
||||
alarm fan to the publish actor's condition quality. Both used to answer "which refs are this driver's?"
|
||||
`BadNoCommunication` at every ref the flapping driver owns — **three** fans: the value nodes (the
|
||||
external OPC UA surface), the dependency mux, and the publish actor's condition quality. Both used to answer "which refs are this driver's?"
|
||||
by **scanning the node's entire ref map and filtering** — the maps are keyed by the composite
|
||||
`(DriverInstanceId, RawPath)`, so one driver's flap walked all ~50k keys on a large node to find its own
|
||||
few thousand. Each now has a secondary index, `DriverInstanceId → [(RawPath, node set)]`
|
||||
@@ -418,6 +449,37 @@ its last one — everything on the subscription path was already wrapped in `Att
|
||||
`DependencyMuxActor` still registers the per-value receive, but in production it is dead code; it stays
|
||||
for tests and direct senders.
|
||||
|
||||
#### The third fan: the value nodes (archreview 03/S15)
|
||||
|
||||
The two fans above reach *internal* consumers and *conditions*. Until archreview round 3 the driver's raw
|
||||
value variables and their UNS mirrors — what an OPC UA client actually subscribes to — got nothing, so a
|
||||
client saw the last value at **Good** quality for the whole outage. That was invisible on Modbus (which
|
||||
degrades per tag within one poll interval on its own) and total on every driver whose failure shape is
|
||||
"the reader stops producing": a coalesced-read throw, a dropped push session, or simply a driver whose
|
||||
last act was a Good publish.
|
||||
|
||||
`FanCommsLossToValueNodes` closes it with **one status-only batch** — a new sink primitive,
|
||||
`IOpcUaAddressSpaceSink.WriteStatuses` → `OtOpcUaNodeManager.WriteStatuses` — that changes each node's
|
||||
StatusCode and Timestamp and **leaves its Value alone**. Status-only is not an optimisation: the host
|
||||
holds no value cache (v3's single-source design deliberately removed the independent buffer), so only the
|
||||
node itself can retain the last known value, and a Bad node still carrying its last reading is the OPC UA
|
||||
Part 8 convention a SCADA consumer expects. Two details:
|
||||
|
||||
- It is **ungated by redundancy role**, like the condition annotation: a Secondary keeps its address space
|
||||
honest for failover.
|
||||
- An unmaterialised NodeId is **skipped, never created** — a status carries nothing to create a node from,
|
||||
and the per-driver index can legitimately name a node a surgical remove has just dropped.
|
||||
|
||||
**Nothing is written on reconnect.** Publishing Good there would assert a quality the host has not
|
||||
observed. Recovery is the driver's own re-publish: for a poll driver the watchdog's `DetachSubscription`
|
||||
forces a fresh `PollGroupEngine` subscription whose initial poll runs `forceRaise: true`; for MQTT it is
|
||||
the initial push added by 05/STAB-18. What the host does instead is **audit** it — `CommsRecoveryAudit`
|
||||
fires `DefaultCommsRecoveryAuditDelay` (30 s) after the driver reconnects and reports any ref still
|
||||
latched Bad, as a Warning naming the driver plus the `otopcua.driver.comms_latched_refs` counter (tagged
|
||||
`driver`, `driver_type`). That counter going non-zero means a driver reconnected without re-publishing —
|
||||
a push driver missing its initial push, or a value-only dedup swallowing an unchanged value. It is a
|
||||
tripwire, not a repair: nothing downstream depends on it.
|
||||
|
||||
### The connect path feeds the circuit breaker (R2-10 T10 closure)
|
||||
|
||||
`IDriver.InitializeAsync` is routed through the capability invoker as
|
||||
|
||||
+11
@@ -123,6 +123,17 @@ the full design. Highlights:
|
||||
rejects an A→B→A oscillation loop naming the members. Compile is not hard-gated (VirtualTag
|
||||
parity); a runtime compile/throw/timeout lands as Bad quality + a `script-logs` entry.
|
||||
|
||||
## Quality: a raw tag's node when its driver stops communicating
|
||||
|
||||
A raw tag's node — and every UNS mirror of it — goes **`BadNoCommunication` with its last value
|
||||
retained** when the owning driver is declared comms-lost, and clears when the driver re-publishes. It is
|
||||
never cleared by the server on reconnect: recovery is a value, never an assumption. The rule and the
|
||||
seams that implement it are stated once in
|
||||
[`docs/DriverLifecycle.md` §"The quality contract"](DriverLifecycle.md#the-quality-contract).
|
||||
|
||||
Calc tags authored here inherit the same rule from their inputs — see
|
||||
[`docs/drivers/Calculation.md` §"Input quality"](drivers/Calculation.md#input-quality).
|
||||
|
||||
## Retirement note
|
||||
|
||||
The routed `/clusters/{id}/drivers` driver-authoring flow (`DriverTypePicker`,
|
||||
|
||||
@@ -399,6 +399,29 @@ reference map identically, so their plans stay byte-parity.
|
||||
|
||||
---
|
||||
|
||||
## Input quality: what a script sees, and when it runs at all
|
||||
|
||||
The rule is stated once, in
|
||||
[`docs/DriverLifecycle.md` §"The quality contract"](DriverLifecycle.md#the-quality-contract), and it
|
||||
applies identically to virtual-tag scripts and to `Calculation` driver scripts — they share
|
||||
`VirtualTagContext` and this editor, so one policy keeps this page true for every `ctx.GetTag` call
|
||||
site.
|
||||
|
||||
- **`ctx.GetTag(path).StatusCode` and `.SourceTimestampUtc` are real.** The read cache used to stamp
|
||||
every dependency `Good` with an evaluation-time timestamp, so a quality-aware script could not be
|
||||
written at all — it was told Good even for an input whose driver had stopped communicating.
|
||||
- **A script is NOT run while any of its declared inputs is Bad.** The tag instead reports that
|
||||
input's own status (`BadNoCommunication`, `BadWaitingForInitialData`, …) carrying its last known
|
||||
value. So the `(double)ctx.GetTag("x").Value` idiom is safe against a dead input: the script simply
|
||||
does not run, rather than throwing on every trigger and filling `/script-log`.
|
||||
- **An Uncertain input IS passed to the script**, and the output inherits that Uncertain sub-code. An
|
||||
Uncertain value is present and a script can make a qualified decision from it; that is the alarm
|
||||
engine's long-standing rule, now shared.
|
||||
- **The output never reports better than its worst input.** A Good result over an Uncertain input is
|
||||
published Uncertain; nothing is published Good while an input is Bad.
|
||||
- ⚠️ **If you wrote a script that deliberately handled `null` for a dead input, it no longer runs
|
||||
while that input is Bad.** Assume your inputs are usable — the runtime has already decided they are.
|
||||
|
||||
## Known Limitations / Follow-ups
|
||||
|
||||
- **Tag-path completion shows resolvable keys only.** Surfacing the UNS browse
|
||||
|
||||
+11
@@ -476,6 +476,17 @@ not — clients pick those up on their next browse. And a **degraded** non-rebui
|
||||
(some node failed to materialise) deliberately does not record its composition as
|
||||
applied, so the next deploy re-diffs and repairs it.
|
||||
|
||||
## Quality on a UNS node
|
||||
|
||||
A UNS reference variable mirrors its backing raw tag exactly — value, status and timestamps — so when
|
||||
that tag's driver stops communicating the mirror goes **`BadNoCommunication` with its last value
|
||||
retained**, in the same fan-out message as the raw node, and clears only when the driver re-publishes.
|
||||
A VirtualTag on the same equipment follows the same rule from its own inputs: it holds while any input
|
||||
is Bad and reports that input's sub-code. The contract, and every seam that implements it, is stated
|
||||
once in [`docs/DriverLifecycle.md` §"The quality contract"](DriverLifecycle.md#the-quality-contract);
|
||||
what a script sees is in
|
||||
[`docs/ScriptEditor.md`](ScriptEditor.md#input-quality-what-a-script-sees-and-when-it-runs-at-all).
|
||||
|
||||
## See also
|
||||
|
||||
- [Configuration.md](Configuration.md) — the underlying config entities.
|
||||
|
||||
@@ -81,10 +81,43 @@ Two deploy-time gates run over the result:
|
||||
- **Nothing publishes until every declared dependency has arrived at least once.** Before that a read
|
||||
returns `BadWaitingForInitialData`. This is the same gate `VirtualTagActor` uses, and it exists so a
|
||||
calculation cannot publish a confident-looking value derived from defaults.
|
||||
- **Equal results are deduped** — an evaluation producing the same value as last time does not
|
||||
re-publish.
|
||||
- **Equal results are deduped on (value, *status*)** — an evaluation producing the same value *and*
|
||||
the same status as last time does not re-publish. The status half is load-bearing: a device that
|
||||
comes back after an outage usually comes back holding the reading it had before, so a value-only
|
||||
dedup would swallow the recovery and leave the node Bad forever.
|
||||
- **Timers are grouped by interval**, one timer per distinct interval, not one per tag.
|
||||
|
||||
## Input quality
|
||||
|
||||
The contract is stated once, in
|
||||
[`docs/DriverLifecycle.md` §"The quality contract"](../DriverLifecycle.md#the-quality-contract): a
|
||||
value consumer never reports a quality better than its source's. A calc tag's inputs are other tags'
|
||||
live values, so its output inherits their quality:
|
||||
|
||||
| Worst input status | What the driver does | What the calc tag publishes |
|
||||
|---|---|---|
|
||||
| Good | evaluates | the computed value, `Good` |
|
||||
| Uncertain (any sub-code) | **evaluates** — an Uncertain value is present and usable | the computed value, stamped with that Uncertain sub-code |
|
||||
| Bad (any sub-code) | **holds** — the script does not run | the **last known value**, stamped with that input's Bad sub-code, once per change of status |
|
||||
| a dependency never pushed | holds (the pre-existing warm-up gate) | nothing — the node stays `BadWaitingForInitialData` |
|
||||
|
||||
Notes worth knowing before authoring against this:
|
||||
|
||||
- **The sub-code propagates**, so the output says *why*: `BadNoCommunication` means go look at the
|
||||
device feeding the input; `BadInternalError` means this tag's own script failed.
|
||||
- **Holding on Bad is what removes the NRE storm.** A comms-lost input used to arrive as *Good, null*,
|
||||
so `(double)ctx.GetTag("x").Value` threw on every trigger and filled `/script-log` with identical
|
||||
"script threw" entries. Now the script simply is not run, and **no script-log entry is written** — an
|
||||
input dying is not a script fault.
|
||||
- ⚠️ **A script that relied on receiving `null` for a dead input no longer runs at all** while that
|
||||
input is Bad. Rewrite such a script to assume its inputs are usable; the driver has already decided
|
||||
they are.
|
||||
- **`ctx.GetTag(path).StatusCode` and `.SourceTimestampUtc` are real.** The read cache used to stamp
|
||||
every input `Good` at evaluation time; it now passes the input's own snapshot through. Since Bad
|
||||
inputs never reach a script, what a script can observe is Good or Uncertain.
|
||||
- **`ReadAsync` reports the same status the subscription carried**, so a polled read and a subscribed
|
||||
client never disagree about the same node.
|
||||
|
||||
## Error semantics
|
||||
|
||||
An evaluator failure publishes **Bad quality carrying the last-known value** — the value is not
|
||||
|
||||
@@ -415,6 +415,41 @@ one session, and carrying it across made the host's re-subscribe resolve an empt
|
||||
SUBSCRIBE at all, and leave the driver Connected/Healthy and permanently deaf. Sparkplug's
|
||||
`SparkplugIngestor.AttachTo` resets its own session state for the same reason.
|
||||
|
||||
## Quality on connection loss and re-subscribe
|
||||
|
||||
The quality contract this implements is stated once, in
|
||||
[`docs/DriverLifecycle.md` §"The quality contract"](../DriverLifecycle.md#the-quality-contract).
|
||||
Two edge behaviours make MQTT — a push-model driver — honour it. A poll driver gets both free from
|
||||
`PollGroupEngine` (a fresh subscription's first poll runs with `forceRaise: true`); MQTT had neither,
|
||||
which is archreview 05/STAB-18.
|
||||
|
||||
- **Connection loss stales, keeping the value.** On the `Connected → Reconnecting` edge
|
||||
`MqttConnection.ConnectionLost` fires, and both ingest paths mark every **already-observed**
|
||||
reference `BadNoCommunication` with its **last value retained** and publish the change. A
|
||||
never-observed reference (still `BadWaitingForInitialData`) and an already-Bad one are left alone.
|
||||
The event does **not** fire for a failed *initial* connect or during dispose — neither is an outage.
|
||||
- **Every (re)subscribe pushes the cached snapshots.** `SubscribeAsync` republishes each observed
|
||||
reference's cached snapshot onto the new handle before any broker traffic. This is what refills a
|
||||
node an address-space rebuild reseeded `BadWaitingForInitialData` on a quiet topic — a rebuild
|
||||
triggers no retained replay and, in Sparkplug, no rebirth.
|
||||
|
||||
**Why the stale is what makes recovery work.** The plain path suppresses a message whose value *and*
|
||||
status equal the driver-owned cache, and that cache survives reconnects and redeploys. So the retained
|
||||
message a broker replays on reconnect — routinely the identical payload — was suppressed, and every
|
||||
node the host had latched Bad stayed Bad until the plant value happened to change. Staling the status
|
||||
first makes the identical replay a change. (A regression test for this must replay the **identical**
|
||||
payload; a different value defeats the suppression on its own and proves nothing.)
|
||||
|
||||
**Two "not communicating" codes, deliberately.** `BadNoCommunication` (0x80310000) means *the link is
|
||||
down* — the connection-loss stale, and what the host's comms-loss fan-out writes onto the value nodes.
|
||||
`BadCommunicationError` (0x80050000) keeps its narrower Sparkplug meaning: the **source announced its
|
||||
own death** (NDEATH/DDEATH), or a birth voided the scope feeding the tag. A refused SUBSCRIBE filter
|
||||
also degrades to `BadCommunicationError`.
|
||||
|
||||
**Operator-visible change:** an MQTT tag now goes Bad when the broker session drops, rather than
|
||||
holding its last value at Good until the host's 90 s comms-loss watchdog notices — which, since
|
||||
MQTTnet usually reconnects well inside that window, in practice meant never.
|
||||
|
||||
## Write-through
|
||||
|
||||
`IWritable` landed with [#508](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/508). Before it,
|
||||
|
||||
Reference in New Issue
Block a user