dohertj2/lmxopcua
1
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases Wiki Activity

Drivers report Healthy with 0 errors while silently ignoring tags they skipped at init #541

New Issue
Closed
opened 2026-08-05 17:32:44 -04:00 by dohertj2 · 1 comment
dohertj2 commented 2026-08-05 17:32:44 -04:00
Owner
Copy Link
Copy Source

Split out of #540, whose addressing half is fixed. This is the half that made that defect invisible rather than merely wrong, and it is not a FOCAS problem.

The behaviour

Every driver builds its RawPath -> definition table at InitializeAsync and tolerates a per-tag miss: an unmappable TagConfig, an unparseable address, a capability-matrix rejection. The tag is logged and skipped, never enters the poll table, and every read of it returns BadNodeIdUnknown.

Tolerance is the right call — one bad tag must not fail a whole driver. The problem is what an operator can see afterwards: nothing. Skips happen at init and are not errors, so:

  • driver state stays Healthy
  • errors/5 min stays 0
  • Last error stays —
  • Last read advances live, because the driver's other tags are fine

On the Makino in #540, focas-z-34184 showed exactly that while ignoring 20 of its 22 tags. There is no surface anywhere in the AdminUI that would have said otherwise.

It is fleet-wide

Eight drivers skip-and-log the same way:

Driver Site
FOCAS FocasDriver.cs:183, :206, :213
Modbus ModbusDriver.cs:228
S7 S7Driver.cs:194
AbCip AbCipDriver.cs:292
AbLegacy AbLegacyDriver.cs:139
TwinCAT TwinCATDriver.cs:140
Sql SqlDriver.cs:710
Mqtt MqttDriver.cs (same shape)

So a per-driver fix would be the wrong shape — it is one missing signal, eight times.

Why the #540 guardrails do not close it

#540 added a real address parse to FocasTagConfigModel.Validate() (authoring) and to FocasTagDefinitionFactory.Inspect() (deploy gate). Those stop the specific cause found there, and only for FOCAS. They do not cover:

  • the other seven drivers' equivalent authoring gates
  • rejection reasons that are not "the address does not parse" — a capability-matrix reject, a strict-enum dataType reject, a TagConfig that maps to nothing
  • anything that changes after the deploy gate ran (a device swapped to a series whose capability matrix rejects an address that was previously fine)

The gate is upstream and per-cause. What is missing is a downstream statement of fact: this driver is not polling N of its M tags.

Suggested direction

Carry the count on the health snapshot and surface it on /hosts, the same way #521 resolved per-host connectivity:

  • a SkippedTags field on DriverHealthChanged (a count plus a bounded list of RawPath + reason), populated from what each driver already logs
  • a column or chip on /hosts, so "Healthy, 0 errors" cannot coexist with "ignoring most of my tags"
  • reuse the Phase 5 telemetry stream — do NOT add a table; a driver-only node has no ConfigDb to write to (the lesson from #521, where DriverHostStatus was dropped for exactly that reason)

Deliberately not Degraded. Latching a driver non-Healthy for a config problem would interact badly with #527: the comms-loss detector trips on 90 s without a successful read AND non-Healthy, so a driver whose tags were ALL skipped would have no successful reads, latch non-Healthy, and reconnect-loop against a perfectly healthy device forever. The signal has to be additive to the state, not a change of it.

An IDriver seam (GetSkippedTags()) would make this uniform across the eight, rather than eight separate counters.

Split out of [#540](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/540), whose addressing half is fixed. This is the half that made that defect *invisible* rather than merely wrong, and it is not a FOCAS problem. ## The behaviour Every driver builds its `RawPath -> definition` table at `InitializeAsync` and **tolerates a per-tag miss**: an unmappable `TagConfig`, an unparseable address, a capability-matrix rejection. The tag is logged and skipped, never enters the poll table, and every read of it returns `BadNodeIdUnknown`. Tolerance is the right call — one bad tag must not fail a whole driver. The problem is what an operator can see afterwards: **nothing**. Skips happen at init and are not errors, so: - driver state stays `Healthy` - `errors/5 min` stays `0` - `Last error` stays `—` - `Last read` advances live, because the driver's *other* tags are fine On the Makino in #540, `focas-z-34184` showed exactly that while ignoring 20 of its 22 tags. There is no surface anywhere in the AdminUI that would have said otherwise. ## It is fleet-wide Eight drivers skip-and-log the same way: | Driver | Site | |---|---| | FOCAS | `FocasDriver.cs:183`, `:206`, `:213` | | Modbus | `ModbusDriver.cs:228` | | S7 | `S7Driver.cs:194` | | AbCip | `AbCipDriver.cs:292` | | AbLegacy | `AbLegacyDriver.cs:139` | | TwinCAT | `TwinCATDriver.cs:140` | | Sql | `SqlDriver.cs:710` | | Mqtt | `MqttDriver.cs` (same shape) | So a per-driver fix would be the wrong shape — it is one missing signal, eight times. ## Why the #540 guardrails do not close it #540 added a real address parse to `FocasTagConfigModel.Validate()` (authoring) and to `FocasTagDefinitionFactory.Inspect()` (deploy gate). Those stop the *specific* cause found there, and only for FOCAS. They do not cover: - the other seven drivers' equivalent authoring gates - rejection reasons that are not "the address does not parse" — a capability-matrix reject, a strict-enum `dataType` reject, a `TagConfig` that maps to nothing - anything that changes *after* the deploy gate ran (a device swapped to a series whose capability matrix rejects an address that was previously fine) The gate is upstream and per-cause. What is missing is a downstream statement of fact: *this driver is not polling N of its M tags.* ## Suggested direction Carry the count on the health snapshot and surface it on `/hosts`, the same way [#521](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/521) resolved per-host connectivity: - a `SkippedTags` field on `DriverHealthChanged` (a count plus a bounded list of `RawPath` + reason), populated from what each driver already logs - a column or chip on `/hosts`, so "Healthy, 0 errors" cannot coexist with "ignoring most of my tags" - reuse the Phase 5 telemetry stream — do NOT add a table; a driver-only node has no ConfigDb to write to (the lesson from #521, where `DriverHostStatus` was dropped for exactly that reason) **Deliberately not `Degraded`.** Latching a driver non-Healthy for a config problem would interact badly with #527: the comms-loss detector trips on *90 s without a successful read AND non-Healthy*, so a driver whose tags were ALL skipped would have no successful reads, latch non-Healthy, and reconnect-loop against a perfectly healthy device forever. The signal has to be additive to the state, not a change of it. An `IDriver` seam (`GetSkippedTags()`) would make this uniform across the eight, rather than eight separate counters.
dohertj2 commented 2026-08-06 02:20:10 -04:00
Author
Owner
Copy Link
Copy Source

Fixed — merged to master and pushed, merge 355e59aa (branch fix/541-skipped-tag-visibility, 4 commits).

What shipped

A new optional capability ISkippedTagReporter over a shared SkippedTagLedger, carrying the authored count, the exact skipped count and a capped sample of RawPath + reason on DriverHealthChanged → a Tags column on /hosts. Transport is the Phase 5 telemetry stream, not a table — the #521 lesson, since a driver-only node has no ConfigDb.

Reading the column:

Cell Meaning
— no answer — the driver cannot report, or it faulted before building a tag table
12 tags it built a table and skipped nothing
20 / 22 skipped (amber) partly working
22 / 22 skipped (red) polling nothing — the state most likely to look healthy, having no failing reads to report

The suggested IDriver.GetSkippedTags() became an optional capability instead, matching the existing IHostConnectivityProbe / IRediscoverable idiom, so "cannot report" stays distinguishable from "reports none". The ledger is shared precisely so the reset/commit/cap rules are one implementation rather than twelve — the duplication this issue was filed about.

Two corrections to the issue

It is twelve drivers, not eight. Every driver in DriverTypeNames.All has an authored-tag skip path. Unlisted here: Calculation (no scriptId), MTConnect (no readable DataItem id), Galaxy (no attributeRef), OpcUaClient (no nodeId). OpcUaClient's was the quietest of the twelve — NamespaceMap drops such an entry with a bare continue and not even a log line. A hand-maintained list would have shipped those four still invisible, so SkippedTagReporterFleetTests discovers IDriver implementations reflectively and a thirteenth driver cannot re-introduce the silence.

Sql drops tags at two points, and the report is the union. The blob that does not map (BuildTagTable) and an identifier the §8.1 catalog gate rejects against the live catalog (ApplyCatalogGateAsync). The second is what schema drift produces — a renamed column, a dropped view — months after a deployment sealed green. Reporting only the first would have that driver claim 0 skipped while polling half its tags, which is worse than reporting nothing, because the column would then be actively reassuring.

Design points that are load-bearing

  • Additive to state, never Degraded — exactly as this issue argued. Confirmed against #527: a driver whose tags were all skipped has no successful reads by construction, so it would satisfy both halves of the conjunction and reconnect-loop forever against a healthy device.
  • The skip digest is part of PublishHealthSnapshot's fingerprint, and covers the sample, not just the counts. A skip leaves every other fingerprint component identical to a clean driver's, so without it the dedup swallows precisely the publish carrying the signal — the trap that already bit the rediscovery and host-status channels. Counts alone would also miss a redeploy that fixes one tag and breaks another.
  • The count is exact; only the sample is capped, and the tooltip says so when it is showing fewer than it counted. A capped list that looked complete would have an operator fix twenty tags and believe the job done.
  • The proto field takes no companion presence flag, unlike host_statuses beside it: a singular message field has real presence in proto3, and a has_ flag would collide with protoc's generated HasSkippedTags.

Live gate — docker-dev, two rounds

Record: docs/plans/2026-08-05-skipped-tag-visibility-live-gate.md.

Round one broke 3 of 13 tags on a Modbus driver: 3 / 13 SKIPPED in amber, driver HEALTHY with 0 errors and an advancing last read — the exact combination this issue describes. It also found a defect the unit tests could not: two drivers that faulted before building a tag table rendered 0 tags against two authored tags apiece, because a never-committed ledger returned a zero report. A zero report is a positive claim — "this driver skipped nothing" — and making it on behalf of a driver that never looked is the same quiet inaccuracy this issue is about. GetSkippedTags() now returns null until the first committed build. Every unit test had driven a driver through a completed build first, which is why none caught it.

Round two closed the two remaining gaps with one setup: 25 unmappable tags on a SITE-A driver gave 25 / 26 SKIPPED with the sample capped at exactly 20 and the tooltip reading (+5 more not shown); trimming to exactly 20 gave 20 / 21 with no note, pinning the off-by-one. SITE-A also settles the central proto path — Phase 6 gave it its own Akka mesh, DPS does not federate, and the lone ClusterClient reference is a point-to-point command path rather than gossip, so TelemetryProtoMapNode → gRPC → TelemetryProtoMapCentral is the only route to the page. Central's view tracking a change made solely on the site node is the positive control.

Unplanned finding: three of the four drivers showing skips on the rig were not planted. sql-oracle-fixture, sql-postgres-fixture and gate-modbus-renamed had been silently ignoring their tags for an unknown length of time, each reporting Healthy with zero errors. The column surfaced them on its first run.

Verification

Red-first three ways: dropping SkippedTagDigest from the fingerprint fails the two dedup tests; removing the seam from one driver fails the fleet guard; dropping the catalog-gate leg fails the Sql union test. Full solution builds with zero new warnings against 4e22f024 (diffed a full -t:Rebuild census — an incremental build under-reports on this tree). All twelve driver suites plus Runtime / ControlPlane / AdminUI / Core / Commons pass.

Not consumer-facing to ScadaBridge — AdminUI observability, no OPC UA wire-contract change. A skipped tag's node still materializes and still reads BadNodeIdUnknown exactly as before; what changed is that an operator can now see it.

Fixed — merged to `master` and pushed, merge `355e59aa` (branch `fix/541-skipped-tag-visibility`, 4 commits). ## What shipped A new optional capability `ISkippedTagReporter` over a shared `SkippedTagLedger`, carrying the **authored count**, the **exact skipped count** and a **capped sample** of `RawPath` + reason on `DriverHealthChanged` → a **Tags column on `/hosts`**. Transport is the Phase 5 telemetry stream, **not a table** — the #521 lesson, since a driver-only node has no ConfigDb. Reading the column: | Cell | Meaning | |---|---| | `—` | **no answer** — the driver cannot report, *or* it faulted before building a tag table | | `12 tags` | it built a table and skipped nothing | | `20 / 22 skipped` (amber) | partly working | | `22 / 22 skipped` (red) | polling **nothing** — the state most likely to look healthy, having no failing reads to report | The suggested `IDriver.GetSkippedTags()` became an *optional* capability instead, matching the existing `IHostConnectivityProbe` / `IRediscoverable` idiom, so "cannot report" stays distinguishable from "reports none". The ledger is shared precisely so the reset/commit/cap rules are one implementation rather than twelve — the duplication this issue was filed about. ## Two corrections to the issue **It is twelve drivers, not eight.** Every driver in `DriverTypeNames.All` has an authored-tag skip path. Unlisted here: **Calculation** (no `scriptId`), **MTConnect** (no readable DataItem id), **Galaxy** (no `attributeRef`), **OpcUaClient** (no `nodeId`). OpcUaClient's was the quietest of the twelve — `NamespaceMap` drops such an entry with a bare `continue` and **not even a log line**. A hand-maintained list would have shipped those four still invisible, so `SkippedTagReporterFleetTests` discovers `IDriver` implementations reflectively and a thirteenth driver cannot re-introduce the silence. **Sql drops tags at two points, and the report is the union.** The blob that does not map (`BuildTagTable`) *and* an identifier the §8.1 catalog gate rejects against the live catalog (`ApplyCatalogGateAsync`). The second is what schema drift produces — a renamed column, a dropped view — months after a deployment sealed green. Reporting only the first would have that driver claim `0 skipped` while polling half its tags, which is worse than reporting nothing, because the column would then be actively reassuring. ## Design points that are load-bearing - **Additive to state, never `Degraded`** — exactly as this issue argued. Confirmed against #527: a driver whose tags were *all* skipped has no successful reads by construction, so it would satisfy both halves of the conjunction and reconnect-loop forever against a healthy device. - **The skip digest is part of `PublishHealthSnapshot`'s fingerprint, and covers the sample, not just the counts.** A skip leaves every other fingerprint component identical to a clean driver's, so without it the dedup swallows precisely the publish carrying the signal — the trap that already bit the rediscovery and host-status channels. Counts alone would also miss a redeploy that fixes one tag and breaks another. - **The count is exact; only the sample is capped,** and the tooltip says so when it is showing fewer than it counted. A capped list that looked complete would have an operator fix twenty tags and believe the job done. - **The proto field takes no companion presence flag,** unlike `host_statuses` beside it: a singular message field has real presence in proto3, and a `has_` flag would collide with protoc's generated `HasSkippedTags`. ## Live gate — docker-dev, two rounds Record: `docs/plans/2026-08-05-skipped-tag-visibility-live-gate.md`. **Round one** broke 3 of 13 tags on a Modbus driver: `3 / 13 SKIPPED` in amber, driver HEALTHY with 0 errors and an advancing last read — the exact combination this issue describes. It also **found a defect the unit tests could not**: two drivers that faulted *before* building a tag table rendered `0 tags` against two authored tags apiece, because a never-committed ledger returned a zero report. A zero report is a positive claim — "this driver skipped nothing" — and making it on behalf of a driver that never looked is the same quiet inaccuracy this issue is about. `GetSkippedTags()` now returns null until the first committed build. Every unit test had driven a driver through a completed build first, which is why none caught it. **Round two** closed the two remaining gaps with one setup: 25 unmappable tags on a **SITE-A** driver gave `25 / 26 SKIPPED` with the sample capped at exactly 20 and the tooltip reading `(+5 more not shown)`; trimming to exactly 20 gave `20 / 21` with **no** note, pinning the off-by-one. SITE-A also settles the central proto path — Phase 6 gave it its own Akka mesh, DPS does not federate, and the lone ClusterClient reference is a point-to-point command path rather than gossip, so `TelemetryProtoMapNode` → gRPC → `TelemetryProtoMapCentral` is the only route to the page. Central's view tracking a change made solely on the site node is the positive control. **Unplanned finding:** three of the four drivers showing skips on the rig were **not planted**. `sql-oracle-fixture`, `sql-postgres-fixture` and `gate-modbus-renamed` had been silently ignoring their tags for an unknown length of time, each reporting Healthy with zero errors. The column surfaced them on its first run. ## Verification Red-first three ways: dropping `SkippedTagDigest` from the fingerprint fails the two dedup tests; removing the seam from one driver fails the fleet guard; dropping the catalog-gate leg fails the Sql union test. Full solution builds with **zero new warnings** against `4e22f024` (diffed a full `-t:Rebuild` census — an incremental build under-reports on this tree). All twelve driver suites plus Runtime / ControlPlane / AdminUI / Core / Commons pass. **Not consumer-facing to ScadaBridge** — AdminUI observability, no OPC UA wire-contract change. A skipped tag's node still materializes and still reads `BadNodeIdUnknown` exactly as before; what changed is that an operator can now see it.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
auto-managed
queue management
 cross-driver
queue management
 driver/abcip
queue management
 driver/ablegacy
queue management
 driver/focas
queue management
 driver/mqtt
MQTT / Sparkplug B broker-subscribe driver
 driver/opcuaclient
queue management
 driver/s7
queue management
 driver/twincat
queue management
 phase/1
queue management
 phase/2
queue management
 phase/3
queue management
 phase/4
queue management
 phase/5
queue management
 phase/6
queue management
 queue/blocked
queue management
 queue/done
queue management
 queue/failed
queue management
 queue/in-progress
queue management
 queue/queued
queue management
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
dohertj2 (dohertj2)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dohertj2/lmxopcua#541
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?