dohertj2/lmxopcua
1
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(mqtt): Sparkplug DataSet / Template / PropertySet metric support #515

New Issue
Closed
opened 2026-07-27 13:51:16 -04:00 by dohertj2 · 2 comments
dohertj2 commented 2026-07-27 13:51:16 -04:00
Owner
Copy Link
Copy Source

SparkplugCodec handles scalar metrics. DataSet, Template (UDT instances) and PropertySet metrics are
skipped with a warning — v1-unsupported by explicit decision, recorded in the plan's Deferred / out of
scope section and design §3.5. This issue tracks closing that.

The current behaviour is the right default and should survive any fix: an unsupported metric is skipped and
warned about, never coerced into a guessed type. A Template flattened into a String, or a DataSet whose
first column is silently taken as "the value", would bind a tag to data that is not what its name says — worse
than not binding it. Whatever shape support takes, do not lose the skip-and-warn fallback for the cases it
still does not cover.

Design questions to settle first, since these are structured values and OPC UA has more than one honest
mapping:

  • Template — map to an OPC UA Structure/UDT, or flatten members into sibling tags (Motor/Speed,
    Motor/Torque)? Flattening interacts with #509: member paths contain /.
  • DataSet — a table has no single scalar value. Array-valued variable, or per-column tags?
  • PropertySet — arguably not a value at all; likely belongs on the node as OPC UA properties/metadata
    rather than as tags.

Bytes/File metrics beyond a base64/raw-String fallback are the same class of gap and can ride along or split
out.

Acceptance. A simulator publishing a Template and a DataSet metric produces addressable, correctly-typed
OPC UA nodes, live-verified; every still-unsupported kind continues to skip with a warning.

`SparkplugCodec` handles scalar metrics. `DataSet`, `Template` (UDT instances) and `PropertySet` metrics are **skipped with a warning** — v1-unsupported by explicit decision, recorded in the plan's *Deferred / out of scope* section and design §3.5. This issue tracks closing that. **The current behaviour is the right default and should survive any fix:** an unsupported metric is skipped and warned about, **never coerced into a guessed type**. A `Template` flattened into a String, or a `DataSet` whose first column is silently taken as "the value", would bind a tag to data that is not what its name says — worse than not binding it. Whatever shape support takes, do not lose the skip-and-warn fallback for the cases it still does not cover. **Design questions to settle first,** since these are structured values and OPC UA has more than one honest mapping: - **`Template`** — map to an OPC UA Structure/UDT, or flatten members into sibling tags (`Motor/Speed`, `Motor/Torque`)? Flattening interacts with #509: member paths contain `/`. - **`DataSet`** — a table has no single scalar value. Array-valued variable, or per-column tags? - **`PropertySet`** — arguably not a value at all; likely belongs on the node as OPC UA properties/metadata rather than as tags. `Bytes`/`File` metrics beyond a base64/raw-String fallback are the same class of gap and can ride along or split out. **Acceptance.** A simulator publishing a `Template` and a `DataSet` metric produces addressable, correctly-typed OPC UA nodes, live-verified; every still-unsupported kind continues to skip with a warning.
dohertj2 added the driver/mqtt label 2026-07-27 13:51:16 -04:00
dohertj2 commented 2026-07-31 06:46:01 -04:00
Author
Owner
Copy Link
Copy Source

Decision (owner, 2026-07-31): flatten to sibling tags. Not yet implemented — recording the settled
design so the mapping question is closed.

Metric kind Mapping
Template (UDT instance) members become sibling tags — Motor/Speed, Motor/Torque
DataSet per-column tags, not one array-valued variable
PropertySet OPC UA properties on the node, not tags — it is metadata, not a value

Chosen over the structured-OPC-UA-types alternative because flattened scalars are consumable by any OPC
UA client (ScadaBridge included), and the result mirrors how the /raw tree already works. Structures
would be truer to Sparkplug's model but much harder to consume, and the address space would stop
mirroring raw.

Two things that must survive the change:

  • Skip-and-warn stays for every kind still unsupported. An unsupported metric is never coerced into a
    guessed type — a Template flattened into a String would bind a tag to data that is not what its name
    says.
  • Member paths contain /, which is exactly the interaction this issue flagged with #509. That is now
    handled: #509 shipped RawBrowseCommitMapper.ToRawSegment, which coerces the tag Name to a legal
    RawPath segment while carrying the original slashed string through as metricName, plus a collision
    guard that refuses two members that would sanitise onto one Name. So the flattening work inherits a
    solved problem rather than reopening it.

Live gate will need the project-owned C# edge-node simulator
(tests/.../SparkplugSimulator) extended to publish a Template and a DataSet metric — it currently
emits scalars only.

**Decision (owner, 2026-07-31): flatten to sibling tags.** Not yet implemented — recording the settled design so the mapping question is closed. | Metric kind | Mapping | |---|---| | `Template` (UDT instance) | members become **sibling tags** — `Motor/Speed`, `Motor/Torque` | | `DataSet` | **per-column tags**, not one array-valued variable | | `PropertySet` | OPC UA **properties on the node**, not tags — it is metadata, not a value | Chosen over the structured-OPC-UA-types alternative because flattened scalars are consumable by any OPC UA client (ScadaBridge included), and the result mirrors how the `/raw` tree already works. Structures would be truer to Sparkplug's model but much harder to consume, and the address space would stop mirroring raw. **Two things that must survive the change:** - **Skip-and-warn stays** for every kind still unsupported. An unsupported metric is never coerced into a guessed type — a `Template` flattened into a String would bind a tag to data that is not what its name says. - **Member paths contain `/`**, which is exactly the interaction this issue flagged with #509. That is now handled: #509 shipped `RawBrowseCommitMapper.ToRawSegment`, which coerces the tag **Name** to a legal RawPath segment while carrying the original slashed string through as `metricName`, plus a collision guard that refuses two members that would sanitise onto one Name. So the flattening work inherits a solved problem rather than reopening it. Live gate will need the project-owned C# edge-node simulator (`tests/.../SparkplugSimulator`) extended to publish a `Template` and a `DataSet` metric — it currently emits scalars only.
dohertj2 commented 2026-07-31 08:15:03 -04:00
Author
Owner
Copy Link
Copy Source

Implemented and live-gated. master 857a10b4 (feature commit c74ccc55).

Flattened to sibling tags, as decided:

Metric kind Mapping Example
Template members become sibling metrics, /-joined, recursively Motor → Motor/Speed, Motor/Stator/Temp
DataSet one metric per cell, {column}/{rowIndex} Recipe → Recipe/Step/0, Recipe/Target/1
PropertySet node metadata, never tags engUnit, Range/Low

One place the decision needed sharpening

"Per-column tags" cannot survive N > 1 rows — a column of N rows is not a scalar — so it is per
cell, and the row index is always present, including on row 0. Emitting a bare {column} for
a single-row table and {column}/{row} otherwise would make a tag's own name depend on how many rows
the plant happened to publish, so a table growing from one row to two would rename every tag already
authored against it. Recipe/Target/0 costs a /0 on the common single-row "current values" shape
and buys a stable address.

The case a naive implementation gets wrong

A DATA metric carries the parent's alias and no name. So member names cannot be composed until
the alias table has resolved the parent — flattening at decode time alone is not enough. Expansion
happens in SparkplugIngestor.PublishStructuredMembers, after resolution, at the single funnel both
the DATA and birth-value paths already pass through (so neither can end up supporting structured
metrics while the other silently does not). The alias table installs a binding per member, which is
also what lets a DDATA member that omits its datatype fall back to the birth's — guessing from the
CLR type of the wire value would read a signed Int32 of −42 out of its unsigned field as
4294967254.

Skip-and-warn survives, and refusal is whole

A template definition (is_definition — shape, no values), an unnamed member, an extension
value, a column/type count mismatch, a short row, or a payload past MaxDepth (5) / MaxMembers
(512) refuses the entire metric rather than publishing part of it. A silently half-flattened UDT
looks exactly like a complete one in the picker, and the operator would author tags against a subtree
that quietly stops partway down.

Two consequences worth recording

  • The browse picker offers the members and deliberately not the parent. The parent has no value
    and no servable datatype, so a leaf for it would offer an address that can only ever commit a tag
    which never reads.
  • The rediscovery signature tracks members, not the parent. A UDT gaining or losing a member
    leaves the parent metric's name identical, so a gate built on raw wire names would never move — and
    for a UDT that is the change that matters most.

Member paths contain /, which #509 had already solved, exactly as this issue anticipated: the
slashed name travels intact as metricName while the derived tag Name is coerced to a legal
RawPath segment, with the collision guard behind it.

Live gate — 3/3

The project-owned simulator was extended to publish a nested Template (Motor with Speed,
Running and a nested Stator/Temp) and a 2×2 DataSet, both moving on the data tick so a fed
member is distinguishable from one published once at birth and then quiet. Legs: template across
birth and alias-only DDATA; dataset cells typed per column; and a template definition still
skipped — that last one with a positive control that must arrive first, or "Speed never arrived"
would be satisfied by any failure to connect at all.

Falsified at three points — dropping the member bindings, not expanding the parent, and refusing to
flatten templates — each turning exactly the expected tests red.

Docs: docs/drivers/Mqtt.md §"Structured metrics".

**Implemented and live-gated.** `master` `857a10b4` (feature commit `c74ccc55`). Flattened to sibling tags, as decided: | Metric kind | Mapping | Example | |---|---|---| | `Template` | members become sibling metrics, `/`-joined, **recursively** | `Motor` → `Motor/Speed`, `Motor/Stator/Temp` | | `DataSet` | one metric per **cell**, `{column}/{rowIndex}` | `Recipe` → `Recipe/Step/0`, `Recipe/Target/1` | | `PropertySet` | node **metadata**, never tags | `engUnit`, `Range/Low` | ### One place the decision needed sharpening "Per-column tags" cannot survive N > 1 rows — a column of N rows is not a scalar — so it is per **cell**, and **the row index is always present, including on row 0**. Emitting a bare `{column}` for a single-row table and `{column}/{row}` otherwise would make a tag's own name depend on how many rows the plant happened to publish, so a table growing from one row to two would rename every tag already authored against it. `Recipe/Target/0` costs a `/0` on the common single-row "current values" shape and buys a stable address. ### The case a naive implementation gets wrong **A DATA metric carries the parent's alias and no name.** So member names cannot be composed until the alias table has resolved the parent — flattening at decode time alone is not enough. Expansion happens in `SparkplugIngestor.PublishStructuredMembers`, after resolution, at the single funnel both the DATA and birth-value paths already pass through (so neither can end up supporting structured metrics while the other silently does not). The alias table installs a binding per member, which is also what lets a DDATA member that omits its datatype fall back to the birth's — guessing from the CLR type of the wire value would read a signed `Int32` of −42 out of its unsigned field as 4294967254. ### Skip-and-warn survives, and refusal is whole A template **definition** (`is_definition` — shape, no values), an unnamed member, an extension value, a column/type count mismatch, a short row, or a payload past `MaxDepth` (5) / `MaxMembers` (512) refuses the **entire** metric rather than publishing part of it. A silently half-flattened UDT looks exactly like a complete one in the picker, and the operator would author tags against a subtree that quietly stops partway down. ### Two consequences worth recording - **The browse picker offers the members and deliberately not the parent.** The parent has no value and no servable datatype, so a leaf for it would offer an address that can only ever commit a tag which never reads. - **The rediscovery signature tracks members, not the parent.** A UDT gaining or losing a member leaves the parent metric's name identical, so a gate built on raw wire names would never move — and for a UDT that is the change that matters most. Member paths contain `/`, which #509 had already solved, exactly as this issue anticipated: the slashed name travels intact as `metricName` while the derived tag **Name** is coerced to a legal RawPath segment, with the collision guard behind it. ### Live gate — 3/3 The project-owned simulator was extended to publish a **nested** Template (`Motor` with `Speed`, `Running` and a nested `Stator/Temp`) and a 2×2 `DataSet`, both moving on the data tick so a fed member is distinguishable from one published once at birth and then quiet. Legs: template across birth **and** alias-only DDATA; dataset cells typed per column; and a template definition still skipped — that last one with a positive control that must arrive first, or "Speed never arrived" would be satisfied by any failure to connect at all. Falsified at three points — dropping the member bindings, not expanding the parent, and refusing to flatten templates — each turning exactly the expected tests red. Docs: `docs/drivers/Mqtt.md` §"Structured metrics".
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
auto-managed
queue management
 cross-driver
queue management
 driver/abcip
queue management
 driver/ablegacy
queue management
 driver/focas
queue management
 driver/mqtt
MQTT / Sparkplug B broker-subscribe driver
 driver/opcuaclient
queue management
 driver/s7
queue management
 driver/twincat
queue management
 phase/1
queue management
 phase/2
queue management
 phase/3
queue management
 phase/4
queue management
 phase/5
queue management
 phase/6
queue management
 queue/blocked
queue management
 queue/done
queue management
 queue/failed
queue management
 queue/in-progress
queue management
 queue/queued
queue management
No Label driver/mqtt
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
dohertj2 (dohertj2)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dohertj2/lmxopcua#515
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?