dohertj2/lmxopcua
1
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(mqtt): flatten Sparkplug Template/DataSet metrics to sibling tags (#515)

Browse Source 
`SparkplugCodec` handled scalars; `DataSet`, `Template` and `PropertySet` were skipped
with a warning. They now flatten to addressable tags (owner decision, recorded on the
issue):

  Template  → sibling metrics, `/`-joined, recursively: Motor → Motor/Speed, Motor/Stator/Temp
  DataSet   → one metric per cell, {column}/{rowIndex}: Recipe → Recipe/Step/0, Recipe/Target/1
  PropertySet → node metadata, never tags — it describes a metric rather than being one

Flattening was chosen over OPC UA Structures because flattened scalars are consumable by
any client with no type dictionary, and the result mirrors how the `/raw` tree works.

**The row index is always present, including on row 0.** Emitting a bare column name for
a single-row table and an indexed one otherwise would make a tag's name depend on how many
rows the plant published — a table growing from one row to two would rename every tag
already authored against it.

**A DATA metric carries the parent's alias and no name**, so member names cannot be
composed until the alias table has resolved the parent. Expansion therefore happens in
`SparkplugIngestor.PublishStructuredMembers` — after resolution, at the single funnel both
the DATA and birth-value paths already pass through — not at decode time. The alias table
installs a binding per member so browse enumerates them and a DDATA member that omits its
datatype falls back to the birth's (guessing from the CLR type would read a signed Int32 of
-42 out of its unsigned field as 4294967254).

**Skip-and-warn survives, and refusal is whole** — the explicit requirement on the issue.
A template definition, an unnamed member, an extension value, a column/type mismatch, a
short row, or a payload past MaxDepth(5)/MaxMembers(512) refuses the ENTIRE metric. A
silently half-flattened UDT looks exactly like a complete one in the picker.

The browse picker offers the members and deliberately NOT the structured parent, which has
no value and no servable datatype. The rediscovery signature likewise tracks members rather
than the parent, or a UDT gaining a member would leave the signature unmoved.

Member paths contain `/`, which #509 already solved: the slashed name travels intact as
`metricName` while the derived tag Name is coerced to a legal RawPath segment.

Live-gated 3/3 against the Mosquitto fixture with the project-owned simulator extended to
publish a nested Template and a 2x2 DataSet. Falsified at three points: dropping the member
bindings, not expanding the parent, and refusing to flatten templates each turn exactly the
expected tests red.

Also fixes a regression from the #511 commit in the same branch: `AddOtOpcUaDriverFactories`
resolved `IDriverPrimaryRoleSource` with `GetRequiredService` but only `AddOtOpcUaRuntime`
registered it, so a graph wiring driver factories without the full runtime failed at
resolution. Two Host integration suites do exactly that and caught it; both extensions now
TryAdd the same concrete `RedundancyRoleView`, so the registration is order-independent and
the fail-loud resolve is kept.

Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
This commit is contained in:
Joseph Doherty
2026-07-31 08:14:04 -04:00
parent 7f2108dcf2
commit c74ccc55b5
15 changed files with 1900 additions and 53 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/drivers/Mqtt.md
+61 -7
View File
@@ -204,7 +204,7 @@ survive a reconnect; a message fans out to *every* matching tag, and unauthored
| `groupId` | string | — | **Required.** Must match the driver's `Sparkplug.GroupId` to ever receive a message |
| `edgeNodeId` | string | — | **Required.** |
| `deviceId` | string | *absent* | **Omit** for a metric published by the edge node itself (NBIRTH/NDATA). Absent is meaningfully different from `""` |
| `metricName` | string | — | **Required.** The tag's stable binding key across rebirths. **May contain `/`** — `Node Control/Rebirth` and `Properties/Serial Number` are canonical Sparkplug names, and the whole name is one value, never split |
| `metricName` | string | — | **Required.** The tag's stable binding key across rebirths. **May contain `/`** — `Node Control/Rebirth` and `Properties/Serial Number` are canonical Sparkplug names, and the whole name is one value, never split. Also how a flattened structured member is addressed: `Motor/Speed`, `Recipe/Target/0` (see [Structured metrics](#structured-metrics-template--dataset--propertyset)) |
| `dataType` | `DriverDataType` | *from the birth* | **Optional override.** Absent ⇒ take whatever type the birth certificate declared. The AdminUI's "(from birth certificate)" option removes the key entirely |
| `isHistorized` / `historianTagname` | — | — | Server-side historization, unchanged from Plain |
@@ -222,6 +222,11 @@ bind. `metricName` is deliberately **exempt**: it is not a topic segment.
// A node-level metric — note deviceId is ABSENT, not blank.
{"groupId":"OtOpcUaSim","edgeNodeId":"EdgeA","metricName":"Temperature"}
// A flattened Template member, and a flattened DataSet cell. `Motor` and `Recipe` are what the wire
// carries; nothing on the wire is ever called by these names.
{"groupId":"OtOpcUaSim","edgeNodeId":"EdgeA","deviceId":"Filler1","metricName":"Motor/Stator/Temp"}
{"groupId":"OtOpcUaSim","edgeNodeId":"EdgeA","deviceId":"Filler1","metricName":"Recipe/Target/1"}
```
## Sparkplug B Ingest
@@ -240,9 +245,11 @@ One subscription (`spBv1.0/{GroupId}/#`) feeds a per-edge-node state machine
- **Death → STALE.** An NDEATH whose `bdSeq` matches the live session (the bdSeq tie is what stops a
stale death from killing a fresh session) marks every metric under that edge node `Bad` /
`BadNoCommunication`. A DDEATH does the same for one device. The next birth restores them.
- **Unsupported datatypes are skipped with a warning, never guessed**: `DataSet`, `Template`,
`PropertySet`, `PropertySetList`, `Unknown`. `Int8`→`Int16` and `UInt8`→`UInt16` widen; `*Array`
variants map to the element type with the array bit set.
- **Structured metrics flatten to sibling tags** — `Template` and `DataSet`; see
[Structured metrics](#structured-metrics-template--dataset--propertyset).
- **Unsupported datatypes are skipped with a warning, never guessed**: `PropertySetList`, `Unknown`,
extension values, and any `Template`/`DataSet` that cannot be flattened honestly. `Int8`→`Int16` and
`UInt8`→`UInt16` widen; `*Array` variants map to the element type with the array bit set.
- **Ingest state is reset at every session/authoring seam** — a reconnect or a `ReinitializeAsync`
must not carry a previous session's alias table into a new one.
@@ -260,6 +267,53 @@ re-births freely — on its own schedule, on every reconnect, and once per rebir
> through the `/raw` browse-commit flow, and grafting nodes at runtime would materialise ones nobody
> approved and no deployment artifact records.
## Structured metrics (`Template` / `DataSet` / `PropertySet`)
Sparkplug metrics are not all scalars. `SparkplugFlattener`
([#515](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/515)) turns the structured ones into
addressable tags:
| Metric kind | Mapping | Example |
|---|---|---|
| `Template` (UDT instance) | members become **sibling metrics**, `/`-joined, recursively | `Motor` → `Motor/Speed`, `Motor/Stator/Temp` |
| `DataSet` | **one metric per cell**, `{column}/{rowIndex}` | `Recipe` → `Recipe/Step/0`, `Recipe/Target/1` |
| `PropertySet` (a metric's `properties`) | node **metadata**, never tags | `engUnit`, `Range/Low` |
Flattened members are authored exactly like any other Sparkplug metric — the composed name goes in
`metricName`. The browse picker offers them directly (the structured **parent** is deliberately *not*
offered: it has no value and no servable datatype, so a tag committed against it could never read).
**Why flattening rather than OPC UA Structures.** Owner decision, 2026-07-31. Structures are truer to
Sparkplug's model but much harder to consume — flattened scalars are readable by any OPC UA client
with no type dictionary, and the result mirrors how the `/raw` tree already works.
### Things worth knowing before you touch this
- **The row index is always present, including on row 0.** `Recipe/Target/0`, never `Recipe/Target`.
Emitting a bare column name for a single-row table and an indexed one otherwise would make a tag's
name depend on how many rows the plant published — so a table growing from one row to two would
rename every tag already authored against it.
- **A DATA metric carries the parent's alias and no name**, so member names cannot be composed until
the alias table has resolved the parent. Flattening at decode time alone is not enough; the
expansion happens in `SparkplugIngestor.PublishStructuredMembers`, after resolution.
- **Member datatype falls back to the birth's.** A DDATA template may omit its members' datatypes.
Guessing from the CLR type of the wire value would read a signed `Int32` of −42 out of its unsigned
field as 4294967254 — plausible, `Good`, and wrong.
- **Skip-and-warn survives, and refusal is whole.** A template **definition** (`is_definition`), an
unnamed member, an extension value, a column/type count mismatch, a short row, or a payload past
`MaxDepth` (5) / `MaxMembers` (512) refuses the **entire** metric rather than publishing part of it.
A silently half-flattened UDT looks exactly like a complete one in the picker.
- **Member paths contain `/`**, which `RawBrowseCommitMapper.ToRawSegment`
([#509](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/509)) already handles: the slashed name
travels intact as `metricName` while the derived tag **Name** is coerced to a legal RawPath segment,
with a collision guard for two members that would sanitise onto one name.
- **The rediscovery signature tracks members, not the parent.** A UDT gaining or losing a member leaves
the parent metric's name identical, so a gate built on raw wire names would never move.
`PropertySet` is projected as metadata rather than as sibling tags because it *describes* a metric
(units, ranges, quality hints) — surfacing it as tags would invent plant signals out of documentation.
`PropertySetList` and property extension values are dropped rather than stringified.
## Primary host (`actAsPrimaryHost`)
A Sparkplug **Primary Host Application** is the thing edge nodes watch to decide whether anyone is
@@ -515,7 +569,7 @@ identical words. See [TestConnectProbes.md](TestConnectProbes.md).
reconnect/rebuild branches, CONNACK classification, and the Sparkplug half — golden-payload decode,
topic parse/format, datatype map, alias/birth-cache rebuild, sequence + bdSeq-death ties, the
rediscovery change gate, and the browse session's "publishes nothing" assertion.
- **Live (env-gated)** — `tests/Drivers/…Driver.Mqtt.IntegrationTests` (**26 tests**) against the
- **Live (env-gated)** — `tests/Drivers/…Driver.Mqtt.IntegrationTests` (**29 tests**) against the
Mosquitto broker + the C# Sparkplug edge-node simulator; skips cleanly when `MQTT_FIXTURE_ENDPOINT`
is unset. See [Mqtt-Test-Fixture.md](Mqtt-Test-Fixture.md) and `infra/README.md` §3.
<br/>The primary-host legs (`MqttPrimaryHostLiveTests`) include the one the ordinary lifecycle cannot
@@ -541,7 +595,8 @@ identical words. See [TestConnectProbes.md](TestConnectProbes.md).
Every row is tracked. None is a silent failure — each either refuses in words or logs.
**Closed 2026-07-31:** write-through ([#508](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/508), live-gated), primary-host STATE publishing
([#511](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/511), live-gated), the slashed-metric browse-commit
([#511](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/511), live-gated), structured
`Template`/`DataSet`/`PropertySet` metrics ([#515](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/515), live-gated), the slashed-metric browse-commit
([#509](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/509)), the stale rebirth affordance ([#510](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/510)), the `.Browser` layering exception
([#512](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/512)), the stray Sparkplug `payloadFormat` ([#513](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/513)) and rebirth on a
not-yet-birthed plant ([#514](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/514)).
@@ -549,4 +604,3 @@ not-yet-birthed plant ([#514](https://gitea.dohertylan.com/dohertj2/lmxopcua/iss
| Gap | Issue | Effect | Notes |
|---|:---:|---|---|
| **Rediscovery is advisory, not a rebuild** | [#507](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/507) (closed, superseded) | A DBIRTH introducing a metric does not change the served OPC UA tree; an operator must re-browse the device and commit | **No longer "inert".** `DriverInstanceActor` now subscribes to `OnRediscoveryNeeded` and the raise rides the driver-health snapshot to the AdminUI `/hosts` page as a **re-browse chip**. It deliberately does not inject nodes: v3 authors raw tags through the `/raw` browse-commit flow, and grafting nodes at runtime would materialise ones nobody approved and no deployment artifact records. #507's injection path was deleted rather than revived |
| **`DataSet` / `Template` / `PropertySet` metrics** | [#515](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/515) | Skipped with a warning | v1-unsupported; never coerced into a guessed type |