feat(mqtt): flatten Sparkplug Template/DataSet metrics to sibling tags (#515)

`SparkplugCodec` handled scalars; `DataSet`, `Template` and `PropertySet` were skipped
with a warning. They now flatten to addressable tags (owner decision, recorded on the
issue):

  Template  → sibling metrics, `/`-joined, recursively: Motor → Motor/Speed, Motor/Stator/Temp
  DataSet   → one metric per cell, {column}/{rowIndex}: Recipe → Recipe/Step/0, Recipe/Target/1
  PropertySet → node metadata, never tags — it describes a metric rather than being one

Flattening was chosen over OPC UA Structures because flattened scalars are consumable by
any client with no type dictionary, and the result mirrors how the `/raw` tree works.

**The row index is always present, including on row 0.** Emitting a bare column name for
a single-row table and an indexed one otherwise would make a tag's name depend on how many
rows the plant published — a table growing from one row to two would rename every tag
already authored against it.

**A DATA metric carries the parent's alias and no name**, so member names cannot be
composed until the alias table has resolved the parent. Expansion therefore happens in
`SparkplugIngestor.PublishStructuredMembers` — after resolution, at the single funnel both
the DATA and birth-value paths already pass through — not at decode time. The alias table
installs a binding per member so browse enumerates them and a DDATA member that omits its
datatype falls back to the birth's (guessing from the CLR type would read a signed Int32 of
-42 out of its unsigned field as 4294967254).

**Skip-and-warn survives, and refusal is whole** — the explicit requirement on the issue.
A template definition, an unnamed member, an extension value, a column/type mismatch, a
short row, or a payload past MaxDepth(5)/MaxMembers(512) refuses the ENTIRE metric. A
silently half-flattened UDT looks exactly like a complete one in the picker.

The browse picker offers the members and deliberately NOT the structured parent, which has
no value and no servable datatype. The rediscovery signature likewise tracks members rather
than the parent, or a UDT gaining a member would leave the signature unmoved.

Member paths contain `/`, which #509 already solved: the slashed name travels intact as
`metricName` while the derived tag Name is coerced to a legal RawPath segment.

Live-gated 3/3 against the Mosquitto fixture with the project-owned simulator extended to
publish a nested Template and a 2x2 DataSet. Falsified at three points: dropping the member
bindings, not expanding the parent, and refusing to flatten templates each turn exactly the
expected tests red.

Also fixes a regression from the #511 commit in the same branch: `AddOtOpcUaDriverFactories`
resolved `IDriverPrimaryRoleSource` with `GetRequiredService` but only `AddOtOpcUaRuntime`
registered it, so a graph wiring driver factories without the full runtime failed at
resolution. Two Host integration suites do exactly that and caught it; both extensions now
TryAdd the same concrete `RedundancyRoleView`, so the registration is order-independent and
the fail-loud resolve is kept.

Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
This commit is contained in:
Joseph Doherty
2026-07-31 08:14:04 -04:00
parent 7f2108dcf2
commit c74ccc55b5
15 changed files with 1900 additions and 53 deletions
@@ -45,10 +45,19 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Sparkplug;
/// <i>wire</i>, not of the tag.
/// </para>
/// <para>
/// <b><c>DataSet</c> and <c>Template</c> metrics are out of scope for v1.</b> They decode to
/// <b><c>Template</c> and <c>DataSet</c> metrics decode to
/// <see cref="SparkplugValueKind.Structured"/> with their members flattened</b> into
/// <see cref="SparkplugMetric.Members"/> (Gitea #515) — a template member <c>Speed</c> of metric
/// <c>Motor</c> becomes the addressable <c>Motor/Speed</c>. See <see cref="SparkplugFlattener"/>
/// for the mapping and its bounds. Anything that cannot be flattened honestly stays
/// <see cref="SparkplugValueKind.Unsupported"/> with a null value — deliberately visible, so a
/// consumer can warn about a metric it cannot serve instead of silently treating it as one
/// that was never published.
/// consumer warns about a metric it cannot serve instead of silently treating it as one that was
/// never published, and never coerces it into a guessed scalar.
/// </para>
/// <para>
/// <b>A metric's <c>PropertySet</c> is projected to <see cref="SparkplugMetric.Properties"/></b>
/// as node metadata, never as sibling values: a PropertySet describes a metric (units, ranges,
/// quality hints); it is not something the metric <i>is</i>.
/// </para>
/// </remarks>
public static class SparkplugCodec
@@ -161,7 +170,7 @@ public static class SparkplugCodec
/// <returns>The projection.</returns>
private static SparkplugMetric ProjectMetric(Payload.Types.Metric metric)
{
var (kind, value) = ProjectValue(metric);
var (kind, value, members) = ProjectValue(metric);
return new SparkplugMetric(
Name: metric.HasName ? metric.Name : null,
@@ -173,7 +182,22 @@ public static class SparkplugCodec
DataType: metric.HasDatatype ? (TahuDataType)metric.Datatype : null,
TimestampMs: metric.HasTimestamp ? metric.Timestamp : null,
ValueKind: kind,
Value: value);
Value: value,
Members: members,
Properties: SparkplugFlattener.ProjectProperties(metric.Properties));
}
/// <summary>
/// Resolves a <b>template member's</b> value oneof. Internal to the assembly: shared with
/// <see cref="SparkplugFlattener"/>, which walks nested templates and needs the identical
/// projection so a member and a top-level metric of the same type never decode differently.
/// </summary>
/// <param name="metric">The template member, itself a generated <c>Metric</c>.</param>
/// <returns>The value kind and the projected raw value.</returns>
internal static (SparkplugValueKind Kind, object? Value) ProjectMemberValue(Payload.Types.Metric metric)
{
var (kind, value, _) = ProjectValue(metric);
return (kind, value);
}
/// <summary>Resolves a metric's value <c>oneof</c> to a kind plus a raw CLR value.</summary>
@@ -184,34 +208,52 @@ public static class SparkplugCodec
/// that some datatypes have no spare sentinel, so a publisher that sets it means "null" even if
/// a value field is also populated.
/// </remarks>
private static (SparkplugValueKind Kind, object? Value) ProjectValue(Payload.Types.Metric metric)
private static (SparkplugValueKind Kind, object? Value, IReadOnlyList<SparkplugMember> Members) ProjectValue(
Payload.Types.Metric metric)
{
if (metric.HasIsNull && metric.IsNull)
{
return (SparkplugValueKind.Null, null);
return (SparkplugValueKind.Null, null, []);
}
// Gitea #515: structured values flatten to addressable members. A refusal to flatten falls
// through to Unsupported — the pre-#515 behaviour — rather than to a guessed scalar.
switch (metric.ValueCase)
{
case Payload.Types.Metric.ValueOneofCase.TemplateValue:
return SparkplugFlattener.TryFlattenTemplate(metric.TemplateValue, out var templateMembers)
? (SparkplugValueKind.Structured, null, templateMembers)
: (SparkplugValueKind.Unsupported, null, []);
case Payload.Types.Metric.ValueOneofCase.DatasetValue:
return SparkplugFlattener.TryFlattenDataSet(metric.DatasetValue, out var cells)
? (SparkplugValueKind.Structured, null, cells)
: (SparkplugValueKind.Unsupported, null, []);
}
return metric.ValueCase switch
{
Payload.Types.Metric.ValueOneofCase.IntValue => (SparkplugValueKind.Scalar, metric.IntValue),
Payload.Types.Metric.ValueOneofCase.LongValue => (SparkplugValueKind.Scalar, metric.LongValue),
Payload.Types.Metric.ValueOneofCase.FloatValue => (SparkplugValueKind.Scalar, metric.FloatValue),
Payload.Types.Metric.ValueOneofCase.DoubleValue => (SparkplugValueKind.Scalar, metric.DoubleValue),
Payload.Types.Metric.ValueOneofCase.BooleanValue => (SparkplugValueKind.Scalar, metric.BooleanValue),
Payload.Types.Metric.ValueOneofCase.StringValue => (SparkplugValueKind.Scalar, metric.StringValue),
Payload.Types.Metric.ValueOneofCase.IntValue => (SparkplugValueKind.Scalar, metric.IntValue, Empty),
Payload.Types.Metric.ValueOneofCase.LongValue => (SparkplugValueKind.Scalar, metric.LongValue, Empty),
Payload.Types.Metric.ValueOneofCase.FloatValue => (SparkplugValueKind.Scalar, metric.FloatValue, Empty),
Payload.Types.Metric.ValueOneofCase.DoubleValue => (SparkplugValueKind.Scalar, metric.DoubleValue, Empty),
Payload.Types.Metric.ValueOneofCase.BooleanValue => (SparkplugValueKind.Scalar, metric.BooleanValue, Empty),
Payload.Types.Metric.ValueOneofCase.StringValue => (SparkplugValueKind.Scalar, metric.StringValue, Empty),
// Copied out of the ByteString: the projection must outlive the parsed message.
Payload.Types.Metric.ValueOneofCase.BytesValue =>
(SparkplugValueKind.Scalar, metric.BytesValue.ToByteArray()),
(SparkplugValueKind.Scalar, metric.BytesValue.ToByteArray(), Empty),
// v1 scope boundary — decoded as a visible refusal, not as an exception or a silent drop.
Payload.Types.Metric.ValueOneofCase.DatasetValue => (SparkplugValueKind.Unsupported, null),
Payload.Types.Metric.ValueOneofCase.TemplateValue => (SparkplugValueKind.Unsupported, null),
Payload.Types.Metric.ValueOneofCase.ExtensionValue => (SparkplugValueKind.Unsupported, null),
// An extension value is a third-party escape hatch with no defined semantics — there is
// nothing honest to flatten it into.
Payload.Types.Metric.ValueOneofCase.ExtensionValue => (SparkplugValueKind.Unsupported, null, Empty),
_ => (SparkplugValueKind.Absent, null),
_ => (SparkplugValueKind.Absent, null, Empty),
};
}
/// <summary>The shared empty member list — every scalar metric would otherwise allocate one.</summary>
private static readonly IReadOnlyList<SparkplugMember> Empty = [];
}
/// <summary>
@@ -279,13 +321,31 @@ public sealed record SparkplugPayload(
/// switch over a dozen arms.
/// </para>
/// </param>
/// <param name="Members">
/// The flattened members of a <see cref="SparkplugValueKind.Structured"/> metric — a
/// <c>Template</c> or <c>DataSet</c> broken into addressable scalars (Gitea #515). Empty for every
/// other kind. Paths are <b>relative to this metric</b>; see <see cref="SparkplugMember.Path"/>.
/// </param>
/// <param name="Properties">
/// The metric's <c>PropertySet</c>, projected as node metadata (Gitea #515). Empty when the metric
/// carried none. <b>Never a value</b> — a PropertySet describes the metric rather than being one.
/// </param>
public readonly record struct SparkplugMetric(
string? Name,
ulong? Alias,
TahuDataType? DataType,
ulong? TimestampMs,
SparkplugValueKind ValueKind,
object? Value);
object? Value,
IReadOnlyList<SparkplugMember>? Members = null,
IReadOnlyList<SparkplugProperty>? Properties = null)
{
/// <summary>The flattened members, never null.</summary>
public IReadOnlyList<SparkplugMember> MemberList => Members ?? [];
/// <summary>The projected properties, never null.</summary>
public IReadOnlyList<SparkplugProperty> PropertyList => Properties ?? [];
}
/// <summary>What a decoded metric's <see cref="SparkplugMetric.Value"/> represents.</summary>
/// <remarks>
@@ -305,9 +365,17 @@ public enum SparkplugValueKind
Scalar,
/// <summary>
/// A <c>DataSet</c>, <c>Template</c> or extension value — decoded and reported, but not
/// supported in v1. The consumer should warn and skip the metric rather than treat it as
/// missing.
/// An extension value, or a <c>Template</c>/<c>DataSet</c> that could not be flattened
/// honestly (see <see cref="SparkplugFlattener"/>). Decoded and reported, but not servable:
/// the consumer warns and skips the metric rather than treating it as missing — and never
/// coerces it into a guessed scalar.
/// </summary>
Unsupported,
/// <summary>
/// A <c>Template</c> or <c>DataSet</c> whose members were flattened into
/// <see cref="SparkplugMetric.Members"/> (Gitea #515). <see cref="SparkplugMetric.Value"/> is
/// null — the parent has no scalar value of its own; its members carry the values.
/// </summary>
Structured,
}