feat(mqtt): flatten Sparkplug Template/DataSet metrics to sibling tags (#515)
`SparkplugCodec` handled scalars; `DataSet`, `Template` and `PropertySet` were skipped
with a warning. They now flatten to addressable tags (owner decision, recorded on the
issue):
Template → sibling metrics, `/`-joined, recursively: Motor → Motor/Speed, Motor/Stator/Temp
DataSet → one metric per cell, {column}/{rowIndex}: Recipe → Recipe/Step/0, Recipe/Target/1
PropertySet → node metadata, never tags — it describes a metric rather than being one
Flattening was chosen over OPC UA Structures because flattened scalars are consumable by
any client with no type dictionary, and the result mirrors how the `/raw` tree works.
**The row index is always present, including on row 0.** Emitting a bare column name for
a single-row table and an indexed one otherwise would make a tag's name depend on how many
rows the plant published — a table growing from one row to two would rename every tag
already authored against it.
**A DATA metric carries the parent's alias and no name**, so member names cannot be
composed until the alias table has resolved the parent. Expansion therefore happens in
`SparkplugIngestor.PublishStructuredMembers` — after resolution, at the single funnel both
the DATA and birth-value paths already pass through — not at decode time. The alias table
installs a binding per member so browse enumerates them and a DDATA member that omits its
datatype falls back to the birth's (guessing from the CLR type would read a signed Int32 of
-42 out of its unsigned field as 4294967254).
**Skip-and-warn survives, and refusal is whole** — the explicit requirement on the issue.
A template definition, an unnamed member, an extension value, a column/type mismatch, a
short row, or a payload past MaxDepth(5)/MaxMembers(512) refuses the ENTIRE metric. A
silently half-flattened UDT looks exactly like a complete one in the picker.
The browse picker offers the members and deliberately NOT the structured parent, which has
no value and no servable datatype. The rediscovery signature likewise tracks members rather
than the parent, or a UDT gaining a member would leave the signature unmoved.
Member paths contain `/`, which #509 already solved: the slashed name travels intact as
`metricName` while the derived tag Name is coerced to a legal RawPath segment.
Live-gated 3/3 against the Mosquitto fixture with the project-owned simulator extended to
publish a nested Template and a 2x2 DataSet. Falsified at three points: dropping the member
bindings, not expanding the parent, and refusing to flatten templates each turn exactly the
expected tests red.
Also fixes a regression from the #511 commit in the same branch: `AddOtOpcUaDriverFactories`
resolved `IDriverPrimaryRoleSource` with `GetRequiredService` but only `AddOtOpcUaRuntime`
registered it, so a graph wiring driver factories without the full runtime failed at
resolution. Two Host integration suites do exactly that and caught it; both extensions now
TryAdd the same concrete `RedundancyRoleView`, so the registration is order-independent and
the fail-loud resolve is kept.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
This commit is contained in:
Joseph Doherty
@@ -585,20 +585,77 @@ internal sealed class MqttBrowseSession : IRebirthCapableBrowseSession
|
||||
continue;
|
||||
}
|
||||
|
||||
var metricPath = ownerPath + MetricSeparator + name;
|
||||
if (metricPath.Length > MaxTopicChars)
|
||||
// Gitea #515: a Template/DataSet metric contributes its flattened MEMBERS as leaves, never
|
||||
// itself. The parent has no value and no servable datatype, so a leaf for it would offer
|
||||
// the operator an address that can only ever commit a tag which never reads.
|
||||
if (metric.ValueKind == SparkplugValueKind.Structured)
|
||||
{
|
||||
Volatile.Write(ref _oversized, 1);
|
||||
foreach (var member in metric.MemberList)
|
||||
{
|
||||
var memberName = $"{name}{SparkplugFlattener.MemberSeparator}{member.Path}";
|
||||
if (!IsUsableLabel(memberName))
|
||||
{
|
||||
Volatile.Write(ref _oversized, 1);
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!AddMetricLeaf(owner, ownerPath, ownerScope, memberName, member.DataType, parsed.Type, alias: null))
|
||||
{
|
||||
return; // node cap tripped
|
||||
}
|
||||
}
|
||||
|
||||
continue;
|
||||
}
|
||||
|
||||
var node = GetOrCreate(owner.Children, metricPath, name, BrowseNodeKind.Leaf, ownerScope);
|
||||
if (node is null) return; // node cap tripped
|
||||
|
||||
node.RecordMetric(new SparkplugMetricInfo(name, metric.DataType, parsed.Type, metric.Alias));
|
||||
if (!AddMetricLeaf(owner, ownerPath, ownerScope, name, metric.DataType, parsed.Type, metric.Alias))
|
||||
{
|
||||
return; // node cap tripped
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Places one metric leaf under <paramref name="owner"/>.
|
||||
/// </summary>
|
||||
/// <param name="owner">The owning edge-node / device folder.</param>
|
||||
/// <param name="ownerPath">That folder's path.</param>
|
||||
/// <param name="ownerScope">The scope stamped on the leaf.</param>
|
||||
/// <param name="metricName">
|
||||
/// The metric's Sparkplug name — for a flattened structured member, the composed
|
||||
/// <c>{parent}/{memberPath}</c>. It travels intact: it is the binding key, and only the
|
||||
/// <i>display</i> name is later coerced to a legal RawPath segment (Gitea #509).
|
||||
/// </param>
|
||||
/// <param name="dataType">The declared datatype, or null.</param>
|
||||
/// <param name="messageType">The birth message type this leaf was seen in.</param>
|
||||
/// <param name="alias">The metric's alias, or null — flattened members carry none of their own.</param>
|
||||
/// <returns><see langword="false"/> when the node cap tripped and the caller must stop.</returns>
|
||||
private bool AddMetricLeaf(
|
||||
TopicNode owner,
|
||||
string ownerPath,
|
||||
SparkplugScope ownerScope,
|
||||
string metricName,
|
||||
Org.Eclipse.Tahu.Protobuf.DataType? dataType,
|
||||
SparkplugMessageType messageType,
|
||||
ulong? alias)
|
||||
{
|
||||
var metricPath = ownerPath + MetricSeparator + metricName;
|
||||
if (metricPath.Length > MaxTopicChars)
|
||||
{
|
||||
Volatile.Write(ref _oversized, 1);
|
||||
return true;
|
||||
}
|
||||
|
||||
var node = GetOrCreate(owner.Children, metricPath, metricName, BrowseNodeKind.Leaf, ownerScope);
|
||||
if (node is null)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
node.RecordMetric(new SparkplugMetricInfo(metricName, dataType, messageType, alias));
|
||||
return true;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Finds or creates one tree node under <paramref name="level"/>, honouring the node cap.
|
||||
/// </summary>
|
||||
@@ -698,6 +755,40 @@ internal sealed class MqttBrowseSession : IRebirthCapableBrowseSession
|
||||
ObserveBirth(groupId, edgeNodeId, device, metric);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Test seam feeding a birth whose metric is a structured <c>Template</c> (Gitea #515).
|
||||
/// </summary>
|
||||
/// <param name="groupId">The Sparkplug group id.</param>
|
||||
/// <param name="edgeNodeId">The Sparkplug edge-node id.</param>
|
||||
/// <param name="device">The device id, or <see langword="null"/> for an NBIRTH.</param>
|
||||
/// <param name="metricName">The template metric's own name.</param>
|
||||
/// <param name="members">The template's members, as <c>(name, datatype)</c>.</param>
|
||||
internal void ObserveTemplateBirthForTest(
|
||||
string groupId,
|
||||
string edgeNodeId,
|
||||
string? device,
|
||||
string metricName,
|
||||
params (string Name, SparkplugDataType DataType)[] members)
|
||||
{
|
||||
var template = new Payload.Types.Template { Version = "1", IsDefinition = false };
|
||||
foreach (var (name, dataType) in members)
|
||||
{
|
||||
var member = new Payload.Types.Metric { Name = name, Datatype = (uint)dataType };
|
||||
|
||||
// A value is required: a member carrying only a name and a datatype is indistinguishable
|
||||
// from a template DEFINITION's member, which the flattener deliberately refuses.
|
||||
member.DoubleValue = 0;
|
||||
template.Metrics.Add(member);
|
||||
}
|
||||
|
||||
ObserveBirth(groupId, edgeNodeId, device, new Payload.Types.Metric
|
||||
{
|
||||
Name = metricName,
|
||||
Datatype = (uint)SparkplugDataType.Template,
|
||||
TemplateValue = template,
|
||||
});
|
||||
}
|
||||
|
||||
/// <summary>Test seam feeding a birth certificate that declares no metrics at all.</summary>
|
||||
/// <param name="groupId">The Sparkplug group id.</param>
|
||||
/// <param name="edgeNodeId">The Sparkplug edge-node id.</param>
|
||||
|
||||
@@ -45,10 +45,19 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Sparkplug;
|
||||
/// <i>wire</i>, not of the tag.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b><c>DataSet</c> and <c>Template</c> metrics are out of scope for v1.</b> They decode to
|
||||
/// <b><c>Template</c> and <c>DataSet</c> metrics decode to
|
||||
/// <see cref="SparkplugValueKind.Structured"/> with their members flattened</b> into
|
||||
/// <see cref="SparkplugMetric.Members"/> (Gitea #515) — a template member <c>Speed</c> of metric
|
||||
/// <c>Motor</c> becomes the addressable <c>Motor/Speed</c>. See <see cref="SparkplugFlattener"/>
|
||||
/// for the mapping and its bounds. Anything that cannot be flattened honestly stays
|
||||
/// <see cref="SparkplugValueKind.Unsupported"/> with a null value — deliberately visible, so a
|
||||
/// consumer can warn about a metric it cannot serve instead of silently treating it as one
|
||||
/// that was never published.
|
||||
/// consumer warns about a metric it cannot serve instead of silently treating it as one that was
|
||||
/// never published, and never coerces it into a guessed scalar.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>A metric's <c>PropertySet</c> is projected to <see cref="SparkplugMetric.Properties"/></b>
|
||||
/// as node metadata, never as sibling values: a PropertySet describes a metric (units, ranges,
|
||||
/// quality hints); it is not something the metric <i>is</i>.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public static class SparkplugCodec
|
||||
@@ -161,7 +170,7 @@ public static class SparkplugCodec
|
||||
/// <returns>The projection.</returns>
|
||||
private static SparkplugMetric ProjectMetric(Payload.Types.Metric metric)
|
||||
{
|
||||
var (kind, value) = ProjectValue(metric);
|
||||
var (kind, value, members) = ProjectValue(metric);
|
||||
|
||||
return new SparkplugMetric(
|
||||
Name: metric.HasName ? metric.Name : null,
|
||||
@@ -173,7 +182,22 @@ public static class SparkplugCodec
|
||||
DataType: metric.HasDatatype ? (TahuDataType)metric.Datatype : null,
|
||||
TimestampMs: metric.HasTimestamp ? metric.Timestamp : null,
|
||||
ValueKind: kind,
|
||||
Value: value);
|
||||
Value: value,
|
||||
Members: members,
|
||||
Properties: SparkplugFlattener.ProjectProperties(metric.Properties));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Resolves a <b>template member's</b> value oneof. Internal to the assembly: shared with
|
||||
/// <see cref="SparkplugFlattener"/>, which walks nested templates and needs the identical
|
||||
/// projection so a member and a top-level metric of the same type never decode differently.
|
||||
/// </summary>
|
||||
/// <param name="metric">The template member, itself a generated <c>Metric</c>.</param>
|
||||
/// <returns>The value kind and the projected raw value.</returns>
|
||||
internal static (SparkplugValueKind Kind, object? Value) ProjectMemberValue(Payload.Types.Metric metric)
|
||||
{
|
||||
var (kind, value, _) = ProjectValue(metric);
|
||||
return (kind, value);
|
||||
}
|
||||
|
||||
/// <summary>Resolves a metric's value <c>oneof</c> to a kind plus a raw CLR value.</summary>
|
||||
@@ -184,34 +208,52 @@ public static class SparkplugCodec
|
||||
/// that some datatypes have no spare sentinel, so a publisher that sets it means "null" even if
|
||||
/// a value field is also populated.
|
||||
/// </remarks>
|
||||
private static (SparkplugValueKind Kind, object? Value) ProjectValue(Payload.Types.Metric metric)
|
||||
private static (SparkplugValueKind Kind, object? Value, IReadOnlyList<SparkplugMember> Members) ProjectValue(
|
||||
Payload.Types.Metric metric)
|
||||
{
|
||||
if (metric.HasIsNull && metric.IsNull)
|
||||
{
|
||||
return (SparkplugValueKind.Null, null);
|
||||
return (SparkplugValueKind.Null, null, []);
|
||||
}
|
||||
|
||||
// Gitea #515: structured values flatten to addressable members. A refusal to flatten falls
|
||||
// through to Unsupported — the pre-#515 behaviour — rather than to a guessed scalar.
|
||||
switch (metric.ValueCase)
|
||||
{
|
||||
case Payload.Types.Metric.ValueOneofCase.TemplateValue:
|
||||
return SparkplugFlattener.TryFlattenTemplate(metric.TemplateValue, out var templateMembers)
|
||||
? (SparkplugValueKind.Structured, null, templateMembers)
|
||||
: (SparkplugValueKind.Unsupported, null, []);
|
||||
|
||||
case Payload.Types.Metric.ValueOneofCase.DatasetValue:
|
||||
return SparkplugFlattener.TryFlattenDataSet(metric.DatasetValue, out var cells)
|
||||
? (SparkplugValueKind.Structured, null, cells)
|
||||
: (SparkplugValueKind.Unsupported, null, []);
|
||||
}
|
||||
|
||||
return metric.ValueCase switch
|
||||
{
|
||||
Payload.Types.Metric.ValueOneofCase.IntValue => (SparkplugValueKind.Scalar, metric.IntValue),
|
||||
Payload.Types.Metric.ValueOneofCase.LongValue => (SparkplugValueKind.Scalar, metric.LongValue),
|
||||
Payload.Types.Metric.ValueOneofCase.FloatValue => (SparkplugValueKind.Scalar, metric.FloatValue),
|
||||
Payload.Types.Metric.ValueOneofCase.DoubleValue => (SparkplugValueKind.Scalar, metric.DoubleValue),
|
||||
Payload.Types.Metric.ValueOneofCase.BooleanValue => (SparkplugValueKind.Scalar, metric.BooleanValue),
|
||||
Payload.Types.Metric.ValueOneofCase.StringValue => (SparkplugValueKind.Scalar, metric.StringValue),
|
||||
Payload.Types.Metric.ValueOneofCase.IntValue => (SparkplugValueKind.Scalar, metric.IntValue, Empty),
|
||||
Payload.Types.Metric.ValueOneofCase.LongValue => (SparkplugValueKind.Scalar, metric.LongValue, Empty),
|
||||
Payload.Types.Metric.ValueOneofCase.FloatValue => (SparkplugValueKind.Scalar, metric.FloatValue, Empty),
|
||||
Payload.Types.Metric.ValueOneofCase.DoubleValue => (SparkplugValueKind.Scalar, metric.DoubleValue, Empty),
|
||||
Payload.Types.Metric.ValueOneofCase.BooleanValue => (SparkplugValueKind.Scalar, metric.BooleanValue, Empty),
|
||||
Payload.Types.Metric.ValueOneofCase.StringValue => (SparkplugValueKind.Scalar, metric.StringValue, Empty),
|
||||
|
||||
// Copied out of the ByteString: the projection must outlive the parsed message.
|
||||
Payload.Types.Metric.ValueOneofCase.BytesValue =>
|
||||
(SparkplugValueKind.Scalar, metric.BytesValue.ToByteArray()),
|
||||
(SparkplugValueKind.Scalar, metric.BytesValue.ToByteArray(), Empty),
|
||||
|
||||
// v1 scope boundary — decoded as a visible refusal, not as an exception or a silent drop.
|
||||
Payload.Types.Metric.ValueOneofCase.DatasetValue => (SparkplugValueKind.Unsupported, null),
|
||||
Payload.Types.Metric.ValueOneofCase.TemplateValue => (SparkplugValueKind.Unsupported, null),
|
||||
Payload.Types.Metric.ValueOneofCase.ExtensionValue => (SparkplugValueKind.Unsupported, null),
|
||||
// An extension value is a third-party escape hatch with no defined semantics — there is
|
||||
// nothing honest to flatten it into.
|
||||
Payload.Types.Metric.ValueOneofCase.ExtensionValue => (SparkplugValueKind.Unsupported, null, Empty),
|
||||
|
||||
_ => (SparkplugValueKind.Absent, null),
|
||||
_ => (SparkplugValueKind.Absent, null, Empty),
|
||||
};
|
||||
}
|
||||
|
||||
/// <summary>The shared empty member list — every scalar metric would otherwise allocate one.</summary>
|
||||
private static readonly IReadOnlyList<SparkplugMember> Empty = [];
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
@@ -279,13 +321,31 @@ public sealed record SparkplugPayload(
|
||||
/// switch over a dozen arms.
|
||||
/// </para>
|
||||
/// </param>
|
||||
/// <param name="Members">
|
||||
/// The flattened members of a <see cref="SparkplugValueKind.Structured"/> metric — a
|
||||
/// <c>Template</c> or <c>DataSet</c> broken into addressable scalars (Gitea #515). Empty for every
|
||||
/// other kind. Paths are <b>relative to this metric</b>; see <see cref="SparkplugMember.Path"/>.
|
||||
/// </param>
|
||||
/// <param name="Properties">
|
||||
/// The metric's <c>PropertySet</c>, projected as node metadata (Gitea #515). Empty when the metric
|
||||
/// carried none. <b>Never a value</b> — a PropertySet describes the metric rather than being one.
|
||||
/// </param>
|
||||
public readonly record struct SparkplugMetric(
|
||||
string? Name,
|
||||
ulong? Alias,
|
||||
TahuDataType? DataType,
|
||||
ulong? TimestampMs,
|
||||
SparkplugValueKind ValueKind,
|
||||
object? Value);
|
||||
object? Value,
|
||||
IReadOnlyList<SparkplugMember>? Members = null,
|
||||
IReadOnlyList<SparkplugProperty>? Properties = null)
|
||||
{
|
||||
/// <summary>The flattened members, never null.</summary>
|
||||
public IReadOnlyList<SparkplugMember> MemberList => Members ?? [];
|
||||
|
||||
/// <summary>The projected properties, never null.</summary>
|
||||
public IReadOnlyList<SparkplugProperty> PropertyList => Properties ?? [];
|
||||
}
|
||||
|
||||
/// <summary>What a decoded metric's <see cref="SparkplugMetric.Value"/> represents.</summary>
|
||||
/// <remarks>
|
||||
@@ -305,9 +365,17 @@ public enum SparkplugValueKind
|
||||
Scalar,
|
||||
|
||||
/// <summary>
|
||||
/// A <c>DataSet</c>, <c>Template</c> or extension value — decoded and reported, but not
|
||||
/// supported in v1. The consumer should warn and skip the metric rather than treat it as
|
||||
/// missing.
|
||||
/// An extension value, or a <c>Template</c>/<c>DataSet</c> that could not be flattened
|
||||
/// honestly (see <see cref="SparkplugFlattener"/>). Decoded and reported, but not servable:
|
||||
/// the consumer warns and skips the metric rather than treating it as missing — and never
|
||||
/// coerces it into a guessed scalar.
|
||||
/// </summary>
|
||||
Unsupported,
|
||||
|
||||
/// <summary>
|
||||
/// A <c>Template</c> or <c>DataSet</c> whose members were flattened into
|
||||
/// <see cref="SparkplugMetric.Members"/> (Gitea #515). <see cref="SparkplugMetric.Value"/> is
|
||||
/// null — the parent has no scalar value of its own; its members carry the values.
|
||||
/// </summary>
|
||||
Structured,
|
||||
}
|
||||
|
||||
@@ -0,0 +1,368 @@
|
||||
using Org.Eclipse.Tahu.Protobuf;
|
||||
using TahuDataType = Org.Eclipse.Tahu.Protobuf.DataType;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Sparkplug;
|
||||
|
||||
/// <summary>
|
||||
/// Flattens Sparkplug's structured metric values — <c>Template</c> (UDT instances) and
|
||||
/// <c>DataSet</c> — into addressable scalar members, and projects a metric's <c>PropertySet</c>
|
||||
/// into node metadata (Gitea #515).
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>Flatten to siblings, by owner decision (2026-07-31).</b> The alternative was mapping a
|
||||
/// Template to an OPC UA Structure/UDT, which is truer to Sparkplug's model but far harder to
|
||||
/// consume: flattened scalars are readable by any OPC UA client (ScadaBridge included) with no
|
||||
/// type dictionary, and the result mirrors how the <c>/raw</c> tree already works. A member of
|
||||
/// template metric <c>Motor</c> called <c>Speed</c> becomes the metric <c>Motor/Speed</c>.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Member paths contain <c>/</c>, which is already solved.</b> That interaction is what
|
||||
/// #515 flagged against #509, and #509 shipped first:
|
||||
/// <c>RawBrowseCommitMapper.ToRawSegment</c> coerces the tag <i>Name</i> to a legal RawPath
|
||||
/// segment while carrying the original slashed string through as <c>metricName</c>, plus a
|
||||
/// collision guard that refuses two members which would sanitise onto one name. So the
|
||||
/// flattened name travels intact as the binding key and only the display name is coerced.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Skip-and-warn survives, and that is a requirement rather than a leftover.</b> Anything
|
||||
/// this cannot flatten honestly — an extension value, a nested member with no name, a payload
|
||||
/// past the bounds below — stays <see cref="SparkplugValueKind.Unsupported"/> and is skipped
|
||||
/// with a warning. A structured value coerced into a guessed scalar would bind a tag to data
|
||||
/// that is not what its name says, which is worse than not binding it at all.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Bounded, because this walks an attacker-shaped object graph.</b> A Template may contain
|
||||
/// Templates, so the natural recursion is unbounded in a format anyone on the broker can
|
||||
/// publish. <see cref="MaxDepth"/> and <see cref="MaxMembers"/> cap it; exceeding either
|
||||
/// abandons the whole metric to <c>Unsupported</c> rather than publishing a truncated member
|
||||
/// set, because a silently half-flattened UDT looks exactly like a complete one.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Nothing here throws</b> — same contract as <see cref="SparkplugCodec"/>, for the same
|
||||
/// reason: this runs on MQTTnet's shared dispatcher thread behind an unauthenticated firehose.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public static class SparkplugFlattener
|
||||
{
|
||||
/// <summary>
|
||||
/// How deeply nested a Template may be before the whole metric is refused. Five is far past
|
||||
/// anything a real UDT library produces and still bounds the walk.
|
||||
/// </summary>
|
||||
public const int MaxDepth = 5;
|
||||
|
||||
/// <summary>
|
||||
/// Ceiling on the flattened members one metric may produce. A DataSet is columns × rows, so a
|
||||
/// modest-looking table is the realistic way to reach this.
|
||||
/// </summary>
|
||||
public const int MaxMembers = 512;
|
||||
|
||||
/// <summary>
|
||||
/// Separator between a structured metric's name and its member path. The same character
|
||||
/// Sparkplug itself uses for metric-name hierarchy, so <c>Motor/Speed</c> is what an edge node
|
||||
/// would have called a flat metric of the same meaning.
|
||||
/// </summary>
|
||||
public const char MemberSeparator = '/';
|
||||
|
||||
/// <summary>
|
||||
/// Flattens a Template into member paths relative to the owning metric.
|
||||
/// </summary>
|
||||
/// <param name="template">The template value.</param>
|
||||
/// <param name="members">The flattened members on success; empty otherwise.</param>
|
||||
/// <returns>
|
||||
/// <see langword="true"/> when the whole template flattened. <see langword="false"/> means the
|
||||
/// caller must treat the metric as <see cref="SparkplugValueKind.Unsupported"/>.
|
||||
/// </returns>
|
||||
public static bool TryFlattenTemplate(Payload.Types.Template template, out IReadOnlyList<SparkplugMember> members)
|
||||
{
|
||||
var collected = new List<SparkplugMember>();
|
||||
var ok = template is not null && AppendTemplate(template, prefix: "", depth: 1, collected);
|
||||
members = ok ? collected : [];
|
||||
return ok;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Flattens a DataSet into one member per cell, named <c>{column}/{rowIndex}</c>.
|
||||
/// </summary>
|
||||
/// <param name="dataSet">The dataset value.</param>
|
||||
/// <param name="members">The flattened members on success; empty otherwise.</param>
|
||||
/// <returns>
|
||||
/// <see langword="true"/> when the whole dataset flattened. <see langword="false"/> means the
|
||||
/// caller must treat the metric as <see cref="SparkplugValueKind.Unsupported"/>.
|
||||
/// </returns>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>Per cell, not per column — and the row index is always present.</b> The recorded
|
||||
/// decision says "per-column tags, not one array-valued variable", and a column of N rows
|
||||
/// is not a scalar, so per-column can only mean per-cell once N > 1. Emitting a bare
|
||||
/// <c>{column}</c> for the single-row case and <c>{column}/{row}</c> otherwise would make
|
||||
/// a tag's own name depend on how many rows the plant happened to publish — so a table
|
||||
/// that grew from one row to two would rename every existing tag out from under its
|
||||
/// authored binding. Uniform naming costs a <c>/0</c> on the common single-row "current
|
||||
/// values" shape and buys a stable address.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// A row that disappears simply stops publishing, and its tag holds its last value — the
|
||||
/// same as any metric an edge node stops sending.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public static bool TryFlattenDataSet(Payload.Types.DataSet dataSet, out IReadOnlyList<SparkplugMember> members)
|
||||
{
|
||||
members = [];
|
||||
if (dataSet is null)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
// Columns and types are parallel arrays. A publisher that disagrees with itself has produced a
|
||||
// dataset whose cells cannot be typed, and guessing which array to trust would type a column
|
||||
// by its neighbour — refuse instead.
|
||||
if (dataSet.Columns.Count == 0 || dataSet.Types_.Count != dataSet.Columns.Count)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
if ((long)dataSet.Columns.Count * dataSet.Rows.Count > MaxMembers)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
var collected = new List<SparkplugMember>();
|
||||
for (var row = 0; row < dataSet.Rows.Count; row++)
|
||||
{
|
||||
var elements = dataSet.Rows[row].Elements;
|
||||
|
||||
// A short row is a malformed dataset, not a row of nulls: emitting the cells that ARE
|
||||
// present would publish a table silently missing its right-hand columns.
|
||||
if (elements.Count != dataSet.Columns.Count)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
for (var col = 0; col < dataSet.Columns.Count; col++)
|
||||
{
|
||||
var column = dataSet.Columns[col];
|
||||
if (string.IsNullOrEmpty(column))
|
||||
{
|
||||
return false; // An unnamed column has no address.
|
||||
}
|
||||
|
||||
var (kind, value) = ProjectDataSetValue(elements[col]);
|
||||
collected.Add(new SparkplugMember(
|
||||
Path: $"{column}{MemberSeparator}{row.ToString(System.Globalization.CultureInfo.InvariantCulture)}",
|
||||
DataType: (TahuDataType)dataSet.Types_[col],
|
||||
ValueKind: kind,
|
||||
Value: value));
|
||||
}
|
||||
}
|
||||
|
||||
members = collected;
|
||||
return true;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Projects a metric's <c>PropertySet</c> into flat name/value metadata.
|
||||
/// </summary>
|
||||
/// <param name="properties">The property set, or <see langword="null"/>.</param>
|
||||
/// <returns>The projected properties; empty when there are none or the set is malformed.</returns>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>Metadata, never tags</b> — the recorded decision, and the honest reading of the
|
||||
/// spec: a PropertySet describes a metric (engineering units, ranges, quality hints), it is
|
||||
/// not a value the metric has. Surfacing it as sibling tags would invent plant signals out
|
||||
/// of documentation.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Nested <c>PropertySet</c>/<c>PropertySetList</c> values are flattened one level with a
|
||||
/// <c>/</c>-joined key, matching the metric-member convention; anything deeper, or an
|
||||
/// extension value, is dropped rather than stringified.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public static IReadOnlyList<SparkplugProperty> ProjectProperties(Payload.Types.PropertySet? properties)
|
||||
{
|
||||
if (properties is null || properties.Keys.Count == 0 || properties.Keys.Count != properties.Values.Count)
|
||||
{
|
||||
return [];
|
||||
}
|
||||
|
||||
var collected = new List<SparkplugProperty>();
|
||||
AppendProperties(properties, prefix: "", depth: 1, collected);
|
||||
return collected;
|
||||
}
|
||||
|
||||
private static void AppendProperties(
|
||||
Payload.Types.PropertySet set, string prefix, int depth, List<SparkplugProperty> into)
|
||||
{
|
||||
if (set.Keys.Count != set.Values.Count || depth > 2 || into.Count >= MaxMembers)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
for (var i = 0; i < set.Keys.Count && into.Count < MaxMembers; i++)
|
||||
{
|
||||
var key = set.Keys[i];
|
||||
if (string.IsNullOrEmpty(key))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var name = prefix.Length == 0 ? key : $"{prefix}{MemberSeparator}{key}";
|
||||
var value = set.Values[i];
|
||||
|
||||
if (value.HasIsNull && value.IsNull)
|
||||
{
|
||||
into.Add(new SparkplugProperty(name, null));
|
||||
continue;
|
||||
}
|
||||
|
||||
switch (value.ValueCase)
|
||||
{
|
||||
case Payload.Types.PropertyValue.ValueOneofCase.IntValue:
|
||||
into.Add(new SparkplugProperty(name, value.IntValue));
|
||||
break;
|
||||
case Payload.Types.PropertyValue.ValueOneofCase.LongValue:
|
||||
into.Add(new SparkplugProperty(name, value.LongValue));
|
||||
break;
|
||||
case Payload.Types.PropertyValue.ValueOneofCase.FloatValue:
|
||||
into.Add(new SparkplugProperty(name, value.FloatValue));
|
||||
break;
|
||||
case Payload.Types.PropertyValue.ValueOneofCase.DoubleValue:
|
||||
into.Add(new SparkplugProperty(name, value.DoubleValue));
|
||||
break;
|
||||
case Payload.Types.PropertyValue.ValueOneofCase.BooleanValue:
|
||||
into.Add(new SparkplugProperty(name, value.BooleanValue));
|
||||
break;
|
||||
case Payload.Types.PropertyValue.ValueOneofCase.StringValue:
|
||||
into.Add(new SparkplugProperty(name, value.StringValue));
|
||||
break;
|
||||
case Payload.Types.PropertyValue.ValueOneofCase.PropertysetValue:
|
||||
AppendProperties(value.PropertysetValue, name, depth + 1, into);
|
||||
break;
|
||||
default:
|
||||
// PropertySetList and extension values are dropped rather than stringified — a
|
||||
// property whose displayed value is not what its type says is worse than an absent
|
||||
// one, and these are documentation, not signal.
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Walks one template level, appending members under <paramref name="prefix"/>.</summary>
|
||||
/// <returns><see langword="false"/> as soon as anything cannot be flattened honestly.</returns>
|
||||
private static bool AppendTemplate(
|
||||
Payload.Types.Template template, string prefix, int depth, List<SparkplugMember> into)
|
||||
{
|
||||
if (depth > MaxDepth)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
// A template DEFINITION carries the UDT's shape with no instance values. It is metadata an
|
||||
// edge node publishes so consumers can understand instances — flattening it would materialise
|
||||
// a parallel set of tags that never carry a value.
|
||||
if (template.HasIsDefinition && template.IsDefinition)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
foreach (var member in template.Metrics)
|
||||
{
|
||||
if (into.Count >= MaxMembers)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
if (!member.HasName || string.IsNullOrEmpty(member.Name))
|
||||
{
|
||||
// A template member is addressed by name; there is no alias fallback inside a template.
|
||||
return false;
|
||||
}
|
||||
|
||||
var path = prefix.Length == 0 ? member.Name : $"{prefix}{MemberSeparator}{member.Name}";
|
||||
|
||||
if (member.ValueCase == Payload.Types.Metric.ValueOneofCase.TemplateValue)
|
||||
{
|
||||
if (!AppendTemplate(member.TemplateValue, path, depth + 1, into))
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
continue;
|
||||
}
|
||||
|
||||
if (member.ValueCase == Payload.Types.Metric.ValueOneofCase.DatasetValue)
|
||||
{
|
||||
if (!TryFlattenDataSet(member.DatasetValue, out var cells)
|
||||
|| into.Count + cells.Count > MaxMembers)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
foreach (var cell in cells)
|
||||
{
|
||||
into.Add(cell with { Path = $"{path}{MemberSeparator}{cell.Path}" });
|
||||
}
|
||||
|
||||
continue;
|
||||
}
|
||||
|
||||
var (kind, value) = SparkplugCodec.ProjectMemberValue(member);
|
||||
if (kind == SparkplugValueKind.Unsupported)
|
||||
{
|
||||
return false; // An extension value inside a template: refuse the whole metric.
|
||||
}
|
||||
|
||||
into.Add(new SparkplugMember(
|
||||
Path: path,
|
||||
DataType: member.HasDatatype ? (TahuDataType)member.Datatype : null,
|
||||
ValueKind: kind,
|
||||
Value: value));
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
private static (SparkplugValueKind Kind, object? Value) ProjectDataSetValue(
|
||||
Payload.Types.DataSet.Types.DataSetValue value) => value.ValueCase switch
|
||||
{
|
||||
Payload.Types.DataSet.Types.DataSetValue.ValueOneofCase.IntValue => (SparkplugValueKind.Scalar, value.IntValue),
|
||||
Payload.Types.DataSet.Types.DataSetValue.ValueOneofCase.LongValue => (SparkplugValueKind.Scalar, value.LongValue),
|
||||
Payload.Types.DataSet.Types.DataSetValue.ValueOneofCase.FloatValue => (SparkplugValueKind.Scalar, value.FloatValue),
|
||||
Payload.Types.DataSet.Types.DataSetValue.ValueOneofCase.DoubleValue => (SparkplugValueKind.Scalar, value.DoubleValue),
|
||||
Payload.Types.DataSet.Types.DataSetValue.ValueOneofCase.BooleanValue => (SparkplugValueKind.Scalar, value.BooleanValue),
|
||||
Payload.Types.DataSet.Types.DataSetValue.ValueOneofCase.StringValue => (SparkplugValueKind.Scalar, value.StringValue),
|
||||
|
||||
// A DataSetValue has no is_null field, so an empty oneof IS the null cell — unlike a Metric,
|
||||
// where Absent and explicit-null are genuinely different states.
|
||||
_ => (SparkplugValueKind.Null, null),
|
||||
};
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// One flattened member of a structured Sparkplug metric (Gitea #515).
|
||||
/// </summary>
|
||||
/// <param name="Path">
|
||||
/// The member path <b>relative to the owning metric</b>, <c>/</c>-joined — <c>Speed</c> for a
|
||||
/// top-level template member, <c>Motor/Speed</c> for a nested one, <c>Temp/0</c> for a dataset
|
||||
/// cell. The consumer composes the full metric name as <c>{parentName}/{Path}</c>; it is kept
|
||||
/// relative here because a DATA metric carries only the parent's alias, so the parent's name is
|
||||
/// not known until the alias table has resolved it.
|
||||
/// </param>
|
||||
/// <param name="DataType">
|
||||
/// The member's declared datatype, or <see langword="null"/> when it carried none — in which case
|
||||
/// the consumer falls back to the datatype the birth installed for this member path.
|
||||
/// </param>
|
||||
/// <param name="ValueKind">What <paramref name="Value"/> means; never <c>Unsupported</c> here.</param>
|
||||
/// <param name="Value">The raw wire value, with the same signed-integer caveat as a scalar metric.</param>
|
||||
public readonly record struct SparkplugMember(
|
||||
string Path,
|
||||
TahuDataType? DataType,
|
||||
SparkplugValueKind ValueKind,
|
||||
object? Value);
|
||||
|
||||
/// <summary>One entry of a metric's <c>PropertySet</c>, projected as node metadata (Gitea #515).</summary>
|
||||
/// <param name="Name">The property key, <c>/</c>-joined for a nested set.</param>
|
||||
/// <param name="Value">
|
||||
/// The boxed scalar value, or <see langword="null"/> for an explicitly-null property.
|
||||
/// </param>
|
||||
public readonly record struct SparkplugProperty(string Name, object? Value);
|
||||
@@ -128,6 +128,36 @@ public sealed class AliasTable
|
||||
aliasCollisions++;
|
||||
(collidedAliases ??= []).Add(alias);
|
||||
}
|
||||
|
||||
// Gitea #515: a Template/DataSet metric also installs one binding per flattened member,
|
||||
// named `{parent}/{memberPath}`. The parent's own binding above is kept — a DATA metric
|
||||
// carries only the parent's ALIAS, so resolving that alias to the parent's name is the only
|
||||
// way the member names can be composed at all. But the parent is not itself servable (its
|
||||
// Sparkplug datatype maps to no DriverDataType), so it is the member bindings that browse
|
||||
// enumerates and that authored tags bind to.
|
||||
//
|
||||
// Members carry NO alias, deliberately: aliases are per-metric on the wire, so inventing
|
||||
// one here would create a number the edge node never assigned, which a later birth could
|
||||
// legitimately hand to a different metric.
|
||||
foreach (var member in metric.MemberList)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(member.Path))
|
||||
{
|
||||
rejectedUnnamed++;
|
||||
continue;
|
||||
}
|
||||
|
||||
var memberBinding = new SparkplugMetricBinding(
|
||||
$"{binding.Name}{SparkplugFlattener.MemberSeparator}{member.Path}",
|
||||
Alias: null,
|
||||
member.DataType ?? TahuDataType.Unknown);
|
||||
|
||||
if (!byName.TryAdd(memberBinding.Name, memberBinding))
|
||||
{
|
||||
duplicateNames++;
|
||||
byName[memberBinding.Name] = memberBinding;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (collidedAliases is not null)
|
||||
|
||||
@@ -1177,6 +1177,15 @@ public sealed class SparkplugIngestor
|
||||
SparkplugMetric metric,
|
||||
DateTime? sourceUtc)
|
||||
{
|
||||
// Gitea #515: a Template/DataSet has no value of its own — its flattened members do. Expand
|
||||
// here, at the single funnel both the DATA and the birth-value paths already pass through, so
|
||||
// neither can end up supporting structured metrics while the other silently does not.
|
||||
if (metric.ValueKind == SparkplugValueKind.Structured)
|
||||
{
|
||||
PublishStructuredMembers(scope, binding, metric, sourceUtc);
|
||||
return;
|
||||
}
|
||||
|
||||
// Bound by NAME. The alias got us here; it is never the key.
|
||||
var key = new SparkplugMetricKey(scope.GroupId, scope.EdgeNodeId, scope.DeviceId, binding.Name);
|
||||
if (!_authored.ByMetric.TryGetValue(key, out var defs))
|
||||
@@ -1287,6 +1296,62 @@ public sealed class SparkplugIngestor
|
||||
: BadTypeMismatch(def, binding, target, stamp, serverUtc);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Publishes each flattened member of a structured (<c>Template</c> / <c>DataSet</c>) metric as
|
||||
/// the sibling metric <c>{parent}/{memberPath}</c> (Gitea #515).
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>Each member is re-entered through <see cref="PublishMetric"/> as a scalar</b>, rather
|
||||
/// than duplicating the coercion, array-gate and authored-tag fan-out here. A second copy of
|
||||
/// that logic would be free to drift — and the drift would be silent, since both paths
|
||||
/// publish values that look plausible.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>The member's datatype falls back to the BIRTH's.</b> A DDATA template usually
|
||||
/// re-declares its members' datatypes, but it is not obliged to; the binding installed at
|
||||
/// birth is the authority when it does not. Guessing from the CLR type of the wire value
|
||||
/// would type a signed Int32 of -42 as a UInt32 of 4294967254 — the exact
|
||||
/// plausible-and-wrong failure <c>ReinterpretSigned</c> exists to prevent.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
private void PublishStructuredMembers(
|
||||
SparkplugScope scope,
|
||||
SparkplugMetricBinding parent,
|
||||
SparkplugMetric metric,
|
||||
DateTime? sourceUtc)
|
||||
{
|
||||
var table = Births.Find(scope);
|
||||
|
||||
foreach (var member in metric.MemberList)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(member.Path))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var name = $"{parent.Name}{SparkplugFlattener.MemberSeparator}{member.Path}";
|
||||
|
||||
// Wire datatype first, birth binding second. Unknown last — which the typed layer refuses
|
||||
// once, loudly, rather than coercing.
|
||||
var dataType = member.DataType
|
||||
?? table?.ResolveByName(name)?.DataType
|
||||
?? Org.Eclipse.Tahu.Protobuf.DataType.Unknown;
|
||||
|
||||
PublishMetric(
|
||||
scope,
|
||||
new SparkplugMetricBinding(name, Alias: null, dataType),
|
||||
new SparkplugMetric(
|
||||
Name: name,
|
||||
Alias: null,
|
||||
DataType: dataType,
|
||||
TimestampMs: metric.TimestampMs,
|
||||
ValueKind: member.ValueKind,
|
||||
Value: member.Value),
|
||||
sourceUtc);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The metric decoded, but does not fit the tag's effective type. Warned once per tag; the value
|
||||
/// itself is deliberately never logged — a payload value is plant data, not diagnostics.
|
||||
@@ -1370,9 +1435,7 @@ public sealed class SparkplugIngestor
|
||||
|
||||
try
|
||||
{
|
||||
subscribers(this, new SparkplugBirthObservedEventArgs(
|
||||
scope,
|
||||
[.. payload.Metrics.Where(m => m.Name is not null).Select(m => m.Name!)]));
|
||||
subscribers(this, new SparkplugBirthObservedEventArgs(scope, [.. DiscoverableNames(payload)]));
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
@@ -1380,6 +1443,38 @@ public sealed class SparkplugIngestor
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The metric names a birth contributes to the <i>discoverable address space</i> — which is
|
||||
/// what the rediscovery change gate must compare, not the raw wire names.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Gitea #515: a structured metric contributes its flattened MEMBERS and not itself. Comparing
|
||||
/// raw wire names instead would make the gate blind to the change that matters most for a UDT:
|
||||
/// a template gaining or losing a member leaves the parent metric's name identical, so the
|
||||
/// signature would not move and the operator would never be told the tree changed.
|
||||
/// </remarks>
|
||||
private static IEnumerable<string> DiscoverableNames(SparkplugPayload payload)
|
||||
{
|
||||
foreach (var metric in payload.Metrics)
|
||||
{
|
||||
if (metric.Name is not { } name)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
if (metric.ValueKind != SparkplugValueKind.Structured)
|
||||
{
|
||||
yield return name;
|
||||
continue;
|
||||
}
|
||||
|
||||
foreach (var member in metric.MemberList)
|
||||
{
|
||||
yield return $"{name}{SparkplugFlattener.MemberSeparator}{member.Path}";
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ---- sequence + rebirth ----
|
||||
|
||||
private SequenceTracker TrackerFor(SparkplugNodeKey node) =>
|
||||
|
||||
Reference in New Issue
Block a user