feat(mtconnect): serve TIME_SERIES observations as variable-length double[] (#627)

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
This commit is contained in:
Joseph Doherty
2026-10-08 14:25:45 -04:00
parent d898262c4a
commit da33f27544
8 changed files with 229 additions and 40 deletions
+35 -7
View File
@@ -230,10 +230,37 @@ Other status codes an observation can surface:
|---|---|
| `BadTypeMismatch` (`0x80740000`) | The Agent's text isn't a value of the tag's coerced type at all |
| `BadOutOfRange` (`0x803C0000`) | The Agent reported a number the coerced type can't represent |
| `BadNotSupported` (`0x803D0000`) | A shape this build doesn't materialize — a `TIME_SERIES` vector, or a structured `DATA_SET`/`TABLE` observation (deferred to P1.5; see [Known limitations](#known-limitations)) |
| `BadDecodingError` (`0x80070000`) | A `TIME_SERIES` vector whose element count disagrees with the observation's own `sampleCount` (or the tag's fixed `ArrayDim`) — see [TIME_SERIES](#time_series-arrays-627) |
| `BadNotSupported` (`0x803D0000`) | A shape this build doesn't materialize — a structured `DATA_SET`/`TABLE` observation (decided: stays unsupported, #627), or an array tag declared as anything other than Float64 |
| `BadWaitingForInitialData` | Tag is authored but the Agent hasn't reported it yet |
| `BadNodeIdUnknown` | DataItem id is neither authored nor ever observed |
### TIME_SERIES arrays (#627)
A `SAMPLE` DataItem with `representation="TIME_SERIES"` is served as a
**variable-length `double[]`** (`ValueRank = 1`, no fixed `ArrayDimensions` —
a live Agent never declares `sampleCount` on the DataItem, see Known
limitations 4; a probe that does declare one yields a fixed `ArrayDim`).
Parsing: the observation text is split on any whitespace, each token parsed
invariant-culture. The result is **all-or-nothing** — one non-numeric /
non-finite token is `BadTypeMismatch`, a literal `UNAVAILABLE` token or an
empty body is `BadNoCommunication`, and a count that disagrees with the
observation's own `sampleCount` attribute (or a fixed `ArrayDim`) is
`BadDecodingError`. A partial vector is never served as Good. Only a
**Float64** array tag is served; any other declared element type stays
`BadNotSupported`. An authored tag needs `"isArray": true` and
`"dataType": "Float64"` in its TagConfig (the AdminUI editor shows an advisory
for any other array type; a browse-commit does not stamp `isArray`, so set it
in the Tag modal's array section).
**Sample rate is not exposed on the served node.** The observation's
`sampleRate` and `sampleCount` attributes are parsed (`MTConnectObservation`)
but the OPC UA node has no description/property slot through `DriverAttributeInfo`,
and a sibling property node would need a new address-space mechanism. Decision:
document it — the rate is a property of the Agent's DataItem (`sampleRate` in
`/probe`), read it there. **`DATA_SET` / `TABLE` stay `BadNotSupported`** (decision, #627): they are
key/value maps with no scalar or array OPC UA shape that preserves the keys.
### Subscribe — the `/sample` pump
One shared `/sample` long-poll stream per driver instance (the Agent streams
@@ -410,7 +437,7 @@ for anything beyond values-and-conditions.
`/probe` metadata a browse-commit stamped (`mtCategory`/`mtType`), and shows an
advisory line — it never blocks a save (the deploy gate does not refuse it) and
is a no-op for hand-authored blobs with no stamped metadata. `representation`
is not on the model, so TIME_SERIES arrays are not covered. Beyond that,
is not on the model; TIME_SERIES arrays get their own `isArray` advisory (declare Float64). Beyond that,
nothing validates a raw tag's declared OPC UA `DataType` against the
blob's coercion type. The `/probe`-sourced inference
(`MTConnectDataTypeInference`) narrows how often an author picks a wrong
@@ -427,9 +454,11 @@ for anything beyond values-and-conditions.
declare a `sampleCount`, is unrealistic on this point and exists only to
pin the parsing rule itself.
5. **CONDITION is a plain `String`, not a native Part 9 alarm** (see above).
6. **`TIME_SERIES` arrays and structured `DATA_SET`/`TABLE` observations
surface as `BadNotSupported`**, not an approximation — see the Quality
mapping table.
6. **Structured `DATA_SET`/`TABLE` observations surface as
`BadNotSupported`**, not an approximation — see the Quality mapping table.
`TIME_SERIES` is served as `double[]` (#627, see
[TIME_SERIES arrays](#time_series-arrays-627)); the sample rate is not
surfaced on the node.
## Deferred (not in this build)
@@ -447,8 +476,7 @@ for anything beyond values-and-conditions.
- **P1.5 fast-follow** (design
[§9](../plans/2026-07-15-mtconnect-driver-design.md#9-phasing--effort)):
CONDITION → native OPC UA Part 9 alarms via `IAlarmSource` (the Galaxy
native-alarm pattern); `TIME_SERIES` SAMPLE arrays materialized as real
OPC UA arrays; EVENT controlled-vocabulary values → OPC UA enumerations.
native-alarm pattern); EVENT controlled-vocabulary values → OPC UA enumerations.
- **P2 (on demand): SHDR adapter ingest.** A `SourceMode: "Agent" | "Shdr"`
switch that opens the raw pipe-delimited SHDR TCP socket directly. Loses
auto-discovery (no device model without an Agent in front, so
@@ -158,11 +158,22 @@ public sealed record MTConnectStreamsResult(
/// <c>false</c>, the shape of every ordinary scalar observation.
/// </para>
/// </param>
/// <param name="SampleCount">
/// The observation's own <c>sampleCount</c> attribute — the number of values a <c>TIME_SERIES</c>
/// vector claims to carry (#627). The index checks the parsed vector against it. <c>null</c> when absent.
/// </param>
/// <param name="SampleRate">
/// The observation's own <c>sampleRate</c> attribute (samples per second) on a <c>TIME_SERIES</c>
/// observation. Carried for the parser contract; not yet surfaced on the served node (see
/// docs/drivers/MTConnect.md). <c>null</c> when absent.
/// </param>
public sealed record MTConnectObservation(
string DataItemId,
string Value,
DateTime TimestampUtc,
bool IsStructured = false);
bool IsStructured = false,
int? SampleCount = null,
double? SampleRate = null);
/// <summary>
/// Recursive helpers over the <see cref="IMTConnectComponentContainer"/> device/component tree.
@@ -49,8 +49,8 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.MTConnect;
/// </para>
/// <para>
/// <b>Shapes this build does not represent</b> are coded <c>BadNotSupported</c> with a null
/// value rather than approximated: a <c>TIME_SERIES</c> vector (array materialization is
/// deferred to P1.5) and a <c>DATA_SET</c> / <c>TABLE</c> observation. The latter carries its
/// value rather than approximated: a non-Float64 array tag and a <c>DATA_SET</c> / <c>TABLE</c>
/// observation (a Float64 <c>TIME_SERIES</c> vector is served as <c>double[]</c>, #627). The latter carries its
/// real content in <c>&lt;Entry key=…&gt;</c> child elements, which reach a text-valued
/// observation concatenated with the keys discarded — a two-entry set reads as the single
/// token <c>"12"</c> — and because the type inference correctly demotes those
@@ -85,9 +85,12 @@ internal sealed class MTConnectObservationIndex
/// <summary>The Agent reported a number the authored type cannot represent.</summary>
private const uint BadOutOfRange = 0x803C0000u;
/// <summary>The observation's shape is real data this build cannot represent (TIME_SERIES vectors).</summary>
/// <summary>The observation's shape is real data this build cannot represent (a DATA_SET/TABLE, or a non-Float64 array tag).</summary>
private const uint BadNotSupported = 0x803D0000u;
/// <summary>A TIME_SERIES vector whose element count disagrees with its declared <c>sampleCount</c>.</summary>
private const uint BadDecodingError = 0x80070000u;
/// <summary>The Agent reported text that is not a value of the authored type at all.</summary>
private const uint BadTypeMismatch = 0x80740000u;
@@ -268,13 +271,11 @@ internal sealed class MTConnectObservationIndex
_tags.TryGetValue(observation.DataItemId, out var tag);
// TIME_SERIES vectors arrive as one space-separated string. P1 does not materialize OPC UA
// arrays (deferred to P1.5), and the honest answer is "real data I cannot represent" — not a
// type mismatch, and emphatically not the first element parsed out and served as Good.
// Checked AFTER the sentinel so an unavailable array tag still reports the truthful no-comms.
// TIME_SERIES vectors arrive as one whitespace-separated string (#627). Checked AFTER the
// sentinel so an unavailable array tag still reports the truthful no-comms.
if (tag is { IsArray: true })
{
return new DataValueSnapshot(null, BadNotSupported, sourceTimestamp, serverTimestamp);
return BuildArraySnapshot(observation, tag, sourceTimestamp, serverTimestamp);
}
// An unauthored observation has no declared target type; String is the only coercion that
@@ -286,6 +287,40 @@ internal sealed class MTConnectObservationIndex
: new DataValueSnapshot(null, status, sourceTimestamp, serverTimestamp);
}
/// <summary>
/// Parses a <c>TIME_SERIES</c> vector into a <c>double[]</c>. Only <see cref="DriverDataType.Float64"/>
/// arrays are served (the MTConnect TIME_SERIES element type is a floating-point sample); any other
/// authored array type is <c>BadNotSupported</c> rather than an approximation. All-or-nothing: one
/// unparseable or non-finite token, or a count that disagrees with the observation's own
/// <c>sampleCount</c> (or the tag's fixed <c>ArrayDim</c>), makes the whole vector Bad — a partial
/// vector served as Good would be a silently wrong waveform.
/// </summary>
private static DataValueSnapshot BuildArraySnapshot(
MTConnectObservation observation, MTConnectTagDefinition tag, DateTime sourceTimestamp, DateTime serverTimestamp)
{
DataValueSnapshot Bad(uint status) => new(null, status, sourceTimestamp, serverTimestamp);
if (tag.DriverDataType != DriverDataType.Float64) return Bad(BadNotSupported);
var tokens = observation.Value.Split((char[]?)null, StringSplitOptions.RemoveEmptyEntries);
var values = new double[tokens.Length];
for (var i = 0; i < tokens.Length; i++)
{
if (string.Equals(tokens[i], UnavailableSentinel, StringComparison.Ordinal)) return Bad(BadNoCommunication);
if (!double.TryParse(tokens[i], NumberStyles.Float, CultureInfo.InvariantCulture, out values[i]) ||
!double.IsFinite(values[i]))
{
return Bad(BadTypeMismatch);
}
}
if (observation.SampleCount is { } declared && declared != values.Length) return Bad(BadDecodingError);
if (tag.ArrayDim > 0 && tag.ArrayDim != values.Length) return Bad(BadDecodingError);
return new DataValueSnapshot(values, Good, sourceTimestamp, serverTimestamp);
}
/// <summary>
/// Converts the Agent's text to the tag's authored type. Every parse is
/// <see cref="CultureInfo.InvariantCulture"/>: MTConnect is an invariant-format wire, and a
@@ -377,6 +377,8 @@ internal static class MTConnectStreamsParser
string? dataItemId = null;
var timestampUtc = default(DateTime);
string? conditionState = null;
int? sampleCount = null;
double? sampleRate = null;
if (isObservation)
{
@@ -385,6 +387,24 @@ internal static class MTConnectStreamsParser
timestampUtc = ReadTimestampUtc(reader, dataItemId);
conditionState = isCondition ? ConditionState(name) : null;
// TIME_SERIES metadata rides the observation element itself. Read here, before the subtree
// walk moves the reader off this element's attributes. Lenient: a vendor's unparseable
// attribute is dropped rather than failing the whole document (the vector is checked
// against the declared sampleCount only when one parses).
if (!isCondition)
{
sampleCount = int.TryParse(
MTConnectXml.OptionalAttribute(reader, "sampleCount"),
NumberStyles.Integer,
CultureInfo.InvariantCulture,
out var count) && count >= 0 ? count : null;
sampleRate = double.TryParse(
MTConnectXml.OptionalAttribute(reader, "sampleRate"),
NumberStyles.Float,
CultureInfo.InvariantCulture,
out var rate) && double.IsFinite(rate) ? rate : null;
}
// The slot is reserved BEFORE the subtree is read, and filled after: an observation is
// only complete once its text and its child-element flag are known, but appending it
// then would report a nested observation ahead of the one enclosing it. The tree walk
@@ -430,7 +450,9 @@ internal static class MTConnectStreamsParser
// MTConnectObservation.IsStructured. Never true for a CONDITION: its value comes from
// the element NAME, so child content cannot corrupt it, and flagging one would throw
// away a perfectly well-determined state.
IsStructured: !isCondition && hasElements);
IsStructured: !isCondition && hasElements,
SampleCount: sampleCount,
SampleRate: sampleRate);
}
/// <summary>
@@ -229,10 +229,25 @@ public sealed class MTConnectTagConfigModel
/// passed as null; it can only DEMOTE an item to String (DATA_SET/TABLE), which cannot create a false
/// warning because a numeric declaration on a String-inferred item is warned either way, and a SAMPLE
/// inferred Float64 is only warned about for a non-numeric, non-String declaration or an integer one.
/// TIME_SERIES arrays are not detectable without it and are not covered.
/// TIME_SERIES arrays are covered separately by the isArray rule in <see cref="Warnings"/>.
/// </remarks>
/// <returns>Zero or more human-readable warnings.</returns>
public IReadOnlyList<string> Warnings()
{
// #627: an array (TIME_SERIES) tag is served as double[] only. Independent of the probe metadata —
// the array flag is authored on the tag, and any other declared element type reads BadNotSupported.
var arrayWarning = _bag.TryGetPropertyValue("isArray", out var arr)
&& arr is JsonValue av && av.TryGetValue<bool>(out var isArray) && isArray
&& DataType != DriverDataType.Float64
? $"Data type {DataType} on an array tag: MTConnect TIME_SERIES arrays are served as Float64 (double[]) only; "
+ "declare Float64 or the tag reads BadNotSupported. Saving is still allowed."
: null;
var scalar = ScalarWarnings();
return arrayWarning is null ? scalar : [arrayWarning, .. scalar];
}
private IReadOnlyList<string> ScalarWarnings()
{
if (string.IsNullOrWhiteSpace(MtCategory)) { return []; }
@@ -30,6 +30,7 @@ public sealed class MTConnectObservationIndexTests
private const uint BadOutOfRange = 0x803C0000u;
private const uint BadNotSupported = 0x803D0000u;
private const uint BadTypeMismatch = 0x80740000u;
private const uint BadDecodingError = 0x80070000u;
/// <summary>Every dataItemId both /current fixtures report.</summary>
private static readonly string[] AllFixtureDataItemIds =
@@ -48,7 +49,7 @@ public sealed class MTConnectObservationIndexTests
/// <summary>Wraps one synthetic observation in the minimal legal streams result.</summary>
private static MTConnectStreamsResult Synthetic(
string dataItemId, string value, DateTime? timestampUtc = null, bool isStructured = false) =>
string dataItemId, string value, DateTime? timestampUtc = null, bool isStructured = false, int? sampleCount = null) =>
new(
InstanceId: 1L,
NextSequence: 2L,
@@ -57,7 +58,8 @@ public sealed class MTConnectObservationIndexTests
dataItemId,
value,
timestampUtc ?? new DateTime(2026, 7, 24, 12, 0, 0, DateTimeKind.Utc),
isStructured)]);
isStructured,
sampleCount)]);
private static MTConnectTagDefinition Tag(string id, DriverDataType type) => new(id, type);
@@ -412,13 +414,9 @@ public sealed class MTConnectObservationIndexTests
// -------------------------------------------------------------------------------- TIME_SERIES
/// <summary>
/// P1 defers TIME_SERIES array materialization to P1.5. An array-authored tag therefore
/// reports BadNotSupported rather than pretending: the vector is real data this build
/// cannot represent, which is neither a type mismatch nor a comms failure.
/// </summary>
/// <summary>#627: a Float64 array tag serves the whole vector as <c>double[]</c>, Good.</summary>
[Fact]
public void Time_series_authored_as_an_array_reports_BadNotSupported()
public void Time_series_authored_as_a_float64_array_parses_to_double_array()
{
var idx = new MTConnectObservationIndex(
[new MTConnectTagDefinition("dev1_vibration_ts", DriverDataType.Float64, IsArray: true, ArrayDim: 10)]);
@@ -426,25 +424,93 @@ public sealed class MTConnectObservationIndexTests
var snap = idx.Get("dev1_vibration_ts");
snap.StatusCode.ShouldBe(BadNotSupported);
snap.StatusCode.ShouldBe(Good);
snap.Value.ShouldBeOfType<double[]>()
.ShouldBe([1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 2.0]);
}
/// <summary>The variable-length shape (no declared ArrayDim) is served the same way.</summary>
[Fact]
public void Time_series_without_a_declared_length_is_served_variable_length()
{
var idx = new MTConnectObservationIndex(
[new MTConnectTagDefinition("ts", DriverDataType.Float64, IsArray: true)]);
idx.Apply(Synthetic("ts", "1 2.5\n 3"));
var snap = idx.Get("ts");
snap.StatusCode.ShouldBe(Good);
snap.Value.ShouldBeOfType<double[]>().ShouldBe([1.0, 2.5, 3.0]);
}
[Fact]
public void Time_series_count_mismatching_the_observations_sampleCount_is_Bad()
{
var idx = new MTConnectObservationIndex(
[new MTConnectTagDefinition("ts", DriverDataType.Float64, IsArray: true)]);
idx.Apply(Synthetic("ts", "1 2 3", sampleCount: 4));
var snap = idx.Get("ts");
snap.StatusCode.ShouldBe(BadDecodingError);
snap.Value.ShouldBeNull();
}
/// <summary>
/// The falsifiable half of the deferral: the vector must never be silently reduced to its
/// first element. A Good 1.1 here would be a wrong value an operator would trust.
/// </summary>
[Fact]
public void Time_series_vector_is_never_silently_parsed_as_its_first_element()
public void Time_series_count_mismatching_a_fixed_ArrayDim_is_Bad()
{
var idx = new MTConnectObservationIndex(
[new MTConnectTagDefinition("dev1_vibration_ts", DriverDataType.Float64, IsArray: true, ArrayDim: 10)]);
[new MTConnectTagDefinition("ts", DriverDataType.Float64, IsArray: true, ArrayDim: 5)]);
idx.Apply(Synthetic("ts", "1 2 3"));
idx.Get("ts").StatusCode.ShouldBe(BadDecodingError);
}
[Theory]
[InlineData("1 2 abc")]
[InlineData("1 NaN 3")]
[InlineData("1 UNAVAILABLE 3")]
public void Time_series_with_an_unusable_token_is_Bad_never_a_partial_vector(string text)
{
var idx = new MTConnectObservationIndex(
[new MTConnectTagDefinition("ts", DriverDataType.Float64, IsArray: true)]);
idx.Apply(Synthetic("ts", text));
var snap = idx.Get("ts");
snap.StatusCode.ShouldNotBe(Good);
snap.Value.ShouldBeNull();
}
[Fact]
public void Empty_time_series_reports_no_communication()
{
var idx = new MTConnectObservationIndex(
[new MTConnectTagDefinition("ts", DriverDataType.Float64, IsArray: true)]);
idx.Apply(Synthetic("ts", " "));
idx.Get("ts").StatusCode.ShouldBe(BadNoCommunication);
}
[Fact]
public void A_non_float64_array_tag_stays_BadNotSupported()
{
var idx = new MTConnectObservationIndex(
[new MTConnectTagDefinition("dev1_vibration_ts", DriverDataType.Int32, IsArray: true)]);
idx.Apply(ParseFixture(CurrentFixture));
var snap = idx.Get("dev1_vibration_ts");
idx.Get("dev1_vibration_ts").StatusCode.ShouldBe(BadNotSupported);
}
snap.Value.ShouldNotBe(1.1);
snap.StatusCode.ShouldNotBe(Good);
/// <summary>The parser carries the observation's own sampleCount / sampleRate attributes.</summary>
[Fact]
public void Parser_carries_time_series_sampleCount_and_sampleRate()
{
var obs = ParseFixture(CurrentFixture).Observations.Single(o => o.DataItemId == "dev1_vibration_ts");
obs.SampleCount.ShouldBe(10);
obs.SampleRate.ShouldBe(100.0);
ParseFixture(CurrentFixture).Observations.Single(o => o.DataItemId == "dev1_pos").SampleCount.ShouldBeNull();
}
/// <summary>
@@ -452,7 +518,7 @@ public sealed class MTConnectObservationIndexTests
/// representable state and must report it rather than the deferral code.
/// </summary>
[Fact]
public void Unavailable_wins_over_the_unsupported_array_shape()
public void Unavailable_array_tag_reports_no_communication()
{
var idx = new MTConnectObservationIndex(
[new MTConnectTagDefinition("dev1_vibration_ts", DriverDataType.Float64, IsArray: true, ArrayDim: 10)]);
@@ -219,15 +219,16 @@ public sealed class MTConnectReadTests
res[0].SourceTimestampUtc.ShouldNotBeNull();
}
/// <summary>A TIME_SERIES vector is real data this build cannot represent (P1.5 deferral).</summary>
/// <summary>A probe-declared TIME_SERIES item is read end to end as a <c>double[]</c> (#627).</summary>
[Fact]
public async Task Read_of_a_time_series_ref_reports_BadNotSupported()
public async Task Read_of_a_time_series_ref_returns_the_double_vector()
{
var (driver, _) = await InitializedDriverAsync();
var res = await driver.ReadAsync(["dev1_vibration_ts"], TestContext.Current.CancellationToken);
res[0].StatusCode.ShouldBe(BadNotSupported);
res[0].StatusCode.ShouldBe(0u);
res[0].Value.ShouldBeOfType<double[]>().Length.ShouldBe(10);
}
/// <summary>A blank reference is data, not a programming error — it codes Bad, it does not throw.</summary>
@@ -328,4 +328,15 @@ public sealed class MTConnectTagConfigModelTests
public void Warnings_is_a_noop_without_probe_metadata()
=> MTConnectTagConfigModel.FromJson("{\"fullName\":\"x\",\"dataType\":\"Boolean\"}")
.Warnings().ShouldBeEmpty();
[Fact]
public void Warnings_accepts_a_float64_array_for_time_series_and_flags_other_array_types()
{
MTConnectTagConfigModel.FromJson("{\"fullName\":\"x\",\"dataType\":\"Float64\",\"isArray\":true}")
.Warnings().ShouldBeEmpty();
var m = MTConnectTagConfigModel.FromJson("{\"fullName\":\"x\",\"dataType\":\"Int32\",\"isArray\":true}");
m.Warnings().ShouldHaveSingleItem().ShouldContain("Float64");
m.Validate().ShouldBeNull(); // advisory only
}
}