Files
lmxopcua/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect/MTConnectDriver.cs
T
Joseph Doherty 8a6f0e3c84 fix(mtconnect): the Agent refuses count ABOVE bufferSize, not at it (#825)
Live gate against the cppagent 2.7 fixture: count=16384 with bufferSize
16384 streams; 16385 is OUT_OF_RANGE ('count' must be less than 16385).
Warn only when SampleCount > bufferSize, pin both sides of the edge
against the live Agent, and correct the #804 wording that said 'at or
above'.
2026-11-11 01:38:48 -05:00

4514 lines
227 KiB
C#
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
using System.Collections.Frozen;
using System.Globalization;
using System.Text.Json;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.MTConnect;
/// <summary>
/// MTConnect Agent implementation of <see cref="IDriver"/> — the lifecycle half. Everything the
/// driver serves sits behind one <see cref="IMTConnectAgentClient"/>, so the whole class is
/// exercised against canned XML with no socket.
/// </summary>
/// <remarks>
/// <para>
/// <b>The constructor opens nothing and builds nothing.</b> It does not even call
/// <c>agentClientFactory</c> — the universal discovery browser constructs a throwaway
/// instance purely to ask whether the driver can browse, and a constructor that materialized
/// a client would open a connection pool per browse probe against a possibly-unreachable
/// Agent. Every connection is owned by <see cref="InitializeAsync"/>.
/// </para>
/// <para>
/// <b>The <c>driverConfigJson</c> argument is honoured, not decorative.</b> <see cref="IDriver.InitializeAsync"/>
/// and <see cref="ReinitializeAsync"/> both take the document, and the contract is that the driver
/// serves it — a driver that read only its constructor options would turn an in-place config
/// change into a silent no-op. (The runtime itself now applies a config change by respawning,
/// .) This class therefore
/// re-parses the document (<see cref="ParseOptions"/>) whenever it carries a real body,
/// matching S7/AbCip/TwinCAT/Galaxy rather than the Modbus/FOCAS/OpcUaClient drivers that
/// ignore the argument. A blank / <c>{}</c> / <c>[]</c> document means "no config supplied"
/// and keeps the constructor options, which is what unit tests and the factory path rely on.
/// </para>
/// <para>
/// <b>Failure is never dressed as success.</b> A failed initialize disposes whatever
/// client it built, drops the probe cache and the observation index, publishes
/// <see cref="DriverState.Faulted"/> with the error text, and rethrows — the runtime's
/// retry/backoff loop is the recovery path. In particular a <c>/probe</c> that succeeds while
/// the priming <c>/current</c> fails is Faulted, not "Healthy with no data": the index IS the
/// read surface, so serving that state would render every tag
/// <c>BadWaitingForInitialData</c> indefinitely behind a green driver, and the
/// <c>/sample</c> cursor only exists because <c>/current</c> reported a
/// <c>nextSequence</c>.
/// </para>
/// <para>
/// <b>Scope.</b> This type implements <see cref="IDriver"/>, <see cref="IReadable"/>,
/// <see cref="IWritable"/>, <see cref="ISubscribable"/>, <see cref="ITagDiscovery"/>,
/// <see cref="IHostConnectivityProbe"/> and <see cref="IRediscoverable"/> — the last two
/// built on state this class already caches (the Agent <c>instanceId</c> and the probe
/// model) — plus <see cref="IAlarmSource"/> for CONDITION data items (in
/// <c>MTConnectDriver.Alarms.cs</c>).
/// </para>
/// <para>
/// <b><see cref="IWritable"/> is <i>observation injection</i>, not machine command, and it is
/// off by default.</b> A write goes to the Agent's <c>AllowPut</c> surface
/// (<c>POST {agent}/{device}?{dataItemId}={value}</c>) and sets what the Agent <i>reports</i>
/// for that DataItem — visible to every consumer of that Agent, and overwritten by the next
/// adapter/SHDR update. Nothing reaches the CNC. It is therefore gated twice, and off at both
/// gates by default: the Agent's own <c>AllowPut</c> (<c>agent.cfg</c>) and this driver's
/// <see cref="MTConnectWriteOptions.Enabled"/>. Either gate alone leaves the shipped read-only
/// behaviour byte-for-byte unchanged. CONDITION data items are excluded — they carry alarm
/// semantics and a different <c>level|text</c> wire format. MTConnect <b>Interfaces</b>, the
/// protocol's own request/response handshake, remains deferred and is a different mechanism
/// entirely (see <c>docs/plans/2026-08-05-mtconnect-writeback-design.md</c> §2.1).
/// </para>
/// <para>
/// <b>The read path never writes the shared observation index</b> — see
/// <see cref="ReadAsync"/>. That index has exactly one writer: the <c>/sample</c> pump
/// (<see cref="RunSampleStreamAsync"/>). The same single-writer discipline covers the two
/// connectivity-probe fields: the <b>connectivity probe loop</b> is the sole writer of
/// <see cref="HostState"/>, and it publishes <b>nothing</b> into
/// <see cref="AgentSession.ProbeModel"/> (see <see cref="RunProbeLoopAsync"/>).
/// </para>
/// </remarks>
public sealed partial class MTConnectDriver
: IDriver, IReadable, IWritable, ISubscribable, ITagDiscovery, IHostConnectivityProbe, IRediscoverable,
ISkippedTagReporter
{
/// <summary>Authored tags this driver could not bind to an Agent DataItem.</summary>
private readonly SkippedTagLedger _skippedTags = new();
/// <inheritdoc />
public SkippedTagReport? GetSkippedTags() => _skippedTags.Snapshot();
/// <summary>
/// The Agent could not be reached, answered unusably, or blew the per-call deadline. Distinct
/// from the index's <c>BadNoCommunication</c>, which means the Agent answered and said the
/// machine has no value — the two failures need different operator responses. On the write
/// path it also covers an Agent that answered and <i>refused</i> — see <see cref="WriteAsync"/>.
/// </summary>
private const uint BadCommunicationError = 0x80050000u;
// ---- Write status codes ----
//
// NAMED constants, never inline literals: StatusCodeParityTests checks each against the pinned SDK
// by reflection and can only see a named const. The names must match
// Opc.Ua.StatusCodes members exactly. Spelled WITHOUT the "Status" prefix MqttDriver uses, to match
// the two this file already declares (the parity guard strips either spelling).
/// <summary>
/// The reference names no DataItem this driver serves, or names one whose owning device cannot
/// be resolved — there is no Agent URL a write could be addressed to.
/// </summary>
private const uint BadNodeIdUnknown = 0x80340000u;
/// <summary>
/// Writing is switched off (<c>write.enabled</c>), or the DataItem is CONDITION-category. The
/// honest answer to "can I write this?", and deliberately not a communication failure: nothing
/// was attempted and no Agent was involved.
/// </summary>
private const uint BadNotWritable = 0x803B0000u;
/// <summary>
/// The supplied value has no invariant string form a PUT could carry — <c>null</c>, or a
/// non-scalar (see <see cref="TryFormatValue"/>).
/// </summary>
private const uint BadTypeMismatch = 0x80740000u;
/// <summary>
/// There is no Agent client at all: the driver has not been initialized, its initialize
/// faulted, or it has been shut down. Deliberately not one of the index's codes — none of
/// them would say "this driver is not running", which is the only true statement available.
/// </summary>
private const uint BadNotConnected = 0x808A0000u;
/// <summary>
/// Floor the reconnect backoff grows from, in milliseconds. Load-bearing: the shipped
/// <see cref="MTConnectReconnectOptions.MinBackoffMs"/> default is <b>0</b> (an immediate
/// first retry, which is what an operator wants after a one-off blip) and
/// <c>0 × multiplier</c> is still 0 — so without a floor, the "geometric" backoff would be
/// an unbounded hot loop against an Agent that stays down. Mirrors
/// <c>ModbusTcpTransport.ConnectWithBackoffAsync</c>.
/// </summary>
private const int BackoffGrowthFloorMs = 100;
/// <summary>Growth factor used when the authored multiplier could not grow the delay (≤ 1).</summary>
private const double DefaultBackoffMultiplier = 2.0;
/// <summary>
/// Backoff cap used when <see cref="MTConnectReconnectOptions.MaxBackoffMs"/> is authored
/// non-positive. Treated as "unset" rather than as "no cap" for the same reason a
/// multiplier ≤ 1 falls back to <see cref="DefaultBackoffMultiplier"/>: a zero cap clamps
/// EVERY delay to zero, which turns the reconnect loop into a spin against a refused
/// connection — one Warning per iteration, as fast as the socket can fail. Mirrors the
/// shipped <see cref="MTConnectReconnectOptions.MaxBackoffMs"/> default.
/// </summary>
private const int DefaultMaxBackoffMs = 30_000;
/// <summary>
/// How long teardown waits for a background loop (the <c>/sample</c> pump, the connectivity
/// probe) to unwind before abandoning the wait and carrying on.
/// </summary>
/// <remarks>
/// <b>An unbounded wait here is not merely slow — it wedges an actor-system thread.</b>
/// <c>DriverInstanceActor.PostStop</c> calls <see cref="ShutdownAsync"/> with
/// <c>GetAwaiter().GetResult()</c>, i.e. a synchronous block on an Akka dispatcher thread,
/// while this driver holds <see cref="_lifecycle"/>. So a subscriber callback that never
/// returns would take the dispatcher thread AND every subsequent lifecycle call on this
/// driver with it, permanently. Five seconds matches the budget
/// <c>DriverInstanceActor</c> already puts around <see cref="UnsubscribeAsync"/>.
/// </remarks>
private static readonly TimeSpan TeardownWait = TimeSpan.FromSeconds(5);
/// <summary>
/// How many consecutive <c>/current</c> re-baselines with no <c>/sample</c> chunk delivered between them
/// report the stream Degraded (#804; the Java driver's <c>MAX_REBASELINES_WITHOUT_PROGRESS</c>). One
/// re-baseline is a genuine ring-buffer overrun resolving; three in a row is an Agent and a cursor/count
/// that cannot agree.
/// </summary>
internal const int MaxRebaselinesWithoutProgress = 3;
/// <summary>The type-only <c>LastError</c> a stalled stream (<see cref="MaxRebaselinesWithoutProgress"/>) reports.</summary>
internal const string StalledStreamReason = "repeated /sample re-baselines with no observations in between";
/// <summary>
/// Config-JSON reader options, mirroring the sibling driver factories. Note there is
/// deliberately <b>no</b> <c>JsonStringEnumConverter</c>: enum-carrying DTO fields stay
/// <c>string?</c> and go through <see cref="ParseEnum{T}"/>, which is the project-wide
/// guard against the numerically-serialized-enum trap that faults a driver at deploy time.
/// </summary>
private static readonly JsonSerializerOptions JsonOptions = new()
{
PropertyNameCaseInsensitive = true,
ReadCommentHandling = JsonCommentHandling.Skip,
AllowTrailingCommas = true,
};
/// <summary>
/// Reader options for the emptiness probe in <see cref="HasConfigBody"/>. Kept in step with
/// <see cref="JsonOptions"/> so a document the real parse would accept is never judged
/// malformed here (a stray trailing comma must not change which branch a config takes).
/// </summary>
private static readonly JsonDocumentOptions DocumentOptions = new()
{
CommentHandling = JsonCommentHandling.Skip,
AllowTrailingCommas = true,
};
private readonly string _driverInstanceId;
private readonly ILogger<MTConnectDriver> _logger;
private readonly Func<MTConnectDriverOptions, IMTConnectAgentClient> _agentClientFactory;
/// <summary>
/// How the connectivity probe loop waits between ticks. Injected so the cadence is a seam
/// rather than a wall clock: the loop is a timer, and a suite that drove it by sleeping would
/// be flaky by construction. Defaults to <see cref="Task.Delay(TimeSpan, CancellationToken)"/>.
/// </summary>
private readonly Func<TimeSpan, CancellationToken, Task> _probeScheduler;
/// <summary>
/// Serializes the three lifecycle entry points against each other. Initialize / Reinitialize
/// / Shutdown all swap the client and the served caches; overlapping them would let a
/// shutdown dispose a client an in-flight initialize is about to publish.
/// </summary>
/// <remarks>
/// <para>
/// <b>⚠️ NON-REENTRANT — never call a public lifecycle method from code that already
/// holds this.</b> <see cref="SemaphoreSlim"/> has no owner tracking, so a re-entrant
/// call does not fail: it <b>deadlocks the driver permanently</b>, with no exception and
/// no log line to diagnose it from.
/// </para>
/// <para>
/// <b>This is aimed squarely at the <c>/sample</c> pump.</b> Its ring-buffer
/// re-baseline ("on a sequence gap, re-<c>/current</c> and resume") is naturally written
/// as "just call the existing re-init logic" — and
/// <see cref="ReinitializeAsync"/>/<see cref="InitializeAsync"/> both take this
/// semaphore, so from inside the pump loop that is a hang, not a re-prime. Re-baseline
/// must instead call <see cref="IMTConnectAgentClient.CurrentAsync"/> directly on the
/// client the pump already holds (through <see cref="BoundedAsync"/>, to keep the
/// deadline) and <c>Apply</c> the result to <see cref="ObservationIndex"/>. Any future
/// shared step belongs in a <b>private, non-locking</b> helper that both the pump and the
/// lock-holding lifecycle methods can call — never in the public entry points.
/// </para>
/// <para>
/// The inverse direction is already safe: the lifecycle methods call
/// <see cref="StopSampleStreamAsync"/> while holding this, so that method — and whatever
/// the sample pump fills it with — must never re-enter the lifecycle either.
/// </para>
/// </remarks>
private readonly SemaphoreSlim _lifecycle = new(1, 1);
/// <summary>
/// Guards the subscription registry and the pump's control block (<see cref="_pumpCts"/>,
/// <see cref="_pumpTask"/>, <see cref="_streamUnsupported"/>).
/// </summary>
/// <remarks>
/// <para>
/// <b>Lock order is <see cref="_lifecycle"/> → this, never the reverse.</b> The lifecycle
/// methods take this lock (to start/stop the pump) while holding the semaphore;
/// <see cref="UnsubscribeAsync"/> therefore releases this lock <i>before</i> awaiting
/// <see cref="StopSampleStreamAsync"/>, and nothing under this lock ever waits on
/// <see cref="_lifecycle"/>.
/// </para>
/// <para>
/// <b>Nothing is awaited while it is held, and no subscriber callback is raised under
/// it.</b> A handler is caller code: it may block, throw, or re-enter this driver, and
/// holding a lock across it would turn a slow subscriber into a stalled pump.
/// </para>
/// </remarks>
private readonly Lock _subscriptionLock = new();
/// <summary>Live subscriptions by id. Guarded by <see cref="_subscriptionLock"/>.</summary>
private readonly Dictionary<long, SubscriptionState> _subscriptions = [];
/// <summary>
/// Guards the two host-connectivity fields, which the probe loop writes and
/// <see cref="GetHostStatuses"/> reads from an arbitrary caller thread. Held for a field
/// read/write only — <b>never</b> across an await, and never while a status handler runs.
/// </summary>
private readonly Lock _probeLock = new();
/// <summary>
/// The Agent's last observed reachability. Written <b>only</b> by
/// <see cref="RunProbeLoopAsync"/>, so the state and the event stream can never disagree:
/// every value this field has ever held was announced by exactly one
/// <see cref="OnHostStatusChanged"/>. That is why a successful <see cref="InitializeAsync"/>
/// does not seed it <see cref="HostState.Running"/> — raising the event from there would run
/// caller code while the non-reentrant <see cref="_lifecycle"/> semaphore is held (the same
/// hazard that makes the pump, not <see cref="StartCoreAsync"/>, republish to subscribers),
/// and seeding it silently would leave a consumer that tracks the event stream disagreeing
/// with <see cref="GetHostStatuses"/>. <see cref="HostState.Unknown"/> until the first tick
/// is the honest answer. Guarded by <see cref="_probeLock"/>.
/// </summary>
private HostState _hostState = HostState.Unknown;
/// <summary>When <see cref="_hostState"/> last changed. Guarded by <see cref="_probeLock"/>.</summary>
private DateTime _hostStateChangedUtc = DateTime.UtcNow;
/// <summary>
/// Cancels the connectivity probe loop. Written only under <see cref="_lifecycle"/>, which
/// serializes every start and stop of the loop.
/// </summary>
private CancellationTokenSource? _probeCts;
/// <summary>The connectivity probe loop's task, or <c>null</c> when none runs. See <see cref="_probeCts"/>.</summary>
private Task? _probeTask;
/// <summary>
/// The Agent <c>instanceId</c> the last <see cref="OnRediscoveryNeeded"/> was raised for —
/// seeded at the commit point of <see cref="StartCoreAsync"/> with the id the priming
/// <c>/current</c> reported. This is what makes rediscovery fire once per <b>change</b>
/// rather than once per chunk: every chunk after a restart carries the new id, and a
/// re-baseline that fails leaves the pump still holding the old one.
/// </summary>
private long _announcedInstanceId;
/// <summary>
/// The options document currently in force. Installed at the <c>StartCoreAsync</c> commit
/// point alongside <see cref="_binding"/> / <see cref="_session"/> /
/// <see cref="_announcedInstanceId"/>, and read lock-free from <c>ReadAsync</c>,
/// <c>WriteAsync</c>, <c>DiscoverAsync</c>, <c>HostName</c> and the health paths.
/// <para>
/// <b>Published with <c>Volatile.Write</c> and read with <c>Volatile.Read</c>, exactly
/// like its siblings</b>, so a reader that
/// takes this field BEFORE it takes <see cref="_binding"/> — which both <c>ReadAsync</c>
/// and <c>WriteAsync</c> do — has the ordering that stops it judging a batch against the
/// previous document's <c>Write.Enabled</c> / <c>RequestTimeoutMs</c> paired with the new
/// tag table. Pinned by
/// <c>MTConnectOptionsPublicationTests</c>, which fails on any unfenced touch.
/// </para>
/// </summary>
private MTConnectDriverOptions _options;
/// <summary>
/// The tag tables derived from <see cref="_options"/> — the <c>RawPath ↔ dataItemId</c>
/// translation the whole data plane runs through, plus the definition set the observation
/// index coerces against. Rebuilt (never mutated) whenever options are installed, and always
/// published in the same breath as them, so a lock-free reader can never pair one options
/// document's tags with another's RawPaths.
/// </summary>
private TagBinding _binding;
private MTConnectObservationIndex _index;
private DriverHealth _health = new(DriverState.Unknown, null, null);
private long _nextSubscriptionId;
/// <summary>Cancels the shared <c>/sample</c> pump. Guarded by <see cref="_subscriptionLock"/>.</summary>
private CancellationTokenSource? _pumpCts;
/// <summary>The shared <c>/sample</c> pump task. Guarded by <see cref="_subscriptionLock"/>.</summary>
private Task? _pumpTask;
/// <summary>
/// The <see cref="_sessionGeneration"/> the running pump was started against, guarded by
/// <see cref="_subscriptionLock"/> like the rest of the control block. A pump whose generation is
/// not the current one is bound to a client a lifecycle restart has retired.
/// </summary>
private long _pumpGeneration;
/// <summary>
/// Bumped each time <see cref="StartCoreAsync"/> installs a NEW <see cref="AgentSession"/> (not on a
/// <c>with</c>-copy that only advances the cursor or the probe cache). Written under
/// <see cref="_subscriptionLock"/>.
/// </summary>
private long _sessionGeneration;
/// <summary>
/// Pumps retired by <see cref="StartSampleStreamCore"/> because they were bound to a replaced
/// session, kept so <see cref="StopSampleStreamAsync"/> can join them — a pump nobody
/// awaits can still run its default-verdict arm (<c>Degrade</c>, the reconnect counter) after
/// the new session has written Healthy. Guarded by <see cref="_subscriptionLock"/>.
/// </summary>
private readonly List<(CancellationTokenSource Cts, Task Task)> _supersededPumps = [];
/// <summary>
/// The Agent answered <c>/sample</c> with something that cannot be streamed at all, so the
/// pump gave up. Latched — reconnecting reproduces the identical answer forever, and a
/// later <see cref="SubscribeAsync"/> is not new information. Cleared only by a lifecycle
/// start, which is the one event that means an operator changed something. Guarded by
/// <see cref="_subscriptionLock"/>.
/// </summary>
private bool _streamUnsupported;
/// <summary>
/// Whether the <c>/sample</c> pump is currently failing. Kept separate from
/// <see cref="_readDegraded"/> because <c>/sample</c> and <c>/current</c> are different
/// Agent requests that fail independently — see <see cref="PublishHealthy"/>.
/// </summary>
private volatile bool _streamDegraded;
/// <summary>Whether the <c>/current</c> read path is currently failing. See <see cref="PublishHealthy"/>.</summary>
private volatile bool _readDegraded;
/// <summary>
/// Whether the last Agent write got no reply (review I-3 / coordinator ruling: an unanswered write degrades).
/// Cleared by the next answered <c>/current</c> read, answered write (<see cref="RecordReadAnswered"/> /
/// <see cref="RecordWriteAnswered"/>) or delivered <c>/sample</c> chunk of the current session
/// (<see cref="TryMarkStreamDelivered"/>, r2 N-5) — so it cannot stick on a subscription-only driver. A read served
/// from the live index does not clear it directly; the chunks behind that index do.
/// </summary>
private volatile bool _writeDegraded;
/// <summary>The last <c>/current</c> refusal text logged, so a persistent refusal warns once, not per read.</summary>
private string? _lastReadRefusal;
/// <summary>
/// Whether the current-generation <c>/sample</c> pump has delivered since it started (a chunk, a heartbeat or
/// a re-baseline) — the precondition, with <see cref="_streamDegraded"/> clear, for <see cref="ReadAsync"/>
/// to answer from the pump's index instead of a <c>/current</c> (R06:T-4).
/// </summary>
private volatile bool _pumpLive;
/// <summary>
/// Consecutive <c>/sample</c> reconnect attempts that have <b>not</b> yet been vindicated by
/// a delivered chunk — the input to <see cref="BackoffFor"/>. Reset by any stream that
/// actually delivered something, so an agent that drops a connection once an hour is never
/// treated as one that has been failing all day.
/// </summary>
private volatile int _reconnectAttempts;
/// <summary>
/// Latch behind the once-per-stream-generation subscriber-fault Warning. A consumer that
/// throws on every value would otherwise emit one Warning per observation <i>per handle</i>
/// — thousands a second on a busy Agent, which buries the very log it is trying to raise.
/// </summary>
private int _subscriberFaultLogged;
/// <summary>
/// The live agent client plus the <c>/probe</c> cache derived from it, held as <b>one
/// immutable snapshot behind a single reference</b> and only ever replaced by a
/// <see cref="Volatile.Write{T}"/> of a whole new instance — the same discipline
/// <see cref="_index"/> uses, and for the same reason.
/// </summary>
/// <remarks>
/// <para>
/// <b>Why one field instead of several.</b> These values are read <i>outside</i>
/// <see cref="_lifecycle"/> — by <see cref="ReadAsync"/> and by
/// <see cref="GetOrFetchProbeModelAsync"/> (the browse) — while every write
/// happens under it. As separate fields they could be observed mid-update, pairing one
/// initialize's client with the previous one's model. Packing them makes every
/// observation internally consistent by construction, with no lock on the read path.
/// (A third value, a cached data-item count, lived here until it was deleted when its
/// only reader — <c>GetMemoryFootprint</c> — along with the rest of that dead surface.)
/// </para>
/// <para>
/// <b>Why not simply take the lock in those readers.</b> Both readers <c>await</c> the
/// network, so they would hold the lifecycle lock across an HTTP round trip — a browse
/// could stall a <see cref="ShutdownAsync"/> for a full <c>RequestTimeoutMs</c> — and
/// every added lock site widens the non-reentrancy hazard documented on
/// <see cref="_lifecycle"/>. It would also make the read path inconsistent with
/// <see cref="ReadAsync"/>, which is deliberately lock-free.
/// </para>
/// <para>
/// <b>What this does NOT fix.</b> A reader can still capture a session moments before a
/// concurrent teardown disposes its client, and then call it — no lock-free scheme can
/// prevent that, and holding the lock across the call is the trade rejected above.
/// What it guarantees is that the failure is <i>named</i>: the resulting
/// <see cref="ObjectDisposedException"/> is caught at the seam and reported as the same
/// "driver is not connected" error a caller already has to handle, rather than escaping
/// raw out of a browse.
/// </para>
/// </remarks>
private AgentSession? _session;
/// <summary>Creates a driver instance. Opens no connection and builds no agent client.</summary>
/// <param name="options">
/// The typed configuration this instance starts from. A config document supplied to
/// <see cref="InitializeAsync"/> / <see cref="ReinitializeAsync"/> supersedes it.
/// </param>
/// <param name="driverInstanceId">Stable logical id of this driver instance, from the config DB.</param>
/// <param name="agentClientFactory">
/// Builds the Agent client from the effective options. Defaults to the production
/// <see cref="MTConnectAgentClient"/>; tests inject a canned-XML fake. Called from
/// <see cref="InitializeAsync"/>, never from this constructor.
/// </param>
/// <param name="logger">Optional logger; defaults to the null logger.</param>
/// <param name="probeScheduler">
/// How the connectivity probe loop waits between ticks. Defaults to
/// <see cref="Task.Delay(TimeSpan, CancellationToken)"/>; tests inject a hand-driven ticker so
/// the probe's cadence is asserted deterministically instead of slept through.
/// </param>
public MTConnectDriver(
MTConnectDriverOptions options,
string driverInstanceId,
Func<MTConnectDriverOptions, IMTConnectAgentClient>? agentClientFactory = null,
ILogger<MTConnectDriver>? logger = null,
Func<TimeSpan, CancellationToken, Task>? probeScheduler = null)
{
ArgumentNullException.ThrowIfNull(options);
ArgumentException.ThrowIfNullOrWhiteSpace(driverInstanceId);
_driverInstanceId = driverInstanceId;
_logger = logger ?? NullLogger<MTConnectDriver>.Instance;
_agentClientFactory = agentClientFactory ?? (o => new MTConnectAgentClient(o, logger));
_probeScheduler = probeScheduler ?? Task.Delay;
// Sets _options AND the tag tables derived from it. No /probe model is available here (the
// constructor dials nothing), so a raw tag whose blob declares no type falls back to String
// until InitializeAsync rebuilds the binding against the Agent's own declaration.
Volatile.Write(ref _options, options);
_binding = BuildBinding(options, probeModel: null);
// Never null, so every consumer (ReadAsync included) can ask it for a snapshot
// without a null check: an un-primed index answers BadWaitingForInitialData for an authored
// tag and BadNodeIdUnknown otherwise, which is exactly the truth before Initialize runs.
_index = new MTConnectObservationIndex(_binding.IndexTags);
}
/// <inheritdoc/>
public string DriverInstanceId => _driverInstanceId;
/// <inheritdoc />
// Sourced from the constant rather than a literal — the
// repo-wide fix for the historical TwinCat/Focas drift, where a driver's own
// spelling diverged from the constant every dispatch map keys off.
public string DriverType => DriverTypeNames.MTConnect;
/// <summary>
/// The live <c>dataItemId → snapshot</c> map that backs the <b>subscription</b> surface:
/// primed by <see cref="InitializeAsync"/> and thereafter written only by the <c>/sample</c>
/// pump. Exposed to tests as the observable proof that initialize actually primed
/// it — and, in <c>MTConnectReadTests</c>, that <see cref="ReadAsync"/> leaves it alone.
/// </summary>
/// <remarks>
/// <b>Read does not consume this.</b> <see cref="ReadAsync"/> indexes its own <c>/current</c>
/// into a per-call snapshot instead, so the pump keeps a single writer — see the failure
/// analysis in <see cref="ReadAsync"/>'s remarks.
/// </remarks>
internal MTConnectObservationIndex ObservationIndex => Volatile.Read(ref _index);
/// <summary>
/// The cached <c>/probe</c> device model, or <c>null</c> when it has not been fetched or was
/// dropped by <see cref="InvalidateProbeModel"/> on an Agent restart. Prefer
/// <see cref="GetOrFetchProbeModelAsync"/>, which cannot observe the dropped hole.
/// </summary>
internal MTConnectProbeModel? CachedProbeModel => Volatile.Read(ref _session)?.ProbeModel;
/// <summary>
/// The Agent's <c>instanceId</c> as of the last successful <c>/current</c> — including the
/// re-baseline the <c>/sample</c> pump runs when it sees the id change mid-stream. The driver
/// watches this for change to raise rediscovery (an Agent restart invalidates every cached
/// id).
/// </summary>
internal long? AgentInstanceId => Volatile.Read(ref _session)?.InstanceId;
/// <summary>
/// The sequence the next <c>/sample</c> stream will open at — the priming <c>/current</c>'s
/// <c>nextSequence</c>, advanced by every re-baseline and by the pump as it stops. Paired
/// with <see cref="AgentInstanceId"/> in one session record, so the two are never observed
/// out of step.
/// </summary>
internal long? AgentNextSequence => Volatile.Read(ref _session)?.NextSequence;
/// <summary>
/// Consecutive <c>/sample</c> reconnect attempts not yet vindicated by a delivered chunk —
/// the input to <see cref="BackoffFor"/>, exposed so the reset policy can be asserted
/// without measuring how long anything slept.
/// </summary>
internal int ReconnectAttempts => _reconnectAttempts;
/// <summary>
/// The shared <c>/sample</c> pump's task while one is running, else <c>null</c>. Exposed to
/// tests as the deterministic teardown barrier: "the pump has stopped" is a task completion,
/// never a sleep. Guarded, so a test never observes a half-installed pump.
/// </summary>
internal Task? SampleStreamTask
{
get
{
lock (_subscriptionLock)
{
return _pumpTask;
}
}
}
/// <summary>
/// The connectivity probe loop's task while one is running, else <c>null</c>. Exposed to
/// tests as the deterministic teardown barrier — "the probe loop has stopped" is a task
/// completion, never a sleep — and as the proof that <c>Probe.Enabled = false</c> starts
/// nothing at all.
/// </summary>
internal Task? ProbeLoopTask => Volatile.Read(ref _probeTask);
/// <summary>The options currently in force — the constructor's, or the last document applied.</summary>
internal MTConnectDriverOptions EffectiveOptions => Volatile.Read(ref _options);
/// <inheritdoc />
// Builds the Agent client from the effective options, runs one deadline-bounded
// /probe (caching the device model and its data-item count), then one
// deadline-bounded /current to prime the observation index and record the Agent's
// instanceId. Publishes on success; on any failure
// disposes the client, clears the caches, publishes with
// the error text, and rethrows.
public async Task InitializeAsync(string driverConfigJson, CancellationToken cancellationToken)
{
await _lifecycle.WaitAsync(cancellationToken).ConfigureAwait(false);
try
{
// Parse BEFORE tearing down — see ResolveIncomingOptions. Initialize is reachable on a
// live instance (DriverInstanceActor re-initialises in Connecting/Reconnecting), and
// destroying a working client before discovering the new document is unreadable would
// be the exact outcome ReinitializeAsync is written to avoid.
var options = ResolveIncomingOptions(driverConfigJson);
// A re-Initialize on a live instance must not orphan the previous client — nor the pump
// that is enumerating it. Stopping the stream FIRST is the same ordering
// ReinitializeAsync and ShutdownAsync use, and for the same reason: a pump left
// enumerating a disposed client reads the disposal as a dropped connection and
// reconnects against an object that no longer exists.
await StopSampleStreamAsync(cancellationToken).ConfigureAwait(false);
await StopProbeLoopAsync(cancellationToken).ConfigureAwait(false);
await TeardownCoreAsync().ConfigureAwait(false);
await StartCoreAsync(options, existingClient: null, cancellationToken).ConfigureAwait(false);
}
finally
{
_lifecycle.Release();
}
}
/// <inheritdoc />
// Stops the /sample stream, then takes one of two paths depending on whether the
// new document changes anything the reads at
// construction:
// Endpoint/transport changed (AgentUri, DeviceName,
// RequestTimeoutMs, HeartbeatMs, SampleIntervalMs,
// SampleCount) ⇒ full teardown and rebuild. The plan names only the first two,
// but the other four are baked into the client's deadlines, its watchdog window, and its
// /sample query string, so keeping the old client because the URI happened not to
// change would silently discard the operator's edit — the same class of defect as
// ignoring the config document altogether.
// Otherwise ("config-only": authored tags, probe cadence, reconnect backoff) ⇒
// the live client and its connection pool are kept, and everything derived from config
// is rebuilt: the probe model is re-fetched, the observation index is recreated against
// the new tag types, and one /current re-primes it. Nothing config-derived
// survives, so a retyped tag cannot keep coercing to its old type.
public async Task ReinitializeAsync(string driverConfigJson, CancellationToken cancellationToken)
{
await _lifecycle.WaitAsync(cancellationToken).ConfigureAwait(false);
try
{
var incoming = ResolveIncomingOptions(driverConfigJson);
var existing = Volatile.Read(ref _session)?.Client;
await StopSampleStreamAsync(cancellationToken).ConfigureAwait(false);
await StopProbeLoopAsync(cancellationToken).ConfigureAwait(false);
if (existing is null || RequiresClientRebuild(Volatile.Read(ref _options), incoming))
{
await TeardownCoreAsync().ConfigureAwait(false);
await StartCoreAsync(incoming, existingClient: null, cancellationToken).ConfigureAwait(false);
return;
}
await StartCoreAsync(incoming, existing, cancellationToken).ConfigureAwait(false);
}
finally
{
_lifecycle.Release();
}
}
/// <inheritdoc />
// Stops the /sample stream, disposes the Agent client, and drops the probe cache and
// the observation index — values sourced from a torn-down connection must never keep
// reading Good. LastSuccessfulRead is retained because it is diagnostic ("when did
// this driver last hear from the Agent"), not served data. Idempotent.
public async Task ShutdownAsync(CancellationToken cancellationToken)
{
await _lifecycle.WaitAsync(cancellationToken).ConfigureAwait(false);
try
{
var lastRead = ReadHealth().LastSuccessfulRead;
await StopSampleStreamAsync(cancellationToken).ConfigureAwait(false);
await StopProbeLoopAsync(cancellationToken).ConfigureAwait(false);
await TeardownCoreAsync().ConfigureAwait(false);
WriteHealth(new DriverHealth(DriverState.Unknown, lastRead, null));
}
finally
{
_lifecycle.Release();
}
}
/// <inheritdoc/>
public DriverHealth GetHealth() => ReadHealth();
/// <inheritdoc />
// One /current per batch, whatever the batch size. MTConnect's
// /current answers for the whole device (or the whole Agent), so N references
// cost one deadline-bounded round-trip and are answered out of that one document. There
// is no per-reference request to make and none is made.
// The response is indexed into a per-call snapshot, NOT into the driver's shared
// observation index. That index is written by the /sample pump and
// is deliberately last-write-wins with no timestamp arbitration. A read that folded its
// /current into it would be a second writer racing the first, and the
// /current document is frequently the older of the two — the Agent
// composes it while the pump's newer delta is already in flight. Losing that race rolls
// a subscribed value backwards and leaves it there until the data item next
// changes, which for an EVENT such as Execution can be hours. Keeping the
// read path a pure function of (document, authored tags) costs one index construction
// per batch — negligible beside the HTTP round-trip that precedes it — and makes it
// impossible for a read to corrupt subscription state. The reference source-of-truth is
// unchanged either way: both paths coerce through the same
// logic, so a read and a subscription report a
// given observation identically.
// Nothing but genuine caller cancellation crosses this boundary. An unreachable
// Agent, a blown deadline, an unusable answer — each fails the batch's values
// (every reference codes ) rather than the batch: an
// exception here would also fail the references that were perfectly readable, and the
// node-manager cannot tell a thrown read from a broken driver. This is the opposite
// posture to , which is specified to throw so the
// runtime's retry/backoff loop can own recovery.
// The status-code distinctions the index draws — BadWaitingForInitialData for an
// authored-but-unreported id, BadNodeIdUnknown for an unknown one,
// BadNoCommunication for the Agent's UNAVAILABLE — are passed through
// unmodified. They tell an operator three different things.
// is null — a caller bug, not device data, and the
// one thing this method is loud about (matching 's
// documented posture).
public async Task<IReadOnlyList<DataValueSnapshot>> ReadAsync(
IReadOnlyList<string> fullReferences, CancellationToken cancellationToken)
{
ArgumentNullException.ThrowIfNull(fullReferences);
// Checked before anything else: an empty batch must cost the Agent nothing, and there is no
// failure it could report.
if (fullReferences.Count == 0)
{
return [];
}
// One read of the session reference: the client used below is the one that was live at this
// instant, never a half-swapped pairing (see the _session remarks).
IMTConnectAgentClient? client;
long readGeneration;
lock (_subscriptionLock)
{
client = Volatile.Read(ref _session)?.Client;
readGeneration = _sessionGeneration;
}
var options = Volatile.Read(ref _options);
// Read once, for the same reason the session is: every reference in this batch must be
// resolved against ONE tag table, not half against the table a concurrent re-initialize
// just replaced.
var binding = Volatile.Read(ref _binding);
if (client is null)
{
// Before InitializeAsync, after a faulted one, or after ShutdownAsync. Health is left
// exactly as it is: a read attempted against a driver that was never started (or was
// deliberately stopped) is not evidence of degradation.
return BadBatch(fullReferences.Count, BadNotConnected);
}
// R06:T-4 — while the /sample pump is live and healthy its index IS the Agent's current state (to within
// one chunk), so a read is answered from it: READ only — the pump stays the index's single writer, which
// is all the single-writer rule below forbids. /current stays the fallback for every other case (no
// subscription, a stream not yet delivering, a degraded stream).
if (TryReadFromLivePump(fullReferences, binding) is { } live)
{
return live;
}
try
{
var current = await BoundedAsync(client.CurrentAsync, "/current", options.RequestTimeoutMs, cancellationToken)
.ConfigureAwait(false);
// R06:T-2 — Agent-restart detection used to live only on the pump, so a driver with no subscription
// (browse-only) kept serving a cached /probe model of a process that no longer existed and never raised
// rediscovery. Only while the session this read used is still installed: a read that raced a
// re-initialise against a different Agent would otherwise announce a "restart" the operator caused.
if (current.InstanceId != Interlocked.Read(ref _announcedInstanceId)
&& IsCurrentSessionGeneration(readGeneration))
{
AnnounceAgentRestart(current.InstanceId, options);
// Review m6: the session's cursor still named the dead instance, so the next pump would open there and
// spend its first chunk on a clear + re-baseline (a Bad flicker). Move it to the new process — through
// the same generation-checked publish the pump uses, so it can only land on the session this read used.
PublishAgentCursor(readGeneration, current.InstanceId, current.NextSequence);
}
// Built from the SAME authored tags as the shared index, so the authored-vs-unknown
// distinction and every coercion behave identically — it is a snapshot, not a variant.
var snapshot = MTConnectObservationIndex.OverFrozenTags(binding.IndexTagMap);
snapshot.Apply(current, baseline: true);
var results = new DataValueSnapshot[fullReferences.Count];
for (var i = 0; i < results.Length; i++)
{
// Each reference is a RawPath, translated to the DataItem id the index is keyed by.
// An unmapped reference falls through unchanged and misses — Bad, never a throw.
// Get never throws and never returns null, including for a null/blank reference.
results[i] = snapshot.Get(binding.ToDataItemId(fullReferences[i]));
}
// m7: the next refusal after a recovery is news again.
Volatile.Write(ref _lastReadRefusal, null);
RecordReadAnswered(DateTime.UtcNow);
return results;
}
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
{
// The caller asked us to stop. This is the ONLY exception that may propagate: a blown
// per-call deadline arrives as the TimeoutException BoundedAsync raises instead, and is
// handled below as the Agent failure it is.
throw;
}
catch (InvalidDataException ex) when (MTConnectXml.IsAgentRefusal(ex))
{
// The health contract: an MTConnectError document is the Agent REFUSING the request (a bad device
// scope, an unsupported request) — it answered. The batch is Bad, but this is not a comms fault: the
// read path is not degraded by it (that would reconnect-loop a driver over a config problem), the stamp
// advances, and the Agent's text is logged once per distinct refusal.
RecordReadAnswered(DateTime.UtcNow);
if (!string.Equals(Interlocked.Exchange(ref _lastReadRefusal, ex.Message), ex.Message, StringComparison.Ordinal))
{
_logger.LogWarning(
"MTConnect driver {DriverInstanceId}: {AgentUri} refused /current — {AgentError}. Every read is reported Bad until the Agent accepts the request; the driver stays connected.",
_driverInstanceId, AgentDisplay(options), ex.Message);
}
return BadBatch(fullReferences.Count, BadCommunicationError);
}
catch (Exception ex)
{
DegradeAfterReadFailure(ex);
return BadBatch(fullReferences.Count, BadCommunicationError);
}
}
/// <summary>
/// Answers a read batch from the live pump's index, or returns <see langword="null"/> when that index cannot be
/// trusted as the Agent's current state: no pump, a pump bound to a retired session, one that has not yet
/// delivered, or a degraded stream. Touches no health — the pump owns the stream's.
/// </summary>
/// <remarks>
/// <b>Stale window (review m1).</b> A stream that stalls WITHOUT failing (no chunk, no heartbeat) keeps
/// <c>_streamDegraded</c> clear until the client's idle watchdog fires (2 × <c>HeartbeatMs</c> +
/// <c>RequestTimeoutMs</c>); for that bounded window reads answer Good from an index nothing is updating, where a
/// <c>/current</c> would have noticed. Subscriptions have the identical exposure; the window is the same one
/// the watchdog already accepts for the data plane.
/// </remarks>
/// <param name="fullReferences">The batch.</param>
/// <param name="binding">The tag table the batch is resolved against.</param>
/// <returns>One snapshot per reference in order, or <see langword="null"/> to fall back to <c>/current</c>.</returns>
private DataValueSnapshot[]? TryReadFromLivePump(IReadOnlyList<string> fullReferences, TagBinding binding)
{
lock (_subscriptionLock)
{
if (_pumpTask is not { IsCompleted: false } || _pumpGeneration != _sessionGeneration)
{
return null;
}
}
if (!_pumpLive || _streamDegraded)
{
return null;
}
var index = ObservationIndex;
var results = new DataValueSnapshot[fullReferences.Count];
for (var i = 0; i < results.Length; i++)
{
results[i] = index.Get(binding.ToDataItemId(fullReferences[i]));
}
// The read path is answering (from a stream that is delivering), so a degradation a failed /current left
// behind is no longer true of it. The stamp stays the pump's — this read heard nothing from the Agent itself.
// Clears only the READ degradation: an index read is derived from the stream's evidence — the chunks that fed the
// index are what clear an unanswered write (TryMarkStreamDelivered, r2 N-5).
if (_readDegraded)
{
_readDegraded = false;
PublishHealthy(ReadHealth().LastSuccessfulRead ?? DateTime.UtcNow);
}
return results;
}
// ---- IWritable: observation injection through the Agent's AllowPut surface ----
/// <inheritdoc />
// Sets the Agent's reported observation for each requested reference, through its
// AllowPut surface. Returns one status per request, in request order.
// The requested writes.
// Cancellation for the batch.
// One per request, in request order.
// This writes the Agent, not the machine — see the class remarks. Good here means
// "the Agent took the observation", which is a stronger statement than MQTT's optimistic
// "published" (the Agent answers) and a weaker one than a device acknowledgement (no
// MTConnect Agent speaks for the machine). The next adapter update for that DataItem
// overwrites what was written, and the ordinary /sample ingest path is what
// corrects the node.
// Never throws, and one bad slot never fails the batch — matching
// . Failing the whole call would let one stale reference reject
// every other write in the same OPC UA request. Even cancellation returns a full-length
// batch rather than propagating, so the node-manager can tell which slots were attempted.
// Refusal order is fixed, and each step answers a different operator question:
// writing switched off ⇒ BadNotWritable; no live client ⇒ BadNotConnected
// (this driver is not running — 's convention); the reference
// resolves to no DataItem, or to one whose owning device cannot be named ⇒
// BadNodeIdUnknown; CONDITION category ⇒ BadNotWritable; no invariant string
// form ⇒ BadTypeMismatch; and only then the Agent, whose refusal or whose
// unreachability ⇒ BadCommunicationError.
// The Agent's refusal text is logged, never stored. It goes to a Warning line and
// nowhere near LastError, which is type-only by repo-wide discipline. A refused write is
// not evidence the read path degraded — but it IS the Agent answering, so any answer
// (accepted or refused) advances LastSuccessfulRead with the state following the read /
// stream flags; a write that got no answer degrades (stamp preserved) until the next answered read or write.
// Slots that survive the local refusals go out as ONE request per device (R06:T-6).
// is null — a caller bug.
public async Task<IReadOnlyList<WriteResult>> WriteAsync(
IReadOnlyList<WriteRequest> writes, CancellationToken cancellationToken)
{
ArgumentNullException.ThrowIfNull(writes);
if (writes.Count == 0)
{
return [];
}
// Read once each, for the same reason ReadAsync does: every slot in this batch is judged
// against ONE options document and ONE tag table, never half against a table a concurrent
// re-initialize just replaced.
var options = Volatile.Read(ref _options);
var client = Volatile.Read(ref _session)?.Client;
var binding = Volatile.Read(ref _binding);
var results = new WriteResult[writes.Count];
// Pass 1 — every local refusal, slot by slot, in the fixed order (see above). What survives is a pending
// Agent write.
var pending = new List<PendingWrite>(writes.Count);
for (var i = 0; i < writes.Count; i++)
{
var request = writes[i];
if (!options.Write.Enabled)
{
results[i] = new WriteResult(BadNotWritable);
continue;
}
if (client is null)
{
results[i] = new WriteResult(BadNotConnected);
continue;
}
if (!binding.TryResolveWriteTarget(request.FullReference, out var dataItemId, out var target)
|| target.DeviceName is null)
{
results[i] = new WriteResult(BadNodeIdUnknown);
continue;
}
if (target.IsCondition)
{
results[i] = new WriteResult(BadNotWritable);
continue;
}
if (!TryFormatValue(request.Value, out var text))
{
results[i] = new WriteResult(BadTypeMismatch);
continue;
}
pending.Add(new PendingWrite(i, target.DeviceName, dataItemId, text));
}
if (pending.Count == 0)
{
return results;
}
// Pass 2 — R06:T-6: the Agent's PUT surface takes several id=value pairs per device, so the pending slots go
// out as ONE request per device (it used to be one POST per slot). Request order is kept: a batch closes at
// the first slot naming a different device or repeating an id already in it, so two writes to one DataItem
// land in the order the caller sent them.
var batches = new List<List<PendingWrite>>();
foreach (var write in pending)
{
var open = batches.Count > 0 ? batches[^1] : null;
if (open is null
|| !string.Equals(open[0].DeviceName, write.DeviceName, StringComparison.Ordinal)
|| open.Exists(w => string.Equals(w.DataItemId, write.DataItemId, StringComparison.Ordinal)))
{
batches.Add([write]);
continue;
}
open.Add(write);
}
for (var b = 0; b < batches.Count; b++)
{
var batch = batches[b];
try
{
await WriteBatchAsync(client!, options, batch, results, cancellationToken).ConfigureAwait(false);
}
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
{
// Caller-cancelled: fill every slot not yet answered rather than throwing, so the batch shape holds
// and a slot that already landed keeps its real outcome.
for (var r = b; r < batches.Count; r++)
{
foreach (var write in batches[r])
{
results[write.Slot] ??= new WriteResult(BadCommunicationError);
}
}
return results;
}
}
return results;
}
/// <summary>One slot that survived the local refusals, waiting for the Agent.</summary>
private sealed record PendingWrite(int Slot, string DeviceName, string DataItemId, string Text);
/// <summary>
/// Sends one same-device batch and maps the Agent's answer onto its slots. Accepted ⇒ every slot Good. Refused
/// ⇒ a single-slot batch is that slot's <c>BadCommunicationError</c> (the Agent's text logged); a multi-slot
/// batch is retried slot by slot so one refused DataItem cannot fail the others — the batch contract above. A
/// transport failure fails the batch's slots. Caller cancellation propagates to <see cref="WriteAsync"/>.
/// </summary>
private async Task WriteBatchAsync(
IMTConnectAgentClient client,
MTConnectDriverOptions options,
List<PendingWrite> batch,
WriteResult[] results,
CancellationToken cancellationToken)
{
var device = batch[0].DeviceName;
MTConnectWriteOutcome outcome;
try
{
outcome = batch.Count == 1
? await client.WriteObservationAsync(device, batch[0].DataItemId, batch[0].Text, cancellationToken)
.ConfigureAwait(false)
: await client.WriteObservationsAsync(
device,
[.. batch.Select(w => new KeyValuePair<string, string>(w.DataItemId, w.Text))],
cancellationToken)
.ConfigureAwait(false);
}
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
{
throw;
}
catch (Exception ex)
{
// A transport failure, a blown per-call deadline, or a disposed client — no answer arrived. Logged with
// the exception (its TYPE is the diagnosis), reported per slot, and Degraded with the stamp preserved
// (RecordWriteNoReply) until the next answered read or write.
_logger.LogWarning(
ex,
"MTConnect driver {DriverInstanceId} could not write {Count} DataItem(s) ({DataItemIds}) on device '{DeviceName}' to {AgentUri}.",
_driverInstanceId, batch.Count, string.Join(", ", batch.Select(w => w.DataItemId)), device, AgentDisplay(options));
foreach (var write in batch)
{
results[write.Slot] = new WriteResult(BadCommunicationError);
}
RecordWriteNoReply(options, ex);
return;
}
// An answer — accepted or refused — is the Agent talking (the health contract: a refusal is a reply).
RecordWriteAnswered();
if (outcome.Accepted)
{
foreach (var write in batch)
{
results[write.Slot] = new WriteResult(0u);
}
return;
}
if (batch.Count == 1)
{
_logger.LogWarning(
"MTConnect driver {DriverInstanceId}: {AgentUri} refused the write of '{Value}' to DataItem '{DataItemId}' on device '{DeviceName}' — {AgentError}. Check the Agent's AllowPut / AllowPutFrom settings and that the DataItem exists.",
_driverInstanceId, AgentDisplay(options), batch[0].Text, batch[0].DataItemId, device, outcome.AgentError);
results[batch[0].Slot] = new WriteResult(BadCommunicationError);
return;
}
// The Agent refused the request as a whole: find out which slot(s) it objects to, one by one. (Review m4:
// an Agent may have applied some pairs before refusing — cppagent collects put errors per key — so the retry
// can re-send a pair that already landed. Harmless for value-setting: the Agent's duplicate filter drops an
// identical repeat.)
foreach (var write in batch)
{
await WriteBatchAsync(client, options, [write], results, cancellationToken).ConfigureAwait(false);
}
}
/// <summary>
/// A write got an answer from the Agent: <see cref="DriverHealth.LastSuccessfulRead"/> advances, a prior
/// unanswered-write degradation clears, and the state follows the read / stream flags (a write cannot clear
/// either of THOSE paths' degradation, but it is evidence the Agent is reachable).
/// </summary>
private void RecordWriteAnswered()
{
_writeDegraded = false;
if (ReadHealth().State is DriverState.Healthy or DriverState.Degraded)
{
PublishHealthy(DateTime.UtcNow);
}
}
/// <summary>
/// A write got NO answer (transport failure, deadline): Degraded with the stamp preserved — the health contract
/// for a comms fault. Type-only text. Cleared by the next answered read or write.
/// </summary>
private void RecordWriteNoReply(MTConnectDriverOptions options, Exception ex)
{
_writeDegraded = true;
Degrade($"MTConnect write to {AgentDisplay(options)} got no reply ({ex.GetType().Name}).");
}
/// <summary>
/// The ONE success writer the read path and the write path share (review I-3): an answered read clears the
/// read AND write degradations (the Agent is answering requests), then publishes Healthy unless the
/// <c>/sample</c> stream is still failing.
/// </summary>
/// <param name="at">The stamp to publish.</param>
private void RecordReadAnswered(DateTime at)
{
_readDegraded = false;
_writeDegraded = false;
PublishHealthy(at);
}
/// <summary>
/// Renders a value to the invariant string a PUT query carries, or reports that it has no such
/// form.
/// </summary>
/// <param name="value">The value the OPC UA client supplied.</param>
/// <param name="text">The rendered value; empty when this returns <see langword="false"/>.</param>
/// <returns><see langword="false"/> for a value that must be refused as a type mismatch.</returns>
/// <remarks>
/// <para>
/// <b><see cref="CultureInfo.InvariantCulture"/> is load-bearing.</b> A comma decimal
/// separator produces a value the Agent accepts and every consumer reads back as a
/// different number. Booleans render as <c>True</c>/<c>False</c> —
/// <see cref="bool.ToString()"/> is culture-independent, so that IS the invariant form, and
/// it is what an MTConnect consumer will see verbatim.
/// </para>
/// <para>
/// <b>Anything that is not <see cref="IConvertible"/> is refused rather than rendered.</b>
/// <see cref="Convert.ToString(object, IFormatProvider)"/> falls back to
/// <see cref="object.ToString"/> for a non-<see cref="IConvertible"/> value, so an array
/// would be written as the literal text <c>System.Double[]</c> — which the Agent stores
/// happily, making the corruption invisible until someone reads the observation back.
/// The guard is therefore written against that property itself rather than against
/// <see cref="Array"/>, the one type it was first noticed on: an
/// <c>Opc.Ua.ExtensionObject</c>, an <c>XmlElement</c> or a <c>List&lt;T&gt;</c> from a
/// non-SDK caller corrupts an observation in exactly the same way.
/// <c>string</c>, <c>bool</c>, <see cref="DateTime"/> and every numeric type ARE
/// <see cref="IConvertible"/>, so the accepted set is unchanged. TIME_SERIES / DATA_SET
/// writes are out of scope for v1.
/// </para>
/// <para>
/// <b>Every formatting throw is caught, not just the documented conversion ones.</b> The
/// rendering runs the value's own <see cref="object.ToString"/> /
/// <see cref="IFormattable"/>, which is arbitrary code — and this runs OUTSIDE
/// <see cref="WriteAsync"/>'s per-slot <c>try</c>, so anything that escapes here escapes
/// the whole batch and breaks its never-throws contract. It costs its own slot a
/// <c>BadTypeMismatch</c> and is logged.
/// </para>
/// </remarks>
private bool TryFormatValue(object? value, out string text)
{
text = string.Empty;
// A PUT query has no null to send, and the empty string is a real observation rather than an
// absence — so "no value" is refused instead of being silently turned into one. Everything
// that is not IConvertible is refused with it: Convert.ToString renders such a value as its
// TYPE NAME, which the Agent stores under a Good status (see the remarks).
if (value is null or Array or not IConvertible)
{
return false;
}
string? rendered;
try
{
rendered = Convert.ToString(value, CultureInfo.InvariantCulture);
}
catch (Exception ex)
{
// Catch-all, and deliberately so. Convert.ToString falls back to the value's OWN
// ToString/IFormattable, which is arbitrary caller code that may throw ANYTHING — and this
// call sits outside WriteAsync's per-slot try, so a narrower filter (FormatException /
// InvalidCastException / OverflowException) let every other exception escape WriteAsync
// uncaught and break the documented never-throws contract for the whole batch. One
// unformattable value must cost its own slot, nothing more.
_logger.LogWarning(
ex,
"MTConnect driver {DriverInstanceId} could not render a write value of type {ValueType} to its invariant string form; the write is refused as a type mismatch.",
_driverInstanceId,
value.GetType());
return false;
}
if (rendered is null)
{
return false;
}
text = rendered;
return true;
}
// ---- ISubscribable: ONE shared /sample long poll behind every handle ----
/// <summary>Raised when a subscribed data item's value changes.</summary>
public event EventHandler<DataChangeEventArgs>? OnDataChange;
/// <inheritdoc />
// Initial data comes from the primed /current, not from the wire. Every
// subscribed reference gets one callback immediately (the OPC UA Part 4 convention),
// answered out of the observation index — including the references with no value yet,
// which report BadWaitingForInitialData. Waiting for the Agent's next chunk
// instead would leave a fresh subscription silent for up to a whole heartbeat,
// and permanently silent for any data item that simply is not changing.
// Subscribe does not wait for the stream. The first subscription starts the
// shared pump; the pump then opens /sample on its own task under its own
// cancellation token. This is deliberate on both counts. The opening handshake cannot
// be awaited inside the caller's Subscribe timeout in any useful sense — the first
// chunk may legitimately be a heartbeat away — and the long-lived poll must NOT run
// under the caller's token, which belongs to one Subscribe call and is disposed the
// moment it returns. What bounds the running stream is the client's own heartbeat
// watchdog, and what stops it is or the lifecycle.
// is not honoured per subscription, and
// cannot be: MTConnect's publish cadence is the Agent-side interval query
// parameter of the single shared /sample request
// (), fixed when the client is
// built. Honouring a per-handle interval would mean one stream per subscription — the
// Agent load this driver exists to avoid — or client-side throttling that would drop
// transient EVENT values the Agent bothered to send. The parameter is accepted and
// ignored, which is also what the natively-subscribing sibling drivers do.
// Subscribing before is legal: the interest is recorded,
// every reference reports "no value yet", and the initialize that follows starts the
// pump for it. It does not throw and it does not dial an Agent that is not there yet.
// is null — a caller bug, and the only thing this
// method is loud about, matching 's posture.
public Task<ISubscriptionHandle> SubscribeAsync(
IReadOnlyList<string> fullReferences, TimeSpan publishingInterval, CancellationToken cancellationToken)
{
ArgumentNullException.ThrowIfNull(fullReferences);
// De-duplicated: a reference asked for twice in one subscription is one subscribed value,
// and would otherwise raise two identical callbacks per chunk forever.
var references = new HashSet<string>(StringComparer.Ordinal);
foreach (var reference in fullReferences)
{
if (reference is not null)
{
references.Add(reference);
}
}
var handle = new MTConnectSampleHandle(Interlocked.Increment(ref _nextSubscriptionId));
lock (_subscriptionLock)
{
_subscriptions[handle.SubscriptionId] = new SubscriptionState(handle, references);
// No-op when a pump is already running: every handle shares the one stream.
StartSampleStreamCore(republishOnStart: false);
}
// Raised OUTSIDE the lock — a handler is caller code (see the _subscriptionLock remarks).
// The callback carries the caller's OWN reference (the RawPath); only the index lookup
// behind it is translated to the DataItem id.
var index = ObservationIndex;
var binding = Volatile.Read(ref _binding);
foreach (var reference in references)
{
RaiseDataChange(handle, reference, index.Get(binding.ToDataItemId(reference)));
}
_logger.LogDebug(
"MTConnect driver {DriverInstanceId} opened subscription {SubscriptionId} over {ReferenceCount} reference(s).",
_driverInstanceId, handle.DiagnosticId, references.Count);
return Task.FromResult<ISubscriptionHandle>(handle);
}
/// <inheritdoc />
// Drops the handle's references and stops the shared stream when the last subscription
// goes. An unknown handle — already unsubscribed, or issued by a different driver — is a
// no-op rather than an error: this runs on teardown paths, including failure paths, where
// throwing would mask whatever prompted the teardown.
// is null.
public async Task UnsubscribeAsync(ISubscriptionHandle handle, CancellationToken cancellationToken)
{
ArgumentNullException.ThrowIfNull(handle);
if (handle is not MTConnectSampleHandle ours)
{
return;
}
bool stopStream;
lock (_subscriptionLock)
{
if (!_subscriptions.Remove(ours.SubscriptionId))
{
return;
}
stopStream = !HasSubscribedReferencesCore();
}
if (stopStream)
{
// Outside the lock: StopSampleStreamAsync awaits the pump, and the pump takes this same
// lock to read the subscription registry. Holding it here would deadlock the two.
// The caller's token bounds the wait: DriverInstanceActor puts a 5 s budget around this
// call, and until now that budget was inert.
await StopSampleStreamAsync(cancellationToken).ConfigureAwait(false);
}
_logger.LogDebug(
"MTConnect driver {DriverInstanceId} closed subscription {SubscriptionId}{StreamNote}.",
_driverInstanceId, ours.DiagnosticId, stopStream ? " and stopped the shared /sample stream" : string.Empty);
}
// ---- ITagDiscovery: the /probe device model, streamed as folders + variables ----
/// <inheritdoc />
// This one member is the whole browse opt-in. The universal
// DiscoveryDriverBrowser turns any driver that answers true here into a live
// address picker by capturing — so this driver ships no browser
// code, no per-driver picker component, and no entry in the browser's browse-patch table
// (its discovery is unconditional; there is no "enable browse" option to turn on first).
// It is answered on an un-initialized instance without dialling anything, because
// CanBrowse constructs a throwaway driver purely to read it — which is also why the
// constructor opens nothing.
public bool SupportsOnlineDiscovery => true;
/// <inheritdoc />
// Once, not the default UntilStable: streams the
// complete device model within the single call — /probe is one synchronous request
// that either answers with the whole hierarchy or fails — so nothing fills in asynchronously
// after connect and a settle loop would only re-fetch an identical answer. (The one thing
// that does change the model is an Agent restart, and that is
// IRediscoverable's job, not a retry cadence's.)
public DiscoveryRediscoverPolicy RediscoverPolicy => DiscoveryRediscoverPolicy.Once;
/// <inheritdoc />
// Streams the Agent's /probe hierarchy verbatim: one folder per Device,
// one per nested Component to arbitrary depth, and one variable per
// DataItem under whichever container declares it — including the data items a
// device declares directly, outside any component (an availability item almost
// always lives there, and a walker that only recursed Components would silently
// drop it).
// is the DataItem@id, never the
// browse name. The universal browser commits a browsed leaf's FullName
// straight into TagConfig.FullName, and that is the key
// and the /sample pump resolve an observation against
// ( is keyed by DataItem@id). Committing
// the name instead would produce a picker that looks right and authors tags that
// can never report a value. The browse name — cosmetic — prefers the DataItem's
// name and falls back to its id, which for this protocol is the common
// case rather than an edge case.
// The type shape comes from 's returned
// record, in full — and
// included. Recomputing the array shape
// locally from representation/sampleCount is the tempting shortcut and it
// is wrong: the inference also demotes DATA_SET/TABLE to
// even on a SAMPLE, and honours
// TIME_SERIES only on a SAMPLE. The AdminUI typed editor calls the same
// function, so any divergence here makes the picker and the editor disagree about one
// data item. The type attribute is passed verbatim for the same reason:
// the probe document spells it UPPER_SNAKE (PART_COUNT) where the streams
// document uses PascalCase, and the inference is separator-insensitive precisely to
// bridge the two — pre-normalising it here would defeat that.
// is this seam's job, deliberately not
// the inference's: a CONDITION data item is an alarm.
// IVariableHandle.MarkAsAlarmCondition is not called — that annotation
// needs the driver-side refs of an alarm's sibling attributes (in-alarm flag, priority,
// ack target), and an MTConnect condition has none: it is one text observation whose
// value is the state word. Core's generic node manager enriches on the flag alone.
// The model comes from , never from
// . may have
// dropped the cache after an Agent restart; reading the field directly would make such
// a driver browse as an Agent with no data items, permanently.
// This throws, and that is the opposite of 's posture on
// purpose. A read has a per-reference Bad status code to report a failure in-band; a
// discovery stream has no such channel, so the only way to "fail quietly" would be to
// emit an empty tree — indistinguishable from an Agent that genuinely declares nothing,
// and read by an operator in the picker as a working connection to an empty machine
// (empty is not an answer).
// What the callers actually do with the throw — worth stating precisely, because
// the reassuring version ("it is logged and retried") is not true here. The live
// consumer is the /raw browse-commit path
// (DiscoveryDriverBrowser/BrowserSessionService), which surfaces it to the
// operator as a failed browse — that is the case this posture is chosen for.
// DriverInstanceActor.HandleRediscoverAsync catches it, logs a Warning, and
// publishes an EMPTY node set; it does not reschedule, because retrying is a
// behaviour and this driver is
// . And that publish currently reaches
// nothing: DriverHostActor.HandleDiscoveredNodes short-circuits unconditionally
// (discovered-node injection is dormant in v3 — raw tags are authored through
// browse-commit instead), so at that seam the throw-versus-empty choice has no runtime
// effect today. It is made for the browse path, and for whatever re-enables injection.
// Nothing here writes the observation index. Discovery is a pure read of the
// device model — the /sample pump remains its sole writer.
// is null.
// The driver is not initialized, or it was torn down while the /probe fetch was in
// flight (see ).
public async Task DiscoverAsync(IAddressSpaceBuilder builder, CancellationToken cancellationToken)
{
ArgumentNullException.ThrowIfNull(builder);
var model = await GetOrFetchProbeModelAsync(cancellationToken).ConfigureAwait(false);
// Read once: a concurrent re-initialize must not scope half this walk to one device and half
// to another.
// Both read once, together: a concurrent re-initialize must not scope half this walk to one
// device and half to another, nor stamp half its leaves writable and half not.
var options = Volatile.Read(ref _options);
var deviceScope = options.DeviceName?.Trim();
var writeEnabled = options.Write.Enabled;
var devices = 0;
var variables = 0;
// Siblings that resolve to one folder name share ONE folder (#804) — see SiblingFolders.
var deviceFolders = new SiblingFolders(builder);
foreach (var device in model.Devices)
{
if (device is null || !InDeviceScope(device, deviceScope))
{
continue;
}
var name = FolderName(device.Name, device.Id);
if (string.IsNullOrWhiteSpace(name))
{
// Neither a name nor an id: there is no browse path that could identify this device,
// and folding it in would emit a blank tree segment (and a blank NodeId component
// downstream). Its data items are unreachable either way.
_logger.LogWarning(
"MTConnect driver {DriverInstanceId} skipped a device in {AgentUri}'s /probe model that declares neither a name nor an id; it cannot be given a browse path.",
_driverInstanceId, AgentDisplay(options));
continue;
}
devices++;
variables += StreamContainer(device, deviceFolders.Get(name), writeEnabled);
}
// A configured scope that matches NO device is an authoring error — almost always a typo or
// a device renamed in the Agent — and its only symptom is a browse that succeeds and shows
// nothing. Reported at Warning, because at Debug the operator's evidence is an empty picker
// and no explanation. An agent-wide browse that finds nothing is a different statement (the
// Agent declares no devices) and stays a Debug tally.
if (devices == 0 && !string.IsNullOrEmpty(deviceScope))
{
_logger.LogWarning(
"MTConnect driver {DriverInstanceId} is scoped to device '{DeviceScope}', but {AgentUri}'s /probe model declares no device with that name or id ({DeclaredCount} declared). The browse is empty because the scope matched nothing, not because the Agent has nothing.",
_driverInstanceId, deviceScope, AgentDisplay(options), model.Devices.Count);
return;
}
_logger.LogDebug(
"MTConnect driver {DriverInstanceId} streamed {DeviceCount} device(s) and {VariableCount} data item(s) from {AgentUri}'s /probe model{DeviceScope}.",
_driverInstanceId,
devices,
variables,
AgentDisplay(options),
deviceScope is null or "" ? string.Empty : $" (scoped to '{deviceScope}')");
}
/// <summary>
/// Streams one device/component's own data items into <paramref name="scope"/>, then recurses
/// into each nested component under a folder of its own. Returns how many variables it
/// streamed, for the diagnostic tally.
/// </summary>
/// <remarks>
/// Walks the tree rather than using <c>AllDataItems()</c>: that helper flattens, and the
/// folder structure — which item belongs to which component — is precisely what a browse
/// picker exists to show. Recursion depth is bounded by the parser, which refuses a probe
/// document nested beyond its own limit.
/// </remarks>
/// <param name="container">The device or component whose data items are being streamed.</param>
/// <param name="scope">The builder folder they belong under.</param>
/// <param name="writeEnabled">
/// <see cref="MTConnectWriteOptions.Enabled"/>, read once by the caller so one walk cannot
/// stamp half its leaves under an old options document and half under a new one.
/// </param>
private static int StreamContainer(
IMTConnectComponentContainer container, IAddressSpaceBuilder scope, bool writeEnabled)
{
var streamed = 0;
foreach (var dataItem in container.DataItems)
{
// An id-less data item cannot be read, subscribed, or committed as a tag — there would be
// nothing to key it by. The parser already requires the attribute; this is the guard for a
// model built any other way.
if (dataItem is null || string.IsNullOrWhiteSpace(dataItem.Id))
{
continue;
}
// Every field the DataItem declares, straight through — the array shape is READ OFF the
// result, never recomputed here. See the DiscoverAsync remarks.
var inferred = MTConnectDataTypeInference.Infer(
dataItem.Category,
dataItem.Type,
dataItem.Units,
dataItem.Representation,
dataItem.SampleCount);
var browseName = string.IsNullOrWhiteSpace(dataItem.Name) ? dataItem.Id : dataItem.Name;
scope.Variable(
browseName,
browseName,
new DriverAttributeInfo(
FullName: dataItem.Id,
DriverDataType: inferred.DataType,
IsArray: inferred.IsArray,
ArrayDim: inferred.ArrayDim,
// Operate exactly for the leaves WriteAsync would accept — the "a node cannot
// advertise writability the driver would refuse" principle. With write.enabled
// off (the default) every leaf stays ViewOnly, which is what this driver shipped.
//
// ⚠️ This does NOT make the committed tag writable over OPC UA, and must not be
// read as doing so. Nothing persists SecurityClass: RawBrowseCommitMapper.MapLeaf
// hardcodes TagAccessLevel.Read, Tag has no SecurityClass column, and the
// CurrentWrite bit is composed from Tag.AccessLevel alone
// (OtOpcUaNodeManager.ComposeAccessLevel). The stamp reaches the browse session's
// attribute side-channel and stops there, so an operator still has to author
// ReadWrite on the tag. It is kept honest anyway — a browse that advertised
// ViewOnly for a leaf this driver now writes would be a lie an operator reads.
SecurityClass: CanWrite(writeEnabled, dataItem)
? SecurityClassification.Operate
: SecurityClassification.ViewOnly,
// Historization is authored per tag, never inferred from a device model.
IsHistorized: false,
IsAlarm: IsCondition(dataItem.Category)));
streamed++;
}
var componentFolders = new SiblingFolders(scope);
foreach (var component in container.Components)
{
if (component is null)
{
continue;
}
// Guarded on the RESOLVED folder name, not on the id alone: a component with a blank id
// but a real name browses perfectly well (the id is only the fallback), while one with
// neither would open a folder named "" — a blank segment in every browse path beneath
// it. The data-item guard above is stricter for a different reason: THAT id is the
// correlation key an observation is resolved by, so a blank one is unreadable whatever
// it is called.
var name = FolderName(component.Name, component.Id);
if (string.IsNullOrWhiteSpace(name))
{
continue;
}
streamed += StreamContainer(component, componentFolders.Get(name), writeEnabled);
}
return streamed;
}
/// <summary>
/// One builder folder per distinct name under one parent scope: two sibling containers that resolve to the
/// same folder name (two components named alike, or one's <c>name</c> equal to another's <c>id</c>) are
/// streamed into a SINGLE shared folder, deterministically, rather than as two same-named folders.
/// </summary>
/// <remarks>
/// <b>Why (#804, Java-driver parity).</b> A browse picker keys a folder by its path, so two same-named sibling
/// folders share one key: the universal browser's capture (<c>CapturingAddressSpaceBuilder</c>) indexes the
/// FIRST and every expand of that path returns the first one's children — the second container's data items
/// were listed in the tree but unreachable. Merging is what the Java driver does for the same collision, and
/// it loses nothing: every data item is still streamed, keyed by its own DataItem id. A data item whose name
/// equals a sibling folder's is NOT a collision here — a leaf is keyed by its DataItem id, never its path.
/// </remarks>
private sealed class SiblingFolders(IAddressSpaceBuilder parent)
{
private readonly Dictionary<string, IAddressSpaceBuilder> _folders = new(StringComparer.Ordinal);
/// <summary>The folder named <paramref name="name"/> under the parent, created on first use.</summary>
public IAddressSpaceBuilder Get(string name)
{
if (!_folders.TryGetValue(name, out var folder))
{
folder = parent.Folder(name, name);
_folders[name] = folder;
}
return folder;
}
}
/// <summary>
/// Whether <see cref="WriteAsync"/> would accept a write to <paramref name="dataItem"/> — the
/// browse-time answer, asked of the same two facts the runtime asks of.
/// </summary>
/// <param name="writeEnabled">Whether <c>write.enabled</c> is on.</param>
/// <param name="dataItem">The declared data item.</param>
/// <remarks>
/// The device-name leg of the runtime refusal is not asked here and does not need to be: a
/// leaf reached by this walk is <i>inside</i> a device the model declares, so its owning device
/// is known by construction.
/// </remarks>
private static bool CanWrite(bool writeEnabled, MTConnectDataItem dataItem) =>
writeEnabled && !IsCondition(dataItem.Category);
/// <summary>
/// Whether a device is in this instance's configured scope. An unset scope is agent-wide.
/// </summary>
/// <remarks>
/// <para>
/// Matched against the device's <c>name</c> <b>or</b> its <c>id</c> — a device model may
/// omit the name, and its id is then what an operator would put in the config and what a
/// scoped Agent URL would carry.
/// </para>
/// <para>
/// <b>Belt-and-braces, not dead code.</b> The production client already narrows the
/// request to <c>{AgentUri}/{DeviceName}/probe</c>, so a conforming Agent answers with the
/// one device anyway. But an Agent (or an intermediary) that ignored the path segment and
/// answered agent-wide would otherwise publish every other machine's data items into an
/// instance scoped to one, and the operator's only evidence would be a picker full of ids
/// they never configured. Matching is <b>exact</b> for the same reason the URL is: a
/// differently-cased device name is not the device the Agent would have served.
/// </para>
/// </remarks>
private static bool InDeviceScope(MTConnectDevice device, string? deviceScope) =>
string.IsNullOrEmpty(deviceScope)
|| string.Equals(device.Name, deviceScope, StringComparison.Ordinal)
|| string.Equals(device.Id, deviceScope, StringComparison.Ordinal);
/// <summary>The folder name for a device/component: its <c>name</c>, or its <c>id</c> when it declares none.</summary>
private static string FolderName(string? name, string id) => string.IsNullOrWhiteSpace(name) ? id : name;
/// <summary>
/// Whether a data item's category makes it an alarm. Lives here rather than in
/// <see cref="MTConnectDataTypeInference"/> because that function answers a <i>type</i>
/// question; this is an address-space one.
/// </summary>
private static bool IsCondition(string? category) =>
category.AsSpan().Trim().Equals("CONDITION", StringComparison.OrdinalIgnoreCase);
// ---- IHostConnectivityProbe: one host per Agent, on its own cheap tick ----
/// <summary>Raised when the Agent host connectivity state changes.</summary>
/// <remarks>
/// <b>Raised from the probe loop's task, never under a lock and never under
/// <see cref="_lifecycle"/>.</b> A handler is arbitrary caller code: one that blocks forever
/// blocks the loop with it (and therefore the teardown that awaits it), exactly as an
/// <see cref="OnDataChange"/> handler does to the pump. One that <i>throws</i> is absorbed.
/// </remarks>
public event EventHandler<HostStatusChangedEventArgs>? OnHostStatusChanged;
/// <summary>
/// The single host this driver instance talks to, named by its configured
/// <see cref="MTConnectDriverOptions.AgentUri"/>. Mirrors <c>ModbusDriver.HostName</c>'s
/// <c>host:port</c> convention: several MTConnect drivers in one server disambiguate on the
/// Admin dashboard by endpoint rather than by driver-instance id.
/// </summary>
public string HostName => AgentDisplay(Volatile.Read(ref _options));
/// <summary>
/// R06:T-8 — the redacted display form of <paramref name="options"/>' <c>AgentUri</c> (no <c>user:pass@</c>),
/// used for <see cref="HostName"/>, every log line and every message that can reach health text. Computed once
/// per options document and cached by reference.
/// </summary>
/// <param name="options">The options whose Agent is being named.</param>
/// <returns>The display form; identical to the configured value when it carries no credentials.</returns>
private string AgentDisplay(MTConnectDriverOptions options)
{
var cached = Volatile.Read(ref _agentDisplay);
if (cached is not null && ReferenceEquals(cached.Options, options))
{
return cached.Display;
}
var fresh = new AgentDisplayCache(options, MTConnectUriRedaction.Redact(options.AgentUri));
Volatile.Write(ref _agentDisplay, fresh);
return fresh.Display;
}
/// <summary>The last computed <see cref="AgentDisplay"/>, keyed by the options instance it was computed for.</summary>
private AgentDisplayCache? _agentDisplay;
private sealed record AgentDisplayCache(MTConnectDriverOptions Options, string Display);
/// <inheritdoc />
// One entry, always — an Agent is a single host, and the driver knows of it whether or not
// it has been reached. Before the first probe tick (and for the whole life of an instance
// with Probe.Enabled = false) that entry reads , which
// is the truth: nothing has measured it. See for why a successful
// initialize does not seed it .
public IReadOnlyList<HostConnectivityStatus> GetHostStatuses()
{
lock (_probeLock)
{
return [new HostConnectivityStatus(HostName, _hostState, _hostStateChangedUtc)];
}
}
/// <summary>
/// Starts the periodic connectivity probe, if the operator asked for one. <b>Caller must hold
/// <see cref="_lifecycle"/>.</b>
/// </summary>
/// <param name="client">The client this session owns; the loop dials that one, never a later one.</param>
/// <param name="options">The options this session started with.</param>
private void StartProbeLoopCore(IMTConnectAgentClient client, MTConnectDriverOptions options)
{
if (!options.Probe.Enabled)
{
return;
}
var cts = new CancellationTokenSource();
_probeCts = cts;
Volatile.Write(ref _probeTask, Task.Run(() => RunProbeLoopAsync(client, options, cts.Token), CancellationToken.None));
}
/// <summary>
/// The connectivity probe: wait, ask the Agent whether it is answering, publish the verdict,
/// repeat. Never throws — it is a detached background task, and an escaping exception would
/// silently end host reporting while <see cref="GetHostStatuses"/> kept serving the last
/// thing it saw.
/// </summary>
/// <remarks>
/// <para>
/// <b>It calls <see cref="IMTConnectAgentClient.ProbeAsync"/> directly and publishes
/// nothing.</b> Both halves of that are deliberate.
/// </para>
/// <para>
/// Routing it through <see cref="GetOrFetchProbeModelAsync"/> is the obvious shortcut and
/// it is <b>inert</b>: that accessor answers from the cache <see cref="InitializeAsync"/>
/// already warmed, so the loop would issue no request at all and report
/// <see cref="HostState.Running"/> forever no matter what the Agent was doing — a feature
/// that ticks, logs, and measures nothing.
/// </para>
/// <para>
/// Publishing the answer into <see cref="AgentSession.ProbeModel"/> is the opposite
/// mistake: it would put a second writer on a field the lifecycle owns under a
/// compare-and-swap, and would silently re-warm a cache that an Agent restart — see
/// <see cref="AnnounceAgentRestart"/> — deliberately dropped. The parsed model is
/// discarded; only the fact that a request completed is used.
/// </para>
/// <para>
/// <b>Why <c>/probe</c> rather than the <c>/sample</c> heartbeat.</b> The heartbeat is
/// free, but it only exists while something is subscribed — an instance with no
/// subscriptions would report <see cref="HostState.Unknown"/> indefinitely, and host
/// reachability would become a function of OPC UA client behaviour. A dedicated tick is
/// the one signal that is true whether or not anyone is watching. Its cost used to be a parse
/// of the whole device model per interval; it is now one request whose response headers are
/// read and whose body is dropped (<see cref="IMTConnectAgentClient.PingAsync"/>, R06:T-7).
/// </para>
/// <para>
/// <b>It waits first, then probes.</b> <see cref="StartCoreAsync"/> has just completed a
/// successful <c>/probe</c> one moment earlier; a tick that fired immediately would spend
/// a second identical request to learn nothing, and would do it on every re-initialize.
/// </para>
/// <para>
/// <b>It never calls a lifecycle method</b> — see the <see cref="_lifecycle"/> remarks.
/// <see cref="StopProbeLoopAsync"/> awaits this task while holding that semaphore, so
/// reaching back into one would be a permanent hang with no exception and no log line.
/// </para>
/// </remarks>
private async Task RunProbeLoopAsync(
IMTConnectAgentClient client, MTConnectDriverOptions options, CancellationToken ct)
{
try
{
while (!ct.IsCancellationRequested)
{
try
{
await _probeScheduler(options.Probe.Interval, ct).ConfigureAwait(false);
}
catch (OperationCanceledException)
{
return;
}
if (ct.IsCancellationRequested)
{
return;
}
bool reachable;
try
{
using var deadline = CancellationTokenSource.CreateLinkedTokenSource(ct);
deadline.CancelAfter(options.Probe.Timeout);
// Headers only — the device model is neither downloaded in full nor parsed (R06:T-7).
await client.PingAsync(deadline.Token).ConfigureAwait(false);
reachable = true;
}
catch (OperationCanceledException) when (ct.IsCancellationRequested)
{
return;
}
catch (Exception ex)
{
// Unreachable, unusable, disposed, or past its deadline — from the host's point
// of view these are one fact, and the data paths report the detail.
reachable = false;
_logger.LogDebug(
ex,
"MTConnect driver {DriverInstanceId} could not reach {AgentUri} on its connectivity probe.",
_driverInstanceId, AgentDisplay(options));
}
TransitionHostTo(reachable ? HostState.Running : HostState.Stopped);
}
}
catch (Exception ex)
{
_logger.LogError(
ex,
"MTConnect driver {DriverInstanceId} stopped its connectivity probe on an unexpected fault; host status for {AgentUri} will not be updated until the driver is re-initialized.",
_driverInstanceId, AgentDisplay(options));
}
}
/// <summary>
/// Publishes a host state and announces it — <b>only when it actually changed</b>. Raising on
/// every tick would emit one event per interval per Agent forever: Core would re-fan Bad
/// quality across the host's subtree each time, and an operator watching the feed could not
/// tell a new outage from an ongoing one.
/// </summary>
private void TransitionHostTo(HostState newState)
{
HostState old;
lock (_probeLock)
{
old = _hostState;
if (old == newState)
{
return;
}
_hostState = newState;
_hostStateChangedUtc = DateTime.UtcNow;
}
_logger.LogInformation(
"MTConnect driver {DriverInstanceId} host {AgentUri} went {Previous} -> {Current}.",
_driverInstanceId, HostName, old, newState);
// Outside the lock: a handler is caller code.
try
{
OnHostStatusChanged?.Invoke(this, new HostStatusChangedEventArgs(HostName, old, newState));
}
catch (Exception ex)
{
_logger.LogWarning(
ex,
"MTConnect driver {DriverInstanceId} caught a fault from a host-status subscriber for {AgentUri}; the probe continues.",
_driverInstanceId, HostName);
}
}
/// <summary>
/// Stops the connectivity probe loop and waits for it to be gone. Idempotent, and never
/// throws — it runs on teardown paths where a second exception would mask the first.
/// </summary>
/// <remarks>
/// <b>⚠️ This runs while <see cref="_lifecycle"/> is held</b> and it awaits the loop, on
/// exactly the same terms as <see cref="StopSampleStreamAsync"/>: safe only because the loop
/// never touches the lifecycle and every await it makes observes the token cancelled below.
/// The wait is bounded by <see cref="TeardownWait"/> and the caller's token for the same
/// reason: an <see cref="OnHostStatusChanged"/> handler that never returns would otherwise
/// wedge the Akka dispatcher thread <c>DriverInstanceActor.PostStop</c> blocks on.
/// </remarks>
/// <param name="cancellationToken">The caller's deadline for the teardown wait.</param>
private async Task StopProbeLoopAsync(CancellationToken cancellationToken = default)
{
var cts = _probeCts;
var task = Volatile.Read(ref _probeTask);
_probeCts = null;
Volatile.Write(ref _probeTask, null);
if (cts is null)
{
return;
}
try
{
await cts.CancelAsync().ConfigureAwait(false);
}
catch (ObjectDisposedException)
{
// Raced another stop; the loop is already going away.
}
var stopped = true;
if (task is not null)
{
try
{
await task.WaitAsync(TeardownWait, cancellationToken).ConfigureAwait(false);
}
catch (Exception ex) when (ex is TimeoutException or OperationCanceledException)
{
stopped = false;
_logger.LogError(
ex,
"MTConnect driver {DriverInstanceId} gave up waiting for its connectivity probe to stop after {TeardownWait}; the usual cause is an OnHostStatusChanged subscriber that blocks. Teardown continues.",
_driverInstanceId, TeardownWait);
}
catch (Exception ex)
{
// RunProbeLoopAsync is written not to throw, so this is belt-and-braces.
_logger.LogDebug(
ex,
"MTConnect driver {DriverInstanceId} swallowed a fault from its connectivity probe while stopping it.",
_driverInstanceId);
}
}
if (stopped)
{
// Only once the loop is provably gone — see StopSampleStreamAsync for why.
cts.Dispose();
}
}
// ---- IRediscoverable: the Agent instanceId watch ----
/// <summary>Raised when the Agent instance id changes, so the host should rediscover.</summary>
/// <remarks>
/// <para>
/// <b>This event is what makes <see cref="RediscoverPolicy"/> =
/// <see cref="DiscoveryRediscoverPolicy.Once"/> safe.</b> The runtime runs exactly one
/// discovery pass per connect, so an Agent that restarts mid-run and renumbers, adds, or
/// drops data items would otherwise leave a stale address space sitting behind a
/// perfectly Healthy driver.
/// </para>
/// <para>
/// <b>Raised from the <c>/sample</c> pump's task, outside every lock.</b> A handler that
/// throws is absorbed and logged — a consumer bug must not tear down the stream serving
/// every subscriber. A handler that <i>blocks</i> blocks the pump, and one that
/// synchronously awaits a lifecycle method on this driver deadlocks it: teardown waits
/// for the pump, and the pump is waiting inside the handler. Core's consumer schedules
/// the rediscovery rather than running it inline, which is the only safe shape.
/// </para>
/// </remarks>
public event EventHandler<RediscoveryEventArgs>? OnRediscoveryNeeded;
/// <summary>
/// Handles an Agent restart observed on the wire: drop the device model it invalidated, then
/// tell Core to re-discover — <b>once per distinct <c>instanceId</c></b>.
/// </summary>
/// <remarks>
/// <para>
/// <b>Dropping the cached <c>/probe</c> model is not housekeeping — it is what stops this
/// whole capability being inert.</b> Core's response to the event is to re-run
/// <see cref="DiscoverAsync"/>, which reads
/// <see cref="GetOrFetchProbeModelAsync"/>; with the cache still warm that call answers
/// from a model describing a process that no longer exists, and the "rediscovery" would
/// diff the stale tree against itself and change nothing. Dropped through a
/// compare-and-swap, so a lifecycle change landing mid-drop wins rather than being
/// undone.
/// </para>
/// <para>
/// <b>Once per change, not once per chunk.</b> Every chunk after a restart carries the new
/// id, and a re-baseline that fails leaves the pump still comparing against the old one —
/// so the guard is a latch on the id that was last announced, not on the pump's cursor
/// state. A second, different restart is a second announcement; an id the Agent reuses
/// after some other id is one too, because that is genuinely a new process.
/// </para>
/// </remarks>
/// <param name="instanceId">The <c>instanceId</c> the Agent's chunk reported.</param>
/// <param name="options">The options in force, for the log line's endpoint.</param>
private void AnnounceAgentRestart(long instanceId, MTConnectDriverOptions options)
{
if (Interlocked.Exchange(ref _announcedInstanceId, instanceId) == instanceId)
{
return;
}
InvalidateProbeModel();
_logger.LogInformation(
"MTConnect driver {DriverInstanceId} dropped its cached /probe device model for {AgentUri} and raised rediscovery: agent instanceId is now {InstanceId}.",
_driverInstanceId, AgentDisplay(options), instanceId);
try
{
OnRediscoveryNeeded?.Invoke(this, new RediscoveryEventArgs("MTConnect agent instanceId changed", null));
}
catch (Exception ex)
{
_logger.LogWarning(
ex,
"MTConnect driver {DriverInstanceId} caught a fault from a rediscovery subscriber; the /sample stream continues.",
_driverInstanceId);
}
}
/// <summary>
/// Drops the cached <c>/probe</c> device model so the next consumer re-fetches it.
/// <c>internal</c> rather than <c>private</c> so the discovery tests can exercise the real
/// drop path; production calls it from <see cref="AnnounceAgentRestart"/>.
/// Compare-and-swap, and <b>retried</b> — this drop is a correctness requirement rather
/// than a courtesy: losing the race must not leave a model from a dead Agent installed.
/// (The courtesy version, <c>FlushOptionalCachesAsync</c>, took the same lock-free
/// approach but gave up on a lost race; it went with the rest of the unconsumed
/// memory-pressure surface.)
/// </summary>
internal void InvalidateProbeModel()
{
while (true)
{
var session = Volatile.Read(ref _session);
if (session?.ProbeModel is null)
{
return;
}
var dropped = session with { ProbeModel = null };
if (ReferenceEquals(Interlocked.CompareExchange(ref _session, dropped, session), session))
{
return;
}
}
}
/// <summary>Disposes and forgets superseded pumps that have finished. <b>Caller must hold <see cref="_subscriptionLock"/>.</b></summary>
private void PruneSupersededPumpsCore()
{
for (var i = _supersededPumps.Count - 1; i >= 0; i--)
{
if (!_supersededPumps[i].Task.IsCompleted)
{
continue;
}
_supersededPumps[i].Cts.Dispose();
_supersededPumps.RemoveAt(i);
}
}
/// <summary>The retired-but-unjoined pump tasks (test barrier).</summary>
internal Task[] SupersededPumpTasks
{
get
{
lock (_subscriptionLock)
{
return [.. _supersededPumps.Select(p => p.Task)];
}
}
}
/// <summary>
/// Starts the shared <c>/sample</c> pump, if there is anything to pump and anything to pump
/// from. <b>Caller must hold <see cref="_subscriptionLock"/>.</b>
/// </summary>
/// <param name="republishOnStart">
/// Whether the pump should republish the observation index to every subscriber before it
/// opens the stream — true when a lifecycle start replaced the baseline the subscribers'
/// current values came from.
/// </param>
private void StartSampleStreamCore(bool republishOnStart)
{
// `IsCompleted: false`, not `is not null`: a pump that has already exited is not a running
// stream, and conflating the two means a pump killed by an unexpected fault could never be
// revived by a resubscribe. The genuinely-unrepeatable case is the latch below, not this.
if (_pumpTask is { IsCompleted: false })
{
if (_pumpGeneration == _sessionGeneration)
{
return;
}
// A Subscribe can land between StopSampleStreamAsync clearing the registry slots and
// the teardown that retires the client — StopSampleStreamAsync is still awaiting the OLD pump
// then — and starts a pump on the session about to be disposed. Treating that pump as "the
// running stream" would leave the freshly-installed session with no live stream at all
// (the stale pump just reads its disposed client as a dropped connection). Retire it: it is
// cancelled and can only fail against its disposed client.
try { _pumpCts?.Cancel(); }
catch (ObjectDisposedException) { /* already going away */ }
// Retain (don't drop) the retired pump so shutdown can join it; its source is disposed
// by whoever observes it complete.
PruneSupersededPumpsCore();
if (_pumpCts is not null && _pumpTask is not null)
{
_supersededPumps.Add((_pumpCts, _pumpTask));
_pumpCts = null;
}
_pumpTask = null;
_logger.LogInformation(
"MTConnect driver {DriverInstanceId} replaced a /sample pump bound to a retired Agent session.",
_driverInstanceId);
}
if (_streamUnsupported)
{
// Refusing IS correct — reconnecting reproduces the identical answer forever — but it
// must never be silent. DriverInstanceActor handles a tag-set change as
// Unsubscribe-then-Subscribe with NO re-initialize, and the unsubscribe stops a stream
// that is not running; without this, the resubscribe returns a handle, the caller is
// told SubscriptionEstablished, and the next successful read reports a perfectly green
// driver over a subscription that can never deliver a value.
var agentUri = AgentDisplay(Volatile.Read(ref _options));
_streamDegraded = true;
Degrade($"The MTConnect Agent at '{agentUri}' does not serve a framed /sample stream; subscriptions cannot deliver until the driver is re-initialized.");
_logger.LogWarning(
"MTConnect driver {DriverInstanceId} refused to start a /sample stream for {AgentUri}: the endpoint was already found not to stream. The subscription is registered but will receive NO updates until the driver is re-initialized.",
_driverInstanceId, agentUri);
return;
}
if (!HasSubscribedReferencesCore())
{
return;
}
// One read of the session: the client the pump enumerates and the cursor it opens at are
// guaranteed to belong to the same initialize.
var session = Volatile.Read(ref _session);
if (session is null)
{
return;
}
// A previous pump ran to completion; its cancellation source is ours to release before the
// field is overwritten, or every revived stream leaks one.
_pumpCts?.Dispose();
_pumpTask = null;
// A new stream generation gets a fresh subscriber-fault Warning budget (see
// _subscriberFaultLogged).
Interlocked.Exchange(ref _subscriberFaultLogged, 0);
var options = Volatile.Read(ref _options);
var cts = new CancellationTokenSource();
_pumpCts = cts;
_pumpGeneration = _sessionGeneration;
var generation = _sessionGeneration;
// Not live until it has actually delivered (or re-baselined): a pump revived by a resubscribe opens at a
// cursor from when the last one stopped, and until it catches up the index is that old.
_pumpLive = false;
_pumpTask = Task.Run(
() => RunSampleStreamAsync(
session.Client, options, session.NextSequence, session.InstanceId, republishOnStart, generation, cts.Token),
CancellationToken.None);
}
/// <summary>
/// The shared <c>/sample</c> pump: reads chunks, keeps the observation index and the
/// subscribers up to date, and owns every way the stream can stop.
/// </summary>
/// <remarks>
/// <para>
/// <b>Never throws.</b> It is a detached background task; an escaping exception would be
/// an unobserved fault that silently ends the driver's subscription surface while every
/// handle still looks alive.
/// </para>
/// <para>
/// <b>Never calls a lifecycle method.</b> Its re-baseline goes straight to
/// <see cref="IMTConnectAgentClient.CurrentAsync"/> on the client it already holds — see
/// <see cref="RebaselineAsync"/> and the <see cref="_lifecycle"/> remarks for why
/// "just re-initialize" would be a silent permanent deadlock.
/// </para>
/// <para>
/// <b>Backoff applies to failures only — and to a re-baseline that resolved nothing.</b>
/// A re-baseline behind a stream that delivered is a protocol event, not a fault, so it
/// reopens immediately and resets the attempt count. One on a connection that delivered
/// nothing advances <see cref="BackoffFor"/> like any other failure, because the
/// <see cref="InvalidDataException"/> path that reaches it also carries conditions a
/// <c>/current</c> cannot resolve (an unparseable chunk, an over-sized part) and would
/// otherwise reopen-and-fail as fast as the Agent can answer.
/// </para>
/// </remarks>
private async Task RunSampleStreamAsync(
IMTConnectAgentClient client,
MTConnectDriverOptions options,
long from,
long instanceId,
bool republishOnStart,
long generation,
CancellationToken ct)
{
var cursor = from;
var attempt = 0;
_reconnectAttempts = 0;
var counts = new SampleCountState(options.SampleCount);
Volatile.Write(ref _effectiveSampleCount, counts.Count);
// #804 (Java-driver parity) — consecutive re-baselines with no /sample chunk delivered in between.
var rebaselinesWithoutProgress = 0;
try
{
if (republishOnStart)
{
FanOut(SubscribedReferences(), ObservationIndex);
// and the conditions, diffed against what the server was last told — so a restart
// delivers what changed while the driver was down and re-raises nothing that did not.
EmitConditionTransitions(touchedDataItemIds: null, resync: false);
}
while (!ct.IsCancellationRequested)
{
if (attempt > 0)
{
var delay = BackoffFor(attempt, options.Reconnect);
if (delay > TimeSpan.Zero)
{
try
{
await Task.Delay(delay, ct).ConfigureAwait(false);
}
catch (OperationCanceledException)
{
return;
}
}
}
var outcome = await ConsumeStreamAsync(
client, options, cursor, instanceId, generation, counts,
holdStalled: rebaselinesWithoutProgress >= MaxRebaselinesWithoutProgress, ct)
.ConfigureAwait(false);
if (outcome.DeliveredAny)
{
rebaselinesWithoutProgress = 0;
}
cursor = outcome.Cursor;
instanceId = outcome.InstanceId;
// A pump whose session a restart replaced (retired, or abandoned after the bounded teardown wait)
// acts on no verdict: its Degrade / latch would land on the NEW session's health (R06:T-1).
if (!IsCurrentSessionGeneration(generation))
{
return;
}
// R06:T-5 — the adaptive count. A stream that delivered vindicates the count it was opened with; a
// RAISED count whose stream delivered nothing may be one the Agent refuses (count above its buffer),
// so step back to the last count that worked and stop raising for this pump.
if (outcome.DeliveredAny)
{
counts.LastDelivered = counts.Count;
}
else if (counts.Count > counts.LastDelivered && outcome.Verdict is not StreamVerdict.Cancelled)
{
_logger.LogWarning(
outcome.Error,
"MTConnect driver {DriverInstanceId}: the /sample stream opened with the raised count {Count} delivered nothing; stepping back to {Previous} and not raising it again for this stream.",
_driverInstanceId, counts.Count, counts.LastDelivered);
counts.Count = counts.LastDelivered;
counts.Ceiling = counts.Count;
Volatile.Write(ref _effectiveSampleCount, counts.Count);
}
if (outcome.RaiseCountTo is { } raised)
{
counts.Count = raised;
Volatile.Write(ref _effectiveSampleCount, raised);
}
switch (outcome.Verdict)
{
case StreamVerdict.Cancelled:
return;
case StreamVerdict.Unsupported:
// Configuration, not weather: reconnecting reproduces the identical answer
// forever, so retrying would be an unbounded hot loop against an endpoint
// that will never stream until an operator changes something. Latch it, say
// so loudly, and stop. A re-initialize clears the latch.
// r1 N-4: the latch is set inside the same generation-checked section as the degrade, so a
// superseded pump can never latch Unsupported on the session that replaced it.
if (!TryDegradeStream(generation, outcome.Error!.Message, latchUnsupported: true))
{
return;
}
_logger.LogError(
outcome.Error,
"MTConnect driver {DriverInstanceId} cannot stream /sample from {AgentUri}: the endpoint did not answer with a framed multipart stream. Subscriptions stay registered but will receive no updates until the driver is re-initialized; check for a proxy or load balancer in front of the Agent.",
_driverInstanceId, AgentDisplay(options));
return;
case StreamVerdict.Reopen:
// A re-baseline behind a stream that DELIVERED is the protocol event this
// verdict was written for — a sequence gap or an OUT_OF_RANGE rejection, both
// of which the /current genuinely resolves — so it reopens immediately on a
// clean ladder.
//
// One that delivered NOTHING is not necessarily resolved by anything. The
// InvalidDataException catch that routes OUT_OF_RANGE here also catches an
// unparseable chunk and an over-sized part, and a /current cannot fix either:
// resetting the ladder there reopens instantly, fails identically, and spends
// one /sample AND one /current per iteration forever — the unbounded hot loop
// every other arm of this pump (the Unsupported latch, the backoff floor, the
// non-positive cap) exists to prevent. The discriminator is the same one the
// transient arm below uses, for the same reason.
attempt = outcome.DeliveredAny ? 0 : attempt + 1;
_reconnectAttempts = attempt;
// #804 — the backoff above PACES a stream that only ever re-baselines, but each /current
// also marked the stream delivered, so it read Healthy for ever while no /sample chunk
// arrived (a SampleCount above the Agent's bufferSize is refused OUT_OF_RANGE on
// EVERY open). After MaxRebaselinesWithoutProgress in a row it is reported Degraded, and
// stays so (holdStalled) until a chunk is delivered.
if (!outcome.DeliveredAny &&
++rebaselinesWithoutProgress == MaxRebaselinesWithoutProgress)
{
if (!TryHoldStreamStalled(generation, StalledStreamReason))
{
return;
}
_logger.LogWarning(
outcome.Error,
"MTConnect driver {DriverInstanceId}: {Count} consecutive /current re-baselines of {AgentUri} without a single /sample chunk in between; reporting the stream Degraded until one is delivered. Values are re-baselined, not streamed. An Agent answering every /sample with OUT_OF_RANGE usually means SampleCount is above its bufferSize.",
_driverInstanceId, rebaselinesWithoutProgress, AgentDisplay(options));
}
break;
default:
if (!TryDegradeStream(generation, outcome.Error!.Message))
{
return;
}
// A stream that DELIVERED before it dropped proves the endpoint works, so
// its failure starts a fresh backoff ladder. Without this the counter is
// monotonic for the life of the process: a healthy agent that drops a
// connection once an hour would be pinned at MaxBackoffMs within a day, and
// "the first retry is immediate" would be true exactly once ever.
attempt = outcome.DeliveredAny ? 1 : attempt + 1;
_reconnectAttempts = attempt;
_logger.LogWarning(
outcome.Error,
"MTConnect driver {DriverInstanceId} lost the /sample stream from {AgentUri}; reconnecting from sequence {Cursor} (attempt {Attempt}, delivered={Delivered}).",
_driverInstanceId, AgentDisplay(options), cursor, attempt, outcome.DeliveredAny);
break;
}
}
}
catch (Exception ex)
{
_logger.LogError(
ex,
"MTConnect driver {DriverInstanceId} stopped its /sample pump on an unexpected fault; subscriptions will receive no further updates until the driver is re-initialized.",
_driverInstanceId);
TryDegradeStream(generation, ex.Message);
}
finally
{
// The session's cursor is the opening `from` of the NEXT pump, and until now it only
// ever held the priming /current's. A last-unsubscribe + resubscribe (no re-initialize)
// would therefore reopen at a sequence the Agent has long since evicted: it self-heals
// via OUT_OF_RANGE, but only after a wasted round trip, and a still-buffered stale
// cursor replays historical observations to subscribers first. One publish per pump
// lifetime, dropped if a lifecycle change has already installed a newer session (R06:T-1: the
// generation this pump was started against is what says so — the session alone cannot).
PublishAgentCursor(generation, instanceId, cursor);
if (IsCurrentSessionGeneration(generation))
{
_pumpLive = false;
}
}
}
/// <summary>
/// Enumerates one <c>/sample</c> connection until it ends, and classifies how it ended.
/// Re-baselines are performed here, <b>inside</b> the enumeration, so the index and the
/// subscribers are up to date before the stream is reopened.
/// </summary>
private async Task<StreamOutcome> ConsumeStreamAsync(
IMTConnectAgentClient client,
MTConnectDriverOptions options,
long cursor,
long instanceId,
long generation,
SampleCountState counts,
bool holdStalled,
CancellationToken ct)
{
// The RUNNING cursor the gap check is made against: the previous chunk's nextSequence, equal
// to the opening `from` only for the first chunk. Comparing every chunk against the opening
// `from` instead reports a gap on every chunk once the Agent's buffer rolls past it — an
// endless /current re-baseline storm against a perfectly healthy stream.
var expected = cursor;
// Whether THIS connection ever handed us a chunk. A stream that delivered and then dropped
// is a working endpoint having a bad moment; one that never delivered may be an endpoint
// that cannot work at all. They must not share a backoff ladder — see RunSampleStreamAsync.
var delivered = false;
// Consecutive chunks that said this stream is behind the Agent (R06:T-5).
var behindChunks = 0;
try
{
await foreach (var chunk in client.SampleAsync(cursor, counts.Count, ct).ConfigureAwait(false))
{
if (chunk is null)
{
continue;
}
// R06:T-1 — a pump whose session a lifecycle restart has replaced (it was retired, or its
// teardown wait was abandoned) must not fold a late chunk of the OLD Agent connection into
// the NEW session's index — nor read it as an Agent restart and clear that index.
if (!IsCurrentSessionGeneration(generation))
{
return new StreamOutcome(StreamVerdict.Cancelled, expected, instanceId, delivered, null);
}
// CHECKED BEFORE THE GAP: an Agent restart changes instanceId *and* resets
// sequences, so a restart usually trips IsSequenceGap too — and the two need
// different handling. A gap keeps the held values (they are still this device's);
// a restart invalidates every one of them, because the device model they describe
// no longer exists. Testing the gap first would misdiagnose the restart and keep
// serving values the new Agent may never report again.
if (chunk.InstanceId != instanceId)
{
_logger.LogWarning(
"MTConnect driver {DriverInstanceId} saw {AgentUri} restart (instanceId {Previous} -> {Current}); dropping every held observation and re-baselining.",
_driverInstanceId, AgentDisplay(options), instanceId, chunk.InstanceId);
// Dropped BEFORE the re-baseline, and the subscribers are told: if the /current
// then fails, holding stale values from a dead device model would be worse than
// holding none.
if (!TryDropForAgentRestart(generation, chunk.InstanceId, options))
{
return new StreamOutcome(StreamVerdict.Cancelled, expected, instanceId, delivered, null);
}
return await RebaselineAsync(
client, options, expected, chunk.InstanceId, delivered, generation, holdStalled, ct)
.ConfigureAwait(false);
}
if (IMTConnectAgentClient.IsSequenceGap(expected, chunk))
{
_logger.LogWarning(
"MTConnect driver {DriverInstanceId} fell out of {AgentUri}'s buffer (expected sequence {Expected}, oldest retained {FirstSequence}); re-baselining from /current.",
_driverInstanceId, AgentDisplay(options), expected, chunk.FirstSequence);
return await RebaselineAsync(client, options, expected, instanceId, delivered, generation, holdStalled, ct)
.ConfigureAwait(false);
}
if (!ApplyAndFanOut(chunk, generation))
{
return new StreamOutcome(StreamVerdict.Cancelled, expected, instanceId, delivered, null);
}
delivered = true;
// EVERY chunk advances the cursor, including an observation-free heartbeat: the
// Agent sends those precisely so a quiet connection can be told from a dead one, and
// they carry the sequence forward. Not advancing on one makes the NEXT chunk look
// like a gap.
expected = chunk.NextSequence;
// Review I-1: a pump wedged inside the fan-out above can come back after a restart replaced its
// session — its "the stream delivered" must then land nowhere.
if (!TryMarkStreamDelivered(generation))
{
return new StreamOutcome(StreamVerdict.Cancelled, expected, instanceId, delivered, null);
}
// R06:T-5 — a stream persistently behind the Agent reopens at the cursor it reached with a larger
// count, rather than falling out of the ring buffer and re-baselining every buffer-length.
if (!MeasureSampleLag(chunk, counts.Count))
{
behindChunks = 0;
}
else if (++behindChunks >= BehindChunksBeforeRaise)
{
behindChunks = 0;
if (NextSampleCount(counts, options, expected) is { } next)
{
return new StreamOutcome(StreamVerdict.Reopen, expected, instanceId, delivered, null, next);
}
}
}
// The seam contracts that cancellation is the ONLY way this enumeration ends quietly, so
// a quiet end under a live token is a non-conforming client — reported as the lost
// stream it is rather than treated as "the subscription is simply idle".
return ct.IsCancellationRequested
? new StreamOutcome(StreamVerdict.Cancelled, expected, instanceId, delivered, null)
: new StreamOutcome(
StreamVerdict.Transient,
expected,
instanceId,
delivered,
new MTConnectStreamEndedException(
MTConnectStreamEndReason.ConnectionClosed, AgentDisplay(options), 0));
}
catch (OperationCanceledException) when (ct.IsCancellationRequested)
{
return new StreamOutcome(StreamVerdict.Cancelled, expected, instanceId, delivered, null);
}
catch (MTConnectStreamNotSupportedException ex)
{
return new StreamOutcome(StreamVerdict.Unsupported, expected, instanceId, delivered, ex);
}
catch (InvalidDataException ex)
{
// THE OTHER HALF OF RING-BUFFER OVERFLOW, and the one IsSequenceGap cannot see: a real
// cppagent answers a `from` that has already fallen out of its buffer with an
// MTConnectError / OUT_OF_RANGE document served under HTTP 200, which the client
// surfaces here rather than as a gap-bearing chunk. Re-baselining on the gap alone would
// mean the driver recovers from falling a little behind but not from falling a lot.
_logger.LogWarning(
ex,
"MTConnect driver {DriverInstanceId} could not resume {AgentUri}'s /sample stream at sequence {Cursor} (the Agent rejected it as out of range); re-baselining from /current.",
_driverInstanceId, AgentDisplay(options), cursor);
return await RebaselineAsync(client, options, expected, instanceId, delivered, generation, holdStalled, ct)
.ConfigureAwait(false);
}
catch (Exception ex)
{
// Transient by default: a dropped connection, a closing boundary, the heartbeat
// watchdog's TimeoutException. Reconnect under backoff.
return new StreamOutcome(StreamVerdict.Transient, expected, instanceId, delivered, ex);
}
}
/// <summary>
/// Re-primes the observation index from a fresh <c>/current</c> and reports where the stream
/// should resume. The recovery path for every way the incremental sequence can be broken:
/// a ring-buffer overflow (detected as a gap, or refused outright as <c>OUT_OF_RANGE</c>)
/// and an Agent restart.
/// </summary>
/// <remarks>
/// <b>This calls the client directly and takes no lock.</b> Re-baselining by calling
/// <see cref="ReinitializeAsync"/> — the obvious way to write it, since that method already
/// does exactly this work — would take the non-reentrant <see cref="_lifecycle"/> semaphore
/// from inside a pump that a lifecycle method may already be waiting on, and hang the driver
/// permanently with no exception and no log line.
/// </remarks>
private async Task<StreamOutcome> RebaselineAsync(
IMTConnectAgentClient client,
MTConnectDriverOptions options,
long fallbackCursor,
long fallbackInstanceId,
bool deliveredAny,
long generation,
bool holdStalled,
CancellationToken ct)
{
try
{
var current = await BoundedAsync(client.CurrentAsync, "/current", options.RequestTimeoutMs, ct)
.ConfigureAwait(false);
if (!IsCurrentSessionGeneration(generation))
{
return new StreamOutcome(
StreamVerdict.Cancelled, fallbackCursor, fallbackInstanceId, deliveredAny, null);
}
// #804 — the restart test runs on the SNAPSHOT too (the Java driver's rule). On a real 2.x Agent this
// is how a restart is usually first seen: the restarted process refuses the dead one's cursor with
// HTTP 400 OUT_OF_RANGE, so no chunk ever reaches the in-stream instanceId check above, and the
// re-baseline used to adopt the new instanceId silently — no rediscovery prompt, and the dead model's
// held values (data items the new Agent may not declare) kept being served.
if (current.InstanceId != fallbackInstanceId)
{
_logger.LogWarning(
"MTConnect driver {DriverInstanceId} saw {AgentUri} restart while re-baselining (instanceId {Previous} -> {Current}); dropping every held observation.",
_driverInstanceId, AgentDisplay(options), fallbackInstanceId, current.InstanceId);
if (!TryDropForAgentRestart(generation, current.InstanceId, options))
{
return new StreamOutcome(
StreamVerdict.Cancelled, fallbackCursor, fallbackInstanceId, deliveredAny, null);
}
}
PublishAgentCursor(generation, current.InstanceId, current.NextSequence);
// A stalled stream (#804) keeps its Degraded verdict across the re-baseline: the /current answered, but
// it is not the /sample progress the verdict is about. Only a delivered chunk clears it.
if (!ApplyAndFanOut(current, generation, baseline: true) ||
!(holdStalled && !deliveredAny
? TryHoldStreamStalled(generation, StalledStreamReason)
: TryMarkStreamDelivered(generation)))
{
return new StreamOutcome(
StreamVerdict.Cancelled, fallbackCursor, fallbackInstanceId, deliveredAny, null);
}
_logger.LogInformation(
"MTConnect driver {DriverInstanceId} re-baselined from {AgentUri}'s /current ({ObservationCount} observation(s), agent instanceId {InstanceId}) and resumes /sample from sequence {NextSequence}.",
_driverInstanceId,
AgentDisplay(options),
current.Observations.Count,
current.InstanceId,
current.NextSequence);
return new StreamOutcome(
StreamVerdict.Reopen, current.NextSequence, current.InstanceId, deliveredAny, null);
}
catch (OperationCanceledException) when (ct.IsCancellationRequested)
{
return new StreamOutcome(
StreamVerdict.Cancelled, fallbackCursor, fallbackInstanceId, deliveredAny, null);
}
catch (Exception ex)
{
// The Agent is unreachable or unusable right now. Treated as an ordinary transient
// failure so the reconnect backoff applies — without it, an Agent that is down would be
// re-baselined against as fast as the loop can spin. The cursor is deliberately left
// where it was: reopening there either works or lands right back here.
return new StreamOutcome(
StreamVerdict.Transient, fallbackCursor, fallbackInstanceId, deliveredAny, ex);
}
}
/// <summary>
/// The reconnect delay before attempt <paramref name="attempt"/>, as a pure function — the
/// backoff policy is asserted directly rather than inferred from how long a test slept.
/// </summary>
/// <remarks>
/// The first retry honours <see cref="MTConnectReconnectOptions.MinBackoffMs"/> (<c>0</c> by
/// default = immediate, which is what an operator wants after a one-off blip); every later
/// one multiplies, from a <see cref="BackoffGrowthFloorMs"/> floor, up to
/// <see cref="MTConnectReconnectOptions.MaxBackoffMs"/>. Operator-authored nonsense cannot
/// un-bound the loop: a multiplier that cannot grow the delay falls back to
/// <see cref="DefaultBackoffMultiplier"/>, a non-positive cap falls back to
/// <see cref="DefaultMaxBackoffMs"/> (clamping to it would make EVERY delay zero — a spin,
/// not a backoff), and a negative minimum clamps to zero.
/// </remarks>
/// <param name="attempt">The 1-based reconnect attempt about to be made.</param>
/// <param name="reconnect">The authored backoff options.</param>
/// <returns>How long to wait before that attempt.</returns>
internal static TimeSpan BackoffFor(int attempt, MTConnectReconnectOptions reconnect)
{
ArgumentNullException.ThrowIfNull(reconnect);
// A non-positive cap is "unset", NOT "no cap". Math.Max(0, ...) would clamp every delay to
// zero and turn the reconnect loop into a spin against a refused connection.
var capMs = reconnect.MaxBackoffMs > 0 ? reconnect.MaxBackoffMs : DefaultMaxBackoffMs;
var minMs = Math.Max(0, reconnect.MinBackoffMs);
if (attempt <= 1)
{
return TimeSpan.FromMilliseconds(Math.Min(minMs, capMs));
}
var multiplier = reconnect.BackoffMultiplier > 1.0 ? reconnect.BackoffMultiplier : DefaultBackoffMultiplier;
var delayMs = (double)Math.Max(minMs, BackoffGrowthFloorMs);
for (var step = 2; step <= attempt; step++)
{
delayMs *= multiplier;
if (delayMs >= capMs)
{
return TimeSpan.FromMilliseconds(capMs);
}
}
return TimeSpan.FromMilliseconds(Math.Min(delayMs, capMs));
}
/// <summary>
/// Folds a <c>/current</c> snapshot or one <c>/sample</c> chunk into the shared index and
/// publishes every reference it touched to the handles that subscribe it.
/// </summary>
/// <param name="document">The document to fold in.</param>
/// <param name="generation">The session generation of the pump doing it.</param>
/// <param name="baseline">Whether it is a <c>/current</c> baseline.</param>
/// <returns><see langword="false"/> (nothing applied) when the pump's session has been replaced.</returns>
private bool ApplyAndFanOut(MTConnectStreamsResult document, long generation, bool baseline = false)
{
// Captured with the generation check under one lock (review m3): a stale chunk can never fold into the
// session state a restart installed after the pump's last check.
if (!TryCaptureSessionState(generation, out var index, out var tracker))
{
return false;
}
index.Apply(document, baseline);
// CONDITION observations also feed the alarm path, which keeps its own multi-condition state
// (a /sample Normal for one nativeCode is not a return to normal while another code is active).
var touchedConditions = tracker.Apply(document, baseline);
EmitConditionTransitions(touchedConditions, resync: false);
if (document.Observations is not { Count: > 0 })
{
return true;
}
// De-duplicated in document order: one Agent document may report a data item more than once
// (a Condition container carries several simultaneously-active states), and the index has
// already reconciled those into a single snapshot.
//
// The Agent reports DataItem ids; subscribers hold RawPaths. Each touched id is therefore
// EXPANDED into the references it is published under — every RawPath bound to it (a DataItem
// may legitimately back more than one authored raw tag), plus the id itself for the
// dataItemId-keyed authoring surface (the driver CLI). Publishing the id instead of the
// RawPath would miss DriverHostActor's NodeId table and drop the value silently.
var binding = Volatile.Read(ref _binding);
var touched = new List<string>(document.Observations.Count);
var seen = new HashSet<string>(StringComparer.Ordinal);
foreach (var observation in document.Observations)
{
if (observation is null || string.IsNullOrWhiteSpace(observation.DataItemId))
{
continue;
}
if (seen.Add(observation.DataItemId))
{
binding.AppendReferences(observation.DataItemId, touched);
}
}
FanOut(touched, index);
return true;
}
/// <summary>
/// The observation index and condition tracker of the session <paramref name="generation"/> names, read with
/// the generation check under <see cref="_subscriptionLock"/> — the lock <see cref="StartCoreAsync"/> installs
/// them, the session and the generation under. <see langword="false"/> once that session is gone.
/// </summary>
private bool TryCaptureSessionState(
long generation, out MTConnectObservationIndex index, out MTConnectConditionTracker tracker)
{
lock (_subscriptionLock)
{
index = Volatile.Read(ref _index);
tracker = Volatile.Read(ref _conditions);
return _sessionGeneration == generation;
}
}
/// <summary>
/// The stream of the pump started under <paramref name="generation"/> failed: degrade it — under the generation
/// check, in one critical section, so a superseded pump's failure never lands on the session that replaced it.
/// </summary>
/// <param name="generation">The pump's session generation.</param>
/// <param name="message">The failure (redacted text only).</param>
/// <returns><see langword="false"/> when the pump's session has been replaced (nothing written).</returns>
/// <param name="latchUnsupported">Also latch the "endpoint cannot stream" verdict (r1 N-4).</param>
private bool TryDegradeStream(long generation, string message, bool latchUnsupported = false)
{
// The mirror of TryMarkStreamDelivered: a superseded pump's failure is not the new session's.
lock (_subscriptionLock)
{
if (_sessionGeneration != generation)
{
return false;
}
if (latchUnsupported)
{
_streamUnsupported = true;
}
_streamDegraded = true;
Degrade(message);
return true;
}
}
/// <summary>
/// An Agent restart (a changed <c>instanceId</c>) seen by the pump started under <paramref name="generation"/>:
/// drop every held observation and condition state — the device model they describe may no longer exist —
/// tell the subscribers, and raise rediscovery (once per distinct id).
/// </summary>
/// <returns><see langword="false"/> when the pump's session has been replaced (nothing dropped).</returns>
private bool TryDropForAgentRestart(long generation, long newInstanceId, MTConnectDriverOptions options)
{
if (!TryCaptureSessionState(generation, out var restartIndex, out var restartTracker))
{
return false;
}
// Not live while the index is empty (review m2): reads go to /current until the re-baseline lands.
_pumpLive = false;
restartIndex.Clear();
restartTracker.Clear();
FanOut(SubscribedReferences(), restartIndex);
// The driver's own state is consistent by this point, so it is safe to hand the restart to arbitrary
// caller code — which may call straight back into DiscoverAsync.
AnnounceAgentRestart(newInstanceId, options);
return true;
}
/// <summary>
/// #804: the pump started under <paramref name="generation"/> has re-baselined
/// <see cref="MaxRebaselinesWithoutProgress"/> times in a row without one <c>/sample</c> chunk: degrade the
/// stream — generation-checked, like <see cref="TryDegradeStream"/> — but stamp
/// <c>LastSuccessfulRead</c> now, because each <c>/current</c> it re-baselined from IS the Agent answering.
/// That is deliberate: the comms-loss watchdog reconnects on non-Healthy AND a frozen stamp, and a reconnect
/// cannot fix an Agent that refuses the count or cursor — it would only reconnect-loop the driver (the
/// "an Agent refusal is a reply" health contract).
/// </summary>
/// <param name="generation">The pump's session generation.</param>
/// <param name="message">The type-only reason.</param>
/// <returns><see langword="false"/> when the pump's session has been replaced (nothing written).</returns>
private bool TryHoldStreamStalled(long generation, string message)
{
lock (_subscriptionLock)
{
if (_sessionGeneration != generation)
{
return false;
}
_streamDegraded = true;
if (ReadHealth().State != DriverState.Faulted)
{
WriteHealth(new DriverHealth(DriverState.Degraded, DateTime.UtcNow, message));
}
return true;
}
}
/// <summary>
/// Review I-1: the stream of the pump started under <paramref name="generation"/> delivered — clear its
/// degradation, mark it live and publish Healthy, all under the generation check and in one critical section,
/// so a superseded pump (wedged in a subscriber past its last check) can never restore a replaced session's
/// health or make the read path trust a stream that is down. Nothing here calls caller code.
/// </summary>
/// <param name="generation">The pump's session generation.</param>
/// <returns><see langword="false"/> when the pump's session has been replaced (nothing written).</returns>
private bool TryMarkStreamDelivered(long generation)
{
lock (_subscriptionLock)
{
if (_sessionGeneration != generation)
{
return false;
}
_streamDegraded = false;
_pumpLive = true;
// r2 N-5 (coordinator): a delivered chunk proves the Agent is alive — the MTConnect analogue of OpcUaClient's
// good keep-alive clearing a read/write Degraded — so it also clears an unanswered-write degradation, which
// otherwise stuck forever on a subscription-only driver. Generation-checked with the rest (above).
_writeDegraded = false;
PublishHealthy(DateTime.UtcNow);
return true;
}
}
/// <summary>
/// Publishes the index's current snapshot of <paramref name="references"/> to every handle
/// that subscribes them. References nobody subscribed raise nothing: the Agent streams the
/// whole device, and republishing all of it would flood the server with values no client
/// asked for.
/// </summary>
/// <remarks>
/// <paramref name="references"/> are <b>subscriber</b> references — RawPaths under v3 — and
/// each is carried through to the callback unchanged; only the index lookup behind it is
/// translated to the DataItem id the Agent reports under. That asymmetry is the point: the
/// value must arrive under the reference the caller subscribed, or it is routed nowhere.
/// </remarks>
private void FanOut(IReadOnlyList<string> references, MTConnectObservationIndex index)
{
if (references.Count == 0)
{
return;
}
SubscriptionState[] states;
lock (_subscriptionLock)
{
if (_subscriptions.Count == 0)
{
return;
}
states = [.. _subscriptions.Values];
}
var binding = Volatile.Read(ref _binding);
// The lock is released before any callback: handlers are caller code.
foreach (var reference in references)
{
DataValueSnapshot? snapshot = null;
foreach (var state in states)
{
if (!state.References.Contains(reference))
{
continue;
}
// Read once per reference, so every handle sees the identical snapshot.
snapshot ??= index.Get(binding.ToDataItemId(reference));
RaiseDataChange(state.Handle, reference, snapshot);
}
}
}
/// <summary>
/// Raises one data-change callback to <b>every</b> subscriber, absorbing anything each one
/// throws. A consumer bug must not tear down the pump that serves the others — nor rob them
/// of the value.
/// </summary>
/// <remarks>
/// <para>
/// <b>The invocation list is walked by hand, with the try/catch INSIDE the loop.</b> A
/// plain <c>OnDataChange?.Invoke(...)</c> is a multicast call: the first handler that
/// throws aborts the rest of the list, so one broken consumer silently starves every
/// other subscriber of that observation — and a single try/catch around the whole
/// invoke absorbs the exception while leaving the starvation in place, which reads as
/// "isolated" in a log and is not.
/// </para>
/// <para>
/// One <see cref="DataChangeEventArgs"/> is shared across the list deliberately: it is
/// an immutable record, and allocating per handler on the hot path would buy nothing.
/// </para>
/// </remarks>
private void RaiseDataChange(MTConnectSampleHandle handle, string reference, DataValueSnapshot snapshot)
{
var handlers = OnDataChange;
if (handlers is null)
{
return;
}
var args = new DataChangeEventArgs(handle, reference, snapshot);
foreach (var registration in handlers.GetInvocationList())
{
try
{
((EventHandler<DataChangeEventArgs>)registration).Invoke(this, args);
}
catch (Exception ex)
{
LogSubscriberFault(ex, reference, handle);
}
}
}
/// <summary>
/// Reports a faulting data-change subscriber <b>once per stream generation</b> at Warning,
/// and at Debug thereafter.
/// </summary>
/// <remarks>
/// A consumer that throws on every value would otherwise emit one Warning per observation
/// per handle — on a busy Agent, thousands a second, which buries the incident it is
/// reporting along with everything else in the log. Nothing is lost: the later faults are
/// still emitted, at Debug, and the latch is cleared whenever the pump (re)starts.
/// </remarks>
private void LogSubscriberFault(Exception ex, string reference, MTConnectSampleHandle handle)
{
if (Interlocked.Exchange(ref _subscriberFaultLogged, 1) == 0)
{
_logger.LogWarning(
ex,
"MTConnect driver {DriverInstanceId} caught a fault from a data-change subscriber for {Reference} on {SubscriptionId}; the stream continues and every other subscriber still received the value. Further subscriber faults are logged at Debug until the stream restarts.",
_driverInstanceId, reference, handle.DiagnosticId);
return;
}
_logger.LogDebug(
ex,
"MTConnect driver {DriverInstanceId} caught a further fault from a data-change subscriber for {Reference} on {SubscriptionId}.",
_driverInstanceId, reference, handle.DiagnosticId);
}
/// <summary>Every reference any live subscription is interested in.</summary>
private List<string> SubscribedReferences()
{
lock (_subscriptionLock)
{
var all = new HashSet<string>(StringComparer.Ordinal);
foreach (var state in _subscriptions.Values)
{
all.UnionWith(state.References);
}
return [.. all];
}
}
/// <summary>
/// Whether any live subscription actually names a reference. <b>Caller must hold
/// <see cref="_subscriptionLock"/>.</b> A subscription over an empty reference list is legal
/// (a caller that adds tags later) and must not open a stream by itself.
/// </summary>
private bool HasSubscribedReferencesCore()
{
// a CONDITION alarm subscription needs the same stream — a driver whose only authored tags are
// alarms would otherwise never hear a condition change.
if (HasAlarmReferencesCore())
{
return true;
}
foreach (var state in _subscriptions.Values)
{
if (state.References.Count > 0)
{
return true;
}
}
return false;
}
/// <summary>
/// Records where the Agent now is — its <c>instanceId</c> <b>and</b> the sequence a stream
/// should resume from — onto whichever session is installed.
/// </summary>
/// <remarks>
/// <para>
/// <b>Both fields move together, in one compare-and-swap.</b> They are one fact about
/// one Agent process, and the session's own contract is that a reader gets the client
/// and its cursor as a consistent pair. Publishing the id alone (as this did) left the
/// session holding a NEW instanceId beside the cursor of the process that had just
/// died — the exact tear the packing exists to prevent.
/// </para>
/// <para>
/// <b>There is no "the id did not change, so skip" early return</b>, because the two
/// re-baselines that do NOT change the id — a sequence gap and an <c>OUT_OF_RANGE</c>
/// rejection — are precisely the ones whose whole purpose is to move the cursor.
/// Suppressing the write there would leave the next pump reopening at the stale
/// sequence that had just been rejected.
/// </para>
/// <para>
/// Compare-and-swap for the same reason <see cref="InvalidateProbeModel"/> uses one:
/// a lifecycle change landing mid-update has newer state, so this write is dropped
/// rather than replayed over it.
/// </para>
/// </remarks>
/// <param name="instanceId">The Agent instanceId last observed.</param>
/// <param name="nextSequence">The sequence a <c>/sample</c> stream should resume from.</param>
/// <param name="generation">The session generation the publishing pump was started against.</param>
private void PublishAgentCursor(long generation, long instanceId, long nextSequence)
{
// R06:T-1 — under _subscriptionLock, the lock StartCoreAsync installs a new session AND bumps the
// generation under: inside it the two are always observed together, so a pump bound to a retired
// session can never land its cursor on the session that replaced it (a lock-free generation check
// could read the new session beside the not-yet-bumped generation). Nothing is awaited here.
lock (_subscriptionLock)
{
if (_sessionGeneration != generation)
{
return;
}
while (true)
{
var session = Volatile.Read(ref _session);
if (session is null ||
(session.InstanceId == instanceId && session.NextSequence == nextSequence))
{
return;
}
// Still a compare-and-swap: InvalidateProbeModel / GetOrFetchProbeModelAsync swap the session
// lock-free, and a plain write here could undo a probe-model drop.
var updated = session with { InstanceId = instanceId, NextSequence = nextSequence };
if (ReferenceEquals(Interlocked.CompareExchange(ref _session, updated, session), session))
{
return;
}
}
}
}
/// <summary>Whether <paramref name="generation"/> is still the installed session's generation.</summary>
/// <param name="generation">A pump's session generation.</param>
/// <returns><see langword="false"/> once a lifecycle restart has installed a newer session.</returns>
private bool IsCurrentSessionGeneration(long generation)
{
lock (_subscriptionLock)
{
return _sessionGeneration == generation;
}
}
/// <summary>
/// The <c>/probe</c> device model: the cached copy when warm, otherwise a fresh
/// deadline-bounded <c>/probe</c> that re-populates the cache. This is the accessor every
/// model consumer (<c>DiscoverAsync</c>) must use, so that
/// <see cref="InvalidateProbeModel"/> can never leave browse permanently disabled.
/// </summary>
/// <remarks>
/// Lock-free by design — see the <see cref="_session"/> remarks for why this does not take
/// <see cref="_lifecycle"/>. It reads the session <b>once</b>, fetches against that
/// session's client, and re-caches only if that same session is still installed, so a
/// concurrent <see cref="ShutdownAsync"/>/<see cref="ReinitializeAsync"/> can neither be
/// undone by a late re-cache nor leak a model belonging to a client that is gone. A teardown
/// that lands mid-fetch surfaces as the <see cref="InvalidOperationException"/> below rather
/// than as a raw <see cref="ObjectDisposedException"/> from the disposed client.
/// </remarks>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The Agent's device model.</returns>
/// <exception cref="InvalidOperationException">
/// The driver is not initialized, or it was torn down while this fetch was in flight.
/// </exception>
internal async Task<MTConnectProbeModel> GetOrFetchProbeModelAsync(CancellationToken cancellationToken)
{
var session = Volatile.Read(ref _session) ?? throw NotConnected();
if (session.ProbeModel is not null)
{
return session.ProbeModel;
}
MTConnectProbeModel model;
try
{
model = await BoundedAsync(
session.Client.ProbeAsync, "/probe", Volatile.Read(ref _options).RequestTimeoutMs, cancellationToken)
.ConfigureAwait(false);
}
catch (ObjectDisposedException ex)
{
// The lock-free window the _session remarks name: a teardown disposed this client after
// we captured it. Reported as the not-connected error the caller already handles, so
// browse has exactly one failure mode to reason about.
throw NotConnected(ex);
}
// Publish only onto the session we actually fetched against.
var recached = session with { ProbeModel = model };
_ = Interlocked.CompareExchange(ref _session, recached, session);
return model;
}
private InvalidOperationException NotConnected(Exception? inner = null) =>
new($"MTConnect driver '{_driverInstanceId}' has no live agent client; the /probe device model cannot be fetched unless InitializeAsync has succeeded and the driver has not since been shut down or re-initialized.",
inner);
/// <summary>
/// Builds driver options from a <c>DriverConfig</c> JSON document.
/// </summary>
/// <remarks>
/// Lives here rather than on the factory because the driver itself needs it: the runtime
/// delivers config changes as JSON to a live instance (see the class remarks). The
/// <c>MTConnectDriverFactoryExtensions.CreateInstance</c> should <b>delegate to this method</b>
/// rather than deserializing a second DTO — two parsers for one document drift.
/// </remarks>
/// <param name="driverInstanceId">Driver instance id, used only in error text.</param>
/// <param name="driverConfigJson">The driver's <c>DriverConfig</c> JSON.</param>
/// <returns>The parsed options.</returns>
/// <exception cref="InvalidOperationException">The document is unusable or omits <c>agentUri</c>.</exception>
internal static MTConnectDriverOptions ParseOptions(string driverInstanceId, string driverConfigJson)
{
ArgumentException.ThrowIfNullOrWhiteSpace(driverInstanceId);
ArgumentException.ThrowIfNullOrWhiteSpace(driverConfigJson);
MTConnectDriverConfigDto? dto;
try
{
dto = JsonSerializer.Deserialize<MTConnectDriverConfigDto>(driverConfigJson, JsonOptions);
}
catch (JsonException ex)
{
throw new InvalidOperationException(
$"MTConnect driver config for '{driverInstanceId}' is not valid JSON: {ex.Message}", ex);
}
if (dto is null)
{
throw new InvalidOperationException(
$"MTConnect driver config for '{driverInstanceId}' deserialised to null");
}
if (string.IsNullOrWhiteSpace(dto.AgentUri))
{
throw new InvalidOperationException(
$"MTConnect driver config for '{driverInstanceId}' missing required AgentUri");
}
return new MTConnectDriverOptions
{
AgentUri = dto.AgentUri.Trim(),
DeviceName = string.IsNullOrWhiteSpace(dto.DeviceName) ? null : dto.DeviceName.Trim(),
RequestTimeoutMs = dto.RequestTimeoutMs ?? 5_000,
SampleIntervalMs = dto.SampleIntervalMs ?? 1_000,
SampleCount = dto.SampleCount ?? 1_000,
HeartbeatMs = dto.HeartbeatMs ?? 10_000,
Tags = dto.Tags is { Count: > 0 }
? [.. dto.Tags.Select(t => BuildTag(t, driverInstanceId))]
: [],
// v3: the deploy artifact injects the driver's authored raw tags here (see
// DriverDeviceConfigMerger). Bound verbatim — the per-entry TagConfig blob is mapped
// later, at binding-build time, where a bad blob can be skipped rather than failing the
// whole deployment over one tag.
RawTags = dto.RawTags is { Count: > 0 } ? [.. dto.RawTags] : [],
Probe = new MTConnectProbeOptions
{
Enabled = dto.Probe?.Enabled ?? true,
Interval = TimeSpan.FromMilliseconds(dto.Probe?.IntervalMs ?? 5_000),
Timeout = TimeSpan.FromMilliseconds(dto.Probe?.TimeoutMs ?? 2_000),
},
Reconnect = new MTConnectReconnectOptions
{
MinBackoffMs = dto.Reconnect?.MinBackoffMs ?? 0,
MaxBackoffMs = dto.Reconnect?.MaxBackoffMs ?? 30_000,
BackoffMultiplier = dto.Reconnect?.BackoffMultiplier ?? 2.0,
},
// Absent block == disabled: write-back is opt-in, and the agent's own AllowPut has to be
// on as well, so neither layer can arm the other by omission.
Write = new MTConnectWriteOptions
{
Enabled = dto.Write?.Enabled ?? false,
},
};
}
// ---- lifecycle internals (all called with _lifecycle held) ----
/// <summary>
/// Brings the driver up against <paramref name="options"/>, reusing
/// <paramref name="existingClient"/> when the caller determined the transport is unchanged.
/// Nothing is published until BOTH agent calls have succeeded, so a failure can never leave a
/// half-initialized instance behind.
/// </summary>
private async Task StartCoreAsync(
MTConnectDriverOptions options, IMTConnectAgentClient? existingClient, CancellationToken ct)
{
var lastRead = ReadHealth().LastSuccessfulRead;
WriteHealth(new DriverHealth(DriverState.Initializing, lastRead, null));
IMTConnectAgentClient? built = null;
try
{
// arch-review 01/S-6: a 0 authored here must never come to mean "wait forever". Checked
// in the driver, not only in the production client's constructor, so the guard holds for
// any client implementation behind the seam.
RequirePositive(options.RequestTimeoutMs, nameof(MTConnectDriverOptions.RequestTimeoutMs));
RequirePositive(options.HeartbeatMs, nameof(MTConnectDriverOptions.HeartbeatMs));
RequirePositive(options.SampleIntervalMs, nameof(MTConnectDriverOptions.SampleIntervalMs));
RequirePositive(options.SampleCount, nameof(MTConnectDriverOptions.SampleCount));
// Same guard for the probe's two knobs, gated on the feature that reads them. A 0
// interval turns the "periodic" probe into an unthrottled hot loop against the Agent, and
// a 0 (or negative) timeout cancels every probe before it can be answered — pinning the
// host to Stopped forever and reporting an outage that does not exist. Checked only when
// probing is ON because a disabled probe reads neither value, and faulting a deployment
// over a field the driver never touches would buy nothing; flipping Enabled back on is a
// config edit, and ReinitializeAsync runs this same validation.
if (options.Probe.Enabled)
{
RequirePositive(options.Probe.Interval, $"{nameof(MTConnectDriverOptions.Probe)}.{nameof(MTConnectProbeOptions.Interval)}");
RequirePositive(options.Probe.Timeout, $"{nameof(MTConnectDriverOptions.Probe)}.{nameof(MTConnectProbeOptions.Timeout)}");
}
var client = existingClient;
if (client is null)
{
built = _agentClientFactory(options)
?? throw new InvalidOperationException(
$"The MTConnect agent-client factory for driver '{_driverInstanceId}' returned null.");
client = built;
}
var probe = await BoundedAsync(client.ProbeAsync, "/probe", options.RequestTimeoutMs, ct)
.ConfigureAwait(false);
var current = await BoundedAsync(client.CurrentAsync, "/current", options.RequestTimeoutMs, ct)
.ConfigureAwait(false);
// ---- commit point: everything below is non-throwing bookkeeping ----
Volatile.Write(ref _options, options);
// lmxopcua#825 — a SampleCount the Agent will refuse is a configuration error; say so at deploy rather than
// leaving it to surface as three progress-free re-baselines and a Degraded stream.
Interlocked.Exchange(ref _agentBufferSize, probe.BufferSize ?? 0);
if (SampleCountBufferWarning(options.SampleCount, probe.BufferSize) is { } bufferWarning)
{
_logger.LogWarning("MTConnect driver {DriverInstanceId} ({AgentUri}): {Warning}",
_driverInstanceId, AgentDisplay(options), bufferWarning);
}
// Derived from the options AND the /probe model just fetched: a raw tag whose TagConfig
// blob declares no type takes the type the Agent itself declares for that DataItem, so a
// browse-committed tag (whose blob carries only the address) publishes as the number it
// is rather than as text. Published before the index, which is built from it.
var binding = BuildBinding(options, probe);
Volatile.Write(ref _binding, binding);
built = null;
// The Agent this session talks to, as of now. A restart is measured against THIS, so a
// re-initialize against a different Agent process never re-announces the change the
// operator's own action caused.
Volatile.Write(ref _announcedInstanceId, current.InstanceId);
// One publish, so a concurrent lock-free reader sees the new client, the new probe
// cache, and the pump's opening cursor together or none of them.
// The generation bump and the session write are ONE critical section: a Subscribe landing
// between them would start a pump on the OLD session stamped with the NEW generation, and
// the restart's own StartSampleStreamCore would then treat it as current.
var index = new MTConnectObservationIndex(binding.IndexTags);
index.Apply(current, baseline: true);
// the CONDITION tracker is per-session state like the index. Nothing is RAISED here — the
// lifecycle semaphore is held; the restarted pump delivers the diff as its first act.
var conditions = new MTConnectConditionTracker();
conditions.Apply(current, baseline: true);
// The index and the tracker are installed in the SAME critical section as the session and its
// generation (R06:T-1 / review m3): a pump that reads them under this lock with its generation still
// current gets this session's state, never the previous one's, and never the next one's.
lock (_subscriptionLock)
{
Volatile.Write(
ref _session,
new AgentSession(client, probe, current.InstanceId, current.NextSequence));
_sessionGeneration++;
Volatile.Write(ref _index, index);
Volatile.Write(ref _conditions, conditions);
}
// A fresh session has no failures behind it — including the /sample verdict, because a
// re-initialize is precisely the event that means an operator changed something.
_streamDegraded = false;
_readDegraded = false;
_writeDegraded = false;
WriteHealth(new DriverHealth(DriverState.Healthy, DateTime.UtcNow, null));
// Standing subscriptions survive a lifecycle restart; everything they hold came from the
// baseline just thrown away, so the restarted pump republishes as its first act. It is
// the PUMP that republishes rather than this method, deliberately: raising subscriber
// callbacks from here would run caller code on the thread holding the non-reentrant
// lifecycle semaphore, and a handler that touched the driver's lifecycle would deadlock.
lock (_subscriptionLock)
{
_streamUnsupported = false;
StartSampleStreamCore(republishOnStart: true);
}
StartProbeLoopCore(client, options);
_logger.LogInformation(
"MTConnect driver {DriverInstanceId} initialized against {AgentUri}{DeviceScope}: {DeviceCount} device(s), {ObservationCount} primed observation(s), agent instanceId {InstanceId}.",
_driverInstanceId,
AgentDisplay(options),
options.DeviceName is null ? string.Empty : $"/{options.DeviceName}",
probe.Devices.Count,
index.Count,
current.InstanceId);
}
catch (Exception ex)
{
// land Faulted BEFORE anything is disposed, not after.
//
// A successful connect writes Healthy above, and the steps that follow it (stream start, probe
// loop) can still throw. Without this first write the driver would report Healthy for the whole
// duration of the teardown below, with its client already disposed. That window is genuinely
// observable: DriverInstanceActor invokes InitializeAsync on a fire-and-forget Task.Run, so its
// health-poll tick keeps calling GetHealth() on the actor thread throughout.
//
// The write is then REPEATED after the teardown, and both are load-bearing. Teardown drains
// in-flight loops, and those loops write health too — so a driver that only moved the write
// earlier could have a late loop overwrite Faulted with something rosier and leave THAT as the
// terminal state. Conservative during the window, correct at the end.
WriteHealth(new DriverHealth(DriverState.Faulted, lastRead, ex.Message));
// A client this attempt built is unreachable from anywhere else; one the caller handed in
// is disposed by the teardown below. Either way nothing survives a failed start.
SafeDispose(built);
await TeardownCoreAsync().ConfigureAwait(false);
WriteHealth(new DriverHealth(DriverState.Faulted, lastRead, ex.Message));
_logger.LogError(
ex,
"MTConnect driver {DriverInstanceId} failed to initialize against {AgentUri}; the driver is Faulted and serves no values.",
_driverInstanceId, AgentDisplay(options));
throw;
}
}
/// <summary>
/// Releases the agent client and drops every piece of served state. Never throws — it runs
/// on the failure path of <see cref="StartCoreAsync"/>, where a second exception would mask
/// the real one.
/// </summary>
private Task TeardownCoreAsync()
{
// Retire the whole session in one write, BEFORE disposing: a lock-free reader that still
// holds the old reference gets a disposed client (caught + named at each seam), never a
// session that is half-torn-down.
// The retirement bumps the session generation too, so a pump abandoned by the teardown wait can no
// longer write health, flags or index state over what the shutdown / next start writes (review I-1).
AgentSession? session;
lock (_subscriptionLock)
{
session = Volatile.Read(ref _session);
Volatile.Write(ref _session, null);
_sessionGeneration++;
}
SafeDispose(session?.Client);
// A fresh index rather than Clear(): the tag table it coerces against belongs to the options
// that are being torn down. The binding is left in place — it is a pure function of the
// options still installed, and a reference must keep resolving to the tag it names so a read
// against a torn-down driver answers BadWaitingForInitialData ("authored, no value") rather
// than BadNodeIdUnknown ("no such tag").
Volatile.Write(ref _index, new MTConnectObservationIndex(Volatile.Read(ref _binding).IndexTags));
Volatile.Write(ref _conditions, new MTConnectConditionTracker());
return Task.CompletedTask;
}
/// <summary>
/// Stops the shared <c>/sample</c> long-poll stream and waits for the pump to actually be
/// gone. Idempotent, and never throws — it runs on teardown paths where a second exception
/// would mask the first.
/// </summary>
/// <remarks>
/// <para>
/// <b>It waits, rather than merely signalling.</b> Every caller is about to dispose the
/// client the pump is enumerating (or install a new observation index the old pump would
/// write into). Returning while the pump is still unwinding would let it publish one more
/// value into state its owner has already retired — the exact "torn down but still
/// reporting" shape the lifecycle is written to prevent.
/// </para>
/// <para>
/// <b>⚠️ This runs while <see cref="_lifecycle"/> is held</b> (from
/// <see cref="InitializeAsync"/>, <see cref="ReinitializeAsync"/> and
/// <see cref="ShutdownAsync"/>) — and it awaits the pump. That is only safe because the
/// pump never touches <see cref="_lifecycle"/>: its re-baseline calls
/// <see cref="IMTConnectAgentClient.CurrentAsync"/> directly (see
/// <see cref="RebaselineAsync"/>) rather than re-entering a lifecycle method, and every
/// await it makes observes the pump token cancelled below. Break either property and
/// this await is a permanent hang with no exception and no log line.
/// </para>
/// <para>
/// <b>The wait is bounded</b> (<see cref="TeardownWait"/>, and the caller's token) — the
/// one thing outside the guarantee above is a subscriber callback, and an
/// <see cref="OnDataChange"/> handler that never returns would otherwise block teardown
/// forever. That is not merely slow: <c>DriverInstanceActor.PostStop</c> blocks an Akka
/// dispatcher thread on <see cref="ShutdownAsync"/> while this driver holds
/// <see cref="_lifecycle"/>, so an unbounded wait wedges an actor-system thread and
/// every later lifecycle call with it. Abandoning the wait is safe because the pump's
/// registry slots are cleared BEFORE it: a pump left running cannot be resurrected into
/// <see cref="_pumpTask"/>, and it is already cancelled.
/// </para>
/// </remarks>
/// <param name="cancellationToken">The caller's deadline for the teardown wait.</param>
private async Task StopSampleStreamAsync(CancellationToken cancellationToken = default)
{
CancellationTokenSource? cts;
Task? task;
(CancellationTokenSource Cts, Task Task)[] retired;
lock (_subscriptionLock)
{
cts = _pumpCts;
task = _pumpTask;
_pumpCts = null;
_pumpTask = null;
retired = [.. _supersededPumps];
_supersededPumps.Clear();
}
// Join pumps retired by a session replacement (already cancelled): they must not outlive
// teardown and write health / the reconnect counter over the next session.
foreach (var (retiredCts, retiredTask) in retired)
{
try
{
await retiredCts.CancelAsync().ConfigureAwait(false);
await retiredTask.WaitAsync(TeardownWait, cancellationToken).ConfigureAwait(false);
retiredCts.Dispose();
}
catch (Exception ex)
{
_logger.LogDebug(
ex,
"MTConnect driver {DriverInstanceId} could not join a superseded /sample pump within {TeardownWait}; it is cancelled and abandoned.",
_driverInstanceId, TeardownWait);
}
}
if (cts is null)
{
return;
}
try
{
await cts.CancelAsync().ConfigureAwait(false);
}
catch (ObjectDisposedException)
{
// Raced another stop; the pump is already going away.
}
var stopped = true;
if (task is not null)
{
try
{
await task.WaitAsync(TeardownWait, cancellationToken).ConfigureAwait(false);
}
catch (Exception ex) when (ex is TimeoutException or OperationCanceledException)
{
stopped = false;
_logger.LogError(
ex,
"MTConnect driver {DriverInstanceId} gave up waiting for its /sample pump to stop after {TeardownWait}. The pump is cancelled and can publish nothing further, but something inside it is not returning — the usual cause is an OnDataChange subscriber that blocks. Teardown continues.",
_driverInstanceId, TeardownWait);
}
catch (Exception ex)
{
// RunSampleStreamAsync is written not to throw, so this is belt-and-braces — but a
// teardown that rethrew would replace whatever failure prompted it.
_logger.LogDebug(
ex,
"MTConnect driver {DriverInstanceId} swallowed a fault from its /sample pump while stopping the stream.",
_driverInstanceId);
}
}
if (stopped)
{
// Disposed ONLY once the pump is provably gone: an abandoned pump still holds this
// source's token, and disposing underneath it turns its next linked-token creation into
// an ObjectDisposedException. One leaked CTS beats a mystery exception.
cts.Dispose();
}
// Nothing to be degraded about once the stream is gone — UNLESS the endpoint is latched as
// unable to stream at all, in which case there is no prospect of one coming back, and
// clearing this would let the next successful read report a green driver over a subscription
// that can never deliver. See StartSampleStreamCore.
lock (_subscriptionLock)
{
if (!_streamUnsupported)
{
_streamDegraded = false;
}
}
}
/// <summary>
/// Resolves the options a lifecycle call should apply, reporting an unreadable document
/// consistently for both entry points before any state is destroyed.
/// </summary>
/// <remarks>
/// <para>
/// One rule, deliberately shared by <see cref="InitializeAsync"/> and
/// <see cref="ReinitializeAsync"/>: <b>a config document that cannot be read never
/// destroys working state; it faults only a driver that had none.</b>
/// </para>
/// <para>
/// With a live session the driver keeps serving its previous, valid configuration and
/// health is left alone — rejecting the edit already fails the deployment
/// (<c>DriverInstanceActor</c> turns the throw into <c>ApplyResult(false)</c>), and
/// downing a working driver over an unreadable document is a far larger blast radius
/// than the change that was refused. Publishing <see cref="DriverState.Faulted"/> while
/// still serving values would also be a lie in the other direction.
/// </para>
/// <para>
/// With no live session there is nothing to protect and every tag is already dark, so
/// the failure must reach <see cref="GetHealth"/> as <see cref="DriverState.Faulted"/> —
/// otherwise the dashboard keeps showing whatever state the driver was in before, which
/// is the failure-that-looks-like-success shape this codebase exists to avoid.
/// </para>
/// </remarks>
private MTConnectDriverOptions ResolveIncomingOptions(string driverConfigJson)
{
try
{
return HasConfigBody(driverConfigJson)
? ParseOptions(_driverInstanceId, driverConfigJson)
: Volatile.Read(ref _options);
}
catch (Exception ex)
{
if (Volatile.Read(ref _session) is not null)
{
_logger.LogError(
ex,
"MTConnect driver {DriverInstanceId} rejected a new driver config document and kept running its previous configuration.",
_driverInstanceId);
}
else
{
WriteHealth(new DriverHealth(DriverState.Faulted, ReadHealth().LastSuccessfulRead, ex.Message));
_logger.LogError(
ex,
"MTConnect driver {DriverInstanceId} could not read its driver config document and has no running configuration to fall back on; the driver is Faulted.",
_driverInstanceId);
}
throw;
}
}
/// <summary>
/// True when <paramref name="driverConfigJson"/> carries a real config body — i.e. a JSON
/// object or array with <b>at least one element</b>. The bootstrapper always passes a
/// populated document; tests and probe-only callers pass a semantically empty one, which
/// keeps the constructor-supplied options.
/// </summary>
/// <remarks>
/// <para>
/// <b>Emptiness is decided by parsing, not by string comparison.</b> Matching the
/// literals <c>"{}"</c> / <c>"[]"</c> misses every other spelling of the same document —
/// <c>"{ }"</c>, a pretty-printed <c>"{\n}"</c>, <c>"{ /* nothing yet */ }"</c> — and
/// each of those would then reach <see cref="ParseOptions"/>, fail the required-AgentUri
/// check, and turn a semantically empty document into a cold-start fault or a rejected
/// deployment. That is precisely the asymmetry the keep-the-constructor-options rule
/// exists to prevent.
/// </para>
/// <para>
/// <b>A malformed document is emphatically NOT empty</b> and returns <c>true</c>, so
/// <see cref="ParseOptions"/> runs and produces the real, quoted parse error. Treating
/// unreadable text as "no config supplied" would silently swallow a corrupt document and
/// start the driver on stale options — a failure wearing success's clothes.
/// </para>
/// </remarks>
private static bool HasConfigBody(string? driverConfigJson)
{
if (string.IsNullOrWhiteSpace(driverConfigJson))
{
return false;
}
try
{
using var document = JsonDocument.Parse(driverConfigJson, DocumentOptions);
return document.RootElement.ValueKind switch
{
JsonValueKind.Object => document.RootElement.EnumerateObject().Any(),
JsonValueKind.Array => document.RootElement.EnumerateArray().Any(),
// A bare `null` document is the JSON spelling of "nothing supplied".
JsonValueKind.Null => false,
// A scalar root is not a config object at all; let ParseOptions say so.
_ => true,
};
}
catch (JsonException)
{
return true;
}
}
/// <summary>
/// Whether the new options change anything <see cref="IMTConnectAgentClient"/> reads at
/// construction, and therefore cannot be applied to a live client. See
/// <see cref="ReinitializeAsync"/> for why this is wider than the plan's AgentUri/DeviceName.
/// </summary>
private static bool RequiresClientRebuild(MTConnectDriverOptions current, MTConnectDriverOptions incoming) =>
!string.Equals(current.AgentUri, incoming.AgentUri, StringComparison.Ordinal)
|| !string.Equals(current.DeviceName, incoming.DeviceName, StringComparison.Ordinal)
|| current.RequestTimeoutMs != incoming.RequestTimeoutMs
|| current.HeartbeatMs != incoming.HeartbeatMs
|| current.SampleIntervalMs != incoming.SampleIntervalMs
|| current.SampleCount != incoming.SampleCount;
/// <summary>
/// Runs one unary agent call under a deadline linked to the caller's token. Belt-and-braces
/// over the production client's own per-call deadline: the seam does not oblige an
/// implementation to bound itself, and an unbounded initialize would wedge the driver-host
/// actor.
/// </summary>
private async Task<T> BoundedAsync<T>(
Func<CancellationToken, Task<T>> call, string request, int timeoutMs, CancellationToken ct)
{
using var deadline = CancellationTokenSource.CreateLinkedTokenSource(ct);
deadline.CancelAfter(TimeSpan.FromMilliseconds(timeoutMs));
try
{
return await call(deadline.Token).ConfigureAwait(false);
}
catch (OperationCanceledException) when (!ct.IsCancellationRequested)
{
// A bare "A task was canceled" names neither the Agent nor the request that stalled.
throw new TimeoutException(
$"MTConnect {request} request for driver '{_driverInstanceId}' against '{AgentDisplay(Volatile.Read(ref _options))}' did not complete within {timeoutMs} ms.");
}
}
/// <summary>
/// One Bad-coded snapshot per requested reference — the shape every read failure degrades
/// to. Arity is preserved because the caller indexes the result positionally against the
/// references it asked for.
/// </summary>
private static DataValueSnapshot[] BadBatch(int count, uint status)
{
// No source timestamp: nothing was observed. The server timestamp is when we gave up.
var serverTimestamp = DateTime.UtcNow;
var results = new DataValueSnapshot[count];
for (var i = 0; i < count; i++)
{
results[i] = new DataValueSnapshot(null, status, null, serverTimestamp);
}
return results;
}
/// <summary>
/// Routes a read failure to the health surface (the fleet-wide shape): log it and
/// degrade, preserving <c>LastSuccessfulRead</c> — the "when did this driver last hear from
/// the Agent" diagnostic. Never downgrades <see cref="DriverState.Faulted"/>, which is a
/// stronger, config-level signal that a transient read failure must not paper over.
/// </summary>
private void DegradeAfterReadFailure(Exception ex)
{
_readDegraded = true;
Degrade(ex);
_logger.LogWarning(
ex,
"MTConnect driver {DriverInstanceId} could not read /current from {AgentUri}; the batch is reported Bad.",
_driverInstanceId, AgentDisplay(Volatile.Read(ref _options)));
}
/// <summary>Publishes <see cref="DriverState.Degraded"/> without downgrading a Faulted driver.</summary>
private void Degrade(Exception ex) => Degrade(ex.Message);
/// <summary>
/// Publishes <see cref="DriverState.Degraded"/> with an explanatory message, without
/// downgrading a Faulted driver.
/// </summary>
/// <remarks>
/// <b>Deliberately not <see cref="DriverState.Faulted"/>, even for the "an operator must fix
/// this" cases</b> such as an endpoint that cannot stream. In this codebase Faulted is not
/// merely a stronger word: <c>DriverHealthReport</c> maps any Faulted driver to a
/// <c>/readyz</c> <b>503</b>, taking the whole server node out of readiness. A driver whose
/// reads are perfectly healthy and whose subscriptions are dead is degraded, not unready —
/// de-readying the node over one misconfigured Agent would be a far larger blast radius than
/// the fault it reports. Faulted stays reserved for the case that earns it: an initialize
/// that failed, after which the driver serves nothing at all.
/// </remarks>
private void Degrade(string message)
{
var current = ReadHealth();
if (current.State != DriverState.Faulted)
{
WriteHealth(new DriverHealth(DriverState.Degraded, current.LastSuccessfulRead, message));
}
}
/// <summary>
/// Publishes <see cref="DriverState.Healthy"/> — <b>unless the other data path is currently
/// failing</b>.
/// </summary>
/// <remarks>
/// <para>
/// <b>Health precedence, stated deliberately.</b> This driver has two independent Agent
/// data paths: <c>/current</c> (the <see cref="IReadable"/> batch read) and
/// <c>/sample</c> (the subscription pump). They are different HTTP requests and fail
/// separately — a proxy that collapses <c>multipart/x-mixed-replace</c> breaks streaming
/// while reads stay perfect; an Agent that stalls composing a whole-device snapshot
/// breaks reads while the stream keeps flowing.
/// </para>
/// <para>
/// The naive last-writer-wins publish would let either path paper over the other's
/// failure, and "Healthy" while half the surface returns Bad is precisely the
/// failure-wearing-success shape this codebase exists to avoid. So each path owns a
/// flag, <b>clears only its own</b>, and Healthy requires both clear: the driver reports
/// Degraded while <i>anything</i> is failing and recovers as soon as the failing path
/// itself recovers. <see cref="DriverState.Faulted"/> — a config-level verdict — is
/// never downgraded by either.
/// </para>
/// </remarks>
/// <param name="at">When the Agent was last successfully heard from.</param>
private void PublishHealthy(DateTime at)
{
var current = ReadHealth();
if (_streamDegraded || _readDegraded || _writeDegraded)
{
if (current.State != DriverState.Faulted)
{
WriteHealth(new DriverHealth(DriverState.Degraded, at, current.LastError));
}
return;
}
WriteHealth(new DriverHealth(DriverState.Healthy, at, null));
}
/// <summary>
/// Disposes an agent client without letting a disposal fault escape. Teardown runs on the
/// failure path of <see cref="StartCoreAsync"/>, where a second exception would replace the
/// real one — but the swallowed exception is still <b>traced</b> rather than vanishing: a
/// client that cannot be released is how a socket-handle leak starts, and an empty catch is
/// the only place in this driver where a failure would leave no evidence at all. Debug
/// level, because it is diagnostic detail about an already-reported failure.
/// </summary>
private void SafeDispose(IMTConnectAgentClient? client)
{
if (client is null)
{
return;
}
try
{
client.Dispose();
}
catch (Exception ex)
{
_logger.LogDebug(
ex,
"MTConnect driver {DriverInstanceId} swallowed a fault while disposing its agent client during teardown; the client may not have released its connections.",
_driverInstanceId);
}
}
private static void RequirePositive(int value, string optionName)
{
if (value <= 0)
{
throw new ArgumentOutOfRangeException(
nameof(MTConnectDriverOptions),
value,
$"{optionName} must be positive; a non-positive value would either un-bound a deadline or ask the Agent for nothing.");
}
}
/// <summary>
/// The <see cref="TimeSpan"/> half of the 01/S-6 guard, for the connectivity probe's two
/// knobs. A non-positive interval is an unthrottled loop; a non-positive timeout cancels
/// every request before it can be answered.
/// </summary>
private static void RequirePositive(TimeSpan value, string optionName)
{
if (value <= TimeSpan.Zero)
{
throw new ArgumentOutOfRangeException(
nameof(MTConnectDriverOptions),
value,
$"{optionName} must be positive; a non-positive value would either spin the connectivity probe against the Agent or cancel every probe before it could be answered.");
}
}
/// <summary>Barrier-protected read of the cross-thread health snapshot.</summary>
private DriverHealth ReadHealth() => Volatile.Read(ref _health);
/// <summary>Barrier-protected publish of a new health snapshot.</summary>
private void WriteHealth(DriverHealth value) => Volatile.Write(ref _health, value);
private static MTConnectTagDefinition BuildTag(MTConnectTagDto dto, string driverInstanceId)
{
if (string.IsNullOrWhiteSpace(dto.FullName))
{
throw new InvalidOperationException(
$"MTConnect driver config for '{driverInstanceId}' has a tag with no FullName (the Agent DataItem id it binds).");
}
return new MTConnectTagDefinition(
dto.FullName.Trim(),
// Absent means "no declared type": String is the one coercion that cannot fail, and it
// carries the Agent's text through untouched.
dto.DriverDataType is null
? DriverDataType.String
: ParseEnum<DriverDataType>(dto.DriverDataType, dto.FullName, driverInstanceId, nameof(MTConnectTagDefinition.DriverDataType)),
dto.IsArray ?? false,
dto.ArrayDim ?? 0,
dto.MtCategory);
}
// ---- v3 tag binding: RawPath ↔ dataItemId, and the definitions the index coerces against ----
/// <summary>
/// Names the distinct, non-blank <see cref="RawTagEntry.DeviceName"/> values when there are two or
/// more, else <c>null</c>. The driver never routes by that field — one instance owns one
/// Agent client — so differing values mean the author expects a split the driver silently does
/// not perform. A single value is NOT reported: it is the <c>/raw</c> Device's own name, which
/// routinely differs from the Agent's device name and would make the warning noise.
/// </summary>
/// <param name="rawTags">The authored raw tags.</param>
/// <returns>A comma-joined, sorted list of the distinct names, or <c>null</c> when fewer than two.</returns>
internal static string? DescribeIgnoredDeviceNames(IReadOnlyList<RawTagEntry> rawTags)
{
var names = new SortedSet<string>(StringComparer.Ordinal);
foreach (var entry in rawTags)
{
if (entry is not null && !string.IsNullOrWhiteSpace(entry.DeviceName))
{
names.Add(entry.DeviceName.Trim());
}
}
return names.Count >= 2 ? string.Join(", ", names) : null;
}
/// <summary>
/// Derives the driver's tag tables from an options document — the <c>RawPath → dataItemId</c>
/// translation the data plane runs through, its reverse, and the definition set the
/// observation index coerces against.
/// </summary>
/// <remarks>
/// <para>
/// <b>A bad raw tag is skipped, never thrown.</b> One unreadable <c>TagConfig</c> blob
/// must not fail a whole driver — and therefore a whole deployment — over a tag the rest
/// of the plant does not depend on. The skipped tag's RawPath then resolves to nothing
/// and reports <c>BadNodeIdUnknown</c>, which is the documented
/// <see cref="EquipmentTagRefResolver{TDef}"/> posture ("a miss is a miss") and is
/// visible to an operator, unlike a silent default.
/// </para>
/// <para>
/// <b>Coercion type precedence</b> (see <see cref="MTConnectDriverOptions.RawTags"/>):
/// the blob's own <c>driverDataType</c>, else a <see cref="MTConnectDriverOptions.Tags"/>
/// entry naming the same DataItem id, else the Agent's <c>/probe</c> declaration for that
/// data item, else <see cref="DriverDataType.String"/>.
/// </para>
/// <para>
/// <b>A browse-committed tag never reaches the third step, and no longer needs to.</b>
/// The MTConnect branch of <c>RawBrowseCommitMapper.BuildTagConfig</c> emits a whole
/// <c>MTConnectTagConfigModel</c>, whose <c>ToJson</c> unconditionally writes
/// <c>"dataType"</c> — always a non-blank authored type, so step 1 always wins. Since
/// the value it writes is the browsed leaf's own inferred
/// <see cref="DriverDataType"/>, i.e. what this step would have computed, so it agrees
/// with the raw-tag node's OPC UA type (taken from the same inference by
/// <c>RawBrowseCommitMapper.MapLeaf</c>). Historically it wrote the model's
/// <see cref="DriverDataType.String"/> default instead, and every browse-committed tag
/// coerced to text inside a <c>Double</c>/<c>Int64</c> node.
/// The step remains the rescue for a blob authored by hand, by CSV or by API that states
/// no type at all.
/// </para>
/// </remarks>
/// <param name="options">The options document to derive from.</param>
/// <param name="probeModel">The Agent's device model when one has been fetched; else null.</param>
private TagBinding BuildBinding(MTConnectDriverOptions options, MTConnectProbeModel? probeModel)
{
// The dataItemId-keyed authoring surface first (the driver CLI, and every pre-v3 caller).
// Last-wins on a duplicate id, matching MTConnectObservationIndex's own documented rule.
var definitions = new Dictionary<string, MTConnectTagDefinition>(StringComparer.Ordinal);
foreach (var tag in options.Tags)
{
if (tag is null || string.IsNullOrWhiteSpace(tag.FullName)) continue;
definitions[tag.FullName] = tag;
}
_skippedTags.BeginRebuild(options.RawTags.Count);
if (options.RawTags.Count == 0)
{
MTConnectTagDefinition[] only = [.. definitions.Values];
_skippedTags.Commit();
return new TagBinding(
FrozenDictionary<string, string>.Empty,
FrozenDictionary<string, IReadOnlyList<string>>.Empty,
only,
BuildWriteTargets(only, probeModel, options.DeviceName));
}
if (DescribeIgnoredDeviceNames(options.RawTags) is { } ignoredDevices)
{
_logger.LogWarning(
"MTConnect driver {DriverInstanceId}: raw tags carry differing DeviceName values ({DeviceNames}), but RawTagEntry.DeviceName is not used for routing — every tag is served from the one Agent connection scoped by the driver-level DeviceName ({ConfiguredDevice}). Author one MTConnect driver instance per Agent device if they must be kept apart.",
_driverInstanceId, ignoredDevices, options.DeviceName ?? "<whole Agent>");
}
var declared = IndexProbeDataItems(probeModel);
var idByRawPath = new Dictionary<string, string>(options.RawTags.Count, StringComparer.Ordinal);
var rawPathsById = new Dictionary<string, List<string>>(StringComparer.Ordinal);
foreach (var entry in options.RawTags)
{
if (entry is null || string.IsNullOrWhiteSpace(entry.RawPath)) continue;
if (!MTConnectTagDefinitionFactory.TryFromTagConfig(entry.TagConfig, out var dataItemId, out var authored))
{
_skippedTags.Skip(
entry.RawPath,
"names no readable DataItem id (expected 'fullName', 'dataItemId' or 'address') "
+ "or declares an unrecognised driverDataType");
_logger.LogWarning(
"MTConnect driver {DriverInstanceId} could not map the raw tag {RawPath} to an Agent DataItem; it names no readable id (expected 'fullName', 'dataItemId' or 'address') or declares an unrecognised driverDataType. The tag is skipped and reports BadNodeIdUnknown; every other tag is unaffected.",
_driverInstanceId, entry.RawPath);
continue;
}
idByRawPath[entry.RawPath] = dataItemId;
if (!rawPathsById.TryGetValue(dataItemId, out var paths))
{
rawPathsById[dataItemId] = paths = [];
}
if (!paths.Contains(entry.RawPath, StringComparer.Ordinal))
{
// One DataItem may legitimately back several authored raw tags; a duplicate RawPath
// (the same tag listed twice) must still publish once.
paths.Add(entry.RawPath);
}
if (authored is not null)
{
// The blob declared a type: it is the deploy-time authority and supersedes an older
// Tags entry for the same DataItem.
definitions[dataItemId] = authored;
}
else if (!definitions.ContainsKey(dataItemId))
{
definitions[dataItemId] = InferDefinition(dataItemId, declared);
}
}
MTConnectTagDefinition[] indexTags = [.. definitions.Values];
_skippedTags.Commit();
return new TagBinding(
idByRawPath.ToFrozenDictionary(StringComparer.Ordinal),
rawPathsById.ToFrozenDictionary(
pair => pair.Key,
pair => (IReadOnlyList<string>)pair.Value,
StringComparer.Ordinal),
indexTags,
BuildWriteTargets(indexTags, probeModel, options.DeviceName));
}
/// <summary>
/// Derives the write-addressing table: for every DataItem this driver serves, the device the
/// Agent's <c>AllowPut</c> URL must name, and whether the item is CONDITION-category.
/// </summary>
/// <param name="definitions">Every DataItem the binding serves (the union of both authoring surfaces).</param>
/// <param name="probeModel">The Agent's device model when one has been fetched; else null.</param>
/// <param name="configuredDeviceName">
/// <see cref="MTConnectDriverOptions.DeviceName"/> — the fallback for a DataItem the
/// <c>/probe</c> model does not declare.
/// </param>
/// <remarks>
/// <para>
/// <b>The device name is derived, not authored.</b> The Agent's PUT surface is addressed
/// per device even on a single-device Agent, and the probe model already says which device
/// owns which DataItem — so a multi-device Agent is addressed correctly with no extra
/// config. Config is the fallback, for the dataItemId-keyed authoring surface (the driver
/// CLI) and for a DataItem the Agent has since stopped declaring.
/// </para>
/// <para>
/// <b>A null <see cref="WriteTarget.DeviceName"/> is a refusal, not a default.</b> Neither
/// the model nor config names a device, so there is no URL to POST to; guessing one yields
/// a 404 an operator reads as "the Agent is broken". <see cref="WriteAsync"/> reports it as
/// <c>BadNodeIdUnknown</c>.
/// </para>
/// <para>
/// <b>The category comes from the probe first, the authored blob second.</b> The Agent is
/// the authority on what a DataItem is; a blob's <c>mtCategory</c> is optional metadata
/// that may be absent (browse-commit writes only the address) or stale.
/// </para>
/// </remarks>
private static FrozenDictionary<string, WriteTarget> BuildWriteTargets(
IReadOnlyList<MTConnectTagDefinition> definitions,
MTConnectProbeModel? probeModel,
string? configuredDeviceName)
{
if (definitions.Count == 0)
{
return FrozenDictionary<string, WriteTarget>.Empty;
}
var owners = IndexProbeOwners(probeModel);
var fallback = string.IsNullOrWhiteSpace(configuredDeviceName) ? null : configuredDeviceName.Trim();
var targets = new Dictionary<string, WriteTarget>(definitions.Count, StringComparer.Ordinal);
foreach (var definition in definitions)
{
if (definition is null || string.IsNullOrWhiteSpace(definition.FullName)) continue;
var owner = owners.TryGetValue(definition.FullName, out var declared) ? declared : null;
targets[definition.FullName] = new WriteTarget(
owner?.DeviceName ?? fallback,
IsCondition(owner?.Category ?? definition.MtCategory));
}
return targets.ToFrozenDictionary(StringComparer.Ordinal);
}
/// <summary>
/// Flattens a device model into a <c>dataItemId → owning device</c> lookup; null ⇒ empty.
/// Separate from <see cref="IndexProbeDataItems"/> because this one must remember which device
/// the walk was inside, which a flat DataItem lookup has thrown away.
/// </summary>
/// <param name="probeModel">The Agent's device model, or null.</param>
private static FrozenDictionary<string, DataItemOwner> IndexProbeOwners(MTConnectProbeModel? probeModel)
{
if (probeModel is null) return FrozenDictionary<string, DataItemOwner>.Empty;
var map = new Dictionary<string, DataItemOwner>(StringComparer.Ordinal);
foreach (var device in probeModel.Devices)
{
if (device is null) continue;
// The same name/id fallback the browse tree uses: an Agent addresses a device by its
// name, and by its id when it declares no name.
var deviceName = FolderName(device.Name, device.Id);
var owner = new DataItemOwner(
string.IsNullOrWhiteSpace(deviceName) ? null : deviceName, Category: null);
foreach (var dataItem in device.AllDataItems())
{
if (dataItem is null || string.IsNullOrWhiteSpace(dataItem.Id)) continue;
map[dataItem.Id] = owner with { Category = dataItem.Category };
}
}
return map.ToFrozenDictionary(StringComparer.Ordinal);
}
/// <summary>
/// The definition for a raw tag whose blob declared no type: the Agent's own <c>/probe</c>
/// declaration when the model is in hand, else <see cref="DriverDataType.String"/> — the one
/// coercion that cannot fail, carrying the Agent's text through untouched.
/// </summary>
private static MTConnectTagDefinition InferDefinition(
string dataItemId, FrozenDictionary<string, MTConnectDataItem> declared)
{
if (!declared.TryGetValue(dataItemId, out var dataItem))
{
return new MTConnectTagDefinition(dataItemId, DriverDataType.String);
}
var inferred = MTConnectDataTypeInference.Infer(
dataItem.Category, dataItem.Type, dataItem.Units, dataItem.Representation, dataItem.SampleCount);
return new MTConnectTagDefinition(
dataItemId,
inferred.DataType,
inferred.IsArray,
(int)(inferred.ArrayDim ?? 0),
dataItem.Category);
}
/// <summary>Flattens a device model into a <c>dataItemId → DataItem</c> lookup; null ⇒ empty.</summary>
private static FrozenDictionary<string, MTConnectDataItem> IndexProbeDataItems(MTConnectProbeModel? probeModel)
{
if (probeModel is null) return FrozenDictionary<string, MTConnectDataItem>.Empty;
var map = new Dictionary<string, MTConnectDataItem>(StringComparer.Ordinal);
foreach (var device in probeModel.Devices)
{
if (device is null) continue;
foreach (var dataItem in device.AllDataItems())
{
if (dataItem is null || string.IsNullOrWhiteSpace(dataItem.Id)) continue;
map[dataItem.Id] = dataItem;
}
}
return map.ToFrozenDictionary(StringComparer.Ordinal);
}
/// <summary>
/// Parses an enum authored as a <b>name</b>. Config surfaces serialize enums as names
/// project-wide; a numerically-serialized enum is the known trap that faults a driver at
/// deploy, so this reports the offending field rather than silently taking member 0.
/// </summary>
private static T ParseEnum<T>(string raw, string tagName, string driverInstanceId, string field)
where T : struct, Enum =>
Enum.TryParse<T>(raw, ignoreCase: true, out var parsed) && Enum.IsDefined(parsed)
? parsed
: throw new InvalidOperationException(string.Create(
CultureInfo.InvariantCulture,
$"MTConnect driver config for '{driverInstanceId}' tag '{tagName}' has an unrecognised {field} '{raw}'. Expected one of: {string.Join(", ", Enum.GetNames<T>())}."));
/// <summary>
/// The live agent client and the <c>/probe</c> cache derived from it, as one immutable
/// unit. Immutable so that publishing a change is a single reference swap — see the
/// <see cref="_session"/> remarks for why these values must move together.
/// </summary>
/// <param name="Client">The agent client this session owns and will dispose on teardown.</param>
/// <param name="ProbeModel">
/// The cached device model, or <c>null</c> once <see cref="InvalidateProbeModel"/> has
/// dropped it. Never null at the moment a session is first published.
/// </param>
/// <param name="InstanceId">
/// The Agent's <c>instanceId</c> as last observed. Updated in place (by
/// <see cref="PublishAgentCursor"/>) when the pump sees the Agent restart, so the
/// session always names the Agent process the client is actually talking to.
/// </param>
/// <param name="NextSequence">
/// The sequence the <c>/sample</c> pump should open at — the priming <c>/current</c>'s
/// <c>nextSequence</c>. Lives here so that starting the pump reads the client and its cursor
/// as one consistent pair; a torn read would open the stream at another session's sequence.
/// </param>
private sealed record AgentSession(
IMTConnectAgentClient Client,
MTConnectProbeModel? ProbeModel,
long InstanceId,
long NextSequence);
/// <summary>
/// The tag tables one options document derives, as one immutable unit — so installing new
/// options is a single reference swap and no reader can pair one document's RawPaths with
/// another's definitions.
/// </summary>
/// <param name="DataItemIdByRawPath">
/// The v3 wire translation: the RawPath the server subscribes / reads / routes by, mapped to
/// the Agent DataItem <c>id</c> the observation index is keyed by.
/// </param>
/// <param name="RawPathsByDataItemId">
/// The reverse, one-to-many: every RawPath a DataItem must be published under. A DataItem may
/// legitimately back more than one authored raw tag, and each of those tags is a separate
/// OPC UA node that must receive the value.
/// </param>
/// <param name="IndexTags">
/// The definition set <see cref="MTConnectObservationIndex"/> coerces against — the union of
/// the authored <see cref="MTConnectDriverOptions.Tags"/> and the definitions derived from
/// <see cref="MTConnectDriverOptions.RawTags"/>. This is how a raw tag's declared (or
/// probe-inferred) <c>DriverDataType</c> reaches the coercion.
/// </param>
/// <param name="WriteTargetsByDataItemId">
/// How each DataItem is addressed on the Agent's <c>AllowPut</c> surface, and whether it is
/// CONDITION-category — see <see cref="BuildWriteTargets"/>. Keyed by DataItem id, so it
/// doubles as the write path's "is this a reference I serve at all?" test: a miss is
/// <c>BadNodeIdUnknown</c>, exactly as an index miss is on the read path.
/// </param>
private sealed record TagBinding(
FrozenDictionary<string, string> DataItemIdByRawPath,
FrozenDictionary<string, IReadOnlyList<string>> RawPathsByDataItemId,
IReadOnlyList<MTConnectTagDefinition> IndexTags,
FrozenDictionary<string, WriteTarget> WriteTargetsByDataItemId)
{
/// <summary><see cref="IndexTags"/> frozen once per binding, for the per-read index (R06:T-4).</summary>
public FrozenDictionary<string, MTConnectTagDefinition> IndexTagMap { get; } =
MTConnectObservationIndex.FreezeTags(IndexTags);
/// <summary>
/// Resolves a write reference to the DataItem id and the Agent address it writes through.
/// </summary>
/// <param name="reference">The caller's reference (a RawPath under v3, or a DataItem id).</param>
/// <param name="dataItemId">The resolved DataItem id; empty when this returns false.</param>
/// <param name="target">Where and whether it can be written; <c>default</c> when false.</param>
/// <returns><see langword="false"/> when no DataItem this driver serves answers to that reference.</returns>
/// <remarks>
/// Stricter than <see cref="ToDataItemId"/>, deliberately. A read may fall through to an
/// index miss and report itself; a write that fell through would POST an arbitrary
/// operator string at the Agent as if it were a DataItem id — the Agent's answer to which
/// is a refusal that reads like a comms fault. Unknown is answered here, before the wire.
/// </remarks>
public bool TryResolveWriteTarget(string? reference, out string dataItemId, out WriteTarget target)
{
dataItemId = ToDataItemId(reference);
if (dataItemId.Length != 0 && WriteTargetsByDataItemId.TryGetValue(dataItemId, out target))
{
return true;
}
target = default;
return false;
}
/// <summary>
/// Translates a subscriber/read reference to the DataItem id the index is keyed by.
/// </summary>
/// <remarks>
/// An unmapped reference is returned <b>unchanged</b> rather than rejected, which serves
/// two callers at once: the dataItemId-keyed authoring surface (the driver CLI, and a
/// driver configured with <c>Tags</c> and no <c>RawTags</c>) resolves normally, and a
/// RawPath that no authored raw tag claims falls through to an index miss —
/// <c>BadNodeIdUnknown</c>, never an exception. There is no third behaviour to add here:
/// the index already distinguishes "authored but not yet reported" from "unknown".
/// </remarks>
/// <param name="reference">The caller's reference (a RawPath under v3).</param>
/// <returns>The DataItem id to look up.</returns>
public string ToDataItemId(string? reference) =>
reference is not null && DataItemIdByRawPath.TryGetValue(reference, out var dataItemId)
? dataItemId
: reference ?? string.Empty;
/// <summary>
/// Appends every reference <paramref name="dataItemId"/> must be published under: each
/// RawPath bound to it, plus the id itself for the dataItemId-keyed authoring surface.
/// </summary>
/// <param name="dataItemId">The DataItem the Agent just reported.</param>
/// <param name="into">The reference list being accumulated for the fan-out.</param>
public void AppendReferences(string dataItemId, List<string> into)
{
if (RawPathsByDataItemId.TryGetValue(dataItemId, out var rawPaths))
{
into.AddRange(rawPaths);
// Only when a RawPath is spelled identically to the id — otherwise a subscriber
// holding that one reference would be raised twice for one observation.
if (rawPaths.Contains(dataItemId, StringComparer.Ordinal)) return;
}
into.Add(dataItemId);
}
}
/// <summary>
/// Everything <see cref="WriteAsync"/> needs to decide one slot before it touches the network.
/// </summary>
/// <param name="DeviceName">
/// The device path segment of the Agent PUT URL, or <c>null</c> when neither the probe model
/// nor config names one — which is a refusal, not a default (see <see cref="BuildWriteTargets"/>).
/// </param>
/// <param name="IsCondition">
/// Whether the DataItem is CONDITION-category. Excluded from v1 writes: a condition PUT carries
/// alarm semantics and a different <c>level|text</c> wire format.
/// </param>
private readonly record struct WriteTarget(string? DeviceName, bool IsCondition);
/// <summary>The device a DataItem was declared under, and the category it was declared with.</summary>
/// <param name="DeviceName">The owning device's name (or id); null when it declares neither.</param>
/// <param name="Category">The DataItem's <c>category</c> attribute as the Agent spelled it.</param>
private sealed record DataItemOwner(string? DeviceName, string? Category);
/// <summary>
/// One live subscription: its handle and the references it wants. Both are effectively
/// immutable once registered, so the pump reads <see cref="References"/> without a lock —
/// it takes <see cref="_subscriptionLock"/> only to snapshot the registry itself.
/// </summary>
/// <param name="Handle">The identity handed to the caller and stamped on every event.</param>
/// <param name="References">The de-duplicated DataItem ids this subscription publishes.</param>
private sealed record SubscriptionState(MTConnectSampleHandle Handle, HashSet<string> References);
/// <summary>How one <c>/sample</c> connection ended, and therefore what the pump does next.</summary>
private enum StreamVerdict
{
/// <summary>Cancelled by teardown — stop, publish nothing, report nothing.</summary>
Cancelled = 0,
/// <summary>A transient failure: degrade and reconnect under the backoff.</summary>
Transient = 1,
/// <summary>Re-baselined successfully: reopen immediately at the new cursor, no backoff.</summary>
Reopen = 2,
/// <summary>The endpoint cannot stream at all: latch, report loudly, and stop retrying.</summary>
Unsupported = 3
}
/// <param name="Verdict">How the connection ended.</param>
/// <param name="Cursor">The sequence the next connection must open at.</param>
/// <param name="InstanceId">The Agent instance the cursor belongs to.</param>
/// <param name="DeliveredAny">
/// Whether this connection yielded at least one chunk before it ended. Distinguishes a
/// working endpoint having a bad moment from one that has never worked, which is what keeps
/// the reconnect backoff from ratcheting monotonically over a process lifetime.
/// </param>
/// <param name="Error">The failure, when there was one.</param>
/// <param name="RaiseCountTo">
/// Set on a <see cref="StreamVerdict.Reopen"/> the adaptive count asked for: the stream was persistently behind
/// the Agent, and the next one opens with this <c>count</c> (R06:T-5).
/// </param>
private sealed record StreamOutcome(
StreamVerdict Verdict, long Cursor, long InstanceId, bool DeliveredAny, Exception? Error, int? RaiseCountTo = null);
// ---- config DTOs ----
/// <summary>
/// Nullable-init mirror of the driver's <c>DriverConfig</c> JSON. Every property is nullable
/// so an omitted key means "use the default" rather than "reset to zero".
/// </summary>
private sealed class MTConnectDriverConfigDto
{
/// <summary>The Agent base URI.</summary>
public string? AgentUri { get; init; }
/// <summary>The Agent device name to bind.</summary>
public string? DeviceName { get; init; }
/// <summary>Per-request timeout in milliseconds.</summary>
public int? RequestTimeoutMs { get; init; }
/// <summary>The /sample interval in milliseconds.</summary>
public int? SampleIntervalMs { get; init; }
/// <summary>The /sample count per request.</summary>
public int? SampleCount { get; init; }
/// <summary>The /sample heartbeat in milliseconds.</summary>
public int? HeartbeatMs { get; init; }
/// <summary>The authored tags.</summary>
public List<MTConnectTagDto>? Tags { get; init; }
/// <summary>
/// The v3 authored raw tags the deploy artifact injects (RawPath + driver <c>TagConfig</c>
/// blob). Bound verbatim into <see cref="MTConnectDriverOptions.RawTags"/>; this is the
/// collection a deployed instance's data plane is keyed by.
/// </summary>
public List<RawTagEntry>? RawTags { get; init; }
/// <summary>Connectivity-probe settings.</summary>
public MTConnectProbeDto? Probe { get; init; }
/// <summary>Reconnect-backoff settings.</summary>
public MTConnectReconnectDto? Reconnect { get; init; }
/// <summary>Write-back settings.</summary>
public MTConnectWriteDto? Write { get; init; }
}
/// <summary>One authored tag. <c>DriverDataType</c> stays a string — see <see cref="ParseEnum{T}"/>.</summary>
private sealed class MTConnectTagDto
{
/// <summary>The tag's full name.</summary>
public string? FullName { get; init; }
/// <summary>The coercion type name.</summary>
public string? DriverDataType { get; init; }
/// <summary>Whether the tag is an array.</summary>
public bool? IsArray { get; init; }
/// <summary>The array length when <c>IsArray</c> is set.</summary>
public int? ArrayDim { get; init; }
/// <summary>The MTConnect DataItem category.</summary>
public string? MtCategory { get; init; }
// mtType / mtSubType / units were bound here and read by nothing (review 2026-10-21 T-3); a document that
// still carries them parses — System.Text.Json skips unmapped members.
}
/// <summary>Background connectivity-probe knobs (consumed by the probe loop).</summary>
private sealed class MTConnectProbeDto
{
/// <summary>Whether the probe loop runs.</summary>
public bool? Enabled { get; init; }
/// <summary>Probe interval in milliseconds.</summary>
public int? IntervalMs { get; init; }
/// <summary>Probe timeout in milliseconds.</summary>
public int? TimeoutMs { get; init; }
}
/// <summary>Reconnect-backoff knobs (consumed by the /sample pump).</summary>
private sealed class MTConnectReconnectDto
{
/// <summary>Minimum reconnect backoff in milliseconds.</summary>
public int? MinBackoffMs { get; init; }
/// <summary>Maximum reconnect backoff in milliseconds.</summary>
public int? MaxBackoffMs { get; init; }
/// <summary>Backoff growth multiplier.</summary>
public double? BackoffMultiplier { get; init; }
}
/// <summary>Write-back knobs — the driver-side half of the two-layer opt-in on observation injection.</summary>
private sealed class MTConnectWriteDto
{
/// <summary>Whether write-back to the Agent is enabled.</summary>
public bool? Enabled { get; init; }
}
}