dohertj2/lmxopcua
1
0
Fork 0
Code Issues 21 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(drivers): tag-set drift detection, gated on SupportsOnlineDiscovery

Browse Source 
The second thing #516/§8.2 deliberately did not build. The original objection
stands — Modbus, S7, MQTT, AbLegacy and Sql echo authored config from
DiscoverAsync, so diffing discovered against authored is a tautology for them
— but it is now ENCODED rather than used as a reason not to build.

The discriminator already existed: ITagDiscovery.SupportsOnlineDiscovery is
documented as "enumerates the tag set from the live backend rather than
replaying pre-declared/authored tags", and it separates the fleet cleanly —
FOCAS, TwinCAT, MTConnect and AbCip true; the five config-echo drivers
explicitly false. The check runs only for a driver that is
SupportsOnlineDiscovery AND reports the new AuthoredDiscoveryRefs. That is
nullable rather than empty-by-default on purpose: an empty collection
legitimately means "nothing authored, everything discovered is new", so
conflating the two would report total drift for a driver that never opted in.

Where the value is: AbCip and FOCAS browse a live backend but implement no
IRediscoverable, so before this they had no change signal at all. A periodic
compare is the only way to notice a PLC re-download.

A finding that changed the design: FOCAS, TwinCAT and AbCip all re-emit their
authored tags into the same DiscoverAsync stream as device-derived ones, so
the browse picker can show an operator their existing tags. Discovered is
therefore always a superset of authored, and a VANISHED tag can never appear
missing — one whole direction is structurally undetectable for three of the
four. Shipping a Vanished list that is permanently empty for them would have
been the half-inert seam this whole exercise removes. So the driver declares
DiscoveryStreamIncludesAuthoredTags, the detector does not compute the
undetectable half, and the operator message says "missing-tag detection
unavailable for this driver" rather than letting the absence of a missing-tag
clause read as reassurance. MTConnect is the only driver where both directions
work — its stream is purely probe-model-derived.

Behaviour: a failed browse is not drift (an unreachable device would otherwise
report every authored tag as vanished); a truncated capture is skipped for the
same reason; an unchanged drift is reported once rather than re-stamping its
timestamp; drift that resolves clears the prompt. Interval defaults to 5
minutes — it browses a real device, and a tag set changing is an engineering
event, not a runtime one. It reuses the Stage-2 surfacing, raising the same
/hosts re-browse prompt, and never rebuilds the served address space.

Comparison logic is a pure, separately-tested type rather than actor-inline.
14 new tests. The gate was verified load-bearing by deleting the
SupportsOnlineDiscovery half: the tautology-driver test goes red, and it
asserts discovery was never invoked rather than merely "no prompt raised",
which would have passed for the wrong reason if the timer never fired.

Full suite green except the same three pre-existing fixture-gated integration
suites, identical counts.
This commit is contained in:
Joseph Doherty
2026-07-28 00:48:39 -04:00
parent 5184a2e107
commit e77c8a3569
11 changed files with 770 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverInstanceActor.cs
+157 -3
View File
@@ -1,5 +1,6 @@
using Akka.Actor;
using Akka.Event;
using ZB.MOM.WW.OtOpcUa.Commons.Browsing;
using ZB.MOM.WW.OtOpcUa.Commons.Observability;
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
using ZB.MOM.WW.OtOpcUa.Commons.Types;
@@ -32,6 +33,11 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
{
public static readonly TimeSpan DefaultReconnectInterval = TimeSpan.FromSeconds(10);
/// <summary>Default period of the tag-set drift check. Deliberately slow: it browses a real device, and
/// a tag set changing is an engineering event (a PLC re-download, a CNC option install), not a runtime
/// one. Five minutes bounds the wire cost while still surfacing drift the same shift it happens.</summary>
public static readonly TimeSpan DefaultDriftCheckInterval = TimeSpan.FromMinutes(5);
public sealed record InitializeRequested(string DriverConfigJson);
public sealed record InitializeSucceeded(int Generation);
public sealed record InitializeFailed(string Reason, int Generation);
@@ -121,6 +127,15 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
/// between connects), and dropping it in one state would lose the signal silently.</summary>
private sealed record RediscoveryRaised(RediscoveryEventArgs Args);
/// <summary>Self-sent on a slow timer to re-browse a genuinely-browsable driver and compare what the
/// remote offers against what an operator authored. Only ever scheduled for a driver that opts in — see
/// <see cref="QualifiesForDriftCheck"/>.</summary>
private sealed record DriftCheckTick
{
public static readonly DriftCheckTick Instance = new();
private DriftCheckTick() { }
}
public sealed class RetryConnect
{
public static readonly RetryConnect Instance = new();
@@ -181,6 +196,14 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
private DateTime? _rediscoveryNeededUtc;
private string? _rediscoveryReason;
/// <summary>Period of the tag-set drift check; tests inject a tiny value.</summary>
private readonly TimeSpan _driftCheckInterval;
/// <summary>Signature of the last drift REPORTED, so an unchanged drift is not re-announced on every
/// check. Without this the operator's prompt would re-stamp its timestamp every interval forever,
/// which reads as "it just happened again" rather than "it is still true".</summary>
private string? _lastDriftSignature;
/// <summary>The references the host wants kept subscribed (set by <see cref="SetDesiredSubscriptions"/>).
/// Re-applied on every entry into <c>Connected</c> so values resume after a reconnect or redeploy.</summary>
private IReadOnlyList<string> _desiredRefs = Array.Empty<string>();
@@ -214,6 +237,8 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
/// defaults to an empty string when not provided (e.g. in unit tests).</param>
/// <param name="invoker">Optional Phase 6.1 resilience invoker wrapping this driver's capability
/// calls; defaults to <see cref="NullDriverCapabilityInvoker"/> (pass-through) when not supplied.</param>
/// <param name="driftCheckInterval">Optional period of the tag-set drift check; defaults to
/// <see cref="DefaultDriftCheckInterval"/>. Tests inject a tiny value so the check runs without a wait.</param>
/// <param name="healthPollInterval">Optional period of the health-poll heartbeat, which also drives the
/// subscription reconcile (<see cref="ReconcileSubscription"/>); defaults to
/// <see cref="HealthPollInterval"/>. Exists so a test can drive the reconcile without waiting 30 s.</param>
@@ -225,7 +250,8 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
IDriverHealthPublisher? healthPublisher = null,
string? clusterId = null,
IDriverCapabilityInvoker? invoker = null,
TimeSpan? healthPollInterval = null) =>
TimeSpan? healthPollInterval = null,
TimeSpan? driftCheckInterval = null) =>
Akka.Actor.Props.Create(() => new DriverInstanceActor(
driver,
reconnectInterval ?? DefaultReconnectInterval,
@@ -233,7 +259,8 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
healthPublisher ?? NullDriverHealthPublisher.Instance,
clusterId ?? string.Empty,
invoker,
healthPollInterval));
healthPollInterval,
driftCheckInterval));
/// <summary>
/// Returns true when the driver should boot in DEV-STUB mode based on host platform and
@@ -264,6 +291,8 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
/// <param name="clusterId">Cluster identifier forwarded in health snapshots.</param>
/// <param name="invoker">Phase 6.1 resilience invoker wrapping this driver's capability calls;
/// defaults to <see cref="NullDriverCapabilityInvoker"/> (pass-through) when null.</param>
/// <param name="driftCheckInterval">Period of the tag-set drift check; defaults to
/// <see cref="DefaultDriftCheckInterval"/>.</param>
/// <param name="healthPollInterval">Period of the health-poll heartbeat, which also drives the
/// subscription reconcile; defaults to <see cref="HealthPollInterval"/> when null.</param>
public DriverInstanceActor(
@@ -273,13 +302,15 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
IDriverHealthPublisher? healthPublisher = null,
string? clusterId = null,
IDriverCapabilityInvoker? invoker = null,
TimeSpan? healthPollInterval = null)
TimeSpan? healthPollInterval = null,
TimeSpan? driftCheckInterval = null)
{
_driver = driver;
_invoker = invoker ?? NullDriverCapabilityInvoker.Instance;
_driverInstanceId = driver.DriverInstanceId;
_clusterId = clusterId ?? string.Empty;
_healthPublisher = healthPublisher ?? NullDriverHealthPublisher.Instance;
_driftCheckInterval = driftCheckInterval ?? DefaultDriftCheckInterval;
_reconnectInterval = reconnectInterval;
_healthPollInterval = healthPollInterval ?? HealthPollInterval;
OtOpcUaTelemetry.DriverInstanceLifecycle.Add(1,
@@ -350,6 +381,7 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
ResubscribeDesired();
AttachAlarmSource();
SubscribeDesiredAlarms();
StartDriftCheck();
});
Receive<InitializeFailed>(msg =>
{
@@ -395,6 +427,7 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
_driverInstanceId, msg.Reason);
DetachSubscription();
RecordFault();
StopDriftCheck();
Become(Reconnecting);
PublishHealthSnapshot();
});
@@ -402,6 +435,7 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
{
_log.Info("DriverInstance {Id}: ForceReconnect requested by admin; re-entering Reconnecting", _driverInstanceId);
DetachSubscription();
StopDriftCheck();
Become(Reconnecting);
PublishHealthSnapshot();
});
@@ -424,6 +458,7 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
SubscribeDesiredAlarms();
});
ReceiveAsync<SubscribeAlarms>(HandleSubscribeAlarmsAsync);
ReceiveAsync<DriftCheckTick>(HandleDriftCheckAsync);
Receive<DataChangeForward>(OnDataChangeForward);
// Native alarm transition marshaled onto the actor thread from the driver's OnAlarmEvent;
// project it to the parent the same way DataChangeForward projects AttributeValuePublished.
@@ -518,6 +553,7 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
ResubscribeDesired();
AttachAlarmSource();
SubscribeDesiredAlarms();
StartDriftCheck();
});
// A failure here is a no-op regardless of generation — the retry timer keeps trying the
// current config; only a (generation-matched) InitializeSucceeded transitions state.
@@ -762,6 +798,10 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
}
}
/// <summary>Stops the drift check. Called on every Connected exit — browsing a disconnected driver would
/// fail every pass, and a failed browse is deliberately NOT treated as drift.</summary>
private void StopDriftCheck() => Timers.Cancel("drift-check");
/// <summary>Tear down the data-change + native-alarm event handlers + null the handle. Called from the
/// Unsubscribe path, on PostStop, and on Connected → Reconnecting transitions so a stale handler doesn't
/// push data-change / alarm events to an actor that has lost its driver connection.</summary>
@@ -809,6 +849,119 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
_rediscoveryHandler = null;
}
/// <summary>
/// True when this driver is worth drift-checking: it must genuinely BROWSE its remote
/// (<see cref="ITagDiscovery.SupportsOnlineDiscovery"/>) and must report the refs its authored tags
/// bind (<see cref="ITagDiscovery.AuthoredDiscoveryRefs"/>).
/// <para>The first condition is what makes the check meaningful rather than reassuring. Most drivers
/// — Modbus, S7, AbLegacy, Sql, Mqtt — stream their AUTHORED tags back out of <c>DiscoverAsync</c>
/// rather than enumerating the device, so diffing the two sets for them is a tautology that would
/// report "no drift" forever. A check that cannot fail is worse than no check: it looks like
/// coverage.</para>
/// </summary>
private bool QualifiesForDriftCheck() =>
_driver is ITagDiscovery { SupportsOnlineDiscovery: true, AuthoredDiscoveryRefs: not null };
/// <summary>Starts the slow drift check on a Connected entry, for qualifying drivers only.</summary>
private void StartDriftCheck()
{
if (!QualifiesForDriftCheck()) return;
// Periodic, not fire-on-connect: a driver whose discovered shape fills in asynchronously after
// connect (the FOCAS FixedTree) would otherwise be compared against a half-populated browse and
// report phantom drift on every reconnect.
Timers.StartPeriodicTimer("drift-check", DriftCheckTick.Instance, _driftCheckInterval);
}
/// <summary>
/// Re-browses the remote and compares it against the driver's authored refs. A CHANGED drift raises
/// the same operator prompt an <see cref="IRediscoverable"/> raise does — this is the signal for the
/// browsable drivers that have no native change notification (AbCip, FOCAS), where a periodic
/// compare is the only way to notice a PLC program re-download.
/// <para>Advisory, like every rediscovery signal: the served address space is not touched. Raw tags
/// are authored through the <c>/raw</c> browse-commit flow, so an operator re-browses and commits.</para>
/// </summary>
private async Task HandleDriftCheckAsync(DriftCheckTick _)
{
if (_driver is not ITagDiscovery discovery) return;
var authored = discovery.AuthoredDiscoveryRefs;
if (authored is null) return;
CapturedTree tree;
try
{
var builder = new CapturingAddressSpaceBuilder();
// Bounded: ReceiveAsync suspends the mailbox for the whole handler, so an unbounded browse would
// block writes, reconnects and health polls behind it.
using var cts = new CancellationTokenSource(DriftBrowseTimeout);
await _invoker.ExecuteAsync(
DriverCapability.Discover,
_driverInstanceId,
async ct => await discovery.DiscoverAsync(builder, ct),
cts.Token);
tree = builder.Build();
}
catch (Exception ex)
{
// A failed browse is not drift — the device may simply be unreachable, which the health surface
// already reports. Reporting drift here would turn every comms blip into "all your tags vanished".
_log.Debug(ex, "DriverInstance {Id}: drift check browse failed; skipping this pass", _driverInstanceId);
return;
}
if (tree.Truncated)
{
// A truncated capture is a PARTIAL view, so every un-captured authored ref would look vanished.
_log.Warning(
"DriverInstance {Id}: drift check skipped — the browse hit the capture cap and is partial",
_driverInstanceId);
return;
}
var discovered = new List<string>();
CollectLeafRefs(tree.Root, discovered);
var drift = TagSetDriftDetector.Compare(
discovered, authored, detectVanished: !discovery.DiscoveryStreamIncludesAuthoredTags);
if (!drift.HasDrift)
{
// Recovered: the device matches the authored set again, so drop the prompt and let the next
// genuine drift re-raise with a fresh timestamp.
if (_lastDriftSignature is not null)
{
_log.Info("DriverInstance {Id}: tag-set drift cleared — device matches the authored set",
_driverInstanceId);
_lastDriftSignature = null;
_rediscoveryNeededUtc = null;
_rediscoveryReason = null;
PublishHealthSnapshot();
}
return;
}
// Report a CHANGED drift once. An unchanged one re-stamping its timestamp every interval would read
// as "it just happened again" rather than "it is still true".
if (string.Equals(_lastDriftSignature, drift.Signature, StringComparison.Ordinal)) return;
_lastDriftSignature = drift.Signature;
_rediscoveryNeededUtc = DateTime.UtcNow;
_rediscoveryReason = "device tag set differs from authored: " + drift.Describe();
_log.Info(
"DriverInstance {Id}: tag-set drift detected ({Drift}) — surfaced for operator re-browse; the served address space is unchanged",
_driverInstanceId, drift.Describe());
PublishHealthSnapshot();
}
/// <summary>Per-pass timeout for the drift browse. Bounds the mailbox suspension.</summary>
private static readonly TimeSpan DriftBrowseTimeout = TimeSpan.FromSeconds(30);
/// <summary>Flattens a captured tree to its leaf ids — the driver-side <c>FullName</c>s, which is the
/// vocabulary <c>AuthoredDiscoveryRefs</c> is defined in.</summary>
private static void CollectLeafRefs(CapturedNode node, List<string> into)
{
if (node.IsLeaf) into.Add(node.Id);
foreach (var child in node.Children) CollectLeafRefs(child, into);
}
/// <summary>Records the driver's rediscovery raise and re-publishes health so the signal reaches the
/// AdminUI promptly rather than waiting for the next 30 s heartbeat.
/// <para><b>Advisory only.</b> The served address space is deliberately NOT rebuilt: v3 authors raw tags
@@ -984,6 +1137,7 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
/// <inheritdoc />
protected override void PostStop()
{
StopDriftCheck();
DetachSubscription();
// MUST happen: the IDriver instance can outlive this actor (the host respawns a child around the
// same driver object), so a missing unsubscribe accumulates a handler per respawn holding a dead Self.
src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/TagSetDrift.cs
+92
View File
@@ -0,0 +1,92 @@
namespace ZB.MOM.WW.OtOpcUa.Runtime.Drivers;
/// <summary>
/// The difference between what a driver <b>discovers</b> on its remote and what an operator
/// <b>authored</b> — i.e. whether the device's tag set has moved out from under the deployed config.
/// </summary>
/// <param name="Appeared">Refs the remote now offers that no authored tag binds.</param>
/// <param name="Vanished">
/// Refs authored tags bind that the remote no longer offers. <b>Always empty when
/// <paramref name="VanishedDetectable"/> is false</b> — see that parameter.
/// </param>
/// <param name="VanishedDetectable">
/// False when the driver re-emits its authored tags into the same discovery stream as the device-derived
/// ones (<c>ITagDiscovery.DiscoveryStreamIncludesAuthoredTags</c>). The discovered set then always
/// contains the authored set, so a vanished tag cannot be seen. Carried explicitly so
/// <see cref="Describe"/> can say "new-on-device only" rather than implying a clean bill of health.
/// </param>
public sealed record TagSetDrift(
IReadOnlyList<string> Appeared,
IReadOnlyList<string> Vanished,
bool VanishedDetectable)
{
/// <summary>True when the two sets differ in either direction.</summary>
public bool HasDrift => Appeared.Count > 0 || Vanished.Count > 0;
/// <summary>
/// A stable identity for this drift, used to report a CHANGED drift once rather than re-reporting an
/// unchanged one on every check. Ordinal-ordered so set enumeration order cannot make an identical
/// drift look new.
/// </summary>
public string Signature { get; } =
string.Join('', Appeared) + '' + string.Join('', Vanished);
/// <summary>An operator-facing one-liner naming what moved, suitable for a UI tooltip.</summary>
/// <returns>A short human-readable description, or "no drift" when there is none.</returns>
public string Describe()
{
if (!HasDrift) return "no drift";
var parts = new List<string>(3);
if (Appeared.Count > 0) parts.Add($"{Appeared.Count} new on device (e.g. {Sample(Appeared)})");
if (Vanished.Count > 0) parts.Add($"{Vanished.Count} authored missing (e.g. {Sample(Vanished)})");
// Say so rather than let the absence of a "missing" clause read as "nothing is missing".
if (!VanishedDetectable) parts.Add("missing-tag detection unavailable for this driver");
return string.Join("; ", parts);
}
/// <summary>Names at most two refs so the message stays readable when a whole PLC program is swapped.</summary>
private static string Sample(IReadOnlyList<string> refs) =>
refs.Count <= 2 ? string.Join(", ", refs) : $"{refs[0]}, {refs[1]}, +{refs.Count - 2} more";
}
/// <summary>
/// Compares a driver's discovered refs against its authored refs. Pure and allocation-light so the
/// actor's periodic check stays cheap; kept out of <c>DriverInstanceActor</c> so it is directly testable
/// without an actor system.
/// </summary>
public static class TagSetDriftDetector
{
/// <summary>
/// Computes the two-way difference.
/// <para><b>Ordinal comparison</b>, matching how every driver keys its tag tables — a device that
/// genuinely exposes both <c>Speed</c> and <c>speed</c> must not have them collapsed.</para>
/// </summary>
/// <param name="discovered">Refs streamed by the driver's most recent discovery pass.</param>
/// <param name="authored">Refs the driver's authored tags bind (<c>ITagDiscovery.AuthoredDiscoveryRefs</c>).</param>
/// <param name="detectVanished">
/// False when the driver re-emits authored tags into its discovery stream, which makes a vanished tag
/// structurally invisible. The vanished half is then not computed at all rather than computed to a
/// misleading empty — see <see cref="TagSetDrift.VanishedDetectable"/>.
/// </param>
/// <returns>The drift, which may be empty.</returns>
public static TagSetDrift Compare(
IReadOnlyCollection<string> discovered,
IReadOnlyCollection<string> authored,
bool detectVanished = true)
{
ArgumentNullException.ThrowIfNull(discovered);
ArgumentNullException.ThrowIfNull(authored);
var authoredSet = new HashSet<string>(authored, StringComparer.Ordinal);
var discoveredSet = new HashSet<string>(discovered, StringComparer.Ordinal);
var appeared = discoveredSet.Where(r => !authoredSet.Contains(r))
.OrderBy(r => r, StringComparer.Ordinal).ToArray();
var vanished = detectVanished
? authoredSet.Where(r => !discoveredSet.Contains(r))
.OrderBy(r => r, StringComparer.Ordinal).ToArray()
: [];
return new TagSetDrift(appeared, vanished, detectVanished);
}
}