fix(opcua,drivers): carry driver status sub-codes to clients (#519) + health-write ordering (#517)

#519 — a driver's OPC UA status code now reaches a client with its sub-code intact.
DriverInstanceActor projected the driver's uint onto a 3-state Good/Uncertain/Bad enum
and OtOpcUaNodeManager re-expanded Bad to a bare 0x80000000, so BadNoCommunication,
BadTypeMismatch and BadNotSupported were indistinguishable on the wire and every
driver's status constants (incl. the 16 #497 had just corrected) were diagnostic in
logs only. Resolved by option (b): the raw uint rides AttributeValuePublished ->
AttributeValueUpdate -> IOpcUaAddressSpaceSink.WriteValue, OpcUaQuality is deleted, and
severity classification is a helper (OpcUaStatus.IsGood/IsUncertain/IsBad).

The surface was wider than the issue costed (41 files, not "the sink and 3
implementations"): OpcUaQuality had non-driver producers with no uint to carry.
Each got its natural code — BadInternalError for a failed virtual-tag script,
BadNoCommunication for driver comms loss — and ScriptedAlarmHostActor's pair of
mutually-inverse uint<->enum mappers is deleted, having round-tripped a code the
engine already held. AddressSpaceApplier's removal writes deliberately keep the
generic Bad: not losing a code the driver had is this issue; inventing one for the
deploy path is a separate call.

Behaviour change: WriteAlarmQuality fires its Part 9 event on a status-code change
rather than a severity-bucket change. While the annotation was 3-state those were the
same predicate; now a move between two Bad reasons changes what a subscriber reads.

New SDK-free named constants live on Commons.OpcUa.OpcUaStatus, and
StatusCodeParityTests is widened to cover them — that guard is reflection-driven, so
it can only see a named constant, never an inline literal.

#517 — the symptom is real but the issue named the wrong path and its fix was unsafe.
ShutdownAsync cannot race: DriverInstanceActor calls it from PostStop, after the
mailbox drains. (Core/Hosting/DriverHost, the one class with a concurrent GetHealth,
has no production call site.) InitializeAsync can: it runs on a fire-and-forget
Task.Run, and MTConnectDriver writes Healthy then starts loops that can throw into a
catch which disposes the client before writing Faulted.

Moving the write earlier, as suggested, inverts the terminal state — teardown drains
in-flight loops that write health as they unwind, so a late loop could leave Healthy at
rest with every connection closed. The fix is a bracket (conservative before, terminal
after), applied to MTConnect's init-failure path only; shutdown keeps its single
trailing write, and the rule plus the reachability table are documented on
IDriver.ShutdownAsync.

CalculationDriver.GetHealth returned a hard-coded Healthy that consulted nothing —
Healthy before initialize, after shutdown and after dispose. Now derived from _running.
Its test, named Health_is_always_Connected, asserted the defect and is replaced.

Every new guard was falsified by breaking the production code and watching it go red.
The two sub-code tests each assert two different Bad codes, since a single-code
assertion could pass against a hard-coded constant under the old projection.

Pre-existing failures, verified unchanged on a stashed tree and not chased: 4 in
Driver.AbLegacy.IntegrationTests (slc500/micrologix/plc5 fixtures not up) and 3 in
Host.IntegrationTests.

Closes #517
Closes #519

Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo

src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/AlarmConditionSnapshot.cs

@@ -16,11 +16,13 @@ namespace ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
 /// <param name="Shelving">The shelving mode (ShelvingState): unshelved, one-shot, or timed.</param>
 /// <param name="Severity">OPC UA severity on the 1..1000 scale (the SDK <c>SetSeverity</c> input).</param>
 /// <param name="Message">The human-readable condition message (LocalizedText payload).</param>
-/// <param name="Quality">
-///     Quality of the condition's source data (Part 9 <c>ConditionType.Quality</c>). Carried so a
+/// <param name="StatusCode">
+///     Status code of the condition's source data (Part 9 <c>ConditionType.Quality</c>). Carried so a
 ///     comms-lost native source can report a non-<c>Good</c> condition instead of the accidentally-Good
 ///     default (issue #477). It is a pure annotation — it never alters Active/Acked/Retain. Defaults to
-///     <see cref="OpcUaQuality.Good"/> so scripted-alarm callers (which stay Good in v1) need not supply it.
+///     <see cref="OpcUaStatus.Good"/> so scripted-alarm callers (which stay Good in v1) need not supply it.
+///     Carries the full sub-code since #519, so a comms-lost condition reports
+///     <see cref="OpcUaStatus.BadNoCommunication"/> rather than a bare <see cref="OpcUaStatus.Bad"/>.
 /// </param>
 public sealed record AlarmConditionSnapshot(
     bool Active,
@@ -30,7 +32,7 @@ public sealed record AlarmConditionSnapshot(
     AlarmShelvingKind Shelving,
     ushort Severity,
     string Message,
-    OpcUaQuality Quality = OpcUaQuality.Good);
+    uint StatusCode = OpcUaStatus.Good);
 
 /// <summary>
 ///     Commons-local mirror of the Core <c>ShelvingKind</c> enum so this assembly carries no

src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/DeferredAddressSpaceSink.cs

@@ -23,15 +23,15 @@ public sealed class DeferredAddressSpaceSink : IOpcUaAddressSpaceSink, ISurgical
         _inner = sink ?? NullOpcUaAddressSpaceSink.Instance;
 
     /// <inheritdoc />
-    public void WriteValue(string nodeId, object? value, OpcUaQuality quality, DateTime sourceTimestampUtc, AddressSpaceRealm realm)
-        => _inner.WriteValue(nodeId, value, quality, sourceTimestampUtc, realm);
+    public void WriteValue(string nodeId, object? value, uint statusCode, DateTime sourceTimestampUtc, AddressSpaceRealm realm)
+        => _inner.WriteValue(nodeId, value, statusCode, sourceTimestampUtc, realm);
 
     /// <inheritdoc />
     public void WriteAlarmCondition(string alarmNodeId, AlarmConditionSnapshot state, DateTime sourceTimestampUtc, AddressSpaceRealm realm)
         => _inner.WriteAlarmCondition(alarmNodeId, state, sourceTimestampUtc, realm);
 
-    public void WriteAlarmQuality(string alarmNodeId, OpcUaQuality quality, DateTime sourceTimestampUtc, AddressSpaceRealm realm)
-        => _inner.WriteAlarmQuality(alarmNodeId, quality, sourceTimestampUtc, realm);
+    public void WriteAlarmQuality(string alarmNodeId, uint statusCode, DateTime sourceTimestampUtc, AddressSpaceRealm realm)
+        => _inner.WriteAlarmQuality(alarmNodeId, statusCode, sourceTimestampUtc, realm);
 
     /// <inheritdoc />
     public void MaterialiseAlarmCondition(string alarmNodeId, string equipmentNodeId, string displayName, string alarmType, int severity, AddressSpaceRealm realm, bool isNative = false)

src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/IOpcUaAddressSpaceSink.cs

@@ -8,16 +8,19 @@ namespace ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
 /// </summary>
 public interface IOpcUaAddressSpaceSink
 {
-    /// <summary>Write a Variable node's current value + quality + source timestamp.</summary>
+    /// <summary>Write a Variable node's current value + status code + source timestamp.</summary>
     /// <param name="nodeId">The OPC UA node ID of the variable.</param>
     /// <param name="value">The value to write.</param>
-    /// <param name="quality">The quality status of the value.</param>
+    /// <param name="statusCode">The full OPC UA status code, sub-code included. Passed to the SDK verbatim
+    /// (#519): a driver's <see cref="OpcUaStatus.BadNoCommunication"/> reaches the client as
+    /// <c>BadNoCommunication</c>, not as a generic <see cref="OpcUaStatus.Bad"/>. Use
+    /// <see cref="OpcUaStatus"/>' named constants rather than a literal.</param>
     /// <param name="sourceTimestampUtc">The source timestamp in UTC.</param>
     /// <param name="realm">Which of the two v3 namespaces (<see cref="AddressSpaceRealm.Raw"/> device tree or
     /// <see cref="AddressSpaceRealm.Uns"/> equipment tree) the <paramref name="nodeId"/> lives in — the sink
     /// resolves the namespace index from the realm rather than parsing it out of the id string. WP3's fan-out
     /// calls this once per NodeId (raw + every referencing UNS node) with the matching realm.</param>
-    void WriteValue(string nodeId, object? value, OpcUaQuality quality, DateTime sourceTimestampUtc, AddressSpaceRealm realm);
+    void WriteValue(string nodeId, object? value, uint statusCode, DateTime sourceTimestampUtc, AddressSpaceRealm realm);
 
     /// <summary>Write an alarm-condition's full Part 9 state. When a real condition node has been
     /// materialised for <paramref name="alarmNodeId"/> via <see cref="MaterialiseAlarmCondition"/>,
@@ -34,15 +37,17 @@ public interface IOpcUaAddressSpaceSink
     void WriteAlarmCondition(string alarmNodeId, AlarmConditionSnapshot state, DateTime sourceTimestampUtc, AddressSpaceRealm realm);
 
     /// <summary>#477 — annotate a materialised condition's source-data quality OUT OF BAND from any alarm
-    /// transition (used by the driver-connectivity path: comms lost → <see cref="OpcUaQuality.Bad"/>,
-    /// restored → <see cref="OpcUaQuality.Good"/>). Sets ONLY the condition's Quality — never
+    /// transition (used by the driver-connectivity path: comms lost → <see cref="OpcUaStatus.BadNoCommunication"/>,
+    /// restored → <see cref="OpcUaStatus.Good"/>). Sets ONLY the condition's Quality — never
     /// Active/Acked/Severity/Retain (a comms-lost active alarm stays active) — and fires one Part 9 event
-    /// only on a quality-bucket change. A no-op for an unmaterialised / non-condition node.</summary>
+    /// only on a change to the status code itself — since #519 that includes a move between two Bad
+    /// sub-codes, which genuinely changes what a subscriber reads. A no-op for an unmaterialised /
+    /// non-condition node.</summary>
     /// <param name="alarmNodeId">The condition node id (RawPath for a native alarm).</param>
-    /// <param name="quality">The source-data quality to annotate.</param>
+    /// <param name="statusCode">The source-data status code to annotate, sub-code included.</param>
     /// <param name="sourceTimestampUtc">The connectivity transition timestamp in UTC.</param>
     /// <param name="realm">The namespace realm the condition was materialised under.</param>
-    void WriteAlarmQuality(string alarmNodeId, OpcUaQuality quality, DateTime sourceTimestampUtc, AddressSpaceRealm realm);
+    void WriteAlarmQuality(string alarmNodeId, uint statusCode, DateTime sourceTimestampUtc, AddressSpaceRealm realm);
 
     /// <summary>
     ///     Materialise a real OPC UA Part 9 alarm-condition node under its equipment folder so clients
@@ -160,10 +165,6 @@ public interface IOpcUaAddressSpaceSink
                       string referenceType = "Organizes");
 }
 
-/// <summary>OPC UA status code projection — Good / Uncertain / Bad. Real SDK has finer-grained
-/// codes; the engine actors only need this 3-state classification.</summary>
-public enum OpcUaQuality { Good, Uncertain, Bad }
-
 /// <summary>No-op sink. Bound by default so the actors are safe to run in dev / Mac /
 /// integration tests without a real SDK behind them.</summary>
 public sealed class NullOpcUaAddressSpaceSink : IOpcUaAddressSpaceSink
@@ -172,11 +173,11 @@ public sealed class NullOpcUaAddressSpaceSink : IOpcUaAddressSpaceSink
     private NullOpcUaAddressSpaceSink() { }
 
     /// <inheritdoc />
-    public void WriteValue(string nodeId, object? value, OpcUaQuality quality, DateTime sourceTimestampUtc, AddressSpaceRealm realm) { }
+    public void WriteValue(string nodeId, object? value, uint statusCode, DateTime sourceTimestampUtc, AddressSpaceRealm realm) { }
 
     /// <inheritdoc />
     public void WriteAlarmCondition(string alarmNodeId, AlarmConditionSnapshot state, DateTime sourceTimestampUtc, AddressSpaceRealm realm) { }
-    public void WriteAlarmQuality(string alarmNodeId, OpcUaQuality quality, DateTime sourceTimestampUtc, AddressSpaceRealm realm) { }
+    public void WriteAlarmQuality(string alarmNodeId, uint statusCode, DateTime sourceTimestampUtc, AddressSpaceRealm realm) { }
 
     /// <inheritdoc />
     public void MaterialiseAlarmCondition(string alarmNodeId, string equipmentNodeId, string displayName, string alarmType, int severity, AddressSpaceRealm realm, bool isNative = false) { }

src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/OpcUaStatus.cs

@@ -0,0 +1,68 @@
+namespace ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
+
+/// <summary>
+///     The handful of OPC UA status codes the engine layer needs to spell, plus the Part 4 severity
+///     classification helpers that replaced the retired <c>OpcUaQuality</c> 3-state enum (Gitea #519).
+///
+///     <para><b>Why these are bare <c>uint</c> literals.</b> Commons and Runtime deliberately do not
+///     reference the OPC UA SDK — <see cref="IOpcUaAddressSpaceSink"/> exists precisely so the Runtime
+///     project can publish values without dragging in <c>Opc.Ua.Server</c>. That is the same layering
+///     decision the driver assemblies make, and it carries the same single failure mode: nothing checks
+///     the literal against the name. <c>StatusCodeParityTests</c> closes it by reflection, and it can
+///     only see a <b>named</b> constant — so add codes here rather than writing a literal at a call
+///     site, exactly as Gitea #497 concluded for the drivers.</para>
+///
+///     <para><b>Severity, not equality, is what callers should branch on.</b> A status code's top two
+///     bits carry its Part 4 severity (00 = Good, 01 = Uncertain, 10/11 = Bad); everything below is the
+///     sub-code. Before #519 the engine collapsed every value onto a 3-state enum here and the node
+///     manager re-expanded <c>Bad</c> to a bare <c>0x80000000</c>, so no driver's sub-code ever reached
+///     a client. The sub-code now rides through to the SDK verbatim, and any code that only needs the
+///     bucket uses <see cref="IsGood"/> / <see cref="IsUncertain"/> / <see cref="IsBad"/> rather than
+///     comparing against <see cref="Good"/> or <see cref="Bad"/>.</para>
+/// </summary>
+public static class OpcUaStatus
+{
+    /// <summary>The value is good. Part 4 severity <c>00</c>; the whole code is zero.</summary>
+    public const uint Good = 0x00000000;
+
+    /// <summary>Generic uncertain. Prefer a specific <c>Uncertain*</c> sub-code where one applies.</summary>
+    public const uint Uncertain = 0x40000000;
+
+    /// <summary>Generic bad, carrying no diagnostic beyond the severity. This is the value the retired
+    /// 3-state projection produced for <em>every</em> bad status; a caller that has something more
+    /// specific should say so instead.</summary>
+    public const uint Bad = 0x80000000;
+
+    /// <summary>No value has been received from the device yet. Written at materialisation and on every
+    /// address-space reset, so a freshly-deployed or just-rebuilt node reads as "not yet polled" rather
+    /// than as a device fault.</summary>
+    public const uint BadWaitingForInitialData = 0x80320000;
+
+    /// <summary>Communication with the source has been lost. The driver-connectivity annotation for a
+    /// native alarm condition, and the sentinel several drivers map an explicit "unavailable" reading to.</summary>
+    public const uint BadNoCommunication = 0x80310000;
+
+    /// <summary>An internal failure produced no usable value — the virtual-tag path's code for a script
+    /// that threw or timed out.</summary>
+    public const uint BadInternalError = 0x80020000;
+
+    /// <summary>Part 4 severity bucket of <paramref name="statusCode"/>: the top two bits.</summary>
+    /// <param name="statusCode">The status code to classify.</param>
+    /// <returns>0 for Good, 1 for Uncertain, 2 or 3 for Bad.</returns>
+    public static uint Severity(uint statusCode) => statusCode >> 30;
+
+    /// <summary>True when <paramref name="statusCode"/> is in the Good severity bucket.</summary>
+    /// <param name="statusCode">The status code to classify.</param>
+    /// <returns><see langword="true"/> when the value is usable.</returns>
+    public static bool IsGood(uint statusCode) => Severity(statusCode) == 0;
+
+    /// <summary>True when <paramref name="statusCode"/> is in the Uncertain severity bucket.</summary>
+    /// <param name="statusCode">The status code to classify.</param>
+    /// <returns><see langword="true"/> when the value is questionable but present.</returns>
+    public static bool IsUncertain(uint statusCode) => Severity(statusCode) == 1;
+
+    /// <summary>True when <paramref name="statusCode"/> is in the Bad severity bucket.</summary>
+    /// <param name="statusCode">The status code to classify.</param>
+    /// <returns><see langword="true"/> when the value must not be trusted.</returns>
+    public static bool IsBad(uint statusCode) => Severity(statusCode) >= 2;
+}

src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/IDriver.cs

@@ -38,11 +38,30 @@ public interface IDriver
     Task ReinitializeAsync(string driverConfigJson, CancellationToken cancellationToken);
 
     /// <summary>Stop the driver, close connections, release resources. Called on shutdown or driver removal.</summary>
+    /// <remarks>
+    ///     <para><b>Land the terminal health value AFTER the teardown, never only before it</b> (Gitea #517).
+    ///     Teardown drains in-flight poll / probe loops, and those loops write health as they unwind — so a
+    ///     driver that wrote <see cref="DriverState.Unknown"/> first and then tore down could have a late
+    ///     loop overwrite it with <see cref="DriverState.Healthy"/> and leave THAT as the resting state,
+    ///     with every connection already closed. This is why the fleet writes last, and it is the reason
+    ///     #517's original suggestion — "write the transitional value before disposing" — is only safe as an
+    ///     ADDITIONAL write, bracketing the teardown, rather than as a move.</para>
+    ///     <para>Bracketing matters wherever a <see cref="GetHealth"/> can land mid-teardown. It cannot on
+    ///     this path today: <c>DriverInstanceActor</c> calls this from <c>PostStop</c>, on the actor thread,
+    ///     after the mailbox has drained, so its health poll is serialised against it. It CAN on the
+    ///     initialize path, which the same actor runs on a fire-and-forget <c>Task.Run</c> while the health
+    ///     tick keeps firing — so a driver whose <see cref="InitializeAsync"/> disposes anything on its
+    ///     failure path must write the failed state before disposing as well as after.</para>
+    /// </remarks>
     /// <param name="cancellationToken">Cancellation token for the operation.</param>
     /// <returns>A task that represents the asynchronous operation.</returns>
     Task ShutdownAsync(CancellationToken cancellationToken);
 
-    /// <summary>Current health snapshot, polled by Core for the status dashboard and ServiceLevel.</summary>
+    /// <summary>Current health snapshot, polled by Core for the status dashboard and ServiceLevel.
+    /// <para>Must reflect the driver's ACTUAL state rather than a constant — a hard-coded
+    /// <see cref="DriverState.Healthy"/> reports a healthy driver after shutdown and before initialize
+    /// alike, which is what Gitea #517 found in <c>CalculationDriver</c>. A driver with no transport still
+    /// has a running / not-running distinction worth reporting.</para></summary>
     /// <returns>The driver's current health snapshot.</returns>
     DriverHealth GetHealth();
 

src/Core/ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms/ScriptedAlarmEngine.cs

@@ -1011,9 +1011,10 @@ public sealed record ScriptedAlarmEvent(
     DateTime TimestampUtc,
     string? Comment = null,
     bool HistorizeToAveva = true,
-    // #478 — the worst OPC UA StatusCode across the alarm's input tags at evaluation time. A raw uint
-    // (Core.ScriptedAlarms does not reference Commons/OpcUaQuality); the host maps it to OpcUaQuality by
-    // the top-2 severity bits. Default 0u == Good keeps every existing constructor call unchanged.
+    // #478 — the worst OPC UA StatusCode across the alarm's input tags at evaluation time. A raw uint,
+    // since Core.ScriptedAlarms does not reference Commons. Since #519 the host carries it to the condition
+    // node UNCHANGED — it used to be collapsed to a 3-state bucket and re-expanded, losing the sub-code.
+    // Default 0u == Good keeps every existing constructor call unchanged.
     uint WorstInputStatusCode = 0u);
 
 /// <summary>