Closes arch-review remediation residual #1 (DCL unsubscribe-during-reconnect
count staleness).
DataConnectionActor tracked TotalSubscribedTags/ResolvedTags as two int fields
incremented and decremented at five independent sites. ReSubscribeAll clears the
very maps those decrements key off (_subscriptionIds, _unresolvedTags) while
deliberately preserving _subscriptionsByInstance, so an unsubscribe landing
inside a reconnect window matched NEITHER decrement branch: the total leaked +1
per subscribe/reconnect/unsubscribe churn cycle, permanently and cumulatively.
The 37f13e2e discard gate stopped the orphan-handle half of that race; it could
not stop the counters drifting, because they were state of their own.
Both counts are now DERIVED at report time from the authoritative per-tag
collections, which makes the drift unrepresentable rather than merely guarded:
total = _instancesByTag.Count (the per-tag counted set the residual
called for — distinct tags with at least
one subscribing instance)
resolved = _subscriptionIds.Count (tags for which the adapter holds a handle)
Two semantic corrections fall out of the derivation:
- A tag whose subscribe failed at CONNECTION level now counts toward the total.
It was excluded before, yet the reconnect re-subscribe re-issued it from
_subscriptionsByInstance and booked it as resolved — resolved above total, and
a total driven negative by the eventual unsubscribe.
- _tagSubscriberCount is deleted. It duplicated _instancesByTag exactly, so
HandleUnsubscribe's last-subscriber test is now "did UnindexTag drop the key?"
— still O(1), with no parallel count that can disagree about when a handle is
released. The subscribe-success promotion split (fresh vs. unresolved→resolved)
also goes: it existed only to pick which scalar to bump; set sizes get
DataConnectionLayer-020's double-count cases right for free.
Behavior is otherwise unchanged — same logging, same handle release, same
unresolved-tag probing, same in-flight-unsubscribe discard semantics (the long
comment block there is updated for the mechanics that changed).
Tests: five TagResolutionCounts_* cases in DataConnectionActorBatchTests
covering the churn repro (3 cycles), a shared tag losing one instance mid
reconnect, connection-level failure then recovery, plain subscribe/unsubscribe
cycles, and a completed reconnect re-subscribe. Verified failing against the
pre-fix actor (churn: total 1 not 0; connection-level: total 0 not 1) and
passing after. Full DCL suite 319/319; solution builds with 0 warnings.
Docs: Component-DataConnectionLayer.md health-reporting section describes the
derived counts; residuals register item 1 marked RESOLVED.
MES alarm-status API §6.4 (docs/plans/2026-06-30-mes-alarm-status-api.md,
Phase 1 task 1). MES needs a real AckDT for a triggered alarm, and the mirror
carried acked-vs-unacked but never WHEN. AckTime now rides the whole path:
DCL transition -> AlarmStateChanged -> gRPC AlarmStateUpdate -> site SQLite.
Stamping rule, identical on both protocols: non-null ONLY while the condition
is active AND acknowledged. That single predicate yields all three required
behaviours -- null while unacked, cleared on re-raise (a re-raise arrives
unacknowledged), and no phantom ack on a return-to-normal. The last one is
load-bearing for MxGateway, which maps INACTIVE to Acknowledged = true; without
the active check every clear would claim an ack the system never observed.
Provenance is honest, never fabricated:
- OPC UA A&C supplies a TRUE ack instant, so we now select it:
AcknowledgeableConditionType/AckedState/TransitionTime at SelectClause
index 18, APPENDED so the positional reads at 0-17 keep their meaning.
Servers that omit the field fall back to the event's own Time.
- MxAccess Gateway supplies none, so the ack transition's own timestamp is
used -- accurate to when the system SAW the ack. An ACTIVE_ACKED
re-subscribe snapshot restores one from LastTransitionTimestamp rather
than dropping it.
The decision lives in pure mappers (Opc/Mx AlarmMapper.DeriveAckTime), so it is
unit-tested with no live server or gateway.
Additive-only throughout: init-only property on AlarmStateChanged, trailing
optional positional on NativeAlarmTransition (all 14-arg call sites untouched),
proto field 24 (never reusing a number) regenerated via docker/regen-proto.sh
sitestream with the csproj diff verified empty.
Persistence rides native_alarm_state's existing metadata_json blob rather than
a new column -- deliberately. That table is RegisterReplicated in
SiteLocalDbSetup and LocalDb builds its CDC triggers from the column list at
registration time, so an additive JSON property changes no schema, no triggers
and no replication contract; metadata_json is exactly the extension point UA4
introduced for this. Rows written before the field deserialize it as null.
Tests: 4 OPC UA + 6 MxGateway mapper cases, 3 NativeAlarmActor (emit,
failover rehydrate, pre-AckTime row), 1 proto round-trip incl. the null case,
4 Commons additive/back-compat. The OPC UA SelectClause count lock-in moves
18 -> 19 with an index-18 assertion -- intended, the clause is appended, which
is precisely what that guard exists to make visible.
Code review of a1abbff7 found two edge cases: a discovery/advertised URL with no
explicit port emitted a malformed ':-1' (opc.tcp has no Uri default port), and an
IPv6 literal host could lose/double its brackets. Omit the port when neither URL
carries one; bracket an IPv6 host only when missing. +4 tests (12 total).
An OPC UA server advertises its base address from its own config, not the route
the client took — commonly a 0.0.0.0 wildcard bind or an internal container/NAT
hostname. The OPC Foundation session dials EndpointDescription.EndpointUrl
verbatim, so a 0.0.0.0 advertisement resolved to the client's own loopback and
the connect failed. RealOpcUaClient now swaps the advertised authority for the
reachable host/port (ConnectAsync + VerifyEndpointAsync), preserving scheme+path;
no-op when already reachable. Surfaced live-gating OtOpcUa v3 (#14).
A native OPC UA alarm source's SourceReference has to be two things at once:
it is parsed as a NodeId to open the monitored item, and matched as a plain-name
prefix against the event's SourceName to route the transition to an instance. No
string is both, so a NodeId-form binding subscribed correctly and then silently
dropped every transition — "pymodbus/plc/HR200".StartsWith("nsu=...;s=pymodbus/plc/HR200")
is false. Only the empty (Server-object) binding worked, because StartsWith("")
matches everything, which is why the sole live smoke test never caught it.
Each OPC UA alarm feed is opened for exactly one binding, so every transition on
it belongs to that binding. The adapter now tags each transition's routing
identity (SourceObjectReference) with the binding string verbatim via the pure
OpcUaAlarmMapper.BuildIdentity, making DataConnectionActor's routing key an exact
match regardless of whether the binding is stored as ns=<index> or the durable
nsu=<uri> form. The Server-object aggregate feed keeps an empty routing identity,
so it reaches only "mirror everything" subscribers and never leaks into a
specific-node binding. The per-condition SourceReference key stays the readable
SourceName.ConditionName, so persistence and display are unchanged, and MxGateway
is untouched — its bindings are names and its mapper already emits matching names.
Unblocked by lmxopcua#473 (OtOpcUa now populates SourceNode/SourceName/EventType
on conditions); SourceName is the RawPath, so the per-condition key is unique.
Live end-to-end verification against native alarms still needs a v3 rig.
Fixes: Gitea #17
A stored ns=<index> reference is only meaningful against one server's namespace
table. A server that adds, removes or reorders a namespace silently re-points
every binding: best case BadNodeIdUnknown, worst case the index now names a
different namespace holding a colliding identifier and the binding resolves to
the wrong node with Good quality. Nothing could detect that, because ScadaBridge
stored no namespace URI anywhere.
OtOpcUa v3.0 makes this concrete: v2's sole custom namespace and v3's raw
namespace both sit at index 2, so v2-era bindings resolve against v3 without
error while meaning something else entirely.
Adds OpcUaNodeReference as the single translation seam between stored references
and the wire. Resolve() accepts both ns=<index>;s=<id> (existing bindings, which
keep working unchanged) and the durable nsu=<uri>;s=<id>, mapping the URI to the
live index at use time; it is wired into every parse site — subscribe, read,
write, alarm-subscribe and browse. An unpublished URI now throws naming the URI
rather than binding to whatever occupies that index, and a svr= reference to
another server is rejected instead of being resolved against the wrong address
space.
The browser emits ToDurable(), so what the picker shows is what gets stored and
newly-authored bindings are index-proof from the start. That also closes a
round-trip gap: browse previously emitted ExpandedNodeId.ToString(), which for a
URI- or server-index-carrying reference produced a string NodeId.Parse could not
read back — the same method already resolved it correctly 57 lines later.
Bindings stored before this change keep their ns= form and keep working; they are
only as durable as the server's namespace order. Re-authoring against the picker
is what makes them durable, and that re-bind still needs a live v3 rig.
Refs: Gitea #14
Fixes the 8 findings from the 2026-06-24 re-review (commit c42bb485), with a
regression test per Medium finding:
- DataConnectionLayer-029 (Med): HandleAlarmSubscribeCompleted now mirrors the
tag-path re-check — if a feed is already stored for the source, release the
redundant just-created subscription instead of overwriting + leaking the first
one (the double-subscribe window DCL-023 reopened). +regression test.
- InboundAPI-031 (Med): remove WaitForAttribute's local 5s grace backstop (tighter
than the CommunicationService Ask's timeout+IntegrationTimeout round-trip budget,
so a slow-but-valid timed-out 'false' got cancelled into a 500). Link only the
client-abort + explicit caller tokens; the lower layer owns the backstop. +test.
- SiteRuntime-032 (Med): derive the deployed count from an authoritative set of
deployed config names (HashSet) instead of a map-presence-gated int, so deleting
a DISABLED instance decrements correctly (SiteRuntime-029's gate leaked it).
+deploy->disable->delete regression test.
- StoreAndForward-028 (Med): reset _bufferedCount in StopAsync alongside the
register-guard so a same-instance Stop->Start re-seeds from a clean base (no ~2N
gauge double-count). +restart regression test.
- AuditLog-017 (Low): test the OnIngestAsync scope-resolution guard (actor survives,
replies empty, counts the failure) — no longer unpinned.
- CentralUI-037 / ScriptAnalysis-009 / SiteRuntime-033 (Low): doc-comment + spec
fixes (Database-throws in the inbound sandbox; baseReferences param wording;
native-alarm cap return-to-normal + per-condition NativeAlarmDropped eviction).
Targeted suites green: SiteRuntime 5, StoreAndForward 6, InboundAPI 31,
DataConnectionLayer 10, AuditLog 5, ScriptAnalysis 40, CentralUI ScriptAnalysis 52.
Remediation from the full per-module code review at 4307c381 (findings recorded
separately in code-reviews/).
Highs fixed:
- DeploymentManager-025/SiteRuntime-031: stop broadcasting notification lists + SMTP
configs (incl. credentials) to sites; site purges already-persisted rows on apply
(enforces the central-only delivery design; clears plaintext SMTP creds at rest).
- DataConnectionLayer-023: guard the native-alarm subscribe path against the
mid-flight-unsubscribe adapter-feed leak (mirrors the DCL-021 tag-path fix).
- SiteEventLogging-024: normalize From/To query bounds to UTC (the -016 fix the
audit trail claimed but never committed).
- KpiHistory-001: add an in-flight guard to the recorder sample tick.
- ScriptAnalysis-001: harden the trust analyzer's TPA-absent fallback (resolve
forbidden anchors in the minimal reference set; warn on degraded mode) — anchors
added to validation references only, never the compile gate.
(InboundAPI-026 left to the feat/ipsen-movein effort per owner decision.)
Medium/Low: DM-026 deterministic deploy-status tiebreaker; SR-027/028/029/030
native-alarm leak/phantom-active/delete-during-redeploy fixes; AL-013/014/016;
TE-024 (folder-mutation audit rows now persisted)/025; SF-025 gauge-provider
clear-on-stop; ESG-025/026; SEC-023/024/025; SCA-007/008/009; plus doc/test
accuracy COM-023/024, HOST-025/026, HM-024/025, NS-027/028.
Full-solution build 0 warnings; ~3560 tests across 18 touched suites green.
STATIC tags (no further OnDataChange after advise) depend entirely on the
seed read. Pre-fix HandleSubscribe seeded only on Success && Value != null,
silently dropping a seed that raced the just-created advise (VT_EMPTY) — so a
static tag stayed Uncertain forever while the source read Good. ReSubscribeAll
did no seeding at all, so a static tag could not self-heal across reconnect.
- New SeedTagsAsync helper: per-tag ReadAsync (not a bulk read — some gateways
time out on large batches) with round-based bounded retry
(SeedReadMaxAttempts/SeedReadRetryDelay), logging any tag that never yields a
value (named — previously zero log trace).
- HandleSubscribe seed loop delegates to SeedTagsAsync.
- ReSubscribeAll re-seeds re-advised tags after reconnect via the
generation-guarded TagValueReceived path (fan-out keys off
_subscriptionsByInstance, preserved across reconnect).
Diagnosed live on wonder-app-vd03 2026-06-17 (see scadabridge-dcl-static-tag-false-bad).
Mechanism #2 (single transient-bad push) left as a follow-up.
Surface native (Galaxy/MxGateway) alarms by their object-relative reference
(e.g. "Z28061.HeartbeatTimeoutAlarm") instead of the gateway's full provider
reference ("Galaxy!<area>.<object>.<alarm>"). The area is already preserved in
Category and the object reference is globally unique within the galaxy, so the
full provider prefix added only noise to the alarm identity operators see.
MxGatewayAlarmMapper.MapTransition/MapSnapshot now set SourceReference from
SourceObjectReference, falling back to AlarmFullReference only when the gateway
omits the object reference. +2 mapper tests; full DCL suite green (158).
The MxGateway alarm mapper emits the SnapshotComplete framing sentinel with
empty SourceReference/SourceObjectReference. HandleAlarmTransitionReceived
routed every transition by prefix match against the subscriber's source, so
the empty-ref sentinel ('' .StartsWith("<src>.") == false) was dropped for
any specific source. The NativeAlarmActor buffers snapshot conditions and only
flushes them on SnapshotComplete, so statically-active native alarms delivered
only in the initial snapshot (no later live transition) never surfaced.
Broadcast the SnapshotComplete sentinel to all alarm subscribers (bypassing the
source match + type filter) so each NativeAlarmActor's snapshot swap completes.
Adds a regression test using the real empty-ref sentinel against a specific
(prefix) source.
DataConnectionActor seeded a tag's initial value by Tell-ing TagValueReceived
from HandleSubscribe's background task, which runs BEFORE HandleSubscribeCompleted
registers the instance's tags in _subscriptionsByInstance. HandleTagValueReceived's
fan-out then found no subscriber and dropped the value. A tag that soon gets a
data-change notification recovers, but a STATIC tag (e.g. an idle MES field that
never changes) was left Uncertain forever — the dropped seed was its only value.
Seeds now ride back on SubscribeCompleted and are delivered after registration,
reusing HandleTagValueReceived's generation guard, fan-out and quality accounting.
+1 regression test (DCL026).
OPC UA (RealOpcUaClient):
- Append 5 new SelectClauses at indices 13–17 (never renumber 0–12):
- 13: AlarmConditionType/ActiveState/TransitionTime → OriginalRaiseTime
- 14–17: LimitAlarmType HighHighLimit/HighLimit/LowLimit/LowLowLimit → LimitValue
- New OpcUaAlarmMapper.PickLimitValue helper: first non-null in HiHi→Hi→Lo→LoLo
priority order, InvariantCulture-formatted; empty string for non-limit alarm types.
- HandleAlarmEvent reads new indices with fields.Count > N guards; hard minimum (6)
unchanged so base ConditionType events still process without the limit fields.
- Document unavailable-by-protocol fields (Category, Description, OperatorUser,
CurrentValue) inline in BuildAlarmEventFilter and HandleAlarmEvent.
MxGateway (MxGatewayAlarmMapper):
- MapTransition: CurrentValue and LimitValue now populated via MxValueToString
(uses MxValueExtensions.ToClrValue + InvariantCulture) from OnAlarmTransitionEvent
proto fields current_value/limit_value.
- MapSnapshot: same — populated from ActiveAlarmSnapshot.current_value/limit_value.
- MxValueToString helper (internal): null-safe MxValue → string conversion.
Tests (17 new, 40 total pass):
- OpcUaAlarmMapperTests: PickLimitValue priority, InvariantCulture, all-null case.
- MxGatewayAlarmMapperTests: CurrentValue/LimitValue populate from double/string
MxValue; absent fields yield empty strings.
- RealOpcUaClientAlarmFilterTests: index alignment assertions (count=18, per-index
TypeDefinitionId+BrowsePath), regression guard on existing indices 0–12.
HandleAlarmEvent set AlarmTypeName to the event-type NodeId string ("i=9341"),
but the client-side conditionFilter gate (and the OPC UA WhereClause) use friendly
type names — so a friendly-name filter built a correct server WhereClause yet the
client gate dropped every event (zero alarms delivered). Resolve the event-type
NodeId to its friendly name via an inverse of KnownConditionTypeIds (NodeId-string
fallback for custom types) so both sides agree. Also fix a dead-code ternary in
the SourceName derivation.
conditionFilter was plumbed end-to-end but applied nowhere — a filtered source
silently mirrored all conditions. Define the filter as a comma-separated,
case-insensitive list of condition type names (blank = all); enforce it
authoritatively client-side in DataConnectionActor routing (uniform across OPC UA
+ MxGateway) and, for OPC UA, additionally build a server-side EventFilter
WhereClause as a bandwidth optimization.
The NativeAlarmActor sends SubscribeAlarmsRequest to the DCL manager, but the
manager only routed tag/write/browse messages to the per-connection
DataConnectionActor — alarm subscribe/unsubscribe were unhandled and dead-lettered,
so native alarms never subscribed at runtime. Caught by live T28 deployment.
Mirrors the existing HandleRoute forwarding.
Adds IAlarmSubscribableConnection to MxGatewayDataConnection (shared session-less
feed, ref-counted), IMxGatewayClient.RunAlarmStreamAsync over the package
StreamAlarmsAsync with internal reconnect, and MxGatewayAlarmMapper
(AlarmFeedMessage/OnAlarmTransitionEvent -> NativeAlarmTransition). Behavior
verified against a live gateway in Task 28; mapper unit-tested.
Expanding a Galaxy object in the tag picker hung on "loading…": the browse
reply inlined every child's full attribute set (~152 KB), exceeding Akka's
128 KB remote frame, and remoting silently discarded the oversized reply.
Browse path (DataConnectionLayer):
- RealMxGatewayClient: navigation now uses BrowseChildren(include_attributes=
false) — child objects only — and an object's own attributes load lazily via
DiscoverHierarchy(root, max_depth=0) when it's expanded. Payload drops from
~152 KB/level to a few KB. Seam contract unchanged.
- DataConnectionActor.CapBrowseChildren: protocol-agnostic byte-budget cap
(~100 KB) on every BrowseNodeResult before it crosses the site→central
frame, OR-ing the adapter's own Truncated flag. Byte budget, not a count —
the only bound that holds regardless of NodeId/attribute-name length.
- RealOpcUaClient: requestedMaxReferencesPerNode 1000 → 500 to narrow the
window before the byte budget applies.
- Graceful gRPC Unimplemented handling → NotSupportedException →
BrowseFailureKind.NotBrowsable with an actionable message (older gateway
builds lacking BrowseChildren).
Picker UI (CentralUI):
- NodeBrowserDialog: modal-lg → modal-xl; new scoped .razor.css caps the tree
at 55vh with its own scrollbar so manual entry + Select/Cancel stay visible.
- Protocol-agnostic failure messages (was hardcoded "OPC UA …"); renamed the
leftover opcua-browser-tree class to node-browser-tree.
Tests: new frame-budget cap test + NotSupported=>NotBrowsable mapping test;
DCL suite 88/88. Doc: Component-DataConnectionLayer.md records the lazy
attribute-light browse and the frame-size guard.
DataConnectionFactory registers 'MxGateway' -> MxGatewayDataConnection over the
real client; AddDataConnectionLayer binds MxGatewayGlobalOptions; DeploymentManager
FlattenConnectionConfig gains an MxGateway arm using the typed serializer. Factory
test confirms Create("MxGateway") returns the adapter.
Implements IDataConnection + IBrowsableDataConnection over the IMxGatewayClient
seam: connect/disconnect with once-only Disconnected guard + background event
loop, subscribe/unsubscribe with tag routing, read/write batch with per-tag
error classification, WriteBatchAndWait, and Galaxy browse mapping. Covers plan
Tasks 6-10. Full unit coverage via FakeMxGatewayClient (12 tests).
Adds a Test Bindings button to the Connection Bindings table on the Configure
Instance page that opens a modal showing the live current value of every bound
attribute. Reuses the routing path that the OPC UA tag browser landed on:
Central: TestBindingsDialog → IBindingTester → CommunicationService
→ ReadTagValuesCommand → SiteEnvelope (Ask)
Site: SiteCommunicationActor → DeploymentManagerActor singleton
→ DataConnectionManagerActor → child DataConnectionActor
→ _adapter.ReadBatchAsync
Split mirrors the browse handler:
• Manager owns ConnectionNotFound (only it sees the per-site connection set).
• Child owns ConnectionNotConnected (pre-call status check, never stash —
read is interactive design-time), Timeout (OperationCanceledException),
ServerError (any other exception). Per-tag failures from ReadBatchAsync
become failure TagReadOutcomes without aborting the batch.
CentralUI:
• IBindingTester / BindingTester — Design-role guard via HasClaim against
JwtTokenService.RoleClaimType (not IsInRole — see c1e16cf), typed
transport-failure translation.
• TestBindingsDialog — ShowAsync(siteId, rows, instanceLabel) method-arg
pattern (no Razor parameter race; see 2c138b6), groups rows by connection
and issues one ReadAsync per connection in parallel, per-row error subline
+ per-connection banner, Refresh button re-issues the reads.
• InstanceConfigure.razor — Test Bindings button next to Save Bindings,
disabled when no testable rows. OPC UA only today (other protocols have
no ReadTagValuesCommand wiring yet).
Tests:
• Commons: ReadTagValuesCommand discovered by ManagementCommandRegistry.
• DataConnectionLayer: unknown connection → ConnectionNotFound,
not-connected adapter → ConnectionNotConnected (ReadBatchAsync NOT called),
success-path mapping (Good/Bad + per-tag error), cancellation → Timeout.
• CentralUI: register IBindingTester (and the previously-missing
IOpcUaBrowseService) on the existing InstanceConfigureAuditDrillinTests
Bunit container so the page renders cleanly with the new dialog.
Joseph Doherty