An in-flight subscribe launched against adapter generation N can take up to
SeedOverallTimeout (120s) to answer. If the adapter is swapped in that window
(failover, or the row-97 in-place reconfigure), the completion was applied
unconditionally: it stored a dead session's handle under the tag key, so the
LIVE completion for the same tag read as a duplicate and had its working
monitored item released (tag resolved forever, no data); it re-stamped the old
endpoint's seed values with the CURRENT generation, bypassing the stale-data
guard; and a connection-level failure row from the dead adapter knocked the
healthy new connection back into Reconnecting. The alarm twin released the new
live alarm feed the same way.
Both completions now carry the generation AND the adapter instance they ran
against, and both handlers drop a completion whose generation no longer
matches - no handle store, no seed application, no per-tag or connection-level
failure processing, no duplicate-release logic. Handles the dropped completion
did create are released best-effort against the CAPTURED adapter (handles are
adapter-scoped; MxGateway's are plain ints that collide across sessions), and
the existing release sites in both handlers were switched to the message's
adapter for the same reason. The in-flight markers are deliberately left alone
(ReSubscribeAll/ReSubscribeAllAlarms clear them wholesale and may already have
re-registered the tag/source), and the requester gets a failure reply so its
retry timer re-sends.
Tests: DataConnectionStaleCompletionTests - stale handle not stored / live item
not released, release lands on the old adapter only, seed values not delivered,
stale connection-level failure does not re-enter Reconnecting, stale alarm
completion does not release the live feed. All five fail against the unfixed
handlers.
`DataConnectionManagerActor.HandleCreateConnection` is create-only (warns and
returns for a name it already holds), so `EnsureDclConnection`'s documented
"a changed config re-issues the create so the DCL adopts it" was never true on
ANY node: a corrected endpoint/credential/backup/failover-count took effect only
when the connection was next built from scratch (node restart, singleton
failover, remove-then-re-add), while the DM's hash cache recorded it as applied.
Rows 89/90 made that honest with a Warning; this closes it.
New `ReconfigureConnectionCommand` (Commons, same shape as the create):
- `DeploymentManagerActor.EnsureDclConnection` now has three outcomes off its one
per-name hash slot — unchanged -> nothing at all, new -> create, CHANGED ->
reconfigure — and logs "Reconfigured DCL connection {Connection}
(protocol={Protocol}) in place" at Information in place of the old Warning. The
winner-resolution / arbitration / tombstone logic is untouched; only what
happens after a change is detected changed.
- `DataConnectionManagerActor` FORWARDS the command to the owning connection
actor, falling back to a create for a name it does not hold, so the command is
safe whatever the manager's state.
- `DataConnectionActor` handles it in ALL THREE states and never stashes it — a
connection pointed at a wrong or dead endpoint lives in Connecting/Reconnecting,
which is exactly when a correction arrives; stashed it would wait on the broken
configuration forever. Handling replaces protocol/primary/backup/failover-count,
re-arms endpoint selection to Primary (the new config's primary is the intent,
whatever the old config had failed over to), and calls the new shared
`SwapAdapter` helper — now used by both failover sites too, so the swap exists
once: unhook + fire-and-forget dispose -> _factory.Create -> rewire -> bump
_adapterGeneration LAST. `BecomeReconnecting(afterReconfigure: true)` then
reconnects immediately (StartSingleTimer replaces any armed reconnect, so no
racing attempts) with the unstable-disconnect accounting and failover check
suppressed — the link was dropped on purpose, and counting it as an unstable
cycle could trip a failover onto the NEW backup before the new primary had been
tried once. ReSubscribeAll/ReSubscribeAllAlarms then re-issue everything from
the untouched subscription registries. Idempotent: a reconfigure carrying the
configuration already in force is a no-op.
Stop-and-recreate was rejected as destructive: subscriptions live inside
DataConnectionActor and InstanceActor cancels its subscribe-retry timer on first
success, so a rebuilt actor would have no subscribers and nothing would ever
re-subscribe.
Tests: new DataConnectionReconfigureTests (7; 5 fail / 2 pass against a
behavioural revert). DclConnectionArbitrationTests gains two row-97 tests and its
CDC-arrival test now asserts the second push is a reconfigure; its ping-pong
guard also asserts zero reconfigures, since a flip would now show up there.
SiteRuntime-010 renamed to ..._ReissuesReconfigureCommand. 3 fail against a
reverted EnsureDclConnection. DataConnectionLayer.Tests 327/327,
SiteRuntime.Tests 643/643, solution build 0 warnings.
Docs: Component-DataConnectionLayer.md gains an "In-place reconfiguration"
section under Endpoint Redundancy; Component-SiteRuntime.md, and the
docs/components/ mirrors, updated; the arbitration decision memo gains an
appended §9 postscript resolving its residual 1 (residuals 2 and 3 stay open).
Closes arch-review remediation residual #1 (DCL unsubscribe-during-reconnect
count staleness).
DataConnectionActor tracked TotalSubscribedTags/ResolvedTags as two int fields
incremented and decremented at five independent sites. ReSubscribeAll clears the
very maps those decrements key off (_subscriptionIds, _unresolvedTags) while
deliberately preserving _subscriptionsByInstance, so an unsubscribe landing
inside a reconnect window matched NEITHER decrement branch: the total leaked +1
per subscribe/reconnect/unsubscribe churn cycle, permanently and cumulatively.
The 37f13e2e discard gate stopped the orphan-handle half of that race; it could
not stop the counters drifting, because they were state of their own.
Both counts are now DERIVED at report time from the authoritative per-tag
collections, which makes the drift unrepresentable rather than merely guarded:
total = _instancesByTag.Count (the per-tag counted set the residual
called for — distinct tags with at least
one subscribing instance)
resolved = _subscriptionIds.Count (tags for which the adapter holds a handle)
Two semantic corrections fall out of the derivation:
- A tag whose subscribe failed at CONNECTION level now counts toward the total.
It was excluded before, yet the reconnect re-subscribe re-issued it from
_subscriptionsByInstance and booked it as resolved — resolved above total, and
a total driven negative by the eventual unsubscribe.
- _tagSubscriberCount is deleted. It duplicated _instancesByTag exactly, so
HandleUnsubscribe's last-subscriber test is now "did UnindexTag drop the key?"
— still O(1), with no parallel count that can disagree about when a handle is
released. The subscribe-success promotion split (fresh vs. unresolved→resolved)
also goes: it existed only to pick which scalar to bump; set sizes get
DataConnectionLayer-020's double-count cases right for free.
Behavior is otherwise unchanged — same logging, same handle release, same
unresolved-tag probing, same in-flight-unsubscribe discard semantics (the long
comment block there is updated for the mechanics that changed).
Tests: five TagResolutionCounts_* cases in DataConnectionActorBatchTests
covering the churn repro (3 cycles), a shared tag losing one instance mid
reconnect, connection-level failure then recovery, plain subscribe/unsubscribe
cycles, and a completed reconnect re-subscribe. Verified failing against the
pre-fix actor (churn: total 1 not 0; connection-level: total 0 not 1) and
passing after. Full DCL suite 319/319; solution builds with 0 warnings.
Docs: Component-DataConnectionLayer.md health-reporting section describes the
derived counts; residuals register item 1 marked RESOLVED.
Expanding a Galaxy object in the tag picker hung on "loading…": the browse
reply inlined every child's full attribute set (~152 KB), exceeding Akka's
128 KB remote frame, and remoting silently discarded the oversized reply.
Browse path (DataConnectionLayer):
- RealMxGatewayClient: navigation now uses BrowseChildren(include_attributes=
false) — child objects only — and an object's own attributes load lazily via
DiscoverHierarchy(root, max_depth=0) when it's expanded. Payload drops from
~152 KB/level to a few KB. Seam contract unchanged.
- DataConnectionActor.CapBrowseChildren: protocol-agnostic byte-budget cap
(~100 KB) on every BrowseNodeResult before it crosses the site→central
frame, OR-ing the adapter's own Truncated flag. Byte budget, not a count —
the only bound that holds regardless of NodeId/attribute-name length.
- RealOpcUaClient: requestedMaxReferencesPerNode 1000 → 500 to narrow the
window before the byte budget applies.
- Graceful gRPC Unimplemented handling → NotSupportedException →
BrowseFailureKind.NotBrowsable with an actionable message (older gateway
builds lacking BrowseChildren).
Picker UI (CentralUI):
- NodeBrowserDialog: modal-lg → modal-xl; new scoped .razor.css caps the tree
at 55vh with its own scrollbar so manual entry + Select/Cancel stay visible.
- Protocol-agnostic failure messages (was hardcoded "OPC UA …"); renamed the
leftover opcua-browser-tree class to node-browser-tree.
Tests: new frame-budget cap test + NotSupported=>NotBrowsable mapping test;
DCL suite 88/88. Doc: Component-DataConnectionLayer.md records the lazy
attribute-light browse and the frame-size guard.
Adds a Test Bindings button to the Connection Bindings table on the Configure
Instance page that opens a modal showing the live current value of every bound
attribute. Reuses the routing path that the OPC UA tag browser landed on:
Central: TestBindingsDialog → IBindingTester → CommunicationService
→ ReadTagValuesCommand → SiteEnvelope (Ask)
Site: SiteCommunicationActor → DeploymentManagerActor singleton
→ DataConnectionManagerActor → child DataConnectionActor
→ _adapter.ReadBatchAsync
Split mirrors the browse handler:
• Manager owns ConnectionNotFound (only it sees the per-site connection set).
• Child owns ConnectionNotConnected (pre-call status check, never stash —
read is interactive design-time), Timeout (OperationCanceledException),
ServerError (any other exception). Per-tag failures from ReadBatchAsync
become failure TagReadOutcomes without aborting the batch.
CentralUI:
• IBindingTester / BindingTester — Design-role guard via HasClaim against
JwtTokenService.RoleClaimType (not IsInRole — see c1e16cf), typed
transport-failure translation.
• TestBindingsDialog — ShowAsync(siteId, rows, instanceLabel) method-arg
pattern (no Razor parameter race; see 2c138b6), groups rows by connection
and issues one ReadAsync per connection in parallel, per-row error subline
+ per-connection banner, Refresh button re-issues the reads.
• InstanceConfigure.razor — Test Bindings button next to Save Bindings,
disabled when no testable rows. OPC UA only today (other protocols have
no ReadTagValuesCommand wiring yet).
Tests:
• Commons: ReadTagValuesCommand discovered by ManagementCommandRegistry.
• DataConnectionLayer: unknown connection → ConnectionNotFound,
not-connected adapter → ConnectionNotConnected (ReadBatchAsync NOT called),
success-path mapping (Good/Bad + per-tag error), cancellation → Timeout.
• CentralUI: register IBindingTester (and the previously-missing
IOpcUaBrowseService) on the existing InstanceConfigureAuditDrillinTests
Bunit container so the page renders cleanly with the new dialog.
- BrowseOpcUaNodeCommand: int DataConnectionId -> string ConnectionName
(site DataConnectionManagerActor indexes children by name; CentralUI
already has the connection name in scope via the dropdown — no extra
plumbing across the trust boundary).
- IOpcUaBrowseService / OpcUaBrowseService: parameter renamed accordingly.
- OpcUaBrowserDialog: collapse the duplicate ConnectionName parameters
(display label and routing key are the same string).
- Task 10: DataConnectionManagerActor forwards BrowseOpcUaNodeCommand to
its child by name (owns ConnectionNotFound); DataConnectionActor adds
the receive across all three lifecycle states (Connecting / Connected
/ Reconnecting) and maps adapter outcomes to BrowseFailureKind
(NotBrowsable / ConnectionNotConnected / Timeout / ServerError).
- Task 17: SetFailure in OpcUaBrowserDialog implements the full
BrowseFailureKind switch with friendly UI messages.
- Tests: DataConnectionManagerBrowseHandlerTests covers ConnectionNotFound,
NotBrowsable, success, and ConnectionNotConnectedException paths.
Joseph Doherty