Remediates RESILIENCE-DISPATCH-GAP (surfaced by the 07/C-1 analyzer): the Phase 6.1
CapabilityInvoker resilience pipeline (retry / breaker / bulkhead / telemetry) was
constructed only in tests — the production dispatch layer called driver-capability
methods directly, bypassing it entirely.
The filed plan (thread CapabilityInvoker directly into DriverInstanceActor) is
INFEASIBLE: Runtime is deliberately Polly-free (references Core.Abstractions, not
Core — the same boundary IDriverFactory documents). So this introduces a seam,
mirroring IDriverFactory exactly:
- Core.Abstractions: IDriverCapabilityInvoker (+ NullDriverCapabilityInvoker
pass-through) and IDriverCapabilityInvokerFactory (+ null factory).
- Core: CapabilityInvoker now implements the interface; new
DriverCapabilityInvokerFactory builds a per-instance invoker over the
process-singleton pipeline builder + status tracker + tier resolver.
- Runtime: DriverInstanceActor takes an IDriverCapabilityInvoker (default =
pass-through) and routes all 6 dispatch sites (write, alarm-ack, subscribe,
unsubscribe, alarm-subscribe, discover) through it; per-host key resolved via
IPerCallHostResolver for single-ref calls, driver-instance key for bulk calls.
DriverHostActor builds + injects the real invoker per spawned driver.
- Host: DriverFactoryBootstrap registers the tracker, pipeline builder, and
concrete factory (needs DriverFactoryRegistry.GetTier). Runtime SCE resolves the
factory from DI like IDriverFactory; pass-through on nodes without it bound.
- Analyzer: IDriverCapabilityInvoker.ExecuteAsync/ExecuteWriteAsync are now
recognized wrapper homes (interface-typed invoker calls must suppress OTOPCUA0001
too). All 6 RESILIENCE-DISPATCH-GAP pragmas removed — the analyzer (an error in
Runtime) is now the standing regression guard.
Also adds retry / breaker-open / breaker-close LOGGING to the pipeline builder —
the operator-facing observability surface (the Admin /hosts reader, Phase 6.1
Stream E.2/E.3, was never built, so the pipeline otherwise runs silently).
Scope notes:
- Tier-DEFAULT policy only: DriverInstanceSpec (the deploy artifact) does not carry
the per-instance ResilienceConfig JSON, so overrides aren't applied yet (tracked
follow-up: plumb ResilienceConfig through the composer/artifact).
- GenericDriverNodeManager's call is NOT wired: that class is test-only scaffolding
(zero production references — the Server has its own address-space path); its
pragma is re-annotated accordingly, not left as a "tracked gap".
Verification (unit + analyzer; live behavioral gate still pending — see FOLLOWUP-10):
- Negative control: unwrapping any site fails the Runtime build with OTOPCUA0001.
- New Runtime guard (recording invoker): write routes via ExecuteWriteAsync with the
IPerCallHostResolver host; subscribe via ExecuteAsync — proves runtime routing.
- New analyzer test: the interface is a valid wrapper home.
- New pipeline-builder test: retry events are logged.
- Full solution builds clean (0 errors); Runtime.Tests 357, Core.Tests 238,
Analyzers.Tests 32 all green; pass-through default keeps existing dispatch tests
byte-for-byte unchanged.
Inject the custom UnwrappedCapabilityCallAnalyzer as an OutputItemType=Analyzer
ProjectReference from Directory.Build.props (excluding the analyzer + its test
project) so OTOPCUA0001 runs on every src/ and tests/ compilation — it previously
enforced its CapabilityInvoker-wrapping rule against nothing but its own 31 unit
tests (the 'built-but-never-wired' failure mode).
Triage of the ~280 surfaced hits, three categories:
1. RESILIENCE-DISPATCH-GAP (7 sites, DriverInstanceActor x6 + GenericDriverNodeManager
x1): a REAL, previously-untracked gap the analyzer caught on first wiring — the
Phase 6.1 CapabilityInvoker resilience pipeline (retry/breaker/bulkhead/telemetry)
is constructed ONLY in tests and was never wired into the production dispatch
layer. Scoped per-site #pragma with a greppable RESILIENCE-DISPATCH-GAP marker
explicitly noting these are tracked-but-not-intentional, pending the dispatch-wiring
remediation (filed as a follow-up). Keeps the analyzer live everywhere else in
those projects so a NEW unwrapped call still fails the build.
2. Driver-INTERNAL self-calls (3 sites, AbCipAlarmProjection x2 + S7Driver x1):
a driver's own poll/ack path calling its own capability method. The invoker wraps
the driver from the dispatch layer OUTWARD; a driver re-wrapping its own internal
calls would double-wrap. Genuinely intentional — scoped #pragma with that rationale.
3. Wire-level test suites + manual-testing CLIs (12 projects): invoke drivers directly
by design — the analyzer's own documented intentional case. Project-level NoWarn
with a comment.
Verified: full solution build green, 0 OTOPCUA0001 hits; analyzer's 31 tests pass;
negative control — dropping one dispatch-gap pragma re-fires OTOPCUA0001 and fails
the Runtime build, proving the analyzer is genuinely live tree-wide, not disabled.
Adds <summary>/<param>/<returns>/<inheritdoc> where missing and removes
project bookkeeping IDs (task/tracking refs) from shipped code comments,
so the docs read cleanly and CommentChecker is quiet except for known
false positives (PLC/protocol terms, event/IEqualityComparer inheritdoc).
Doc/comment-only; no logic changed; solution builds clean.
The HistorianGateway driver is now the sole historian read/write+alarm backend, so the
Wonderware sidecar projects are dead code. Removes the 5 Wonderware projects (driver,
.Client, .Client.Contracts, + their 2 test projects) from the solution and tree, and fully
retires the vestigial 'Historian.Wonderware' driver type (UI/probe-only; it had no driver
factory): the Host probe registration, the AdminUI driver-config surface (driver page,
tag-config editor/model/validator entry, address picker/builder, driver-type catalog +
dropdown + edit-router entries), and their tests. Prunes the now-unused Wonderware
connection fields (Host/Port/UseTls/ServerCertThumbprint/SharedSecret) from
AlarmHistorianOptions (keeping Enabled + the SQLite store-and-forward knobs) and refreshes
the stale XML docs that named Wonderware as the production backend.
Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
GalaxyDriver's StreamAlarms feed is session-less and survives an in-place
reconnect, so DriverInstanceActor re-subscribed on every Connected re-entry
(after dropping its own cached handle without an Unsubscribe — sync teardown).
The re-subscribe was additive: _alarmSubscriptions.Add grew the list by one
untracked handle per reconnect cycle — a slow unbounded leak. Functionally
harmless (the gate is Count>0 and OnAlarmFeedTransition only reads [0], firing
once regardless), but it accumulated forever.
Fix: SubscribeAlarmsAsync clears the set before adding, collapsing to a single
live handle (under the existing _alarmHandlersLock, atomic w.r.t. the fan-out
reader). There is exactly one consumer per driver instance (factory-per-actor
lifecycle), so replacing the set with the latest handle is faithful. Chosen
over making the actor's sync DetachAlarmSource call UnsubscribeAlarmsAsync
async/fire-and-forget — disproportionate for a minor leak.
Regression test Re_subscribe_collapses_to_a_single_handle_no_accumulation
(TDD-verified: FAILS without the Clear — releasing the latest handle leaves
the feed open because stale handles remain; PASSES with the fix). Galaxy tests
263 pass / 3 skip; Runtime native-alarm 24 pass. Code-reviewed (approved).
- Correct the misleading DetachAlarmSource comment: a session-less feed (Galaxy) is NOT
torn down on an in-place reconnect, so re-subscribe is additive (harmless; gate reads [0]).
- Add trace-only SubscribeAlarms drop handlers in Connecting/Reconnecting (symmetry with
NativeAlarmRaised) so a self-tell overtaken by a queued disconnect doesn't dead-letter.
- Document the deliberate no-unsubscribe-on-empty asymmetry vs the value path.
Behavior-neutral for the un-gate path. Minor handle-accumulation leak tracked as follow-up.
Phase B native alarms never fired end-to-end: GalaxyDriver suppresses OnAlarmEvent until
an alarm subscription exists (_alarmSubscriptions.Count > 0), but the runtime only attached
the OnAlarmEvent handler and never called SubscribeAlarmsAsync — so the central feed stayed
gated and no transition reached the Part 9 condition / /alerts. Unit tests passed because
they inject through the IAlarmSource seam directly; the deferred live /run surfaced it.
DriverHostActor computes per-driver alarm refs (alarm-bearing tags' FullNames) and hands them
via SetDesiredSubscriptions; DriverInstanceActor calls SubscribeAlarmsAsync for IAlarmSource
drivers on Connected entry and whenever alarm refs are pushed while Connected (the deploy path),
idempotent via a cached handle reset on detach so reconnect re-subscribes.
A DriverInstanceActor stuck Reconnecting/Connecting now adopts a config delivered via ApplyDelta and
re-initialises with it, instead of dead-lettering and retrying the stale config forever. A monotonic
init generation supersedes the in-flight init so the corrected config always wins.
Materialised SystemPlatform/Galaxy variables previously stayed
BadWaitingForInitialData because nothing told the driver to subscribe
(OpcUaPublishActor TODO 'on a future SubscribeBulk pass') and published
values were only forwarded to the VirtualTag mux, never the OPC UA sink.
DriverHostActor now, after each apply, groups the deployment's galaxy tag
MXAccess refs by driver and sends DriverInstanceActor.SetDesiredSubscriptions;
the actor retains the set and (re)subscribes on every Connected entry, so
values resume after reconnects/redeploys (closes the F8b/#113 gap). Published
values are also forwarded to OpcUaPublishActor as AttributeValueUpdate
(NodeId == galaxy MxAccessRef) so the materialised variable shows live data.
Verified live in docker-dev: galaxy TestMachine_001 tags go Good with a
changing TestChangingInt. +1 unit test.
- Topic-name drift fix: DriverHealthChanged.TopicName and
DriverControlTopic.Name now live on the message contracts in
Commons. AkkaDriverHealthPublisher, DriverStatusSignalRBridge,
DriverHostActor, and AdminOperationsActor all delegate to the
single constant so a rename can't silently desynchronise
publisher and subscriber.
- DriverStatusPanel._opResultClearTimer switched from
System.Timers.Timer to System.Threading.Timer + awaited
DisposeAsync. Prevents an in-flight 8s clear-callback from
invoking StateHasChanged on a component whose hub has already
been released.
- PublishHealthSnapshot deduplicates against the last published
(state, lastSuccess, lastError, errorCount) fingerprint. The
30s heartbeat no longer floods the SignalR layer with identical
Healthy snapshots — newly-joined clients still warm up via the
snapshot store on JoinDriver.
- DriverInstanceSpec carries ClusterId from the deployment artifact;
DriverHostActor threads the real cluster identity into
DriverInstanceActor instead of the local NodeId. Old pre-PR
artifacts without a ClusterId field fall back to the NodeId so
in-flight deployments keep working.
- DriverHostActor.ChildEntry holds the full DriverInstanceSpec
(was only carrying DriverType + LastConfigJson). Restart respawns
preserve RowId, Name, Enabled, ClusterId — no placeholder values.
- Drop the unnecessary _faultLock on DriverInstanceActor — every
read/write site runs inside an Akka message handler which is
single-threaded per actor instance.
- DriverStatusPanel.DisposeAsync awaits Timer.DisposeAsync so an
in-flight 5s tick can't invoke StateHasChanged on a component
whose hub has already been torn down.
- RestartDriver / ReconnectDriver messages + AdminOperationsActor
handlers (broadcast via driver-control DPS topic; audited via
ConfigEdits).
- DriverHostActor subscribes to driver-control; locates the
matching child DriverInstanceActor and stops+respawns it
(Restart) or sends it a ForceReconnect internal message
(Reconnect — re-enters Reconnecting state without full stop).
DriverInstanceSpec constructor call uses named args to handle
the full 6-parameter signature.
- New DriverOperator authorization policy mapped to DriverOperator
or FleetAdmin role; documented in docs/security.md. Map LDAP
group via GroupToRole (e.g. "ot-driver-operator": "DriverOperator").
- DriverStatusPanel renders Reconnect + Restart buttons when the
user holds the DriverOperator policy (hidden otherwise). Restart
requires an in-page Razor confirm block (no JS confirm, keeps
SignalR event loop unblocked). Both buttons show a spinner and
are disabled during in-flight; result chip auto-clears after 8s.
Username sourced from AuthenticationStateProvider.
Reconnect resolves to "ForceReconnect" (re-enter Reconnecting,
not full stop+respawn) — transport drops and retries while actor
and in-memory state are preserved. All DriverInstanceActor states
handle ForceReconnect safely (no-op when already in transition).
- IDriverHealthPublisher in Core.Abstractions + NullDriverHealthPublisher
no-op for tests/dev-stub paths.
- AkkaDriverHealthPublisher in Runtime forwards to the cluster-wide
`driver-health` DPS topic.
- DriverInstanceActor instrumented to publish snapshots on every
observable state change + a periodic 30s heartbeat so the AdminUI
snapshot store warms up for newly-joined SignalR clients.
- Sliding 5-minute Faulted-count tracked per actor via Queue<DateTime>.
- DriverHostActor.SpawnChild threads clusterId (_localNode.Value) and
the health publisher down to every DriverInstanceActor child.
- ServiceCollectionExtensions.AddOtOpcUaRuntime registers
AkkaDriverHealthPublisher as IDriverHealthPublisher singleton.
Adds <summary>, <param>, <typeparam>, and <inheritdoc/> tags to public
members surfaced by commentchecker — resolves 5,847 of 5,869 issues
(99.6%) across three /fixdocs passes.
User confirmed the mxaccessgw client (Galaxy driver) doesn't need Windows
— only the gateway worker has that constraint. This wires the Galaxy
driver into the docker-dev fleet:
- docker-compose.yml: GALAXY_MXGW_API_KEY env var on every host service
(admin nodes harmlessly ignore it; driver-role nodes pick it up when
the seeded DriverInstance resolves ApiKeySecretRef=env:GALAXY_MXGW_API_KEY).
Default value matches the key the operator provided; override via shell
env (GALAXY_MXGW_API_KEY=... docker compose up -d) to rotate without
editing compose.
- seed-clusters.sql: now creates a SystemPlatform Namespace
(MAIN-galaxy, urn:zb:docker-dev:galaxy) plus a GalaxyMxGateway
DriverInstance (MAIN-galaxy-mxgw) in the MAIN cluster pointing at
http://10.100.0.48:5120 with UseTls=false. Idempotent via IF NOT EXISTS.
- DriverInstanceActor.ShouldStub: clarified the doc comment — only the
legacy "Galaxy" type name and "Historian.Wonderware" are Windows-only;
the v2 "GalaxyMxGateway" driver is .NET 10 cross-platform (gRPC to an
external gateway) and is NOT stubbed.
- README: documents the final operator step — sign in, click "Deploy
current configuration" on /deployments to materialise the seeded
Galaxy driver into a running gRPC connection. Raw DriverInstance rows
don't spawn drivers on their own; the v2 lifecycle requires a sealed
Deployment first.
OtOpcUaTelemetry (Commons/Observability) centralizes the project's Meter
+ ActivitySource so all instrumentation points emit through a single
named surface. Counters cover the hot paths:
otopcua.deploy.applied (outcome=ack|reject)
otopcua.deploy.apply.duration (s, histogram)
otopcua.driver.lifecycle (event=spawn|spawn_stub|stop|fault)
otopcua.virtualtag.eval (outcome=ok|fail|skip)
otopcua.scriptedalarm.transition (state=activated|acknowledged|cleared)
otopcua.opcua.sink.write (kind=value|alarm|rebuild)
otopcua.redundancy.service_level_change (level=byte)
Plus two ActivitySource spans:
otopcua.deploy.apply wraps DriverHostActor.ApplyAndAck
otopcua.opcua.address_space_rebuild wraps OpcUaPublishActor.HandleRebuild
Instruments are no-op until a listener attaches, so tests + dev hosts
pay nothing for unread telemetry.
Host Program.cs gains AddOtOpcUaObservability() (binds the OtOpcUa Meter
+ ActivitySource to OpenTelemetry, attaches a Prometheus exporter) and
MapOtOpcUaMetrics() (mounts /metrics scrape endpoint). Driver-side
internals + ASP.NET request metrics deliberately stay off — the scrape
payload is scoped to OtOpcUa signals only.
Tests use MeterListener + ActivityListener to verify
VirtualTagActor.eval, OpcUaPublishActor.AttributeValueUpdate, and
RebuildAddressSpace actually emit on the central instruments. Runtime
suite is 72 / 72 green (+3).
Closes#105. Path A (F13b/c/d) complete; next batch options: #85 UNS
folder hierarchy in SDK, or F8b/F9b production engine bindings.
Three pieces landed in one batch, closing F7-residual + Host DI #106:
Runtime/DriverInstanceActor:
- Subscribe / Unsubscribe message contracts; the Connected state
handles them via IDriver.ISubscribable. On every OnDataChange
event the actor publishes AttributeValuePublished to its parent
(DriverHostActor → OpcUaPublishActor). OPC UA StatusCode is
mapped to the 3-state OpcUaQuality enum via severity bits
(00=Good, 01=Uncertain, 10/11=Bad).
- DetachSubscription tears the handler off the driver on
DisconnectObserved, Unsubscribe, and PostStop so a stale handler
never pushes to a dead actor.
- WriteAttribute now dispatches IWritable.WriteAsync (batch of one)
with a 5s CancellationTokenSource; status-code propagated to
WriteAttributeResult on non-Good results.
Host:
- New ProjectReferences to Core + every cross-platform driver
assembly (AbCip/AbLegacy/FOCAS/Galaxy/Modbus/S7/TwinCAT).
Galaxy is net10 (gRPC client to mxaccessgw); the COM-bound net48
Wonderware Historian driver stays out of the Host's reference
closure — its .Client gRPC wrapper is what binds for historian
needs.
- New DriverFactoryBootstrap.AddOtOpcUaDriverFactories() registers
a singleton DriverFactoryRegistry, invokes each driver's
Register(registry, loggerFactory), and binds IDriverFactory to
DriverFactoryRegistryAdapter. Replaces the F7 NullDriverFactory
default so deploys actually materialise real IDriver instances
on driver-role nodes. ShouldStub() still gates per-platform
behaviour at spawn time.
- Program.cs wires AddOtOpcUaDriverFactories() before AddAkka so
the runtime extension can resolve IDriverFactory from DI.
Tests: Runtime 46 -> 52 (+6):
- Write returns success when StatusCode = Good
- Write propagates non-Good status code in failure Reason
- Subscribe forwards OnDataChange to parent as AttributeValuePublished
- Quality translation: Uncertain (0x40...) and Bad (0x80...)
- Subscribe against non-ISubscribable returns failure
- DisconnectObserved detaches handler so late events are dropped
All 6 v2 test suites green: 152 tests passing.
Closes F7. F7-residual sub-tasks #110 (subscribe) and #111 (write)
both shipped. Host DI binding #106 shipped.
Joseph Doherty