The DriverInstance.ResilienceConfig column was authored in AdminUI, persisted to the
entity, and serialized into the deployment artifact by ConfigComposer — but the runtime
read path dropped it: DriverInstanceSpec didn't carry it and the invoker factory always
passed null, so every driver got tier defaults regardless of its configured overrides
(a silent dead-config gap — #10's residual sub-finding).
Read-path plumbing (write side was already complete):
- DriverInstanceSpec gains ResilienceConfig; DeploymentArtifact.TryReadSpec reads the column.
- IDriverCapabilityInvokerFactory.Create takes resilienceConfigJson; DriverHostActor.SpawnChild
threads spec.ResilienceConfig; the concrete factory parses it (ParseOrDefaults, layering on the
tier), logs any parse diagnostic (never throws), and builds the invoker with the merged options.
- Invalidate-on-change: the pipeline cache keys on (instance, host, capability) and ignores options
on a hit, so Create() now Invalidate()s the instance's cached pipelines first (no-op on first
spawn) — a respawn with changed options rebuilds them.
- DriverSpawnPlanner treats a ResilienceConfig change as a stop+respawn (the invoker/options are
bound to the child at spawn); a pure DriverConfig change stays an in-place delta (no reconnect).
- Host DI passes a logger to the factory for the parse diagnostic.
Verification (deterministic): factory Create applies a retryCount:0 override to actual execution
(control test proves tier default retries), invalidates the instance's cache on re-create (scoped —
sibling survives), malformed config logs+falls-back; planner respawns on ResilienceConfig change
(incl null→json) and stays delta on a pure config change; artifact parse carries/omits the column.
Core.Tests 243 (+5), Runtime.Tests 363 (+5), Host builds clean.
Remediates RESILIENCE-DISPATCH-GAP (surfaced by the 07/C-1 analyzer): the Phase 6.1
CapabilityInvoker resilience pipeline (retry / breaker / bulkhead / telemetry) was
constructed only in tests — the production dispatch layer called driver-capability
methods directly, bypassing it entirely.
The filed plan (thread CapabilityInvoker directly into DriverInstanceActor) is
INFEASIBLE: Runtime is deliberately Polly-free (references Core.Abstractions, not
Core — the same boundary IDriverFactory documents). So this introduces a seam,
mirroring IDriverFactory exactly:
- Core.Abstractions: IDriverCapabilityInvoker (+ NullDriverCapabilityInvoker
pass-through) and IDriverCapabilityInvokerFactory (+ null factory).
- Core: CapabilityInvoker now implements the interface; new
DriverCapabilityInvokerFactory builds a per-instance invoker over the
process-singleton pipeline builder + status tracker + tier resolver.
- Runtime: DriverInstanceActor takes an IDriverCapabilityInvoker (default =
pass-through) and routes all 6 dispatch sites (write, alarm-ack, subscribe,
unsubscribe, alarm-subscribe, discover) through it; per-host key resolved via
IPerCallHostResolver for single-ref calls, driver-instance key for bulk calls.
DriverHostActor builds + injects the real invoker per spawned driver.
- Host: DriverFactoryBootstrap registers the tracker, pipeline builder, and
concrete factory (needs DriverFactoryRegistry.GetTier). Runtime SCE resolves the
factory from DI like IDriverFactory; pass-through on nodes without it bound.
- Analyzer: IDriverCapabilityInvoker.ExecuteAsync/ExecuteWriteAsync are now
recognized wrapper homes (interface-typed invoker calls must suppress OTOPCUA0001
too). All 6 RESILIENCE-DISPATCH-GAP pragmas removed — the analyzer (an error in
Runtime) is now the standing regression guard.
Also adds retry / breaker-open / breaker-close LOGGING to the pipeline builder —
the operator-facing observability surface (the Admin /hosts reader, Phase 6.1
Stream E.2/E.3, was never built, so the pipeline otherwise runs silently).
Scope notes:
- Tier-DEFAULT policy only: DriverInstanceSpec (the deploy artifact) does not carry
the per-instance ResilienceConfig JSON, so overrides aren't applied yet (tracked
follow-up: plumb ResilienceConfig through the composer/artifact).
- GenericDriverNodeManager's call is NOT wired: that class is test-only scaffolding
(zero production references — the Server has its own address-space path); its
pragma is re-annotated accordingly, not left as a "tracked gap".
Verification (unit + analyzer; live behavioral gate still pending — see FOLLOWUP-10):
- Negative control: unwrapping any site fails the Runtime build with OTOPCUA0001.
- New Runtime guard (recording invoker): write routes via ExecuteWriteAsync with the
IPerCallHostResolver host; subscribe via ExecuteAsync — proves runtime routing.
- New analyzer test: the interface is a valid wrapper home.
- New pipeline-builder test: retry events are logged.
- Full solution builds clean (0 errors); Runtime.Tests 357, Core.Tests 238,
Analyzers.Tests 32 all green; pass-through default keeps existing dispatch tests
byte-for-byte unchanged.
Adds <summary>/<param>/<returns>/<inheritdoc> where missing and removes
project bookkeeping IDs (task/tracking refs) from shipped code comments,
so the docs read cleanly and CommentChecker is quiet except for known
false positives (PLC/protocol terms, event/IEqualityComparer inheritdoc).
Doc/comment-only; no logic changed; solution builds clean.
The OPC UA address-space build pipeline was named after a v2-roadmap
milestone number rather than its domain. Rename the family to describe
what it does (build/diff/apply the OPC UA address space):
Phase7Composer -> AddressSpaceComposer
Phase7CompositionResult -> AddressSpaceComposition
Phase7Planner -> AddressSpacePlanner
Phase7Plan -> AddressSpacePlan
Phase7Applier -> AddressSpaceApplier
Phase7ApplyOutcome -> AddressSpaceApplyOutcome
The 9 Phase7*Tests suites follow suit; Phase7ScriptingEntitiesTests ->
ScriptingEntitiesTests (it tests the scripting migration, not the
pipeline). Log-message prefixes move to the new class names.
Pure mechanical rename, no behavioral change. EF migration classes/IDs
(AddPhase7ScriptingTables, ExtendComputeGenerationDiffWithPhase7) are
immutable and left untouched, as are historical design docs.
Build clean; OpcUaServer 261/261, Runtime 272/272, ScriptingEntities
12/12 green.
HandleRestartDriver stopped + respawned the child within one synchronous
message handler, reusing the base actor name drv-<id>. Context.Stop is async
(the child processes its own stop on its own mailbox), so the old child was
ALWAYS still registered when the respawn ran — Context.ActorOf threw
InvalidActorNameException deterministically on every AdminUI Restart press,
crashing + restarting the host.
Fix: a monotonic _childSpawnGeneration counter (single-threaded actor) feeds a
-g<gen> suffix on every spawned child name, so a respawn can never collide with
the still-terminating predecessor. Children are tracked by the _children dict
(by IActorRef), never by actor path, so the suffix is invisible to callers.
This also closes the same-shaped latent race in the reconcile path (a removed-
then-readded instance, and a driver-type-change ToStop+ToSpawn in one plan).
Regression test RestartDriver_respawns_the_child_without_an_actor_name_collision
(verified: FAILS on the old code with the exact InvalidActorNameException,
PASSES with the fix). Runtime.Tests 238/238 green. Code-reviewed (approved).
Phase B native alarms never fired end-to-end: GalaxyDriver suppresses OnAlarmEvent until
an alarm subscription exists (_alarmSubscriptions.Count > 0), but the runtime only attached
the OnAlarmEvent handler and never called SubscribeAlarmsAsync — so the central feed stayed
gated and no transition reached the Part 9 condition / /alerts. Unit tests passed because
they inject through the IAlarmSource seam directly; the deferred live /run surfaced it.
DriverHostActor computes per-driver alarm refs (alarm-bearing tags' FullNames) and hands them
via SetDesiredSubscriptions; DriverInstanceActor calls SubscribeAlarmsAsync for IAlarmSource
drivers on Connected entry and whenever alarm refs are pushed while Connected (the deploy path),
idempotent via a cached handle reset on detach so reconnect re-subscribes.
Replace "SystemPlatform mirror tag", "Galaxy alias", and "SystemPlatform-kind" in doc-comments and
test names with neutral accurate wording ("FolderPath-scoped tag", "EquipmentId == null", etc.).
No code, logic, or test bodies changed — comments and one test method name only.
Resolves the code-review notes on 95be607a + the AdminUI bundle: the
EnsureVariable docs (IOpcUaAddressSpaceSink, OtOpcUaNodeManager) and the Tag
entity doc no longer say 'Galaxy / SystemPlatform / alias'; the DriverHostActor
ForwardToMux comment now states the real equipment-tag value-routing gap (the
FullName→NodeId 'live values' milestone) instead of claiming Galaxy values map
straight through.
Log a WARNING on startup when IVirtualTagEvaluator is not registered so a DI misconfig on a
driver-role node is visible in logs instead of silently evaluating all VirtualTags to NoChange.
Add a comment in PushDesiredSubscriptions noting that TryRecoverFromStale does not call this
method, so VirtualTags remain empty after a Stale recovery until the next deployment dispatch
(intentional, consistent with driver recovery).
Two ordering/lifecycle gaps surfaced once tag values began streaming:
1. OpcUaPublishActor.HandleRebuild loaded the latest *Sealed* artifact, but the
rebuild fires at apply time — before this deployment seals — so it materialised
the PREVIOUS revision while SubscribeBulk subscribed to the applied one. The two
disagreed (4 variables materialised vs 396 subscribed) and every config needed
two deploys. RebuildAddressSpace now carries the applied DeploymentId and the
rebuild loads that exact artifact.
2. On restart a node recovered its revision from NodeDeploymentState but left the
driver children + address space empty (and an identical-config redeploy no-ops on
the unchanged revision), so a rebuilt node served nothing until a config change.
Bootstrap now calls RestoreApplied: re-spawn drivers, rebuild from the applied
artifact, re-push SubscribeBulk — no re-ack.
Verified live: recreating the driver nodes auto-restores all 396 galaxy mirror
tags across 40 machines with Good live values, no deploy required.
Materialised SystemPlatform/Galaxy variables previously stayed
BadWaitingForInitialData because nothing told the driver to subscribe
(OpcUaPublishActor TODO 'on a future SubscribeBulk pass') and published
values were only forwarded to the VirtualTag mux, never the OPC UA sink.
DriverHostActor now, after each apply, groups the deployment's galaxy tag
MXAccess refs by driver and sends DriverInstanceActor.SetDesiredSubscriptions;
the actor retains the set and (re)subscribes on every Connected entry, so
values resume after reconnects/redeploys (closes the F8b/#113 gap). Published
values are also forwarded to OpcUaPublishActor as AttributeValueUpdate
(NodeId == galaxy MxAccessRef) so the materialised variable shows live data.
Verified live in docker-dev: galaxy TestMachine_001 tags go Good with a
changing TestChangingInt. +1 unit test.
- Topic-name drift fix: DriverHealthChanged.TopicName and
DriverControlTopic.Name now live on the message contracts in
Commons. AkkaDriverHealthPublisher, DriverStatusSignalRBridge,
DriverHostActor, and AdminOperationsActor all delegate to the
single constant so a rename can't silently desynchronise
publisher and subscriber.
- DriverStatusPanel._opResultClearTimer switched from
System.Timers.Timer to System.Threading.Timer + awaited
DisposeAsync. Prevents an in-flight 8s clear-callback from
invoking StateHasChanged on a component whose hub has already
been released.
- PublishHealthSnapshot deduplicates against the last published
(state, lastSuccess, lastError, errorCount) fingerprint. The
30s heartbeat no longer floods the SignalR layer with identical
Healthy snapshots — newly-joined clients still warm up via the
snapshot store on JoinDriver.
- DriverInstanceSpec carries ClusterId from the deployment artifact;
DriverHostActor threads the real cluster identity into
DriverInstanceActor instead of the local NodeId. Old pre-PR
artifacts without a ClusterId field fall back to the NodeId so
in-flight deployments keep working.
- DriverHostActor.ChildEntry holds the full DriverInstanceSpec
(was only carrying DriverType + LastConfigJson). Restart respawns
preserve RowId, Name, Enabled, ClusterId — no placeholder values.
- Drop the unnecessary _faultLock on DriverInstanceActor — every
read/write site runs inside an Akka message handler which is
single-threaded per actor instance.
- DriverStatusPanel.DisposeAsync awaits Timer.DisposeAsync so an
in-flight 5s tick can't invoke StateHasChanged on a component
whose hub has already been torn down.
- RestartDriver / ReconnectDriver messages + AdminOperationsActor
handlers (broadcast via driver-control DPS topic; audited via
ConfigEdits).
- DriverHostActor subscribes to driver-control; locates the
matching child DriverInstanceActor and stops+respawns it
(Restart) or sends it a ForceReconnect internal message
(Reconnect — re-enters Reconnecting state without full stop).
DriverInstanceSpec constructor call uses named args to handle
the full 6-parameter signature.
- New DriverOperator authorization policy mapped to DriverOperator
or FleetAdmin role; documented in docs/security.md. Map LDAP
group via GroupToRole (e.g. "ot-driver-operator": "DriverOperator").
- DriverStatusPanel renders Reconnect + Restart buttons when the
user holds the DriverOperator policy (hidden otherwise). Restart
requires an in-page Razor confirm block (no JS confirm, keeps
SignalR event loop unblocked). Both buttons show a spinner and
are disabled during in-flight; result chip auto-clears after 8s.
Username sourced from AuthenticationStateProvider.
Reconnect resolves to "ForceReconnect" (re-enter Reconnecting,
not full stop+respawn) — transport drops and retries while actor
and in-memory state are preserved. All DriverInstanceActor states
handle ForceReconnect safely (no-op when already in transition).
- IDriverHealthPublisher in Core.Abstractions + NullDriverHealthPublisher
no-op for tests/dev-stub paths.
- AkkaDriverHealthPublisher in Runtime forwards to the cluster-wide
`driver-health` DPS topic.
- DriverInstanceActor instrumented to publish snapshots on every
observable state change + a periodic 30s heartbeat so the AdminUI
snapshot store warms up for newly-joined SignalR clients.
- Sliding 5-minute Faulted-count tracked per actor via Queue<DateTime>.
- DriverHostActor.SpawnChild threads clusterId (_localNode.Value) and
the health publisher down to every DriverInstanceActor child.
- ServiceCollectionExtensions.AddOtOpcUaRuntime registers
AkkaDriverHealthPublisher as IDriverHealthPublisher singleton.
Adds <summary>, <param>, <typeparam>, and <inheritdoc/> tags to public
members surfaced by commentchecker — resolves 5,847 of 5,869 issues
(99.6%) across three /fixdocs passes.
OtOpcUaTelemetry (Commons/Observability) centralizes the project's Meter
+ ActivitySource so all instrumentation points emit through a single
named surface. Counters cover the hot paths:
otopcua.deploy.applied (outcome=ack|reject)
otopcua.deploy.apply.duration (s, histogram)
otopcua.driver.lifecycle (event=spawn|spawn_stub|stop|fault)
otopcua.virtualtag.eval (outcome=ok|fail|skip)
otopcua.scriptedalarm.transition (state=activated|acknowledged|cleared)
otopcua.opcua.sink.write (kind=value|alarm|rebuild)
otopcua.redundancy.service_level_change (level=byte)
Plus two ActivitySource spans:
otopcua.deploy.apply wraps DriverHostActor.ApplyAndAck
otopcua.opcua.address_space_rebuild wraps OpcUaPublishActor.HandleRebuild
Instruments are no-op until a listener attaches, so tests + dev hosts
pay nothing for unread telemetry.
Host Program.cs gains AddOtOpcUaObservability() (binds the OtOpcUa Meter
+ ActivitySource to OpenTelemetry, attaches a Prometheus exporter) and
MapOtOpcUaMetrics() (mounts /metrics scrape endpoint). Driver-side
internals + ASP.NET request metrics deliberately stay off — the scrape
payload is scoped to OtOpcUa signals only.
Tests use MeterListener + ActivityListener to verify
VirtualTagActor.eval, OpcUaPublishActor.AttributeValueUpdate, and
RebuildAddressSpace actually emit on the central instruments. Runtime
suite is 72 / 72 green (+3).
Closes#105. Path A (F13b/c/d) complete; next batch options: #85 UNS
folder hierarchy in SDK, or F8b/F9b production engine bindings.
Closes the loop between F10b (SDK NodeManager) and F14 (Phase7Plan +
Phase7Applier). DriverHostActor's successful apply now triggers a
RebuildAddressSpace on the publish actor, which loads the latest
deployment artifact + walks composer → planner → applier through the
sink. The OPC UA address space tracks the deployed composition.
DeploymentArtifact:
- New ParseComposition(blob) → Phase7CompositionResult that decodes
Equipment + DriverInstance + ScriptedAlarm arrays into the
projection records Phase7Planner consumes. Pascal-case property
names mirror ConfigComposer.SnapshotAndFlattenAsync's output.
- Each entity reader is tolerant: missing-id rows are dropped,
natural-key sort matches Phase7Composer's contract.
OpcUaPublishActor:
- New Props params: dbFactory + applier. When wired, RebuildAddressSpace
does:
1. LoadLatestArtifact (most recent Sealed Deployment.ArtifactBlob)
2. ParseComposition → Phase7CompositionResult
3. Phase7Planner.Compute(lastApplied, next) → Phase7Plan
4. Empty plan ⇒ no-op (deploy of unchanged composition is benign)
5. applier.Apply(plan) drives sink.RebuildAddressSpace +
WriteAlarmState for removed nodes
6. lastApplied = next so the next rebuild diffs forward
- Without dbFactory/applier wiring, falls back to raw
sink.RebuildAddressSpace — the dev/Mac path before #108 binds prod.
DriverHostActor:
- New Props param opcUaPublishActor (IActorRef?). After successful
ApplyAndAck (status Applied, ACK sent), tells the publish actor
RebuildAddressSpace with the same correlation id so the audit trail
threads through. Null publish actor ⇒ no trigger (admin-only nodes).
Tests: Runtime 63 -> 69 (+6):
- ParseComposition reads Equipment/Driver/Alarm sorted by natural key
- ParseComposition returns empty for empty blob
- Rebuild with dbFactory + sealed deployment artifact triggers exactly
one sink.Rebuild call (Equipment topology added)
- Rebuild with no artifact is idempotent no-op
- Second rebuild with same composition is empty-plan no-op
- Rebuild without dbFactory falls back to raw sink.Rebuild (legacy path)
All 6 v2 test suites green: 173 tests passing.
Closes#109. Engine-wiring data flow is now end-to-end through:
Deploy → DriverHostActor.ApplyAndAck → driver spawn + ACK +
RebuildAddressSpace → OpcUaPublishActor → Phase7Applier → SDK
NodeManager → subscribed OPC UA clients see the change.
End-to-end data path is now wired on the read side: driver subscriptions
fire AttributeValuePublished → DriverHostActor → DependencyMuxActor →
DependencyValueChanged to every interested VirtualTagActor. Previously
the publish hit a dead-letter at the host.
DependencyMuxActor:
- Per-node fan-out router. Maintains tagRef → Set<IActorRef> with a
reverse subscriber → refs index so unregister/replace are O(refs).
- Watches subscribers; Terminated triggers automatic unregister so
dead virtual-tag actors stop receiving publishes.
- Re-register replaces the prior interest set — no stale-ref leaks
on actor restart.
- Drops publishes for refs with no interested subscribers.
VirtualTagActor:
- New Props params: dependencyRefs + mux ActorRef.
- PreStart sends RegisterInterest to the mux; PostStop sends
UnregisterInterest. Default both null so older callers stay quiet.
DriverHostActor:
- New dependencyMux Props param. Steady + Applying states now
receive AttributeValuePublished from their DriverInstance children
and forward to the mux. Null mux is a no-op (dev/Mac).
ServiceCollectionExtensions:
- WithOtOpcUaRuntimeActors spawns DependencyMuxActor before
DriverHostActor and threads its ActorRef into the host's Props.
New DependencyMuxActorKey + DependencyMuxActorName.
Tests: Runtime 57 -> 63 (+6):
- Mux forwards to only subscribers interested in each ref
- Publish for unregistered ref is dropped silently
- Unregister stops forwarding
- Re-register replaces prior interest set
- VirtualTagActor PreStart registration drives end-to-end eval
(uses AwaitAssert to race-safely settle the PreStart Tell)
- DriverHostActor forwards AttributeValuePublished through to mux
All 6 v2 test suites green: 163 tests passing.
F8 (#79) state updated — dep subscribe seam shipped, Core.VirtualTags
production engine binding (compile + ITagUpstreamSource subscribe) is
the residual.
Joseph Doherty