Four continuous-historization types (IHistorizationOutbox, HistorizationOutboxEntry,
IHistorianValueWriter/HistorizationValue, HistorizationCommitMode) drifted into a
ZB.MOM.WW.OtOpcUa.Core.Abstractions.Historian sub-namespace while the other six files
in the same folder correctly use the root namespace. This violated decision #59's
flat-public-surface rule and left InterfaceIndependenceTests.AllPublicTypes_LiveInRootNamespace
red on master (pre-existing, predates round-2 remediation).
Move all four to the root namespace and drop/retarget the now-redundant
'using ...Core.Abstractions.Historian;' in the ~11 consumers. No behavioral change.
Verified: Core.Abstractions.Tests 129/129 (was 128/129), Runtime.Tests Historian+DI 72/72,
Gateway driver unit 102/102, full solution build 0 errors.
Findings 01/S-1 (AddressSpaceApplyOutcome failure field + apply.failed logging,
no optimistic success), 06/S-1 (Galaxy write fails closed -> BadCommunicationError
+ #5 revert, no knowingly-lost raw Write), 03/S4 (PrimaryGatePolicy default-deny
unknown-role-multi-driver on all gates + scripted-alarm emit gate). T13/T15 (2-node
live gates) deferred -- T15 is the behavior-affecting S4 live gate, must run in heavy
pass. Clean merge, build clean.
- CapabilityInvoker: 3 methods that implement IDriverCapabilityInvoker now use
<inheritdoc /> instead of duplicating the interface docs (MissingInheritDoc).
- TwinCAT NativeRegistration ctor: added the 8 missing <param> tags (MissingParam).
- Stripped the arch-review bookkeeping IDs STAB-1 (IS7Plc, S7Driver) + STAB-2
(TwinCATDriver x4) from comments, keeping the descriptive text.
Left the other 86 flagged items untouched — they are checker false positives on
legitimate technical content: interface-implemented events / IEqualityComparer /
IDisposable members whose <inheritdoc/> IS correct (10 InheritDocMisused), and domain
terms the heuristic misreads as tracking IDs (IEEE-754, S7-1200/300/400/1500, PLC-5,
D2-260, S7/Melsec memory addresses, OPC UA Part 9, phase-N design-doc filename refs,
UNS level-N descriptions, and prose like 'the #1 bug source' / failover 'pick #3').
The DriverInstance.ResilienceConfig column was authored in AdminUI, persisted to the
entity, and serialized into the deployment artifact by ConfigComposer — but the runtime
read path dropped it: DriverInstanceSpec didn't carry it and the invoker factory always
passed null, so every driver got tier defaults regardless of its configured overrides
(a silent dead-config gap — #10's residual sub-finding).
Read-path plumbing (write side was already complete):
- DriverInstanceSpec gains ResilienceConfig; DeploymentArtifact.TryReadSpec reads the column.
- IDriverCapabilityInvokerFactory.Create takes resilienceConfigJson; DriverHostActor.SpawnChild
threads spec.ResilienceConfig; the concrete factory parses it (ParseOrDefaults, layering on the
tier), logs any parse diagnostic (never throws), and builds the invoker with the merged options.
- Invalidate-on-change: the pipeline cache keys on (instance, host, capability) and ignores options
on a hit, so Create() now Invalidate()s the instance's cached pipelines first (no-op on first
spawn) — a respawn with changed options rebuilds them.
- DriverSpawnPlanner treats a ResilienceConfig change as a stop+respawn (the invoker/options are
bound to the child at spawn); a pure DriverConfig change stays an in-place delta (no reconnect).
- Host DI passes a logger to the factory for the parse diagnostic.
Verification (deterministic): factory Create applies a retryCount:0 override to actual execution
(control test proves tier default retries), invalidates the instance's cache on re-create (scoped —
sibling survives), malformed config logs+falls-back; planner respawns on ResilienceConfig change
(incl null→json) and stays delta on a pure config change; artifact parse carries/omits the column.
Core.Tests 243 (+5), Runtime.Tests 363 (+5), Host builds clean.
Akka.Cluster.Hosting's WithClustering enables an SBR downing provider by default
(applies SplitBrainResolverOption.Default when ClusterOptions.SplitBrainResolver is
null), which reads the pre-existing akka.conf keep-oldest block. So the cluster was
NOT running NoDowning before Critical 1 and hard-crash failover already worked — the
typed KeepOldestOption is reinforcing/explicit-in-code, not the sole activator.
Corrects the inaccurate 'HOCON inert / NoDowning / never fails over' framing in:
- ServiceCollectionExtensions.BuildClusterOptions XML comment
- akka.conf split-brain-resolver comment
- docs/Redundancy.md Split-brain section
- SplitBrainResolverActivationTests summary + assertion message (+ method rename)
No code revert (the typed option is correct belt-and-suspenders). Cluster.Tests 29/29.
Surfaced by the #9 hard-kill failover negative control.
Remediates RESILIENCE-DISPATCH-GAP (surfaced by the 07/C-1 analyzer): the Phase 6.1
CapabilityInvoker resilience pipeline (retry / breaker / bulkhead / telemetry) was
constructed only in tests — the production dispatch layer called driver-capability
methods directly, bypassing it entirely.
The filed plan (thread CapabilityInvoker directly into DriverInstanceActor) is
INFEASIBLE: Runtime is deliberately Polly-free (references Core.Abstractions, not
Core — the same boundary IDriverFactory documents). So this introduces a seam,
mirroring IDriverFactory exactly:
- Core.Abstractions: IDriverCapabilityInvoker (+ NullDriverCapabilityInvoker
pass-through) and IDriverCapabilityInvokerFactory (+ null factory).
- Core: CapabilityInvoker now implements the interface; new
DriverCapabilityInvokerFactory builds a per-instance invoker over the
process-singleton pipeline builder + status tracker + tier resolver.
- Runtime: DriverInstanceActor takes an IDriverCapabilityInvoker (default =
pass-through) and routes all 6 dispatch sites (write, alarm-ack, subscribe,
unsubscribe, alarm-subscribe, discover) through it; per-host key resolved via
IPerCallHostResolver for single-ref calls, driver-instance key for bulk calls.
DriverHostActor builds + injects the real invoker per spawned driver.
- Host: DriverFactoryBootstrap registers the tracker, pipeline builder, and
concrete factory (needs DriverFactoryRegistry.GetTier). Runtime SCE resolves the
factory from DI like IDriverFactory; pass-through on nodes without it bound.
- Analyzer: IDriverCapabilityInvoker.ExecuteAsync/ExecuteWriteAsync are now
recognized wrapper homes (interface-typed invoker calls must suppress OTOPCUA0001
too). All 6 RESILIENCE-DISPATCH-GAP pragmas removed — the analyzer (an error in
Runtime) is now the standing regression guard.
Also adds retry / breaker-open / breaker-close LOGGING to the pipeline builder —
the operator-facing observability surface (the Admin /hosts reader, Phase 6.1
Stream E.2/E.3, was never built, so the pipeline otherwise runs silently).
Scope notes:
- Tier-DEFAULT policy only: DriverInstanceSpec (the deploy artifact) does not carry
the per-instance ResilienceConfig JSON, so overrides aren't applied yet (tracked
follow-up: plumb ResilienceConfig through the composer/artifact).
- GenericDriverNodeManager's call is NOT wired: that class is test-only scaffolding
(zero production references — the Server has its own address-space path); its
pragma is re-annotated accordingly, not left as a "tracked gap".
Verification (unit + analyzer; live behavioral gate still pending — see FOLLOWUP-10):
- Negative control: unwrapping any site fails the Runtime build with OTOPCUA0001.
- New Runtime guard (recording invoker): write routes via ExecuteWriteAsync with the
IPerCallHostResolver host; subscribe via ExecuteAsync — proves runtime routing.
- New analyzer test: the interface is a valid wrapper home.
- New pipeline-builder test: retry events are logged.
- Full solution builds clean (0 errors); Runtime.Tests 357, Core.Tests 238,
Analyzers.Tests 32 all green; pass-through default keeps existing dispatch tests
byte-for-byte unchanged.
Inject the custom UnwrappedCapabilityCallAnalyzer as an OutputItemType=Analyzer
ProjectReference from Directory.Build.props (excluding the analyzer + its test
project) so OTOPCUA0001 runs on every src/ and tests/ compilation — it previously
enforced its CapabilityInvoker-wrapping rule against nothing but its own 31 unit
tests (the 'built-but-never-wired' failure mode).
Triage of the ~280 surfaced hits, three categories:
1. RESILIENCE-DISPATCH-GAP (7 sites, DriverInstanceActor x6 + GenericDriverNodeManager
x1): a REAL, previously-untracked gap the analyzer caught on first wiring — the
Phase 6.1 CapabilityInvoker resilience pipeline (retry/breaker/bulkhead/telemetry)
is constructed ONLY in tests and was never wired into the production dispatch
layer. Scoped per-site #pragma with a greppable RESILIENCE-DISPATCH-GAP marker
explicitly noting these are tracked-but-not-intentional, pending the dispatch-wiring
remediation (filed as a follow-up). Keeps the analyzer live everywhere else in
those projects so a NEW unwrapped call still fails the build.
2. Driver-INTERNAL self-calls (3 sites, AbCipAlarmProjection x2 + S7Driver x1):
a driver's own poll/ack path calling its own capability method. The invoker wraps
the driver from the dispatch layer OUTWARD; a driver re-wrapping its own internal
calls would double-wrap. Genuinely intentional — scoped #pragma with that rationale.
3. Wire-level test suites + manual-testing CLIs (12 projects): invoke drivers directly
by design — the analyzer's own documented intentional case. Project-level NoWarn
with a comment.
Verified: full solution build green, 0 OTOPCUA0001 hits; analyzer's 31 tests pass;
negative control — dropping one dispatch-gap pragma re-fires OTOPCUA0001 and fails
the Runtime build, proving the analyzer is genuinely live tree-wide, not disabled.
U2 (Critical): RoslynVirtualTagEvaluator ran scripts with a CancellationToken
that a synchronous Roslyn script can never observe, so one CPU-bound/infinite-
loop virtual-tag script hung the owning VirtualTagActor forever (no timeout, no
recovery). Route evaluation through the existing TimedScriptEvaluator (Task.Run
+ WaitAsync) so the wall-clock budget is real; a runaway script now returns Bad
within the timeout instead of wedging the actor.
U3 (High): the live path cached evaluators in a raw ConcurrentDictionary never
cleared on republish, leaking a collectible AssemblyLoadContext per edited
script forever. Swap in the purpose-built CompiledScriptCache (Lazy single-
compile, dispose-on-Clear) and add an apply-boundary clear: a new narrow
IScriptCacheOwner seam the VirtualTagHostActor calls on each ApplyVirtualTags
generation (mirrors ScriptedAlarmEngine's per-generation clear).
Tests: infinite-loop-fails-within-timeout (bounded by WaitAsync so a regression
fails instead of hanging), happy-path-after-wrapping, ClearCompiledScripts-
empties-cache, and a host-actor wiring guard asserting each apply generation
calls ClearCompiledScripts (theme #1 production-caller assertion). Host.Integration
VT suite 18/18, Runtime VT suite 12/12.
The split-brain-resolver HOCON block in akka.conf was inert — nothing
registered a downing provider, so the cluster ran Akka's default NoDowning:
a hard-crashed node was never downed, cluster singletons + the driver
role-leader never failed over, and a partition left both redundancy sides
at ServiceLevel 240 indefinitely.
Activate the resolver via the typed ClusterOptions.SplitBrainResolver
(KeepOldestOption { DownIfAlone = true }) in a new testable
BuildClusterOptions helper. keep-oldest + down-if-alone is correct for a
2-node warm-redundancy pair. HOCON keeps stable-after (typed option can't
express it) + a cross-reference comment. Deterministic unit guard added;
end-to-end verified the resolved ActorSystem config carries the provider
and stable-after survives the HOCON/Hosting merge. Redundancy.md corrected.
Hard-kill failover integration test tracked as a follow-up (effort-M harness).
Adds <summary>/<param>/<returns>/<inheritdoc> where missing and removes
project bookkeeping IDs (task/tracking refs) from shipped code comments,
so the docs read cleanly and CommentChecker is quiet except for known
false positives (PLC/protocol terms, event/IEqualityComparer inheritdoc).
Doc/comment-only; no logic changed; solution builds clean.
The continuous-historization recorder conflated two identifiers into one string:
the dependency mux fans DependencyValueChanged keyed by the driver FullName
(the mux ref), but a value must be historized under the resolved historian name
(HistorianTagname override, else FullName). In the common no-override case the
two are equal, so it worked; with an override they diverge and the recorder
registered mux interest under a key the mux never fans — that tag's values were
never captured (and, had they been, would have been written under the mux ref).
Carry BOTH identifiers through the seam: a new HistorizedTagRef(MuxRef,
HistorianName) record on IHistorizedTagSubscriptionSink. The applier resolves
MuxRef = FullName and HistorianName = override-or-FullName. The recorder now
keeps a muxRef->historianName map: it registers/filters mux interest by MuxRef
but writes the outbox entry (and drains) under HistorianName. The convergence
handler re-registers the mux only when the registered key-set changes, so an
override-only rename (same FullName) updates the write target without mux churn.
Tests: a divergent-override recorder test (interest by mux ref, value written
under the override name, never the mux ref) + an override-rename no-churn test;
the applier feed tests now assert the full (mux ref, historian name) pairs.
Runtime 348/0, OpcUaServer 327/0; 0 warnings. Closes FU-3.
Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
The ContinuousHistorizationRecorder was spawned with an EMPTY historized-ref
set, so it registered interest in nothing and historized nothing. This feeds it
the currently-historized tag refs on every address-space deploy/redeploy so its
DependencyMuxActor interest converges to exactly the historized set (the same
refs the EnsureTags provisioning hook resolves: override-or-FullName).
Design — delta convergence (the plan is a pure DIFF):
- New seam IHistorizedTagSubscriptionSink (Core.Abstractions/Historian) with a
Null no-op singleton, mirroring how IHistorianProvisioning decouples the T15
hook. AddressSpaceApplier gains a DEFAULTED ctor param (Null sink) so all ~80
existing call sites + the production site compile unchanged.
- Apply() only ever sees a plan diff (an incremental/surgical apply carries a
delta, not the full set), so the applier feeds an add/remove DELTA computed
from AddedEquipmentTags / RemovedEquipmentTags / ChangedEquipmentTags. The
recorder keeps the full set and re-registers it. The feed is a single
non-blocking Tell behind the sink, wrapped in try/catch so a faulting feed
never blocks or breaks a deploy (same discipline as the provisioning hook).
- Recorder.UpdateHistorizedRefs(added, removed) converges the tracked set, then
— only when it actually changed — sends ONE RegisterInterest with the full set
(the mux's RegisterInterest is a full-REPLACE) or one UnregisterInterest when
it drains to empty (the mux has no per-ref unregister). An unchanged delta is
a no-op (no mux churn).
- DI: the recorder is now spawned BEFORE the applier so the adapter
(ActorHistorizedTagSubscriptionSink) can wrap its IActorRef; the Null sink is
used when continuous historization is off/unwired.
Tests: recorder convergence (add-from-empty, add+remove converge, idempotent,
drain-to-empty unregisters); applier feeds resolved added refs, removed+renamed
deltas, and survives a throwing sink. Build clean (0 warnings on touched
projects); Runtime/OpcUaServer/Gateway/AdminUI suites green.
Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
The HistorianGateway driver is now the sole historian read/write+alarm backend, so the
Wonderware sidecar projects are dead code. Removes the 5 Wonderware projects (driver,
.Client, .Client.Contracts, + their 2 test projects) from the solution and tree, and fully
retires the vestigial 'Historian.Wonderware' driver type (UI/probe-only; it had no driver
factory): the Host probe registration, the AdminUI driver-config surface (driver page,
tag-config editor/model/validator entry, address picker/builder, driver-type catalog +
dropdown + edit-router entries), and their tests. Prunes the now-unused Wonderware
connection fields (Host/Port/UseTls/ServerCertThumbprint/SharedSecret) from
AlarmHistorianOptions (keeping Enabled + the SQLite store-and-forward knobs) and refreshes
the stale XML docs that named Wonderware as the production backend.
Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
Continuous-historization engine for non-Galaxy driver tags. Registers
interest with the per-node DependencyMuxActor for the historized refs and
taps the VirtualTagActor.DependencyValueChanged values the mux fans:
coerce to numeric -> append to the durable IHistorizationOutbox (crash
boundary) -> off-thread drain writes batches through IHistorianValueWriter
and acks (FIFO-truncates) on success, backing off (exponential, capped) on
failure. Non-numeric values are dropped + metered (SQL analog path is
numeric-only).
- New seam IHistorianValueWriter + HistorizationValue in Core.Abstractions
so Runtime stays free of the gRPC driver.
- GatewayHistorianValueWriter (driver) adapts IHistorianGatewayClient.
WriteLiveValues: HistorizationValue -> HistorianLiveValue proto, WriteAck
Success||Queued -> true; non-throwing (errors -> false for retry).
- Drain runs via PipeTo(Self) so the mailbox never blocks on the gateway
write; appends awaited on the actor thread to stay serialized.
Adaptation vs plan: the mux fans DependencyValueChanged (TagId/Value/
TimestampUtc, no quality), not DriverInstanceActor.AttributeValuePublished,
so values are recorded Good-quality (192) by the same convention the
scripted-alarm host uses.
Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
DriverHostActor/DriverInstanceActor and cluster events log via Akka's ILoggingAdapter, which had no Serilog bridge — under the Windows service host the default StandardOutLogger output is discarded, making the driver-role actor graph invisible (this masked the data-plane diagnosis).
- Add Akka.Logger.Serilog 1.5.60 (deps satisfied by Akka 1.5.62 / Serilog 4.3.1).
- WithOtOpcUaClusterBootstrap: ConfigureLoggers(DebugLevel; ClearLoggers(); AddLogger<SerilogLogger>()) — Akka.Hosting owns logger setup, so HOCON akka.loggers alone is not honored.
- Program.cs: set static Serilog.Log.Logger from the DI root logger after Build() (AddZbSerilog registers the MEL provider but not the static logger, which the Akka SerilogLogger and the startup banner both need).
A rename-only deploy produced an IsEmpty plan that short-circuited before MaterialiseHierarchy,
leaving the OPC UA folder DisplayName stale. AddressSpacePlanner now diffs UnsAreas/UnsLines by
stable id into a RenamedFolders set (counted in IsEmpty); the applier refreshes the folder in
place via a new UpdateFolderDisplayName on ISurgicalAddressSpaceSink (forwarded through
DeferredAddressSpaceSink so it is NOT inert on driver hosts; falls back to rebuild when the sink
is non-surgical). DeploymentArtifact byte-parity untouched (rename rides the existing Name
round-trip). No EF migration, no serialized wire/proto contract change. +13 OpcUaServer tests, Runtime rebuild test.
Joseph Doherty