Code review found a residual silent-data-loss path: a single driver ref (mux
ref) can back SEVERAL historized equipment tags via aliasing (identical machines
sharing a register — DriverHostActor._nodeIdByDriverRef is a HashSet), each with
its own HistorianTagname. The muxRef->single-name map collapsed last-wins, so
under alias + divergent overrides only one historian tag got the value and the
rest were silently dropped — the exact failure class FU-3 exists to eliminate.
Model the fan as muxRef -> HashSet<historianName> and append ONE outbox entry per
name in OnValueChangedAsync (a per-name append failure drops only that name and
continues). Convergence removes/adds each (muxRef, name) pair individually from
the per-ref set, dropping the mux key only when its last name is removed — so
removing one alias leaves the shared ref fanning for the others with no mux churn.
Tests: aliased-refs-each-get-the-value (one fan → both historian names written),
removing-one-alias-keeps-the-ref-registered, and the override-rename test now
feeds a value post-rename to prove the write target actually moved to the new
name. Runtime 350/0, OpcUaServer 327/0; 0 warnings.
Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
The continuous-historization recorder conflated two identifiers into one string:
the dependency mux fans DependencyValueChanged keyed by the driver FullName
(the mux ref), but a value must be historized under the resolved historian name
(HistorianTagname override, else FullName). In the common no-override case the
two are equal, so it worked; with an override they diverge and the recorder
registered mux interest under a key the mux never fans — that tag's values were
never captured (and, had they been, would have been written under the mux ref).
Carry BOTH identifiers through the seam: a new HistorizedTagRef(MuxRef,
HistorianName) record on IHistorizedTagSubscriptionSink. The applier resolves
MuxRef = FullName and HistorianName = override-or-FullName. The recorder now
keeps a muxRef->historianName map: it registers/filters mux interest by MuxRef
but writes the outbox entry (and drains) under HistorianName. The convergence
handler re-registers the mux only when the registered key-set changes, so an
override-only rename (same FullName) updates the write target without mux churn.
Tests: a divergent-override recorder test (interest by mux ref, value written
under the override name, never the mux ref) + an override-rename no-churn test;
the applier feed tests now assert the full (mux ref, historian name) pairs.
Runtime 348/0, OpcUaServer 327/0; 0 warnings. Closes FU-3.
Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
The ContinuousHistorizationRecorder was spawned with an EMPTY historized-ref
set, so it registered interest in nothing and historized nothing. This feeds it
the currently-historized tag refs on every address-space deploy/redeploy so its
DependencyMuxActor interest converges to exactly the historized set (the same
refs the EnsureTags provisioning hook resolves: override-or-FullName).
Design — delta convergence (the plan is a pure DIFF):
- New seam IHistorizedTagSubscriptionSink (Core.Abstractions/Historian) with a
Null no-op singleton, mirroring how IHistorianProvisioning decouples the T15
hook. AddressSpaceApplier gains a DEFAULTED ctor param (Null sink) so all ~80
existing call sites + the production site compile unchanged.
- Apply() only ever sees a plan diff (an incremental/surgical apply carries a
delta, not the full set), so the applier feeds an add/remove DELTA computed
from AddedEquipmentTags / RemovedEquipmentTags / ChangedEquipmentTags. The
recorder keeps the full set and re-registers it. The feed is a single
non-blocking Tell behind the sink, wrapped in try/catch so a faulting feed
never blocks or breaks a deploy (same discipline as the provisioning hook).
- Recorder.UpdateHistorizedRefs(added, removed) converges the tracked set, then
— only when it actually changed — sends ONE RegisterInterest with the full set
(the mux's RegisterInterest is a full-REPLACE) or one UnregisterInterest when
it drains to empty (the mux has no per-ref unregister). An unchanged delta is
a no-op (no mux churn).
- DI: the recorder is now spawned BEFORE the applier so the adapter
(ActorHistorizedTagSubscriptionSink) can wrap its IActorRef; the Null sink is
used when continuous historization is off/unwired.
Tests: recorder convergence (add-from-empty, add+remove converge, idempotent,
drain-to-empty unregisters); applier feeds resolved added refs, removed+renamed
deltas, and survives a throwing sink. Build clean (0 warnings on touched
projects); Runtime/OpcUaServer/Gateway/AdminUI suites green.
Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
HistorianGateway is now the sole historian backend (read + alarm SendEvent +
continuous WriteLiveValues). Document the final state and retire the Wonderware
sidecar from the docs/config/labels:
- CLAUDE.md: rewrite the Historian section — ServerHistorian /
ContinuousHistorization / AlarmHistorian config keys, the IHistorianProvisioning
EnsureTags hook, the GatewayAlarmHistorianWriter SendEvent path + ReadEvents
dependency on gateway RuntimeDb:EventReadsEnabled=true, gateway-side
prerequisites (RuntimeDb flags + historian:read/write/tags:write scopes),
migration note, and two KNOWN-LIMITATION callouts (live-validation gate +
empty historized-ref-set recorder follow-on).
- appsettings.json: fix the stale ServerHistorian block (Host/Port/SharedSecret/
ServerCertThumbprint -> Endpoint/ApiKey/UseTls/AllowUntrustedServerCertificate/
CaCertificatePath/CallTimeout, keep MaxTieClusterOverfetch); add a disabled
ContinuousHistorization block; prune the orphaned Wonderware keys from
AlarmHistorian (keep the SQLite knobs). ApiKey env-supplied via
ServerHistorian__ApiKey (commented; valid strict JSON via _comment keys).
- README.md + docs (Historian.md, AlarmHistorian.md, Configuration.md,
ServiceHosting.md, DriverLifecycle.md, drivers/README.md, Uns.md, VirtualTags.md,
AlarmTracking.md, Client.UI.md, README.md, TestConnectProbes.md): retire the
Wonderware historian backend from current-backend descriptions; fix the stale
ServerHistorian/AlarmHistorian config tables (now gateway shape); convert
drivers/Historian.Wonderware.md to a retired stub pointing at the gateway.
- Source/UI labels (descriptive text only, no behavior change):
OtOpcUaServerHostedService.cs, HistoryPaging.cs, OtOpcUaSdkServer.cs,
HistorianAdapterActor.cs, VirtualTagModal.razor, ScriptedAlarmModal.razor,
AlarmsHistorian.razor now name the HistorianGateway backend.
Build clean (0 errors); AdminUI.Tests green (514 passed).
Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
The HistorianGateway driver is now the sole historian read/write+alarm backend, so the
Wonderware sidecar projects are dead code. Removes the 5 Wonderware projects (driver,
.Client, .Client.Contracts, + their 2 test projects) from the solution and tree, and fully
retires the vestigial 'Historian.Wonderware' driver type (UI/probe-only; it had no driver
factory): the Host probe registration, the AdminUI driver-config surface (driver page,
tag-config editor/model/validator entry, address picker/builder, driver-type catalog +
dropdown + edit-router entries), and their tests. Prunes the now-unused Wonderware
connection fields (Host/Port/UseTls/ServerCertThumbprint/SharedSecret) from
AlarmHistorianOptions (keeping Enabled + the SQLite store-and-forward knobs) and refreshes
the stale XML docs that named Wonderware as the production backend.
Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
Addresses T18 review: GatewayHistorianValueWriter is a DI singleton holding a gRPC
channel — make it IAsyncDisposable so the container closes the channel gracefully at
shutdown. Tighten the blank-OutboxPath warning to state startup will fail.
Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
Bind ContinuousHistorizationOptions (Enabled/OutboxPath/CommitMode/
CommitIntervalMs/DrainBatchSize/DrainIntervalSeconds/Capacity/backoff) with a
warn-only Validate(); gated on Enabled AND the ServerHistorian gateway being
configured, the Host registers the durable FasterLogHistorizationOutbox (container
-disposed) + a gateway-backed GatewayHistorianValueWriter, and binds outbox
depth/dropped observable gauges on the central scraped meter. WithOtOpcUaRuntimeActors
spawns the recorder (over the same dependency-mux ref) when the options + writer +
outbox resolve, registering ContinuousHistorizationRecorderKey. Spawned with an EMPTY
historized-ref set: the deployed address space builds later, so ref population is a
documented follow-on (a later SetHistorizedRefs feed) — T18 wires the actor + outbox
+ writer + meters; the ref feed is the known remaining gap.
Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
I-1: Wrap the OnValueChangedAsync AppendAsync in try/catch so a durable-boundary
failure (e.g. a PerEntry fsync hitting disk-full/I-O error) can no longer propagate
out of the handler and trip Akka supervision into a restart loop. A canceled append
during shutdown returns quietly; any other exception increments a new
_outboxAppendFailures counter, logs a Warning (exception type name only), and drops
the value without recording it or nudging the drain. The counter is surfaced on
RecorderStatus (new OutboxAppendFailures field).
I-2: Strengthen Writer_failure_keeps_entry_for_retry to prove the drain actually ran
— assert the writer was invoked (the fake records even on Succeed=false) AND the
outbox stayed at 1 (RemoveAsync not called), via AwaitAssertAsync.
M-3: Capture Sender before the await in the GetStatus handler, then Tell the reply.
M-4: Add Retry_after_writer_failure_eventually_acks proving the retry -> success ->
ack path; FakeValueWriter gains a FailFirstN option + CallCount (Succeed behaviour
unchanged). Short minBackoff keeps it fast and deterministic (AwaitAssert, no sleep).
M-5: Deregister mux interest on PostStop via DependencyMuxActor.UnregisterInterest,
mirroring VirtualTagActor.PostStop, closing the dead-letter window before Terminated.
Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
Addresses T15 review: treat a canceled EnsureTags task like a faulted one so the
fire-and-forget continuation never reaches t.Result (which would re-throw and leave
the discarded task unobserved).
Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
Continuous-historization engine for non-Galaxy driver tags. Registers
interest with the per-node DependencyMuxActor for the historized refs and
taps the VirtualTagActor.DependencyValueChanged values the mux fans:
coerce to numeric -> append to the durable IHistorizationOutbox (crash
boundary) -> off-thread drain writes batches through IHistorianValueWriter
and acks (FIFO-truncates) on success, backing off (exponential, capped) on
failure. Non-numeric values are dropped + metered (SQL analog path is
numeric-only).
- New seam IHistorianValueWriter + HistorizationValue in Core.Abstractions
so Runtime stays free of the gRPC driver.
- GatewayHistorianValueWriter (driver) adapts IHistorianGatewayClient.
WriteLiveValues: HistorizationValue -> HistorianLiveValue proto, WriteAck
Success||Queued -> true; non-throwing (errors -> false for retry).
- Drain runs via PipeTo(Self) so the mailbox never blocks on the gateway
write; appends awaited on the actor thread to stay serialized.
Adaptation vs plan: the mux fans DependencyValueChanged (TagId/Value/
TimestampUtc, no quality), not DriverInstanceActor.AttributeValuePublished,
so values are recorded Good-quality (192) by the same convention the
scripted-alarm host uses.
Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
Addresses Task 9 review: add the enabled+nonpositive MaxTieClusterOverfetch warning
test; update the AddServerHistorian XML doc to describe the gateway-backed data source
(the alarm-path Wonderware doc stays until T13).
Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
DriverHostActor/DriverInstanceActor and cluster events log via Akka's ILoggingAdapter, which had no Serilog bridge — under the Windows service host the default StandardOutLogger output is discarded, making the driver-role actor graph invisible (this masked the data-plane diagnosis).
- Add Akka.Logger.Serilog 1.5.60 (deps satisfied by Akka 1.5.62 / Serilog 4.3.1).
- WithOtOpcUaClusterBootstrap: ConfigureLoggers(DebugLevel; ClearLoggers(); AddLogger<SerilogLogger>()) — Akka.Hosting owns logger setup, so HOCON akka.loggers alone is not honored.
- Program.cs: set static Serilog.Log.Logger from the DI root logger after Build() (AddZbSerilog registers the MEL provider but not the static logger, which the Akka SerilogLogger and the startup banner both need).
A rename-only deploy produced an IsEmpty plan that short-circuited before MaterialiseHierarchy,
leaving the OPC UA folder DisplayName stale. AddressSpacePlanner now diffs UnsAreas/UnsLines by
stable id into a RenamedFolders set (counted in IsEmpty); the applier refreshes the folder in
place via a new UpdateFolderDisplayName on ISurgicalAddressSpaceSink (forwarded through
DeferredAddressSpaceSink so it is NOT inert on driver hosts; falls back to rebuild when the sink
is non-surgical). DeploymentArtifact byte-parity untouched (rename rides the existing Name
round-trip). No EF migration, no serialized wire/proto contract change. +13 OpcUaServer tests, Runtime rebuild test.
- OpcUaServer-002: HistoryRead-Events NumValuesPerNode==0 now maps to unbounded (int.MaxValue) instead of the backend default-cap sentinel; no Core.Abstractions contract change (+EventMaxEvents helper tests)
- OpcUaServer-004: EnsureAddressSpaceCreated guard on public mutators -> clear InvalidOperationException instead of bare NRE if called pre-start (+tests)
- OpcUaServer-003: Deferred (endUtc inclusive/exclusive needs live Wonderware boundary confirmation)
- Configuration-013: wire DraftValidator.ValidateClusterTopology into AdminOperationsActor deploy gate (read-only, no migration) (+2 tests)
Cross-module fix from the review sweep. -007 (Medium): OnTimedUnshelve built its AlarmCommand
with User=string.Empty, so Part9StateMachine.ApplyUnshelve rejected it (ArgumentException,
swallowed) and a TimedShelve never auto-expired. Pass the canonical 'system' user; the
AlarmAck-gate bypass is preserved. Repurposed the test that had encoded the bug.
Joseph Doherty