Item 1: IHistorianDataSource.MaxRawValuesPerRead (Gateway: learned ceiling - 16 = L; 0 elsewhere, guard-tested).
RawPageLimit plans every raw page on top of #692's lookahead: with L > 0 the read (page P + boundary ties k
+ lookahead) is <= L, P = min(N, L - k - 1) (uncapped: L - k - 1); no room => straight to the tie-cluster
over-fetch (never an uncapped or underflowing request). One re-plan + re-read when the read taught a new
limit or was refused for crowding pre-window rows (HistoryReadLimitExceededException.PreWindowRows): the
page shrinks by them. Over-fetch bound min(MaxTieClusterOverfetch, L - 1); slice size never 0.
Review fixes: F1 a window starting at/before the epoch counts AVEVA's synthetic row as pre-window (a
ceiling-truncated read was reported complete); F2 the page path disables ReverseRawReader's own re-run
(cold end-only ceiling 20/N=3000 cost 3,023 reads, now <= 40); F3 crowded look-back re-read around, not
refused; F4 the #687 crowding re-read room at least doubles (up to 8 re-reads), so a cluster of hundreds
just before a resume point no longer fails on a default gateway; a crowding refusal that leaves no room
is BadTooManyOperations, never a tick-by-tick cursor.
Item 2: ServerHistorian:EventReadHorizon (default 00:00:00 = exact, unchanged). When set, every fresh
HistoryReadEvents window's earlier bound is raised to now - horizon (both directions, all backends).
Lossy by design; documented.
Item 3 (native TimeAverage/Total re-check) stays blocked on historiangw#27.
Gateway HistoryRead-AtTime passed AVEVA's own interpolation through: live it
answered the previous value as Good at a stored NULL's own time, null + Bad at
a Good sample right after a NULL, a drifted 99.99999999944 for a stored 100,
and no Interpolated bit. It is now served from raw gateway reads and evaluated
with Sql.Common's AtTimeEvaluator, as the SQL backends do (Part 11 §6.5.4).
GatewayAtTimeReader: one <=1,024-row forward window read [first-1ms, last+1ms]
for the batch, a walk-back (density-seeded, x4 growth, whole-remainder shortcut,
bisects dense history) for the first time's prev, and an 18-row probe for the
last time's next; truncated window reads continue or jump with a density-sized
margin. All bounds whole ms; a row counts only strictly inside its request
(AVEVA's boundary row never a sample); Bad rows value-less (#690); never past
now + 1 day; strict-gateway ceiling learned (#691.3) without re-probing.
10 dense times = 1 read (exact) / 3 (interpolated); 10 hourly = 12.
Kill switch ServerHistorian:GatewayAtTimeFromRaw (default true; false = the
old AVEVA passthrough). Known limit: the wire carries no data type, so numeric
tags interpolate as Double (Boolean/integer nodes are not stepped/rounded as on
the SQL backends).
1. The at-or-after bound read on a forward-only backend (Gateway) ends at
now + ReverseRawReader.LookAhead instead of DateTime.MaxValue (the gateway
only checks start <= end, so a year-9999 end reached AVEVA); a bound beyond
the horizon is not found without a read. SqlServer/QuestDB keep MaxValue:
they bind it safely and may hold samples further ahead. The Gateway fake now
refuses far-future ends, so the missing-end-bound test is no longer vacuous.
3. No data and no bound found (whole read empty) => GoodNoData, no values —
the pre-bounds answer (6.5.3.2: success only if one or both bounds exist).
A one-value page whose start bound is missing first checks there is data or
a found end bound.
4. ServerHistorian:HonorReturnBounds (default true); false ignores ReturnBounds
as before #691, including the modified-read Bad_InvalidArgument.
ReverseRawReader aimed every segment at half of MaxTieClusterOverfetch
(~32k rows by default) and carried that width page to page, so a paged
'latest N' read on a dense Gateway tag fetched ~32,770 rows per page
(2-5 s live) to return N. The walk now lives in a generic ReverseWalk
core: segments are aimed at ~N+8 rows (page budget 2N+16), a discovery
segment (first without a hint / after an empty one) may read 1,024, and
the tie budget is only the ceiling a single-timestamp cluster escalates
to. A truncated read never re-enters the stretch below its newest row
until the gap above it is bisected. Tests: per-page row counts on a
dense tag (node manager + walk), randomized exactness fuzz (walk and
node-manager paging), dense-tag-below-a-long-gap read bound.
1. Uncapped reverse raw read on a forward-only backend = ONE forward read of the
window, filtered and reversed in memory (was a segment-by-segment walk).
2. Segment width is aimed at ~half the row budget from the density just observed
(overflow: from the truncated read's own sample span) and carried page to page
in HistoryContinuationState.SegmentWidthTicks.
3. The walk starts no later than now + 1 day and never reads before the epoch.
4. Request bounds are rounded outward to the millisecond, so a row AVEVA
fabricates at a rounded-up query start cannot land inside the segment.
5. Budget M = max(1, MaxTieClusterOverfetch); reads ask M + 2 rows, completeness is
'short OR last row past the segment', and a read crowded by rows before the
segment is re-read once (<= 2M + 4). A cluster of exactly M, neighbouring
clusters each within M, and M <= 1 are all served; only one timestamp over M
is refused. ServerHistorianOptions text updated.
6. The raw and events tie-path oversized-cluster refusal is BadTooManyOperations
(was BadHistoryOperationUnsupported), consistent with the walk.
7. GatewayHistorianDataSource.ReadRawAsync: a caller cancellation is not a backend
failure; NumValuesPerNode = 0 is sent as int.MaxValue (the client package and
the gateway reject max_values <= 0, so every uncapped Gateway raw read failed);
a MaxValuesCeiling refusal is a HistoryReadLimitExceededException.
8. Tests: upper-edge filter, epoch floor, look-ahead, ms alignment, exact/adjacent
clusters, minimal budget, carried density, and the walk over the real Gateway
source + FakeHistorianGatewayClient (OpcUaServer internals visible to it).
- HistoryUpdate items with SourceTimestamp > now + ServerHistorian:HistoryUpdateMaxFutureSkew
(default 1 day, <=0 disables) => BadOutOfRange in the node manager (every backend, never sent);
clock injected via OtOpcUaNodeManager.Clock (host TimeProvider).
- HistoryUpdate is Primary-only: off-Primary => BadInvalidState per node, after authorization,
same IDriverPrimaryRoleSource verdict as the native-ack gate; HistoryRead ungated.
- SqlServer: base schema requirement back to v1; SupportsModifiedTrail (v>=2). On v1 with
AutoCreateSchema=false: writes/reads/events/retention work, modified reads + HistoryUpdate =>
BadHistoryOperationUnsupported with nothing sent, retention skips SampleModified, one Warning per boot.
- QuestDB integration test pinning wall-clock TTL (far-future row evicts only partitions older
than the TTL by the wall clock), with a positive control.
- Docs: Historian.md, Configuration.md, security.md, CLAUDE.md (incl. QuestDB retry duplicates,
#674.4 anonymous-readable trail with ACL off, #674.5 not implemented).
Co-Authored-By: Claude <noreply@anthropic.com>
M1: the time-weighted family (incl. Total, formerly a pushed-down SUM) streams
raw window rows into the C# evaluator. New shared knob
ServerHistorian:MaxTimeWeightedRawRows (default 1,000,000; <=0 unbounded with a
Validate warning) flows through HistorianBackendRegistry into both providers.
The window query is capped at cap+1 (SQL Server TOP (@maxRows), QuestDB LIMIT $4);
one row past the cap throws the new Core.Abstractions
HistoryReadLimitExceededException, which the node manager maps to
BadTooManyOperations for that node. Not counted as a backend health failure;
bound rows are not counted; per-sample aggregates are not capped.
M2: reverse Interpolative is anchored at the later time (Part 13): points at
End - i*interval, emitted latest first. 10:00->09:00 @1h yields 10:00.
L1: Partial bit (0x4, = Opc.Ua.AggregateBits.Partial) on non-Bad results of a
truncated last interval (time-weighted family; per-sample family is a follow-up).
L2/L3: aligned with the SDK AggregateCalculator.GetTimeBasedStatusCode
(decompiled from Opc.Ua.Server 1.5.378.106): TreatUncertainAsBad=false counts
Uncertain time (Uncertain samples, held-into-outage, extrapolation) as Good
time; data but no non-Bad time => generic Bad (0x80000000); BadNoData only
when there is no data at all. Consequence: a live tag's latest interval reads
Good (documented).
L5: release note no longer claims Total matches the Gateway Integral exactly
(AVEVA's divisor is unverified); it notes Total is now evaluated in C# and
subject to the cap.
Tests: unit (reverse Interpolative x3, Partial, Bad vs BadNoData incl. an
open-ended outage, cap helpers, SDK bit oracles); conformance (reverse
Interpolative, raw-row cap at/over/0, bounds not counted, health unaffected);
node manager BadTooManyOperations; options default + warning. Each new
behaviour falsified by mutation (aggregator x9, data-source cap checks on
both backends against the real DBs, node-manager catch).
A SIGKILLed Primary stays an unreachable-but-not-downed member until
auto-down, and the redundancy singleton usually dies with it, so the
standby's cached snapshot kept naming it Primary and nobody captured for
~30 s. DriverHostActor now holds its own ReachabilityEvent (+MemberUp)
subscription, caches the last snapshot, and excludes unreachable peers
from the historization peer-is-Primary predicate only, republishing just
the capture verdict on a reachability change. Alarm drain and the Primary
data-plane verdict are unchanged. Gated by
ContinuousHistorization:CaptureWhilePrimaryUnreachable (default true).
Tests: message-driven cases with real ReachabilityEvents (incl. a
falsification control on the drain/data-plane verdicts) plus a real
two-node cluster crash test with downing disabled.
- Historian.md: Empty samples always Bad (writer/updater/read clamp),
long-name refusal, 5 s bootstrap failure cache, AtTime after now, info-bit
masking, HistoryUpdate does NOT type-check (M-1 decision), DeleteRaw
needs both bounds, reverse Raw paging, the historian/historization metric
prefixes, and provisioning numeric-only lines qualified as Gateway.
- CLAUDE.md numeric-only lines qualified as Gateway backend.
- Stale 'sole historian' lines in DriverLifecycle, ServiceHosting and the
retired Wonderware stub; Client.CLI exit-code contract.
- ServerHistorianOptions: Endpoint/Backend XML docs untangled, Validate()
indented; Program.cs comments are backend-neutral; T10 week-test day count.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
The DroppedUnsupportedKind counter still counts every value; only the Debug
line is latched (cleared by the next captured value), so a Gateway-backend
comms loss no longer logs one line per tag.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
- OtOpcUaNodeManager overrides HistoryUpdateData / HistoryDeleteRawModified / HistoryDeleteAtTime,
block-bridging to the optional IHistorianDataUpdater under the shared HistoryRead batch limiter +
deadline. Insert/Replace/Update map to HistoryUpdateMode; Remove, IsDeleteModified, events and
structure data are BadHistoryOperationUnsupported. Per-item codes stitched back in request order
(MinValue SourceTimestamp => BadInvalidTimestamp, untypeable value => BadTypeMismatch); success
clears the base's pre-seeded Unsupported error.
- HistoryWrite AccessLevel bit OR-ed in for historized nodes when an updater is wired (raw + UNS);
it is dispatch-load-bearing in the SDK base. OtOpcUaSdkServer.SetHistorianDataUpdater; the hosted
service assigns it from an optional HistorianUpdaterHolder before SetSink, clears it on stop.
- AuthorizeHistoryUpdate: ACL on => scoped OpcUaOperation.HistoryUpdate decision (replaces the role
check); ACL off => #563 key re-check + new OpcUaDataPlaneRoles.HistoryUpdate role, conferred by
ApiKeyInterimRoleMapper only from a Cluster-scoped HistoryUpdate grant (WriteOperate does not).
- AdminUI ACL editor offers NodePermissions.HistoryUpdate.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
Review round 1: the #481 FanCommsLossToDependencyMux fan-out DOES reach the recorder. Per the
controller ruling it is historized as an Empty/BadNoCommunication outage marker when the backend
supports Empty and counted in DroppedUnsupportedKind otherwise; remarks and the counter doc now say
so. DefaultSupportedKinds is a FrozenSet. The drop-leg test now runs under the default (gateway)
kinds, and the Empty leg pins status 0x80310000.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
The recorder now appends the typed HistorianValue and the mux's own StatusCode instead of a
double + hard-coded Good, and drops any kind not in the backend's SupportedKinds (resolved
optionally from IHistorianBackendProvider; absent => {Double, Int64, Boolean}, today's behaviour).
RecorderStatus.DroppedNonNumeric -> DroppedUnsupportedKind; GoodQuality const removed; stale
"mux carries no quality" remarks rewritten (comms-loss node stamps are not historized).
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
The batched ingest's append-failure catch had zero coverage, and two doc
statements overstated the loss: the Warning said the burst's values were
"dropped", and RecorderStatus.OutboxAppendFailures said "the value was
dropped, not recorded". The outbox contract is prefix-never-orphan, so a
partial-prefix failure leaves entries durably queued that drain normally.
- New test pins the branch: one 3-item burst whose group commit commits a
1-entry prefix then throws => OutboxAppendFailures == 1 (per burst, not
per value), TotalRecorded == 0, actor alive, and the prefix still reaches
the writer via the post-failure DrainTick (drainInterval 30s, so the
steady cadence cannot account for it). Falsified twice: counting per item
and dropping the drain nudge each turn it red.
- Second test pins the OperationCanceledException filter from its
observable side - a cancel that is NOT shutdown is metered, not swallowed.
Removing the 'when (_lifetime.IsCancellationRequested)' turns it red. The
shutdown side has no cheap seam (the counter dies with the actor).
- Docs corrected: OutboxAppendFailures counts FAILED BATCH APPENDS, one per
burst whatever its size (a granularity change from the per-value era), and
is not a count of values lost; TotalRecorded under-counts when a commit
fails after a partial prefix. Log message says "failed to append" and
names the prefix. No functional change beyond the message string.
Also folds in the DrainBatchSize default alignment behind 0298a0b9: the
recorder's two ctor defaults and its non-positive fallback still said 64
while ContinuousHistorizationOptions.DrainBatchSize now defaults to 1024.
No test pinned the old 64.
Both defects were found by #491's live gate against a real HistorianGateway and filed rather than
fixed at the time, because each one's fix is a design decision rather than a patch.
#537 — both nodes of a redundant pair captured every value, so the historian received duplicate
live samples for every historized tag (measured: the gateway's write counter advanced twice per
value and halved when one central was stopped).
The recorder now reads a capture gate per value, fed from the new
PrimaryGatePolicy.ShouldRecordHistorizedValues through IHistorizationRoleView — a third verdict on
the same RedundancyRoleView object, registered unconditionally on driver nodes rather than
alongside the alarm section's IRedundancyRoleView (which only exists when that section is enabled;
piggybacking is the #511 registration accident).
Two things are deliberate and easy to get backwards:
* It gates the CAPTURE, not the drain — the opposite of the alarm sink. That queue is in LocalDb
and replicates, so the Secondary already holds the Primary's rows. The historization outbox is
a node-local FasterLog replicating to nobody, so a drain-only gate would leave the standby
appending forever to a queue no one empties: unbounded disk, then a flood of stale duplicates
the moment it is promoted. Entries a node already captured are still drained by that node.
* The peer it stands down for must RECORD THE SAME REFS, not merely hold the same rows — a
Primary in this node's own per-cluster mesh. A node carrying no cluster-scoped role cannot
establish that and never stands down: under a fleet-wide election the elected Primary is
usually running a different cluster's configuration, and deferring to it would end this
cluster's historization silently, with the drain reporting success on an empty outbox forever.
That is the scope mismatch that once suspended the alarm drain on every node of the rig.
An unknown role captures, matching the alarm drain — a duplicate sample is recoverable, a hole in
the trend is not.
#538 — the drain writes every per-tag group and then acks all-or-nothing (deliberate: RemoveAsync
truncates THROUGH an id, so partial acking could drop older unwritten entries). One permanently
failing tag therefore pinned the FIFO head while every healthy tag in the batch was re-submitted
to the historian on every pass, indefinitely — duplicate samples and an outbox that never
truncated, observed live at 8.3 MB from a single unprovisionable tag.
The drain now tracks consecutive failures per historian tag and abandons that tag's entries once
it exceeds MaxWriteAttempts (new, default 5), so the head advances. The counter only advances on a
pass where some OTHER tag wrote successfully: that is what separates "this tag is poison" from
"the backend is down". A count-only quarantine looks identical on the poison-tag case and silently
discards the whole backlog during an ordinary gateway restart, turning a duplicate-samples bug
into a lost-history one. Abandonment is also deferred until every failing tag is over budget, so
nothing is discarded on a pass that could not have advanced the head anyway. Each abandonment logs
a Warning naming the tag; the total rides RecorderStatus.AbandonedEntries.
Capacity also gains a non-zero default (500,000 entries) as the disk backstop for the one case the
quarantine deliberately cannot resolve — a historian refusing every write, where retaining is
correct. Set 0 to restore unbounded retention.
Every new test was falsified by reverting the production change it guards, and each one reddens
alone:
* drop the backend-reachable evidence -> only A_backend_that_fails_every_tag_abandons_nothing
* never abandon -> only A_permanently_failing_tag_is_abandoned_so_healthy_tags_stop_being_rewritten
* latch the capture gate at construction -> only The_capture_gate_is_re_read_per_value...
* treat any Primary as a scope peer -> only the two scope-mismatch cases
Full suite green. (Four OpcUaClient.Browser failures during the run were the opc-plc fixture being
down on the shared Docker host; brought it back up and they pass.)
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
Ran #491's live gate against a real ZB.MOM.WW.HistorianGateway talking to a real AVEVA
Historian, and it found a production-fatal defect on the first value it ever saw.
ContinuousHistorizationRecorder.OnValueChangedAsync touched `Self` after awaiting
IHistorizationOutbox.AppendAsync(...).ConfigureAwait(false). ConfigureAwait(false) discards
Akka's ActorTaskScheduler synchronization context, so `Self` — which reads ActorBase.Context —
threw NotSupportedException. The throw escaped into supervision, which RESTARTED the actor;
because production seeds an EMPTY ref set (the deployed set arrives later via
UpdateHistorizedRefs), the restart silently discarded every historized ref. Net effect: one
value captured, then nothing, with no further error to show for it. Fixed by capturing
`IActorRef self = Self` before the await — the idiom the GetStatus handler already applied to
Sender.
Thirteen unit tests missed it because InMemoryOutbox.AppendAsync returned
ValueTask.CompletedTask: a completed ValueTask never yields, so the continuation ran inline
with the context still installed. The production FasterLog outbox fsyncs per entry and does
yield. The first regression test was itself vacuous — `await Task.Yield()` posts back to the
CAPTURED context, so the append still completed on the actor thread and reverting the fix left
the test green. The fake must complete on a POOL thread (Task.Delay(1).ConfigureAwait(false)),
as a real fsync does; with that, reverting the fix reddens exactly one test.
The rig could not run this gate at all: docker-dev carried ServerHistorian__* but no
ContinuousHistorization__* keys, so the recorder could never spawn. Both centrals now carry
them (default OFF — the recorder needs ContinuousHistorization:Enabled AND
ServerHistorian:Enabled), with the outbox on the per-node named volume so it survives a
container recreate. ServerHistorian__UseTls was hard-coded true and is now parameterized; a
plaintext h2c gateway was unreachable without it.
Gate results: outbox append, drain, WriteLiveValues accepted by a real historian, and restart
convergence (capture resumes after a restart with no new deployment) all pass. The read-back
hop is NOT proven — the LAN historian accepts writes but serves no reads under a "License
violation: Database tag count exceeds license tag count", and the licensed corporate one needs
the VPN. EnsureTags provisioning is likewise unexercised there (RemoteGrpc-only, and that CAP
is not listening). #491 stays open for the read-back hop alone.
Also recorded, not fixed: the recorder has NO Primary gate, unlike the alarm store-and-forward
sink, so both nodes of a redundant pair send duplicate live values. Gating it would introduce a
failover window with no capture — that is a design decision, not a cleanup.
Evidence: docs/plans/2026-08-01-continuous-historization-live-gate.md
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
Build: 0 errors solution-wide, and 0 warnings from every project this branch
touches. The ~816 solution-wide warnings are pre-existing xUnit1051 /
OTOPCUA0001 / CS86xx in untouched driver + client test projects.
Tests: full solution run compared against a full run on a detached worktree at
the pre-branch baseline 2e46d054. The two failure SETS are identical -- all 13
tests, same names, zero new failures. Net +26 tests: +3 Core.AlarmHistorian
(drain gate), +6 Runtime (role view), +17 Host.IntegrationTests (migrator +
convergence). Set comparison rather than counts, because the suite carries
standing environment- and load-dependent failures a count would hide.
The greps found real drift Task 6 missed -- eight live sites still naming the
deleted SqliteStoreAndForwardSink, including the AdminUI /alarms/historian
panel text, which is user-visible, and a <see cref> in HistorianAdapterActor
that resolved to nothing without warning. All repointed at
LocalDbStoreAndForwardSink; CLAUDE.md's alarm-history paragraph now also
records the LocalDb buffer and the primary-gated drain, and drops DatabasePath
from the knob list. docs/AlarmTracking.md still promised an
AlarmHistorianOptions.Validate() startup warning for a relative DatabasePath
and an empty SharedSecret; both branches are gone, so it now says so.
Code references to AlarmHistorian:DatabasePath reduce to exactly two intentional
ones: AlarmSfLegacyMigrator.LegacyPathKey and its test. No `new SqliteConnection`
remains anywhere in Core.AlarmHistorian.
Recon doc gains the durable verification record: guard-deletion evidence for
both vacuous passes, the two exact-set replicated-table pins (both assert set
equality, so an added or a dropped registration fails), and the baseline test
comparison with a per-failure account of why each of the 13 is not this
branch's.
Stops here per the plan. Task 8's live gate needs explicit go-ahead; nothing on
this branch is to be merged.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Moves the alarm store-and-forward buffer out of its own alarm-historian.db
and into the node's consolidated LocalDb, where it replicates to the
redundant pair peer. A node that dies holding undelivered alarm history no
longer takes it to the grave.
Tasks 2 and 3 land together, as the plan anticipated. They are not separable:
the gate is a constructor argument of the rewritten sink, and a commit that
replicated the queue without gating the drain would be a commit in which both
nodes of a pair deliver every alarm event, continuously.
That drain gate is the load-bearing part of the change, and the recon
explains why it is new work rather than a refinement. Exactly-once delivery
across a pair is enforced today on the ENQUEUE side, by
HistorianAdapterActor: only the Primary enqueues, so the Secondary's queue is
empty and its ungated drain has nothing to send. Replicating the table
destroys that invariant.
The gate is a Func<bool> the caller supplies, because the drain runs on a
timer the sink owns rather than on a mailbox, and Core.AlarmHistorian cannot
reference PrimaryGatePolicy in Runtime. Runtime supplies it via a new
IRedundancyRoleView singleton that DriverHostActor publishes its Primary-gate
verdict to -- the same verdict the inbound-write and native-ack gates use, so
there is no second notion of am-I-the-Primary to drift.
Two failure modes are deliberately closed:
- The view is seeded OPEN, matching the policy's own answer for an unknown
role with no driver peer. A deployment that runs no redundancy never
publishes to it, and defaulting closed would silently stop its alarm
history forever.
- A gate that throws is read as not-now, never as permission, and a closed
gate reports the new HistorianDrainState.NotPrimary rather than Idle. A
Secondary's rising queue is supposed to look different from a stalled
drain, and if BOTH nodes report NotPrimary the pair is misconfigured and
says so instead of quietly filling toward the capacity ceiling.
Row ids are a hash of the payload rather than fresh GUIDs. Both adapters
accept the same fanned transition in the window before the first redundancy
snapshot arrives, and under last-writer-wins an equal key converges those two
accepts into one row instead of duplicating them.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
OtOpcUa commits no plaintext secrets — the only secret-shaped committed value is
ServerHistorian:ApiKey='' (empty, disabled by default). Because the pre-host expander
is fail-closed AND section-agnostic, hard-committing a ${secret:} token would break
every dev/CI/integration boot for zero at-rest benefit. Instead document the token
delivery convention for all five config secrets (jwt/ldap/deploy/configdb/historian)
in docs/security.md and note the token option in the ServerHistorian comments.
Mechanism proven live: unseeded token boot fails with SecretNotFoundException at
pre-host expansion; seeded token resolves and boot proceeds. Behavior-neutral
(comment + doc only).
Four continuous-historization types (IHistorizationOutbox, HistorizationOutboxEntry,
IHistorianValueWriter/HistorizationValue, HistorizationCommitMode) drifted into a
ZB.MOM.WW.OtOpcUa.Core.Abstractions.Historian sub-namespace while the other six files
in the same folder correctly use the root namespace. This violated decision #59's
flat-public-surface rule and left InterfaceIndependenceTests.AllPublicTypes_LiveInRootNamespace
red on master (pre-existing, predates round-2 remediation).
Move all four to the root namespace and drop/retarget the now-redundant
'using ...Core.Abstractions.Historian;' in the ~11 consumers. No behavioral change.
Verified: Core.Abstractions.Tests 129/129 (was 128/129), Runtime.Tests Historian+DI 72/72,
Gateway driver unit 102/102, full solution build 0 errors.
Adds <summary>/<param>/<returns>/<inheritdoc> where missing and removes
project bookkeeping IDs (task/tracking refs) from shipped code comments,
so the docs read cleanly and CommentChecker is quiet except for known
false positives (PLC/protocol terms, event/IEqualityComparer inheritdoc).
Doc/comment-only; no logic changed; solution builds clean.
Code review found a residual silent-data-loss path: a single driver ref (mux
ref) can back SEVERAL historized equipment tags via aliasing (identical machines
sharing a register — DriverHostActor._nodeIdByDriverRef is a HashSet), each with
its own HistorianTagname. The muxRef->single-name map collapsed last-wins, so
under alias + divergent overrides only one historian tag got the value and the
rest were silently dropped — the exact failure class FU-3 exists to eliminate.
Model the fan as muxRef -> HashSet<historianName> and append ONE outbox entry per
name in OnValueChangedAsync (a per-name append failure drops only that name and
continues). Convergence removes/adds each (muxRef, name) pair individually from
the per-ref set, dropping the mux key only when its last name is removed — so
removing one alias leaves the shared ref fanning for the others with no mux churn.
Tests: aliased-refs-each-get-the-value (one fan → both historian names written),
removing-one-alias-keeps-the-ref-registered, and the override-rename test now
feeds a value post-rename to prove the write target actually moved to the new
name. Runtime 350/0, OpcUaServer 327/0; 0 warnings.
Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
The continuous-historization recorder conflated two identifiers into one string:
the dependency mux fans DependencyValueChanged keyed by the driver FullName
(the mux ref), but a value must be historized under the resolved historian name
(HistorianTagname override, else FullName). In the common no-override case the
two are equal, so it worked; with an override they diverge and the recorder
registered mux interest under a key the mux never fans — that tag's values were
never captured (and, had they been, would have been written under the mux ref).
Carry BOTH identifiers through the seam: a new HistorizedTagRef(MuxRef,
HistorianName) record on IHistorizedTagSubscriptionSink. The applier resolves
MuxRef = FullName and HistorianName = override-or-FullName. The recorder now
keeps a muxRef->historianName map: it registers/filters mux interest by MuxRef
but writes the outbox entry (and drains) under HistorianName. The convergence
handler re-registers the mux only when the registered key-set changes, so an
override-only rename (same FullName) updates the write target without mux churn.
Tests: a divergent-override recorder test (interest by mux ref, value written
under the override name, never the mux ref) + an override-rename no-churn test;
the applier feed tests now assert the full (mux ref, historian name) pairs.
Runtime 348/0, OpcUaServer 327/0; 0 warnings. Closes FU-3.
Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
The ContinuousHistorizationRecorder was spawned with an EMPTY historized-ref
set, so it registered interest in nothing and historized nothing. This feeds it
the currently-historized tag refs on every address-space deploy/redeploy so its
DependencyMuxActor interest converges to exactly the historized set (the same
refs the EnsureTags provisioning hook resolves: override-or-FullName).
Design — delta convergence (the plan is a pure DIFF):
- New seam IHistorizedTagSubscriptionSink (Core.Abstractions/Historian) with a
Null no-op singleton, mirroring how IHistorianProvisioning decouples the T15
hook. AddressSpaceApplier gains a DEFAULTED ctor param (Null sink) so all ~80
existing call sites + the production site compile unchanged.
- Apply() only ever sees a plan diff (an incremental/surgical apply carries a
delta, not the full set), so the applier feeds an add/remove DELTA computed
from AddedEquipmentTags / RemovedEquipmentTags / ChangedEquipmentTags. The
recorder keeps the full set and re-registers it. The feed is a single
non-blocking Tell behind the sink, wrapped in try/catch so a faulting feed
never blocks or breaks a deploy (same discipline as the provisioning hook).
- Recorder.UpdateHistorizedRefs(added, removed) converges the tracked set, then
— only when it actually changed — sends ONE RegisterInterest with the full set
(the mux's RegisterInterest is a full-REPLACE) or one UnregisterInterest when
it drains to empty (the mux has no per-ref unregister). An unchanged delta is
a no-op (no mux churn).
- DI: the recorder is now spawned BEFORE the applier so the adapter
(ActorHistorizedTagSubscriptionSink) can wrap its IActorRef; the Null sink is
used when continuous historization is off/unwired.
Tests: recorder convergence (add-from-empty, add+remove converge, idempotent,
drain-to-empty unregisters); applier feeds resolved added refs, removed+renamed
deltas, and survives a throwing sink. Build clean (0 warnings on touched
projects); Runtime/OpcUaServer/Gateway/AdminUI suites green.
Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
HistorianGateway is now the sole historian backend (read + alarm SendEvent +
continuous WriteLiveValues). Document the final state and retire the Wonderware
sidecar from the docs/config/labels:
- CLAUDE.md: rewrite the Historian section — ServerHistorian /
ContinuousHistorization / AlarmHistorian config keys, the IHistorianProvisioning
EnsureTags hook, the GatewayAlarmHistorianWriter SendEvent path + ReadEvents
dependency on gateway RuntimeDb:EventReadsEnabled=true, gateway-side
prerequisites (RuntimeDb flags + historian:read/write/tags:write scopes),
migration note, and two KNOWN-LIMITATION callouts (live-validation gate +
empty historized-ref-set recorder follow-on).
- appsettings.json: fix the stale ServerHistorian block (Host/Port/SharedSecret/
ServerCertThumbprint -> Endpoint/ApiKey/UseTls/AllowUntrustedServerCertificate/
CaCertificatePath/CallTimeout, keep MaxTieClusterOverfetch); add a disabled
ContinuousHistorization block; prune the orphaned Wonderware keys from
AlarmHistorian (keep the SQLite knobs). ApiKey env-supplied via
ServerHistorian__ApiKey (commented; valid strict JSON via _comment keys).
- README.md + docs (Historian.md, AlarmHistorian.md, Configuration.md,
ServiceHosting.md, DriverLifecycle.md, drivers/README.md, Uns.md, VirtualTags.md,
AlarmTracking.md, Client.UI.md, README.md, TestConnectProbes.md): retire the
Wonderware historian backend from current-backend descriptions; fix the stale
ServerHistorian/AlarmHistorian config tables (now gateway shape); convert
drivers/Historian.Wonderware.md to a retired stub pointing at the gateway.
- Source/UI labels (descriptive text only, no behavior change):
OtOpcUaServerHostedService.cs, HistoryPaging.cs, OtOpcUaSdkServer.cs,
HistorianAdapterActor.cs, VirtualTagModal.razor, ScriptedAlarmModal.razor,
AlarmsHistorian.razor now name the HistorianGateway backend.
Build clean (0 errors); AdminUI.Tests green (514 passed).
Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
The HistorianGateway driver is now the sole historian read/write+alarm backend, so the
Wonderware sidecar projects are dead code. Removes the 5 Wonderware projects (driver,
.Client, .Client.Contracts, + their 2 test projects) from the solution and tree, and fully
retires the vestigial 'Historian.Wonderware' driver type (UI/probe-only; it had no driver
factory): the Host probe registration, the AdminUI driver-config surface (driver page,
tag-config editor/model/validator entry, address picker/builder, driver-type catalog +
dropdown + edit-router entries), and their tests. Prunes the now-unused Wonderware
connection fields (Host/Port/UseTls/ServerCertThumbprint/SharedSecret) from
AlarmHistorianOptions (keeping Enabled + the SQLite store-and-forward knobs) and refreshes
the stale XML docs that named Wonderware as the production backend.
Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
Addresses T18 review: GatewayHistorianValueWriter is a DI singleton holding a gRPC
channel — make it IAsyncDisposable so the container closes the channel gracefully at
shutdown. Tighten the blank-OutboxPath warning to state startup will fail.
Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
Bind ContinuousHistorizationOptions (Enabled/OutboxPath/CommitMode/
CommitIntervalMs/DrainBatchSize/DrainIntervalSeconds/Capacity/backoff) with a
warn-only Validate(); gated on Enabled AND the ServerHistorian gateway being
configured, the Host registers the durable FasterLogHistorizationOutbox (container
-disposed) + a gateway-backed GatewayHistorianValueWriter, and binds outbox
depth/dropped observable gauges on the central scraped meter. WithOtOpcUaRuntimeActors
spawns the recorder (over the same dependency-mux ref) when the options + writer +
outbox resolve, registering ContinuousHistorizationRecorderKey. Spawned with an EMPTY
historized-ref set: the deployed address space builds later, so ref population is a
documented follow-on (a later SetHistorizedRefs feed) — T18 wires the actor + outbox
+ writer + meters; the ref feed is the known remaining gap.
Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
I-1: Wrap the OnValueChangedAsync AppendAsync in try/catch so a durable-boundary
failure (e.g. a PerEntry fsync hitting disk-full/I-O error) can no longer propagate
out of the handler and trip Akka supervision into a restart loop. A canceled append
during shutdown returns quietly; any other exception increments a new
_outboxAppendFailures counter, logs a Warning (exception type name only), and drops
the value without recording it or nudging the drain. The counter is surfaced on
RecorderStatus (new OutboxAppendFailures field).
I-2: Strengthen Writer_failure_keeps_entry_for_retry to prove the drain actually ran
— assert the writer was invoked (the fake records even on Succeed=false) AND the
outbox stayed at 1 (RemoveAsync not called), via AwaitAssertAsync.
M-3: Capture Sender before the await in the GetStatus handler, then Tell the reply.
M-4: Add Retry_after_writer_failure_eventually_acks proving the retry -> success ->
ack path; FakeValueWriter gains a FailFirstN option + CallCount (Succeed behaviour
unchanged). Short minBackoff keeps it fast and deterministic (AwaitAssert, no sleep).
M-5: Deregister mux interest on PostStop via DependencyMuxActor.UnregisterInterest,
mirroring VirtualTagActor.PostStop, closing the dead-letter window before Terminated.
Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
Continuous-historization engine for non-Galaxy driver tags. Registers
interest with the per-node DependencyMuxActor for the historized refs and
taps the VirtualTagActor.DependencyValueChanged values the mux fans:
coerce to numeric -> append to the durable IHistorizationOutbox (crash
boundary) -> off-thread drain writes batches through IHistorianValueWriter
and acks (FIFO-truncates) on success, backing off (exponential, capped) on
failure. Non-numeric values are dropped + metered (SQL analog path is
numeric-only).
- New seam IHistorianValueWriter + HistorizationValue in Core.Abstractions
so Runtime stays free of the gRPC driver.
- GatewayHistorianValueWriter (driver) adapts IHistorianGatewayClient.
WriteLiveValues: HistorizationValue -> HistorianLiveValue proto, WriteAck
Success||Queued -> true; non-throwing (errors -> false for retry).
- Drain runs via PipeTo(Self) so the mailbox never blocks on the gateway
write; appends awaited on the actor thread to stay serialized.
Adaptation vs plan: the mux fans DependencyValueChanged (TagId/Value/
TimestampUtc, no quality), not DriverInstanceActor.AttributeValuePublished,
so values are recorded Good-quality (192) by the same convention the
scripted-alarm host uses.
Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
Joseph Doherty