Both defects were found by #491's live gate against a real HistorianGateway and filed rather than
fixed at the time, because each one's fix is a design decision rather than a patch.
#537 — both nodes of a redundant pair captured every value, so the historian received duplicate
live samples for every historized tag (measured: the gateway's write counter advanced twice per
value and halved when one central was stopped).
The recorder now reads a capture gate per value, fed from the new
PrimaryGatePolicy.ShouldRecordHistorizedValues through IHistorizationRoleView — a third verdict on
the same RedundancyRoleView object, registered unconditionally on driver nodes rather than
alongside the alarm section's IRedundancyRoleView (which only exists when that section is enabled;
piggybacking is the #511 registration accident).
Two things are deliberate and easy to get backwards:
* It gates the CAPTURE, not the drain — the opposite of the alarm sink. That queue is in LocalDb
and replicates, so the Secondary already holds the Primary's rows. The historization outbox is
a node-local FasterLog replicating to nobody, so a drain-only gate would leave the standby
appending forever to a queue no one empties: unbounded disk, then a flood of stale duplicates
the moment it is promoted. Entries a node already captured are still drained by that node.
* The peer it stands down for must RECORD THE SAME REFS, not merely hold the same rows — a
Primary in this node's own per-cluster mesh. A node carrying no cluster-scoped role cannot
establish that and never stands down: under a fleet-wide election the elected Primary is
usually running a different cluster's configuration, and deferring to it would end this
cluster's historization silently, with the drain reporting success on an empty outbox forever.
That is the scope mismatch that once suspended the alarm drain on every node of the rig.
An unknown role captures, matching the alarm drain — a duplicate sample is recoverable, a hole in
the trend is not.
#538 — the drain writes every per-tag group and then acks all-or-nothing (deliberate: RemoveAsync
truncates THROUGH an id, so partial acking could drop older unwritten entries). One permanently
failing tag therefore pinned the FIFO head while every healthy tag in the batch was re-submitted
to the historian on every pass, indefinitely — duplicate samples and an outbox that never
truncated, observed live at 8.3 MB from a single unprovisionable tag.
The drain now tracks consecutive failures per historian tag and abandons that tag's entries once
it exceeds MaxWriteAttempts (new, default 5), so the head advances. The counter only advances on a
pass where some OTHER tag wrote successfully: that is what separates "this tag is poison" from
"the backend is down". A count-only quarantine looks identical on the poison-tag case and silently
discards the whole backlog during an ordinary gateway restart, turning a duplicate-samples bug
into a lost-history one. Abandonment is also deferred until every failing tag is over budget, so
nothing is discarded on a pass that could not have advanced the head anyway. Each abandonment logs
a Warning naming the tag; the total rides RecorderStatus.AbandonedEntries.
Capacity also gains a non-zero default (500,000 entries) as the disk backstop for the one case the
quarantine deliberately cannot resolve — a historian refusing every write, where retaining is
correct. Set 0 to restore unbounded retention.
Every new test was falsified by reverting the production change it guards, and each one reddens
alone:
* drop the backend-reachable evidence -> only A_backend_that_fails_every_tag_abandons_nothing
* never abandon -> only A_permanently_failing_tag_is_abandoned_so_healthy_tags_stop_being_rewritten
* latch the capture gate at construction -> only The_capture_gate_is_re_read_per_value...
* treat any Primary as a scope peer -> only the two scope-mismatch cases
Full suite green. (Four OpcUaClient.Browser failures during the run were the opc-plc fixture being
down on the shared Docker host; brought it back up and they pass.)
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
The 2026-08-01 run closed every leg except read-back, which no reachable historian could
serve that day (the LAN instance accepts writes but serves no reads under a licence tag-count
violation; the licensed one needed the VPN). With the VPN up the run was repeated against
wonder-sql-vd03 and the remaining legs pass:
- EnsureTags provisioning: ensured=2, failed=0 — the path the LAN box could not exercise at
all (it is RemoteGrpc-only and that CAP was not listening).
- Read-back: two Modbus registers carrying DISTINCT NON-ZERO constants were historized, and
each read back as its own value (HR20 -> 8224, HR3 -> 120, quality 192). A first attempt
read back 0, which proves nothing — the same trap #536 recorded, so the proof was rebuilt
on values a crossed or fabricated answer could not fake.
Found during the run and FILED rather than fixed:
- #538 — one permanently-failing tag re-writes every healthy tag on every drain pass, forever.
DrainOnceAsync writes EVERY per-tag group and then acks all-or-nothing, so a tag whose
historian tag does not exist stops the FIFO head advancing while the healthy tags' values
are re-submitted every drain interval; the outbox never truncates (Capacity defaults to 0 =
unbounded, observed at 8.3 MB). The all-or-nothing ack is deliberate — RemoveAsync truncates
THROUGH an id, so partial acking could drop older unwritten entries — so the fix changes the
outbox contract and does not belong in this commit.
Also re-confirmed live: EnsureTags provisioning fires only for newly-ADDED historized tags, so
flipping an existing tag's historianTagname provisions nothing (Issue 5 of
2026-06-27-historian-gateway-integration-issues.md).
No production code changes — the recorder fix shipped in ad744734.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
Ran #491's live gate against a real ZB.MOM.WW.HistorianGateway talking to a real AVEVA
Historian, and it found a production-fatal defect on the first value it ever saw.
ContinuousHistorizationRecorder.OnValueChangedAsync touched `Self` after awaiting
IHistorizationOutbox.AppendAsync(...).ConfigureAwait(false). ConfigureAwait(false) discards
Akka's ActorTaskScheduler synchronization context, so `Self` — which reads ActorBase.Context —
threw NotSupportedException. The throw escaped into supervision, which RESTARTED the actor;
because production seeds an EMPTY ref set (the deployed set arrives later via
UpdateHistorizedRefs), the restart silently discarded every historized ref. Net effect: one
value captured, then nothing, with no further error to show for it. Fixed by capturing
`IActorRef self = Self` before the await — the idiom the GetStatus handler already applied to
Sender.
Thirteen unit tests missed it because InMemoryOutbox.AppendAsync returned
ValueTask.CompletedTask: a completed ValueTask never yields, so the continuation ran inline
with the context still installed. The production FasterLog outbox fsyncs per entry and does
yield. The first regression test was itself vacuous — `await Task.Yield()` posts back to the
CAPTURED context, so the append still completed on the actor thread and reverting the fix left
the test green. The fake must complete on a POOL thread (Task.Delay(1).ConfigureAwait(false)),
as a real fsync does; with that, reverting the fix reddens exactly one test.
The rig could not run this gate at all: docker-dev carried ServerHistorian__* but no
ContinuousHistorization__* keys, so the recorder could never spawn. Both centrals now carry
them (default OFF — the recorder needs ContinuousHistorization:Enabled AND
ServerHistorian:Enabled), with the outbox on the per-node named volume so it survives a
container recreate. ServerHistorian__UseTls was hard-coded true and is now parameterized; a
plaintext h2c gateway was unreachable without it.
Gate results: outbox append, drain, WriteLiveValues accepted by a real historian, and restart
convergence (capture resumes after a restart with no new deployment) all pass. The read-back
hop is NOT proven — the LAN historian accepts writes but serves no reads under a "License
violation: Database tag count exceeds license tag count", and the licensed corporate one needs
the VPN. EnsureTags provisioning is likewise unexercised there (RemoteGrpc-only, and that CAP
is not listening). #491 stays open for the read-back hop alone.
Also recorded, not fixed: the recorder has NO Primary gate, unlike the alarm store-and-forward
sink, so both nodes of a redundant pair send duplicate live values. Gating it would introduce a
failover window with no capture — that is a design decision, not a cleanup.
Evidence: docs/plans/2026-08-01-continuous-historization-live-gate.md
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
Adds deferment.md — a source-verified inventory of new-driver status, all 17
open issues, in-code deferrals, open live gates and plan bookkeeping. Every
claim was checked against src/ and git, not against documentation.
Headline finding: three subsystems are authored, persisted and shipped but
never executed —
* Node ACLs: IPermissionEvaluator/TriePermissionEvaluator have zero
production consumers and OtOpcUaNodeManager never references them, yet
ClusterAcls.razor authors NodeAcl rows and ConfigComposer.cs:51 ships them
in every artifact. An authored deny rule has no effect.
* IRediscoverable / IHostConnectivityProbe raise into the void (#518/#507);
nothing ever writes a DriverHostStatus row.
* DriverTypeRegistry is vestigial, so no factory passes a tier and every
driver runs Tier A with the Tier-C protections dormant.
Also records the Calculation driver as picker-visible but unauthorable
(DriverConfigModal has no case) — the "registered but unauthorable" class
recurring after the Sql picker defect — and notes that no parity test guards
DriverConfigModal/DeviceModal, which is why it survived review.
Documentation corrections (source-verified):
* ReadWriteOperations.md claimed "a denied read never hits the driver" via
four types that do not exist in src/. Bannered + struck through; a security
review reading that page would have concluded a per-node ACL gate exists.
* CLAUDE.md: the Change Detection sentence was false (DriverHost consumes
nothing); the mesh Phase 4 and Phase 5 live gates and the auto-down 1-vs-1
gate had all PASSED; the ScriptedAlarmState table was already dropped.
* driver-expansion tracking: Modbus RTU and SQL poll are merged, not
pending — its command table would have made someone rebuild two shipped
drivers in a fresh worktree.
* drivers/README.md: dead DriverTypeRegistry paragraph, retired
SystemPlatform namespace kind, missing Sql + Calculation rows, missing
Modbus RTU-over-TCP transport.
* TwinCAT.md/Galaxy.md promised an address-space rebuild that never happens.
* Historian.md gained the #491 unproven-value-capture pointer.
* IncrementalSync.md and AddressSpace.md bannered as wholesale v2-era (the
rewrite is recorded as still owed, not done here); four v2 status docs
bannered as historical; three wrong-name-for-a-live-type fixes.
Left deliberately untouched: the secrets NoOpSecretReplicator line, which
makes a security claim and needs the real behaviour identified rather than a
rename.
The historian's Int1 analog-tag creation path is server-degenerate:
EnsureTags(Int1) stores an unusable stub (live-probed on the deployed
2023 R2 historian; root-caused + documented gateway-side in
HistorianGateway #11 / PR #16). HistorianTypeMapper mapped
DriverDataType.Boolean -> HistorianDataType.Int1, so Boolean historized
tags failed auto-provisioning.
Map Boolean -> Int2 instead (store 0/1). Int2 is a proven-creatable
analog type, and the value-write path already sends every value as a
double on WriteLiveValues regardless of tag type, so Boolean 0.0/1.0
round-trips cleanly. Both Float and Int2 are supported, so a
pre-existing Float Boolean tag retypes cleanly on its next deploy.
- HistorianTypeMapper: Boolean => Int2 (+ remarks)
- Unit pins: HistorianTypeMapperTests, GatewayTagProvisionerTests
(Boolean_definition_carries_explicit_Int2_presence)
- Live gate: EnsureTags_boolean_sandbox_provisions_as_Int2 (was _Int1);
retype probe reframed Float->Int2 (drops the dead Int1 leg)
- Docs: docs/Historian.md 'Boolean historization maps to Int2', CLAUDE.md,
archreview STATUS.md (R2-06 live gate -> 6/6)
Resolves#441.
- StorageRateMs bug found+fixed (PR #439); live suite 3/6 -> 5/6
- U-7: EnsureTags is a real upsert; supported-type retype ACCEPTED (Float->Double);
Int1 not creatable on this histsdk (ProtocolEvidenceMissingException, Int1-only)
- record the observed retype policy in docs/Historian.md (fills the placeholder)
- pending.md, current.md: git rm --cached (untrack, keep on disk) + root-anchored
.gitignore entries — making the files' own declared 'never stage' rule true.
- looseends.md: deleted (2026-05-18 snapshot; sole open item + all follow-ups
target the retired Wonderware historian sidecar rig, moot since the gateway
cutover; recoverable from history).
- stillpending.md -> docs/plans/2026-06-18-stillpending-backlog-audit.md (git mv;
referenced by docs/Historian.md + memory index, preserved under dated-plan convention).
- HISTORIAN-GATEWAY-INTEGRATION-ISSUES.md -> docs/plans/2026-06-27-historian-gateway-integration-issues.md (git mv).
- docs/Historian.md §'Closed backlog §2 items' repointed at the archived path.
Historical docs/plans/* references to the bare stillpending.md filename are frozen
point-in-time records, left untouched per the documentation-audit convention.
PR #423 shipped GatewayTagProvisioner + unit tests but never registered it in
DI nor passed it into the AddressSpaceApplier, so deploying historized tags used
the no-op NullHistorianProvisioning and never called the gateway's EnsureTags
(confirmed live on wonder-app-vd03: zero EnsureTags calls on a historized deploy).
Addresses HISTORIAN-GATEWAY-INTEGRATION-ISSUES.md.
Issue 1 (wire provisioner):
- Runtime: AddHistorianProvisioning extension (gated on ServerHistorian:Enabled,
mirrors AddServerHistorian) + NullHistorianProvisioning TryAdd default in
AddOtOpcUaRuntime; WithOtOpcUaRuntimeActors resolves IHistorianProvisioning and
passes it into the applier.
- Gateway driver: GatewayHistorian.CreateProvisioner factory (mirrors CreateDataSource).
- Host: Program.cs calls AddHistorianProvisioning after AddServerHistorian.
- Tests: AddHistorianProvisioningTests (config-gated registration + the
register->resolve->applier->EnsureTags chain).
Issue 2 (observability): AddressSpaceApplier logs the provisioning tally on every
successful dispatch (was gated behind Failed/Skipped > 0), including dispatched=N
so a dispatched=N/requested=0 line flags the dormant no-op. +2 tests.
Issue 3 (30s HistoryRead on unprovisioned tags): root coupling fixed by Issue 1;
documented the CallTimeout knob + coupling. Default left at 30s pending the
multi-data-point investigation the issue requests (lowering risks truncating
legitimate large reads).
Issue 4 (docs): docs/Historian.md gains a "Tag auto-provisioning (EnsureTags)"
section and CLAUDE.md a wiring/gating note (both stress ServerHistorian:Enabled).
Sibling scadaproj/CLAUDE.md carries no false claim -> unchanged.
Pre-existing Serilog observation: anchor CWD to AppContext.BaseDirectory before
AddZbSerilog so the relative file sink stops landing in C:\Windows\System32 under
the Windows-service CWD.
Builds 0-error; Runtime.Tests 355, OpcUaServer.Tests 329, Gateway.Tests 99 (+4
live-skipped) all green.
HistorianGateway is now the sole historian backend (read + alarm SendEvent +
continuous WriteLiveValues). Document the final state and retire the Wonderware
sidecar from the docs/config/labels:
- CLAUDE.md: rewrite the Historian section — ServerHistorian /
ContinuousHistorization / AlarmHistorian config keys, the IHistorianProvisioning
EnsureTags hook, the GatewayAlarmHistorianWriter SendEvent path + ReadEvents
dependency on gateway RuntimeDb:EventReadsEnabled=true, gateway-side
prerequisites (RuntimeDb flags + historian:read/write/tags:write scopes),
migration note, and two KNOWN-LIMITATION callouts (live-validation gate +
empty historized-ref-set recorder follow-on).
- appsettings.json: fix the stale ServerHistorian block (Host/Port/SharedSecret/
ServerCertThumbprint -> Endpoint/ApiKey/UseTls/AllowUntrustedServerCertificate/
CaCertificatePath/CallTimeout, keep MaxTieClusterOverfetch); add a disabled
ContinuousHistorization block; prune the orphaned Wonderware keys from
AlarmHistorian (keep the SQLite knobs). ApiKey env-supplied via
ServerHistorian__ApiKey (commented; valid strict JSON via _comment keys).
- README.md + docs (Historian.md, AlarmHistorian.md, Configuration.md,
ServiceHosting.md, DriverLifecycle.md, drivers/README.md, Uns.md, VirtualTags.md,
AlarmTracking.md, Client.UI.md, README.md, TestConnectProbes.md): retire the
Wonderware historian backend from current-backend descriptions; fix the stale
ServerHistorian/AlarmHistorian config tables (now gateway shape); convert
drivers/Historian.Wonderware.md to a retired stub pointing at the gateway.
- Source/UI labels (descriptive text only, no behavior change):
OtOpcUaServerHostedService.cs, HistoryPaging.cs, OtOpcUaSdkServer.cs,
HistorianAdapterActor.cs, VirtualTagModal.razor, ScriptedAlarmModal.razor,
AlarmsHistorian.razor now name the HistorianGateway backend.
Build clean (0 errors); AdminUI.Tests green (514 passed).
Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
C1 (critical): a boundary tie cluster larger than NumValuesPerNode could
silently truncate a resumed read to GoodNoData, permanently dropping the
un-emitted ties — the (timestamp, skip) cursor cannot advance past a single
timestamp the fixed-(start,end,cap) backend keeps re-returning. Now detected
and failed LOUDLY per node with BadHistoryOperationUnsupported + a log naming
the tag/timestamp/cap; documented in Historian.md with the larger-cap remedy.
Regression test Raw_tie_cluster_larger_than_page_fails_loudly_not_silently.
I3: build HistoryData before Save() so a projection failure can never orphan a
stored continuation cursor.
N1 (YAGNI): drop the never-produced HistoryReadKind enum + Processed-only
Aggregate/IntervalTicks fields from HistoryContinuationState — only Raw pages.
N3: ComputeResumeCursor guards its documented non-empty precondition.
I1: document InMemoryHistoryContinuationStore's eventual-consistency (test double).
Build clean, 182/182 OpcUaServer tests pass.
The Wonderware historian backend is single-shot — it returns up to
NumValuesPerNode samples with a null continuation point — so paging is
synthesised server-side, time-based, for the only count-capped arm (Raw):
- A full page (count == NumValuesPerNode, NumValuesPerNode > 0) emits an
opaque 16-byte continuation point and stores a resume cursor; a short page
(or NumValuesPerNode == 0 "all values") emits none.
- A resume read takes the stored cursor, reads the next page from the boundary
forward, and emits a fresh CP only if that page is also full.
- The resume cursor is tie-safe (HistoryPaging.ComputeResumeCursor /
TrimBoundaryDuplicates): the next page resumes from the boundary timestamp
INCLUSIVE and drops the head ties already returned, so samples sharing the
boundary SourceTimestamp are neither duplicated nor skipped.
Continuation points are bound to the OPC UA session via the SDK's
ISession.SaveHistoryContinuationPoint / RestoreHistoryContinuationPoint store
(SessionHistoryContinuationStore) — capped by ServerConfiguration.
MaxHistoryContinuationPoints (default 100, oldest-evicted) and disposed on
session close. releaseContinuationPoints is honoured via an override of
HistoryReleaseContinuationPoints (the base dispatcher routes release-only reads
there, never to the per-details arms). An unknown / evicted / released point
resumes to BadContinuationPointInvalid.
Processed and AtTime stay single-shot: neither details type carries a client
count cap, so the single-shot backend returns the complete result in one read
and there is no "full page" signal to page on (spec-conformant). Modified-value
history remains out of scope.
The pure paging decisions + CP store contract are unit-tested via HistoryPaging
+ InMemoryHistoryContinuationStore; the full multi-page round trip is driven
end-to-end through the node manager with an in-memory store + a series-backed
fake historian (the in-process harness is session-less).
Joseph Doherty