The #687/F4 crowding re-read doubled its room up to 8 re-reads with no cap, so a pathological single-ms
cluster on a streaming gateway was streamed several times over for one page. The room is now capped at
ServerHistorian:MaxTieClusterOverfetch (plumbed into GatewayHistorianDataSource as maxPreWindowRows,
default 65536); past it, or once a read has already seen more, the read is BadTooManyOperations with
PreWindowRows set.
Gateway HistoryRead-AtTime passed AVEVA's own interpolation through: live it
answered the previous value as Good at a stored NULL's own time, null + Bad at
a Good sample right after a NULL, a drifted 99.99999999944 for a stored 100,
and no Interpolated bit. It is now served from raw gateway reads and evaluated
with Sql.Common's AtTimeEvaluator, as the SQL backends do (Part 11 §6.5.4).
GatewayAtTimeReader: one <=1,024-row forward window read [first-1ms, last+1ms]
for the batch, a walk-back (density-seeded, x4 growth, whole-remainder shortcut,
bisects dense history) for the first time's prev, and an 18-row probe for the
last time's next; truncated window reads continue or jump with a density-sized
margin. All bounds whole ms; a row counts only strictly inside its request
(AVEVA's boundary row never a sample); Bad rows value-less (#690); never past
now + 1 day; strict-gateway ceiling learned (#691.3) without re-probing.
10 dense times = 1 read (exact) / 3 (interpolated); 10 hourly = 12.
Kill switch ServerHistorian:GatewayAtTimeFromRaw (default true; false = the
old AVEVA passthrough). Known limit: the wire carries no data type, so numeric
tags interpolate as Double (Boolean/integer nodes are not stepped/rounded as on
the SQL backends).
Average, Count, Minimum, Maximum, Range, Start, End, Delta and Minimum/MaximumActualTime are evaluated
on the Gateway backend from forward ReadRaw reads with BucketPlan + AggregateEvaluator, so Gateway
matches SqlServer/QuestDB exactly (values, status bits, interval-start stamps, #677 reverse).
- Window read [start - 1 ms, end] so AVEVA's initial-value row never lands in a bucket; a pre-window
row proves earlier data only with a non-Bad OPC quality (histsdk puts a value on every NULL row).
- Earlier data otherwise from a 2-row [1970, start) probe; later data from a 2-row [end, now+1d]
look-ahead; both ignore rows stamped at their query start/end.
- No gateway stream is ever abandoned (a client-side cancel makes the gateway evict its shared
historian session); past ServerHistorian:MaxTimeWeightedRawRows the bounded stream is drained, then
BadTooManyOperations. The gateway's max_values-ceiling InvalidArgument maps to the same limit; a
caller cancellation is not a backend fault.
- TimeAverage/Total stay AVEVA-native (no prior-sample read for exact Part 13 bounds), now stamped with
the interval start (PENDING the live leg: histsdk's tail+43 start_time field is unverified) and
carrying the Calculated bits.
- Served set: GatewayHistorianDataSource.ServedAggregates (12). Live AVEVA leg open (VPN).
Build: 0 errors solution-wide, and 0 warnings from every project this branch
touches. The ~816 solution-wide warnings are pre-existing xUnit1051 /
OTOPCUA0001 / CS86xx in untouched driver + client test projects.
Tests: full solution run compared against a full run on a detached worktree at
the pre-branch baseline 2e46d054. The two failure SETS are identical -- all 13
tests, same names, zero new failures. Net +26 tests: +3 Core.AlarmHistorian
(drain gate), +6 Runtime (role view), +17 Host.IntegrationTests (migrator +
convergence). Set comparison rather than counts, because the suite carries
standing environment- and load-dependent failures a count would hide.
The greps found real drift Task 6 missed -- eight live sites still naming the
deleted SqliteStoreAndForwardSink, including the AdminUI /alarms/historian
panel text, which is user-visible, and a <see cref> in HistorianAdapterActor
that resolved to nothing without warning. All repointed at
LocalDbStoreAndForwardSink; CLAUDE.md's alarm-history paragraph now also
records the LocalDb buffer and the primary-gated drain, and drops DatabasePath
from the knob list. docs/AlarmTracking.md still promised an
AlarmHistorianOptions.Validate() startup warning for a relative DatabasePath
and an empty SharedSecret; both branches are gone, so it now says so.
Code references to AlarmHistorian:DatabasePath reduce to exactly two intentional
ones: AlarmSfLegacyMigrator.LegacyPathKey and its test. No `new SqliteConnection`
remains anywhere in Core.AlarmHistorian.
Recon doc gains the durable verification record: guard-deletion evidence for
both vacuous passes, the two exact-set replicated-table pins (both assert set
equality, so an added or a dropped registration fails), and the baseline test
comparison with a per-failure account of why each of the 13 is not this
branch's.
Stops here per the plan. Task 8's live gate needs explicit go-ahead; nothing on
this branch is to be merged.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
PR #423 shipped GatewayTagProvisioner + unit tests but never registered it in
DI nor passed it into the AddressSpaceApplier, so deploying historized tags used
the no-op NullHistorianProvisioning and never called the gateway's EnsureTags
(confirmed live on wonder-app-vd03: zero EnsureTags calls on a historized deploy).
Addresses HISTORIAN-GATEWAY-INTEGRATION-ISSUES.md.
Issue 1 (wire provisioner):
- Runtime: AddHistorianProvisioning extension (gated on ServerHistorian:Enabled,
mirrors AddServerHistorian) + NullHistorianProvisioning TryAdd default in
AddOtOpcUaRuntime; WithOtOpcUaRuntimeActors resolves IHistorianProvisioning and
passes it into the applier.
- Gateway driver: GatewayHistorian.CreateProvisioner factory (mirrors CreateDataSource).
- Host: Program.cs calls AddHistorianProvisioning after AddServerHistorian.
- Tests: AddHistorianProvisioningTests (config-gated registration + the
register->resolve->applier->EnsureTags chain).
Issue 2 (observability): AddressSpaceApplier logs the provisioning tally on every
successful dispatch (was gated behind Failed/Skipped > 0), including dispatched=N
so a dispatched=N/requested=0 line flags the dormant no-op. +2 tests.
Issue 3 (30s HistoryRead on unprovisioned tags): root coupling fixed by Issue 1;
documented the CallTimeout knob + coupling. Default left at 30s pending the
multi-data-point investigation the issue requests (lowering risks truncating
legitimate large reads).
Issue 4 (docs): docs/Historian.md gains a "Tag auto-provisioning (EnsureTags)"
section and CLAUDE.md a wiring/gating note (both stress ServerHistorian:Enabled).
Sibling scadaproj/CLAUDE.md carries no false claim -> unchanged.
Pre-existing Serilog observation: anchor CWD to AppContext.BaseDirectory before
AddZbSerilog so the relative file sink stops landing in C:\Windows\System32 under
the Windows-service CWD.
Builds 0-error; Runtime.Tests 355, OpcUaServer.Tests 329, Gateway.Tests 99 (+4
live-skipped) all green.
Joseph Doherty