Commit Graph

62 Commits

Author SHA1 Message Date
Joseph Doherty dbffb6bdb4 fix(historian): #691.3 strict-ceiling gateway: re-ask once at the named ceiling, never answer a truncated read 2026-10-12 16:13:14 -04:00
Joseph Doherty 39552e15b9 fix(historian): #690 Gateway raw/AtTime Bad rows carry no value (a stored NULL was value 0) 2026-10-12 16:08:54 -04:00
Joseph Doherty a3ee917fb4 fix(historian): #687 re-review — a row at/before the Unix epoch is never a stored sample
With StartTime at or before the epoch, RequestStartFor has no margin left, so
AVEVA's synthetic no-data/boundary row is stamped exactly at StartTime and was
kept: raw reads returned a fabricated Bad sample at 1970-01-01 and the per-sample
aggregates put it in bucket 0 (Start/End fabricated, Min/Max status Uncertain).
IsWindowSample and the aggregate loop now drop any row with t <= ProbeFloor
(RawAggregateReader.IsAfterFloor), matching the #686 walk-back's assumption.
2026-10-12 15:47:39 -04:00
Joseph Doherty 8114f13a28 fix(historian): #687 review — one bounded re-read instead of failing, raw budget unclamped, exclusive forward end
- M1/M2: when the 1 ms pre-window margin (boundary row + stored samples, e.g. a
  >16-sample tie cluster at start-1ms / lastTs-1ms) uses up the cap + 16 budget,
  re-read ONCE asking for cap + m + 16 (m = pre-window rows seen); fail
  BadTooManyOperations only if the re-read is still short because of them. The
  raw budget is no longer clamped at the gateway default ceiling (AVEVA always
  emits the boundary row, so page sizes 999,985-1,000,000 always came back
  short); the #688 clamp stays on the per-sample aggregate path only.
- Merge trap: NumValuesPerNode 0 => int.MaxValue, never through the clamp.
- L3: the forward end is exclusive ([start, end), the IHistorianDataSource
  contract); a point read (start == end, the tie-cluster over-fetch) inclusive.
- RequestStartFor never goes before the Unix epoch (#686's walk-back floor).
- L4 (docs): an Uncertain native row for an all-NULL interval is unverified live.
2026-10-12 15:19:34 -04:00
Joseph Doherty 6796ba4126 fix(historian): #688 clamp the Gateway per-sample row budget at the gateway's default ceiling
The default MaxTimeWeightedRawRows (1,000,000) + 18 exceeded the gateway's
default MaxValuesCeiling (1,000,000), so with AllowStreamingBeyondCeiling=false
every Gateway per-sample aggregate read was refused. The allowance is now clamped
at 1,000,000 while the cap is at or under it (RawAggregateReader.
ClampToGatewayCeiling); a cap above it keeps its allowance. Raw reads are not
clamped (rebased onto #686).
2026-10-12 15:15:35 -04:00
Joseph Doherty 9e47532283 fix(historian): #687 Gateway raw reads drop AVEVA's synthetic start-boundary row
An AVEVA Full raw read leads with a row stamped at the query start (the value in
effect, Quality=133, or a no-data NULL row); it was returned as a stored sample.
Raw reads now use the #681 classification (RawAggregateReader.RequestStartFor /
IsWindowSample — one authority, by timestamp, since a stored NULL looks the same
on the wire): request from 1 ms early with NumValuesPerNode + 16 rows, keep rows
at or after StartTime, drain the whole stream. A full page still holds exactly
the cap, so #669 paging/resume/tie-cluster reads are unchanged; a margin that
swallows the allowance fails the node rather than ending paging early.
ReturnBounds is not plumbed on any backend (documented).
2026-10-12 15:15:16 -04:00
Joseph Doherty 44a03bedc0 fix(historian): #685 Gateway native TimeAverage/Total — drop AVEVA's pre-window row, keep Uncertain values
AVEVA answers one extra row for the interval before the window; it was mapped as
a result stamped before StartTime (N+1 values). Rows whose interval lies outside
the requested span (either direction) are now dropped after the drained stream.
An Uncertain (opc 64-191) interval kept no value (opc >= 192 was the only data
test); it now carries its value as UncertainDataSubNormal | Calculated.
Docs: AVEVA's TWA excludes NULL-gap time while the SQL backends bridge it;
end-to-end per-interval values also need historiangw#27.
2026-10-12 15:14:56 -04:00
Joseph Doherty cf1af55720 fix(historian): #686 review — bounded, density-aware reverse raw walk; uncapped Gateway raw reads
1. Uncapped reverse raw read on a forward-only backend = ONE forward read of the
   window, filtered and reversed in memory (was a segment-by-segment walk).
2. Segment width is aimed at ~half the row budget from the density just observed
   (overflow: from the truncated read's own sample span) and carried page to page
   in HistoryContinuationState.SegmentWidthTicks.
3. The walk starts no later than now + 1 day and never reads before the epoch.
4. Request bounds are rounded outward to the millisecond, so a row AVEVA
   fabricates at a rounded-up query start cannot land inside the segment.
5. Budget M = max(1, MaxTieClusterOverfetch); reads ask M + 2 rows, completeness is
   'short OR last row past the segment', and a read crowded by rows before the
   segment is re-read once (<= 2M + 4). A cluster of exactly M, neighbouring
   clusters each within M, and M <= 1 are all served; only one timestamp over M
   is refused. ServerHistorianOptions text updated.
6. The raw and events tie-path oversized-cluster refusal is BadTooManyOperations
   (was BadHistoryOperationUnsupported), consistent with the walk.
7. GatewayHistorianDataSource.ReadRawAsync: a caller cancellation is not a backend
   failure; NumValuesPerNode = 0 is sent as int.MaxValue (the client package and
   the gateway reject max_values <= 0, so every uncapped Gateway raw read failed);
   a MaxValuesCeiling refusal is a HistoryReadLimitExceededException.
8. Tests: upper-edge filter, epoch floor, look-ahead, ms alignment, exact/adjacent
   clusters, minimal budget, carried density, and the walk over the real Gateway
   source + FakeHistorianGatewayClient (OpcUaServer internals visible to it).
2026-10-12 15:11:21 -04:00
Joseph Doherty 410c8d8976 fix(historian): #686 serve reverse/end-only raw reads on forward-only backends (Gateway)
#669 turned an end-only capped HistoryRead-Raw into a reverse read (start > end)
for every backend; the gateway rejects start > end (InvalidArgument), so on the
default Gateway backend every end-only capped and every explicit reverse raw read
failed the node and was counted as a backend health failure.

- IHistorianDataSource.SupportsReverseRaw (default false; SqlServer/QuestDB/Null
  true, Gateway false; guard test pins that every production backend declares it).
- OtOpcUaNodeManager sends a reverse window only to a backend that declares it;
  otherwise ReverseRawReader walks (lo, hi] back in bounded forward segment reads
  (1 min, x4 while sparse; each <= MaxTieClusterOverfetch + 1 rows; a full read is
  re-read narrower, never answered truncated; filtered on sample timestamps, which
  also drops AVEVA's fabricated query-start rows; stops at the Unix epoch). A
  one-timestamp tie cluster over the bound => BadTooManyOperations.
- ServeRawPagedAsync maps HistoryReadLimitExceededException to BadTooManyOperations.
- GatewayHistorianDataSource.ReadRawAsync refuses start > end unsent, without
  recording a health failure.
2026-10-12 14:42:27 -04:00
Joseph Doherty 5f225f0b44 fix(historian): #681 on #682/#684 — Gateway served set is the 12-member ServedAggregates; raw buckets use Good-only values
- #682: ServedAggregates (ten raw-evaluated per-sample + TimeAverage/Total) replaces the placeholder
  AggregateModeMapper.Served; NativeAggregates is derived from the mapper's switch. The Gateway
  expectations in AggregateModeMapperTests, HistorianBackendRegistryTests and
  AdvertisedAggregateFunctionsWireTests move to the 12-member set (the wire tests' 'not served by
  Gateway' example is now TimeAverage2).
- #684: RawBucketAccumulator applies the SQL backends' Good-only value columns (Avg/Min/Max/Sum and
  the ActualTime extreme over Good numeric samples), UncertainMin/UncertainMax over Uncertain numeric
  samples, and GoodNonNumeric. The parity oracle carries the #684 InMemoryRows and the DenseUncertain /
  UncertainBeyond / UncertainText datasets.
2026-10-12 10:00:30 -04:00
Joseph Doherty 78f604e0cd feat(historian): Gateway serves the per-sample aggregates from raw reads with the shared Sql.Common evaluator (#681)
Average, Count, Minimum, Maximum, Range, Start, End, Delta and Minimum/MaximumActualTime are evaluated
on the Gateway backend from forward ReadRaw reads with BucketPlan + AggregateEvaluator, so Gateway
matches SqlServer/QuestDB exactly (values, status bits, interval-start stamps, #677 reverse).

- Window read [start - 1 ms, end] so AVEVA's initial-value row never lands in a bucket; a pre-window
  row proves earlier data only with a non-Bad OPC quality (histsdk puts a value on every NULL row).
- Earlier data otherwise from a 2-row [1970, start) probe; later data from a 2-row [end, now+1d]
  look-ahead; both ignore rows stamped at their query start/end.
- No gateway stream is ever abandoned (a client-side cancel makes the gateway evict its shared
  historian session); past ServerHistorian:MaxTimeWeightedRawRows the bounded stream is drained, then
  BadTooManyOperations. The gateway's max_values-ceiling InvalidArgument maps to the same limit; a
  caller cancellation is not a backend fault.
- TimeAverage/Total stay AVEVA-native (no prior-sample read for exact Part 13 bounds), now stamped with
  the interval start (PENDING the live leg: histsdk's tail+43 start_time field is unverified) and
  carrying the Calculated bits.
- Served set: GatewayHistorianDataSource.ServedAggregates (12). Live AVEVA leg open (VPN).
2026-10-12 09:56:54 -04:00
Joseph Doherty ede8610804 refactor(historian): Gateway served set behind GatewayHistorianDataSource.ServedAggregates (#682, merge seam for #681)
Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-12 08:57:19 -04:00
Joseph Doherty 6ea3ba44da feat(historian): advertise only the backend's aggregates under AggregateFunctions (#682)
IHistorianDataSource.SupportedAggregates (default: every HistoryAggregateType)
is the one authority for both ServerCapabilities/AggregateFunctions (i=2997) and
HistoryServerCapabilities/AggregateFunctions (i=11201) and for the up-front
BadAggregateNotSupported in HistoryRead-Processed. Gateway declares
AggregateModeMapper.Served (4), SqlServer/QuestDB AggregateFamilies.Served (17),
Null none. Only the folder references are trimmed; the SDK's calculator
factories stay so live AggregateFilter subscriptions keep working.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-12 08:52:47 -04:00
Joseph Doherty fb5ec80fe6 fix(historian): #679 Gateway — Average/Count not supported (no AVEVA equivalent), TimeAverage → TimeWeightedAverage 2026-10-11 15:19:21 -04:00
Joseph Doherty 945ec2c069 Merge feat/677-aggregates: reverse processed reads SDK-anchored, start-of-data Partial (forward), AggregateConfiguration accept-defaults, IntegralDivisor=1, SDK parity tests (#677) 2026-10-11 11:47:52 -04:00
Joseph Doherty 93d6c1da67 fix(historian): #675 review follow-ups — SupportsReverseEvents capability, no catch-as-unsupported
- IHistorianDataSource.SupportsReverseEvents (default false; SqlServer/QuestDB true; Gateway/Null declare false).
  The node manager checks it BEFORE calling the source: an end-only capped events read becomes reverse only on
  a supporting backend (Gateway, the default, keeps the pre-#675 forward [MinValue, EndTime) paging), and an
  explicit reverse read on a non-supporting backend is BadHistoryOperationUnsupported without a call.
- Removed the broad catch (NotSupportedException) in the events path: a NotSupportedException from inside a
  backend is now logged as a backend fault (Error), not an 'unsupported' Warning.
- Null declares the reverse overload itself; guard test: every production IHistorianDataSource declares both
  the reverse overload and the capability (positive-control scan of all four backends).
- Conformance: capability reported + oversized reverse tie cluster (forward over-fetch reversed == reverse
  read order; capped reverse read stopping inside the cluster flags truncation) on SqlServer + QuestDB.
- docs/Historian.md: end-only capped events read is backward on SqlServer/QuestDB, forward on Gateway.
2026-10-11 11:36:12 -04:00
Joseph Doherty 9120182314 Merge master into feat/677-aggregates
# Conflicts:
#	docs/Historian.md
2026-10-11 11:30:46 -04:00
Joseph Doherty 27ca6cd791 feat(historian): #677 Part 13/SDK reverse anchoring, start-of-data Partial, AggregateConfiguration, IntegralDivisor
- BucketPlan: reverse processed reads anchored at the later time for every
  aggregate: window (end, start], partial interval earliest, results stamped
  with the later edge, latest first (SDK CreateSlice/GetTimestamp/backward
  GetValues). SqlServer adds @bucketAdjustTicks + flips the window predicate;
  QuestDB binds BucketBaseUnixTicks. IndexOf is the C# statement of the math.
- Partial bit on the interval the tag's data starts inside (SDK m_startOfData),
  every aggregate but Average; per-sample reads fetch the earliest sample.
- Node manager: non-default client AggregateConfiguration =>
  BadAggregateConfigurationRejected per node (ServerAggregateConfiguration,
  pinned to the SDK AggregateManager defaults).
- GatewayTagProvisioner sends IntegralDivisor = 1.
- SDK parity oracle (Opc.Ua.Server calculators) in unit + both backends'
  conformance suites, forward and reverse; remaining divergences documented.
- docs/Historian.md + CLAUDE.md release note (client-visible).
2026-10-11 11:26:00 -04:00
Joseph Doherty 8d154c6a81 feat(historian): #675 reverse HistoryRead-Events paging + end-only backward reads
- IHistorianDataSource.ReadEventsAsync(..., bool reverse, ...) overload: reverse = Part 11 window (lo, hi]
  newest-first by (time, id) DESC. Default forwards forward reads and throws NotSupportedException for
  reverse (Null); Gateway refuses explicitly (ascending-only stream). Server answers
  BadHistoryOperationUnsupported instead of silently returning the earliest N ascending.
- SqlServer / QuestDB serve reverse natively (ORDER BY time DESC, id DESC + TOP/LIMIT).
- Node manager: reverse event reads page with continuation points; direction stored in the cursor
  (HistoryContinuationState.Reverse) so a resume continues in the cursor's direction; reverse tie clusters
  over-fetched forward and sliced from the newest end (SliceTieCluster(reverse:)); end-only reads with a
  cap page backward from EndTime (mirrors raw).
- Resumed event pages over-fetch by the boundary skip, so every non-final page is a full cap page.
- Tests: node-manager reverse paging (ties, clusters, coarse resolution, overfetch bound, resume
  direction both ways, end-only, unsupported backend); shared EventReadConformance on SqlServer +
  QuestDB; over-the-wire reverse paging; offline Gateway pin of the UTC DateTime.MaxValue bound through
  the published client's proto conversion (in-process gRPC capture) and FILETIME.
- docs/Historian.md event paging section.
2026-10-11 11:19:54 -04:00
Joseph Doherty c6d4134795 test(historian): gateway writer acks an all-non-numeric batch without a call; fix stale writer docs
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 14:24:45 -04:00
Joseph Doherty 4fa6cf8fee Merge ph-T4 (pluggable historian)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 12:57:52 -04:00
Joseph Doherty 29893ef1b4 feat(historian): outbox format v2 carries typed values + status; legacy entries still read
Serialize always writes v2 ("HV2\0" | ver 2 | id | tag | kind | payload | status u32 | ts).
TryDeserialize reads v2 by magic, falls back to the legacy exact-length layout (also when the
magic matches but v2 validation fails), else returns false - never throws.

FasterLogHistorizationOutbox: RecoverState and PeekBatchAsync both use TryDeserialize. An
unreadable record is indexed under a synthetic id (FIFO truncation passes it), skipped by peek,
counted once (UnreadableCount), and dropped by peek once it reaches the head so it can never
pin the outbox.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 12:50:25 -04:00
Joseph Doherty 8c8e243de6 feat(historian): GatewayHistorianProvider; status→DA quality reverse mapping
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 12:47:09 -04:00
Joseph Doherty ff7c6c531a feat(historian): typed HistorianValue contracts, IHistorianDataUpdater, IHistorianBackendProvider (no behaviour change)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 12:44:19 -04:00
Joseph Doherty 55eac5f68b feat(history): full Part 13 aggregate set; unsupported-by-backend aggregates answer BadAggregateNotSupported (#285)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 14:24:25 -04:00
Joseph Doherty 7c4c7f9653 fix(galaxy): seam hardening - advise unmark on throw, dup attributeRef surfaced, bounded subscribe-once read; delete dormant historian connection probe (#605)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 06:52:21 -04:00
Joseph Doherty 6b46c485ef test(historian): bounded-wait durability read; honest nondeterminism notes on the #491 controls 2026-08-17 04:07:08 -04:00
Joseph Doherty 7320586512 fix(historian): incremental batch-append bookkeeping; genuine group-commit durability pin 2026-08-16 04:18:38 -04:00
Joseph Doherty 6ff64ca067 perf(historian): group-commit batch append on the historization outbox 2026-08-15 21:40:04 -04:00
Joseph Doherty a86a09767c test(historian-gateway): cover GatewayHistorianValueWriter + HistorizationOutboxEntrySerializer offline
Driver.Historian.Gateway-003 (Medium): both were exercised only by the
env-gated Category=LiveIntegration suite (skips cleanly offline) or only
indirectly through FasterLogHistorizationOutboxTests with single-char
ASCII tags. Adds direct offline unit coverage for the value writer's
ack-to-bool mapping, timestamp presence/absence, and non-throwing
contract (via the existing FakeHistorianGatewayClient double), and for
the serializer's Serialize/Deserialize round trip incl. a multi-byte
UTF-8 tag, an empty tag, and ushort/double boundaries.
2026-08-10 06:53:47 -04:00
Joseph Doherty 55b12dc217 fix(historian-gateway): map alarm severity numerically; dispose provisioner's gRPC channel
Driver.Historian.Gateway-001 (High): AlarmEventMapper wrote the alarm's
Severity wire property as the .NET enum name (e.g. "High"), but
EventMapper's read side int.TryParse()s that same property and silently
falls back to the minimum severity on any unparseable value - so every
alarm event's severity round-tripped through this module's own mappers
collapsed to 1 (Low), contradicting AlarmHistorianEvent.Severity's own
documented contract ("mapped to Historian's numeric priority"). Now
mapped through a numeric priority mirroring the existing
NativeAlarmProjector.MapSeverity convention (200/500/700/900).

Driver.Historian.Gateway-002 (Low): GatewayTagProvisioner never disposed
its IHistorianGatewayClient, unlike its GatewayHistorianDataSource /
GatewayAlarmHistorianWriter siblings. Now implements IAsyncDisposable so
the DI-owned singleton's gRPC channel closes gracefully at shutdown.

Both TDD: new/corrected tests confirmed RED against the pre-fix code.
2026-08-10 06:53:41 -04:00
Joseph Doherty b95b99ba8e fix(drivers): correct 16 wrong OPC UA status-code constants + guard them by reflection (#497)
Every value verified against Opc.Ua.StatusCodes in the pinned SDK (1.5.378.106) by
reflection, not by copying siblings. All are client-visible: OPC UA clients branch on
status.

The three named in #497:
  - Historian.Gateway SampleMapper "BadNoData"  0x800E0000 -> 0x809B0000
    (0x800E0000 is BadServerHalted)
  - Galaxy "BadTimeout"                         0x800B0000 -> 0x800A0000
    (0x800B0000 is BadServiceUnsupported)
  - TwinCAT BadTypeMismatch                     0x80730000 -> 0x80740000
    (0x80730000 is BadWriteNotSupported)

BadTypeMismatch was wrong in three MORE drivers the issue did not name — FOCAS,
AbLegacy and AbCip carried the same 0x80730000. Every call site is a
FormatException/InvalidCastException conversion failure, so the name was right and
the value was wrong in all four.

Adding the reflection guard then surfaced nine further defects offline tests had
never checked:
  - Galaxy GoodLocalOverride  0x00D80000 -> 0x00960000  (not a UA code at all)
  - Galaxy UncertainLastUsableValue        0x40A40000 -> 0x40900000
  - Galaxy UncertainSensorNotAccurate      0x408D0000 -> 0x40930000
  - Galaxy UncertainEngineeringUnitsExceeded 0x408E0000 -> 0x40940000
  - Galaxy UncertainSubNormal              0x408F0000 -> 0x40950000
  - TwinCAT BadOutOfService   0x80BE0000 -> 0x808D0000 (was BadProtocolVersionUnsupported)
  - TwinCAT BadInvalidState   0x80350000 -> 0x80AF0000 (was BadAttributeIdInvalid)
  - AbCip/AbLegacy GoodMoreData 0x00A70000 -> 0x00A60000 (was GoodCommunicationEvent)
  - Historian.Gateway GatewayQualityMapper Good_LocalOverride 0x00D80000 -> 0x00960000

Galaxy's whole Uncertain block read as though the OPC DA quality byte could be
shifted into the UA substatus position; it cannot — the substatuses are an unrelated
enumeration. GatewayQualityMapper already held the correct table, which is what the
Galaxy values are now reconciled against.

Guard: StatusCodeParityTests (Core.Abstractions.Tests, which already project-
references every driver) reflects over every `const uint` in the deployed
ZB.MOM.WW.OtOpcUa.Driver.*.dll whose name reads as a status code, and asserts it
equals Opc.Ua.StatusCodes.<name>. Discovery is by convention, so a new driver
assembly referenced by that project is covered with no edit here. It went red on all
nine defects above before the fixes and now checks 109 constants green.

Because reflection can only see a NAMED constant, two inline literals were hoisted so
the guard can reach them: Galaxy's BadTimeout/Bad into StatusCodeMap, and
GatewayQualityMapper's 15-entry table into a new HistorianStatusCodes. An inline
`0x800B0000u, // BadTimeout` at a call site is exactly how the Galaxy defect survived.

The drivers stay SDK-free by design; only the test project references Opc.Ua.Core,
as the oracle.

Also corrected: tests that pinned the wrong values (Galaxy StatusCodeMapTests,
SampleMapperTests, GatewayQualityMapperTests — all written from the same bad source),
a mislabelled comment in DriverInstanceActorTests, and stale constants in two design
docs, one of them the unexecuted MTConnect driver design that would have propagated
BadNoData's wrong value into a new driver.

The CLI's SnapshotFormatter value->name table was checked against the SDK and is
correct; no change needed.
2026-07-25 15:34:31 -04:00
Joseph Doherty f9f1b8fcee fix(localdb): phase-2 live gate — 4 production defects found and fixed
Gate record: docs/plans/2026-07-20-localdb-phase2-live-gate.md.
Checks 1, 2, 5, 6 pass. Checks 3 and 4 are NOT satisfied — the defect they
were meant to confirm turned out to be the opposite of what the plan assumed.

Three defects crash-looped every driver node before check 1 could even run:

1. An empty ServerHistorian:ApiKey kills the host. ServerHistorianOptions-
   Validator exists to turn exactly that class of failure into a named
   OptionsValidationException, but its documented fail tier explicitly excluded
   ApiKey on the reasoning that a keyless client "degrades — the gateway rejects
   calls". It does not: the client validates its own options at construction, so
   the process dies during Akka startup and never makes a call.

2. UseTls disagreeing with the endpoint scheme kills the host too, in both
   directions (both messages confirmed in the shipped client assembly). Moving an
   endpoint from https to http without clearing UseTls is an ordinary migration
   slip.

3. Plaintext h2c was UNREACHABLE. HistorianGatewayClientAdapter forwarded the
   TLS-only options unconditionally, and AllowUntrustedServerCertificate defaults
   to false, so it always sent RequireCertificateValidation=true — which the
   client rejects outright when UseTls=false. Every http:// deployment crashed,
   though the scheme is documented as the supported way to select h2c, and the
   only workaround was to assert a certificate posture for a connection that has
   no certificate.

The fourth was the blocker, and it is Phase 2's own:

4. The drain gate deferred to a Primary that cannot deliver. Redundancy roles are
   elected CLUSTER-WIDE; the alarm queue is PAIR-LOCAL. On the rig the elected
   driver Primary is central-1 — it carries the driver Akka role, replicates
   nobody's LocalDb and does not even run the alarm historian — so every driver
   node logged "Historian drain suspended", including the two site-b nodes that
   have no peer at all. Nothing drained anywhere, where before Phase 2 it drained
   fine. The cost is not a duplicate; it is the buffer growing to the capacity
   wall and evicting the audit trail it exists to protect.

   Fixed in three layers: a separate ShouldDrainAlarmHistory policy (unknown role
   drains; the two gates now deliberately disagree, and a test pins that); peer-
   host matching in DriverHostActor so a node stands down only for a Primary
   holding its rows; and AddAlarmHistorian short-circuiting the gate when
   replication is unconfigured — testing BOTH Replication:PeerAddress and
   SyncListenPort, since only the dialing half sets the former while both halves
   share the queue.

   Every one of these follows from the asymmetry: a false allow costs a duplicate
   row, which at-least-once delivery already accepts and payload-hash ids
   collapse; a false deny loses data silently.

A third vacuous test, caught by the same delete-the-guard discipline: the
role-view tests stayed green with the guard removed, because AwaitAssert polls
until an assertion passes and the assertion was "reads open" — which is the
SEEDED value, satisfied at the first poll before the actor processed anything.
They now assert the sequence of published values through a recording view; the
control then goes red for exactly the cases that matter.

Migration evidence: 11 legacy rows across two deliberately overlapping files
converged to exactly 9 identical rows on both nodes, proving D-6's payload-hash
identity on real nodes rather than in a fixture.

Open design fork, recorded in the gate doc rather than decided here: a pair
cannot currently identify its own Primary, so both halves drain. Safe in every
topology — nothing loses data — but the gate's de-duplication benefit is
unrealised until roles are scoped per pair.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-21 06:13:01 -04:00
Joseph Doherty 2e4ccf7fe9 chore(localdb): phase-2 DoD sweep
Build: 0 errors solution-wide, and 0 warnings from every project this branch
touches. The ~816 solution-wide warnings are pre-existing xUnit1051 /
OTOPCUA0001 / CS86xx in untouched driver + client test projects.

Tests: full solution run compared against a full run on a detached worktree at
the pre-branch baseline 2e46d054. The two failure SETS are identical -- all 13
tests, same names, zero new failures. Net +26 tests: +3 Core.AlarmHistorian
(drain gate), +6 Runtime (role view), +17 Host.IntegrationTests (migrator +
convergence). Set comparison rather than counts, because the suite carries
standing environment- and load-dependent failures a count would hide.

The greps found real drift Task 6 missed -- eight live sites still naming the
deleted SqliteStoreAndForwardSink, including the AdminUI /alarms/historian
panel text, which is user-visible, and a <see cref> in HistorianAdapterActor
that resolved to nothing without warning. All repointed at
LocalDbStoreAndForwardSink; CLAUDE.md's alarm-history paragraph now also
records the LocalDb buffer and the primary-gated drain, and drops DatabasePath
from the knob list. docs/AlarmTracking.md still promised an
AlarmHistorianOptions.Validate() startup warning for a relative DatabasePath
and an empty SharedSecret; both branches are gone, so it now says so.

Code references to AlarmHistorian:DatabasePath reduce to exactly two intentional
ones: AlarmSfLegacyMigrator.LegacyPathKey and its test. No `new SqliteConnection`
remains anywhere in Core.AlarmHistorian.

Recon doc gains the durable verification record: guard-deletion evidence for
both vacuous passes, the two exact-set replicated-table pins (both assert set
equality, so an added or a dropped registration fails), and the baseline test
comparison with a per-failure account of why each of the 13 is not this
branch's.

Stops here per the plan. Task 8's live gate needs explicit go-ahead; nothing on
this branch is to be merged.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-21 05:03:57 -04:00
Joseph Doherty 9f62310b49 fix(historian): map Boolean historization to Int2, not degenerate Int1 (#441)
v2-ci / build (pull_request) Successful in 3m24s
v2-ci / unit-tests (pull_request) Failing after 8m19s
The historian's Int1 analog-tag creation path is server-degenerate:
EnsureTags(Int1) stores an unusable stub (live-probed on the deployed
2023 R2 historian; root-caused + documented gateway-side in
HistorianGateway #11 / PR #16). HistorianTypeMapper mapped
DriverDataType.Boolean -> HistorianDataType.Int1, so Boolean historized
tags failed auto-provisioning.

Map Boolean -> Int2 instead (store 0/1). Int2 is a proven-creatable
analog type, and the value-write path already sends every value as a
double on WriteLiveValues regardless of tag type, so Boolean 0.0/1.0
round-trips cleanly. Both Float and Int2 are supported, so a
pre-existing Float Boolean tag retypes cleanly on its next deploy.

- HistorianTypeMapper: Boolean => Int2 (+ remarks)
- Unit pins: HistorianTypeMapperTests, GatewayTagProvisionerTests
  (Boolean_definition_carries_explicit_Int2_presence)
- Live gate: EnsureTags_boolean_sandbox_provisions_as_Int2 (was _Int1);
  retype probe reframed Float->Int2 (drops the dead Int1 leg)
- Docs: docs/Historian.md 'Boolean historization maps to Int2', CLAUDE.md,
  archreview STATUS.md (R2-06 live gate -> 6/6)

Resolves #441.
2026-07-14 11:57:27 -04:00
Joseph Doherty 4da2d62b9a fix(historian): stamp a positive StorageRateMs on provisioned tags
v2-ci / build (pull_request) Successful in 4m7s
v2-ci / unit-tests (pull_request) Failing after 10m50s
GatewayTagProvisioner built HistorianTagDefinition without a StorageRateMs, so
the proto uint32 defaulted to 0. The gateway's EnsureTags maps StorageRateMs
straight to the AVEVA SDK's storageRateMs, which requires a positive value —
0 makes the SDK throw ArgumentOutOfRangeException ('Storage rate must be > 0 ms'),
so EVERY historized-tag auto-provision (the deploy-time IHistorianProvisioning
hook) silently failed at the gateway with failed=N.

Stamp DefaultStorageRateMs=1000 (matches the gateway's own live provisioner
convention; quantized 1000/5000/10000). Add a regression unit test asserting the
definition carries a positive rate, and set StorageRateMs on the live-suite's
inline definitions so the retype probe reaches a real EnsureTags.

Found via the R2-06 live gate against a real 0.2.0 gateway + AVEVA historian:
with this fixed, Float/Int2/UInt2/Int4/UInt4/Double all provision and the live
suite goes 3/6 -> 5/6. (Int1/Boolean remains blocked by the gateway histsdk's
ProtocolEvidenceMissingException for Int1 — a gateway-side type-support gap, not
this bug.)

Gateway driver unit 103/103.
2026-07-13 19:29:49 -04:00
Joseph Doherty 56ddf45e1c chore(core): flatten Historian sub-namespace to Core.Abstractions root
v2-ci / build (pull_request) Successful in 4m32s
v2-ci / unit-tests (pull_request) Failing after 10m2s
Four continuous-historization types (IHistorizationOutbox, HistorizationOutboxEntry,
IHistorianValueWriter/HistorizationValue, HistorizationCommitMode) drifted into a
ZB.MOM.WW.OtOpcUa.Core.Abstractions.Historian sub-namespace while the other six files
in the same folder correctly use the root namespace. This violated decision #59's
flat-public-surface rule and left InterfaceIndependenceTests.AllPublicTypes_LiveInRootNamespace
red on master (pre-existing, predates round-2 remediation).

Move all four to the root namespace and drop/retarget the now-redundant
'using ...Core.Abstractions.Historian;' in the ~11 consumers. No behavioral change.

Verified: Core.Abstractions.Tests 129/129 (was 128/129), Runtime.Tests Historian+DI 72/72,
Gateway driver unit 102/102, full solution build 0 errors.
2026-07-13 18:29:01 -04:00
Joseph Doherty 1aa89d754f test(r2-06): HISTGW_BOOL_SANDBOX_TAG fixture + Boolean EnsureTags leg + retype probe (skip-clean offline) (06/U-7) 2026-07-13 10:01:12 -04:00
Joseph Doherty dc2f3fa970 test(r2-06): pin 0.2.0 Boolean->Int1 explicit-presence (HasDataType) provisioning (06/U-7) 2026-07-13 09:58:07 -04:00
Joseph Doherty 3a049a13a4 fix(r2-06): RED repro — misconfigured ServerHistorian:Endpoint should fail with a named error, not UriFormatException (06/S-11) 2026-07-13 09:51:31 -04:00
Joseph Doherty 7a94f7adf0 test(historian-gateway,live): assert FU-1 alarm SendEvent->ReadEvents round-trip (C4 fixed)
v2-ci / build (pull_request) Failing after 41s
v2-ci / unit-tests (tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests) (pull_request) Has been skipped
v2-ci / unit-tests (tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests) (pull_request) Has been skipped
v2-ci / unit-tests (tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests) (pull_request) Has been skipped
v2-ci / unit-tests (tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests) (pull_request) Has been skipped
v2-ci / unit-tests (tests/Server/ZB.MOM.WW.OtOpcUa.Security.Tests) (pull_request) Has been skipped
v2-ci / integration (tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests) (pull_request) Has been skipped
v2-ci / integration (tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests) (pull_request) Has been skipped
The gateway now populates Runtime.dbo.Events.Source_Object from the event
SourceName (gateway C4 fix: HistorianProtoMapper threads SourceName into the
`source_object` event property). So an ad-hoc alarm SendEvent is now source-
filterable on readback, and Alarm_SendEvent_then_ReadEvents asserts the round-
trip instead of skipping with the old "ad-hoc sends land without Source_Object"
reason. Poll window widened to 60s for the live event-view flush latency.

Live-proven at the gateway level (HistorianGateway EventSourceObjectProbeTests,
2026-06-27); this asserts the same round-trip through the OtOpcUa alarm writer +
data source adapters.

Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
2026-06-27 16:48:16 -04:00
Joseph Doherty 240c967576 docs(historian-gateway): correct the alarm-readback skip reason (SQL reader works)
v2-ci / build (pull_request) Failing after 44s
v2-ci / unit-tests (tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests) (pull_request) Has been skipped
v2-ci / unit-tests (tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests) (pull_request) Has been skipped
v2-ci / unit-tests (tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests) (pull_request) Has been skipped
v2-ci / unit-tests (tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests) (pull_request) Has been skipped
v2-ci / unit-tests (tests/Server/ZB.MOM.WW.OtOpcUa.Security.Tests) (pull_request) Has been skipped
v2-ci / integration (tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests) (pull_request) Has been skipped
v2-ci / integration (tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests) (pull_request) Has been skipped
Live investigation showed the earlier 'C2 server-gated event reads' attribution was
wrong: the gateway's SQL event reader works (a source-filtered ReadEvents returns a
real Galaxy-sourced event's history; a time-only ReadEvents returns 50 events). The
alarm round-trip's source-filtered readback is empty only because an ad-hoc SendEvent
is recorded in Runtime.dbo.Events WITHOUT a Source_Object — so reading existing Galaxy
alarm/event history by source works, but round-tripping OtOpcUa's own sends by source
needs the gateway's SendEvent to populate the event source. Skip message corrected.

Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
2026-06-26 23:59:56 -04:00
Joseph Doherty 44644ddc7f fix(historian-gateway): alarm SendEvent must not set wire event Id (live-validated)
v2-ci / build (pull_request) Failing after 45s
v2-ci / unit-tests (tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests) (pull_request) Has been skipped
v2-ci / unit-tests (tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests) (pull_request) Has been skipped
v2-ci / unit-tests (tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests) (pull_request) Has been skipped
v2-ci / unit-tests (tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests) (pull_request) Has been skipped
v2-ci / unit-tests (tests/Server/ZB.MOM.WW.OtOpcUa.Security.Tests) (pull_request) Has been skipped
v2-ci / integration (tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests) (pull_request) Has been skipped
v2-ci / integration (tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests) (pull_request) Has been skipped
Live validation against wonder-sql-vd03 caught that the gateway's SendEvent handler
throws when the wire event carries a client-supplied Id — so every alarm send from
OtOpcUa failed (PermanentFail). AlarmEventMapper now leaves HistorianEvent.Id unset
(the historian assigns event identity) and preserves the alarm's id as an 'AlarmId'
property. With this, the live alarm send acks.

Also harden the env-gated live tests against two gateway/historian-side limitations
surfaced during validation (neither an OtOpcUa defect): the write readback uses a
timezone-tolerant window (an explicit-timestamp WriteLiveValues lands offset by the
deployment's local-vs-UTC delta — reproducible via raw grpcurl; OtOpcUa sends correct
UTC), and the alarm ReadEvents readback skips with a clear reason when the historian's
server-gated event reads (C2, won't-fix) return nothing. Read + write-persist +
alarm-send are all live-validated green; the alarm send-ack is split into its own test.

Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
2026-06-26 23:48:52 -04:00
Joseph Doherty b32436902a test(historian-gateway): env-gated live validation vs wonder-sql-vd03 (read/write/alarm round-trips)
Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
2026-06-26 19:01:36 -04:00
Joseph Doherty 035bde0562 fix(historian-gateway): dispose alarm-write channel at shutdown + ServerHistorian startup diagnostic
Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
2026-06-26 17:55:44 -04:00
Joseph Doherty 22711444cc fix(historian-gateway): cancellation-safe alarm writer + dispose-safe outbox + provisioner polish + outbox tests
I-1: GatewayAlarmHistorianWriter no longer dead-letters events cancelled
mid-drain at shutdown. WriteBatchAsync short-circuits remaining events to
RetryPlease once cancellation is requested, and SendOneAsync catches
OperationCanceledException (when the token is cancelled) -> RetryPlease,
so in-flight events stay queued instead of being permanently dropped.

I-2: FasterLogHistorizationOutbox.Dispose now guards the awaited periodic
loop with a broad catch (Exception) after the OperationCanceledException
catch, so a non-Faster teardown fault (e.g. ObjectDisposedException) can
never escape Dispose.

M-1: GatewayTagProvisioner skips the empty EnsureTags round-trip when every
request is non-historizable (early return).

M-2: GatewayTagProvisioner handles plain shutdown cancellation quietly
(Debug, not Warning), counting the unsent batch as Failed, never throwing.

M-3/M-4: Added remove-last-entry (TailAddress truncation branch) and
FIFO implicit-ack (RemoveAsync acks up to and including the target)
durability tests, both reopen-and-survive.

M-5: Clarifying comment in RecoverState on the transient over-capacity
rebuild after a crash between append-commit and drop-truncation-commit.

Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
2026-06-26 17:47:20 -04:00
Joseph Doherty 0be79219fc feat(historian-gateway): alarm-write cutover — AddAlarmHistorian drains to GatewayAlarmHistorianWriter
Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
2026-06-26 17:40:23 -04:00
Joseph Doherty 8559905e8a feat(historian-gateway): IHistorianProvisioning + GatewayTagProvisioner (EnsureTags, non-blocking)
Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
2026-06-26 17:30:03 -04:00
Joseph Doherty d3081a659f feat(historian-gateway): GatewayAlarmHistorianWriter — SendEvent + gRPC->outcome mapping
Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
2026-06-26 17:27:03 -04:00
Joseph Doherty 555bd477f1 feat(historian-gateway): FasterLog historization outbox (PerEntry/Periodic, drop-oldest)
Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
2026-06-26 17:20:06 -04:00