Commit Graph

2298 Commits

Author SHA1 Message Date
Joseph Doherty 0e2bffc74a feat(historian): QuestDb provider + Host wiring; conformance suite green on both backends
AtTime conformance stamps are the requested time (M3; identical on SQL Server). The
QuestDB test fixture prunes idle pooled connections after 1 s: undisposed per-test data
sources exhausted QuestDB's 64-connection PG-wire limit in a full run.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 02:53:40 -04:00
Joseph Doherty 3244f9781a refactor(historian): one QuestDbTimeClusters helper for writer + updater existence reads; redelivery tombstone test
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 02:32:11 -04:00
Joseph Doherty d2be213e86 fix(historian): page by the backend's timestamp resolution; QuestDB AtTime stamps the requested time
IHistorianDataSource.TimestampResolutionTicks (default 1; QuestDB 10). A drained tie
cluster now steps off T by the resolution, so a forward one-value-per-page read on a
microsecond backend no longer re-reads the same sample forever (C1). QuestDB AtTime
results carry the client's original request time (M3); regression for the NEXT seek
skipping a tombstone (M1). Conformance backend wires the real writer and updater.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 02:30:25 -04:00
Joseph Doherty 13d628a07f feat(historian): QuestDb provider, registry + consumer-gated validator, compose env
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 02:23:21 -04:00
Joseph Doherty 18e85265f9 Merge qdb/T5: QuestDB reads and aggregates
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 02:19:32 -04:00
Joseph Doherty 3790a21acb Merge qdb/T4: QuestDB writes (value writer, provisioner, alarm writer)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 02:16:49 -04:00
Joseph Doherty 9d90dc9457 fix(historian): clustered existence reads + tag-predicate tests (T4 review)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 02:16:14 -04:00
Joseph Doherty c4b4b2e2bc feat(historian): QuestDB reads and aggregates (pushed down, shared evaluation)
QuestDbHistorianDataSource serves Raw (LIMIT $n, no continuation, R8), AtTime
(chunked LIMIT 1 neighbour seeks, <=50 times per statement), Events and
Processed (one statement per node, bucket = (cast(ts AS LONG)*10 - start)/interval,
shared BucketPlan/AggregateEvaluator/AtTimeEvaluator). Every sample read filters
NOT deleted; every row goes through StoredSample.FromStored; RC2 long.MinValue
read repair. QuestDbConformanceBackend (seed/read-back) for the conformance suite;
writer/updater factories are T7 placeholders.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 02:13:00 -04:00
Joseph Doherty 5c0671e5b5 fix(historian): per-tag update lock + clustered existence reads (T6 review)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 02:12:09 -04:00
Joseph Doherty 4c053cc4d0 feat(historian): QuestDB live writes, provisioning and alarm events
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 02:08:04 -04:00
Joseph Doherty ee174c0179 feat(historian): QuestDB HistoryUpdate with soft delete and read-your-writes
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 02:04:48 -04:00
Joseph Doherty ecc0f914cc Merge qdb/T3: QuestDB historian skeleton
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 01:55:28 -04:00
Joseph Doherty 11b703708c feat(historian): QuestDB backend skeleton — options, connection, schema bootstrap with TTL
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 01:52:57 -04:00
Joseph Doherty c5b33aa123 test(historian): backend conformance suite (shared source), run against SQL Server
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 01:50:51 -04:00
Joseph Doherty 082e808d0d test(historian): pin Uncertain-input degradation (T1 review)
Two Sql.Common tests close review finding I-1: an Uncertain NEXT neighbour makes an AtTime
interpolation Uncertain, and an Uncertain (non-Bad) sample in a bucket makes the aggregate
Uncertain. Each falsified with the reviewer's surviving mutation.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 01:42:22 -04:00
Joseph Doherty f0700571e8 test(historian): direct Sql.Common evaluator tests (1b)
51 new test cases (34 methods) in Sql.Common.Tests covering AtTimeEvaluator, AggregateEvaluator
(hand-built tag-A bucket rows, ProcessedRead literals), BucketPlan, LiveWritePreparer and
HistoryUpdatePlanner (incl. resolutionTicks=10 truncate-before-dedup and the superseded-duplicate
outcome), plus EventContinuation, HistorianHealthCounters, ClampSeverity and StoredSample.FromStored.
Each was falsified by reverting the rule it pins (37 mutations, all red, all restored).

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 01:35:51 -04:00
Joseph Doherty 8c6d6aff6a refactor(historian): extract Driver.Historian.Sql.Common from the SQL Server backend (no behaviour change) (1a)
Moves the database-independent logic of the SQL Server historian backend into a new
shared library for the upcoming QuestDB backend (#666): status codes and bits, value/column
mapping, StoredSample + quality clamp, the AtTime and aggregate evaluators, BucketPlan,
LiveWritePreparer and HistoryUpdatePlanner (both with a resolutionTicks parameter, default 1),
tag-name rules, the bootstrap failure cache, health counters, alarm event ids, the event
continuation and severity clamp. The SQL Server backend now calls it; its T-SQL, TVP
builders, connection, bootstrapper and catalog stay.

Tests: StatusCodeOracleTests, ValueWriterPrepareTests, the column half of SampleRowMapperTests
and ProcessedReadTests.Every_enum_member_is_either_served_or_refused move to the new
Sql.Common.Tests under the same names. SqlServer integration tests get mechanical renames
only. Test-name set across the three projects is identical to the baseline (160 tests).

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 01:26:49 -04:00
Joseph Doherty 5fdf205caa test(historian): live gate PASSED on docker-dev (SqlServer backend); CLI prints node-level HistoryUpdate result
Gate: Primary-only typed recording with real status, raw+UNS HistoryRead/Processed parity, HistoryUpdate
insert/duplicate/replace/delete + WriteOperate-only and anonymous denied, failover at 1x, Modbus device loss
stored as BadCommunicationError then a BadNoCommunication outage marker. Gate found the CLI hid the
node-level refusal code (fixed + pinned) and a pre-existing OpcUaClient comms-loss gap (filed separately).
docker-dev compose gains ServerHistorian__Backend / __SqlServer__ConnectionString (default Gateway).

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 15:14:46 -04:00
Joseph Doherty 60a1e7e669 docs(historian): fix-wave docs and stale comments
- Historian.md: Empty samples always Bad (writer/updater/read clamp),
  long-name refusal, 5 s bootstrap failure cache, AtTime after now, info-bit
  masking, HistoryUpdate does NOT type-check (M-1 decision), DeleteRaw
  needs both bounds, reverse Raw paging, the historian/historization metric
  prefixes, and provisioning numeric-only lines qualified as Gateway.
- CLAUDE.md numeric-only lines qualified as Gateway backend.
- Stale 'sole historian' lines in DriverLifecycle, ServiceHosting and the
  retired Wonderware stub; Client.CLI exit-code contract.
- ServerHistorianOptions: Endpoint/Backend XML docs untangled, Validate()
  indented; Program.cs comments are backend-neutral; T10 week-test day count.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 14:33:30 -04:00
Joseph Doherty 7fafeb433c test(opcua): flag-OFF HistoryUpdate over the wire - Cluster HistoryUpdate grant allowed, WriteOperate alone denied
The rig's key roles are now settable; the test derives them with the
production ApiKeyInterimRoleMapper from the same grants it installs.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 14:31:01 -04:00
Joseph Doherty f8ad74dc19 fix(opcua): DeleteRawModified with an unspecified bound is BadInvalidArgument; HistoryUpdate node-manager tests
- A DateTime.MinValue start or end on DeleteRawModified is refused
  BadInvalidArgument before the backend.
- Tests: anonymous without the HistoryUpdate role denied (with it allowed),
  UpdateTagAttributes enabling historization sets HistoryWrite, a throwing
  updater is BadInternalError, the MinValue bounds case; corrected the
  test helper's comment on how the SDK base seeds unadmitted slots.
- GatewayHistorianValueWriter remarks describe the recorder's real status
  capture (and that it does not persist on the gateway).

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 14:27:33 -04:00
Joseph Doherty c6d4134795 test(historian): gateway writer acks an all-non-numeric batch without a call; fix stale writer docs
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 14:24:45 -04:00
Joseph Doherty 96ec751a0b fix(historian): latch the recorder's dropped-kind Debug log per run; pin the outbox.unreadable gauge
The DroppedUnsupportedKind counter still counts every value; only the Debug
line is latched (cleared by the next captured value), so a Gateway-backend
comms loss no longer logs one line per tag.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 14:23:42 -04:00
Joseph Doherty 7c787b8b64 test(host): an invalid SqlServer:Schema is consumer-gated, not a Resolve-time throw
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 14:22:41 -04:00
Joseph Doherty 6e10e45bb9 fix(cli): historyupdate exits non-zero unless every item is Good, and never prints a blank status
A Good service result with missing or too few per-item results, a missing
result entry, or an Uncertain item now fails the command; a status code
the SDK has no name for prints as hex.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 14:21:32 -04:00
Joseph Doherty 8d51380ff1 fix(opcua): reverse HistoryRead-Raw paging no longer stops after the first page
A reverse read (start > end) with a small NumValuesPerNode resumed through
the tie-cluster path, which drained the cluster and stepped FORWARD to
T + 1 tick - outside a reverse window - so the read ended with GoodNoData
after one sample. SliceTieCluster now steps down to T - 1 tick for a
reverse read (terminal at the excluded end), and the node manager slices
the over-fetched cluster in the read's own direction so the skip count is
not applied from the wrong end. Forward paging is unchanged.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 14:20:00 -04:00
Joseph Doherty 821763a09d fix(historian): SQL quality contract, long-name refusal, info-bit masking, AtTime-after-now, bootstrap hardening
- I-1: an Empty sample is always Bad. The writer stores a non-Bad NaN/Inf as
  Empty+BadOutOfRange and any other non-Bad Empty as BadNoData; the updater
  refuses a non-Bad Empty with BadInvalidArgument; reads clamp a legacy
  non-Bad Empty row to BadNoData (Raw and AtTime).
- Tag/source names over 400 chars are refused, never truncated: writer
  per-tag failure, updater BadInvalidArgument, reads not-found, events none.
- HistorianInfoBits masks the historian bits before setting one.
- AtTime past now, stepped from the last sample => UncertainDataSubNormal.
- Bootstrapper: logger, 5 s negative cache, ClearPool + Warning when
  sp_releaseapplock fails.
- Provider builds its shared state lazily (an invalid Schema no longer
  throws at Resolve); Kinds is a FrozenSet; Gateway reuses the recorder's
  DefaultSupportedKinds.
- Cleanup: SqlHistorianDataTypes in its own file (updater dup removed),
  dead Invalidate and unused status consts deleted, named severity masks,
  alarm writer brace.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 14:16:31 -04:00
Joseph Doherty 7500106c2f feat(historian): SqlServerHistorianProvider + SqlServer backend registration
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 13:42:38 -04:00
Joseph Doherty a611cc358f Merge ph-T13 (pluggable historian)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 13:35:22 -04:00
Joseph Doherty 0a8deba94c Merge ph-T11 (pluggable historian)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 13:34:53 -04:00
Joseph Doherty a8c0919365 Merge ph-T10 (pluggable historian)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 13:34:20 -04:00
Joseph Doherty d0876440df Merge ph-T9 (pluggable historian)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 13:33:48 -04:00
Joseph Doherty 8f859acb2c fix(historian): T10 review — tick-exact bucketing over the full DateTime range; AtTime outage keeps its Bad code
Important 1: bucket offsets are now 100 ns ticks as bigint
(DATEDIFF_BIG(microsecond)*10 plus the sub-microsecond tick digits), so a
1601→9999 window no longer overflows (SQL 535) and nothing scales the interval
to nanoseconds in C#. Interval wider than the window clamps to one bucket and
the ceiling division cannot overflow. Exactness to the tick is unchanged.

Important 2 (spec §4 ruling d6afb394): a Bad previous neighbour means the time
is inside an outage — null value with that neighbour's Bad code, never
Uncertain. Usable previous + Bad next still holds the previous value Uncertain.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 13:30:54 -04:00
Joseph Doherty c2ab68ceb5 fix(historian): T9 review round 1 - dup-key retry counter, alarm Classify retries unknown, test hardening
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 13:30:42 -04:00
Joseph Doherty 38d37f5c23 feat(historian): SQL Server read side — raw, at-time, events, push-down processed (ph-T10)
SqlServerHistorianDataSource implements IHistorianDataSource over the T8 schema:
raw forward/reverse/point with the 0 = unbounded sentinel; AtTime in one statement
(TimeRow TVP + two OUTER APPLYs) with linear/stepped/no-prior rules and no
interpolation across a Bad neighbour or Empty outage marker; events with a
maxEvents+1 truncation probe and a non-null continuation point; processed
aggregates pushed down to T-SQL (SqlAggregateQueries) with nanosecond-exact
bucket edges, Calculated bits on every result, Uncertain on any non-Good input,
BadAggregateInvalidInputs for numeric aggregates over String/DateTime, a
100 000-bucket cap and HistoryAggregateNotSupportedException for the rest.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 13:13:07 -04:00
Joseph Doherty bef47f89da feat(historian): SQL Server value writer, tag provisioner, alarm writer (T9)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 13:08:43 -04:00
Joseph Doherty 2d4fbc857f feat(historian): SQL Server HistoryUpdate data updater (T11)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 13:07:44 -04:00
Joseph Doherty 32852382cc test(historian): over-the-wire HistoryUpdate integration tests
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 13:07:19 -04:00
Joseph Doherty 31d342eff3 feat(historian): backend registry, ServerHistorian:Backend, Host wiring via provider
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 13:05:53 -04:00
Joseph Doherty 65e6352dad Merge ph-T8 (pluggable historian)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 13:00:51 -04:00
Joseph Doherty 1447a50140 Merge ph-T7 (pluggable historian)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 13:00:22 -04:00
Joseph Doherty 08c340add7 Merge ph-T6 (pluggable historian)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 12:59:51 -04:00
Joseph Doherty 4fa6cf8fee Merge ph-T4 (pluggable historian)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 12:57:52 -04:00
Joseph Doherty 31239d51ff Merge ph-T3 (pluggable historian)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 12:57:20 -04:00
Joseph Doherty 2ef8ee3ade feat(historian): OPC UA HistoryUpdate in the node manager + HistoryUpdate authorization (T6)
- OtOpcUaNodeManager overrides HistoryUpdateData / HistoryDeleteRawModified / HistoryDeleteAtTime,
  block-bridging to the optional IHistorianDataUpdater under the shared HistoryRead batch limiter +
  deadline. Insert/Replace/Update map to HistoryUpdateMode; Remove, IsDeleteModified, events and
  structure data are BadHistoryOperationUnsupported. Per-item codes stitched back in request order
  (MinValue SourceTimestamp => BadInvalidTimestamp, untypeable value => BadTypeMismatch); success
  clears the base's pre-seeded Unsupported error.
- HistoryWrite AccessLevel bit OR-ed in for historized nodes when an updater is wired (raw + UNS);
  it is dispatch-load-bearing in the SDK base. OtOpcUaSdkServer.SetHistorianDataUpdater; the hosted
  service assigns it from an optional HistorianUpdaterHolder before SetSink, clears it on stop.
- AuthorizeHistoryUpdate: ACL on => scoped OpcUaOperation.HistoryUpdate decision (replaces the role
  check); ACL off => #563 key re-check + new OpcUaDataPlaneRoles.HistoryUpdate role, conferred by
  ApiKeyInterimRoleMapper only from a Cluster-scoped HistoryUpdate grant (WriteOperate does not).
- AdminUI ACL editor offers NodePermissions.HistoryUpdate.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 12:55:42 -04:00
Joseph Doherty 1c822eeeed fix(historian): document comms-loss outage markers; freeze DefaultSupportedKinds
Review round 1: the #481 FanCommsLossToDependencyMux fan-out DOES reach the recorder. Per the
controller ruling it is historized as an Empty/BadNoCommunication outage marker when the backend
supports Empty and counted in DroppedUnsupportedKind otherwise; remarks and the counter doc now say
so. DefaultSupportedKinds is a FrozenSet. The drop-leg test now runs under the default (gateway)
kinds, and the Empty leg pins status 0x80310000.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 12:52:32 -04:00
Joseph Doherty 17962d4233 feat(historian): SQL Server backend skeleton - schema DDL, locked bootstrapper, options, connection/mapper/tag-catalog helpers, status consts, env-gated fixture
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 12:52:15 -04:00
Joseph Doherty 29893ef1b4 feat(historian): outbox format v2 carries typed values + status; legacy entries still read
Serialize always writes v2 ("HV2\0" | ver 2 | id | tag | kind | payload | status u32 | ts).
TryDeserialize reads v2 by magic, falls back to the legacy exact-length layout (also when the
magic matches but v2 validation fails), else returns false - never throws.

FasterLogHistorizationOutbox: RecoverState and PeekBatchAsync both use TryDeserialize. An
unreadable record is indexed under a synthetic id (FIFO truncation passes it), skipped by peek,
counted once (UnreadableCount), and dropped by peek once it reaches the head so it can never
pin the outbox.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 12:50:25 -04:00
Joseph Doherty 04b882f5c6 fix(historian): record the real status code and typed values; backend-supported kinds gate capture
The recorder now appends the typed HistorianValue and the mux's own StatusCode instead of a
double + hard-coded Good, and drops any kind not in the backend's SupportedKinds (resolved
optionally from IHistorianBackendProvider; absent => {Double, Int64, Boolean}, today's behaviour).
RecorderStatus.DroppedNonNumeric -> DroppedUnsupportedKind; GoodQuality const removed; stale
"mux carries no quality" remarks rewritten (comms-loss node stamps are not historized).

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 12:49:05 -04:00
Joseph Doherty 70dde115f0 feat(cli): historyupdate command (insert/replace/update/delete-raw/delete-at-time)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 12:47:53 -04:00