Commit Graph

4030 Commits

Author SHA1 Message Date
Joseph Doherty 99c13e4459 fix(opcuaclient): #670 live-gate follow-ups — historize the outage Bad, drop late replays
F1: the driver's outage BadNoCommunication carried the last Good sample's
source timestamp. The recorder historizes SourceTimestampUtc ??
ServerTimestampUtc, so the Bad landed on the Good row's (TagId,
SourceTimeUtc) key and the insert-if-absent historian dropped it - no
outage marker until the host's watchdog marker ~95 s in, none at all for
a short blip. Now stamped with the detection time (source + server), as
the host's comms-loss fan does.

F2: after a same-session recovery the SDK delivered a missed notification
seconds late with an older source time and the node regressed. A
notification older than the newest Good upstream source time for its ref
(notification or recovery read) is dropped and counted
(stale_notification_dropped); equal timestamps pass; the outage Bad never
moves the comparison; a re-initialise resets it.
2026-10-11 09:11:34 -04:00
Joseph Doherty db6446ce73 Merge fix/670-opcuaclient-session-loss: OpcUaClient upstream session loss goes Bad and recovers (#670) 2026-10-11 08:41:48 -04:00
Joseph Doherty b028322568 fix(opcuaclient): #670 review — subscription loss with a live session, stale recovery latch
M1: a subscription that dies while the session lives (outage longer than
the subscription lifetime but shorter than the session timeout) used to
end Healthy with fresh keep-alive stamps and frozen values. PublishError
BadNoSubscription / BadSubscriptionIdInvalid naming a tracked
subscription, or a tracked subscription whose SDK PublishingStopped
persists one more keep-alive window + 5 s across good keep-alives, now
declares it lost: Degraded, outage published once, and nothing advances
LastSuccessfulRead until the session is released, so the #527 watchdog
re-initialises and re-subscribes. Both detectors wire-tested.

M2: in-place recovery is bound to the session whose keep-alive recovered;
one overtaken by a re-initialise no longer re-arms the new session's
outage latch or degrades its health (ReadCoreAsync expectedSession).

L1 CAS health updates; L2 re-check before each recovery publish; L3 skip
CloseSession on a lost session; L4 SDK log forwarding rate-limited
(20/min); L5 per-driver SDK LoggerFactory disposed with the driver.
Wire test asserts the read stamp stays frozen while the server is down.
2026-10-11 08:41:27 -04:00
Joseph Doherty d803e0eab4 Merge fix/669-event-continuation: event HistoryRead continuation points + open-ended reads (#669) 2026-10-11 08:37:39 -04:00
Joseph Doherty 50b504d293 fix(opcua): raw open-ended/end-only reads per Part 11; extreme-bound conformance (#669) 2026-10-11 08:37:24 -04:00
Joseph Doherty acce99de9a fix(opcua): event paging review fixes (#669): open-ended end, node-bound cursors, save after projection 2026-10-11 08:30:36 -04:00
Joseph Doherty 91434e7f61 Merge fix/672-capture-gap: capture while the recording Primary is unreachable (#672) 2026-10-11 08:29:59 -04:00
Joseph Doherty b3db64af6c Merge feat/667-sqlserver-retention: SQL Server historian retention/purge (#667) 2026-10-11 08:27:47 -04:00
Joseph Doherty e4123ada91 fix(historian): retention applock release hardening, shutdown-quiet cancellation, release tests, docs (#667 review) 2026-10-11 08:27:31 -04:00
Joseph Doherty 2a8f9af5e4 fix(opcuaclient): upstream session loss goes Bad and recovers (#670)
A bad keep-alive now degrades health (LastSuccessfulRead preserved) and,
once per outage, publishes BadNoCommunication (value retained) for every
subscribed ref plus a quality-only annotation on subscribed alarm
conditions. Good keep-alives and data/event notifications advance
LastSuccessfulRead, so an idle-but-alive subscription never looks dead
to the #527 watchdog. A keep-alive that recovers on the same session
re-reads the Bad refs and republishes them Good; a session that does not
come back is recovered by the watchdog's ShutdownAsync + InitializeAsync
+ re-subscribe.

Deletes the SessionReconnectHandler wiring (it retried forever in SDK
1.5.378, cloned subscriptions the driver never re-tracked and kept health
Healthy), OnReconnectComplete, TryRearmReconnect and the reconnect_*
counters (new keepalive_recovered). Teardown of a lost session skips
wire deletes and bounds CloseSession. SDK warnings reach the driver log.
Wire test reworked: restart => Bad, then watchdog-style reinit => Good.
2026-10-11 08:27:04 -04:00
Joseph Doherty 1b2c36eb87 fix(historian): capture while the recording Primary is unreachable (#672)
A SIGKILLed Primary stays an unreachable-but-not-downed member until
auto-down, and the redundancy singleton usually dies with it, so the
standby's cached snapshot kept naming it Primary and nobody captured for
~30 s. DriverHostActor now holds its own ReachabilityEvent (+MemberUp)
subscription, caches the last snapshot, and excludes unreachable peers
from the historization peer-is-Primary predicate only, republishing just
the capture verdict on a reachability change. Alarm drain and the Primary
data-plane verdict are unchanged. Gated by
ContinuousHistorization:CaptureWhilePrimaryUnreachable (default true).

Tests: message-driven cases with real ReachabilityEvents (incl. a
falsification control on the drain/data-plane verdicts) plus a real
two-node cluster crash test with downing disabled.
2026-10-11 08:26:39 -04:00
Joseph Doherty 924eff8d3c feat(historian): SQL Server retention purge (RetentionDays, Primary-gated, applock) (#667) 2026-10-11 08:21:30 -04:00
Joseph Doherty f9fd495292 fix(opcua): issue and resume continuation points for event HistoryRead (#669)
A capped events read now pages: the source's truncation flag triggers a
session-bound cursor (time inclusive + tie skip), oversized tie clusters
are paged within via a bounded half-open over-fetch, and release/unknown/
cross-kind points behave as for Raw.
2026-10-11 08:21:02 -04:00
Joseph Doherty 7d75478f10 Merge fix/673-focas-flake: FOCAS LastStatus publication order (#673) 2026-10-11 08:16:16 -04:00
Joseph Doherty e5d12d391a fix(focas): publish LastStatus after the fixed-tree stamp; test waits on full readiness (#673) 2026-10-11 08:16:00 -04:00
Joseph Doherty db6a53036b docs(plans): issue batch #667-#673 design 2026-10-11 08:14:38 -04:00
Joseph Doherty ee67b6370a Merge feat/questdb-historian: QuestDB historian backend (#666)
v2-ci / build (push) Successful in 5m6s
v2-ci / unit-tests (push) Failing after 23m29s
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 04:40:42 -04:00
Joseph Doherty 1569a59ad2 test(historian): QuestDB live gate PASSED on docker-dev (6/6 legs); RetentionDays compose key
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 04:00:03 -04:00
Joseph Doherty 2ebe0a4226 Merge qdb/final: final-review fix wave (pool cap, catalog metadata, warn rate-limit, disposal, docs)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 03:37:55 -04:00
Joseph Doherty 8d82d049e9 fix(historian): QuestDB final-review fix wave (I-1, M-1, M-3, M-5, M-8, M-9)
- I-1: QuestDbConnection caps Npgsql Maximum Pool Size at 24 (DefaultMaxPoolSize)
  when the connection string does not set it; an explicit value is kept. The
  effective value is logged once at Information.
- M-1: QuestDbTagCatalog.EnsureAsync waits for the tag table's sequenced commits
  and re-checks before inserting, so it no longer overwrites a provisioner row
  (unit/description) that is committed but not yet applied. Integration test
  reproduces the clobber against real QuestDB (falsified).
- M-3: the pre-1970 drop Warning is rate-limited per tag (once per 10 min);
  DroppedPre1970 still counts every sample.
- M-5: QuestDbHistorianProvider is IAsyncDisposable/IDisposable and disposes the
  shared Npgsql pool; Program.cs registers the provider through a factory and
  every seam resolves it from the container, so DI disposes it at shutdown.
- M-8, M-9: docs fixes (HistoryUpdate transaction wording; folded duplicate
  HistoryUpdate paragraph). Pool-size docs updated for the new default cap.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 03:35:35 -04:00
Joseph Doherty e177a50046 Merge qdb/T8: QuestDB backend documentation (as built)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 03:10:56 -04:00
Joseph Doherty 94691ec50f docs(historian): QuestDB backend documentation (as built)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 03:09:11 -04:00
Joseph Doherty 17cff2ba16 Merge qdb/T7: QuestDb provider + Host wiring; conformance green on both backends
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 03:04:43 -04:00
Joseph Doherty f661c084d8 test(historian): QuestDB-backed paging regression (T7 review)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 03:03:13 -04:00
Joseph Doherty 0e2bffc74a feat(historian): QuestDb provider + Host wiring; conformance suite green on both backends
AtTime conformance stamps are the requested time (M3; identical on SQL Server). The
QuestDB test fixture prunes idle pooled connections after 1 s: undisposed per-test data
sources exhausted QuestDB's 64-connection PG-wire limit in a full run.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 02:53:40 -04:00
Joseph Doherty 3244f9781a refactor(historian): one QuestDbTimeClusters helper for writer + updater existence reads; redelivery tombstone test
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 02:32:11 -04:00
Joseph Doherty d2be213e86 fix(historian): page by the backend's timestamp resolution; QuestDB AtTime stamps the requested time
IHistorianDataSource.TimestampResolutionTicks (default 1; QuestDB 10). A drained tie
cluster now steps off T by the resolution, so a forward one-value-per-page read on a
microsecond backend no longer re-reads the same sample forever (C1). QuestDB AtTime
results carry the client's original request time (M3); regression for the NEXT seek
skipping a tombstone (M1). Conformance backend wires the real writer and updater.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 02:30:25 -04:00
Joseph Doherty 13d628a07f feat(historian): QuestDb provider, registry + consumer-gated validator, compose env
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 02:23:21 -04:00
Joseph Doherty 18e85265f9 Merge qdb/T5: QuestDB reads and aggregates
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 02:19:32 -04:00
Joseph Doherty 3790a21acb Merge qdb/T4: QuestDB writes (value writer, provisioner, alarm writer)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 02:16:49 -04:00
Joseph Doherty 9d90dc9457 fix(historian): clustered existence reads + tag-predicate tests (T4 review)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 02:16:14 -04:00
Joseph Doherty e5b9783ba3 Merge qdb/T6: QuestDB HistoryUpdate (soft delete)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 02:13:25 -04:00
Joseph Doherty c4b4b2e2bc feat(historian): QuestDB reads and aggregates (pushed down, shared evaluation)
QuestDbHistorianDataSource serves Raw (LIMIT $n, no continuation, R8), AtTime
(chunked LIMIT 1 neighbour seeks, <=50 times per statement), Events and
Processed (one statement per node, bucket = (cast(ts AS LONG)*10 - start)/interval,
shared BucketPlan/AggregateEvaluator/AtTimeEvaluator). Every sample read filters
NOT deleted; every row goes through StoredSample.FromStored; RC2 long.MinValue
read repair. QuestDbConformanceBackend (seed/read-back) for the conformance suite;
writer/updater factories are T7 placeholders.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 02:13:00 -04:00
Joseph Doherty 5c0671e5b5 fix(historian): per-tag update lock + clustered existence reads (T6 review)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 02:12:09 -04:00
Joseph Doherty 4c053cc4d0 feat(historian): QuestDB live writes, provisioning and alarm events
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 02:08:04 -04:00
Joseph Doherty ee174c0179 feat(historian): QuestDB HistoryUpdate with soft delete and read-your-writes
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 02:04:48 -04:00
Joseph Doherty ecc0f914cc Merge qdb/T3: QuestDB historian skeleton
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 01:55:28 -04:00
Joseph Doherty fabcc6eed6 Merge qdb/T2: historian conformance suite over SQL Server
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 01:52:59 -04:00
Joseph Doherty 11b703708c feat(historian): QuestDB backend skeleton — options, connection, schema bootstrap with TTL
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 01:52:57 -04:00
Joseph Doherty c5b33aa123 test(historian): backend conformance suite (shared source), run against SQL Server
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 01:50:51 -04:00
Joseph Doherty ca2c20f852 Merge qdb/T1: extract Driver.Historian.Sql.Common
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 01:43:01 -04:00
Joseph Doherty 082e808d0d test(historian): pin Uncertain-input degradation (T1 review)
Two Sql.Common tests close review finding I-1: an Uncertain NEXT neighbour makes an AtTime
interpolation Uncertain, and an Uncertain (non-Bad) sample in a bucket makes the aggregate
Uncertain. Each falsified with the reviewer's surviving mutation.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 01:42:22 -04:00
Joseph Doherty f0700571e8 test(historian): direct Sql.Common evaluator tests (1b)
51 new test cases (34 methods) in Sql.Common.Tests covering AtTimeEvaluator, AggregateEvaluator
(hand-built tag-A bucket rows, ProcessedRead literals), BucketPlan, LiveWritePreparer and
HistoryUpdatePlanner (incl. resolutionTicks=10 truncate-before-dedup and the superseded-duplicate
outcome), plus EventContinuation, HistorianHealthCounters, ClampSeverity and StoredSample.FromStored.
Each was falsified by reverting the rule it pins (37 mutations, all red, all restored).

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 01:35:51 -04:00
Joseph Doherty 8c6d6aff6a refactor(historian): extract Driver.Historian.Sql.Common from the SQL Server backend (no behaviour change) (1a)
Moves the database-independent logic of the SQL Server historian backend into a new
shared library for the upcoming QuestDB backend (#666): status codes and bits, value/column
mapping, StoredSample + quality clamp, the AtTime and aggregate evaluators, BucketPlan,
LiveWritePreparer and HistoryUpdatePlanner (both with a resolutionTicks parameter, default 1),
tag-name rules, the bootstrap failure cache, health counters, alarm event ids, the event
continuation and severity clamp. The SQL Server backend now calls it; its T-SQL, TVP
builders, connection, bootstrapper and catalog stay.

Tests: StatusCodeOracleTests, ValueWriterPrepareTests, the column half of SampleRowMapperTests
and ProcessedReadTests.Every_enum_member_is_either_served_or_refused move to the new
Sql.Common.Tests under the same names. SqlServer integration tests get mechanical renames
only. Test-name set across the three projects is identical to the baseline (160 tests).

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 01:26:49 -04:00
Joseph Doherty 27991434a0 spike(historian): T0 review fixes
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 01:10:31 -04:00
Joseph Doherty cdea173bd1 spike(historian): QuestDB behaviours confirmed against pinned image; spec §2–§4 amended
questdb/questdb:10.0.1 (pushed 2026-08-24). Q1–Q13 answered with harness output in
docs/plans/2026-10-10-questdb-spike.md; confirmed DDL, visibility wait, bucket/neighbour
SQL and connection keywords folded into the design spec, marked T0-confirmed.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 01:07:38 -04:00
Joseph Doherty f54ab04e64 docs(historian): fold Fable pre-execution review into QuestDB plan + spec (6 blockers, 15 should-fix, 8 nits)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-10 09:09:42 -04:00
Joseph Doherty b3031dfea9 docs(historian): QuestDB backend implementation plan (#666)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-10 08:56:15 -04:00
Joseph Doherty 9998d816f1 docs(historian): QuestDB historian backend design (#666)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-10 08:46:48 -04:00
Joseph Doherty 8ec7a83c83 Merge feat/pluggable-historian: pluggable historian backends (Gateway | SqlServer), typed values + real status, OPC UA HistoryUpdate
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 15:29:10 -04:00