Files

T

Joseph Doherty b3031dfea9 docs(historian): QuestDB backend implementation plan (#666 )

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe

2026-10-10 08:56:15 -04:00

58 KiB

Raw Blame History

QuestDB Historian Backend Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: use superpowers-extended-cc:subagent-driven-development (recommended) or superpowers-extended-cc:executing-plans to run this plan task by task. Steps use checkbox (- [ ]) syntax.

Goal: add ServerHistorian:Backend=QuestDb. It reaches full parity with the SQL Server backend, covering recording, reads, the 13 aggregates and all five HistoryUpdate operations through soft delete, and adds table-TTL retention.

Architecture: move the database-independent logic of Driver.Historian.SqlServer into a new shared library, Driver.Historian.Sql.Common. Build Driver.Historian.QuestDb on top of it, using Npgsql over the Postgres wire protocol. A shared-source conformance suite runs the same hand-computed expectations against both backends.

Tech stack: .NET 10, Npgsql 10.0.3 (already pinned), QuestDB as a WAL table with DEDUP UPSERT KEYS, xUnit v3 + Shouldly, docker-dev.

Spec: docs/plans/2026-10-10-questdb-historian-design.md. Its §0 holds the user decisions. The T0 spike amends §2–§4 before W1 starts.

Global Constraints

These are binding on every task and every reviewer.

Packages and build

Add no NuGet packages and change no versions. Never edit Directory.Packages.props. Referencing Npgsql from the new QuestDb project is allowed, because it is already centrally pinned.
Every new src csproj sets TreatWarningsAsErrors=true, GenerateDocumentationFile=true and NoWarn CS1591, as Driver.Historian.SqlServer.csproj does. TargetFramework and Nullable are inherited from Directory.Build.props.
New src project names MUST match ZB.MOM.WW.OtOpcUa.Driver.*. StatusCodeParityTests scans only ZB.MOM.WW.OtOpcUa.Driver.*.dll.

Status codes

Every OPC UA status code a class emits is a named const uint whose name starts with Good, Uncertain or Bad. Bit masks use names that do NOT start with those words (CodeBitsMask, SeverityBadBit).
Any new Driver assembly must be a ProjectReference of tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests, so its DLL lands in that test's bin and gets scanned.

SQL Server backend

Its observable behaviour must not change. The SqlServer unit and integration suites stay green, and no test there is deleted. A test may only move into the conformance suite, keeping the same name and the same literals.

Quality and write rules

Quality contract: an Empty (null-value) sample is always served with a Bad status. NaN/±Inf and Empty-with-non-Bad are stored as BadOutOfRange or BadNoData on the live write, and rejected with BadInvalidArgument by HistoryUpdate.
Live writes are insert-if-absent, and the first duplicate in a batch wins. On QuestDB, an existing tombstone also counts as present, so it blocks the insert and the row is never resurrected.
QuestDB writes: every write is an INSERT, and the dedup upsert does the replacing. Never issue QuestDB UPDATE or DELETE for sample or event data. DROP TABLE is allowed only in test cleanup.
Soft delete: a delete writes a tombstone row (deleted = true, value columns NULL) at the same (ts, tag). Every read and aggregate filters NOT deleted.

Time and buckets

Timestamps: QuestDB stores microseconds. The QuestDB writer and updater truncate to microseconds (ticks - ticks % 10) before de-duplicating. Conformance tests compare timestamps through IConformanceBackend.Round(DateTime).
Bucket edges must be identical to SQL Server's. The QuestDB bucket index is (cast(ts AS LONG) * 10 - @startUnixTicks) / @intervalTicks, using integer ticks relative to the Unix epoch. Never use SAMPLE BY.

Infrastructure and tests

docker-dev: QuestDB is an overlay file, docker-dev/docker-compose.questdb.yml. The rig uses no compose profiles:. Host ports are PG wire 18812 and web console 19000.
- These ports are already taken and must not be reused: 14330, 14340, 14350, 4840–4845, 4850, 4851, 4860, 4861, 9200, 9210, 9220, 8089.
Test environment variables: HISTORIAN_QUESTDB_TEST_CONNECTION, alongside the existing HISTORIAN_SQL_TEST_CONNECTION. When a variable is absent, its tests skip through Assert.Skip.
Regression tests must be falsified: revert the production change, watch the test go red, restore it, and record what you observed.
Shell: zsh, so quote globs and avoid a bare ====.
Building: put two projects in one dotnet build and it fails with MSB1008, so build them separately or use the .slnx.

User decisions (already made):

"Soft delete (Recommended)": the deleted flag plus tombstones, with all five HistoryUpdate operations.
"PG wire only (Recommended)": Npgsql for everything, no ILP.
"Include TTL (Recommended)": RetentionDays becomes table TTL.
Approach B: a shared dialect-free core plus per-backend SQL, and one conformance suite run against both backends.
Microsecond timestamps; two samples in one batch landing on the same microsecond collapse to one (first wins).
Every design section was approved as written (2026-10-10). The plan gets a Fable review, then the session is prepared for compaction before executing with subagents.

Plan rulings (made while planning, from the code inventory):

R1: the spec §6 compose profile becomes an overlay file. The rig has no profiles; each optional fixture is docker-compose.<name>.yml.
R2: the conformance suite is a shared-source directory, tests/Drivers/Historian.Conformance/, which has no csproj. Each integration project includes it with <Compile Include="..\Historian.Conformance\**\*.cs" LinkBase="Conformance" />. The reason is that only xunit.v3 is pinned, and a class library referencing it would become a test project.
R3: the shared types become public in Sql.Common, so no InternalsVisibleTo is needed for the two backends. They are renamed from their Sql* names to neutral ones (table in T1). The parity test keys on field names, not class names.

Waves, models and parallel safety

Wave	Tasks (‖ = parallel)	Models
W0	T0 QuestDB spike + overlay compose + spec amendments	controller (live docker)
W1	T1 extract `Sql.Common`	opus
W2	T2 conformance suite over SqlServer ‖ T3 QuestDB skeleton	sonnet ‖ sonnet
W3	T4 writes ‖ T5 reads + aggregates ‖ T6 updater	sonnet ‖ opus ‖ sonnet
W4	T7 provider + Host wiring + QuestDB conformance	sonnet
W5	T8 docs → fable whole-branch review → one fix wave (opus) + scoped re-review (sonnet) → T9 live gate	sonnet / fable / opus / controller

Reviewers:

opus for T1 and T5;
sonnet for the rest;
sonnet for scoped re-reviews.

Each task runs in its own worktree, .worktrees/qdb-T<N>, branched from the current feat/questdb-historian head. Merges happen in ascending task order within a wave, with dotnet build ZB.MOM.WW.OtOpcUa.slnx after each merge.

File ownership. A task edits only the files it owns. Anything else needs a controller ruling before it is touched.

Path	Owner
`docker-dev/docker-compose.questdb.yml`, `docs/plans/2026-10-10-questdb-spike.md`, spec §2–§4 amendments	T0
`src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/`, `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/`	T1
`src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/`, `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests/`	T1
`ZB.MOM.WW.OtOpcUa.slnx`, `tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests/*.csproj`	T1 adds Sql.Common; then T3 adds the QuestDb lines
`tests/Drivers/Historian.Conformance/`, `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/`	T2
`src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/{csproj,QuestDbHistorianOptions.cs,QuestDbConnection.cs,QuestDbSchemaBootstrapper.cs,QuestDbTables.cs}`	T3
`tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/**` (csproj + options/bootstrap unit tests)	T3
`tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/{csproj,QuestDbFixture.cs,BootstrapTests.cs}`	T3
QuestDb `QuestDbHistorianValueWriter.cs`, `QuestDbTagProvisioner.cs`, `QuestDbAlarmHistorianWriter.cs`; IT `WriteTests.cs`, `ProvisioningTests.cs`, `AlarmWriterTests.cs`	T4
QuestDb `QuestDbHistorianDataSource.cs`, `QuestDbAggregateQueries.cs`; IT `ReadTests.cs`	T5
QuestDb `QuestDbHistorianDataUpdater.cs`, `QuestDbWalVisibility.cs`; IT `UpdateTests.cs`, `TombstoneTests.cs`	T6
QuestDb `QuestDbHistorianProvider.cs`; IT `QuestDbConformanceBackend.cs`, `Conformance*.cs`; Host `HistorianBackendRegistry.cs`, `ServerHistorianOptionsValidator.cs`, `Host.csproj`; registry and validator tests; `docker-dev/docker-compose.yml` central env	T7
`docs/Historian.md`, `docs/Configuration.md`, `CLAUDE.md`, `docker-dev/README.md`, the spec (final as-built pass)	T8
`docs/plans/2026-10-10-questdb-historian-live-gate.md`	T9

In W3, T4, T5 and T6 must NOT edit T3's files. If one of them needs a connection helper T3 did not provide, it adds a private static helper inside its own file, and the fix wave dedupes. None of T4–T6 edit any csproj or the slnx, which T3 already wired.

Task 0: QuestDB spike, overlay compose and spec amendments

Goal: prove every QuestDB behaviour the design assumes, against a pinned image, before any code is built on it. The findings are written into spec §2–§4.

Files:

Create: docker-dev/docker-compose.questdb.yml
Create: docs/plans/2026-10-10-questdb-spike.md
Modify: docs/plans/2026-10-10-questdb-historian-design.md (§2–§4: replace assumptions with confirmed forms, and mark each one "T0-confirmed")

Acceptance Criteria:

The overlay pins an exact questdb/questdb:<x.y.z> tag. The tag's release date is at least two weeks before 2026-10-10, and is recorded in the spike doc.
The overlay exposes 18812:8812 and 19000:9000, uses a named volume otopcua-questdb-data, and sets the container name otopcua-dev-questdb-1. It is brought up with docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d questdb.
The spike doc records a working Npgsql connection string and answers Q1–Q12 below, each with the exact SQL or C# that worked and its observed output.
Spec §2.1 has the confirmed DDL, §2.3 the confirmed visibility-wait query, §3 the confirmed bucket-statistics and neighbour query forms, and §4 the confirmed connection-string keywords.

Verify: docker exec otopcua-dev-questdb-1 true exits 0. The spike harness output is pasted into the spike doc.

Steps:

Step 1: Create the overlay.

# docker-dev/docker-compose.questdb.yml — optional QuestDB fixture for the QuestDB historian backend.
# Up: docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d questdb
services:
  questdb:
    image: questdb/questdb:<PINNED>   # T0 pins an exact tag ≥ 2 weeks old
    container_name: otopcua-dev-questdb-1
    ports:
      - "18812:8812"   # PG wire
      - "19000:9000"   # web console / REST
    volumes:
      - otopcua-questdb-data:/var/lib/questdb
    healthcheck:
      test: ["CMD-SHELL", "curl -fs http://localhost:9003/status || exit 1"]
      interval: 10s
      timeout: 5s
      retries: 12
volumes:
  otopcua-questdb-data:

Step 2: Write a throwaway C# harness in the scratchpad, not in the repo. It is a console project that references Npgsql, restored from the existing central pin via a temporary Directory.Packages.props copy, or run with dotnet run inside a scratch folder under the repo with ManagePackageVersionsCentrally. Answer each question with real output:
- Q1. The connection-string keywords that work: test ServerCompatibilityMode=NoTypeLoading with and without it. Also the parameter types needed for TIMESTAMP (UTC DateTime), BYTE, SYMBOL, VARCHAR, BOOLEAN and LONG.
- Q2. Whether a parameterised multi-row INSERT INTO t VALUES (…),(…) works, and the largest row count per statement that works reliably. Pick the batch size: ≤ 1000 rows, capped by any parameter limit found.
- Q3. CREATE TABLE … WAL DEDUP UPSERT KEYS(ts, tag): re-inserting the same key replaces the row. Show before and after.
- Q4. The wal_tables() columns. After an insert, how to know the transaction we produced and wait until it is applied (writerTxn >= sequencerTxn). Measure the typical lag over 20 runs.
- Q5. TTL: the syntax at create (TTL n DAYS), and ALTER TABLE … SET TTL n DAYS. Where TTL shows up in tables() metadata.
- Q6. Timestamp precision through Npgsql: a value with 100 ns ticks (…1234567) comes back as …123456 (truncated) or …123457 (rounded)? Confirm that our pre-truncation makes the round trip exact.
- Q7. Two connections run CREATE TABLE IF NOT EXISTS for the same table at once, 20 times. Record any error text, and the catch rule that makes it idempotent.
- Q8. Bucket statistics: one GROUP BY over the expression (cast(ts AS LONG)*10 - $1) / $2, with sum(CASE WHEN … THEN 1 ELSE 0 END), avg, min, max, sum. Plus the first and last usable sample per bucket, with its value and time. Try first()/last() with a filter, or a window row_number() OVER (PARTITION BY b ORDER BY ts). Plus arg-min/arg-max time, via a window function or a second statement.
- Q9. AtTime neighbour lookup: chunked LIMIT 1 seeks (ts <= t ORDER BY ts DESC LIMIT 1 and ts > t ORDER BY ts LIMIT 1, combined with UNION ALL per time) versus ASOF JOIN. Time both for 100 timestamps and pick one.
- Q10. VARCHAR and SYMBOL accept a 400-character string, and SYMBOL capacity is fine for tens of thousands of tags.
- Q11. An existing-timestamp lookup for a batch: WHERE tag=$1 AND ts >= $2 AND ts <= $3 (a range), or an IN list. Pick the one that works with parameters.
- Q12. DROP TABLE IF EXISTS works for test cleanup, and tables() lists the tables with a given prefix.
Step 3: Write the spike doc with the image tag and date, the connection string, and Q1–Q12 (SQL, output and decision for each).
Step 4: Amend the spec §2–§4 in place with the confirmed forms. If any assumption failed, write the replacement design inline and add a ledger ruling. If the failure breaks a user decision, stop and ask.
Step 5: Commit

git add docker-dev/docker-compose.questdb.yml docs/plans/2026-10-10-questdb-spike.md docs/plans/2026-10-10-questdb-historian-design.md
git commit -m "spike(historian): QuestDB behaviours confirmed against pinned image; spec §2–§4 amended

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe"

{"files":["docker-dev/docker-compose.questdb.yml","docs/plans/2026-10-10-questdb-spike.md","docs/plans/2026-10-10-questdb-historian-design.md"],"verifyCommand":"docker exec otopcua-dev-questdb-1 true","acceptanceCriteria":["pinned image ≥2 weeks old","overlay ports 18812/19000","Q1–Q12 answered with output","spec §2–§4 amended"],"modelTier":"frontier","model":"controller","wave":0}

Task 1: Extract `Driver.Historian.Sql.Common` (no behaviour change)

Goal: move every database-independent piece of Driver.Historian.SqlServer into a new shared library. The SQL Server backend then calls it, with byte-identical behaviour.

Files:

Create: src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.csproj
Create in that project: HistorianStatusCodes.cs, StatusBits.cs, HistorianInfoBits.cs, HistorianDataTypes.cs, SampleColumns.cs, HistorianSampleRow.cs, StoredSample.cs, BucketRow.cs, AggregateFamily.cs, AggregateEvaluator.cs, AtTimeEvaluator.cs, BucketPlan.cs, LiveWritePreparer.cs, HistoryUpdatePlanner.cs, HistorianTagNames.cs, BootstrapFailureCache.cs, HistorianSchemaMissingException.cs, AlarmEventIds.cs
Create: tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/ (csproj + tests)
Modify: every file in src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/ that contained the moved code (SqlServerHistorianDataSource.cs, SqlAggregateQueries.cs, SqlHistorianDataTypes.cs (delete), SqlHistorianStatusCodes.cs (keep only SQL Server-only codes, or delete if none), SampleRowMapper.cs, SqlServerHistorianValueWriter.cs, SqlServerHistorianDataUpdater.cs, SqlServerTagCatalog.cs, SqlServerSchemaBootstrapper.cs, the csproj)
Modify: the SqlServer unit tests that tested moved code. Move them into Sql.Common.Tests under the same test name.
Modify: ZB.MOM.WW.OtOpcUa.slnx (add both new projects alphabetically, next to lines 30 and 104)
Modify: tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests.csproj (add a ProjectReference to Sql.Common)

Acceptance Criteria:

Sql.Common has no reference to Microsoft.Data.SqlClient or any other database client. Its references are Core.Abstractions, Server/Runtime (only if needed for HistorizationValue) and Microsoft.Extensions.Logging.Abstractions.
The public API matches the signatures in Step 1 exactly. T2–T7 depend on them.
The SqlServer integration suite passes with the same test count as before the change, which is recorded in the report. The SqlServer unit suite plus the moved tests in Sql.Common.Tests has the same total count.
New Sql.Common.Tests unit tests cover AtTimeEvaluator, AggregateEvaluator, BucketPlan, LiveWritePreparer and HistoryUpdatePlanner directly, using the literals from the SqlServer integration tests. Each one is falsified once.
StatusCodeParity passes, and its constant count does not drop.

Verify:

dotnet build ZB.MOM.WW.OtOpcUa.slnx gives 0 errors.
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests passes.
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests passes.
With HISTORIAN_SQL_TEST_CONNECTION set, dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests passes with the same count as before.
dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests --filter "FullyQualifiedName~StatusCodeParity" passes.

Steps:

Step 0: Record the baseline. Run both SqlServer test projects and the parity filter, and write the exact pass counts into the report.
Step 1: Create the shared API. Namespace: ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common. Each moved body is transplanted verbatim from the source named in the comment; only names and visibility change.

// HistorianStatusCodes.cs — from SqlHistorianStatusCodes + SqlHistorianUpdateStatusCodes (all consts, same values)
public static class HistorianStatusCodes
{
    public const uint Good = 0x00000000u, GoodEntryInserted = 0x00A20000u, GoodEntryReplaced = 0x00A30000u;
    public const uint BadNoData = 0x809B0000u, BadEntryExists = 0x809F0000u, BadNoEntryExists = 0x80A00000u;
    public const uint BadInvalidTimestamp = 0x80230000u, BadAggregateInvalidInputs = 0x80D60000u;
    public const uint UncertainDataSubNormal = 0x40A40000u, BadOutOfRange = 0x803C0000u, BadInvalidArgument = 0x80AB0000u;
}
// StatusBits.cs / HistorianInfoBits.cs — moved verbatim from SqlHistorianStatusCodes.cs (made public)
// HistorianDataTypes.cs — from SqlHistorianDataTypes: DefaultForEmpty, byte FromKind(HistorianValueKind)
// SampleColumns.cs — record struct SampleColumns(byte ValueKind, double? NumValue, long? IntValue, string? StrValue)
//   + static class SampleColumnMapper { ToColumns(HistorianValue); FromColumns(byte,double?,long?,string?);
//     int ToDbStatus(uint); uint FromDbStatus(int); long ToLongStatus(uint) => status; uint FromLongStatus(long) }
// HistorianSampleRow.cs — record struct HistorianSampleRow(DateTime TimeUtc, HistorianValue Value, uint StatusCode)  (was SqlSampleRow)
public readonly record struct StoredSample(DateTime TimeUtc, HistorianValue Value, uint Status)
{
    public bool IsUsable { get; }   // verbatim rule from SqlServerHistorianDataSource
    public bool IsGood { get; }
    /// <summary>Applies the quality clamp (NULL value + non-Bad ⇒ BadNoData) — from ReadStored line 533.</summary>
    public static StoredSample FromStored(DateTime timeUtc, HistorianValue value, uint status);
}
public sealed class BucketRow   // fields verbatim from the private BucketRow, as public get/set properties
{
    public long Bucket { get; set; } public int Total { get; set; } public int Usable { get; set; }
    public int Good { get; set; } public int Bad { get; set; } public int NonNumeric { get; set; }
    public double? Avg { get; set; } public double? Min { get; set; } public double? Max { get; set; } public double? Sum { get; set; }
    public HistorianValue First { get; set; } public HistorianValue Last { get; set; }
    public double? Extreme { get; set; } public DateTime? ExtremeTime { get; set; }
}
public enum AggregateFamily { Simple, StartEnd, ActualTime }
public static class AggregateFamilies { public static AggregateFamily? FamilyOf(HistoryAggregateType a); }  // from SqlAggregateQueries.FamilyOf
public static class AggregateEvaluator
{
    public static DataValueSnapshot Evaluate(HistoryAggregateType a, BucketRow r, DateTime bucketStart, DateTime now); // verbatim
    public static DataValueSnapshot NoData(DateTime t, DateTime now);                                                  // verbatim
}
public static class AtTimeEvaluator
{
    public static DataValueSnapshot Evaluate(DateTime t, StoredSample? prev, StoredSample? next, DateTime serverTime, DateTime now); // verbatim AtTimeValue
}
public sealed class BucketPlan   // from ReadProcessedAsync lines 355-370 and 395-408
{
    public const int MaxBuckets = 100_000;
    public DateTime StartUtc { get; }  public DateTime EndUtc { get; }  public bool Reversed { get; }
    public long IntervalTicks { get; } public long BucketCount { get; }
    public long StartUnixTicks => (StartUtc - DateTime.UnixEpoch).Ticks;   // for QuestDB's (cast(ts AS LONG)*10 - start)/interval
    /// <exception cref="ArgumentOutOfRangeException">BucketCount > MaxBuckets (param name "interval") — same as today.</exception>
    public static BucketPlan Create(DateTime startUtc, DateTime endUtc, TimeSpan interval);
    public DateTime BucketStart(long index);
    /// <summary>One snapshot per bucket in window order; missing buckets get <paramref name="empty"/>; reversed if the window was.</summary>
    public IReadOnlyList<DataValueSnapshot> Assemble(IReadOnlyDictionary<long, DataValueSnapshot> byBucket, Func<DateTime, DataValueSnapshot> empty);
}
public static class LiveWritePreparer   // from SqlServerHistorianValueWriter.Prepare, plus a resolution parameter
{
    /// <param name="resolutionTicks">1 = keep 100 ns (SQL Server); 10 = truncate to µs BEFORE de-duplication (QuestDB).</param>
    public static List<HistorianSampleRow> Prepare(IReadOnlyList<HistorizationValue> values, Func<DateTime> utcNow, long resolutionTicks = 1);
}
public enum UpsertOutcome { Inserted, Replaced }
public sealed class UpdatePlan
{
    public uint?[] Results { get; }   // pre-decided codes (invalid ts, NaN, superseded duplicate…); null = decided by the backend
    public IReadOnlyList<(int Index, HistorianSampleRow Row)> Winners { get; }   // rows to send, de-duplicated per mode
}
public static class HistoryUpdatePlanner   // pre/post passes from SqlServerHistorianDataUpdater.UpdateDataAsync lines 56-108
{
    public static UpdatePlan Plan(string tag, HistoryUpdateMode mode, IReadOnlyList<HistorizationValue> values, long resolutionTicks = 1);
    /// <param name="outcomeByTick">ticks of each winner that the backend actually wrote → Inserted/Replaced; absent ⇒ not written.</param>
    public static IReadOnlyList<uint> Complete(UpdatePlan plan, HistoryUpdateMode mode, IReadOnlyDictionary<long, UpsertOutcome> outcomeByTick);
    public static IReadOnlyList<uint> DeleteAtTimeResults(IReadOnlyList<DateTime> times, ISet<long> deletedTicks, bool tagKnown);
    public static uint DeleteRawResult(bool tagKnown, long rowsAffected);
}
public static class HistorianTagNames { public const int MaxLength = 400; public static bool IsValid(string? name); }  // from SqlServerTagCatalog.IsValidName
public sealed class BootstrapFailureCache   // the 5 s negative-cache mechanism from SqlServerSchemaBootstrapper
{
    public BootstrapFailureCache(TimeProvider clock, TimeSpan duration);
    public void ThrowIfCached();  public void Record(Exception ex);  public void Clear();
}
// HistorianSchemaMissingException.cs — moved from SqlServerHistorianOptions.cs (public sealed, ctor(string message))
// AlarmEventIds.cs — public static string ComputeEventId(...) moved verbatim from SqlServerAlarmHistorianWriter (same signature)

Step 2: Point the SqlServer backend at the shared lib.
- Replace each moved body with a call into Sql.Common.
- Delete the now-empty SqlServer classes.
- Keep SqlAggregateQueries' SQL text and ReadBucket, now filling the shared BucketRow.
- Keep SampleRowMapper's TVP builders, which take HistorianSampleRow.
Step 3: Move the SqlServer unit tests for the moved code into Sql.Common.Tests, keeping test names and literals: StatusCodeOracleTests, ValueWriterPrepareTests, the column round-trip parts of SampleRowMapperTests, and the data-type table test. The TVP parts stay in SqlServer.Tests.
Step 4: Add direct unit tests that did not exist before. For example:

[Fact] public void AtTime_inside_an_outage_returns_the_bad_neighbours_code_with_no_value()
{
    var t0 = new DateTime(2026,10,9,8,0,0,DateTimeKind.Utc);
    var prev = new StoredSample(t0, HistorianValue.Empty, 0x80310000u);
    var next = new StoredSample(t0.AddSeconds(10), HistorianValue.FromDouble(5), 0);
    var r = AtTimeEvaluator.Evaluate(t0.AddSeconds(5), prev, next, t0, t0.AddHours(1));
    r.StatusCode.ShouldBe(0x80310000u); r.Value.ShouldBeNull();
}
[Fact] public void BucketPlan_rejects_more_than_100k_buckets()
    => Should.Throw<ArgumentOutOfRangeException>(() =>
        BucketPlan.Create(DateTime.UnixEpoch, DateTime.UnixEpoch.AddSeconds(100_001), TimeSpan.FromSeconds(1)));
[Fact] public void Prepare_truncates_to_microseconds_before_dedup_when_resolution_is_10()
{
    var t = new DateTime(2026,10,9,8,0,0,DateTimeKind.Utc).AddTicks(1234567);
    var rows = LiveWritePreparer.Prepare([new(t, HistorianValue.FromDouble(1), 0), new(t.AddTicks(3), HistorianValue.FromDouble(2), 0)], () => t, 10);
    rows.Count.ShouldBe(1); rows[0].TimeUtc.Ticks.ShouldBe(t.Ticks - t.Ticks % 10); rows[0].Value.Number.ShouldBe(1);
}

Add the matching AggregateEvaluator cases, using the 8-row ProcessedRead literals from ProcessedReadTests.Numeric_aggregates_over_the_fixture. Build the BucketRows by hand from tag A's samples.

Step 5: Wire the solution and parity test. Add both projects to the slnx, and the ProjectReference to the Core.Abstractions.Tests csproj.
Step 6: Run the Verify commands. Counts must equal the Step 0 baseline, plus the new direct tests. Falsify each new direct test once by reverting the rule it pins.
Step 7: Commit with refactor(historian): extract Driver.Historian.Sql.Common from the SQL Server backend (no behaviour change) and the Claude-Session trailer.

{"files":["src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/","tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/","src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/","tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests/","ZB.MOM.WW.OtOpcUa.slnx","tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests.csproj"],"verifyCommand":"dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests (env set) — same count as baseline","acceptanceCriteria":["no DB client ref in Sql.Common","public API exactly as Step 1","SqlServer suites same counts","direct evaluator tests falsified","parity green"],"modelTier":"frontier","model":"opus","wave":1}

Task 2: Conformance suite (shared source), run against SQL Server

Goal: one backend-neutral suite of hand-computed behaviour tests, compiled into the SqlServer integration project and passing there. T7 later compiles the same suite into the QuestDB project.

Files:

Create: tests/Drivers/Historian.Conformance/IConformanceBackend.cs, ConformanceData.cs, RawReadConformance.cs, AtTimeConformance.cs, ProcessedConformance.cs, QualityContractConformance.cs, LiveWriteConformance.cs, UpdateConformance.cs, TagNameConformance.cs
Modify: tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/*.csproj (add <Compile Include="..\Historian.Conformance\**\*.cs" LinkBase="Conformance" /> and a ProjectReference to Sql.Common)
Create in the SqlServer IT project: SqlServerConformanceBackend.cs, SqlServerConformance.cs (one concrete subclass per conformance class)
Modify in the SqlServer IT project: remove from the original files the generic tests that moved into the suite (RawReadTests, AtTimeReadTests, ProcessedReadTests, QualityContractTests, and the generic parts of WriteTests, UpdateTests and LongNameTests). SQL Server-specific tests stay where they are.

Acceptance Criteria:

Every moved test keeps its method name and literal expectations. git log -p shows a move, not a rewrite.
The SqlServer IT total count after the change is at least the pre-change count, and the report has the before and after count.
Suite code references only IConformanceBackend and Core.Abstractions types. There is no SqlServer* type and no T-SQL in tests/Drivers/Historian.Conformance.
Timestamp assertions go through B.Round(t).
Any test that cannot hold for every backend is a virtual member with a doc comment naming the difference. None should be needed for SQL Server.

Verify: with HISTORIAN_SQL_TEST_CONNECTION set, dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests passes with a count of at least the baseline. Without the variable, every test skips.

Steps:

Step 1: Define the backend contract.

namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance;
public interface IConformanceBackend : IAsyncDisposable
{
    /// <summary>1 for SQL Server (100 ns), 10 for QuestDB (µs).</summary>
    long ResolutionTicks { get; }
    DateTime Round(DateTime t) => new(t.Ticks - t.Ticks % ResolutionTicks, DateTimeKind.Utc);
    IHistorianDataSource CreateDataSource(TimeProvider? clock = null);
    IHistorianValueWriter CreateValueWriter(Func<DateTime>? utcNow = null);
    IHistorianDataUpdater CreateUpdater();
    /// <summary>Writes rows directly in the backend's schema (never through the writer), status verbatim.</summary>
    Task SeedAsync(string tag, params (DateTime T, HistorianValue V, uint Status)[] samples);
    /// <summary>Live (non-deleted) rows for a tag, ascending, read directly from storage.</summary>
    Task<IReadOnlyList<(DateTime T, HistorianValue V, uint Status)>> ReadBackAsync(string tag);
}
public abstract class ConformanceBase : IAsyncLifetime
{
    protected IConformanceBackend B { get; private set; } = null!;
    protected abstract Task<IConformanceBackend> CreateBackendAsync();   // calls fixture.SkipIfUnavailable()
    public async ValueTask InitializeAsync() => B = await CreateBackendAsync();
    public async ValueTask DisposeAsync() => await B.DisposeAsync();
}

ConformanceData holds T0, At(seconds), the status consts and SeedAAsync(B), verbatim from ReadTestDb.

Step 2: Move the generic tests listed in the T1 inventory (Raw, AtTime, Processed literals, the quality contract, live-write redelivery and first-wins, update result codes, tag names over 400 characters) into the suite classes. Write each one as an abstract class RawReadConformance : ConformanceBase with [Fact] methods.
Step 3: Implement SqlServerConformanceBackend. It reuses SqlHistorianFixture, a unique schema and the existing seeding SQL from ReadTestDb. Add one concrete class per suite: [Collection(SqlHistorianCollection.Name)] public sealed class SqlServerRawReadConformance(SqlHistorianFixture fx) : RawReadConformance.
Step 4: Run the tests and compare counts. Commit with test(historian): backend conformance suite (shared source), run against SQL Server and the trailer.

{"files":["tests/Drivers/Historian.Conformance/","tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/"],"verifyCommand":"dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests (env set) ≥ baseline count","acceptanceCriteria":["tests moved by name with same literals","count ≥ baseline","suite is backend-neutral","Round() for timestamps"],"modelTier":"standard","model":"sonnet","wave":2}

Task 3: QuestDB skeleton (options, connection, bootstrapper, fixture)

Goal: a building Driver.Historian.QuestDb project. It has validated options, a pooled Npgsql connection, an idempotent and concurrency-safe schema bootstrapper with TTL, and an env-gated integration fixture.

Files:

Create: src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.csproj (references Core.Abstractions, Core.AlarmHistorian, Server/Runtime and Sql.Common; packages Npgsql and Microsoft.Extensions.Logging.Abstractions; InternalsVisibleTo the two QuestDb test assemblies)
Create: QuestDbHistorianOptions.cs, QuestDbConnection.cs, QuestDbTables.cs, QuestDbSchemaBootstrapper.cs
Create: tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/ (csproj, QuestDbHistorianOptionsTests.cs, QuestDbTablesTests.cs)
Create: tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/ (csproj with the conformance-include line and a ProjectReference to Sql.Common, QuestDbFixture.cs, BootstrapTests.cs)
Modify: ZB.MOM.WW.OtOpcUa.slnx, and the Core.Abstractions.Tests csproj (ProjectReference to the QuestDb project)

Acceptance Criteria:

QuestDbHistorianOptions has ConnectionString (default ""), TablePrefix ("hist_", regex ^[A-Za-z0-9_]+$), RetentionDays (int?, ≥ 1), CommandTimeout (30 s, > 0), AutoCreateSchema (true) and VisibilityTimeout (5 s, ≥ 0). Validate() returns error strings that name the key ServerHistorian:QuestDb:<Key>.
QuestDbTables produces the three CREATE TABLE IF NOT EXISTS statements, as confirmed by T0, for a given prefix and retention. Unit tests pin the exact text.
The bootstrapper is idempotent and survives two concurrent instances, using the catch rule from T0 Q7. It re-applies a changed TTL. It uses BootstrapFailureCache for the 5 s negative cache.
QuestDbFixture reads HISTORIAN_QUESTDB_TEST_CONNECTION and skips when it is absent. UniquePrefix() returns "t" + 12 hex + "_". DropPrefixAsync(prefix) drops every table with that prefix, found through tables().
BootstrapTests cover:
- tables created;
- a second run is a no-op;
- two concurrent bootstrappers both succeed;
- TTL is visible in metadata when RetentionDays=7 and absent when unset;
- AutoCreateSchema=false against a missing table throws HistorianSchemaMissingException from Sql.Common.

Verify:

dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests passes.
With HISTORIAN_QUESTDB_TEST_CONNECTION set, dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests --filter "FullyQualifiedName~Bootstrap" passes.
Without the variable, every test skips.

Steps:

Step 1: write the options with Validate() and its unit tests (red, then green).
Step 2: write QuestDbTables DDL builders (use the spec §2.1 as amended by T0), with unit tests pinning the text for prefix hist_ with and without TTL.
Step 3: write QuestDbConnection.
- Members:
  - NpgsqlDataSource built once;
  - OpenAsync(ct);
  - CreateCommand(conn, sql) with CommandTimeout;
  - static bool IsTransient(Exception), true for NpgsqlException with IsTransient, socket and IO errors, and timeouts;
  - static DateTime ReadUtc(NpgsqlDataReader r, int i);
  - string T(string table), which returns prefix + table.
- Use the parameter typing from T0 Q1.
Step 4: write the bootstrapper and fixture. Bring up the overlay (docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d questdb), then write and run BootstrapTests.
Step 5: wire the slnx and the parity-test reference, then build the solution.
Step 6: commit feat(historian): QuestDB backend skeleton — options, connection, schema bootstrap with TTL with the trailer.

{"files":["src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/","tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/","tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/","ZB.MOM.WW.OtOpcUa.slnx"],"verifyCommand":"dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests --filter FullyQualifiedName~Bootstrap (env set)","acceptanceCriteria":["options validate naming keys","DDL pinned by unit tests","bootstrap idempotent + concurrent + TTL","fixture env-gated with unique prefix"],"modelTier":"standard","model":"sonnet","wave":2}

Task 4: QuestDB writes (value writer, tag provisioner, alarm writer)

Goal: the recorder, provisioning and alarm-history seams write to QuestDB with the same semantics as SQL Server.

Files:

Create: src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianValueWriter.cs, QuestDbTagProvisioner.cs, QuestDbAlarmHistorianWriter.cs
Create: tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/WriteTests.cs, ProvisioningTests.cs, AlarmWriterTests.cs

Acceptance Criteria:

The value writer works as follows:
- It calls LiveWritePreparer.Prepare(values, utcNow, resolutionTicks: 10).
- It rejects an invalid tag name (HistorianTagNames.IsValid) and returns false.
- It reads the existing timestamps for the tag in the batch range, including tombstones, using the T0 Q11 form.
- It inserts only the rows that are not yet present, in multi-row parameterised INSERTs of at most the T0 Q2 batch size, with deleted=false and server_ts=utcNow().
- It returns true with no SQL when every row already existed.
- Any exception returns false. It never throws, and cancellation also returns false.
The provisioner upserts hist_tag (ts = 1970-01-01, data type from HistorianDataTypes.FromKind, eng unit, description). Both the request field types and the result tally match the SqlServer provisioner's.
The alarm writer works as follows:
- It uses the same deterministic event_id as the SQL Server writer, through AlarmEventIds.ComputeEventId from Sql.Common.
- It inserts events through the dedup key (ts, event_id), so a redelivery is idempotent.
- Classify maps an unknown or non-database exception to RetryPlease, and maps permanent errors as the SqlServer writer does.
WriteTests cover:
- typed round trip for all six kinds with a non-Good status;
- a redelivered batch leaves one row and the first value;
- a tombstoned key is not resurrected by redelivery (seed a tombstone directly);
- NaN ⇒ BadOutOfRange Empty;
- a microsecond collision within a batch keeps the first;
- a stopped port (Port=1) returns false without throwing.
Each regression test is falsified once.

Verify: with HISTORIAN_QUESTDB_TEST_CONNECTION set, dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests --filter "FullyQualifiedName~Write|FullyQualifiedName~Provisioning|FullyQualifiedName~AlarmWriter" passes.

Steps:

Step 1: write WriteTests (red).
Step 2: implement the value writer.
Step 3: make WriteTests green, then falsify the first-wins, tombstone and NaN tests.
Step 4: do the provisioner, then the alarm writer, each tests-first.
Step 5: commit feat(historian): QuestDB live writes, provisioning and alarm events with the trailer.

{"files":["src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianValueWriter.cs","src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbTagProvisioner.cs","src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbAlarmHistorianWriter.cs","tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/WriteTests.cs","tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/ProvisioningTests.cs","tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/AlarmWriterTests.cs"],"verifyCommand":"dotnet test …QuestDb.IntegrationTests --filter Write|Provisioning|AlarmWriter (env set)","acceptanceCriteria":["insert-if-absent incl tombstones","never throws","event id deterministic + idempotent","falsified"],"modelTier":"standard","model":"sonnet","wave":3}

Task 5: QuestDB reads and aggregates

Goal: QuestDbHistorianDataSource serves Raw, AtTime, Events and Processed. Results are identical to SQL Server's, except for microsecond rounding.

Files:

Create: src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataSource.cs, QuestDbAggregateQueries.cs
Create: tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/ReadTests.cs

Acceptance Criteria:

Every query filters NOT deleted. Every row goes through StoredSample.FromStored.
Raw: forward and reverse, LIMIT n+1, and the same paging contract and continuation semantics as the SqlServer data source (re-read from the last timestamp, inclusive). Unbounded when maxValuesPerNode == 0. Unknown tag ⇒ an empty result. The tag is resolved through the hist_tag existence or the sample table, so the read never creates a tag.
AtTime: the result list matches the input in length and order. Neighbours come from the T0 Q9 form, and AtTimeEvaluator.Evaluate decides each value.
Processed:
- It uses BucketPlan.Create and AggregateFamilies.FamilyOf.
- It runs one statement per node, grouped by (cast(ts AS LONG)*10 - @startUnixTicks) / @intervalTicks.
- It fills the shared BucketRow with the same usable/good/bad/non-numeric definitions as SQL Server: usable = status severity not Bad and kind ≠ Empty; good = usable and status < 0x40000000; non-numeric = usable and kind in (4, 5).
- AggregateEvaluator.Evaluate produces the values, and BucketPlan.Assemble fills empty buckets and handles reversal.
- An unsupported aggregate throws HistoryAggregateNotSupportedException.
Events: filtered by source and time range, ascending, limited to maxEvents. The ContinuationPoint is non-null when truncated.
Health counters (GetHealthSnapshot) are implemented as in the SqlServer data source (lock-guarded success/failure counters plus the last error). A failure fails only that call and is recorded in GetHealthSnapshot.
ReadTests cover only QuestDB-specific behaviour:
- tombstoned rows are invisible to Raw, AtTime and Processed (seed a tombstone directly);
- a whole-range Processed read from 1601 to 9999 with interval 0 does not overflow;
- a reverse paging walk with maxValuesPerNode=1 returns every row. The shared literals run through the conformance suite in T7.

Verify: with HISTORIAN_QUESTDB_TEST_CONNECTION set, dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests --filter "FullyQualifiedName~ReadTests" passes.

Steps:

Step 1: write ReadTests (red).
Step 2: implement Raw, then AtTime, Events and Processed.
Step 3: temporarily compile the conformance read classes into this task's worktree. Add a throwaway concrete subclass and a throwaway backend that seeds through direct INSERTs. Run the Raw, AtTime and Processed conformance classes locally and get them green.
Step 4: delete the throwaway files before committing. T7 owns the real ones. Paste the conformance run output into the report.
Step 5: falsify the tombstone filter and the bucket expression (an off-by-one start).
Step 6: commit feat(historian): QuestDB reads and aggregates (pushed down, shared evaluation) with the trailer.

{"files":["src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataSource.cs","src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbAggregateQueries.cs","tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/ReadTests.cs"],"verifyCommand":"dotnet test …QuestDb.IntegrationTests --filter FullyQualifiedName~ReadTests (env set) + throwaway conformance run in report","acceptanceCriteria":["NOT deleted everywhere","bucket expr exact vs SqlServer","shared evaluators used","conformance read classes green locally"],"modelTier":"frontier","model":"opus","wave":3}

Task 6: QuestDB HistoryUpdate (soft delete) with read-your-writes

Goal: QuestDbHistorianDataUpdater implements all five operations through insert-only upserts and tombstones. It returns codes identical to SQL Server's, and waits for WAL visibility before returning.

Files:

Create: src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataUpdater.cs, QuestDbWalVisibility.cs
Create: tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/UpdateTests.cs, TombstoneTests.cs

Acceptance Criteria:

UpdateDataAsync:
1. HistoryUpdatePlanner.Plan(tag, mode, values, resolutionTicks: 10) builds the plan.
2. It reads which winner timestamps currently have a live row, meaning present and not deleted.
3. It writes:
  - Insert: only the absent winners;
  - Replace: only the present winners;
  - Update: every winner. Each write is an INSERT with deleted=false.
4. It builds the outcome map (present ⇒ Replaced, absent ⇒ Inserted, for rows it wrote).
5. HistoryUpdatePlanner.Complete(...) produces the codes.
DeleteAtTimeAsync writes a tombstone for each time with a live row, with deleted=true, value columns NULL and status=BadNoData. Codes come from HistoryUpdatePlanner.DeleteAtTimeResults.
DeleteRawAsync writes a tombstone for every live row in [start, end). The code comes from HistoryUpdatePlanner.DeleteRawResult.
On an unknown tag, Insert and Update create the hist_tag row; Replace and Delete do not, and return the shared unknown-tag codes.
QuestDbWalVisibility.WaitAsync(table, ct) works as follows:
- It polls the T0 Q4 query until the table's applied transaction is at least the sequencer transaction captured after our write.
- It gives up after VisibilityTimeout and logs a Warning, without changing the codes. VisibilityTimeout = 0 skips the wait.
- Every updater operation calls it once, after writing.
UpdateTests cover the five operations' codes (QuestDB-specific checks only; the literal codes run through conformance in T7) and that a read made right after UpdateDataAsync returns sees the change.
TombstoneTests cover:
- delete then Raw read ⇒ the row is gone;
- delete, then a QuestDbHistorianValueWriter redelivery of the same sample ⇒ the row stays gone (this needs T4's writer: write the test against the interface, and if T4 is not merged yet, seed a tombstone and assert the writer's existence query counts it);
- insert over a tombstone ⇒ GoodEntryInserted and a visible row.

Verify: with HISTORIAN_QUESTDB_TEST_CONNECTION set, dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests --filter "FullyQualifiedName~UpdateTests|FullyQualifiedName~TombstoneTests" passes.

Steps:

Step 1: write the tests (red).
Step 2: implement QuestDbWalVisibility.
Step 3: implement the updater.
Step 4: get to green. Falsify the "live row" filter by dropping NOT deleted from the existence check, and the visibility wait by setting the timeout to 0 in the read-after-write test.
Step 5: commit feat(historian): QuestDB HistoryUpdate with soft delete and read-your-writes with the trailer.

{"files":["src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataUpdater.cs","src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbWalVisibility.cs","tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/UpdateTests.cs","tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/TombstoneTests.cs"],"verifyCommand":"dotnet test …QuestDb.IntegrationTests --filter UpdateTests|TombstoneTests (env set)","acceptanceCriteria":["insert-only upserts, no UPDATE/DELETE","codes from shared planner","tombstones never resurrected","read-your-writes wait"],"modelTier":"standard","model":"sonnet","wave":3}

Task 7: Provider, Host wiring and the conformance suite on QuestDB

Goal: Backend=QuestDb resolves, validates and builds all five seams. The full conformance suite passes against QuestDB.

Files:

Create: src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianProvider.cs
Create: tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformanceBackend.cs, QuestDbConformance.cs (one concrete subclass per conformance class)
Create: tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbHistorianProviderTests.cs
Modify:
- src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/HistorianBackendRegistry.cs (add a QuestDbBackendName const, BindQuestDb(IConfiguration) and a Factories entry);
- ServerHistorianOptionsValidator.cs (a QuestDb block before the if (!options.IsGatewayBackend) return; line, mirroring the SqlServer block);
- src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj (a ProjectReference);
- tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/Configuration/HistorianBackendRegistryTests.cs;
- tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/ServerHistorianOptionsValidatorTests.cs;
- docker-dev/docker-compose.yml: in the central-1 and central-2 ServerHistorian__* blocks, add ServerHistorian__QuestDb__ConnectionString: "${OTOPCUA_HISTORIAN_QUESTDB:-<T0 connection string with Host=questdb;Port=8812>}" plus a comment line naming the overlay.

Acceptance Criteria:

The provider has Name="QuestDb", SupportedKinds = all six kinds as a FrozenSet, and one lazily built shared connection and bootstrapper across the seams. The constructor does no I/O and no validation, like the SqlServer provider. CreateUpdater returns the soft-delete updater.
The registry resolves QuestDb case-insensitively.
The validator, when the backend is consumed and is QuestDb:
- reports every QuestDbHistorianOptions.Validate() error, naming the key;
- does not require the Gateway keys;
- ignores an invalid TablePrefix when not consumed.
Registry and validator tests mirror the five SqlServer ones (lines 71 and 319–368).
The QuestDB integration project runs the entire conformance suite through QuestDbConformanceBackend:
- ResolutionTicks = 10;
- seeding is direct INSERTs into the prefixed tables;
- ReadBackAsync filters NOT deleted. Every class passes. Any named override has a doc comment and a ledger ruling.

Verify:

With both environment variables set, the SqlServer and QuestDB integration projects both pass. The report shows the conformance class counts side by side.
dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests --filter "FullyQualifiedName~Historian" passes.
dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter "FullyQualifiedName~Historian" passes.

Steps:

Step 1: write the provider unit tests (red), then the provider.
Step 2: write the registry and validator tests (red), then the wiring.
Step 3: write QuestDbConformanceBackend and the subclasses, and run the full suite. Fix any divergence in the QuestDB code, even though those files belong to T4–T6; this task is the integration owner in W4.
Step 4: make the compose env edit.
Step 5: build the solution, then commit feat(historian): QuestDb provider + Host wiring; conformance suite green on both backends with the trailer.

{"files":["src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianProvider.cs","tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformanceBackend.cs","tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformance.cs","src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/HistorianBackendRegistry.cs","src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/ServerHistorianOptionsValidator.cs","src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj","docker-dev/docker-compose.yml"],"verifyCommand":"both IT projects green with env set; Host historian filters green","acceptanceCriteria":["provider shape","registry + consumer-gated validator","full conformance suite green on QuestDB","compose env"],"modelTier":"standard","model":"sonnet","wave":4}

Task 8: Documentation

Goal: the docs describe the QuestDB backend as built, and every claim is checked against the merged code.

Files: docs/Historian.md (a new §"QuestDB backend"), docs/Configuration.md (the ServerHistorian:QuestDb table), CLAUDE.md (one short bullet in the Pluggable backends subsection), docker-dev/README.md (the overlay recipe), and the spec (an as-built pass)

Acceptance Criteria:

Every ServerHistorian:QuestDb key is documented with its default and validation rule.
The QuestDB-specific behaviour is documented:
- soft delete and tombstones;
- insert-if-absent including tombstones;
- microsecond truncation and collisions;
- WAL visibility and the read-your-writes wait;
- TTL;
- the overlay recipe and HISTORIAN_QUESTDB_TEST_CONNECTION.
Each statement is cross-checked against the code, and any spec-versus-code disagreement is fixed in the spec inline and listed in the report.

Verify: the controller reads the diff, and grep -n "QuestDb" docs/Configuration.md shows the table.

{"files":["docs/Historian.md","docs/Configuration.md","CLAUDE.md","docker-dev/README.md","docs/plans/2026-10-10-questdb-historian-design.md"],"verifyCommand":"grep -n QuestDb docs/Configuration.md","acceptanceCriteria":["keys documented","QuestDB-specific semantics documented","claims cross-checked"],"modelTier":"standard","model":"sonnet","wave":5}

Final review, then Task 9: live gate on docker-dev (controller)

Final review: a fable reviewer reads git diff master...feat/questdb-historian together with the spec. It must report PASS or FAIL on each of these checks:

SqlServer behaviour is unchanged: identical conformance results, and no deleted SqlServer test.
No QuestDB UPDATE or DELETE is issued on data.
NOT deleted is present in every read path.
Every new status literal is a named const.
No package or version changes.
The bucket expression is exact.
#491 still holds: no Self/Context access after an await.

Then one fix dispatch (opus) and a scoped re-review (sonnet), followed by adjudication.

Task 9: live gate (controller only).

Goal: prove Backend=QuestDb end to end on the docker-dev rig, against the overlay questdb service, with captured output for every leg.

USER-ORDERED GATE — NON-SKIPPABLE. This task was requested by the user in the current conversation. It MUST NOT be closed by walking around it, by declaring it "verified inline", or by substituting a cheaper check. Close only after every item in acceptanceCriteria has been re-validated independently, with output captured.

Files: docs/plans/2026-10-10-questdb-historian-live-gate.md

Acceptance Criteria: run the six legs of spec §7 on docker-dev with OTOPCUA_HISTORIAN_BACKEND=QuestDb. Each leg needs captured output.

Verify: the evidence doc has each leg's CLI output plus the direct QuestDB query output.

Steps:

Bring up the overlay questdb.
Rebuild the central image and recreate the central nodes with OTOPCUA_HISTORIAN_ENABLED=true, OTOPCUA_HISTORIAN_BACKEND=QuestDb and OTOPCUA_CONTHIST_ENABLED=true.
Historize FastUInt1, a string calculation tag and HR5. Add a UNS reference. Mint the two gate API keys, as in the SQL Server gate recipe (docs/plans/2026-10-09-pluggable-historian-live-gate.md), and deploy.
Run legs 1–6 with direct psql or REST queries. QuestDB REST is curl -G localhost:19000/exec --data-urlencode "query=…".
Write the evidence doc.
Restore the rig: recreate the centrals with defaults, revert the config DB, redeploy, down the overlay, and stop the fixtures.

{"files":["docs/plans/2026-10-10-questdb-historian-live-gate.md"],"verifyCommand":"Client.CLI historyread/historyupdate against opc.tcp://localhost:4840 + QuestDB REST /exec on :19000","acceptanceCriteria":["Primary-only typed recording one row per value","raw+UNS HistoryRead/Processed identical and match QuestDB","HistoryUpdate codes + delete not resurrected by redelivery + WriteOperate-only and anonymous denied","failover 1x","Modbus loss: BadCommunicationError then BadNoCommunication marker then Good","TTL visible in tables() metadata"],"userGate":true,"tags":["user-gate"],"modelTier":"frontier","model":"controller","wave":5}

Finish

Merge to master and push only with the user's go-ahead.
After merging, append the scadaproj sister-project-history/otopcua.md entry on scadaproj main and push it.
Close #666.

58 KiB Raw Blame History Unescape Escape

QuestDB Historian Backend Implementation Plan

Global Constraints

Waves, models and parallel safety

Task 0: QuestDB spike, overlay compose and spec amendments

Task 1: Extract Driver.Historian.Sql.Common (no behaviour change)

Task 2: Conformance suite (shared source), run against SQL Server

Task 3: QuestDB skeleton (options, connection, bootstrapper, fixture)

Task 4: QuestDB writes (value writer, tag provisioner, alarm writer)

Task 5: QuestDB reads and aggregates

Task 6: QuestDB HistoryUpdate (soft delete) with read-your-writes

Task 7: Provider, Host wiring and the conformance suite on QuestDB

Task 8: Documentation

Final review, then Task 9: live gate on docker-dev (controller)

Finish

58 KiB

Raw Blame History

Task 1: Extract `Driver.Historian.Sql.Common` (no behaviour change)