Joseph Doherty
1081 lines
72 KiB
Markdown
1081 lines
72 KiB
Markdown
# QuestDB Historian Backend Implementation Plan
|
||
|
||
> **For agentic workers:** REQUIRED SUB-SKILL: use superpowers-extended-cc:subagent-driven-development (recommended) or
|
||
> superpowers-extended-cc:executing-plans to run this plan task by task. Steps use checkbox (`- [ ]`) syntax.
|
||
|
||
**Goal:** add `ServerHistorian:Backend=QuestDb`. It reaches full parity with the SQL Server backend, covering
|
||
recording, reads, the 13 aggregates and all five HistoryUpdate operations through soft delete, and adds table-TTL
|
||
retention.
|
||
|
||
**Architecture:** move the database-independent logic of `Driver.Historian.SqlServer` into a new shared library,
|
||
`Driver.Historian.Sql.Common`. Build `Driver.Historian.QuestDb` on top of it, using Npgsql over the Postgres wire
|
||
protocol. A shared-source conformance suite runs the same hand-computed expectations against both backends.
|
||
|
||
**Tech stack:** .NET 10, Npgsql 10.0.3 (already pinned), QuestDB as a WAL table with `DEDUP UPSERT KEYS`, xUnit v3 +
|
||
Shouldly, docker-dev.
|
||
|
||
**Spec:** `docs/plans/2026-10-10-questdb-historian-design.md`. Its §0 holds the user decisions. The T0 spike amends
|
||
§2–§4 before W1 starts.
|
||
|
||
## Global Constraints
|
||
|
||
These are binding on every task and every reviewer.
|
||
|
||
**Packages and build**
|
||
- Add no NuGet packages and change no versions. Never edit `Directory.Packages.props`. Referencing `Npgsql` from the
|
||
new QuestDb project is allowed, because it is already centrally pinned.
|
||
- Every new `src` csproj sets `TreatWarningsAsErrors=true`, `GenerateDocumentationFile=true` and `NoWarn CS1591`, as
|
||
`Driver.Historian.SqlServer.csproj` does. `TargetFramework` and `Nullable` are inherited from
|
||
`Directory.Build.props`.
|
||
- New `src` project names MUST match `ZB.MOM.WW.OtOpcUa.Driver.*`. `StatusCodeParityTests` scans only
|
||
`ZB.MOM.WW.OtOpcUa.Driver.*.dll`.
|
||
|
||
**Status codes**
|
||
- Every OPC UA status code a class emits is a named `const uint` whose name starts with `Good`, `Uncertain` or `Bad`.
|
||
Bit masks use names that do NOT start with those words (`CodeBitsMask`, `SeverityBadBit`).
|
||
- Any new Driver assembly must be a ProjectReference of `tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests`, so its
|
||
DLL lands in that test's bin and gets scanned.
|
||
|
||
**SQL Server backend**
|
||
- Its observable behaviour must not change. The SqlServer unit and integration suites stay green, and no test there
|
||
is deleted. A test may only *move* into the conformance suite, keeping the same name and the same literals.
|
||
|
||
**Quality and write rules**
|
||
- **Quality contract:** an Empty (null-value) sample is always served with a Bad status. NaN/±Inf and Empty-with-non-Bad
|
||
are stored as `BadOutOfRange` or `BadNoData` on the live write, and rejected with `BadInvalidArgument` by HistoryUpdate.
|
||
- **Live writes are insert-if-absent**, and the first duplicate in a batch wins. On QuestDB, an existing *tombstone*
|
||
also counts as present, so it blocks the insert and the row is never resurrected.
|
||
- **QuestDB writes:** every write is an `INSERT`, and the dedup upsert does the replacing. Never issue QuestDB `UPDATE`
|
||
or `DELETE` for sample or event data. `DROP TABLE` is allowed only in test cleanup.
|
||
- **Soft delete:** a delete writes a tombstone row (`deleted = true`, value columns NULL) at the same `(ts, tag)`. Every
|
||
read and aggregate filters `NOT deleted`.
|
||
|
||
**Time and buckets**
|
||
- **Timestamps:** QuestDB stores microseconds. The QuestDB writer and updater truncate to microseconds
|
||
(`ticks - ticks % 10`) before de-duplicating. Conformance tests compare timestamps through
|
||
`IConformanceBackend.Round(DateTime)`.
|
||
- **Bucket edges must be identical to SQL Server's.** The QuestDB bucket index is
|
||
`(cast(ts AS LONG) * 10 - @startUnixTicks) / @intervalTicks`, using integer ticks relative to the Unix epoch. Never use
|
||
`SAMPLE BY`.
|
||
|
||
**Infrastructure and tests**
|
||
- **docker-dev:** QuestDB is an *overlay file*, `docker-dev/docker-compose.questdb.yml`. The rig uses no compose
|
||
`profiles:`. Host ports are PG wire `18812` and web console `19000`.
|
||
- These ports are already taken and must not be reused: 14330, 14340, 14350, 4840–4845, 4850, 4851, 4860, 4861, 9200,
|
||
9210, 9220, 8089.
|
||
- **Test environment variables:** `HISTORIAN_QUESTDB_TEST_CONNECTION`, alongside the existing
|
||
`HISTORIAN_SQL_TEST_CONNECTION`. When a variable is absent, its tests skip through `Assert.Skip`.
|
||
- **Regression tests** must be falsified: revert the production change, watch the test go red, restore it, and record
|
||
what you observed.
|
||
- **Shell:** zsh, so quote globs and avoid a bare `====`.
|
||
- **Building:** put two projects in one `dotnet build` and it fails with MSB1008, so build them separately or use the
|
||
`.slnx`.
|
||
|
||
**User decisions (already made):**
|
||
- "Soft delete (Recommended)": the `deleted` flag plus tombstones, with all five HistoryUpdate operations.
|
||
- "PG wire only (Recommended)": Npgsql for everything, no ILP.
|
||
- "Include TTL (Recommended)": `RetentionDays` becomes table TTL.
|
||
- Approach B: a shared dialect-free core plus per-backend SQL, and one conformance suite run against both backends.
|
||
- Microsecond timestamps; two samples in one batch landing on the same microsecond collapse to one (first wins).
|
||
- Every design section was approved as written (2026-10-10). The plan gets a Fable review, then the session is
|
||
prepared for compaction before executing with subagents.
|
||
|
||
**Plan rulings (made while planning, from the code inventory):**
|
||
- **R1:** the spec §6 compose *profile* becomes an overlay file. The rig has no profiles; each optional fixture is
|
||
`docker-compose.<name>.yml`.
|
||
- **R2:** the conformance suite is a **shared-source directory**, `tests/Drivers/Historian.Conformance/`, which has no
|
||
csproj. Each integration project includes it with
|
||
`<Compile Include="..\Historian.Conformance\**\*.cs" LinkBase="Conformance" />`. The reason is that only `xunit.v3`
|
||
is pinned, and a class library referencing it would become a test project.
|
||
- **R3:** the shared types become `public` in `Sql.Common`, so no InternalsVisibleTo is needed for the two backends.
|
||
They are renamed from their `Sql*` names to neutral ones (table in T1). The parity test keys on field names, not
|
||
class names.
|
||
|
||
## Review amendments (Fable, 2026-10-10)
|
||
|
||
These are binding. They override anything else in this plan that conflicts with them. The full review is
|
||
summarised here as B = blocker, S = should-fix, N = nit.
|
||
|
||
**Global rulings**
|
||
- **R4 — resolution units (B2).** Conformance tests express every sub-second offset in units of `B.ResolutionTicks`:
|
||
- `AddTicks(k)` becomes `AddTicks(k * B.ResolutionTicks)`.
|
||
- Seeds are at least one resolution unit apart.
|
||
- A "near miss" is `+ B.ResolutionTicks`, and the last tick of a bucket is `edge.AddTicks(-B.ResolutionTicks)`.
|
||
- The 100k-bucket cap test uses `TimeSpan.FromTicks(B.ResolutionTicks)` intervals over a
|
||
`100_000 * B.ResolutionTicks` window.
|
||
- Every timestamp assertion, including `.Ticks` comparisons and `ServerTimestampUtc`, goes through `B.Round(..)`.
|
||
- Fixture bases (`T0`, `T(n)`) stay deliberately not µs-aligned.
|
||
- On SQL Server (`ResolutionTicks = 1`) every rewritten expression evaluates to today's literal.
|
||
- **R5 — both write paths wait for WAL visibility (B6).** `QuestDbWalVisibility` is a T3 file. The value writer
|
||
(T4) and the updater (T6) each call `WaitAsync` once after writing, bounded by `VisibilityTimeout`. A timeout logs a
|
||
Warning and does not change the return value or result codes. Spec §2.2's "The recorder does not wait for WAL
|
||
visibility" is replaced by "The recorder waits for WAL visibility, bounded by `VisibilityTimeout`."
|
||
- **R6 — microsecond domain at the boundary (B3, B4).**
|
||
- `QuestDbConnection.Round(DateTime)` (= `ticks - ticks % 10`, UTC kind) is applied to EVERY inbound timestamp:
|
||
- Raw and Events window bounds;
|
||
- each AtTime request time;
|
||
- DeleteRaw bounds and DeleteAtTime times;
|
||
- the Processed window, BEFORE `BucketPlan.Create`.
|
||
- Results are stamped with the rounded request time.
|
||
- `QuestDbConnection.BindTimestamp(cmd, name, DateTime)` is the ONLY way a DateTime reaches Npgsql. It rounds, and it
|
||
never sends a sentinel: `DateTime.MinValue` becomes `MinValue.AddTicks(10)`, and `MaxValue` rounds to `MaxValue − 9`
|
||
ticks. Binding a raw DateTime is a review FAIL.
|
||
- The Global Constraints "Bucket edges" rule therefore holds exactly. T8 documents that a request start with sub-µs
|
||
ticks admits a sample stored up to 9 ticks earlier.
|
||
- **R7 — one tag authority (B5).** T3 creates `QuestDbTagCatalog.cs`, mirroring `SqlServerTagCatalog`:
|
||
- `Task<bool> ExistsAsync(name, ct)`: positive results cached; an invalid name ⇒ false; runs bootstrap `EnsureAsync`
|
||
first.
|
||
- `Task EnsureAsync(name, byte dataType, ct)`: inserts into `hist_tag` with `ts = 1970-01-01` when absent.
|
||
- **The `hist_tag` row is the authority for "known tag".** T4's writer and T6's Insert/Update call `EnsureAsync`.
|
||
T5's reads and T6's Replace/DeleteRaw/DeleteAtTime call `ExistsAsync` and never create.
|
||
- T4–T6 add no private tag resolver.
|
||
- **R8 — Raw paging (S1).** `LIMIT @top`, where `top = maxValuesPerNode == 0 ? int.MaxValue : maxValuesPerNode`. Return
|
||
at most `top` samples with `ContinuationPoint: null`, exactly as the SqlServer data source does. The node manager does
|
||
the paging (it re-reads from the last timestamp, inclusive). The spec §3 "LIMIT n+1" is wrong and is replaced.
|
||
- **R9 — value columns (S3).** In the QuestDB DDL the value columns are `num_value`, `int_value` and `str_value`, to
|
||
avoid the reserved word `int`. The `SampleColumns` names in Sql.Common are unchanged.
|
||
- **R10 — models (S9, S10).**
|
||
- T7 runs on **opus**. It may fix T4/T6 files and the Raw/Events plumbing in `QuestDbHistorianDataSource.cs`.
|
||
- A divergence inside `QuestDbAggregateQueries.cs` or the AtTime neighbour logic is reported, not fixed; the
|
||
controller dispatches an opus fix.
|
||
- T1 runs as **two sequential opus dispatches**:
|
||
- **1a:** extract, SqlServer re-point, rename fixes, slnx and parity wiring, moved unit tests. Its gate is the Step 0
|
||
counts, reproduced exactly.
|
||
- **1b:** the new direct `Sql.Common.Tests`, each falsified.
|
||
- Two commits, one review each.
|
||
- **R11 — ownership (B1, N6, S8).**
|
||
- `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/**`: T1 in W1, for **mechanical
|
||
rename and using fixes only** (no test moved, renamed, re-literalled or deleted; the count stays at baseline), then
|
||
T2 in W2.
|
||
- `IT/QuestDbConformanceBackend.cs` (seed and read-back only) is owned by **T5**. T7 owns the subclasses
|
||
(`QuestDbConformance.cs`).
|
||
- `IT/TombstoneTests.Redelivery.cs` is owned by T7 (see the T6 amendment).
|
||
- **N4.** `Platforms AnyCPU;x64` goes on both new src csprojs.
|
||
|
||
---
|
||
|
||
## Waves, models and parallel safety
|
||
|
||
| Wave | Tasks (‖ = parallel) | Models |
|
||
|---|---|---|
|
||
| W0 | T0 QuestDB spike + overlay compose + spec amendments | **controller** (live docker) |
|
||
| W1 | T1 extract `Sql.Common` | **opus** |
|
||
| W2 | T2 conformance suite over SqlServer ‖ T3 QuestDB skeleton | sonnet ‖ sonnet |
|
||
| W3 | T4 writes ‖ T5 reads + aggregates ‖ T6 updater | sonnet ‖ **opus** ‖ sonnet |
|
||
| W4 | T7 provider + Host wiring + QuestDB conformance | **opus** (R10) |
|
||
| W5 | T8 docs → **fable** whole-branch review → one fix wave (**opus**) + scoped re-review (sonnet) → T9 live gate | sonnet / fable / opus / controller |
|
||
|
||
Reviewers:
|
||
- opus for T1 and T5;
|
||
- sonnet for the rest;
|
||
- sonnet for scoped re-reviews.
|
||
|
||
Each task runs in its own worktree, `.worktrees/qdb-T<N>`, branched from the current `feat/questdb-historian` head.
|
||
Merges happen in ascending task order within a wave, with `dotnet build ZB.MOM.WW.OtOpcUa.slnx` after each merge.
|
||
|
||
**File ownership.** A task edits only the files it owns. Anything else needs a controller ruling before it is
|
||
touched.
|
||
|
||
| Path | Owner |
|
||
|---|---|
|
||
| `docker-dev/docker-compose.questdb.yml`, `docs/plans/2026-10-10-questdb-spike.md`, spec §2–§4 amendments | T0 |
|
||
| `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/**`, `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/**` | T1 |
|
||
| `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/**`, `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests/**` | T1 |
|
||
| `ZB.MOM.WW.OtOpcUa.slnx`, `tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests/*.csproj` | T1 adds Sql.Common; then T3 adds the QuestDb lines |
|
||
| `tests/Drivers/Historian.Conformance/**`, `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/**` | T2 |
|
||
| `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/{csproj,QuestDbHistorianOptions.cs,QuestDbConnection.cs,QuestDbSchemaBootstrapper.cs,QuestDbTables.cs}` | T3 |
|
||
| `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/**` (csproj + options/bootstrap unit tests) | T3 |
|
||
| `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/{csproj,QuestDbFixture.cs,BootstrapTests.cs}` | T3 |
|
||
| QuestDb `QuestDbHistorianValueWriter.cs`, `QuestDbTagProvisioner.cs`, `QuestDbAlarmHistorianWriter.cs`; IT `WriteTests.cs`, `ProvisioningTests.cs`, `AlarmWriterTests.cs` | T4 |
|
||
| QuestDb `QuestDbHistorianDataSource.cs`, `QuestDbAggregateQueries.cs`; IT `ReadTests.cs` | T5 |
|
||
| QuestDb `QuestDbHistorianDataUpdater.cs`, `QuestDbWalVisibility.cs`; IT `UpdateTests.cs`, `TombstoneTests.cs` | T6 |
|
||
| QuestDb `QuestDbHistorianProvider.cs`; IT `QuestDbConformanceBackend.cs`, `Conformance*.cs`; Host `HistorianBackendRegistry.cs`, `ServerHistorianOptionsValidator.cs`, `Host.csproj`; registry and validator tests; `docker-dev/docker-compose.yml` central env | T7 |
|
||
| `docs/Historian.md`, `docs/Configuration.md`, `CLAUDE.md`, `docker-dev/README.md`, the spec (final as-built pass) | T8 |
|
||
| `docs/plans/2026-10-10-questdb-historian-live-gate.md` | T9 |
|
||
|
||
In W3, T4, T5 and T6 must NOT edit T3's files. If one of them needs a connection helper T3 did not provide, it adds a
|
||
`private static` helper inside its own file, and the fix wave dedupes. None of T4–T6 edit any csproj or the slnx, which
|
||
T3 already wired.
|
||
|
||
---
|
||
|
||
### Task 0: QuestDB spike, overlay compose and spec amendments
|
||
|
||
> **Review amendments (Fable) — binding:**
|
||
> - **Q1 is expanded (S2).** Record, with output:
|
||
> - the DateTime `Kind`/`NpgsqlDbType` pair that writes `TIMESTAMP` without error (watch for
|
||
> `Cannot write DateTime with Kind=UTC to … timestamp without time zone`);
|
||
> - whether open → close → reopen on one `NpgsqlDataSource` works without `No Reset On Close=true` (DISCARD ALL);
|
||
> - the CLR type returned for `BYTE` (likely `int2`, so use `GetInt16`), `SYMBOL`, `VARCHAR`, `LONG`, `BOOLEAN` and
|
||
> `TIMESTAMP`;
|
||
> - the parameter typing needed for `$n` in integer arithmetic (`NpgsqlDbType.Bigint`);
|
||
> - that a DOUBLE NaN reads as NULL.
|
||
> The confirmed keywords go into spec §4 verbatim.
|
||
> - **Q3 (S4):** also test `hist_event` with `event_id VARCHAR` as a dedup key. **Fallback:** no dedup on `hist_event`.
|
||
> The alarm writer checks `SELECT 1 … WHERE event_id=$1` before inserting, and `ReadEventsAsync` de-duplicates by
|
||
> `event_id` client-side.
|
||
> - **Q4 fallback (S11):** if `wal_tables()` lacks `writerTxn`/`sequencerTxn`, the visibility wait polls for the written
|
||
> key (`WHERE tag=$1 AND ts=$2 AND server_ts=$3`). Record which form is used.
|
||
> - **Q5 (S11):** the image must be ≥ 8.2.3, the first release with TTL. Record the floor.
|
||
> - **Q7 fallback (S11):** catch a `PostgresException` whose message contains `already exists`, then re-check
|
||
> `tables()`. Never swallow blindly.
|
||
> - **Q8 fallback (S11):** if neither a window `ORDER BY num` nor the `GROUP BY` expression works, arg-min/arg-max is a
|
||
> second statement per node, joined client-side. One-query-per-node is relaxed to two for the ActualTime family only.
|
||
> - **Q13 (B4):** bind `DateTime.MinValue`, `DateTime.MaxValue`, `0001-01-01`, `1601-01-01`,
|
||
> `9999-12-31T23:59:59.9999999`, and a pre-2000 timestamp with sub-µs ticks, in `WHERE ts >= $1 AND ts < $2`. Record
|
||
> what arrives and what reads back. This confirms R6's `BindTimestamp` rule.
|
||
> - **Healthcheck (S14):** check whether `curl` exists in the image. If not, use `wget -qO-`, or drop the healthcheck.
|
||
> - **Q2 also fixes the page size** that T6 uses for DeleteRaw.
|
||
|
||
|
||
**Goal:** prove every QuestDB behaviour the design assumes, against a pinned image, before any code is built on it.
|
||
The findings are written into spec §2–§4.
|
||
|
||
**Files:**
|
||
- Create: `docker-dev/docker-compose.questdb.yml`
|
||
- Create: `docs/plans/2026-10-10-questdb-spike.md`
|
||
- Modify: `docs/plans/2026-10-10-questdb-historian-design.md` (§2–§4: replace assumptions with confirmed forms, and mark
|
||
each one "T0-confirmed")
|
||
|
||
**Acceptance Criteria:**
|
||
- [ ] The overlay pins an exact `questdb/questdb:<x.y.z>` tag. The tag's release date is at least two weeks before
|
||
2026-10-10, and is recorded in the spike doc.
|
||
- [ ] The overlay exposes `18812:8812` and `19000:9000`, uses a named volume `otopcua-questdb-data`, and sets the
|
||
container name `otopcua-dev-questdb-1`. It is brought up with
|
||
`docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d questdb`.
|
||
- [ ] The spike doc records a working Npgsql connection string and answers Q1–Q12 below, each with the exact SQL or
|
||
C# that worked and its observed output.
|
||
- [ ] Spec §2.1 has the confirmed DDL, §2.3 the confirmed visibility-wait query, §3 the confirmed bucket-statistics and
|
||
neighbour query forms, and §4 the confirmed connection-string keywords.
|
||
|
||
**Verify:** `docker exec otopcua-dev-questdb-1 true` exits 0. The spike harness output is pasted into the spike doc.
|
||
|
||
**Steps:**
|
||
|
||
- [ ] **Step 1: Create the overlay.**
|
||
|
||
```yaml
|
||
# docker-dev/docker-compose.questdb.yml — optional QuestDB fixture for the QuestDB historian backend.
|
||
# Up: docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d questdb
|
||
services:
|
||
questdb:
|
||
image: questdb/questdb:<PINNED> # T0 pins an exact tag ≥ 2 weeks old
|
||
container_name: otopcua-dev-questdb-1
|
||
ports:
|
||
- "18812:8812" # PG wire
|
||
- "19000:9000" # web console / REST
|
||
volumes:
|
||
- otopcua-questdb-data:/var/lib/questdb
|
||
healthcheck:
|
||
test: ["CMD-SHELL", "curl -fs http://localhost:9003/status || exit 1"]
|
||
interval: 10s
|
||
timeout: 5s
|
||
retries: 12
|
||
volumes:
|
||
otopcua-questdb-data:
|
||
```
|
||
|
||
- [ ] **Step 2: Write a throwaway C# harness** in the scratchpad, not in the repo. It is a console project that
|
||
references `Npgsql`, restored from the existing central pin via a temporary `Directory.Packages.props` copy, or
|
||
run with `dotnet run` inside a scratch folder under the repo with `ManagePackageVersionsCentrally`. Answer each
|
||
question with real output:
|
||
- **Q1.** The connection-string keywords that work: test `ServerCompatibilityMode=NoTypeLoading` with and without
|
||
it. Also the parameter types needed for `TIMESTAMP` (UTC `DateTime`), `BYTE`, `SYMBOL`, `VARCHAR`, `BOOLEAN` and
|
||
`LONG`.
|
||
- **Q2.** Whether a parameterised multi-row `INSERT INTO t VALUES (…),(…)` works, and the largest row count per
|
||
statement that works reliably. Pick the batch size: ≤ 1000 rows, capped by any parameter limit found.
|
||
- **Q3.** `CREATE TABLE … WAL DEDUP UPSERT KEYS(ts, tag)`: re-inserting the same key replaces the row. Show
|
||
before and after.
|
||
- **Q4.** The `wal_tables()` columns. After an insert, how to know the transaction we produced and wait until it is
|
||
applied (`writerTxn >= sequencerTxn`). Measure the typical lag over 20 runs.
|
||
- **Q5.** TTL: the syntax at create (`TTL n DAYS`), and `ALTER TABLE … SET TTL n DAYS`. Where TTL shows up in
|
||
`tables()` metadata.
|
||
- **Q6.** Timestamp precision through Npgsql: a value with 100 ns ticks (`…1234567`) comes back as `…123456` (truncated)
|
||
or `…123457` (rounded)? Confirm that our pre-truncation makes the round trip exact.
|
||
- **Q7.** Two connections run `CREATE TABLE IF NOT EXISTS` for the same table at once, 20 times. Record any error
|
||
text, and the catch rule that makes it idempotent.
|
||
- **Q8.** Bucket statistics: one `GROUP BY` over the expression `(cast(ts AS LONG)*10 - $1) / $2`, with
|
||
`sum(CASE WHEN … THEN 1 ELSE 0 END)`, `avg`, `min`, `max`, `sum`. Plus the first and last usable sample per bucket,
|
||
with its value and time. Try `first()`/`last()` with a filter, or a window `row_number() OVER (PARTITION BY b
|
||
ORDER BY ts)`. Plus arg-min/arg-max time, via a window function or a second statement.
|
||
- **Q9.** AtTime neighbour lookup: chunked `LIMIT 1` seeks (`ts <= t ORDER BY ts DESC LIMIT 1` and
|
||
`ts > t ORDER BY ts LIMIT 1`, combined with `UNION ALL` per time) versus `ASOF JOIN`. Time both for 100 timestamps
|
||
and pick one.
|
||
- **Q10.** VARCHAR and SYMBOL accept a 400-character string, and `SYMBOL` capacity is fine for tens of thousands of
|
||
tags.
|
||
- **Q11.** An existing-timestamp lookup for a batch: `WHERE tag=$1 AND ts >= $2 AND ts <= $3` (a range), or an `IN`
|
||
list. Pick the one that works with parameters.
|
||
- **Q12.** `DROP TABLE IF EXISTS` works for test cleanup, and `tables()` lists the tables with a given prefix.
|
||
|
||
- [ ] **Step 3: Write the spike doc** with the image tag and date, the connection string, and Q1–Q12 (SQL, output and
|
||
decision for each).
|
||
- [ ] **Step 4: Amend the spec §2–§4** in place with the confirmed forms. If any assumption failed, write the
|
||
replacement design inline and add a ledger ruling. If the failure breaks a user decision, stop and ask.
|
||
- [ ] **Step 5: Commit**
|
||
|
||
```bash
|
||
git add docker-dev/docker-compose.questdb.yml docs/plans/2026-10-10-questdb-spike.md docs/plans/2026-10-10-questdb-historian-design.md
|
||
git commit -m "spike(historian): QuestDB behaviours confirmed against pinned image; spec §2–§4 amended
|
||
|
||
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe"
|
||
```
|
||
|
||
```json:metadata
|
||
{"files":["docker-dev/docker-compose.questdb.yml","docs/plans/2026-10-10-questdb-spike.md","docs/plans/2026-10-10-questdb-historian-design.md"],"verifyCommand":"docker exec otopcua-dev-questdb-1 true","acceptanceCriteria":["pinned image ≥2 weeks old","overlay ports 18812/19000","Q1–Q12 answered with output","spec §2–§4 amended"],"modelTier":"frontier","model":"controller","wave":0}
|
||
```
|
||
|
||
---
|
||
|
||
### Task 1: Extract `Driver.Historian.Sql.Common` (no behaviour change)
|
||
|
||
> **Review amendments (Fable) — binding:**
|
||
> - Runs as dispatch **1a** then **1b** (R10).
|
||
> - **Files gain** `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/**`, for mechanical
|
||
> rename and using fixes only (R11 / B1). This includes replacing `SqlServerSchemaBootstrapper.FailureCacheDuration`
|
||
> with the `BootstrapFailureCache` constant.
|
||
> - `ProcessedReadTests.Every_enum_member_is_either_served_or_refused` moves to `Sql.Common.Tests` (B2 list).
|
||
> - **Step 1 API additions (S6):**
|
||
> - `AlarmEventIds.ToNumericSeverity(AlarmSeverity)` (moved from the private SqlServer method);
|
||
> - `public sealed class HistorianHealthCounters` (lock-guarded): `RecordSuccess()`, `RecordFailure(Exception)`
|
||
> (`OperationCanceledException` is not a failure), and `Snapshot()` returning `HistorianHealthSnapshot` with the
|
||
> `ProcessConnectionOpen = successes > 0 && consecutive == 0` rule. `SqlServerHistorianDataSource` uses it too.
|
||
> - `public static class EventContinuation { string Encode(DateTime, string eventId) }`, in the existing
|
||
> `"{ticks}|{EventId}"` format.
|
||
> - `HistoricalEventRows.ClampSeverity(int) → ushort`.
|
||
> - `StatusBits.SeverityUncertainBit = 0x40000000u`.
|
||
> - Copy `Platforms AnyCPU;x64` into the new csproj (N4).
|
||
|
||
|
||
**Goal:** move every database-independent piece of `Driver.Historian.SqlServer` into a new shared library. The SQL
|
||
Server backend then calls it, with byte-identical behaviour.
|
||
|
||
**Files:**
|
||
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.csproj`
|
||
- Create in that project: `HistorianStatusCodes.cs`, `StatusBits.cs`, `HistorianInfoBits.cs`, `HistorianDataTypes.cs`,
|
||
`SampleColumns.cs`, `HistorianSampleRow.cs`, `StoredSample.cs`, `BucketRow.cs`, `AggregateFamily.cs`,
|
||
`AggregateEvaluator.cs`, `AtTimeEvaluator.cs`, `BucketPlan.cs`, `LiveWritePreparer.cs`, `HistoryUpdatePlanner.cs`,
|
||
`HistorianTagNames.cs`, `BootstrapFailureCache.cs`, `HistorianSchemaMissingException.cs`, `AlarmEventIds.cs`
|
||
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/` (csproj + tests)
|
||
- Modify: every file in `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/` that contained the moved code
|
||
(`SqlServerHistorianDataSource.cs`, `SqlAggregateQueries.cs`, `SqlHistorianDataTypes.cs` (delete),
|
||
`SqlHistorianStatusCodes.cs` (keep only SQL Server-only codes, or delete if none), `SampleRowMapper.cs`,
|
||
`SqlServerHistorianValueWriter.cs`, `SqlServerHistorianDataUpdater.cs`, `SqlServerTagCatalog.cs`,
|
||
`SqlServerSchemaBootstrapper.cs`, the csproj)
|
||
- Modify: the SqlServer unit tests that tested moved code. Move them into `Sql.Common.Tests` under the same test name.
|
||
- Modify: `ZB.MOM.WW.OtOpcUa.slnx` (add both new projects alphabetically, next to lines 30 and 104)
|
||
- Modify: `tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests.csproj` (add a
|
||
ProjectReference to Sql.Common)
|
||
|
||
**Acceptance Criteria:**
|
||
- [ ] `Sql.Common` has no reference to `Microsoft.Data.SqlClient` or any other database client. Its references are
|
||
Core.Abstractions, Server/Runtime (only if needed for `HistorizationValue`) and
|
||
`Microsoft.Extensions.Logging.Abstractions`.
|
||
- [ ] The public API matches the signatures in Step 1 exactly. T2–T7 depend on them.
|
||
- [ ] The SqlServer integration suite passes with the same test count as before the change, which is recorded in the
|
||
report. The SqlServer unit suite plus the moved tests in `Sql.Common.Tests` has the same total count.
|
||
- [ ] New `Sql.Common.Tests` unit tests cover `AtTimeEvaluator`, `AggregateEvaluator`, `BucketPlan`, `LiveWritePreparer`
|
||
and `HistoryUpdatePlanner` directly, using the literals from the SqlServer integration tests. Each one is
|
||
falsified once.
|
||
- [ ] StatusCodeParity passes, and its constant count does not drop.
|
||
|
||
**Verify:**
|
||
- `dotnet build ZB.MOM.WW.OtOpcUa.slnx` gives 0 errors.
|
||
- `dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests` passes.
|
||
- `dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests` passes.
|
||
- With `HISTORIAN_SQL_TEST_CONNECTION` set, `dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests`
|
||
passes with the same count as before.
|
||
- `dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests --filter "FullyQualifiedName~StatusCodeParity"`
|
||
passes.
|
||
|
||
**Steps:**
|
||
|
||
- [ ] **Step 0: Record the baseline.** Run both SqlServer test projects and the parity filter, and write the exact
|
||
pass counts into the report.
|
||
|
||
- [ ] **Step 1: Create the shared API.** Namespace: `ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common`. Each moved body is
|
||
transplanted verbatim from the source named in the comment; only names and visibility change.
|
||
|
||
```csharp
|
||
// HistorianStatusCodes.cs — from SqlHistorianStatusCodes + SqlHistorianUpdateStatusCodes (all consts, same values)
|
||
public static class HistorianStatusCodes
|
||
{
|
||
public const uint Good = 0x00000000u, GoodEntryInserted = 0x00A20000u, GoodEntryReplaced = 0x00A30000u;
|
||
public const uint BadNoData = 0x809B0000u, BadEntryExists = 0x809F0000u, BadNoEntryExists = 0x80A00000u;
|
||
public const uint BadInvalidTimestamp = 0x80230000u, BadAggregateInvalidInputs = 0x80D60000u;
|
||
public const uint UncertainDataSubNormal = 0x40A40000u, BadOutOfRange = 0x803C0000u, BadInvalidArgument = 0x80AB0000u;
|
||
}
|
||
// StatusBits.cs / HistorianInfoBits.cs — moved verbatim from SqlHistorianStatusCodes.cs (made public)
|
||
// HistorianDataTypes.cs — from SqlHistorianDataTypes: DefaultForEmpty, byte FromKind(HistorianValueKind)
|
||
// SampleColumns.cs — record struct SampleColumns(byte ValueKind, double? NumValue, long? IntValue, string? StrValue)
|
||
// + static class SampleColumnMapper { ToColumns(HistorianValue); FromColumns(byte,double?,long?,string?);
|
||
// int ToDbStatus(uint); uint FromDbStatus(int); long ToLongStatus(uint) => status; uint FromLongStatus(long) }
|
||
// HistorianSampleRow.cs — record struct HistorianSampleRow(DateTime TimeUtc, HistorianValue Value, uint StatusCode) (was SqlSampleRow)
|
||
public readonly record struct StoredSample(DateTime TimeUtc, HistorianValue Value, uint Status)
|
||
{
|
||
public bool IsUsable { get; } // verbatim rule from SqlServerHistorianDataSource
|
||
public bool IsGood { get; }
|
||
/// <summary>Applies the quality clamp (NULL value + non-Bad ⇒ BadNoData) — from ReadStored line 533.</summary>
|
||
public static StoredSample FromStored(DateTime timeUtc, HistorianValue value, uint status);
|
||
}
|
||
public sealed class BucketRow // fields verbatim from the private BucketRow, as public get/set properties
|
||
{
|
||
public long Bucket { get; set; } public int Total { get; set; } public int Usable { get; set; }
|
||
public int Good { get; set; } public int Bad { get; set; } public int NonNumeric { get; set; }
|
||
public double? Avg { get; set; } public double? Min { get; set; } public double? Max { get; set; } public double? Sum { get; set; }
|
||
public HistorianValue First { get; set; } public HistorianValue Last { get; set; }
|
||
public double? Extreme { get; set; } public DateTime? ExtremeTime { get; set; }
|
||
}
|
||
public enum AggregateFamily { Simple, StartEnd, ActualTime }
|
||
public static class AggregateFamilies { public static AggregateFamily? FamilyOf(HistoryAggregateType a); } // from SqlAggregateQueries.FamilyOf
|
||
public static class AggregateEvaluator
|
||
{
|
||
public static DataValueSnapshot Evaluate(HistoryAggregateType a, BucketRow r, DateTime bucketStart, DateTime now); // verbatim
|
||
public static DataValueSnapshot NoData(DateTime t, DateTime now); // verbatim
|
||
}
|
||
public static class AtTimeEvaluator
|
||
{
|
||
public static DataValueSnapshot Evaluate(DateTime t, StoredSample? prev, StoredSample? next, DateTime serverTime, DateTime now); // verbatim AtTimeValue
|
||
}
|
||
public sealed class BucketPlan // from ReadProcessedAsync lines 355-370 and 395-408
|
||
{
|
||
public const int MaxBuckets = 100_000;
|
||
public DateTime StartUtc { get; } public DateTime EndUtc { get; } public bool Reversed { get; }
|
||
public long IntervalTicks { get; } public long BucketCount { get; }
|
||
public long StartUnixTicks => (StartUtc - DateTime.UnixEpoch).Ticks; // for QuestDB's (cast(ts AS LONG)*10 - start)/interval
|
||
/// <exception cref="ArgumentOutOfRangeException">BucketCount > MaxBuckets (param name "interval") — same as today.</exception>
|
||
public static BucketPlan Create(DateTime startUtc, DateTime endUtc, TimeSpan interval);
|
||
public DateTime BucketStart(long index);
|
||
/// <summary>One snapshot per bucket in window order; missing buckets get <paramref name="empty"/>; reversed if the window was.</summary>
|
||
public IReadOnlyList<DataValueSnapshot> Assemble(IReadOnlyDictionary<long, DataValueSnapshot> byBucket, Func<DateTime, DataValueSnapshot> empty);
|
||
}
|
||
public static class LiveWritePreparer // from SqlServerHistorianValueWriter.Prepare, plus a resolution parameter
|
||
{
|
||
/// <param name="resolutionTicks">1 = keep 100 ns (SQL Server); 10 = truncate to µs BEFORE de-duplication (QuestDB).</param>
|
||
public static List<HistorianSampleRow> Prepare(IReadOnlyList<HistorizationValue> values, Func<DateTime> utcNow, long resolutionTicks = 1);
|
||
}
|
||
public enum UpsertOutcome { Inserted, Replaced }
|
||
public sealed class UpdatePlan
|
||
{
|
||
public uint?[] Results { get; } // pre-decided codes (invalid ts, NaN, superseded duplicate…); null = decided by the backend
|
||
public IReadOnlyList<(int Index, HistorianSampleRow Row)> Winners { get; } // rows to send, de-duplicated per mode
|
||
}
|
||
public static class HistoryUpdatePlanner // pre/post passes from SqlServerHistorianDataUpdater.UpdateDataAsync lines 56-108
|
||
{
|
||
public static UpdatePlan Plan(string tag, HistoryUpdateMode mode, IReadOnlyList<HistorizationValue> values, long resolutionTicks = 1);
|
||
/// <param name="outcomeByTick">ticks of each winner that the backend actually wrote → Inserted/Replaced; absent ⇒ not written.</param>
|
||
public static IReadOnlyList<uint> Complete(UpdatePlan plan, HistoryUpdateMode mode, IReadOnlyDictionary<long, UpsertOutcome> outcomeByTick);
|
||
public static IReadOnlyList<uint> DeleteAtTimeResults(IReadOnlyList<DateTime> times, ISet<long> deletedTicks, bool tagKnown);
|
||
public static uint DeleteRawResult(bool tagKnown, long rowsAffected);
|
||
}
|
||
public static class HistorianTagNames { public const int MaxLength = 400; public static bool IsValid(string? name); } // from SqlServerTagCatalog.IsValidName
|
||
public sealed class BootstrapFailureCache // the 5 s negative-cache mechanism from SqlServerSchemaBootstrapper
|
||
{
|
||
public BootstrapFailureCache(TimeProvider clock, TimeSpan duration);
|
||
public void ThrowIfCached(); public void Record(Exception ex); public void Clear();
|
||
}
|
||
// HistorianSchemaMissingException.cs — moved from SqlServerHistorianOptions.cs (public sealed, ctor(string message))
|
||
// AlarmEventIds.cs — public static string ComputeEventId(...) moved verbatim from SqlServerAlarmHistorianWriter (same signature)
|
||
```
|
||
|
||
- [ ] **Step 2: Point the SqlServer backend at the shared lib.**
|
||
- Replace each moved body with a call into `Sql.Common`.
|
||
- Delete the now-empty SqlServer classes.
|
||
- Keep `SqlAggregateQueries`' SQL text and `ReadBucket`, now filling the shared `BucketRow`.
|
||
- Keep `SampleRowMapper`'s TVP builders, which take `HistorianSampleRow`.
|
||
- [ ] **Step 3: Move the SqlServer unit tests for the moved code** into `Sql.Common.Tests`, keeping test names and
|
||
literals: `StatusCodeOracleTests`, `ValueWriterPrepareTests`, the column round-trip parts of
|
||
`SampleRowMapperTests`, and the data-type table test. The TVP parts stay in SqlServer.Tests.
|
||
- [ ] **Step 4: Add direct unit tests that did not exist before.** For example:
|
||
|
||
```csharp
|
||
[Fact] public void AtTime_inside_an_outage_returns_the_bad_neighbours_code_with_no_value()
|
||
{
|
||
var t0 = new DateTime(2026,10,9,8,0,0,DateTimeKind.Utc);
|
||
var prev = new StoredSample(t0, HistorianValue.Empty, 0x80310000u);
|
||
var next = new StoredSample(t0.AddSeconds(10), HistorianValue.FromDouble(5), 0);
|
||
var r = AtTimeEvaluator.Evaluate(t0.AddSeconds(5), prev, next, t0, t0.AddHours(1));
|
||
r.StatusCode.ShouldBe(0x80310000u); r.Value.ShouldBeNull();
|
||
}
|
||
[Fact] public void BucketPlan_rejects_more_than_100k_buckets()
|
||
=> Should.Throw<ArgumentOutOfRangeException>(() =>
|
||
BucketPlan.Create(DateTime.UnixEpoch, DateTime.UnixEpoch.AddSeconds(100_001), TimeSpan.FromSeconds(1)));
|
||
[Fact] public void Prepare_truncates_to_microseconds_before_dedup_when_resolution_is_10()
|
||
{
|
||
var t = new DateTime(2026,10,9,8,0,0,DateTimeKind.Utc).AddTicks(1234567);
|
||
var rows = LiveWritePreparer.Prepare([new(t, HistorianValue.FromDouble(1), 0), new(t.AddTicks(3), HistorianValue.FromDouble(2), 0)], () => t, 10);
|
||
rows.Count.ShouldBe(1); rows[0].TimeUtc.Ticks.ShouldBe(t.Ticks - t.Ticks % 10); rows[0].Value.Number.ShouldBe(1);
|
||
}
|
||
```
|
||
|
||
Add the matching `AggregateEvaluator` cases, using the 8-row ProcessedRead literals from
|
||
`ProcessedReadTests.Numeric_aggregates_over_the_fixture`. Build the `BucketRow`s by hand from tag A's samples.
|
||
- [ ] **Step 5: Wire the solution and parity test.** Add both projects to the slnx, and the ProjectReference to the
|
||
Core.Abstractions.Tests csproj.
|
||
- [ ] **Step 6: Run the Verify commands.** Counts must equal the Step 0 baseline, plus the new direct tests. Falsify
|
||
each new direct test once by reverting the rule it pins.
|
||
- [ ] **Step 7: Commit** with `refactor(historian): extract Driver.Historian.Sql.Common from the SQL Server backend (no
|
||
behaviour change)` and the Claude-Session trailer.
|
||
|
||
```json:metadata
|
||
{"files":["src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/","tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/","src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/","tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests/","ZB.MOM.WW.OtOpcUa.slnx","tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests.csproj"],"verifyCommand":"dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests (env set) — same count as baseline","acceptanceCriteria":["no DB client ref in Sql.Common","public API exactly as Step 1","SqlServer suites same counts","direct evaluator tests falsified","parity green"],"modelTier":"frontier","model":"opus","wave":1}
|
||
```
|
||
|
||
---
|
||
|
||
### Task 2: Conformance suite (shared source), run against SQL Server
|
||
|
||
> **Review amendments (Fable) — binding:**
|
||
> - **AC bullet 1 becomes:** "Every moved test keeps its method name and its expected VALUES/STATUS literals. Tick
|
||
> offsets are rewritten per R4. The report lists each test whose offsets were rewritten."
|
||
> - **AC bullet 5 becomes:** "A `virtual` override is allowed only for a difference R4 cannot express. The report names
|
||
> it, and the controller rules on it before T7."
|
||
> - **Step 2 explicit list (B2). Move:**
|
||
> - `RawReadTests` — all except `A_missing_schema_surfaces_in_health_LastError_and_is_not_latched`. The SQL tag-count
|
||
> line in `Unknown_tag_is_an_empty_result_and_reads_never_create_tags` becomes `B.TagExistsAsync`.
|
||
> - `AtTimeReadTests` — all.
|
||
> - `ProcessedReadTests` — all except `Every_enum_member_is_either_served_or_refused`, which went to Sql.Common.Tests
|
||
> in T1.
|
||
> - `QualityContractTests` — all. Direct status-column reads become `B.ReadBackAsync` status checks.
|
||
> - `WriteTests` — `Every_kind_round_trips…`, `Writing_the_same_batch_twice_is_idempotent`,
|
||
> `Within_batch_duplicate_timestamps…`, `Non_finite_doubles…`, `Empty_batch_succeeds…` and `Tag_is_created…`.
|
||
> - `UpdateTests` — all.
|
||
> - `LongNameTests` — all except `An_event_read_filtered_by_a_long_source_matches_nothing`.
|
||
> - **Stay in SqlServer:**
|
||
> - `A_missing_schema…`
|
||
> - `Ten_thousand_values_write_in_under_five_seconds`
|
||
> - `Concurrent_overlapping_writers…`
|
||
> - `A_SQL_failure_returns_false…`
|
||
> - `EventReadTests`
|
||
> - `ProvisioningTests`
|
||
> - `AlarmWriterTests`
|
||
> - `SchemaBootstrapIntegrationTests`
|
||
> - **Interface (S13):** add `Task<bool> TagExistsAsync(string tag);`. `ReadBackAsync` returns the stored status
|
||
> verbatim, with no read-side clamp; say so in its doc comment.
|
||
> - **Skip check (S7):** first run with the env var UNSET and confirm the tests are reported Skipped, not Failed.
|
||
> Fallback: `ConformanceBase` exposes a lazy, memoised `protected async Task<IConformanceBackend> BackendAsync()`,
|
||
> disposed in `DisposeAsync`, and each test starts with `var B = await BackendAsync();`.
|
||
> - **Verify (N8):** the report lists every moved method name, plus per-file `[Fact]`/`[Theory]` counts before and
|
||
> after.
|
||
|
||
|
||
**Goal:** one backend-neutral suite of hand-computed behaviour tests, compiled into the SqlServer integration project
|
||
and passing there. T7 later compiles the same suite into the QuestDB project.
|
||
|
||
**Files:**
|
||
- Create: `tests/Drivers/Historian.Conformance/IConformanceBackend.cs`, `ConformanceData.cs`,
|
||
`RawReadConformance.cs`, `AtTimeConformance.cs`, `ProcessedConformance.cs`, `QualityContractConformance.cs`,
|
||
`LiveWriteConformance.cs`, `UpdateConformance.cs`, `TagNameConformance.cs`
|
||
- Modify: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/*.csproj` (add
|
||
`<Compile Include="..\Historian.Conformance\**\*.cs" LinkBase="Conformance" />` and a ProjectReference to Sql.Common)
|
||
- Create in the SqlServer IT project: `SqlServerConformanceBackend.cs`, `SqlServerConformance.cs` (one concrete
|
||
subclass per conformance class)
|
||
- Modify in the SqlServer IT project: remove from the original files the generic tests that moved into the suite
|
||
(`RawReadTests`, `AtTimeReadTests`, `ProcessedReadTests`, `QualityContractTests`, and the generic parts of
|
||
`WriteTests`, `UpdateTests` and `LongNameTests`). SQL Server-specific tests stay where they are.
|
||
|
||
**Acceptance Criteria:**
|
||
- [ ] Every moved test keeps its method name and literal expectations. `git log -p` shows a move, not a rewrite.
|
||
- [ ] The SqlServer IT total count after the change is at least the pre-change count, and the report has the before
|
||
and after count.
|
||
- [ ] Suite code references only `IConformanceBackend` and Core.Abstractions types. There is no `SqlServer*` type and
|
||
no T-SQL in `tests/Drivers/Historian.Conformance`.
|
||
- [ ] Timestamp assertions go through `B.Round(t)`.
|
||
- [ ] Any test that cannot hold for every backend is a `virtual` member with a doc comment naming the difference.
|
||
None should be needed for SQL Server.
|
||
|
||
**Verify:** with `HISTORIAN_SQL_TEST_CONNECTION` set, `dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests`
|
||
passes with a count of at least the baseline. Without the variable, every test skips.
|
||
|
||
**Steps:**
|
||
|
||
- [ ] **Step 1: Define the backend contract.**
|
||
|
||
```csharp
|
||
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance;
|
||
public interface IConformanceBackend : IAsyncDisposable
|
||
{
|
||
/// <summary>1 for SQL Server (100 ns), 10 for QuestDB (µs).</summary>
|
||
long ResolutionTicks { get; }
|
||
DateTime Round(DateTime t) => new(t.Ticks - t.Ticks % ResolutionTicks, DateTimeKind.Utc);
|
||
IHistorianDataSource CreateDataSource(TimeProvider? clock = null);
|
||
IHistorianValueWriter CreateValueWriter(Func<DateTime>? utcNow = null);
|
||
IHistorianDataUpdater CreateUpdater();
|
||
/// <summary>Writes rows directly in the backend's schema (never through the writer), status verbatim.</summary>
|
||
Task SeedAsync(string tag, params (DateTime T, HistorianValue V, uint Status)[] samples);
|
||
/// <summary>Live (non-deleted) rows for a tag, ascending, read directly from storage.</summary>
|
||
Task<IReadOnlyList<(DateTime T, HistorianValue V, uint Status)>> ReadBackAsync(string tag);
|
||
}
|
||
public abstract class ConformanceBase : IAsyncLifetime
|
||
{
|
||
protected IConformanceBackend B { get; private set; } = null!;
|
||
protected abstract Task<IConformanceBackend> CreateBackendAsync(); // calls fixture.SkipIfUnavailable()
|
||
public async ValueTask InitializeAsync() => B = await CreateBackendAsync();
|
||
public async ValueTask DisposeAsync() => await B.DisposeAsync();
|
||
}
|
||
```
|
||
|
||
`ConformanceData` holds `T0`, `At(seconds)`, the status consts and `SeedAAsync(B)`, verbatim from `ReadTestDb`.
|
||
- [ ] **Step 2: Move the generic tests** listed in the T1 inventory (Raw, AtTime, Processed literals, the quality
|
||
contract, live-write redelivery and first-wins, update result codes, tag names over 400 characters) into the
|
||
suite classes. Write each one as an abstract class `RawReadConformance : ConformanceBase` with `[Fact]` methods.
|
||
- [ ] **Step 3: Implement `SqlServerConformanceBackend`.** It reuses `SqlHistorianFixture`, a unique schema and the
|
||
existing seeding SQL from `ReadTestDb`. Add one concrete class per suite:
|
||
`[Collection(SqlHistorianCollection.Name)] public sealed class SqlServerRawReadConformance(SqlHistorianFixture fx) : RawReadConformance`.
|
||
- [ ] **Step 4: Run the tests and compare counts.** Commit with
|
||
`test(historian): backend conformance suite (shared source), run against SQL Server` and the trailer.
|
||
|
||
```json:metadata
|
||
{"files":["tests/Drivers/Historian.Conformance/","tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/"],"verifyCommand":"dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests (env set) ≥ baseline count","acceptanceCriteria":["tests moved by name with same literals","count ≥ baseline","suite is backend-neutral","Round() for timestamps"],"modelTier":"standard","model":"sonnet","wave":2}
|
||
```
|
||
|
||
---
|
||
|
||
### Task 3: QuestDB skeleton (options, connection, bootstrapper, fixture)
|
||
|
||
> **Review amendments (Fable) — binding:**
|
||
> - **Files gain** `QuestDbTagCatalog.cs` (R7) and `QuestDbWalVisibility.cs` (R5, moved from T6).
|
||
> - `QuestDbConnection` gains `Round(DateTime)` and `BindTimestamp` (R6), plus `BindByte` and `ReadKind`, so the T0 Q1
|
||
> answers live in one file (S2).
|
||
> - The DDL uses `num_value`/`int_value`/`str_value` (R9).
|
||
> - **New tests:**
|
||
> - `BootstrapTests.ExistsAsync_is_false_for_an_unknown_tag_and_true_after_EnsureAsync`;
|
||
> - `Wait_returns_once_writerTxn_reaches_the_captured_sequencerTxn` (or the Q4 fallback form);
|
||
> - `Timeout_zero_skips_the_wait`;
|
||
> - a `QuestDb.Tests` unit test pinning that `BindTimestamp` never emits `DateTime.MinValue`/`MaxValue`.
|
||
> - Copy `Platforms AnyCPU;x64` (N4).
|
||
|
||
|
||
**Goal:** a building `Driver.Historian.QuestDb` project. It has validated options, a pooled Npgsql connection, an
|
||
idempotent and concurrency-safe schema bootstrapper with TTL, and an env-gated integration fixture.
|
||
|
||
**Files:**
|
||
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.csproj`
|
||
(references Core.Abstractions, Core.AlarmHistorian, Server/Runtime and Sql.Common; packages `Npgsql` and
|
||
`Microsoft.Extensions.Logging.Abstractions`; InternalsVisibleTo the two QuestDb test assemblies)
|
||
- Create: `QuestDbHistorianOptions.cs`, `QuestDbConnection.cs`, `QuestDbTables.cs`, `QuestDbSchemaBootstrapper.cs`
|
||
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/` (csproj, `QuestDbHistorianOptionsTests.cs`,
|
||
`QuestDbTablesTests.cs`)
|
||
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/` (csproj with the
|
||
conformance-include line and a ProjectReference to Sql.Common, `QuestDbFixture.cs`, `BootstrapTests.cs`)
|
||
- Modify: `ZB.MOM.WW.OtOpcUa.slnx`, and the Core.Abstractions.Tests csproj (ProjectReference to the QuestDb project)
|
||
|
||
**Acceptance Criteria:**
|
||
- [ ] `QuestDbHistorianOptions` has `ConnectionString` (default `""`), `TablePrefix` (`"hist_"`, regex
|
||
`^[A-Za-z0-9_]+$`), `RetentionDays` (`int?`, ≥ 1), `CommandTimeout` (30 s, > 0), `AutoCreateSchema` (`true`) and
|
||
`VisibilityTimeout` (5 s, ≥ 0). `Validate()` returns error strings that name the key
|
||
`ServerHistorian:QuestDb:<Key>`.
|
||
- [ ] `QuestDbTables` produces the three `CREATE TABLE IF NOT EXISTS` statements, as confirmed by T0, for a given prefix
|
||
and retention. Unit tests pin the exact text.
|
||
- [ ] The bootstrapper is idempotent and survives two concurrent instances, using the catch rule from T0 Q7. It
|
||
re-applies a changed TTL. It uses `BootstrapFailureCache` for the 5 s negative cache.
|
||
- [ ] `QuestDbFixture` reads `HISTORIAN_QUESTDB_TEST_CONNECTION` and skips when it is absent. `UniquePrefix()` returns
|
||
`"t" + 12 hex + "_"`. `DropPrefixAsync(prefix)` drops every table with that prefix, found through `tables()`.
|
||
- [ ] `BootstrapTests` cover:
|
||
- tables created;
|
||
- a second run is a no-op;
|
||
- two concurrent bootstrappers both succeed;
|
||
- TTL is visible in metadata when `RetentionDays=7` and absent when unset;
|
||
- `AutoCreateSchema=false` against a missing table throws `HistorianSchemaMissingException` from Sql.Common.
|
||
|
||
**Verify:**
|
||
- `dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests` passes.
|
||
- With `HISTORIAN_QUESTDB_TEST_CONNECTION` set, `dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests --filter "FullyQualifiedName~Bootstrap"`
|
||
passes.
|
||
- Without the variable, every test skips.
|
||
|
||
**Steps:**
|
||
- [ ] Step 1: write the options with `Validate()` and its unit tests (red, then green).
|
||
- [ ] Step 2: write `QuestDbTables` DDL builders (use the spec §2.1 *as amended by T0*), with unit tests pinning the
|
||
text for prefix `hist_` with and without TTL.
|
||
- [ ] Step 3: write `QuestDbConnection`.
|
||
- Members:
|
||
- `NpgsqlDataSource` built once;
|
||
- `OpenAsync(ct)`;
|
||
- `CreateCommand(conn, sql)` with `CommandTimeout`;
|
||
- `static bool IsTransient(Exception)`, true for `NpgsqlException` with `IsTransient`, socket and IO errors, and
|
||
timeouts;
|
||
- `static DateTime ReadUtc(NpgsqlDataReader r, int i)`;
|
||
- `string T(string table)`, which returns prefix + table.
|
||
- Use the parameter typing from T0 Q1.
|
||
- [ ] Step 4: write the bootstrapper and fixture. Bring up the overlay
|
||
(`docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.questdb.yml up -d questdb`), then
|
||
write and run `BootstrapTests`.
|
||
- [ ] Step 5: wire the slnx and the parity-test reference, then build the solution.
|
||
- [ ] Step 6: commit `feat(historian): QuestDB backend skeleton — options, connection, schema bootstrap with TTL` with
|
||
the trailer.
|
||
|
||
```json:metadata
|
||
{"files":["src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/","tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/","tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/","ZB.MOM.WW.OtOpcUa.slnx"],"verifyCommand":"dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests --filter FullyQualifiedName~Bootstrap (env set)","acceptanceCriteria":["options validate naming keys","DDL pinned by unit tests","bootstrap idempotent + concurrent + TTL","fixture env-gated with unique prefix"],"modelTier":"standard","model":"sonnet","wave":2}
|
||
```
|
||
|
||
---
|
||
|
||
### Task 4: QuestDB writes (value writer, tag provisioner, alarm writer)
|
||
|
||
> **Review amendments (Fable) — binding:**
|
||
> - The writer calls `QuestDbTagCatalog.EnsureAsync` before inserting (R7). After inserting, it calls
|
||
> `QuestDbWalVisibility.WaitAsync`; a timeout does not change the return value (R5).
|
||
> - **Provisioner (S5):** read the existing `hist_tag` rows per chunk (`WHERE name IN (…)`), merge
|
||
> `COALESCE(request, stored)` for `eng_unit` and `description` client-side, then insert. Pin this with the SqlServer
|
||
> `Upsert_creates_then_changes_DataType_on_rerun…` literal.
|
||
> - The alarm writer uses `AlarmEventIds.ToNumericSeverity`, plus the Q3 fallback if VARCHAR dedup was refused.
|
||
> - Every DateTime is bound via `BindTimestamp` (R6).
|
||
|
||
|
||
**Goal:** the recorder, provisioning and alarm-history seams write to QuestDB with the same semantics as SQL Server.
|
||
|
||
**Files:**
|
||
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianValueWriter.cs`,
|
||
`QuestDbTagProvisioner.cs`, `QuestDbAlarmHistorianWriter.cs`
|
||
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/WriteTests.cs`,
|
||
`ProvisioningTests.cs`, `AlarmWriterTests.cs`
|
||
|
||
**Acceptance Criteria:**
|
||
- [ ] The value writer works as follows:
|
||
- It calls `LiveWritePreparer.Prepare(values, utcNow, resolutionTicks: 10)`.
|
||
- It rejects an invalid tag name (`HistorianTagNames.IsValid`) and returns `false`.
|
||
- It reads the existing timestamps for the tag in the batch range, including tombstones, using the T0 Q11 form.
|
||
- It inserts only the rows that are not yet present, in multi-row parameterised `INSERT`s of at most the T0 Q2
|
||
batch size, with `deleted=false` and `server_ts=utcNow()`.
|
||
- It returns `true` with no SQL when every row already existed.
|
||
- Any exception returns `false`. It never throws, and cancellation also returns `false`.
|
||
- [ ] The provisioner upserts `hist_tag` (`ts = 1970-01-01`, data type from `HistorianDataTypes.FromKind`, eng unit,
|
||
description). Both the request field types and the result tally match the SqlServer provisioner's.
|
||
- [ ] The alarm writer works as follows:
|
||
- It uses the same deterministic `event_id` as the SQL Server writer, through `AlarmEventIds.ComputeEventId` from
|
||
Sql.Common.
|
||
- It inserts events through the dedup key `(ts, event_id)`, so a redelivery is idempotent.
|
||
- `Classify` maps an unknown or non-database exception to RetryPlease, and maps permanent errors as the SqlServer
|
||
writer does.
|
||
- [ ] `WriteTests` cover:
|
||
- typed round trip for all six kinds with a non-Good status;
|
||
- a redelivered batch leaves one row and the first value;
|
||
- a tombstoned key is not resurrected by redelivery (seed a tombstone directly);
|
||
- NaN ⇒ `BadOutOfRange` Empty;
|
||
- a microsecond collision within a batch keeps the first;
|
||
- a stopped port (`Port=1`) returns `false` without throwing.
|
||
- [ ] Each regression test is falsified once.
|
||
|
||
**Verify:** with `HISTORIAN_QUESTDB_TEST_CONNECTION` set,
|
||
`dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests --filter "FullyQualifiedName~Write|FullyQualifiedName~Provisioning|FullyQualifiedName~AlarmWriter"`
|
||
passes.
|
||
|
||
**Steps:**
|
||
- [ ] Step 1: write `WriteTests` (red).
|
||
- [ ] Step 2: implement the value writer.
|
||
- [ ] Step 3: make `WriteTests` green, then falsify the first-wins, tombstone and NaN tests.
|
||
- [ ] Step 4: do the provisioner, then the alarm writer, each tests-first.
|
||
- [ ] Step 5: commit `feat(historian): QuestDB live writes, provisioning and alarm events` with the trailer.
|
||
|
||
```json:metadata
|
||
{"files":["src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianValueWriter.cs","src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbTagProvisioner.cs","src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbAlarmHistorianWriter.cs","tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/WriteTests.cs","tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/ProvisioningTests.cs","tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/AlarmWriterTests.cs"],"verifyCommand":"dotnet test …QuestDb.IntegrationTests --filter Write|Provisioning|AlarmWriter (env set)","acceptanceCriteria":["insert-if-absent incl tombstones","never throws","event id deterministic + idempotent","falsified"],"modelTier":"standard","model":"sonnet","wave":3}
|
||
```
|
||
|
||
---
|
||
|
||
### Task 5: QuestDB reads and aggregates
|
||
|
||
> **Review amendments (Fable) — binding:**
|
||
> - **Raw per R8:** `LIMIT @top`, at most `top` samples, `ContinuationPoint: null`. Delete the "LIMIT n+1" text.
|
||
> - Every inbound timestamp is rounded via `QuestDbConnection.Round`, and every DateTime is bound via `BindTimestamp`
|
||
> (R6). The Processed window is rounded before `BucketPlan.Create`.
|
||
> - Tags are resolved through `QuestDbTagCatalog.ExistsAsync` only (R7).
|
||
> - **SQL classification (S6):** usable ⇔ `status < {StatusBits.SeverityBadBit}` AND `kind <> 0`; good ⇔ usable AND
|
||
> `status < {StatusBits.SeverityUncertainBit}`. The literals are interpolated from the consts, never typed by hand. Note
|
||
> that QuestDB's "not Bad" is an unsigned LONG comparison, not T-SQL's `>= 0`.
|
||
> - Health uses the shared `HistorianHealthCounters`. Events use `EventContinuation` and
|
||
> `HistoricalEventRows.ClampSeverity`.
|
||
> - **T5 owns `IT/QuestDbConformanceBackend.cs`** (seed and read-back only; N6). Step 3's throwaway backend IS this file,
|
||
> which is kept. Only the throwaway subclasses are deleted.
|
||
> - **Extra test (N2):** `ReadTests` includes one seeded Int64-kind row with `num_value = NULL`, proving the kind, not
|
||
> the column, decides the value.
|
||
|
||
|
||
**Goal:** `QuestDbHistorianDataSource` serves Raw, AtTime, Events and Processed. Results are identical to SQL Server's,
|
||
except for microsecond rounding.
|
||
|
||
**Files:**
|
||
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataSource.cs`,
|
||
`QuestDbAggregateQueries.cs`
|
||
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/ReadTests.cs`
|
||
|
||
**Acceptance Criteria:**
|
||
- [ ] Every query filters `NOT deleted`. Every row goes through `StoredSample.FromStored`.
|
||
- [ ] **Raw:** forward and reverse, `LIMIT n+1`, and the same paging contract and continuation semantics as the
|
||
SqlServer data source (re-read from the last timestamp, inclusive). Unbounded when `maxValuesPerNode == 0`.
|
||
Unknown tag ⇒ an empty result. The tag is resolved through the `hist_tag` existence or the sample table, so the
|
||
read never creates a tag.
|
||
- [ ] **AtTime:** the result list matches the input in length and order. Neighbours come from the T0 Q9 form, and
|
||
`AtTimeEvaluator.Evaluate` decides each value.
|
||
- [ ] **Processed:**
|
||
- It uses `BucketPlan.Create` and `AggregateFamilies.FamilyOf`.
|
||
- It runs one statement per node, grouped by `(cast(ts AS LONG)*10 - @startUnixTicks) / @intervalTicks`.
|
||
- It fills the shared `BucketRow` with the same usable/good/bad/non-numeric definitions as SQL Server: usable = status
|
||
severity not Bad and kind ≠ Empty; good = usable and status < 0x40000000; non-numeric = usable and kind in (4, 5).
|
||
- `AggregateEvaluator.Evaluate` produces the values, and `BucketPlan.Assemble` fills empty buckets and handles
|
||
reversal.
|
||
- An unsupported aggregate throws `HistoryAggregateNotSupportedException`.
|
||
- [ ] **Events:** filtered by source and time range, ascending, limited to `maxEvents`. The `ContinuationPoint` is
|
||
non-null when truncated.
|
||
- [ ] Health counters (`GetHealthSnapshot`) are implemented as in the SqlServer data source (lock-guarded success/failure counters plus the last error). A failure fails only that call and is recorded in
|
||
`GetHealthSnapshot`.
|
||
- [ ] `ReadTests` cover only QuestDB-specific behaviour:
|
||
- tombstoned rows are invisible to Raw, AtTime and Processed (seed a tombstone directly);
|
||
- a whole-range Processed read from 1601 to 9999 with interval 0 does not overflow;
|
||
- a reverse paging walk with `maxValuesPerNode=1` returns every row.
|
||
The shared literals run through the conformance suite in T7.
|
||
|
||
**Verify:** with `HISTORIAN_QUESTDB_TEST_CONNECTION` set,
|
||
`dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests --filter "FullyQualifiedName~ReadTests"`
|
||
passes.
|
||
|
||
**Steps:**
|
||
- [ ] Step 1: write `ReadTests` (red).
|
||
- [ ] Step 2: implement Raw, then AtTime, Events and Processed.
|
||
- [ ] Step 3: temporarily compile the conformance read classes into this task's worktree. Add a throwaway concrete
|
||
subclass and a throwaway backend that seeds through direct INSERTs. Run the Raw, AtTime and Processed
|
||
conformance classes locally and get them green.
|
||
- [ ] Step 4: delete the throwaway files before committing. T7 owns the real ones. Paste the conformance run output
|
||
into the report.
|
||
- [ ] Step 5: falsify the tombstone filter and the bucket expression (an off-by-one start).
|
||
- [ ] Step 6: commit `feat(historian): QuestDB reads and aggregates (pushed down, shared evaluation)` with the trailer.
|
||
|
||
```json:metadata
|
||
{"files":["src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataSource.cs","src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbAggregateQueries.cs","tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/ReadTests.cs"],"verifyCommand":"dotnet test …QuestDb.IntegrationTests --filter FullyQualifiedName~ReadTests (env set) + throwaway conformance run in report","acceptanceCriteria":["NOT deleted everywhere","bucket expr exact vs SqlServer","shared evaluators used","conformance read classes green locally"],"modelTier":"frontier","model":"opus","wave":3}
|
||
```
|
||
|
||
---
|
||
|
||
### Task 6: QuestDB HistoryUpdate (soft delete) with read-your-writes
|
||
|
||
> **Review amendments (Fable) — binding:**
|
||
> - `QuestDbWalVisibility.cs` moves to T3, so T6 consumes it (R5).
|
||
> - Insert and Update call `QuestDbTagCatalog.EnsureAsync`; Replace and the deletes call `ExistsAsync` (R7).
|
||
> - **Existence read (S15):** one `WHERE tag=$1 AND ts >= $min AND ts <= $max AND NOT deleted` range query, filtered to
|
||
> the winners client-side. Never an `IN` list of `n` parameters.
|
||
> - **DeleteRaw (S12):** read live rows in `LIMIT` pages of the Q2 batch size, keyed on the last `ts`, and tombstone them
|
||
> page by page. `rowsAffected` is the total.
|
||
> - Inbound times are rounded and bound via `BindTimestamp` (R6).
|
||
> - The TombstoneTests "redelivery after delete" case moves to T7 as
|
||
> `IT/TombstoneTests.Redelivery.cs::Tombstone_is_not_resurrected_by_a_writer_redelivery` (S8). T6 keeps the other two.
|
||
|
||
|
||
**Goal:** `QuestDbHistorianDataUpdater` implements all five operations through insert-only upserts and tombstones. It
|
||
returns codes identical to SQL Server's, and waits for WAL visibility before returning.
|
||
|
||
**Files:**
|
||
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataUpdater.cs`,
|
||
`QuestDbWalVisibility.cs`
|
||
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/UpdateTests.cs`,
|
||
`TombstoneTests.cs`
|
||
|
||
**Acceptance Criteria:**
|
||
- [ ] `UpdateDataAsync`:
|
||
1. `HistoryUpdatePlanner.Plan(tag, mode, values, resolutionTicks: 10)` builds the plan.
|
||
2. It reads which winner timestamps currently have a **live** row, meaning present and not deleted.
|
||
3. It writes:
|
||
- Insert: only the absent winners;
|
||
- Replace: only the present winners;
|
||
- Update: every winner.
|
||
Each write is an `INSERT` with `deleted=false`.
|
||
4. It builds the outcome map (present ⇒ Replaced, absent ⇒ Inserted, for rows it wrote).
|
||
5. `HistoryUpdatePlanner.Complete(...)` produces the codes.
|
||
- [ ] `DeleteAtTimeAsync` writes a tombstone for each time with a live row, with `deleted=true`, value columns NULL and
|
||
`status=BadNoData`. Codes come from `HistoryUpdatePlanner.DeleteAtTimeResults`.
|
||
- [ ] `DeleteRawAsync` writes a tombstone for every live row in `[start, end)`. The code comes from
|
||
`HistoryUpdatePlanner.DeleteRawResult`.
|
||
- [ ] On an unknown tag, Insert and Update create the `hist_tag` row; Replace and Delete do not, and return the shared
|
||
unknown-tag codes.
|
||
- [ ] `QuestDbWalVisibility.WaitAsync(table, ct)` works as follows:
|
||
- It polls the T0 Q4 query until the table's applied transaction is at least the sequencer transaction captured
|
||
after our write.
|
||
- It gives up after `VisibilityTimeout` and logs a Warning, without changing the codes. `VisibilityTimeout = 0`
|
||
skips the wait.
|
||
- Every updater operation calls it once, after writing.
|
||
- [ ] `UpdateTests` cover the five operations' codes (QuestDB-specific checks only; the literal codes run through
|
||
conformance in T7) and that a read made right after `UpdateDataAsync` returns sees the change.
|
||
- [ ] `TombstoneTests` cover:
|
||
- delete then Raw read ⇒ the row is gone;
|
||
- delete, then a `QuestDbHistorianValueWriter` redelivery of the same sample ⇒ the row stays gone (this needs T4's
|
||
writer: write the test against the interface, and if T4 is not merged yet, seed a tombstone and assert the
|
||
writer's existence query counts it);
|
||
- insert over a tombstone ⇒ `GoodEntryInserted` and a visible row.
|
||
|
||
**Verify:** with `HISTORIAN_QUESTDB_TEST_CONNECTION` set,
|
||
`dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests --filter "FullyQualifiedName~UpdateTests|FullyQualifiedName~TombstoneTests"`
|
||
passes.
|
||
|
||
**Steps:**
|
||
- [ ] Step 1: write the tests (red).
|
||
- [ ] Step 2: implement `QuestDbWalVisibility`.
|
||
- [ ] Step 3: implement the updater.
|
||
- [ ] Step 4: get to green. Falsify the "live row" filter by dropping `NOT deleted` from the existence check, and the
|
||
visibility wait by setting the timeout to 0 in the read-after-write test.
|
||
- [ ] Step 5: commit `feat(historian): QuestDB HistoryUpdate with soft delete and read-your-writes` with the trailer.
|
||
|
||
```json:metadata
|
||
{"files":["src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataUpdater.cs","src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbWalVisibility.cs","tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/UpdateTests.cs","tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/TombstoneTests.cs"],"verifyCommand":"dotnet test …QuestDb.IntegrationTests --filter UpdateTests|TombstoneTests (env set)","acceptanceCriteria":["insert-only upserts, no UPDATE/DELETE","codes from shared planner","tombstones never resurrected","read-your-writes wait"],"modelTier":"standard","model":"sonnet","wave":3}
|
||
```
|
||
|
||
---
|
||
|
||
### Task 7: Provider, Host wiring and the conformance suite on QuestDB
|
||
|
||
> **Review amendments (Fable) — binding:**
|
||
> - The model is **opus** (R10). T7 may fix T4/T6 files and the data-source Raw/Events plumbing. Aggregate and AtTime
|
||
> divergences are reported to the controller, not fixed.
|
||
> - T5 owns `QuestDbConformanceBackend.cs`; T7 owns the subclasses (N6).
|
||
> - T7 adds `IT/TombstoneTests.Redelivery.cs` (S8).
|
||
> - **Compose default (S14):** the exact Q1-confirmed connection string, with `Host=questdb;Port=8812` substituted,
|
||
> keeping every keyword T0 found necessary.
|
||
|
||
|
||
**Goal:** `Backend=QuestDb` resolves, validates and builds all five seams. The full conformance suite passes against
|
||
QuestDB.
|
||
|
||
**Files:**
|
||
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianProvider.cs`
|
||
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformanceBackend.cs`,
|
||
`QuestDbConformance.cs` (one concrete subclass per conformance class)
|
||
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbHistorianProviderTests.cs`
|
||
- Modify:
|
||
- `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/HistorianBackendRegistry.cs` (add a `QuestDbBackendName`
|
||
const, `BindQuestDb(IConfiguration)` and a `Factories` entry);
|
||
- `ServerHistorianOptionsValidator.cs` (a QuestDb block before the `if (!options.IsGatewayBackend) return;` line,
|
||
mirroring the SqlServer block);
|
||
- `src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj` (a ProjectReference);
|
||
- `tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/Configuration/HistorianBackendRegistryTests.cs`;
|
||
- `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/ServerHistorianOptionsValidatorTests.cs`;
|
||
- `docker-dev/docker-compose.yml`: in the central-1 and central-2 `ServerHistorian__*` blocks, add
|
||
`ServerHistorian__QuestDb__ConnectionString: "${OTOPCUA_HISTORIAN_QUESTDB:-<T0 connection string with Host=questdb;Port=8812>}"`
|
||
plus a comment line naming the overlay.
|
||
|
||
**Acceptance Criteria:**
|
||
- [ ] The provider has `Name="QuestDb"`, `SupportedKinds` = all six kinds as a FrozenSet, and one lazily built shared
|
||
connection and bootstrapper across the seams. The constructor does no I/O and no validation, like the SqlServer
|
||
provider. `CreateUpdater` returns the soft-delete updater.
|
||
- [ ] The registry resolves `QuestDb` case-insensitively.
|
||
- [ ] The validator, when the backend is consumed and is `QuestDb`:
|
||
- reports every `QuestDbHistorianOptions.Validate()` error, naming the key;
|
||
- does not require the Gateway keys;
|
||
- ignores an invalid `TablePrefix` when not consumed.
|
||
- [ ] Registry and validator tests mirror the five SqlServer ones (lines 71 and 319–368).
|
||
- [ ] The QuestDB integration project runs the **entire** conformance suite through `QuestDbConformanceBackend`:
|
||
- `ResolutionTicks = 10`;
|
||
- seeding is direct INSERTs into the prefixed tables;
|
||
- `ReadBackAsync` filters `NOT deleted`.
|
||
Every class passes. Any named override has a doc comment and a ledger ruling.
|
||
|
||
**Verify:**
|
||
- With both environment variables set, the SqlServer and QuestDB integration projects both pass. The report shows the
|
||
conformance class counts side by side.
|
||
- `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests --filter "FullyQualifiedName~Historian"` passes.
|
||
- `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter "FullyQualifiedName~Historian"` passes.
|
||
|
||
**Steps:**
|
||
- [ ] Step 1: write the provider unit tests (red), then the provider.
|
||
- [ ] Step 2: write the registry and validator tests (red), then the wiring.
|
||
- [ ] Step 3: write `QuestDbConformanceBackend` and the subclasses, and run the full suite. Fix any divergence in the
|
||
QuestDB code, even though those files belong to T4–T6; this task is the integration owner in W4.
|
||
- [ ] Step 4: make the compose env edit.
|
||
- [ ] Step 5: build the solution, then commit `feat(historian): QuestDb provider + Host wiring; conformance suite green
|
||
on both backends` with the trailer.
|
||
|
||
```json:metadata
|
||
{"files":["src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianProvider.cs","tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformanceBackend.cs","tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformance.cs","src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/HistorianBackendRegistry.cs","src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/ServerHistorianOptionsValidator.cs","src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj","docker-dev/docker-compose.yml"],"verifyCommand":"both IT projects green with env set; Host historian filters green","acceptanceCriteria":["provider shape","registry + consumer-gated validator","full conformance suite green on QuestDB","compose env"],"modelTier":"frontier","model":"opus","wave":4}
|
||
```
|
||
|
||
---
|
||
|
||
### Task 8: Documentation
|
||
|
||
> **Review amendments (Fable) — binding:**
|
||
> - Document these, in addition to the existing list:
|
||
> - R6's sub-µs request-start consequence;
|
||
> - QuestDB `LONG` NULL = `long.MinValue`, so a stored `FromInt64(long.MinValue)` reads back Empty (N1);
|
||
> - TTL is enforced on write commits only, so a quiet table never drops (N3);
|
||
> - all write paths wait for WAL visibility (R5).
|
||
> - **CLAUDE.md (N5):** check the Historian section's opening paragraph. If it still says
|
||
> "HistorianGateway (sole historian backend)", rewrite it to name all three backends. Otherwise add QuestDb to the
|
||
> existing "Pluggable backends" subsection.
|
||
|
||
|
||
**Goal:** the docs describe the QuestDB backend as built, and every claim is checked against the merged code.
|
||
|
||
**Files:** `docs/Historian.md` (a new §"QuestDB backend"), `docs/Configuration.md` (the `ServerHistorian:QuestDb`
|
||
table), `CLAUDE.md` (one short bullet in the Pluggable backends subsection), `docker-dev/README.md` (the overlay
|
||
recipe), and the spec (an as-built pass)
|
||
|
||
**Acceptance Criteria:**
|
||
- [ ] Every `ServerHistorian:QuestDb` key is documented with its default and validation rule.
|
||
- [ ] The QuestDB-specific behaviour is documented:
|
||
- soft delete and tombstones;
|
||
- insert-if-absent including tombstones;
|
||
- microsecond truncation and collisions;
|
||
- WAL visibility and the read-your-writes wait;
|
||
- TTL;
|
||
- the overlay recipe and `HISTORIAN_QUESTDB_TEST_CONNECTION`.
|
||
- [ ] Each statement is cross-checked against the code, and any spec-versus-code disagreement is fixed in the spec
|
||
inline and listed in the report.
|
||
|
||
**Verify:** the controller reads the diff, and `grep -n "QuestDb" docs/Configuration.md` shows the table.
|
||
|
||
```json:metadata
|
||
{"files":["docs/Historian.md","docs/Configuration.md","CLAUDE.md","docker-dev/README.md","docs/plans/2026-10-10-questdb-historian-design.md"],"verifyCommand":"grep -n QuestDb docs/Configuration.md","acceptanceCriteria":["keys documented","QuestDB-specific semantics documented","claims cross-checked"],"modelTier":"standard","model":"sonnet","wave":5}
|
||
```
|
||
|
||
---
|
||
|
||
### Final review, then Task 9: live gate on docker-dev (controller)
|
||
|
||
**Final review:** a **fable** reviewer reads `git diff master...feat/questdb-historian` together with the spec. It must
|
||
report PASS or FAIL on each of these checks:
|
||
1. SqlServer behaviour is unchanged: identical conformance results, and no deleted SqlServer test.
|
||
2. No QuestDB `UPDATE` or `DELETE` is issued on data.
|
||
3. `NOT deleted` is present in every read path.
|
||
4. Every new status literal is a named const.
|
||
5. No package or version changes.
|
||
6. The bucket expression is exact.
|
||
7. #491 still holds: no Self/Context access after an await.
|
||
|
||
Then one fix dispatch (opus) and a scoped re-review (sonnet), followed by adjudication.
|
||
|
||
**Task 9: live gate (controller only).**
|
||
|
||
**Goal:** prove `Backend=QuestDb` end to end on the docker-dev rig, against the overlay `questdb` service, with captured
|
||
output for every leg.
|
||
|
||
> **USER-ORDERED GATE — NON-SKIPPABLE.** This task was requested by the user in the current conversation. It MUST NOT
|
||
> be closed by walking around it, by declaring it "verified inline", or by substituting a cheaper check. Close only
|
||
> after every item in `acceptanceCriteria` has been re-validated independently, with output captured.
|
||
|
||
**Files:** `docs/plans/2026-10-10-questdb-historian-live-gate.md`
|
||
|
||
**Acceptance Criteria:** run the six legs of spec §7 on docker-dev with `OTOPCUA_HISTORIAN_BACKEND=QuestDb`. Each leg
|
||
needs captured output.
|
||
|
||
**Verify:** the evidence doc has each leg's CLI output plus the direct QuestDB query output.
|
||
|
||
**Steps:**
|
||
1. Bring up the overlay `questdb`.
|
||
2. Rebuild the central image and recreate the central nodes with `OTOPCUA_HISTORIAN_ENABLED=true`,
|
||
`OTOPCUA_HISTORIAN_BACKEND=QuestDb` and `OTOPCUA_CONTHIST_ENABLED=true`.
|
||
3. Historize FastUInt1, a string calculation tag and `HR5`. Add a UNS reference. Mint the two gate API keys, as in the
|
||
SQL Server gate recipe (`docs/plans/2026-10-09-pluggable-historian-live-gate.md`), and deploy.
|
||
4. Run legs 1–6 with direct `psql` or REST queries. QuestDB REST is `curl -G localhost:19000/exec --data-urlencode "query=…"`.
|
||
5. Write the evidence doc.
|
||
6. Restore the rig: recreate the centrals with defaults, revert the config DB, redeploy, `down` the overlay, and stop
|
||
the fixtures.
|
||
|
||
```json:metadata
|
||
{"files":["docs/plans/2026-10-10-questdb-historian-live-gate.md"],"verifyCommand":"Client.CLI historyread/historyupdate against opc.tcp://localhost:4840 + QuestDB REST /exec on :19000","acceptanceCriteria":["Primary-only typed recording one row per value","raw+UNS HistoryRead/Processed identical and match QuestDB","HistoryUpdate codes + delete not resurrected by redelivery + WriteOperate-only and anonymous denied","failover 1x","Modbus loss: BadCommunicationError then BadNoCommunication marker then Good","TTL visible in tables() metadata"],"userGate":true,"tags":["user-gate"],"modelTier":"frontier","model":"controller","wave":5}
|
||
```
|
||
|
||
---
|
||
|
||
## Finish
|
||
|
||
- Merge to `master` and push only with the user's go-ahead.
|
||
- After merging, append the scadaproj `sister-project-history/otopcua.md` entry on scadaproj `main` and push it.
|
||
- Close #666.
|