Compare commits
123 Commits
f627cee9c0
..
main
| Author | SHA1 | Date | |
|---|---|---|---|
| 6ad15fcf93 | |||
| 795af1da71 | |||
| 65c6b99cf2 | |||
| 420c2e0783 | |||
| 3cacdca861 | |||
| ade49c0935 | |||
| 02480d4fbc | |||
| 87bb647efb | |||
| a0cec479b6 | |||
| 91baba8fcc | |||
| 9688707d82 | |||
| 509114360d | |||
| cfd371ce18 | |||
| 51a956a878 | |||
| 2a3baae301 | |||
| 10fc22b87e | |||
| 8825cd966f | |||
| 69a7f8e1b5 | |||
| 01a6313fe7 | |||
| e7cfa21fda | |||
| 68f1b56acf | |||
| b4da0deacb | |||
| 180403cb17 | |||
| 67c182ce18 | |||
| 05042e5e36 | |||
| aed5fbf046 | |||
| 00ef22823f | |||
| c05a6d93b0 | |||
| ec2615a9d2 | |||
| 2a0b34e056 | |||
| 4555a3bdad | |||
| 681606dad0 | |||
| aa7093b53b | |||
| 17cb91a315 | |||
| 0adfb244b2 | |||
| c6e60cb6c4 | |||
| c713925ac7 | |||
| 0b056c6fee | |||
| 484b176cd6 | |||
| dd1eea2106 | |||
| 63cc772e8d | |||
| 3b4d81e630 | |||
| b6f5eb2535 | |||
| a8f5f66647 | |||
| df21d365ec | |||
| a2b40b7922 | |||
| 3c91f0f813 | |||
| a6c4161719 | |||
| 78ca2a6744 | |||
| 21a95c7f50 | |||
| c43d7ee1ad | |||
| d0b66bf105 | |||
| 44421dc5aa | |||
| 8fa7631ec4 | |||
| b62cfad48d | |||
| 625f207e08 | |||
| 29bfa8fbf9 | |||
| b0582eb66a | |||
| 9090b1b511 | |||
| e3d146aa21 | |||
| 177881c423 | |||
| 3c42069673 | |||
| 00ac436ae2 | |||
| 302d1245cd | |||
| fa0ca66a05 | |||
| 711e8beb2b | |||
| 9ac6ed51d3 | |||
| efd5d881dc | |||
| 0793159cbb | |||
| 51b6f1ab9f | |||
| 29d0c50f07 | |||
| adbf2eb182 | |||
| b392b84f21 | |||
| 5c73be486d | |||
| 308742b7a6 | |||
| 8cfe076012 | |||
| 646af887d1 | |||
| 1e0bfe03d4 | |||
| 9934bea05a | |||
| d0079ba1a1 | |||
| 4dda768801 | |||
| f6eef9d8eb | |||
| 16a84f9df4 | |||
| 3c92bf54b0 | |||
| fb33924473 | |||
| 46fa06a612 | |||
| 5ee31dbf7b | |||
| 98868a60d0 | |||
| 3903c9a196 | |||
| ef487957b2 | |||
| 895e881807 | |||
| 54f111e8b1 | |||
| 370a9eea96 | |||
| 4988c24618 | |||
| fe6216c533 | |||
| 10990a43c8 | |||
| e538ea36e3 | |||
| 63e6d3f867 | |||
| c72271aa8a | |||
| 2e545b50da | |||
| 6acc5c1908 | |||
| c54cd808f4 | |||
| de21669ad5 | |||
| dcee4199ab | |||
| b5da1c8c57 | |||
| c1e34c307e | |||
| 2969805e4e | |||
| 3f11704d10 | |||
| 300094e792 | |||
| 75c1897522 | |||
| 5e02c1c38f | |||
| 9663279594 | |||
| 122940c0e4 | |||
| e9dc13689a | |||
| f62b017cc7 | |||
| dbcbc5bd41 | |||
| 1145df682e | |||
| 3e5a5fde81 | |||
| b62981ff80 | |||
| 31a7c8a88a | |||
| 7757b9c6fc | |||
| 0038d46c9a | |||
| cea3930b94 |
@@ -0,0 +1,16 @@
|
||||
---
|
||||
name: sister-project-history
|
||||
description: Use when you need the detailed change history, live-gate records, redundancy/auth/ACL behaviour, or ScadaBridge consumer-impact notes for a sister project (OtOpcUa, HistorianGateway, BTDB) — the scadaproj root CLAUDE.md keeps only a short summary row per project; the full narrative lives in this directory.
|
||||
---
|
||||
|
||||
# Sister project history
|
||||
|
||||
The root `CLAUDE.md` index keeps one short row per sister project. The full narrative rows that used to live
|
||||
in that table were moved here **verbatim** on 2026-09-03 (they were ~57k characters loading into every session):
|
||||
|
||||
- `otopcua.md` — OtOpcUa: v3 dual-namespace address space, every driver, redundancy/mesh program, data-plane auth + ACL enforcement, deferment register, the authoring-vs-deploy-gate parity arc, and the per-issue consumer-impact record for ScadaBridge.
|
||||
- `historiangateway.md` — HistorianGateway: ArchReview cycles, pending.md items, session pools, datatype gaps, SQL-path workarounds, PR/commit history.
|
||||
- `btdb.md` — BTDB (Batch Tracking Database, added 2026-09-03): schema + sproc write contract, Historian/CLR coupling, Galaxy/MESAPI/ScadaBridge edges, the DARS `BatchTracker_DB` fork, and the gap analysis.
|
||||
|
||||
Read the file for the project you are working against. When a sister repo tells you to "update its entry in
|
||||
`scadaproj/CLAUDE.md`", append the detail to the file here and keep the index row to a few lines.
|
||||
@@ -0,0 +1,207 @@
|
||||
# BTDB (Batch Tracking Database) — index history + analysis
|
||||
|
||||
> Added 2026-09-03 as a sister project. **This file is the authoritative place for BTDB detail — append here and keep the root index row short.** Everything below was verified from source on 2026-09-03 unless marked _(inferred)_.
|
||||
|
||||
| Project | Location | Stack | Repo | Summary |
|
||||
|---|---|---|---|---|
|
||||
| **BTDB** (Batch Tracker) | `~/Desktop/btdb` (case-insensitive FS: also `BTDB`) | SQL Server 2016+ SSDT project + SQL CLR (net472) · manager: **Blazor Web App net10.0** on `feat/manager-blazor` (§8; `master` still carries the EOL net6.0 MVC app) | **`ZimmerBiomet/BTDB`** on the corporate Gitea `wonder-app-vd03.zmr.zimmer.com:3000` (`master` @ `d147c10e`, 2023-07-26). **Not** on `gitea.dohertylan.com`. | The `BT` SQL Server database that is the plant's **batch/cycle/work-order system of record for data collection**: Galaxy machine objects call its `StartMachineBatch`/`StartMachineCycle`/`End*`/`AddMachineBatchWorkOrder`/`AddOtherData*` sprocs; reports pull time-series per batch out of the AVEVA Historian through its CLR sprocs (`OPENQUERY(INSQL, … WideHistory …)` + `Runtime.dbo.*`); MESAPI and ScadaBridge's inbound API resolve `SAPID → Machine.Code` from `dbo.Machine`. `BTDBManager` is a small Windows-service MVC site for machine/group config, batch edits (audited) and cross-environment config import. |
|
||||
|
||||
## 1. What is on disk, and where the truth is
|
||||
|
||||
- `~/Desktop/btdb` is **not a git checkout** — no `.git`, no `CLAUDE.md`, no `README`. It is a file-for-file snapshot of `ZimmerBiomet/BTDB` `master` (the only branch) at its head commit `d147c10e` (2023-07-26, "Update notification system tables to add urgent flag"). Verified by comparing the Gitea tree with the local tree: 170 vs 168 files, the only difference being the remote's `.gitattributes`/`.gitignore`. The repo's `updated_at` of 2024-11-07 is not a commit (the branch head is still 2023-07-26).
|
||||
- `~/Desktop/DARS/WW_ARCH/BatchTracker_DB/` is a **fork of the `Database` project only** (no Manager), modified 2026-08-13, also not under git (DARS is not a repo). Substantive differences vs the snapshot, all matching the DARS sprint-1 spec `DARS/sprints/sprint1/bt_db_changes.md`:
|
||||
- new table `MachineBatchTransfer` (source/dest batch chain for mid-run reactor failure; `TransferType` CHECK `'mid_run_failure'`; three indexes) + sproc `AddMachineBatchTransfer` (linear-chain guard);
|
||||
- `MachineBatchWO` gains `ContainerID NVARCHAR(64) NULL` + `MoveOutTimestamp DATETIME2 NULL` with filtered indexes (Camstar container traceability; mirrors the `SAPID`/`ZTag` idiom);
|
||||
- new CLR sproc `GetLinkedBatchTSV` (+ `_Wrapper`) that walks the transfer chain and stitches TSVs, and `LinkedBatchAlarm_Wrapper`;
|
||||
- `StartMachineBatch.sql`, `MachineBatch.sql`, `AuditLogMachineBatch.sql` differ **only** in BOM/CRLF → LF normalization (no logic change).
|
||||
- Reference doc for the schema: `DARS/WW_ARCH/BatchTracker_Reference.md` (states 22 tables / 35 SQL sprocs / 4 CLR sprocs / 5 CLR UDFs / 6 table types / 1 aggregate).
|
||||
- **Resolved 2026-09-03 — the divergence is closed locally:** `~/Desktop/btdb` was turned into a real checkout (`git init` + fetch of `ZimmerBiomet/BTDB`, `master` tracking `origin/master`; the snapshot was tree-identical apart from exec bits copied from Windows, which were cleared). The DARS fork's transfer-chain work was migrated onto branch **`feat/machine-batch-transfer`** as one commit: the five new objects, the two `MachineBatchWO` columns, and the `.sqlproj` wiring **including the `Compile` entry for `GetLinkedBatchTSV.cs` that the fork had omitted** (without it the CLR sproc is never built and its wrapper fails at deploy). The BOM/CRLF re-saves were not carried over. The sprint-1 plan (`bt_db_changes.md`, overview, diagrams) and `BatchTracker_Reference.md` now live under `btdb/docs/`. **The branch is local only — not pushed to the corporate Gitea.** `DARS/WW_ARCH/BatchTracker_DB/` is now superseded and should be deleted or replaced by a pointer once the branch is pushed. A second commit on the same branch carried the two columns through the Manager's batch-edit path: `MachineBatchWOType` (appended last — the TVP binds by ordinal and `ToDataTable` emits properties in declaration order, so the view model's property order must keep matching), `AuditLogMachineBatchWO` (`Prev_`/current pairs), the DTOs and view model, `SaveBatchEdit.sql` (MERGE change-detection with `COALESCE` because both columns are NULL-able, OUTPUT and the three audit inserts), and the Kendo work-order grid in `Views/BatchEdit/Edit.cshtml` (masked timestamp editor like the cycle grid). `LookupBatchInfo.sql` is `SELECT *` so it needed nothing. Not touched: the Telerik `BatchEditAuditLogReport.trdp` (binary package; open it in the Telerik designer to add the audit columns) and the batch search grid (`SearchBatches.sql`), which still lists only work-order numbers. Nothing here was built or run: the Manager needs Telerik + Windows, and the SSDT project needs MSBuild; a throwaway compile of the Manager retargeted to .NET 10 on the Mac showed no errors in the touched files.
|
||||
|
||||
## 2. Solution shape
|
||||
|
||||
`BTDB.sln` (VS 2022) = two projects:
|
||||
|
||||
| Project | Kind | Notes |
|
||||
|---|---|---|
|
||||
| `Database/Database.sqlproj` | SSDT, `Sql130DatabaseSchemaProvider` (SQL Server 2016+), CLR assembly targets **net472**, C# namespace `BT` | 22 tables, 6 table types, ~35 T-SQL sprocs, 3 CLR sprocs (`GetTSV`, `GetMachineBatchTSV`, `GetCycleTSV`), 5 CLR UDFs (`GetAggregateValues`, `CheckTriggered`, `WhenTriggered`, `WhenCleared`, `StringCompare`), 1 CLR aggregate (`StringJoin`), 1 T-SQL TVF (`GenerateShifts`), 1 trigger (`TrgNotification` → `AuditLogNotification`). Requires `CLR enabled` on the instance. |
|
||||
| `Manager/BTDBManager.csproj` (**`master` only — deleted on `feat/manager-blazor`, see §8**) | ASP.NET Core MVC, **net6.0** (out of support since 2024-11), Dapper 2.0, `System.Data.SqlClient`, `System.DirectoryServices` (Windows-only LDAP), `Microsoft.Extensions.Hosting.WindowsServices`, Kendo UI + Bootstrap, **Telerik Reporting R2 2022** referenced by an absolute `HintPath` into `Program Files (x86)` | Kestrel `http://*:9876`; runs as a Windows service; 13 embedded `.sql` query resources; 3 `.trdp` Telerik audit reports. `Manager/install.bat`/`uninstall.bat` are **leftovers from a different service** (`WWNotificationSystem.exe`) and do not install the Manager. |
|
||||
|
||||
## 3. Database model (what the family writes and reads)
|
||||
|
||||
**Core hierarchy:** `Machine` (`Code` unique; optional unique `ZTag`, `SAPID`; `TimeZone`; `MultipleBatch`/`MultipleCycle` flags; `OtherData` JSON) → `MachineBatch` (start/end operator + UTC timestamps, `OtherData`) → `MachineCycle` (same shape) and `MachineBatchWO` (`WorkOrderNumber`, `ItemNumber`). `MachineGroup` + `MachineGroupMembership` give roll-ups (DARS models one `Machine` per reactor *side*, one `MachineGroup` per unit). `MachineAlarm` is a per-machine alarm catalog (`Severity` 0–999, `FlaggedForMES`).
|
||||
|
||||
**Lifecycle sprocs (the write contract):**
|
||||
|
||||
| Sproc | Behaviour |
|
||||
|---|---|
|
||||
| `StartMachineBatch(@Code, @OperatorID, @o_MachineBatchID OUT)` | Looks up machine by `Code`; unless `MultipleCycle`/`MultipleBatch`, **auto-closes** any open cycle/batch first; inserts with `GETUTCDATE()`. Throws 50000 if machine unknown. |
|
||||
| `EndMachineBatch(@MachineBatchID, @EndOperatorID)` | Closes the open cycle too; throws if not found / already closed. |
|
||||
| `StartMachineCycle(@MachineBatchID, @OperatorID, @o_MachineCycleID OUT)` / `EndMachineCycle` | Cycle must belong to an open batch. |
|
||||
| `AddMachineBatchWorkOrder(@MachineBatchID, @WorkOrderNumber, @ItemNumber, @o OUT)` | Only on open batches. |
|
||||
| `AddOtherDataMachineBatch` / `…MachineBatchWorkOrder` / `…MachineCycle (@ID, @DataKey, @DataValue)` | `JSON_MODIFY` into `OtherData` — via **dynamic SQL that concatenates `@DataKey`/`@DataValue`** into the statement (`sp_executesql` of a built string). Injection surface; callers today are trusted Galaxy scripts. |
|
||||
|
||||
All timestamps are `GETUTCDATE()`; `Machine.TimeZone` is used only at retrieval time (`wwTimeZone=` in report wrappers). This matches the family's all-UTC rule.
|
||||
|
||||
**Historian coupling (the read contract):** the CLR code resolves a machine-relative tag (`<Code>.<local>`; `*.<attr>` shorthand; `|literal|` static columns) to the Historian's global tagname via `Runtime.dbo.TagExtendedPropertyInfo` (`HierarchicalName`), then runs `SELECT … FROM OPENQUERY(INSQL, 'SELECT DateTime, [tag]… FROM WideHistory WHERE … wwTimeZone="UTC" AND <queryParams>')` (`wwRetrievalMode`, `wwResolution`, `wwTimeZone` are passed through **verbatim** from `@queryParams`). Discrete checks read `Runtime.dbo.DiscreteHistory`; alarm wrappers read `Runtime.dbo.Events` (`IsAlarm=1`, `Alarm_OriginationTime`, `Alarm_DurationMs`, `Alarm_UnAckDurationMs`). **Consequence: the `BT` database must live on (or three-part-name-reach) the Historian's SQL instance with the `INSQL` linked server** — it is co-tenant with `Runtime`, which is why it sits on the `wonder-sql-*` boxes.
|
||||
|
||||
**Report wrappers** are per machine family and are the consumers of the CLR layer: Crest (`CrestWide`/`CrestMinMax`/`CrestAlarm`/`CrestCleaner*`), Elma (`ElmaData`, `Elma_Bulk_Query_Nitric`), `ExtruderWide`, `PultrusionWide`, Haas (`HaasStatusReport`, `HaasAlarm`, `QV_Haas*` = QlikView), Solar (`SolarAlarms`/`SolarEvents`), DryHeat (`DryHeatCycleReport`/`Search`/`BulkReport`), PoncePolyPress, `Z002432Wide`/`Z002433Wide`. `GenerateShifts` builds a 3-shift calendar table for these.
|
||||
|
||||
**Auth tables that are not the Manager's:** `UserAuth`, `UserAuthDetails`, `UserAuthRole`, `ApiKey` are ServiceStack's schema — they belong to **MESAPI/WWSupport** (the Camstar-facing REST service, see `mes-delmia-integration-api.md` §1.1: `ApiKeyAuthProvider` + `LdapAuthProvider`, role `MESAPI`), which shares the `BT` database. The Manager authenticates via LDAP only.
|
||||
|
||||
**Notification tables** (`Notification`, `NotificationLog`, audited by trigger) back the separate `WWNotificationSystem` Windows service, which is **not in this repo**.
|
||||
|
||||
## 4. BTDBManager (the config/edit UI) — `master` state; the replacement is §8
|
||||
|
||||
- **Auth:** cookie scheme; `LdapAuthenticationService` binds `LDAP://nam.zimmer.com` with the user's own credentials via `System.DirectoryServices` (so it only runs on a domain-joined Windows host), reads `memberOf`, and keeps only the three configured group DNs. Policies `READONLY` / `MACHINECONFIG` / `BATCHEDIT` map to `ReadOnlySecurityGroup` / `MachineConfigSecurityGroup` / `BatchEditSecurityGroup`; the checked-in DEV config points all three at `UG_WW_APP_ADMIN_DEV`. A global `AuthorizeFilter` makes every controller (incl. `/Api/GetMachines`, `/Api/GetMachineGroups`) require login. This is exactly the AD-group→role problem the family's `ZB.MOM.WW.Auth` 0.3.0 + `ldap_mapping.md` already solve.
|
||||
- **Features:** Machine CRUD (with group memberships + alarm catalog via TVPs `MachineGroupMembershipType`/`MachineAlarmType`), MachineGroup CRUD, batch search/edit (`SaveBatchEdit` writes `AuditLogBatchEdit` + prev/new `AuditLogMachineBatch`/`…WO`/`…Cycle` rows), **MachineConfigImport** (copies machine/group config from another environment's `BT` — sources `QA` and `PROD` are hard-wired in `appsettings.json`), and Telerik `.trdp` audit reports.
|
||||
- **Secrets are checked in:** `appsettings.json` carries plaintext `wonderapp` SQL passwords for DEV/QA/PROD, and `install.bat` carries a domain service-account password. Treat all of them as compromised-by-repo; do not copy them into any other file or reply. This is the pattern `ZB.MOM.WW.Secrets` (`${secret:}`) exists to replace.
|
||||
- **Environment hosts as recorded in 2023:** DEV `wonder-sql-vd01/BT`, QA `wondersqlqa/BT` (`MultiSubnetFailover`), PROD `wondersqlpd/BT`. ScadaBridge's 2026-06-16 design says the DEV BTDB is on **`wonder-sql-vd03/BT`** (db-connection id 1). `mes-delmia-integration-api.md` also says `vd01`. **Host drift is unresolved** — do not assume either without probing.
|
||||
|
||||
## 5. How BTDB relates to the rest of the family
|
||||
|
||||
```
|
||||
Galaxy machine objects ──(script: EXEC StartMachineBatch / StartMachineCycle /
|
||||
($Reactor_Side etc.) AddMachineBatchWorkOrder / AddOtherData* …)──────────┐
|
||||
▼
|
||||
Camstar MES ──HTTPS──▶ MESAPI/WWSupport ──(SELECT Machine by SAPID; alarm catalog; ┌────────────────────┐
|
||||
(ServiceStack, own UserAuth/ApiKey tables) ────────────────▶ │ SQL Server "BT" │
|
||||
│ (BTDB) │
|
||||
ScadaBridge inbound API ──(Database helper, connection "BTDB": SELECT Code FROM │ co-tenant with │
|
||||
(IpsenMESMoveIn, MES alarm-status) ─────────────────────────────────────────────▶ │ Historian Runtime │
|
||||
ScadaBridge site scripts ──(Database.Connection / CachedWrite — customer scripts)─▶ │ + INSQL linked │
|
||||
└─────────┬──────────┘
|
||||
BTDBManager (config, batch edits, env import) ─────────────────────────────────────▶ │ CLR: OPENQUERY(INSQL) WideHistory,
|
||||
SSRS / QlikView / Telerik reports ──(EXEC *_Wrapper, GetMachineBatchTSV …)──────────▶ │ Runtime.dbo.{TagExtendedPropertyInfo,
|
||||
▼ DiscreteHistory, Events}
|
||||
AVEVA Historian (Runtime DB)
|
||||
```
|
||||
|
||||
- **Galaxy → BT** is the primary write path (batches/cycles/work orders/OtherData JSON). DARS `model_info.md` §: "Existing per-machine BT sprocs (`StartMachineBatch`, `StartMachineCycle`, `GetCycleTSV`, `Alarm_Wrapper`, …) work unchanged" — the sprocs are the contract Galaxy scripts bind to.
|
||||
- **ScadaBridge → BT (verified):** the inbound-API script context has a `Database` helper bound to named db-connections; production defines connection **`BTDB`** (id 1), and the `IpsenMESMoveIn` (design 2026-06-16) and MES alarm-status (2026-06-30) methods do `SELECT Code FROM dbo.Machine WHERE SAPID=@s` before `Route.To(instance)`. Template/site scripts expose `Database.Connection(name)` + `Database.CachedWrite(...)`, so site scripts **can** write batches into BT — but **no checked-in ScadaBridge script calls a BT sproc**; any batch logging is customer-site script configuration, not platform code, and the docker rig does **not** seed a `BT` database (`infra/mssql/setup.sql` creates only `ScadaBridgeConfig` + `ScadaBridgeMachineData`; the `InboundDatabaseHelperTests` use a stand-in). ScadaBridge's inbound API is the designated successor of MESAPI's move-in/move-out surface, which is why it needs the same `SAPID → Code` lookup.
|
||||
- **HistorianGateway ↔ BT:** no direct coupling, but both read `Runtime.dbo.Events` (HG's SQL-path `ReadEvents`) and both need the Historian SQL instance. BT's CLR `GetTSV`/`GetAggregateValues` is a bespoke, in-database equivalent of what HG's `ReadRaw`/aggregate retrieval modes do over gRPC — a candidate for consolidation if the report layer is ever re-platformed.
|
||||
- **OtOpcUa / mxaccessgw:** no coupling.
|
||||
- **Break surface:** the sproc signatures + `Machine.Code`/`SAPID`/`ZTag` semantics (Galaxy scripts, MESAPI, ScadaBridge inbound methods all bind to them), the `OtherData` JSON key conventions (DARS sprint 1 defines ~14 new keys with single writers), and the `INSQL`/`Runtime` co-location assumption.
|
||||
|
||||
## 6. Analysis — gaps and recommendations (2026-09-03)
|
||||
|
||||
1. **Push the branch.** ~~Put both copies under git~~ — done locally 2026-09-03 (`feat/machine-batch-transfer`). Remaining: push it to `ZimmerBiomet/BTDB`, review/merge, decide whether BTDB also gets a mirror on `gitea.dohertylan.com` like the other sisters, then delete the superseded `DARS/WW_ARCH/BatchTracker_DB/` copy.
|
||||
2. **Rotate and externalize the checked-in credentials** (three SQL passwords, one domain account). _Externalised on `feat/manager-blazor` (`${secret:btdb/...}` refs, store outside the content root); rotation is a cut-over step (§8)._ The family standard is `ZB.MOM.WW.Secrets` `${secret:}` with an absolute `Secrets:SqlitePath` outside the content root.
|
||||
3. **Manager re-platform** — _done on `feat/manager-blazor`, see §8_: net6.0 is EOL; Telerik is pinned by an absolute machine path; `System.DirectoryServices` binds the user's own credentials on a domain-joined box. A .NET 10 rewrite on the family libs is mostly assembly: Auth 0.3.0 (LDAP + `UG_WW_*` group→role mapping, `FallbackServers`), Theme 0.4.1, Health 0.3.0 (`MapZbHealth`, so Overview can watch it), Telemetry, Configuration, Audit (the batch-edit audit is a natural `ZB.MOM.WW.Audit` writer), Secrets. Keep Dapper + the embedded queries; replace Telerik `.trdp` with server-side rendering only if the three audit reports are still used.
|
||||
4. **Injection surfaces in the SQL layer:** `AddOtherData*` build dynamic SQL from `@DataKey`/`@DataValue`; `GetTSV`/`GetAggregateValues` interpolate tag names and append `@queryParams` raw into `OPENQUERY`. Trusted callers today, but ScadaBridge scripts and the inbound API widen the caller set. Fix `AddOtherData*` with parameterised `JSON_MODIFY` (no dynamic SQL needed), and whitelist `queryParams` to the three `ww*` keys.
|
||||
5. **ServiceStack auth tables** (`UserAuth*`, `ApiKey`) are MESAPI's, not BT's domain. Once ScadaBridge's inbound API fully replaces MESAPI (Ipsen MoveIn is the first slice; ScadaBridge keys already live in the `zbauth` schema via `Auth.ApiKeys.SqlServer` 0.3.0), those tables are dead weight in the schema and should be dropped from the sqlproj rather than carried forward.
|
||||
6. **Host inventory drift:** 2023 config says DEV BT is on `wonder-sql-vd01`; ScadaBridge's 2026 design says `wonder-sql-vd03`. Probe and record the answer in `deployment.md` §2.1 (which currently has no BT row at all).
|
||||
7. **`install.bat`/`uninstall.bat` are wrong for this project** (they install `WWNotificationSystem`). _Deleted on `feat/manager-blazor`; replaced by `deploy/install-service.bat` (NSSM, service `ZB.MOM.WW.BTDB.Manager`)._
|
||||
8. **CLR + Historian co-location is a hard deployment constraint** the index did not previously record: BT cannot be moved to the app-tier SQL (`wonder-app-vd03,1433`, where ScadaBridge's ConfigDb lives) without breaking every report wrapper. Any consolidation plan must keep BT on the Historian instance or replace the CLR with HistorianGateway calls.
|
||||
|
||||
## 7. Primary commands (Windows only)
|
||||
|
||||
```powershell
|
||||
# from ~/Desktop/btdb (master)
|
||||
dotnet build Manager\BTDBManager.csproj # net6.0; needs Telerik Reporting R2 2022 at the HintPath
|
||||
msbuild Database\Database.sqlproj /p:Configuration=Release # SSDT (VS 2022 + SSDT) → bin\Release\Database.dacpac
|
||||
SqlPackage /Action:Publish /SourceFile:Database\bin\Release\Database.dacpac /TargetServerName:<historian-sql> /TargetDatabaseName:BT
|
||||
# Instance prerequisites: CLR enabled, `INSQL` linked server, `Runtime` DB reachable by three-part name
|
||||
```
|
||||
|
||||
```bash
|
||||
# feat/manager-blazor (any OS for the app; SSDT still Windows-only)
|
||||
./build.sh # restore + build (Release, warnings-as-errors) + test BTDB.slnx
|
||||
dotnet run --project src/ZB.MOM.WW.BTDB.Manager # appsettings.Development.json → local glauth; needs a BT
|
||||
pwsh deploy/publish.ps1 # self-contained win-x64 → deploy/out/ (+ NSSM install scripts)
|
||||
```
|
||||
|
||||
```bash
|
||||
# Local Docker rig (2026-09-04, any OS) — see §9
|
||||
docker/deploy-db.sh # BT on scadabridge-mssql from Database/ (dacpac + SQL CLR); --recreate drops it first
|
||||
docker/deploy.sh # publish FDD on the host → zb-btdb-manager:local, up on :9877 (glauth login enabled)
|
||||
```
|
||||
|
||||
On `master` nothing builds on macOS (SSDT/CLR need Windows MSBuild, the MVC Manager needs Telerik + `System.DirectoryServices`). On the branch the app and its tests build and run on macOS; only `Database.sqlproj` (kept alone in `BTDB.sln`) still needs Windows — but §9's side projects build the same sources into a dacpac and a CLR assembly anywhere, for the local rig.
|
||||
|
||||
## 8. Blazor conversion — branch `feat/manager-blazor` (2026-09-04)
|
||||
|
||||
Plan: `btdb/docs/plans/manager_blazor_conversion.md` (+ `.tasks.json` tracker beside it; mockups `docs/mockups/manager_blazor_mockups.html`). Executed 2026-09-04 in worktree `~/.config/superpowers/worktrees/BTDB/feat-manager-blazor`; **uncommitted and unpushed at the time of writing** — phases 1–3 (scaffold, core, auth, widgets, queries, all pages) plus the local half of phase 4 are done; the DEV-box half of phase 4 (parity run, secrets seed + rotation, NSSM side-by-side, port swap) is written up as a runbook and not yet performed.
|
||||
|
||||
- **Shape on the branch:** `BTDB.slnx` = `src/ZB.MOM.WW.BTDB.Manager` (Blazor Web App, InteractiveServer, net10.0) + `src/ZB.MOM.WW.BTDB.Manager.Core` (Dapper data layer moved verbatim from the MVC app, hardened) + two xunit/bUnit test projects (334 tests, warnings-as-errors gate via root `Directory.Build.props`, `build.sh`/`build.ps1`). `BTDB.sln` keeps only `Database.sqlproj`. `Manager/` (MVC, Kendo, the three Telerik `.trdp` reports with embedded connection strings, the mis-named `install.bat`) is **deleted**; `/Api/GetMachines`+`GetMachineGroups` are gone (no external consumer — ScadaBridge/MESAPI read `dbo.Machine` by SQL).
|
||||
- **Family libs:** Theme 0.4.1, Health 0.3.0 (`MapZbHealth`; readiness = a `SELECT 1` against `BT` with a 3 s timeout, so `/health/ready` is 503 until the DB answers — register it in Overview), Telemetry + Serilog 0.1.0, Configuration 0.1.0 (`ConfigPreflight` requires `ConnectionStrings:BT` and `Btdb:Security:Ldap:Server`), Secrets 0.6.2 (`${secret:btdb/connectionstrings/{bt,qa,prod}}`, `${secret:btdb/ldap/serviceaccountpassword}`; SQLite store `C:\ProgramData\ZB.MOM.WW\btdb-manager\secrets.db`, KEK from `ZB_SECRETS_MASTER_KEY`), Auth.Ldap/AspNetCore 0.3.0 (service-account bind, `Btdb:Security:GroupToRole` → `CanonicalRole`; policies Viewer / BatchEdit (Operator+) / MachineConfig (Administrator); cookie `btdb.auth`, plain-HTTP cookie because the box has no certificate). `Microsoft.Data.SqlClient` 7.x with `Encrypt=False` forced by `BtConnectionStrings.Normalize` unless the string sets it.
|
||||
- **Behavioural deltas vs the MVC app worth knowing:** config import derives the group set from the selected machines' memberships (no separate group pick) and has a dry-run preview (`MachineConfigImportPreview.sql`: same MERGEs, `OUTPUT` into a table variable, always `ROLLBACK`); batch-edit timestamps are 7-fraction UTC strings validated on blur (no Kendo mask); the three audit reports are HTML tables with a print stylesheet instead of Telerik PDFs; `MachineAlarm` DTO property order/types were wrong in the MVC app (alarm import always threw) and are fixed against `MachineAlarmType`; every `Save*` runs in a transaction and `THROW 5001x` maps to `BtdbSaveException`.
|
||||
- **Cut-over material:** `docs/operations/cut-over.md` (parity checklist per route + THROW paths + audit-username check, secrets seed via the `secret` console + rotation, NSSM side-by-side on `:9877`, port swap to `:9876`, Overview registry entry), `deploy/` (`publish.ps1`, `install-service.bat` / `uninstall-service.bat` for NSSM service `ZB.MOM.WW.BTDB.Manager`), `tools/smoke-new-queries.sql` (runs the six never-executed queries read-only against DEV with sqlcmd). Still open after cut-over: Overview lacks the mockup's per-machine "open > 24 h" rows; Ldap `Server`/`SearchBase`/`ServiceAccountDn` are blank in `appsettings.json` by design (per-environment).
|
||||
|
||||
## 9. Local Docker rig — `docker/` (2026-09-04)
|
||||
|
||||
Mirrors the HistorianGateway / Overview pattern: publish framework-dependent on the host (Gitea feed creds), copy into `aspnet:10.0`, never restore in Docker. Compose project `zb-btdb`, container `zb-btdb-manager`, image `zb-btdb-manager:local`, port `:9877` (same as the DEV side-by-side port, so `appsettings.json`'s Kestrel section is the single binding authority; the Dockerfile clears `ASPNETCORE_URLS`/`ASPNETCORE_HTTP_PORTS`). `ASPNETCORE_ENVIRONMENT=Production`; all environment-specific values are compose env vars, dev-only and committed under the family's rig exception.
|
||||
|
||||
- **Database:** no MSSQL of its own — joins the external `scadabridge-net` and uses ScadaBridge's `scadabridge-mssql` (the instance that already carries `ScadaBridgeConfig` + `ZbSecretsHub`). `docker/deploy-db.sh`: (1) `docker/db/BT.Clr.csproj` compiles `Database/**/*.cs` to `Database.dll` (net472 via the auto-referenced reference assemblies — builds on macOS); (2) `docker/db/BT.Local.sqlproj` (SDK-style `Microsoft.Build.Sql/2.2.0`, `EnableDefaultSqlItems=false`, globs the four `Database/` folders, Sql130 model, suppresses the legacy build's own SQL71562/71558/71502) → `BT.Local.dacpac`; (3) `db/setup.sql` — `BT` + login `btdb_app` (db_owner); (4) `db/clr-objects.sql` — `clr enabled` on the instance, `sp_add_trusted_assembly` by SHA2-512 (strict security stays on), `CREATE ASSEMBLY [Database] … SAFE` from the bytes streamed as 2000-char `0x` chunks (a single 57 KB literal gets split by sqlcmd), then `StringJoin` aggregate, `StringCompare`/`CheckTriggered`/`WhenTriggered`/`WhenCleared`/`GetAggregateValues`, `GetTSV`/`GetMachineBatchTSV`/`GetCycleTSV`/`GetLinkedBatchTSV` with SSDT's type mapping; (5) `sqlpackage /Action:Publish` from a one-shot container (`db/Dockerfile.sqlpackage`, SDK image + the dotnet tool — the host's 10.0.5 runtime could not run the tool's 10.0.11 pin) with `DropObjectsNotInSource=false`, `BlockOnPossibleDataLoss=false`, `AllowIncompatiblePlatform=true`; (6) smoke. Result on the rig: 23 tables, 37 T-SQL procs + 1 TVF, 10 CLR objects, `Database` assembly `SAFE_ACCESS`. The script diffs the legacy `<Build Include>` list against the folders and warns on drift (found: `ElmaData_Nitric.sql` + `Elma_Bulk_Query_Nitric.sql` were on disk but not in `Database.sqlproj` — added). `ScadaBridge/infra/teardown.sh`/`reseed.sh` drop the volume and BT with it; re-run `deploy-db.sh`.
|
||||
- **Why not convert `Database.sqlproj` itself:** `Microsoft.Build.Sql` does not compile SQL CLR, and the Manager depends on `dbo.StringJoin` (batch search) and `dbo.StringCompare` (machine saves). The legacy SSDT project stays the artifact of record; the two side projects reference the same files, nothing is copied.
|
||||
- **Login:** enabled, `host.docker.internal:3893` + `FallbackServers` `:3894` (glauth-redundant), bind `cn=serviceaccount,dc=zb,dc=local`; `GroupToRole` `SCADA-Admins→Administrator`, `SCADA-Designers→Operator`, `SCADA-Viewers→Viewer`. Verified: `admin`/`designer` sign in, `gw-viewer` refused (fail closed). Literal `ConnectionStrings__BT` / LDAP password env vars override the `${secret:}` refs, so the `/data/btdb-secrets.db` store stays empty and no master key is set (confirmed to boot without one). The two config-import sources point at the local BT.
|
||||
- **Health / Overview:** `/healthz` 200, `/health/ready` 200 with the `sql` entry (503 when BT is down), `/health/active` 200, `/metrics`. Compose `healthcheck` = bash `/dev/tcp` GET of `/healthz` (liveness on purpose). Registered in `ZB.MOM.WW.Overview/src/…/appsettings.Docker.json` as application **BTDB Manager**, instance `manager`, `BaseUrl http://zb-btdb-manager:9877`, `HasActiveRole false`; the containerised Overview was republished and shows the tile Up with `sql` healthy (the dashboard already sits on `scadabridge-net`, so no extra network).
|
||||
|
||||
|
||||
## 10. State as of 2026-09-11 (supersedes §1, §4, §6.1, §6.3, §6.7, §8 where they differ)
|
||||
|
||||
- **Everything is on local `master`** (`78195b8`): the Blazor conversion (§8) was merged, the
|
||||
`feat/manager-blazor` worktree is gone, and the 2026-09-04 full review
|
||||
(`docs/reviews/2026-09-04-full-review.md`, reviewed at `c624d41`) was remediated in 12 work-package
|
||||
branches — `fix/save-batch-edit-null-compare`, `fix/db-lifecycle-sprocs`, `fix/clr-openquery-escaping`,
|
||||
`fix/config-import-parents-and-nulls`, `test/db-integration`, `fix/manager-robustness`,
|
||||
`fix/auth-hardening`, `ops/runbook-and-deploy`, `perf/indexes`, `db/schema-hygiene`,
|
||||
`fix/concurrency-and-async`, `chore/nits` — all merged; 60 findings fixed, 3 partial by decision
|
||||
(DB-15 `QV_*` impersonation, DB-16 alarm-CTE consolidation, DB-21 ServiceStack table drop), 0 open.
|
||||
`feat/machine-batch-transfer` (DARS sprint 1, §1 item 1) is merged too, and so is
|
||||
`feat/transfer-machine-batch` (merged `be976a6`, 2026-09-11): **`dbo.TransferMachineBatch`**, a tenth
|
||||
Galaxy-facing writer that ends the source batch if still open, starts the destination batch by machine
|
||||
`Code`, copies the `MachineBatchWO` rows still on the source (`MoveOutTimestamp IS NULL`, narrowed to
|
||||
`@ContainerID`; blank = whole batch; source rows never modified) and records the `MachineBatchTransfer`
|
||||
link — by composing `EndMachineBatch` / `StartMachineBatch` / `AddMachineBatchTransfer` in one owned
|
||||
transaction. Outputs `@o_DestMachineBatchID`, `@o_MachineBatchTransferID`. Documented in
|
||||
`docs/BatchTracker_Reference.md`; proven by `ValidateLifecycleSprocs.sql` §F (82 PASS on the rig).
|
||||
`fix/transfer-lot-walk` (merged `5f0da01`, 2026-09-11) **reopened review DB-6**: the 2026-09-04 loop guard
|
||||
in `AddMachineBatchTransfer` walked backward (toward the batches that fed the destination), which the
|
||||
already-transferred check makes unreachable for any source, so `A→B` then `B→A` was still accepted; it
|
||||
now walks forward along source→dest links with `HOLDLOCK`, and the destination row is held like the
|
||||
source. The chain walks in `GetLinkedBatchTSV.cs` and `LinkedBatchAlarm_Wrapper.sql` picked one root
|
||||
with `TOP 1` and dropped the other branch of a lot merge (`A→C, B→C`); both now resolve the whole lot
|
||||
(forward to the final batch, then every batch that fed it) with one CTE pair pinned equal by
|
||||
`SchemaHygieneTests`. Rig proofs: `ValidateLifecycleSprocs.sql` 87/0, `docker/db/validate-clr.sql`
|
||||
27/0 over a merged lot with a `Runtime.dbo.Events` stand-in.
|
||||
`feat/manager-transfers` (merged `695192d`, 2026-09-11) gave the **Manager** the transfer side it lacked: the
|
||||
batch edit page edits the batch's outgoing `MachineBatchTransfer` row as a fourth MERGEd collection in
|
||||
`SaveBatchEdit.sql` (guards `50015` self, `50016` destination missing, `50017` loop, `50018` more than one;
|
||||
destination row and chain walk held like `AddMachineBatchTransfer`), audited into the new
|
||||
`AuditLogMachineBatchTransfer` under the batch-edit header (new TVP `MachineBatchTransferType`); a read-only
|
||||
lot panel (`GetBatchLot.sql`, the same `Down`/`Lot` CTE as the CLR, pinned equal by a static test); a lineage
|
||||
column on Find Batch; a Transfers section on the batch-edit audit page and PDF. Rig proofs: 28 integration
|
||||
tests (three new: round trip with audit, loop refused, merged lot), smoke 17/17, lifecycle 87/0, transfer
|
||||
chains 4/0, validate-clr 27/0; gate 961 tests, 0 warnings. No branch is in flight.
|
||||
- **`master` is ~198 commits ahead of `origin/master` (`d147c10`) and unpushed.** Pushing is the user's
|
||||
call — the remote is the corporate `ZimmerBiomet/BTDB`.
|
||||
- **Solution shape now:** `BTDB.slnx` = `src/ZB.MOM.WW.BTDB.Manager` (Blazor host) +
|
||||
`src/ZB.MOM.WW.BTDB.Manager.Core` (Dapper data layer, embedded `Queries/*.sql`) + five test projects
|
||||
(`Manager.Core.Tests` static query invariants, `Manager.Tests` bUnit/boot, `Database.Tests` static SQL
|
||||
+ sqlproj invariants, `Clr.Tests` compiles the CLR sources into net10, `Manager.Core.IntegrationTests`
|
||||
opt-in on the rig behind `BTDB_TEST_CONNECTION`). `BTDB.sln` keeps `Database/Database.sqlproj`
|
||||
(Windows MSBuild only). `.gitea/workflows/build.yml` runs `./build.sh` (needs `GITEA_NUGET_TOKEN`).
|
||||
- **Invariants a future session must not break** (all test-enforced): TVPs bind by ordinal so model
|
||||
property order + lengths must equal `Database/Types/*.sql`; every write query is transaction-free and
|
||||
audited (the C# wrapper owns the transaction); `dbo.X` only, never `BT.dbo.X`; `SYSUTCDATETIME()` in
|
||||
Manager queries; every writer sproc follows the own-or-join `TRY/CATCH` template with locked lookups;
|
||||
every `.sql` on disk must be in `Database.sqlproj`'s `<Build Include>` list; optimistic concurrency via
|
||||
`ExpectedLastUpdate` → `THROW 50014`.
|
||||
- **Secrets/credentials (§6.2):** `${secret:btdb/connectionstrings/{bt,qa,prod}}` (the LDAP
|
||||
service-account secret is gone since 2026-10-01 — see below; only `dotnet run` against local glauth uses it), store at `C:\ProgramData\ZB.MOM.WW\btdb-manager\secrets.db`,
|
||||
master key only in `ZB_SECRETS_MASTER_KEY`. Rotation is cut-over step 4.2.
|
||||
- **Claude artifacts are gitignored + untracked in btdb** (2026-09-11): `CLAUDE.md`, `.claude/`,
|
||||
`docs/plans/`, `docs/reviews/`, `*.tasks.json` — they exist on disk (btdb's `CLAUDE.md` is the repo
|
||||
orientation and still loads) but never reach the corporate remote. `docs/operations/`,
|
||||
`docs/BatchTracker_Reference.md`, `docs/mockups/` stay tracked.
|
||||
- **Open step:** phase-4 cut-over on the DEV box (`docs/operations/cut-over.md`; 4.1 parity walk-through
|
||||
gates the rest) — **blocked by the 2026-09-11 remote-access hold** (no psbridge, no winrm; UNC copies and
|
||||
direct SQL logins still allowed). §6 item 6 (which SQL host carries DEV `BT`) is still unrecorded.
|
||||
- **2026-10-01 — LDAP sign-in by user bind** (merged `a81a480`): Auth 0.4.0/0.4.1 added opt-in
|
||||
`LdapOptions.BindMode=User`; BTDB ships it. Sign-in binds as `UserBindTemplate` (`<DOMAIN>\{0}` —
|
||||
down-level maps exactly to the sAMAccountName the group search uses), the same connection reads the
|
||||
user's own `memberOf`/`displayName`, and the username returned is the directory's spelling. No service
|
||||
account, no LDAP secret; the preflight requires the template and refuses `AllowInsecure`/`Transport=None`
|
||||
with user bind outside Development. Local dev + docker rig stay `BindMode=ServiceAccount` (GLAuth bind DNs
|
||||
embed the user's OU). Rollout to the three Manager nodes (vd03 first) and retiring the stored secret are
|
||||
the open steps (plan `docs/plans/2026-10-01-ldap-user-bind.md`, gitignored).
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1,5 @@
|
||||
# ScadaBridge — index history
|
||||
|
||||
- **2026-10-13 existing fixes verified and issues closed.** Main `513fa1aa` already includes #35 fail-closed unavailable configured body/SQL redactors (`006202f3`) and #37 Flow Debugger structured/raw validated method inputs (`b5eb08f1`). Fresh full AuditLog project 407 passed/0 failed; FlowDebuggerPageTests 47 passed/0 failed; both terminal exit0. Remote ancestry confirmed; #35/#37 closed with evidence. No source changes/deployment/MES transactions. #36 runtime reset-echo overload is implemented, but QA/prod script port remains tracked; #30 per-site PSK deployment gate remains open.
|
||||
|
||||
- **2026-10-13 #36 runtime regression proof refreshed.** Rebuilt HandshakeResetEchoTests passed all three cases with local fakes, terminal exit0; an earlier no-build attempt matched no tests and is excluded from evidence. Tracker now distinguishes implemented runtime b623fdb0 from remaining QA/prod database-owned script rollout (deployment register row119, scripts13/14/36 and15/16/25/26/37/38). No database writes, deployments or fresh live rig claims.
|
||||
+4
-2
@@ -7,8 +7,10 @@ obj/
|
||||
[Aa]rtifacts/
|
||||
*.user
|
||||
|
||||
# Claude Code tooling / agent worktrees (local-only)
|
||||
.claude/
|
||||
# Claude Code tooling / agent worktrees (local-only) — but .claude/skills/ IS tracked:
|
||||
# the sister-project-history skill holds the per-project narratives the root index points at.
|
||||
.claude/*
|
||||
!.claude/skills/
|
||||
|
||||
# Local ops/infra notes — internal access details, never publish
|
||||
zimmer-vpn-tunnel.md
|
||||
|
||||
@@ -0,0 +1,37 @@
|
||||
# ZB.MOM.WW.Audit
|
||||
|
||||
Shared audit event model + writer seam for the **ZB.MOM.WW SCADA family** (single package; DEEP-adopted by all apps as the canonical record). Build/test from this directory: `dotnet test`. Component spec + gaps: `../components/audit/`.
|
||||
|
||||
## Index history (migrated from the scadaproj root CLAUDE.md)
|
||||
|
||||
> Moved verbatim out of the scadaproj root `CLAUDE.md` on 2026-09-03 so it loads only when needed. **This file is now the authoritative place for this history — append new entries here and keep the root index row short.**
|
||||
|
||||
### Narrative
|
||||
|
||||
The audit component is fully populated: a normalized [`spec`](components/audit/spec/SPEC.md), an
|
||||
[`event-model`](components/audit/spec/EVENT-MODEL.md) reference, a
|
||||
[`shared-contract`](components/audit/shared-contract/ZB.MOM.WW.Audit.md), three
|
||||
[`current-state`](components/audit/current-state/) docs, and an adoption [`GAPS`](components/audit/GAPS.md)
|
||||
backlog. Common ground = canonical `AuditEvent` record + `AuditOutcome` enum + `IAuditWriter` /
|
||||
`IAuditRedactor` seams + helpers (`NullAuditRedactor`, `TruncatingAuditRedactor`, `NoOpAuditWriter`,
|
||||
`CompositeAuditWriter`, `RedactingAuditWriter`) + `AddZbAudit` DI registration; left per-project =
|
||||
transport/storage and domain vocabulary. Closes the loop on Auth — audit's `Actor` field = the Auth
|
||||
principal. `IAuditRedactor` is aligned with Telemetry's `ILogRedactor` seam convention.
|
||||
|
||||
The shared library is **built and lives in this repo** at [`ZB.MOM.WW.Audit/`](ZB.MOM.WW.Audit/)
|
||||
(.NET 10; 1 package — `ZB.MOM.WW.Audit`; only non-BCL dependency `Microsoft.Extensions.DependencyInjection.Abstractions`;
|
||||
23 tests (verified 2026-07-18); `dotnet pack` → 1 nupkg @ 0.1.0). Repo: `https://gitea.dohertylan.com/dohertj2/zb-mom-ww-audit`.
|
||||
**Adopted across all three apps on 2026-06-02** (audit GAPS #1–#6) on each repo's `feat/adopt-zb-audit` branch
|
||||
(stacked on `feat/adopt-zb-auth`) — committed + reviewed, then **merged into the repo's local default (main/master)
|
||||
and PUSHED to origin (gitea) on 2026-06-03** (in sync). Depth =
|
||||
**DEEP adopt** (the canonical 9-field `AuditEvent` is the record everywhere; domain fields ride in `DetailsJson`).
|
||||
OtOpcUa: canonical record + `AuditWriterActor : IAuditWriter` + `Outcome` column/migration + `ClusterAudit` fix.
|
||||
MxGateway: new canonical SQLite `audit_event` store + `IAuditWriter` + `IApiKeyAuditStore`→canonical adapter.
|
||||
**ScadaBridge: a full audit-subsystem re-architecture** (the program's largest task) — canonical record everywhere via a
|
||||
deterministic codec; site SQLite split into `audit_event` + an `audit_forward_state` forwarding sidecar; central
|
||||
partitioned `dbo.AuditLog` collapsed to 10 canonical cols + persisted computed cols (`CollapseAuditLogToCanonical`
|
||||
migration, MSSQL-verified). Phase 3 wires `Actor` from the Auth principal at authenticated emit sites (per-app
|
||||
`IAuditActorAccessor`). Per-repo detail in [`components/audit/GAPS.md`](components/audit/GAPS.md) +
|
||||
`docs/plans/2026-06-02-auth-audit-normalization-phase2-deep.md` + `…-scadabridge-audit-rearch.md`.
|
||||
Build/test from `ZB.MOM.WW.Audit/`: `dotnet test`. Consumer matrix: all three apps consume the single
|
||||
`ZB.MOM.WW.Audit` package (OtOpcUa, MxAccessGateway, ScadaBridge — DEEP-adopted as the canonical record).
|
||||
@@ -1,4 +1,10 @@
|
||||
#!/usr/bin/env bash
|
||||
# pack.sh — produce the ZB.MOM.WW.Audit NuGet package into ./artifacts.
|
||||
set -euo pipefail
|
||||
dotnet pack -c Release -o ./artifacts
|
||||
# cd to the repo root and name the solution + an absolute output path: `dotnet pack` with
|
||||
# neither a project argument nor a fixed working directory packs whatever solution happens
|
||||
# to sit in the CALLER's directory (the bug behind the LocalDb feed incident, 0038d46).
|
||||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "$REPO_ROOT"
|
||||
|
||||
dotnet pack ZB.MOM.WW.Audit.slnx -c Release -o "$REPO_ROOT/artifacts"
|
||||
|
||||
@@ -16,9 +16,20 @@ set -euo pipefail
|
||||
: "${GITEA_NUGET_SOURCE:?set GITEA_NUGET_SOURCE to your Gitea NuGet feed URL}"
|
||||
: "${GITEA_NUGET_KEY:?set GITEA_NUGET_KEY to your Gitea access token}"
|
||||
|
||||
dotnet pack -c Release -o ./artifacts
|
||||
# Named solution, named output, and a cd to the repo root, all for one reason: `dotnet pack`
|
||||
# with neither a project argument nor a fixed working directory packs whatever solution
|
||||
# happens to sit in the CALLER's directory — and then pushes it to a live feed under this
|
||||
# script's name. Invoked by absolute path from another repo's checkout, the relative form
|
||||
# packed and published 25 unrelated packages (LocalDb's copy of this script, fixed in
|
||||
# 0038d46; recovered with `dotnet nuget delete`, but only because the token happened to
|
||||
# carry delete rights). A publish script must be unambiguous about WHAT it publishes no
|
||||
# matter where it is run.
|
||||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "$REPO_ROOT"
|
||||
|
||||
dotnet nuget push "./artifacts/*.nupkg" \
|
||||
dotnet pack ZB.MOM.WW.Audit.slnx -c Release -o "$REPO_ROOT/artifacts"
|
||||
|
||||
dotnet nuget push "$REPO_ROOT/artifacts/*.nupkg" \
|
||||
--source "$GITEA_NUGET_SOURCE" \
|
||||
--api-key "$GITEA_NUGET_KEY" \
|
||||
--skip-duplicate
|
||||
|
||||
@@ -0,0 +1,41 @@
|
||||
# ZB.MOM.WW.Auth
|
||||
|
||||
Shared login / identity / API-key library for the **ZB.MOM.WW SCADA family** (4 packages + `ApiKeys.SqlServer`; consumed by OtOpcUa, ScadaBridge, MxAccessGateway, HistorianGateway). Build/test from this directory: `dotnet test` (the `ApiKeys.SqlServer` live suite is gated on `AUTH_SQLSERVER_CONNSTR`). Component spec + gaps: `../components/auth/`.
|
||||
|
||||
## Index history (migrated from the scadaproj root CLAUDE.md)
|
||||
|
||||
> Moved verbatim out of the scadaproj root `CLAUDE.md` on 2026-09-03 so it loads only when needed. **This file is now the authoritative place for this history — append new entries here and keep the root index row short.**
|
||||
|
||||
### Component-table row as it read in the index
|
||||
|
||||
| Component | Status | Goal | Design | Implementation |
|
||||
|---|---|---|---|---|
|
||||
| Auth (login / identity / authz) | Adopted (**feed carries `0.3.0` as of 2026-09-02 — five packages, the new one being `ZB.MOM.WW.Auth.ApiKeys.SqlServer`; ALL FOUR apps pin `0.3.0` as of 2026-09-03**: OtOpcUa `master @ b8b4c18f`, ScadaBridge `main @ e45c4af9` (the `inbound-api-key-shared-store` work, merged), MxAccessGateway `main @ 4f931e4`, HistorianGateway `main @ d8d890c` — each built + tested with no new failures versus its pre-bump baseline, all PUSHED to origin. The preceding `0.2.1` pins landed 2026-08-13 (OtOpcUa `08e33fc7`, ScadaBridge `d28824d9`, MxAccessGateway `1cb14d2`, HistorianGateway `e42ff31`). **`0.2.0` (2026-08-10) adds LDAP backup-DC sticky failover**, **`0.2.1` (2026-08-12) fixes a defect that broke AD logins outright** — an AD continuation referral escaped `NovellLdapConnection.Search`, so any `SearchBase` at a domain root failed every login as `ServiceAccountBindFailed` against a perfectly healthy directory. Live-proven against `DC=nam,DC=zimmer,DC=com`; the app-level exposure is closed by the pin bumps. **`FallbackServers` is live-gated** via the local redundant GLAuth pair ([`infra/glauth-redundant/`](infra/glauth-redundant/), primary :3893 + backup :3894 on the dev Mac): HistorianGateway dashboard drill PASS 2026-08-13 (failover, stickiness, walk-back). | Shared `ZB.MOM.WW.Auth` lib | [`components/auth/`](components/auth/) | [`ZB.MOM.WW.Auth/`](ZB.MOM.WW.Auth/) |
|
||||
|
||||
### Narrative
|
||||
|
||||
The auth component is fully populated: a normalized [`spec`](components/auth/spec/SPEC.md), a
|
||||
proposed [`shared-contract`](components/auth/shared-contract/ZB.MOM.WW.Auth.md), three
|
||||
[`current-state`](components/auth/current-state/) docs, and an adoption [`GAPS`](components/auth/GAPS.md)
|
||||
backlog. Common ground = LDAP/GLAuth identity + peppered-HMAC API keys; left per-project =
|
||||
the authz vocabularies (OPC-UA permissions / gRPC scopes / roles + site-scoping).
|
||||
|
||||
The shared library is **built and lives in this repo** at [`ZB.MOM.WW.Auth/`](ZB.MOM.WW.Auth/)
|
||||
(a plain directory of this repo — no nested `.git`, verified 2026-08-07; .NET 10; 4 packages — `Abstractions`, `Ldap`, `ApiKeys`, `AspNetCore`;
|
||||
250 tests passing + 2 skipped (verified 2026-08-10 at `0.2.0`; was 215+1 at `0.1.5`); feed carries `0.1.0`–`0.2.0`). The implementation plan is at
|
||||
[`docs/plans/2026-06-01-zb-mom-ww-auth-shared-library.md`](docs/plans/2026-06-01-zb-mom-ww-auth-shared-library.md).
|
||||
**Adopted across all three apps on 2026-06-02** (auth GAPS #1–#8) on each repo's `feat/adopt-zb-auth` branch —
|
||||
committed + reviewed, then **fast-forward-merged into the repo's local default (main/master) and PUSHED to origin
|
||||
(gitea) on 2026-06-03** (in sync; the `feat/*` branches kept locally as history). Cutover: shared `Auth.Ldap`,
|
||||
`Auth.ApiKeys` (ScadaBridge inbound fully re-architected to the keyId/Bearer model), `IGroupRoleMapper<TRole>` seam,
|
||||
`Transport`-enum config, canonical `ZbClaimTypes`/`ZbCookieDefaults`, unified dev base DN `dc=zb,dc=local`, and the
|
||||
canonical-six role vocabulary (with ScadaBridge's accepted auditor/admin SoD collapse). Consumer pins: **all four apps pin `0.3.0` as of 2026-09-03** (commits listed in the table row above, all pushed to origin); all four pinned `0.2.1` from 2026-08-13 before that. Prior history — all four pinned `0.1.5` (verified 2026-08-07): MxGateway via `e107019` 2026-07-18 (earlier bumped `0.1.2`→`0.1.4` 2026-07-09 for the G-2 API-key-expiry verifier enforcement — commit `197731a`; also required implementing the 0.1.3 `IApiKeyAdminStore.SetScopes/SetEnabled` members on a test fake), ScadaBridge `50d79ed1` + HistorianGateway `df211ee` 2026-07-18, OtOpcUa `1adaa2fc` 2026-07-20 (PR #484, family alignment from `0.1.1`). Per-repo detail in [`components/auth/GAPS.md`](components/auth/GAPS.md) +
|
||||
`docs/plans/2026-06-02-auth-audit-normalization*.md`.
|
||||
Build/test from `ZB.MOM.WW.Auth/`: `dotnet test` (the `ApiKeys.SqlServer` live suite is gated on `AUTH_SQLSERVER_CONNSTR`, skipped otherwise). Consumer matrix: OtOpcUa → Abstractions+Ldap+AspNetCore;
|
||||
MxAccessGateway & HistorianGateway → the original four; ScadaBridge → those four **plus `ApiKeys.SqlServer`** (ApiKeys not used by OtOpcUa).
|
||||
**`0.1.4` (2026-07-09, archreview G-2):** optional `ApiKeyRecord.ExpiresUtc` (nullable = never-expires), `ApiKeyFailure.KeyExpired`, `ApiKeyVerifier` expiry enforcement (inclusive, injected clock), a `CreateKeyAsync(expiresUtc)` overload, and a SQLite **schema v3** additive `expires_utc` column (idempotent `ALTER`; donor v2 DBs upgrade in place, no key invalidated). **All four Auth pkgs published to the Gitea feed at `0.1.4`** (2026-07-09). **HistorianGateway adopted the expiry surface at `0.1.4`** — `apikey create --expires` + dashboard Expires column (Task-10 done). **`0.1.5` (2026-07-18)** is `0.1.4` plus the `SQLitePCLRaw.lib.e_sqlite3 2.1.12` transitive security pin (GHSA-2m69-gcr7-jv3q), no API change; all four apps now pin it (see consumer pins above). **`0.2.0` (2026-08-10) — LDAP backup-DC sticky failover** (plan `docs/plans/2026-08-10-ldap-backup-dc-failover.md`, design `…-design.md`; **4 pkgs published to the feed, restore-verified**; family `Directory.Build.props` now `0.2.0`): one additive `LdapOptions.FallbackServers` list (`"host"` or `"host:port"`, empty default = failover off, single-endpoint behaviour byte-unchanged) makes `LdapAuthService` walk an ordered endpoint list (primary + fallbacks) on **system-side failures only** — connect/TLS, service-account bind, or search. A user-side answer from a healthy directory (`Success`/`UserNotFound`/`AmbiguousUser`/`BadCredentials`/`GroupLookupFailed`) is **final and never retried elsewhere** (re-asking a second DC about a rejected password would multiply one bad-password event into several against the account lockout policy). Endpoint preference is **sticky** — the last endpoint that gave *any* authoritative answer (success **or** a user-side failure) is tried first next time — so a primary outage costs its connect timeout once, not per login; an exhausted walk leaves the preference untouched. Fail-closed contract preserved per-endpoint; boot validation rejects a malformed `FallbackServers` entry naming the field; optional `ILogger<LdapAuthService>` emits a Warning per failed hop (endpoint + stage, no secrets) and an Information on sticky change; wired via `AddZbLdapAuth` (`GetService`, so logging is optional). Motivation: the ZB production directory is NAM AD with two DCs. Unit suite covers the walk/stickiness (14 tests) + env-gated live GLAuth failover test (`ZB_LDAP_IT=1`). **No `Abstractions` contract change beyond the new option; no `LdapAuthFailure` enum change.** App adoption (pin bump to `0.2.0` + authoring `FallbackServers`) is a follow-up.
|
||||
**`0.2.1` (2026-08-12) — AD continuation referrals no longer fail the search** (**4 pkgs published to the feed, restore-verified**; the assembly restored *from the feed* carries the fix and `0.2.0` does not — checked as a positive control, not by trusting the push). Active Directory answers **any search based at a domain root** with continuation references (to `CN=Configuration`, `DC=ForestDnsZones`, …). With referral following disabled — the default, and what we want, since chasing one would re-bind somewhere we never authenticated — Novell surfaces each by throwing `LdapReferralException` from `Next()`. That escaped `NovellLdapConnection.Search`, and `LdapAuthService` maps a failed search to `ServiceAccountBindFailed`, so **every login against `SearchBase = DC=nam,DC=zimmer,DC=com` failed system-side on both DCs while the directory, the credentials and the service account were all healthy** — ~546 ms, identical on each endpoint, so failover could not help either. Found by the new `tools/LdapProbe` (below) and live-proven fixed against the ZB NAM domain root. The search loop now skips referrals (`ReadEntries(ILdapSearchResults)`, extracted so it is unit-testable through the Novell seam without a live directory) and continues; a 1000-referral cap guards a client that throws without advancing, since that would spin forever inside a fail-closed auth path. **Non-referral faults still propagate** — swallowing those would turn "the directory is broken" into "no such user", which is the wrong answer to give about someone's credentials. 7 new tests (`NovellSearchReferralTests`), **6 verified red against the pre-fix code**, the seventh being the non-referral control that must stay green; suite 257 pass / 2 skip / 0 fail. **No `Abstractions` change; no config change.** **Consumer impact: an app on `0.1.5` cannot authenticate against an AD domain root at all**, so the pin bump is a functional fix, not housekeeping. **Closed 2026-08-13: all four apps adopted `0.2.1`** (see the table row for commits), and `FallbackServers` was live-gated the same day against the redundant local GLAuth pair (`infra/glauth-redundant/`).
|
||||
|
||||
**`0.3.0` (2026-09-02) — `ZB.MOM.WW.Auth.ApiKeys.SqlServer`, a SQL Server provider for the API-key store** (design `ScadaBridge/docs/plans/2026-09-02-inbound-api-key-shared-store-design.md`; **5 pkgs published to the feed**, `ApiKeys.SqlServer` search-verified from the ScadaBridge checkout; family `Directory.Build.props` now `0.3.0`). Motivation: `Auth.ApiKeys`' SQLite store is a per-node file, so on a two-node active/standby host (ScadaBridge central) a key minted on one node exists only there and a failover 401s every inbound caller (ScadaBridge deferred-work row 108). The new package implements `IApiKeyStore`/`IApiKeyAdminStore`/`IApiKeyAuditStore` over `Microsoft.Data.SqlClient` in a `zbauth` schema (v1: `schema_version`, `api_keys`, `api_key_audit`; all timestamps `DATETIMEOFFSET(7)`, tick-exact) with a concurrent-boot-safe migrator (Serializable tx, retry on 1205/2714/1913/2627 with **jittered** backoff — the Secrets replicator's flat `100*attempt` backoff exhausted five retries about once per sixty 8-way boots in the port's soak test, so the same latent defect is an open follow-up there) and a `SqliteApiKeyStoreImporter` that copies an existing SQLite file into the schema insert-where-absent (idempotent, so both nodes of a pair can import their own file and the result is the union). Wiring: `services.AddZbApiKeyAuthSqlServer(config, apiKeySection, sqlServerSection)` registers the SQL Server stores + migrator **before** delegating to `AddZbApiKeyAuth`, whose `TryAdd*` registrations then yield (same displacement mechanism as Secrets 0.5.0). One `Abstractions` addition, additive: `IApiKeyStoreMigrator`, which `SqliteAuthStoreMigrator` now implements and `ApiKeyAdminCommands` accepts via a new ctor (old `SqliteAuthStoreMigrator` ctor retained — the three `0.2.1` consumers compile unchanged). Store semantics are pinned to the SQLite contract tests re-run live (mark-used skips revoked keys, delete only removes revoked keys, rotate clears last-used/revoked, list newest-first). 92 tests in the new project, 351 across the solution, 0 failed. A DBA-run copy of the DDL ships at `docs/sql/zbauth-schema.sql`.
|
||||
|
||||
**`tools/LdapProbe`** (`ldapprobe`, not packable, outside the Auth solution) is the diagnostic that found the above: it drives the **real** `LdapAuthService` so its verdict is what the four apps produce, then separately queries the directory for what the library cannot report — full group DNs, **nested** groups via `LDAP_MATCHING_RULE_IN_CHAIN`, the **primary group** (which AD omits from `memberOf`, so it can never be a role group), group type (security vs distribution), decoded `userAccountControl`, referral counts, and the TLS chain with key sizes. `--list-groups '<pat>'` enumerates every group in the directory matching a pattern, which a membership dump structurally cannot do. Findings recorded in [`ldap_setup.md`](ldap_setup.md), including the **`UG_WW_*` Wonderware group convention (39 groups)** that partly overturns that doc's earlier "no group→role convention exists" claim — it was true of Ignition, false of the domain.
|
||||
@@ -5,10 +5,12 @@
|
||||
<Nullable>enable</Nullable>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
<LangVersion>latest</LangVersion>
|
||||
<!-- 0.2.1: NovellLdapConnection.Search now skips AD continuation referrals instead of
|
||||
letting them fail the search stage. Without it, a SearchBase at a domain root made
|
||||
every login fail as ServiceAccountBindFailed against a healthy directory. -->
|
||||
<Version>0.2.1</Version>
|
||||
<!-- 0.4.0: opt-in LdapOptions.BindMode = User — authenticate by binding as the signing-in user
|
||||
(UserBindTemplate) and read their own groups on the same connection; no service account.
|
||||
ServiceAccount stays the default, so existing consumers need no change. -->
|
||||
<!-- 0.4.1: Username canonicalised from the entry's UserNameAttribute; empty passwords refused
|
||||
in both bind modes (ServiceAccount-mode behaviour change); template docs prefer EXAMPLE\{0}. -->
|
||||
<Version>0.4.1</Version>
|
||||
<ManagePackageVersionsCentrally>true</ManagePackageVersionsCentrally>
|
||||
</PropertyGroup>
|
||||
|
||||
|
||||
@@ -15,6 +15,14 @@
|
||||
<!-- Transitive security pin: Microsoft.Data.Sqlite 10.0.7 pulls SQLitePCLRaw.lib.e_sqlite3 2.1.11,
|
||||
which carries advisory GHSA-2m69-gcr7-jv3q (NU1903). Pin the patched 2.1.12 native lib. -->
|
||||
<PackageVersion Include="SQLitePCLRaw.lib.e_sqlite3" Version="2.1.12" />
|
||||
<!-- Shared SQL-Server API-key store (ZB.MOM.WW.Auth.ApiKeys.SqlServer). 6.0.2 matches the
|
||||
family pin used by ZB.MOM.WW.Secrets' SQL-Server store, so a consumer taking both gets one
|
||||
SqlClient rather than a downgrade warning. -->
|
||||
<PackageVersion Include="Microsoft.Data.SqlClient" Version="6.0.2" />
|
||||
<!-- 9.0.4, not 9.0.0: Microsoft.Data.SqlClient 6.0.2 pulls System.Configuration.ConfigurationManager
|
||||
9.0.4, which requires ProtectedData >= 9.0.4. With transitive pinning on, the lower central
|
||||
version is a hard NU1109 downgrade error rather than a silent resolve. -->
|
||||
<PackageVersion Include="System.Security.Cryptography.ProtectedData" Version="9.0.4" />
|
||||
|
||||
<!-- Extensions -->
|
||||
<PackageVersion Include="Microsoft.Extensions.Options" Version="10.0.7" />
|
||||
|
||||
@@ -11,17 +11,18 @@ Authentication and authorisation libraries for the **ZB.MOM.WW SCADA family** (O
|
||||
| `ZB.MOM.WW.Auth.Abstractions` | Auth contracts, canonical role constants, and shared types (`LdapOptions`, `LdapAuthResult`, `ILdapAuthService`, `IApiKeyStore`). No runtime dependencies beyond the BCL. | — |
|
||||
| `ZB.MOM.WW.Auth.Ldap` | LDAP authentication service: bind-then-search-then-bind against GLAuth or Active Directory; RFC 4514-aware group extraction; fail-closed. | `Abstractions`, `Novell.Directory.Ldap.NETStandard` |
|
||||
| `ZB.MOM.WW.Auth.ApiKeys` | SQLite-backed API-key store with **pepper-keyed HMAC-SHA256** secret hashing, rotation, and audit log. DI wiring is `AddZbApiKeyAuth`; an opt-in `MigrationHostedService` runs schema migrations on startup. | `Abstractions`, `Microsoft.Data.Sqlite` |
|
||||
| `ZB.MOM.WW.Auth.ApiKeys.SqlServer` | SQL Server-backed API-key store sharing **one row set** across clustered hosts (e.g. a central pair), instead of each node keeping its own SQLite file. DI wiring is `AddZbApiKeyAuthSqlServer(config, apiKeySection, sqlServerSection)`, called **instead of** `AddZbApiKeyAuth` (it calls that internally). Also ships `SqliteApiKeyStoreImporter` — a one-shot per-node SQLite → shared store copy a host runs once at boot. | `Abstractions`, `ApiKeys`, `Microsoft.Data.SqlClient` |
|
||||
| `ZB.MOM.WW.Auth.AspNetCore` | ASP.NET Core wiring for the **LDAP** provider only: `AddZbLdapAuth` (binds + start-time-validates `LdapOptions`, registers `ILdapAuthService`), plus `ZbCookieDefaults.Apply` (hardened cookie helper the consumer calls itself) and `ZbClaimTypes` constants. It does **not** wire API keys or cookie middleware — API-key DI is `AddZbApiKeyAuth` in the `ApiKeys` package. | `Abstractions`, `Ldap`, `Microsoft.AspNetCore.App` |
|
||||
|
||||
---
|
||||
|
||||
## Consumer Matrix
|
||||
|
||||
| Consumer | Abstractions | Ldap | ApiKeys | AspNetCore |
|
||||
|---|:---:|:---:|:---:|:---:|
|
||||
| **OtOpcUa** | yes | yes | — | yes |
|
||||
| **MxAccessGateway** | yes | yes | yes | yes |
|
||||
| **ScadaBridge** | yes | yes | yes | yes |
|
||||
| Consumer | Abstractions | Ldap | ApiKeys | ApiKeys.SqlServer | AspNetCore |
|
||||
|---|:---:|:---:|:---:|:---:|:---:|
|
||||
| **OtOpcUa** | yes | yes | — | — | yes |
|
||||
| **MxAccessGateway** | yes | yes | yes | — | yes |
|
||||
| **ScadaBridge** | yes | yes | yes | yes (central pair, 2026-09-02) | yes |
|
||||
|
||||
`ApiKeys` is NOT used by OtOpcUa (that app authenticates human operators via LDAP + cookies only; machine-to-machine access is out of scope).
|
||||
|
||||
@@ -29,7 +30,41 @@ Authentication and authorisation libraries for the **ZB.MOM.WW SCADA family** (O
|
||||
|
||||
## Versioning
|
||||
|
||||
All four packages are versioned **lockstep** from `Directory.Build.props`. The current release is **0.1.4**. A single version bump in `Directory.Build.props` bumps all four packages simultaneously — consumers should reference the same version for all ZB.MOM.WW.Auth packages.
|
||||
All five packages are versioned **lockstep** from `Directory.Build.props`. The current release is **0.4.1**. A single version bump in `Directory.Build.props` bumps all five packages simultaneously — consumers should reference the same version for all ZB.MOM.WW.Auth packages.
|
||||
|
||||
### 0.4.1 — LDAP review fixes
|
||||
|
||||
- **Canonical username (both modes):** the search now also requests `UserNameAttribute`, and `LdapAuthResult.Username` is the entry's first value of it (the directory's spelling), so `JDoe` and `jdoe` sign in as one identity in consumers' audit columns. Falls back to the normalised typed name when the entry has no such value.
|
||||
- **Behaviour change for `ServiceAccount`-mode consumers:** an empty (or null) password is now refused by the library as `BadCredentials` before any network call, in both bind modes. Previously only `User` mode refused it; in `ServiceAccount` mode the re-bind as the user's DN with an empty password is an unauthenticated bind that AD reports as success, so a consumer without its own edge check admitted anyone.
|
||||
- **Template guidance:** prefer the down-level `UserBindTemplate` `EXAMPLE\{0}` — it maps exactly to the `sAMAccountName` the group search uses. A UPN template (`{0}@example.com`) is only correct where every account's UPN prefix equals its `sAMAccountName`; AD resolves UPN binds against the explicit `userPrincipalName`, so a mismatch can fail sign-in or bind a different account than the one whose groups are read.
|
||||
- No `Abstractions` contract change.
|
||||
|
||||
### 0.4.0 — LDAP sign-in by user bind (opt-in)
|
||||
|
||||
- **`LdapOptions.BindMode`** (`ServiceAccount` default | `User`), **`UserBindTemplate`** (`{0}` = the login name, e.g. `EXAMPLE\{0}` — preferred, see 0.4.1 — or `{0}@example.com`) and **`AcceptDomainQualifiedNames`** (default `true`).
|
||||
- In `User` mode `LdapAuthService` binds as the user; success authenticates them and the same connection reads their entry (`UserNameAttribute`, `DisplayNameAttribute`, `GroupAttribute`). No `ServiceAccountDn` / `ServiceAccountPassword` is needed or validated.
|
||||
- Failure mapping: empty password or an unusable name → `BadCredentials` with no network call (AD treats an empty-password simple bind as an unauthenticated success). Bind result 49 → `BadCredentials`, final — never retried on a fallback DC, so one wrong password is one lockout strike. Any other bind error, a connect failure or a search failure → the system-side bucket (`ServiceAccountBindFailed`) and the next endpoint. Zero/several entries or no groups after a good bind → `GroupLookupFailed`.
|
||||
- `DOMAIN\name` / `name@suffix` are reduced to `name` (or refused when `AcceptDomainQualifiedNames` is `false`), so the result's `Username` has one spelling (0.4.1 also canonicalises its case from the entry).
|
||||
- No `Abstractions` contract change beyond the new enum and option properties; `ServiceAccount` mode is unchanged.
|
||||
- `UserBindTemplate` output is used verbatim as the bind name, so use a down-level (`EXAMPLE\{0}`, preferred) or UPN (`{0}@example.com`, only where every UPN prefix equals the `sAMAccountName`) template; a DN template is not escaped.
|
||||
|
||||
### 0.3.0 — SQL Server API-key store (ScadaBridge central pair)
|
||||
|
||||
- **`IApiKeyStoreMigrator`** added to `Abstractions`: creates or upgrades an API-key store's schema; idempotent and safe to run concurrently from several nodes. `SqliteAuthStoreMigrator` implements it unchanged. `AddZbApiKeyAuth` registers the SQLite implementation via `TryAdd`, so a provider that registers its own migrator first (see below) wins — that ordering is the whole mechanism by which a different backend is swapped in.
|
||||
- **New package `ZB.MOM.WW.Auth.ApiKeys.SqlServer`** + `AddZbApiKeyAuthSqlServer(config, apiKeySection, sqlServerSection)`, called instead of `AddZbApiKeyAuth` (it calls it internally after registering the SQL Server stores and migrator first).
|
||||
- **`zbauth` schema v1** (three tables: `schema_version`, `api_keys`, `api_key_audit`; `DATETIMEOFFSET(7)` timestamps). `SqlServerApiKeyStoreMigrator` is concurrent-boot safe — a central pair's two nodes both migrate at every boot — with retry on SQL error numbers 1205/2714/1913/2627 and jittered backoff (measured: the flat backoff it replaced exhausted its retries roughly 1 in 60 eight-way boots). DBA-facing DDL: `docs/sql/zbauth-schema.sql`.
|
||||
- **`SqliteApiKeyStoreImporter`** — idempotent insert-where-not-exists copy of an existing node's per-node SQLite keys and audit rows into the shared store; a row already present in SQL Server is never modified, so running it from both nodes of a pair converges on the union of their files.
|
||||
- **`ApiKeyAdminCommands`** gained a ctor taking `IApiKeyStoreMigrator`; the existing ctor taking the concrete `SqliteAuthStoreMigrator` is retained (MxAccessGateway, HistorianGateway and ScadaBridge all construct it that way).
|
||||
- SQLite remains the default store; **existing consumers need no code change to move to 0.3.0**.
|
||||
- Live tests for the SQL Server package are gated on `AUTH_SQLSERVER_CONNSTR` and skip (not fail) when it is unset.
|
||||
|
||||
### 0.2.1 — LDAP AD continuation-referral fix
|
||||
|
||||
- `NovellLdapConnection.Search` now skips Active Directory continuation referrals (`LdapReferralException` from `Next()`) instead of letting one fail the whole search — a `SearchBase` at a domain root made every login fail as `ServiceAccountBindFailed` against a healthy directory. Non-referral faults still propagate untouched.
|
||||
|
||||
### 0.2.0 — version bump only
|
||||
|
||||
- No functional change; bumped the lockstep family version ahead of the 0.2.1 LDAP fix.
|
||||
|
||||
### 0.1.4 — API-key expiry (archreview G-2)
|
||||
|
||||
|
||||
@@ -2,11 +2,13 @@
|
||||
<Folder Name="/src/">
|
||||
<Project Path="src/ZB.MOM.WW.Auth.Abstractions/ZB.MOM.WW.Auth.Abstractions.csproj" />
|
||||
<Project Path="src/ZB.MOM.WW.Auth.ApiKeys/ZB.MOM.WW.Auth.ApiKeys.csproj" />
|
||||
<Project Path="src/ZB.MOM.WW.Auth.ApiKeys.SqlServer/ZB.MOM.WW.Auth.ApiKeys.SqlServer.csproj" />
|
||||
<Project Path="src/ZB.MOM.WW.Auth.AspNetCore/ZB.MOM.WW.Auth.AspNetCore.csproj" />
|
||||
<Project Path="src/ZB.MOM.WW.Auth.Ldap/ZB.MOM.WW.Auth.Ldap.csproj" />
|
||||
</Folder>
|
||||
<Folder Name="/tests/">
|
||||
<Project Path="tests/ZB.MOM.WW.Auth.ApiKeys.Tests/ZB.MOM.WW.Auth.ApiKeys.Tests.csproj" />
|
||||
<Project Path="tests/ZB.MOM.WW.Auth.ApiKeys.SqlServer.Tests/ZB.MOM.WW.Auth.ApiKeys.SqlServer.Tests.csproj" />
|
||||
<Project Path="tests/ZB.MOM.WW.Auth.AspNetCore.Tests/ZB.MOM.WW.Auth.AspNetCore.Tests.csproj" />
|
||||
<Project Path="tests/ZB.MOM.WW.Auth.Ldap.Tests/ZB.MOM.WW.Auth.Ldap.Tests.csproj" />
|
||||
</Folder>
|
||||
|
||||
@@ -6,4 +6,10 @@
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
dotnet pack -c Release -o ./artifacts
|
||||
# cd to the repo root and name the solution + an absolute output path: `dotnet pack` with
|
||||
# neither a project argument nor a fixed working directory packs whatever solution happens
|
||||
# to sit in the CALLER's directory (the bug behind the LocalDb feed incident, 0038d46).
|
||||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "$REPO_ROOT"
|
||||
|
||||
dotnet pack ZB.MOM.WW.Auth.slnx -c Release -o "$REPO_ROOT/artifacts"
|
||||
|
||||
@@ -16,9 +16,20 @@ set -euo pipefail
|
||||
: "${GITEA_NUGET_SOURCE:?set GITEA_NUGET_SOURCE to your Gitea NuGet feed URL}"
|
||||
: "${GITEA_NUGET_KEY:?set GITEA_NUGET_KEY to your Gitea access token}"
|
||||
|
||||
dotnet pack -c Release -o ./artifacts
|
||||
# Named solution, named output, and a cd to the repo root, all for one reason: `dotnet pack`
|
||||
# with neither a project argument nor a fixed working directory packs whatever solution
|
||||
# happens to sit in the CALLER's directory — and then pushes it to a live feed under this
|
||||
# script's name. Invoked by absolute path from another repo's checkout, the relative form
|
||||
# packed and published 25 unrelated packages (LocalDb's copy of this script, fixed in
|
||||
# 0038d46; recovered with `dotnet nuget delete`, but only because the token happened to
|
||||
# carry delete rights). A publish script must be unambiguous about WHAT it publishes no
|
||||
# matter where it is run.
|
||||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "$REPO_ROOT"
|
||||
|
||||
dotnet nuget push "./artifacts/*.nupkg" \
|
||||
dotnet pack ZB.MOM.WW.Auth.slnx -c Release -o "$REPO_ROOT/artifacts"
|
||||
|
||||
dotnet nuget push "$REPO_ROOT/artifacts/*.nupkg" \
|
||||
--source "$GITEA_NUGET_SOURCE" \
|
||||
--api-key "$GITEA_NUGET_KEY" \
|
||||
--skip-duplicate
|
||||
|
||||
@@ -0,0 +1,69 @@
|
||||
-- ZB.MOM.WW.Auth.ApiKeys.SqlServer — shared API-key schema, version 1.
|
||||
--
|
||||
-- For DBAs whose application login lacks DDL rights. The host normally provisions this itself at
|
||||
-- startup (SqlServerApiKeyStoreMigrator, which runs the identical statements and is safe to run
|
||||
-- concurrently from every node); where the app login may only read and write rows, run this script
|
||||
-- once against the target database instead. The migrator then finds the version row already at the
|
||||
-- version it supports and does nothing further beyond re-stamping applied_utc.
|
||||
--
|
||||
-- The schema name is hard-coded to zbauth here — the default of SqlServerApiKeyOptions.Schema. If a
|
||||
-- deployment overrides that option, substitute the name in every bracketed identifier below.
|
||||
--
|
||||
-- Kept byte-for-byte consistent with SqlServerAuthSchema.CreateSchemaDdl apart from that
|
||||
-- substitution and the literal version row at the end; change both together.
|
||||
|
||||
IF NOT EXISTS (SELECT 1 FROM sys.schemas WHERE name = N'zbauth')
|
||||
EXEC(N'CREATE SCHEMA [zbauth]');
|
||||
|
||||
IF OBJECT_ID(N'[zbauth].[schema_version]', N'U') IS NULL
|
||||
CREATE TABLE [zbauth].[schema_version] (
|
||||
id INT NOT NULL CONSTRAINT PK_zbauth_schema_version PRIMARY KEY CHECK (id = 1),
|
||||
version INT NOT NULL,
|
||||
applied_utc DATETIMEOFFSET(7) NOT NULL
|
||||
);
|
||||
|
||||
IF OBJECT_ID(N'[zbauth].[api_keys]', N'U') IS NULL
|
||||
CREATE TABLE [zbauth].[api_keys] (
|
||||
key_id NVARCHAR(64) NOT NULL CONSTRAINT PK_zbauth_api_keys PRIMARY KEY,
|
||||
key_prefix NVARCHAR(16) NOT NULL,
|
||||
secret_hash VARBINARY(64) NOT NULL,
|
||||
display_name NVARCHAR(256) NOT NULL,
|
||||
scopes NVARCHAR(MAX) NOT NULL,
|
||||
constraints NVARCHAR(MAX) NULL,
|
||||
created_utc DATETIMEOFFSET(7) NOT NULL,
|
||||
last_used_utc DATETIMEOFFSET(7) NULL,
|
||||
revoked_utc DATETIMEOFFSET(7) NULL,
|
||||
expires_utc DATETIMEOFFSET(7) NULL
|
||||
);
|
||||
|
||||
IF OBJECT_ID(N'[zbauth].[api_key_audit]', N'U') IS NULL
|
||||
CREATE TABLE [zbauth].[api_key_audit] (
|
||||
audit_id BIGINT IDENTITY(1,1) NOT NULL CONSTRAINT PK_zbauth_api_key_audit PRIMARY KEY,
|
||||
key_id NVARCHAR(64) NULL,
|
||||
event_type NVARCHAR(64) NOT NULL,
|
||||
remote_address NVARCHAR(64) NULL,
|
||||
created_utc DATETIMEOFFSET(7) NOT NULL,
|
||||
details NVARCHAR(MAX) NULL
|
||||
);
|
||||
|
||||
IF NOT EXISTS (
|
||||
SELECT 1 FROM sys.indexes
|
||||
WHERE name = N'ix_api_keys_revoked_utc'
|
||||
AND object_id = OBJECT_ID(N'[zbauth].[api_keys]'))
|
||||
CREATE INDEX ix_api_keys_revoked_utc
|
||||
ON [zbauth].[api_keys] (revoked_utc);
|
||||
|
||||
IF NOT EXISTS (
|
||||
SELECT 1 FROM sys.indexes
|
||||
WHERE name = N'ix_api_key_audit_key_id_created_utc'
|
||||
AND object_id = OBJECT_ID(N'[zbauth].[api_key_audit]'))
|
||||
CREATE INDEX ix_api_key_audit_key_id_created_utc
|
||||
ON [zbauth].[api_key_audit] (key_id, created_utc);
|
||||
|
||||
MERGE [zbauth].[schema_version] AS target
|
||||
USING (SELECT 1 AS id, 1 AS version, SYSDATETIMEOFFSET() AS applied_utc) AS source
|
||||
ON target.id = source.id
|
||||
WHEN MATCHED THEN
|
||||
UPDATE SET version = source.version, applied_utc = source.applied_utc
|
||||
WHEN NOT MATCHED THEN
|
||||
INSERT (id, version, applied_utc) VALUES (1, source.version, source.applied_utc);
|
||||
@@ -78,3 +78,16 @@ public interface IApiKeyAuditStore
|
||||
Task AppendAsync(ApiKeyAuditEntry entry, CancellationToken ct);
|
||||
Task<IReadOnlyList<ApiKeyAuditEntry>> ListRecentAsync(int limit, CancellationToken ct);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Creates or upgrades the API-key store schema to the version this build supports. Each store
|
||||
/// provider (SQLite, SQL Server) supplies one; hosts run it at startup and before admin commands.
|
||||
/// Implementations must be idempotent and safe to run concurrently from several nodes.
|
||||
/// </summary>
|
||||
public interface IApiKeyStoreMigrator
|
||||
{
|
||||
/// <summary>Migrates the store schema.</summary>
|
||||
/// <param name="ct">A token to cancel the operation.</param>
|
||||
/// <returns>A task that completes when the schema is current.</returns>
|
||||
Task MigrateAsync(CancellationToken ct);
|
||||
}
|
||||
|
||||
@@ -4,6 +4,20 @@ namespace ZB.MOM.WW.Auth.Abstractions.Ldap;
|
||||
|
||||
public enum LdapTransport { Ldaps, StartTls, None }
|
||||
|
||||
/// <summary>How <see cref="LdapOptions"/> authenticates a sign-in.</summary>
|
||||
public enum LdapBindMode
|
||||
{
|
||||
/// <summary>Bind as the service account, search for the user, re-bind as the user (0.3.0 behaviour).</summary>
|
||||
ServiceAccount,
|
||||
|
||||
/// <summary>
|
||||
/// Bind as the signing-in user (identity built from <see cref="LdapOptions.UserBindTemplate"/>);
|
||||
/// a successful bind authenticates them, and the same connection reads their own entry for groups.
|
||||
/// No service account is used.
|
||||
/// </summary>
|
||||
User,
|
||||
}
|
||||
|
||||
public sealed record LdapOptions
|
||||
{
|
||||
public bool Enabled { get; init; } = true;
|
||||
@@ -19,6 +33,33 @@ public sealed record LdapOptions
|
||||
public string GroupAttribute { get; init; } = "memberOf";
|
||||
public int ConnectionTimeoutMs { get; init; } = 10_000;
|
||||
|
||||
/// <summary>
|
||||
/// Selects the authentication flow. <see cref="LdapBindMode.ServiceAccount"/> (the default) is the
|
||||
/// 0.3.0 bind-then-search-then-bind flow; <see cref="LdapBindMode.User"/> binds as the user and needs
|
||||
/// no <see cref="ServiceAccountDn"/> / <see cref="ServiceAccountPassword"/>.
|
||||
/// </summary>
|
||||
public LdapBindMode BindMode { get; init; } = LdapBindMode.ServiceAccount;
|
||||
|
||||
/// <summary>
|
||||
/// <see cref="LdapBindMode.User"/> only: the simple-bind identity, with <c>{0}</c> replaced by the
|
||||
/// normalised login name. Must contain <c>{0}</c> exactly once.
|
||||
/// <para>
|
||||
/// Prefer the down-level form <c>EXAMPLE\{0}</c>: it maps exactly to the <c>sAMAccountName</c> the
|
||||
/// group search uses, so the account that binds is the account whose groups are read. A UPN
|
||||
/// template (<c>{0}@example.com</c>) is only correct where every account's UPN prefix equals its
|
||||
/// <c>sAMAccountName</c> — AD resolves a UPN bind against the explicit <c>userPrincipalName</c>, so
|
||||
/// a mismatch can fail sign-in or bind a different account than the one whose groups are read.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public string UserBindTemplate { get; init; } = "";
|
||||
|
||||
/// <summary>
|
||||
/// <see cref="LdapBindMode.User"/> only: when <see langword="true"/> (default) a login typed as
|
||||
/// <c>DOMAIN\name</c> or <c>name@suffix</c> is reduced to <c>name</c>; when <see langword="false"/>
|
||||
/// such input is refused as bad credentials without contacting the directory.
|
||||
/// </summary>
|
||||
public bool AcceptDomainQualifiedNames { get; init; } = true;
|
||||
|
||||
/// <summary>
|
||||
/// Ordered fallback LDAP endpoints tried when the primary (and any earlier fallback)
|
||||
/// fails with a system-side error (connect/TLS, service-account bind, or search).
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
using Microsoft.Data.SqlClient;
|
||||
|
||||
namespace ZB.MOM.WW.Auth.ApiKeys.SqlServer;
|
||||
|
||||
/// <summary>
|
||||
/// Opens connections to the shared SQL Server API-key database and carries the settings every
|
||||
/// command needs (schema name, command timeout).
|
||||
/// </summary>
|
||||
/// <param name="options">The SQL Server API-key store options; validated in the constructor.</param>
|
||||
/// <exception cref="InvalidOperationException"><paramref name="options"/> is not usable.</exception>
|
||||
public sealed class AuthSqlServerConnectionFactory(SqlServerApiKeyOptions options)
|
||||
{
|
||||
private readonly SqlServerApiKeyOptions _options = Validate(options);
|
||||
|
||||
private static SqlServerApiKeyOptions Validate(SqlServerApiKeyOptions options)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
options.Validate();
|
||||
return options;
|
||||
}
|
||||
|
||||
/// <summary>The schema that owns the three API-key tables.</summary>
|
||||
public string Schema => _options.Schema;
|
||||
|
||||
/// <summary>Command timeout, in whole seconds, for every command issued against this store.</summary>
|
||||
public int CommandTimeoutSeconds => Math.Max(1, (int)_options.CommandTimeout.TotalSeconds);
|
||||
|
||||
/// <summary>Opens a connection to the shared API-key database.</summary>
|
||||
/// <param name="cancellationToken">A token to cancel the connect.</param>
|
||||
/// <returns>An opened connection. The caller owns disposal.</returns>
|
||||
public async Task<SqlConnection> OpenConnectionAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
var connection = new SqlConnection(_options.ConnectionString);
|
||||
try
|
||||
{
|
||||
await connection.OpenAsync(cancellationToken).ConfigureAwait(false);
|
||||
return connection;
|
||||
}
|
||||
catch
|
||||
{
|
||||
await connection.DisposeAsync().ConfigureAwait(false);
|
||||
throw;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Creates a command on <paramref name="connection"/> with the configured timeout applied.
|
||||
/// </summary>
|
||||
/// <param name="connection">The open connection.</param>
|
||||
/// <param name="commandText">The command text.</param>
|
||||
/// <param name="transaction">The ambient transaction, if any.</param>
|
||||
/// <returns>The configured command. The caller owns disposal.</returns>
|
||||
public SqlCommand CreateCommand(
|
||||
SqlConnection connection, string commandText, SqlTransaction? transaction = null)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(connection);
|
||||
|
||||
SqlCommand command = connection.CreateCommand();
|
||||
command.CommandText = commandText;
|
||||
command.CommandTimeout = CommandTimeoutSeconds;
|
||||
command.Transaction = transaction;
|
||||
return command;
|
||||
}
|
||||
}
|
||||
+96
@@ -0,0 +1,96 @@
|
||||
using Microsoft.Extensions.Configuration;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.DependencyInjection.Extensions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.Auth.Abstractions.ApiKeys;
|
||||
using ZB.MOM.WW.Auth.ApiKeys.DependencyInjection;
|
||||
|
||||
namespace ZB.MOM.WW.Auth.ApiKeys.SqlServer.DependencyInjection;
|
||||
|
||||
/// <summary>
|
||||
/// Dependency-injection entry point that swaps the API-key store backend from the SQLite default
|
||||
/// to a shared SQL Server database — one row set read and written by every node of a cluster (e.g.
|
||||
/// a ScadaBridge central pair), instead of the per-node file <see cref="AddZbApiKeyAuth"/> alone
|
||||
/// would give each node.
|
||||
/// </summary>
|
||||
public static class SqlServerApiKeyServiceCollectionExtensions
|
||||
{
|
||||
/// <summary>
|
||||
/// Registers the SQL Server-backed <see cref="IApiKeyStore"/>, <see cref="IApiKeyAdminStore"/>,
|
||||
/// <see cref="IApiKeyAuditStore"/> and <see cref="IApiKeyStoreMigrator"/>, then calls
|
||||
/// <see cref="ApiKeyServiceCollectionExtensions.AddZbApiKeyAuth"/> to complete the registration
|
||||
/// (verifier, pepper provider, migration hosted service).
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>Call this INSTEAD of <see cref="ApiKeyServiceCollectionExtensions.AddZbApiKeyAuth"/></b>,
|
||||
/// never both — this method calls it internally. The SQL Server stores and migrator are
|
||||
/// registered here, BEFORE that inner call, precisely because
|
||||
/// <see cref="ApiKeyServiceCollectionExtensions.AddZbApiKeyAuth"/> registers everything with
|
||||
/// <c>TryAdd*</c>: whichever implementation of <see cref="IApiKeyStore"/> (etc.) is registered
|
||||
/// first wins, and a later <c>TryAdd</c> for the same service is a no-op. That ordering is the
|
||||
/// entire mechanism by which this method displaces the SQLite default.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <see cref="ApiKeyServiceCollectionExtensions.AddZbApiKeyAuth"/> still registers a SQLite
|
||||
/// connection factory from <c>ApiKeyOptions.SqlitePath</c> even when SQL Server wins every
|
||||
/// store slot. That is harmless — the SQLite factory is lazy and the connection it would open
|
||||
/// is never opened when nothing resolves it — and it is actually useful: it keeps the one path
|
||||
/// a node's pre-existing per-node SQLite file lives at in a single option, which is exactly the
|
||||
/// source path <see cref="SqliteApiKeyStoreImporter"/> needs for the one-shot copy into the
|
||||
/// shared store.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <see cref="SqliteApiKeyStoreImporter"/> is registered here so a host can resolve it and run
|
||||
/// it once at boot (e.g. from a startup task or its own hosted service) to copy an existing
|
||||
/// node's per-node keys and audit rows into the now-shared store; it performs no work by being
|
||||
/// merely registered.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>The connection string must be a literal value or an environment-variable override — never
|
||||
/// a <c>${secret:}</c> reference</b> on a host whose secret store is itself SQL-backed: resolving
|
||||
/// that reference would require a working secret store, which on such a host requires this very
|
||||
/// database to already be reachable, an unbreakable circular dependency at boot.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
/// <param name="services">The service collection to add to.</param>
|
||||
/// <param name="config">The application configuration.</param>
|
||||
/// <param name="apiKeySectionPath">Path of the configuration section holding the core API-key options.</param>
|
||||
/// <param name="sqlServerSectionPath">Path of the configuration section holding <see cref="SqlServerApiKeyOptions"/>.</param>
|
||||
/// <returns>The same <paramref name="services"/> instance, for chaining.</returns>
|
||||
public static IServiceCollection AddZbApiKeyAuthSqlServer(
|
||||
this IServiceCollection services,
|
||||
IConfiguration config,
|
||||
string apiKeySectionPath,
|
||||
string sqlServerSectionPath)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(services);
|
||||
ArgumentNullException.ThrowIfNull(config);
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(apiKeySectionPath);
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(sqlServerSectionPath);
|
||||
|
||||
services.Configure<SqlServerApiKeyOptions>(config.GetSection(sqlServerSectionPath));
|
||||
|
||||
services.TryAddSingleton(sp =>
|
||||
new AuthSqlServerConnectionFactory(sp.GetRequiredService<IOptions<SqlServerApiKeyOptions>>().Value));
|
||||
|
||||
// Registered BEFORE AddZbApiKeyAuth so its TryAdd calls find these already present and
|
||||
// leave them alone — that is the whole mechanism by which the SQL Server store displaces
|
||||
// SQLite (see the class remarks).
|
||||
services.TryAddSingleton<IApiKeyStore>(sp =>
|
||||
new SqlServerApiKeyStore(sp.GetRequiredService<AuthSqlServerConnectionFactory>()));
|
||||
services.TryAddSingleton<IApiKeyAdminStore>(sp =>
|
||||
new SqlServerApiKeyAdminStore(sp.GetRequiredService<AuthSqlServerConnectionFactory>()));
|
||||
services.TryAddSingleton<IApiKeyAuditStore>(sp =>
|
||||
new SqlServerApiKeyAuditStore(sp.GetRequiredService<AuthSqlServerConnectionFactory>()));
|
||||
|
||||
services.TryAddSingleton<SqlServerApiKeyStoreMigrator>();
|
||||
services.TryAddSingleton<IApiKeyStoreMigrator>(sp => sp.GetRequiredService<SqlServerApiKeyStoreMigrator>());
|
||||
|
||||
// Registered so a host can resolve it and run the one-shot per-node SQLite -> shared store
|
||||
// copy at boot; registering it here does no work on its own.
|
||||
services.TryAddSingleton<SqliteApiKeyStoreImporter>();
|
||||
|
||||
return services.AddZbApiKeyAuth(config, apiKeySectionPath);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,247 @@
|
||||
using System.Data;
|
||||
using Microsoft.Data.SqlClient;
|
||||
using ZB.MOM.WW.Auth.Abstractions.ApiKeys;
|
||||
using ZB.MOM.WW.Auth.ApiKeys.Sqlite;
|
||||
|
||||
namespace ZB.MOM.WW.Auth.ApiKeys.SqlServer;
|
||||
|
||||
/// <summary>
|
||||
/// SQL Server-backed administration store for API keys (create, revoke, rotate, delete,
|
||||
/// set-scopes, enable/disable, list).
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Semantics are a deliberate one-for-one port of <c>SqliteApiKeyAdminStore</c> — same WHERE and SET
|
||||
/// clauses, same rows-affected return contract — so an admin command behaves identically whichever
|
||||
/// provider a host composed. The point of this provider is only WHERE the rows live: one shared row
|
||||
/// set for a clustered pair instead of a file per node, so a revoke on either node is a revoke
|
||||
/// everywhere.
|
||||
/// </remarks>
|
||||
/// <param name="connectionFactory">Factory for connections to the shared API-key database.</param>
|
||||
public sealed class SqlServerApiKeyAdminStore(AuthSqlServerConnectionFactory connectionFactory) : IApiKeyAdminStore
|
||||
{
|
||||
/// <inheritdoc />
|
||||
/// <remarks>
|
||||
/// A plain INSERT: a duplicate <c>key_id</c> surfaces as a primary-key
|
||||
/// <see cref="SqlException"/> rather than silently overwriting an existing key (the SQLite store
|
||||
/// raises its own <c>SqliteException</c> in the same situation).
|
||||
/// </remarks>
|
||||
public async Task CreateAsync(ApiKeyRecord record, CancellationToken ct)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(record);
|
||||
|
||||
string schema = connectionFactory.Schema;
|
||||
|
||||
await using SqlConnection connection =
|
||||
await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false);
|
||||
|
||||
await using SqlCommand command = connectionFactory.CreateCommand(connection, $"""
|
||||
INSERT INTO [{schema}].[{SqlServerAuthSchema.ApiKeysTable}] (
|
||||
key_id, key_prefix, secret_hash, display_name, scopes,
|
||||
constraints, created_utc, last_used_utc, revoked_utc, expires_utc)
|
||||
VALUES (
|
||||
@key_id, @key_prefix, @secret_hash, @display_name, @scopes,
|
||||
@constraints, @created_utc, @last_used_utc, @revoked_utc, @expires_utc);
|
||||
""");
|
||||
command.Parameters.Add("@key_id", SqlDbType.NVarChar, 64).Value = record.KeyId;
|
||||
command.Parameters.Add("@key_prefix", SqlDbType.NVarChar, 16).Value = record.KeyPrefix;
|
||||
command.Parameters.Add("@secret_hash", SqlDbType.VarBinary, 64).Value = record.SecretHash;
|
||||
command.Parameters.Add("@display_name", SqlDbType.NVarChar, 256).Value = record.DisplayName;
|
||||
command.Parameters.Add("@scopes", SqlDbType.NVarChar, -1).Value = ScopeSerializer.Serialize(record.Scopes);
|
||||
command.Parameters.Add("@constraints", SqlDbType.NVarChar, -1).Value = (object?)record.ConstraintsJson ?? DBNull.Value;
|
||||
command.Parameters.Add("@created_utc", SqlDbType.DateTimeOffset).Value = record.CreatedUtc;
|
||||
command.Parameters.Add("@last_used_utc", SqlDbType.DateTimeOffset).Value = (object?)record.LastUsedUtc ?? DBNull.Value;
|
||||
command.Parameters.Add("@revoked_utc", SqlDbType.DateTimeOffset).Value = (object?)record.RevokedUtc ?? DBNull.Value;
|
||||
command.Parameters.Add("@expires_utc", SqlDbType.DateTimeOffset).Value = (object?)record.ExpiresUtc ?? DBNull.Value;
|
||||
|
||||
await command.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
/// <remarks>
|
||||
/// Guarded on <c>revoked_utc IS NULL</c>, so the returned flag means "this call revoked it" —
|
||||
/// a second revoke of the same key returns <see langword="false"/> and preserves the original
|
||||
/// revocation instant.
|
||||
/// </remarks>
|
||||
public async Task<bool> RevokeAsync(string keyId, DateTimeOffset whenUtc, CancellationToken ct)
|
||||
{
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(keyId);
|
||||
|
||||
string schema = connectionFactory.Schema;
|
||||
|
||||
await using SqlConnection connection =
|
||||
await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false);
|
||||
|
||||
await using SqlCommand command = connectionFactory.CreateCommand(connection, $"""
|
||||
UPDATE [{schema}].[{SqlServerAuthSchema.ApiKeysTable}]
|
||||
SET revoked_utc = @revoked_utc
|
||||
WHERE key_id = @key_id AND revoked_utc IS NULL;
|
||||
""");
|
||||
command.Parameters.Add("@key_id", SqlDbType.NVarChar, 64).Value = keyId;
|
||||
command.Parameters.Add("@revoked_utc", SqlDbType.DateTimeOffset).Value = whenUtc;
|
||||
|
||||
int rows = await command.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
|
||||
|
||||
return rows > 0;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
/// <remarks>
|
||||
/// Rotating also clears <c>last_used_utc</c> and <c>revoked_utc</c> — a rotated key is a fresh,
|
||||
/// active credential — while leaving <c>expires_utc</c> alone, so an absolute expiry survives a
|
||||
/// rotation.
|
||||
/// </remarks>
|
||||
public async Task<bool> RotateAsync(string keyId, byte[] newSecretHash, CancellationToken ct)
|
||||
{
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(keyId);
|
||||
ArgumentNullException.ThrowIfNull(newSecretHash);
|
||||
|
||||
string schema = connectionFactory.Schema;
|
||||
|
||||
await using SqlConnection connection =
|
||||
await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false);
|
||||
|
||||
await using SqlCommand command = connectionFactory.CreateCommand(connection, $"""
|
||||
UPDATE [{schema}].[{SqlServerAuthSchema.ApiKeysTable}]
|
||||
SET secret_hash = @secret_hash,
|
||||
last_used_utc = NULL,
|
||||
revoked_utc = NULL
|
||||
WHERE key_id = @key_id;
|
||||
""");
|
||||
command.Parameters.Add("@key_id", SqlDbType.NVarChar, 64).Value = keyId;
|
||||
command.Parameters.Add("@secret_hash", SqlDbType.VarBinary, 64).Value = newSecretHash;
|
||||
|
||||
int rows = await command.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
|
||||
|
||||
return rows > 0;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task<bool> SetScopesAsync(string keyId, IReadOnlySet<string> scopes, CancellationToken ct)
|
||||
{
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(keyId);
|
||||
ArgumentNullException.ThrowIfNull(scopes);
|
||||
|
||||
string schema = connectionFactory.Schema;
|
||||
|
||||
await using SqlConnection connection =
|
||||
await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false);
|
||||
|
||||
await using SqlCommand command = connectionFactory.CreateCommand(connection, $"""
|
||||
UPDATE [{schema}].[{SqlServerAuthSchema.ApiKeysTable}]
|
||||
SET scopes = @scopes
|
||||
WHERE key_id = @key_id;
|
||||
""");
|
||||
command.Parameters.Add("@key_id", SqlDbType.NVarChar, 64).Value = keyId;
|
||||
command.Parameters.Add("@scopes", SqlDbType.NVarChar, -1).Value = ScopeSerializer.Serialize(scopes);
|
||||
|
||||
int rows = await command.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
|
||||
|
||||
return rows > 0;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task<bool> SetEnabledAsync(string keyId, bool enabled, DateTimeOffset whenUtc, CancellationToken ct)
|
||||
{
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(keyId);
|
||||
|
||||
string schema = connectionFactory.Schema;
|
||||
|
||||
await using SqlConnection connection =
|
||||
await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false);
|
||||
|
||||
// Reversible toggle: NO `revoked_utc IS NULL` guard (unlike RevokeAsync), so it works
|
||||
// regardless of current state. Deliberately leaves secret_hash and last_used_utc untouched
|
||||
// — that is what distinguishes re-enable from RotateAsync.
|
||||
SqlCommand command;
|
||||
|
||||
if (enabled)
|
||||
{
|
||||
command = connectionFactory.CreateCommand(connection, $"""
|
||||
UPDATE [{schema}].[{SqlServerAuthSchema.ApiKeysTable}]
|
||||
SET revoked_utc = NULL
|
||||
WHERE key_id = @key_id;
|
||||
""");
|
||||
command.Parameters.Add("@key_id", SqlDbType.NVarChar, 64).Value = keyId;
|
||||
}
|
||||
else
|
||||
{
|
||||
command = connectionFactory.CreateCommand(connection, $"""
|
||||
UPDATE [{schema}].[{SqlServerAuthSchema.ApiKeysTable}]
|
||||
SET revoked_utc = @revoked_utc
|
||||
WHERE key_id = @key_id;
|
||||
""");
|
||||
command.Parameters.Add("@key_id", SqlDbType.NVarChar, 64).Value = keyId;
|
||||
command.Parameters.Add("@revoked_utc", SqlDbType.DateTimeOffset).Value = whenUtc;
|
||||
}
|
||||
|
||||
await using (command)
|
||||
{
|
||||
int rows = await command.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
|
||||
|
||||
return rows > 0;
|
||||
}
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
/// <remarks>
|
||||
/// Only a REVOKED key can be deleted (<c>revoked_utc IS NOT NULL</c>), so an active credential
|
||||
/// cannot vanish in one step — revoke first, and the revocation stands on its own until then.
|
||||
/// </remarks>
|
||||
public async Task<bool> DeleteAsync(string keyId, CancellationToken ct)
|
||||
{
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(keyId);
|
||||
|
||||
string schema = connectionFactory.Schema;
|
||||
|
||||
await using SqlConnection connection =
|
||||
await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false);
|
||||
|
||||
await using SqlCommand command = connectionFactory.CreateCommand(connection, $"""
|
||||
DELETE FROM [{schema}].[{SqlServerAuthSchema.ApiKeysTable}]
|
||||
WHERE key_id = @key_id AND revoked_utc IS NOT NULL;
|
||||
""");
|
||||
command.Parameters.Add("@key_id", SqlDbType.NVarChar, 64).Value = keyId;
|
||||
|
||||
int rows = await command.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
|
||||
|
||||
return rows > 0;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task<IReadOnlyList<ApiKeyListItem>> ListAsync(CancellationToken ct)
|
||||
{
|
||||
string schema = connectionFactory.Schema;
|
||||
|
||||
await using SqlConnection connection =
|
||||
await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false);
|
||||
|
||||
// Deliberately omits secret_hash so listing can never leak secret material. key_id is the
|
||||
// tie-break so keys minted in the same instant still list in a stable order.
|
||||
await using SqlCommand command = connectionFactory.CreateCommand(connection, $"""
|
||||
SELECT key_id, key_prefix, display_name, scopes, constraints,
|
||||
created_utc, last_used_utc, revoked_utc, expires_utc
|
||||
FROM [{schema}].[{SqlServerAuthSchema.ApiKeysTable}]
|
||||
ORDER BY created_utc DESC, key_id DESC;
|
||||
""");
|
||||
|
||||
List<ApiKeyListItem> items = [];
|
||||
|
||||
await using SqlDataReader reader = await command.ExecuteReaderAsync(ct).ConfigureAwait(false);
|
||||
|
||||
while (await reader.ReadAsync(ct).ConfigureAwait(false))
|
||||
{
|
||||
items.Add(new ApiKeyListItem(
|
||||
KeyId: reader.GetString(0),
|
||||
KeyPrefix: reader.GetString(1),
|
||||
DisplayName: reader.GetString(2),
|
||||
Scopes: ScopeSerializer.Deserialize(reader.GetString(3)),
|
||||
ConstraintsJson: reader.IsDBNull(4) ? null : reader.GetString(4),
|
||||
CreatedUtc: reader.GetDateTimeOffset(5),
|
||||
LastUsedUtc: reader.IsDBNull(6) ? null : reader.GetDateTimeOffset(6),
|
||||
RevokedUtc: reader.IsDBNull(7) ? null : reader.GetDateTimeOffset(7),
|
||||
ExpiresUtc: reader.IsDBNull(8) ? null : reader.GetDateTimeOffset(8)));
|
||||
}
|
||||
|
||||
return items;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,80 @@
|
||||
using System.Data;
|
||||
using Microsoft.Data.SqlClient;
|
||||
using ZB.MOM.WW.Auth.Abstractions.ApiKeys;
|
||||
|
||||
namespace ZB.MOM.WW.Auth.ApiKeys.SqlServer;
|
||||
|
||||
/// <summary>
|
||||
/// SQL Server-backed, append-only audit store for API-key events. A one-for-one port of
|
||||
/// <c>SqliteApiKeyAuditStore</c>; on a clustered host the rows from every node land in one table,
|
||||
/// so the audit trail no longer has to be stitched together per node.
|
||||
/// </summary>
|
||||
/// <param name="connectionFactory">Factory for connections to the shared API-key database.</param>
|
||||
public sealed class SqlServerApiKeyAuditStore(AuthSqlServerConnectionFactory connectionFactory) : IApiKeyAuditStore
|
||||
{
|
||||
/// <inheritdoc />
|
||||
public async Task AppendAsync(ApiKeyAuditEntry entry, CancellationToken ct)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(entry);
|
||||
|
||||
string schema = connectionFactory.Schema;
|
||||
|
||||
await using SqlConnection connection =
|
||||
await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false);
|
||||
|
||||
await using SqlCommand command = connectionFactory.CreateCommand(connection, $"""
|
||||
INSERT INTO [{schema}].[{SqlServerAuthSchema.ApiKeyAuditTable}] (
|
||||
key_id, event_type, remote_address, created_utc, details)
|
||||
VALUES (@key_id, @event_type, @remote_address, @created_utc, @details);
|
||||
""");
|
||||
command.Parameters.Add("@key_id", SqlDbType.NVarChar, 64).Value = (object?)entry.KeyId ?? DBNull.Value;
|
||||
command.Parameters.Add("@event_type", SqlDbType.NVarChar, 64).Value = entry.EventType;
|
||||
command.Parameters.Add("@remote_address", SqlDbType.NVarChar, 64).Value = (object?)entry.RemoteAddress ?? DBNull.Value;
|
||||
command.Parameters.Add("@created_utc", SqlDbType.DateTimeOffset).Value = entry.CreatedUtc;
|
||||
command.Parameters.Add("@details", SqlDbType.NVarChar, -1).Value = (object?)entry.Details ?? DBNull.Value;
|
||||
|
||||
await command.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
/// <remarks>
|
||||
/// Ordered by the IDENTITY <c>audit_id</c>, not by <c>created_utc</c>: two events stamped in the
|
||||
/// same instant still come back in insertion order. A non-positive limit short-circuits without
|
||||
/// opening a connection — <c>SELECT TOP (-1)</c> is not even legal T-SQL.
|
||||
/// </remarks>
|
||||
public async Task<IReadOnlyList<ApiKeyAuditEntry>> ListRecentAsync(int limit, CancellationToken ct)
|
||||
{
|
||||
if (limit <= 0)
|
||||
{
|
||||
return [];
|
||||
}
|
||||
|
||||
string schema = connectionFactory.Schema;
|
||||
|
||||
await using SqlConnection connection =
|
||||
await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false);
|
||||
|
||||
await using SqlCommand command = connectionFactory.CreateCommand(connection, $"""
|
||||
SELECT TOP (@limit) key_id, event_type, remote_address, created_utc, details
|
||||
FROM [{schema}].[{SqlServerAuthSchema.ApiKeyAuditTable}]
|
||||
ORDER BY audit_id DESC;
|
||||
""");
|
||||
command.Parameters.Add("@limit", SqlDbType.Int).Value = limit;
|
||||
|
||||
List<ApiKeyAuditEntry> entries = [];
|
||||
|
||||
await using SqlDataReader reader = await command.ExecuteReaderAsync(ct).ConfigureAwait(false);
|
||||
|
||||
while (await reader.ReadAsync(ct).ConfigureAwait(false))
|
||||
{
|
||||
entries.Add(new ApiKeyAuditEntry(
|
||||
KeyId: reader.IsDBNull(0) ? null : reader.GetString(0),
|
||||
EventType: reader.GetString(1),
|
||||
RemoteAddress: reader.IsDBNull(2) ? null : reader.GetString(2),
|
||||
CreatedUtc: reader.GetDateTimeOffset(3),
|
||||
Details: reader.IsDBNull(4) ? null : reader.GetString(4)));
|
||||
}
|
||||
|
||||
return entries;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,65 @@
|
||||
namespace ZB.MOM.WW.Auth.ApiKeys.SqlServer;
|
||||
|
||||
/// <summary>
|
||||
/// Options for the shared SQL Server API-key store (section supplied by the host, e.g.
|
||||
/// <c>ScadaBridge:InboundApi:ApiKeyStore:SqlServer</c>).
|
||||
/// </summary>
|
||||
public sealed class SqlServerApiKeyOptions
|
||||
{
|
||||
/// <summary>
|
||||
/// Full SQL Server connection string. Required. Never a <c>${secret:}</c> reference on hosts
|
||||
/// whose secret store is itself SQL-backed.
|
||||
/// </summary>
|
||||
public string ConnectionString { get; set; } = string.Empty;
|
||||
|
||||
/// <summary>Schema that owns the three tables. Letters, digits and underscore only (it is interpolated into DDL).</summary>
|
||||
public string Schema { get; set; } = "zbauth";
|
||||
|
||||
/// <summary>
|
||||
/// Command timeout applied to every store operation. Defaults to 30 seconds — long enough for
|
||||
/// a contended shared database, short enough that a hung server surfaces as an error rather
|
||||
/// than stalling a resolve indefinitely.
|
||||
/// </summary>
|
||||
public TimeSpan CommandTimeout { get; set; } = TimeSpan.FromSeconds(30);
|
||||
|
||||
/// <summary>Fails fast on a blank connection string, an unsafe schema identifier, or a non-positive command timeout.</summary>
|
||||
/// <exception cref="InvalidOperationException">The options are not usable.</exception>
|
||||
public void Validate()
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(ConnectionString))
|
||||
throw new InvalidOperationException("SQL Server API-key store requires a ConnectionString.");
|
||||
|
||||
ValidateIdentifier(Schema);
|
||||
|
||||
if (CommandTimeout <= TimeSpan.Zero)
|
||||
throw new InvalidOperationException($"SQL Server API-key store CommandTimeout must be positive (was {CommandTimeout}).");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Strict allow-list for a SQL identifier that will be interpolated into DDL or query text.
|
||||
/// Rejects everything that is not <c>[A-Za-z_][A-Za-z0-9_]{0,63}</c> — no brackets, quotes, dots
|
||||
/// or spaces — so no bracket-escape (<c>]]</c>) trick can break out of the quoting applied at the
|
||||
/// use site. Uses the ASCII-only character tests deliberately, so homoglyph and full-width
|
||||
/// look-alikes are rejected rather than accepted as "letters".
|
||||
/// </summary>
|
||||
/// <param name="schemaName">The candidate identifier.</param>
|
||||
/// <returns>The same value, once validated.</returns>
|
||||
/// <exception cref="InvalidOperationException">The value is not a plain identifier.</exception>
|
||||
/// <remarks>
|
||||
/// Public because <see cref="SqlServerAuthSchema.CreateSchemaDdl(string)"/> is public and must
|
||||
/// defend itself rather than trust that every caller routed through <see cref="Validate"/>.
|
||||
/// </remarks>
|
||||
public static string ValidateIdentifier(string schemaName)
|
||||
{
|
||||
if (string.IsNullOrEmpty(schemaName)
|
||||
|| schemaName.Length > 64
|
||||
|| (!char.IsAsciiLetter(schemaName[0]) && schemaName[0] != '_')
|
||||
|| !schemaName.All(c => char.IsAsciiLetterOrDigit(c) || c == '_'))
|
||||
{
|
||||
throw new InvalidOperationException(
|
||||
$"SQL Server API-key store schema name '{schemaName}' is not a safe identifier.");
|
||||
}
|
||||
|
||||
return schemaName;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,100 @@
|
||||
using System.Data;
|
||||
using Microsoft.Data.SqlClient;
|
||||
using ZB.MOM.WW.Auth.Abstractions.ApiKeys;
|
||||
using ZB.MOM.WW.Auth.ApiKeys.Sqlite;
|
||||
|
||||
namespace ZB.MOM.WW.Auth.ApiKeys.SqlServer;
|
||||
|
||||
/// <summary>
|
||||
/// SQL Server-backed read store for API-key records — the verification hot path.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Semantics are a deliberate one-for-one port of <c>SqliteApiKeyStore</c>, down to the WHERE
|
||||
/// clauses, so a host can switch providers without any behaviour changing. One connection per
|
||||
/// operation: the shared store is reached over the network and pooling, not a held connection, is
|
||||
/// what keeps a central pair's concurrent verifies cheap.
|
||||
/// </remarks>
|
||||
/// <param name="connectionFactory">Factory for connections to the shared API-key database.</param>
|
||||
public sealed class SqlServerApiKeyStore(AuthSqlServerConnectionFactory connectionFactory) : IApiKeyStore
|
||||
{
|
||||
private const string SelectColumns =
|
||||
"key_id, key_prefix, secret_hash, display_name, scopes, constraints, created_utc, last_used_utc, revoked_utc, expires_utc";
|
||||
|
||||
/// <inheritdoc />
|
||||
public Task<ApiKeyRecord?> FindByKeyIdAsync(string keyId, CancellationToken ct)
|
||||
{
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(keyId);
|
||||
return FindAsync(keyId, requireActive: false, ct);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public Task<ApiKeyRecord?> FindActiveByKeyIdAsync(string keyId, CancellationToken ct)
|
||||
{
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(keyId);
|
||||
return FindAsync(keyId, requireActive: true, ct);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
/// <remarks>
|
||||
/// Guarded on <c>revoked_utc IS NULL</c>: a revoked key's last-used stamp is frozen at its last
|
||||
/// legitimate use, so it never records the rejected attempts that follow.
|
||||
/// </remarks>
|
||||
public async Task MarkUsedAsync(string keyId, DateTimeOffset whenUtc, CancellationToken ct)
|
||||
{
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(keyId);
|
||||
|
||||
string schema = connectionFactory.Schema;
|
||||
|
||||
await using SqlConnection connection =
|
||||
await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false);
|
||||
|
||||
await using SqlCommand command = connectionFactory.CreateCommand(connection, $"""
|
||||
UPDATE [{schema}].[{SqlServerAuthSchema.ApiKeysTable}]
|
||||
SET last_used_utc = @last_used_utc
|
||||
WHERE key_id = @key_id AND revoked_utc IS NULL;
|
||||
""");
|
||||
command.Parameters.Add("@key_id", SqlDbType.NVarChar, 64).Value = keyId;
|
||||
command.Parameters.Add("@last_used_utc", SqlDbType.DateTimeOffset).Value = whenUtc;
|
||||
|
||||
await command.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
private async Task<ApiKeyRecord?> FindAsync(string keyId, bool requireActive, CancellationToken ct)
|
||||
{
|
||||
string schema = connectionFactory.Schema;
|
||||
string table = $"[{schema}].[{SqlServerAuthSchema.ApiKeysTable}]";
|
||||
|
||||
await using SqlConnection connection =
|
||||
await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false);
|
||||
|
||||
// Active means NOT REVOKED only. Expiry is deliberately not filtered here — the verifier
|
||||
// owns it, so it can tell an expired key from an unknown one (ApiKeyFailure.KeyExpired).
|
||||
await using SqlCommand command = connectionFactory.CreateCommand(
|
||||
connection,
|
||||
requireActive
|
||||
? $"SELECT {SelectColumns} FROM {table} WHERE key_id = @key_id AND revoked_utc IS NULL;"
|
||||
: $"SELECT {SelectColumns} FROM {table} WHERE key_id = @key_id;");
|
||||
command.Parameters.Add("@key_id", SqlDbType.NVarChar, 64).Value = keyId;
|
||||
|
||||
await using SqlDataReader reader = await command.ExecuteReaderAsync(ct).ConfigureAwait(false);
|
||||
|
||||
if (!await reader.ReadAsync(ct).ConfigureAwait(false))
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
return ReadRecord(reader);
|
||||
}
|
||||
|
||||
internal static ApiKeyRecord ReadRecord(SqlDataReader reader) => new(
|
||||
KeyId: reader.GetString(0),
|
||||
KeyPrefix: reader.GetString(1),
|
||||
SecretHash: reader.GetFieldValue<byte[]>(2),
|
||||
DisplayName: reader.GetString(3),
|
||||
Scopes: ScopeSerializer.Deserialize(reader.GetString(4)),
|
||||
ConstraintsJson: reader.IsDBNull(5) ? null : reader.GetString(5),
|
||||
CreatedUtc: reader.GetDateTimeOffset(6),
|
||||
LastUsedUtc: reader.IsDBNull(7) ? null : reader.GetDateTimeOffset(7),
|
||||
RevokedUtc: reader.IsDBNull(8) ? null : reader.GetDateTimeOffset(8),
|
||||
ExpiresUtc: reader.IsDBNull(9) ? null : reader.GetDateTimeOffset(9));
|
||||
}
|
||||
@@ -0,0 +1,207 @@
|
||||
using System.Data;
|
||||
using System.Globalization;
|
||||
using Microsoft.Data.SqlClient;
|
||||
using ZB.MOM.WW.Auth.Abstractions.ApiKeys;
|
||||
using ZB.MOM.WW.Auth.ApiKeys.Sqlite;
|
||||
|
||||
namespace ZB.MOM.WW.Auth.ApiKeys.SqlServer;
|
||||
|
||||
/// <summary>
|
||||
/// Creates the shared SQL Server API-key schema and records the applied version. Idempotent: safe
|
||||
/// to run on every node at every startup, including concurrently — lost provisioning races surface
|
||||
/// as retryable <see cref="SqlException"/>s (see <see cref="IsRetryableMigrationErrorNumber"/>) and
|
||||
/// the retry re-runs the <c>IF NOT EXISTS</c> path against the objects the winner just created.
|
||||
/// Refuses to run against a database whose recorded version is newer than this build supports.
|
||||
/// </summary>
|
||||
/// <param name="connectionFactory">Factory for connections to the shared API-key database.</param>
|
||||
public sealed class SqlServerApiKeyStoreMigrator(AuthSqlServerConnectionFactory connectionFactory)
|
||||
: IApiKeyStoreMigrator
|
||||
{
|
||||
// Every node runs this at every startup, so a virgin database being provisioned by N nodes at
|
||||
// once is the NORMAL case, not a rare race — and an unretried failure aborts that node's whole
|
||||
// host at startup. Two failure shapes come out of that race, and retrying is correct for both
|
||||
// precisely because the migration is idempotent:
|
||||
//
|
||||
// - Deadlock victim (1205): concurrent Serializable DDL against the system catalogs can pick
|
||||
// a victim even though the IF NOT EXISTS guards make the logic itself safe.
|
||||
// - Lost CREATE / lost INSERT (2714, 1913, 2627): Serializable range locks do NOT serialize
|
||||
// the metadata existence checks against another session's in-flight CREATE — observed live
|
||||
// on the sibling shared secret store (ScadaBridge central-pair first boot, 2026-08-07):
|
||||
// both nodes passed the sys.schemas IF NOT EXISTS probe, the loser's CREATE SCHEMA then
|
||||
// died with 2714 "There is already an object named 'zbsecrets'". The object the loser tried
|
||||
// to create now exists, which is the desired end state — the retry's IF NOT EXISTS pass
|
||||
// skips it and proceeds.
|
||||
//
|
||||
// The retry cap keeps this from masking a GENUINE conflict (e.g. an unrelated pre-existing
|
||||
// object colliding with the schema name): that reproduces on every attempt, so after
|
||||
// MaxMigrationRetries the real SqlException propagates and the node fails closed.
|
||||
private const int MaxMigrationRetries = 5;
|
||||
|
||||
/// <summary>Deadlock victim.</summary>
|
||||
private const int SqlDeadlockVictimErrorNumber = 1205;
|
||||
|
||||
/// <summary>"There is already an object named '%ls'" — lost CREATE SCHEMA / CREATE TABLE race.</summary>
|
||||
private const int SqlObjectAlreadyExistsErrorNumber = 2714;
|
||||
|
||||
/// <summary>"An index or statistics with name '%ls' already exists" — lost CREATE INDEX race.</summary>
|
||||
private const int SqlIndexAlreadyExistsErrorNumber = 1913;
|
||||
|
||||
/// <summary>Primary-key violation — lost version-row MERGE-insert race.</summary>
|
||||
private const int SqlPrimaryKeyViolationErrorNumber = 2627;
|
||||
|
||||
/// <inheritdoc />
|
||||
/// <exception cref="AuthStoreMigrationException">
|
||||
/// The recorded schema version is newer than <see cref="SqlServerAuthSchema.CurrentVersion"/>.
|
||||
/// </exception>
|
||||
public async Task MigrateAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
for (int attempt = 1; ; attempt++)
|
||||
{
|
||||
try
|
||||
{
|
||||
await MigrateOnceAsync(cancellationToken).ConfigureAwait(false);
|
||||
return;
|
||||
}
|
||||
catch (SqlException ex)
|
||||
when (IsRetryableMigrationError(ex) && attempt < MaxMigrationRetries)
|
||||
{
|
||||
await Task.Delay(BackoffFor(attempt), cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Back-off before retry attempt <paramref name="attempt"/> + 1: a growing base plus RANDOM
|
||||
/// jitter. The jitter is the load-bearing half. The sibling secret-store migrator this is ported
|
||||
/// from delays a flat <c>100 * attempt</c>, which is the SAME value for every victim of the same
|
||||
/// round — so N deadlock victims wake together and re-collide, exactly the lockstep its comment
|
||||
/// claims to prevent. Measured here against a live SQL Server with eight concurrent migrators on
|
||||
/// a virgin schema, the unjittered form exhausted the five-attempt budget on repeated 1205s in
|
||||
/// roughly 1 run in 60; with jitter it did not.
|
||||
/// </summary>
|
||||
/// <param name="attempt">The 1-based attempt that just failed.</param>
|
||||
/// <returns>How long to wait before the next attempt.</returns>
|
||||
private static TimeSpan BackoffFor(int attempt) =>
|
||||
TimeSpan.FromMilliseconds((100 * attempt) + Random.Shared.Next(0, 100 * attempt));
|
||||
|
||||
/// <summary>
|
||||
/// Whether any error carried by <paramref name="exception"/> is a retryable provisioning-race
|
||||
/// outcome. Scans the whole <see cref="SqlException.Errors"/> collection rather than only
|
||||
/// <see cref="SqlException.Number"/> because a failed DDL batch stacks follow-on errors (e.g.
|
||||
/// 2714 arrives with 2759 "CREATE SCHEMA failed due to previous errors") and the retryable one
|
||||
/// is not guaranteed to be first.
|
||||
/// </summary>
|
||||
/// <param name="exception">The exception the migration attempt failed with.</param>
|
||||
/// <returns><see langword="true"/> when a retry may succeed.</returns>
|
||||
internal static bool IsRetryableMigrationError(SqlException exception)
|
||||
{
|
||||
foreach (SqlError error in exception.Errors)
|
||||
{
|
||||
if (IsRetryableMigrationErrorNumber(error.Number))
|
||||
{
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
return IsRetryableMigrationErrorNumber(exception.Number);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Classifies a SQL Server error number as a retryable outcome of N nodes provisioning the same
|
||||
/// (possibly virgin) database at once. Everything here is a race whose retry re-runs the
|
||||
/// idempotent <c>IF NOT EXISTS</c> path; anything else (login failures, missing database,
|
||||
/// permissions, syntax) reproduces identically on retry and must propagate. 2705 (duplicate
|
||||
/// column) is deliberately absent: every column lives inside a single <c>CREATE TABLE</c>
|
||||
/// statement, so no cross-session race can produce it.
|
||||
/// </summary>
|
||||
/// <param name="errorNumber">The <see cref="SqlError.Number"/> to classify.</param>
|
||||
/// <returns><see langword="true"/> when the number is a known provisioning-race outcome.</returns>
|
||||
internal static bool IsRetryableMigrationErrorNumber(int errorNumber) => errorNumber
|
||||
is SqlDeadlockVictimErrorNumber
|
||||
or SqlObjectAlreadyExistsErrorNumber
|
||||
or SqlIndexAlreadyExistsErrorNumber
|
||||
or SqlPrimaryKeyViolationErrorNumber;
|
||||
|
||||
private async Task MigrateOnceAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
string schema = connectionFactory.Schema;
|
||||
|
||||
await using SqlConnection connection =
|
||||
await connectionFactory.OpenConnectionAsync(cancellationToken).ConfigureAwait(false);
|
||||
|
||||
// Serializable: unlike the single-writer SQLite store, EVERY node in the cluster runs this
|
||||
// migration at startup, so concurrent first-boots are the normal case rather than a rare
|
||||
// race. The isolation level narrows the race window but does NOT close it — the IF NOT
|
||||
// EXISTS metadata probes are not serialized against another session's in-flight CREATE, so
|
||||
// a loser can still fail with "object exists" (2714/1913) or a deadlock (1205). The retry
|
||||
// filter in MigrateAsync is what makes simultaneous provision converge: by the time the
|
||||
// loser retries, the winner's objects exist and every guard here skips them.
|
||||
await using SqlTransaction transaction = (SqlTransaction)
|
||||
await connection.BeginTransactionAsync(IsolationLevel.Serializable, cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
int existingVersion =
|
||||
await ReadExistingSchemaVersionAsync(connection, transaction, schema, cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
if (existingVersion > SqlServerAuthSchema.CurrentVersion)
|
||||
{
|
||||
throw new AuthStoreMigrationException(
|
||||
$"Shared API-key database schema version {existingVersion} is newer than supported version " +
|
||||
$"{SqlServerAuthSchema.CurrentVersion}. Upgrade this node before it writes to the shared store.");
|
||||
}
|
||||
|
||||
await using (SqlCommand ddl = connectionFactory.CreateCommand(
|
||||
connection, SqlServerAuthSchema.CreateSchemaDdl(schema), transaction))
|
||||
{
|
||||
await ddl.ExecuteNonQueryAsync(cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
// MERGE rather than insert-if-missing: WHEN MATCHED re-stamps the row, so a future schema
|
||||
// version actually lands in the table instead of leaving the first-ever value behind.
|
||||
await using (SqlCommand version = connectionFactory.CreateCommand(
|
||||
connection,
|
||||
$"""
|
||||
MERGE [{schema}].[{SqlServerAuthSchema.SchemaVersionTable}] AS target
|
||||
USING (SELECT 1 AS id, @version AS version, @applied_utc AS applied_utc) AS source
|
||||
ON target.id = source.id
|
||||
WHEN MATCHED THEN
|
||||
UPDATE SET version = source.version, applied_utc = source.applied_utc
|
||||
WHEN NOT MATCHED THEN
|
||||
INSERT (id, version, applied_utc) VALUES (1, source.version, source.applied_utc);
|
||||
""",
|
||||
transaction))
|
||||
{
|
||||
version.Parameters.Add("@version", SqlDbType.Int).Value = SqlServerAuthSchema.CurrentVersion;
|
||||
version.Parameters.Add("@applied_utc", SqlDbType.DateTimeOffset).Value = DateTimeOffset.UtcNow;
|
||||
await version.ExecuteNonQueryAsync(cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
await transaction.CommitAsync(cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
private async Task<int> ReadExistingSchemaVersionAsync(
|
||||
SqlConnection connection,
|
||||
SqlTransaction transaction,
|
||||
string schema,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
// OBJECT_ID returns NULL for an absent table, so this reads 0 on a virgin database without
|
||||
// needing the table to exist first.
|
||||
await using SqlCommand command = connectionFactory.CreateCommand(
|
||||
connection,
|
||||
$"""
|
||||
IF OBJECT_ID(N'[{schema}].[{SqlServerAuthSchema.SchemaVersionTable}]', N'U') IS NULL
|
||||
SELECT 0;
|
||||
ELSE
|
||||
SELECT ISNULL((SELECT version FROM [{schema}].[{SqlServerAuthSchema.SchemaVersionTable}] WHERE id = 1), 0);
|
||||
""",
|
||||
transaction);
|
||||
|
||||
object? version = await command.ExecuteScalarAsync(cancellationToken).ConfigureAwait(false);
|
||||
|
||||
return version is null || version == DBNull.Value
|
||||
? 0
|
||||
: Convert.ToInt32(version, CultureInfo.InvariantCulture);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,116 @@
|
||||
namespace ZB.MOM.WW.Auth.ApiKeys.SqlServer;
|
||||
|
||||
/// <summary>
|
||||
/// Schema constants and DDL for the shared SQL Server API-key store. Mirrors
|
||||
/// <c>ZB.MOM.WW.Auth.ApiKeys.Sqlite.SqliteAuthSchema</c> column-for-column so a row means exactly
|
||||
/// the same thing in either store.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>Timestamps are <c>DATETIMEOFFSET(7)</c>, not ISO-8601 text.</b> The sibling
|
||||
/// <c>ZB.MOM.WW.Secrets</c> SQL Server store deliberately stores its timestamps as
|
||||
/// <c>nvarchar</c> because its rows <em>replicate</em> with a SQLite store and last-writer-wins
|
||||
/// compares those timestamps for equality, where a sub-tick round-trip difference would turn a tie
|
||||
/// into a spurious "newer". No such rationale applies here: this store never replicates with the
|
||||
/// SQLite one — a host picks exactly one provider, and the one-shot SQLite import is a copy, not an
|
||||
/// ongoing sync — so the native type is used, which sorts and range-scans correctly and keeps the
|
||||
/// tables readable to a DBA.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <c>SqliteAuthSchema.CurrentVersion</c> is 3 only because that store carries the donor
|
||||
/// MxAccessGateway's version history (v1/v2 were byte-identical; v3 added <c>expires_utc</c>). This
|
||||
/// schema is created whole, with <c>expires_utc</c> present from the start, so its own version line
|
||||
/// starts at 1.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public static class SqlServerAuthSchema
|
||||
{
|
||||
/// <summary>
|
||||
/// The schema version this build creates and supports. The migrator refuses a database whose
|
||||
/// recorded version is <em>newer</em> than this.
|
||||
/// </summary>
|
||||
public const int CurrentVersion = 1;
|
||||
|
||||
/// <summary>Default schema the API-key tables live in.</summary>
|
||||
public const string DefaultSchemaName = "zbauth";
|
||||
|
||||
/// <summary>Name of the single-row table tracking the applied schema version.</summary>
|
||||
public const string SchemaVersionTable = "schema_version";
|
||||
|
||||
/// <summary>Name of the table storing API-key records.</summary>
|
||||
public const string ApiKeysTable = "api_keys";
|
||||
|
||||
/// <summary>Name of the append-only audit table.</summary>
|
||||
public const string ApiKeyAuditTable = "api_key_audit";
|
||||
|
||||
/// <summary>
|
||||
/// Returns the idempotent DDL that provisions the schema, the three tables and the two
|
||||
/// supporting indexes in <paramref name="schemaName"/>.
|
||||
/// </summary>
|
||||
/// <param name="schemaName">The SQL schema name to create the objects in.</param>
|
||||
/// <returns>The DDL batch.</returns>
|
||||
/// <remarks>
|
||||
/// <paramref name="schemaName"/> is interpolated, not parameterized — T-SQL has no parameter
|
||||
/// form for an object name. It is re-validated here against the same strict identifier
|
||||
/// allow-list <see cref="SqlServerApiKeyOptions.Validate"/> enforces, and bracket-quoted below.
|
||||
/// The duplicate check is deliberate: this method is <c>public</c>, so it must defend itself
|
||||
/// rather than trust that every caller routed through the options type. The version row is NOT
|
||||
/// written here — <see cref="SqlServerApiKeyStoreMigrator"/> writes it with a MERGE that also
|
||||
/// updates on match, so a future upgrade re-stamps the stored version.
|
||||
/// </remarks>
|
||||
/// <exception cref="InvalidOperationException"><paramref name="schemaName"/> is not a plain identifier.</exception>
|
||||
public static string CreateSchemaDdl(string schemaName)
|
||||
{
|
||||
SqlServerApiKeyOptions.ValidateIdentifier(schemaName);
|
||||
|
||||
return $"""
|
||||
IF NOT EXISTS (SELECT 1 FROM sys.schemas WHERE name = N'{schemaName}')
|
||||
EXEC(N'CREATE SCHEMA [{schemaName}]');
|
||||
|
||||
IF OBJECT_ID(N'[{schemaName}].[{SchemaVersionTable}]', N'U') IS NULL
|
||||
CREATE TABLE [{schemaName}].[{SchemaVersionTable}] (
|
||||
id INT NOT NULL CONSTRAINT PK_{schemaName}_{SchemaVersionTable} PRIMARY KEY CHECK (id = 1),
|
||||
version INT NOT NULL,
|
||||
applied_utc DATETIMEOFFSET(7) NOT NULL
|
||||
);
|
||||
|
||||
IF OBJECT_ID(N'[{schemaName}].[{ApiKeysTable}]', N'U') IS NULL
|
||||
CREATE TABLE [{schemaName}].[{ApiKeysTable}] (
|
||||
key_id NVARCHAR(64) NOT NULL CONSTRAINT PK_{schemaName}_{ApiKeysTable} PRIMARY KEY,
|
||||
key_prefix NVARCHAR(16) NOT NULL,
|
||||
secret_hash VARBINARY(64) NOT NULL,
|
||||
display_name NVARCHAR(256) NOT NULL,
|
||||
scopes NVARCHAR(MAX) NOT NULL,
|
||||
constraints NVARCHAR(MAX) NULL,
|
||||
created_utc DATETIMEOFFSET(7) NOT NULL,
|
||||
last_used_utc DATETIMEOFFSET(7) NULL,
|
||||
revoked_utc DATETIMEOFFSET(7) NULL,
|
||||
expires_utc DATETIMEOFFSET(7) NULL
|
||||
);
|
||||
|
||||
IF OBJECT_ID(N'[{schemaName}].[{ApiKeyAuditTable}]', N'U') IS NULL
|
||||
CREATE TABLE [{schemaName}].[{ApiKeyAuditTable}] (
|
||||
audit_id BIGINT IDENTITY(1,1) NOT NULL CONSTRAINT PK_{schemaName}_{ApiKeyAuditTable} PRIMARY KEY,
|
||||
key_id NVARCHAR(64) NULL,
|
||||
event_type NVARCHAR(64) NOT NULL,
|
||||
remote_address NVARCHAR(64) NULL,
|
||||
created_utc DATETIMEOFFSET(7) NOT NULL,
|
||||
details NVARCHAR(MAX) NULL
|
||||
);
|
||||
|
||||
IF NOT EXISTS (
|
||||
SELECT 1 FROM sys.indexes
|
||||
WHERE name = N'ix_api_keys_revoked_utc'
|
||||
AND object_id = OBJECT_ID(N'[{schemaName}].[{ApiKeysTable}]'))
|
||||
CREATE INDEX ix_api_keys_revoked_utc
|
||||
ON [{schemaName}].[{ApiKeysTable}] (revoked_utc);
|
||||
|
||||
IF NOT EXISTS (
|
||||
SELECT 1 FROM sys.indexes
|
||||
WHERE name = N'ix_api_key_audit_key_id_created_utc'
|
||||
AND object_id = OBJECT_ID(N'[{schemaName}].[{ApiKeyAuditTable}]'))
|
||||
CREATE INDEX ix_api_key_audit_key_id_created_utc
|
||||
ON [{schemaName}].[{ApiKeyAuditTable}] (key_id, created_utc);
|
||||
""";
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,300 @@
|
||||
using System.Data;
|
||||
using Microsoft.Data.SqlClient;
|
||||
using Microsoft.Data.Sqlite;
|
||||
using ZB.MOM.WW.Auth.Abstractions.ApiKeys;
|
||||
using ZB.MOM.WW.Auth.ApiKeys.Sqlite;
|
||||
|
||||
namespace ZB.MOM.WW.Auth.ApiKeys.SqlServer;
|
||||
|
||||
/// <summary>Result of a one-shot import.</summary>
|
||||
/// <param name="KeysInserted">Keys copied into SQL Server by this run.</param>
|
||||
/// <param name="KeysSkipped">Keys whose <c>key_id</c> already existed in SQL Server.</param>
|
||||
/// <param name="AuditInserted">Audit rows copied into SQL Server by this run.</param>
|
||||
/// <param name="AuditSkipped">Audit rows that already existed in SQL Server.</param>
|
||||
/// <param name="SourceFound">Whether the SQLite file existed at all.</param>
|
||||
public sealed record ApiKeyImportResult(
|
||||
int KeysInserted, int KeysSkipped, int AuditInserted, int AuditSkipped, bool SourceFound);
|
||||
|
||||
/// <summary>
|
||||
/// Copies every key and audit row from a per-node SQLite store into the shared SQL Server store,
|
||||
/// insert-where-not-exists (keys by key_id; audit rows by key_id + event_type + created_utc), so it is
|
||||
/// idempotent and every node of a pair may run it: the result is the UNION of the nodes' files. A row
|
||||
/// that already exists in SQL Server is never modified — a revoke recorded on one node but not the
|
||||
/// other therefore survives only if the revoked copy is imported first; operators reconcile by listing.
|
||||
/// </summary>
|
||||
/// <param name="sql">Factory for connections to the shared SQL Server API-key database.</param>
|
||||
public sealed class SqliteApiKeyStoreImporter(AuthSqlServerConnectionFactory sql)
|
||||
{
|
||||
/// <summary>
|
||||
/// Imports the keys and audit rows held in the SQLite store at <paramref name="sqlitePath"/>.
|
||||
/// </summary>
|
||||
/// <param name="sqlitePath">Path of the per-node SQLite API-key file.</param>
|
||||
/// <param name="ct">A token to cancel the import.</param>
|
||||
/// <returns>
|
||||
/// The per-category insert/skip counts. When the file does not exist the result is all zeroes
|
||||
/// with <see cref="ApiKeyImportResult.SourceFound"/> <see langword="false"/> and nothing is
|
||||
/// opened — a host that has already imported (and renamed the file) pays nothing.
|
||||
/// </returns>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// The SQLite file is opened READ-ONLY and the schema migrator is deliberately NOT run against
|
||||
/// it: the source is evidence, not something to upgrade. A file with no <c>api_keys</c> table is
|
||||
/// treated as an empty source rather than an error, and a pre-v3 file (no <c>expires_utc</c>
|
||||
/// column) reads its expiry as NULL = never expires, which is exactly what those keys meant.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Every row is read and the SQLite connection closed BEFORE any SQL Server write, and all the
|
||||
/// writes then run inside ONE transaction. So the import is all-or-nothing: a failure part way
|
||||
/// through rolls the transaction back and the exception propagates to the caller, leaving the
|
||||
/// shared store exactly as it was. The host decides what that means — for a boot-time import,
|
||||
/// logging and carrying on is usually right, because keys already in SQL Server keep working.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
/// <exception cref="SqlException">A SQL Server write failed; nothing was committed.</exception>
|
||||
/// <exception cref="SqliteException">The source file could not be read.</exception>
|
||||
public async Task<ApiKeyImportResult> ImportAsync(string sqlitePath, CancellationToken ct)
|
||||
{
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(sqlitePath);
|
||||
|
||||
if (!File.Exists(sqlitePath))
|
||||
{
|
||||
return new ApiKeyImportResult(0, 0, 0, 0, SourceFound: false);
|
||||
}
|
||||
|
||||
(IReadOnlyList<ApiKeyRecord> keys, IReadOnlyList<ApiKeyAuditEntry> auditEntries) =
|
||||
await ReadSourceAsync(sqlitePath, ct).ConfigureAwait(false);
|
||||
|
||||
return await WriteAsync(keys, auditEntries, ct).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
// --- read side (SQLite) ---
|
||||
|
||||
private static async Task<(IReadOnlyList<ApiKeyRecord> Keys, IReadOnlyList<ApiKeyAuditEntry> Audit)>
|
||||
ReadSourceAsync(string sqlitePath, CancellationToken ct)
|
||||
{
|
||||
SqliteConnectionStringBuilder builder = new()
|
||||
{
|
||||
DataSource = sqlitePath,
|
||||
Mode = SqliteOpenMode.ReadOnly,
|
||||
};
|
||||
|
||||
await using var connection = new SqliteConnection(builder.ToString());
|
||||
await connection.OpenAsync(ct).ConfigureAwait(false);
|
||||
|
||||
IReadOnlyList<ApiKeyRecord> keys =
|
||||
await TableExistsAsync(connection, SqliteAuthSchema.ApiKeysTable, ct).ConfigureAwait(false)
|
||||
? await ReadKeysAsync(connection, ct).ConfigureAwait(false)
|
||||
: [];
|
||||
|
||||
IReadOnlyList<ApiKeyAuditEntry> audit =
|
||||
await TableExistsAsync(connection, SqliteAuthSchema.ApiKeyAuditTable, ct).ConfigureAwait(false)
|
||||
? await ReadAuditAsync(connection, ct).ConfigureAwait(false)
|
||||
: [];
|
||||
|
||||
return (keys, audit);
|
||||
}
|
||||
|
||||
private static async Task<bool> TableExistsAsync(
|
||||
SqliteConnection connection, string tableName, CancellationToken ct)
|
||||
{
|
||||
await using SqliteCommand command = connection.CreateCommand();
|
||||
command.CommandText = """
|
||||
SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = $table_name;
|
||||
""";
|
||||
command.Parameters.AddWithValue("$table_name", tableName);
|
||||
|
||||
return (long)(await command.ExecuteScalarAsync(ct).ConfigureAwait(false) ?? 0L) > 0;
|
||||
}
|
||||
|
||||
private static async Task<IReadOnlyList<ApiKeyRecord>> ReadKeysAsync(
|
||||
SqliteConnection connection, CancellationToken ct)
|
||||
{
|
||||
// A pre-v3 file has no expires_utc column, so select a literal NULL in its place rather than
|
||||
// failing the import: NULL is precisely what those keys mean (never expires).
|
||||
bool hasExpires = await ColumnExistsAsync(
|
||||
connection, SqliteAuthSchema.ApiKeysTable, SqliteAuthSchema.ExpiresUtcColumn, ct).ConfigureAwait(false);
|
||||
|
||||
string expiresColumn = hasExpires ? SqliteAuthSchema.ExpiresUtcColumn : "NULL";
|
||||
|
||||
await using SqliteCommand command = connection.CreateCommand();
|
||||
command.CommandText =
|
||||
"SELECT key_id, key_prefix, secret_hash, display_name, scopes, constraints, " +
|
||||
$"created_utc, last_used_utc, revoked_utc, {expiresColumn} FROM api_keys;";
|
||||
|
||||
List<ApiKeyRecord> keys = [];
|
||||
|
||||
await using SqliteDataReader reader = await command.ExecuteReaderAsync(ct).ConfigureAwait(false);
|
||||
|
||||
while (await reader.ReadAsync(ct).ConfigureAwait(false))
|
||||
{
|
||||
keys.Add(new ApiKeyRecord(
|
||||
KeyId: reader.GetString(0),
|
||||
KeyPrefix: reader.GetString(1),
|
||||
SecretHash: reader.GetFieldValue<byte[]>(2),
|
||||
DisplayName: reader.GetString(3),
|
||||
Scopes: ScopeSerializer.Deserialize(reader.GetString(4)),
|
||||
ConstraintsJson: reader.IsDBNull(5) ? null : reader.GetString(5),
|
||||
CreatedUtc: SqliteValueParsing.ParseUtc(reader.GetString(6)),
|
||||
LastUsedUtc: SqliteValueParsing.ReadNullableUtc(reader, 7),
|
||||
RevokedUtc: SqliteValueParsing.ReadNullableUtc(reader, 8),
|
||||
ExpiresUtc: SqliteValueParsing.ReadNullableUtc(reader, 9)));
|
||||
}
|
||||
|
||||
return keys;
|
||||
}
|
||||
|
||||
// SQLite has no information_schema; PRAGMA table_info is the column probe (the same one
|
||||
// SqliteAuthStoreMigrator uses to decide whether its v3 ALTER TABLE is needed).
|
||||
private static async Task<bool> ColumnExistsAsync(
|
||||
SqliteConnection connection, string tableName, string columnName, CancellationToken ct)
|
||||
{
|
||||
await using SqliteCommand command = connection.CreateCommand();
|
||||
// table_info takes an identifier, not a bindable parameter; the name is an internal constant.
|
||||
command.CommandText = $"PRAGMA table_info({tableName});";
|
||||
|
||||
await using SqliteDataReader reader = await command.ExecuteReaderAsync(ct).ConfigureAwait(false);
|
||||
|
||||
while (await reader.ReadAsync(ct).ConfigureAwait(false))
|
||||
{
|
||||
// PRAGMA table_info columns: cid(0), name(1), type(2), ...
|
||||
if (string.Equals(reader.GetString(1), columnName, StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
private static async Task<IReadOnlyList<ApiKeyAuditEntry>> ReadAuditAsync(
|
||||
SqliteConnection connection, CancellationToken ct)
|
||||
{
|
||||
await using SqliteCommand command = connection.CreateCommand();
|
||||
// Insertion order, so the shared table's IDENTITY re-creates the source's ordering.
|
||||
command.CommandText = """
|
||||
SELECT key_id, event_type, remote_address, created_utc, details
|
||||
FROM api_key_audit
|
||||
ORDER BY audit_id;
|
||||
""";
|
||||
|
||||
List<ApiKeyAuditEntry> entries = [];
|
||||
|
||||
await using SqliteDataReader reader = await command.ExecuteReaderAsync(ct).ConfigureAwait(false);
|
||||
|
||||
while (await reader.ReadAsync(ct).ConfigureAwait(false))
|
||||
{
|
||||
entries.Add(new ApiKeyAuditEntry(
|
||||
KeyId: reader.IsDBNull(0) ? null : reader.GetString(0),
|
||||
EventType: reader.GetString(1),
|
||||
RemoteAddress: reader.IsDBNull(2) ? null : reader.GetString(2),
|
||||
CreatedUtc: SqliteValueParsing.ParseUtc(reader.GetString(3)),
|
||||
Details: reader.IsDBNull(4) ? null : reader.GetString(4)));
|
||||
}
|
||||
|
||||
return entries;
|
||||
}
|
||||
|
||||
// --- write side (SQL Server) ---
|
||||
|
||||
private async Task<ApiKeyImportResult> WriteAsync(
|
||||
IReadOnlyList<ApiKeyRecord> keys,
|
||||
IReadOnlyList<ApiKeyAuditEntry> auditEntries,
|
||||
CancellationToken ct)
|
||||
{
|
||||
string schema = sql.Schema;
|
||||
|
||||
await using SqlConnection connection = await sql.OpenConnectionAsync(ct).ConfigureAwait(false);
|
||||
await using SqlTransaction transaction =
|
||||
(SqlTransaction)await connection.BeginTransactionAsync(ct).ConfigureAwait(false);
|
||||
|
||||
int keysInserted = 0;
|
||||
int auditInserted = 0;
|
||||
|
||||
foreach (ApiKeyRecord key in keys)
|
||||
{
|
||||
if (await InsertKeyIfAbsentAsync(connection, transaction, schema, key, ct).ConfigureAwait(false))
|
||||
{
|
||||
keysInserted++;
|
||||
}
|
||||
}
|
||||
|
||||
foreach (ApiKeyAuditEntry entry in auditEntries)
|
||||
{
|
||||
if (await InsertAuditIfAbsentAsync(connection, transaction, schema, entry, ct).ConfigureAwait(false))
|
||||
{
|
||||
auditInserted++;
|
||||
}
|
||||
}
|
||||
|
||||
await transaction.CommitAsync(ct).ConfigureAwait(false);
|
||||
|
||||
return new ApiKeyImportResult(
|
||||
KeysInserted: keysInserted,
|
||||
KeysSkipped: keys.Count - keysInserted,
|
||||
AuditInserted: auditInserted,
|
||||
AuditSkipped: auditEntries.Count - auditInserted,
|
||||
SourceFound: true);
|
||||
}
|
||||
|
||||
// Returns true when this call inserted the row. The guard is inside the statement rather than a
|
||||
// separate SELECT so the check and the insert cannot straddle a concurrent writer.
|
||||
private async Task<bool> InsertKeyIfAbsentAsync(
|
||||
SqlConnection connection,
|
||||
SqlTransaction transaction,
|
||||
string schema,
|
||||
ApiKeyRecord key,
|
||||
CancellationToken ct)
|
||||
{
|
||||
await using SqlCommand command = sql.CreateCommand(connection, $"""
|
||||
IF NOT EXISTS (
|
||||
SELECT 1 FROM [{schema}].[{SqlServerAuthSchema.ApiKeysTable}] WHERE key_id = @key_id)
|
||||
INSERT INTO [{schema}].[{SqlServerAuthSchema.ApiKeysTable}] (
|
||||
key_id, key_prefix, secret_hash, display_name, scopes,
|
||||
constraints, created_utc, last_used_utc, revoked_utc, expires_utc)
|
||||
VALUES (
|
||||
@key_id, @key_prefix, @secret_hash, @display_name, @scopes,
|
||||
@constraints, @created_utc, @last_used_utc, @revoked_utc, @expires_utc);
|
||||
""", transaction);
|
||||
command.Parameters.Add("@key_id", SqlDbType.NVarChar, 64).Value = key.KeyId;
|
||||
command.Parameters.Add("@key_prefix", SqlDbType.NVarChar, 16).Value = key.KeyPrefix;
|
||||
command.Parameters.Add("@secret_hash", SqlDbType.VarBinary, 64).Value = key.SecretHash;
|
||||
command.Parameters.Add("@display_name", SqlDbType.NVarChar, 256).Value = key.DisplayName;
|
||||
command.Parameters.Add("@scopes", SqlDbType.NVarChar, -1).Value = ScopeSerializer.Serialize(key.Scopes);
|
||||
command.Parameters.Add("@constraints", SqlDbType.NVarChar, -1).Value = (object?)key.ConstraintsJson ?? DBNull.Value;
|
||||
command.Parameters.Add("@created_utc", SqlDbType.DateTimeOffset).Value = key.CreatedUtc;
|
||||
command.Parameters.Add("@last_used_utc", SqlDbType.DateTimeOffset).Value = (object?)key.LastUsedUtc ?? DBNull.Value;
|
||||
command.Parameters.Add("@revoked_utc", SqlDbType.DateTimeOffset).Value = (object?)key.RevokedUtc ?? DBNull.Value;
|
||||
command.Parameters.Add("@expires_utc", SqlDbType.DateTimeOffset).Value = (object?)key.ExpiresUtc ?? DBNull.Value;
|
||||
|
||||
return await command.ExecuteNonQueryAsync(ct).ConfigureAwait(false) > 0;
|
||||
}
|
||||
|
||||
// Audit rows carry no natural key (audit_id is per-file IDENTITY), so identity is the triple
|
||||
// key_id + event_type + created_utc. key_id is nullable, hence the explicit NULL-safe arm: a
|
||||
// plain `key_id = @key_id` is UNKNOWN, never true, when both sides are NULL.
|
||||
private async Task<bool> InsertAuditIfAbsentAsync(
|
||||
SqlConnection connection,
|
||||
SqlTransaction transaction,
|
||||
string schema,
|
||||
ApiKeyAuditEntry entry,
|
||||
CancellationToken ct)
|
||||
{
|
||||
await using SqlCommand command = sql.CreateCommand(connection, $"""
|
||||
IF NOT EXISTS (
|
||||
SELECT 1 FROM [{schema}].[{SqlServerAuthSchema.ApiKeyAuditTable}]
|
||||
WHERE ((key_id = @key_id) OR (key_id IS NULL AND @key_id IS NULL))
|
||||
AND event_type = @event_type
|
||||
AND created_utc = @created_utc)
|
||||
INSERT INTO [{schema}].[{SqlServerAuthSchema.ApiKeyAuditTable}] (
|
||||
key_id, event_type, remote_address, created_utc, details)
|
||||
VALUES (@key_id, @event_type, @remote_address, @created_utc, @details);
|
||||
""", transaction);
|
||||
command.Parameters.Add("@key_id", SqlDbType.NVarChar, 64).Value = (object?)entry.KeyId ?? DBNull.Value;
|
||||
command.Parameters.Add("@event_type", SqlDbType.NVarChar, 64).Value = entry.EventType;
|
||||
command.Parameters.Add("@remote_address", SqlDbType.NVarChar, 64).Value = (object?)entry.RemoteAddress ?? DBNull.Value;
|
||||
command.Parameters.Add("@created_utc", SqlDbType.DateTimeOffset).Value = entry.CreatedUtc;
|
||||
command.Parameters.Add("@details", SqlDbType.NVarChar, -1).Value = (object?)entry.Details ?? DBNull.Value;
|
||||
|
||||
return await command.ExecuteNonQueryAsync(ct).ConfigureAwait(false) > 0;
|
||||
}
|
||||
}
|
||||
+42
@@ -0,0 +1,42 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\ZB.MOM.WW.Auth.ApiKeys\ZB.MOM.WW.Auth.ApiKeys.csproj" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<PackageReference Include="Microsoft.Data.SqlClient" />
|
||||
<!--
|
||||
Lightweight Microsoft.Extensions.* abstractions back the DI extension
|
||||
(AddZbApiKeyAuthSqlServer, shipped by Task 6): IServiceCollection / TryAdd* / Configure
|
||||
(DependencyInjection.Abstractions), IOptions (Options), IConfiguration
|
||||
(Configuration.Abstractions), and IHostedService / AddHostedService (Hosting.Abstractions).
|
||||
-->
|
||||
<PackageReference Include="Microsoft.Extensions.DependencyInjection.Abstractions" />
|
||||
<PackageReference Include="Microsoft.Extensions.Options" />
|
||||
<!-- Supplies the Configure<TOptions>(IConfiguration) binding overload used by the DI extension. -->
|
||||
<PackageReference Include="Microsoft.Extensions.Options.ConfigurationExtensions" />
|
||||
<PackageReference Include="Microsoft.Extensions.Configuration.Abstractions" />
|
||||
<PackageReference Include="Microsoft.Extensions.Hosting.Abstractions" />
|
||||
</ItemGroup>
|
||||
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
<Nullable>enable</Nullable>
|
||||
</PropertyGroup>
|
||||
|
||||
<PropertyGroup>
|
||||
<IsPackable>true</IsPackable>
|
||||
<PackageId>ZB.MOM.WW.Auth.ApiKeys.SqlServer</PackageId>
|
||||
<Authors>ZB.MOM.WW</Authors>
|
||||
<Description>SQL Server API-key store for the ZB.MOM.WW SCADA family — one shared row set for clustered hosts (central pairs).</Description>
|
||||
<PackageProjectUrl>https://gitea.dohertylan.com/dohertj2/zb-mom-ww-auth</PackageProjectUrl>
|
||||
<RepositoryUrl>https://gitea.dohertylan.com/dohertj2/zb-mom-ww-auth</RepositoryUrl>
|
||||
</PropertyGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<InternalsVisibleTo Include="ZB.MOM.WW.Auth.ApiKeys.SqlServer.Tests" />
|
||||
</ItemGroup>
|
||||
|
||||
</Project>
|
||||
@@ -33,10 +33,15 @@ public sealed class ApiKeyAdminCommands
|
||||
private readonly IApiKeyAdminStore _adminStore;
|
||||
private readonly IApiKeyAuditStore _auditStore;
|
||||
private readonly IApiKeyPepperProvider _pepperProvider;
|
||||
private readonly SqliteAuthStoreMigrator _migrator;
|
||||
private readonly IApiKeyStoreMigrator _migrator;
|
||||
private readonly TimeProvider _clock;
|
||||
|
||||
/// <summary>Creates the command set over the supplied stores and options.</summary>
|
||||
/// <summary>
|
||||
/// Creates the command set over the supplied stores and options.
|
||||
/// Retained for existing consumers (MxAccessGateway, HistorianGateway, ScadaBridge) that
|
||||
/// construct this type against the concrete SQLite migrator; forwards to the
|
||||
/// <see cref="IApiKeyStoreMigrator"/> overload.
|
||||
/// </summary>
|
||||
/// <param name="options">API-key options (token prefix, store path, ...).</param>
|
||||
/// <param name="adminStore">Mutating store (create / revoke / rotate / delete / list).</param>
|
||||
/// <param name="auditStore">Append-only audit store wired into every mutating verb.</param>
|
||||
@@ -50,6 +55,25 @@ public sealed class ApiKeyAdminCommands
|
||||
IApiKeyPepperProvider pepperProvider,
|
||||
SqliteAuthStoreMigrator migrator,
|
||||
TimeProvider? clock = null)
|
||||
: this(options, adminStore, auditStore, pepperProvider, (IApiKeyStoreMigrator)migrator, clock)
|
||||
{
|
||||
}
|
||||
|
||||
/// <summary>Creates the command set over the supplied stores and options.</summary>
|
||||
/// <param name="options">API-key options (token prefix, store path, ...).</param>
|
||||
/// <param name="adminStore">Mutating store (create / revoke / rotate / delete / list).</param>
|
||||
/// <param name="auditStore">Append-only audit store wired into every mutating verb.</param>
|
||||
/// <param name="pepperProvider">Resolves the pepper used to hash secrets.</param>
|
||||
/// <param name="migrator">Schema migrator used by <see cref="InitDbAsync"/> — whichever store
|
||||
/// provider's implementation (SQLite, SQL Server, ...) DI has registered.</param>
|
||||
/// <param name="clock">Optional clock; defaults to <see cref="TimeProvider.System"/>.</param>
|
||||
public ApiKeyAdminCommands(
|
||||
ApiKeyOptions options,
|
||||
IApiKeyAdminStore adminStore,
|
||||
IApiKeyAuditStore auditStore,
|
||||
IApiKeyPepperProvider pepperProvider,
|
||||
IApiKeyStoreMigrator migrator,
|
||||
TimeProvider? clock = null)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
ArgumentNullException.ThrowIfNull(adminStore);
|
||||
|
||||
+6
-5
@@ -1,17 +1,18 @@
|
||||
using Microsoft.Extensions.Hosting;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.Auth.Abstractions.ApiKeys;
|
||||
using ZB.MOM.WW.Auth.ApiKeys.Sqlite;
|
||||
|
||||
namespace ZB.MOM.WW.Auth.ApiKeys.DependencyInjection;
|
||||
|
||||
/// <summary>
|
||||
/// Runs the API-key SQLite schema migration at application startup when
|
||||
/// <see cref="ApiKeyOptions.RunMigrationsOnStartup"/> is <see langword="true"/>.
|
||||
/// The migration is idempotent, so repeated restarts are safe.
|
||||
/// Runs the registered <see cref="IApiKeyStoreMigrator"/> at application startup when
|
||||
/// <see cref="ApiKeyOptions.RunMigrationsOnStartup"/> is <see langword="true"/>. The migrator
|
||||
/// registered in DI is whichever store provider's implementation won the <c>TryAdd</c> race —
|
||||
/// SQLite by default, or a provider package (e.g. SQL Server) registered before
|
||||
/// <c>AddZbApiKeyAuth</c>. The migration is idempotent, so repeated restarts are safe.
|
||||
/// </summary>
|
||||
internal sealed class ApiKeyMigrationHostedService(
|
||||
SqliteAuthStoreMigrator migrator,
|
||||
IApiKeyStoreMigrator migrator,
|
||||
IOptions<ApiKeyOptions> options) : IHostedService
|
||||
{
|
||||
/// <inheritdoc />
|
||||
|
||||
+4
-2
@@ -61,10 +61,12 @@ public static class ApiKeyServiceCollectionExtensions
|
||||
sp.GetRequiredService<IApiKeyStore>(),
|
||||
sp.GetRequiredService<IApiKeyPepperProvider>()));
|
||||
|
||||
// Migrator: singleton, constructed from the already-registered connection factory.
|
||||
// Needed before any store operations so the schema exists.
|
||||
// Migrator: the SQLite implementation is the default; a provider package that registers
|
||||
// IApiKeyStoreMigrator (and the three stores) BEFORE this call displaces it — that is the
|
||||
// whole mechanism by which AddZbApiKeyAuthSqlServer swaps the backend.
|
||||
services.TryAddSingleton(sp =>
|
||||
new SqliteAuthStoreMigrator(sp.GetRequiredService<AuthSqliteConnectionFactory>()));
|
||||
services.TryAddSingleton<IApiKeyStoreMigrator>(sp => sp.GetRequiredService<SqliteAuthStoreMigrator>());
|
||||
|
||||
// Hosted service that runs migrations on startup when ApiKeyOptions.RunMigrationsOnStartup.
|
||||
services.AddHostedService<ApiKeyMigrationHostedService>();
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
using System.Globalization;
|
||||
using Microsoft.Data.Sqlite;
|
||||
using ZB.MOM.WW.Auth.Abstractions.ApiKeys;
|
||||
|
||||
namespace ZB.MOM.WW.Auth.ApiKeys.Sqlite;
|
||||
|
||||
@@ -11,7 +12,7 @@ public sealed class AuthStoreMigrationException(string message) : InvalidOperati
|
||||
/// is safe to run repeatedly. Refuses to run against a database whose on-disk version
|
||||
/// is newer than this build supports.
|
||||
/// </summary>
|
||||
public sealed class SqliteAuthStoreMigrator(AuthSqliteConnectionFactory connectionFactory)
|
||||
public sealed class SqliteAuthStoreMigrator(AuthSqliteConnectionFactory connectionFactory) : IApiKeyStoreMigrator
|
||||
{
|
||||
/// <summary>Applies the schema migration to the auth store.</summary>
|
||||
/// <param name="cancellationToken">Cancellation token.</param>
|
||||
|
||||
@@ -38,6 +38,9 @@
|
||||
|
||||
<ItemGroup>
|
||||
<InternalsVisibleTo Include="ZB.MOM.WW.Auth.ApiKeys.Tests" />
|
||||
<!-- Task 5's SqliteApiKeyStoreImporter parses SQLite values with SqliteValueParsing (internal)
|
||||
rather than duplicating the parser. -->
|
||||
<InternalsVisibleTo Include="ZB.MOM.WW.Auth.ApiKeys.SqlServer" />
|
||||
</ItemGroup>
|
||||
|
||||
</Project>
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
namespace ZB.MOM.WW.Auth.Ldap.Internal;
|
||||
|
||||
/// <summary>
|
||||
/// Reduces a typed login to the bare account name used by <see cref="Abstractions.Ldap.LdapBindMode.User"/>:
|
||||
/// the qualifier is stripped before the name reaches the bind template and the search filter, so audit
|
||||
/// rows never carry <c>DOMAIN\jdoe</c> for one sign-in and <c>jdoe</c> for the next. Case is NOT
|
||||
/// normalised here: the returned username is canonicalised from the directory entry's
|
||||
/// <c>UserNameAttribute</c> value, so <c>JDoe</c> and <c>jdoe</c> also yield one identity; this name is
|
||||
/// only the fallback when the entry has no such value.
|
||||
/// </summary>
|
||||
internal static class LdapUserName
|
||||
{
|
||||
/// <summary>
|
||||
/// Trims <paramref name="input"/> and, when <paramref name="acceptDomainQualified"/>, strips one
|
||||
/// leading <c>DOMAIN\</c> or one trailing <c>@suffix</c>. Returns <see langword="false"/> (with
|
||||
/// <paramref name="name"/> empty) for blank input, for both forms at once, for a repeated
|
||||
/// separator, for an empty side, or for any qualified form when <paramref name="acceptDomainQualified"/>
|
||||
/// is <see langword="false"/>.
|
||||
/// </summary>
|
||||
public static bool TryNormalize(string? input, bool acceptDomainQualified, out string name)
|
||||
{
|
||||
name = "";
|
||||
if (string.IsNullOrWhiteSpace(input))
|
||||
return false;
|
||||
|
||||
var s = input.Trim();
|
||||
var slash = s.IndexOf('\\');
|
||||
var at = s.IndexOf('@');
|
||||
|
||||
if (slash >= 0 || at >= 0)
|
||||
{
|
||||
if (!acceptDomainQualified || (slash >= 0 && at >= 0))
|
||||
return false;
|
||||
|
||||
if (slash >= 0)
|
||||
{
|
||||
if (slash == 0 || s.IndexOf('\\', slash + 1) >= 0)
|
||||
return false;
|
||||
s = s[(slash + 1)..];
|
||||
}
|
||||
else
|
||||
{
|
||||
// Unlike the slash branch (whose extracted suffix is checked for emptiness by the
|
||||
// final length check below), the extracted piece here is the *prefix* — so an empty
|
||||
// domain part (a trailing "@", e.g. "jdoe@") must be rejected explicitly.
|
||||
if (at == s.Length - 1 || s.IndexOf('@', at + 1) >= 0)
|
||||
return false;
|
||||
s = s[..at];
|
||||
}
|
||||
|
||||
s = s.Trim();
|
||||
}
|
||||
|
||||
if (s.Length == 0)
|
||||
return false;
|
||||
|
||||
name = s;
|
||||
return true;
|
||||
}
|
||||
}
|
||||
@@ -38,6 +38,10 @@ namespace ZB.MOM.WW.Auth.Ldap;
|
||||
/// only mutable state; concurrent logins may race on it, which is benign — the worst outcome
|
||||
/// is one login starting at a stale endpoint and walking on.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// With <c>BindMode = User</c> the service-account bind and the user re-bind are replaced by a
|
||||
/// single bind as the user (<see cref="LdapOptions.UserBindTemplate"/>); see <see cref="CompleteAsUser"/>.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class LdapAuthService : ILdapAuthService
|
||||
{
|
||||
@@ -141,6 +145,21 @@ public sealed class LdapAuthService : ILdapAuthService
|
||||
// filter and the returned result (avoids two identities for one person).
|
||||
username = username.Trim();
|
||||
|
||||
// 3b. An empty password must never reach the directory, in EITHER bind mode: AD answers a
|
||||
// simple bind with an empty password as an UNAUTHENTICATED bind and reports success —
|
||||
// the user bind (or the ServiceAccount-mode re-bind as the user's DN) would admit anyone.
|
||||
// Refused here, before any network call, so no consumer depends on its own edge check.
|
||||
if (string.IsNullOrEmpty(password))
|
||||
return LdapAuthResult.Fail(LdapAuthFailure.BadCredentials);
|
||||
|
||||
// 3c. User-bind mode: reduce the name to the bare account name the bind template and the
|
||||
// search filter share. (The returned username is canonicalised from the entry later.)
|
||||
if (_options.BindMode == LdapBindMode.User
|
||||
&& !LdapUserName.TryNormalize(username, _options.AcceptDomainQualifiedNames, out username))
|
||||
{
|
||||
return LdapAuthResult.Fail(LdapAuthFailure.BadCredentials);
|
||||
}
|
||||
|
||||
// 4. Walk the endpoint list, preferred one first. A single-endpoint configuration
|
||||
// (no FallbackServers) collapses to exactly one attempt — unchanged behaviour.
|
||||
var preferred = Volatile.Read(ref _preferredIndex);
|
||||
@@ -235,74 +254,9 @@ public sealed class LdapAuthService : ILdapAuthService
|
||||
throw new StageFailure(LdapAuthFailure.ServiceAccountBindFailed, "connect");
|
||||
}
|
||||
|
||||
// 6. Service-account bind so we can search for the user's DN. A bind failure
|
||||
// here is a service-account misconfiguration — DISTINCT from a user-credential
|
||||
// failure — so it maps to ServiceAccountBindFailed.
|
||||
try
|
||||
{
|
||||
conn.Bind(_options.ServiceAccountDn, _options.ServiceAccountPassword);
|
||||
}
|
||||
catch (LdapException)
|
||||
{
|
||||
throw new StageFailure(LdapAuthFailure.ServiceAccountBindFailed, "service-bind");
|
||||
}
|
||||
|
||||
// 7. Search for the user entry by the configured username attribute. A search
|
||||
// failure is infrastructure (directory unreachable / unhealthy) — system-side.
|
||||
// NOTE: same enum limitation as Connect above; ServiceAccountBindFailed is the
|
||||
// closest system-side bucket until a DirectoryUnavailable value exists.
|
||||
IReadOnlyList<LdapSearchEntry> entries;
|
||||
try
|
||||
{
|
||||
var filter = $"({_options.UserNameAttribute}={LdapEscaping.Filter(username)})";
|
||||
entries = conn.Search(_options.SearchBase, filter, BuildSearchAttributes());
|
||||
}
|
||||
catch (LdapException)
|
||||
{
|
||||
throw new StageFailure(LdapAuthFailure.ServiceAccountBindFailed, "search");
|
||||
}
|
||||
|
||||
// 8. Require exactly one match. Zero -> UserNotFound; two or more -> AmbiguousUser.
|
||||
// Both are authoritative answers from a healthy directory, so they end the walk.
|
||||
// We never attempt a user bind against an ambiguous DN.
|
||||
if (entries.Count == 0)
|
||||
return LdapAuthResult.Fail(LdapAuthFailure.UserNotFound);
|
||||
if (entries.Count >= 2)
|
||||
return LdapAuthResult.Fail(LdapAuthFailure.AmbiguousUser);
|
||||
|
||||
var entry = entries[0];
|
||||
|
||||
// 9. User bind: re-bind as the resolved DN to verify the password. A bind failure
|
||||
// here is the end user's bad credentials — the directory answered, so this is
|
||||
// returned directly and never retried against another endpoint.
|
||||
try
|
||||
{
|
||||
conn.Bind(entry.Dn, password);
|
||||
}
|
||||
catch (LdapException)
|
||||
{
|
||||
return LdapAuthResult.Fail(LdapAuthFailure.BadCredentials);
|
||||
}
|
||||
|
||||
// 10. Group extraction. Fail closed: an empty/missing group set, or any error while
|
||||
// extracting groups, is a GroupLookupFailed — never a zero-group success. This
|
||||
// too is an answer about the user, not about the endpoint, so it ends the walk.
|
||||
IReadOnlyList<string> groups;
|
||||
try
|
||||
{
|
||||
groups = ExtractGroups(entry);
|
||||
}
|
||||
catch
|
||||
{
|
||||
return LdapAuthResult.Fail(LdapAuthFailure.GroupLookupFailed);
|
||||
}
|
||||
|
||||
if (groups.Count == 0)
|
||||
return LdapAuthResult.Fail(LdapAuthFailure.GroupLookupFailed);
|
||||
|
||||
// 11. Success — and only here, with a verified password and >= 1 group.
|
||||
var displayName = ExtractDisplayName(entry, username);
|
||||
return LdapAuthResult.Success(username, displayName, groups);
|
||||
return _options.BindMode == LdapBindMode.User
|
||||
? CompleteAsUser(conn, username, password)
|
||||
: CompleteWithServiceAccount(conn, username, password);
|
||||
}
|
||||
catch (StageFailure stage)
|
||||
{
|
||||
@@ -323,6 +277,121 @@ public sealed class LdapAuthService : ILdapAuthService
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Steps 6–11 of the 0.3.0 flow: service-account bind, search, user re-bind, groups.</summary>
|
||||
private LdapAuthResult CompleteWithServiceAccount(ZB.MOM.WW.Auth.Ldap.Internal.ILdapConnection conn, string username, string password)
|
||||
{
|
||||
// 6. Service-account bind so we can search for the user's DN. A bind failure here is a
|
||||
// service-account misconfiguration — DISTINCT from a user-credential failure — so it
|
||||
// maps to ServiceAccountBindFailed.
|
||||
try
|
||||
{
|
||||
conn.Bind(_options.ServiceAccountDn, _options.ServiceAccountPassword);
|
||||
}
|
||||
catch (LdapException)
|
||||
{
|
||||
throw new StageFailure(LdapAuthFailure.ServiceAccountBindFailed, "service-bind");
|
||||
}
|
||||
|
||||
// 7–8. Require exactly one match. Zero -> UserNotFound; two or more -> AmbiguousUser.
|
||||
// Both are authoritative answers from a healthy directory, so they end the walk.
|
||||
// We never attempt a user bind against an ambiguous DN.
|
||||
var entries = SearchUser(conn, username);
|
||||
if (entries.Count == 0)
|
||||
return LdapAuthResult.Fail(LdapAuthFailure.UserNotFound);
|
||||
if (entries.Count >= 2)
|
||||
return LdapAuthResult.Fail(LdapAuthFailure.AmbiguousUser);
|
||||
|
||||
var entry = entries[0];
|
||||
|
||||
// 9. User bind: re-bind as the resolved DN to verify the password. A bind failure here is
|
||||
// the end user's bad credentials — the directory answered, so this is returned directly
|
||||
// and never retried against another endpoint.
|
||||
try
|
||||
{
|
||||
conn.Bind(entry.Dn, password);
|
||||
}
|
||||
catch (LdapException)
|
||||
{
|
||||
return LdapAuthResult.Fail(LdapAuthFailure.BadCredentials);
|
||||
}
|
||||
|
||||
return SucceedFromEntry(entry, username);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// <see cref="LdapBindMode.User"/>: the user's own bind is the authentication; the same
|
||||
/// connection then reads their entry. Only result 49 (invalid credentials — AD's answer for a
|
||||
/// wrong password AND for unknown, locked, disabled or expired accounts) is the user's failure
|
||||
/// and final; any other bind error means this endpoint gave no answer, so the walk moves on.
|
||||
/// </summary>
|
||||
private LdapAuthResult CompleteAsUser(ZB.MOM.WW.Auth.Ldap.Internal.ILdapConnection conn, string username, string password)
|
||||
{
|
||||
var bindName = _options.UserBindTemplate.Replace("{0}", username, StringComparison.Ordinal);
|
||||
try
|
||||
{
|
||||
conn.Bind(bindName, password);
|
||||
}
|
||||
catch (LdapException ex) when (ex.ResultCode == LdapException.InvalidCredentials)
|
||||
{
|
||||
return LdapAuthResult.Fail(LdapAuthFailure.BadCredentials);
|
||||
}
|
||||
catch (LdapException)
|
||||
{
|
||||
throw new StageFailure(LdapAuthFailure.ServiceAccountBindFailed, "user-bind");
|
||||
}
|
||||
|
||||
// After a good bind, anything but exactly one readable entry is a failure to establish the
|
||||
// user's groups — never a success, and never UserNotFound (the directory just accepted them).
|
||||
var entries = SearchUser(conn, username);
|
||||
if (entries.Count != 1)
|
||||
return LdapAuthResult.Fail(LdapAuthFailure.GroupLookupFailed);
|
||||
|
||||
return SucceedFromEntry(entries[0], username);
|
||||
}
|
||||
|
||||
/// <summary>Step 7: search for the user entry; a search fault is system-side.</summary>
|
||||
private IReadOnlyList<LdapSearchEntry> SearchUser(ZB.MOM.WW.Auth.Ldap.Internal.ILdapConnection conn, string username)
|
||||
{
|
||||
// A search failure is infrastructure (directory unreachable / unhealthy) — system-side.
|
||||
// NOTE: the LdapAuthFailure enum has no DirectoryUnavailable value, so a failed search maps
|
||||
// to ServiceAccountBindFailed, the closest system-side bucket (same limitation as
|
||||
// Connect), until a DirectoryUnavailable value exists.
|
||||
try
|
||||
{
|
||||
var filter = $"({_options.UserNameAttribute}={LdapEscaping.Filter(username)})";
|
||||
return conn.Search(_options.SearchBase, filter, BuildSearchAttributes());
|
||||
}
|
||||
catch (LdapException)
|
||||
{
|
||||
throw new StageFailure(LdapAuthFailure.ServiceAccountBindFailed, "search");
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Steps 10–11: fail-closed group extraction, then success. The returned username is the
|
||||
/// entry's own <see cref="LdapOptions.UserNameAttribute"/> value (the directory's spelling), so
|
||||
/// <c>JDoe</c> and <c>jdoe</c> sign in as one identity; the typed (normalised) name is the
|
||||
/// fallback when the entry carries no such value.
|
||||
/// </summary>
|
||||
private LdapAuthResult SucceedFromEntry(LdapSearchEntry entry, string username)
|
||||
{
|
||||
IReadOnlyList<string> groups;
|
||||
try
|
||||
{
|
||||
groups = ExtractGroups(entry);
|
||||
}
|
||||
catch
|
||||
{
|
||||
return LdapAuthResult.Fail(LdapAuthFailure.GroupLookupFailed);
|
||||
}
|
||||
|
||||
if (groups.Count == 0)
|
||||
return LdapAuthResult.Fail(LdapAuthFailure.GroupLookupFailed);
|
||||
|
||||
var canonical = ExtractUserName(entry, username);
|
||||
return LdapAuthResult.Success(canonical, ExtractDisplayName(entry, canonical), groups);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Internal control-flow exception that carries an already-mapped <see cref="LdapAuthFailure"/>
|
||||
/// out of a stage to the single fail-closed catch site. Never escapes this type.
|
||||
@@ -337,21 +406,42 @@ public sealed class LdapAuthService : ILdapAuthService
|
||||
|
||||
public LdapAuthFailure Failure { get; }
|
||||
|
||||
/// <summary>Which stage faulted (<c>connect</c> / <c>service-bind</c> / <c>search</c>), for diagnostics.</summary>
|
||||
/// <summary>Which stage faulted (<c>connect</c> / <c>service-bind</c> / <c>user-bind</c> / <c>search</c>), for diagnostics.</summary>
|
||||
public string Stage { get; }
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Builds the distinct attribute list requested from the directory. The display-name
|
||||
/// and group attributes are de-duplicated so we never request the same attribute twice
|
||||
/// when an operator configures them to the same value.
|
||||
/// Builds the distinct attribute list requested from the directory: display name, groups and
|
||||
/// the username attribute (read back as the canonical username). They are de-duplicated,
|
||||
/// case-insensitively, so we never request the same attribute twice when an operator
|
||||
/// configures two of them to the same value.
|
||||
/// </summary>
|
||||
private IReadOnlyList<string> BuildSearchAttributes()
|
||||
{
|
||||
if (string.Equals(_options.DisplayNameAttribute, _options.GroupAttribute, StringComparison.OrdinalIgnoreCase))
|
||||
return new[] { _options.DisplayNameAttribute };
|
||||
var attributes = new List<string>(3);
|
||||
foreach (var name in new[] { _options.DisplayNameAttribute, _options.GroupAttribute, _options.UserNameAttribute })
|
||||
{
|
||||
if (!attributes.Contains(name, StringComparer.OrdinalIgnoreCase))
|
||||
attributes.Add(name);
|
||||
}
|
||||
|
||||
return new[] { _options.DisplayNameAttribute, _options.GroupAttribute };
|
||||
return attributes;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Returns the first non-blank value of the configured username attribute — the directory's
|
||||
/// canonical spelling — falling back to the (already normalised) typed username.
|
||||
/// </summary>
|
||||
private string ExtractUserName(LdapSearchEntry entry, string username)
|
||||
{
|
||||
if (entry.Attributes.TryGetValue(_options.UserNameAttribute, out var values)
|
||||
&& values.Count > 0
|
||||
&& !string.IsNullOrWhiteSpace(values[0]))
|
||||
{
|
||||
return values[0];
|
||||
}
|
||||
|
||||
return username;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
|
||||
@@ -18,8 +18,10 @@ namespace ZB.MOM.WW.Auth.Ldap;
|
||||
/// <item><see cref="LdapOptions.Server"/> must be specified (no sane default host);</item>
|
||||
/// <item><see cref="LdapOptions.SearchBase"/> must be specified (the DN root every
|
||||
/// search runs against);</item>
|
||||
/// <item><see cref="LdapOptions.ServiceAccountDn"/> must be specified — an empty value
|
||||
/// would bind anonymously, defeating the search-then-bind authentication flow;</item>
|
||||
/// <item>in <see cref="LdapBindMode.ServiceAccount"/> mode <see cref="LdapOptions.ServiceAccountDn"/>
|
||||
/// must be specified — an empty value would bind anonymously, defeating the search-then-bind
|
||||
/// authentication flow; in <see cref="LdapBindMode.User"/> mode <see cref="LdapOptions.UserBindTemplate"/>
|
||||
/// must contain <c>{0}</c> exactly once instead;</item>
|
||||
/// <item>every <see cref="LdapOptions.FallbackServers"/> entry must parse as
|
||||
/// <c>"host"</c> or <c>"host:port"</c> — a typo'd backup DC must not lie dormant
|
||||
/// until the primary outage it exists to survive.</item>
|
||||
@@ -63,7 +65,20 @@ public sealed class LdapOptionsValidator : IValidateOptions<LdapOptions>
|
||||
"set it to the search-base DN (e.g. \"dc=example,dc=com\").");
|
||||
}
|
||||
|
||||
if (string.IsNullOrWhiteSpace(options.ServiceAccountDn))
|
||||
if (options.BindMode == LdapBindMode.User)
|
||||
{
|
||||
// User mode binds as the signing-in user, so there is no service account to require.
|
||||
// The template is the bind identity: without exactly one {0} every login would bind as
|
||||
// the same fixed name (or as a name with the user spliced in twice).
|
||||
if (CountPlaceholders(options.UserBindTemplate) != 1)
|
||||
{
|
||||
return ValidateOptionsResult.Fail(
|
||||
$"{nameof(LdapOptions.UserBindTemplate)} must contain {{0}} exactly once when " +
|
||||
$"{nameof(LdapOptions.BindMode)} is {nameof(LdapBindMode.User)} — e.g. \"{{0}}@example.com\" " +
|
||||
"or \"EXAMPLE\\{0}\".");
|
||||
}
|
||||
}
|
||||
else if (string.IsNullOrWhiteSpace(options.ServiceAccountDn))
|
||||
{
|
||||
return ValidateOptionsResult.Fail(
|
||||
$"{nameof(LdapOptions.ServiceAccountDn)} is required but was empty or whitespace — " +
|
||||
@@ -88,4 +103,16 @@ public sealed class LdapOptionsValidator : IValidateOptions<LdapOptions>
|
||||
|
||||
return ValidateOptionsResult.Success;
|
||||
}
|
||||
|
||||
private static int CountPlaceholders(string? template)
|
||||
{
|
||||
if (string.IsNullOrEmpty(template))
|
||||
return 0;
|
||||
|
||||
var count = 0;
|
||||
for (var i = template.IndexOf("{0}", StringComparison.Ordinal); i >= 0;
|
||||
i = template.IndexOf("{0}", i + 3, StringComparison.Ordinal))
|
||||
count++;
|
||||
return count;
|
||||
}
|
||||
}
|
||||
|
||||
+101
@@ -0,0 +1,101 @@
|
||||
using Microsoft.Data.SqlClient;
|
||||
|
||||
namespace ZB.MOM.WW.Auth.ApiKeys.SqlServer.Tests.Live;
|
||||
|
||||
/// <summary>
|
||||
/// Shared connection details for the env-gated live SQL Server suite.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Gated on <c>AUTH_SQLSERVER_CONNSTR</c> and skips cleanly when it is unset, matching the family
|
||||
/// live-test idiom (see <c>ZB.MOM.WW.Secrets.Replicator.SqlServer</c>) — the offline suite must stay
|
||||
/// runnable on any machine. Set it to a database the test is allowed to create and drop a schema in,
|
||||
/// for example <c>Server=localhost,1433;Database=ZbAuthTest;User Id=sa;Password=...;TrustServerCertificate=true</c>.
|
||||
/// Each test class works in a throwaway <em>schema</em> rather than a throwaway database: it needs no
|
||||
/// CREATE DATABASE right, and a virgin schema still reproduces the CREATE SCHEMA race the migrator's
|
||||
/// retry loop exists for.
|
||||
/// </remarks>
|
||||
public static class LiveSqlServer
|
||||
{
|
||||
/// <summary>Environment variable holding the live connection string.</summary>
|
||||
public const string ConnectionStringVariable = "AUTH_SQLSERVER_CONNSTR";
|
||||
|
||||
/// <summary>The configured connection string, or <see langword="null"/> when the suite is not enabled.</summary>
|
||||
public static string? ConnectionString =>
|
||||
Environment.GetEnvironmentVariable(ConnectionStringVariable);
|
||||
|
||||
/// <summary>Whether the live suite should run.</summary>
|
||||
public static bool IsEnabled => !string.IsNullOrWhiteSpace(ConnectionString);
|
||||
|
||||
/// <summary>Skip reason shown when the suite is not enabled.</summary>
|
||||
public const string SkipReason =
|
||||
"Live SQL Server suite disabled; set AUTH_SQLSERVER_CONNSTR to enable.";
|
||||
|
||||
/// <summary>
|
||||
/// Builds options against a throwaway schema so parallel runs and repeat runs cannot collide,
|
||||
/// and so a failed run leaves no trace in a shared database.
|
||||
/// </summary>
|
||||
/// <param name="schemaName">The unique schema name for this test.</param>
|
||||
/// <returns>Options targeting that schema.</returns>
|
||||
public static SqlServerApiKeyOptions OptionsFor(string schemaName) => new()
|
||||
{
|
||||
ConnectionString = ConnectionString!,
|
||||
Schema = schemaName,
|
||||
};
|
||||
|
||||
/// <summary>Generates a unique, allow-list-legal schema name for one test.</summary>
|
||||
/// <returns>A fresh schema name.</returns>
|
||||
public static string NewSchemaName() => $"zbauth_test_{Guid.NewGuid():N}"[..32];
|
||||
|
||||
/// <summary>Drops the throwaway schema and its three tables.</summary>
|
||||
/// <param name="schemaName">The schema to drop.</param>
|
||||
/// <returns>A task that completes when the schema is gone.</returns>
|
||||
public static async Task DropSchemaAsync(string schemaName)
|
||||
{
|
||||
if (!IsEnabled)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
await using var connection = new SqlConnection(ConnectionString);
|
||||
await connection.OpenAsync();
|
||||
|
||||
await using SqlCommand command = connection.CreateCommand();
|
||||
command.CommandText = $"""
|
||||
IF OBJECT_ID(N'[{schemaName}].[api_key_audit]', N'U') IS NOT NULL DROP TABLE [{schemaName}].[api_key_audit];
|
||||
IF OBJECT_ID(N'[{schemaName}].[api_keys]', N'U') IS NOT NULL DROP TABLE [{schemaName}].[api_keys];
|
||||
IF OBJECT_ID(N'[{schemaName}].[schema_version]', N'U') IS NOT NULL DROP TABLE [{schemaName}].[schema_version];
|
||||
IF EXISTS (SELECT 1 FROM sys.schemas WHERE name = N'{schemaName}') EXEC(N'DROP SCHEMA [{schemaName}]');
|
||||
""";
|
||||
|
||||
await command.ExecuteNonQueryAsync();
|
||||
}
|
||||
|
||||
/// <summary>Runs a scalar query against the live database.</summary>
|
||||
/// <param name="commandText">The query text.</param>
|
||||
/// <returns>The first column of the first row, or <see langword="null"/>.</returns>
|
||||
public static async Task<object?> ScalarAsync(string commandText)
|
||||
{
|
||||
await using var connection = new SqlConnection(ConnectionString);
|
||||
await connection.OpenAsync();
|
||||
|
||||
await using SqlCommand command = connection.CreateCommand();
|
||||
command.CommandText = commandText;
|
||||
|
||||
object? value = await command.ExecuteScalarAsync();
|
||||
return value == DBNull.Value ? null : value;
|
||||
}
|
||||
|
||||
/// <summary>Runs a non-query statement against the live database.</summary>
|
||||
/// <param name="commandText">The statement text.</param>
|
||||
/// <returns>A task that completes when the statement has run.</returns>
|
||||
public static async Task ExecuteAsync(string commandText)
|
||||
{
|
||||
await using var connection = new SqlConnection(ConnectionString);
|
||||
await connection.OpenAsync();
|
||||
|
||||
await using SqlCommand command = connection.CreateCommand();
|
||||
command.CommandText = commandText;
|
||||
|
||||
await command.ExecuteNonQueryAsync();
|
||||
}
|
||||
}
|
||||
+499
@@ -0,0 +1,499 @@
|
||||
using ZB.MOM.WW.Auth.Abstractions.ApiKeys;
|
||||
|
||||
namespace ZB.MOM.WW.Auth.ApiKeys.SqlServer.Tests.Live;
|
||||
|
||||
/// <summary>
|
||||
/// Live contract tests for <see cref="SqlServerApiKeyAdminStore"/> (and the audit store it is
|
||||
/// exercised alongside).
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// A one-for-one port of <c>SqliteApiKeyAdminStoreTests</c> — same test names, same assertions — so
|
||||
/// the create/revoke/rotate/scope/enable/delete semantics are pinned identically across the two
|
||||
/// providers. xUnit constructs a new test-class instance per test, so each test provisions and drops
|
||||
/// its own throwaway schema.
|
||||
/// </remarks>
|
||||
[Collection("live-sqlserver")]
|
||||
public sealed class SqlServerApiKeyAdminStoreLiveTests : IAsyncLifetime
|
||||
{
|
||||
private readonly string _schema = LiveSqlServer.NewSchemaName();
|
||||
|
||||
private AuthSqlServerConnectionFactory _factory = null!;
|
||||
private SqlServerApiKeyAdminStore _admin = null!;
|
||||
private SqlServerApiKeyStore _read = null!;
|
||||
private SqlServerApiKeyAuditStore _audit = null!;
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task InitializeAsync()
|
||||
{
|
||||
if (!LiveSqlServer.IsEnabled)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
_factory = new AuthSqlServerConnectionFactory(LiveSqlServer.OptionsFor(_schema));
|
||||
await new SqlServerApiKeyStoreMigrator(_factory).MigrateAsync(CancellationToken.None);
|
||||
_admin = new SqlServerApiKeyAdminStore(_factory);
|
||||
_read = new SqlServerApiKeyStore(_factory);
|
||||
_audit = new SqlServerApiKeyAuditStore(_factory);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public Task DisposeAsync() => LiveSqlServer.DropSchemaAsync(_schema);
|
||||
|
||||
// --- Create ---
|
||||
|
||||
[SkippableFact]
|
||||
public async Task Create_ThenFindByKeyId_ReturnsRecord()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
ApiKeyRecord record = SampleRecord("key-1");
|
||||
|
||||
await _admin.CreateAsync(record, CancellationToken.None);
|
||||
|
||||
ApiKeyRecord? found = await _read.FindByKeyIdAsync("key-1", CancellationToken.None);
|
||||
Assert.NotNull(found);
|
||||
Assert.Equal(record.SecretHash, found!.SecretHash);
|
||||
Assert.True(record.Scopes.SetEquals(found.Scopes));
|
||||
Assert.Equal(record.ConstraintsJson, found.ConstraintsJson);
|
||||
Assert.Null(found.LastUsedUtc);
|
||||
Assert.Null(found.RevokedUtc);
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public async Task Create_WithExpiresUtc_RoundTripsThroughFindAndList()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
var expiry = new DateTimeOffset(2026, 12, 31, 23, 59, 59, TimeSpan.Zero);
|
||||
ApiKeyRecord record = SampleRecord("key-exp") with { ExpiresUtc = expiry };
|
||||
|
||||
await _admin.CreateAsync(record, CancellationToken.None);
|
||||
|
||||
ApiKeyRecord? found = await _read.FindByKeyIdAsync("key-exp", CancellationToken.None);
|
||||
Assert.Equal(expiry, found!.ExpiresUtc);
|
||||
|
||||
IReadOnlyList<ApiKeyListItem> listed = await _admin.ListAsync(CancellationToken.None);
|
||||
ApiKeyListItem item = Assert.Single(listed, k => k.KeyId == "key-exp");
|
||||
Assert.Equal(expiry, item.ExpiresUtc);
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public async Task Create_WithoutExpiresUtc_ReadsBackAsNull()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
await _admin.CreateAsync(SampleRecord("key-1"), CancellationToken.None);
|
||||
|
||||
ApiKeyRecord? found = await _read.FindByKeyIdAsync("key-1", CancellationToken.None);
|
||||
Assert.Null(found!.ExpiresUtc);
|
||||
|
||||
IReadOnlyList<ApiKeyListItem> listed = await _admin.ListAsync(CancellationToken.None);
|
||||
Assert.Null(Assert.Single(listed, k => k.KeyId == "key-1").ExpiresUtc);
|
||||
}
|
||||
|
||||
// --- Revoke ---
|
||||
|
||||
[SkippableFact]
|
||||
public async Task Revoke_ActiveKey_SetsRevokedAndFindActiveReturnsNull()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
await _admin.CreateAsync(SampleRecord("key-1"), CancellationToken.None);
|
||||
var when = new DateTimeOffset(2026, 5, 31, 9, 0, 0, TimeSpan.Zero);
|
||||
|
||||
bool result = await _admin.RevokeAsync("key-1", when, CancellationToken.None);
|
||||
|
||||
Assert.True(result);
|
||||
Assert.Null(await _read.FindActiveByKeyIdAsync("key-1", CancellationToken.None));
|
||||
ApiKeyRecord? found = await _read.FindByKeyIdAsync("key-1", CancellationToken.None);
|
||||
Assert.Equal(when, found!.RevokedUtc);
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public async Task Revoke_UnknownKey_ReturnsFalse()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
bool result = await _admin.RevokeAsync("missing", DateTimeOffset.UtcNow, CancellationToken.None);
|
||||
|
||||
Assert.False(result);
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public async Task Revoke_AlreadyRevoked_ReturnsFalse()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
await _admin.CreateAsync(SampleRecord("key-1"), CancellationToken.None);
|
||||
await _admin.RevokeAsync("key-1", DateTimeOffset.UtcNow, CancellationToken.None);
|
||||
|
||||
bool result = await _admin.RevokeAsync("key-1", DateTimeOffset.UtcNow, CancellationToken.None);
|
||||
|
||||
Assert.False(result);
|
||||
}
|
||||
|
||||
// --- Rotate ---
|
||||
|
||||
[SkippableFact]
|
||||
public async Task Rotate_ChangesHashAndReactivates()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
await _admin.CreateAsync(SampleRecord("key-1"), CancellationToken.None);
|
||||
await _admin.RevokeAsync("key-1", DateTimeOffset.UtcNow, CancellationToken.None);
|
||||
await _read.MarkUsedAsync("key-1", DateTimeOffset.UtcNow, CancellationToken.None);
|
||||
byte[] newHash = [9, 9, 9, 9];
|
||||
|
||||
bool result = await _admin.RotateAsync("key-1", newHash, CancellationToken.None);
|
||||
|
||||
Assert.True(result);
|
||||
ApiKeyRecord? found = await _read.FindByKeyIdAsync("key-1", CancellationToken.None);
|
||||
Assert.Equal(newHash, found!.SecretHash);
|
||||
Assert.Null(found.RevokedUtc);
|
||||
Assert.Null(found.LastUsedUtc);
|
||||
// Reactivated: now visible as active.
|
||||
Assert.NotNull(await _read.FindActiveByKeyIdAsync("key-1", CancellationToken.None));
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public async Task Rotate_UnknownKey_ReturnsFalse()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
bool result = await _admin.RotateAsync("missing", [1], CancellationToken.None);
|
||||
|
||||
Assert.False(result);
|
||||
}
|
||||
|
||||
// --- SetScopes ---
|
||||
|
||||
[SkippableFact]
|
||||
public async Task SetScopesAsync_ReplacesScopes_AndReturnsTrue()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
await _admin.CreateAsync(
|
||||
SampleRecord("key-1") with { Scopes = new HashSet<string>(["a"], StringComparer.Ordinal) },
|
||||
CancellationToken.None);
|
||||
|
||||
bool result = await _admin.SetScopesAsync(
|
||||
"key-1",
|
||||
new HashSet<string>(["b", "c"], StringComparer.Ordinal),
|
||||
CancellationToken.None);
|
||||
|
||||
Assert.True(result);
|
||||
IReadOnlyList<ApiKeyListItem> listed = await _admin.ListAsync(CancellationToken.None);
|
||||
ApiKeyListItem item = Assert.Single(listed, k => k.KeyId == "key-1");
|
||||
Assert.True(item.Scopes.SetEquals(new HashSet<string>(["b", "c"], StringComparer.Ordinal)));
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public async Task SetScopesAsync_UnknownKey_ReturnsFalse()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
bool result = await _admin.SetScopesAsync(
|
||||
"missing",
|
||||
new HashSet<string>(["b"], StringComparer.Ordinal),
|
||||
CancellationToken.None);
|
||||
|
||||
Assert.False(result);
|
||||
}
|
||||
|
||||
// --- SetEnabled ---
|
||||
|
||||
[SkippableFact]
|
||||
public async Task SetEnabledAsync_False_DisablesKey()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
await _admin.CreateAsync(SampleRecord("key-1"), CancellationToken.None);
|
||||
var when = new DateTimeOffset(2026, 5, 31, 9, 0, 0, TimeSpan.Zero);
|
||||
|
||||
bool result = await _admin.SetEnabledAsync("key-1", enabled: false, when, CancellationToken.None);
|
||||
|
||||
Assert.True(result);
|
||||
Assert.Null(await _read.FindActiveByKeyIdAsync("key-1", CancellationToken.None));
|
||||
ApiKeyRecord? found = await _read.FindByKeyIdAsync("key-1", CancellationToken.None);
|
||||
Assert.Equal(when, found!.RevokedUtc);
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public async Task SetEnabledAsync_True_ReenablesKey_WithoutChangingSecret()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
ApiKeyRecord original = SampleRecord("key-1");
|
||||
await _admin.CreateAsync(original, CancellationToken.None);
|
||||
// Record some usage so we can prove last_used_utc is left untouched on re-enable.
|
||||
var used = new DateTimeOffset(2026, 5, 20, 12, 0, 0, TimeSpan.Zero);
|
||||
await _read.MarkUsedAsync("key-1", used, CancellationToken.None);
|
||||
|
||||
// Disable, then re-enable.
|
||||
await _admin.SetEnabledAsync(
|
||||
"key-1", enabled: false, new DateTimeOffset(2026, 5, 31, 9, 0, 0, TimeSpan.Zero), CancellationToken.None);
|
||||
bool result = await _admin.SetEnabledAsync(
|
||||
"key-1", enabled: true, new DateTimeOffset(2026, 6, 1, 9, 0, 0, TimeSpan.Zero), CancellationToken.None);
|
||||
|
||||
Assert.True(result);
|
||||
|
||||
// Active again, and the secret hash + last-used timestamp are unchanged.
|
||||
ApiKeyRecord? active = await _read.FindActiveByKeyIdAsync("key-1", CancellationToken.None);
|
||||
Assert.NotNull(active);
|
||||
Assert.True(active!.SecretHash.SequenceEqual(original.SecretHash));
|
||||
Assert.Null(active.RevokedUtc);
|
||||
Assert.Equal(used, active.LastUsedUtc);
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public async Task SetEnabledAsync_UnknownKey_ReturnsFalse()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
bool result = await _admin.SetEnabledAsync(
|
||||
"missing", enabled: false, DateTimeOffset.UtcNow, CancellationToken.None);
|
||||
|
||||
Assert.False(result);
|
||||
}
|
||||
|
||||
// --- Delete ---
|
||||
|
||||
[SkippableFact]
|
||||
public async Task Delete_ActiveKey_ReturnsFalseAndKeyStillPresent()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
await _admin.CreateAsync(SampleRecord("key-1"), CancellationToken.None);
|
||||
|
||||
bool result = await _admin.DeleteAsync("key-1", CancellationToken.None);
|
||||
|
||||
Assert.False(result);
|
||||
Assert.NotNull(await _read.FindByKeyIdAsync("key-1", CancellationToken.None));
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public async Task Delete_RevokedKey_ReturnsTrueAndKeyGone()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
await _admin.CreateAsync(SampleRecord("key-1"), CancellationToken.None);
|
||||
await _admin.RevokeAsync("key-1", DateTimeOffset.UtcNow, CancellationToken.None);
|
||||
|
||||
bool result = await _admin.DeleteAsync("key-1", CancellationToken.None);
|
||||
|
||||
Assert.True(result);
|
||||
Assert.Null(await _read.FindByKeyIdAsync("key-1", CancellationToken.None));
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public async Task Delete_UnknownKey_ReturnsFalse()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
bool result = await _admin.DeleteAsync("missing", CancellationToken.None);
|
||||
|
||||
Assert.False(result);
|
||||
}
|
||||
|
||||
// --- keyId guard tests ---
|
||||
|
||||
[SkippableTheory]
|
||||
[InlineData(null)]
|
||||
[InlineData("")]
|
||||
[InlineData(" ")]
|
||||
public async Task Revoke_NullOrWhitespaceKeyId_ThrowsArgumentException(string? keyId)
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
// ArgumentNullException (null) and ArgumentException (empty/whitespace) are both acceptable;
|
||||
// ThrowIfNullOrWhiteSpace throws ArgumentNullException for null, ArgumentException for whitespace.
|
||||
await Assert.ThrowsAnyAsync<ArgumentException>(
|
||||
() => _admin.RevokeAsync(keyId!, DateTimeOffset.UtcNow, CancellationToken.None));
|
||||
}
|
||||
|
||||
[SkippableTheory]
|
||||
[InlineData(null)]
|
||||
[InlineData("")]
|
||||
[InlineData(" ")]
|
||||
public async Task Rotate_NullOrWhitespaceKeyId_ThrowsArgumentException(string? keyId)
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
await Assert.ThrowsAnyAsync<ArgumentException>(
|
||||
() => _admin.RotateAsync(keyId!, [1, 2, 3], CancellationToken.None));
|
||||
}
|
||||
|
||||
[SkippableTheory]
|
||||
[InlineData(null)]
|
||||
[InlineData("")]
|
||||
[InlineData(" ")]
|
||||
public async Task Delete_NullOrWhitespaceKeyId_ThrowsArgumentException(string? keyId)
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
await Assert.ThrowsAnyAsync<ArgumentException>(
|
||||
() => _admin.DeleteAsync(keyId!, CancellationToken.None));
|
||||
}
|
||||
|
||||
[SkippableTheory]
|
||||
[InlineData(null)]
|
||||
[InlineData("")]
|
||||
[InlineData(" ")]
|
||||
public async Task SetScopesAsync_NullOrWhitespaceKeyId_ThrowsArgumentException(string? keyId)
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
await Assert.ThrowsAnyAsync<ArgumentException>(
|
||||
() => _admin.SetScopesAsync(
|
||||
keyId!,
|
||||
new HashSet<string>(["read"], StringComparer.Ordinal),
|
||||
CancellationToken.None));
|
||||
}
|
||||
|
||||
[SkippableTheory]
|
||||
[InlineData(null)]
|
||||
[InlineData("")]
|
||||
[InlineData(" ")]
|
||||
public async Task SetEnabledAsync_NullOrWhitespaceKeyId_ThrowsArgumentException(string? keyId)
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
await Assert.ThrowsAnyAsync<ArgumentException>(
|
||||
() => _admin.SetEnabledAsync(keyId!, enabled: false, DateTimeOffset.UtcNow, CancellationToken.None));
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public async Task SetScopesAsync_NullScopes_ThrowsArgumentNullException()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
await _admin.CreateAsync(SampleRecord("key-1"), CancellationToken.None);
|
||||
|
||||
await Assert.ThrowsAsync<ArgumentNullException>(
|
||||
() => _admin.SetScopesAsync("key-1", null!, CancellationToken.None));
|
||||
}
|
||||
|
||||
// --- SetEnabled idempotence ---
|
||||
|
||||
[SkippableFact]
|
||||
public async Task SetEnabledAsync_OnAlreadyActiveKey_ReturnsTrue()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
await _admin.CreateAsync(SampleRecord("key-1"), CancellationToken.None);
|
||||
|
||||
bool result = await _admin.SetEnabledAsync(
|
||||
"key-1", enabled: true, DateTimeOffset.UtcNow, CancellationToken.None);
|
||||
|
||||
Assert.True(result);
|
||||
ApiKeyRecord? active = await _read.FindActiveByKeyIdAsync("key-1", CancellationToken.None);
|
||||
Assert.NotNull(active);
|
||||
Assert.Null(active!.RevokedUtc);
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public async Task SetEnabledAsync_OnAlreadyDisabledKey_OverwritesTimestamp_ReturnsTrue()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
await _admin.CreateAsync(SampleRecord("key-1"), CancellationToken.None);
|
||||
var t1 = new DateTimeOffset(2026, 5, 1, 10, 0, 0, TimeSpan.Zero);
|
||||
var t2 = new DateTimeOffset(2026, 5, 15, 10, 0, 0, TimeSpan.Zero);
|
||||
|
||||
// Disable at t1.
|
||||
await _admin.SetEnabledAsync("key-1", enabled: false, t1, CancellationToken.None);
|
||||
|
||||
// Disable again at a later t2 (idempotent overwrite — no guard on revoked_utc).
|
||||
bool result = await _admin.SetEnabledAsync("key-1", enabled: false, t2, CancellationToken.None);
|
||||
|
||||
Assert.True(result);
|
||||
IReadOnlyList<ApiKeyListItem> listed = await _admin.ListAsync(CancellationToken.None);
|
||||
ApiKeyListItem item = Assert.Single(listed, k => k.KeyId == "key-1");
|
||||
Assert.Equal(t2, item.RevokedUtc);
|
||||
}
|
||||
|
||||
// --- Audit ---
|
||||
|
||||
[SkippableFact]
|
||||
public async Task Audit_AppendThenListRecent_ReturnsEntry()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
var entry = new ApiKeyAuditEntry(
|
||||
KeyId: "key-1",
|
||||
EventType: "created",
|
||||
RemoteAddress: "10.0.0.1",
|
||||
CreatedUtc: new DateTimeOffset(2026, 5, 31, 10, 0, 0, TimeSpan.Zero),
|
||||
Details: "by admin");
|
||||
|
||||
await _audit.AppendAsync(entry, CancellationToken.None);
|
||||
|
||||
IReadOnlyList<ApiKeyAuditEntry> recent = await _audit.ListRecentAsync(10, CancellationToken.None);
|
||||
Assert.Single(recent);
|
||||
Assert.Equal("key-1", recent[0].KeyId);
|
||||
Assert.Equal("created", recent[0].EventType);
|
||||
Assert.Equal("10.0.0.1", recent[0].RemoteAddress);
|
||||
Assert.Equal("by admin", recent[0].Details);
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public async Task Audit_ListRecent_ReturnsNewestFirst()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
await _audit.AppendAsync(
|
||||
new ApiKeyAuditEntry("k", "first", null, DateTimeOffset.UtcNow, null), CancellationToken.None);
|
||||
await _audit.AppendAsync(
|
||||
new ApiKeyAuditEntry("k", "second", null, DateTimeOffset.UtcNow, null), CancellationToken.None);
|
||||
await _audit.AppendAsync(
|
||||
new ApiKeyAuditEntry("k", "third", null, DateTimeOffset.UtcNow, null), CancellationToken.None);
|
||||
|
||||
IReadOnlyList<ApiKeyAuditEntry> recent = await _audit.ListRecentAsync(10, CancellationToken.None);
|
||||
|
||||
Assert.Equal(["third", "second", "first"], recent.Select(e => e.EventType));
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public async Task Audit_ListRecent_RespectsLimit()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
for (int i = 0; i < 5; i++)
|
||||
{
|
||||
await _audit.AppendAsync(
|
||||
new ApiKeyAuditEntry("k", $"e{i}", null, DateTimeOffset.UtcNow, null), CancellationToken.None);
|
||||
}
|
||||
|
||||
IReadOnlyList<ApiKeyAuditEntry> recent = await _audit.ListRecentAsync(2, CancellationToken.None);
|
||||
|
||||
Assert.Equal(2, recent.Count);
|
||||
Assert.Equal(["e4", "e3"], recent.Select(e => e.EventType));
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public async Task Audit_NullableFields_RoundTripAsNull()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
await _audit.AppendAsync(
|
||||
new ApiKeyAuditEntry(null, "anon", null, DateTimeOffset.UtcNow, null), CancellationToken.None);
|
||||
|
||||
IReadOnlyList<ApiKeyAuditEntry> recent = await _audit.ListRecentAsync(10, CancellationToken.None);
|
||||
Assert.Single(recent);
|
||||
Assert.Null(recent[0].KeyId);
|
||||
Assert.Null(recent[0].RemoteAddress);
|
||||
Assert.Null(recent[0].Details);
|
||||
}
|
||||
|
||||
private static ApiKeyRecord SampleRecord(string keyId) => new(
|
||||
KeyId: keyId,
|
||||
KeyPrefix: "mxgw_ab12",
|
||||
SecretHash: [1, 2, 3, 4, 5, 6, 7, 8],
|
||||
DisplayName: "Test Key " + keyId,
|
||||
Scopes: new HashSet<string>(["read", "write"], StringComparer.Ordinal),
|
||||
ConstraintsJson: """{"ipAllow":["10.0.0.0/8"]}""",
|
||||
CreatedUtc: new DateTimeOffset(2026, 5, 1, 8, 30, 0, TimeSpan.Zero),
|
||||
LastUsedUtc: null,
|
||||
RevokedUtc: null);
|
||||
}
|
||||
+74
@@ -0,0 +1,74 @@
|
||||
using ZB.MOM.WW.Auth.Abstractions.ApiKeys;
|
||||
|
||||
namespace ZB.MOM.WW.Auth.ApiKeys.SqlServer.Tests.Live;
|
||||
|
||||
/// <summary>
|
||||
/// Live tests for <see cref="SqlServerApiKeyAuditStore"/> that the ported SQLite suite does not
|
||||
/// already cover: newest-first ordering under a limit, and the non-positive-limit short circuit.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// xUnit constructs a new test-class instance per test, so each test provisions and drops its own
|
||||
/// throwaway schema.
|
||||
/// </remarks>
|
||||
[Collection("live-sqlserver")]
|
||||
public sealed class SqlServerApiKeyAuditStoreLiveTests : IAsyncLifetime
|
||||
{
|
||||
private readonly string _schema = LiveSqlServer.NewSchemaName();
|
||||
|
||||
private SqlServerApiKeyAuditStore _audit = null!;
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task InitializeAsync()
|
||||
{
|
||||
if (!LiveSqlServer.IsEnabled)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
var factory = new AuthSqlServerConnectionFactory(LiveSqlServer.OptionsFor(_schema));
|
||||
await new SqlServerApiKeyStoreMigrator(factory).MigrateAsync(CancellationToken.None);
|
||||
_audit = new SqlServerApiKeyAuditStore(factory);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public Task DisposeAsync() => LiveSqlServer.DropSchemaAsync(_schema);
|
||||
|
||||
[SkippableFact]
|
||||
public async Task Append_ThenListRecent_ReturnsNewestFirstWithinLimit()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
// Every row carries the SAME created_utc, so the newest-first order can only come from the
|
||||
// IDENTITY audit_id the ORDER BY names — the timestamp cannot accidentally supply it.
|
||||
var when = new DateTimeOffset(2026, 7, 1, 0, 0, 0, TimeSpan.Zero);
|
||||
for (int i = 0; i < 6; i++)
|
||||
{
|
||||
await _audit.AppendAsync(
|
||||
new ApiKeyAuditEntry("k", $"e{i}", "10.0.0.1", when, $"detail {i}"),
|
||||
CancellationToken.None);
|
||||
}
|
||||
|
||||
IReadOnlyList<ApiKeyAuditEntry> recent = await _audit.ListRecentAsync(3, CancellationToken.None);
|
||||
|
||||
Assert.Equal(3, recent.Count);
|
||||
Assert.Equal(["e5", "e4", "e3"], recent.Select(e => e.EventType));
|
||||
Assert.Equal("detail 5", recent[0].Details);
|
||||
Assert.Equal(when, recent[0].CreatedUtc);
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public async Task ListRecent_NonPositiveLimit_ReturnsEmpty()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
await _audit.AppendAsync(
|
||||
new ApiKeyAuditEntry("k", "created", null, DateTimeOffset.UtcNow, null),
|
||||
CancellationToken.None);
|
||||
|
||||
// A non-positive limit short circuits WITHOUT touching the database — SELECT TOP (0) would
|
||||
// be legal but SELECT TOP (-1) is a syntax error, so the guard is load-bearing here in a way
|
||||
// it is not for SQLite's LIMIT.
|
||||
Assert.Empty(await _audit.ListRecentAsync(0, CancellationToken.None));
|
||||
Assert.Empty(await _audit.ListRecentAsync(-5, CancellationToken.None));
|
||||
}
|
||||
}
|
||||
+340
@@ -0,0 +1,340 @@
|
||||
using System.Data;
|
||||
using Microsoft.Data.SqlClient;
|
||||
using ZB.MOM.WW.Auth.Abstractions.ApiKeys;
|
||||
using ZB.MOM.WW.Auth.ApiKeys.Sqlite;
|
||||
|
||||
namespace ZB.MOM.WW.Auth.ApiKeys.SqlServer.Tests.Live;
|
||||
|
||||
/// <summary>
|
||||
/// Live contract tests for <see cref="SqlServerApiKeyStore"/>.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// A one-for-one port of <c>SqliteApiKeyStoreTests</c> (same test names, same assertions) so the two
|
||||
/// providers are pinned to identical semantics: whichever store a host composes, a row means the
|
||||
/// same thing. xUnit constructs a new test-class instance per test, so each test provisions and
|
||||
/// drops its own throwaway schema — the isolation the SQLite suite gets from a per-test temp file.
|
||||
/// </remarks>
|
||||
[Collection("live-sqlserver")]
|
||||
public sealed class SqlServerApiKeyStoreLiveTests : IAsyncLifetime
|
||||
{
|
||||
private readonly string _schema = LiveSqlServer.NewSchemaName();
|
||||
|
||||
private AuthSqlServerConnectionFactory _factory = null!;
|
||||
private SqlServerApiKeyStore _store = null!;
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task InitializeAsync()
|
||||
{
|
||||
if (!LiveSqlServer.IsEnabled)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
_factory = new AuthSqlServerConnectionFactory(LiveSqlServer.OptionsFor(_schema));
|
||||
await new SqlServerApiKeyStoreMigrator(_factory).MigrateAsync(CancellationToken.None);
|
||||
_store = new SqlServerApiKeyStore(_factory);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public Task DisposeAsync() => LiveSqlServer.DropSchemaAsync(_schema);
|
||||
|
||||
[SkippableFact]
|
||||
public async Task FindByKeyId_AfterInsert_ReturnsEqualRecord()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
ApiKeyRecord record = SampleRecord("key-1");
|
||||
await InsertAsync(record);
|
||||
|
||||
ApiKeyRecord? found = await _store.FindByKeyIdAsync("key-1", CancellationToken.None);
|
||||
|
||||
Assert.NotNull(found);
|
||||
AssertRecordEqual(record, found!);
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public async Task FindByKeyId_ReturnsRevokedRecord()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
ApiKeyRecord record = SampleRecord("key-revoked") with
|
||||
{
|
||||
RevokedUtc = new DateTimeOffset(2026, 1, 2, 3, 4, 5, TimeSpan.Zero),
|
||||
};
|
||||
await InsertAsync(record);
|
||||
|
||||
ApiKeyRecord? found = await _store.FindByKeyIdAsync("key-revoked", CancellationToken.None);
|
||||
|
||||
Assert.NotNull(found);
|
||||
Assert.NotNull(found!.RevokedUtc);
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public async Task FindActiveByKeyId_RevokedKey_ReturnsNull()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
ApiKeyRecord record = SampleRecord("key-revoked") with
|
||||
{
|
||||
RevokedUtc = new DateTimeOffset(2026, 1, 2, 3, 4, 5, TimeSpan.Zero),
|
||||
};
|
||||
await InsertAsync(record);
|
||||
|
||||
ApiKeyRecord? found = await _store.FindActiveByKeyIdAsync("key-revoked", CancellationToken.None);
|
||||
|
||||
Assert.Null(found);
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public async Task FindActiveByKeyId_ActiveKey_ReturnsRecord()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
await InsertAsync(SampleRecord("key-active"));
|
||||
|
||||
ApiKeyRecord? found = await _store.FindActiveByKeyIdAsync("key-active", CancellationToken.None);
|
||||
|
||||
Assert.NotNull(found);
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public async Task FindByKeyId_UnknownKey_ReturnsNull()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
ApiKeyRecord? found = await _store.FindByKeyIdAsync("missing", CancellationToken.None);
|
||||
|
||||
Assert.Null(found);
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public async Task MarkUsed_ActiveKey_UpdatesLastUsed()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
await InsertAsync(SampleRecord("key-active"));
|
||||
var when = new DateTimeOffset(2026, 5, 31, 12, 0, 0, TimeSpan.Zero);
|
||||
|
||||
await _store.MarkUsedAsync("key-active", when, CancellationToken.None);
|
||||
|
||||
ApiKeyRecord? found = await _store.FindByKeyIdAsync("key-active", CancellationToken.None);
|
||||
Assert.Equal(when, found!.LastUsedUtc);
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public async Task MarkUsed_RevokedKey_DoesNotUpdateLastUsed()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
ApiKeyRecord record = SampleRecord("key-revoked") with
|
||||
{
|
||||
RevokedUtc = new DateTimeOffset(2026, 1, 2, 3, 4, 5, TimeSpan.Zero),
|
||||
};
|
||||
await InsertAsync(record);
|
||||
var when = new DateTimeOffset(2026, 5, 31, 12, 0, 0, TimeSpan.Zero);
|
||||
|
||||
await _store.MarkUsedAsync("key-revoked", when, CancellationToken.None);
|
||||
|
||||
ApiKeyRecord? found = await _store.FindByKeyIdAsync("key-revoked", CancellationToken.None);
|
||||
Assert.Null(found!.LastUsedUtc);
|
||||
}
|
||||
|
||||
// --- SQL-Server-specific: DATETIMEOFFSET(7) must not truncate to milliseconds ---
|
||||
|
||||
[SkippableFact]
|
||||
public async Task FindByKeyId_RoundTripsDateTimeOffsetToTheTick()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
// Sub-millisecond ticks: SQLite persists round-trip ("O") text and keeps them for free,
|
||||
// whereas SQL Server keeps them only because the columns are DATETIMEOFFSET(7) and the
|
||||
// parameters are bound as SqlDbType.DateTimeOffset. A DATETIME/DATETIME2(3) column or an
|
||||
// untyped parameter would silently round here.
|
||||
var created = new DateTimeOffset(2026, 3, 4, 5, 6, 7, TimeSpan.Zero).AddTicks(1234567);
|
||||
var expires = new DateTimeOffset(2027, 3, 4, 5, 6, 7, TimeSpan.Zero).AddTicks(7654321);
|
||||
ApiKeyRecord record = SampleRecord("key-ticks") with { CreatedUtc = created, ExpiresUtc = expires };
|
||||
await InsertAsync(record);
|
||||
|
||||
ApiKeyRecord? found = await _store.FindByKeyIdAsync("key-ticks", CancellationToken.None);
|
||||
|
||||
Assert.NotNull(found);
|
||||
Assert.Equal(created, found!.CreatedUtc);
|
||||
Assert.Equal(created.Ticks, found.CreatedUtc.Ticks);
|
||||
Assert.Equal(expires, found.ExpiresUtc);
|
||||
Assert.Equal(expires.Ticks, found.ExpiresUtc!.Value.Ticks);
|
||||
}
|
||||
|
||||
// --- keyId guard tests ---
|
||||
|
||||
[SkippableTheory]
|
||||
[InlineData(null)]
|
||||
[InlineData("")]
|
||||
[InlineData(" ")]
|
||||
public async Task FindByKeyId_NullOrWhitespaceKeyId_ThrowsArgumentException(string? keyId)
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
// ArgumentNullException (null) and ArgumentException (empty/whitespace) are both acceptable;
|
||||
// ThrowIfNullOrWhiteSpace throws ArgumentNullException for null, ArgumentException for whitespace.
|
||||
await Assert.ThrowsAnyAsync<ArgumentException>(
|
||||
() => _store.FindByKeyIdAsync(keyId!, CancellationToken.None));
|
||||
}
|
||||
|
||||
[SkippableTheory]
|
||||
[InlineData(null)]
|
||||
[InlineData("")]
|
||||
[InlineData(" ")]
|
||||
public async Task FindActiveByKeyId_NullOrWhitespaceKeyId_ThrowsArgumentException(string? keyId)
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
await Assert.ThrowsAnyAsync<ArgumentException>(
|
||||
() => _store.FindActiveByKeyIdAsync(keyId!, CancellationToken.None));
|
||||
}
|
||||
|
||||
[SkippableTheory]
|
||||
[InlineData(null)]
|
||||
[InlineData("")]
|
||||
[InlineData(" ")]
|
||||
public async Task MarkUsed_NullOrWhitespaceKeyId_ThrowsArgumentException(string? keyId)
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
await Assert.ThrowsAnyAsync<ArgumentException>(
|
||||
() => _store.MarkUsedAsync(keyId!, DateTimeOffset.UtcNow, CancellationToken.None));
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public void ScopeSerializer_RoundTripsAndSortsOrdinally()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
var unsorted = new HashSet<string>(["zeta", "alpha", "mike"], StringComparer.Ordinal);
|
||||
var differentOrder = new HashSet<string>(["mike", "zeta", "alpha"], StringComparer.Ordinal);
|
||||
|
||||
string a = ScopeSerializer.Serialize(unsorted);
|
||||
string b = ScopeSerializer.Serialize(differentOrder);
|
||||
|
||||
// Equal sets must produce identical column text regardless of insertion order.
|
||||
Assert.Equal(a, b);
|
||||
Assert.Equal("""["alpha","mike","zeta"]""", a);
|
||||
|
||||
IReadOnlySet<string> roundTripped = ScopeSerializer.Deserialize(a);
|
||||
Assert.True(roundTripped.SetEquals(unsorted));
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public void ScopeSerializer_DeserializeNullOrEmpty_ReturnsEmptySet()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
Assert.Empty(ScopeSerializer.Deserialize(null));
|
||||
Assert.Empty(ScopeSerializer.Deserialize(""));
|
||||
}
|
||||
|
||||
// --- Auth-003: corrupt scopes JSON must fail closed (empty set), never throw JsonException ---
|
||||
|
||||
[SkippableTheory]
|
||||
[InlineData("not json at all")]
|
||||
[InlineData("{")]
|
||||
[InlineData("{\"a\":1}")] // valid JSON, but an object, not a string[]
|
||||
[InlineData("42")] // valid JSON, but a number
|
||||
[InlineData("[\"read\",")] // truncated/partial write
|
||||
public void ScopeSerializer_DeserializeMalformed_ReturnsEmptySet_DoesNotThrow(string value)
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
// A poisoned scopes column (tampering, partial write, format change, buggy writer) must
|
||||
// degrade to a zero-scope set rather than throwing on the verification hot path.
|
||||
IReadOnlySet<string> scopes = ScopeSerializer.Deserialize(value);
|
||||
Assert.Empty(scopes);
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public async Task FindByKeyId_CorruptScopesColumn_ReturnsRecordWithEmptyScopes_DoesNotThrow()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
// Insert a row whose scopes column holds malformed (non-array) JSON, then read it through
|
||||
// the store. The store must NOT propagate a JsonException out of FindByKeyIdAsync (which the
|
||||
// verifier relies on for its "only exception path is cancellation" contract).
|
||||
await InsertWithRawScopesAsync("key-corrupt", scopesJson: "{ this is not valid json");
|
||||
|
||||
ApiKeyRecord? found = await _store.FindByKeyIdAsync("key-corrupt", CancellationToken.None);
|
||||
|
||||
Assert.NotNull(found);
|
||||
Assert.Empty(found!.Scopes);
|
||||
}
|
||||
|
||||
private static ApiKeyRecord SampleRecord(string keyId) => new(
|
||||
KeyId: keyId,
|
||||
KeyPrefix: "mxgw_ab12",
|
||||
SecretHash: [1, 2, 3, 4, 5, 6, 7, 8],
|
||||
DisplayName: "Test Key " + keyId,
|
||||
Scopes: new HashSet<string>(["read", "write"], StringComparer.Ordinal),
|
||||
ConstraintsJson: """{"ipAllow":["10.0.0.0/8"]}""",
|
||||
CreatedUtc: new DateTimeOffset(2026, 5, 1, 8, 30, 0, TimeSpan.Zero),
|
||||
LastUsedUtc: null,
|
||||
RevokedUtc: null);
|
||||
|
||||
private static void AssertRecordEqual(ApiKeyRecord expected, ApiKeyRecord actual)
|
||||
{
|
||||
Assert.Equal(expected.KeyId, actual.KeyId);
|
||||
Assert.Equal(expected.KeyPrefix, actual.KeyPrefix);
|
||||
Assert.Equal(expected.SecretHash, actual.SecretHash);
|
||||
Assert.Equal(expected.DisplayName, actual.DisplayName);
|
||||
Assert.True(expected.Scopes.SetEquals(actual.Scopes));
|
||||
Assert.Equal(expected.ConstraintsJson, actual.ConstraintsJson);
|
||||
Assert.Equal(expected.CreatedUtc, actual.CreatedUtc);
|
||||
Assert.Equal(expected.LastUsedUtc, actual.LastUsedUtc);
|
||||
Assert.Equal(expected.RevokedUtc, actual.RevokedUtc);
|
||||
}
|
||||
|
||||
private async Task InsertAsync(ApiKeyRecord record)
|
||||
{
|
||||
await using SqlConnection connection =
|
||||
await _factory.OpenConnectionAsync(CancellationToken.None);
|
||||
await using SqlCommand command = _factory.CreateCommand(connection, $"""
|
||||
INSERT INTO [{_schema}].[{SqlServerAuthSchema.ApiKeysTable}] (
|
||||
key_id, key_prefix, secret_hash, display_name, scopes,
|
||||
constraints, created_utc, last_used_utc, revoked_utc, expires_utc)
|
||||
VALUES (
|
||||
@key_id, @key_prefix, @secret_hash, @display_name, @scopes,
|
||||
@constraints, @created_utc, @last_used_utc, @revoked_utc, @expires_utc);
|
||||
""");
|
||||
command.Parameters.Add("@key_id", SqlDbType.NVarChar, 64).Value = record.KeyId;
|
||||
command.Parameters.Add("@key_prefix", SqlDbType.NVarChar, 16).Value = record.KeyPrefix;
|
||||
command.Parameters.Add("@secret_hash", SqlDbType.VarBinary, 64).Value = record.SecretHash;
|
||||
command.Parameters.Add("@display_name", SqlDbType.NVarChar, 256).Value = record.DisplayName;
|
||||
command.Parameters.Add("@scopes", SqlDbType.NVarChar, -1).Value = ScopeSerializer.Serialize(record.Scopes);
|
||||
command.Parameters.Add("@constraints", SqlDbType.NVarChar, -1).Value = (object?)record.ConstraintsJson ?? DBNull.Value;
|
||||
command.Parameters.Add("@created_utc", SqlDbType.DateTimeOffset).Value = record.CreatedUtc;
|
||||
command.Parameters.Add("@last_used_utc", SqlDbType.DateTimeOffset).Value = (object?)record.LastUsedUtc ?? DBNull.Value;
|
||||
command.Parameters.Add("@revoked_utc", SqlDbType.DateTimeOffset).Value = (object?)record.RevokedUtc ?? DBNull.Value;
|
||||
command.Parameters.Add("@expires_utc", SqlDbType.DateTimeOffset).Value = (object?)record.ExpiresUtc ?? DBNull.Value;
|
||||
await command.ExecuteNonQueryAsync(CancellationToken.None);
|
||||
}
|
||||
|
||||
private async Task InsertWithRawScopesAsync(string keyId, string scopesJson)
|
||||
{
|
||||
// Writes the scopes column verbatim (NOT via ScopeSerializer.Serialize) so a malformed
|
||||
// value can be persisted to simulate tampering / a partial or buggy write.
|
||||
await using SqlConnection connection =
|
||||
await _factory.OpenConnectionAsync(CancellationToken.None);
|
||||
await using SqlCommand command = _factory.CreateCommand(connection, $"""
|
||||
INSERT INTO [{_schema}].[{SqlServerAuthSchema.ApiKeysTable}] (
|
||||
key_id, key_prefix, secret_hash, display_name, scopes,
|
||||
constraints, created_utc, last_used_utc, revoked_utc, expires_utc)
|
||||
VALUES (
|
||||
@key_id, @key_prefix, @secret_hash, @display_name, @scopes,
|
||||
NULL, @created_utc, NULL, NULL, NULL);
|
||||
""");
|
||||
command.Parameters.Add("@key_id", SqlDbType.NVarChar, 64).Value = keyId;
|
||||
command.Parameters.Add("@key_prefix", SqlDbType.NVarChar, 16).Value = "mxgw";
|
||||
command.Parameters.Add("@secret_hash", SqlDbType.VarBinary, 64).Value = new byte[] { 1, 2, 3 };
|
||||
command.Parameters.Add("@display_name", SqlDbType.NVarChar, 256).Value = "Corrupt Key";
|
||||
command.Parameters.Add("@scopes", SqlDbType.NVarChar, -1).Value = scopesJson;
|
||||
command.Parameters.Add("@created_utc", SqlDbType.DateTimeOffset).Value = DateTimeOffset.UnixEpoch;
|
||||
await command.ExecuteNonQueryAsync(CancellationToken.None);
|
||||
}
|
||||
}
|
||||
+145
@@ -0,0 +1,145 @@
|
||||
using System.Globalization;
|
||||
using ZB.MOM.WW.Auth.ApiKeys.Sqlite;
|
||||
|
||||
namespace ZB.MOM.WW.Auth.ApiKeys.SqlServer.Tests.Live;
|
||||
|
||||
/// <summary>
|
||||
/// Live provisioning tests for <see cref="SqlServerApiKeyStoreMigrator"/>.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Every test provisions its own throwaway schema — the concurrency case in particular needs a
|
||||
/// virgin one, because the race the retry loop exists for only happens while the schema and tables
|
||||
/// do not yet exist. All schemas created by the class are dropped in <see cref="DisposeAsync"/>.
|
||||
/// </remarks>
|
||||
[Collection("live-sqlserver")]
|
||||
public sealed class SqlServerApiKeyStoreMigratorLiveTests : IAsyncLifetime
|
||||
{
|
||||
private readonly List<string> _schemas = [];
|
||||
|
||||
/// <inheritdoc />
|
||||
public Task InitializeAsync() => Task.CompletedTask;
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task DisposeAsync()
|
||||
{
|
||||
foreach (string schema in _schemas)
|
||||
{
|
||||
await LiveSqlServer.DropSchemaAsync(schema);
|
||||
}
|
||||
}
|
||||
|
||||
private string NewSchema()
|
||||
{
|
||||
string schema = LiveSqlServer.NewSchemaName();
|
||||
_schemas.Add(schema);
|
||||
return schema;
|
||||
}
|
||||
|
||||
private static SqlServerApiKeyStoreMigrator MigratorFor(string schema) =>
|
||||
new(new AuthSqlServerConnectionFactory(LiveSqlServer.OptionsFor(schema)));
|
||||
|
||||
[SkippableFact]
|
||||
public async Task Migrate_OnEmptyDatabase_CreatesSchemaTablesAndVersionRow()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
string schema = NewSchema();
|
||||
|
||||
await MigratorFor(schema).MigrateAsync(CancellationToken.None);
|
||||
|
||||
Assert.Equal(
|
||||
1,
|
||||
Convert.ToInt32(
|
||||
await LiveSqlServer.ScalarAsync(
|
||||
$"SELECT COUNT(*) FROM sys.schemas WHERE name = N'{schema}'"),
|
||||
CultureInfo.InvariantCulture));
|
||||
|
||||
foreach (string table in new[]
|
||||
{
|
||||
SqlServerAuthSchema.SchemaVersionTable,
|
||||
SqlServerAuthSchema.ApiKeysTable,
|
||||
SqlServerAuthSchema.ApiKeyAuditTable,
|
||||
})
|
||||
{
|
||||
Assert.NotNull(
|
||||
await LiveSqlServer.ScalarAsync($"SELECT OBJECT_ID(N'[{schema}].[{table}]', N'U')"));
|
||||
}
|
||||
|
||||
Assert.Equal(
|
||||
SqlServerAuthSchema.CurrentVersion,
|
||||
Convert.ToInt32(
|
||||
await LiveSqlServer.ScalarAsync(
|
||||
$"SELECT version FROM [{schema}].[{SqlServerAuthSchema.SchemaVersionTable}] WHERE id = 1"),
|
||||
CultureInfo.InvariantCulture));
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public async Task Migrate_Twice_IsIdempotent()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
string schema = NewSchema();
|
||||
SqlServerApiKeyStoreMigrator migrator = MigratorFor(schema);
|
||||
|
||||
await migrator.MigrateAsync(CancellationToken.None);
|
||||
await migrator.MigrateAsync(CancellationToken.None);
|
||||
|
||||
Assert.Equal(
|
||||
1,
|
||||
Convert.ToInt32(
|
||||
await LiveSqlServer.ScalarAsync(
|
||||
$"SELECT COUNT(*) FROM [{schema}].[{SqlServerAuthSchema.SchemaVersionTable}]"),
|
||||
CultureInfo.InvariantCulture));
|
||||
Assert.Equal(
|
||||
SqlServerAuthSchema.CurrentVersion,
|
||||
Convert.ToInt32(
|
||||
await LiveSqlServer.ScalarAsync(
|
||||
$"SELECT version FROM [{schema}].[{SqlServerAuthSchema.SchemaVersionTable}] WHERE id = 1"),
|
||||
CultureInfo.InvariantCulture));
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public async Task Migrate_Concurrently_FromEightTasks_AllSucceed()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
// A virgin schema provisioned by eight nodes at once — the central-pair first boot, scaled
|
||||
// up. Each task gets its own factory (and therefore its own connection) so the DDL really
|
||||
// does run on eight sessions.
|
||||
string schema = NewSchema();
|
||||
|
||||
await Task.WhenAll(Enumerable.Range(0, 8).Select(_ =>
|
||||
Task.Run(() => MigratorFor(schema).MigrateAsync(CancellationToken.None))));
|
||||
|
||||
Assert.Equal(
|
||||
1,
|
||||
Convert.ToInt32(
|
||||
await LiveSqlServer.ScalarAsync(
|
||||
$"SELECT COUNT(*) FROM [{schema}].[{SqlServerAuthSchema.SchemaVersionTable}]"),
|
||||
CultureInfo.InvariantCulture));
|
||||
Assert.Equal(
|
||||
SqlServerAuthSchema.CurrentVersion,
|
||||
Convert.ToInt32(
|
||||
await LiveSqlServer.ScalarAsync(
|
||||
$"SELECT version FROM [{schema}].[{SqlServerAuthSchema.SchemaVersionTable}] WHERE id = 1"),
|
||||
CultureInfo.InvariantCulture));
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public async Task Migrate_WhenStoredVersionIsNewer_Throws()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
string schema = NewSchema();
|
||||
SqlServerApiKeyStoreMigrator migrator = MigratorFor(schema);
|
||||
|
||||
await migrator.MigrateAsync(CancellationToken.None);
|
||||
await LiveSqlServer.ExecuteAsync(
|
||||
$"UPDATE [{schema}].[{SqlServerAuthSchema.SchemaVersionTable}] SET version = 99 WHERE id = 1");
|
||||
|
||||
AuthStoreMigrationException exception = await Assert.ThrowsAsync<AuthStoreMigrationException>(
|
||||
() => migrator.MigrateAsync(CancellationToken.None));
|
||||
|
||||
Assert.Contains("99", exception.Message, StringComparison.Ordinal);
|
||||
}
|
||||
}
|
||||
+283
@@ -0,0 +1,283 @@
|
||||
using Microsoft.Data.Sqlite;
|
||||
using ZB.MOM.WW.Auth.Abstractions.ApiKeys;
|
||||
using ZB.MOM.WW.Auth.ApiKeys.Sqlite;
|
||||
|
||||
namespace ZB.MOM.WW.Auth.ApiKeys.SqlServer.Tests.Live;
|
||||
|
||||
/// <summary>
|
||||
/// Live tests for <see cref="SqliteApiKeyStoreImporter"/> — the one-shot per-node SQLite to shared
|
||||
/// SQL Server copy.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Each test owns a throwaway SQL Server schema AND a throwaway SQLite file under the temp
|
||||
/// directory; both are torn down in <see cref="DisposeAsync"/>, so a failed run leaves nothing
|
||||
/// behind on a shared database or the developer's disk.
|
||||
/// </remarks>
|
||||
[Collection("live-sqlserver")]
|
||||
public sealed class SqliteApiKeyStoreImporterLiveTests : IAsyncLifetime
|
||||
{
|
||||
private readonly string _schema = LiveSqlServer.NewSchemaName();
|
||||
private readonly string _sqlitePath =
|
||||
Path.Combine(Path.GetTempPath(), $"zbauth-import-{Guid.NewGuid():N}.db");
|
||||
|
||||
private AuthSqlServerConnectionFactory _factory = null!;
|
||||
private SqlServerApiKeyAdminStore _sqlAdmin = null!;
|
||||
private SqlServerApiKeyStore _sqlRead = null!;
|
||||
private SqlServerApiKeyAuditStore _sqlAudit = null!;
|
||||
private SqliteApiKeyStoreImporter _importer = null!;
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task InitializeAsync()
|
||||
{
|
||||
if (!LiveSqlServer.IsEnabled)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
_factory = new AuthSqlServerConnectionFactory(LiveSqlServer.OptionsFor(_schema));
|
||||
await new SqlServerApiKeyStoreMigrator(_factory).MigrateAsync(CancellationToken.None);
|
||||
_sqlAdmin = new SqlServerApiKeyAdminStore(_factory);
|
||||
_sqlRead = new SqlServerApiKeyStore(_factory);
|
||||
_sqlAudit = new SqlServerApiKeyAuditStore(_factory);
|
||||
_importer = new SqliteApiKeyStoreImporter(_factory);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task DisposeAsync()
|
||||
{
|
||||
// SQLite pools connections by connection string; the handle must be released before the
|
||||
// file can be removed cleanly.
|
||||
SqliteConnection.ClearAllPools();
|
||||
|
||||
foreach (string path in new[] { _sqlitePath, _sqlitePath + "-wal", _sqlitePath + "-shm" })
|
||||
{
|
||||
if (File.Exists(path))
|
||||
{
|
||||
File.Delete(path);
|
||||
}
|
||||
}
|
||||
|
||||
await LiveSqlServer.DropSchemaAsync(_schema);
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public async Task Import_CopiesKeysAndAuditRows_AndReturnsCounts()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
(ApiKeyRecord keyA, ApiKeyRecord keyB) = await SeedSqliteAsync();
|
||||
|
||||
ApiKeyImportResult result = await _importer.ImportAsync(_sqlitePath, CancellationToken.None);
|
||||
|
||||
Assert.True(result.SourceFound);
|
||||
Assert.Equal(2, result.KeysInserted);
|
||||
Assert.Equal(0, result.KeysSkipped);
|
||||
Assert.Equal(3, result.AuditInserted);
|
||||
Assert.Equal(0, result.AuditSkipped);
|
||||
|
||||
await AssertRecordMatchesAsync(keyA);
|
||||
await AssertRecordMatchesAsync(keyB);
|
||||
|
||||
IReadOnlyList<ApiKeyAuditEntry> audit = await _sqlAudit.ListRecentAsync(10, CancellationToken.None);
|
||||
Assert.Equal(3, audit.Count);
|
||||
Assert.Contains(audit, e => e.KeyId is null && e.EventType == "auth_failed");
|
||||
Assert.Contains(audit, e => e.KeyId == "key-a" && e.EventType == "created");
|
||||
Assert.Contains(audit, e => e.KeyId == "key-b" && e.EventType == "created");
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public async Task Import_Twice_IsIdempotent()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
await SeedSqliteAsync();
|
||||
|
||||
await _importer.ImportAsync(_sqlitePath, CancellationToken.None);
|
||||
ApiKeyImportResult second = await _importer.ImportAsync(_sqlitePath, CancellationToken.None);
|
||||
|
||||
Assert.True(second.SourceFound);
|
||||
Assert.Equal(0, second.KeysInserted);
|
||||
Assert.Equal(2, second.KeysSkipped);
|
||||
Assert.Equal(0, second.AuditInserted);
|
||||
Assert.Equal(3, second.AuditSkipped);
|
||||
|
||||
Assert.Equal(2, (await _sqlAdmin.ListAsync(CancellationToken.None)).Count);
|
||||
Assert.Equal(3, (await _sqlAudit.ListRecentAsync(10, CancellationToken.None)).Count);
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public async Task Import_UnionsWithExistingRows()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
var lastUsed = new DateTimeOffset(2026, 7, 4, 12, 0, 0, TimeSpan.Zero);
|
||||
ApiKeyRecord keyC = SampleRecord("key-c") with { LastUsedUtc = lastUsed };
|
||||
await _sqlAdmin.CreateAsync(keyC, CancellationToken.None);
|
||||
|
||||
await SeedSqliteAsync();
|
||||
|
||||
ApiKeyImportResult result = await _importer.ImportAsync(_sqlitePath, CancellationToken.None);
|
||||
|
||||
Assert.Equal(2, result.KeysInserted);
|
||||
Assert.Equal(0, result.KeysSkipped);
|
||||
|
||||
IReadOnlyList<ApiKeyListItem> listed = await _sqlAdmin.ListAsync(CancellationToken.None);
|
||||
Assert.Equal(
|
||||
["key-a", "key-b", "key-c"],
|
||||
listed.Select(k => k.KeyId).OrderBy(k => k, StringComparer.Ordinal));
|
||||
|
||||
// The pre-existing row is untouched — the importer only ever inserts.
|
||||
Assert.Equal(lastUsed, Assert.Single(listed, k => k.KeyId == "key-c").LastUsedUtc);
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public async Task Import_MissingFile_ReturnsZeroWithoutThrowing()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
ApiKeyImportResult result = await _importer.ImportAsync(
|
||||
Path.Combine(Path.GetTempPath(), $"zbauth-absent-{Guid.NewGuid():N}.db"),
|
||||
CancellationToken.None);
|
||||
|
||||
Assert.False(result.SourceFound);
|
||||
Assert.Equal(0, result.KeysInserted);
|
||||
Assert.Equal(0, result.KeysSkipped);
|
||||
Assert.Equal(0, result.AuditInserted);
|
||||
Assert.Equal(0, result.AuditSkipped);
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public async Task Import_PreV3FileWithoutExpiresColumn_ReadsExpiryAsNull()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
await CreatePreV3SqliteFileAsync();
|
||||
|
||||
ApiKeyImportResult result = await _importer.ImportAsync(_sqlitePath, CancellationToken.None);
|
||||
|
||||
Assert.True(result.SourceFound);
|
||||
Assert.Equal(1, result.KeysInserted);
|
||||
|
||||
ApiKeyRecord? found = await _sqlRead.FindByKeyIdAsync("key-old", CancellationToken.None);
|
||||
Assert.NotNull(found);
|
||||
Assert.Null(found!.ExpiresUtc);
|
||||
Assert.Equal("legacy key", found.DisplayName);
|
||||
Assert.Equal(new DateTimeOffset(2026, 1, 2, 3, 4, 5, TimeSpan.Zero), found.CreatedUtc);
|
||||
}
|
||||
|
||||
// --- helpers ---
|
||||
|
||||
/// <summary>
|
||||
/// Seeds the throwaway SQLite file with two keys (one revoked, one with an expiry) and three
|
||||
/// audit rows (one of them key-less), then returns the two key records as SQLite holds them.
|
||||
/// </summary>
|
||||
private async Task<(ApiKeyRecord KeyA, ApiKeyRecord KeyB)> SeedSqliteAsync()
|
||||
{
|
||||
var sqliteFactory = new AuthSqliteConnectionFactory(_sqlitePath);
|
||||
await new SqliteAuthStoreMigrator(sqliteFactory).MigrateAsync(CancellationToken.None);
|
||||
|
||||
var admin = new SqliteApiKeyAdminStore(sqliteFactory);
|
||||
var audit = new SqliteApiKeyAuditStore(sqliteFactory);
|
||||
var read = new SqliteApiKeyStore(sqliteFactory);
|
||||
|
||||
ApiKeyRecord keyA = SampleRecord("key-a");
|
||||
ApiKeyRecord keyB = SampleRecord("key-b") with
|
||||
{
|
||||
ExpiresUtc = new DateTimeOffset(2027, 3, 1, 0, 0, 0, TimeSpan.Zero),
|
||||
};
|
||||
|
||||
await admin.CreateAsync(keyA, CancellationToken.None);
|
||||
await admin.CreateAsync(keyB, CancellationToken.None);
|
||||
await admin.RevokeAsync("key-a", new DateTimeOffset(2026, 6, 1, 7, 8, 9, TimeSpan.Zero), CancellationToken.None);
|
||||
|
||||
await audit.AppendAsync(
|
||||
new ApiKeyAuditEntry("key-a", "created", "10.0.0.1", new DateTimeOffset(2026, 5, 1, 8, 30, 0, TimeSpan.Zero), "by admin"),
|
||||
CancellationToken.None);
|
||||
await audit.AppendAsync(
|
||||
new ApiKeyAuditEntry("key-b", "created", null, new DateTimeOffset(2026, 5, 1, 8, 31, 0, TimeSpan.Zero), null),
|
||||
CancellationToken.None);
|
||||
await audit.AppendAsync(
|
||||
new ApiKeyAuditEntry(null, "auth_failed", "10.0.0.9", new DateTimeOffset(2026, 5, 2, 9, 0, 0, TimeSpan.Zero), "unknown key"),
|
||||
CancellationToken.None);
|
||||
|
||||
// Re-read so the assertions compare against exactly what SQLite holds, revocation included.
|
||||
ApiKeyRecord storedA = (await read.FindByKeyIdAsync("key-a", CancellationToken.None))!;
|
||||
ApiKeyRecord storedB = (await read.FindByKeyIdAsync("key-b", CancellationToken.None))!;
|
||||
|
||||
return (storedA, storedB);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Writes a v2-shaped SQLite file by hand — <c>api_keys</c> WITHOUT <c>expires_utc</c> — and
|
||||
/// inserts one key row, reproducing a store created before schema v3.
|
||||
/// </summary>
|
||||
private async Task CreatePreV3SqliteFileAsync()
|
||||
{
|
||||
await using var connection = new SqliteConnection(
|
||||
new SqliteConnectionStringBuilder { DataSource = _sqlitePath }.ToString());
|
||||
await connection.OpenAsync();
|
||||
|
||||
await using SqliteCommand ddl = connection.CreateCommand();
|
||||
ddl.CommandText = """
|
||||
CREATE TABLE api_keys (
|
||||
key_id TEXT PRIMARY KEY,
|
||||
key_prefix TEXT NOT NULL,
|
||||
secret_hash BLOB NOT NULL,
|
||||
display_name TEXT NOT NULL,
|
||||
scopes TEXT NOT NULL,
|
||||
constraints TEXT NULL,
|
||||
created_utc TEXT NOT NULL,
|
||||
last_used_utc TEXT NULL,
|
||||
revoked_utc TEXT NULL
|
||||
);
|
||||
CREATE TABLE api_key_audit (
|
||||
audit_id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
key_id TEXT NULL,
|
||||
event_type TEXT NOT NULL,
|
||||
remote_address TEXT NULL,
|
||||
created_utc TEXT NOT NULL,
|
||||
details TEXT NULL
|
||||
);
|
||||
""";
|
||||
await ddl.ExecuteNonQueryAsync();
|
||||
|
||||
await using SqliteCommand insert = connection.CreateCommand();
|
||||
insert.CommandText = """
|
||||
INSERT INTO api_keys (
|
||||
key_id, key_prefix, secret_hash, display_name, scopes,
|
||||
constraints, created_utc, last_used_utc, revoked_utc)
|
||||
VALUES ('key-old', 'sbk_old1', $hash, 'legacy key', 'read', NULL, $created, NULL, NULL);
|
||||
""";
|
||||
insert.Parameters.Add("$hash", SqliteType.Blob).Value = new byte[] { 4, 5, 6 };
|
||||
insert.Parameters.AddWithValue(
|
||||
"$created", new DateTimeOffset(2026, 1, 2, 3, 4, 5, TimeSpan.Zero).ToString("O"));
|
||||
await insert.ExecuteNonQueryAsync();
|
||||
}
|
||||
|
||||
private async Task AssertRecordMatchesAsync(ApiKeyRecord expected)
|
||||
{
|
||||
ApiKeyRecord? actual = await _sqlRead.FindByKeyIdAsync(expected.KeyId, CancellationToken.None);
|
||||
|
||||
Assert.NotNull(actual);
|
||||
Assert.Equal(expected.KeyPrefix, actual!.KeyPrefix);
|
||||
Assert.True(expected.SecretHash.SequenceEqual(actual.SecretHash));
|
||||
Assert.Equal(expected.DisplayName, actual.DisplayName);
|
||||
Assert.True(expected.Scopes.SetEquals(actual.Scopes));
|
||||
Assert.Equal(expected.ConstraintsJson, actual.ConstraintsJson);
|
||||
Assert.Equal(expected.CreatedUtc, actual.CreatedUtc);
|
||||
Assert.Equal(expected.LastUsedUtc, actual.LastUsedUtc);
|
||||
Assert.Equal(expected.RevokedUtc, actual.RevokedUtc);
|
||||
Assert.Equal(expected.ExpiresUtc, actual.ExpiresUtc);
|
||||
}
|
||||
|
||||
private static ApiKeyRecord SampleRecord(string keyId) => new(
|
||||
KeyId: keyId,
|
||||
KeyPrefix: "sbk_ab12",
|
||||
SecretHash: [1, 2, 3, 4, 5, 6, 7, 8],
|
||||
DisplayName: "Test Key " + keyId,
|
||||
Scopes: new HashSet<string>(["read", "write"], StringComparer.Ordinal),
|
||||
ConstraintsJson: """{"ipAllow":["10.0.0.0/8"]}""",
|
||||
CreatedUtc: new DateTimeOffset(2026, 5, 1, 8, 30, 0, TimeSpan.Zero),
|
||||
LastUsedUtc: null,
|
||||
RevokedUtc: null);
|
||||
}
|
||||
+16
@@ -0,0 +1,16 @@
|
||||
namespace ZB.MOM.WW.Auth.ApiKeys.SqlServer.Tests;
|
||||
|
||||
public sealed class SqlServerApiKeyOptionsTests
|
||||
{
|
||||
[Fact]
|
||||
public void Validate_RejectsMissingConnectionString() =>
|
||||
Assert.Throws<InvalidOperationException>(() => new SqlServerApiKeyOptions { ConnectionString = " " }.Validate());
|
||||
|
||||
[Fact]
|
||||
public void Validate_RejectsUnsafeSchemaName() =>
|
||||
Assert.Throws<InvalidOperationException>(() => new SqlServerApiKeyOptions { ConnectionString = "Server=x", Schema = "zb;auth" }.Validate());
|
||||
|
||||
[Fact]
|
||||
public void Schema_DefaultsToZbauth() =>
|
||||
Assert.Equal("zbauth", new SqlServerApiKeyOptions().Schema);
|
||||
}
|
||||
+66
@@ -0,0 +1,66 @@
|
||||
using Microsoft.Extensions.Configuration;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Hosting;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.Auth.Abstractions.ApiKeys;
|
||||
using ZB.MOM.WW.Auth.ApiKeys.SqlServer.DependencyInjection;
|
||||
|
||||
namespace ZB.MOM.WW.Auth.ApiKeys.SqlServer.Tests;
|
||||
|
||||
public sealed class SqlServerApiKeyServiceCollectionExtensionsTests
|
||||
{
|
||||
private static IConfiguration MinimalConfig() =>
|
||||
new ConfigurationBuilder().AddInMemoryCollection(new Dictionary<string, string?>
|
||||
{
|
||||
["ApiKeys:TokenPrefix"] = "sbk",
|
||||
["ApiKeys:PepperSecretName"] = "Pepper",
|
||||
["Pepper"] = "0123456789abcdef",
|
||||
["ApiKeys:SqlServer:ConnectionString"] = "Server=unused;TrustServerCertificate=true",
|
||||
}).Build();
|
||||
|
||||
[Fact]
|
||||
public void AddZbApiKeyAuthSqlServer_DisplacesSqliteStoresAndMigrator()
|
||||
{
|
||||
var services = new ServiceCollection();
|
||||
services.AddZbApiKeyAuthSqlServer(MinimalConfig(), "ApiKeys", "ApiKeys:SqlServer");
|
||||
using ServiceProvider sp = services.BuildServiceProvider();
|
||||
|
||||
Assert.IsType<SqlServerApiKeyStore>(sp.GetRequiredService<IApiKeyStore>());
|
||||
Assert.IsType<SqlServerApiKeyAdminStore>(sp.GetRequiredService<IApiKeyAdminStore>());
|
||||
Assert.IsType<SqlServerApiKeyAuditStore>(sp.GetRequiredService<IApiKeyAuditStore>());
|
||||
Assert.IsType<SqlServerApiKeyStoreMigrator>(sp.GetRequiredService<IApiKeyStoreMigrator>());
|
||||
Assert.NotNull(sp.GetRequiredService<SqliteApiKeyStoreImporter>());
|
||||
Assert.NotNull(sp.GetRequiredService<IApiKeyVerifier>());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void AddZbApiKeyAuthSqlServer_BindsSqlServerOptionsFromSection()
|
||||
{
|
||||
var config = new ConfigurationBuilder().AddInMemoryCollection(new Dictionary<string, string?>
|
||||
{
|
||||
["ApiKeys:TokenPrefix"] = "sbk",
|
||||
["ApiKeys:PepperSecretName"] = "Pepper",
|
||||
["Pepper"] = "0123456789abcdef",
|
||||
["ApiKeys:SqlServer:ConnectionString"] = "Server=my-shared-sql;TrustServerCertificate=true",
|
||||
["ApiKeys:SqlServer:Schema"] = "zbauth_custom",
|
||||
}).Build();
|
||||
var services = new ServiceCollection();
|
||||
services.AddZbApiKeyAuthSqlServer(config, "ApiKeys", "ApiKeys:SqlServer");
|
||||
using ServiceProvider sp = services.BuildServiceProvider();
|
||||
|
||||
SqlServerApiKeyOptions options = sp.GetRequiredService<IOptions<SqlServerApiKeyOptions>>().Value;
|
||||
Assert.Equal("Server=my-shared-sql;TrustServerCertificate=true", options.ConnectionString);
|
||||
Assert.Equal("zbauth_custom", options.Schema);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void AddZbApiKeyAuthSqlServer_RegistersMigrationHostedService()
|
||||
{
|
||||
var services = new ServiceCollection();
|
||||
services.AddZbApiKeyAuthSqlServer(MinimalConfig(), "ApiKeys", "ApiKeys:SqlServer");
|
||||
using ServiceProvider sp = services.BuildServiceProvider();
|
||||
|
||||
IEnumerable<IHostedService> hostedServices = sp.GetServices<IHostedService>();
|
||||
Assert.Contains(hostedServices, s => s.GetType().Name == "ApiKeyMigrationHostedService");
|
||||
}
|
||||
}
|
||||
+43
@@ -0,0 +1,43 @@
|
||||
namespace ZB.MOM.WW.Auth.ApiKeys.SqlServer.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Offline pins for the migrator's retry classification. A <see cref="Microsoft.Data.SqlClient.SqlException"/>
|
||||
/// cannot be constructed normally, so the error-number predicate is extracted as an internal static
|
||||
/// and pinned here by number; the concurrent-provision behaviour itself is exercised by the
|
||||
/// env-gated live suite. These run on any machine, so the retry filter stays pinned even when the
|
||||
/// live suite skips.
|
||||
/// </summary>
|
||||
public sealed class SqlServerApiKeyStoreMigratorTests
|
||||
{
|
||||
/// <summary>
|
||||
/// The provisioning-race outcomes the ScadaBridge central shared-store live gate (2026-08-07)
|
||||
/// proved reachable — or that the multi-statement DDL makes reachable — when N nodes provision
|
||||
/// one virgin database at once. Losing such a race means the object now exists, which is the
|
||||
/// desired end state, so a retry of the idempotent IF NOT EXISTS path must be taken.
|
||||
/// </summary>
|
||||
/// <param name="errorNumber">The SQL Server error number under test.</param>
|
||||
[Theory]
|
||||
[InlineData(1205)] // deadlock victim
|
||||
[InlineData(2714)] // "There is already an object named ..." — lost CREATE SCHEMA/TABLE (the live-gate defect)
|
||||
[InlineData(1913)] // "index ... already exists" — lost CREATE INDEX
|
||||
[InlineData(2627)] // PK violation — lost schema_version MERGE-insert
|
||||
public void Provisioning_race_error_numbers_are_retryable(int errorNumber)
|
||||
=> Assert.True(SqlServerApiKeyStoreMigrator.IsRetryableMigrationErrorNumber(errorNumber));
|
||||
|
||||
/// <summary>
|
||||
/// Errors that reproduce identically on retry must propagate so the node fails closed instead
|
||||
/// of spinning through the retry budget against a genuine misconfiguration.
|
||||
/// </summary>
|
||||
/// <param name="errorNumber">The SQL Server error number under test.</param>
|
||||
[Theory]
|
||||
[InlineData(2705)] // duplicate column — unreachable from the race (single CREATE TABLE statement)
|
||||
[InlineData(208)] // invalid object name
|
||||
[InlineData(229)] // permission denied
|
||||
[InlineData(2759)] // "CREATE SCHEMA failed due to previous errors" — retryable only via its companion 2714
|
||||
[InlineData(4060)] // cannot open database
|
||||
[InlineData(18456)] // login failed
|
||||
[InlineData(0)]
|
||||
[InlineData(-2)] // client-side timeout
|
||||
public void Non_race_error_numbers_are_not_retryable(int errorNumber)
|
||||
=> Assert.False(SqlServerApiKeyStoreMigrator.IsRetryableMigrationErrorNumber(errorNumber));
|
||||
}
|
||||
+30
@@ -0,0 +1,30 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
|
||||
<PropertyGroup>
|
||||
<IsPackable>false</IsPackable>
|
||||
</PropertyGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<PackageReference Include="coverlet.collector" />
|
||||
<PackageReference Include="Microsoft.NET.Test.Sdk" />
|
||||
<PackageReference Include="xunit" />
|
||||
<PackageReference Include="xunit.runner.visualstudio" />
|
||||
<!-- Live tests (Task 3+) skip rather than fail when AUTH_SQLSERVER_CONNSTR is unset. -->
|
||||
<PackageReference Include="Xunit.SkippableFact" />
|
||||
<PackageReference Include="Microsoft.Extensions.DependencyInjection" />
|
||||
<PackageReference Include="Microsoft.Extensions.Configuration" />
|
||||
<PackageReference Include="Microsoft.Extensions.Configuration.Abstractions" />
|
||||
<PackageReference Include="Microsoft.Extensions.Hosting.Abstractions" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<Using Include="Xunit" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\..\src\ZB.MOM.WW.Auth.ApiKeys.SqlServer\ZB.MOM.WW.Auth.ApiKeys.SqlServer.csproj" />
|
||||
<ProjectReference Include="..\..\src\ZB.MOM.WW.Auth.ApiKeys\ZB.MOM.WW.Auth.ApiKeys.csproj" />
|
||||
<ProjectReference Include="..\..\src\ZB.MOM.WW.Auth.Abstractions\ZB.MOM.WW.Auth.Abstractions.csproj" />
|
||||
</ItemGroup>
|
||||
|
||||
</Project>
|
||||
+34
@@ -5,6 +5,7 @@ using Microsoft.Extensions.Hosting;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.Auth.Abstractions.ApiKeys;
|
||||
using ZB.MOM.WW.Auth.ApiKeys.DependencyInjection;
|
||||
using ZB.MOM.WW.Auth.ApiKeys.Sqlite;
|
||||
|
||||
namespace ZB.MOM.WW.Auth.ApiKeys.Tests;
|
||||
|
||||
@@ -197,4 +198,37 @@ public class ApiKeyServiceCollectionExtensionsTests
|
||||
File.Delete(sqlitePath);
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void AddZbApiKeyAuth_RegistersSqliteMigratorAsIApiKeyStoreMigrator()
|
||||
{
|
||||
IConfiguration config = BuildConfiguration(TempSqlitePath());
|
||||
var services = new ServiceCollection();
|
||||
|
||||
services.AddZbApiKeyAuth(config, ApiKeySection);
|
||||
|
||||
using ServiceProvider sp = services.BuildServiceProvider();
|
||||
|
||||
Assert.IsType<SqliteAuthStoreMigrator>(sp.GetRequiredService<IApiKeyStoreMigrator>());
|
||||
Assert.Same(sp.GetRequiredService<SqliteAuthStoreMigrator>(), sp.GetRequiredService<IApiKeyStoreMigrator>());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void AddZbApiKeyAuth_LeavesPreRegisteredMigratorInPlace()
|
||||
{
|
||||
var services = new ServiceCollection();
|
||||
var fake = new FakeMigrator();
|
||||
services.AddSingleton<IApiKeyStoreMigrator>(fake);
|
||||
|
||||
services.AddZbApiKeyAuth(BuildConfiguration(TempSqlitePath()), ApiKeySection);
|
||||
|
||||
using ServiceProvider sp = services.BuildServiceProvider();
|
||||
|
||||
Assert.Same(fake, sp.GetRequiredService<IApiKeyStoreMigrator>());
|
||||
}
|
||||
|
||||
private sealed class FakeMigrator : IApiKeyStoreMigrator
|
||||
{
|
||||
public Task MigrateAsync(CancellationToken ct) => Task.CompletedTask;
|
||||
}
|
||||
}
|
||||
|
||||
+26
@@ -134,4 +134,30 @@ public class ServiceCollectionExtensionsTests
|
||||
await host.StartAsync();
|
||||
await host.StopAsync();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void AddZbLdapAuth_BindsUserBindModeKeys()
|
||||
{
|
||||
IConfiguration config = new ConfigurationBuilder()
|
||||
.AddInMemoryCollection(new Dictionary<string, string?>
|
||||
{
|
||||
[$"{LdapSection}:Server"] = LdapServer,
|
||||
[$"{LdapSection}:SearchBase"] = "dc=example,dc=com",
|
||||
[$"{LdapSection}:Transport"] = nameof(LdapTransport.Ldaps),
|
||||
[$"{LdapSection}:BindMode"] = "User",
|
||||
[$"{LdapSection}:UserBindTemplate"] = "{0}@corp.test",
|
||||
[$"{LdapSection}:AcceptDomainQualifiedNames"] = "false",
|
||||
})
|
||||
.Build();
|
||||
var services = new ServiceCollection();
|
||||
|
||||
services.AddZbLdapAuth(config, LdapSection);
|
||||
|
||||
using ServiceProvider provider = services.BuildServiceProvider();
|
||||
var options = provider.GetRequiredService<IOptions<LdapOptions>>().Value;
|
||||
|
||||
Assert.Equal(LdapBindMode.User, options.BindMode);
|
||||
Assert.Equal("{0}@corp.test", options.UserBindTemplate);
|
||||
Assert.False(options.AcceptDomainQualifiedNames);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -16,6 +16,8 @@ internal sealed class FakeLdapConnection : ILdapConnection
|
||||
private bool _throwOnConnect;
|
||||
private bool _throwOnServiceBind;
|
||||
private bool _throwOnUserBind;
|
||||
private int? _firstBindResultCode;
|
||||
private bool _throwOnSearch;
|
||||
|
||||
// ---- observation -----
|
||||
|
||||
@@ -32,13 +34,24 @@ internal sealed class FakeLdapConnection : ILdapConnection
|
||||
/// </summary>
|
||||
public int BindAttempts { get; private set; }
|
||||
|
||||
/// <summary>The attribute list passed to the most recent <see cref="Search"/> call (the fake still returns every scripted attribute).</summary>
|
||||
public IReadOnlyList<string>? SearchAttributes { get; private set; }
|
||||
|
||||
/// <summary>Every call in order: <c>connect</c>, <c>bind:<dn></c>, <c>search:<filter></c>.</summary>
|
||||
public List<string> Calls { get; } = new();
|
||||
|
||||
// ---- builder methods -----
|
||||
|
||||
/// <summary>
|
||||
/// Scripts a user entry that will be returned by the next <see cref="Search"/> call.
|
||||
/// Builds a minimal attribute bag with <c>memberOf</c> and optional <c>displayName</c>.
|
||||
/// Builds a minimal attribute bag with <c>memberOf</c>, optional <c>displayName</c> and any
|
||||
/// <paramref name="extraAttributes"/> (e.g. <c>sAMAccountName</c>) the test needs.
|
||||
/// </summary>
|
||||
public FakeLdapConnection WithUserEntry(string dn, string[] memberOf, string? displayName = null)
|
||||
public FakeLdapConnection WithUserEntry(
|
||||
string dn,
|
||||
string[] memberOf,
|
||||
string? displayName = null,
|
||||
IReadOnlyDictionary<string, string[]>? extraAttributes = null)
|
||||
{
|
||||
var attrs = new Dictionary<string, IReadOnlyList<string>>(StringComparer.OrdinalIgnoreCase)
|
||||
{
|
||||
@@ -46,6 +59,11 @@ internal sealed class FakeLdapConnection : ILdapConnection
|
||||
};
|
||||
if (displayName is not null)
|
||||
attrs["displayName"] = new[] { displayName };
|
||||
if (extraAttributes is not null)
|
||||
{
|
||||
foreach (var (name, values) in extraAttributes)
|
||||
attrs[name] = values;
|
||||
}
|
||||
|
||||
_scriptedEntries.Add(new LdapSearchEntry(dn, attrs));
|
||||
return this;
|
||||
@@ -83,6 +101,24 @@ internal sealed class FakeLdapConnection : ILdapConnection
|
||||
return this;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Throw <see cref="Novell.Directory.Ldap.LdapException"/> with <paramref name="resultCode"/> on the
|
||||
/// FIRST bind. In user-bind mode the first bind is the user's own, so 49 simulates a wrong
|
||||
/// password and e.g. 52 (Unavailable) a domain controller that cannot answer.
|
||||
/// </summary>
|
||||
public FakeLdapConnection ThrowOnFirstBind(int resultCode)
|
||||
{
|
||||
_firstBindResultCode = resultCode;
|
||||
return this;
|
||||
}
|
||||
|
||||
/// <summary>Throw <see cref="Novell.Directory.Ldap.LdapException"/> from <see cref="Search"/> (unhealthy directory).</summary>
|
||||
public FakeLdapConnection ThrowOnSearch()
|
||||
{
|
||||
_throwOnSearch = true;
|
||||
return this;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Throw <see cref="Novell.Directory.Ldap.LdapException"/> from <see cref="Connect"/> to
|
||||
/// simulate an unreachable directory (infrastructure failure).
|
||||
@@ -120,6 +156,7 @@ internal sealed class FakeLdapConnection : ILdapConnection
|
||||
int timeoutMs,
|
||||
RemoteCertificateValidationCallback? serverCertificateValidationCallback = null)
|
||||
{
|
||||
Calls.Add("connect");
|
||||
ConnectArgs = (host, port, transport, allowInsecure, timeoutMs);
|
||||
ConnectCertCallback = serverCertificateValidationCallback;
|
||||
if (_throwOnConnect)
|
||||
@@ -130,8 +167,12 @@ internal sealed class FakeLdapConnection : ILdapConnection
|
||||
public void Bind(string dn, string password)
|
||||
{
|
||||
BindAttempts++;
|
||||
Calls.Add($"bind:{dn}");
|
||||
var isServiceBind = BindAttempts == 1;
|
||||
|
||||
if (isServiceBind && _firstBindResultCode is { } code)
|
||||
throw new Novell.Directory.Ldap.LdapException("Bind refused", code, dn);
|
||||
|
||||
if ((_throwOnServiceBind && isServiceBind)
|
||||
|| (_throwOnUserBind && !isServiceBind)
|
||||
|| _throwBindDns.Contains(dn))
|
||||
@@ -147,7 +188,14 @@ internal sealed class FakeLdapConnection : ILdapConnection
|
||||
string searchBase,
|
||||
string filter,
|
||||
IReadOnlyList<string> attributes)
|
||||
=> _scriptedEntries.AsReadOnly();
|
||||
{
|
||||
Calls.Add($"search:{filter}");
|
||||
SearchAttributes = attributes;
|
||||
if (_throwOnSearch)
|
||||
throw new Novell.Directory.Ldap.LdapException(
|
||||
"Search failed", Novell.Directory.Ldap.LdapException.Unavailable, searchBase);
|
||||
return _scriptedEntries.AsReadOnly();
|
||||
}
|
||||
|
||||
public void Dispose() { /* nothing to clean up */ }
|
||||
}
|
||||
|
||||
@@ -147,4 +147,45 @@ public class LdapAuthServiceTests
|
||||
Assert.True(r.Succeeded);
|
||||
Assert.Equal(new[] { "Eng,ineers", "A,B" }, r.Groups);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Username_IsCanonicalisedFromTheEntry_InServiceAccountMode()
|
||||
{
|
||||
var fake = new FakeLdapConnection().WithUserEntry(
|
||||
"cn=alice,dc=x", new[] { "cn=g,dc=x" }, "Alice",
|
||||
new Dictionary<string, string[]> { ["cn"] = new[] { "alice" } });
|
||||
var svc = new LdapAuthService(Opts(), new FakeLdapConnectionFactory(fake));
|
||||
|
||||
var r = await svc.AuthenticateAsync("ALICE", "pw", default);
|
||||
|
||||
Assert.True(r.Succeeded);
|
||||
Assert.Equal("alice", r.Username);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Username_FallsBackToTheTypedName_InServiceAccountMode_WhenAttributeAbsent()
|
||||
{
|
||||
var fake = new FakeLdapConnection().WithUserEntry("cn=alice,dc=x", new[] { "cn=g,dc=x" }, "Alice");
|
||||
var svc = new LdapAuthService(Opts(), new FakeLdapConnectionFactory(fake));
|
||||
|
||||
var r = await svc.AuthenticateAsync("ALICE", "pw", default);
|
||||
|
||||
Assert.True(r.Succeeded);
|
||||
Assert.Equal("ALICE", r.Username);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData("")]
|
||||
[InlineData(null)]
|
||||
public async Task EmptyPassword_IsRefused_InServiceAccountMode_WithoutContactingTheDirectory(string? password)
|
||||
{
|
||||
var factory = new SequenceLdapConnectionFactory(
|
||||
new FakeLdapConnection().WithUserEntry("cn=alice,dc=x", new[] { "cn=g,dc=x" }, "Alice"));
|
||||
var svc = new LdapAuthService(Opts(), factory);
|
||||
|
||||
var r = await svc.AuthenticateAsync("alice", password!, default);
|
||||
|
||||
Assert.Equal(LdapAuthFailure.BadCredentials, r.Failure);
|
||||
Assert.Empty(factory.Created);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,222 @@
|
||||
using Novell.Directory.Ldap;
|
||||
using ZB.MOM.WW.Auth.Abstractions.Ldap;
|
||||
|
||||
namespace ZB.MOM.WW.Auth.Ldap.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// <see cref="LdapBindMode.User"/>: bind as the signing-in user, then read their own entry on the
|
||||
/// same connection. The service account in <see cref="Opts"/> is deliberately populated so the
|
||||
/// tests prove it is never used.
|
||||
/// </summary>
|
||||
public class LdapAuthServiceUserBindTests
|
||||
{
|
||||
private static LdapOptions Opts(params string[] fallbacks) => new()
|
||||
{
|
||||
Enabled = true,
|
||||
Server = "dc1",
|
||||
Port = 636,
|
||||
Transport = LdapTransport.Ldaps,
|
||||
SearchBase = "dc=x",
|
||||
UserNameAttribute = "sAMAccountName",
|
||||
DisplayNameAttribute = "displayName",
|
||||
GroupAttribute = "memberOf",
|
||||
BindMode = LdapBindMode.User,
|
||||
UserBindTemplate = "{0}@corp.test",
|
||||
ServiceAccountDn = "cn=svc,dc=x",
|
||||
ServiceAccountPassword = "svcpw",
|
||||
FallbackServers = fallbacks,
|
||||
};
|
||||
|
||||
private static FakeLdapConnection Jane() => new FakeLdapConnection()
|
||||
.WithUserEntry("CN=Jane Doe,OU=u,DC=x", new[] { "CN=UG_A,OU=g,DC=x", "CN=UG_B,OU=g,DC=x" }, "Jane Doe");
|
||||
|
||||
private static Task<LdapAuthResult> Login(LdapAuthService svc, string user, string pw = "pw") =>
|
||||
svc.AuthenticateAsync(user, pw, CancellationToken.None);
|
||||
|
||||
[Fact]
|
||||
public async Task Success_BindsAsUser_ThenSearchesOnTheSameConnection()
|
||||
{
|
||||
var factory = new SequenceLdapConnectionFactory(Jane());
|
||||
var result = await Login(new LdapAuthService(Opts(), factory), "jdoe");
|
||||
|
||||
Assert.True(result.Succeeded);
|
||||
Assert.Equal("jdoe", result.Username);
|
||||
Assert.Equal("Jane Doe", result.DisplayName);
|
||||
Assert.Equal(new[] { "UG_A", "UG_B" }, result.Groups);
|
||||
var conn = Assert.Single(factory.Created);
|
||||
Assert.Equal(new[] { "connect", "bind:jdoe@corp.test", "search:(sAMAccountName=jdoe)" }, conn.Calls);
|
||||
Assert.Equal(new[] { "jdoe@corp.test" }, conn.BoundDns);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(@"NAM\jdoe")]
|
||||
[InlineData("jdoe@corp.test")]
|
||||
[InlineData(" jdoe ")]
|
||||
public async Task QualifiedNames_BindAsTheTemplate_AndReturnTheBareName(string typed)
|
||||
{
|
||||
var factory = new SequenceLdapConnectionFactory(Jane());
|
||||
var result = await Login(new LdapAuthService(Opts(), factory), typed);
|
||||
|
||||
Assert.True(result.Succeeded);
|
||||
Assert.Equal("jdoe", result.Username);
|
||||
Assert.Equal(new[] { "jdoe@corp.test" }, factory.Created[0].BoundDns);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task EmptyPassword_IsRefused_WithoutContactingTheDirectory()
|
||||
{
|
||||
var factory = new SequenceLdapConnectionFactory(Jane());
|
||||
var result = await Login(new LdapAuthService(Opts(), factory), "jdoe", "");
|
||||
|
||||
Assert.Equal(LdapAuthFailure.BadCredentials, result.Failure);
|
||||
Assert.Empty(factory.Created);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData("@x")]
|
||||
[InlineData(@"a\b\c")]
|
||||
[InlineData("a@b@c")]
|
||||
public async Task UnusableNames_AreRefused_WithoutContactingTheDirectory(string typed)
|
||||
{
|
||||
var factory = new SequenceLdapConnectionFactory(Jane());
|
||||
var result = await Login(new LdapAuthService(Opts(), factory), typed);
|
||||
|
||||
Assert.Equal(LdapAuthFailure.BadCredentials, result.Failure);
|
||||
Assert.Empty(factory.Created);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task StrictMode_RefusesQualifiedNames_WithoutContactingTheDirectory()
|
||||
{
|
||||
var factory = new SequenceLdapConnectionFactory(Jane());
|
||||
var svc = new LdapAuthService(Opts() with { AcceptDomainQualifiedNames = false }, factory);
|
||||
|
||||
var result = await Login(svc, @"NAM\jdoe");
|
||||
|
||||
Assert.Equal(LdapAuthFailure.BadCredentials, result.Failure);
|
||||
Assert.Empty(factory.Created);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task WrongPassword_IsFinal_AndNeverAsksTheFallback()
|
||||
{
|
||||
var factory = new SequenceLdapConnectionFactory(
|
||||
Jane().ThrowOnFirstBind(LdapException.InvalidCredentials), Jane());
|
||||
var result = await Login(new LdapAuthService(Opts("dc2"), factory), "jdoe", "wrong");
|
||||
|
||||
Assert.Equal(LdapAuthFailure.BadCredentials, result.Failure);
|
||||
Assert.Single(factory.Created);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task NonCredentialBindError_FailsOver()
|
||||
{
|
||||
var factory = new SequenceLdapConnectionFactory(
|
||||
Jane().ThrowOnFirstBind(LdapException.Unavailable), Jane());
|
||||
var result = await Login(new LdapAuthService(Opts("dc2"), factory), "jdoe");
|
||||
|
||||
Assert.True(result.Succeeded);
|
||||
Assert.Equal(2, factory.Created.Count);
|
||||
Assert.Equal("dc2", factory.Created[1].ConnectArgs!.Value.Host);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ConnectFailure_FailsOver()
|
||||
{
|
||||
var factory = new SequenceLdapConnectionFactory(new FakeLdapConnection().ThrowOnConnect(), Jane());
|
||||
var result = await Login(new LdapAuthService(Opts("dc2"), factory), "jdoe");
|
||||
|
||||
Assert.True(result.Succeeded);
|
||||
Assert.Equal(2, factory.Created.Count);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task SearchFailure_FailsOver()
|
||||
{
|
||||
var factory = new SequenceLdapConnectionFactory(Jane().ThrowOnSearch(), Jane());
|
||||
var result = await Login(new LdapAuthService(Opts("dc2"), factory), "jdoe");
|
||||
|
||||
Assert.True(result.Succeeded);
|
||||
Assert.Equal(2, factory.Created.Count);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task EveryEndpointDown_IsSystemSide()
|
||||
{
|
||||
var factory = new SequenceLdapConnectionFactory(
|
||||
new FakeLdapConnection().ThrowOnConnect(), new FakeLdapConnection().ThrowOnConnect());
|
||||
var result = await Login(new LdapAuthService(Opts("dc2"), factory), "jdoe");
|
||||
|
||||
Assert.Equal(LdapAuthFailure.ServiceAccountBindFailed, result.Failure);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task NoEntryAfterAGoodBind_IsGroupLookupFailed()
|
||||
{
|
||||
var factory = new SequenceLdapConnectionFactory(new FakeLdapConnection().WithNoMatch());
|
||||
var result = await Login(new LdapAuthService(Opts("dc2"), factory), "jdoe");
|
||||
|
||||
Assert.Equal(LdapAuthFailure.GroupLookupFailed, result.Failure);
|
||||
Assert.Single(factory.Created);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task TwoEntriesAfterAGoodBind_IsGroupLookupFailed()
|
||||
{
|
||||
var factory = new SequenceLdapConnectionFactory(new FakeLdapConnection().WithDuplicateMatch());
|
||||
var result = await Login(new LdapAuthService(Opts(), factory), "jdoe");
|
||||
|
||||
Assert.Equal(LdapAuthFailure.GroupLookupFailed, result.Failure);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task NoGroups_IsGroupLookupFailed()
|
||||
{
|
||||
var factory = new SequenceLdapConnectionFactory(
|
||||
new FakeLdapConnection().WithUserEntry("CN=Jane,DC=x", Array.Empty<string>(), "Jane"));
|
||||
var result = await Login(new LdapAuthService(Opts(), factory), "jdoe");
|
||||
|
||||
Assert.Equal(LdapAuthFailure.GroupLookupFailed, result.Failure);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Username_IsCanonicalisedFromTheEntry_NotTheTypedCase()
|
||||
{
|
||||
var factory = new SequenceLdapConnectionFactory(new FakeLdapConnection().WithUserEntry(
|
||||
"CN=Jane Doe,OU=u,DC=x", new[] { "CN=UG_A,OU=g,DC=x" }, "Jane Doe",
|
||||
new Dictionary<string, string[]> { ["sAMAccountName"] = new[] { "jdoe" } }));
|
||||
var result = await Login(new LdapAuthService(Opts(), factory), @"NAM\JDOE");
|
||||
|
||||
Assert.True(result.Succeeded);
|
||||
Assert.Equal("jdoe", result.Username);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Username_FallsBackToTheTypedName_WhenTheEntryLacksTheAttribute()
|
||||
{
|
||||
var factory = new SequenceLdapConnectionFactory(Jane());
|
||||
var result = await Login(new LdapAuthService(Opts(), factory), "JDOE");
|
||||
|
||||
Assert.True(result.Succeeded);
|
||||
Assert.Equal("JDOE", result.Username);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Search_RequestsTheUserNameAttribute()
|
||||
{
|
||||
var factory = new SequenceLdapConnectionFactory(Jane());
|
||||
await Login(new LdapAuthService(Opts(), factory), "jdoe");
|
||||
|
||||
Assert.Equal(new[] { "displayName", "memberOf", "sAMAccountName" }, factory.Created[0].SearchAttributes);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Search_DeDuplicatesTheUserNameAttribute_CaseInsensitively()
|
||||
{
|
||||
var factory = new SequenceLdapConnectionFactory(Jane());
|
||||
var svc = new LdapAuthService(Opts() with { DisplayNameAttribute = "SAMACCOUNTNAME" }, factory);
|
||||
await Login(svc, "jdoe");
|
||||
|
||||
Assert.Equal(new[] { "SAMACCOUNTNAME", "memberOf" }, factory.Created[0].SearchAttributes);
|
||||
}
|
||||
}
|
||||
@@ -116,4 +116,38 @@ public class LdapOptionsValidatorTests
|
||||
Assert.False(new LdapOptionsValidator()
|
||||
.Validate(null, Opts() with { Enabled = false, FallbackServers = new[] { ":::" } })
|
||||
.Failed);
|
||||
|
||||
private static LdapOptions UserOpts() => Opts() with
|
||||
{
|
||||
BindMode = LdapBindMode.User,
|
||||
UserBindTemplate = "{0}@corp.test",
|
||||
ServiceAccountDn = "",
|
||||
ServiceAccountPassword = "",
|
||||
};
|
||||
|
||||
[Fact]
|
||||
public void Defaults_KeepServiceAccountMode()
|
||||
{
|
||||
var o = new LdapOptions();
|
||||
Assert.Equal(LdapBindMode.ServiceAccount, o.BindMode);
|
||||
Assert.Equal("", o.UserBindTemplate);
|
||||
Assert.True(o.AcceptDomainQualifiedNames);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Validator_UserMode_DoesNotRequireServiceAccount() =>
|
||||
Assert.True(new LdapOptionsValidator().Validate(null, UserOpts()).Succeeded);
|
||||
|
||||
[Theory]
|
||||
[InlineData("")]
|
||||
[InlineData(" ")]
|
||||
[InlineData("jdoe@corp.test")]
|
||||
[InlineData("{0}{0}@corp.test")]
|
||||
public void Validator_UserMode_RequiresExactlyOnePlaceholder(string template)
|
||||
{
|
||||
var result = new LdapOptionsValidator().Validate(null, UserOpts() with { UserBindTemplate = template });
|
||||
|
||||
Assert.True(result.Failed);
|
||||
Assert.Contains(nameof(LdapOptions.UserBindTemplate), result.FailureMessage, StringComparison.Ordinal);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,47 @@
|
||||
using ZB.MOM.WW.Auth.Ldap.Internal;
|
||||
|
||||
namespace ZB.MOM.WW.Auth.Ldap.Tests;
|
||||
|
||||
public class LdapUserNameTests
|
||||
{
|
||||
[Theory]
|
||||
[InlineData("jdoe")]
|
||||
[InlineData(" jdoe ")]
|
||||
[InlineData(@"NAM\jdoe")]
|
||||
[InlineData("jdoe@corp.test")]
|
||||
public void Accepting_ReducesToBareName(string input)
|
||||
{
|
||||
Assert.True(LdapUserName.TryNormalize(input, acceptDomainQualified: true, out var name));
|
||||
Assert.Equal("jdoe", name);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(@"NAM\jdoe")]
|
||||
[InlineData("jdoe@corp.test")]
|
||||
public void Strict_RefusesQualifiedNames(string input) =>
|
||||
Assert.False(LdapUserName.TryNormalize(input, acceptDomainQualified: false, out _));
|
||||
|
||||
[Fact]
|
||||
public void Strict_AcceptsBareName()
|
||||
{
|
||||
Assert.True(LdapUserName.TryNormalize(" jdoe ", acceptDomainQualified: false, out var name));
|
||||
Assert.Equal("jdoe", name);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(null)]
|
||||
[InlineData("")]
|
||||
[InlineData(" ")]
|
||||
[InlineData("@x")]
|
||||
[InlineData(@"\jdoe")]
|
||||
[InlineData("jdoe@")]
|
||||
[InlineData(@"NAM\")]
|
||||
[InlineData(@"a\b\c")]
|
||||
[InlineData("a@b@c")]
|
||||
[InlineData(@"NAM\jdoe@corp.test")]
|
||||
public void Unusable_IsRefused(string? input)
|
||||
{
|
||||
Assert.False(LdapUserName.TryNormalize(input, acceptDomainQualified: true, out var name));
|
||||
Assert.Equal("", name);
|
||||
}
|
||||
}
|
||||
@@ -75,3 +75,33 @@ Design documentation:
|
||||
- `~/Desktop/scadaproj/components/configuration/spec/SPEC.md` — normalized validation target
|
||||
- `~/Desktop/scadaproj/components/configuration/shared-contract/ZB.MOM.WW.Configuration.md` — proposed shared-library API
|
||||
- `~/Desktop/scadaproj/components/configuration/current-state/` — per-project current state (code-verified)
|
||||
|
||||
## Index history (migrated from the scadaproj root CLAUDE.md)
|
||||
|
||||
> Moved verbatim out of the scadaproj root `CLAUDE.md` on 2026-09-03 so it loads only when needed. **This file is now the authoritative place for this history — append new entries here and keep the root index row short.**
|
||||
|
||||
### Narrative
|
||||
|
||||
The configuration component is fully populated: a normalized [`spec`](components/configuration/spec/SPEC.md), a
|
||||
[`shared-contract`](components/configuration/shared-contract/ZB.MOM.WW.Configuration.md), three
|
||||
[`current-state`](components/configuration/current-state/) docs, and an adoption [`GAPS`](components/configuration/GAPS.md)
|
||||
backlog. Shared = the `IValidateOptions<T>` failure-accumulation base (`OptionsValidatorBase<T>`) +
|
||||
reusable rule primitives (`ValidationBuilder`: port / host:port / required / positive-duration / one-of /
|
||||
min-count) + `AddValidatedOptions<TOptions,TValidator>()` (bind + validate + `ValidateOnStart`) + the
|
||||
pre-host `ConfigPreflight` aggregator (generalizes ScadaBridge's `StartupValidator`, byte-compatible
|
||||
message); left per-project = each app's options classes + domain rules, and OtOpcUa's
|
||||
draft/generation-content validation (DB-side `sp_ValidateDraft`; its C# `DraftValidator` is dormant).
|
||||
|
||||
The shared library is **built and lives in this repo** at [`ZB.MOM.WW.Configuration/`](ZB.MOM.WW.Configuration/)
|
||||
(.NET 10; single package `ZB.MOM.WW.Configuration`; 42 tests (verified 2026-07-18); `dotnet pack` → 1 nupkg @ 0.1.0).
|
||||
The implementation plan is at
|
||||
[`docs/plans/2026-06-01-zb-mom-ww-configuration-shared-library.md`](docs/plans/2026-06-01-zb-mom-ww-configuration-shared-library.md).
|
||||
**Adopted across all three apps on 2026-06-01** (OtOpcUa, MxAccessGateway, ScadaBridge) on each repo's
|
||||
local default branch (`main`/`master`) — merged, **not yet pushed** to remotes; the package was first
|
||||
published to the Gitea feed. Behaviour-preserving onto `OptionsValidatorBase`/`AddValidatedOptions`
|
||||
for MxGateway + ScadaBridge (validator messages byte-identical), `StartupValidator` → `ConfigPreflight`
|
||||
for ScadaBridge, and net-new `Ldap`/`OpcUa` validators for OtOpcUa. Per-app result tracked in
|
||||
[`components/configuration/GAPS.md`](components/configuration/GAPS.md).
|
||||
Build/test from `ZB.MOM.WW.Configuration/`: `dotnet test`. Consumer matrix: all three apps consume the
|
||||
single package; ScadaBridge is the heaviest adopter (per-module validators + `StartupValidator` →
|
||||
`ConfigPreflight`); OtOpcUa adoption is additive (it has no `IValidateOptions` usage today).
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
# ZB.MOM.WW.GalaxyRepository
|
||||
|
||||
Shared Galaxy object-hierarchy SQL browse + `galaxy_repository.v1` gRPC service for the **ZB.MOM.WW SCADA family** (consumed by HistorianGateway and mxaccessgw). Build/test from this directory: `dotnet test`.
|
||||
|
||||
## Index history (migrated from the scadaproj root CLAUDE.md)
|
||||
|
||||
> Moved verbatim out of the scadaproj root `CLAUDE.md` on 2026-09-03 so it loads only when needed. **This file is now the authoritative place for this history — append new entries here and keep the root index row short.**
|
||||
|
||||
### Narrative
|
||||
|
||||
The Galaxy Repository component normalizes the **Galaxy object-hierarchy SQL browse + reusable gRPC service**
|
||||
that was previously embedded in `mxaccessgw`. Shared = canonical `galaxy_repository.v1` proto (wire-compatible
|
||||
with `mxaccessgw`'s existing contract so OtOpcUa's `GalaxyRepositoryClient` is unaffected), the SQL browse
|
||||
provider (`HierarchySql` / `AttributesSql` validated reverse-engineered queries), in-memory hierarchy cache +
|
||||
snapshot + deploy-poll refresh `BackgroundService`, `GalaxyHierarchyProjector`, and `AddZbGalaxyRepository` /
|
||||
`MapZbGalaxyRepository` DI extension. Left per-consumer = section path, subtree auth filtering, and any
|
||||
app-specific paging defaults.
|
||||
|
||||
The shared library is **built and lives in this repo** at [`ZB.MOM.WW.GalaxyRepository/`](ZB.MOM.WW.GalaxyRepository/)
|
||||
(.NET 10; single package `ZB.MOM.WW.GalaxyRepository`; `dotnet pack` → 1 nupkg @ 0.1.0, **published to
|
||||
the Gitea NuGet feed** `gitea.dohertylan.com/api/packages/dohertj2/nuget`). The design doc is at
|
||||
[`docs/plans/2026-06-23-historian-gateway-design.md`](docs/plans/2026-06-23-historian-gateway-design.md) (§10, component 1).
|
||||
**Consumed by HistorianGateway** as a `PackageReference` from that Gitea feed, pinned at `0.2.0` (originally a
|
||||
cross-repo `ProjectReference` to this scadaproj tree; switched to the feed package 2026-06-24; bumped `0.1.0` → `0.2.0` 2026-06-25).
|
||||
**mxaccessgw adoption is a tracked follow-on** — once adopted, mxaccessgw's inline Galaxy browse code is replaced
|
||||
by the shared lib (the `galaxy_repository.v1` wire contract is already identical, so OtOpcUa and ScadaBridge
|
||||
clients are unaffected). Build/test from `ZB.MOM.WW.GalaxyRepository/`: `dotnet test`.
|
||||
Consumer matrix: HistorianGateway (initial); mxaccessgw (follow-on adoption).
|
||||
@@ -75,3 +75,30 @@ Design documentation:
|
||||
- `~/Desktop/scadaproj/components/health/spec/SPEC.md` — normalized three-tier target
|
||||
- `~/Desktop/scadaproj/components/health/shared-contract/ZB.MOM.WW.Health.md` — proposed API (aligned to shipped code)
|
||||
- `~/Desktop/scadaproj/components/health/current-state/` — per-project current state (code-verified)
|
||||
|
||||
## Index history (migrated from the scadaproj root CLAUDE.md)
|
||||
|
||||
> Moved verbatim out of the scadaproj root `CLAUDE.md` on 2026-09-03 so it loads only when needed. **This file is now the authoritative place for this history — append new entries here and keep the root index row short.**
|
||||
|
||||
### Component-table row as it read in the index
|
||||
|
||||
| Component | Status | Goal | Design | Implementation |
|
||||
|---|---|---|---|---|
|
||||
| Health (readiness / liveness / active-node) | **Adopted** (feed carries `0.1.0`–`0.3.0`, current lib version **`0.3.0`**; **all 4 apps** reference it + wire `MapZbHealth`). **`0.2.0` (2026-07-24)** added an optional per-entry `data` object to the canonical health JSON and made `AkkaClusterHealthCheck` publish its cluster view (`leader`/`selfAddress`/`selfRoles`/`memberCount`/`unreachableCount`) — additive, so pre-0.2.0 payloads still parse. **`0.2.1` (2026-07-24) is a behaviour fix with a live consumer impact:** the active tier now returns **Unhealthy (503)** on a role-member-but-not-leader node instead of `Degraded`, which the spec maps to 200. The old value made `/health/active` answer 200 on *every* node, so it could not distinguish active from standby over HTTP at all — and OtOpcUa's Traefik routes the admin UI by exactly this probe, so its leader-pinning had silently never worked ([`lmxopcua#494`](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/494)). **`0.3.0` (2026-07-24) finished that fix and ENDED a family-wide duplication.** 0.2.1 made the tier a real 503 but still selected by `RoleLeader`; on a rebuilt rig the two orderings diverge live (akka leader `central-1`, oldest admin `central-2` — the singleton host), so it would have pinned Traefik to the node *not* hosting the work. **Leadership is address-ordered (host, then port) and has no relationship to time; singleton placement is age-ordered.** They agree on a freshly-formed cluster — which is why single-node and happy-path tests never caught it — and diverge permanently after any restart. Both Akka apps had already discovered this independently and each written a private replacement rather than use the shared check (ScadaBridge `OldestNodeActiveHealthCheck`, OtOpcUa `ClusterPrimaryHealthCheck`), so the package's entire active-node surface had **zero consumers — not unused but avoided, twice, for the same reason.** 0.3.0 promotes those two into one `ClusterActiveNode` primitive (oldest `Up` member, optional role scope, plus an ordered role-preference so a fused node answers for the role its singletons are pinned to); `ActiveNodeHealthCheck` and `AkkaActiveNodeGate` both delegate to it, and results now carry `activeRole`/`selfAddress`/`activeNode` in the 0.2.0 `data` object so a **standby reports who IS active**. Two further behaviour changes: a node holding **none** of the preferred roles is now **Unhealthy**, not Healthy — the old "not applicable ⇒ Healthy" is exactly what made the tier answer 200 everywhere, and the case is reachable (OtOpcUa's `RoleParser` admits `dev`-only and `cluster-`only nodes); and identity compares `UniqueAddress`, so a node restarted on the same host:port is correctly a different member. **Both apps migrated and their private copies deleted** — OtOpcUa (`feat/health-0.3.0-active-node` @ `88adec04`) also routes `RedundancyStateActor.SelectOldestUpMemberOfRole` through the shared primitive, so the tier and the OPC UA `ServiceLevel` 250/240 split cannot drift; ScadaBridge (`feat/site-node-health` @ `5d4853a1`) routes `ActiveNodeEvaluator` through it, unifying the S&F delivery gate, the heartbeat `IsActive` stamp, the inbound-API gate and the health tier on one rule. ScadaBridge keeps `SitePairActiveNodeHealthCheck` deliberately — it is a thin adapter over the site's own role-scoped `IClusterNodeProvider` (itself now backed by the shared rule), not a second copy of it. mxgw/HistorianGateway have no Akka cluster and are unaffected. **Both app branches are MERGED and PUSHED** (verified 2026-08-07): OtOpcUa `88adec04` is on `origin/master`; ScadaBridge merged via `b3dc17a5` (2026-07-24) on `origin/main`, private-copy deletions included. | Shared `ZB.MOM.WW.Health` lib | [`components/health/`](components/health/) | [`ZB.MOM.WW.Health/`](ZB.MOM.WW.Health/) |
|
||||
|
||||
### Narrative
|
||||
|
||||
The health component is fully populated: a normalized [`spec`](components/health/spec/SPEC.md), a
|
||||
[`shared-contract`](components/health/shared-contract/ZB.MOM.WW.Health.md), three
|
||||
[`current-state`](components/health/current-state/) docs, and an adoption [`GAPS`](components/health/GAPS.md)
|
||||
backlog. Shared = three-tier endpoint convention (ready/active/healthz) + canonical JSON writer +
|
||||
`IActiveNodeGate` seam + `GrpcDependencyHealthCheck` + `AkkaClusterHealthCheck` + `ActiveNodeHealthCheck`
|
||||
+ `DatabaseHealthCheck<TContext>`; left per-project = which probes each app registers,
|
||||
orchestrator wiring, and ScadaBridge's distributed health-monitoring pipeline.
|
||||
|
||||
The shared library is **built and lives in this repo** at [`ZB.MOM.WW.Health/`](ZB.MOM.WW.Health/)
|
||||
(.NET 10; 3 packages — `ZB.MOM.WW.Health`, `ZB.MOM.WW.Health.Akka`, `ZB.MOM.WW.Health.EntityFrameworkCore`;
|
||||
64 tests (verified 2026-07-18); `dotnet pack` → 3 nupkgs @ 0.1.0).
|
||||
**Adopted by all four apps** — verified 2026-07-18 (each references the package and wires `MapZbHealth`); the previous "not yet adopted" claim was stale. Per-app probe coverage vs the spec has NOT been re-verified — see [`components/health/GAPS.md`](components/health/GAPS.md).
|
||||
Build/test from `ZB.MOM.WW.Health/`: `dotnet test`. Consumer matrix: MxAccessGateway → core only;
|
||||
OtOpcUa & ScadaBridge → all three packages.
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -5,7 +5,7 @@
|
||||
<Nullable>enable</Nullable>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
<LangVersion>latest</LangVersion>
|
||||
<Version>0.2.1</Version>
|
||||
<Version>0.3.4</Version>
|
||||
<ManagePackageVersionsCentrally>true</ManagePackageVersionsCentrally>
|
||||
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
|
||||
<!-- NU1510 (package-pruning advice): the Replication and Tests projects carry a
|
||||
|
||||
+126
-18
@@ -10,16 +10,100 @@ Change capture uses SQLite triggers writing a single HLC-stamped oplog inside th
|
||||
consumer's own write transaction (crash-consistent, non-bypassable). Convergence is
|
||||
last-writer-wins over a hybrid logical clock. All timestamps are UTC.
|
||||
|
||||
## Packages (all `0.2.1`)
|
||||
## Packages (all `0.3.4`)
|
||||
|
||||
| Package | Contents |
|
||||
|---|---|
|
||||
| `ZB.MOM.WW.LocalDb` | Embedded DB host: `ILocalDb` SQL surface (`ExecuteAsync` / `QueryAsync` / transactions), `RegisterReplicated(table[, baselineExistingRows])` (PK validation + trigger install + oplog schema), `DeregisterReplicated(table)` (trigger removal + oplog/ledger prune), `HybridLogicalClock`, `AddZbLocalDb()` DI. |
|
||||
| `ZB.MOM.WW.LocalDb` | Embedded DB host: `ILocalDb` SQL surface (`ExecuteAsync` / `QueryAsync` / transactions), `RegisterReplicated(table[, baselineExistingRows])` (PK + BLOB + foreign-key validation, trigger install, oplog schema), `DeregisterReplicated(table)` (trigger removal + oplog/ledger prune), `HybridLogicalClock`, `AddZbLocalDb()` DI. |
|
||||
| `ZB.MOM.WW.LocalDb.Contracts` | `localdb_sync.v1` proto + generated gRPC types (packable; a future peer could be non-.NET). |
|
||||
| `ZB.MOM.WW.LocalDb.Replication` | gRPC sync engine: passive `LocalDbSyncService`, initiating `SyncBackgroundService`, `MaintenanceBackgroundService`, LWW apply, watermark acks + pruning, snapshot resync, `ISyncStatus`, `LocalDbMetrics`, `AddZbLocalDbReplication()` / `MapZbLocalDbSync()`. |
|
||||
|
||||
> Status: **published to the Gitea feed; adopted by ScadaBridge (Phases 1+2) and OtOpcUa (Phase 1).**
|
||||
>
|
||||
> - `0.3.4` (2026-08-20) — **a converged pair no longer re-streams a full snapshot on every
|
||||
> reconnect once the sender's oplog has been ack-pruned to empty.** Found on the ScadaBridge
|
||||
> docker rig (its register row 106): a rebuilt node (watermark 0) was healed by a snapshot
|
||||
> stamped `as_of_seq = MAX(seq)` — which is 0 on an empty oplog — so its MAX-guarded watermark
|
||||
> stayed 0, and the healthy side's empty-oplog horizon rule (`0.1.2`: streamable from
|
||||
> `last_acked_seq + 1`) saw a gap on every later handshake and re-sent the whole database
|
||||
> (127k rows per restart on the rig) until it happened to write a new row. The as-of is now
|
||||
> `max(MAX(seq), last_acked_seq)`: every delta pruned below `last_acked_seq` is already folded
|
||||
> into the rows the snapshot carries, so that is the floor it may claim. Sender-side only,
|
||||
> wire-compatible, no schema change; pinned by
|
||||
> `WipedPeer_SnapshotLiftsItsWatermark_SoTheNextReconnectSendsNoSecondSnapshot`.
|
||||
> - `0.3.3` (2026-08-20) — two latent apply-path hazards from the ScadaBridge adversarial
|
||||
> architecture review (P4-7). Neither had a live trigger in any current consumer; both are the
|
||||
> kind that surface as silent divergence, not as an error. Wire-compatible: no proto change, no
|
||||
> `lib_schema_version` bump, no on-disk schema change.
|
||||
> - **A replicated table may no longer be the parent of a row-mutating `ON DELETE` foreign key.**
|
||||
> The apply path writes remote rows with `INSERT OR REPLACE`, and REPLACE resolves a PK conflict
|
||||
> by DELETING the existing row first. Connections run `PRAGMA foreign_keys=ON`, so that implicit
|
||||
> delete fires the child's `ON DELETE CASCADE` / `SET NULL` / `SET DEFAULT` — while the capture
|
||||
> guard is raised. The child rows change on the applying node ONLY, produce no oplog entry, and
|
||||
> the pair diverges with nothing able to detect it (LWW compares only rows it was told about).
|
||||
> `RegisterReplicated` now inspects every foreign key in the database and refuses such a table
|
||||
> with a `LocalDbRegistrationException` naming the offending child column and action.
|
||||
> `RESTRICT` / `NO ACTION` remain fine — they abort the statement instead of mutating rows
|
||||
> behind capture's back, which the apply path already handles as per-entry poison. Checked at
|
||||
> registration; SQLite cannot add a foreign key to an existing table, so a child table created
|
||||
> later is caught at the next startup's registration rather than at the DDL.
|
||||
> - **An entry's row write and its ledger write are now atomic with each other.** They are two
|
||||
> statements, and SQLite aborts only the failing one — so a statement-level fault between them
|
||||
> (the case the per-entry poison catch exists for) committed a half-applied entry: a tombstone
|
||||
> whose table row was gone while `__localdb_row_version` still called it live, which the snapshot
|
||||
> streamer can only warn about and skip, so the peer never learned the row died; or an upsert
|
||||
> whose new row carried the OLD version and therefore lost the next comparison it should have
|
||||
> won. The pair is wrapped in a per-entry `SAVEPOINT` that is rolled back before dead-lettering,
|
||||
> so a poison entry leaves both writes or neither and the rest of the batch is untouched.
|
||||
> - `0.3.2` (2026-08-19) — two Medium findings from the ScadaBridge adversarial architecture review
|
||||
> (P4-1, P4-3). Wire-compatible with every earlier peer: no proto change, no `lib_schema_version`
|
||||
> bump, no on-disk schema change — the keepalive is transport-level and the rest is local policy.
|
||||
> - **A live session now ends itself when a cap prune eats deltas it has not sent.** The pump reads
|
||||
> `seq > sentThruSeq`, which resumes SILENTLY above a pruned hole; the peer acks the later seqs
|
||||
> and the pair is durably divergent for the whole life of that session. `needs_snapshot` (set in
|
||||
> the prune's own transaction) only heals a RECONNECT, and nothing forced one — so on a stable
|
||||
> LAN with a slow-but-alive peer the divergence was unbounded. Every pump iteration now compares
|
||||
> the oplog horizon (`MIN(seq)`, one b-tree seek) against its sent watermark and throws
|
||||
> `LocalDbOplogGapException` when deltas it still owes are gone. The horizon, not the flag,
|
||||
> because the flag is also raised for reasons that do not invalidate the running session.
|
||||
> - **The sync stream has HTTP/2 keepalive, and the server's single-stream guard is newest-wins.**
|
||||
> Neither end ever timed out on silence (a converged pair is legitimately quiet), so a hard crash
|
||||
> or partition of the dialling node left the passive node's handler parked in `MoveNext` on a
|
||||
> half-open stream — holding the guard, reporting itself connected, and rejecting the rebooted
|
||||
> peer's reconnects with `AlreadyExists`, indefinitely on an idle pair. The initiator now dials
|
||||
> with `KeepAlivePingDelay` / `KeepAlivePingTimeout` (new options, 60s/20s, `Always` policy so
|
||||
> idle connections ping too), and a fresh authenticated `Sync` cancels the incumbent and waits
|
||||
> for it to unwind before starting — the guard serializes sessions, it was never a fence, and an
|
||||
> authenticated same-key dialler is by definition the peer. An incumbent that will not yield
|
||||
> within the takeover timeout still gets `AlreadyExists`, because two live sessions over one
|
||||
> database is worse than a refused dial.
|
||||
> - `0.3.1` (2026-08-18) — a faulting session flushes its control lane before cancelling its writer,
|
||||
> so a locally-detected handshake mismatch actually reaches the peer instead of leaving it to
|
||||
> reconnect-loop against a generic disconnect.
|
||||
> - `0.3.0` (2026-08-15) — **the snapshot-resync flag is per-table.** On-disk bookkeeping schema
|
||||
> goes to **v3** (new `__localdb_snapshot_state`, upgraded in place on open); wire-compatible with
|
||||
> `0.1.x`/`0.2.x` peers, and `lib_schema_version` deliberately stays `1` because it is compared
|
||||
> fail-closed for equality.
|
||||
> - **`RegisterReplicated(table, baselineExistingRows: true)` now flags a resync for THAT TABLE.**
|
||||
> Through `0.2.x` the flag was one bit for the whole database, so baselining one table
|
||||
> re-streamed every registered table in full — and on a pair enabling replication late, both
|
||||
> nodes did that to each other at once. A snapshot now streams only the flagged tables.
|
||||
> - **The database-wide flag survives, unchanged, for the cases that genuinely need it**: an oplog
|
||||
> cap prune or an impossible peer ack loses deltas for the oplog as a whole, so nothing short of
|
||||
> every table is sound. What moved is only where a *baseline* records its debt.
|
||||
> - **Additive wire extension, negotiated by capability, not by version.** `Handshake` gains
|
||||
> `supports_partial_snapshot`; `SnapshotBegin` gains a `tables` scope where **empty means "all
|
||||
> tables"** — which is precisely what a pre-`0.3.0` sender emits and what a pre-`0.3.0` receiver
|
||||
> assumes. A peer that does not advertise the capability is served a **full** snapshot, because a
|
||||
> scoped one would make it advance `last_applied_remote_seq` past deltas it never received.
|
||||
> - **A partial snapshot moves no watermark on either side.** The receiver keeps
|
||||
> `last_applied_remote_seq` (it is one number for the whole peer oplog, and the untouched tables'
|
||||
> deltas still sit below the snapshot's as-of); the sender keeps pumping from where it was. The
|
||||
> overlap re-delivers rows the snapshot already carried, which LWW absorbs.
|
||||
> - **Snapshot debt now clears on acknowledgement, not on enqueue** (new `SnapshotAck`, exchanged
|
||||
> only between `0.3.0` peers). A crash part-way through a snapshot re-streams the same scope on
|
||||
> reconnect instead of losing it. Against a pre-`0.3.0` peer the `0.2.x` clear-on-enqueue
|
||||
> behaviour is kept — there is no ack to wait for.
|
||||
> - `0.2.1` (2026-08-14) — two adversarial-review findings against `0.2.0`. Wire-compatible with
|
||||
> `0.1.x`/`0.2.0` peers; no proto change, no `lib_schema_version` bump, no on-disk schema change.
|
||||
> - **`DeregisterReplicated` no longer leaves the HLC without a crash anchor.** Clock recovery on
|
||||
@@ -177,10 +261,15 @@ All options are validated at startup (`ZB.MOM.WW.Configuration` validators).
|
||||
delta/snapshot batches), so receiving never blocks on the ability to ack.
|
||||
- **Resync** — when a peer's watermark falls below the pruned oplog horizon, it is sent a
|
||||
full snapshot of the registered tables (streamed, keyset-paged) and deltas resume above
|
||||
the snapshot's as-of seq. **Each node decides from its OWN state** whether it owes the
|
||||
peer a snapshot. The snapshot is streamed *after* the handshake, alongside a running
|
||||
receive loop, so a node sending one still applies what the peer sends it — a mutual
|
||||
snapshot exchanges under flow control instead of being buffered whole in memory.
|
||||
the snapshot's as-of seq. When instead a single table was *baselined*, only that table is
|
||||
streamed and **no watermark moves on either side** — a partial snapshot speaks for nothing
|
||||
outside its own tables, so the deltas it overlaps are still pumped and merged by LWW.
|
||||
**Each node decides from its OWN state** whether it owes the peer a snapshot, and a peer
|
||||
that does not advertise `supports_partial_snapshot` is sent a full one. The snapshot is
|
||||
streamed *after* the handshake, alongside a running receive loop, so a node sending one
|
||||
still applies what the peer sends it — a mutual snapshot exchanges under flow control
|
||||
instead of being buffered whole in memory. Between `0.3.0` peers the sender's flags are
|
||||
cleared by the receiver's `SnapshotAck`, so a crash mid-snapshot re-streams it.
|
||||
- **Maintenance** — a background service on **both** roles enforces the oplog row/age caps
|
||||
and prunes expired tombstones every `MaintenanceInterval`.
|
||||
|
||||
@@ -206,20 +295,30 @@ All options are validated at startup (`ZB.MOM.WW.Configuration` validators).
|
||||
`RegisterReplicated`, because `json_object` cannot capture it). A *typeless* column can
|
||||
still hold a BLOB value at runtime, which then surfaces as a write-time error, not a
|
||||
registration error.
|
||||
- **A replicated table may not be the parent of a row-mutating `ON DELETE` foreign key**
|
||||
(`CASCADE` / `SET NULL` / `SET DEFAULT` from any other table, or from itself). Applying a
|
||||
remote row uses `INSERT OR REPLACE`, whose implicit delete fires that action with change
|
||||
capture suppressed — the child rows would change on the applying node only and never
|
||||
replicate, diverging the pair undetectably. `RegisterReplicated` refuses such a table
|
||||
(`0.3.3`); use `ON DELETE NO ACTION`/`RESTRICT` and delete children explicitly, or leave
|
||||
the parent unreplicated. The check runs at registration and reads the whole database's
|
||||
foreign keys, so a child table created *after* registration is caught at the next startup,
|
||||
not at the moment of the `CREATE TABLE`.
|
||||
- **CDC semantics** — rows that existed **before** `RegisterReplicated` are not captured
|
||||
and are not snapshotted; only changes after registration are replicated. **Opt out with
|
||||
`RegisterReplicated(table, baselineExistingRows: true)`**, which seeds the row-version
|
||||
ledger for existing rows at the baseline floor (HLC `0`) and flags a snapshot resync so
|
||||
they reach the peer. The floor is below every genuine stamp, so a baselined row loses
|
||||
ledger for existing rows at the baseline floor (HLC `0`) and flags a snapshot resync
|
||||
**for that table** so they reach the peer. The floor is below every genuine stamp, so a baselined row loses
|
||||
every comparison against a real remote write and wins only where the peer has no version
|
||||
of that key at all. Two nodes that baseline the same key both hold HLC `0` and resolve by
|
||||
the node-id tie-break — convergent, but arbitrary as to which content survives.
|
||||
- **The snapshot flag is per-database, not per-table.** Baselining one table flags a resync
|
||||
that streams **every** registered table in full, and on a pair enabling replication late
|
||||
both nodes do that to each other simultaneously — the largest transfer this library
|
||||
performs. Bounded (batch budget out, bounded inbox in), but **purge or trim high-volume
|
||||
tables such as event logs before enabling replication late**, and expect one slow first
|
||||
sync. A per-table flag needs an on-disk schema change and is not in `0.2.x`.
|
||||
- **A snapshot still ships whole tables.** The flag is per-table as of `0.3.0`, so baselining
|
||||
one table streams that table and no other — but *that* table travels in full, and two
|
||||
cases still stream **everything**: an oplog cap prune / impossible peer ack (delta
|
||||
continuity is lost for the oplog as a whole), and any peer older than `0.3.0` (which
|
||||
cannot be told a snapshot is partial). So **purge or trim high-volume tables such as event
|
||||
logs before baselining them**, and expect one slow first sync. Bounded either way (batch
|
||||
budget out, bounded inbox in).
|
||||
- **REAL ±Infinity** is carried in SQLite's JSON dialect on the wire (fine for the
|
||||
SQLite↔SQLite peers this library targets).
|
||||
- `AddZbLocalDb` / `AddZbLocalDbReplication` are **first-registration-wins** (idempotent).
|
||||
@@ -256,8 +355,15 @@ provider is wired or the poll threw, so a DB failure never reads as "0 backlog")
|
||||
|
||||
## Ops notes
|
||||
|
||||
- When the oplog row/age caps are exceeded, maintenance prunes to the ceiling and flags
|
||||
`needs_snapshot` (disk safety over delta continuity); the peer then snapshot-resyncs.
|
||||
- When the oplog row/age caps are exceeded, maintenance prunes to the ceiling and flags the
|
||||
**database-wide** `needs_snapshot` (disk safety over delta continuity); the peer then gets a
|
||||
full snapshot. Per-table debt lives separately in `__localdb_snapshot_state` and is what
|
||||
baselining writes.
|
||||
- **Upgrading a pair to `0.3.0`** needs no coordination: the two nodes negotiate scoped snapshots
|
||||
by capability, so a `0.3.0` node paired with a `0.2.x` one simply keeps sending full snapshots
|
||||
until both are upgraded. Rolling the file forward is one-way — a `0.2.x` build reopening a v3
|
||||
file leaves `__localdb_snapshot_state` in place and ignores it, so any per-table debt recorded
|
||||
by `0.3.0` is invisible to it (roll forward again, or re-baseline).
|
||||
- **Turning a table off** — `DeregisterReplicated(table)` drops its capture triggers and prunes
|
||||
its oplog + ledger rows. It must be applied **symmetrically**: the handshake compares
|
||||
registered-table digests fail-closed, so a node that drops a table its peer still
|
||||
@@ -281,14 +387,16 @@ provider is wired or the poll threw, so a DB failure never reads as "0 backlog")
|
||||
|
||||
## Testing
|
||||
|
||||
188 tests, all offline / no live dependencies — unit (HLC ordering + restart
|
||||
199 tests, all offline / no live dependencies — unit (HLC ordering + restart
|
||||
persistence, deregistration's HLC anchor surviving an ungraceful reopen, trigger SQL
|
||||
generation, LWW decision table, watermark/pruning, digest computation, schema-version
|
||||
upgrade, deregistration idempotency, baseline-floor LWW safety, O(1) depth vs a seeded
|
||||
backlog, byte-budget chunking, bounded-inbox backpressure under a slow reader), integration
|
||||
over two real SQLite DBs on an in-memory (and real) gRPC pipe (bidirectional convergence,
|
||||
delete/update races, offline-accumulate → reconnect, prune → snapshot resync, mutual
|
||||
snapshot over a capacity-1 transport), plus seeded randomized property tests (random interleaved op sequences → both DBs identical after quiesce).
|
||||
snapshot over a capacity-1 transport, single-table baseline streaming only that table,
|
||||
mixed-version interop where a peer without `supports_partial_snapshot` gets a full one,
|
||||
snapshot debt surviving until the peer's `SnapshotAck`), plus seeded randomized property tests (random interleaved op sequences → both DBs identical after quiesce).
|
||||
Family conventions apply (`TreatWarningsAsErrors`).
|
||||
|
||||
Build / test from this directory:
|
||||
|
||||
@@ -16,9 +16,19 @@ set -euo pipefail
|
||||
: "${GITEA_NUGET_SOURCE:?set GITEA_NUGET_SOURCE to your Gitea NuGet feed URL}"
|
||||
: "${GITEA_NUGET_KEY:?set GITEA_NUGET_KEY to your Gitea access token}"
|
||||
|
||||
dotnet pack -c Release -o ./artifacts
|
||||
# Named solution, named output, and a cd to the repo root, all for one reason: `dotnet pack`
|
||||
# with neither a project argument nor a fixed working directory packs whatever solution
|
||||
# happens to sit in the CALLER's directory — and then pushes it to a live feed under this
|
||||
# script's name. Invoked as `bash /path/to/ZB.MOM.WW.LocalDb/build/push.sh` from another
|
||||
# repo's checkout, the relative form packed and published 25 unrelated packages (recovered
|
||||
# with `dotnet nuget delete`, but only because the token happened to carry delete rights).
|
||||
# A publish script must be unambiguous about WHAT it publishes no matter where it is run.
|
||||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "$REPO_ROOT"
|
||||
|
||||
dotnet nuget push "./artifacts/*.nupkg" \
|
||||
dotnet pack ZB.MOM.WW.LocalDb.slnx -c Release -o "$REPO_ROOT/artifacts"
|
||||
|
||||
dotnet nuget push "$REPO_ROOT/artifacts/*.nupkg" \
|
||||
--source "$GITEA_NUGET_SOURCE" \
|
||||
--api-key "$GITEA_NUGET_KEY" \
|
||||
--skip-duplicate
|
||||
|
||||
@@ -16,6 +16,7 @@ message SyncMessage {
|
||||
SnapshotBegin snapshot_begin = 5; // sender switches to snapshot mode
|
||||
SnapshotBatch snapshot_batch = 6;
|
||||
SnapshotComplete snapshot_complete = 7;
|
||||
SnapshotAck snapshot_ack = 8; // 0.3.0+; only sent when the peer advertised support
|
||||
}
|
||||
}
|
||||
|
||||
@@ -24,6 +25,13 @@ message Handshake {
|
||||
uint32 lib_schema_version = 2;
|
||||
repeated TableDigest tables = 3; // fail-closed on any mismatch
|
||||
int64 last_applied_remote_seq = 4; // highest of YOUR oplog seqs I have applied
|
||||
// 0.3.0+ capability, additive: I understand a snapshot scoped to a SUBSET of the registered
|
||||
// tables (SnapshotBegin.tables) and I will answer a completed snapshot with a SnapshotAck.
|
||||
// A pre-0.3.0 peer leaves this false — proto3 default — and a 0.3.0 sender then promotes any
|
||||
// scoped snapshot to a full one and clears its flags on enqueue, exactly as 0.2.x did. This is
|
||||
// why lib_schema_version stays 1: it is compared fail-closed for EQUALITY, so bumping it would
|
||||
// stop 0.1.x/0.2.x pairs syncing outright.
|
||||
bool supports_partial_snapshot = 5;
|
||||
}
|
||||
message TableDigest { string table_name = 1; string digest = 2; }
|
||||
message HandshakeAck { string node_id = 1; bool snapshot_required = 2; }
|
||||
@@ -40,7 +48,12 @@ message OplogEntry {
|
||||
message DeltaBatch { repeated OplogEntry entries = 1; }
|
||||
message DeltaAck { int64 applied_thru_seq = 1; }
|
||||
|
||||
message SnapshotBegin { int64 as_of_seq = 1; }
|
||||
// tables: the SCOPE of this snapshot (0.3.0+). EMPTY means every registered table — a full
|
||||
// snapshot, which is the only thing a pre-0.3.0 sender ever emits and the only thing a
|
||||
// pre-0.3.0 receiver can safely assume. A non-empty list means the snapshot is PARTIAL, so the
|
||||
// receiver must NOT advance last_applied_remote_seq to as_of_seq (deltas below it for the tables
|
||||
// NOT listed have not been delivered); it merges the rows through LWW and leaves the watermark.
|
||||
message SnapshotBegin { int64 as_of_seq = 1; repeated string tables = 2; }
|
||||
message SnapshotBatch { string table_name = 1; repeated SnapshotRow rows = 2; }
|
||||
message SnapshotRow {
|
||||
string pk_json = 1;
|
||||
@@ -50,3 +63,8 @@ message SnapshotRow {
|
||||
bool is_tombstone = 5;
|
||||
}
|
||||
message SnapshotComplete { int64 as_of_seq = 1; }
|
||||
// Sent by the RECEIVER once a snapshot has been durably applied, so the sender clears the
|
||||
// per-table snapshot flags only after the peer actually holds the rows (a crash mid-snapshot then
|
||||
// re-streams the same scope on reconnect). Only exchanged when both peers advertised
|
||||
// supports_partial_snapshot; a pre-0.3.0 pair keeps the 0.2.x clear-on-enqueue behaviour.
|
||||
message SnapshotAck { int64 as_of_seq = 1; }
|
||||
|
||||
@@ -12,7 +12,8 @@ internal sealed record ApplyResult(long AppliedThruSeq, int Applied, int Discard
|
||||
/// Applies a peer's oplog batch under last-writer-wins in one transaction: the applying guard is
|
||||
/// raised so capture triggers skip these writes, per-entry LWW resolves against the local
|
||||
/// row-version (hlc, then node_id ordinal tie-break), poison rows are dead-lettered while the
|
||||
/// batch continues, and the remote watermark advances in the SAME transaction as the data so a
|
||||
/// batch continues — each entry's row write and ledger write sharing a savepoint so a dead-lettered
|
||||
/// entry leaves neither — and the remote watermark advances in the SAME transaction as the data so a
|
||||
/// crash-replay of the batch is a no-op (LWW's strict-greater compare loses on equal versions).
|
||||
/// This never writes <c>__localdb_oplog</c>: the 2-node topology does not re-forward applied deltas.
|
||||
/// </summary>
|
||||
@@ -88,15 +89,28 @@ internal sealed class LwwApplier(SqliteLocalDb db, Func<DateTimeOffset>? utcNow
|
||||
continue;
|
||||
}
|
||||
|
||||
var table = db.ReplicatedTables[entry.TableName];
|
||||
|
||||
// An entry is TWO statements — the row change and its ledger row — and SQLite aborts
|
||||
// only the failing statement, so a fault between them used to commit a half-applied
|
||||
// entry: a tombstone whose table row is gone while __localdb_row_version still calls it
|
||||
// live (which SnapshotStreamer can only warn about and skip, so the peer never learns
|
||||
// the row died), or an upsert whose new row carries the OLD version and therefore loses
|
||||
// the next LWW comparison it should win. A per-entry savepoint makes the pair
|
||||
// all-or-nothing without widening the poison catch to the whole batch.
|
||||
//
|
||||
// Opened OUTSIDE the try on purpose: a SAVEPOINT that failed opened nothing, so the
|
||||
// handler's ROLLBACK TO would name a savepoint that does not exist and fault in turn.
|
||||
await sql.EnterEntryAsync(ct);
|
||||
try
|
||||
{
|
||||
var table = db.ReplicatedTables[entry.TableName];
|
||||
if (entry.IsTombstone)
|
||||
await sql.DeleteAsync(table, entry.PkJson, ct);
|
||||
else
|
||||
await sql.UpsertAsync(table, entry.RowJson!, ct);
|
||||
|
||||
await sql.WriteRowVersionAsync(entry, utcNow, ct);
|
||||
await sql.ReleaseEntryAsync(ct);
|
||||
applied++;
|
||||
}
|
||||
catch (SqliteException ex)
|
||||
@@ -104,8 +118,29 @@ internal sealed class LwwApplier(SqliteLocalDb db, Func<DateTimeOffset>? utcNow
|
||||
// Deliberately narrow: only statement-level SQLite errors (e.g. json_extract on
|
||||
// malformed row_json) are treated as per-entry poison — SQLite aborts just the
|
||||
// statement, not the transaction, so the batch continues after dead-lettering.
|
||||
// Any other exception propagates and rolls back the whole batch (fail-closed).
|
||||
await sql.DeadLetterAsync(entry, ex.Message, utcNow, ct);
|
||||
// Any other exception propagates with the savepoint still open, and disposing the
|
||||
// outer transaction unwinds the whole batch (fail-closed).
|
||||
try
|
||||
{
|
||||
await sql.RollbackEntryAsync(ct);
|
||||
await sql.DeadLetterAsync(entry, ex.Message, utcNow, ct);
|
||||
}
|
||||
catch (Exception recoveryFailure) when (recoveryFailure is not OperationCanceledException)
|
||||
{
|
||||
// Recovering from the poison entry itself failed — a wedged connection, an I/O
|
||||
// error, a dead-letter INSERT that faults. The outcome is unchanged and still
|
||||
// fail-closed (nothing commits; disposing the batch transaction unwinds it),
|
||||
// but reporting only the recovery failure would throw away the ORIGINAL cause,
|
||||
// which is the one that says WHY this entry is poison. Both are carried out:
|
||||
// the original as InnerException (AggregateException surfaces the first inner
|
||||
// there), the recovery failure alongside it with its own stack intact.
|
||||
throw new AggregateException(
|
||||
$"Failed to dead-letter a poison oplog entry for table '{entry.TableName}' " +
|
||||
$"(seq {entry.Seq}): {recoveryFailure.Message}. The entry's own fault is the " +
|
||||
"inner exception. The batch is not committed.",
|
||||
ex, recoveryFailure);
|
||||
}
|
||||
|
||||
deadLettered++;
|
||||
}
|
||||
}
|
||||
@@ -138,7 +173,13 @@ internal sealed class LwwApplier(SqliteLocalDb db, Func<DateTimeOffset>? utcNow
|
||||
private readonly Dictionary<string, SqliteCommand> _deletes = new(StringComparer.Ordinal);
|
||||
private readonly Dictionary<string, SqliteCommand> _upserts = new(StringComparer.Ordinal);
|
||||
|
||||
/// <summary>Name of the per-entry savepoint. Fixed: only one is ever open at a time.</summary>
|
||||
private const string EntrySavepoint = "zb_lww_entry";
|
||||
|
||||
private SqliteCommand? _applying;
|
||||
private SqliteCommand? _enterEntry;
|
||||
private SqliteCommand? _releaseEntry;
|
||||
private SqliteCommand? _rollbackEntry;
|
||||
private SqliteCommand? _lookup;
|
||||
private SqliteCommand? _rowVersion;
|
||||
private SqliteCommand? _deadLetter;
|
||||
@@ -155,6 +196,32 @@ internal sealed class LwwApplier(SqliteLocalDb db, Func<DateTimeOffset>? utcNow
|
||||
return cmd.ExecuteNonQueryAsync(ct);
|
||||
}
|
||||
|
||||
/// <summary>Opens the per-entry savepoint inside the batch transaction.</summary>
|
||||
public Task EnterEntryAsync(CancellationToken ct)
|
||||
{
|
||||
var cmd = _enterEntry ??= Create($"SAVEPOINT {EntrySavepoint}");
|
||||
return cmd.ExecuteNonQueryAsync(ct);
|
||||
}
|
||||
|
||||
/// <summary>Commits the entry's work into the batch transaction (pops the savepoint).</summary>
|
||||
public Task ReleaseEntryAsync(CancellationToken ct)
|
||||
{
|
||||
var cmd = _releaseEntry ??= Create($"RELEASE {EntrySavepoint}");
|
||||
return cmd.ExecuteNonQueryAsync(ct);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Undoes everything the entry did and pops the savepoint, leaving the batch transaction
|
||||
/// open and usable. Two statements, because ROLLBACK TO rewinds to the savepoint but leaves
|
||||
/// it ON the stack — without the RELEASE, every dead-lettered entry would nest another one.
|
||||
/// </summary>
|
||||
public async Task RollbackEntryAsync(CancellationToken ct)
|
||||
{
|
||||
var cmd = _rollbackEntry ??= Create($"ROLLBACK TO {EntrySavepoint}");
|
||||
await cmd.ExecuteNonQueryAsync(ct);
|
||||
await ReleaseEntryAsync(ct);
|
||||
}
|
||||
|
||||
public async Task<(long Hlc, string NodeId)?> ReadLocalVersionAsync(
|
||||
string tableName, string pkJson, CancellationToken ct)
|
||||
{
|
||||
|
||||
@@ -24,8 +24,9 @@ internal sealed class OplogStore(ILocalDb db, ReplicationOptions options, Func<D
|
||||
|
||||
/// <summary>
|
||||
/// Reads up to <paramref name="maxBatch"/> oplog entries with <c>seq > afterSeq</c> in ascending
|
||||
/// seq order. Caller contract: check <c>needs_snapshot</c> before trusting incremental reads —
|
||||
/// after cap-pruning, reads silently resume past a gap.
|
||||
/// seq order. Caller contract: this read resumes SILENTLY past a cap-pruned gap, so a caller
|
||||
/// streaming incrementally must first confirm its position is still below the horizon — see
|
||||
/// <see cref="GetMinSeqAsync"/>, which <c>SyncSession</c>'s pump checks on every iteration.
|
||||
/// </summary>
|
||||
public Task<IReadOnlyList<OplogEntryRecord>> ReadBatchAboveAsync(long afterSeq, int maxBatch, CancellationToken ct = default) =>
|
||||
db.QueryAsync(
|
||||
@@ -148,6 +149,24 @@ internal sealed class OplogStore(ILocalDb db, ReplicationOptions options, Func<D
|
||||
return rows[0];
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The oldest oplog seq still present — the delta horizon — or 0 when the oplog is empty.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// One b-tree seek (<c>seq</c> is the rowid and SQLite optimizes MIN into a seek), which is what
|
||||
/// makes it affordable for the pump to check on every flush interval. Compared against a
|
||||
/// session's sent watermark it answers the only question the pump has: is everything I have not
|
||||
/// yet sent still here? Ack-pruning deletes <c>seq <= last_acked_seq</c> and the peer can only
|
||||
/// ack what it received, so in a healthy session the horizon never climbs past
|
||||
/// <c>sentThruSeq + 1</c>; a higher one means a cap prune ate unsent deltas.
|
||||
/// </remarks>
|
||||
public async Task<long> GetMinSeqAsync(CancellationToken ct = default)
|
||||
{
|
||||
var rows = await db.QueryAsync(
|
||||
"SELECT COALESCE(MIN(seq), 0) FROM __localdb_oplog", static r => r.GetInt64(0), null, ct);
|
||||
return rows[0];
|
||||
}
|
||||
|
||||
/// <summary>The highest oplog seq present, or 0 when the oplog is empty.</summary>
|
||||
public async Task<long> GetMaxSeqAsync(CancellationToken ct = default)
|
||||
{
|
||||
@@ -233,10 +252,61 @@ internal sealed class OplogStore(ILocalDb db, ReplicationOptions options, Func<D
|
||||
db.ExecuteAsync(
|
||||
"UPDATE __localdb_peer_state SET last_applied_remote_seq = 0, last_seen_hlc = 0 WHERE id = 1", null, ct);
|
||||
|
||||
/// <summary>Sets or clears the snapshot-resync-required flag.</summary>
|
||||
/// <summary>
|
||||
/// Sets or clears the DATABASE-WIDE snapshot-resync flag — the one that means "every registered
|
||||
/// table". Set by the oplog-horizon paths (cap prune, impossible peer ack), which invalidate
|
||||
/// delta continuity for all tables at once. Per-table flags live in
|
||||
/// <c>__localdb_snapshot_state</c>; see <see cref="GetTablesNeedingSnapshotAsync"/>.
|
||||
/// </summary>
|
||||
public Task SetNeedsSnapshotAsync(bool value, CancellationToken ct = default) =>
|
||||
db.ExecuteAsync("UPDATE __localdb_peer_state SET needs_snapshot = @v WHERE id = 1", new { v = value ? 1 : 0 }, ct);
|
||||
|
||||
/// <summary>
|
||||
/// The tables individually flagged as owing the peer a snapshot (schema v3), in ordinal name
|
||||
/// order. Independent of the database-wide flag: baselining one table flags only that table.
|
||||
/// </summary>
|
||||
public Task<IReadOnlyList<string>> GetTablesNeedingSnapshotAsync(CancellationToken ct = default) =>
|
||||
db.QueryAsync(
|
||||
"SELECT table_name FROM __localdb_snapshot_state WHERE needs_snapshot <> 0 ORDER BY table_name",
|
||||
static r => r.GetString(0), null, ct);
|
||||
|
||||
/// <summary>
|
||||
/// Clears the snapshot debt a completed snapshot discharged: the named tables when
|
||||
/// <paramref name="tables"/> is non-empty, otherwise <b>everything</b> — the database-wide flag
|
||||
/// and every per-table row, because a full snapshot subsumes each of them.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// Rows are deleted rather than zeroed: absence is the resting state, and keeping a row per
|
||||
/// table that has ever been baselined would accumulate bookkeeping that means nothing. Runs in
|
||||
/// one transaction so a full snapshot's two clears cannot half-apply and leave the pair
|
||||
/// re-snapshotting on every reconnect.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// A scoped clear deliberately does NOT touch the database-wide flag. If the cap enforcer
|
||||
/// flagged a whole-database resync while this snapshot was in flight, that debt is real and
|
||||
/// unrelated — the snapshot only carried some of the tables — so it must survive. The full
|
||||
/// clear may take it down because a full snapshot carries every row state as of its as-of seq,
|
||||
/// which is exactly what the pruned deltas would have delivered.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public async Task ClearSnapshotDebtAsync(IReadOnlyList<string> tables, CancellationToken ct = default)
|
||||
{
|
||||
await using var tx = await db.BeginTransactionAsync(ct);
|
||||
if (tables.Count == 0)
|
||||
{
|
||||
await tx.ExecuteAsync("UPDATE __localdb_peer_state SET needs_snapshot = 0 WHERE id = 1", null, ct);
|
||||
await tx.ExecuteAsync("DELETE FROM __localdb_snapshot_state", null, ct);
|
||||
}
|
||||
else
|
||||
{
|
||||
foreach (var table in tables)
|
||||
await tx.ExecuteAsync(
|
||||
"DELETE FROM __localdb_snapshot_state WHERE table_name = @t", new { t = table }, ct);
|
||||
}
|
||||
await tx.CommitAsync(ct);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Records the highest remote seq applied locally plus the peer HLC observed with it; also stamps
|
||||
/// <c>last_sync_utc</c>. Both watermarks are MAX-guarded (monotonic): a snapshot completing with a
|
||||
|
||||
@@ -19,21 +19,66 @@ internal interface ISnapshotApplier
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Streams a full snapshot of every registered table to a peer that has fallen behind the pruned
|
||||
/// oplog horizon. Reads the authoritative per-row version from <c>__localdb_row_version</c> joined
|
||||
/// Which tables a snapshot covers. An EMPTY <see cref="Tables"/> is the wire's own encoding of
|
||||
/// "all registered tables" (see <c>SnapshotBegin.tables</c>) and is what every pre-0.3.0 peer both
|
||||
/// sends and expects, so it is deliberately the same value here rather than a separate flag.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The distinction is not cosmetic: a FULL snapshot carries every row state as of its
|
||||
/// <c>as_of_seq</c>, so the receiver may jump its watermark there and the sender may skip the
|
||||
/// deltas it covered. A PARTIAL one carries only the listed tables, so neither side may move a
|
||||
/// watermark that also speaks for the tables it left out.
|
||||
/// </remarks>
|
||||
internal sealed record SnapshotScope(IReadOnlyList<string> Tables)
|
||||
{
|
||||
/// <summary>Every registered table — the only scope a pre-0.3.0 peer can be sent.</summary>
|
||||
public static readonly SnapshotScope Full = new([]);
|
||||
|
||||
public bool IsPartial => Tables.Count > 0;
|
||||
|
||||
public bool Includes(string tableName) =>
|
||||
Tables.Count == 0 || Tables.Contains(tableName, StringComparer.Ordinal);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Streams a snapshot of the tables in the requested <see cref="SnapshotScope"/> — every registered
|
||||
/// table for a peer that has fallen behind the pruned oplog horizon, or just the baselined ones for
|
||||
/// a scoped resync. Reads the authoritative per-row version from <c>__localdb_row_version</c> joined
|
||||
/// to each live table, pages it into <see cref="SnapshotBatch"/> messages through the gated data
|
||||
/// lane, and brackets them with SnapshotBegin/SnapshotComplete carrying the max oplog seq as-of.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Clearing the snapshot debt is deliberately NOT done here: whether the flags may be cleared at all
|
||||
/// depends on the peer's capability (a 0.3.0 peer acknowledges the snapshot, a pre-0.3.0 one cannot),
|
||||
/// which is session state. <see cref="SyncSession"/> owns that decision; this type only streams.
|
||||
/// </remarks>
|
||||
internal sealed class SnapshotStreamer(SqliteLocalDb db, OplogStore store, ReplicationOptions options, ILogger logger)
|
||||
{
|
||||
/// <summary>Streams the full snapshot; returns the as-of oplog seq it covers, so the session can skip already-covered tail deltas.</summary>
|
||||
public async Task<long> SendAsync(Func<SyncMessage, CancellationToken, Task> send, CancellationToken ct)
|
||||
/// <summary>
|
||||
/// Streams the snapshot for <paramref name="scope"/>; returns the as-of oplog seq it covers, so
|
||||
/// the session can skip already-covered tail deltas (a FULL snapshot only — a partial one speaks
|
||||
/// for nothing outside its tables).
|
||||
/// </summary>
|
||||
public async Task<long> SendAsync(SnapshotScope scope, Func<SyncMessage, CancellationToken, Task> send, CancellationToken ct)
|
||||
{
|
||||
var asOfSeq = await store.GetMaxSeqAsync(ct);
|
||||
await send(new SyncMessage { SnapshotBegin = new SnapshotBegin { AsOfSeq = asOfSeq } }, ct);
|
||||
// The as-of is the highest seq this snapshot can stand in for. MAX(seq) alone is wrong once
|
||||
// ack-pruning has emptied the oplog: it reads 0, and a receiver whose watermark genuinely IS
|
||||
// 0 (a rebuilt node) is then never lifted by the snapshot — its MAX-guarded watermark stays
|
||||
// 0, the empty-oplog horizon rule (streamable from last_acked_seq + 1) sees a gap on the
|
||||
// next handshake, and the whole database is re-streamed on every reconnect until this node
|
||||
// happens to write again. Every delta pruned below last_acked_seq is already folded into the
|
||||
// rows this snapshot carries, so last_acked_seq is the floor the as-of must carry.
|
||||
var asOfSeq = Math.Max(
|
||||
await store.GetMaxSeqAsync(ct),
|
||||
(await store.GetPeerStateAsync(ct)).LastAckedSeq);
|
||||
var begin = new SnapshotBegin { AsOfSeq = asOfSeq };
|
||||
begin.Tables.AddRange(scope.Tables);
|
||||
await send(new SyncMessage { SnapshotBegin = begin }, ct);
|
||||
|
||||
var totalRows = 0L;
|
||||
foreach (var table in db.ReplicatedTables.Values.OrderBy(t => t.Name, StringComparer.Ordinal))
|
||||
foreach (var table in db.ReplicatedTables.Values
|
||||
.Where(t => scope.Includes(t.Name))
|
||||
.OrderBy(t => t.Name, StringComparer.Ordinal))
|
||||
{
|
||||
var sql = BuildPageSql(table);
|
||||
// Keyset cursor (pk_json > last seen), not LIMIT/OFFSET: concurrent inserts/prunes
|
||||
@@ -111,16 +156,9 @@ internal sealed class SnapshotStreamer(SqliteLocalDb db, OplogStore store, Repli
|
||||
}
|
||||
|
||||
await send(new SyncMessage { SnapshotComplete = new SnapshotComplete { AsOfSeq = asOfSeq } }, ct);
|
||||
// The sender owed this snapshot (its own needs_snapshot); clearing it here — after the whole
|
||||
// snapshot is on the wire — stops it re-streaming on the next reconnect. Narrow window: the
|
||||
// messages are enqueued on the data lane, not yet transport-delivered, so a crash/drop after
|
||||
// this clear could lose the snapshot with the flag already down. The reconnect handshake's
|
||||
// gap check (peer watermark below the pruned horizon) re-detects that in every case where the
|
||||
// oplog is non-empty; the sole undetectable corner is an EMPTY oplog (nothing to gap-check),
|
||||
// where the peer stays behind until new local writes recreate a horizon. Accepted: closing it
|
||||
// would need transport-level delivery acks for snapshot frames.
|
||||
await store.SetNeedsSnapshotAsync(false, ct);
|
||||
logger.LogInformation("Snapshot sent (as_of_seq {AsOfSeq}, {Rows} rows).", asOfSeq, totalRows);
|
||||
logger.LogInformation(
|
||||
"Snapshot sent (as_of_seq {AsOfSeq}, {Rows} rows, scope {Scope}).",
|
||||
asOfSeq, totalRows, scope.IsPartial ? string.Join(",", scope.Tables) : "all");
|
||||
return asOfSeq;
|
||||
}
|
||||
|
||||
@@ -181,6 +219,7 @@ internal sealed class SnapshotApplier : ISnapshotApplier
|
||||
private long _asOfSeq;
|
||||
private long _rows;
|
||||
private long _maxHlc;
|
||||
private bool _partial;
|
||||
|
||||
// db is part of the collaborator's canonical shape (mirrors the streamer) even though the merge
|
||||
// reaches the database through the applier and store; kept for construction symmetry in the factory.
|
||||
@@ -197,7 +236,13 @@ internal sealed class SnapshotApplier : ISnapshotApplier
|
||||
_asOfSeq = begin.AsOfSeq;
|
||||
_rows = 0;
|
||||
_maxHlc = 0;
|
||||
_logger.LogInformation("Snapshot receive begun (as_of_seq {AsOfSeq}).", _asOfSeq);
|
||||
// A non-empty table list scopes the snapshot; empty (which is all a pre-0.3.0 sender can
|
||||
// emit) means every registered table. Read here rather than on complete because
|
||||
// SnapshotComplete does not repeat it — the receive loop guarantees a Begin precedes both.
|
||||
_partial = begin.Tables.Count > 0;
|
||||
_logger.LogInformation(
|
||||
"Snapshot receive begun (as_of_seq {AsOfSeq}, scope {Scope}).",
|
||||
_asOfSeq, _partial ? string.Join(",", begin.Tables) : "all");
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
|
||||
@@ -219,9 +264,20 @@ internal sealed class SnapshotApplier : ISnapshotApplier
|
||||
|
||||
public async Task OnCompleteAsync(SnapshotComplete complete, CancellationToken ct)
|
||||
{
|
||||
// A PARTIAL snapshot may not move the inbound seq watermark. That watermark is one number
|
||||
// for the whole peer oplog, and jumping it to as_of_seq would declare every table applied
|
||||
// through that point — including the ones this snapshot deliberately left out, whose deltas
|
||||
// below as_of_seq the sender would then never resend. Only the HLC is carried across (seq 0
|
||||
// is a no-op against the MAX guard), so the clock still cannot regress below what the peer
|
||||
// has stamped. The sender makes the mirror-image concession: it does not advance its own
|
||||
// sent-thru watermark past a partial snapshot, so those deltas are still pumped normally
|
||||
// and merely re-deliver rows the snapshot already carried, which LWW absorbs.
|
||||
var seq = _partial ? 0 : complete.AsOfSeq;
|
||||
// SetLastAppliedRemoteSeqAsync is MAX-guarded, so a delta applied mid-snapshot with a seq
|
||||
// above the snapshot's as-of (or an HLC above what the snapshot carried) is never regressed.
|
||||
await _store.SetLastAppliedRemoteSeqAsync(complete.AsOfSeq, _maxHlc, ct);
|
||||
_logger.LogInformation("Snapshot receive complete (as_of_seq {AsOfSeq}, {Rows} rows).", complete.AsOfSeq, _rows);
|
||||
await _store.SetLastAppliedRemoteSeqAsync(seq, _maxHlc, ct);
|
||||
_logger.LogInformation(
|
||||
"Snapshot receive complete (as_of_seq {AsOfSeq}, {Rows} rows, partial {Partial}).",
|
||||
complete.AsOfSeq, _rows, _partial);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -27,6 +27,10 @@ internal sealed class SyncSession
|
||||
// inbound batch rate (the peer's flow control), so its lane can safely be unbounded.
|
||||
private const int DataLaneCapacity = 16;
|
||||
|
||||
// How long a faulting session's teardown lets the writer keep flushing already-enqueued
|
||||
// messages before the backstop cancel; only ever reached when the transport itself stalls.
|
||||
private static readonly TimeSpan FaultFlushGrace = TimeSpan.FromSeconds(2);
|
||||
|
||||
private readonly SqliteLocalDb _db;
|
||||
private readonly OplogStore _store;
|
||||
private readonly LwwApplier _applier;
|
||||
@@ -47,6 +51,17 @@ internal sealed class SyncSession
|
||||
// single receive-loop task, so no synchronization is needed.
|
||||
private bool _snapshotReceiving;
|
||||
|
||||
// Whether the peer speaks the 0.3.0 snapshot extensions: scoped snapshots
|
||||
// (SnapshotBegin.tables) and the SnapshotAck that answers a completed one. Written once during
|
||||
// the handshake — which completes before either loop starts — then read-only.
|
||||
private bool _peerSupportsPartialSnapshot;
|
||||
|
||||
// The scope of the snapshot this node is streaming, held until the peer's SnapshotAck discharges
|
||||
// it; null on every session that owes none. Assigned on the pump task strictly BEFORE the
|
||||
// snapshot's first message is enqueued and read on the receive loop when the ack lands, so the
|
||||
// ordering is causal — volatile only to make the cross-thread publication explicit.
|
||||
private SnapshotScope? _sentSnapshotScope;
|
||||
|
||||
public SyncSession(
|
||||
SqliteLocalDb db, OplogStore store, LwwApplier applier, ReplicationOptions options, ILogger logger,
|
||||
Func<DateTimeOffset>? utcNow = null, LocalDbMetrics? metrics = null, SyncStatus? status = null)
|
||||
@@ -63,11 +78,12 @@ internal sealed class SyncSession
|
||||
|
||||
/// <summary>
|
||||
/// Invoked when THIS node owes the peer a snapshot (its own snapshot-required computation, not
|
||||
/// the peer's) before deltas resume. Receives a gated send delegate that enqueues on the bounded
|
||||
/// data lane (never the raw transport); returns the as-of oplog seq the snapshot covers so the
|
||||
/// pump can skip already-covered tail deltas. Wired to <see cref="SnapshotStreamer.SendAsync"/>.
|
||||
/// the peer's) before deltas resume. Receives the negotiated <see cref="SnapshotScope"/> and a
|
||||
/// gated send delegate that enqueues on the bounded data lane (never the raw transport); returns
|
||||
/// the as-of oplog seq the snapshot covers so the pump can skip already-covered tail deltas.
|
||||
/// Wired to <see cref="SnapshotStreamer.SendAsync"/>.
|
||||
/// </summary>
|
||||
public Func<Func<SyncMessage, CancellationToken, Task>, CancellationToken, Task<long>>? SnapshotSender { get; set; }
|
||||
public Func<SnapshotScope, Func<SyncMessage, CancellationToken, Task>, CancellationToken, Task<long>>? SnapshotSender { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// Consumes an inbound snapshot inline on the receive loop: SnapshotBegin opens the receive,
|
||||
@@ -99,7 +115,7 @@ internal sealed class SyncSession
|
||||
var writer = WriterLoopAsync(duplex, control.Reader, data.Reader, linked.Token);
|
||||
try
|
||||
{
|
||||
var snapshotRequired = await HandshakeAsync(duplex, control.Writer, linked.Token);
|
||||
var snapshotScope = await HandshakeAsync(duplex, control.Writer, linked.Token);
|
||||
|
||||
// The receive loop starts BEFORE any snapshot this node owes is streamed. That ordering
|
||||
// is load-bearing, not incidental: a snapshot sent from inside the handshake left the
|
||||
@@ -111,7 +127,7 @@ internal sealed class SyncSession
|
||||
// ordinary flow control. Interleaving is already safe by design: snapshot rows and
|
||||
// deltas both merge through LWW, and the snapshot's watermark write is MAX-guarded.
|
||||
var receive = ReceiveLoopAsync(duplex, control.Writer, linked.Token);
|
||||
var pump = SnapshotThenPumpLoopAsync(snapshotRequired, GatedDataSend, data.Writer, linked.Token);
|
||||
var pump = SnapshotThenPumpLoopAsync(snapshotScope, GatedDataSend, data.Writer, linked.Token);
|
||||
var loops = new[] { pump, receive, writer };
|
||||
|
||||
// The first COMPLETED task is the primary outcome. Cancel the siblings, observe every
|
||||
@@ -128,19 +144,32 @@ internal sealed class SyncSession
|
||||
{
|
||||
// Handshake-phase faults (and the primary rethrow above) land here: the writer may
|
||||
// still be running, so stop and observe it before the primary exception surfaces.
|
||||
linked.Cancel();
|
||||
// It must be allowed to DRAIN, not be cancelled outright: the Handshake enqueued at
|
||||
// the top of HandshakeAsync can still be sitting in the control lane (the enqueue →
|
||||
// receive → validate → fault path has no yields, while the writer's wakeup waits in
|
||||
// the thread-pool queue), and that message is the only thing that lets the peer
|
||||
// compute its own fail-closed verdict — mismatch detection is symmetric, so this
|
||||
// node's locally-thrown verdict reaches the peer only as the Handshake it transmits.
|
||||
// Cancelling first killed it almost every time the peer's handshake was already
|
||||
// buffered, leaving the healthy peer to misread a schema mismatch as a generic
|
||||
// disconnect and reconnect-loop at the fast backoff. Completing the lanes lets the
|
||||
// writer send what is already enqueued and exit on its own; the deferred cancel is
|
||||
// the backstop that keeps a wedged transport from holding teardown hostage.
|
||||
control.Writer.TryComplete();
|
||||
data.Writer.TryComplete();
|
||||
linked.CancelAfter(FaultFlushGrace);
|
||||
await DrainSecondaryAsync(writer);
|
||||
throw;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Runs the two-phase fail-closed handshake and returns whether THIS node owes the peer a full
|
||||
/// snapshot (its own computation from its own state — the peer's answer drives the peer's send
|
||||
/// direction, symmetrically). Sending that snapshot is deliberately NOT part of the handshake;
|
||||
/// see <see cref="RunAsync"/>.
|
||||
/// Runs the two-phase fail-closed handshake and returns the snapshot THIS node owes the peer —
|
||||
/// null for none, otherwise the <see cref="SnapshotScope"/> to stream (its own computation from
|
||||
/// its own state; the peer's answer drives the peer's send direction, symmetrically). Sending
|
||||
/// that snapshot is deliberately NOT part of the handshake; see <see cref="RunAsync"/>.
|
||||
/// </summary>
|
||||
private async Task<bool> HandshakeAsync(
|
||||
private async Task<SnapshotScope?> HandshakeAsync(
|
||||
SyncDuplex duplex, ChannelWriter<SyncMessage> controlLane, CancellationToken ct)
|
||||
{
|
||||
var peerState = await _store.GetPeerStateAsync(ct);
|
||||
@@ -158,6 +187,11 @@ internal sealed class SyncSession
|
||||
|
||||
ValidateHandshake(localHandshake, peerHandshake);
|
||||
|
||||
// Read before anything can act on it. A pre-0.3.0 peer leaves the field at its proto3
|
||||
// default, so "false" covers both "explicitly unsupported" and "older build" without a
|
||||
// version check — which is the point of advertising a capability rather than a version.
|
||||
_peerSupportsPartialSnapshot = peerHandshake.SupportsPartialSnapshot;
|
||||
|
||||
// A different node id on the far end means the peer was replaced, not merely restarted — a
|
||||
// rebuilt host, a restored-from-nothing volume. Its oplog numbers from 1 again, so the
|
||||
// watermark we hold (in the OLD peer's seq space) would silently swallow its first N
|
||||
@@ -191,16 +225,16 @@ internal sealed class SyncSession
|
||||
_sentThruSeq = peerAppliedThruSeq;
|
||||
Interlocked.Exchange(ref _peerAckedSeq, peerAppliedThruSeq);
|
||||
|
||||
var snapshotRequired = await ComputeSnapshotRequiredAsync(peerHandshake, peerState, ct);
|
||||
var snapshotScope = await ComputeSnapshotScopeAsync(peerHandshake, peerState, ct);
|
||||
await controlLane.WriteAsync(
|
||||
new SyncMessage { HandshakeAck = new HandshakeAck { NodeId = _db.NodeId, SnapshotRequired = snapshotRequired } }, ct);
|
||||
new SyncMessage { HandshakeAck = new HandshakeAck { NodeId = _db.NodeId, SnapshotRequired = snapshotScope is not null } }, ct);
|
||||
|
||||
var peerAckMsg = await ReceiveOneAsync(duplex, ct);
|
||||
if (peerAckMsg.MsgCase != SyncMessage.MsgOneofCase.HandshakeAck)
|
||||
throw new InvalidOperationException(
|
||||
$"Replication protocol error: expected a HandshakeAck, got {peerAckMsg.MsgCase}.");
|
||||
|
||||
return snapshotRequired;
|
||||
return snapshotScope;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
@@ -216,25 +250,55 @@ internal sealed class SyncSession
|
||||
/// than racing the transport.
|
||||
/// </remarks>
|
||||
private async Task SnapshotThenPumpLoopAsync(
|
||||
bool snapshotRequired, Func<SyncMessage, CancellationToken, Task> dataSend,
|
||||
SnapshotScope? snapshotScope, Func<SyncMessage, CancellationToken, Task> dataSend,
|
||||
ChannelWriter<SyncMessage> dataLane, CancellationToken ct)
|
||||
{
|
||||
if (snapshotRequired)
|
||||
if (snapshotScope is not null)
|
||||
{
|
||||
if (SnapshotSender is null)
|
||||
throw new NotSupportedException("snapshot required but no snapshot sender configured");
|
||||
var snapshotAsOfSeq = await SnapshotSender(dataSend, ct);
|
||||
// Published BEFORE the first frame is enqueued, so the peer's SnapshotAck can never
|
||||
// arrive on the receive loop while this is still null.
|
||||
Volatile.Write(ref _sentSnapshotScope, snapshotScope);
|
||||
var snapshotAsOfSeq = await SnapshotSender(snapshotScope, dataSend, ct);
|
||||
_metrics?.RecordSnapshotSent();
|
||||
_status?.SetLastSyncUtc(_utcNow());
|
||||
// The snapshot already carries every row state up to as-of; skipping the pump past it
|
||||
// avoids re-sending tail deltas the peer would only discard via LWW.
|
||||
if (snapshotAsOfSeq > _sentThruSeq)
|
||||
// A FULL snapshot carries every row state up to as-of, so skipping the pump past it
|
||||
// avoids re-sending tail deltas the peer would only discard via LWW. A PARTIAL one
|
||||
// speaks only for its own tables, so the pump must still deliver every delta below
|
||||
// as-of — the peer has not moved its watermark either (see SnapshotApplier.OnComplete).
|
||||
if (!snapshotScope.IsPartial && snapshotAsOfSeq > _sentThruSeq)
|
||||
_sentThruSeq = snapshotAsOfSeq;
|
||||
|
||||
// A peer that cannot acknowledge the snapshot leaves only the 0.2.x option: clear once
|
||||
// the frames are on the data lane. The window that opens (enqueued, not yet delivered —
|
||||
// a crash there loses the snapshot with the debt already discharged) is why 0.3.0 peers
|
||||
// wait for the ack instead. The reconnect handshake's gap check re-detects it whenever
|
||||
// the oplog is non-empty; an EMPTY oplog remains the undetectable corner.
|
||||
if (!_peerSupportsPartialSnapshot)
|
||||
await ClearSentSnapshotDebtAsync(ct);
|
||||
}
|
||||
|
||||
await PumpLoopAsync(dataLane, ct);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Discharges the snapshot debt this session's outbound snapshot covered, exactly once.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Taking the scope out of the field before clearing makes this idempotent, which matters
|
||||
/// because two paths reach it — the pre-0.3.0 clear-on-enqueue above and the peer's SnapshotAck
|
||||
/// — and a peer that acknowledges anyway (a newer build than its handshake claimed) must not
|
||||
/// clear a second time on top of flags a later baseline may already have re-raised.
|
||||
/// </remarks>
|
||||
private async Task ClearSentSnapshotDebtAsync(CancellationToken ct)
|
||||
{
|
||||
var scope = Interlocked.Exchange(ref _sentSnapshotScope, null);
|
||||
if (scope is null)
|
||||
return;
|
||||
await _store.ClearSnapshotDebtAsync(scope.Tables, ct);
|
||||
}
|
||||
|
||||
// The ONLY caller of duplex.Send. Control (handshake + acks) always preempts data so the peer's
|
||||
// inbound processing is never starved of its acks behind a backlog of DeltaBatches.
|
||||
private static async Task WriterLoopAsync(
|
||||
@@ -292,9 +356,14 @@ internal sealed class SyncSession
|
||||
while (true)
|
||||
{
|
||||
ct.ThrowIfCancellationRequested();
|
||||
// BEFORE the read, because the read itself cannot tell a pruned gap from an idle oplog.
|
||||
ThrowIfPruneGap(await _store.GetMinSeqAsync(ct));
|
||||
var batch = await _store.ReadBatchAboveAsync(_sentThruSeq, _options.MaxBatchSize, ct);
|
||||
if (batch.Count > 0)
|
||||
{
|
||||
// The exact guard on the rows about to go out, for a prune that landed between the
|
||||
// probe and the read. Free (the seq is already in hand) and strictly tighter.
|
||||
ThrowIfPruneGap(batch[0].Seq);
|
||||
var index = 0;
|
||||
while (index < batch.Count)
|
||||
{
|
||||
@@ -326,6 +395,40 @@ internal sealed class SyncSession
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Faults the session when the oplog horizon has climbed above this session's read position —
|
||||
/// i.e. a cap prune deleted deltas between <see cref="_sentThruSeq"/> and
|
||||
/// <paramref name="oldestAvailableSeq"/>. An empty oplog (0) is not a gap.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>Why a live session has to die for this.</b> <c>EnforceCapsAsync</c> sets
|
||||
/// <c>needs_snapshot</c> in the same transaction as the prune, so a RECONNECT heals — the
|
||||
/// handshake serves a full snapshot instead of deltas. Nothing, however, made the live session
|
||||
/// notice: the pump reads <c>seq > _sentThruSeq</c>, which silently resumes above the hole, and
|
||||
/// the peer acks those later seqs as if the pair were converged. On a stable LAN with a slow but
|
||||
/// alive peer that session can run for days, and the pair is durably divergent for all of it —
|
||||
/// the flag that would fix it is only ever read at handshake time. Tearing down is what turns a
|
||||
/// persisted "needs snapshot" into an actual snapshot.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Why the horizon, not the flag.</b> <c>needs_snapshot</c> is also set for reasons that do
|
||||
/// not invalidate THIS session's position (and a full snapshot this session just streamed clears
|
||||
/// it only when the peer's <c>SnapshotAck</c> lands, which is after the pump has started).
|
||||
/// Comparing the horizon against the sent watermark asks the precise question instead, so a
|
||||
/// session is faulted when — and only when — rows it still owes the peer are gone.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
private void ThrowIfPruneGap(long oldestAvailableSeq)
|
||||
{
|
||||
if (oldestAvailableSeq == 0 || oldestAvailableSeq <= _sentThruSeq + 1)
|
||||
return;
|
||||
throw new LocalDbOplogGapException(
|
||||
$"Oplog deltas above the sent watermark {_sentThruSeq} were pruned (oldest surviving seq " +
|
||||
$"{oldestAvailableSeq}): this session can no longer stream incrementally without skipping " +
|
||||
"them. Ending the session so the reconnect handshake serves a snapshot resync.");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Serialized bytes one entry contributes to its batch message: the entry's own encoding plus
|
||||
/// the repeated field's tag and length prefix, as protobuf itself computes them.
|
||||
@@ -384,6 +487,20 @@ internal sealed class SyncSession
|
||||
_snapshotReceiving = false;
|
||||
_metrics?.RecordSnapshotReceived();
|
||||
_status?.SetLastSyncUtc(_utcNow());
|
||||
// Acknowledge only to a peer that asked for it. Sent AFTER the applier's
|
||||
// durable watermark write, so the ack means "I hold these rows", which is
|
||||
// what lets the sender drop its per-table flags. On the unbounded control
|
||||
// lane, like DeltaAck: an ack must never queue behind a data backlog.
|
||||
if (_peerSupportsPartialSnapshot)
|
||||
await controlLane.WriteAsync(
|
||||
new SyncMessage { SnapshotAck = new SnapshotAck { AsOfSeq = msg.SnapshotComplete.AsOfSeq } }, ct);
|
||||
break;
|
||||
case SyncMessage.MsgOneofCase.SnapshotAck:
|
||||
// The peer has durably applied the snapshot THIS node streamed, so the
|
||||
// tables it covered no longer owe one. Nothing to reconcile against the
|
||||
// ack's as-of seq: only one snapshot is ever sent per session, so the scope
|
||||
// held in _sentSnapshotScope is unambiguously the one being acknowledged.
|
||||
await ClearSentSnapshotDebtAsync(ct);
|
||||
break;
|
||||
default:
|
||||
throw new InvalidOperationException(
|
||||
@@ -565,6 +682,9 @@ internal sealed class SyncSession
|
||||
NodeId = _db.NodeId,
|
||||
LibSchemaVersion = LibSchemaVersion,
|
||||
LastAppliedRemoteSeq = peerState.LastAppliedRemoteSeq,
|
||||
// Unconditional: this build always understands a scoped snapshot and always answers a
|
||||
// completed one. It is a statement about the code, not about configuration or state.
|
||||
SupportsPartialSnapshot = true,
|
||||
};
|
||||
foreach (var table in _db.ReplicatedTables.Values.OrderBy(t => t.Name, StringComparer.Ordinal))
|
||||
handshake.Tables.Add(new TableDigest { TableName = table.Name, Digest = table.Digest });
|
||||
@@ -595,11 +715,63 @@ internal sealed class SyncSession
|
||||
"Replication handshake rejected (fail-closed): " + string.Join("; ", parts) + ".");
|
||||
}
|
||||
|
||||
private async Task<bool> ComputeSnapshotRequiredAsync(Handshake peerHandshake, PeerState peerState, CancellationToken ct)
|
||||
/// <summary>
|
||||
/// Decides what snapshot this node owes the peer: null for none, <see cref="SnapshotScope.Full"/>
|
||||
/// for every registered table, or a scoped snapshot naming just the tables that need one.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// The two sources of snapshot debt are genuinely different in reach, which is why they are
|
||||
/// stored separately. An <b>oplog-horizon</b> gap — the peer's watermark below the oldest delta
|
||||
/// still streamable, which is what a cap prune or an impossible peer ack leaves behind — means
|
||||
/// deltas have been lost for the oplog as a whole, so nothing short of every table is sound. A
|
||||
/// <b>baseline</b> means one table's pre-existing rows were never captured, and says nothing at
|
||||
/// all about the others.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// A scope that happens to name every registered table is returned as Full rather than as a
|
||||
/// list: it covers exactly what a full snapshot covers, so it should also get a full snapshot's
|
||||
/// watermark advance on both sides, and it needs no promotion for a pre-0.3.0 peer.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Flags naming tables that are not registered are ignored (they can only stream nothing).
|
||||
/// <c>DeregisterReplicated</c> drops them with the table's ledger rows, so the surviving way to
|
||||
/// hold one is to drop the SQL table without deregistering — which the handshake's fail-closed
|
||||
/// digest comparison rejects for its own reasons.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
private async Task<SnapshotScope?> ComputeSnapshotScopeAsync(Handshake peerHandshake, PeerState peerState, CancellationToken ct)
|
||||
{
|
||||
if (peerState.NeedsSnapshot)
|
||||
return true;
|
||||
if (peerState.NeedsSnapshot || await HasHorizonGapAsync(peerHandshake, peerState, ct))
|
||||
return SnapshotScope.Full;
|
||||
|
||||
var flagged = await _store.GetTablesNeedingSnapshotAsync(ct);
|
||||
if (flagged.Count == 0)
|
||||
return null;
|
||||
|
||||
var scoped = flagged.Where(_db.ReplicatedTables.ContainsKey).ToList();
|
||||
if (scoped.Count == 0)
|
||||
return null;
|
||||
if (scoped.Count >= _db.ReplicatedTables.Count)
|
||||
return SnapshotScope.Full;
|
||||
|
||||
// A peer that would read SnapshotBegin.tables as "all tables" and then advance its watermark
|
||||
// past deltas this snapshot did not carry gets the full snapshot instead. Wider than asked
|
||||
// for, but it is the only shape the older protocol can express correctly — and it is what
|
||||
// 0.2.x did for every baseline.
|
||||
if (!_peerSupportsPartialSnapshot)
|
||||
{
|
||||
_logger.LogInformation(
|
||||
"Peer {PeerNodeId} does not support scoped snapshots; promoting the snapshot of {Tables} to a full one.",
|
||||
PeerNodeId, string.Join(",", scoped));
|
||||
return SnapshotScope.Full;
|
||||
}
|
||||
|
||||
return new SnapshotScope(scoped);
|
||||
}
|
||||
|
||||
private async Task<bool> HasHorizonGapAsync(Handshake peerHandshake, PeerState peerState, CancellationToken ct)
|
||||
{
|
||||
var minRows = await _db.QueryAsync(
|
||||
"SELECT COALESCE(MIN(seq), 0) FROM __localdb_oplog", static r => r.GetInt64(0), null, ct);
|
||||
var minSeq = minRows[0];
|
||||
|
||||
@@ -6,7 +6,7 @@ namespace ZB.MOM.WW.LocalDb.Replication.Internal;
|
||||
/// <summary>
|
||||
/// Shared construction of a <see cref="SyncSession"/> and its dependencies (OplogStore + LwwApplier)
|
||||
/// from a <see cref="SqliteLocalDb"/> and the bound options. Both the service and client adapters use
|
||||
/// this so they build an identical session; snapshot hooks are left null (Task 12 fills them in).
|
||||
/// this so they build an identical session.
|
||||
/// </summary>
|
||||
internal static class SyncSessionFactory
|
||||
{
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
namespace ZB.MOM.WW.LocalDb.Replication;
|
||||
|
||||
/// <summary>
|
||||
/// Thrown by a live sync session when the oplog horizon has moved ABOVE the position that session
|
||||
/// is streaming from: a cap prune deleted deltas it had not yet sent, so every later delta it could
|
||||
/// ship would land on the far side of a hole the peer will never receive.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// This is not a transport failure and not a data error — it is the session discovering that its own
|
||||
/// incremental position has become unsound. Nothing can repair it in-session (the deleted rows are
|
||||
/// gone), so the session tears itself down; the same prune set <c>needs_snapshot</c>, which makes the
|
||||
/// reconnect handshake serve a snapshot instead of deltas. Catching this and continuing would
|
||||
/// reintroduce exactly the silent divergence it exists to stop.
|
||||
/// </remarks>
|
||||
public sealed class LocalDbOplogGapException : Exception
|
||||
{
|
||||
/// <summary>Creates the exception with a message describing the gap.</summary>
|
||||
public LocalDbOplogGapException(string message) : base(message) { }
|
||||
|
||||
/// <summary>Creates the exception with a message and the underlying cause.</summary>
|
||||
public LocalDbOplogGapException(string message, Exception innerException)
|
||||
: base(message, innerException) { }
|
||||
}
|
||||
@@ -21,8 +21,26 @@ public sealed class LocalDbSyncService : LocalDbSync.LocalDbSyncBase
|
||||
private readonly LocalDbMetrics? _metrics;
|
||||
private readonly SyncStatus? _status;
|
||||
|
||||
// 0 = idle, 1 = a stream is active. Interlocked-guarded across concurrent calls (singleton service).
|
||||
private int _active;
|
||||
// The stream currently holding the guard, or null when idle. Guarded by _gate (singleton service).
|
||||
private readonly object _gate = new();
|
||||
private Registration? _active;
|
||||
|
||||
/// <summary>
|
||||
/// How long a displacing call waits for the incumbent it evicted to actually finish before
|
||||
/// giving up and refusing itself. Test seam; the default is generous because eviction only has
|
||||
/// to outrun the operator, not the network.
|
||||
/// </summary>
|
||||
internal TimeSpan TakeoverTimeout { get; set; } = TimeSpan.FromSeconds(10);
|
||||
|
||||
/// <summary>One in-flight <c>Sync</c> handler: its cancellation handle plus a completion signal
|
||||
/// a displacing handler can await, so two sessions never run against the database at once.</summary>
|
||||
private sealed class Registration(CancellationTokenSource cts)
|
||||
{
|
||||
public CancellationTokenSource Cts { get; } = cts;
|
||||
|
||||
public TaskCompletionSource Finished { get; } =
|
||||
new(TaskCreationOptions.RunContinuationsAsynchronously);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Creates the passive sync service. <paramref name="status"/> is taken as the public
|
||||
@@ -41,21 +59,47 @@ public sealed class LocalDbSyncService : LocalDbSync.LocalDbSyncBase
|
||||
_status = status as SyncStatus;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Accepts one inbound sync stream, evicting any stream already holding the guard.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>Newest wins.</b> The guard exists to SERIALIZE sessions — a node syncs with exactly one
|
||||
/// peer, and two sessions against one database would fight over the same watermarks — not to
|
||||
/// fence out an impostor; authentication is the interceptor's job, and an authenticated caller
|
||||
/// on this key IS the peer by definition. Refusing the newcomer therefore protected nothing and
|
||||
/// cost everything in the case that actually happens: the dialling node crashes or is
|
||||
/// partitioned, this handler stays parked forever inside <c>MoveNext</c> on a half-open stream
|
||||
/// that will never produce another byte, and every reconnect from the rebooted peer is rejected
|
||||
/// <c>AlreadyExists</c> — potentially for as long as the pair stays idle, while this side keeps
|
||||
/// reporting itself connected. The newest dialler is the one with a live link, so it takes over.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Eviction is cancel-then-await, not cancel-and-go: the incumbent's cancellation unwinds its
|
||||
/// session and reader pump, and the newcomer waits for that to complete before starting its own,
|
||||
/// so the "one session at a time" property the guard was built for still holds. An incumbent
|
||||
/// that cannot be evicted within <see cref="TakeoverTimeout"/> is the one case where the old
|
||||
/// answer is still right — the newcomer refuses itself rather than double-running.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public override async Task Sync(
|
||||
IAsyncStreamReader<SyncMessage> requestStream,
|
||||
IServerStreamWriter<SyncMessage> responseStream,
|
||||
ServerCallContext context)
|
||||
{
|
||||
if (Interlocked.CompareExchange(ref _active, 1, 0) != 0)
|
||||
throw new RpcException(new Status(StatusCode.AlreadyExists, "a sync stream is already active"));
|
||||
// The linked source exists so the reader pump is deterministically torn down on EVERY exit
|
||||
// path — including a session fault for a local (non-transport) reason, where the call token
|
||||
// never fires and the pump would otherwise keep reading a stream whose handler has exited.
|
||||
// It is also the handle a later call cancels to displace this one.
|
||||
using var linkedCts = CancellationTokenSource.CreateLinkedTokenSource(context.CancellationToken);
|
||||
var mine = new Registration(linkedCts);
|
||||
|
||||
try
|
||||
{
|
||||
// The linked source exists so the reader pump is deterministically torn down on EVERY
|
||||
// exit path — including a session fault for a local (non-transport) reason, where the
|
||||
// call token never fires and the pump would otherwise keep reading a stream whose
|
||||
// handler has exited.
|
||||
using var linkedCts = CancellationTokenSource.CreateLinkedTokenSource(context.CancellationToken);
|
||||
// Inside the try so the finally below always releases the guard — including when the
|
||||
// takeover itself is abandoned because THIS caller gave up mid-handover.
|
||||
await TakeOverAsync(mine, context.CancellationToken);
|
||||
|
||||
var (duplex, readerTask) = GrpcSyncDuplex.Create(requestStream, responseStream, linkedCts.Token);
|
||||
var session = SyncSessionFactory.Create(_db, _options, _loggerFactory, _metrics, _status);
|
||||
_status?.SessionStarted();
|
||||
@@ -81,13 +125,66 @@ public sealed class LocalDbSyncService : LocalDbSync.LocalDbSyncBase
|
||||
{
|
||||
// Client disconnect / server shutdown: the normal end of a long-lived streaming RPC.
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
// Displaced by a newer stream: an orderly handover, not a failure of this node.
|
||||
_logger.LogInformation("Sync stream ended: displaced by a newer stream from the peer.");
|
||||
}
|
||||
catch (LocalDbSchemaMismatchException ex)
|
||||
{
|
||||
throw new RpcException(new Status(StatusCode.FailedPrecondition, ex.Message));
|
||||
}
|
||||
finally
|
||||
{
|
||||
Interlocked.Exchange(ref _active, 0);
|
||||
lock (_gate)
|
||||
{
|
||||
if (ReferenceEquals(_active, mine))
|
||||
_active = null;
|
||||
}
|
||||
mine.Finished.TrySetResult();
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Claims the single-stream guard for <paramref name="mine"/>, cancelling and awaiting whatever
|
||||
/// held it. Throws <c>AlreadyExists</c> only when the incumbent refuses to end.
|
||||
/// </summary>
|
||||
private async Task TakeOverAsync(Registration mine, CancellationToken ct)
|
||||
{
|
||||
Registration? displaced;
|
||||
lock (_gate)
|
||||
{
|
||||
displaced = _active;
|
||||
_active = mine;
|
||||
}
|
||||
if (displaced is null)
|
||||
return;
|
||||
|
||||
_logger.LogWarning(
|
||||
"A new sync stream arrived while one was already active; displacing the incumbent (newest wins).");
|
||||
try
|
||||
{
|
||||
// The incumbent may already be disposing its own source between its finally and here.
|
||||
displaced.Cts.Cancel();
|
||||
}
|
||||
catch (ObjectDisposedException) { }
|
||||
|
||||
try
|
||||
{
|
||||
await displaced.Finished.Task.WaitAsync(TakeoverTimeout, ct);
|
||||
}
|
||||
catch (TimeoutException)
|
||||
{
|
||||
// Could not evict it: two live sessions over one database is worse than a refused dial,
|
||||
// so fall back to the historical answer — and hand the guard back to the incumbent.
|
||||
lock (_gate)
|
||||
{
|
||||
if (ReferenceEquals(_active, mine))
|
||||
_active = displaced;
|
||||
}
|
||||
throw new RpcException(new Status(
|
||||
StatusCode.AlreadyExists,
|
||||
"a sync stream is already active and did not yield within the takeover timeout"));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -54,7 +54,7 @@ public sealed class MaintenanceBackgroundService : BackgroundService
|
||||
{
|
||||
_snapshotFlagWarned = true;
|
||||
_logger.LogWarning(
|
||||
"Oplog backlog/age caps exceeded: pruned to the ceiling and flagged needs_snapshot — the peer must snapshot-resync. Not re-logged until a clean tick.");
|
||||
"Oplog backlog/age caps exceeded: pruned to the ceiling and flagged needs_snapshot — the peer must snapshot-resync. A session streaming from below the new horizon ends itself on its next pump tick so the reconnect serves that snapshot. Not re-logged until a clean tick.");
|
||||
}
|
||||
else if (!snapshotFlagged)
|
||||
{
|
||||
|
||||
@@ -64,6 +64,34 @@ public sealed class ReplicationOptions
|
||||
/// <summary>Upper bound on the exponential reconnect backoff.</summary>
|
||||
public TimeSpan ReconnectBackoffMax { get; set; } = TimeSpan.FromSeconds(60);
|
||||
|
||||
/// <summary>
|
||||
/// How long the initiator's HTTP/2 connection may sit without a keepalive PING before one is
|
||||
/// sent. Default 60s.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// A sync stream is long-lived and, on a converged pair, almost entirely silent — which is
|
||||
/// exactly the shape a half-open TCP connection hides behind. Without pings, a hard crash or a
|
||||
/// partition of one node leaves the other holding a stream that will never produce another byte
|
||||
/// and never fail either: the initiator sits in a session it believes is healthy, and the
|
||||
/// passive node's handler sits inside its single-stream guard, refusing the reconnect that would
|
||||
/// heal the pair. Pings turn that into a detected failure on both ends.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Sent with <c>KeepAlivePingPolicy.Always</c>, so they flow on an idle connection too — an
|
||||
/// active-streams-only policy would not fire in precisely the silent-pair case above. Safe over
|
||||
/// h2c: a PING is an HTTP/2 frame, nothing to do with TLS.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public TimeSpan KeepAlivePingDelay { get; set; } = TimeSpan.FromSeconds(60);
|
||||
|
||||
/// <summary>
|
||||
/// How long the initiator waits for a keepalive PING ack before tearing the connection down.
|
||||
/// Default 20s. Detection therefore lands within roughly
|
||||
/// <see cref="KeepAlivePingDelay"/> + this.
|
||||
/// </summary>
|
||||
public TimeSpan KeepAlivePingTimeout { get; set; } = TimeSpan.FromSeconds(20);
|
||||
|
||||
/// <summary>How often the maintenance service enforces the oplog caps and prunes expired tombstones.</summary>
|
||||
public TimeSpan MaintenanceInterval { get; set; } = TimeSpan.FromSeconds(60);
|
||||
|
||||
|
||||
@@ -40,6 +40,10 @@ public sealed class ReplicationOptionsValidator : IValidateOptions<ReplicationOp
|
||||
failures.Add($"LocalDb:Replication:ReconnectBackoffMax must be greater than zero; got {options.ReconnectBackoffMax}.");
|
||||
else if (options.ReconnectBackoffMax > ReconnectBackoffCeiling)
|
||||
failures.Add($"LocalDb:Replication:ReconnectBackoffMax must be at most {ReconnectBackoffCeiling}; got {options.ReconnectBackoffMax}.");
|
||||
if (options.KeepAlivePingDelay <= TimeSpan.Zero)
|
||||
failures.Add($"LocalDb:Replication:KeepAlivePingDelay must be greater than zero; got {options.KeepAlivePingDelay}.");
|
||||
if (options.KeepAlivePingTimeout <= TimeSpan.Zero)
|
||||
failures.Add($"LocalDb:Replication:KeepAlivePingTimeout must be greater than zero; got {options.KeepAlivePingTimeout}.");
|
||||
if (options.MaintenanceInterval <= TimeSpan.Zero)
|
||||
failures.Add($"LocalDb:Replication:MaintenanceInterval must be greater than zero; got {options.MaintenanceInterval}.");
|
||||
if (options.MaxHlcDriftAhead <= TimeSpan.Zero)
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
using System.Net.Http;
|
||||
using Grpc.Core;
|
||||
using Grpc.Net.Client;
|
||||
using Microsoft.Extensions.Hosting;
|
||||
@@ -76,7 +77,7 @@ public sealed class SyncBackgroundService : BackgroundService
|
||||
{
|
||||
var channel = ChannelFactory is { } factory
|
||||
? factory()
|
||||
: ownedChannel = GrpcChannel.ForAddress(_options.PeerAddress);
|
||||
: ownedChannel = DialPeer();
|
||||
|
||||
var client = new LocalDbSync.LocalDbSyncClient(channel);
|
||||
var headers = new Metadata();
|
||||
@@ -152,6 +153,39 @@ public sealed class SyncBackgroundService : BackgroundService
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Dials the peer over a channel that sends HTTP/2 keepalive PINGs, so a half-open link is
|
||||
/// detected instead of being waited on forever.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// The bare <c>GrpcChannel.ForAddress(address)</c> this replaces had NO keepalive of any kind,
|
||||
/// and neither end of a sync stream ever times out on silence by design (a converged pair is
|
||||
/// legitimately quiet). A hard crash or partition of the peer therefore left this node inside a
|
||||
/// session that could never make progress and could never fail — and its own half of that stream
|
||||
/// pinned on the peer, where the single-stream guard turns it into a refusal of the very
|
||||
/// reconnect that heals the pair.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <c>DisposeHttpClient</c> hands the handler's lifetime to the channel, which this loop already
|
||||
/// disposes on every exit path.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
private GrpcChannel DialPeer() =>
|
||||
GrpcChannel.ForAddress(_options.PeerAddress, new GrpcChannelOptions
|
||||
{
|
||||
HttpHandler = new SocketsHttpHandler
|
||||
{
|
||||
KeepAlivePingDelay = _options.KeepAlivePingDelay,
|
||||
KeepAlivePingTimeout = _options.KeepAlivePingTimeout,
|
||||
// Always, not WithActiveRequests: the case being closed is a SILENT stream on a dead
|
||||
// link, and an active-requests policy stops pinging once the peer stops responding.
|
||||
KeepAlivePingPolicy = HttpKeepAlivePingPolicy.Always,
|
||||
EnableMultipleHttp2Connections = true,
|
||||
},
|
||||
DisposeHttpClient = true,
|
||||
});
|
||||
|
||||
private static bool IsSchemaMismatch(Exception? ex) =>
|
||||
ex is LocalDbSchemaMismatchException
|
||||
|| (ex is RpcException rpc && rpc.StatusCode == StatusCode.FailedPrecondition);
|
||||
|
||||
@@ -38,8 +38,20 @@ public interface ILocalDb
|
||||
/// schema digest, and installs the three AFTER triggers that capture every change into the
|
||||
/// oplog + row-version tables within the same transaction as the consumer's write. Idempotent
|
||||
/// (re-registering regenerates the triggers). Throws <see cref="LocalDbRegistrationException"/>
|
||||
/// if the name is unsafe, the table is missing, or it has no primary key.
|
||||
/// if the name is unsafe, the table is missing, it has no primary key, or it is the parent of a
|
||||
/// foreign key whose <c>ON DELETE</c> action mutates the child (<c>CASCADE</c> / <c>SET NULL</c> /
|
||||
/// <c>SET DEFAULT</c>) — see the remarks for why that last one cannot be replicated.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <b>Replicated tables may not be the parent of a cascading foreign key.</b> The apply path
|
||||
/// writes remote rows with <c>INSERT OR REPLACE</c>; REPLACE deletes the conflicting row first,
|
||||
/// and with <c>PRAGMA foreign_keys=ON</c> that delete fires the child's <c>ON DELETE</c> action
|
||||
/// while change capture is suppressed. The child rows would be mutated on the applying node only,
|
||||
/// never captured and never shipped, diverging the pair with nothing able to detect it. The check
|
||||
/// runs at registration and inspects the whole database's foreign keys; SQLite cannot add a
|
||||
/// foreign key to an existing table, so a later-created child table is caught on the next
|
||||
/// registration (i.e. the next startup), not at the moment of the DDL.
|
||||
/// </remarks>
|
||||
ReplicatedTable RegisterReplicated(string tableName);
|
||||
|
||||
/// <summary>
|
||||
@@ -50,7 +62,8 @@ public interface ILocalDb
|
||||
/// <param name="baselineExistingRows">
|
||||
/// When true, every row already in the table that has no <c>__localdb_row_version</c> entry is
|
||||
/// given one at the <b>baseline floor</b> — HLC <c>0</c>, stamped with this node's id — and, if
|
||||
/// anything was seeded, a snapshot resync is flagged so the peer actually receives them.
|
||||
/// anything was seeded, a snapshot resync is flagged <b>for this table</b> so the peer actually
|
||||
/// receives them.
|
||||
/// </param>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
@@ -80,16 +93,29 @@ public interface ILocalDb
|
||||
/// and no snapshot is flagged. Passing true on every boot is therefore free after the first.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Cost, and why it is worth planning for.</b> The snapshot flag is a single per-DATABASE
|
||||
/// flag, and a snapshot streams EVERY registered table in full — so baselining one table
|
||||
/// re-sends all of them, and a pair that enables replication late has both nodes doing that to
|
||||
/// each other at once. On a database holding a high-volume table (an event log at its retention
|
||||
/// cap, say) that is the largest transfer this library ever performs, in both directions
|
||||
/// simultaneously. It is bounded work, not runaway: outbound messages respect
|
||||
/// <c>MaxBatchBytes</c>, and the receive side is bounded by the transport's inbox. But the
|
||||
/// operational advice is to <b>purge or trim high-volume tables before enabling replication
|
||||
/// late</b>, and to expect one slow first sync rather than a slow steady state. A per-table
|
||||
/// snapshot flag would narrow this; it needs an on-disk schema change and is not in 0.2.x.
|
||||
/// <b>Cost, and what bounds it (0.3.0).</b> The snapshot flag is <b>per-table</b>
|
||||
/// (<c>__localdb_snapshot_state</c>, on-disk schema v3), so baselining one table asks the peer
|
||||
/// for that table alone — the tables around it are not re-streamed. Through 0.2.x the flag was a
|
||||
/// single per-database bit and baselining anything re-sent everything, in both directions at
|
||||
/// once on a pair enabling replication late.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Two cases still stream every registered table, by design. A <b>pre-0.3.0 peer</b> cannot be
|
||||
/// told a snapshot is partial (it would advance its watermark past deltas it never received),
|
||||
/// so a scoped snapshot is promoted to a full one for it. And the <b>oplog-horizon</b> paths —
|
||||
/// a cap prune, an impossible peer ack — invalidate delta continuity for every table at once
|
||||
/// and still set the database-wide flag. So the operational advice survives, narrowed: baseline
|
||||
/// a high-volume table (an event log at its retention cap, say) deliberately, and <b>purge or
|
||||
/// trim it first</b>, because that ONE table's rows still travel in full. The work is bounded
|
||||
/// either way — outbound messages respect <c>MaxBatchBytes</c>, the receive side is bounded by
|
||||
/// the transport's inbox.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Crash safety.</b> Between 0.3.0 peers a table's flag is cleared only when the receiver
|
||||
/// acknowledges the completed snapshot, so an ungraceful exit part-way through re-streams the
|
||||
/// same scope on reconnect rather than losing it. Against a pre-0.3.0 peer (which sends no such
|
||||
/// acknowledgement) the 0.2.x behaviour is kept: the flag clears once the snapshot is on the
|
||||
/// wire, and the handshake's horizon gap check is the net.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
ReplicatedTable RegisterReplicated(string tableName, bool baselineExistingRows);
|
||||
|
||||
@@ -27,9 +27,20 @@ internal static class LocalDbSchema
|
||||
/// <c>hlc</c>-predicate delete. It cost a b-tree insert on the CAPTURE path — i.e. inside every
|
||||
/// consumer write transaction — to serve nothing.
|
||||
/// </description></item>
|
||||
/// <item><description>
|
||||
/// <b>3</b> — adds <c>__localdb_snapshot_state</c>, the PER-TABLE snapshot-resync flag. Until
|
||||
/// v3 the only flag was <c>__localdb_peer_state.needs_snapshot</c>, one bit for the whole
|
||||
/// database, so baselining a single table re-streamed EVERY registered table in full — in both
|
||||
/// directions at once on a pair enabling replication late. The database-wide bit survives with
|
||||
/// its original meaning (the oplog-horizon and cap-prune paths genuinely invalidate every
|
||||
/// table); the new table narrows the baseline case to the tables that actually need it. The
|
||||
/// transition itself is a version stamp only — <see cref="Ddl"/> creates the table with
|
||||
/// <c>IF NOT EXISTS</c> on every open, so an older file gains it whatever its stamp says, and
|
||||
/// an empty flag table means exactly what it meant before: nothing per-table is owed.
|
||||
/// </description></item>
|
||||
/// </list>
|
||||
/// </remarks>
|
||||
internal const long CurrentSchemaVersion = 2;
|
||||
internal const long CurrentSchemaVersion = 3;
|
||||
|
||||
private const string Ddl = """
|
||||
CREATE TABLE IF NOT EXISTS __localdb_meta (
|
||||
@@ -73,6 +84,14 @@ internal static class LocalDbSchema
|
||||
id INTEGER PRIMARY KEY CHECK (id = 1),
|
||||
applying INTEGER NOT NULL DEFAULT 0
|
||||
);
|
||||
-- Per-table snapshot-resync flags (schema v3). A row exists only while that table owes the
|
||||
-- peer a snapshot; the whole-database __localdb_peer_state.needs_snapshot bit still means
|
||||
-- "every registered table" and is what the cap/horizon paths set.
|
||||
-- WITHOUT ROWID: table_name IS the only access path (read as a set, cleared by name).
|
||||
CREATE TABLE IF NOT EXISTS __localdb_snapshot_state (
|
||||
table_name TEXT NOT NULL PRIMARY KEY,
|
||||
needs_snapshot INTEGER NOT NULL DEFAULT 0
|
||||
) WITHOUT ROWID;
|
||||
CREATE TABLE IF NOT EXISTS __localdb_dead_letter (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
received_utc TEXT NOT NULL,
|
||||
@@ -143,6 +162,12 @@ internal static class LocalDbSchema
|
||||
if (current < 2)
|
||||
steps.Add("DROP INDEX IF EXISTS __localdb_oplog_hlc;");
|
||||
|
||||
// 2 -> 3: __localdb_snapshot_state. No step: the shared Ddl above already ran
|
||||
// CREATE TABLE IF NOT EXISTS for it in this same transaction, and an empty flag table is
|
||||
// the correct v3 state for a file that has never baselined a table. Repeating the DDL here
|
||||
// would be a second source of truth for the same layout. The stamp still moves, because
|
||||
// schema_version records what the file's layout IS, not which statements produced it.
|
||||
|
||||
if (steps.Count > 0)
|
||||
{
|
||||
using var ddl = connection.CreateCommand();
|
||||
|
||||
@@ -199,6 +199,8 @@ internal sealed class SqliteLocalDb : ILocalDb, IDisposable
|
||||
throw new LocalDbRegistrationException(
|
||||
$"Cannot register table '{tableName}': replication requires an explicit primary key.");
|
||||
|
||||
EnsureNoInboundActionOnDelete(tableName);
|
||||
|
||||
var allColumns = columns.Select(c => (c.Name, c.Type)).ToList();
|
||||
var digest = ReplicatedTable.ComputeDigest(tableName, pkColumns, allColumns);
|
||||
var table = new ReplicatedTable(tableName, pkColumns, allColumns, digest);
|
||||
@@ -276,7 +278,12 @@ internal sealed class SqliteLocalDb : ILocalDb, IDisposable
|
||||
prune.Transaction = tx;
|
||||
prune.CommandText =
|
||||
"DELETE FROM __localdb_oplog WHERE table_name = $t;" +
|
||||
"DELETE FROM __localdb_row_version WHERE table_name = $t;";
|
||||
"DELETE FROM __localdb_row_version WHERE table_name = $t;" +
|
||||
// A pending per-table snapshot flag for a table nothing streams any more would
|
||||
// never clear — the streamer skips unregistered tables, so the flag would sit
|
||||
// in the file forever, and re-registering later would find a stale "already
|
||||
// owed" it did not ask for. Dropped with the ledger it refers to.
|
||||
"DELETE FROM __localdb_snapshot_state WHERE table_name = $t;";
|
||||
prune.Parameters.AddWithValue("$t", tableName);
|
||||
prunedRows = prune.ExecuteNonQuery();
|
||||
}
|
||||
@@ -288,6 +295,66 @@ internal sealed class SqliteLocalDb : ILocalDb, IDisposable
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Refuses to register a table that is the PARENT of a foreign key whose <c>ON DELETE</c> action
|
||||
/// mutates the child (<c>CASCADE</c>, <c>SET NULL</c>, <c>SET DEFAULT</c>).
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>Why this is a replication hazard, not a schema opinion.</b> The apply path writes a remote
|
||||
/// row with <c>INSERT OR REPLACE</c>, and REPLACE resolves a primary-key conflict by DELETING the
|
||||
/// existing row before inserting the new one. Connections run <c>PRAGMA foreign_keys=ON</c>, and
|
||||
/// that implicit delete fires foreign-key actions (verified behaviour, not inference). It runs
|
||||
/// with the capture guard raised, so the resulting child-row deletes/nulls are invisible to the
|
||||
/// oplog: they happen on this node and are never shipped. The peer, applying the same parent row,
|
||||
/// keeps its children — and the pair has silently diverged with nothing to detect it, since LWW
|
||||
/// only ever compares rows it was told about.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Checked at registration only.</b> SQLite cannot add a foreign key to an existing table, so
|
||||
/// a new offender can only arrive as a child table created (or a parent table rebuilt) after this
|
||||
/// call. Consumers register their tables at startup, after schema creation/migration, which is
|
||||
/// where the check belongs; a schema change that adds a cascading child to an already-registered
|
||||
/// parent is caught on the next boot, not at the moment of the DDL.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <c>RESTRICT</c> / <c>NO ACTION</c> are deliberately allowed: they abort the offending statement
|
||||
/// with a constraint error rather than mutating rows behind capture's back, which the apply path
|
||||
/// already handles as per-entry poison.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
private void EnsureNoInboundActionOnDelete(string tableName)
|
||||
{
|
||||
var offenders = new List<string>();
|
||||
using (var cmd = _master.CreateCommand())
|
||||
{
|
||||
// pragma_foreign_key_list is the table-valued form of PRAGMA foreign_key_list, so the
|
||||
// whole database is one query: fk."table" is the PARENT this child points at.
|
||||
cmd.CommandText =
|
||||
"SELECT m.name, fk.\"from\", fk.\"on_delete\" " +
|
||||
"FROM sqlite_master m JOIN pragma_foreign_key_list(m.name) fk " +
|
||||
"WHERE m.type = 'table' AND fk.\"table\" = $t COLLATE NOCASE " +
|
||||
"AND UPPER(fk.\"on_delete\") IN ('CASCADE', 'SET NULL', 'SET DEFAULT') " +
|
||||
"ORDER BY m.name, fk.\"from\"";
|
||||
cmd.Parameters.AddWithValue("$t", tableName);
|
||||
using var reader = cmd.ExecuteReader();
|
||||
while (reader.Read())
|
||||
offenders.Add(
|
||||
$"{reader.GetString(0)}.{reader.GetString(1)} ON DELETE {reader.GetString(2).ToUpperInvariant()}");
|
||||
}
|
||||
|
||||
if (offenders.Count == 0)
|
||||
return;
|
||||
|
||||
throw new LocalDbRegistrationException(
|
||||
$"Cannot register table '{tableName}': it is the parent of a foreign key with a row-mutating " +
|
||||
$"ON DELETE action ({string.Join(", ", offenders)}). Replication applies remote rows with " +
|
||||
"INSERT OR REPLACE, whose implicit delete fires that action with change capture suppressed — " +
|
||||
"the child rows would be changed on this node only and never replicated, diverging the pair " +
|
||||
"undetectably. Use ON DELETE NO ACTION/RESTRICT and delete children explicitly, or leave this " +
|
||||
"table unreplicated.");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Seeds <c>__localdb_row_version</c> at the baseline floor for every row of
|
||||
/// <paramref name="table"/> that has no ledger entry, and flags a snapshot resync if it seeded
|
||||
@@ -308,12 +375,12 @@ internal sealed class SqliteLocalDb : ILocalDb, IDisposable
|
||||
/// the same snapshot anyway, via a cap-prune.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>The flag is per-DATABASE, not per-table</b>, and a snapshot streams every registered table
|
||||
/// in full — so baselining one table re-sends all of them, and both nodes of a late-enabled pair
|
||||
/// do it to each other at once. Bounded (batch budget outbound, bounded inbox inbound) but
|
||||
/// potentially the largest transfer this library performs; see
|
||||
/// <see cref="ILocalDb.RegisterReplicated(string, bool)"/> for the operational advice. Narrowing
|
||||
/// it to a per-table flag needs an on-disk schema change and is deferred.
|
||||
/// <b>The flag is PER-TABLE</b> (<c>__localdb_snapshot_state</c>, schema v3): baselining one
|
||||
/// table asks for a snapshot of that table alone, not of every registered table. The
|
||||
/// database-wide <c>__localdb_peer_state.needs_snapshot</c> bit is untouched here and keeps its
|
||||
/// original meaning — the oplog-horizon and cap-prune paths invalidate every table's delta
|
||||
/// continuity at once, and only those paths set it. A peer too old to understand a scoped
|
||||
/// snapshot is served a full one instead; the negotiation lives in the replication package.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
private void BaselineExistingRows(SqliteTransaction tx, ReplicatedTable table)
|
||||
@@ -339,7 +406,10 @@ internal sealed class SqliteLocalDb : ILocalDb, IDisposable
|
||||
|
||||
using var flag = _master.CreateCommand();
|
||||
flag.Transaction = tx;
|
||||
flag.CommandText = "UPDATE __localdb_peer_state SET needs_snapshot = 1 WHERE id = 1";
|
||||
flag.CommandText =
|
||||
"INSERT INTO __localdb_snapshot_state (table_name, needs_snapshot) VALUES ($t, 1) " +
|
||||
"ON CONFLICT(table_name) DO UPDATE SET needs_snapshot = 1";
|
||||
flag.Parameters.AddWithValue("$t", table.Name);
|
||||
flag.ExecuteNonQuery();
|
||||
}
|
||||
|
||||
|
||||
@@ -70,6 +70,20 @@ public sealed class BatchBudgetTests : IDisposable
|
||||
return msg!;
|
||||
}
|
||||
|
||||
// Reads the next scripted-peer message while also observing the session task: a session that
|
||||
// faults would otherwise leave this wait to run out the whole test timeout, reporting an
|
||||
// opaque cancellation instead of the session's real exception.
|
||||
private static async Task<SyncMessage> NextAsync(Task run, ChannelReader<SyncMessage> reader, CancellationToken ct)
|
||||
{
|
||||
var next = NextAsync(reader, ct);
|
||||
if (ReferenceEquals(await Task.WhenAny(run, next), run))
|
||||
{
|
||||
await run;
|
||||
throw new InvalidOperationException("The sync session ended before the expected message arrived.");
|
||||
}
|
||||
return await next;
|
||||
}
|
||||
|
||||
private static async Task SwallowAsync(Task task)
|
||||
{
|
||||
try { await task; }
|
||||
@@ -140,7 +154,7 @@ public sealed class BatchBudgetTests : IDisposable
|
||||
var seen = new List<long>();
|
||||
while (seen.Count < 20)
|
||||
{
|
||||
var msg = await NextAsync(fromSession, cts.Token);
|
||||
var msg = await NextAsync(run, fromSession, cts.Token);
|
||||
if (msg.MsgCase != SyncMessage.MsgOneofCase.DeltaBatch)
|
||||
continue;
|
||||
Assert.True(BatchBytes(msg.DeltaBatch) <= budget,
|
||||
@@ -180,7 +194,7 @@ public sealed class BatchBudgetTests : IDisposable
|
||||
var batches = new List<DeltaBatch>();
|
||||
while (batches.Sum(b => b.Entries.Count) < 2)
|
||||
{
|
||||
var msg = await NextAsync(fromSession, cts.Token);
|
||||
var msg = await NextAsync(run, fromSession, cts.Token);
|
||||
if (msg.MsgCase == SyncMessage.MsgOneofCase.DeltaBatch)
|
||||
batches.Add(msg.DeltaBatch);
|
||||
}
|
||||
@@ -220,7 +234,7 @@ public sealed class BatchBudgetTests : IDisposable
|
||||
var seen = 0;
|
||||
while (seen < 12)
|
||||
{
|
||||
var msg = await NextAsync(fromSession, cts.Token);
|
||||
var msg = await NextAsync(run, fromSession, cts.Token);
|
||||
if (msg.MsgCase != SyncMessage.MsgOneofCase.DeltaBatch)
|
||||
continue;
|
||||
Assert.True(msg.DeltaBatch.Entries.Count <= 5);
|
||||
@@ -250,7 +264,7 @@ public sealed class BatchBudgetTests : IDisposable
|
||||
var streamer = new SnapshotStreamer(db, store, options, NullLogger.Instance);
|
||||
|
||||
var sent = new List<SyncMessage>();
|
||||
await streamer.SendAsync((m, _) => { sent.Add(m); return Task.CompletedTask; }, default);
|
||||
await streamer.SendAsync(SnapshotScope.Full, (m, _) => { sent.Add(m); return Task.CompletedTask; }, default);
|
||||
|
||||
var batches = sent.Where(m => m.MsgCase == SyncMessage.MsgOneofCase.SnapshotBatch)
|
||||
.Select(m => m.SnapshotBatch).ToList();
|
||||
@@ -279,7 +293,7 @@ public sealed class BatchBudgetTests : IDisposable
|
||||
var streamer = new SnapshotStreamer(db, store, options, NullLogger.Instance);
|
||||
|
||||
var sent = new List<SyncMessage>();
|
||||
await streamer.SendAsync((m, _) => { sent.Add(m); return Task.CompletedTask; }, default);
|
||||
await streamer.SendAsync(SnapshotScope.Full, (m, _) => { sent.Add(m); return Task.CompletedTask; }, default);
|
||||
|
||||
var batches = sent.Where(m => m.MsgCase == SyncMessage.MsgOneofCase.SnapshotBatch)
|
||||
.Select(m => m.SnapshotBatch).ToList();
|
||||
@@ -312,7 +326,7 @@ public sealed class BatchBudgetTests : IDisposable
|
||||
}
|
||||
|
||||
SyncMessage ack;
|
||||
do { ack = await NextAsync(fromSession, cts.Token); }
|
||||
do { ack = await NextAsync(run, fromSession, cts.Token); }
|
||||
while (ack.MsgCase != SyncMessage.MsgOneofCase.DeltaAck);
|
||||
|
||||
// The discriminator: uncoalesced, the FIRST ack could only cover seq 1.
|
||||
@@ -348,7 +362,7 @@ public sealed class BatchBudgetTests : IDisposable
|
||||
new SyncMessage { SnapshotBegin = new SnapshotBegin { AsOfSeq = 77 } }, cts.Token);
|
||||
|
||||
SyncMessage ack;
|
||||
do { ack = await NextAsync(fromSession, cts.Token); }
|
||||
do { ack = await NextAsync(run, fromSession, cts.Token); }
|
||||
while (ack.MsgCase != SyncMessage.MsgOneofCase.DeltaAck);
|
||||
Assert.Equal(1, ack.DeltaAck.AppliedThruSeq);
|
||||
|
||||
|
||||
@@ -70,8 +70,12 @@ public sealed class GrpcAdapterTests : IAsyncLifetime
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task SecondConcurrentStream_Rejected()
|
||||
public async Task SecondConcurrentStream_DisplacesTheIncumbent()
|
||||
{
|
||||
// Newest wins (0.3.2). The guard serializes sessions; it was never a fence — the auth
|
||||
// interceptor is, and an authenticated same-key dialler IS the peer. Refusing the newcomer
|
||||
// bought nothing and cost the pair its only route back from a half-open link, where the
|
||||
// incumbent handler can hold the guard indefinitely on a dead stream.
|
||||
var serverHost = await BuildServerAsync(NewDbPath(), apiKey: null, sink: null);
|
||||
var serverDb = serverHost.Services.GetRequiredService<ILocalDb>();
|
||||
var handler = serverHost.GetTestServer().CreateHandler();
|
||||
@@ -85,11 +89,15 @@ public sealed class GrpcAdapterTests : IAsyncLifetime
|
||||
|
||||
// Convergence proves the first stream's server handler is active and holding the single-stream guard.
|
||||
await WaitForAsync(async () => (await ReadOrders(serverDb)).Count == 1, RunTimeout);
|
||||
var attemptsBefore = bg.ConnectionAttempts;
|
||||
|
||||
var client = new LocalDbSync.LocalDbSyncClient(ChannelOver(handler));
|
||||
using var call2 = client.Sync(cancellationToken: cts.Token);
|
||||
var ex = await Assert.ThrowsAsync<RpcException>(() => call2.ResponseStream.MoveNext(cts.Token));
|
||||
Assert.Equal(StatusCode.AlreadyExists, ex.StatusCode);
|
||||
Assert.True(await call2.ResponseStream.MoveNext(cts.Token));
|
||||
Assert.Equal(SyncMessage.MsgOneofCase.Handshake, call2.ResponseStream.Current.MsgCase);
|
||||
|
||||
// The incumbent really ended: the displaced client notices and dials again.
|
||||
await WaitForAsync(() => Task.FromResult(bg.ConnectionAttempts > attemptsBefore), RunTimeout);
|
||||
|
||||
await bg.StopAsync(CancellationToken.None);
|
||||
}
|
||||
@@ -262,9 +270,13 @@ public sealed class GrpcAdapterTests : IAsyncLifetime
|
||||
TombstoneRetention = TimeSpan.Zero,
|
||||
ReconnectBackoffMax = TimeSpan.Zero,
|
||||
MaxHlcDriftAhead = TimeSpan.Zero,
|
||||
KeepAlivePingDelay = TimeSpan.Zero,
|
||||
KeepAlivePingTimeout = TimeSpan.Zero,
|
||||
});
|
||||
|
||||
Assert.True(result.Failed);
|
||||
Assert.Contains(result.Failures, f => f.Contains("KeepAlivePingDelay", StringComparison.Ordinal));
|
||||
Assert.Contains(result.Failures, f => f.Contains("KeepAlivePingTimeout", StringComparison.Ordinal));
|
||||
Assert.Contains(result.Failures, f => f.Contains("FlushInterval", StringComparison.Ordinal));
|
||||
Assert.Contains(result.Failures, f => f.Contains("MaxBatchSize", StringComparison.Ordinal));
|
||||
Assert.Contains(result.Failures, f => f.Contains("MaxOplogRows", StringComparison.Ordinal));
|
||||
|
||||
@@ -51,7 +51,8 @@ public sealed class LocalDbSchemaTests : IDisposable
|
||||
foreach (var expected in new[]
|
||||
{
|
||||
"__localdb_meta", "__localdb_oplog", "__localdb_peer_state",
|
||||
"__localdb_row_version", "__localdb_applying", "__localdb_dead_letter"
|
||||
"__localdb_row_version", "__localdb_applying", "__localdb_dead_letter",
|
||||
"__localdb_snapshot_state",
|
||||
})
|
||||
Assert.Contains(expected, tables);
|
||||
|
||||
@@ -85,6 +86,44 @@ public sealed class LocalDbSchemaTests : IDisposable
|
||||
ScalarLong(conn, "SELECT schema_version FROM __localdb_meta WHERE id=1"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void EnsureCreated_UpgradesV2File_AddsPerTableSnapshotState()
|
||||
{
|
||||
using var conn = Open();
|
||||
|
||||
// A file exactly as schema v2 left it: everything except __localdb_snapshot_state, stamped 2.
|
||||
LocalDbSchema.EnsureCreated(conn);
|
||||
Exec(conn, "DROP TABLE __localdb_snapshot_state;" +
|
||||
"UPDATE __localdb_meta SET schema_version = 2 WHERE id = 1;");
|
||||
|
||||
LocalDbSchema.EnsureCreated(conn);
|
||||
|
||||
Assert.Equal(1, ScalarLong(conn,
|
||||
"SELECT count(*) FROM sqlite_master WHERE type='table' AND name='__localdb_snapshot_state'"));
|
||||
// Empty is the right v3 state for a file that never baselined anything: no table owes a
|
||||
// snapshot, and the database-wide flag it was carrying keeps whatever value it had.
|
||||
Assert.Equal(0, ScalarLong(conn, "SELECT COUNT(*) FROM __localdb_snapshot_state"));
|
||||
Assert.Equal(LocalDbSchema.CurrentSchemaVersion,
|
||||
ScalarLong(conn, "SELECT schema_version FROM __localdb_meta WHERE id=1"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void EnsureCreated_Upgrade_PreservesTheDatabaseWideSnapshotFlag()
|
||||
{
|
||||
// A v2 file mid-resync (cap prune flagged a full snapshot, the peer was down) must still owe
|
||||
// that snapshot after the upgrade. v3 narrows where NEW debt is recorded; it does not
|
||||
// reinterpret debt already on disk.
|
||||
using var conn = Open();
|
||||
LocalDbSchema.EnsureCreated(conn);
|
||||
Exec(conn, "DROP TABLE __localdb_snapshot_state;" +
|
||||
"UPDATE __localdb_meta SET schema_version = 2 WHERE id = 1;" +
|
||||
"UPDATE __localdb_peer_state SET needs_snapshot = 1 WHERE id = 1;");
|
||||
|
||||
LocalDbSchema.EnsureCreated(conn);
|
||||
|
||||
Assert.Equal(1, ScalarLong(conn, "SELECT needs_snapshot FROM __localdb_peer_state WHERE id=1"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void EnsureCreated_Upgrade_PreservesNodeIdentityAndOplogRows()
|
||||
{
|
||||
|
||||
@@ -427,9 +427,11 @@ public sealed class LwwApplierTests : IDisposable
|
||||
var result = await applier.ApplyBatchAsync(entries, default);
|
||||
|
||||
Assert.Equal(202, result.Applied);
|
||||
// Four table-independent statements (applying guard, version lookup, row-version write,
|
||||
// peer state), plus an upsert per table touched and a delete for the one table tombstoned.
|
||||
Assert.Equal(7, applier.LastBatchStatementCount);
|
||||
// Six table-independent statements (applying guard, version lookup, row-version write,
|
||||
// peer state, and the per-entry savepoint's enter/release), plus an upsert per table
|
||||
// touched and a delete for the one table tombstoned. The savepoint's rollback statement is
|
||||
// built on first use like the rest, and nothing in this batch is poison.
|
||||
Assert.Equal(9, applier.LastBatchStatementCount);
|
||||
|
||||
// ...and the batch is still correct, statement reuse notwithstanding.
|
||||
Assert.Null(await ReadOrder(db, 1));
|
||||
@@ -462,6 +464,120 @@ public sealed class LwwApplierTests : IDisposable
|
||||
Assert.Equal(("LAST", (long?)5), await ReadOrder(db, 5));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Arms a BEFORE INSERT trigger on the row-version ledger that aborts the write for one key —
|
||||
/// a genuine statement-level SQLite fault (SQLITE_CONSTRAINT), landing exactly between an
|
||||
/// entry's table write and its ledger write, with no production test seam.
|
||||
/// </summary>
|
||||
private static Task PoisonRowVersionWriteFor(SqliteLocalDb db, string pkJson) =>
|
||||
db.ExecuteAsync(
|
||||
"CREATE TRIGGER poison_rv BEFORE INSERT ON __localdb_row_version " +
|
||||
$"WHEN NEW.pk_json = '{pkJson}' BEGIN SELECT RAISE(ABORT, 'ledger write failed'); END");
|
||||
|
||||
[Fact]
|
||||
public async Task Apply_TombstoneWhoseLedgerWriteFails_LeavesRowAndLedgerUntouched()
|
||||
{
|
||||
// The hazard: DELETE and the ledger write are two statements, and SQLite aborts only the
|
||||
// failing one. Without a per-entry savepoint the row is gone while __localdb_row_version
|
||||
// still calls it live — SnapshotStreamer can then only warn and skip it, so the peer never
|
||||
// learns the row died and the pair diverges. Both or neither.
|
||||
using var db = await NewOrdersDb();
|
||||
await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1, 'LOCAL', 5)");
|
||||
var localHlc = await RowVersionHlc(db, Pk1);
|
||||
await PoisonRowVersionWriteFor(db, Pk1);
|
||||
|
||||
var result = await new LwwApplier(db).ApplyBatchAsync(
|
||||
[Entry(90, Pk1, null, localHlc + 1000, "remote-node", tombstone: true)], default);
|
||||
|
||||
Assert.Equal(0, result.Applied);
|
||||
Assert.Equal(1, result.DeadLettered);
|
||||
Assert.Equal(("LOCAL", (long?)5), await ReadOrder(db, 1));
|
||||
var rv = await db.QueryAsync(
|
||||
"SELECT hlc, is_tombstone FROM __localdb_row_version WHERE table_name = 'orders' AND pk_json = @pk",
|
||||
x => (Hlc: x.GetInt64(0), Tomb: x.GetInt64(1)), new { pk = Pk1 });
|
||||
Assert.Equal(localHlc, rv[0].Hlc);
|
||||
Assert.Equal(0, rv[0].Tomb);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Apply_UpsertWhoseLedgerWriteFails_LeavesRowAndLedgerUntouched()
|
||||
{
|
||||
// Same split, other direction: the new row would sit in the table carrying the OLD version,
|
||||
// so it would lose the very next LWW comparison it should win.
|
||||
using var db = await NewOrdersDb();
|
||||
await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1, 'LOCAL', 5)");
|
||||
var localHlc = await RowVersionHlc(db, Pk1);
|
||||
await PoisonRowVersionWriteFor(db, Pk1);
|
||||
|
||||
var result = await new LwwApplier(db).ApplyBatchAsync(
|
||||
[Entry(91, Pk1, "{\"id\":1,\"sku\":\"REMOTE\",\"qty\":99}", localHlc + 1000, "remote-node")], default);
|
||||
|
||||
Assert.Equal(0, result.Applied);
|
||||
Assert.Equal(1, result.DeadLettered);
|
||||
Assert.Equal(("LOCAL", (long?)5), await ReadOrder(db, 1));
|
||||
Assert.Equal(localHlc, await RowVersionHlc(db, Pk1));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Apply_LedgerWriteFailureMidBatch_DoesNotUndoTheRestOfTheBatch()
|
||||
{
|
||||
// The savepoint must rewind ONE entry, not the batch: entries either side of the poison
|
||||
// still apply, and the batch still commits.
|
||||
using var db = await NewOrdersDb();
|
||||
await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1, 'LOCAL', 5)");
|
||||
var localHlc = await RowVersionHlc(db, Pk1);
|
||||
await PoisonRowVersionWriteFor(db, Pk1);
|
||||
|
||||
var result = await new LwwApplier(db).ApplyBatchAsync(
|
||||
[
|
||||
Entry(92, "{\"id\":2}", "{\"id\":2,\"sku\":\"BEFORE\",\"qty\":2}", localHlc + 1000, "remote-node"),
|
||||
Entry(93, Pk1, null, localHlc + 1001, "remote-node", tombstone: true),
|
||||
Entry(94, "{\"id\":3}", "{\"id\":3,\"sku\":\"AFTER\",\"qty\":3}", localHlc + 1002, "remote-node"),
|
||||
], default);
|
||||
|
||||
Assert.Equal(2, result.Applied);
|
||||
Assert.Equal(1, result.DeadLettered);
|
||||
Assert.Equal(94, result.AppliedThruSeq);
|
||||
Assert.Equal(("BEFORE", (long?)2), await ReadOrder(db, 2));
|
||||
Assert.Equal(("AFTER", (long?)3), await ReadOrder(db, 3));
|
||||
Assert.Equal(("LOCAL", (long?)5), await ReadOrder(db, 1));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Apply_DeadLetterFailure_ReportsBothCausesAndCommitsNothing()
|
||||
{
|
||||
// Two faults stacked: the entry is poison (its ledger write aborts) AND recovering from it
|
||||
// fails (the dead-letter INSERT aborts too). Reporting only the second would discard the
|
||||
// one that says WHY the entry was poison, leaving an operator with "dead-letter failed"
|
||||
// and no cause.
|
||||
using var db = await NewOrdersDb();
|
||||
await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1, 'LOCAL', 5)");
|
||||
var localHlc = await RowVersionHlc(db, Pk1);
|
||||
await PoisonRowVersionWriteFor(db, Pk1);
|
||||
await db.ExecuteAsync(
|
||||
"CREATE TRIGGER poison_dl BEFORE INSERT ON __localdb_dead_letter " +
|
||||
"BEGIN SELECT RAISE(ABORT, 'dead-letter write failed'); END");
|
||||
|
||||
var ex = await Assert.ThrowsAsync<AggregateException>(() => new LwwApplier(db).ApplyBatchAsync(
|
||||
[
|
||||
Entry(95, "{\"id\":2}", "{\"id\":2,\"sku\":\"BEFORE\",\"qty\":2}", localHlc + 1000, "remote-node"),
|
||||
Entry(96, Pk1, null, localHlc + 1001, "remote-node", tombstone: true),
|
||||
], default));
|
||||
|
||||
// The entry's own fault first (InnerException), the recovery fault carried alongside it.
|
||||
Assert.Contains("ledger write failed", ex.InnerException!.Message);
|
||||
Assert.Contains("dead-letter write failed", ex.InnerExceptions[1].Message);
|
||||
Assert.Contains("orders", ex.Message);
|
||||
Assert.Contains("96", ex.Message);
|
||||
|
||||
// Fail-closed: the batch never commits, so even the entry that applied cleanly is gone.
|
||||
Assert.Null(await ReadOrder(db, 2));
|
||||
Assert.Equal(("LOCAL", (long?)5), await ReadOrder(db, 1));
|
||||
var watermark = await db.QueryAsync(
|
||||
"SELECT last_applied_remote_seq FROM __localdb_peer_state WHERE id = 1", x => x.GetInt64(0));
|
||||
Assert.Equal(0, watermark[0]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Apply_ReplaceExistingRow_DoesNotFireCaptureTriggers()
|
||||
{
|
||||
|
||||
@@ -70,6 +70,109 @@ public sealed class RegistrationTests : IDisposable
|
||||
Assert.Contains("BLOB", ex.Message);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Register_CascadeDeleteParent_Throws()
|
||||
{
|
||||
// The apply path writes remote rows with INSERT OR REPLACE, whose implicit delete fires
|
||||
// ON DELETE CASCADE with capture suppressed — the child rows would die on this node only.
|
||||
using var db = Create();
|
||||
await CreateOrders(db);
|
||||
await db.ExecuteAsync(
|
||||
"CREATE TABLE order_lines (id INTEGER PRIMARY KEY, " +
|
||||
"order_id INTEGER REFERENCES orders(id) ON DELETE CASCADE, sku TEXT)");
|
||||
|
||||
var ex = Assert.Throws<LocalDbRegistrationException>(() => db.RegisterReplicated("orders"));
|
||||
Assert.Contains("orders", ex.Message);
|
||||
Assert.Contains("order_lines.order_id", ex.Message);
|
||||
Assert.Contains("ON DELETE CASCADE", ex.Message);
|
||||
|
||||
// Refused, not half-registered.
|
||||
Assert.Empty(await Triggers(db));
|
||||
Assert.False(db.ReplicatedTables.ContainsKey("orders"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Register_CascadeDeleteParent_IsWhatWouldActuallyDiverge()
|
||||
{
|
||||
// Proves the guard is guarding something real: with the parent NOT registered, applying a
|
||||
// replace to it silently wipes child rows that produce no oplog entry.
|
||||
using var db = Create();
|
||||
await CreateOrders(db);
|
||||
await db.ExecuteAsync(
|
||||
"CREATE TABLE order_lines (id INTEGER PRIMARY KEY, " +
|
||||
"order_id INTEGER REFERENCES orders(id) ON DELETE CASCADE, sku TEXT)");
|
||||
await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1, 'A', 1)");
|
||||
await db.ExecuteAsync("INSERT INTO order_lines (id, order_id, sku) VALUES (10, 1, 'L')");
|
||||
|
||||
await db.ExecuteAsync("INSERT OR REPLACE INTO orders (id, sku, qty) VALUES (1, 'B', 2)");
|
||||
|
||||
var lines = await db.QueryAsync("SELECT COUNT(*) FROM order_lines", r => r.GetInt64(0));
|
||||
Assert.Equal(0, lines[0]);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData("SET NULL")]
|
||||
[InlineData("SET DEFAULT")]
|
||||
public async Task Register_RowMutatingOnDeleteParent_Throws(string action)
|
||||
{
|
||||
using var db = Create();
|
||||
await CreateOrders(db);
|
||||
await db.ExecuteAsync(
|
||||
"CREATE TABLE order_lines (id INTEGER PRIMARY KEY, " +
|
||||
$"order_id INTEGER DEFAULT 0 REFERENCES orders(id) ON DELETE {action}, sku TEXT)");
|
||||
|
||||
var ex = Assert.Throws<LocalDbRegistrationException>(() => db.RegisterReplicated("orders"));
|
||||
Assert.Contains($"ON DELETE {action}", ex.Message);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Register_NonMutatingOnDeleteParent_IsAllowed()
|
||||
{
|
||||
// RESTRICT/NO ACTION abort the statement instead of changing rows behind capture's back,
|
||||
// which the apply path already handles as per-entry poison.
|
||||
using var db = Create();
|
||||
await CreateOrders(db);
|
||||
await db.ExecuteAsync(
|
||||
"CREATE TABLE order_lines (id INTEGER PRIMARY KEY, " +
|
||||
"order_id INTEGER REFERENCES orders(id) ON DELETE RESTRICT, sku TEXT)");
|
||||
|
||||
db.RegisterReplicated("orders");
|
||||
|
||||
Assert.Equal(3, (await Triggers(db)).Count);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Register_SelfReferencingCascade_Throws()
|
||||
{
|
||||
// A table can be its own cascading parent: replacing a row would delete that row's
|
||||
// children in the SAME table, uncaptured. The guard reads every foreign key in the
|
||||
// database, so it does not need a special case — but the README promises this, so it is
|
||||
// pinned.
|
||||
using var db = Create();
|
||||
await db.ExecuteAsync(
|
||||
"CREATE TABLE nodes (id INTEGER PRIMARY KEY, " +
|
||||
"parent_id INTEGER REFERENCES nodes(id) ON DELETE CASCADE, name TEXT)");
|
||||
|
||||
var ex = Assert.Throws<LocalDbRegistrationException>(() => db.RegisterReplicated("nodes"));
|
||||
Assert.Contains("nodes.parent_id", ex.Message);
|
||||
Assert.Contains("ON DELETE CASCADE", ex.Message);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Register_CascadeDeleteChild_IsAllowed()
|
||||
{
|
||||
// Only the PARENT side is hazardous: replacing a child row deletes no other table's rows.
|
||||
using var db = Create();
|
||||
await CreateOrders(db);
|
||||
await db.ExecuteAsync(
|
||||
"CREATE TABLE order_lines (id INTEGER PRIMARY KEY, " +
|
||||
"order_id INTEGER REFERENCES orders(id) ON DELETE CASCADE, sku TEXT)");
|
||||
|
||||
var table = db.RegisterReplicated("order_lines");
|
||||
|
||||
Assert.Equal("order_lines", table.Name);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Register_IsIdempotent_RegeneratesTriggers()
|
||||
{
|
||||
@@ -291,6 +394,45 @@ public sealed class RegistrationTests : IDisposable
|
||||
// Without the flag the seeding would be inert: no gap exists at the handshake, so the peer
|
||||
// would be told it needs no snapshot and the baselined rows would never travel.
|
||||
Assert.True(await NeedsSnapshot(db));
|
||||
// ...and it is flagged for THIS TABLE ONLY. Through 0.2.x this set the one database-wide
|
||||
// bit, so baselining `orders` re-streamed every other registered table with it.
|
||||
Assert.False(await NeedsFullSnapshot(db));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Register_WithBaseline_FlagsOnlyTheBaselinedTable()
|
||||
{
|
||||
// The per-table flag's reason to exist: opting one table into replication late must not
|
||||
// schedule a full re-stream of its unrelated neighbours.
|
||||
using var db = Create();
|
||||
await CreateOrders(db);
|
||||
await db.ExecuteAsync("CREATE TABLE widgets (id INTEGER PRIMARY KEY, name TEXT)");
|
||||
await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1,'A',1)");
|
||||
await db.ExecuteAsync("INSERT INTO widgets (id, name) VALUES (1,'W')");
|
||||
|
||||
db.RegisterReplicated("widgets");
|
||||
db.RegisterReplicated("orders", baselineExistingRows: true);
|
||||
|
||||
Assert.True(await NeedsSnapshot(db, "orders"));
|
||||
Assert.False(await NeedsSnapshot(db, "widgets"));
|
||||
Assert.False(await NeedsFullSnapshot(db));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Deregister_DropsThePendingSnapshotFlag()
|
||||
{
|
||||
// A flag naming a table nothing streams any more can never clear itself — the streamer skips
|
||||
// unregistered tables — so it would sit in the file and re-appear as unasked-for debt if the
|
||||
// table were registered again.
|
||||
using var db = Create();
|
||||
await CreateOrders(db);
|
||||
await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1,'A',1)");
|
||||
db.RegisterReplicated("orders", baselineExistingRows: true);
|
||||
Assert.True(await NeedsSnapshot(db));
|
||||
|
||||
db.DeregisterReplicated("orders");
|
||||
|
||||
Assert.False(await NeedsSnapshot(db));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
@@ -301,7 +443,8 @@ public sealed class RegistrationTests : IDisposable
|
||||
await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1,'A',1)");
|
||||
|
||||
db.RegisterReplicated("orders", baselineExistingRows: true);
|
||||
await db.ExecuteAsync("UPDATE __localdb_peer_state SET needs_snapshot = 0 WHERE id = 1");
|
||||
// Stand in for the snapshot having been delivered and acknowledged.
|
||||
await db.ExecuteAsync("DELETE FROM __localdb_snapshot_state");
|
||||
|
||||
db.RegisterReplicated("orders", baselineExistingRows: true);
|
||||
|
||||
@@ -480,7 +623,17 @@ public sealed class RegistrationTests : IDisposable
|
||||
private static async Task<long> CountAsync(SqliteLocalDb db, string sql) =>
|
||||
(await db.QueryAsync(sql, r => r.GetInt64(0)))[0];
|
||||
|
||||
private static async Task<bool> NeedsSnapshot(SqliteLocalDb db) =>
|
||||
/// <summary>Whether <paramref name="table"/> individually owes the peer a snapshot (schema v3).</summary>
|
||||
private static async Task<bool> NeedsSnapshot(SqliteLocalDb db, string table = "orders") =>
|
||||
(await db.QueryAsync(
|
||||
"SELECT COUNT(*) FROM __localdb_snapshot_state WHERE table_name = @t AND needs_snapshot <> 0",
|
||||
r => r.GetInt64(0), new { t = table }))[0] != 0;
|
||||
|
||||
/// <summary>
|
||||
/// The DATABASE-WIDE flag — "every registered table". Baselining must never set it: that is the
|
||||
/// whole point of the per-table flags, and setting it would restore the 0.2.x behaviour silently.
|
||||
/// </summary>
|
||||
private static async Task<bool> NeedsFullSnapshot(SqliteLocalDb db) =>
|
||||
(await db.QueryAsync(
|
||||
"SELECT needs_snapshot FROM __localdb_peer_state WHERE id = 1", r => r.GetInt64(0)))[0] != 0;
|
||||
}
|
||||
|
||||
@@ -52,14 +52,14 @@ public sealed class SnapshotResyncTests : IDisposable
|
||||
// A fresh session stack over an EXISTING database. Reconnecting after a session ends needs a new
|
||||
// session (RunAsync is single-use), so any scenario with two connection episodes over one node's
|
||||
// data — a peer being replaced between them — goes through here.
|
||||
private static Side NewSessionOver(SqliteLocalDb db, ReplicationOptions? options = null)
|
||||
private static Side NewSessionOver(SqliteLocalDb db, ReplicationOptions? options = null, ILogger? streamerLogger = null)
|
||||
{
|
||||
options ??= new ReplicationOptions { FlushInterval = TimeSpan.FromMilliseconds(20) };
|
||||
var store = new OplogStore(db, options);
|
||||
var applier = new LwwApplier(db);
|
||||
var session = new SyncSession(db, store, applier, options, NullLogger.Instance);
|
||||
|
||||
var streamer = new SnapshotStreamer(db, store, options, NullLogger.Instance);
|
||||
var streamer = new SnapshotStreamer(db, store, options, streamerLogger ?? NullLogger.Instance);
|
||||
var snapshotApplier = new SnapshotApplier(db, applier, store, NullLogger.Instance);
|
||||
session.SnapshotSender = streamer.SendAsync;
|
||||
session.SnapshotApplier = snapshotApplier;
|
||||
@@ -89,9 +89,19 @@ public sealed class SnapshotResyncTests : IDisposable
|
||||
return (duplex, toSession.Writer, fromSession.Reader);
|
||||
}
|
||||
|
||||
private static Handshake MatchingHandshake(SqliteLocalDb sessionDb, string peerNodeId, long lastAppliedRemoteSeq = 0)
|
||||
// supportsPartialSnapshot defaults to FALSE, which is exactly what a 0.1.x/0.2.x peer puts on
|
||||
// the wire (the field does not exist in its proto, so proto3 decodes the default). Scripted
|
||||
// peers are therefore old peers unless a test says otherwise.
|
||||
private static Handshake MatchingHandshake(
|
||||
SqliteLocalDb sessionDb, string peerNodeId, long lastAppliedRemoteSeq = 0, bool supportsPartialSnapshot = false)
|
||||
{
|
||||
var h = new Handshake { NodeId = peerNodeId, LibSchemaVersion = 1, LastAppliedRemoteSeq = lastAppliedRemoteSeq };
|
||||
var h = new Handshake
|
||||
{
|
||||
NodeId = peerNodeId,
|
||||
LibSchemaVersion = 1,
|
||||
LastAppliedRemoteSeq = lastAppliedRemoteSeq,
|
||||
SupportsPartialSnapshot = supportsPartialSnapshot,
|
||||
};
|
||||
foreach (var t in sessionDb.ReplicatedTables.Values.OrderBy(t => t.Name, StringComparer.Ordinal))
|
||||
h.Tables.Add(new TableDigest { TableName = t.Name, Digest = t.Digest });
|
||||
return h;
|
||||
@@ -114,14 +124,34 @@ public sealed class SnapshotResyncTests : IDisposable
|
||||
|
||||
// Completes the two-phase handshake against a real session: the peer never requests a snapshot.
|
||||
private static async Task DriveHandshakeAsync(
|
||||
SqliteLocalDb sessionDb, ChannelWriter<SyncMessage> toSession, ChannelReader<SyncMessage> fromSession, CancellationToken ct)
|
||||
SqliteLocalDb sessionDb, ChannelWriter<SyncMessage> toSession, ChannelReader<SyncMessage> fromSession,
|
||||
CancellationToken ct, bool supportsPartialSnapshot = false, long lastAppliedRemoteSeq = 0)
|
||||
{
|
||||
await NextOfCaseAsync(fromSession, SyncMessage.MsgOneofCase.Handshake, ct);
|
||||
await toSession.WriteAsync(new SyncMessage { Handshake = MatchingHandshake(sessionDb, "peer") }, ct);
|
||||
await toSession.WriteAsync(
|
||||
new SyncMessage
|
||||
{
|
||||
Handshake = MatchingHandshake(
|
||||
sessionDb, "peer", lastAppliedRemoteSeq, supportsPartialSnapshot),
|
||||
}, ct);
|
||||
await NextOfCaseAsync(fromSession, SyncMessage.MsgOneofCase.HandshakeAck, ct);
|
||||
await toSession.WriteAsync(new SyncMessage { HandshakeAck = new HandshakeAck { NodeId = "peer", SnapshotRequired = false } }, ct);
|
||||
}
|
||||
|
||||
// Reads outbound messages until (and including) the first of the given kind.
|
||||
private static async Task<List<SyncMessage>> DrainThroughAsync(
|
||||
ChannelReader<SyncMessage> reader, SyncMessage.MsgOneofCase last, CancellationToken ct)
|
||||
{
|
||||
var all = new List<SyncMessage>();
|
||||
while (true)
|
||||
{
|
||||
var msg = await NextAsync(reader, ct);
|
||||
all.Add(msg);
|
||||
if (msg.MsgCase == last)
|
||||
return all;
|
||||
}
|
||||
}
|
||||
|
||||
private static async Task SwallowAsync(Task task)
|
||||
{
|
||||
try { await task; }
|
||||
@@ -159,6 +189,12 @@ public sealed class SnapshotResyncTests : IDisposable
|
||||
return r[0];
|
||||
}
|
||||
|
||||
/// <summary>The tables individually flagged as owing the peer a snapshot (schema v3).</summary>
|
||||
private static async Task<List<string>> PendingSnapshotTables(SqliteLocalDb db) =>
|
||||
(await db.QueryAsync(
|
||||
"SELECT table_name FROM __localdb_snapshot_state WHERE needs_snapshot <> 0 ORDER BY table_name",
|
||||
x => x.GetString(0))).ToList();
|
||||
|
||||
private static async Task<long> OplogCount(SqliteLocalDb db)
|
||||
{
|
||||
var r = await db.QueryAsync("SELECT COUNT(*) FROM __localdb_oplog", x => x.GetInt64(0));
|
||||
@@ -255,6 +291,74 @@ public sealed class SnapshotResyncTests : IDisposable
|
||||
await SwallowAsync(runB);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task WipedPeer_SnapshotLiftsItsWatermark_SoTheNextReconnectSendsNoSecondSnapshot()
|
||||
{
|
||||
// The follow-on to the wiped-peer gap above, found on the ScadaBridge docker rig 2026-08-20
|
||||
// (its deferred-work register row 106): the snapshot DID heal the wiped node, but it was
|
||||
// stamped as_of_seq = MAX(seq) of an oplog that ack-pruning had emptied — i.e. 0 — so the
|
||||
// receiver's (correctly MAX-guarded) watermark stayed at 0. On the next handshake the
|
||||
// healthy side's empty-oplog horizon rule (streamable from last_acked_seq + 1) saw
|
||||
// 0 < last_acked_seq as a gap again and re-streamed the whole database — on EVERY reconnect,
|
||||
// until the healthy side happened to write a new row (127k rows per restart on the rig).
|
||||
// A full snapshot taken now covers every delta ever pruned below last_acked_seq, so that is
|
||||
// the floor the as-of must carry.
|
||||
var a = await NewSideAsync();
|
||||
for (var i = 1; i <= 3; i++)
|
||||
await a.Db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (@id, 'A', @id)", new { id = i });
|
||||
await a.Store.RecordPeerAckAsync(await a.Store.GetMaxSeqAsync());
|
||||
Assert.Equal(0, await OplogCount(a.Db));
|
||||
|
||||
var b = await NewSideAsync();
|
||||
|
||||
// Episode 1: the wiped peer is healed by a full snapshot.
|
||||
var (d1A, d1B) = DuplexPair();
|
||||
using (var cts1 = new CancellationTokenSource(RunTimeout))
|
||||
{
|
||||
var run1A = a.Session.RunAsync(d1A, cts1.Token);
|
||||
var run1B = b.Session.RunAsync(d1B, cts1.Token);
|
||||
await WaitForAsync(async () => (await ReadOrders(b.Db)).Count == 3, RunTimeout);
|
||||
cts1.Cancel();
|
||||
await SwallowAsync(run1A);
|
||||
await SwallowAsync(run1B);
|
||||
}
|
||||
|
||||
// The snapshot must have lifted B's watermark to everything A ever produced (seq 3), not 0.
|
||||
Assert.Equal(3, await LastAppliedRemoteSeq(b.Db));
|
||||
|
||||
// Episode 2: a routine reconnect. A writes one more row so the delta path gives us a
|
||||
// barrier; the snapshot streamer's logger proves whether a snapshot was sent at all.
|
||||
var streamerLog = new CapturingLogger();
|
||||
var a2 = NewSessionOver(a.Db, streamerLogger: streamerLog);
|
||||
var b2 = NewSessionOver(b.Db);
|
||||
var (d2A, d2B) = DuplexPair();
|
||||
using var cts2 = new CancellationTokenSource(RunTimeout);
|
||||
var run2A = a2.Session.RunAsync(d2A, cts2.Token);
|
||||
var run2B = b2.Session.RunAsync(d2B, cts2.Token);
|
||||
await WaitForAsync(() => Task.FromResult(b2.Session.PeerNodeId is not null), RunTimeout);
|
||||
await a.Db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (4, 'DELTA', 4)");
|
||||
await WaitForAsync(async () => (await ReadOrders(b.Db)).Count == 4, RunTimeout);
|
||||
|
||||
Assert.DoesNotContain(streamerLog.Messages, m => m.StartsWith("Snapshot sent", StringComparison.Ordinal));
|
||||
Assert.Equal(4, await LastAppliedRemoteSeq(b.Db));
|
||||
|
||||
cts2.Cancel();
|
||||
await SwallowAsync(run2A);
|
||||
await SwallowAsync(run2B);
|
||||
}
|
||||
|
||||
private sealed class CapturingLogger : ILogger
|
||||
{
|
||||
private readonly List<string> _messages = [];
|
||||
public IReadOnlyList<string> Messages { get { lock (_messages) return _messages.ToList(); } }
|
||||
public IDisposable? BeginScope<TState>(TState state) where TState : notnull => null;
|
||||
public bool IsEnabled(LogLevel logLevel) => true;
|
||||
public void Log<TState>(LogLevel logLevel, EventId eventId, TState state, Exception? exception, Func<TState, Exception?, string> formatter)
|
||||
{
|
||||
lock (_messages) _messages.Add(formatter(state, exception));
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task RebuiltPeer_OwnWritesReachTheHealthyNode_DespiteTheOldPeersWatermark()
|
||||
{
|
||||
@@ -341,6 +445,93 @@ public sealed class SnapshotResyncTests : IDisposable
|
||||
await SwallowAsync(runB);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task CapPrune_IntoALiveSessionsUnsentRange_FaultsIt_AndTheReconnectSnapshots()
|
||||
{
|
||||
// P4-1. The pump reads `seq > sentThruSeq`, which resumes SILENTLY above a hole a cap prune
|
||||
// punched — so it keeps shipping later deltas, the peer acks them, and the pair is durably
|
||||
// divergent. `needs_snapshot` (set in the prune's own transaction) heals only a RECONNECT,
|
||||
// and nothing forced one: on a stable LAN with a slow-but-alive peer the session, and the
|
||||
// divergence, could last for days. Faulting is what turns the persisted flag into a snapshot.
|
||||
var a = await NewSideAsync();
|
||||
for (var i = 1; i <= 6; i++)
|
||||
await a.Db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (@id, 'A', @id)", new { id = i });
|
||||
|
||||
var (duplex, toSession, fromSession) = ScriptedPeer();
|
||||
using var cts = new CancellationTokenSource(RunTimeout);
|
||||
var run = a.Session.RunAsync(duplex, cts.Token);
|
||||
|
||||
// The peer holds everything so far: no snapshot is owed, and the pump idles at seq 6.
|
||||
await DriveHandshakeAsync(a.Db, toSession, fromSession, cts.Token, lastAppliedRemoteSeq: 6);
|
||||
|
||||
// Write three more rows and prune all but the newest, in ONE transaction — which is what a
|
||||
// cap prune is, and what makes this deterministic: the pump's reader can only ever see the
|
||||
// before state (nothing above seq 6) or the after state (seqs 7 and 8 already gone). It can
|
||||
// never catch rows 7/8 in between and legitimately advance past them.
|
||||
await using (var tx = await a.Db.BeginTransactionAsync(cts.Token))
|
||||
{
|
||||
await tx.ExecuteAsync(
|
||||
"INSERT INTO orders (id, sku, qty) VALUES (7,'PRUNED',7),(8,'PRUNED',8),(9,'KEPT',9)", null, cts.Token);
|
||||
await tx.ExecuteAsync("UPDATE __localdb_peer_state SET needs_snapshot = 1 WHERE id = 1", null, cts.Token);
|
||||
await tx.ExecuteAsync(
|
||||
"DELETE FROM __localdb_oplog WHERE seq < (SELECT MAX(seq) FROM __localdb_oplog)", null, cts.Token);
|
||||
await tx.CommitAsync(cts.Token);
|
||||
}
|
||||
|
||||
var fault = await Assert.ThrowsAsync<LocalDbOplogGapException>(() => run);
|
||||
Assert.Contains("6", fault.Message, StringComparison.Ordinal);
|
||||
|
||||
// And it faulted BEFORE shipping anything from above the hole: a delta for seq 9 would have
|
||||
// moved the peer's watermark past rows 7 and 8, which is the divergence itself.
|
||||
while (fromSession.TryRead(out var stray))
|
||||
Assert.NotEqual(SyncMessage.MsgOneofCase.DeltaBatch, stray.MsgCase);
|
||||
|
||||
// The reconnect is what heals: the prune's needs_snapshot makes the handshake owe a full
|
||||
// snapshot, which carries the pruned rows' current state to a real peer.
|
||||
var a2 = NewSessionOver(a.Db);
|
||||
var b = await NewSideAsync();
|
||||
var (da, db) = DuplexPair();
|
||||
var run2A = a2.Session.RunAsync(da, cts.Token);
|
||||
var run2B = b.Session.RunAsync(db, cts.Token);
|
||||
|
||||
await WaitForAsync(async () => (await ReadOrders(b.Db)).Count == 9, RunTimeout);
|
||||
Assert.Equal(await ReadOrders(a.Db), await ReadOrders(b.Db));
|
||||
|
||||
cts.Cancel();
|
||||
await SwallowAsync(run2A);
|
||||
await SwallowAsync(run2B);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task AckPruningToEmpty_DoesNotFaultALiveSession()
|
||||
{
|
||||
// The other half of the horizon check: ordinary ack-pruning deletes seq <= last_acked_seq,
|
||||
// which by construction sits at or below the sent watermark, so a converged pair that has
|
||||
// pruned its whole oplog must keep streaming. A gap test written against `needs_snapshot`,
|
||||
// or one that treated any risen horizon as a fault, would kill healthy sessions here.
|
||||
var a = await NewSideAsync();
|
||||
var b = await NewSideAsync();
|
||||
for (var i = 1; i <= 4; i++)
|
||||
await a.Db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (@id, 'A', @id)", new { id = i });
|
||||
|
||||
var (da, db) = DuplexPair();
|
||||
using var cts = new CancellationTokenSource(RunTimeout);
|
||||
var runA = a.Session.RunAsync(da, cts.Token);
|
||||
var runB = b.Session.RunAsync(db, cts.Token);
|
||||
|
||||
await WaitForAsync(async () => (await ReadOrders(b.Db)).Count == 4, RunTimeout);
|
||||
await WaitForAsync(async () => await OplogCount(a.Db) == 0, RunTimeout);
|
||||
|
||||
// Still live, still incremental: a post-prune write reaches B as an ordinary delta.
|
||||
await a.Db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (5, 'POST', 5)");
|
||||
await WaitForAsync(async () => (await ReadOrders(b.Db)).Any(r => r.Id == 5), RunTimeout);
|
||||
Assert.False(runA.IsFaulted, runA.Exception?.ToString());
|
||||
|
||||
cts.Cancel();
|
||||
await SwallowAsync(runA);
|
||||
await SwallowAsync(runB);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Snapshot_MergesNotTruncates()
|
||||
{
|
||||
@@ -585,6 +776,261 @@ public sealed class SnapshotResyncTests : IDisposable
|
||||
await SwallowAsync(run);
|
||||
}
|
||||
|
||||
// ---- Per-table snapshot scoping (0.3.0) ----------------------------------------------
|
||||
|
||||
private const string WidgetsSql = "CREATE TABLE widgets (id INTEGER PRIMARY KEY, name TEXT)";
|
||||
|
||||
/// <summary>
|
||||
/// A node with two registered tables where exactly one — <c>orders</c> — was baselined.
|
||||
/// <c>widgets</c> is registered BEFORE its rows are written, so ordinary capture ledgers them;
|
||||
/// <c>orders</c> is registered after, so only baselining can. That asymmetry is what makes the
|
||||
/// scope observable: a full snapshot carries both tables' ledgers, a scoped one carries orders
|
||||
/// alone — and both tables hold rows either way, so an empty widgets stream cannot be mistaken
|
||||
/// for "there was nothing to send".
|
||||
/// </summary>
|
||||
private async Task<Side> NewBaselinedSideAsync(ReplicationOptions? options = null)
|
||||
{
|
||||
var path = Path.Combine(Path.GetTempPath(), Guid.NewGuid() + ".db");
|
||||
_paths.Add(path);
|
||||
var db = new SqliteLocalDb(new LocalDbOptions { Path = path });
|
||||
_disposables.Add(db);
|
||||
await db.ExecuteAsync(OrdersSql);
|
||||
await db.ExecuteAsync(WidgetsSql);
|
||||
|
||||
// Predates registration: invisible to CDC, so only baselining puts it in the ledger.
|
||||
await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1,'PRE',1),(2,'PRE',2)");
|
||||
|
||||
db.RegisterReplicated("widgets");
|
||||
await db.ExecuteAsync("INSERT INTO widgets (id, name) VALUES (1,'W1'),(2,'W2')");
|
||||
|
||||
db.RegisterReplicated("orders", baselineExistingRows: true);
|
||||
|
||||
return NewSessionOver(db, options);
|
||||
}
|
||||
|
||||
/// <summary>The peer of <see cref="NewBaselinedSideAsync"/>: the same two tables, empty, owing nothing.</summary>
|
||||
private async Task<Side> NewTwoTableSideAsync(ReplicationOptions? options = null)
|
||||
{
|
||||
var path = Path.Combine(Path.GetTempPath(), Guid.NewGuid() + ".db");
|
||||
_paths.Add(path);
|
||||
var db = new SqliteLocalDb(new LocalDbOptions { Path = path });
|
||||
_disposables.Add(db);
|
||||
await db.ExecuteAsync(OrdersSql);
|
||||
await db.ExecuteAsync(WidgetsSql);
|
||||
db.RegisterReplicated("orders");
|
||||
db.RegisterReplicated("widgets");
|
||||
return NewSessionOver(db, options);
|
||||
}
|
||||
|
||||
private static List<string> SnapshotTablesIn(IEnumerable<SyncMessage> messages) =>
|
||||
messages.Where(m => m.MsgCase == SyncMessage.MsgOneofCase.SnapshotBatch)
|
||||
.Select(m => m.SnapshotBatch.TableName)
|
||||
.Distinct(StringComparer.Ordinal)
|
||||
.Order(StringComparer.Ordinal)
|
||||
.ToList();
|
||||
|
||||
[Fact]
|
||||
public async Task Baselining_OneTable_StreamsOnlyThatTable()
|
||||
{
|
||||
// The whole point of the per-table flag. Through 0.2.x the flag was one bit for the whole
|
||||
// database, so baselining `orders` re-streamed `widgets` (and everything else) in full — in
|
||||
// both directions at once on a pair enabling replication late, which on a database holding
|
||||
// an event log at its retention cap is the largest transfer this library performs.
|
||||
var a = await NewBaselinedSideAsync();
|
||||
Assert.Equal(["orders"], await PendingSnapshotTables(a.Db));
|
||||
Assert.Equal(0, await NeedsSnapshot(a.Db)); // NOT the database-wide flag
|
||||
|
||||
var (duplex, toSession, fromSession) = ScriptedPeer();
|
||||
using var cts = new CancellationTokenSource(RunTimeout);
|
||||
var run = a.Session.RunAsync(duplex, cts.Token);
|
||||
|
||||
await DriveHandshakeAsync(a.Db, toSession, fromSession, cts.Token, supportsPartialSnapshot: true);
|
||||
var frames = await DrainThroughAsync(fromSession, SyncMessage.MsgOneofCase.SnapshotComplete, cts.Token);
|
||||
|
||||
var begin = frames.Single(m => m.MsgCase == SyncMessage.MsgOneofCase.SnapshotBegin).SnapshotBegin;
|
||||
Assert.Equal(["orders"], begin.Tables);
|
||||
Assert.Equal(["orders"], SnapshotTablesIn(frames));
|
||||
// The baselined rows really are in there — the scope narrowed the snapshot, it did not empty it.
|
||||
var pks = frames.Where(m => m.MsgCase == SyncMessage.MsgOneofCase.SnapshotBatch)
|
||||
.SelectMany(m => m.SnapshotBatch.Rows).Select(r => r.PkJson).Order(StringComparer.Ordinal);
|
||||
Assert.Equal(["{\"id\":1}", "{\"id\":2}"], pks);
|
||||
|
||||
cts.Cancel();
|
||||
await SwallowAsync(run);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Baselining_AgainstAPeerWithoutPartialSupport_StreamsEveryTable()
|
||||
{
|
||||
// Mixed-version interop, the hard constraint. A pre-0.3.0 peer ignores SnapshotBegin.tables
|
||||
// and would advance last_applied_remote_seq to as_of_seq on complete — which for a SCOPED
|
||||
// snapshot silently discards every delta below as_of for the tables it left out. There is no
|
||||
// way to express "partial" to that peer, so it is served the 0.2.x answer: everything.
|
||||
var a = await NewBaselinedSideAsync();
|
||||
var (duplex, toSession, fromSession) = ScriptedPeer();
|
||||
using var cts = new CancellationTokenSource(RunTimeout);
|
||||
var run = a.Session.RunAsync(duplex, cts.Token);
|
||||
|
||||
// supportsPartialSnapshot omitted == what a 0.1.x/0.2.x handshake decodes to.
|
||||
await DriveHandshakeAsync(a.Db, toSession, fromSession, cts.Token);
|
||||
var frames = await DrainThroughAsync(fromSession, SyncMessage.MsgOneofCase.SnapshotComplete, cts.Token);
|
||||
|
||||
var begin = frames.Single(m => m.MsgCase == SyncMessage.MsgOneofCase.SnapshotBegin).SnapshotBegin;
|
||||
Assert.Empty(begin.Tables); // empty == full, the only scope the old wire can express
|
||||
Assert.Equal(["orders", "widgets"], SnapshotTablesIn(frames));
|
||||
|
||||
cts.Cancel();
|
||||
await SwallowAsync(run);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task PartialSnapshot_DebtClearsOnlyWhenThePeerAcknowledges()
|
||||
{
|
||||
// Crash safety: an ungraceful exit part-way through a snapshot must re-stream it, so the
|
||||
// flag cannot come down when the frames are merely enqueued. The peer's SnapshotAck is
|
||||
// written after its own durable watermark write, so it means "I hold these rows".
|
||||
var a = await NewBaselinedSideAsync();
|
||||
var (duplex, toSession, fromSession) = ScriptedPeer();
|
||||
using var cts = new CancellationTokenSource(RunTimeout);
|
||||
var run = a.Session.RunAsync(duplex, cts.Token);
|
||||
|
||||
await DriveHandshakeAsync(a.Db, toSession, fromSession, cts.Token, supportsPartialSnapshot: true);
|
||||
var frames = await DrainThroughAsync(fromSession, SyncMessage.MsgOneofCase.SnapshotComplete, cts.Token);
|
||||
var complete = frames[^1].SnapshotComplete;
|
||||
|
||||
// The entire snapshot is on the wire and unacknowledged: the debt must still stand.
|
||||
Assert.Equal(["orders"], await PendingSnapshotTables(a.Db));
|
||||
|
||||
await toSession.WriteAsync(new SyncMessage { SnapshotAck = new SnapshotAck { AsOfSeq = complete.AsOfSeq } }, cts.Token);
|
||||
await WaitForAsync(async () => (await PendingSnapshotTables(a.Db)).Count == 0, RunTimeout);
|
||||
|
||||
cts.Cancel();
|
||||
await SwallowAsync(run);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Snapshot_ToAPeerWithoutPartialSupport_ClearsOnEnqueue()
|
||||
{
|
||||
// The 0.2.x behaviour, kept deliberately: a peer that sends no SnapshotAck would otherwise
|
||||
// leave the debt standing forever and re-stream the snapshot on every reconnect.
|
||||
var a = await NewBaselinedSideAsync();
|
||||
var (duplex, toSession, fromSession) = ScriptedPeer();
|
||||
using var cts = new CancellationTokenSource(RunTimeout);
|
||||
var run = a.Session.RunAsync(duplex, cts.Token);
|
||||
|
||||
await DriveHandshakeAsync(a.Db, toSession, fromSession, cts.Token);
|
||||
await DrainThroughAsync(fromSession, SyncMessage.MsgOneofCase.SnapshotComplete, cts.Token);
|
||||
|
||||
await WaitForAsync(async () => (await PendingSnapshotTables(a.Db)).Count == 0, RunTimeout);
|
||||
|
||||
cts.Cancel();
|
||||
await SwallowAsync(run);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task PartialSnapshot_DoesNotAdvanceTheReceiversWatermark()
|
||||
{
|
||||
// The receiver's half of the same invariant. last_applied_remote_seq is ONE number for the
|
||||
// whole peer oplog; moving it to a partial snapshot's as_of would declare the untouched
|
||||
// tables applied through that point, and their deltas below it would never be resent.
|
||||
var a = await NewSideAsync();
|
||||
await a.Store.SetLastAppliedRemoteSeqAsync(10, 0);
|
||||
var (duplex, toSession, fromSession) = ScriptedPeer();
|
||||
|
||||
using var cts = new CancellationTokenSource(RunTimeout);
|
||||
var run = a.Session.RunAsync(duplex, cts.Token);
|
||||
|
||||
await DriveHandshakeAsync(a.Db, toSession, fromSession, cts.Token, supportsPartialSnapshot: true);
|
||||
|
||||
var begin = new SnapshotBegin { AsOfSeq = 900 };
|
||||
begin.Tables.Add("orders");
|
||||
await toSession.WriteAsync(new SyncMessage { SnapshotBegin = begin }, cts.Token);
|
||||
await toSession.WriteAsync(new SyncMessage
|
||||
{
|
||||
SnapshotBatch = new SnapshotBatch { TableName = "orders", Rows = { SnapRow(1, "SNAP", 6_000_000) } },
|
||||
}, cts.Token);
|
||||
await toSession.WriteAsync(new SyncMessage { SnapshotComplete = new SnapshotComplete { AsOfSeq = 900 } }, cts.Token);
|
||||
|
||||
// The ack proves the complete was processed — and that a 0.3.0 receiver answers one at all.
|
||||
var ack = await NextOfCaseAsync(fromSession, SyncMessage.MsgOneofCase.SnapshotAck, cts.Token);
|
||||
Assert.Equal(900, ack.SnapshotAck.AsOfSeq);
|
||||
|
||||
Assert.Equal(10, await LastAppliedRemoteSeq(a.Db)); // NOT 900
|
||||
Assert.Single(await ReadOrders(a.Db)); // the rows still merged
|
||||
|
||||
cts.Cancel();
|
||||
await SwallowAsync(run);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task FullSnapshot_ToAnOldPeer_DrawsNoSnapshotAck()
|
||||
{
|
||||
// Symmetry check on the negotiation: a 0.3.0 receiver must not put a message a pre-0.3.0
|
||||
// sender has never heard of on the wire back to it.
|
||||
var a = await NewSideAsync();
|
||||
var (duplex, toSession, fromSession) = ScriptedPeer();
|
||||
|
||||
using var cts = new CancellationTokenSource(RunTimeout);
|
||||
var run = a.Session.RunAsync(duplex, cts.Token);
|
||||
|
||||
await DriveHandshakeAsync(a.Db, toSession, fromSession, cts.Token);
|
||||
await toSession.WriteAsync(new SyncMessage { SnapshotBegin = new SnapshotBegin { AsOfSeq = 300 } }, cts.Token);
|
||||
await toSession.WriteAsync(new SyncMessage
|
||||
{
|
||||
SnapshotBatch = new SnapshotBatch { TableName = "orders", Rows = { SnapRow(1, "SNAP", 6_000_000) } },
|
||||
}, cts.Token);
|
||||
await toSession.WriteAsync(new SyncMessage { SnapshotComplete = new SnapshotComplete { AsOfSeq = 300 } }, cts.Token);
|
||||
|
||||
// A full snapshot from an old peer still advances the watermark — that is the 0.2.x contract.
|
||||
await WaitForAsync(async () => await LastAppliedRemoteSeq(a.Db) == 300, RunTimeout);
|
||||
// Barrier: an ack for this delta proves everything before it was processed, so an absent
|
||||
// SnapshotAck is absent rather than merely late.
|
||||
await toSession.WriteAsync(new SyncMessage
|
||||
{
|
||||
DeltaBatch = new DeltaBatch { Entries = { new OplogEntry
|
||||
{
|
||||
Seq = 301, TableName = "orders", PkJson = "{\"id\":2}",
|
||||
RowJson = "{\"id\":2,\"sku\":\"BARRIER\",\"qty\":1}", Hlc = 6_000_001, NodeId = "peer", IsTombstone = false,
|
||||
} } },
|
||||
}, cts.Token);
|
||||
await NextOfCaseAsync(fromSession, SyncMessage.MsgOneofCase.DeltaAck, cts.Token);
|
||||
Assert.False(fromSession.TryPeek(out var stray) && stray.MsgCase == SyncMessage.MsgOneofCase.SnapshotAck);
|
||||
|
||||
cts.Cancel();
|
||||
await SwallowAsync(run);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task BaselinedTable_ReachesARealPeer_WithoutRestreamingTheOthers()
|
||||
{
|
||||
// End to end over two real databases: the baselined table's pre-existing rows arrive via the
|
||||
// scoped snapshot, the untouched table's rows arrive as ordinary deltas, and the debt clears.
|
||||
var options = new ReplicationOptions { FlushInterval = TimeSpan.FromMilliseconds(20) };
|
||||
var a = await NewBaselinedSideAsync(options);
|
||||
var b = await NewTwoTableSideAsync(options);
|
||||
|
||||
var (da, db) = DuplexPair();
|
||||
using var cts = new CancellationTokenSource(RunTimeout);
|
||||
var runA = a.Session.RunAsync(da, cts.Token);
|
||||
var runB = b.Session.RunAsync(db, cts.Token);
|
||||
|
||||
async Task<bool> Converged() =>
|
||||
(await ReadOrders(b.Db)).Count == 2
|
||||
&& (await b.Db.QueryAsync("SELECT COUNT(*) FROM widgets", x => x.GetInt64(0)))[0] == 2;
|
||||
|
||||
await WaitForAsync(Converged, RunTimeout);
|
||||
Assert.Equal(await ReadOrders(a.Db), await ReadOrders(b.Db));
|
||||
await WaitForAsync(async () => (await PendingSnapshotTables(a.Db)).Count == 0, RunTimeout);
|
||||
|
||||
// A partial snapshot speaks for nothing outside its tables, so neither side may skip the
|
||||
// deltas it covered: B's watermark tracks the oplog, not the snapshot's as-of.
|
||||
Assert.Equal(0, await NeedsSnapshot(a.Db));
|
||||
|
||||
cts.Cancel();
|
||||
await SwallowAsync(runA);
|
||||
await SwallowAsync(runB);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Sender_ClearsNeedsSnapshotFlag()
|
||||
{
|
||||
|
||||
@@ -208,6 +208,50 @@ public sealed class SyncSessionTests : IDisposable
|
||||
Assert.Empty(await ReadOrders(b.Db));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Handshake_FaultTeardown_StillTransmitsTheLocalHandshake()
|
||||
{
|
||||
// Schema-mismatch detection is symmetric: each side computes its verdict FROM the other's
|
||||
// Handshake. So a session that faults on the peer's handshake must still flush its own —
|
||||
// otherwise the healthy peer learns nothing and waits out its receive cap (tests) or
|
||||
// misclassifies the mismatch as a generic disconnect and reconnect-loops (production).
|
||||
//
|
||||
// The race being pinned: with the peer's handshake already buffered, the path from
|
||||
// enqueueing our Handshake on the control lane to the validation throw has no yields,
|
||||
// while the writer's wakeup sits in the thread-pool queue — a teardown that cancels the
|
||||
// writer can therefore kill the Handshake before it ever transmits. Preloading makes the
|
||||
// sprint as tight as possible, and running the scenarios concurrently starves the pool
|
||||
// the way a busy test suite does.
|
||||
const int Iterations = 40;
|
||||
var sides = new List<Side>(Iterations);
|
||||
for (var i = 0; i < Iterations; i++)
|
||||
sides.Add(await NewSideAsync());
|
||||
|
||||
using var cts = new CancellationTokenSource(RunTimeout);
|
||||
var runs = new List<(Task Run, ChannelReader<SyncMessage> FromSession)>(Iterations);
|
||||
foreach (var side in sides)
|
||||
{
|
||||
var (duplex, toSession, fromSession) = ScriptedPeer();
|
||||
var mismatched = MatchingHandshake(side.Db, "peer");
|
||||
mismatched.Tables[0].Digest = "not-the-local-digest";
|
||||
await toSession.WriteAsync(new SyncMessage { Handshake = mismatched }, cts.Token);
|
||||
runs.Add((side.Session.RunAsync(duplex, cts.Token), fromSession));
|
||||
}
|
||||
|
||||
var lost = 0;
|
||||
foreach (var (run, fromSession) in runs)
|
||||
{
|
||||
await Assert.ThrowsAsync<LocalDbSchemaMismatchException>(() => run);
|
||||
var sent = false;
|
||||
while (fromSession.TryRead(out var msg))
|
||||
sent |= msg.MsgCase == SyncMessage.MsgOneofCase.Handshake;
|
||||
if (!sent)
|
||||
lost++;
|
||||
}
|
||||
|
||||
Assert.True(lost == 0, $"{lost}/{Iterations} sessions tore down without transmitting their own Handshake");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Handshake_SchemaVersionMismatch_FailsClosed()
|
||||
{
|
||||
@@ -553,8 +597,9 @@ public sealed class SyncSessionTests : IDisposable
|
||||
await a.Store.SetNeedsSnapshotAsync(true, default);
|
||||
var (duplex, toSession, fromSession) = ScriptedPeer();
|
||||
var sent = new TaskCompletionSource();
|
||||
a.Session.SnapshotSender = async (send, token) =>
|
||||
a.Session.SnapshotSender = async (scope, send, token) =>
|
||||
{
|
||||
Assert.False(scope.IsPartial); // the database-wide flag always means every table
|
||||
sent.TrySetResult();
|
||||
await send(new SyncMessage { SnapshotComplete = new SnapshotComplete { AsOfSeq = 7 } }, token);
|
||||
return 7;
|
||||
|
||||
@@ -0,0 +1,181 @@
|
||||
using System.Threading.Channels;
|
||||
using Grpc.Core;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.LocalDb.Contracts;
|
||||
using ZB.MOM.WW.LocalDb.Replication;
|
||||
|
||||
namespace ZB.MOM.WW.LocalDb.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// The passive service's single-stream guard under a HALF-OPEN link: the dialling node is gone, but
|
||||
/// nothing on this side knows it, so the handler sits inside <c>MoveNext</c> forever holding the
|
||||
/// guard. Through 0.3.1 that made the pair unrecoverable without a restart — every reconnect from
|
||||
/// the rebooted peer was refused <c>AlreadyExists</c>, indefinitely on an idle pair, while this node
|
||||
/// went on reporting itself connected. The guard is there to serialize sessions, not to fence out a
|
||||
/// caller the auth interceptor already vouched for, so the newest dialler wins.
|
||||
/// </summary>
|
||||
public sealed class SyncStreamGuardTests : IDisposable
|
||||
{
|
||||
private static readonly TimeSpan Timeout = TimeSpan.FromSeconds(10);
|
||||
|
||||
private readonly List<string> _paths = [];
|
||||
private readonly List<IDisposable> _disposables = [];
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
foreach (var d in _disposables)
|
||||
d.Dispose();
|
||||
Microsoft.Data.Sqlite.SqliteConnection.ClearAllPools();
|
||||
foreach (var p in _paths)
|
||||
{
|
||||
if (File.Exists(p)) File.Delete(p);
|
||||
if (File.Exists(p + "-wal")) File.Delete(p + "-wal");
|
||||
if (File.Exists(p + "-shm")) File.Delete(p + "-shm");
|
||||
}
|
||||
}
|
||||
|
||||
private LocalDbSyncService NewService()
|
||||
{
|
||||
var path = Path.Combine(Path.GetTempPath(), Guid.NewGuid() + ".db");
|
||||
_paths.Add(path);
|
||||
var db = new Internal.SqliteLocalDb(new LocalDbOptions { Path = path });
|
||||
_disposables.Add(db);
|
||||
db.ExecuteAsync("CREATE TABLE orders (id INTEGER PRIMARY KEY, sku TEXT)").GetAwaiter().GetResult();
|
||||
db.RegisterReplicated("orders");
|
||||
|
||||
return new LocalDbSyncService(
|
||||
db,
|
||||
Options.Create(new ReplicationOptions { FlushInterval = TimeSpan.FromMilliseconds(20) }),
|
||||
NullLoggerFactory.Instance);
|
||||
}
|
||||
|
||||
private static ServerCallContext ContextFor(CancellationToken ct) => new FakeCallContext(ct);
|
||||
|
||||
[Fact]
|
||||
public async Task WedgedIncumbent_IsDisplacedByANewStream()
|
||||
{
|
||||
var service = NewService();
|
||||
// Eviction is normally generous; the test only needs it to not be the thing under test.
|
||||
service.TakeoverTimeout = Timeout;
|
||||
|
||||
// The incumbent's peer crashed: its stream will never deliver another message, and neither
|
||||
// end times out on silence — a converged pair is legitimately quiet. The handler parks in
|
||||
// the handshake's first read, holding the guard.
|
||||
using var incumbentCts = new CancellationTokenSource();
|
||||
var incumbentOut = new RecordingWriter();
|
||||
var incumbent = service.Sync(new SilentReader(), incumbentOut, ContextFor(incumbentCts.Token));
|
||||
|
||||
// It really is running and really did claim the guard (its handshake is on the wire).
|
||||
await incumbentOut.NextAsync(Timeout);
|
||||
Assert.False(incumbent.IsCompleted);
|
||||
|
||||
// The rebooted peer dials again. Pre-fix this threw AlreadyExists and the pair stayed wedged.
|
||||
using var newcomerCts = new CancellationTokenSource();
|
||||
var newcomerOut = new RecordingWriter();
|
||||
var newcomer = service.Sync(new SilentReader(), newcomerOut, ContextFor(newcomerCts.Token));
|
||||
|
||||
// The incumbent is torn down — gracefully, not as a fault of this node.
|
||||
await incumbent.WaitAsync(Timeout);
|
||||
// ...and the new stream is genuinely served: its own handshake goes out.
|
||||
var handshake = await newcomerOut.NextAsync(Timeout);
|
||||
Assert.Equal(SyncMessage.MsgOneofCase.Handshake, handshake.MsgCase);
|
||||
Assert.False(newcomer.IsCompleted);
|
||||
|
||||
newcomerCts.Cancel();
|
||||
await SwallowAsync(newcomer);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task DisplacedIncumbent_ReleasesTheGuard_SoAThirdStreamIsAlsoServed()
|
||||
{
|
||||
// Takeover must leave the guard in a state that can be taken over AGAIN — a handover that
|
||||
// leaked the slot would simply move the wedge one reconnect down the line.
|
||||
var service = NewService();
|
||||
service.TakeoverTimeout = Timeout;
|
||||
|
||||
using var firstCts = new CancellationTokenSource();
|
||||
var firstOut = new RecordingWriter();
|
||||
var first = service.Sync(new SilentReader(), firstOut, ContextFor(firstCts.Token));
|
||||
await firstOut.NextAsync(Timeout);
|
||||
|
||||
using var secondCts = new CancellationTokenSource();
|
||||
var secondOut = new RecordingWriter();
|
||||
var second = service.Sync(new SilentReader(), secondOut, ContextFor(secondCts.Token));
|
||||
await first.WaitAsync(Timeout);
|
||||
await secondOut.NextAsync(Timeout);
|
||||
|
||||
using var thirdCts = new CancellationTokenSource();
|
||||
var thirdOut = new RecordingWriter();
|
||||
var third = service.Sync(new SilentReader(), thirdOut, ContextFor(thirdCts.Token));
|
||||
await second.WaitAsync(Timeout);
|
||||
Assert.Equal(SyncMessage.MsgOneofCase.Handshake, (await thirdOut.NextAsync(Timeout)).MsgCase);
|
||||
|
||||
thirdCts.Cancel();
|
||||
await SwallowAsync(third);
|
||||
}
|
||||
|
||||
private static async Task SwallowAsync(Task task)
|
||||
{
|
||||
try { await task.WaitAsync(Timeout); }
|
||||
catch (OperationCanceledException) { }
|
||||
catch (RpcException) { }
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The minimum <see cref="ServerCallContext"/> the handler actually reads: its cancellation
|
||||
/// token. (Grpc.Core.Api ships no test double, and the ASP.NET Core one is not reachable outside
|
||||
/// a live call.)
|
||||
/// </summary>
|
||||
private sealed class FakeCallContext(CancellationToken ct) : ServerCallContext
|
||||
{
|
||||
protected override string MethodCore => "/localdb_sync.v1.LocalDbSync/Sync";
|
||||
protected override string HostCore => "test";
|
||||
protected override string PeerCore => "test";
|
||||
protected override DateTime DeadlineCore => DateTime.MaxValue;
|
||||
protected override Metadata RequestHeadersCore => [];
|
||||
protected override CancellationToken CancellationTokenCore => ct;
|
||||
protected override Metadata ResponseTrailersCore { get; } = [];
|
||||
protected override Status StatusCore { get; set; }
|
||||
protected override WriteOptions? WriteOptionsCore { get; set; }
|
||||
protected override AuthContext AuthContextCore => new(null, new Dictionary<string, List<AuthProperty>>());
|
||||
|
||||
protected override ContextPropagationToken CreatePropagationTokenCore(ContextPropagationOptions? options) =>
|
||||
throw new NotSupportedException();
|
||||
|
||||
protected override Task WriteResponseHeadersAsyncCore(Metadata responseHeaders) => Task.CompletedTask;
|
||||
}
|
||||
|
||||
/// <summary>A stream that never delivers anything and never ends — a half-open link.</summary>
|
||||
private sealed class SilentReader : IAsyncStreamReader<SyncMessage>
|
||||
{
|
||||
public SyncMessage Current => throw new InvalidOperationException("no message");
|
||||
|
||||
public async Task<bool> MoveNext(CancellationToken cancellationToken)
|
||||
{
|
||||
await Task.Delay(System.Threading.Timeout.Infinite, cancellationToken);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
private sealed class RecordingWriter : IServerStreamWriter<SyncMessage>
|
||||
{
|
||||
private readonly Channel<SyncMessage> _written = Channel.CreateUnbounded<SyncMessage>();
|
||||
|
||||
public WriteOptions? WriteOptions { get; set; }
|
||||
|
||||
public Task WriteAsync(SyncMessage message) => WriteAsync(message, CancellationToken.None);
|
||||
|
||||
public Task WriteAsync(SyncMessage message, CancellationToken cancellationToken)
|
||||
{
|
||||
_written.Writer.TryWrite(message);
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
|
||||
public async Task<SyncMessage> NextAsync(TimeSpan timeout)
|
||||
{
|
||||
using var cts = new CancellationTokenSource(timeout);
|
||||
return await _written.Reader.ReadAsync(cts.Token);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,65 @@
|
||||
# ZB.MOM.WW.Overview
|
||||
|
||||
The hosted family dashboard (Blazor Server). Build/test from this directory: `dotnet build ZB.MOM.WW.Overview.slnx`, `dotnet test ZB.MOM.WW.Overview.slnx` (fully offline).
|
||||
|
||||
## Index history (migrated from the scadaproj root CLAUDE.md)
|
||||
|
||||
> Moved verbatim out of the scadaproj root `CLAUDE.md` on 2026-09-03 so it loads only when needed. **This file is now the authoritative place for this history — append new entries here and keep the root index row short.**
|
||||
|
||||
|
||||
Besides the shared libraries, this repo hosts one **application**:
|
||||
[`ZB.MOM.WW.Overview/`](ZB.MOM.WW.Overview/) — a Blazor Server single-pane dashboard that polls
|
||||
every registered instance's anonymous `/health/ready` + `/health/active`, caches one snapshot
|
||||
(pages never poll) and renders per-cluster state including the Akka **leader chip**, which is what
|
||||
the `ZB.MOM.WW.Health` `0.2.0`/`0.3.0` per-entry `data` work exists to feed. **No login, strictly
|
||||
read-only, holds no credentials** — a requirement, not an omission; it references no Auth/Audit/
|
||||
Secrets package. Design + impl plan: `docs/plans/2026-07-22-overview-dashboard-{design,impl-plan}.md`
|
||||
(ledger 19/20; the open task is Phase 4 live acceptance, recorded in `41013dc` as 7 of 9 design §9
|
||||
checks passing, 2 blocked on an OtOpcUa rig redeploy).
|
||||
|
||||
The `Applications` registry ships **empty** on purpose: a deployment supplying none fails startup
|
||||
naming the key rather than booting an empty dashboard. Per-deployment overlays are committed —
|
||||
`appsettings.Docker.json` (the local rigs, joined via external networks) and
|
||||
`appsettings.Wonder.json`.
|
||||
|
||||
**Deployed to `wonder-app-vd03` 2026-08-11** (`main @ 4f97e72`) as that box's **fifth** app:
|
||||
service `ZbOverview` on **:5320** under the shared NSSM, `ASPNETCORE_ENVIRONMENT=Wonder`,
|
||||
5/5 instances Up. Two constraints worth carrying to any future box:
|
||||
|
||||
- **Publish self-contained** (`-r win-x64 --self-contained true`). wonder has **no .NET 10 runtime**
|
||||
— every app there carries its own. An FDD publish installs fine and then fails at service start.
|
||||
- **Registry `BaseUrl` must not put co-located instances on a shared host name.** Leader-address →
|
||||
instance resolution matches on **host only** (Akka and HTTP ports differ by design), which assumes
|
||||
distinct hosts per instance — true on the rig, false on a single box, where the fleet-wide scan
|
||||
lets the first matching entry win the chip for *every* group. On wonder that made ScadaBridge's
|
||||
**site** card claim `Leader · central`, a different Akka cluster. Fixed by using `127.0.0.1`,
|
||||
which matches nothing, so each chip falls back to the raw Akka address.
|
||||
|
||||
**QA runs an instance on EACH pair node — `wonder-app-vq03:5320` and (since 2026-08-14)
|
||||
`wonder-app-vq04:5320`** — both watching the same registry, now **10 instances across 4
|
||||
applications** after the 2026-08-14 VQ04 singleton clone (MxGateway + HistorianGateway added on
|
||||
vq04; see `deployment.md` §7.1). It is the fastest way to read QA's whole topology.
|
||||
Its registry uses **real FQDNs**, and that is correct rather than a divergence: the `127.0.0.1`
|
||||
rule above exists only because VD03 co-locates every instance on one host. QA is a **2-node pair**
|
||||
(`vq03`/`vq04`) with genuinely distinct hosts, so leader resolution works as designed and the chips
|
||||
resolve to real names (`node-a (vq03)`). Carry the *reason*, not the literal `127.0.0.1`.
|
||||
|
||||
## Commands (from the index)
|
||||
|
||||
```bash
|
||||
# ZB.MOM.WW.Overview (~/Desktop/scadaproj/ZB.MOM.WW.Overview — the hosted family dashboard)
|
||||
dotnet build ZB.MOM.WW.Overview.slnx
|
||||
dotnet test ZB.MOM.WW.Overview.slnx # 177 tests, fully offline
|
||||
cd docker && docker compose up -d --build # local rigs → http://localhost:5320 (start the rigs first)
|
||||
# wonder deploy artifact — SELF-CONTAINED, the box has no .NET 10 runtime:
|
||||
dotnet publish src/ZB.MOM.WW.Overview/ZB.MOM.WW.Overview.csproj -c Release -r win-x64 --self-contained true -o <out>
|
||||
```
|
||||
|
||||
**2026-10-01 — BTDB Manager registered on all three dashboards.** `appsettings.Wonder.json` (vd03,
|
||||
committed) gains one instance `wonder` → `http://127.0.0.1:9876` (management link
|
||||
`http://10.220.157.247:9876/`); the QA overlay `E:\ApiInstall\Overview\app\appsettings.QA.json` (deployed
|
||||
only, identical on vq03/vq04) gains `vq03`/`vq04` → `http://wonder-app-vq0N.zmr.zimmer.com:9876`. All
|
||||
`HasActiveRole: false` (single unclustered Blazor app); `/health/ready` carries one `sql` entry (BT
|
||||
reachable). The registry is read at startup only, so `ZbOverview` was restarted on each node (previous
|
||||
files kept as `*.bak-20261001T02xxxx`). Verified: vd03 6/6 Up, vq03 and vq04 12/12 Up.
|
||||
|
||||
@@ -25,11 +25,14 @@ The dev rigs publish only some node ports to the host:
|
||||
| OtOpcUa central-1/2 | container `:9000` | **no** — only Traefik `:9200`, load-balanced across the pair |
|
||||
| OtOpcUa site-\* (4) | container `:8080` | **no** — not published |
|
||||
| HistorianGateway | container `:5220` | yes — `localhost:5220` |
|
||||
| BTDB Manager | container `:9877` | yes — `localhost:9877` |
|
||||
| MxAccessGateway | windev `:5130` | yes, with the VPN up |
|
||||
|
||||
So a host-run dashboard can only see part of the fleet. This stack joins the three rig networks
|
||||
(`otopcua-dev_default`, `scadabridge-net`, `zb-historiangw_default`, all declared `external`) and
|
||||
addresses every node by container DNS name, which is how it sees all of them.
|
||||
addresses every node by container DNS name, which is how it sees all of them. BTDB Manager
|
||||
(`~/Desktop/BTDB/docker`) sits on `scadabridge-net` too, because its BT database lives on
|
||||
ScadaBridge's infra MSSQL container, so no fourth network is needed for it.
|
||||
|
||||
Ports above were established by probing the running rigs on 2026-07-24, **not** read off the
|
||||
compose files — the two disagree. In particular OtOpcUa's site nodes serve health on `:8080`, not
|
||||
|
||||
@@ -25,6 +25,8 @@
|
||||
// LocalDb sync re-bind the primary HTTP port)
|
||||
// ScadaBridge central :5000
|
||||
// ScadaBridge site :8084 (MetricsPort; MapZbHealth lands here from the site-health work)
|
||||
// BTDB Manager :9877 (BTDB/docker; single instance, joins scadabridge-net for BT;
|
||||
// /health/ready is 503 whenever that database is unreachable)
|
||||
//
|
||||
// ManagementUrl values stay host-relative (localhost:*) on purpose: they are followed by the
|
||||
// OPERATOR'S BROWSER, not by this container, so container DNS names would be dead links.
|
||||
@@ -146,6 +148,18 @@
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"Name": "BTDB Manager",
|
||||
"ManagementLabel": "Manager",
|
||||
"Instances": [
|
||||
{
|
||||
"Name": "manager",
|
||||
"BaseUrl": "http://zb-btdb-manager:9877",
|
||||
"ManagementUrl": "http://localhost:9877/",
|
||||
"HasActiveRole": false
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"Name": "MxAccessGateway",
|
||||
"ManagementLabel": "Dashboard",
|
||||
|
||||
@@ -150,6 +150,21 @@
|
||||
"TimeoutSeconds": 10
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"Name": "BTDB Manager",
|
||||
"ManagementLabel": "Manager",
|
||||
"Instances": [
|
||||
{
|
||||
// HasActiveRole false: a single, unclustered Blazor app — no active/standby role.
|
||||
// /health/ready carries one "sql" entry (BT on the DEV SQL box reachable) and is 503
|
||||
// whenever that database is not; /health/active is an empty check set.
|
||||
"Name": "wonder",
|
||||
"BaseUrl": "http://127.0.0.1:9876",
|
||||
"ManagementUrl": "http://10.220.157.247:9876/",
|
||||
"HasActiveRole": false
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
@@ -5,6 +5,27 @@ follows [Keep a Changelog](https://keepachangelog.com/); versions are shared acr
|
||||
package in the family (see `Directory.Build.props`) even when a given release only touches
|
||||
one of them.
|
||||
|
||||
## 0.6.3 — 2026-09-28
|
||||
|
||||
### Added
|
||||
|
||||
- **gRPC hub follower sweeps are observable.** `GrpcSecretSyncService` (package
|
||||
`ZB.MOM.WW.Secrets.Replicator.Grpc`) now logs every sweep outcome instead of only the sweeps
|
||||
that pull a change:
|
||||
- `EventId 4` (Debug) — a sweep found nothing new; logged on every in-sync sweep.
|
||||
- `EventId 5` (Information) — an in-sync heartbeat, at most once per
|
||||
`SecretsGrpcHubClientOptions.InSyncLogInterval` (new, default 1 hour, validated positive), so
|
||||
"the loop is alive and still agrees with the hub" is visible at the default log level without
|
||||
a line every 30 seconds.
|
||||
- `EventId 6` (Information) — the first successful sweep after one or more failures, naming
|
||||
the failure count.
|
||||
- **`ISecretsHubSyncStatus`** — a public, DI-registered read model of the follower loop:
|
||||
`LastSweepUtc`, `LastSuccessUtc`, `ConsecutiveFailures`, `LastError`. Getters are synchronized
|
||||
with the writer, so a health check or status page reads a consistent snapshot.
|
||||
|
||||
No behaviour change to syncing itself, and no wire or schema change. Only hub followers gain
|
||||
anything from this release; other consumers can stay on 0.6.2.
|
||||
|
||||
## 0.6.2 — 2026-08-11
|
||||
|
||||
### Changed
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -5,7 +5,7 @@
|
||||
<Nullable>enable</Nullable>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
<LangVersion>latest</LangVersion>
|
||||
<Version>0.6.2</Version>
|
||||
<Version>0.6.3</Version>
|
||||
<ManagePackageVersionsCentrally>true</ManagePackageVersionsCentrally>
|
||||
<PackageReadmeFile>README.md</PackageReadmeFile>
|
||||
</PropertyGroup>
|
||||
|
||||
@@ -16,9 +16,20 @@ set -euo pipefail
|
||||
: "${GITEA_NUGET_SOURCE:?set GITEA_NUGET_SOURCE to your Gitea NuGet feed URL}"
|
||||
: "${GITEA_NUGET_KEY:?set GITEA_NUGET_KEY to your Gitea access token}"
|
||||
|
||||
dotnet pack -c Release -o ./artifacts
|
||||
# Named solution, named output, and a cd to the repo root, all for one reason: `dotnet pack`
|
||||
# with neither a project argument nor a fixed working directory packs whatever solution
|
||||
# happens to sit in the CALLER's directory — and then pushes it to a live feed under this
|
||||
# script's name. Invoked by absolute path from another repo's checkout, the relative form
|
||||
# packed and published 25 unrelated packages (LocalDb's copy of this script, fixed in
|
||||
# 0038d46; recovered with `dotnet nuget delete`, but only because the token happened to
|
||||
# carry delete rights). A publish script must be unambiguous about WHAT it publishes no
|
||||
# matter where it is run.
|
||||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "$REPO_ROOT"
|
||||
|
||||
dotnet nuget push "./artifacts/*.nupkg" \
|
||||
dotnet pack ZB.MOM.WW.Secrets.slnx -c Release -o "$REPO_ROOT/artifacts"
|
||||
|
||||
dotnet nuget push "$REPO_ROOT/artifacts/*.nupkg" \
|
||||
--source "$GITEA_NUGET_SOURCE" \
|
||||
--api-key "$GITEA_NUGET_KEY" \
|
||||
--skip-duplicate
|
||||
|
||||
+9
@@ -150,6 +150,15 @@ public static class SecretsGrpcHubClientExtensions
|
||||
});
|
||||
}
|
||||
|
||||
// Singleton, and registered for BOTH the concrete type and the public interface: the
|
||||
// service needs the concrete type (it calls the internal RecordSuccess/RecordFailure), and
|
||||
// an application observing hub-sync health (a readiness check, a status page) needs only
|
||||
// the read-only ISecretsHubSyncStatus. TryAddSingleton for the concrete type means the
|
||||
// interface registration below resolves the SAME instance rather than constructing a second
|
||||
// one, which would silently split "what the service records" from "what a consumer reads".
|
||||
services.TryAddSingleton<SecretsHubSyncStatus>();
|
||||
services.TryAddSingleton<ISecretsHubSyncStatus>(sp => sp.GetRequiredService<SecretsHubSyncStatus>());
|
||||
|
||||
services.AddHostedService<GrpcSecretSyncService>();
|
||||
|
||||
return services;
|
||||
|
||||
@@ -36,6 +36,13 @@ internal sealed partial class GrpcSecretSyncService : BackgroundService
|
||||
private readonly SecretsGrpcHubClientOptions _options;
|
||||
private readonly ILogger<GrpcSecretSyncService> _logger;
|
||||
private readonly TimeProvider _timeProvider;
|
||||
private readonly SecretsHubSyncStatus _status;
|
||||
|
||||
// Last time a row actually moved, and last time the in-sync heartbeat was logged — both
|
||||
// instance state because they describe THIS service's observed history, not anything the hub
|
||||
// or the local store can answer on demand.
|
||||
private DateTimeOffset? _lastPullUtc;
|
||||
private DateTimeOffset? _lastInSyncHeartbeatUtc;
|
||||
|
||||
/// <summary>Creates the sync service.</summary>
|
||||
/// <param name="local">The node's local store — the only thing this service writes to.</param>
|
||||
@@ -44,13 +51,20 @@ internal sealed partial class GrpcSecretSyncService : BackgroundService
|
||||
/// <param name="options">Hub-client options.</param>
|
||||
/// <param name="logger">Logger for sweep outcomes and failures.</param>
|
||||
/// <param name="timeProvider">Clock, injectable so tests need not wait a real interval.</param>
|
||||
/// <param name="status">
|
||||
/// The sweep-outcome tracker this service updates. Optional so existing callers that construct
|
||||
/// this type directly (tests, mainly) keep compiling; a real host resolves the same singleton
|
||||
/// <c>AddZbSecretsGrpcHubClient</c> registers for <see cref="ISecretsHubSyncStatus"/>, so the two
|
||||
/// stay the same instance.
|
||||
/// </param>
|
||||
public GrpcSecretSyncService(
|
||||
ISecretStore local,
|
||||
ISecretsHubReader hub,
|
||||
ISecretCacheInvalidator? cacheInvalidator,
|
||||
IOptions<SecretsGrpcHubClientOptions> options,
|
||||
ILogger<GrpcSecretSyncService> logger,
|
||||
TimeProvider? timeProvider = null)
|
||||
TimeProvider? timeProvider = null,
|
||||
SecretsHubSyncStatus? status = null)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
ArgumentNullException.ThrowIfNull(local);
|
||||
@@ -59,6 +73,7 @@ internal sealed partial class GrpcSecretSyncService : BackgroundService
|
||||
_options = options.Value;
|
||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
_timeProvider = timeProvider ?? TimeProvider.System;
|
||||
_status = status ?? new SecretsHubSyncStatus();
|
||||
_reconciler = new SecretReplicationReconciler(
|
||||
local,
|
||||
cacheInvalidator,
|
||||
@@ -82,11 +97,22 @@ internal sealed partial class GrpcSecretSyncService : BackgroundService
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Runs one pull-only reconcile against the hub.
|
||||
/// Runs one pull-only reconcile against the hub. Throws on any failure — an unreachable hub, a
|
||||
/// rejected credential, a deadline — so a direct caller sees the raw outcome. The background
|
||||
/// loop never calls this directly; it goes through <see cref="SweepAndRecordAsync"/>, which
|
||||
/// shares this method's reconcile core but also records the outcome and logs it.
|
||||
/// </summary>
|
||||
/// <param name="ct">A token to cancel the sweep.</param>
|
||||
/// <returns>How many rows were pulled from the hub.</returns>
|
||||
internal async Task<int> SweepAsync(CancellationToken ct)
|
||||
internal async Task<int> SweepAsync(CancellationToken ct) =>
|
||||
(await SweepCoreAsync(ct).ConfigureAwait(false)).Pulled;
|
||||
|
||||
/// <summary>
|
||||
/// The reconcile core shared by <see cref="SweepAsync"/> and <see cref="SweepAndRecordAsync"/>:
|
||||
/// fetches the hub manifest and reconciles the local store against it. Throws on failure —
|
||||
/// callers that want failures recorded/logged instead do that around this call, not inside it.
|
||||
/// </summary>
|
||||
private async Task<(int Pulled, int ManifestCount)> SweepCoreAsync(CancellationToken ct)
|
||||
{
|
||||
IReadOnlyList<SecretManifestEntry> hubManifest =
|
||||
await _hub.GetManifestAsync(ct).ConfigureAwait(false);
|
||||
@@ -97,20 +123,54 @@ internal sealed partial class GrpcSecretSyncService : BackgroundService
|
||||
// it rather than re-deriving the diff here also keeps the batching: a cold-starting node's
|
||||
// pull set is central's ENTIRE inventory, and an unchunked GetSecrets would exceed gRPC's
|
||||
// 4 MB default message cap and then fail identically on every interval forever.
|
||||
return await _reconciler
|
||||
int pulled = await _reconciler
|
||||
.ReconcileAsync(hubManifest, (names, token) => _hub.GetManyAsync(names, token), ct)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
return (pulled, hubManifest.Count);
|
||||
}
|
||||
|
||||
private async Task SweepSafelyAsync(CancellationToken ct)
|
||||
private Task SweepSafelyAsync(CancellationToken ct) => SweepAndRecordAsync(ct);
|
||||
|
||||
/// <summary>
|
||||
/// Test-only entry point for the same outcome-recording sweep the background loop runs — never
|
||||
/// throws, always updates <see cref="ISecretsHubSyncStatus"/>, and logs exactly as a real tick
|
||||
/// would. Exists so a test can drive one sweep at a time without standing up the
|
||||
/// <see cref="BackgroundService"/> host lifecycle.
|
||||
/// </summary>
|
||||
/// <param name="ct">A token to cancel the sweep.</param>
|
||||
internal Task SweepForTestAsync(CancellationToken ct) => SweepAndRecordAsync(ct);
|
||||
|
||||
private async Task SweepAndRecordAsync(CancellationToken ct)
|
||||
{
|
||||
DateTimeOffset now = _timeProvider.GetUtcNow();
|
||||
|
||||
try
|
||||
{
|
||||
int pulled = await SweepAsync(ct).ConfigureAwait(false);
|
||||
(int pulled, int manifestCount) = await SweepCoreAsync(ct).ConfigureAwait(false);
|
||||
|
||||
int priorFailures = _status.RecordSuccess(now);
|
||||
|
||||
if (priorFailures > 0)
|
||||
{
|
||||
LogSweepRecovered(priorFailures);
|
||||
}
|
||||
|
||||
if (pulled > 0)
|
||||
{
|
||||
LogSweepConverged(pulled);
|
||||
_lastPullUtc = now;
|
||||
}
|
||||
else
|
||||
{
|
||||
LogSweepInSync(manifestCount);
|
||||
|
||||
if (_lastInSyncHeartbeatUtc is null ||
|
||||
now - _lastInSyncHeartbeatUtc.Value >= _options.InSyncLogInterval)
|
||||
{
|
||||
LogSweepInSyncHeartbeat(manifestCount, _lastPullUtc);
|
||||
_lastInSyncHeartbeatUtc = now;
|
||||
}
|
||||
}
|
||||
}
|
||||
catch (OperationCanceledException) when (ct.IsCancellationRequested)
|
||||
@@ -124,6 +184,7 @@ internal sealed partial class GrpcSecretSyncService : BackgroundService
|
||||
// has the same correct response: keep serving the local store and try again. Letting any
|
||||
// of it escape would fault the BackgroundService and take the host down over a transient
|
||||
// central outage, which is precisely the availability the pull-only topology exists for.
|
||||
_status.RecordFailure(now, ex);
|
||||
LogSweepFailed(ex);
|
||||
}
|
||||
}
|
||||
@@ -145,4 +206,36 @@ internal sealed partial class GrpcSecretSyncService : BackgroundService
|
||||
Level = LogLevel.Warning,
|
||||
Message = "Discarding replicated secret row {SecretName} that could not be applied locally; the rest of the batch continues.")]
|
||||
private partial void LogRowDiscarded(Exception exception, string secretName);
|
||||
|
||||
/// <summary>
|
||||
/// Logged at Debug on every sweep that found nothing to pull — the full-cadence signal, for a
|
||||
/// verbose log level, that the loop is alive and agrees with the hub even between the rare
|
||||
/// Information-level heartbeats.
|
||||
/// </summary>
|
||||
[LoggerMessage(
|
||||
EventId = 4,
|
||||
Level = LogLevel.Debug,
|
||||
Message = "Secrets hub sweep found nothing new; {Count} secret(s) already in sync.")]
|
||||
private partial void LogSweepInSync(int count);
|
||||
|
||||
/// <summary>
|
||||
/// Logged at Information at most once per <see cref="SecretsGrpcHubClientOptions.InSyncLogInterval"/>
|
||||
/// — an in-sync sweep is otherwise Debug-only, so this is the periodic proof at the default log
|
||||
/// level that the follower is still converged rather than silently stuck.
|
||||
/// </summary>
|
||||
[LoggerMessage(
|
||||
EventId = 5,
|
||||
Level = LogLevel.Information,
|
||||
Message = "Secrets hub in sync ({Count} secret(s)); last change pulled {LastPullUtc}")]
|
||||
private partial void LogSweepInSyncHeartbeat(int count, DateTimeOffset? lastPullUtc);
|
||||
|
||||
/// <summary>
|
||||
/// Logged at Information the first time a sweep succeeds after one or more failures — the
|
||||
/// recovery line that closes out whatever run of <see cref="LogSweepFailed"/> Warnings preceded it.
|
||||
/// </summary>
|
||||
[LoggerMessage(
|
||||
EventId = 6,
|
||||
Level = LogLevel.Information,
|
||||
Message = "Secrets hub sync recovered after {Failures} failed sweep(s)")]
|
||||
private partial void LogSweepRecovered(int failures);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,112 @@
|
||||
namespace ZB.MOM.WW.Secrets.Replicator.Grpc;
|
||||
|
||||
/// <summary>Point-in-time outcome of the hub follower's sweeps. Never carries secret material.</summary>
|
||||
/// <remarks>
|
||||
/// Registered as a singleton by <c>AddZbSecretsGrpcHubClient</c> alongside the background sweep
|
||||
/// service, so a host can surface hub-sync health (a readiness check, a status page) without
|
||||
/// scraping logs. Every field reflects the outcome of the most recently completed sweep only —
|
||||
/// there is deliberately no history here, just "what happened last time".
|
||||
/// </remarks>
|
||||
public interface ISecretsHubSyncStatus
|
||||
{
|
||||
/// <summary>
|
||||
/// When the most recent sweep finished, whether it succeeded or failed. <see langword="null"/>
|
||||
/// before the first sweep has completed.
|
||||
/// </summary>
|
||||
DateTimeOffset? LastSweepUtc { get; }
|
||||
|
||||
/// <summary>
|
||||
/// When the most recent <em>successful</em> sweep finished. <see langword="null"/> if no sweep
|
||||
/// has ever succeeded.
|
||||
/// </summary>
|
||||
DateTimeOffset? LastSuccessUtc { get; }
|
||||
|
||||
/// <summary>
|
||||
/// The number of sweeps that have failed in a row, ending at the most recent sweep. Reset to
|
||||
/// zero the moment a sweep succeeds.
|
||||
/// </summary>
|
||||
int ConsecutiveFailures { get; }
|
||||
|
||||
/// <summary>
|
||||
/// The message from the most recent sweep failure, or <see langword="null"/> if the most recent
|
||||
/// sweep succeeded (or none has run yet). This is the exception's type and message only — never
|
||||
/// a stack trace, and never secret material, since the sweep never has plaintext in scope.
|
||||
/// </summary>
|
||||
string? LastError { get; }
|
||||
}
|
||||
|
||||
/// <summary>The mutable implementation the sync service writes to; consumers see only the interface.</summary>
|
||||
/// <remarks>
|
||||
/// Writes (<see cref="RecordSuccess"/>/<see cref="RecordFailure"/>) run under <see cref="_gate"/>,
|
||||
/// and every getter takes the same lock rather than reading its backing field bare — a plain field
|
||||
/// read races the writer with no synchronization otherwise, and a torn/half-written
|
||||
/// <see cref="DateTimeOffset"/> (a multi-field struct) or an interleaving that pairs one write's
|
||||
/// <c>LastSweepUtc</c> with another write's <c>LastError</c> is exactly the inconsistency a status
|
||||
/// consumer must not see.
|
||||
/// </remarks>
|
||||
internal sealed class SecretsHubSyncStatus : ISecretsHubSyncStatus
|
||||
{
|
||||
private readonly object _gate = new();
|
||||
|
||||
private DateTimeOffset? _lastSweepUtc;
|
||||
private DateTimeOffset? _lastSuccessUtc;
|
||||
private int _consecutiveFailures;
|
||||
private string? _lastError;
|
||||
|
||||
/// <inheritdoc/>
|
||||
public DateTimeOffset? LastSweepUtc
|
||||
{
|
||||
get { lock (_gate) { return _lastSweepUtc; } }
|
||||
}
|
||||
|
||||
/// <inheritdoc/>
|
||||
public DateTimeOffset? LastSuccessUtc
|
||||
{
|
||||
get { lock (_gate) { return _lastSuccessUtc; } }
|
||||
}
|
||||
|
||||
/// <inheritdoc/>
|
||||
public int ConsecutiveFailures
|
||||
{
|
||||
get { lock (_gate) { return _consecutiveFailures; } }
|
||||
}
|
||||
|
||||
/// <inheritdoc/>
|
||||
public string? LastError
|
||||
{
|
||||
get { lock (_gate) { return _lastError; } }
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Records a successful sweep, resetting the failure streak.
|
||||
/// </summary>
|
||||
/// <param name="now">The instant the sweep completed.</param>
|
||||
/// <returns>
|
||||
/// The consecutive-failure count <em>before</em> this success — the caller uses a non-zero
|
||||
/// result to decide whether the sweep is a recovery worth logging.
|
||||
/// </returns>
|
||||
internal int RecordSuccess(DateTimeOffset now)
|
||||
{
|
||||
lock (_gate)
|
||||
{
|
||||
int prior = _consecutiveFailures;
|
||||
_lastSweepUtc = _lastSuccessUtc = now;
|
||||
_consecutiveFailures = 0;
|
||||
_lastError = null;
|
||||
return prior;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Records a failed sweep, extending the failure streak.</summary>
|
||||
/// <param name="now">The instant the sweep failed.</param>
|
||||
/// <param name="ex">The exception the sweep failed with.</param>
|
||||
internal void RecordFailure(DateTimeOffset now, Exception ex)
|
||||
{
|
||||
lock (_gate)
|
||||
{
|
||||
_lastSweepUtc = now;
|
||||
_consecutiveFailures++;
|
||||
_lastError = ex.GetType().Name + ": " + ex.Message;
|
||||
}
|
||||
}
|
||||
}
|
||||
+21
@@ -77,6 +77,20 @@ public sealed class SecretsGrpcHubClientOptions
|
||||
/// </summary>
|
||||
public TimeSpan CallDeadline { get; set; } = TimeSpan.FromSeconds(30);
|
||||
|
||||
/// <summary>
|
||||
/// How often an in-sync sweep (nothing to pull) is allowed to log an Information-level
|
||||
/// heartbeat. Defaults to one hour. Must be positive.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// A converged follower sweeps every <see cref="SyncInterval"/> — 30 seconds by default — and
|
||||
/// finds nothing new almost every time. Logging that outcome at Information on every sweep
|
||||
/// would drown the log; staying silent forever would make "the sweep loop is alive and the hub
|
||||
/// still agrees with this node" unobservable between the rare sweeps that actually pull
|
||||
/// something. Every sweep still logs at Debug regardless of this interval, so a verbose log
|
||||
/// level shows the true cadence when it is needed.
|
||||
/// </remarks>
|
||||
public TimeSpan InSyncLogInterval { get; set; } = TimeSpan.FromHours(1);
|
||||
|
||||
/// <summary>
|
||||
/// Throws if the options are unusable. Called by the DI extension at registration time so a
|
||||
/// misconfigured node fails at startup rather than on the first silent sweep failure.
|
||||
@@ -161,6 +175,13 @@ public sealed class SecretsGrpcHubClientOptions
|
||||
$"gRPC secrets hub client CallDeadline must be positive " +
|
||||
$"({sectionPath}:CallDeadline, was {CallDeadline}).");
|
||||
}
|
||||
|
||||
if (InSyncLogInterval <= TimeSpan.Zero)
|
||||
{
|
||||
throw new InvalidOperationException(
|
||||
$"gRPC secrets hub client InSyncLogInterval must be positive " +
|
||||
$"({sectionPath}:InSyncLogInterval, was {InSyncLogInterval}).");
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>The configuration section these options are bound from unless a caller overrides it.</summary>
|
||||
|
||||
+250
@@ -0,0 +1,250 @@
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.Secrets.Abstractions;
|
||||
|
||||
namespace ZB.MOM.WW.Secrets.Replicator.Grpc.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Covers the hub follower's sweep-outcome observability: the Debug-per-sweep / Information-
|
||||
/// heartbeat-once-per-interval pair for an in-sync sweep (EventId 4/5), the recovered line the
|
||||
/// first success after a run of failures logs (EventId 6), and <see cref="ISecretsHubSyncStatus"/>
|
||||
/// tracking the same outcomes.
|
||||
/// </summary>
|
||||
public sealed class GrpcSecretSyncServiceLoggingTests
|
||||
{
|
||||
[Fact]
|
||||
public async Task InSyncSweep_LogsDebugEveryTime_AndInformationHeartbeatOncePerInterval()
|
||||
{
|
||||
var time = new FakeTimeProvider(DateTimeOffset.Parse("2026-09-28T00:00:00Z"));
|
||||
(GrpcSecretSyncService svc, RecordingListLogger logs, _) = Harness.Create(
|
||||
time, inSyncLogInterval: TimeSpan.FromHours(1), manifestUnchanged: true);
|
||||
|
||||
await svc.SweepForTestAsync(CancellationToken.None);
|
||||
time.Advance(TimeSpan.FromMinutes(30));
|
||||
await svc.SweepForTestAsync(CancellationToken.None);
|
||||
time.Advance(TimeSpan.FromMinutes(31));
|
||||
await svc.SweepForTestAsync(CancellationToken.None);
|
||||
|
||||
Assert.Equal(3, logs.Entries.Count(l => l.EventId.Id == 4 && l.Level == LogLevel.Debug));
|
||||
Assert.Equal(2, logs.Entries.Count(l => l.EventId.Id == 5 && l.Level == LogLevel.Information));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task FirstSuccessAfterFailures_LogsRecovered_AndStatusResets()
|
||||
{
|
||||
var time = new FakeTimeProvider();
|
||||
(GrpcSecretSyncService svc, RecordingListLogger logs, ISecretsHubSyncStatus status) =
|
||||
Harness.Create(time, failFirst: 2);
|
||||
|
||||
await svc.SweepForTestAsync(default);
|
||||
await svc.SweepForTestAsync(default);
|
||||
|
||||
Assert.Equal(2, status.ConsecutiveFailures);
|
||||
|
||||
await svc.SweepForTestAsync(default);
|
||||
|
||||
Assert.Contains(logs.Entries, l => l.EventId.Id == 6 && l.Message.Contains("after 2 failed"));
|
||||
Assert.Equal(0, status.ConsecutiveFailures);
|
||||
Assert.NotNull(status.LastSuccessUtc);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_failed_sweep_never_logs_EventId6_and_status_records_the_error()
|
||||
{
|
||||
var time = new FakeTimeProvider();
|
||||
(GrpcSecretSyncService svc, RecordingListLogger logs, ISecretsHubSyncStatus status) =
|
||||
Harness.Create(time, failFirst: 1);
|
||||
|
||||
await svc.SweepForTestAsync(default);
|
||||
|
||||
Assert.DoesNotContain(logs.Entries, l => l.EventId.Id == 6);
|
||||
Assert.Equal(1, status.ConsecutiveFailures);
|
||||
Assert.NotNull(status.LastError);
|
||||
Assert.Null(status.LastSuccessUtc);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task An_in_sync_sweep_never_logs_the_recovered_line()
|
||||
{
|
||||
var time = new FakeTimeProvider();
|
||||
(GrpcSecretSyncService svc, RecordingListLogger logs, ISecretsHubSyncStatus status) =
|
||||
Harness.Create(time, manifestUnchanged: true);
|
||||
|
||||
await svc.SweepForTestAsync(default);
|
||||
await svc.SweepForTestAsync(default);
|
||||
|
||||
Assert.DoesNotContain(logs.Entries, l => l.EventId.Id == 6);
|
||||
Assert.Equal(0, status.ConsecutiveFailures);
|
||||
Assert.NotNull(status.LastSweepUtc);
|
||||
}
|
||||
|
||||
// Assembles a GrpcSecretSyncService wired to a controllable fake hub, a no-op local store (the
|
||||
// scenarios here never actually apply a row — an empty hub manifest means the reconciler's pull
|
||||
// set is always empty), a log-capturing logger, and the internal status tracker the service
|
||||
// writes to — so a test can both drive sweeps and observe everything they produced.
|
||||
private static class Harness
|
||||
{
|
||||
public static (GrpcSecretSyncService Service, RecordingListLogger Logs, ISecretsHubSyncStatus Status) Create(
|
||||
FakeTimeProvider time,
|
||||
TimeSpan? inSyncLogInterval = null,
|
||||
bool manifestUnchanged = false,
|
||||
int failFirst = 0)
|
||||
{
|
||||
var logger = new RecordingListLogger();
|
||||
var status = new SecretsHubSyncStatus();
|
||||
|
||||
ISecretsHubReader hub = failFirst > 0
|
||||
? new SequencedFailThenEmptyHubReader(failFirst)
|
||||
: new EmptyHubReader();
|
||||
|
||||
IOptions<SecretsGrpcHubClientOptions> options = Options.Create(new SecretsGrpcHubClientOptions
|
||||
{
|
||||
Endpoint = "https://central:5001",
|
||||
BearerToken = "unused-by-the-fake",
|
||||
SyncInterval = TimeSpan.FromHours(1),
|
||||
InSyncLogInterval = inSyncLogInterval ?? TimeSpan.FromHours(1),
|
||||
});
|
||||
|
||||
// manifestUnchanged has no separate fake to select: both readers above always answer an
|
||||
// empty (therefore unchanged, nothing-to-pull) manifest once they are done failing, so
|
||||
// the parameter exists to make each call site self-documenting rather than to branch on.
|
||||
var svc = new GrpcSecretSyncService(
|
||||
new NoOpLocalStore(),
|
||||
hub,
|
||||
cacheInvalidator: null,
|
||||
options,
|
||||
logger,
|
||||
time,
|
||||
status);
|
||||
|
||||
return (svc, logger, status);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>A hub with nothing in it — every sweep is in-sync with zero rows to pull.</summary>
|
||||
private sealed class EmptyHubReader : ISecretsHubReader
|
||||
{
|
||||
public Task<IReadOnlyList<SecretManifestEntry>> GetManifestAsync(CancellationToken ct) =>
|
||||
Task.FromResult<IReadOnlyList<SecretManifestEntry>>([]);
|
||||
|
||||
public Task<IReadOnlyList<StoredSecret>> GetManyAsync(
|
||||
IReadOnlyList<SecretName> names, CancellationToken ct) =>
|
||||
Task.FromResult<IReadOnlyList<StoredSecret>>([]);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Fails the manifest call with a transport-shaped exception for the first
|
||||
/// <paramref name="failuresBeforeSuccess"/> calls, then behaves exactly like
|
||||
/// <see cref="EmptyHubReader"/> forever after.
|
||||
/// </summary>
|
||||
private sealed class SequencedFailThenEmptyHubReader(int failuresBeforeSuccess) : ISecretsHubReader
|
||||
{
|
||||
private int _remaining = failuresBeforeSuccess;
|
||||
|
||||
public Task<IReadOnlyList<SecretManifestEntry>> GetManifestAsync(CancellationToken ct)
|
||||
{
|
||||
if (_remaining > 0)
|
||||
{
|
||||
_remaining--;
|
||||
return Task.FromException<IReadOnlyList<SecretManifestEntry>>(
|
||||
new InvalidOperationException("hub unreachable"));
|
||||
}
|
||||
|
||||
return Task.FromResult<IReadOnlyList<SecretManifestEntry>>([]);
|
||||
}
|
||||
|
||||
public Task<IReadOnlyList<StoredSecret>> GetManyAsync(
|
||||
IReadOnlyList<SecretName> names, CancellationToken ct) =>
|
||||
Task.FromResult<IReadOnlyList<StoredSecret>>([]);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A local store that is never actually exercised by these scenarios (the hub manifest is
|
||||
/// always empty, so the reconciler's pull set is always empty and none of these members are
|
||||
/// called) beyond <see cref="GetManifestAsync"/>, which the reconciler always calls to compute
|
||||
/// the pull set.
|
||||
/// </summary>
|
||||
private sealed class NoOpLocalStore : ISecretStore
|
||||
{
|
||||
public Task<StoredSecret?> GetAsync(SecretName name, CancellationToken ct) =>
|
||||
throw new NotSupportedException("not exercised by the logging test scenarios.");
|
||||
|
||||
public Task UpsertAsync(StoredSecret row, CancellationToken ct) =>
|
||||
throw new NotSupportedException("not exercised by the logging test scenarios.");
|
||||
|
||||
public Task<bool> DeleteAsync(SecretName name, string? actor, CancellationToken ct) =>
|
||||
throw new NotSupportedException("not exercised by the logging test scenarios.");
|
||||
|
||||
public Task<IReadOnlyList<SecretMetadata>> ListAsync(bool includeDeleted, CancellationToken ct) =>
|
||||
throw new NotSupportedException("not exercised by the logging test scenarios.");
|
||||
|
||||
public Task<IReadOnlyList<SecretManifestEntry>> GetManifestAsync(CancellationToken ct) =>
|
||||
Task.FromResult<IReadOnlyList<SecretManifestEntry>>([]);
|
||||
|
||||
public Task ApplyReplicatedAsync(StoredSecret row, CancellationToken ct) =>
|
||||
throw new NotSupportedException("not exercised by the logging test scenarios.");
|
||||
|
||||
public Task<bool> ApplyRewrapAsync(
|
||||
StoredSecret rewrappedRow, byte[] expectedCurrentWrappedDek, CancellationToken ct) =>
|
||||
throw new NotSupportedException("not exercised by the logging test scenarios.");
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>A clock a test advances by hand, so an hourly heartbeat can be tested without waiting an hour.</summary>
|
||||
/// <remarks>
|
||||
/// Hand-rolled rather than taken from <c>Microsoft.Extensions.TimeProvider.Testing</c>, matching
|
||||
/// this project's existing convention (see <c>HubFakeTimeProvider</c> in
|
||||
/// <c>SecretsHubAuthInterceptorTests.cs</c>) of not taking that package dependency for one small type.
|
||||
/// </remarks>
|
||||
internal sealed class FakeTimeProvider : TimeProvider
|
||||
{
|
||||
private DateTimeOffset _now;
|
||||
|
||||
/// <summary>Creates a clock starting at the current wall-clock time.</summary>
|
||||
public FakeTimeProvider() : this(DateTimeOffset.UtcNow)
|
||||
{
|
||||
}
|
||||
|
||||
/// <summary>Creates a clock starting at <paramref name="now"/>.</summary>
|
||||
/// <param name="now">The instant the clock starts at.</param>
|
||||
public FakeTimeProvider(DateTimeOffset now) => _now = now;
|
||||
|
||||
/// <inheritdoc/>
|
||||
public override DateTimeOffset GetUtcNow() => _now;
|
||||
|
||||
/// <summary>Moves the clock forward by <paramref name="delta"/>.</summary>
|
||||
/// <param name="delta">The amount of time to advance.</param>
|
||||
public void Advance(TimeSpan delta) => _now += delta;
|
||||
}
|
||||
|
||||
/// <summary>Captures every log entry's level, event id, and formatted message, across all levels.</summary>
|
||||
/// <remarks>
|
||||
/// Distinct from the existing <c>RecordingLogger<T></c> in <c>TestDoubles.cs</c>, which only
|
||||
/// captures Information/Warning and neither type captures <see cref="EventId"/> — this suite needs
|
||||
/// Debug-level entries and event-id assertions that the existing double does not support, and
|
||||
/// widening it would risk the tests that already depend on its narrower shape.
|
||||
/// </remarks>
|
||||
internal sealed class RecordingListLogger : ILogger<GrpcSecretSyncService>
|
||||
{
|
||||
/// <summary>Every entry logged, in order.</summary>
|
||||
public List<(LogLevel Level, EventId EventId, string Message)> Entries { get; } = [];
|
||||
|
||||
/// <inheritdoc/>
|
||||
public IDisposable? BeginScope<TState>(TState state) where TState : notnull => null;
|
||||
|
||||
/// <inheritdoc/>
|
||||
public bool IsEnabled(LogLevel logLevel) => true;
|
||||
|
||||
/// <inheritdoc/>
|
||||
public void Log<TState>(
|
||||
LogLevel logLevel,
|
||||
EventId eventId,
|
||||
TState state,
|
||||
Exception? exception,
|
||||
Func<TState, Exception?, string> formatter)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(formatter);
|
||||
|
||||
Entries.Add((logLevel, eventId, formatter(state, exception)));
|
||||
}
|
||||
}
|
||||
+7
-2
@@ -291,11 +291,16 @@ public sealed class GrpcSecretSyncServiceTests : IDisposable
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_quiet_sweep_logs_nothing_at_information()
|
||||
public async Task A_quiet_sweep_logs_the_in_sync_heartbeat_once_but_no_warning()
|
||||
{
|
||||
// A quiet (nothing-to-pull) sweep no longer logs NOTHING at Information: the first-ever
|
||||
// in-sync sweep always logs the once-per-InSyncLogInterval heartbeat (EventId 5), because
|
||||
// "never logged before" is one of the two conditions that fires it. See
|
||||
// GrpcSecretSyncServiceLoggingTests for the interval-gated repeat behaviour on later sweeps.
|
||||
// What a quiet sweep still never does is warn.
|
||||
await RunOneStartupSweepAsync(() => _hub.ManifestCallCount > 0);
|
||||
|
||||
Assert.Equal(0, _logger.InformationCount);
|
||||
Assert.Equal(1, _logger.InformationCount);
|
||||
Assert.Equal(0, _logger.WarningCount);
|
||||
}
|
||||
|
||||
|
||||
@@ -75,3 +75,32 @@ Design documentation:
|
||||
- `~/Desktop/scadaproj/components/observability/spec/METRIC-CONVENTIONS.md` — metric naming reference
|
||||
- `~/Desktop/scadaproj/components/observability/shared-contract/ZB.MOM.WW.Telemetry.md` — proposed shared-library API
|
||||
- `~/Desktop/scadaproj/components/observability/current-state/` — per-project current state (code-verified)
|
||||
|
||||
## Index history (migrated from the scadaproj root CLAUDE.md)
|
||||
|
||||
> Moved verbatim out of the scadaproj root `CLAUDE.md` on 2026-09-03 so it loads only when needed. **This file is now the authoritative place for this history — append new entries here and keep the root index row short.**
|
||||
|
||||
### Narrative
|
||||
|
||||
The observability component is fully populated: a normalized [`spec`](components/observability/spec/SPEC.md),
|
||||
a [`metric-conventions`](components/observability/spec/METRIC-CONVENTIONS.md) reference, a
|
||||
[`shared-contract`](components/observability/shared-contract/ZB.MOM.WW.Telemetry.md), three
|
||||
[`current-state`](components/observability/current-state/) docs, and an adoption [`GAPS`](components/observability/GAPS.md)
|
||||
backlog. Shared = OTel Resource (service.name/site.id/node.role identity triple) + standard instrumentation
|
||||
(ASP.NET Core, HttpClient, gRPC client, runtime, process) + Prometheus always-on exporter + OTLP opt-in
|
||||
+ Serilog two-stage bootstrap + SiteId/NodeRole/NodeHostname enrichers + TraceContextEnricher (trace_id/span_id)
|
||||
+ ILogRedactor seam; left per-project = application Meters/ActivitySources, sink config, per-operation
|
||||
enrichers, and redaction policies.
|
||||
|
||||
The shared library is **built and lives in this repo** at [`ZB.MOM.WW.Telemetry/`](ZB.MOM.WW.Telemetry/)
|
||||
(.NET 10; 2 packages — `ZB.MOM.WW.Telemetry`, `ZB.MOM.WW.Telemetry.Serilog`; 34 tests (verified 2026-07-18);
|
||||
`dotnet pack` → 2 nupkgs @ 0.1.0). **Adopted across all three apps on 2026-06-01** (branch
|
||||
`feat/adopt-zb-telemetry` per repo, behaviour-preserving): `AddZbTelemetry` (Resource + standard
|
||||
instrumentation + Prometheus `/metrics`) everywhere; OtOpcUa + MxGateway on `AddZbSerilog` (MxGateway's
|
||||
MEL→Serilog migration + metrics export both landed in this pass — they were *not* actually done
|
||||
beforehand despite an earlier claim); ScadaBridge keeps its `LoggerConfigurationFactory` (min-level
|
||||
governance) and only adds the shared `TraceContextEnricher`. Deferred: MxGateway `ms`→`s` + Meter
|
||||
rename, ScadaBridge app instruments + Site-node HTTP/1.1 metrics listener, OTLP wiring. Per-repo
|
||||
result tracked in [`components/observability/GAPS.md`](components/observability/GAPS.md).
|
||||
Build/test from `ZB.MOM.WW.Telemetry/`: `dotnet test`. Consumer matrix: all three apps consume both
|
||||
packages after adoption (OtOpcUa, MxGateway Server, ScadaBridge Host + any instrumented project).
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user