Index: record mxaccessgw operator-authenticated alarm ack (proto change)

Claude-Session: https://claude.ai/code/session_016J6isfcgqy4SB2RK3ZvFvT

.claude/skills/sister-project-history/SKILL.md

@@ -0,0 +1,16 @@
+---
+name: sister-project-history
+description: Use when you need the detailed change history, live-gate records, redundancy/auth/ACL behaviour, or ScadaBridge consumer-impact notes for a sister project (OtOpcUa, HistorianGateway, BTDB) — the scadaproj root CLAUDE.md keeps only a short summary row per project; the full narrative lives in this directory.
+---
+
+# Sister project history
+
+The root `CLAUDE.md` index keeps one short row per sister project. The full narrative rows that used to live
+in that table were moved here **verbatim** on 2026-09-03 (they were ~57k characters loading into every session):
+
+- `otopcua.md` — OtOpcUa: v3 dual-namespace address space, every driver, redundancy/mesh program, data-plane auth + ACL enforcement, deferment register, the authoring-vs-deploy-gate parity arc, and the per-issue consumer-impact record for ScadaBridge.
+- `historiangateway.md` — HistorianGateway: ArchReview cycles, pending.md items, session pools, datatype gaps, SQL-path workarounds, PR/commit history.
+- `btdb.md` — BTDB (Batch Tracking Database, added 2026-09-03): schema + sproc write contract, Historian/CLR coupling, Galaxy/MESAPI/ScadaBridge edges, the DARS `BatchTracker_DB` fork, and the gap analysis.
+
+Read the file for the project you are working against. When a sister repo tells you to "update its entry in
+`scadaproj/CLAUDE.md`", append the detail to the file here and keep the index row to a few lines.

.claude/skills/sister-project-history/btdb.md

@@ -0,0 +1,207 @@
+# BTDB (Batch Tracking Database) — index history + analysis
+
+> Added 2026-09-03 as a sister project. **This file is the authoritative place for BTDB detail — append here and keep the root index row short.** Everything below was verified from source on 2026-09-03 unless marked _(inferred)_.
+
+| Project | Location | Stack | Repo | Summary |
+|---|---|---|---|---|
+| **BTDB** (Batch Tracker) | `~/Desktop/btdb` (case-insensitive FS: also `BTDB`) | SQL Server 2016+ SSDT project + SQL CLR (net472) · manager: **Blazor Web App net10.0** on `feat/manager-blazor` (§8; `master` still carries the EOL net6.0 MVC app) | **`ZimmerBiomet/BTDB`** on the corporate Gitea `wonder-app-vd03.zmr.zimmer.com:3000` (`master` @ `d147c10e`, 2023-07-26). **Not** on `gitea.dohertylan.com`. | The `BT` SQL Server database that is the plant's **batch/cycle/work-order system of record for data collection**: Galaxy machine objects call its `StartMachineBatch`/`StartMachineCycle`/`End*`/`AddMachineBatchWorkOrder`/`AddOtherData*` sprocs; reports pull time-series per batch out of the AVEVA Historian through its CLR sprocs (`OPENQUERY(INSQL, … WideHistory …)` + `Runtime.dbo.*`); MESAPI and ScadaBridge's inbound API resolve `SAPID → Machine.Code` from `dbo.Machine`. `BTDBManager` is a small Windows-service MVC site for machine/group config, batch edits (audited) and cross-environment config import. |
+
+## 1. What is on disk, and where the truth is
+
+- `~/Desktop/btdb` is **not a git checkout** — no `.git`, no `CLAUDE.md`, no `README`. It is a file-for-file snapshot of `ZimmerBiomet/BTDB` `master` (the only branch) at its head commit `d147c10e` (2023-07-26, "Update notification system tables to add urgent flag"). Verified by comparing the Gitea tree with the local tree: 170 vs 168 files, the only difference being the remote's `.gitattributes`/`.gitignore`. The repo's `updated_at` of 2024-11-07 is not a commit (the branch head is still 2023-07-26).
+- `~/Desktop/DARS/WW_ARCH/BatchTracker_DB/` is a **fork of the `Database` project only** (no Manager), modified 2026-08-13, also not under git (DARS is not a repo). Substantive differences vs the snapshot, all matching the DARS sprint-1 spec `DARS/sprints/sprint1/bt_db_changes.md`:
+  - new table `MachineBatchTransfer` (source/dest batch chain for mid-run reactor failure; `TransferType` CHECK `'mid_run_failure'`; three indexes) + sproc `AddMachineBatchTransfer` (linear-chain guard);
+  - `MachineBatchWO` gains `ContainerID NVARCHAR(64) NULL` + `MoveOutTimestamp DATETIME2 NULL` with filtered indexes (Camstar container traceability; mirrors the `SAPID`/`ZTag` idiom);
+  - new CLR sproc `GetLinkedBatchTSV` (+ `_Wrapper`) that walks the transfer chain and stitches TSVs, and `LinkedBatchAlarm_Wrapper`;
+  - `StartMachineBatch.sql`, `MachineBatch.sql`, `AuditLogMachineBatch.sql` differ **only** in BOM/CRLF → LF normalization (no logic change).
+  - Reference doc for the schema: `DARS/WW_ARCH/BatchTracker_Reference.md` (states 22 tables / 35 SQL sprocs / 4 CLR sprocs / 5 CLR UDFs / 6 table types / 1 aggregate).
+- **Resolved 2026-09-03 — the divergence is closed locally:** `~/Desktop/btdb` was turned into a real checkout (`git init` + fetch of `ZimmerBiomet/BTDB`, `master` tracking `origin/master`; the snapshot was tree-identical apart from exec bits copied from Windows, which were cleared). The DARS fork's transfer-chain work was migrated onto branch **`feat/machine-batch-transfer`** as one commit: the five new objects, the two `MachineBatchWO` columns, and the `.sqlproj` wiring **including the `Compile` entry for `GetLinkedBatchTSV.cs` that the fork had omitted** (without it the CLR sproc is never built and its wrapper fails at deploy). The BOM/CRLF re-saves were not carried over. The sprint-1 plan (`bt_db_changes.md`, overview, diagrams) and `BatchTracker_Reference.md` now live under `btdb/docs/`. **The branch is local only — not pushed to the corporate Gitea.** `DARS/WW_ARCH/BatchTracker_DB/` is now superseded and should be deleted or replaced by a pointer once the branch is pushed. A second commit on the same branch carried the two columns through the Manager's batch-edit path: `MachineBatchWOType` (appended last — the TVP binds by ordinal and `ToDataTable` emits properties in declaration order, so the view model's property order must keep matching), `AuditLogMachineBatchWO` (`Prev_`/current pairs), the DTOs and view model, `SaveBatchEdit.sql` (MERGE change-detection with `COALESCE` because both columns are NULL-able, OUTPUT and the three audit inserts), and the Kendo work-order grid in `Views/BatchEdit/Edit.cshtml` (masked timestamp editor like the cycle grid). `LookupBatchInfo.sql` is `SELECT *` so it needed nothing. Not touched: the Telerik `BatchEditAuditLogReport.trdp` (binary package; open it in the Telerik designer to add the audit columns) and the batch search grid (`SearchBatches.sql`), which still lists only work-order numbers. Nothing here was built or run: the Manager needs Telerik + Windows, and the SSDT project needs MSBuild; a throwaway compile of the Manager retargeted to .NET 10 on the Mac showed no errors in the touched files.
+
+## 2. Solution shape
+
+`BTDB.sln` (VS 2022) = two projects:
+
+| Project | Kind | Notes |
+|---|---|---|
+| `Database/Database.sqlproj` | SSDT, `Sql130DatabaseSchemaProvider` (SQL Server 2016+), CLR assembly targets **net472**, C# namespace `BT` | 22 tables, 6 table types, ~35 T-SQL sprocs, 3 CLR sprocs (`GetTSV`, `GetMachineBatchTSV`, `GetCycleTSV`), 5 CLR UDFs (`GetAggregateValues`, `CheckTriggered`, `WhenTriggered`, `WhenCleared`, `StringCompare`), 1 CLR aggregate (`StringJoin`), 1 T-SQL TVF (`GenerateShifts`), 1 trigger (`TrgNotification` → `AuditLogNotification`). Requires `CLR enabled` on the instance. |
+| `Manager/BTDBManager.csproj` (**`master` only — deleted on `feat/manager-blazor`, see §8**) | ASP.NET Core MVC, **net6.0** (out of support since 2024-11), Dapper 2.0, `System.Data.SqlClient`, `System.DirectoryServices` (Windows-only LDAP), `Microsoft.Extensions.Hosting.WindowsServices`, Kendo UI + Bootstrap, **Telerik Reporting R2 2022** referenced by an absolute `HintPath` into `Program Files (x86)` | Kestrel `http://*:9876`; runs as a Windows service; 13 embedded `.sql` query resources; 3 `.trdp` Telerik audit reports. `Manager/install.bat`/`uninstall.bat` are **leftovers from a different service** (`WWNotificationSystem.exe`) and do not install the Manager. |
+
+## 3. Database model (what the family writes and reads)
+
+**Core hierarchy:** `Machine` (`Code` unique; optional unique `ZTag`, `SAPID`; `TimeZone`; `MultipleBatch`/`MultipleCycle` flags; `OtherData` JSON) → `MachineBatch` (start/end operator + UTC timestamps, `OtherData`) → `MachineCycle` (same shape) and `MachineBatchWO` (`WorkOrderNumber`, `ItemNumber`). `MachineGroup` + `MachineGroupMembership` give roll-ups (DARS models one `Machine` per reactor *side*, one `MachineGroup` per unit). `MachineAlarm` is a per-machine alarm catalog (`Severity` 0–999, `FlaggedForMES`).
+
+**Lifecycle sprocs (the write contract):**
+
+| Sproc | Behaviour |
+|---|---|
+| `StartMachineBatch(@Code, @OperatorID, @o_MachineBatchID OUT)` | Looks up machine by `Code`; unless `MultipleCycle`/`MultipleBatch`, **auto-closes** any open cycle/batch first; inserts with `GETUTCDATE()`. Throws 50000 if machine unknown. |
+| `EndMachineBatch(@MachineBatchID, @EndOperatorID)` | Closes the open cycle too; throws if not found / already closed. |
+| `StartMachineCycle(@MachineBatchID, @OperatorID, @o_MachineCycleID OUT)` / `EndMachineCycle` | Cycle must belong to an open batch. |
+| `AddMachineBatchWorkOrder(@MachineBatchID, @WorkOrderNumber, @ItemNumber, @o OUT)` | Only on open batches. |
+| `AddOtherDataMachineBatch` / `…MachineBatchWorkOrder` / `…MachineCycle (@ID, @DataKey, @DataValue)` | `JSON_MODIFY` into `OtherData` — via **dynamic SQL that concatenates `@DataKey`/`@DataValue`** into the statement (`sp_executesql` of a built string). Injection surface; callers today are trusted Galaxy scripts. |
+
+All timestamps are `GETUTCDATE()`; `Machine.TimeZone` is used only at retrieval time (`wwTimeZone=` in report wrappers). This matches the family's all-UTC rule.
+
+**Historian coupling (the read contract):** the CLR code resolves a machine-relative tag (`<Code>.<local>`; `*.<attr>` shorthand; `|literal|` static columns) to the Historian's global tagname via `Runtime.dbo.TagExtendedPropertyInfo` (`HierarchicalName`), then runs `SELECT … FROM OPENQUERY(INSQL, 'SELECT DateTime, [tag]… FROM WideHistory WHERE … wwTimeZone="UTC" AND <queryParams>')` (`wwRetrievalMode`, `wwResolution`, `wwTimeZone` are passed through **verbatim** from `@queryParams`). Discrete checks read `Runtime.dbo.DiscreteHistory`; alarm wrappers read `Runtime.dbo.Events` (`IsAlarm=1`, `Alarm_OriginationTime`, `Alarm_DurationMs`, `Alarm_UnAckDurationMs`). **Consequence: the `BT` database must live on (or three-part-name-reach) the Historian's SQL instance with the `INSQL` linked server** — it is co-tenant with `Runtime`, which is why it sits on the `wonder-sql-*` boxes.
+
+**Report wrappers** are per machine family and are the consumers of the CLR layer: Crest (`CrestWide`/`CrestMinMax`/`CrestAlarm`/`CrestCleaner*`), Elma (`ElmaData`, `Elma_Bulk_Query_Nitric`), `ExtruderWide`, `PultrusionWide`, Haas (`HaasStatusReport`, `HaasAlarm`, `QV_Haas*` = QlikView), Solar (`SolarAlarms`/`SolarEvents`), DryHeat (`DryHeatCycleReport`/`Search`/`BulkReport`), PoncePolyPress, `Z002432Wide`/`Z002433Wide`. `GenerateShifts` builds a 3-shift calendar table for these.
+
+**Auth tables that are not the Manager's:** `UserAuth`, `UserAuthDetails`, `UserAuthRole`, `ApiKey` are ServiceStack's schema — they belong to **MESAPI/WWSupport** (the Camstar-facing REST service, see `mes-delmia-integration-api.md` §1.1: `ApiKeyAuthProvider` + `LdapAuthProvider`, role `MESAPI`), which shares the `BT` database. The Manager authenticates via LDAP only.
+
+**Notification tables** (`Notification`, `NotificationLog`, audited by trigger) back the separate `WWNotificationSystem` Windows service, which is **not in this repo**.
+
+## 4. BTDBManager (the config/edit UI) — `master` state; the replacement is §8
+
+- **Auth:** cookie scheme; `LdapAuthenticationService` binds `LDAP://nam.zimmer.com` with the user's own credentials via `System.DirectoryServices` (so it only runs on a domain-joined Windows host), reads `memberOf`, and keeps only the three configured group DNs. Policies `READONLY` / `MACHINECONFIG` / `BATCHEDIT` map to `ReadOnlySecurityGroup` / `MachineConfigSecurityGroup` / `BatchEditSecurityGroup`; the checked-in DEV config points all three at `UG_WW_APP_ADMIN_DEV`. A global `AuthorizeFilter` makes every controller (incl. `/Api/GetMachines`, `/Api/GetMachineGroups`) require login. This is exactly the AD-group→role problem the family's `ZB.MOM.WW.Auth` 0.3.0 + `ldap_mapping.md` already solve.
+- **Features:** Machine CRUD (with group memberships + alarm catalog via TVPs `MachineGroupMembershipType`/`MachineAlarmType`), MachineGroup CRUD, batch search/edit (`SaveBatchEdit` writes `AuditLogBatchEdit` + prev/new `AuditLogMachineBatch`/`…WO`/`…Cycle` rows), **MachineConfigImport** (copies machine/group config from another environment's `BT` — sources `QA` and `PROD` are hard-wired in `appsettings.json`), and Telerik `.trdp` audit reports.
+- **Secrets are checked in:** `appsettings.json` carries plaintext `wonderapp` SQL passwords for DEV/QA/PROD, and `install.bat` carries a domain service-account password. Treat all of them as compromised-by-repo; do not copy them into any other file or reply. This is the pattern `ZB.MOM.WW.Secrets` (`${secret:}`) exists to replace.
+- **Environment hosts as recorded in 2023:** DEV `wonder-sql-vd01/BT`, QA `wondersqlqa/BT` (`MultiSubnetFailover`), PROD `wondersqlpd/BT`. ScadaBridge's 2026-06-16 design says the DEV BTDB is on **`wonder-sql-vd03/BT`** (db-connection id 1). `mes-delmia-integration-api.md` also says `vd01`. **Host drift is unresolved** — do not assume either without probing.
+
+## 5. How BTDB relates to the rest of the family
+
+```
+   Galaxy machine objects ──(script: EXEC StartMachineBatch / StartMachineCycle /
+   ($Reactor_Side etc.)      AddMachineBatchWorkOrder / AddOtherData* …)──────────┐
+                                                                                  ▼
+   Camstar MES ──HTTPS──▶ MESAPI/WWSupport ──(SELECT Machine by SAPID; alarm catalog;  ┌────────────────────┐
+                          (ServiceStack, own UserAuth/ApiKey tables) ────────────────▶ │  SQL Server "BT"   │
+                                                                                      │  (BTDB)            │
+   ScadaBridge inbound API ──(Database helper, connection "BTDB": SELECT Code FROM     │  co-tenant with    │
+   (IpsenMESMoveIn, MES alarm-status) ─────────────────────────────────────────────▶ │  Historian Runtime │
+   ScadaBridge site scripts ──(Database.Connection / CachedWrite — customer scripts)─▶ │  + INSQL linked    │
+                                                                                      └─────────┬──────────┘
+   BTDBManager (config, batch edits, env import) ─────────────────────────────────────▶         │ CLR: OPENQUERY(INSQL) WideHistory,
+   SSRS / QlikView / Telerik reports ──(EXEC *_Wrapper, GetMachineBatchTSV …)──────────▶         │ Runtime.dbo.{TagExtendedPropertyInfo,
+                                                                                                ▼ DiscreteHistory, Events}
+                                                                                  AVEVA Historian (Runtime DB)
+```
+
+- **Galaxy → BT** is the primary write path (batches/cycles/work orders/OtherData JSON). DARS `model_info.md` §: "Existing per-machine BT sprocs (`StartMachineBatch`, `StartMachineCycle`, `GetCycleTSV`, `Alarm_Wrapper`, …) work unchanged" — the sprocs are the contract Galaxy scripts bind to.
+- **ScadaBridge → BT (verified):** the inbound-API script context has a `Database` helper bound to named db-connections; production defines connection **`BTDB`** (id 1), and the `IpsenMESMoveIn` (design 2026-06-16) and MES alarm-status (2026-06-30) methods do `SELECT Code FROM dbo.Machine WHERE SAPID=@s` before `Route.To(instance)`. Template/site scripts expose `Database.Connection(name)` + `Database.CachedWrite(...)`, so site scripts **can** write batches into BT — but **no checked-in ScadaBridge script calls a BT sproc**; any batch logging is customer-site script configuration, not platform code, and the docker rig does **not** seed a `BT` database (`infra/mssql/setup.sql` creates only `ScadaBridgeConfig` + `ScadaBridgeMachineData`; the `InboundDatabaseHelperTests` use a stand-in). ScadaBridge's inbound API is the designated successor of MESAPI's move-in/move-out surface, which is why it needs the same `SAPID → Code` lookup.
+- **HistorianGateway ↔ BT:** no direct coupling, but both read `Runtime.dbo.Events` (HG's SQL-path `ReadEvents`) and both need the Historian SQL instance. BT's CLR `GetTSV`/`GetAggregateValues` is a bespoke, in-database equivalent of what HG's `ReadRaw`/aggregate retrieval modes do over gRPC — a candidate for consolidation if the report layer is ever re-platformed.
+- **OtOpcUa / mxaccessgw:** no coupling.
+- **Break surface:** the sproc signatures + `Machine.Code`/`SAPID`/`ZTag` semantics (Galaxy scripts, MESAPI, ScadaBridge inbound methods all bind to them), the `OtherData` JSON key conventions (DARS sprint 1 defines ~14 new keys with single writers), and the `INSQL`/`Runtime` co-location assumption.
+
+## 6. Analysis — gaps and recommendations (2026-09-03)
+
+1. **Push the branch.** ~~Put both copies under git~~ — done locally 2026-09-03 (`feat/machine-batch-transfer`). Remaining: push it to `ZimmerBiomet/BTDB`, review/merge, decide whether BTDB also gets a mirror on `gitea.dohertylan.com` like the other sisters, then delete the superseded `DARS/WW_ARCH/BatchTracker_DB/` copy.
+2. **Rotate and externalize the checked-in credentials** (three SQL passwords, one domain account). _Externalised on `feat/manager-blazor` (`${secret:btdb/...}` refs, store outside the content root); rotation is a cut-over step (§8)._ The family standard is `ZB.MOM.WW.Secrets` `${secret:}` with an absolute `Secrets:SqlitePath` outside the content root.
+3. **Manager re-platform** — _done on `feat/manager-blazor`, see §8_: net6.0 is EOL; Telerik is pinned by an absolute machine path; `System.DirectoryServices` binds the user's own credentials on a domain-joined box. A .NET 10 rewrite on the family libs is mostly assembly: Auth 0.3.0 (LDAP + `UG_WW_*` group→role mapping, `FallbackServers`), Theme 0.4.1, Health 0.3.0 (`MapZbHealth`, so Overview can watch it), Telemetry, Configuration, Audit (the batch-edit audit is a natural `ZB.MOM.WW.Audit` writer), Secrets. Keep Dapper + the embedded queries; replace Telerik `.trdp` with server-side rendering only if the three audit reports are still used.
+4. **Injection surfaces in the SQL layer:** `AddOtherData*` build dynamic SQL from `@DataKey`/`@DataValue`; `GetTSV`/`GetAggregateValues` interpolate tag names and append `@queryParams` raw into `OPENQUERY`. Trusted callers today, but ScadaBridge scripts and the inbound API widen the caller set. Fix `AddOtherData*` with parameterised `JSON_MODIFY` (no dynamic SQL needed), and whitelist `queryParams` to the three `ww*` keys.
+5. **ServiceStack auth tables** (`UserAuth*`, `ApiKey`) are MESAPI's, not BT's domain. Once ScadaBridge's inbound API fully replaces MESAPI (Ipsen MoveIn is the first slice; ScadaBridge keys already live in the `zbauth` schema via `Auth.ApiKeys.SqlServer` 0.3.0), those tables are dead weight in the schema and should be dropped from the sqlproj rather than carried forward.
+6. **Host inventory drift:** 2023 config says DEV BT is on `wonder-sql-vd01`; ScadaBridge's 2026 design says `wonder-sql-vd03`. Probe and record the answer in `deployment.md` §2.1 (which currently has no BT row at all).
+7. **`install.bat`/`uninstall.bat` are wrong for this project** (they install `WWNotificationSystem`). _Deleted on `feat/manager-blazor`; replaced by `deploy/install-service.bat` (NSSM, service `ZB.MOM.WW.BTDB.Manager`)._
+8. **CLR + Historian co-location is a hard deployment constraint** the index did not previously record: BT cannot be moved to the app-tier SQL (`wonder-app-vd03,1433`, where ScadaBridge's ConfigDb lives) without breaking every report wrapper. Any consolidation plan must keep BT on the Historian instance or replace the CLR with HistorianGateway calls.
+
+## 7. Primary commands (Windows only)
+
+```powershell
+# from ~/Desktop/btdb (master)
+dotnet build Manager\BTDBManager.csproj                 # net6.0; needs Telerik Reporting R2 2022 at the HintPath
+msbuild Database\Database.sqlproj /p:Configuration=Release   # SSDT (VS 2022 + SSDT) → bin\Release\Database.dacpac
+SqlPackage /Action:Publish /SourceFile:Database\bin\Release\Database.dacpac /TargetServerName:<historian-sql> /TargetDatabaseName:BT
+# Instance prerequisites: CLR enabled, `INSQL` linked server, `Runtime` DB reachable by three-part name
+```
+
+```bash
+# feat/manager-blazor (any OS for the app; SSDT still Windows-only)
+./build.sh                       # restore + build (Release, warnings-as-errors) + test BTDB.slnx
+dotnet run --project src/ZB.MOM.WW.BTDB.Manager   # appsettings.Development.json → local glauth; needs a BT
+pwsh deploy/publish.ps1          # self-contained win-x64 → deploy/out/ (+ NSSM install scripts)
+```
+
+```bash
+# Local Docker rig (2026-09-04, any OS) — see §9
+docker/deploy-db.sh              # BT on scadabridge-mssql from Database/ (dacpac + SQL CLR); --recreate drops it first
+docker/deploy.sh                 # publish FDD on the host → zb-btdb-manager:local, up on :9877 (glauth login enabled)
+```
+
+On `master` nothing builds on macOS (SSDT/CLR need Windows MSBuild, the MVC Manager needs Telerik + `System.DirectoryServices`). On the branch the app and its tests build and run on macOS; only `Database.sqlproj` (kept alone in `BTDB.sln`) still needs Windows — but §9's side projects build the same sources into a dacpac and a CLR assembly anywhere, for the local rig.
+
+## 8. Blazor conversion — branch `feat/manager-blazor` (2026-09-04)
+
+Plan: `btdb/docs/plans/manager_blazor_conversion.md` (+ `.tasks.json` tracker beside it; mockups `docs/mockups/manager_blazor_mockups.html`). Executed 2026-09-04 in worktree `~/.config/superpowers/worktrees/BTDB/feat-manager-blazor`; **uncommitted and unpushed at the time of writing** — phases 1–3 (scaffold, core, auth, widgets, queries, all pages) plus the local half of phase 4 are done; the DEV-box half of phase 4 (parity run, secrets seed + rotation, NSSM side-by-side, port swap) is written up as a runbook and not yet performed.
+
+- **Shape on the branch:** `BTDB.slnx` = `src/ZB.MOM.WW.BTDB.Manager` (Blazor Web App, InteractiveServer, net10.0) + `src/ZB.MOM.WW.BTDB.Manager.Core` (Dapper data layer moved verbatim from the MVC app, hardened) + two xunit/bUnit test projects (334 tests, warnings-as-errors gate via root `Directory.Build.props`, `build.sh`/`build.ps1`). `BTDB.sln` keeps only `Database.sqlproj`. `Manager/` (MVC, Kendo, the three Telerik `.trdp` reports with embedded connection strings, the mis-named `install.bat`) is **deleted**; `/Api/GetMachines`+`GetMachineGroups` are gone (no external consumer — ScadaBridge/MESAPI read `dbo.Machine` by SQL).
+- **Family libs:** Theme 0.4.1, Health 0.3.0 (`MapZbHealth`; readiness = a `SELECT 1` against `BT` with a 3 s timeout, so `/health/ready` is 503 until the DB answers — register it in Overview), Telemetry + Serilog 0.1.0, Configuration 0.1.0 (`ConfigPreflight` requires `ConnectionStrings:BT` and `Btdb:Security:Ldap:Server`), Secrets 0.6.2 (`${secret:btdb/connectionstrings/{bt,qa,prod}}`, `${secret:btdb/ldap/serviceaccountpassword}`; SQLite store `C:\ProgramData\ZB.MOM.WW\btdb-manager\secrets.db`, KEK from `ZB_SECRETS_MASTER_KEY`), Auth.Ldap/AspNetCore 0.3.0 (service-account bind, `Btdb:Security:GroupToRole` → `CanonicalRole`; policies Viewer / BatchEdit (Operator+) / MachineConfig (Administrator); cookie `btdb.auth`, plain-HTTP cookie because the box has no certificate). `Microsoft.Data.SqlClient` 7.x with `Encrypt=False` forced by `BtConnectionStrings.Normalize` unless the string sets it.
+- **Behavioural deltas vs the MVC app worth knowing:** config import derives the group set from the selected machines' memberships (no separate group pick) and has a dry-run preview (`MachineConfigImportPreview.sql`: same MERGEs, `OUTPUT` into a table variable, always `ROLLBACK`); batch-edit timestamps are 7-fraction UTC strings validated on blur (no Kendo mask); the three audit reports are HTML tables with a print stylesheet instead of Telerik PDFs; `MachineAlarm` DTO property order/types were wrong in the MVC app (alarm import always threw) and are fixed against `MachineAlarmType`; every `Save*` runs in a transaction and `THROW 5001x` maps to `BtdbSaveException`.
+- **Cut-over material:** `docs/operations/cut-over.md` (parity checklist per route + THROW paths + audit-username check, secrets seed via the `secret` console + rotation, NSSM side-by-side on `:9877`, port swap to `:9876`, Overview registry entry), `deploy/` (`publish.ps1`, `install-service.bat` / `uninstall-service.bat` for NSSM service `ZB.MOM.WW.BTDB.Manager`), `tools/smoke-new-queries.sql` (runs the six never-executed queries read-only against DEV with sqlcmd). Still open after cut-over: Overview lacks the mockup's per-machine "open > 24 h" rows; Ldap `Server`/`SearchBase`/`ServiceAccountDn` are blank in `appsettings.json` by design (per-environment).
+
+## 9. Local Docker rig — `docker/` (2026-09-04)
+
+Mirrors the HistorianGateway / Overview pattern: publish framework-dependent on the host (Gitea feed creds), copy into `aspnet:10.0`, never restore in Docker. Compose project `zb-btdb`, container `zb-btdb-manager`, image `zb-btdb-manager:local`, port `:9877` (same as the DEV side-by-side port, so `appsettings.json`'s Kestrel section is the single binding authority; the Dockerfile clears `ASPNETCORE_URLS`/`ASPNETCORE_HTTP_PORTS`). `ASPNETCORE_ENVIRONMENT=Production`; all environment-specific values are compose env vars, dev-only and committed under the family's rig exception.
+
+- **Database:** no MSSQL of its own — joins the external `scadabridge-net` and uses ScadaBridge's `scadabridge-mssql` (the instance that already carries `ScadaBridgeConfig` + `ZbSecretsHub`). `docker/deploy-db.sh`: (1) `docker/db/BT.Clr.csproj` compiles `Database/**/*.cs` to `Database.dll` (net472 via the auto-referenced reference assemblies — builds on macOS); (2) `docker/db/BT.Local.sqlproj` (SDK-style `Microsoft.Build.Sql/2.2.0`, `EnableDefaultSqlItems=false`, globs the four `Database/` folders, Sql130 model, suppresses the legacy build's own SQL71562/71558/71502) → `BT.Local.dacpac`; (3) `db/setup.sql` — `BT` + login `btdb_app` (db_owner); (4) `db/clr-objects.sql` — `clr enabled` on the instance, `sp_add_trusted_assembly` by SHA2-512 (strict security stays on), `CREATE ASSEMBLY [Database] … SAFE` from the bytes streamed as 2000-char `0x` chunks (a single 57 KB literal gets split by sqlcmd), then `StringJoin` aggregate, `StringCompare`/`CheckTriggered`/`WhenTriggered`/`WhenCleared`/`GetAggregateValues`, `GetTSV`/`GetMachineBatchTSV`/`GetCycleTSV`/`GetLinkedBatchTSV` with SSDT's type mapping; (5) `sqlpackage /Action:Publish` from a one-shot container (`db/Dockerfile.sqlpackage`, SDK image + the dotnet tool — the host's 10.0.5 runtime could not run the tool's 10.0.11 pin) with `DropObjectsNotInSource=false`, `BlockOnPossibleDataLoss=false`, `AllowIncompatiblePlatform=true`; (6) smoke. Result on the rig: 23 tables, 37 T-SQL procs + 1 TVF, 10 CLR objects, `Database` assembly `SAFE_ACCESS`. The script diffs the legacy `<Build Include>` list against the folders and warns on drift (found: `ElmaData_Nitric.sql` + `Elma_Bulk_Query_Nitric.sql` were on disk but not in `Database.sqlproj` — added). `ScadaBridge/infra/teardown.sh`/`reseed.sh` drop the volume and BT with it; re-run `deploy-db.sh`.
+- **Why not convert `Database.sqlproj` itself:** `Microsoft.Build.Sql` does not compile SQL CLR, and the Manager depends on `dbo.StringJoin` (batch search) and `dbo.StringCompare` (machine saves). The legacy SSDT project stays the artifact of record; the two side projects reference the same files, nothing is copied.
+- **Login:** enabled, `host.docker.internal:3893` + `FallbackServers` `:3894` (glauth-redundant), bind `cn=serviceaccount,dc=zb,dc=local`; `GroupToRole` `SCADA-Admins→Administrator`, `SCADA-Designers→Operator`, `SCADA-Viewers→Viewer`. Verified: `admin`/`designer` sign in, `gw-viewer` refused (fail closed). Literal `ConnectionStrings__BT` / LDAP password env vars override the `${secret:}` refs, so the `/data/btdb-secrets.db` store stays empty and no master key is set (confirmed to boot without one). The two config-import sources point at the local BT.
+- **Health / Overview:** `/healthz` 200, `/health/ready` 200 with the `sql` entry (503 when BT is down), `/health/active` 200, `/metrics`. Compose `healthcheck` = bash `/dev/tcp` GET of `/healthz` (liveness on purpose). Registered in `ZB.MOM.WW.Overview/src/…/appsettings.Docker.json` as application **BTDB Manager**, instance `manager`, `BaseUrl http://zb-btdb-manager:9877`, `HasActiveRole false`; the containerised Overview was republished and shows the tile Up with `sql` healthy (the dashboard already sits on `scadabridge-net`, so no extra network).
+
+
+## 10. State as of 2026-09-11 (supersedes §1, §4, §6.1, §6.3, §6.7, §8 where they differ)
+
+- **Everything is on local `master`** (`78195b8`): the Blazor conversion (§8) was merged, the
+  `feat/manager-blazor` worktree is gone, and the 2026-09-04 full review
+  (`docs/reviews/2026-09-04-full-review.md`, reviewed at `c624d41`) was remediated in 12 work-package
+  branches — `fix/save-batch-edit-null-compare`, `fix/db-lifecycle-sprocs`, `fix/clr-openquery-escaping`,
+  `fix/config-import-parents-and-nulls`, `test/db-integration`, `fix/manager-robustness`,
+  `fix/auth-hardening`, `ops/runbook-and-deploy`, `perf/indexes`, `db/schema-hygiene`,
+  `fix/concurrency-and-async`, `chore/nits` — all merged; 60 findings fixed, 3 partial by decision
+  (DB-15 `QV_*` impersonation, DB-16 alarm-CTE consolidation, DB-21 ServiceStack table drop), 0 open.
+  `feat/machine-batch-transfer` (DARS sprint 1, §1 item 1) is merged too, and so is
+  `feat/transfer-machine-batch` (merged `be976a6`, 2026-09-11): **`dbo.TransferMachineBatch`**, a tenth
+  Galaxy-facing writer that ends the source batch if still open, starts the destination batch by machine
+  `Code`, copies the `MachineBatchWO` rows still on the source (`MoveOutTimestamp IS NULL`, narrowed to
+  `@ContainerID`; blank = whole batch; source rows never modified) and records the `MachineBatchTransfer`
+  link — by composing `EndMachineBatch` / `StartMachineBatch` / `AddMachineBatchTransfer` in one owned
+  transaction. Outputs `@o_DestMachineBatchID`, `@o_MachineBatchTransferID`. Documented in
+  `docs/BatchTracker_Reference.md`; proven by `ValidateLifecycleSprocs.sql` §F (82 PASS on the rig).
+  `fix/transfer-lot-walk` (merged `5f0da01`, 2026-09-11) **reopened review DB-6**: the 2026-09-04 loop guard
+  in `AddMachineBatchTransfer` walked backward (toward the batches that fed the destination), which the
+  already-transferred check makes unreachable for any source, so `A→B` then `B→A` was still accepted; it
+  now walks forward along source→dest links with `HOLDLOCK`, and the destination row is held like the
+  source. The chain walks in `GetLinkedBatchTSV.cs` and `LinkedBatchAlarm_Wrapper.sql` picked one root
+  with `TOP 1` and dropped the other branch of a lot merge (`A→C, B→C`); both now resolve the whole lot
+  (forward to the final batch, then every batch that fed it) with one CTE pair pinned equal by
+  `SchemaHygieneTests`. Rig proofs: `ValidateLifecycleSprocs.sql` 87/0, `docker/db/validate-clr.sql`
+  27/0 over a merged lot with a `Runtime.dbo.Events` stand-in.
+  `feat/manager-transfers` (merged `695192d`, 2026-09-11) gave the **Manager** the transfer side it lacked: the
+  batch edit page edits the batch's outgoing `MachineBatchTransfer` row as a fourth MERGEd collection in
+  `SaveBatchEdit.sql` (guards `50015` self, `50016` destination missing, `50017` loop, `50018` more than one;
+  destination row and chain walk held like `AddMachineBatchTransfer`), audited into the new
+  `AuditLogMachineBatchTransfer` under the batch-edit header (new TVP `MachineBatchTransferType`); a read-only
+  lot panel (`GetBatchLot.sql`, the same `Down`/`Lot` CTE as the CLR, pinned equal by a static test); a lineage
+  column on Find Batch; a Transfers section on the batch-edit audit page and PDF. Rig proofs: 28 integration
+  tests (three new: round trip with audit, loop refused, merged lot), smoke 17/17, lifecycle 87/0, transfer
+  chains 4/0, validate-clr 27/0; gate 961 tests, 0 warnings. No branch is in flight.
+- **`master` is ~198 commits ahead of `origin/master` (`d147c10`) and unpushed.** Pushing is the user's
+  call — the remote is the corporate `ZimmerBiomet/BTDB`.
+- **Solution shape now:** `BTDB.slnx` = `src/ZB.MOM.WW.BTDB.Manager` (Blazor host) +
+  `src/ZB.MOM.WW.BTDB.Manager.Core` (Dapper data layer, embedded `Queries/*.sql`) + five test projects
+  (`Manager.Core.Tests` static query invariants, `Manager.Tests` bUnit/boot, `Database.Tests` static SQL
+  + sqlproj invariants, `Clr.Tests` compiles the CLR sources into net10, `Manager.Core.IntegrationTests`
+  opt-in on the rig behind `BTDB_TEST_CONNECTION`). `BTDB.sln` keeps `Database/Database.sqlproj`
+  (Windows MSBuild only). `.gitea/workflows/build.yml` runs `./build.sh` (needs `GITEA_NUGET_TOKEN`).
+- **Invariants a future session must not break** (all test-enforced): TVPs bind by ordinal so model
+  property order + lengths must equal `Database/Types/*.sql`; every write query is transaction-free and
+  audited (the C# wrapper owns the transaction); `dbo.X` only, never `BT.dbo.X`; `SYSUTCDATETIME()` in
+  Manager queries; every writer sproc follows the own-or-join `TRY/CATCH` template with locked lookups;
+  every `.sql` on disk must be in `Database.sqlproj`'s `<Build Include>` list; optimistic concurrency via
+  `ExpectedLastUpdate` → `THROW 50014`.
+- **Secrets/credentials (§6.2):** `${secret:btdb/connectionstrings/{bt,qa,prod}}` (the LDAP
+  service-account secret is gone since 2026-10-01 — see below; only `dotnet run` against local glauth uses it), store at `C:\ProgramData\ZB.MOM.WW\btdb-manager\secrets.db`,
+  master key only in `ZB_SECRETS_MASTER_KEY`. Rotation is cut-over step 4.2.
+- **Claude artifacts are gitignored + untracked in btdb** (2026-09-11): `CLAUDE.md`, `.claude/`,
+  `docs/plans/`, `docs/reviews/`, `*.tasks.json` — they exist on disk (btdb's `CLAUDE.md` is the repo
+  orientation and still loads) but never reach the corporate remote. `docs/operations/`,
+  `docs/BatchTracker_Reference.md`, `docs/mockups/` stay tracked.
+- **Open step:** phase-4 cut-over on the DEV box (`docs/operations/cut-over.md`; 4.1 parity walk-through
+  gates the rest) — **blocked by the 2026-09-11 remote-access hold** (no psbridge, no winrm; UNC copies and
+  direct SQL logins still allowed). §6 item 6 (which SQL host carries DEV `BT`) is still unrecorded.
+- **2026-10-01 — LDAP sign-in by user bind** (merged `a81a480`): Auth 0.4.0/0.4.1 added opt-in
+  `LdapOptions.BindMode=User`; BTDB ships it. Sign-in binds as `UserBindTemplate` (`<DOMAIN>\{0}` —
+  down-level maps exactly to the sAMAccountName the group search uses), the same connection reads the
+  user's own `memberOf`/`displayName`, and the username returned is the directory's spelling. No service
+  account, no LDAP secret; the preflight requires the template and refuses `AllowInsecure`/`Transport=None`
+  with user bind outside Development. Local dev + docker rig stay `BindMode=ServiceAccount` (GLAuth bind DNs
+  embed the user's OU). Rollout to the three Manager nodes (vd03 first) and retiring the stored secret are
+  the open steps (plan `docs/plans/2026-10-01-ldap-user-bind.md`, gitignored).

.claude/skills/sister-project-history/scadabridge.md

@@ -0,0 +1,5 @@
+# ScadaBridge — index history
+
+- **2026-10-13 existing fixes verified and issues closed.** Main `513fa1aa` already includes #35 fail-closed unavailable configured body/SQL redactors (`006202f3`) and #37 Flow Debugger structured/raw validated method inputs (`b5eb08f1`). Fresh full AuditLog project 407 passed/0 failed; FlowDebuggerPageTests 47 passed/0 failed; both terminal exit0. Remote ancestry confirmed; #35/#37 closed with evidence. No source changes/deployment/MES transactions. #36 runtime reset-echo overload is implemented, but QA/prod script port remains tracked; #30 per-site PSK deployment gate remains open.
+
+- **2026-10-13 #36 runtime regression proof refreshed.** Rebuilt HandshakeResetEchoTests passed all three cases with local fakes, terminal exit0; an earlier no-build attempt matched no tests and is excluded from evidence. Tracker now distinguishes implemented runtime b623fdb0 from remaining QA/prod database-owned script rollout (deployment register row119, scripts13/14/36 and15/16/25/26/37/38). No database writes, deployments or fresh live rig claims.

.gitignore

@@ -7,8 +7,10 @@ obj/
 [Aa]rtifacts/
 *.user
 
-# Claude Code tooling / agent worktrees (local-only)
-.claude/
+# Claude Code tooling / agent worktrees (local-only) — but .claude/skills/ IS tracked:
+# the sister-project-history skill holds the per-project narratives the root index points at.
+.claude/*
+!.claude/skills/
 
 # Local ops/infra notes — internal access details, never publish
 zimmer-vpn-tunnel.md

ZB.MOM.WW.Audit/CLAUDE.md

@@ -0,0 +1,37 @@
+# ZB.MOM.WW.Audit
+
+Shared audit event model + writer seam for the **ZB.MOM.WW SCADA family** (single package; DEEP-adopted by all apps as the canonical record). Build/test from this directory: `dotnet test`. Component spec + gaps: `../components/audit/`.
+
+## Index history (migrated from the scadaproj root CLAUDE.md)
+
+> Moved verbatim out of the scadaproj root `CLAUDE.md` on 2026-09-03 so it loads only when needed. **This file is now the authoritative place for this history — append new entries here and keep the root index row short.**
+
+### Narrative
+
+The audit component is fully populated: a normalized [`spec`](components/audit/spec/SPEC.md), an
+[`event-model`](components/audit/spec/EVENT-MODEL.md) reference, a
+[`shared-contract`](components/audit/shared-contract/ZB.MOM.WW.Audit.md), three
+[`current-state`](components/audit/current-state/) docs, and an adoption [`GAPS`](components/audit/GAPS.md)
+backlog. Common ground = canonical `AuditEvent` record + `AuditOutcome` enum + `IAuditWriter` /
+`IAuditRedactor` seams + helpers (`NullAuditRedactor`, `TruncatingAuditRedactor`, `NoOpAuditWriter`,
+`CompositeAuditWriter`, `RedactingAuditWriter`) + `AddZbAudit` DI registration; left per-project =
+transport/storage and domain vocabulary. Closes the loop on Auth — audit's `Actor` field = the Auth
+principal. `IAuditRedactor` is aligned with Telemetry's `ILogRedactor` seam convention.
+
+The shared library is **built and lives in this repo** at [`ZB.MOM.WW.Audit/`](ZB.MOM.WW.Audit/)
+(.NET 10; 1 package — `ZB.MOM.WW.Audit`; only non-BCL dependency `Microsoft.Extensions.DependencyInjection.Abstractions`;
+23 tests (verified 2026-07-18); `dotnet pack` → 1 nupkg @ 0.1.0). Repo: `https://gitea.dohertylan.com/dohertj2/zb-mom-ww-audit`.
+**Adopted across all three apps on 2026-06-02** (audit GAPS #1–#6) on each repo's `feat/adopt-zb-audit` branch
+(stacked on `feat/adopt-zb-auth`) — committed + reviewed, then **merged into the repo's local default (main/master)
+and PUSHED to origin (gitea) on 2026-06-03** (in sync). Depth =
+**DEEP adopt** (the canonical 9-field `AuditEvent` is the record everywhere; domain fields ride in `DetailsJson`).
+OtOpcUa: canonical record + `AuditWriterActor : IAuditWriter` + `Outcome` column/migration + `ClusterAudit` fix.
+MxGateway: new canonical SQLite `audit_event` store + `IAuditWriter` + `IApiKeyAuditStore`→canonical adapter.
+**ScadaBridge: a full audit-subsystem re-architecture** (the program's largest task) — canonical record everywhere via a
+deterministic codec; site SQLite split into `audit_event` + an `audit_forward_state` forwarding sidecar; central
+partitioned `dbo.AuditLog` collapsed to 10 canonical cols + persisted computed cols (`CollapseAuditLogToCanonical`
+migration, MSSQL-verified). Phase 3 wires `Actor` from the Auth principal at authenticated emit sites (per-app
+`IAuditActorAccessor`). Per-repo detail in [`components/audit/GAPS.md`](components/audit/GAPS.md) +
+`docs/plans/2026-06-02-auth-audit-normalization-phase2-deep.md` + `…-scadabridge-audit-rearch.md`.
+Build/test from `ZB.MOM.WW.Audit/`: `dotnet test`. Consumer matrix: all three apps consume the single
+`ZB.MOM.WW.Audit` package (OtOpcUa, MxAccessGateway, ScadaBridge — DEEP-adopted as the canonical record).

ZB.MOM.WW.Audit/build/pack.sh

@@ -1,4 +1,10 @@
 #!/usr/bin/env bash
 # pack.sh — produce the ZB.MOM.WW.Audit NuGet package into ./artifacts.
 set -euo pipefail
-dotnet pack -c Release -o ./artifacts
+# cd to the repo root and name the solution + an absolute output path: `dotnet pack` with
+# neither a project argument nor a fixed working directory packs whatever solution happens
+# to sit in the CALLER's directory (the bug behind the LocalDb feed incident, 0038d46).
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+cd "$REPO_ROOT"
+
+dotnet pack ZB.MOM.WW.Audit.slnx -c Release -o "$REPO_ROOT/artifacts"

ZB.MOM.WW.Audit/build/push.sh

@@ -16,9 +16,20 @@ set -euo pipefail
 : "${GITEA_NUGET_SOURCE:?set GITEA_NUGET_SOURCE to your Gitea NuGet feed URL}"
 : "${GITEA_NUGET_KEY:?set GITEA_NUGET_KEY to your Gitea access token}"
 
-dotnet pack -c Release -o ./artifacts
+# Named solution, named output, and a cd to the repo root, all for one reason: `dotnet pack`
+# with neither a project argument nor a fixed working directory packs whatever solution
+# happens to sit in the CALLER's directory — and then pushes it to a live feed under this
+# script's name. Invoked by absolute path from another repo's checkout, the relative form
+# packed and published 25 unrelated packages (LocalDb's copy of this script, fixed in
+# 0038d46; recovered with `dotnet nuget delete`, but only because the token happened to
+# carry delete rights). A publish script must be unambiguous about WHAT it publishes no
+# matter where it is run.
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+cd "$REPO_ROOT"
 
-dotnet nuget push "./artifacts/*.nupkg" \
+dotnet pack ZB.MOM.WW.Audit.slnx -c Release -o "$REPO_ROOT/artifacts"
+
+dotnet nuget push "$REPO_ROOT/artifacts/*.nupkg" \
   --source "$GITEA_NUGET_SOURCE" \
   --api-key "$GITEA_NUGET_KEY" \
   --skip-duplicate

ZB.MOM.WW.Auth/CLAUDE.md

@@ -0,0 +1,41 @@
+# ZB.MOM.WW.Auth
+
+Shared login / identity / API-key library for the **ZB.MOM.WW SCADA family** (4 packages + `ApiKeys.SqlServer`; consumed by OtOpcUa, ScadaBridge, MxAccessGateway, HistorianGateway). Build/test from this directory: `dotnet test` (the `ApiKeys.SqlServer` live suite is gated on `AUTH_SQLSERVER_CONNSTR`). Component spec + gaps: `../components/auth/`.
+
+## Index history (migrated from the scadaproj root CLAUDE.md)
+
+> Moved verbatim out of the scadaproj root `CLAUDE.md` on 2026-09-03 so it loads only when needed. **This file is now the authoritative place for this history — append new entries here and keep the root index row short.**
+
+### Component-table row as it read in the index
+
+| Component | Status | Goal | Design | Implementation |
+|---|---|---|---|---|
+| Auth (login / identity / authz) | Adopted (**feed carries `0.3.0` as of 2026-09-02 — five packages, the new one being `ZB.MOM.WW.Auth.ApiKeys.SqlServer`; ALL FOUR apps pin `0.3.0` as of 2026-09-03**: OtOpcUa `master @ b8b4c18f`, ScadaBridge `main @ e45c4af9` (the `inbound-api-key-shared-store` work, merged), MxAccessGateway `main @ 4f931e4`, HistorianGateway `main @ d8d890c` — each built + tested with no new failures versus its pre-bump baseline, all PUSHED to origin. The preceding `0.2.1` pins landed 2026-08-13 (OtOpcUa `08e33fc7`, ScadaBridge `d28824d9`, MxAccessGateway `1cb14d2`, HistorianGateway `e42ff31`). **`0.2.0` (2026-08-10) adds LDAP backup-DC sticky failover**, **`0.2.1` (2026-08-12) fixes a defect that broke AD logins outright** — an AD continuation referral escaped `NovellLdapConnection.Search`, so any `SearchBase` at a domain root failed every login as `ServiceAccountBindFailed` against a perfectly healthy directory. Live-proven against `DC=nam,DC=zimmer,DC=com`; the app-level exposure is closed by the pin bumps. **`FallbackServers` is live-gated** via the local redundant GLAuth pair ([`infra/glauth-redundant/`](infra/glauth-redundant/), primary :3893 + backup :3894 on the dev Mac): HistorianGateway dashboard drill PASS 2026-08-13 (failover, stickiness, walk-back). | Shared `ZB.MOM.WW.Auth` lib | [`components/auth/`](components/auth/) | [`ZB.MOM.WW.Auth/`](ZB.MOM.WW.Auth/) |
+
+### Narrative
+
+The auth component is fully populated: a normalized [`spec`](components/auth/spec/SPEC.md), a
+proposed [`shared-contract`](components/auth/shared-contract/ZB.MOM.WW.Auth.md), three
+[`current-state`](components/auth/current-state/) docs, and an adoption [`GAPS`](components/auth/GAPS.md)
+backlog. Common ground = LDAP/GLAuth identity + peppered-HMAC API keys; left per-project =
+the authz vocabularies (OPC-UA permissions / gRPC scopes / roles + site-scoping).
+
+The shared library is **built and lives in this repo** at [`ZB.MOM.WW.Auth/`](ZB.MOM.WW.Auth/)
+(a plain directory of this repo — no nested `.git`, verified 2026-08-07; .NET 10; 4 packages — `Abstractions`, `Ldap`, `ApiKeys`, `AspNetCore`;
+250 tests passing + 2 skipped (verified 2026-08-10 at `0.2.0`; was 215+1 at `0.1.5`); feed carries `0.1.0`–`0.2.0`). The implementation plan is at
+[`docs/plans/2026-06-01-zb-mom-ww-auth-shared-library.md`](docs/plans/2026-06-01-zb-mom-ww-auth-shared-library.md).
+**Adopted across all three apps on 2026-06-02** (auth GAPS #1–#8) on each repo's `feat/adopt-zb-auth` branch —
+committed + reviewed, then **fast-forward-merged into the repo's local default (main/master) and PUSHED to origin
+(gitea) on 2026-06-03** (in sync; the `feat/*` branches kept locally as history). Cutover: shared `Auth.Ldap`,
+`Auth.ApiKeys` (ScadaBridge inbound fully re-architected to the keyId/Bearer model), `IGroupRoleMapper<TRole>` seam,
+`Transport`-enum config, canonical `ZbClaimTypes`/`ZbCookieDefaults`, unified dev base DN `dc=zb,dc=local`, and the
+canonical-six role vocabulary (with ScadaBridge's accepted auditor/admin SoD collapse). Consumer pins: **all four apps pin `0.3.0` as of 2026-09-03** (commits listed in the table row above, all pushed to origin); all four pinned `0.2.1` from 2026-08-13 before that. Prior history — all four pinned `0.1.5` (verified 2026-08-07): MxGateway via `e107019` 2026-07-18 (earlier bumped `0.1.2`→`0.1.4` 2026-07-09 for the G-2 API-key-expiry verifier enforcement — commit `197731a`; also required implementing the 0.1.3 `IApiKeyAdminStore.SetScopes/SetEnabled` members on a test fake), ScadaBridge `50d79ed1` + HistorianGateway `df211ee` 2026-07-18, OtOpcUa `1adaa2fc` 2026-07-20 (PR #484, family alignment from `0.1.1`). Per-repo detail in [`components/auth/GAPS.md`](components/auth/GAPS.md) +
+`docs/plans/2026-06-02-auth-audit-normalization*.md`.
+Build/test from `ZB.MOM.WW.Auth/`: `dotnet test` (the `ApiKeys.SqlServer` live suite is gated on `AUTH_SQLSERVER_CONNSTR`, skipped otherwise). Consumer matrix: OtOpcUa → Abstractions+Ldap+AspNetCore;
+MxAccessGateway & HistorianGateway → the original four; ScadaBridge → those four **plus `ApiKeys.SqlServer`** (ApiKeys not used by OtOpcUa).
+**`0.1.4` (2026-07-09, archreview G-2):** optional `ApiKeyRecord.ExpiresUtc` (nullable = never-expires), `ApiKeyFailure.KeyExpired`, `ApiKeyVerifier` expiry enforcement (inclusive, injected clock), a `CreateKeyAsync(expiresUtc)` overload, and a SQLite **schema v3** additive `expires_utc` column (idempotent `ALTER`; donor v2 DBs upgrade in place, no key invalidated). **All four Auth pkgs published to the Gitea feed at `0.1.4`** (2026-07-09). **HistorianGateway adopted the expiry surface at `0.1.4`** — `apikey create --expires` + dashboard Expires column (Task-10 done). **`0.1.5` (2026-07-18)** is `0.1.4` plus the `SQLitePCLRaw.lib.e_sqlite3 2.1.12` transitive security pin (GHSA-2m69-gcr7-jv3q), no API change; all four apps now pin it (see consumer pins above). **`0.2.0` (2026-08-10) — LDAP backup-DC sticky failover** (plan `docs/plans/2026-08-10-ldap-backup-dc-failover.md`, design `…-design.md`; **4 pkgs published to the feed, restore-verified**; family `Directory.Build.props` now `0.2.0`): one additive `LdapOptions.FallbackServers` list (`"host"` or `"host:port"`, empty default = failover off, single-endpoint behaviour byte-unchanged) makes `LdapAuthService` walk an ordered endpoint list (primary + fallbacks) on **system-side failures only** — connect/TLS, service-account bind, or search. A user-side answer from a healthy directory (`Success`/`UserNotFound`/`AmbiguousUser`/`BadCredentials`/`GroupLookupFailed`) is **final and never retried elsewhere** (re-asking a second DC about a rejected password would multiply one bad-password event into several against the account lockout policy). Endpoint preference is **sticky** — the last endpoint that gave *any* authoritative answer (success **or** a user-side failure) is tried first next time — so a primary outage costs its connect timeout once, not per login; an exhausted walk leaves the preference untouched. Fail-closed contract preserved per-endpoint; boot validation rejects a malformed `FallbackServers` entry naming the field; optional `ILogger<LdapAuthService>` emits a Warning per failed hop (endpoint + stage, no secrets) and an Information on sticky change; wired via `AddZbLdapAuth` (`GetService`, so logging is optional). Motivation: the ZB production directory is NAM AD with two DCs. Unit suite covers the walk/stickiness (14 tests) + env-gated live GLAuth failover test (`ZB_LDAP_IT=1`). **No `Abstractions` contract change beyond the new option; no `LdapAuthFailure` enum change.** App adoption (pin bump to `0.2.0` + authoring `FallbackServers`) is a follow-up.
+**`0.2.1` (2026-08-12) — AD continuation referrals no longer fail the search** (**4 pkgs published to the feed, restore-verified**; the assembly restored *from the feed* carries the fix and `0.2.0` does not — checked as a positive control, not by trusting the push). Active Directory answers **any search based at a domain root** with continuation references (to `CN=Configuration`, `DC=ForestDnsZones`, …). With referral following disabled — the default, and what we want, since chasing one would re-bind somewhere we never authenticated — Novell surfaces each by throwing `LdapReferralException` from `Next()`. That escaped `NovellLdapConnection.Search`, and `LdapAuthService` maps a failed search to `ServiceAccountBindFailed`, so **every login against `SearchBase = DC=nam,DC=zimmer,DC=com` failed system-side on both DCs while the directory, the credentials and the service account were all healthy** — ~546 ms, identical on each endpoint, so failover could not help either. Found by the new `tools/LdapProbe` (below) and live-proven fixed against the ZB NAM domain root. The search loop now skips referrals (`ReadEntries(ILdapSearchResults)`, extracted so it is unit-testable through the Novell seam without a live directory) and continues; a 1000-referral cap guards a client that throws without advancing, since that would spin forever inside a fail-closed auth path. **Non-referral faults still propagate** — swallowing those would turn "the directory is broken" into "no such user", which is the wrong answer to give about someone's credentials. 7 new tests (`NovellSearchReferralTests`), **6 verified red against the pre-fix code**, the seventh being the non-referral control that must stay green; suite 257 pass / 2 skip / 0 fail. **No `Abstractions` change; no config change.** **Consumer impact: an app on `0.1.5` cannot authenticate against an AD domain root at all**, so the pin bump is a functional fix, not housekeeping. **Closed 2026-08-13: all four apps adopted `0.2.1`** (see the table row for commits), and `FallbackServers` was live-gated the same day against the redundant local GLAuth pair (`infra/glauth-redundant/`).
+
+**`0.3.0` (2026-09-02) — `ZB.MOM.WW.Auth.ApiKeys.SqlServer`, a SQL Server provider for the API-key store** (design `ScadaBridge/docs/plans/2026-09-02-inbound-api-key-shared-store-design.md`; **5 pkgs published to the feed**, `ApiKeys.SqlServer` search-verified from the ScadaBridge checkout; family `Directory.Build.props` now `0.3.0`). Motivation: `Auth.ApiKeys`' SQLite store is a per-node file, so on a two-node active/standby host (ScadaBridge central) a key minted on one node exists only there and a failover 401s every inbound caller (ScadaBridge deferred-work row 108). The new package implements `IApiKeyStore`/`IApiKeyAdminStore`/`IApiKeyAuditStore` over `Microsoft.Data.SqlClient` in a `zbauth` schema (v1: `schema_version`, `api_keys`, `api_key_audit`; all timestamps `DATETIMEOFFSET(7)`, tick-exact) with a concurrent-boot-safe migrator (Serializable tx, retry on 1205/2714/1913/2627 with **jittered** backoff — the Secrets replicator's flat `100*attempt` backoff exhausted five retries about once per sixty 8-way boots in the port's soak test, so the same latent defect is an open follow-up there) and a `SqliteApiKeyStoreImporter` that copies an existing SQLite file into the schema insert-where-absent (idempotent, so both nodes of a pair can import their own file and the result is the union). Wiring: `services.AddZbApiKeyAuthSqlServer(config, apiKeySection, sqlServerSection)` registers the SQL Server stores + migrator **before** delegating to `AddZbApiKeyAuth`, whose `TryAdd*` registrations then yield (same displacement mechanism as Secrets 0.5.0). One `Abstractions` addition, additive: `IApiKeyStoreMigrator`, which `SqliteAuthStoreMigrator` now implements and `ApiKeyAdminCommands` accepts via a new ctor (old `SqliteAuthStoreMigrator` ctor retained — the three `0.2.1` consumers compile unchanged). Store semantics are pinned to the SQLite contract tests re-run live (mark-used skips revoked keys, delete only removes revoked keys, rotate clears last-used/revoked, list newest-first). 92 tests in the new project, 351 across the solution, 0 failed. A DBA-run copy of the DDL ships at `docs/sql/zbauth-schema.sql`.
+
+**`tools/LdapProbe`** (`ldapprobe`, not packable, outside the Auth solution) is the diagnostic that found the above: it drives the **real** `LdapAuthService` so its verdict is what the four apps produce, then separately queries the directory for what the library cannot report — full group DNs, **nested** groups via `LDAP_MATCHING_RULE_IN_CHAIN`, the **primary group** (which AD omits from `memberOf`, so it can never be a role group), group type (security vs distribution), decoded `userAccountControl`, referral counts, and the TLS chain with key sizes. `--list-groups '<pat>'` enumerates every group in the directory matching a pattern, which a membership dump structurally cannot do. Findings recorded in [`ldap_setup.md`](ldap_setup.md), including the **`UG_WW_*` Wonderware group convention (39 groups)** that partly overturns that doc's earlier "no group→role convention exists" claim — it was true of Ignition, false of the domain.

ZB.MOM.WW.Auth/Directory.Build.props

@@ -5,7 +5,12 @@
     <Nullable>enable</Nullable>
     <ImplicitUsings>enable</ImplicitUsings>
     <LangVersion>latest</LangVersion>
-    <Version>0.1.5</Version>
+    <!-- 0.4.0: opt-in LdapOptions.BindMode = User — authenticate by binding as the signing-in user
+         (UserBindTemplate) and read their own groups on the same connection; no service account.
+         ServiceAccount stays the default, so existing consumers need no change. -->
+    <!-- 0.4.1: Username canonicalised from the entry's UserNameAttribute; empty passwords refused
+         in both bind modes (ServiceAccount-mode behaviour change); template docs prefer EXAMPLE\{0}. -->
+    <Version>0.4.1</Version>
     <ManagePackageVersionsCentrally>true</ManagePackageVersionsCentrally>
   </PropertyGroup>
 

ZB.MOM.WW.Auth/Directory.Packages.props

@@ -15,6 +15,14 @@
     <!-- Transitive security pin: Microsoft.Data.Sqlite 10.0.7 pulls SQLitePCLRaw.lib.e_sqlite3 2.1.11,
          which carries advisory GHSA-2m69-gcr7-jv3q (NU1903). Pin the patched 2.1.12 native lib. -->
     <PackageVersion Include="SQLitePCLRaw.lib.e_sqlite3" Version="2.1.12" />
+    <!-- Shared SQL-Server API-key store (ZB.MOM.WW.Auth.ApiKeys.SqlServer). 6.0.2 matches the
+         family pin used by ZB.MOM.WW.Secrets' SQL-Server store, so a consumer taking both gets one
+         SqlClient rather than a downgrade warning. -->
+    <PackageVersion Include="Microsoft.Data.SqlClient" Version="6.0.2" />
+    <!-- 9.0.4, not 9.0.0: Microsoft.Data.SqlClient 6.0.2 pulls System.Configuration.ConfigurationManager
+         9.0.4, which requires ProtectedData >= 9.0.4. With transitive pinning on, the lower central
+         version is a hard NU1109 downgrade error rather than a silent resolve. -->
+    <PackageVersion Include="System.Security.Cryptography.ProtectedData" Version="9.0.4" />
 
     <!-- Extensions -->
     <PackageVersion Include="Microsoft.Extensions.Options" Version="10.0.7" />
@@ -24,6 +32,7 @@
     <PackageVersion Include="Microsoft.Extensions.Configuration.Abstractions" Version="10.0.7" />
     <PackageVersion Include="Microsoft.Extensions.Configuration" Version="10.0.7" />
     <PackageVersion Include="Microsoft.Extensions.Hosting.Abstractions" Version="10.0.7" />
+    <PackageVersion Include="Microsoft.Extensions.Logging.Abstractions" Version="10.0.7" />
 
     <!-- ASP.NET Core Authentication / Authorization -->
     <PackageVersion Include="Microsoft.AspNetCore.Authentication.JwtBearer" Version="10.0.7" />

ZB.MOM.WW.Auth/README.md

@@ -11,17 +11,18 @@ Authentication and authorisation libraries for the **ZB.MOM.WW SCADA family** (O
 | `ZB.MOM.WW.Auth.Abstractions` | Auth contracts, canonical role constants, and shared types (`LdapOptions`, `LdapAuthResult`, `ILdapAuthService`, `IApiKeyStore`). No runtime dependencies beyond the BCL. | — |
 | `ZB.MOM.WW.Auth.Ldap` | LDAP authentication service: bind-then-search-then-bind against GLAuth or Active Directory; RFC 4514-aware group extraction; fail-closed. | `Abstractions`, `Novell.Directory.Ldap.NETStandard` |
 | `ZB.MOM.WW.Auth.ApiKeys` | SQLite-backed API-key store with **pepper-keyed HMAC-SHA256** secret hashing, rotation, and audit log. DI wiring is `AddZbApiKeyAuth`; an opt-in `MigrationHostedService` runs schema migrations on startup. | `Abstractions`, `Microsoft.Data.Sqlite` |
+| `ZB.MOM.WW.Auth.ApiKeys.SqlServer` | SQL Server-backed API-key store sharing **one row set** across clustered hosts (e.g. a central pair), instead of each node keeping its own SQLite file. DI wiring is `AddZbApiKeyAuthSqlServer(config, apiKeySection, sqlServerSection)`, called **instead of** `AddZbApiKeyAuth` (it calls that internally). Also ships `SqliteApiKeyStoreImporter` — a one-shot per-node SQLite → shared store copy a host runs once at boot. | `Abstractions`, `ApiKeys`, `Microsoft.Data.SqlClient` |
 | `ZB.MOM.WW.Auth.AspNetCore` | ASP.NET Core wiring for the **LDAP** provider only: `AddZbLdapAuth` (binds + start-time-validates `LdapOptions`, registers `ILdapAuthService`), plus `ZbCookieDefaults.Apply` (hardened cookie helper the consumer calls itself) and `ZbClaimTypes` constants. It does **not** wire API keys or cookie middleware — API-key DI is `AddZbApiKeyAuth` in the `ApiKeys` package. | `Abstractions`, `Ldap`, `Microsoft.AspNetCore.App` |
 
 ---
 
 ## Consumer Matrix
 
-| Consumer | Abstractions | Ldap | ApiKeys | AspNetCore |
-|---|:---:|:---:|:---:|:---:|
-| **OtOpcUa** | yes | yes | — | yes |
-| **MxAccessGateway** | yes | yes | yes | yes |
-| **ScadaBridge** | yes | yes | yes | yes |
+| Consumer | Abstractions | Ldap | ApiKeys | ApiKeys.SqlServer | AspNetCore |
+|---|:---:|:---:|:---:|:---:|:---:|
+| **OtOpcUa** | yes | yes | — | — | yes |
+| **MxAccessGateway** | yes | yes | yes | — | yes |
+| **ScadaBridge** | yes | yes | yes | yes (central pair, 2026-09-02) | yes |
 
 `ApiKeys` is NOT used by OtOpcUa (that app authenticates human operators via LDAP + cookies only; machine-to-machine access is out of scope).
 
@@ -29,7 +30,41 @@ Authentication and authorisation libraries for the **ZB.MOM.WW SCADA family** (O
 
 ## Versioning
 
-All four packages are versioned **lockstep** from `Directory.Build.props`. The current release is **0.1.4**. A single version bump in `Directory.Build.props` bumps all four packages simultaneously — consumers should reference the same version for all ZB.MOM.WW.Auth packages.
+All five packages are versioned **lockstep** from `Directory.Build.props`. The current release is **0.4.1**. A single version bump in `Directory.Build.props` bumps all five packages simultaneously — consumers should reference the same version for all ZB.MOM.WW.Auth packages.
+
+### 0.4.1 — LDAP review fixes
+
+- **Canonical username (both modes):** the search now also requests `UserNameAttribute`, and `LdapAuthResult.Username` is the entry's first value of it (the directory's spelling), so `JDoe` and `jdoe` sign in as one identity in consumers' audit columns. Falls back to the normalised typed name when the entry has no such value.
+- **Behaviour change for `ServiceAccount`-mode consumers:** an empty (or null) password is now refused by the library as `BadCredentials` before any network call, in both bind modes. Previously only `User` mode refused it; in `ServiceAccount` mode the re-bind as the user's DN with an empty password is an unauthenticated bind that AD reports as success, so a consumer without its own edge check admitted anyone.
+- **Template guidance:** prefer the down-level `UserBindTemplate` `EXAMPLE\{0}` — it maps exactly to the `sAMAccountName` the group search uses. A UPN template (`{0}@example.com`) is only correct where every account's UPN prefix equals its `sAMAccountName`; AD resolves UPN binds against the explicit `userPrincipalName`, so a mismatch can fail sign-in or bind a different account than the one whose groups are read.
+- No `Abstractions` contract change.
+
+### 0.4.0 — LDAP sign-in by user bind (opt-in)
+
+- **`LdapOptions.BindMode`** (`ServiceAccount` default | `User`), **`UserBindTemplate`** (`{0}` = the login name, e.g. `EXAMPLE\{0}` — preferred, see 0.4.1 — or `{0}@example.com`) and **`AcceptDomainQualifiedNames`** (default `true`).
+- In `User` mode `LdapAuthService` binds as the user; success authenticates them and the same connection reads their entry (`UserNameAttribute`, `DisplayNameAttribute`, `GroupAttribute`). No `ServiceAccountDn` / `ServiceAccountPassword` is needed or validated.
+- Failure mapping: empty password or an unusable name → `BadCredentials` with no network call (AD treats an empty-password simple bind as an unauthenticated success). Bind result 49 → `BadCredentials`, final — never retried on a fallback DC, so one wrong password is one lockout strike. Any other bind error, a connect failure or a search failure → the system-side bucket (`ServiceAccountBindFailed`) and the next endpoint. Zero/several entries or no groups after a good bind → `GroupLookupFailed`.
+- `DOMAIN\name` / `name@suffix` are reduced to `name` (or refused when `AcceptDomainQualifiedNames` is `false`), so the result's `Username` has one spelling (0.4.1 also canonicalises its case from the entry).
+- No `Abstractions` contract change beyond the new enum and option properties; `ServiceAccount` mode is unchanged.
+- `UserBindTemplate` output is used verbatim as the bind name, so use a down-level (`EXAMPLE\{0}`, preferred) or UPN (`{0}@example.com`, only where every UPN prefix equals the `sAMAccountName`) template; a DN template is not escaped.
+
+### 0.3.0 — SQL Server API-key store (ScadaBridge central pair)
+
+- **`IApiKeyStoreMigrator`** added to `Abstractions`: creates or upgrades an API-key store's schema; idempotent and safe to run concurrently from several nodes. `SqliteAuthStoreMigrator` implements it unchanged. `AddZbApiKeyAuth` registers the SQLite implementation via `TryAdd`, so a provider that registers its own migrator first (see below) wins — that ordering is the whole mechanism by which a different backend is swapped in.
+- **New package `ZB.MOM.WW.Auth.ApiKeys.SqlServer`** + `AddZbApiKeyAuthSqlServer(config, apiKeySection, sqlServerSection)`, called instead of `AddZbApiKeyAuth` (it calls it internally after registering the SQL Server stores and migrator first).
+- **`zbauth` schema v1** (three tables: `schema_version`, `api_keys`, `api_key_audit`; `DATETIMEOFFSET(7)` timestamps). `SqlServerApiKeyStoreMigrator` is concurrent-boot safe — a central pair's two nodes both migrate at every boot — with retry on SQL error numbers 1205/2714/1913/2627 and jittered backoff (measured: the flat backoff it replaced exhausted its retries roughly 1 in 60 eight-way boots). DBA-facing DDL: `docs/sql/zbauth-schema.sql`.
+- **`SqliteApiKeyStoreImporter`** — idempotent insert-where-not-exists copy of an existing node's per-node SQLite keys and audit rows into the shared store; a row already present in SQL Server is never modified, so running it from both nodes of a pair converges on the union of their files.
+- **`ApiKeyAdminCommands`** gained a ctor taking `IApiKeyStoreMigrator`; the existing ctor taking the concrete `SqliteAuthStoreMigrator` is retained (MxAccessGateway, HistorianGateway and ScadaBridge all construct it that way).
+- SQLite remains the default store; **existing consumers need no code change to move to 0.3.0**.
+- Live tests for the SQL Server package are gated on `AUTH_SQLSERVER_CONNSTR` and skip (not fail) when it is unset.
+
+### 0.2.1 — LDAP AD continuation-referral fix
+
+- `NovellLdapConnection.Search` now skips Active Directory continuation referrals (`LdapReferralException` from `Next()`) instead of letting one fail the whole search — a `SearchBase` at a domain root made every login fail as `ServiceAccountBindFailed` against a healthy directory. Non-referral faults still propagate untouched.
+
+### 0.2.0 — version bump only
+
+- No functional change; bumped the lockstep family version ahead of the 0.2.1 LDAP fix.
 
 ### 0.1.4 — API-key expiry (archreview G-2)
 

ZB.MOM.WW.Auth/ZB.MOM.WW.Auth.slnx

@@ -2,11 +2,13 @@
   <Folder Name="/src/">
     <Project Path="src/ZB.MOM.WW.Auth.Abstractions/ZB.MOM.WW.Auth.Abstractions.csproj" />
     <Project Path="src/ZB.MOM.WW.Auth.ApiKeys/ZB.MOM.WW.Auth.ApiKeys.csproj" />
+    <Project Path="src/ZB.MOM.WW.Auth.ApiKeys.SqlServer/ZB.MOM.WW.Auth.ApiKeys.SqlServer.csproj" />
     <Project Path="src/ZB.MOM.WW.Auth.AspNetCore/ZB.MOM.WW.Auth.AspNetCore.csproj" />
     <Project Path="src/ZB.MOM.WW.Auth.Ldap/ZB.MOM.WW.Auth.Ldap.csproj" />
   </Folder>
   <Folder Name="/tests/">
     <Project Path="tests/ZB.MOM.WW.Auth.ApiKeys.Tests/ZB.MOM.WW.Auth.ApiKeys.Tests.csproj" />
+    <Project Path="tests/ZB.MOM.WW.Auth.ApiKeys.SqlServer.Tests/ZB.MOM.WW.Auth.ApiKeys.SqlServer.Tests.csproj" />
     <Project Path="tests/ZB.MOM.WW.Auth.AspNetCore.Tests/ZB.MOM.WW.Auth.AspNetCore.Tests.csproj" />
     <Project Path="tests/ZB.MOM.WW.Auth.Ldap.Tests/ZB.MOM.WW.Auth.Ldap.Tests.csproj" />
   </Folder>

ZB.MOM.WW.Auth/build/pack.sh

@@ -6,4 +6,10 @@
 
 set -euo pipefail
 
-dotnet pack -c Release -o ./artifacts
+# cd to the repo root and name the solution + an absolute output path: `dotnet pack` with
+# neither a project argument nor a fixed working directory packs whatever solution happens
+# to sit in the CALLER's directory (the bug behind the LocalDb feed incident, 0038d46).
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+cd "$REPO_ROOT"
+
+dotnet pack ZB.MOM.WW.Auth.slnx -c Release -o "$REPO_ROOT/artifacts"

ZB.MOM.WW.Auth/build/push.sh

@@ -16,9 +16,20 @@ set -euo pipefail
 : "${GITEA_NUGET_SOURCE:?set GITEA_NUGET_SOURCE to your Gitea NuGet feed URL}"
 : "${GITEA_NUGET_KEY:?set GITEA_NUGET_KEY to your Gitea access token}"
 
-dotnet pack -c Release -o ./artifacts
+# Named solution, named output, and a cd to the repo root, all for one reason: `dotnet pack`
+# with neither a project argument nor a fixed working directory packs whatever solution
+# happens to sit in the CALLER's directory — and then pushes it to a live feed under this
+# script's name. Invoked by absolute path from another repo's checkout, the relative form
+# packed and published 25 unrelated packages (LocalDb's copy of this script, fixed in
+# 0038d46; recovered with `dotnet nuget delete`, but only because the token happened to
+# carry delete rights). A publish script must be unambiguous about WHAT it publishes no
+# matter where it is run.
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+cd "$REPO_ROOT"
 
-dotnet nuget push "./artifacts/*.nupkg" \
+dotnet pack ZB.MOM.WW.Auth.slnx -c Release -o "$REPO_ROOT/artifacts"
+
+dotnet nuget push "$REPO_ROOT/artifacts/*.nupkg" \
   --source "$GITEA_NUGET_SOURCE" \
   --api-key "$GITEA_NUGET_KEY" \
   --skip-duplicate

ZB.MOM.WW.Auth/docs/sql/zbauth-schema.sql

@@ -0,0 +1,69 @@
+-- ZB.MOM.WW.Auth.ApiKeys.SqlServer — shared API-key schema, version 1.
+--
+-- For DBAs whose application login lacks DDL rights. The host normally provisions this itself at
+-- startup (SqlServerApiKeyStoreMigrator, which runs the identical statements and is safe to run
+-- concurrently from every node); where the app login may only read and write rows, run this script
+-- once against the target database instead. The migrator then finds the version row already at the
+-- version it supports and does nothing further beyond re-stamping applied_utc.
+--
+-- The schema name is hard-coded to zbauth here — the default of SqlServerApiKeyOptions.Schema. If a
+-- deployment overrides that option, substitute the name in every bracketed identifier below.
+--
+-- Kept byte-for-byte consistent with SqlServerAuthSchema.CreateSchemaDdl apart from that
+-- substitution and the literal version row at the end; change both together.
+
+IF NOT EXISTS (SELECT 1 FROM sys.schemas WHERE name = N'zbauth')
+    EXEC(N'CREATE SCHEMA [zbauth]');
+
+IF OBJECT_ID(N'[zbauth].[schema_version]', N'U') IS NULL
+    CREATE TABLE [zbauth].[schema_version] (
+        id          INT               NOT NULL CONSTRAINT PK_zbauth_schema_version PRIMARY KEY CHECK (id = 1),
+        version     INT               NOT NULL,
+        applied_utc DATETIMEOFFSET(7) NOT NULL
+    );
+
+IF OBJECT_ID(N'[zbauth].[api_keys]', N'U') IS NULL
+    CREATE TABLE [zbauth].[api_keys] (
+        key_id        NVARCHAR(64)      NOT NULL CONSTRAINT PK_zbauth_api_keys PRIMARY KEY,
+        key_prefix    NVARCHAR(16)      NOT NULL,
+        secret_hash   VARBINARY(64)     NOT NULL,
+        display_name  NVARCHAR(256)     NOT NULL,
+        scopes        NVARCHAR(MAX)     NOT NULL,
+        constraints   NVARCHAR(MAX)     NULL,
+        created_utc   DATETIMEOFFSET(7) NOT NULL,
+        last_used_utc DATETIMEOFFSET(7) NULL,
+        revoked_utc   DATETIMEOFFSET(7) NULL,
+        expires_utc   DATETIMEOFFSET(7) NULL
+    );
+
+IF OBJECT_ID(N'[zbauth].[api_key_audit]', N'U') IS NULL
+    CREATE TABLE [zbauth].[api_key_audit] (
+        audit_id       BIGINT IDENTITY(1,1) NOT NULL CONSTRAINT PK_zbauth_api_key_audit PRIMARY KEY,
+        key_id         NVARCHAR(64)      NULL,
+        event_type     NVARCHAR(64)      NOT NULL,
+        remote_address NVARCHAR(64)      NULL,
+        created_utc    DATETIMEOFFSET(7) NOT NULL,
+        details        NVARCHAR(MAX)     NULL
+    );
+
+IF NOT EXISTS (
+    SELECT 1 FROM sys.indexes
+    WHERE name = N'ix_api_keys_revoked_utc'
+      AND object_id = OBJECT_ID(N'[zbauth].[api_keys]'))
+    CREATE INDEX ix_api_keys_revoked_utc
+        ON [zbauth].[api_keys] (revoked_utc);
+
+IF NOT EXISTS (
+    SELECT 1 FROM sys.indexes
+    WHERE name = N'ix_api_key_audit_key_id_created_utc'
+      AND object_id = OBJECT_ID(N'[zbauth].[api_key_audit]'))
+    CREATE INDEX ix_api_key_audit_key_id_created_utc
+        ON [zbauth].[api_key_audit] (key_id, created_utc);
+
+MERGE [zbauth].[schema_version] AS target
+USING (SELECT 1 AS id, 1 AS version, SYSDATETIMEOFFSET() AS applied_utc) AS source
+    ON target.id = source.id
+WHEN MATCHED THEN
+    UPDATE SET version = source.version, applied_utc = source.applied_utc
+WHEN NOT MATCHED THEN
+    INSERT (id, version, applied_utc) VALUES (1, source.version, source.applied_utc);

ZB.MOM.WW.Auth/src/ZB.MOM.WW.Auth.Abstractions/ApiKeys/ApiKeyContracts.cs

@@ -78,3 +78,16 @@ public interface IApiKeyAuditStore
     Task AppendAsync(ApiKeyAuditEntry entry, CancellationToken ct);
     Task<IReadOnlyList<ApiKeyAuditEntry>> ListRecentAsync(int limit, CancellationToken ct);
 }
+
+/// <summary>
+/// Creates or upgrades the API-key store schema to the version this build supports. Each store
+/// provider (SQLite, SQL Server) supplies one; hosts run it at startup and before admin commands.
+/// Implementations must be idempotent and safe to run concurrently from several nodes.
+/// </summary>
+public interface IApiKeyStoreMigrator
+{
+    /// <summary>Migrates the store schema.</summary>
+    /// <param name="ct">A token to cancel the operation.</param>
+    /// <returns>A task that completes when the schema is current.</returns>
+    Task MigrateAsync(CancellationToken ct);
+}

ZB.MOM.WW.Auth/src/ZB.MOM.WW.Auth.Abstractions/Ldap/LdapContracts.cs

@@ -4,6 +4,20 @@ namespace ZB.MOM.WW.Auth.Abstractions.Ldap;
 
 public enum LdapTransport { Ldaps, StartTls, None }
 
+/// <summary>How <see cref="LdapOptions"/> authenticates a sign-in.</summary>
+public enum LdapBindMode
+{
+    /// <summary>Bind as the service account, search for the user, re-bind as the user (0.3.0 behaviour).</summary>
+    ServiceAccount,
+
+    /// <summary>
+    /// Bind as the signing-in user (identity built from <see cref="LdapOptions.UserBindTemplate"/>);
+    /// a successful bind authenticates them, and the same connection reads their own entry for groups.
+    /// No service account is used.
+    /// </summary>
+    User,
+}
+
 public sealed record LdapOptions
 {
     public bool Enabled { get; init; } = true;
@@ -19,6 +33,44 @@ public sealed record LdapOptions
     public string GroupAttribute { get; init; } = "memberOf";
     public int ConnectionTimeoutMs { get; init; } = 10_000;
 
+    /// <summary>
+    /// Selects the authentication flow. <see cref="LdapBindMode.ServiceAccount"/> (the default) is the
+    /// 0.3.0 bind-then-search-then-bind flow; <see cref="LdapBindMode.User"/> binds as the user and needs
+    /// no <see cref="ServiceAccountDn"/> / <see cref="ServiceAccountPassword"/>.
+    /// </summary>
+    public LdapBindMode BindMode { get; init; } = LdapBindMode.ServiceAccount;
+
+    /// <summary>
+    /// <see cref="LdapBindMode.User"/> only: the simple-bind identity, with <c>{0}</c> replaced by the
+    /// normalised login name. Must contain <c>{0}</c> exactly once.
+    /// <para>
+    /// Prefer the down-level form <c>EXAMPLE\{0}</c>: it maps exactly to the <c>sAMAccountName</c> the
+    /// group search uses, so the account that binds is the account whose groups are read. A UPN
+    /// template (<c>{0}@example.com</c>) is only correct where every account's UPN prefix equals its
+    /// <c>sAMAccountName</c> — AD resolves a UPN bind against the explicit <c>userPrincipalName</c>, so
+    /// a mismatch can fail sign-in or bind a different account than the one whose groups are read.
+    /// </para>
+    /// </summary>
+    public string UserBindTemplate { get; init; } = "";
+
+    /// <summary>
+    /// <see cref="LdapBindMode.User"/> only: when <see langword="true"/> (default) a login typed as
+    /// <c>DOMAIN\name</c> or <c>name@suffix</c> is reduced to <c>name</c>; when <see langword="false"/>
+    /// such input is refused as bad credentials without contacting the directory.
+    /// </summary>
+    public bool AcceptDomainQualifiedNames { get; init; } = true;
+
+    /// <summary>
+    /// Ordered fallback LDAP endpoints tried when the primary (and any earlier fallback)
+    /// fails with a system-side error (connect/TLS, service-account bind, or search).
+    /// Each entry is <c>"host"</c> (adopts <see cref="Port"/>) or <c>"host:port"</c>.
+    /// Empty (the default) disables failover entirely — single-endpoint behavior is
+    /// unchanged. The TLS validation policy (<see cref="ServerCertificateValidationCallback"/>
+    /// / <see cref="AllowInsecure"/>) applies to every endpoint. Endpoint preference is
+    /// sticky: the service keeps using the last endpoint that answered until it fails.
+    /// </summary>
+    public IReadOnlyList<string> FallbackServers { get; init; } = Array.Empty<string>();
+
     /// <summary>
     /// Optional hook to harden (or, in dev, relax) TLS server-certificate validation for the
     /// <see cref="LdapTransport.Ldaps"/> / <see cref="LdapTransport.StartTls"/> transports. When

ZB.MOM.WW.Auth/src/ZB.MOM.WW.Auth.ApiKeys.SqlServer/AuthSqlServerConnectionFactory.cs

@@ -0,0 +1,64 @@
+using Microsoft.Data.SqlClient;
+
+namespace ZB.MOM.WW.Auth.ApiKeys.SqlServer;
+
+/// <summary>
+/// Opens connections to the shared SQL Server API-key database and carries the settings every
+/// command needs (schema name, command timeout).
+/// </summary>
+/// <param name="options">The SQL Server API-key store options; validated in the constructor.</param>
+/// <exception cref="InvalidOperationException"><paramref name="options"/> is not usable.</exception>
+public sealed class AuthSqlServerConnectionFactory(SqlServerApiKeyOptions options)
+{
+    private readonly SqlServerApiKeyOptions _options = Validate(options);
+
+    private static SqlServerApiKeyOptions Validate(SqlServerApiKeyOptions options)
+    {
+        ArgumentNullException.ThrowIfNull(options);
+        options.Validate();
+        return options;
+    }
+
+    /// <summary>The schema that owns the three API-key tables.</summary>
+    public string Schema => _options.Schema;
+
+    /// <summary>Command timeout, in whole seconds, for every command issued against this store.</summary>
+    public int CommandTimeoutSeconds => Math.Max(1, (int)_options.CommandTimeout.TotalSeconds);
+
+    /// <summary>Opens a connection to the shared API-key database.</summary>
+    /// <param name="cancellationToken">A token to cancel the connect.</param>
+    /// <returns>An opened connection. The caller owns disposal.</returns>
+    public async Task<SqlConnection> OpenConnectionAsync(CancellationToken cancellationToken)
+    {
+        var connection = new SqlConnection(_options.ConnectionString);
+        try
+        {
+            await connection.OpenAsync(cancellationToken).ConfigureAwait(false);
+            return connection;
+        }
+        catch
+        {
+            await connection.DisposeAsync().ConfigureAwait(false);
+            throw;
+        }
+    }
+
+    /// <summary>
+    /// Creates a command on <paramref name="connection"/> with the configured timeout applied.
+    /// </summary>
+    /// <param name="connection">The open connection.</param>
+    /// <param name="commandText">The command text.</param>
+    /// <param name="transaction">The ambient transaction, if any.</param>
+    /// <returns>The configured command. The caller owns disposal.</returns>
+    public SqlCommand CreateCommand(
+        SqlConnection connection, string commandText, SqlTransaction? transaction = null)
+    {
+        ArgumentNullException.ThrowIfNull(connection);
+
+        SqlCommand command = connection.CreateCommand();
+        command.CommandText = commandText;
+        command.CommandTimeout = CommandTimeoutSeconds;
+        command.Transaction = transaction;
+        return command;
+    }
+}

ZB.MOM.WW.Auth/src/ZB.MOM.WW.Auth.ApiKeys.SqlServer/DependencyInjection/SqlServerApiKeyServiceCollectionExtensions.cs

@@ -0,0 +1,96 @@
+using Microsoft.Extensions.Configuration;
+using Microsoft.Extensions.DependencyInjection;
+using Microsoft.Extensions.DependencyInjection.Extensions;
+using Microsoft.Extensions.Options;
+using ZB.MOM.WW.Auth.Abstractions.ApiKeys;
+using ZB.MOM.WW.Auth.ApiKeys.DependencyInjection;
+
+namespace ZB.MOM.WW.Auth.ApiKeys.SqlServer.DependencyInjection;
+
+/// <summary>
+/// Dependency-injection entry point that swaps the API-key store backend from the SQLite default
+/// to a shared SQL Server database — one row set read and written by every node of a cluster (e.g.
+/// a ScadaBridge central pair), instead of the per-node file <see cref="AddZbApiKeyAuth"/> alone
+/// would give each node.
+/// </summary>
+public static class SqlServerApiKeyServiceCollectionExtensions
+{
+    /// <summary>
+    /// Registers the SQL Server-backed <see cref="IApiKeyStore"/>, <see cref="IApiKeyAdminStore"/>,
+    /// <see cref="IApiKeyAuditStore"/> and <see cref="IApiKeyStoreMigrator"/>, then calls
+    /// <see cref="ApiKeyServiceCollectionExtensions.AddZbApiKeyAuth"/> to complete the registration
+    /// (verifier, pepper provider, migration hosted service).
+    /// </summary>
+    /// <remarks>
+    /// <para>
+    /// <b>Call this INSTEAD of <see cref="ApiKeyServiceCollectionExtensions.AddZbApiKeyAuth"/></b>,
+    /// never both — this method calls it internally. The SQL Server stores and migrator are
+    /// registered here, BEFORE that inner call, precisely because
+    /// <see cref="ApiKeyServiceCollectionExtensions.AddZbApiKeyAuth"/> registers everything with
+    /// <c>TryAdd*</c>: whichever implementation of <see cref="IApiKeyStore"/> (etc.) is registered
+    /// first wins, and a later <c>TryAdd</c> for the same service is a no-op. That ordering is the
+    /// entire mechanism by which this method displaces the SQLite default.
+    /// </para>
+    /// <para>
+    /// <see cref="ApiKeyServiceCollectionExtensions.AddZbApiKeyAuth"/> still registers a SQLite
+    /// connection factory from <c>ApiKeyOptions.SqlitePath</c> even when SQL Server wins every
+    /// store slot. That is harmless — the SQLite factory is lazy and the connection it would open
+    /// is never opened when nothing resolves it — and it is actually useful: it keeps the one path
+    /// a node's pre-existing per-node SQLite file lives at in a single option, which is exactly the
+    /// source path <see cref="SqliteApiKeyStoreImporter"/> needs for the one-shot copy into the
+    /// shared store.
+    /// </para>
+    /// <para>
+    /// <see cref="SqliteApiKeyStoreImporter"/> is registered here so a host can resolve it and run
+    /// it once at boot (e.g. from a startup task or its own hosted service) to copy an existing
+    /// node's per-node keys and audit rows into the now-shared store; it performs no work by being
+    /// merely registered.
+    /// </para>
+    /// <para>
+    /// <b>The connection string must be a literal value or an environment-variable override — never
+    /// a <c>${secret:}</c> reference</b> on a host whose secret store is itself SQL-backed: resolving
+    /// that reference would require a working secret store, which on such a host requires this very
+    /// database to already be reachable, an unbreakable circular dependency at boot.
+    /// </para>
+    /// </remarks>
+    /// <param name="services">The service collection to add to.</param>
+    /// <param name="config">The application configuration.</param>
+    /// <param name="apiKeySectionPath">Path of the configuration section holding the core API-key options.</param>
+    /// <param name="sqlServerSectionPath">Path of the configuration section holding <see cref="SqlServerApiKeyOptions"/>.</param>
+    /// <returns>The same <paramref name="services"/> instance, for chaining.</returns>
+    public static IServiceCollection AddZbApiKeyAuthSqlServer(
+        this IServiceCollection services,
+        IConfiguration config,
+        string apiKeySectionPath,
+        string sqlServerSectionPath)
+    {
+        ArgumentNullException.ThrowIfNull(services);
+        ArgumentNullException.ThrowIfNull(config);
+        ArgumentException.ThrowIfNullOrWhiteSpace(apiKeySectionPath);
+        ArgumentException.ThrowIfNullOrWhiteSpace(sqlServerSectionPath);
+
+        services.Configure<SqlServerApiKeyOptions>(config.GetSection(sqlServerSectionPath));
+
+        services.TryAddSingleton(sp =>
+            new AuthSqlServerConnectionFactory(sp.GetRequiredService<IOptions<SqlServerApiKeyOptions>>().Value));
+
+        // Registered BEFORE AddZbApiKeyAuth so its TryAdd calls find these already present and
+        // leave them alone — that is the whole mechanism by which the SQL Server store displaces
+        // SQLite (see the class remarks).
+        services.TryAddSingleton<IApiKeyStore>(sp =>
+            new SqlServerApiKeyStore(sp.GetRequiredService<AuthSqlServerConnectionFactory>()));
+        services.TryAddSingleton<IApiKeyAdminStore>(sp =>
+            new SqlServerApiKeyAdminStore(sp.GetRequiredService<AuthSqlServerConnectionFactory>()));
+        services.TryAddSingleton<IApiKeyAuditStore>(sp =>
+            new SqlServerApiKeyAuditStore(sp.GetRequiredService<AuthSqlServerConnectionFactory>()));
+
+        services.TryAddSingleton<SqlServerApiKeyStoreMigrator>();
+        services.TryAddSingleton<IApiKeyStoreMigrator>(sp => sp.GetRequiredService<SqlServerApiKeyStoreMigrator>());
+
+        // Registered so a host can resolve it and run the one-shot per-node SQLite -> shared store
+        // copy at boot; registering it here does no work on its own.
+        services.TryAddSingleton<SqliteApiKeyStoreImporter>();
+
+        return services.AddZbApiKeyAuth(config, apiKeySectionPath);
+    }
+}

ZB.MOM.WW.Auth/src/ZB.MOM.WW.Auth.ApiKeys.SqlServer/SqlServerApiKeyAdminStore.cs

@@ -0,0 +1,247 @@
+using System.Data;
+using Microsoft.Data.SqlClient;
+using ZB.MOM.WW.Auth.Abstractions.ApiKeys;
+using ZB.MOM.WW.Auth.ApiKeys.Sqlite;
+
+namespace ZB.MOM.WW.Auth.ApiKeys.SqlServer;
+
+/// <summary>
+/// SQL Server-backed administration store for API keys (create, revoke, rotate, delete,
+/// set-scopes, enable/disable, list).
+/// </summary>
+/// <remarks>
+/// Semantics are a deliberate one-for-one port of <c>SqliteApiKeyAdminStore</c> — same WHERE and SET
+/// clauses, same rows-affected return contract — so an admin command behaves identically whichever
+/// provider a host composed. The point of this provider is only WHERE the rows live: one shared row
+/// set for a clustered pair instead of a file per node, so a revoke on either node is a revoke
+/// everywhere.
+/// </remarks>
+/// <param name="connectionFactory">Factory for connections to the shared API-key database.</param>
+public sealed class SqlServerApiKeyAdminStore(AuthSqlServerConnectionFactory connectionFactory) : IApiKeyAdminStore
+{
+    /// <inheritdoc />
+    /// <remarks>
+    /// A plain INSERT: a duplicate <c>key_id</c> surfaces as a primary-key
+    /// <see cref="SqlException"/> rather than silently overwriting an existing key (the SQLite store
+    /// raises its own <c>SqliteException</c> in the same situation).
+    /// </remarks>
+    public async Task CreateAsync(ApiKeyRecord record, CancellationToken ct)
+    {
+        ArgumentNullException.ThrowIfNull(record);
+
+        string schema = connectionFactory.Schema;
+
+        await using SqlConnection connection =
+            await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false);
+
+        await using SqlCommand command = connectionFactory.CreateCommand(connection, $"""
+            INSERT INTO [{schema}].[{SqlServerAuthSchema.ApiKeysTable}] (
+                key_id, key_prefix, secret_hash, display_name, scopes,
+                constraints, created_utc, last_used_utc, revoked_utc, expires_utc)
+            VALUES (
+                @key_id, @key_prefix, @secret_hash, @display_name, @scopes,
+                @constraints, @created_utc, @last_used_utc, @revoked_utc, @expires_utc);
+            """);
+        command.Parameters.Add("@key_id", SqlDbType.NVarChar, 64).Value = record.KeyId;
+        command.Parameters.Add("@key_prefix", SqlDbType.NVarChar, 16).Value = record.KeyPrefix;
+        command.Parameters.Add("@secret_hash", SqlDbType.VarBinary, 64).Value = record.SecretHash;
+        command.Parameters.Add("@display_name", SqlDbType.NVarChar, 256).Value = record.DisplayName;
+        command.Parameters.Add("@scopes", SqlDbType.NVarChar, -1).Value = ScopeSerializer.Serialize(record.Scopes);
+        command.Parameters.Add("@constraints", SqlDbType.NVarChar, -1).Value = (object?)record.ConstraintsJson ?? DBNull.Value;
+        command.Parameters.Add("@created_utc", SqlDbType.DateTimeOffset).Value = record.CreatedUtc;
+        command.Parameters.Add("@last_used_utc", SqlDbType.DateTimeOffset).Value = (object?)record.LastUsedUtc ?? DBNull.Value;
+        command.Parameters.Add("@revoked_utc", SqlDbType.DateTimeOffset).Value = (object?)record.RevokedUtc ?? DBNull.Value;
+        command.Parameters.Add("@expires_utc", SqlDbType.DateTimeOffset).Value = (object?)record.ExpiresUtc ?? DBNull.Value;
+
+        await command.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
+    }
+
+    /// <inheritdoc />
+    /// <remarks>
+    /// Guarded on <c>revoked_utc IS NULL</c>, so the returned flag means "this call revoked it" —
+    /// a second revoke of the same key returns <see langword="false"/> and preserves the original
+    /// revocation instant.
+    /// </remarks>
+    public async Task<bool> RevokeAsync(string keyId, DateTimeOffset whenUtc, CancellationToken ct)
+    {
+        ArgumentException.ThrowIfNullOrWhiteSpace(keyId);
+
+        string schema = connectionFactory.Schema;
+
+        await using SqlConnection connection =
+            await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false);
+
+        await using SqlCommand command = connectionFactory.CreateCommand(connection, $"""
+            UPDATE [{schema}].[{SqlServerAuthSchema.ApiKeysTable}]
+            SET revoked_utc = @revoked_utc
+            WHERE key_id = @key_id AND revoked_utc IS NULL;
+            """);
+        command.Parameters.Add("@key_id", SqlDbType.NVarChar, 64).Value = keyId;
+        command.Parameters.Add("@revoked_utc", SqlDbType.DateTimeOffset).Value = whenUtc;
+
+        int rows = await command.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
+
+        return rows > 0;
+    }
+
+    /// <inheritdoc />
+    /// <remarks>
+    /// Rotating also clears <c>last_used_utc</c> and <c>revoked_utc</c> — a rotated key is a fresh,
+    /// active credential — while leaving <c>expires_utc</c> alone, so an absolute expiry survives a
+    /// rotation.
+    /// </remarks>
+    public async Task<bool> RotateAsync(string keyId, byte[] newSecretHash, CancellationToken ct)
+    {
+        ArgumentException.ThrowIfNullOrWhiteSpace(keyId);
+        ArgumentNullException.ThrowIfNull(newSecretHash);
+
+        string schema = connectionFactory.Schema;
+
+        await using SqlConnection connection =
+            await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false);
+
+        await using SqlCommand command = connectionFactory.CreateCommand(connection, $"""
+            UPDATE [{schema}].[{SqlServerAuthSchema.ApiKeysTable}]
+            SET secret_hash = @secret_hash,
+                last_used_utc = NULL,
+                revoked_utc = NULL
+            WHERE key_id = @key_id;
+            """);
+        command.Parameters.Add("@key_id", SqlDbType.NVarChar, 64).Value = keyId;
+        command.Parameters.Add("@secret_hash", SqlDbType.VarBinary, 64).Value = newSecretHash;
+
+        int rows = await command.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
+
+        return rows > 0;
+    }
+
+    /// <inheritdoc />
+    public async Task<bool> SetScopesAsync(string keyId, IReadOnlySet<string> scopes, CancellationToken ct)
+    {
+        ArgumentException.ThrowIfNullOrWhiteSpace(keyId);
+        ArgumentNullException.ThrowIfNull(scopes);
+
+        string schema = connectionFactory.Schema;
+
+        await using SqlConnection connection =
+            await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false);
+
+        await using SqlCommand command = connectionFactory.CreateCommand(connection, $"""
+            UPDATE [{schema}].[{SqlServerAuthSchema.ApiKeysTable}]
+            SET scopes = @scopes
+            WHERE key_id = @key_id;
+            """);
+        command.Parameters.Add("@key_id", SqlDbType.NVarChar, 64).Value = keyId;
+        command.Parameters.Add("@scopes", SqlDbType.NVarChar, -1).Value = ScopeSerializer.Serialize(scopes);
+
+        int rows = await command.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
+
+        return rows > 0;
+    }
+
+    /// <inheritdoc />
+    public async Task<bool> SetEnabledAsync(string keyId, bool enabled, DateTimeOffset whenUtc, CancellationToken ct)
+    {
+        ArgumentException.ThrowIfNullOrWhiteSpace(keyId);
+
+        string schema = connectionFactory.Schema;
+
+        await using SqlConnection connection =
+            await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false);
+
+        // Reversible toggle: NO `revoked_utc IS NULL` guard (unlike RevokeAsync), so it works
+        // regardless of current state. Deliberately leaves secret_hash and last_used_utc untouched
+        // — that is what distinguishes re-enable from RotateAsync.
+        SqlCommand command;
+
+        if (enabled)
+        {
+            command = connectionFactory.CreateCommand(connection, $"""
+                UPDATE [{schema}].[{SqlServerAuthSchema.ApiKeysTable}]
+                SET revoked_utc = NULL
+                WHERE key_id = @key_id;
+                """);
+            command.Parameters.Add("@key_id", SqlDbType.NVarChar, 64).Value = keyId;
+        }
+        else
+        {
+            command = connectionFactory.CreateCommand(connection, $"""
+                UPDATE [{schema}].[{SqlServerAuthSchema.ApiKeysTable}]
+                SET revoked_utc = @revoked_utc
+                WHERE key_id = @key_id;
+                """);
+            command.Parameters.Add("@key_id", SqlDbType.NVarChar, 64).Value = keyId;
+            command.Parameters.Add("@revoked_utc", SqlDbType.DateTimeOffset).Value = whenUtc;
+        }
+
+        await using (command)
+        {
+            int rows = await command.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
+
+            return rows > 0;
+        }
+    }
+
+    /// <inheritdoc />
+    /// <remarks>
+    /// Only a REVOKED key can be deleted (<c>revoked_utc IS NOT NULL</c>), so an active credential
+    /// cannot vanish in one step — revoke first, and the revocation stands on its own until then.
+    /// </remarks>
+    public async Task<bool> DeleteAsync(string keyId, CancellationToken ct)
+    {
+        ArgumentException.ThrowIfNullOrWhiteSpace(keyId);
+
+        string schema = connectionFactory.Schema;
+
+        await using SqlConnection connection =
+            await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false);
+
+        await using SqlCommand command = connectionFactory.CreateCommand(connection, $"""
+            DELETE FROM [{schema}].[{SqlServerAuthSchema.ApiKeysTable}]
+            WHERE key_id = @key_id AND revoked_utc IS NOT NULL;
+            """);
+        command.Parameters.Add("@key_id", SqlDbType.NVarChar, 64).Value = keyId;
+
+        int rows = await command.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
+
+        return rows > 0;
+    }
+
+    /// <inheritdoc />
+    public async Task<IReadOnlyList<ApiKeyListItem>> ListAsync(CancellationToken ct)
+    {
+        string schema = connectionFactory.Schema;
+
+        await using SqlConnection connection =
+            await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false);
+
+        // Deliberately omits secret_hash so listing can never leak secret material. key_id is the
+        // tie-break so keys minted in the same instant still list in a stable order.
+        await using SqlCommand command = connectionFactory.CreateCommand(connection, $"""
+            SELECT key_id, key_prefix, display_name, scopes, constraints,
+                   created_utc, last_used_utc, revoked_utc, expires_utc
+            FROM [{schema}].[{SqlServerAuthSchema.ApiKeysTable}]
+            ORDER BY created_utc DESC, key_id DESC;
+            """);
+
+        List<ApiKeyListItem> items = [];
+
+        await using SqlDataReader reader = await command.ExecuteReaderAsync(ct).ConfigureAwait(false);
+
+        while (await reader.ReadAsync(ct).ConfigureAwait(false))
+        {
+            items.Add(new ApiKeyListItem(
+                KeyId: reader.GetString(0),
+                KeyPrefix: reader.GetString(1),
+                DisplayName: reader.GetString(2),
+                Scopes: ScopeSerializer.Deserialize(reader.GetString(3)),
+                ConstraintsJson: reader.IsDBNull(4) ? null : reader.GetString(4),
+                CreatedUtc: reader.GetDateTimeOffset(5),
+                LastUsedUtc: reader.IsDBNull(6) ? null : reader.GetDateTimeOffset(6),
+                RevokedUtc: reader.IsDBNull(7) ? null : reader.GetDateTimeOffset(7),
+                ExpiresUtc: reader.IsDBNull(8) ? null : reader.GetDateTimeOffset(8)));
+        }
+
+        return items;
+    }
+}

ZB.MOM.WW.Auth/src/ZB.MOM.WW.Auth.ApiKeys.SqlServer/SqlServerApiKeyAuditStore.cs

@@ -0,0 +1,80 @@
+using System.Data;
+using Microsoft.Data.SqlClient;
+using ZB.MOM.WW.Auth.Abstractions.ApiKeys;
+
+namespace ZB.MOM.WW.Auth.ApiKeys.SqlServer;
+
+/// <summary>
+/// SQL Server-backed, append-only audit store for API-key events. A one-for-one port of
+/// <c>SqliteApiKeyAuditStore</c>; on a clustered host the rows from every node land in one table,
+/// so the audit trail no longer has to be stitched together per node.
+/// </summary>
+/// <param name="connectionFactory">Factory for connections to the shared API-key database.</param>
+public sealed class SqlServerApiKeyAuditStore(AuthSqlServerConnectionFactory connectionFactory) : IApiKeyAuditStore
+{
+    /// <inheritdoc />
+    public async Task AppendAsync(ApiKeyAuditEntry entry, CancellationToken ct)
+    {
+        ArgumentNullException.ThrowIfNull(entry);
+
+        string schema = connectionFactory.Schema;
+
+        await using SqlConnection connection =
+            await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false);
+
+        await using SqlCommand command = connectionFactory.CreateCommand(connection, $"""
+            INSERT INTO [{schema}].[{SqlServerAuthSchema.ApiKeyAuditTable}] (
+                key_id, event_type, remote_address, created_utc, details)
+            VALUES (@key_id, @event_type, @remote_address, @created_utc, @details);
+            """);
+        command.Parameters.Add("@key_id", SqlDbType.NVarChar, 64).Value = (object?)entry.KeyId ?? DBNull.Value;
+        command.Parameters.Add("@event_type", SqlDbType.NVarChar, 64).Value = entry.EventType;
+        command.Parameters.Add("@remote_address", SqlDbType.NVarChar, 64).Value = (object?)entry.RemoteAddress ?? DBNull.Value;
+        command.Parameters.Add("@created_utc", SqlDbType.DateTimeOffset).Value = entry.CreatedUtc;
+        command.Parameters.Add("@details", SqlDbType.NVarChar, -1).Value = (object?)entry.Details ?? DBNull.Value;
+
+        await command.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
+    }
+
+    /// <inheritdoc />
+    /// <remarks>
+    /// Ordered by the IDENTITY <c>audit_id</c>, not by <c>created_utc</c>: two events stamped in the
+    /// same instant still come back in insertion order. A non-positive limit short-circuits without
+    /// opening a connection — <c>SELECT TOP (-1)</c> is not even legal T-SQL.
+    /// </remarks>
+    public async Task<IReadOnlyList<ApiKeyAuditEntry>> ListRecentAsync(int limit, CancellationToken ct)
+    {
+        if (limit <= 0)
+        {
+            return [];
+        }
+
+        string schema = connectionFactory.Schema;
+
+        await using SqlConnection connection =
+            await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false);
+
+        await using SqlCommand command = connectionFactory.CreateCommand(connection, $"""
+            SELECT TOP (@limit) key_id, event_type, remote_address, created_utc, details
+            FROM [{schema}].[{SqlServerAuthSchema.ApiKeyAuditTable}]
+            ORDER BY audit_id DESC;
+            """);
+        command.Parameters.Add("@limit", SqlDbType.Int).Value = limit;
+
+        List<ApiKeyAuditEntry> entries = [];
+
+        await using SqlDataReader reader = await command.ExecuteReaderAsync(ct).ConfigureAwait(false);
+
+        while (await reader.ReadAsync(ct).ConfigureAwait(false))
+        {
+            entries.Add(new ApiKeyAuditEntry(
+                KeyId: reader.IsDBNull(0) ? null : reader.GetString(0),
+                EventType: reader.GetString(1),
+                RemoteAddress: reader.IsDBNull(2) ? null : reader.GetString(2),
+                CreatedUtc: reader.GetDateTimeOffset(3),
+                Details: reader.IsDBNull(4) ? null : reader.GetString(4)));
+        }
+
+        return entries;
+    }
+}

ZB.MOM.WW.Auth/src/ZB.MOM.WW.Auth.ApiKeys.SqlServer/SqlServerApiKeyOptions.cs

@@ -0,0 +1,65 @@
+namespace ZB.MOM.WW.Auth.ApiKeys.SqlServer;
+
+/// <summary>
+/// Options for the shared SQL Server API-key store (section supplied by the host, e.g.
+/// <c>ScadaBridge:InboundApi:ApiKeyStore:SqlServer</c>).
+/// </summary>
+public sealed class SqlServerApiKeyOptions
+{
+    /// <summary>
+    /// Full SQL Server connection string. Required. Never a <c>${secret:}</c> reference on hosts
+    /// whose secret store is itself SQL-backed.
+    /// </summary>
+    public string ConnectionString { get; set; } = string.Empty;
+
+    /// <summary>Schema that owns the three tables. Letters, digits and underscore only (it is interpolated into DDL).</summary>
+    public string Schema { get; set; } = "zbauth";
+
+    /// <summary>
+    /// Command timeout applied to every store operation. Defaults to 30 seconds — long enough for
+    /// a contended shared database, short enough that a hung server surfaces as an error rather
+    /// than stalling a resolve indefinitely.
+    /// </summary>
+    public TimeSpan CommandTimeout { get; set; } = TimeSpan.FromSeconds(30);
+
+    /// <summary>Fails fast on a blank connection string, an unsafe schema identifier, or a non-positive command timeout.</summary>
+    /// <exception cref="InvalidOperationException">The options are not usable.</exception>
+    public void Validate()
+    {
+        if (string.IsNullOrWhiteSpace(ConnectionString))
+            throw new InvalidOperationException("SQL Server API-key store requires a ConnectionString.");
+
+        ValidateIdentifier(Schema);
+
+        if (CommandTimeout <= TimeSpan.Zero)
+            throw new InvalidOperationException($"SQL Server API-key store CommandTimeout must be positive (was {CommandTimeout}).");
+    }
+
+    /// <summary>
+    /// Strict allow-list for a SQL identifier that will be interpolated into DDL or query text.
+    /// Rejects everything that is not <c>[A-Za-z_][A-Za-z0-9_]{0,63}</c> — no brackets, quotes, dots
+    /// or spaces — so no bracket-escape (<c>]]</c>) trick can break out of the quoting applied at the
+    /// use site. Uses the ASCII-only character tests deliberately, so homoglyph and full-width
+    /// look-alikes are rejected rather than accepted as "letters".
+    /// </summary>
+    /// <param name="schemaName">The candidate identifier.</param>
+    /// <returns>The same value, once validated.</returns>
+    /// <exception cref="InvalidOperationException">The value is not a plain identifier.</exception>
+    /// <remarks>
+    /// Public because <see cref="SqlServerAuthSchema.CreateSchemaDdl(string)"/> is public and must
+    /// defend itself rather than trust that every caller routed through <see cref="Validate"/>.
+    /// </remarks>
+    public static string ValidateIdentifier(string schemaName)
+    {
+        if (string.IsNullOrEmpty(schemaName)
+            || schemaName.Length > 64
+            || (!char.IsAsciiLetter(schemaName[0]) && schemaName[0] != '_')
+            || !schemaName.All(c => char.IsAsciiLetterOrDigit(c) || c == '_'))
+        {
+            throw new InvalidOperationException(
+                $"SQL Server API-key store schema name '{schemaName}' is not a safe identifier.");
+        }
+
+        return schemaName;
+    }
+}

ZB.MOM.WW.Auth/src/ZB.MOM.WW.Auth.ApiKeys.SqlServer/SqlServerApiKeyStore.cs

@@ -0,0 +1,100 @@
+using System.Data;
+using Microsoft.Data.SqlClient;
+using ZB.MOM.WW.Auth.Abstractions.ApiKeys;
+using ZB.MOM.WW.Auth.ApiKeys.Sqlite;
+
+namespace ZB.MOM.WW.Auth.ApiKeys.SqlServer;
+
+/// <summary>
+/// SQL Server-backed read store for API-key records — the verification hot path.
+/// </summary>
+/// <remarks>
+/// Semantics are a deliberate one-for-one port of <c>SqliteApiKeyStore</c>, down to the WHERE
+/// clauses, so a host can switch providers without any behaviour changing. One connection per
+/// operation: the shared store is reached over the network and pooling, not a held connection, is
+/// what keeps a central pair's concurrent verifies cheap.
+/// </remarks>
+/// <param name="connectionFactory">Factory for connections to the shared API-key database.</param>
+public sealed class SqlServerApiKeyStore(AuthSqlServerConnectionFactory connectionFactory) : IApiKeyStore
+{
+    private const string SelectColumns =
+        "key_id, key_prefix, secret_hash, display_name, scopes, constraints, created_utc, last_used_utc, revoked_utc, expires_utc";
+
+    /// <inheritdoc />
+    public Task<ApiKeyRecord?> FindByKeyIdAsync(string keyId, CancellationToken ct)
+    {
+        ArgumentException.ThrowIfNullOrWhiteSpace(keyId);
+        return FindAsync(keyId, requireActive: false, ct);
+    }
+
+    /// <inheritdoc />
+    public Task<ApiKeyRecord?> FindActiveByKeyIdAsync(string keyId, CancellationToken ct)
+    {
+        ArgumentException.ThrowIfNullOrWhiteSpace(keyId);
+        return FindAsync(keyId, requireActive: true, ct);
+    }
+
+    /// <inheritdoc />
+    /// <remarks>
+    /// Guarded on <c>revoked_utc IS NULL</c>: a revoked key's last-used stamp is frozen at its last
+    /// legitimate use, so it never records the rejected attempts that follow.
+    /// </remarks>
+    public async Task MarkUsedAsync(string keyId, DateTimeOffset whenUtc, CancellationToken ct)
+    {
+        ArgumentException.ThrowIfNullOrWhiteSpace(keyId);
+
+        string schema = connectionFactory.Schema;
+
+        await using SqlConnection connection =
+            await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false);
+
+        await using SqlCommand command = connectionFactory.CreateCommand(connection, $"""
+            UPDATE [{schema}].[{SqlServerAuthSchema.ApiKeysTable}]
+            SET last_used_utc = @last_used_utc
+            WHERE key_id = @key_id AND revoked_utc IS NULL;
+            """);
+        command.Parameters.Add("@key_id", SqlDbType.NVarChar, 64).Value = keyId;
+        command.Parameters.Add("@last_used_utc", SqlDbType.DateTimeOffset).Value = whenUtc;
+
+        await command.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
+    }
+
+    private async Task<ApiKeyRecord?> FindAsync(string keyId, bool requireActive, CancellationToken ct)
+    {
+        string schema = connectionFactory.Schema;
+        string table = $"[{schema}].[{SqlServerAuthSchema.ApiKeysTable}]";
+
+        await using SqlConnection connection =
+            await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false);
+
+        // Active means NOT REVOKED only. Expiry is deliberately not filtered here — the verifier
+        // owns it, so it can tell an expired key from an unknown one (ApiKeyFailure.KeyExpired).
+        await using SqlCommand command = connectionFactory.CreateCommand(
+            connection,
+            requireActive
+                ? $"SELECT {SelectColumns} FROM {table} WHERE key_id = @key_id AND revoked_utc IS NULL;"
+                : $"SELECT {SelectColumns} FROM {table} WHERE key_id = @key_id;");
+        command.Parameters.Add("@key_id", SqlDbType.NVarChar, 64).Value = keyId;
+
+        await using SqlDataReader reader = await command.ExecuteReaderAsync(ct).ConfigureAwait(false);
+
+        if (!await reader.ReadAsync(ct).ConfigureAwait(false))
+        {
+            return null;
+        }
+
+        return ReadRecord(reader);
+    }
+
+    internal static ApiKeyRecord ReadRecord(SqlDataReader reader) => new(
+        KeyId: reader.GetString(0),
+        KeyPrefix: reader.GetString(1),
+        SecretHash: reader.GetFieldValue<byte[]>(2),
+        DisplayName: reader.GetString(3),
+        Scopes: ScopeSerializer.Deserialize(reader.GetString(4)),
+        ConstraintsJson: reader.IsDBNull(5) ? null : reader.GetString(5),
+        CreatedUtc: reader.GetDateTimeOffset(6),
+        LastUsedUtc: reader.IsDBNull(7) ? null : reader.GetDateTimeOffset(7),
+        RevokedUtc: reader.IsDBNull(8) ? null : reader.GetDateTimeOffset(8),
+        ExpiresUtc: reader.IsDBNull(9) ? null : reader.GetDateTimeOffset(9));
+}

ZB.MOM.WW.Auth/src/ZB.MOM.WW.Auth.ApiKeys.SqlServer/SqlServerApiKeyStoreMigrator.cs

@@ -0,0 +1,207 @@
+using System.Data;
+using System.Globalization;
+using Microsoft.Data.SqlClient;
+using ZB.MOM.WW.Auth.Abstractions.ApiKeys;
+using ZB.MOM.WW.Auth.ApiKeys.Sqlite;
+
+namespace ZB.MOM.WW.Auth.ApiKeys.SqlServer;
+
+/// <summary>
+/// Creates the shared SQL Server API-key schema and records the applied version. Idempotent: safe
+/// to run on every node at every startup, including concurrently — lost provisioning races surface
+/// as retryable <see cref="SqlException"/>s (see <see cref="IsRetryableMigrationErrorNumber"/>) and
+/// the retry re-runs the <c>IF NOT EXISTS</c> path against the objects the winner just created.
+/// Refuses to run against a database whose recorded version is newer than this build supports.
+/// </summary>
+/// <param name="connectionFactory">Factory for connections to the shared API-key database.</param>
+public sealed class SqlServerApiKeyStoreMigrator(AuthSqlServerConnectionFactory connectionFactory)
+    : IApiKeyStoreMigrator
+{
+    // Every node runs this at every startup, so a virgin database being provisioned by N nodes at
+    // once is the NORMAL case, not a rare race — and an unretried failure aborts that node's whole
+    // host at startup. Two failure shapes come out of that race, and retrying is correct for both
+    // precisely because the migration is idempotent:
+    //
+    //   - Deadlock victim (1205): concurrent Serializable DDL against the system catalogs can pick
+    //     a victim even though the IF NOT EXISTS guards make the logic itself safe.
+    //   - Lost CREATE / lost INSERT (2714, 1913, 2627): Serializable range locks do NOT serialize
+    //     the metadata existence checks against another session's in-flight CREATE — observed live
+    //     on the sibling shared secret store (ScadaBridge central-pair first boot, 2026-08-07):
+    //     both nodes passed the sys.schemas IF NOT EXISTS probe, the loser's CREATE SCHEMA then
+    //     died with 2714 "There is already an object named 'zbsecrets'". The object the loser tried
+    //     to create now exists, which is the desired end state — the retry's IF NOT EXISTS pass
+    //     skips it and proceeds.
+    //
+    // The retry cap keeps this from masking a GENUINE conflict (e.g. an unrelated pre-existing
+    // object colliding with the schema name): that reproduces on every attempt, so after
+    // MaxMigrationRetries the real SqlException propagates and the node fails closed.
+    private const int MaxMigrationRetries = 5;
+
+    /// <summary>Deadlock victim.</summary>
+    private const int SqlDeadlockVictimErrorNumber = 1205;
+
+    /// <summary>"There is already an object named '%ls'" — lost CREATE SCHEMA / CREATE TABLE race.</summary>
+    private const int SqlObjectAlreadyExistsErrorNumber = 2714;
+
+    /// <summary>"An index or statistics with name '%ls' already exists" — lost CREATE INDEX race.</summary>
+    private const int SqlIndexAlreadyExistsErrorNumber = 1913;
+
+    /// <summary>Primary-key violation — lost version-row MERGE-insert race.</summary>
+    private const int SqlPrimaryKeyViolationErrorNumber = 2627;
+
+    /// <inheritdoc />
+    /// <exception cref="AuthStoreMigrationException">
+    /// The recorded schema version is newer than <see cref="SqlServerAuthSchema.CurrentVersion"/>.
+    /// </exception>
+    public async Task MigrateAsync(CancellationToken cancellationToken)
+    {
+        for (int attempt = 1; ; attempt++)
+        {
+            try
+            {
+                await MigrateOnceAsync(cancellationToken).ConfigureAwait(false);
+                return;
+            }
+            catch (SqlException ex)
+                when (IsRetryableMigrationError(ex) && attempt < MaxMigrationRetries)
+            {
+                await Task.Delay(BackoffFor(attempt), cancellationToken).ConfigureAwait(false);
+            }
+        }
+    }
+
+    /// <summary>
+    /// Back-off before retry attempt <paramref name="attempt"/> + 1: a growing base plus RANDOM
+    /// jitter. The jitter is the load-bearing half. The sibling secret-store migrator this is ported
+    /// from delays a flat <c>100 * attempt</c>, which is the SAME value for every victim of the same
+    /// round — so N deadlock victims wake together and re-collide, exactly the lockstep its comment
+    /// claims to prevent. Measured here against a live SQL Server with eight concurrent migrators on
+    /// a virgin schema, the unjittered form exhausted the five-attempt budget on repeated 1205s in
+    /// roughly 1 run in 60; with jitter it did not.
+    /// </summary>
+    /// <param name="attempt">The 1-based attempt that just failed.</param>
+    /// <returns>How long to wait before the next attempt.</returns>
+    private static TimeSpan BackoffFor(int attempt) =>
+        TimeSpan.FromMilliseconds((100 * attempt) + Random.Shared.Next(0, 100 * attempt));
+
+    /// <summary>
+    /// Whether any error carried by <paramref name="exception"/> is a retryable provisioning-race
+    /// outcome. Scans the whole <see cref="SqlException.Errors"/> collection rather than only
+    /// <see cref="SqlException.Number"/> because a failed DDL batch stacks follow-on errors (e.g.
+    /// 2714 arrives with 2759 "CREATE SCHEMA failed due to previous errors") and the retryable one
+    /// is not guaranteed to be first.
+    /// </summary>
+    /// <param name="exception">The exception the migration attempt failed with.</param>
+    /// <returns><see langword="true"/> when a retry may succeed.</returns>
+    internal static bool IsRetryableMigrationError(SqlException exception)
+    {
+        foreach (SqlError error in exception.Errors)
+        {
+            if (IsRetryableMigrationErrorNumber(error.Number))
+            {
+                return true;
+            }
+        }
+
+        return IsRetryableMigrationErrorNumber(exception.Number);
+    }
+
+    /// <summary>
+    /// Classifies a SQL Server error number as a retryable outcome of N nodes provisioning the same
+    /// (possibly virgin) database at once. Everything here is a race whose retry re-runs the
+    /// idempotent <c>IF NOT EXISTS</c> path; anything else (login failures, missing database,
+    /// permissions, syntax) reproduces identically on retry and must propagate. 2705 (duplicate
+    /// column) is deliberately absent: every column lives inside a single <c>CREATE TABLE</c>
+    /// statement, so no cross-session race can produce it.
+    /// </summary>
+    /// <param name="errorNumber">The <see cref="SqlError.Number"/> to classify.</param>
+    /// <returns><see langword="true"/> when the number is a known provisioning-race outcome.</returns>
+    internal static bool IsRetryableMigrationErrorNumber(int errorNumber) => errorNumber
+        is SqlDeadlockVictimErrorNumber
+        or SqlObjectAlreadyExistsErrorNumber
+        or SqlIndexAlreadyExistsErrorNumber
+        or SqlPrimaryKeyViolationErrorNumber;
+
+    private async Task MigrateOnceAsync(CancellationToken cancellationToken)
+    {
+        string schema = connectionFactory.Schema;
+
+        await using SqlConnection connection =
+            await connectionFactory.OpenConnectionAsync(cancellationToken).ConfigureAwait(false);
+
+        // Serializable: unlike the single-writer SQLite store, EVERY node in the cluster runs this
+        // migration at startup, so concurrent first-boots are the normal case rather than a rare
+        // race. The isolation level narrows the race window but does NOT close it — the IF NOT
+        // EXISTS metadata probes are not serialized against another session's in-flight CREATE, so
+        // a loser can still fail with "object exists" (2714/1913) or a deadlock (1205). The retry
+        // filter in MigrateAsync is what makes simultaneous provision converge: by the time the
+        // loser retries, the winner's objects exist and every guard here skips them.
+        await using SqlTransaction transaction = (SqlTransaction)
+            await connection.BeginTransactionAsync(IsolationLevel.Serializable, cancellationToken)
+                .ConfigureAwait(false);
+
+        int existingVersion =
+            await ReadExistingSchemaVersionAsync(connection, transaction, schema, cancellationToken)
+                .ConfigureAwait(false);
+
+        if (existingVersion > SqlServerAuthSchema.CurrentVersion)
+        {
+            throw new AuthStoreMigrationException(
+                $"Shared API-key database schema version {existingVersion} is newer than supported version " +
+                $"{SqlServerAuthSchema.CurrentVersion}. Upgrade this node before it writes to the shared store.");
+        }
+
+        await using (SqlCommand ddl = connectionFactory.CreateCommand(
+            connection, SqlServerAuthSchema.CreateSchemaDdl(schema), transaction))
+        {
+            await ddl.ExecuteNonQueryAsync(cancellationToken).ConfigureAwait(false);
+        }
+
+        // MERGE rather than insert-if-missing: WHEN MATCHED re-stamps the row, so a future schema
+        // version actually lands in the table instead of leaving the first-ever value behind.
+        await using (SqlCommand version = connectionFactory.CreateCommand(
+            connection,
+            $"""
+            MERGE [{schema}].[{SqlServerAuthSchema.SchemaVersionTable}] AS target
+            USING (SELECT 1 AS id, @version AS version, @applied_utc AS applied_utc) AS source
+                ON target.id = source.id
+            WHEN MATCHED THEN
+                UPDATE SET version = source.version, applied_utc = source.applied_utc
+            WHEN NOT MATCHED THEN
+                INSERT (id, version, applied_utc) VALUES (1, source.version, source.applied_utc);
+            """,
+            transaction))
+        {
+            version.Parameters.Add("@version", SqlDbType.Int).Value = SqlServerAuthSchema.CurrentVersion;
+            version.Parameters.Add("@applied_utc", SqlDbType.DateTimeOffset).Value = DateTimeOffset.UtcNow;
+            await version.ExecuteNonQueryAsync(cancellationToken).ConfigureAwait(false);
+        }
+
+        await transaction.CommitAsync(cancellationToken).ConfigureAwait(false);
+    }
+
+    private async Task<int> ReadExistingSchemaVersionAsync(
+        SqlConnection connection,
+        SqlTransaction transaction,
+        string schema,
+        CancellationToken cancellationToken)
+    {
+        // OBJECT_ID returns NULL for an absent table, so this reads 0 on a virgin database without
+        // needing the table to exist first.
+        await using SqlCommand command = connectionFactory.CreateCommand(
+            connection,
+            $"""
+            IF OBJECT_ID(N'[{schema}].[{SqlServerAuthSchema.SchemaVersionTable}]', N'U') IS NULL
+                SELECT 0;
+            ELSE
+                SELECT ISNULL((SELECT version FROM [{schema}].[{SqlServerAuthSchema.SchemaVersionTable}] WHERE id = 1), 0);
+            """,
+            transaction);
+
+        object? version = await command.ExecuteScalarAsync(cancellationToken).ConfigureAwait(false);
+
+        return version is null || version == DBNull.Value
+            ? 0
+            : Convert.ToInt32(version, CultureInfo.InvariantCulture);
+    }
+}

ZB.MOM.WW.Auth/src/ZB.MOM.WW.Auth.ApiKeys.SqlServer/SqlServerAuthSchema.cs

@@ -0,0 +1,116 @@
+namespace ZB.MOM.WW.Auth.ApiKeys.SqlServer;
+
+/// <summary>
+/// Schema constants and DDL for the shared SQL Server API-key store. Mirrors
+/// <c>ZB.MOM.WW.Auth.ApiKeys.Sqlite.SqliteAuthSchema</c> column-for-column so a row means exactly
+/// the same thing in either store.
+/// </summary>
+/// <remarks>
+/// <para>
+/// <b>Timestamps are <c>DATETIMEOFFSET(7)</c>, not ISO-8601 text.</b> The sibling
+/// <c>ZB.MOM.WW.Secrets</c> SQL Server store deliberately stores its timestamps as
+/// <c>nvarchar</c> because its rows <em>replicate</em> with a SQLite store and last-writer-wins
+/// compares those timestamps for equality, where a sub-tick round-trip difference would turn a tie
+/// into a spurious "newer". No such rationale applies here: this store never replicates with the
+/// SQLite one — a host picks exactly one provider, and the one-shot SQLite import is a copy, not an
+/// ongoing sync — so the native type is used, which sorts and range-scans correctly and keeps the
+/// tables readable to a DBA.
+/// </para>
+/// <para>
+/// <c>SqliteAuthSchema.CurrentVersion</c> is 3 only because that store carries the donor
+/// MxAccessGateway's version history (v1/v2 were byte-identical; v3 added <c>expires_utc</c>). This
+/// schema is created whole, with <c>expires_utc</c> present from the start, so its own version line
+/// starts at 1.
+/// </para>
+/// </remarks>
+public static class SqlServerAuthSchema
+{
+    /// <summary>
+    /// The schema version this build creates and supports. The migrator refuses a database whose
+    /// recorded version is <em>newer</em> than this.
+    /// </summary>
+    public const int CurrentVersion = 1;
+
+    /// <summary>Default schema the API-key tables live in.</summary>
+    public const string DefaultSchemaName = "zbauth";
+
+    /// <summary>Name of the single-row table tracking the applied schema version.</summary>
+    public const string SchemaVersionTable = "schema_version";
+
+    /// <summary>Name of the table storing API-key records.</summary>
+    public const string ApiKeysTable = "api_keys";
+
+    /// <summary>Name of the append-only audit table.</summary>
+    public const string ApiKeyAuditTable = "api_key_audit";
+
+    /// <summary>
+    /// Returns the idempotent DDL that provisions the schema, the three tables and the two
+    /// supporting indexes in <paramref name="schemaName"/>.
+    /// </summary>
+    /// <param name="schemaName">The SQL schema name to create the objects in.</param>
+    /// <returns>The DDL batch.</returns>
+    /// <remarks>
+    /// <paramref name="schemaName"/> is interpolated, not parameterized — T-SQL has no parameter
+    /// form for an object name. It is re-validated here against the same strict identifier
+    /// allow-list <see cref="SqlServerApiKeyOptions.Validate"/> enforces, and bracket-quoted below.
+    /// The duplicate check is deliberate: this method is <c>public</c>, so it must defend itself
+    /// rather than trust that every caller routed through the options type. The version row is NOT
+    /// written here — <see cref="SqlServerApiKeyStoreMigrator"/> writes it with a MERGE that also
+    /// updates on match, so a future upgrade re-stamps the stored version.
+    /// </remarks>
+    /// <exception cref="InvalidOperationException"><paramref name="schemaName"/> is not a plain identifier.</exception>
+    public static string CreateSchemaDdl(string schemaName)
+    {
+        SqlServerApiKeyOptions.ValidateIdentifier(schemaName);
+
+        return $"""
+            IF NOT EXISTS (SELECT 1 FROM sys.schemas WHERE name = N'{schemaName}')
+                EXEC(N'CREATE SCHEMA [{schemaName}]');
+
+            IF OBJECT_ID(N'[{schemaName}].[{SchemaVersionTable}]', N'U') IS NULL
+                CREATE TABLE [{schemaName}].[{SchemaVersionTable}] (
+                    id          INT               NOT NULL CONSTRAINT PK_{schemaName}_{SchemaVersionTable} PRIMARY KEY CHECK (id = 1),
+                    version     INT               NOT NULL,
+                    applied_utc DATETIMEOFFSET(7) NOT NULL
+                );
+
+            IF OBJECT_ID(N'[{schemaName}].[{ApiKeysTable}]', N'U') IS NULL
+                CREATE TABLE [{schemaName}].[{ApiKeysTable}] (
+                    key_id        NVARCHAR(64)      NOT NULL CONSTRAINT PK_{schemaName}_{ApiKeysTable} PRIMARY KEY,
+                    key_prefix    NVARCHAR(16)      NOT NULL,
+                    secret_hash   VARBINARY(64)     NOT NULL,
+                    display_name  NVARCHAR(256)     NOT NULL,
+                    scopes        NVARCHAR(MAX)     NOT NULL,
+                    constraints   NVARCHAR(MAX)     NULL,
+                    created_utc   DATETIMEOFFSET(7) NOT NULL,
+                    last_used_utc DATETIMEOFFSET(7) NULL,
+                    revoked_utc   DATETIMEOFFSET(7) NULL,
+                    expires_utc   DATETIMEOFFSET(7) NULL
+                );
+
+            IF OBJECT_ID(N'[{schemaName}].[{ApiKeyAuditTable}]', N'U') IS NULL
+                CREATE TABLE [{schemaName}].[{ApiKeyAuditTable}] (
+                    audit_id       BIGINT IDENTITY(1,1) NOT NULL CONSTRAINT PK_{schemaName}_{ApiKeyAuditTable} PRIMARY KEY,
+                    key_id         NVARCHAR(64)      NULL,
+                    event_type     NVARCHAR(64)      NOT NULL,
+                    remote_address NVARCHAR(64)      NULL,
+                    created_utc    DATETIMEOFFSET(7) NOT NULL,
+                    details        NVARCHAR(MAX)     NULL
+                );
+
+            IF NOT EXISTS (
+                SELECT 1 FROM sys.indexes
+                WHERE name = N'ix_api_keys_revoked_utc'
+                  AND object_id = OBJECT_ID(N'[{schemaName}].[{ApiKeysTable}]'))
+                CREATE INDEX ix_api_keys_revoked_utc
+                    ON [{schemaName}].[{ApiKeysTable}] (revoked_utc);
+
+            IF NOT EXISTS (
+                SELECT 1 FROM sys.indexes
+                WHERE name = N'ix_api_key_audit_key_id_created_utc'
+                  AND object_id = OBJECT_ID(N'[{schemaName}].[{ApiKeyAuditTable}]'))
+                CREATE INDEX ix_api_key_audit_key_id_created_utc
+                    ON [{schemaName}].[{ApiKeyAuditTable}] (key_id, created_utc);
+            """;
+    }
+}

ZB.MOM.WW.Auth/src/ZB.MOM.WW.Auth.ApiKeys.SqlServer/SqliteApiKeyStoreImporter.cs

@@ -0,0 +1,300 @@
+using System.Data;
+using Microsoft.Data.SqlClient;
+using Microsoft.Data.Sqlite;
+using ZB.MOM.WW.Auth.Abstractions.ApiKeys;
+using ZB.MOM.WW.Auth.ApiKeys.Sqlite;
+
+namespace ZB.MOM.WW.Auth.ApiKeys.SqlServer;
+
+/// <summary>Result of a one-shot import.</summary>
+/// <param name="KeysInserted">Keys copied into SQL Server by this run.</param>
+/// <param name="KeysSkipped">Keys whose <c>key_id</c> already existed in SQL Server.</param>
+/// <param name="AuditInserted">Audit rows copied into SQL Server by this run.</param>
+/// <param name="AuditSkipped">Audit rows that already existed in SQL Server.</param>
+/// <param name="SourceFound">Whether the SQLite file existed at all.</param>
+public sealed record ApiKeyImportResult(
+    int KeysInserted, int KeysSkipped, int AuditInserted, int AuditSkipped, bool SourceFound);
+
+/// <summary>
+/// Copies every key and audit row from a per-node SQLite store into the shared SQL Server store,
+/// insert-where-not-exists (keys by key_id; audit rows by key_id + event_type + created_utc), so it is
+/// idempotent and every node of a pair may run it: the result is the UNION of the nodes' files. A row
+/// that already exists in SQL Server is never modified — a revoke recorded on one node but not the
+/// other therefore survives only if the revoked copy is imported first; operators reconcile by listing.
+/// </summary>
+/// <param name="sql">Factory for connections to the shared SQL Server API-key database.</param>
+public sealed class SqliteApiKeyStoreImporter(AuthSqlServerConnectionFactory sql)
+{
+    /// <summary>
+    /// Imports the keys and audit rows held in the SQLite store at <paramref name="sqlitePath"/>.
+    /// </summary>
+    /// <param name="sqlitePath">Path of the per-node SQLite API-key file.</param>
+    /// <param name="ct">A token to cancel the import.</param>
+    /// <returns>
+    /// The per-category insert/skip counts. When the file does not exist the result is all zeroes
+    /// with <see cref="ApiKeyImportResult.SourceFound"/> <see langword="false"/> and nothing is
+    /// opened — a host that has already imported (and renamed the file) pays nothing.
+    /// </returns>
+    /// <remarks>
+    /// <para>
+    /// The SQLite file is opened READ-ONLY and the schema migrator is deliberately NOT run against
+    /// it: the source is evidence, not something to upgrade. A file with no <c>api_keys</c> table is
+    /// treated as an empty source rather than an error, and a pre-v3 file (no <c>expires_utc</c>
+    /// column) reads its expiry as NULL = never expires, which is exactly what those keys meant.
+    /// </para>
+    /// <para>
+    /// Every row is read and the SQLite connection closed BEFORE any SQL Server write, and all the
+    /// writes then run inside ONE transaction. So the import is all-or-nothing: a failure part way
+    /// through rolls the transaction back and the exception propagates to the caller, leaving the
+    /// shared store exactly as it was. The host decides what that means — for a boot-time import,
+    /// logging and carrying on is usually right, because keys already in SQL Server keep working.
+    /// </para>
+    /// </remarks>
+    /// <exception cref="SqlException">A SQL Server write failed; nothing was committed.</exception>
+    /// <exception cref="SqliteException">The source file could not be read.</exception>
+    public async Task<ApiKeyImportResult> ImportAsync(string sqlitePath, CancellationToken ct)
+    {
+        ArgumentException.ThrowIfNullOrWhiteSpace(sqlitePath);
+
+        if (!File.Exists(sqlitePath))
+        {
+            return new ApiKeyImportResult(0, 0, 0, 0, SourceFound: false);
+        }
+
+        (IReadOnlyList<ApiKeyRecord> keys, IReadOnlyList<ApiKeyAuditEntry> auditEntries) =
+            await ReadSourceAsync(sqlitePath, ct).ConfigureAwait(false);
+
+        return await WriteAsync(keys, auditEntries, ct).ConfigureAwait(false);
+    }
+
+    // --- read side (SQLite) ---
+
+    private static async Task<(IReadOnlyList<ApiKeyRecord> Keys, IReadOnlyList<ApiKeyAuditEntry> Audit)>
+        ReadSourceAsync(string sqlitePath, CancellationToken ct)
+    {
+        SqliteConnectionStringBuilder builder = new()
+        {
+            DataSource = sqlitePath,
+            Mode = SqliteOpenMode.ReadOnly,
+        };
+
+        await using var connection = new SqliteConnection(builder.ToString());
+        await connection.OpenAsync(ct).ConfigureAwait(false);
+
+        IReadOnlyList<ApiKeyRecord> keys =
+            await TableExistsAsync(connection, SqliteAuthSchema.ApiKeysTable, ct).ConfigureAwait(false)
+                ? await ReadKeysAsync(connection, ct).ConfigureAwait(false)
+                : [];
+
+        IReadOnlyList<ApiKeyAuditEntry> audit =
+            await TableExistsAsync(connection, SqliteAuthSchema.ApiKeyAuditTable, ct).ConfigureAwait(false)
+                ? await ReadAuditAsync(connection, ct).ConfigureAwait(false)
+                : [];
+
+        return (keys, audit);
+    }
+
+    private static async Task<bool> TableExistsAsync(
+        SqliteConnection connection, string tableName, CancellationToken ct)
+    {
+        await using SqliteCommand command = connection.CreateCommand();
+        command.CommandText = """
+            SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = $table_name;
+            """;
+        command.Parameters.AddWithValue("$table_name", tableName);
+
+        return (long)(await command.ExecuteScalarAsync(ct).ConfigureAwait(false) ?? 0L) > 0;
+    }
+
+    private static async Task<IReadOnlyList<ApiKeyRecord>> ReadKeysAsync(
+        SqliteConnection connection, CancellationToken ct)
+    {
+        // A pre-v3 file has no expires_utc column, so select a literal NULL in its place rather than
+        // failing the import: NULL is precisely what those keys mean (never expires).
+        bool hasExpires = await ColumnExistsAsync(
+            connection, SqliteAuthSchema.ApiKeysTable, SqliteAuthSchema.ExpiresUtcColumn, ct).ConfigureAwait(false);
+
+        string expiresColumn = hasExpires ? SqliteAuthSchema.ExpiresUtcColumn : "NULL";
+
+        await using SqliteCommand command = connection.CreateCommand();
+        command.CommandText =
+            "SELECT key_id, key_prefix, secret_hash, display_name, scopes, constraints, " +
+            $"created_utc, last_used_utc, revoked_utc, {expiresColumn} FROM api_keys;";
+
+        List<ApiKeyRecord> keys = [];
+
+        await using SqliteDataReader reader = await command.ExecuteReaderAsync(ct).ConfigureAwait(false);
+
+        while (await reader.ReadAsync(ct).ConfigureAwait(false))
+        {
+            keys.Add(new ApiKeyRecord(
+                KeyId: reader.GetString(0),
+                KeyPrefix: reader.GetString(1),
+                SecretHash: reader.GetFieldValue<byte[]>(2),
+                DisplayName: reader.GetString(3),
+                Scopes: ScopeSerializer.Deserialize(reader.GetString(4)),
+                ConstraintsJson: reader.IsDBNull(5) ? null : reader.GetString(5),
+                CreatedUtc: SqliteValueParsing.ParseUtc(reader.GetString(6)),
+                LastUsedUtc: SqliteValueParsing.ReadNullableUtc(reader, 7),
+                RevokedUtc: SqliteValueParsing.ReadNullableUtc(reader, 8),
+                ExpiresUtc: SqliteValueParsing.ReadNullableUtc(reader, 9)));
+        }
+
+        return keys;
+    }
+
+    // SQLite has no information_schema; PRAGMA table_info is the column probe (the same one
+    // SqliteAuthStoreMigrator uses to decide whether its v3 ALTER TABLE is needed).
+    private static async Task<bool> ColumnExistsAsync(
+        SqliteConnection connection, string tableName, string columnName, CancellationToken ct)
+    {
+        await using SqliteCommand command = connection.CreateCommand();
+        // table_info takes an identifier, not a bindable parameter; the name is an internal constant.
+        command.CommandText = $"PRAGMA table_info({tableName});";
+
+        await using SqliteDataReader reader = await command.ExecuteReaderAsync(ct).ConfigureAwait(false);
+
+        while (await reader.ReadAsync(ct).ConfigureAwait(false))
+        {
+            // PRAGMA table_info columns: cid(0), name(1), type(2), ...
+            if (string.Equals(reader.GetString(1), columnName, StringComparison.OrdinalIgnoreCase))
+            {
+                return true;
+            }
+        }
+
+        return false;
+    }
+
+    private static async Task<IReadOnlyList<ApiKeyAuditEntry>> ReadAuditAsync(
+        SqliteConnection connection, CancellationToken ct)
+    {
+        await using SqliteCommand command = connection.CreateCommand();
+        // Insertion order, so the shared table's IDENTITY re-creates the source's ordering.
+        command.CommandText = """
+            SELECT key_id, event_type, remote_address, created_utc, details
+            FROM api_key_audit
+            ORDER BY audit_id;
+            """;
+
+        List<ApiKeyAuditEntry> entries = [];
+
+        await using SqliteDataReader reader = await command.ExecuteReaderAsync(ct).ConfigureAwait(false);
+
+        while (await reader.ReadAsync(ct).ConfigureAwait(false))
+        {
+            entries.Add(new ApiKeyAuditEntry(
+                KeyId: reader.IsDBNull(0) ? null : reader.GetString(0),
+                EventType: reader.GetString(1),
+                RemoteAddress: reader.IsDBNull(2) ? null : reader.GetString(2),
+                CreatedUtc: SqliteValueParsing.ParseUtc(reader.GetString(3)),
+                Details: reader.IsDBNull(4) ? null : reader.GetString(4)));
+        }
+
+        return entries;
+    }
+
+    // --- write side (SQL Server) ---
+
+    private async Task<ApiKeyImportResult> WriteAsync(
+        IReadOnlyList<ApiKeyRecord> keys,
+        IReadOnlyList<ApiKeyAuditEntry> auditEntries,
+        CancellationToken ct)
+    {
+        string schema = sql.Schema;
+
+        await using SqlConnection connection = await sql.OpenConnectionAsync(ct).ConfigureAwait(false);
+        await using SqlTransaction transaction =
+            (SqlTransaction)await connection.BeginTransactionAsync(ct).ConfigureAwait(false);
+
+        int keysInserted = 0;
+        int auditInserted = 0;
+
+        foreach (ApiKeyRecord key in keys)
+        {
+            if (await InsertKeyIfAbsentAsync(connection, transaction, schema, key, ct).ConfigureAwait(false))
+            {
+                keysInserted++;
+            }
+        }
+
+        foreach (ApiKeyAuditEntry entry in auditEntries)
+        {
+            if (await InsertAuditIfAbsentAsync(connection, transaction, schema, entry, ct).ConfigureAwait(false))
+            {
+                auditInserted++;
+            }
+        }
+
+        await transaction.CommitAsync(ct).ConfigureAwait(false);
+
+        return new ApiKeyImportResult(
+            KeysInserted: keysInserted,
+            KeysSkipped: keys.Count - keysInserted,
+            AuditInserted: auditInserted,
+            AuditSkipped: auditEntries.Count - auditInserted,
+            SourceFound: true);
+    }
+
+    // Returns true when this call inserted the row. The guard is inside the statement rather than a
+    // separate SELECT so the check and the insert cannot straddle a concurrent writer.
+    private async Task<bool> InsertKeyIfAbsentAsync(
+        SqlConnection connection,
+        SqlTransaction transaction,
+        string schema,
+        ApiKeyRecord key,
+        CancellationToken ct)
+    {
+        await using SqlCommand command = sql.CreateCommand(connection, $"""
+            IF NOT EXISTS (
+                SELECT 1 FROM [{schema}].[{SqlServerAuthSchema.ApiKeysTable}] WHERE key_id = @key_id)
+                INSERT INTO [{schema}].[{SqlServerAuthSchema.ApiKeysTable}] (
+                    key_id, key_prefix, secret_hash, display_name, scopes,
+                    constraints, created_utc, last_used_utc, revoked_utc, expires_utc)
+                VALUES (
+                    @key_id, @key_prefix, @secret_hash, @display_name, @scopes,
+                    @constraints, @created_utc, @last_used_utc, @revoked_utc, @expires_utc);
+            """, transaction);
+        command.Parameters.Add("@key_id", SqlDbType.NVarChar, 64).Value = key.KeyId;
+        command.Parameters.Add("@key_prefix", SqlDbType.NVarChar, 16).Value = key.KeyPrefix;
+        command.Parameters.Add("@secret_hash", SqlDbType.VarBinary, 64).Value = key.SecretHash;
+        command.Parameters.Add("@display_name", SqlDbType.NVarChar, 256).Value = key.DisplayName;
+        command.Parameters.Add("@scopes", SqlDbType.NVarChar, -1).Value = ScopeSerializer.Serialize(key.Scopes);
+        command.Parameters.Add("@constraints", SqlDbType.NVarChar, -1).Value = (object?)key.ConstraintsJson ?? DBNull.Value;
+        command.Parameters.Add("@created_utc", SqlDbType.DateTimeOffset).Value = key.CreatedUtc;
+        command.Parameters.Add("@last_used_utc", SqlDbType.DateTimeOffset).Value = (object?)key.LastUsedUtc ?? DBNull.Value;
+        command.Parameters.Add("@revoked_utc", SqlDbType.DateTimeOffset).Value = (object?)key.RevokedUtc ?? DBNull.Value;
+        command.Parameters.Add("@expires_utc", SqlDbType.DateTimeOffset).Value = (object?)key.ExpiresUtc ?? DBNull.Value;
+
+        return await command.ExecuteNonQueryAsync(ct).ConfigureAwait(false) > 0;
+    }
+
+    // Audit rows carry no natural key (audit_id is per-file IDENTITY), so identity is the triple
+    // key_id + event_type + created_utc. key_id is nullable, hence the explicit NULL-safe arm: a
+    // plain `key_id = @key_id` is UNKNOWN, never true, when both sides are NULL.
+    private async Task<bool> InsertAuditIfAbsentAsync(
+        SqlConnection connection,
+        SqlTransaction transaction,
+        string schema,
+        ApiKeyAuditEntry entry,
+        CancellationToken ct)
+    {
+        await using SqlCommand command = sql.CreateCommand(connection, $"""
+            IF NOT EXISTS (
+                SELECT 1 FROM [{schema}].[{SqlServerAuthSchema.ApiKeyAuditTable}]
+                WHERE ((key_id = @key_id) OR (key_id IS NULL AND @key_id IS NULL))
+                  AND event_type = @event_type
+                  AND created_utc = @created_utc)
+                INSERT INTO [{schema}].[{SqlServerAuthSchema.ApiKeyAuditTable}] (
+                    key_id, event_type, remote_address, created_utc, details)
+                VALUES (@key_id, @event_type, @remote_address, @created_utc, @details);
+            """, transaction);
+        command.Parameters.Add("@key_id", SqlDbType.NVarChar, 64).Value = (object?)entry.KeyId ?? DBNull.Value;
+        command.Parameters.Add("@event_type", SqlDbType.NVarChar, 64).Value = entry.EventType;
+        command.Parameters.Add("@remote_address", SqlDbType.NVarChar, 64).Value = (object?)entry.RemoteAddress ?? DBNull.Value;
+        command.Parameters.Add("@created_utc", SqlDbType.DateTimeOffset).Value = entry.CreatedUtc;
+        command.Parameters.Add("@details", SqlDbType.NVarChar, -1).Value = (object?)entry.Details ?? DBNull.Value;
+
+        return await command.ExecuteNonQueryAsync(ct).ConfigureAwait(false) > 0;
+    }
+}

ZB.MOM.WW.Auth/src/ZB.MOM.WW.Auth.ApiKeys.SqlServer/ZB.MOM.WW.Auth.ApiKeys.SqlServer.csproj

@@ -0,0 +1,42 @@
+<Project Sdk="Microsoft.NET.Sdk">
+
+  <ItemGroup>
+    <ProjectReference Include="..\ZB.MOM.WW.Auth.ApiKeys\ZB.MOM.WW.Auth.ApiKeys.csproj" />
+  </ItemGroup>
+
+  <ItemGroup>
+    <PackageReference Include="Microsoft.Data.SqlClient" />
+    <!--
+      Lightweight Microsoft.Extensions.* abstractions back the DI extension
+      (AddZbApiKeyAuthSqlServer, shipped by Task 6): IServiceCollection / TryAdd* / Configure
+      (DependencyInjection.Abstractions), IOptions (Options), IConfiguration
+      (Configuration.Abstractions), and IHostedService / AddHostedService (Hosting.Abstractions).
+    -->
+    <PackageReference Include="Microsoft.Extensions.DependencyInjection.Abstractions" />
+    <PackageReference Include="Microsoft.Extensions.Options" />
+    <!-- Supplies the Configure<TOptions>(IConfiguration) binding overload used by the DI extension. -->
+    <PackageReference Include="Microsoft.Extensions.Options.ConfigurationExtensions" />
+    <PackageReference Include="Microsoft.Extensions.Configuration.Abstractions" />
+    <PackageReference Include="Microsoft.Extensions.Hosting.Abstractions" />
+  </ItemGroup>
+
+  <PropertyGroup>
+    <TargetFramework>net10.0</TargetFramework>
+    <ImplicitUsings>enable</ImplicitUsings>
+    <Nullable>enable</Nullable>
+  </PropertyGroup>
+
+  <PropertyGroup>
+    <IsPackable>true</IsPackable>
+    <PackageId>ZB.MOM.WW.Auth.ApiKeys.SqlServer</PackageId>
+    <Authors>ZB.MOM.WW</Authors>
+    <Description>SQL Server API-key store for the ZB.MOM.WW SCADA family — one shared row set for clustered hosts (central pairs).</Description>
+    <PackageProjectUrl>https://gitea.dohertylan.com/dohertj2/zb-mom-ww-auth</PackageProjectUrl>
+    <RepositoryUrl>https://gitea.dohertylan.com/dohertj2/zb-mom-ww-auth</RepositoryUrl>
+  </PropertyGroup>
+
+  <ItemGroup>
+    <InternalsVisibleTo Include="ZB.MOM.WW.Auth.ApiKeys.SqlServer.Tests" />
+  </ItemGroup>
+
+</Project>

ZB.MOM.WW.Auth/src/ZB.MOM.WW.Auth.ApiKeys/Admin/ApiKeyAdminCommands.cs

@@ -33,10 +33,15 @@ public sealed class ApiKeyAdminCommands
     private readonly IApiKeyAdminStore _adminStore;
     private readonly IApiKeyAuditStore _auditStore;
     private readonly IApiKeyPepperProvider _pepperProvider;
-    private readonly SqliteAuthStoreMigrator _migrator;
+    private readonly IApiKeyStoreMigrator _migrator;
     private readonly TimeProvider _clock;
 
-    /// <summary>Creates the command set over the supplied stores and options.</summary>
+    /// <summary>
+    /// Creates the command set over the supplied stores and options.
+    /// Retained for existing consumers (MxAccessGateway, HistorianGateway, ScadaBridge) that
+    /// construct this type against the concrete SQLite migrator; forwards to the
+    /// <see cref="IApiKeyStoreMigrator"/> overload.
+    /// </summary>
     /// <param name="options">API-key options (token prefix, store path, ...).</param>
     /// <param name="adminStore">Mutating store (create / revoke / rotate / delete / list).</param>
     /// <param name="auditStore">Append-only audit store wired into every mutating verb.</param>
@@ -50,6 +55,25 @@ public sealed class ApiKeyAdminCommands
         IApiKeyPepperProvider pepperProvider,
         SqliteAuthStoreMigrator migrator,
         TimeProvider? clock = null)
+        : this(options, adminStore, auditStore, pepperProvider, (IApiKeyStoreMigrator)migrator, clock)
+    {
+    }
+
+    /// <summary>Creates the command set over the supplied stores and options.</summary>
+    /// <param name="options">API-key options (token prefix, store path, ...).</param>
+    /// <param name="adminStore">Mutating store (create / revoke / rotate / delete / list).</param>
+    /// <param name="auditStore">Append-only audit store wired into every mutating verb.</param>
+    /// <param name="pepperProvider">Resolves the pepper used to hash secrets.</param>
+    /// <param name="migrator">Schema migrator used by <see cref="InitDbAsync"/> — whichever store
+    /// provider's implementation (SQLite, SQL Server, ...) DI has registered.</param>
+    /// <param name="clock">Optional clock; defaults to <see cref="TimeProvider.System"/>.</param>
+    public ApiKeyAdminCommands(
+        ApiKeyOptions options,
+        IApiKeyAdminStore adminStore,
+        IApiKeyAuditStore auditStore,
+        IApiKeyPepperProvider pepperProvider,
+        IApiKeyStoreMigrator migrator,
+        TimeProvider? clock = null)
     {
         ArgumentNullException.ThrowIfNull(options);
         ArgumentNullException.ThrowIfNull(adminStore);

ZB.MOM.WW.Auth/src/ZB.MOM.WW.Auth.ApiKeys/DependencyInjection/ApiKeyMigrationHostedService.cs

@@ -1,17 +1,18 @@
 using Microsoft.Extensions.Hosting;
 using Microsoft.Extensions.Options;
 using ZB.MOM.WW.Auth.Abstractions.ApiKeys;
-using ZB.MOM.WW.Auth.ApiKeys.Sqlite;
 
 namespace ZB.MOM.WW.Auth.ApiKeys.DependencyInjection;
 
 /// <summary>
-/// Runs the API-key SQLite schema migration at application startup when
-/// <see cref="ApiKeyOptions.RunMigrationsOnStartup"/> is <see langword="true"/>.
-/// The migration is idempotent, so repeated restarts are safe.
+/// Runs the registered <see cref="IApiKeyStoreMigrator"/> at application startup when
+/// <see cref="ApiKeyOptions.RunMigrationsOnStartup"/> is <see langword="true"/>. The migrator
+/// registered in DI is whichever store provider's implementation won the <c>TryAdd</c> race —
+/// SQLite by default, or a provider package (e.g. SQL Server) registered before
+/// <c>AddZbApiKeyAuth</c>. The migration is idempotent, so repeated restarts are safe.
 /// </summary>
 internal sealed class ApiKeyMigrationHostedService(
-    SqliteAuthStoreMigrator migrator,
+    IApiKeyStoreMigrator migrator,
     IOptions<ApiKeyOptions> options) : IHostedService
 {
     /// <inheritdoc />

ZB.MOM.WW.Auth/src/ZB.MOM.WW.Auth.ApiKeys/DependencyInjection/ApiKeyServiceCollectionExtensions.cs

@@ -61,10 +61,12 @@ public static class ApiKeyServiceCollectionExtensions
                 sp.GetRequiredService<IApiKeyStore>(),
                 sp.GetRequiredService<IApiKeyPepperProvider>()));
 
-        // Migrator: singleton, constructed from the already-registered connection factory.
-        // Needed before any store operations so the schema exists.
+        // Migrator: the SQLite implementation is the default; a provider package that registers
+        // IApiKeyStoreMigrator (and the three stores) BEFORE this call displaces it — that is the
+        // whole mechanism by which AddZbApiKeyAuthSqlServer swaps the backend.
         services.TryAddSingleton(sp =>
             new SqliteAuthStoreMigrator(sp.GetRequiredService<AuthSqliteConnectionFactory>()));
+        services.TryAddSingleton<IApiKeyStoreMigrator>(sp => sp.GetRequiredService<SqliteAuthStoreMigrator>());
 
         // Hosted service that runs migrations on startup when ApiKeyOptions.RunMigrationsOnStartup.
         services.AddHostedService<ApiKeyMigrationHostedService>();

ZB.MOM.WW.Auth/src/ZB.MOM.WW.Auth.ApiKeys/Sqlite/SqliteAuthStoreMigrator.cs

@@ -1,5 +1,6 @@
 using System.Globalization;
 using Microsoft.Data.Sqlite;
+using ZB.MOM.WW.Auth.Abstractions.ApiKeys;
 
 namespace ZB.MOM.WW.Auth.ApiKeys.Sqlite;
 
@@ -11,7 +12,7 @@ public sealed class AuthStoreMigrationException(string message) : InvalidOperati
 /// is safe to run repeatedly. Refuses to run against a database whose on-disk version
 /// is newer than this build supports.
 /// </summary>
-public sealed class SqliteAuthStoreMigrator(AuthSqliteConnectionFactory connectionFactory)
+public sealed class SqliteAuthStoreMigrator(AuthSqliteConnectionFactory connectionFactory) : IApiKeyStoreMigrator
 {
     /// <summary>Applies the schema migration to the auth store.</summary>
     /// <param name="cancellationToken">Cancellation token.</param>

ZB.MOM.WW.Auth/src/ZB.MOM.WW.Auth.ApiKeys/ZB.MOM.WW.Auth.ApiKeys.csproj

@@ -38,6 +38,9 @@
 
   <ItemGroup>
     <InternalsVisibleTo Include="ZB.MOM.WW.Auth.ApiKeys.Tests" />
+    <!-- Task 5's SqliteApiKeyStoreImporter parses SQLite values with SqliteValueParsing (internal)
+         rather than duplicating the parser. -->
+    <InternalsVisibleTo Include="ZB.MOM.WW.Auth.ApiKeys.SqlServer" />
   </ItemGroup>
 
 </Project>

ZB.MOM.WW.Auth/src/ZB.MOM.WW.Auth.AspNetCore/ServiceCollectionExtensions.cs

@@ -49,11 +49,16 @@ public static class ServiceCollectionExtensions
         // Fail fast at startup on a misconfigured directory rather than on first login.
         services.AddSingleton<IValidateOptions<LdapOptions>, LdapOptionsValidator>();
 
-        // LdapAuthService is stateless: it holds only a snapshot of LdapOptions and a stateless
-        // connection factory, and opens/disposes a connection per call. It is not IDisposable.
-        // Singleton is correct; TryAdd mirrors the pattern in AddZbApiKeyAuth (idempotency).
+        // LdapAuthService opens/disposes a connection per call and is not IDisposable. Its only
+        // mutable state is the sticky endpoint preference used by FallbackServers failover —
+        // which is precisely why the instance must be shared, so a primary-DC outage is paid
+        // for once rather than on every login. Singleton is correct; TryAdd mirrors the pattern
+        // in AddZbApiKeyAuth (idempotency). The logger is resolved with GetService, not
+        // GetRequiredService, so a host without logging configured still resolves the service.
         services.TryAddSingleton<ILdapAuthService>(sp =>
-            new LdapAuthService(sp.GetRequiredService<IOptions<LdapOptions>>().Value));
+            new LdapAuthService(
+                sp.GetRequiredService<IOptions<LdapOptions>>().Value,
+                sp.GetService<Microsoft.Extensions.Logging.ILogger<LdapAuthService>>()));
 
         return services;
     }

ZB.MOM.WW.Auth/src/ZB.MOM.WW.Auth.Ldap/Internal/LdapEndpoint.cs

@@ -0,0 +1,60 @@
+namespace ZB.MOM.WW.Auth.Ldap.Internal;
+
+/// <summary>A single LDAP server endpoint resolved from configuration.</summary>
+internal sealed record LdapEndpoint(string Host, int Port)
+{
+    /// <summary>Renders the endpoint as <c>host:port</c> — the form used in failover log messages.</summary>
+    public override string ToString() => $"{Host}:{Port}";
+}
+
+/// <summary>
+/// Parses <c>FallbackServers</c> entries: <c>"host"</c> (adopts the caller's default port)
+/// or <c>"host:port"</c>. IPv6 literals are not supported — use a DNS name.
+/// </summary>
+internal static class LdapEndpointParser
+{
+    /// <summary>
+    /// Attempts to parse a single <c>FallbackServers</c> entry.
+    /// </summary>
+    /// <param name="entry">The configured entry, <c>"host"</c> or <c>"host:port"</c>. Surrounding whitespace is tolerated.</param>
+    /// <param name="defaultPort">Port adopted when the entry carries no explicit one.</param>
+    /// <param name="endpoint">The parsed endpoint on success; undefined otherwise.</param>
+    /// <param name="error">A human-readable reason on failure; empty on success.</param>
+    /// <returns><see langword="true"/> when the entry parsed.</returns>
+    public static bool TryParse(string entry, int defaultPort, out LdapEndpoint endpoint, out string error)
+    {
+        endpoint = null!;
+        error = "";
+
+        if (string.IsNullOrWhiteSpace(entry))
+        {
+            error = "entry is empty or whitespace";
+            return false;
+        }
+
+        var parts = entry.Trim().Split(':');
+        if (parts.Length > 2)
+        {
+            error = $"entry \"{entry.Trim()}\" contains more than one ':' " +
+                    "(IPv6 literals are not supported — use a DNS name)";
+            return false;
+        }
+
+        var host = parts[0].Trim();
+        if (host.Length == 0)
+        {
+            error = $"entry \"{entry.Trim()}\" has an empty host";
+            return false;
+        }
+
+        var port = defaultPort;
+        if (parts.Length == 2 && (!int.TryParse(parts[1].Trim(), out port) || port is < 1 or > 65535))
+        {
+            error = $"entry \"{entry.Trim()}\" has an invalid port (must be 1–65535)";
+            return false;
+        }
+
+        endpoint = new LdapEndpoint(host, port);
+        return true;
+    }
+}

ZB.MOM.WW.Auth/src/ZB.MOM.WW.Auth.Ldap/Internal/LdapUserName.cs

@@ -0,0 +1,60 @@
+namespace ZB.MOM.WW.Auth.Ldap.Internal;
+
+/// <summary>
+/// Reduces a typed login to the bare account name used by <see cref="Abstractions.Ldap.LdapBindMode.User"/>:
+/// the qualifier is stripped before the name reaches the bind template and the search filter, so audit
+/// rows never carry <c>DOMAIN\jdoe</c> for one sign-in and <c>jdoe</c> for the next. Case is NOT
+/// normalised here: the returned username is canonicalised from the directory entry's
+/// <c>UserNameAttribute</c> value, so <c>JDoe</c> and <c>jdoe</c> also yield one identity; this name is
+/// only the fallback when the entry has no such value.
+/// </summary>
+internal static class LdapUserName
+{
+    /// <summary>
+    /// Trims <paramref name="input"/> and, when <paramref name="acceptDomainQualified"/>, strips one
+    /// leading <c>DOMAIN\</c> or one trailing <c>@suffix</c>. Returns <see langword="false"/> (with
+    /// <paramref name="name"/> empty) for blank input, for both forms at once, for a repeated
+    /// separator, for an empty side, or for any qualified form when <paramref name="acceptDomainQualified"/>
+    /// is <see langword="false"/>.
+    /// </summary>
+    public static bool TryNormalize(string? input, bool acceptDomainQualified, out string name)
+    {
+        name = "";
+        if (string.IsNullOrWhiteSpace(input))
+            return false;
+
+        var s = input.Trim();
+        var slash = s.IndexOf('\\');
+        var at = s.IndexOf('@');
+
+        if (slash >= 0 || at >= 0)
+        {
+            if (!acceptDomainQualified || (slash >= 0 && at >= 0))
+                return false;
+
+            if (slash >= 0)
+            {
+                if (slash == 0 || s.IndexOf('\\', slash + 1) >= 0)
+                    return false;
+                s = s[(slash + 1)..];
+            }
+            else
+            {
+                // Unlike the slash branch (whose extracted suffix is checked for emptiness by the
+                // final length check below), the extracted piece here is the *prefix* — so an empty
+                // domain part (a trailing "@", e.g. "jdoe@") must be rejected explicitly.
+                if (at == s.Length - 1 || s.IndexOf('@', at + 1) >= 0)
+                    return false;
+                s = s[..at];
+            }
+
+            s = s.Trim();
+        }
+
+        if (s.Length == 0)
+            return false;
+
+        name = s;
+        return true;
+    }
+}

ZB.MOM.WW.Auth/src/ZB.MOM.WW.Auth.Ldap/Internal/NovellLdapConnection.cs

@@ -99,23 +99,73 @@ internal sealed class NovellLdapConnection : ILdapConnection
             attributes.ToArray(),
             typesOnly: false);
 
+        return ReadEntries(results);
+    }
+
+    /// <summary>
+    /// Drains a result set into entries, <b>skipping continuation referrals</b>.
+    /// </summary>
+    /// <remarks>
+    /// <para>
+    /// Active Directory answers any search based at a domain root with one or more
+    /// continuation references (to <c>CN=Configuration</c>, <c>DC=ForestDnsZones</c> and
+    /// friends). With referral following disabled — the default, and what we want, since
+    /// chasing a referral would re-bind somewhere we did not authenticate — Novell surfaces
+    /// each one by throwing <see cref="LdapReferralException"/> from <see cref="ILdapSearchResults.Next"/>.
+    /// </para>
+    /// <para>
+    /// A continuation reference is a <em>pointer to another naming context</em>, not an error:
+    /// it says "there may be more results over there", never that this search failed. Letting
+    /// it escape made the whole search stage fail, which <c>LdapAuthService</c> maps to
+    /// <see cref="Abstractions.Ldap.LdapAuthFailure.ServiceAccountBindFailed"/> — so against a
+    /// real AD domain root, every login failed with a system-side error while the directory,
+    /// the credentials and both DCs were perfectly healthy. Verified against
+    /// <c>DC=nam,DC=zimmer,DC=com</c> on 2026-08-12.
+    /// </para>
+    /// <para>
+    /// Skipping is the correct behaviour rather than a workaround: the entries we can see are
+    /// exactly the ones in the naming context we bound to, which is the set the caller asked
+    /// about. Internal so the referral handling can be unit-tested through the
+    /// <see cref="ILdapSearchResults"/> seam without a live directory.
+    /// </para>
+    /// </remarks>
+    internal static List<LdapSearchEntry> ReadEntries(ILdapSearchResults results)
+    {
         var entries = new List<LdapSearchEntry>();
-        while (results.HasMore())
+        var referrals = 0;
+
+        while (true)
         {
-            var entry = results.Next();
-            var attrs = new Dictionary<string, IReadOnlyList<string>>(StringComparer.OrdinalIgnoreCase);
-
-            foreach (LdapAttribute attr in entry.GetAttributeSet())
+            try
             {
-                attrs[attr.Name] = attr.StringValueArray.ToList();
-            }
+                if (!results.HasMore()) break;
 
-            entries.Add(new LdapSearchEntry(entry.Dn, attrs));
+                var entry = results.Next();
+                var attrs = new Dictionary<string, IReadOnlyList<string>>(StringComparer.OrdinalIgnoreCase);
+
+                foreach (LdapAttribute attr in entry.GetAttributeSet())
+                {
+                    attrs[attr.Name] = attr.StringValueArray.ToList();
+                }
+
+                entries.Add(new LdapSearchEntry(entry.Dn, attrs));
+            }
+            catch (LdapException ex) when (ex is LdapReferralException || ex.ResultCode == LdapException.Referral)
+            {
+                // Novell dequeues the reference before throwing, so the enumeration advances and
+                // this terminates. The cap is a belt-and-braces guard: a client that threw without
+                // advancing would otherwise spin forever inside a fail-closed auth path, which is a
+                // far worse failure than under-reporting an implausible number of referrals.
+                if (++referrals > MaxReferrals) break;
+            }
         }
 
         return entries;
     }
 
+    /// <summary>Upper bound on continuation references tolerated in one search — see <see cref="ReadEntries"/>.</summary>
+    private const int MaxReferrals = 1_000;
+
     /// <inheritdoc/>
     public void Dispose()
     {

ZB.MOM.WW.Auth/src/ZB.MOM.WW.Auth.Ldap/LdapAuthService.cs

@@ -1,3 +1,4 @@
+using Microsoft.Extensions.Logging;
 using Novell.Directory.Ldap;
 using ZB.MOM.WW.Auth.Abstractions.Ldap;
 using ZB.MOM.WW.Auth.Ldap.Internal;
@@ -11,6 +12,7 @@ namespace ZB.MOM.WW.Auth.Ldap;
 /// <see cref="ILdapConnection"/> so the logic is unit-testable without a live server.
 /// </summary>
 /// <remarks>
+/// <para>
 /// Fully fail-closed: authentication never throws to the caller — every error, expected
 /// or unexpected, is mapped to a structured <see cref="LdapAuthResult.Fail(LdapAuthFailure)"/>.
 /// A success is only returned for a user that resolved to exactly one entry, whose password
@@ -18,11 +20,42 @@ namespace ZB.MOM.WW.Auth.Ldap;
 /// Service-account bind failures (<see cref="LdapAuthFailure.ServiceAccountBindFailed"/>) are
 /// kept distinct from end-user bind failures (<see cref="LdapAuthFailure.BadCredentials"/>) so
 /// a system misconfiguration is not mistaken for bad user input.
+/// </para>
+/// <para>
+/// When <see cref="LdapOptions.FallbackServers"/> is non-empty the flow above runs against
+/// an ordered endpoint list (primary first, then fallbacks in declared order) and moves on
+/// to the next endpoint ONLY on a system-side failure — connect/TLS, service-account bind,
+/// or search — i.e. when the endpoint gave no authoritative answer. Any answer from a healthy
+/// directory (success, or a user-side failure such as
+/// <see cref="LdapAuthFailure.BadCredentials"/> / <see cref="LdapAuthFailure.UserNotFound"/>)
+/// is final and is never retried elsewhere: retrying a rejected password against a second DC
+/// would turn one bad-password event into several, tripping account lockouts. Endpoint
+/// preference is sticky — the last endpoint that answered is tried first next time — so a
+/// primary outage costs its connect timeout once, not once per login.
+/// </para>
+/// <para>
+/// The service is safe to share (it is registered as a singleton). The sticky index is the
+/// only mutable state; concurrent logins may race on it, which is benign — the worst outcome
+/// is one login starting at a stale endpoint and walking on.
+/// </para>
+/// <para>
+/// With <c>BindMode = User</c> the service-account bind and the user re-bind are replaced by a
+/// single bind as the user (<see cref="LdapOptions.UserBindTemplate"/>); see <see cref="CompleteAsUser"/>.
+/// </para>
 /// </remarks>
 public sealed class LdapAuthService : ILdapAuthService
 {
     private readonly LdapOptions _options;
     private readonly ILdapConnectionFactory _connectionFactory;
+    private readonly IReadOnlyList<LdapEndpoint> _endpoints;
+    private readonly ILogger<LdapAuthService>? _logger;
+
+    /// <summary>
+    /// Index into <see cref="_endpoints"/> of the endpoint tried first. Written on any
+    /// authoritative answer from a non-preferred endpoint. Accessed via
+    /// <see cref="Volatile"/> only; benign races are accepted (see the type remarks).
+    /// </summary>
+    private int _preferredIndex;
 
     /// <summary>
     /// Production constructor: binds against a live directory via the real
@@ -31,9 +64,15 @@ public sealed class LdapAuthService : ILdapAuthService
     /// <see cref="LdapOptions.AllowInsecure"/> bypass) is carried into the factory so each
     /// connection is built with it.
     /// </summary>
-    public LdapAuthService(LdapOptions options)
+    /// <param name="options">The directory connection and search settings.</param>
+    /// <param name="logger">
+    /// Optional. When supplied, endpoint failover is made visible: a Warning per failed
+    /// endpoint hop and an Information line when the sticky preference moves. Omitting it
+    /// (the default) leaves the service silent — hosts without logging still work.
+    /// </param>
+    public LdapAuthService(LdapOptions options, ILogger<LdapAuthService>? logger = null)
         : this(options, new NovellLdapConnectionFactory(
-            options.AllowInsecure, options.ServerCertificateValidationCallback))
+            options.AllowInsecure, options.ServerCertificateValidationCallback), logger)
     {
     }
 
@@ -42,10 +81,33 @@ public sealed class LdapAuthService : ILdapAuthService
     /// so the bind/search logic can be exercised without a live directory. Internal
     /// because the connection seam is an implementation detail.
     /// </summary>
-    internal LdapAuthService(LdapOptions options, ILdapConnectionFactory connectionFactory)
+    internal LdapAuthService(
+        LdapOptions options,
+        ILdapConnectionFactory connectionFactory,
+        ILogger<LdapAuthService>? logger = null)
     {
         _options = options;
         _connectionFactory = connectionFactory;
+        _logger = logger;
+        _endpoints = BuildEndpoints(options);
+    }
+
+    /// <summary>
+    /// Primary first, then fallbacks in declared order. The validator rejects malformed
+    /// entries at boot; unparseable entries are skipped defensively here so a service
+    /// constructed without options validation degrades to the parseable subset rather
+    /// than throwing from a constructor.
+    /// </summary>
+    private static IReadOnlyList<LdapEndpoint> BuildEndpoints(LdapOptions options)
+    {
+        var endpoints = new List<LdapEndpoint> { new(options.Server, options.Port) };
+        foreach (var entry in options.FallbackServers)
+        {
+            if (LdapEndpointParser.TryParse(entry, options.Port, out var endpoint, out _))
+                endpoints.Add(endpoint);
+        }
+
+        return endpoints;
     }
 
     /// <inheritdoc />
@@ -54,7 +116,10 @@ public sealed class LdapAuthService : ILdapAuthService
     /// bind-then-search-then-bind flow is wrapped so that any error — expected or unexpected —
     /// is mapped to a structured <see cref="LdapAuthResult.Fail(LdapAuthFailure)"/>. A
     /// <c>Succeeded == true</c> result is only ever returned when the user resolved to exactly
-    /// one entry, their password verified, AND at least one group was extracted.
+    /// one entry, their password verified, AND at least one group was extracted. With
+    /// <see cref="LdapOptions.FallbackServers"/> configured the flow may run against more than
+    /// one endpoint before answering — see the type-level remarks for the walk and its
+    /// system-side-only failover rule.
     /// </remarks>
     public Task<LdapAuthResult> AuthenticateAsync(string username, string password, CancellationToken ct)
     {
@@ -80,25 +145,105 @@ public sealed class LdapAuthService : ILdapAuthService
         //    filter and the returned result (avoids two identities for one person).
         username = username.Trim();
 
+        // 3b. An empty password must never reach the directory, in EITHER bind mode: AD answers a
+        //     simple bind with an empty password as an UNAUTHENTICATED bind and reports success —
+        //     the user bind (or the ServiceAccount-mode re-bind as the user's DN) would admit anyone.
+        //     Refused here, before any network call, so no consumer depends on its own edge check.
+        if (string.IsNullOrEmpty(password))
+            return LdapAuthResult.Fail(LdapAuthFailure.BadCredentials);
+
+        // 3c. User-bind mode: reduce the name to the bare account name the bind template and the
+        //     search filter share. (The returned username is canonicalised from the entry later.)
+        if (_options.BindMode == LdapBindMode.User
+            && !LdapUserName.TryNormalize(username, _options.AcceptDomainQualifiedNames, out username))
+        {
+            return LdapAuthResult.Fail(LdapAuthFailure.BadCredentials);
+        }
+
+        // 4. Walk the endpoint list, preferred one first. A single-endpoint configuration
+        //    (no FallbackServers) collapses to exactly one attempt — unchanged behaviour.
+        var preferred = Volatile.Read(ref _preferredIndex);
+        if ((uint)preferred >= (uint)_endpoints.Count)
+            preferred = 0; // defensive: the list is immutable, but never index out of it
+
+        // Seeded so an exhausted walk still returns the conservative system-side bucket
+        // even in the impossible case of an empty endpoint list.
+        var lastSystemSideFailure = LdapAuthResult.Fail(LdapAuthFailure.ServiceAccountBindFailed);
+
+        foreach (var index in WalkOrder(_endpoints.Count, preferred))
+        {
+            var result = AuthenticateAgainst(_endpoints[index], username, password);
+
+            // ServiceAccountBindFailed is the (sole) system-side bucket: this endpoint gave
+            // no authoritative answer, so try the next one. Every other outcome — success, or
+            // a user-side failure like UserNotFound/BadCredentials — came from a healthy
+            // directory and is FINAL: re-asking a second DC about a rejected password would
+            // multiply one bad-password event into several against the account lockout policy.
+            if (result.Failure == LdapAuthFailure.ServiceAccountBindFailed)
+            {
+                lastSystemSideFailure = result;
+                continue;
+            }
+
+            // Sticky preference: any authoritative answer proves this endpoint healthy.
+            // Updating only on Succeeded would re-pay the dead primary's connect timeout on
+            // every mistyped password for the whole outage.
+            if (index != preferred)
+            {
+                Volatile.Write(ref _preferredIndex, index);
+                _logger?.LogInformation("LDAP sticky preference moved to {Endpoint}.", _endpoints[index]);
+            }
+
+            return result;
+        }
+
+        // Every endpoint failed system-side. The preference is deliberately left where it was:
+        // a total outage says nothing about which endpoint to prefer once one recovers.
+        return lastSystemSideFailure;
+    }
+
+    /// <summary>Preferred endpoint first, then every other endpoint in declared order.</summary>
+    private static IEnumerable<int> WalkOrder(int count, int preferred)
+    {
+        yield return preferred;
+        for (var i = 0; i < count; i++)
+        {
+            if (i != preferred)
+                yield return i;
+        }
+    }
+
+    /// <summary>
+    /// Runs one full bind-then-search-then-bind attempt against a single endpoint on a fresh
+    /// connection. The fail-closed contract holds per attempt: this never throws, and it
+    /// returns <see cref="LdapAuthFailure.ServiceAccountBindFailed"/> for — and only for —
+    /// system-side faults, which is what the caller reads as "no authoritative answer, try
+    /// the next endpoint".
+    /// </summary>
+    private LdapAuthResult AuthenticateAgainst(LdapEndpoint endpoint, string username, string password)
+    {
         // The whole flow runs inside an outer fail-closed guard: a StageFailure carries an
         // already-mapped failure out of a stage, and any OTHER unexpected exception defaults to
         // the most conservative system-side bucket. Either way the caller gets a structured result.
         try
         {
+            // A fresh connection per attempt: a Novell connection that has failed to connect
+            // or bind is not safely reusable against another host.
             using var conn = _connectionFactory.Create();
 
-            // 4. Open the connection (transport/TLS handling lives in the adapter). The
+            // 5. Open the connection (transport/TLS handling lives in the adapter). The
             //    per-operation timeout (ConnectionTimeoutMs) is applied by the adapter here.
-            //    A failure to connect/upgrade means the directory is unreachable — a
-            //    system-side fault, not the user's, so map it to ServiceAccountBindFailed.
+            //    A failure to connect/upgrade means this endpoint is unreachable — a
+            //    system-side fault, not the user's, so map it to ServiceAccountBindFailed
+            //    and let the caller try the next endpoint.
             // NOTE: the LdapAuthFailure enum has no dedicated DirectoryUnavailable value;
             //       ServiceAccountBindFailed is the closest system-side bucket. A future
             //       Abstractions change could add DirectoryUnavailable to disambiguate.
             try
             {
                 conn.Connect(
-                    _options.Server,
-                    _options.Port,
+                    endpoint.Host,
+                    endpoint.Port,
                     _options.Transport,
                     _options.AllowInsecure,
                     _options.ConnectionTimeoutMs,
@@ -106,110 +251,197 @@ public sealed class LdapAuthService : ILdapAuthService
             }
             catch (LdapException)
             {
-                throw new StageFailure(LdapAuthFailure.ServiceAccountBindFailed);
+                throw new StageFailure(LdapAuthFailure.ServiceAccountBindFailed, "connect");
             }
 
-            // 5. Service-account bind so we can search for the user's DN. A bind failure
-            //    here is a service-account misconfiguration — DISTINCT from a user-credential
-            //    failure — so it maps to ServiceAccountBindFailed.
-            try
-            {
-                conn.Bind(_options.ServiceAccountDn, _options.ServiceAccountPassword);
-            }
-            catch (LdapException)
-            {
-                throw new StageFailure(LdapAuthFailure.ServiceAccountBindFailed);
-            }
-
-            // 6. Search for the user entry by the configured username attribute. A search
-            //    failure is infrastructure (directory unreachable / unhealthy) — system-side.
-            // NOTE: same enum limitation as Connect above; ServiceAccountBindFailed is the
-            //       closest system-side bucket until a DirectoryUnavailable value exists.
-            IReadOnlyList<LdapSearchEntry> entries;
-            try
-            {
-                var filter = $"({_options.UserNameAttribute}={LdapEscaping.Filter(username)})";
-                entries = conn.Search(_options.SearchBase, filter, BuildSearchAttributes());
-            }
-            catch (LdapException)
-            {
-                throw new StageFailure(LdapAuthFailure.ServiceAccountBindFailed);
-            }
-
-            // 7. Require exactly one match. Zero -> UserNotFound; two or more -> AmbiguousUser.
-            //    We never attempt a user bind against an ambiguous DN.
-            if (entries.Count == 0)
-                return LdapAuthResult.Fail(LdapAuthFailure.UserNotFound);
-            if (entries.Count >= 2)
-                return LdapAuthResult.Fail(LdapAuthFailure.AmbiguousUser);
-
-            var entry = entries[0];
-
-            // 8. User bind: re-bind as the resolved DN to verify the password. A bind failure
-            //    here is the end user's bad credentials.
-            try
-            {
-                conn.Bind(entry.Dn, password);
-            }
-            catch (LdapException)
-            {
-                throw new StageFailure(LdapAuthFailure.BadCredentials);
-            }
-
-            // 9. Group extraction. Fail closed: an empty/missing group set, or any error while
-            //    extracting groups, is a GroupLookupFailed — never a zero-group success.
-            IReadOnlyList<string> groups;
-            try
-            {
-                groups = ExtractGroups(entry);
-            }
-            catch
-            {
-                throw new StageFailure(LdapAuthFailure.GroupLookupFailed);
-            }
-
-            if (groups.Count == 0)
-                return LdapAuthResult.Fail(LdapAuthFailure.GroupLookupFailed);
-
-            // 10. Success — and only here, with a verified password and >= 1 group.
-            var displayName = ExtractDisplayName(entry, username);
-            return LdapAuthResult.Success(username, displayName, groups);
+            return _options.BindMode == LdapBindMode.User
+                ? CompleteAsUser(conn, username, password)
+                : CompleteWithServiceAccount(conn, username, password);
         }
         catch (StageFailure stage)
         {
-            // A stage mapped its own failure; surface it as a structured result.
+            // A stage mapped its own failure; surface it as a structured result. Log at Warning
+            // so a silent failover — the failure mode this feature exists to survive — is
+            // visible in the host's log rather than only in a latency graph.
+            _logger?.LogWarning(
+                "LDAP endpoint {Endpoint} failed system-side at {Stage}; treating as no answer.",
+                endpoint, stage.Stage);
             return LdapAuthResult.Fail(stage.Failure);
         }
-        catch
+        catch (Exception ex)
         {
             // Belt-and-braces: ANY unexpected exception fails closed to the most conservative
             // system-side bucket rather than propagating to the caller.
+            _logger?.LogWarning(ex, "LDAP endpoint {Endpoint} failed system-side with an unexpected error.", endpoint);
             return LdapAuthResult.Fail(LdapAuthFailure.ServiceAccountBindFailed);
         }
     }
 
+    /// <summary>Steps 6–11 of the 0.3.0 flow: service-account bind, search, user re-bind, groups.</summary>
+    private LdapAuthResult CompleteWithServiceAccount(ZB.MOM.WW.Auth.Ldap.Internal.ILdapConnection conn, string username, string password)
+    {
+        // 6. Service-account bind so we can search for the user's DN. A bind failure here is a
+        //    service-account misconfiguration — DISTINCT from a user-credential failure — so it
+        //    maps to ServiceAccountBindFailed.
+        try
+        {
+            conn.Bind(_options.ServiceAccountDn, _options.ServiceAccountPassword);
+        }
+        catch (LdapException)
+        {
+            throw new StageFailure(LdapAuthFailure.ServiceAccountBindFailed, "service-bind");
+        }
+
+        // 7–8. Require exactly one match. Zero -> UserNotFound; two or more -> AmbiguousUser.
+        //      Both are authoritative answers from a healthy directory, so they end the walk.
+        //      We never attempt a user bind against an ambiguous DN.
+        var entries = SearchUser(conn, username);
+        if (entries.Count == 0)
+            return LdapAuthResult.Fail(LdapAuthFailure.UserNotFound);
+        if (entries.Count >= 2)
+            return LdapAuthResult.Fail(LdapAuthFailure.AmbiguousUser);
+
+        var entry = entries[0];
+
+        // 9. User bind: re-bind as the resolved DN to verify the password. A bind failure here is
+        //    the end user's bad credentials — the directory answered, so this is returned directly
+        //    and never retried against another endpoint.
+        try
+        {
+            conn.Bind(entry.Dn, password);
+        }
+        catch (LdapException)
+        {
+            return LdapAuthResult.Fail(LdapAuthFailure.BadCredentials);
+        }
+
+        return SucceedFromEntry(entry, username);
+    }
+
+    /// <summary>
+    /// <see cref="LdapBindMode.User"/>: the user's own bind is the authentication; the same
+    /// connection then reads their entry. Only result 49 (invalid credentials — AD's answer for a
+    /// wrong password AND for unknown, locked, disabled or expired accounts) is the user's failure
+    /// and final; any other bind error means this endpoint gave no answer, so the walk moves on.
+    /// </summary>
+    private LdapAuthResult CompleteAsUser(ZB.MOM.WW.Auth.Ldap.Internal.ILdapConnection conn, string username, string password)
+    {
+        var bindName = _options.UserBindTemplate.Replace("{0}", username, StringComparison.Ordinal);
+        try
+        {
+            conn.Bind(bindName, password);
+        }
+        catch (LdapException ex) when (ex.ResultCode == LdapException.InvalidCredentials)
+        {
+            return LdapAuthResult.Fail(LdapAuthFailure.BadCredentials);
+        }
+        catch (LdapException)
+        {
+            throw new StageFailure(LdapAuthFailure.ServiceAccountBindFailed, "user-bind");
+        }
+
+        // After a good bind, anything but exactly one readable entry is a failure to establish the
+        // user's groups — never a success, and never UserNotFound (the directory just accepted them).
+        var entries = SearchUser(conn, username);
+        if (entries.Count != 1)
+            return LdapAuthResult.Fail(LdapAuthFailure.GroupLookupFailed);
+
+        return SucceedFromEntry(entries[0], username);
+    }
+
+    /// <summary>Step 7: search for the user entry; a search fault is system-side.</summary>
+    private IReadOnlyList<LdapSearchEntry> SearchUser(ZB.MOM.WW.Auth.Ldap.Internal.ILdapConnection conn, string username)
+    {
+        // A search failure is infrastructure (directory unreachable / unhealthy) — system-side.
+        // NOTE: the LdapAuthFailure enum has no DirectoryUnavailable value, so a failed search maps
+        //       to ServiceAccountBindFailed, the closest system-side bucket (same limitation as
+        //       Connect), until a DirectoryUnavailable value exists.
+        try
+        {
+            var filter = $"({_options.UserNameAttribute}={LdapEscaping.Filter(username)})";
+            return conn.Search(_options.SearchBase, filter, BuildSearchAttributes());
+        }
+        catch (LdapException)
+        {
+            throw new StageFailure(LdapAuthFailure.ServiceAccountBindFailed, "search");
+        }
+    }
+
+    /// <summary>
+    /// Steps 10–11: fail-closed group extraction, then success. The returned username is the
+    /// entry's own <see cref="LdapOptions.UserNameAttribute"/> value (the directory's spelling), so
+    /// <c>JDoe</c> and <c>jdoe</c> sign in as one identity; the typed (normalised) name is the
+    /// fallback when the entry carries no such value.
+    /// </summary>
+    private LdapAuthResult SucceedFromEntry(LdapSearchEntry entry, string username)
+    {
+        IReadOnlyList<string> groups;
+        try
+        {
+            groups = ExtractGroups(entry);
+        }
+        catch
+        {
+            return LdapAuthResult.Fail(LdapAuthFailure.GroupLookupFailed);
+        }
+
+        if (groups.Count == 0)
+            return LdapAuthResult.Fail(LdapAuthFailure.GroupLookupFailed);
+
+        var canonical = ExtractUserName(entry, username);
+        return LdapAuthResult.Success(canonical, ExtractDisplayName(entry, canonical), groups);
+    }
+
     /// <summary>
     /// Internal control-flow exception that carries an already-mapped <see cref="LdapAuthFailure"/>
     /// out of a stage to the single fail-closed catch site. Never escapes this type.
     /// </summary>
     private sealed class StageFailure : Exception
     {
-        public StageFailure(LdapAuthFailure failure) => Failure = failure;
+        public StageFailure(LdapAuthFailure failure, string stage)
+        {
+            Failure = failure;
+            Stage = stage;
+        }
 
         public LdapAuthFailure Failure { get; }
+
+        /// <summary>Which stage faulted (<c>connect</c> / <c>service-bind</c> / <c>user-bind</c> / <c>search</c>), for diagnostics.</summary>
+        public string Stage { get; }
     }
 
     /// <summary>
-    /// Builds the distinct attribute list requested from the directory. The display-name
-    /// and group attributes are de-duplicated so we never request the same attribute twice
-    /// when an operator configures them to the same value.
+    /// Builds the distinct attribute list requested from the directory: display name, groups and
+    /// the username attribute (read back as the canonical username). They are de-duplicated,
+    /// case-insensitively, so we never request the same attribute twice when an operator
+    /// configures two of them to the same value.
     /// </summary>
     private IReadOnlyList<string> BuildSearchAttributes()
     {
-        if (string.Equals(_options.DisplayNameAttribute, _options.GroupAttribute, StringComparison.OrdinalIgnoreCase))
-            return new[] { _options.DisplayNameAttribute };
+        var attributes = new List<string>(3);
+        foreach (var name in new[] { _options.DisplayNameAttribute, _options.GroupAttribute, _options.UserNameAttribute })
+        {
+            if (!attributes.Contains(name, StringComparer.OrdinalIgnoreCase))
+                attributes.Add(name);
+        }
 
-        return new[] { _options.DisplayNameAttribute, _options.GroupAttribute };
+        return attributes;
+    }
+
+    /// <summary>
+    /// Returns the first non-blank value of the configured username attribute — the directory's
+    /// canonical spelling — falling back to the (already normalised) typed username.
+    /// </summary>
+    private string ExtractUserName(LdapSearchEntry entry, string username)
+    {
+        if (entry.Attributes.TryGetValue(_options.UserNameAttribute, out var values)
+            && values.Count > 0
+            && !string.IsNullOrWhiteSpace(values[0]))
+        {
+            return values[0];
+        }
+
+        return username;
     }
 
     /// <summary>

ZB.MOM.WW.Auth/src/ZB.MOM.WW.Auth.Ldap/LdapOptionsValidator.cs

@@ -10,7 +10,7 @@ namespace ZB.MOM.WW.Auth.Ldap;
 /// </summary>
 /// <remarks>
 /// Validation is skipped entirely when <see cref="LdapOptions.Enabled"/> is <c>false</c>
-/// (a disabled provider's connection fields are inert). When enabled, four conditions
+/// (a disabled provider's connection fields are inert). When enabled, five conditions
 /// are enforced:
 /// <list type="bullet">
 /// <item>plaintext transport (<see cref="LdapTransport.None"/>) is rejected unless
@@ -18,8 +18,13 @@ namespace ZB.MOM.WW.Auth.Ldap;
 /// <item><see cref="LdapOptions.Server"/> must be specified (no sane default host);</item>
 /// <item><see cref="LdapOptions.SearchBase"/> must be specified (the DN root every
 /// search runs against);</item>
-/// <item><see cref="LdapOptions.ServiceAccountDn"/> must be specified — an empty value
-/// would bind anonymously, defeating the search-then-bind authentication flow.</item>
+/// <item>in <see cref="LdapBindMode.ServiceAccount"/> mode <see cref="LdapOptions.ServiceAccountDn"/>
+/// must be specified — an empty value would bind anonymously, defeating the search-then-bind
+/// authentication flow; in <see cref="LdapBindMode.User"/> mode <see cref="LdapOptions.UserBindTemplate"/>
+/// must contain <c>{0}</c> exactly once instead;</item>
+/// <item>every <see cref="LdapOptions.FallbackServers"/> entry must parse as
+/// <c>"host"</c> or <c>"host:port"</c> — a typo'd backup DC must not lie dormant
+/// until the primary outage it exists to survive.</item>
 /// </list>
 /// </remarks>
 public sealed class LdapOptionsValidator : IValidateOptions<LdapOptions>
@@ -60,7 +65,20 @@ public sealed class LdapOptionsValidator : IValidateOptions<LdapOptions>
                 "set it to the search-base DN (e.g. \"dc=example,dc=com\").");
         }
 
-        if (string.IsNullOrWhiteSpace(options.ServiceAccountDn))
+        if (options.BindMode == LdapBindMode.User)
+        {
+            // User mode binds as the signing-in user, so there is no service account to require.
+            // The template is the bind identity: without exactly one {0} every login would bind as
+            // the same fixed name (or as a name with the user spliced in twice).
+            if (CountPlaceholders(options.UserBindTemplate) != 1)
+            {
+                return ValidateOptionsResult.Fail(
+                    $"{nameof(LdapOptions.UserBindTemplate)} must contain {{0}} exactly once when " +
+                    $"{nameof(LdapOptions.BindMode)} is {nameof(LdapBindMode.User)} — e.g. \"{{0}}@example.com\" " +
+                    "or \"EXAMPLE\\{0}\".");
+            }
+        }
+        else if (string.IsNullOrWhiteSpace(options.ServiceAccountDn))
         {
             return ValidateOptionsResult.Fail(
                 $"{nameof(LdapOptions.ServiceAccountDn)} is required but was empty or whitespace — " +
@@ -68,6 +86,33 @@ public sealed class LdapOptionsValidator : IValidateOptions<LdapOptions>
                 "(e.g. \"cn=svc,dc=example,dc=com\").");
         }
 
+        // Fallback endpoints are walked by LdapAuthService on system-side failures. A
+        // malformed entry would be silently skipped there (the walk degrades to the
+        // parseable subset rather than throwing), so it must be caught at boot instead —
+        // otherwise an operator's typo'd backup DC would only surface as a failed
+        // failover during the primary outage it was meant to survive.
+        foreach (var entry in options.FallbackServers)
+        {
+            if (!Internal.LdapEndpointParser.TryParse(entry, options.Port, out _, out var error))
+            {
+                return ValidateOptionsResult.Fail(
+                    $"{nameof(LdapOptions.FallbackServers)} contains an invalid entry — {error}. " +
+                    "Each entry must be \"host\" or \"host:port\" (port 1–65535).");
+            }
+        }
+
         return ValidateOptionsResult.Success;
     }
+
+    private static int CountPlaceholders(string? template)
+    {
+        if (string.IsNullOrEmpty(template))
+            return 0;
+
+        var count = 0;
+        for (var i = template.IndexOf("{0}", StringComparison.Ordinal); i >= 0;
+             i = template.IndexOf("{0}", i + 3, StringComparison.Ordinal))
+            count++;
+        return count;
+    }
 }

ZB.MOM.WW.Auth/src/ZB.MOM.WW.Auth.Ldap/ZB.MOM.WW.Auth.Ldap.csproj

@@ -22,6 +22,7 @@
   <ItemGroup>
     <PackageReference Include="Novell.Directory.Ldap.NETStandard" />
     <PackageReference Include="Microsoft.Extensions.Options" />
+    <PackageReference Include="Microsoft.Extensions.Logging.Abstractions" />
   </ItemGroup>
 
   <ItemGroup>

ZB.MOM.WW.Auth/tests/ZB.MOM.WW.Auth.ApiKeys.SqlServer.Tests/Live/LiveSqlServerFixture.cs

@@ -0,0 +1,101 @@
+using Microsoft.Data.SqlClient;
+
+namespace ZB.MOM.WW.Auth.ApiKeys.SqlServer.Tests.Live;
+
+/// <summary>
+/// Shared connection details for the env-gated live SQL Server suite.
+/// </summary>
+/// <remarks>
+/// Gated on <c>AUTH_SQLSERVER_CONNSTR</c> and skips cleanly when it is unset, matching the family
+/// live-test idiom (see <c>ZB.MOM.WW.Secrets.Replicator.SqlServer</c>) — the offline suite must stay
+/// runnable on any machine. Set it to a database the test is allowed to create and drop a schema in,
+/// for example <c>Server=localhost,1433;Database=ZbAuthTest;User Id=sa;Password=...;TrustServerCertificate=true</c>.
+/// Each test class works in a throwaway <em>schema</em> rather than a throwaway database: it needs no
+/// CREATE DATABASE right, and a virgin schema still reproduces the CREATE SCHEMA race the migrator's
+/// retry loop exists for.
+/// </remarks>
+public static class LiveSqlServer
+{
+    /// <summary>Environment variable holding the live connection string.</summary>
+    public const string ConnectionStringVariable = "AUTH_SQLSERVER_CONNSTR";
+
+    /// <summary>The configured connection string, or <see langword="null"/> when the suite is not enabled.</summary>
+    public static string? ConnectionString =>
+        Environment.GetEnvironmentVariable(ConnectionStringVariable);
+
+    /// <summary>Whether the live suite should run.</summary>
+    public static bool IsEnabled => !string.IsNullOrWhiteSpace(ConnectionString);
+
+    /// <summary>Skip reason shown when the suite is not enabled.</summary>
+    public const string SkipReason =
+        "Live SQL Server suite disabled; set AUTH_SQLSERVER_CONNSTR to enable.";
+
+    /// <summary>
+    /// Builds options against a throwaway schema so parallel runs and repeat runs cannot collide,
+    /// and so a failed run leaves no trace in a shared database.
+    /// </summary>
+    /// <param name="schemaName">The unique schema name for this test.</param>
+    /// <returns>Options targeting that schema.</returns>
+    public static SqlServerApiKeyOptions OptionsFor(string schemaName) => new()
+    {
+        ConnectionString = ConnectionString!,
+        Schema = schemaName,
+    };
+
+    /// <summary>Generates a unique, allow-list-legal schema name for one test.</summary>
+    /// <returns>A fresh schema name.</returns>
+    public static string NewSchemaName() => $"zbauth_test_{Guid.NewGuid():N}"[..32];
+
+    /// <summary>Drops the throwaway schema and its three tables.</summary>
+    /// <param name="schemaName">The schema to drop.</param>
+    /// <returns>A task that completes when the schema is gone.</returns>
+    public static async Task DropSchemaAsync(string schemaName)
+    {
+        if (!IsEnabled)
+        {
+            return;
+        }
+
+        await using var connection = new SqlConnection(ConnectionString);
+        await connection.OpenAsync();
+
+        await using SqlCommand command = connection.CreateCommand();
+        command.CommandText = $"""
+            IF OBJECT_ID(N'[{schemaName}].[api_key_audit]', N'U') IS NOT NULL DROP TABLE [{schemaName}].[api_key_audit];
+            IF OBJECT_ID(N'[{schemaName}].[api_keys]', N'U') IS NOT NULL DROP TABLE [{schemaName}].[api_keys];
+            IF OBJECT_ID(N'[{schemaName}].[schema_version]', N'U') IS NOT NULL DROP TABLE [{schemaName}].[schema_version];
+            IF EXISTS (SELECT 1 FROM sys.schemas WHERE name = N'{schemaName}') EXEC(N'DROP SCHEMA [{schemaName}]');
+            """;
+
+        await command.ExecuteNonQueryAsync();
+    }
+
+    /// <summary>Runs a scalar query against the live database.</summary>
+    /// <param name="commandText">The query text.</param>
+    /// <returns>The first column of the first row, or <see langword="null"/>.</returns>
+    public static async Task<object?> ScalarAsync(string commandText)
+    {
+        await using var connection = new SqlConnection(ConnectionString);
+        await connection.OpenAsync();
+
+        await using SqlCommand command = connection.CreateCommand();
+        command.CommandText = commandText;
+
+        object? value = await command.ExecuteScalarAsync();
+        return value == DBNull.Value ? null : value;
+    }
+
+    /// <summary>Runs a non-query statement against the live database.</summary>
+    /// <param name="commandText">The statement text.</param>
+    /// <returns>A task that completes when the statement has run.</returns>
+    public static async Task ExecuteAsync(string commandText)
+    {
+        await using var connection = new SqlConnection(ConnectionString);
+        await connection.OpenAsync();
+
+        await using SqlCommand command = connection.CreateCommand();
+        command.CommandText = commandText;
+
+        await command.ExecuteNonQueryAsync();
+    }
+}

ZB.MOM.WW.Auth/tests/ZB.MOM.WW.Auth.ApiKeys.SqlServer.Tests/Live/SqlServerApiKeyAdminStoreLiveTests.cs

@@ -0,0 +1,499 @@
+using ZB.MOM.WW.Auth.Abstractions.ApiKeys;
+
+namespace ZB.MOM.WW.Auth.ApiKeys.SqlServer.Tests.Live;
+
+/// <summary>
+/// Live contract tests for <see cref="SqlServerApiKeyAdminStore"/> (and the audit store it is
+/// exercised alongside).
+/// </summary>
+/// <remarks>
+/// A one-for-one port of <c>SqliteApiKeyAdminStoreTests</c> — same test names, same assertions — so
+/// the create/revoke/rotate/scope/enable/delete semantics are pinned identically across the two
+/// providers. xUnit constructs a new test-class instance per test, so each test provisions and drops
+/// its own throwaway schema.
+/// </remarks>
+[Collection("live-sqlserver")]
+public sealed class SqlServerApiKeyAdminStoreLiveTests : IAsyncLifetime
+{
+    private readonly string _schema = LiveSqlServer.NewSchemaName();
+
+    private AuthSqlServerConnectionFactory _factory = null!;
+    private SqlServerApiKeyAdminStore _admin = null!;
+    private SqlServerApiKeyStore _read = null!;
+    private SqlServerApiKeyAuditStore _audit = null!;
+
+    /// <inheritdoc />
+    public async Task InitializeAsync()
+    {
+        if (!LiveSqlServer.IsEnabled)
+        {
+            return;
+        }
+
+        _factory = new AuthSqlServerConnectionFactory(LiveSqlServer.OptionsFor(_schema));
+        await new SqlServerApiKeyStoreMigrator(_factory).MigrateAsync(CancellationToken.None);
+        _admin = new SqlServerApiKeyAdminStore(_factory);
+        _read = new SqlServerApiKeyStore(_factory);
+        _audit = new SqlServerApiKeyAuditStore(_factory);
+    }
+
+    /// <inheritdoc />
+    public Task DisposeAsync() => LiveSqlServer.DropSchemaAsync(_schema);
+
+    // --- Create ---
+
+    [SkippableFact]
+    public async Task Create_ThenFindByKeyId_ReturnsRecord()
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        ApiKeyRecord record = SampleRecord("key-1");
+
+        await _admin.CreateAsync(record, CancellationToken.None);
+
+        ApiKeyRecord? found = await _read.FindByKeyIdAsync("key-1", CancellationToken.None);
+        Assert.NotNull(found);
+        Assert.Equal(record.SecretHash, found!.SecretHash);
+        Assert.True(record.Scopes.SetEquals(found.Scopes));
+        Assert.Equal(record.ConstraintsJson, found.ConstraintsJson);
+        Assert.Null(found.LastUsedUtc);
+        Assert.Null(found.RevokedUtc);
+    }
+
+    [SkippableFact]
+    public async Task Create_WithExpiresUtc_RoundTripsThroughFindAndList()
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        var expiry = new DateTimeOffset(2026, 12, 31, 23, 59, 59, TimeSpan.Zero);
+        ApiKeyRecord record = SampleRecord("key-exp") with { ExpiresUtc = expiry };
+
+        await _admin.CreateAsync(record, CancellationToken.None);
+
+        ApiKeyRecord? found = await _read.FindByKeyIdAsync("key-exp", CancellationToken.None);
+        Assert.Equal(expiry, found!.ExpiresUtc);
+
+        IReadOnlyList<ApiKeyListItem> listed = await _admin.ListAsync(CancellationToken.None);
+        ApiKeyListItem item = Assert.Single(listed, k => k.KeyId == "key-exp");
+        Assert.Equal(expiry, item.ExpiresUtc);
+    }
+
+    [SkippableFact]
+    public async Task Create_WithoutExpiresUtc_ReadsBackAsNull()
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        await _admin.CreateAsync(SampleRecord("key-1"), CancellationToken.None);
+
+        ApiKeyRecord? found = await _read.FindByKeyIdAsync("key-1", CancellationToken.None);
+        Assert.Null(found!.ExpiresUtc);
+
+        IReadOnlyList<ApiKeyListItem> listed = await _admin.ListAsync(CancellationToken.None);
+        Assert.Null(Assert.Single(listed, k => k.KeyId == "key-1").ExpiresUtc);
+    }
+
+    // --- Revoke ---
+
+    [SkippableFact]
+    public async Task Revoke_ActiveKey_SetsRevokedAndFindActiveReturnsNull()
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        await _admin.CreateAsync(SampleRecord("key-1"), CancellationToken.None);
+        var when = new DateTimeOffset(2026, 5, 31, 9, 0, 0, TimeSpan.Zero);
+
+        bool result = await _admin.RevokeAsync("key-1", when, CancellationToken.None);
+
+        Assert.True(result);
+        Assert.Null(await _read.FindActiveByKeyIdAsync("key-1", CancellationToken.None));
+        ApiKeyRecord? found = await _read.FindByKeyIdAsync("key-1", CancellationToken.None);
+        Assert.Equal(when, found!.RevokedUtc);
+    }
+
+    [SkippableFact]
+    public async Task Revoke_UnknownKey_ReturnsFalse()
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        bool result = await _admin.RevokeAsync("missing", DateTimeOffset.UtcNow, CancellationToken.None);
+
+        Assert.False(result);
+    }
+
+    [SkippableFact]
+    public async Task Revoke_AlreadyRevoked_ReturnsFalse()
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        await _admin.CreateAsync(SampleRecord("key-1"), CancellationToken.None);
+        await _admin.RevokeAsync("key-1", DateTimeOffset.UtcNow, CancellationToken.None);
+
+        bool result = await _admin.RevokeAsync("key-1", DateTimeOffset.UtcNow, CancellationToken.None);
+
+        Assert.False(result);
+    }
+
+    // --- Rotate ---
+
+    [SkippableFact]
+    public async Task Rotate_ChangesHashAndReactivates()
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        await _admin.CreateAsync(SampleRecord("key-1"), CancellationToken.None);
+        await _admin.RevokeAsync("key-1", DateTimeOffset.UtcNow, CancellationToken.None);
+        await _read.MarkUsedAsync("key-1", DateTimeOffset.UtcNow, CancellationToken.None);
+        byte[] newHash = [9, 9, 9, 9];
+
+        bool result = await _admin.RotateAsync("key-1", newHash, CancellationToken.None);
+
+        Assert.True(result);
+        ApiKeyRecord? found = await _read.FindByKeyIdAsync("key-1", CancellationToken.None);
+        Assert.Equal(newHash, found!.SecretHash);
+        Assert.Null(found.RevokedUtc);
+        Assert.Null(found.LastUsedUtc);
+        // Reactivated: now visible as active.
+        Assert.NotNull(await _read.FindActiveByKeyIdAsync("key-1", CancellationToken.None));
+    }
+
+    [SkippableFact]
+    public async Task Rotate_UnknownKey_ReturnsFalse()
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        bool result = await _admin.RotateAsync("missing", [1], CancellationToken.None);
+
+        Assert.False(result);
+    }
+
+    // --- SetScopes ---
+
+    [SkippableFact]
+    public async Task SetScopesAsync_ReplacesScopes_AndReturnsTrue()
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        await _admin.CreateAsync(
+            SampleRecord("key-1") with { Scopes = new HashSet<string>(["a"], StringComparer.Ordinal) },
+            CancellationToken.None);
+
+        bool result = await _admin.SetScopesAsync(
+            "key-1",
+            new HashSet<string>(["b", "c"], StringComparer.Ordinal),
+            CancellationToken.None);
+
+        Assert.True(result);
+        IReadOnlyList<ApiKeyListItem> listed = await _admin.ListAsync(CancellationToken.None);
+        ApiKeyListItem item = Assert.Single(listed, k => k.KeyId == "key-1");
+        Assert.True(item.Scopes.SetEquals(new HashSet<string>(["b", "c"], StringComparer.Ordinal)));
+    }
+
+    [SkippableFact]
+    public async Task SetScopesAsync_UnknownKey_ReturnsFalse()
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        bool result = await _admin.SetScopesAsync(
+            "missing",
+            new HashSet<string>(["b"], StringComparer.Ordinal),
+            CancellationToken.None);
+
+        Assert.False(result);
+    }
+
+    // --- SetEnabled ---
+
+    [SkippableFact]
+    public async Task SetEnabledAsync_False_DisablesKey()
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        await _admin.CreateAsync(SampleRecord("key-1"), CancellationToken.None);
+        var when = new DateTimeOffset(2026, 5, 31, 9, 0, 0, TimeSpan.Zero);
+
+        bool result = await _admin.SetEnabledAsync("key-1", enabled: false, when, CancellationToken.None);
+
+        Assert.True(result);
+        Assert.Null(await _read.FindActiveByKeyIdAsync("key-1", CancellationToken.None));
+        ApiKeyRecord? found = await _read.FindByKeyIdAsync("key-1", CancellationToken.None);
+        Assert.Equal(when, found!.RevokedUtc);
+    }
+
+    [SkippableFact]
+    public async Task SetEnabledAsync_True_ReenablesKey_WithoutChangingSecret()
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        ApiKeyRecord original = SampleRecord("key-1");
+        await _admin.CreateAsync(original, CancellationToken.None);
+        // Record some usage so we can prove last_used_utc is left untouched on re-enable.
+        var used = new DateTimeOffset(2026, 5, 20, 12, 0, 0, TimeSpan.Zero);
+        await _read.MarkUsedAsync("key-1", used, CancellationToken.None);
+
+        // Disable, then re-enable.
+        await _admin.SetEnabledAsync(
+            "key-1", enabled: false, new DateTimeOffset(2026, 5, 31, 9, 0, 0, TimeSpan.Zero), CancellationToken.None);
+        bool result = await _admin.SetEnabledAsync(
+            "key-1", enabled: true, new DateTimeOffset(2026, 6, 1, 9, 0, 0, TimeSpan.Zero), CancellationToken.None);
+
+        Assert.True(result);
+
+        // Active again, and the secret hash + last-used timestamp are unchanged.
+        ApiKeyRecord? active = await _read.FindActiveByKeyIdAsync("key-1", CancellationToken.None);
+        Assert.NotNull(active);
+        Assert.True(active!.SecretHash.SequenceEqual(original.SecretHash));
+        Assert.Null(active.RevokedUtc);
+        Assert.Equal(used, active.LastUsedUtc);
+    }
+
+    [SkippableFact]
+    public async Task SetEnabledAsync_UnknownKey_ReturnsFalse()
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        bool result = await _admin.SetEnabledAsync(
+            "missing", enabled: false, DateTimeOffset.UtcNow, CancellationToken.None);
+
+        Assert.False(result);
+    }
+
+    // --- Delete ---
+
+    [SkippableFact]
+    public async Task Delete_ActiveKey_ReturnsFalseAndKeyStillPresent()
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        await _admin.CreateAsync(SampleRecord("key-1"), CancellationToken.None);
+
+        bool result = await _admin.DeleteAsync("key-1", CancellationToken.None);
+
+        Assert.False(result);
+        Assert.NotNull(await _read.FindByKeyIdAsync("key-1", CancellationToken.None));
+    }
+
+    [SkippableFact]
+    public async Task Delete_RevokedKey_ReturnsTrueAndKeyGone()
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        await _admin.CreateAsync(SampleRecord("key-1"), CancellationToken.None);
+        await _admin.RevokeAsync("key-1", DateTimeOffset.UtcNow, CancellationToken.None);
+
+        bool result = await _admin.DeleteAsync("key-1", CancellationToken.None);
+
+        Assert.True(result);
+        Assert.Null(await _read.FindByKeyIdAsync("key-1", CancellationToken.None));
+    }
+
+    [SkippableFact]
+    public async Task Delete_UnknownKey_ReturnsFalse()
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        bool result = await _admin.DeleteAsync("missing", CancellationToken.None);
+
+        Assert.False(result);
+    }
+
+    // --- keyId guard tests ---
+
+    [SkippableTheory]
+    [InlineData(null)]
+    [InlineData("")]
+    [InlineData("   ")]
+    public async Task Revoke_NullOrWhitespaceKeyId_ThrowsArgumentException(string? keyId)
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        // ArgumentNullException (null) and ArgumentException (empty/whitespace) are both acceptable;
+        // ThrowIfNullOrWhiteSpace throws ArgumentNullException for null, ArgumentException for whitespace.
+        await Assert.ThrowsAnyAsync<ArgumentException>(
+            () => _admin.RevokeAsync(keyId!, DateTimeOffset.UtcNow, CancellationToken.None));
+    }
+
+    [SkippableTheory]
+    [InlineData(null)]
+    [InlineData("")]
+    [InlineData("   ")]
+    public async Task Rotate_NullOrWhitespaceKeyId_ThrowsArgumentException(string? keyId)
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        await Assert.ThrowsAnyAsync<ArgumentException>(
+            () => _admin.RotateAsync(keyId!, [1, 2, 3], CancellationToken.None));
+    }
+
+    [SkippableTheory]
+    [InlineData(null)]
+    [InlineData("")]
+    [InlineData("   ")]
+    public async Task Delete_NullOrWhitespaceKeyId_ThrowsArgumentException(string? keyId)
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        await Assert.ThrowsAnyAsync<ArgumentException>(
+            () => _admin.DeleteAsync(keyId!, CancellationToken.None));
+    }
+
+    [SkippableTheory]
+    [InlineData(null)]
+    [InlineData("")]
+    [InlineData("   ")]
+    public async Task SetScopesAsync_NullOrWhitespaceKeyId_ThrowsArgumentException(string? keyId)
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        await Assert.ThrowsAnyAsync<ArgumentException>(
+            () => _admin.SetScopesAsync(
+                keyId!,
+                new HashSet<string>(["read"], StringComparer.Ordinal),
+                CancellationToken.None));
+    }
+
+    [SkippableTheory]
+    [InlineData(null)]
+    [InlineData("")]
+    [InlineData("   ")]
+    public async Task SetEnabledAsync_NullOrWhitespaceKeyId_ThrowsArgumentException(string? keyId)
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        await Assert.ThrowsAnyAsync<ArgumentException>(
+            () => _admin.SetEnabledAsync(keyId!, enabled: false, DateTimeOffset.UtcNow, CancellationToken.None));
+    }
+
+    [SkippableFact]
+    public async Task SetScopesAsync_NullScopes_ThrowsArgumentNullException()
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        await _admin.CreateAsync(SampleRecord("key-1"), CancellationToken.None);
+
+        await Assert.ThrowsAsync<ArgumentNullException>(
+            () => _admin.SetScopesAsync("key-1", null!, CancellationToken.None));
+    }
+
+    // --- SetEnabled idempotence ---
+
+    [SkippableFact]
+    public async Task SetEnabledAsync_OnAlreadyActiveKey_ReturnsTrue()
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        await _admin.CreateAsync(SampleRecord("key-1"), CancellationToken.None);
+
+        bool result = await _admin.SetEnabledAsync(
+            "key-1", enabled: true, DateTimeOffset.UtcNow, CancellationToken.None);
+
+        Assert.True(result);
+        ApiKeyRecord? active = await _read.FindActiveByKeyIdAsync("key-1", CancellationToken.None);
+        Assert.NotNull(active);
+        Assert.Null(active!.RevokedUtc);
+    }
+
+    [SkippableFact]
+    public async Task SetEnabledAsync_OnAlreadyDisabledKey_OverwritesTimestamp_ReturnsTrue()
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        await _admin.CreateAsync(SampleRecord("key-1"), CancellationToken.None);
+        var t1 = new DateTimeOffset(2026, 5, 1, 10, 0, 0, TimeSpan.Zero);
+        var t2 = new DateTimeOffset(2026, 5, 15, 10, 0, 0, TimeSpan.Zero);
+
+        // Disable at t1.
+        await _admin.SetEnabledAsync("key-1", enabled: false, t1, CancellationToken.None);
+
+        // Disable again at a later t2 (idempotent overwrite — no guard on revoked_utc).
+        bool result = await _admin.SetEnabledAsync("key-1", enabled: false, t2, CancellationToken.None);
+
+        Assert.True(result);
+        IReadOnlyList<ApiKeyListItem> listed = await _admin.ListAsync(CancellationToken.None);
+        ApiKeyListItem item = Assert.Single(listed, k => k.KeyId == "key-1");
+        Assert.Equal(t2, item.RevokedUtc);
+    }
+
+    // --- Audit ---
+
+    [SkippableFact]
+    public async Task Audit_AppendThenListRecent_ReturnsEntry()
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        var entry = new ApiKeyAuditEntry(
+            KeyId: "key-1",
+            EventType: "created",
+            RemoteAddress: "10.0.0.1",
+            CreatedUtc: new DateTimeOffset(2026, 5, 31, 10, 0, 0, TimeSpan.Zero),
+            Details: "by admin");
+
+        await _audit.AppendAsync(entry, CancellationToken.None);
+
+        IReadOnlyList<ApiKeyAuditEntry> recent = await _audit.ListRecentAsync(10, CancellationToken.None);
+        Assert.Single(recent);
+        Assert.Equal("key-1", recent[0].KeyId);
+        Assert.Equal("created", recent[0].EventType);
+        Assert.Equal("10.0.0.1", recent[0].RemoteAddress);
+        Assert.Equal("by admin", recent[0].Details);
+    }
+
+    [SkippableFact]
+    public async Task Audit_ListRecent_ReturnsNewestFirst()
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        await _audit.AppendAsync(
+            new ApiKeyAuditEntry("k", "first", null, DateTimeOffset.UtcNow, null), CancellationToken.None);
+        await _audit.AppendAsync(
+            new ApiKeyAuditEntry("k", "second", null, DateTimeOffset.UtcNow, null), CancellationToken.None);
+        await _audit.AppendAsync(
+            new ApiKeyAuditEntry("k", "third", null, DateTimeOffset.UtcNow, null), CancellationToken.None);
+
+        IReadOnlyList<ApiKeyAuditEntry> recent = await _audit.ListRecentAsync(10, CancellationToken.None);
+
+        Assert.Equal(["third", "second", "first"], recent.Select(e => e.EventType));
+    }
+
+    [SkippableFact]
+    public async Task Audit_ListRecent_RespectsLimit()
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        for (int i = 0; i < 5; i++)
+        {
+            await _audit.AppendAsync(
+                new ApiKeyAuditEntry("k", $"e{i}", null, DateTimeOffset.UtcNow, null), CancellationToken.None);
+        }
+
+        IReadOnlyList<ApiKeyAuditEntry> recent = await _audit.ListRecentAsync(2, CancellationToken.None);
+
+        Assert.Equal(2, recent.Count);
+        Assert.Equal(["e4", "e3"], recent.Select(e => e.EventType));
+    }
+
+    [SkippableFact]
+    public async Task Audit_NullableFields_RoundTripAsNull()
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        await _audit.AppendAsync(
+            new ApiKeyAuditEntry(null, "anon", null, DateTimeOffset.UtcNow, null), CancellationToken.None);
+
+        IReadOnlyList<ApiKeyAuditEntry> recent = await _audit.ListRecentAsync(10, CancellationToken.None);
+        Assert.Single(recent);
+        Assert.Null(recent[0].KeyId);
+        Assert.Null(recent[0].RemoteAddress);
+        Assert.Null(recent[0].Details);
+    }
+
+    private static ApiKeyRecord SampleRecord(string keyId) => new(
+        KeyId: keyId,
+        KeyPrefix: "mxgw_ab12",
+        SecretHash: [1, 2, 3, 4, 5, 6, 7, 8],
+        DisplayName: "Test Key " + keyId,
+        Scopes: new HashSet<string>(["read", "write"], StringComparer.Ordinal),
+        ConstraintsJson: """{"ipAllow":["10.0.0.0/8"]}""",
+        CreatedUtc: new DateTimeOffset(2026, 5, 1, 8, 30, 0, TimeSpan.Zero),
+        LastUsedUtc: null,
+        RevokedUtc: null);
+}

ZB.MOM.WW.Auth/tests/ZB.MOM.WW.Auth.ApiKeys.SqlServer.Tests/Live/SqlServerApiKeyAuditStoreLiveTests.cs

@@ -0,0 +1,74 @@
+using ZB.MOM.WW.Auth.Abstractions.ApiKeys;
+
+namespace ZB.MOM.WW.Auth.ApiKeys.SqlServer.Tests.Live;
+
+/// <summary>
+/// Live tests for <see cref="SqlServerApiKeyAuditStore"/> that the ported SQLite suite does not
+/// already cover: newest-first ordering under a limit, and the non-positive-limit short circuit.
+/// </summary>
+/// <remarks>
+/// xUnit constructs a new test-class instance per test, so each test provisions and drops its own
+/// throwaway schema.
+/// </remarks>
+[Collection("live-sqlserver")]
+public sealed class SqlServerApiKeyAuditStoreLiveTests : IAsyncLifetime
+{
+    private readonly string _schema = LiveSqlServer.NewSchemaName();
+
+    private SqlServerApiKeyAuditStore _audit = null!;
+
+    /// <inheritdoc />
+    public async Task InitializeAsync()
+    {
+        if (!LiveSqlServer.IsEnabled)
+        {
+            return;
+        }
+
+        var factory = new AuthSqlServerConnectionFactory(LiveSqlServer.OptionsFor(_schema));
+        await new SqlServerApiKeyStoreMigrator(factory).MigrateAsync(CancellationToken.None);
+        _audit = new SqlServerApiKeyAuditStore(factory);
+    }
+
+    /// <inheritdoc />
+    public Task DisposeAsync() => LiveSqlServer.DropSchemaAsync(_schema);
+
+    [SkippableFact]
+    public async Task Append_ThenListRecent_ReturnsNewestFirstWithinLimit()
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        // Every row carries the SAME created_utc, so the newest-first order can only come from the
+        // IDENTITY audit_id the ORDER BY names — the timestamp cannot accidentally supply it.
+        var when = new DateTimeOffset(2026, 7, 1, 0, 0, 0, TimeSpan.Zero);
+        for (int i = 0; i < 6; i++)
+        {
+            await _audit.AppendAsync(
+                new ApiKeyAuditEntry("k", $"e{i}", "10.0.0.1", when, $"detail {i}"),
+                CancellationToken.None);
+        }
+
+        IReadOnlyList<ApiKeyAuditEntry> recent = await _audit.ListRecentAsync(3, CancellationToken.None);
+
+        Assert.Equal(3, recent.Count);
+        Assert.Equal(["e5", "e4", "e3"], recent.Select(e => e.EventType));
+        Assert.Equal("detail 5", recent[0].Details);
+        Assert.Equal(when, recent[0].CreatedUtc);
+    }
+
+    [SkippableFact]
+    public async Task ListRecent_NonPositiveLimit_ReturnsEmpty()
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        await _audit.AppendAsync(
+            new ApiKeyAuditEntry("k", "created", null, DateTimeOffset.UtcNow, null),
+            CancellationToken.None);
+
+        // A non-positive limit short circuits WITHOUT touching the database — SELECT TOP (0) would
+        // be legal but SELECT TOP (-1) is a syntax error, so the guard is load-bearing here in a way
+        // it is not for SQLite's LIMIT.
+        Assert.Empty(await _audit.ListRecentAsync(0, CancellationToken.None));
+        Assert.Empty(await _audit.ListRecentAsync(-5, CancellationToken.None));
+    }
+}

ZB.MOM.WW.Auth/tests/ZB.MOM.WW.Auth.ApiKeys.SqlServer.Tests/Live/SqlServerApiKeyStoreLiveTests.cs

@@ -0,0 +1,340 @@
+using System.Data;
+using Microsoft.Data.SqlClient;
+using ZB.MOM.WW.Auth.Abstractions.ApiKeys;
+using ZB.MOM.WW.Auth.ApiKeys.Sqlite;
+
+namespace ZB.MOM.WW.Auth.ApiKeys.SqlServer.Tests.Live;
+
+/// <summary>
+/// Live contract tests for <see cref="SqlServerApiKeyStore"/>.
+/// </summary>
+/// <remarks>
+/// A one-for-one port of <c>SqliteApiKeyStoreTests</c> (same test names, same assertions) so the two
+/// providers are pinned to identical semantics: whichever store a host composes, a row means the
+/// same thing. xUnit constructs a new test-class instance per test, so each test provisions and
+/// drops its own throwaway schema — the isolation the SQLite suite gets from a per-test temp file.
+/// </remarks>
+[Collection("live-sqlserver")]
+public sealed class SqlServerApiKeyStoreLiveTests : IAsyncLifetime
+{
+    private readonly string _schema = LiveSqlServer.NewSchemaName();
+
+    private AuthSqlServerConnectionFactory _factory = null!;
+    private SqlServerApiKeyStore _store = null!;
+
+    /// <inheritdoc />
+    public async Task InitializeAsync()
+    {
+        if (!LiveSqlServer.IsEnabled)
+        {
+            return;
+        }
+
+        _factory = new AuthSqlServerConnectionFactory(LiveSqlServer.OptionsFor(_schema));
+        await new SqlServerApiKeyStoreMigrator(_factory).MigrateAsync(CancellationToken.None);
+        _store = new SqlServerApiKeyStore(_factory);
+    }
+
+    /// <inheritdoc />
+    public Task DisposeAsync() => LiveSqlServer.DropSchemaAsync(_schema);
+
+    [SkippableFact]
+    public async Task FindByKeyId_AfterInsert_ReturnsEqualRecord()
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        ApiKeyRecord record = SampleRecord("key-1");
+        await InsertAsync(record);
+
+        ApiKeyRecord? found = await _store.FindByKeyIdAsync("key-1", CancellationToken.None);
+
+        Assert.NotNull(found);
+        AssertRecordEqual(record, found!);
+    }
+
+    [SkippableFact]
+    public async Task FindByKeyId_ReturnsRevokedRecord()
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        ApiKeyRecord record = SampleRecord("key-revoked") with
+        {
+            RevokedUtc = new DateTimeOffset(2026, 1, 2, 3, 4, 5, TimeSpan.Zero),
+        };
+        await InsertAsync(record);
+
+        ApiKeyRecord? found = await _store.FindByKeyIdAsync("key-revoked", CancellationToken.None);
+
+        Assert.NotNull(found);
+        Assert.NotNull(found!.RevokedUtc);
+    }
+
+    [SkippableFact]
+    public async Task FindActiveByKeyId_RevokedKey_ReturnsNull()
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        ApiKeyRecord record = SampleRecord("key-revoked") with
+        {
+            RevokedUtc = new DateTimeOffset(2026, 1, 2, 3, 4, 5, TimeSpan.Zero),
+        };
+        await InsertAsync(record);
+
+        ApiKeyRecord? found = await _store.FindActiveByKeyIdAsync("key-revoked", CancellationToken.None);
+
+        Assert.Null(found);
+    }
+
+    [SkippableFact]
+    public async Task FindActiveByKeyId_ActiveKey_ReturnsRecord()
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        await InsertAsync(SampleRecord("key-active"));
+
+        ApiKeyRecord? found = await _store.FindActiveByKeyIdAsync("key-active", CancellationToken.None);
+
+        Assert.NotNull(found);
+    }
+
+    [SkippableFact]
+    public async Task FindByKeyId_UnknownKey_ReturnsNull()
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        ApiKeyRecord? found = await _store.FindByKeyIdAsync("missing", CancellationToken.None);
+
+        Assert.Null(found);
+    }
+
+    [SkippableFact]
+    public async Task MarkUsed_ActiveKey_UpdatesLastUsed()
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        await InsertAsync(SampleRecord("key-active"));
+        var when = new DateTimeOffset(2026, 5, 31, 12, 0, 0, TimeSpan.Zero);
+
+        await _store.MarkUsedAsync("key-active", when, CancellationToken.None);
+
+        ApiKeyRecord? found = await _store.FindByKeyIdAsync("key-active", CancellationToken.None);
+        Assert.Equal(when, found!.LastUsedUtc);
+    }
+
+    [SkippableFact]
+    public async Task MarkUsed_RevokedKey_DoesNotUpdateLastUsed()
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        ApiKeyRecord record = SampleRecord("key-revoked") with
+        {
+            RevokedUtc = new DateTimeOffset(2026, 1, 2, 3, 4, 5, TimeSpan.Zero),
+        };
+        await InsertAsync(record);
+        var when = new DateTimeOffset(2026, 5, 31, 12, 0, 0, TimeSpan.Zero);
+
+        await _store.MarkUsedAsync("key-revoked", when, CancellationToken.None);
+
+        ApiKeyRecord? found = await _store.FindByKeyIdAsync("key-revoked", CancellationToken.None);
+        Assert.Null(found!.LastUsedUtc);
+    }
+
+    // --- SQL-Server-specific: DATETIMEOFFSET(7) must not truncate to milliseconds ---
+
+    [SkippableFact]
+    public async Task FindByKeyId_RoundTripsDateTimeOffsetToTheTick()
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        // Sub-millisecond ticks: SQLite persists round-trip ("O") text and keeps them for free,
+        // whereas SQL Server keeps them only because the columns are DATETIMEOFFSET(7) and the
+        // parameters are bound as SqlDbType.DateTimeOffset. A DATETIME/DATETIME2(3) column or an
+        // untyped parameter would silently round here.
+        var created = new DateTimeOffset(2026, 3, 4, 5, 6, 7, TimeSpan.Zero).AddTicks(1234567);
+        var expires = new DateTimeOffset(2027, 3, 4, 5, 6, 7, TimeSpan.Zero).AddTicks(7654321);
+        ApiKeyRecord record = SampleRecord("key-ticks") with { CreatedUtc = created, ExpiresUtc = expires };
+        await InsertAsync(record);
+
+        ApiKeyRecord? found = await _store.FindByKeyIdAsync("key-ticks", CancellationToken.None);
+
+        Assert.NotNull(found);
+        Assert.Equal(created, found!.CreatedUtc);
+        Assert.Equal(created.Ticks, found.CreatedUtc.Ticks);
+        Assert.Equal(expires, found.ExpiresUtc);
+        Assert.Equal(expires.Ticks, found.ExpiresUtc!.Value.Ticks);
+    }
+
+    // --- keyId guard tests ---
+
+    [SkippableTheory]
+    [InlineData(null)]
+    [InlineData("")]
+    [InlineData("   ")]
+    public async Task FindByKeyId_NullOrWhitespaceKeyId_ThrowsArgumentException(string? keyId)
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        // ArgumentNullException (null) and ArgumentException (empty/whitespace) are both acceptable;
+        // ThrowIfNullOrWhiteSpace throws ArgumentNullException for null, ArgumentException for whitespace.
+        await Assert.ThrowsAnyAsync<ArgumentException>(
+            () => _store.FindByKeyIdAsync(keyId!, CancellationToken.None));
+    }
+
+    [SkippableTheory]
+    [InlineData(null)]
+    [InlineData("")]
+    [InlineData("   ")]
+    public async Task FindActiveByKeyId_NullOrWhitespaceKeyId_ThrowsArgumentException(string? keyId)
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        await Assert.ThrowsAnyAsync<ArgumentException>(
+            () => _store.FindActiveByKeyIdAsync(keyId!, CancellationToken.None));
+    }
+
+    [SkippableTheory]
+    [InlineData(null)]
+    [InlineData("")]
+    [InlineData("   ")]
+    public async Task MarkUsed_NullOrWhitespaceKeyId_ThrowsArgumentException(string? keyId)
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        await Assert.ThrowsAnyAsync<ArgumentException>(
+            () => _store.MarkUsedAsync(keyId!, DateTimeOffset.UtcNow, CancellationToken.None));
+    }
+
+    [SkippableFact]
+    public void ScopeSerializer_RoundTripsAndSortsOrdinally()
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        var unsorted = new HashSet<string>(["zeta", "alpha", "mike"], StringComparer.Ordinal);
+        var differentOrder = new HashSet<string>(["mike", "zeta", "alpha"], StringComparer.Ordinal);
+
+        string a = ScopeSerializer.Serialize(unsorted);
+        string b = ScopeSerializer.Serialize(differentOrder);
+
+        // Equal sets must produce identical column text regardless of insertion order.
+        Assert.Equal(a, b);
+        Assert.Equal("""["alpha","mike","zeta"]""", a);
+
+        IReadOnlySet<string> roundTripped = ScopeSerializer.Deserialize(a);
+        Assert.True(roundTripped.SetEquals(unsorted));
+    }
+
+    [SkippableFact]
+    public void ScopeSerializer_DeserializeNullOrEmpty_ReturnsEmptySet()
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        Assert.Empty(ScopeSerializer.Deserialize(null));
+        Assert.Empty(ScopeSerializer.Deserialize(""));
+    }
+
+    // --- Auth-003: corrupt scopes JSON must fail closed (empty set), never throw JsonException ---
+
+    [SkippableTheory]
+    [InlineData("not json at all")]
+    [InlineData("{")]
+    [InlineData("{\"a\":1}")]   // valid JSON, but an object, not a string[]
+    [InlineData("42")]          // valid JSON, but a number
+    [InlineData("[\"read\",")]  // truncated/partial write
+    public void ScopeSerializer_DeserializeMalformed_ReturnsEmptySet_DoesNotThrow(string value)
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        // A poisoned scopes column (tampering, partial write, format change, buggy writer) must
+        // degrade to a zero-scope set rather than throwing on the verification hot path.
+        IReadOnlySet<string> scopes = ScopeSerializer.Deserialize(value);
+        Assert.Empty(scopes);
+    }
+
+    [SkippableFact]
+    public async Task FindByKeyId_CorruptScopesColumn_ReturnsRecordWithEmptyScopes_DoesNotThrow()
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        // Insert a row whose scopes column holds malformed (non-array) JSON, then read it through
+        // the store. The store must NOT propagate a JsonException out of FindByKeyIdAsync (which the
+        // verifier relies on for its "only exception path is cancellation" contract).
+        await InsertWithRawScopesAsync("key-corrupt", scopesJson: "{ this is not valid json");
+
+        ApiKeyRecord? found = await _store.FindByKeyIdAsync("key-corrupt", CancellationToken.None);
+
+        Assert.NotNull(found);
+        Assert.Empty(found!.Scopes);
+    }
+
+    private static ApiKeyRecord SampleRecord(string keyId) => new(
+        KeyId: keyId,
+        KeyPrefix: "mxgw_ab12",
+        SecretHash: [1, 2, 3, 4, 5, 6, 7, 8],
+        DisplayName: "Test Key " + keyId,
+        Scopes: new HashSet<string>(["read", "write"], StringComparer.Ordinal),
+        ConstraintsJson: """{"ipAllow":["10.0.0.0/8"]}""",
+        CreatedUtc: new DateTimeOffset(2026, 5, 1, 8, 30, 0, TimeSpan.Zero),
+        LastUsedUtc: null,
+        RevokedUtc: null);
+
+    private static void AssertRecordEqual(ApiKeyRecord expected, ApiKeyRecord actual)
+    {
+        Assert.Equal(expected.KeyId, actual.KeyId);
+        Assert.Equal(expected.KeyPrefix, actual.KeyPrefix);
+        Assert.Equal(expected.SecretHash, actual.SecretHash);
+        Assert.Equal(expected.DisplayName, actual.DisplayName);
+        Assert.True(expected.Scopes.SetEquals(actual.Scopes));
+        Assert.Equal(expected.ConstraintsJson, actual.ConstraintsJson);
+        Assert.Equal(expected.CreatedUtc, actual.CreatedUtc);
+        Assert.Equal(expected.LastUsedUtc, actual.LastUsedUtc);
+        Assert.Equal(expected.RevokedUtc, actual.RevokedUtc);
+    }
+
+    private async Task InsertAsync(ApiKeyRecord record)
+    {
+        await using SqlConnection connection =
+            await _factory.OpenConnectionAsync(CancellationToken.None);
+        await using SqlCommand command = _factory.CreateCommand(connection, $"""
+            INSERT INTO [{_schema}].[{SqlServerAuthSchema.ApiKeysTable}] (
+                key_id, key_prefix, secret_hash, display_name, scopes,
+                constraints, created_utc, last_used_utc, revoked_utc, expires_utc)
+            VALUES (
+                @key_id, @key_prefix, @secret_hash, @display_name, @scopes,
+                @constraints, @created_utc, @last_used_utc, @revoked_utc, @expires_utc);
+            """);
+        command.Parameters.Add("@key_id", SqlDbType.NVarChar, 64).Value = record.KeyId;
+        command.Parameters.Add("@key_prefix", SqlDbType.NVarChar, 16).Value = record.KeyPrefix;
+        command.Parameters.Add("@secret_hash", SqlDbType.VarBinary, 64).Value = record.SecretHash;
+        command.Parameters.Add("@display_name", SqlDbType.NVarChar, 256).Value = record.DisplayName;
+        command.Parameters.Add("@scopes", SqlDbType.NVarChar, -1).Value = ScopeSerializer.Serialize(record.Scopes);
+        command.Parameters.Add("@constraints", SqlDbType.NVarChar, -1).Value = (object?)record.ConstraintsJson ?? DBNull.Value;
+        command.Parameters.Add("@created_utc", SqlDbType.DateTimeOffset).Value = record.CreatedUtc;
+        command.Parameters.Add("@last_used_utc", SqlDbType.DateTimeOffset).Value = (object?)record.LastUsedUtc ?? DBNull.Value;
+        command.Parameters.Add("@revoked_utc", SqlDbType.DateTimeOffset).Value = (object?)record.RevokedUtc ?? DBNull.Value;
+        command.Parameters.Add("@expires_utc", SqlDbType.DateTimeOffset).Value = (object?)record.ExpiresUtc ?? DBNull.Value;
+        await command.ExecuteNonQueryAsync(CancellationToken.None);
+    }
+
+    private async Task InsertWithRawScopesAsync(string keyId, string scopesJson)
+    {
+        // Writes the scopes column verbatim (NOT via ScopeSerializer.Serialize) so a malformed
+        // value can be persisted to simulate tampering / a partial or buggy write.
+        await using SqlConnection connection =
+            await _factory.OpenConnectionAsync(CancellationToken.None);
+        await using SqlCommand command = _factory.CreateCommand(connection, $"""
+            INSERT INTO [{_schema}].[{SqlServerAuthSchema.ApiKeysTable}] (
+                key_id, key_prefix, secret_hash, display_name, scopes,
+                constraints, created_utc, last_used_utc, revoked_utc, expires_utc)
+            VALUES (
+                @key_id, @key_prefix, @secret_hash, @display_name, @scopes,
+                NULL, @created_utc, NULL, NULL, NULL);
+            """);
+        command.Parameters.Add("@key_id", SqlDbType.NVarChar, 64).Value = keyId;
+        command.Parameters.Add("@key_prefix", SqlDbType.NVarChar, 16).Value = "mxgw";
+        command.Parameters.Add("@secret_hash", SqlDbType.VarBinary, 64).Value = new byte[] { 1, 2, 3 };
+        command.Parameters.Add("@display_name", SqlDbType.NVarChar, 256).Value = "Corrupt Key";
+        command.Parameters.Add("@scopes", SqlDbType.NVarChar, -1).Value = scopesJson;
+        command.Parameters.Add("@created_utc", SqlDbType.DateTimeOffset).Value = DateTimeOffset.UnixEpoch;
+        await command.ExecuteNonQueryAsync(CancellationToken.None);
+    }
+}

ZB.MOM.WW.Auth/tests/ZB.MOM.WW.Auth.ApiKeys.SqlServer.Tests/Live/SqlServerApiKeyStoreMigratorLiveTests.cs

@@ -0,0 +1,145 @@
+using System.Globalization;
+using ZB.MOM.WW.Auth.ApiKeys.Sqlite;
+
+namespace ZB.MOM.WW.Auth.ApiKeys.SqlServer.Tests.Live;
+
+/// <summary>
+/// Live provisioning tests for <see cref="SqlServerApiKeyStoreMigrator"/>.
+/// </summary>
+/// <remarks>
+/// Every test provisions its own throwaway schema — the concurrency case in particular needs a
+/// virgin one, because the race the retry loop exists for only happens while the schema and tables
+/// do not yet exist. All schemas created by the class are dropped in <see cref="DisposeAsync"/>.
+/// </remarks>
+[Collection("live-sqlserver")]
+public sealed class SqlServerApiKeyStoreMigratorLiveTests : IAsyncLifetime
+{
+    private readonly List<string> _schemas = [];
+
+    /// <inheritdoc />
+    public Task InitializeAsync() => Task.CompletedTask;
+
+    /// <inheritdoc />
+    public async Task DisposeAsync()
+    {
+        foreach (string schema in _schemas)
+        {
+            await LiveSqlServer.DropSchemaAsync(schema);
+        }
+    }
+
+    private string NewSchema()
+    {
+        string schema = LiveSqlServer.NewSchemaName();
+        _schemas.Add(schema);
+        return schema;
+    }
+
+    private static SqlServerApiKeyStoreMigrator MigratorFor(string schema) =>
+        new(new AuthSqlServerConnectionFactory(LiveSqlServer.OptionsFor(schema)));
+
+    [SkippableFact]
+    public async Task Migrate_OnEmptyDatabase_CreatesSchemaTablesAndVersionRow()
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        string schema = NewSchema();
+
+        await MigratorFor(schema).MigrateAsync(CancellationToken.None);
+
+        Assert.Equal(
+            1,
+            Convert.ToInt32(
+                await LiveSqlServer.ScalarAsync(
+                    $"SELECT COUNT(*) FROM sys.schemas WHERE name = N'{schema}'"),
+                CultureInfo.InvariantCulture));
+
+        foreach (string table in new[]
+        {
+            SqlServerAuthSchema.SchemaVersionTable,
+            SqlServerAuthSchema.ApiKeysTable,
+            SqlServerAuthSchema.ApiKeyAuditTable,
+        })
+        {
+            Assert.NotNull(
+                await LiveSqlServer.ScalarAsync($"SELECT OBJECT_ID(N'[{schema}].[{table}]', N'U')"));
+        }
+
+        Assert.Equal(
+            SqlServerAuthSchema.CurrentVersion,
+            Convert.ToInt32(
+                await LiveSqlServer.ScalarAsync(
+                    $"SELECT version FROM [{schema}].[{SqlServerAuthSchema.SchemaVersionTable}] WHERE id = 1"),
+                CultureInfo.InvariantCulture));
+    }
+
+    [SkippableFact]
+    public async Task Migrate_Twice_IsIdempotent()
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        string schema = NewSchema();
+        SqlServerApiKeyStoreMigrator migrator = MigratorFor(schema);
+
+        await migrator.MigrateAsync(CancellationToken.None);
+        await migrator.MigrateAsync(CancellationToken.None);
+
+        Assert.Equal(
+            1,
+            Convert.ToInt32(
+                await LiveSqlServer.ScalarAsync(
+                    $"SELECT COUNT(*) FROM [{schema}].[{SqlServerAuthSchema.SchemaVersionTable}]"),
+                CultureInfo.InvariantCulture));
+        Assert.Equal(
+            SqlServerAuthSchema.CurrentVersion,
+            Convert.ToInt32(
+                await LiveSqlServer.ScalarAsync(
+                    $"SELECT version FROM [{schema}].[{SqlServerAuthSchema.SchemaVersionTable}] WHERE id = 1"),
+                CultureInfo.InvariantCulture));
+    }
+
+    [SkippableFact]
+    public async Task Migrate_Concurrently_FromEightTasks_AllSucceed()
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        // A virgin schema provisioned by eight nodes at once — the central-pair first boot, scaled
+        // up. Each task gets its own factory (and therefore its own connection) so the DDL really
+        // does run on eight sessions.
+        string schema = NewSchema();
+
+        await Task.WhenAll(Enumerable.Range(0, 8).Select(_ =>
+            Task.Run(() => MigratorFor(schema).MigrateAsync(CancellationToken.None))));
+
+        Assert.Equal(
+            1,
+            Convert.ToInt32(
+                await LiveSqlServer.ScalarAsync(
+                    $"SELECT COUNT(*) FROM [{schema}].[{SqlServerAuthSchema.SchemaVersionTable}]"),
+                CultureInfo.InvariantCulture));
+        Assert.Equal(
+            SqlServerAuthSchema.CurrentVersion,
+            Convert.ToInt32(
+                await LiveSqlServer.ScalarAsync(
+                    $"SELECT version FROM [{schema}].[{SqlServerAuthSchema.SchemaVersionTable}] WHERE id = 1"),
+                CultureInfo.InvariantCulture));
+    }
+
+    [SkippableFact]
+    public async Task Migrate_WhenStoredVersionIsNewer_Throws()
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        string schema = NewSchema();
+        SqlServerApiKeyStoreMigrator migrator = MigratorFor(schema);
+
+        await migrator.MigrateAsync(CancellationToken.None);
+        await LiveSqlServer.ExecuteAsync(
+            $"UPDATE [{schema}].[{SqlServerAuthSchema.SchemaVersionTable}] SET version = 99 WHERE id = 1");
+
+        AuthStoreMigrationException exception = await Assert.ThrowsAsync<AuthStoreMigrationException>(
+            () => migrator.MigrateAsync(CancellationToken.None));
+
+        Assert.Contains("99", exception.Message, StringComparison.Ordinal);
+    }
+}

ZB.MOM.WW.Auth/tests/ZB.MOM.WW.Auth.ApiKeys.SqlServer.Tests/Live/SqliteApiKeyStoreImporterLiveTests.cs

@@ -0,0 +1,283 @@
+using Microsoft.Data.Sqlite;
+using ZB.MOM.WW.Auth.Abstractions.ApiKeys;
+using ZB.MOM.WW.Auth.ApiKeys.Sqlite;
+
+namespace ZB.MOM.WW.Auth.ApiKeys.SqlServer.Tests.Live;
+
+/// <summary>
+/// Live tests for <see cref="SqliteApiKeyStoreImporter"/> — the one-shot per-node SQLite to shared
+/// SQL Server copy.
+/// </summary>
+/// <remarks>
+/// Each test owns a throwaway SQL Server schema AND a throwaway SQLite file under the temp
+/// directory; both are torn down in <see cref="DisposeAsync"/>, so a failed run leaves nothing
+/// behind on a shared database or the developer's disk.
+/// </remarks>
+[Collection("live-sqlserver")]
+public sealed class SqliteApiKeyStoreImporterLiveTests : IAsyncLifetime
+{
+    private readonly string _schema = LiveSqlServer.NewSchemaName();
+    private readonly string _sqlitePath =
+        Path.Combine(Path.GetTempPath(), $"zbauth-import-{Guid.NewGuid():N}.db");
+
+    private AuthSqlServerConnectionFactory _factory = null!;
+    private SqlServerApiKeyAdminStore _sqlAdmin = null!;
+    private SqlServerApiKeyStore _sqlRead = null!;
+    private SqlServerApiKeyAuditStore _sqlAudit = null!;
+    private SqliteApiKeyStoreImporter _importer = null!;
+
+    /// <inheritdoc />
+    public async Task InitializeAsync()
+    {
+        if (!LiveSqlServer.IsEnabled)
+        {
+            return;
+        }
+
+        _factory = new AuthSqlServerConnectionFactory(LiveSqlServer.OptionsFor(_schema));
+        await new SqlServerApiKeyStoreMigrator(_factory).MigrateAsync(CancellationToken.None);
+        _sqlAdmin = new SqlServerApiKeyAdminStore(_factory);
+        _sqlRead = new SqlServerApiKeyStore(_factory);
+        _sqlAudit = new SqlServerApiKeyAuditStore(_factory);
+        _importer = new SqliteApiKeyStoreImporter(_factory);
+    }
+
+    /// <inheritdoc />
+    public async Task DisposeAsync()
+    {
+        // SQLite pools connections by connection string; the handle must be released before the
+        // file can be removed cleanly.
+        SqliteConnection.ClearAllPools();
+
+        foreach (string path in new[] { _sqlitePath, _sqlitePath + "-wal", _sqlitePath + "-shm" })
+        {
+            if (File.Exists(path))
+            {
+                File.Delete(path);
+            }
+        }
+
+        await LiveSqlServer.DropSchemaAsync(_schema);
+    }
+
+    [SkippableFact]
+    public async Task Import_CopiesKeysAndAuditRows_AndReturnsCounts()
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        (ApiKeyRecord keyA, ApiKeyRecord keyB) = await SeedSqliteAsync();
+
+        ApiKeyImportResult result = await _importer.ImportAsync(_sqlitePath, CancellationToken.None);
+
+        Assert.True(result.SourceFound);
+        Assert.Equal(2, result.KeysInserted);
+        Assert.Equal(0, result.KeysSkipped);
+        Assert.Equal(3, result.AuditInserted);
+        Assert.Equal(0, result.AuditSkipped);
+
+        await AssertRecordMatchesAsync(keyA);
+        await AssertRecordMatchesAsync(keyB);
+
+        IReadOnlyList<ApiKeyAuditEntry> audit = await _sqlAudit.ListRecentAsync(10, CancellationToken.None);
+        Assert.Equal(3, audit.Count);
+        Assert.Contains(audit, e => e.KeyId is null && e.EventType == "auth_failed");
+        Assert.Contains(audit, e => e.KeyId == "key-a" && e.EventType == "created");
+        Assert.Contains(audit, e => e.KeyId == "key-b" && e.EventType == "created");
+    }
+
+    [SkippableFact]
+    public async Task Import_Twice_IsIdempotent()
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        await SeedSqliteAsync();
+
+        await _importer.ImportAsync(_sqlitePath, CancellationToken.None);
+        ApiKeyImportResult second = await _importer.ImportAsync(_sqlitePath, CancellationToken.None);
+
+        Assert.True(second.SourceFound);
+        Assert.Equal(0, second.KeysInserted);
+        Assert.Equal(2, second.KeysSkipped);
+        Assert.Equal(0, second.AuditInserted);
+        Assert.Equal(3, second.AuditSkipped);
+
+        Assert.Equal(2, (await _sqlAdmin.ListAsync(CancellationToken.None)).Count);
+        Assert.Equal(3, (await _sqlAudit.ListRecentAsync(10, CancellationToken.None)).Count);
+    }
+
+    [SkippableFact]
+    public async Task Import_UnionsWithExistingRows()
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        var lastUsed = new DateTimeOffset(2026, 7, 4, 12, 0, 0, TimeSpan.Zero);
+        ApiKeyRecord keyC = SampleRecord("key-c") with { LastUsedUtc = lastUsed };
+        await _sqlAdmin.CreateAsync(keyC, CancellationToken.None);
+
+        await SeedSqliteAsync();
+
+        ApiKeyImportResult result = await _importer.ImportAsync(_sqlitePath, CancellationToken.None);
+
+        Assert.Equal(2, result.KeysInserted);
+        Assert.Equal(0, result.KeysSkipped);
+
+        IReadOnlyList<ApiKeyListItem> listed = await _sqlAdmin.ListAsync(CancellationToken.None);
+        Assert.Equal(
+            ["key-a", "key-b", "key-c"],
+            listed.Select(k => k.KeyId).OrderBy(k => k, StringComparer.Ordinal));
+
+        // The pre-existing row is untouched — the importer only ever inserts.
+        Assert.Equal(lastUsed, Assert.Single(listed, k => k.KeyId == "key-c").LastUsedUtc);
+    }
+
+    [SkippableFact]
+    public async Task Import_MissingFile_ReturnsZeroWithoutThrowing()
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        ApiKeyImportResult result = await _importer.ImportAsync(
+            Path.Combine(Path.GetTempPath(), $"zbauth-absent-{Guid.NewGuid():N}.db"),
+            CancellationToken.None);
+
+        Assert.False(result.SourceFound);
+        Assert.Equal(0, result.KeysInserted);
+        Assert.Equal(0, result.KeysSkipped);
+        Assert.Equal(0, result.AuditInserted);
+        Assert.Equal(0, result.AuditSkipped);
+    }
+
+    [SkippableFact]
+    public async Task Import_PreV3FileWithoutExpiresColumn_ReadsExpiryAsNull()
+    {
+        Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
+
+        await CreatePreV3SqliteFileAsync();
+
+        ApiKeyImportResult result = await _importer.ImportAsync(_sqlitePath, CancellationToken.None);
+
+        Assert.True(result.SourceFound);
+        Assert.Equal(1, result.KeysInserted);
+
+        ApiKeyRecord? found = await _sqlRead.FindByKeyIdAsync("key-old", CancellationToken.None);
+        Assert.NotNull(found);
+        Assert.Null(found!.ExpiresUtc);
+        Assert.Equal("legacy key", found.DisplayName);
+        Assert.Equal(new DateTimeOffset(2026, 1, 2, 3, 4, 5, TimeSpan.Zero), found.CreatedUtc);
+    }
+
+    // --- helpers ---
+
+    /// <summary>
+    /// Seeds the throwaway SQLite file with two keys (one revoked, one with an expiry) and three
+    /// audit rows (one of them key-less), then returns the two key records as SQLite holds them.
+    /// </summary>
+    private async Task<(ApiKeyRecord KeyA, ApiKeyRecord KeyB)> SeedSqliteAsync()
+    {
+        var sqliteFactory = new AuthSqliteConnectionFactory(_sqlitePath);
+        await new SqliteAuthStoreMigrator(sqliteFactory).MigrateAsync(CancellationToken.None);
+
+        var admin = new SqliteApiKeyAdminStore(sqliteFactory);
+        var audit = new SqliteApiKeyAuditStore(sqliteFactory);
+        var read = new SqliteApiKeyStore(sqliteFactory);
+
+        ApiKeyRecord keyA = SampleRecord("key-a");
+        ApiKeyRecord keyB = SampleRecord("key-b") with
+        {
+            ExpiresUtc = new DateTimeOffset(2027, 3, 1, 0, 0, 0, TimeSpan.Zero),
+        };
+
+        await admin.CreateAsync(keyA, CancellationToken.None);
+        await admin.CreateAsync(keyB, CancellationToken.None);
+        await admin.RevokeAsync("key-a", new DateTimeOffset(2026, 6, 1, 7, 8, 9, TimeSpan.Zero), CancellationToken.None);
+
+        await audit.AppendAsync(
+            new ApiKeyAuditEntry("key-a", "created", "10.0.0.1", new DateTimeOffset(2026, 5, 1, 8, 30, 0, TimeSpan.Zero), "by admin"),
+            CancellationToken.None);
+        await audit.AppendAsync(
+            new ApiKeyAuditEntry("key-b", "created", null, new DateTimeOffset(2026, 5, 1, 8, 31, 0, TimeSpan.Zero), null),
+            CancellationToken.None);
+        await audit.AppendAsync(
+            new ApiKeyAuditEntry(null, "auth_failed", "10.0.0.9", new DateTimeOffset(2026, 5, 2, 9, 0, 0, TimeSpan.Zero), "unknown key"),
+            CancellationToken.None);
+
+        // Re-read so the assertions compare against exactly what SQLite holds, revocation included.
+        ApiKeyRecord storedA = (await read.FindByKeyIdAsync("key-a", CancellationToken.None))!;
+        ApiKeyRecord storedB = (await read.FindByKeyIdAsync("key-b", CancellationToken.None))!;
+
+        return (storedA, storedB);
+    }
+
+    /// <summary>
+    /// Writes a v2-shaped SQLite file by hand — <c>api_keys</c> WITHOUT <c>expires_utc</c> — and
+    /// inserts one key row, reproducing a store created before schema v3.
+    /// </summary>
+    private async Task CreatePreV3SqliteFileAsync()
+    {
+        await using var connection = new SqliteConnection(
+            new SqliteConnectionStringBuilder { DataSource = _sqlitePath }.ToString());
+        await connection.OpenAsync();
+
+        await using SqliteCommand ddl = connection.CreateCommand();
+        ddl.CommandText = """
+            CREATE TABLE api_keys (
+                key_id TEXT PRIMARY KEY,
+                key_prefix TEXT NOT NULL,
+                secret_hash BLOB NOT NULL,
+                display_name TEXT NOT NULL,
+                scopes TEXT NOT NULL,
+                constraints TEXT NULL,
+                created_utc TEXT NOT NULL,
+                last_used_utc TEXT NULL,
+                revoked_utc TEXT NULL
+            );
+            CREATE TABLE api_key_audit (
+                audit_id INTEGER PRIMARY KEY AUTOINCREMENT,
+                key_id TEXT NULL,
+                event_type TEXT NOT NULL,
+                remote_address TEXT NULL,
+                created_utc TEXT NOT NULL,
+                details TEXT NULL
+            );
+            """;
+        await ddl.ExecuteNonQueryAsync();
+
+        await using SqliteCommand insert = connection.CreateCommand();
+        insert.CommandText = """
+            INSERT INTO api_keys (
+                key_id, key_prefix, secret_hash, display_name, scopes,
+                constraints, created_utc, last_used_utc, revoked_utc)
+            VALUES ('key-old', 'sbk_old1', $hash, 'legacy key', 'read', NULL, $created, NULL, NULL);
+            """;
+        insert.Parameters.Add("$hash", SqliteType.Blob).Value = new byte[] { 4, 5, 6 };
+        insert.Parameters.AddWithValue(
+            "$created", new DateTimeOffset(2026, 1, 2, 3, 4, 5, TimeSpan.Zero).ToString("O"));
+        await insert.ExecuteNonQueryAsync();
+    }
+
+    private async Task AssertRecordMatchesAsync(ApiKeyRecord expected)
+    {
+        ApiKeyRecord? actual = await _sqlRead.FindByKeyIdAsync(expected.KeyId, CancellationToken.None);
+
+        Assert.NotNull(actual);
+        Assert.Equal(expected.KeyPrefix, actual!.KeyPrefix);
+        Assert.True(expected.SecretHash.SequenceEqual(actual.SecretHash));
+        Assert.Equal(expected.DisplayName, actual.DisplayName);
+        Assert.True(expected.Scopes.SetEquals(actual.Scopes));
+        Assert.Equal(expected.ConstraintsJson, actual.ConstraintsJson);
+        Assert.Equal(expected.CreatedUtc, actual.CreatedUtc);
+        Assert.Equal(expected.LastUsedUtc, actual.LastUsedUtc);
+        Assert.Equal(expected.RevokedUtc, actual.RevokedUtc);
+        Assert.Equal(expected.ExpiresUtc, actual.ExpiresUtc);
+    }
+
+    private static ApiKeyRecord SampleRecord(string keyId) => new(
+        KeyId: keyId,
+        KeyPrefix: "sbk_ab12",
+        SecretHash: [1, 2, 3, 4, 5, 6, 7, 8],
+        DisplayName: "Test Key " + keyId,
+        Scopes: new HashSet<string>(["read", "write"], StringComparer.Ordinal),
+        ConstraintsJson: """{"ipAllow":["10.0.0.0/8"]}""",
+        CreatedUtc: new DateTimeOffset(2026, 5, 1, 8, 30, 0, TimeSpan.Zero),
+        LastUsedUtc: null,
+        RevokedUtc: null);
+}

ZB.MOM.WW.Auth/tests/ZB.MOM.WW.Auth.ApiKeys.SqlServer.Tests/SqlServerApiKeyOptionsTests.cs

@@ -0,0 +1,16 @@
+namespace ZB.MOM.WW.Auth.ApiKeys.SqlServer.Tests;
+
+public sealed class SqlServerApiKeyOptionsTests
+{
+    [Fact]
+    public void Validate_RejectsMissingConnectionString() =>
+        Assert.Throws<InvalidOperationException>(() => new SqlServerApiKeyOptions { ConnectionString = " " }.Validate());
+
+    [Fact]
+    public void Validate_RejectsUnsafeSchemaName() =>
+        Assert.Throws<InvalidOperationException>(() => new SqlServerApiKeyOptions { ConnectionString = "Server=x", Schema = "zb;auth" }.Validate());
+
+    [Fact]
+    public void Schema_DefaultsToZbauth() =>
+        Assert.Equal("zbauth", new SqlServerApiKeyOptions().Schema);
+}

ZB.MOM.WW.Auth/tests/ZB.MOM.WW.Auth.ApiKeys.SqlServer.Tests/SqlServerApiKeyServiceCollectionExtensionsTests.cs

@@ -0,0 +1,66 @@
+using Microsoft.Extensions.Configuration;
+using Microsoft.Extensions.DependencyInjection;
+using Microsoft.Extensions.Hosting;
+using Microsoft.Extensions.Options;
+using ZB.MOM.WW.Auth.Abstractions.ApiKeys;
+using ZB.MOM.WW.Auth.ApiKeys.SqlServer.DependencyInjection;
+
+namespace ZB.MOM.WW.Auth.ApiKeys.SqlServer.Tests;
+
+public sealed class SqlServerApiKeyServiceCollectionExtensionsTests
+{
+    private static IConfiguration MinimalConfig() =>
+        new ConfigurationBuilder().AddInMemoryCollection(new Dictionary<string, string?>
+        {
+            ["ApiKeys:TokenPrefix"] = "sbk",
+            ["ApiKeys:PepperSecretName"] = "Pepper",
+            ["Pepper"] = "0123456789abcdef",
+            ["ApiKeys:SqlServer:ConnectionString"] = "Server=unused;TrustServerCertificate=true",
+        }).Build();
+
+    [Fact]
+    public void AddZbApiKeyAuthSqlServer_DisplacesSqliteStoresAndMigrator()
+    {
+        var services = new ServiceCollection();
+        services.AddZbApiKeyAuthSqlServer(MinimalConfig(), "ApiKeys", "ApiKeys:SqlServer");
+        using ServiceProvider sp = services.BuildServiceProvider();
+
+        Assert.IsType<SqlServerApiKeyStore>(sp.GetRequiredService<IApiKeyStore>());
+        Assert.IsType<SqlServerApiKeyAdminStore>(sp.GetRequiredService<IApiKeyAdminStore>());
+        Assert.IsType<SqlServerApiKeyAuditStore>(sp.GetRequiredService<IApiKeyAuditStore>());
+        Assert.IsType<SqlServerApiKeyStoreMigrator>(sp.GetRequiredService<IApiKeyStoreMigrator>());
+        Assert.NotNull(sp.GetRequiredService<SqliteApiKeyStoreImporter>());
+        Assert.NotNull(sp.GetRequiredService<IApiKeyVerifier>());
+    }
+
+    [Fact]
+    public void AddZbApiKeyAuthSqlServer_BindsSqlServerOptionsFromSection()
+    {
+        var config = new ConfigurationBuilder().AddInMemoryCollection(new Dictionary<string, string?>
+        {
+            ["ApiKeys:TokenPrefix"] = "sbk",
+            ["ApiKeys:PepperSecretName"] = "Pepper",
+            ["Pepper"] = "0123456789abcdef",
+            ["ApiKeys:SqlServer:ConnectionString"] = "Server=my-shared-sql;TrustServerCertificate=true",
+            ["ApiKeys:SqlServer:Schema"] = "zbauth_custom",
+        }).Build();
+        var services = new ServiceCollection();
+        services.AddZbApiKeyAuthSqlServer(config, "ApiKeys", "ApiKeys:SqlServer");
+        using ServiceProvider sp = services.BuildServiceProvider();
+
+        SqlServerApiKeyOptions options = sp.GetRequiredService<IOptions<SqlServerApiKeyOptions>>().Value;
+        Assert.Equal("Server=my-shared-sql;TrustServerCertificate=true", options.ConnectionString);
+        Assert.Equal("zbauth_custom", options.Schema);
+    }
+
+    [Fact]
+    public void AddZbApiKeyAuthSqlServer_RegistersMigrationHostedService()
+    {
+        var services = new ServiceCollection();
+        services.AddZbApiKeyAuthSqlServer(MinimalConfig(), "ApiKeys", "ApiKeys:SqlServer");
+        using ServiceProvider sp = services.BuildServiceProvider();
+
+        IEnumerable<IHostedService> hostedServices = sp.GetServices<IHostedService>();
+        Assert.Contains(hostedServices, s => s.GetType().Name == "ApiKeyMigrationHostedService");
+    }
+}

ZB.MOM.WW.Auth/tests/ZB.MOM.WW.Auth.ApiKeys.SqlServer.Tests/SqlServerApiKeyStoreMigratorTests.cs

@@ -0,0 +1,43 @@
+namespace ZB.MOM.WW.Auth.ApiKeys.SqlServer.Tests;
+
+/// <summary>
+/// Offline pins for the migrator's retry classification. A <see cref="Microsoft.Data.SqlClient.SqlException"/>
+/// cannot be constructed normally, so the error-number predicate is extracted as an internal static
+/// and pinned here by number; the concurrent-provision behaviour itself is exercised by the
+/// env-gated live suite. These run on any machine, so the retry filter stays pinned even when the
+/// live suite skips.
+/// </summary>
+public sealed class SqlServerApiKeyStoreMigratorTests
+{
+    /// <summary>
+    /// The provisioning-race outcomes the ScadaBridge central shared-store live gate (2026-08-07)
+    /// proved reachable — or that the multi-statement DDL makes reachable — when N nodes provision
+    /// one virgin database at once. Losing such a race means the object now exists, which is the
+    /// desired end state, so a retry of the idempotent IF NOT EXISTS path must be taken.
+    /// </summary>
+    /// <param name="errorNumber">The SQL Server error number under test.</param>
+    [Theory]
+    [InlineData(1205)] // deadlock victim
+    [InlineData(2714)] // "There is already an object named ..." — lost CREATE SCHEMA/TABLE (the live-gate defect)
+    [InlineData(1913)] // "index ... already exists" — lost CREATE INDEX
+    [InlineData(2627)] // PK violation — lost schema_version MERGE-insert
+    public void Provisioning_race_error_numbers_are_retryable(int errorNumber)
+        => Assert.True(SqlServerApiKeyStoreMigrator.IsRetryableMigrationErrorNumber(errorNumber));
+
+    /// <summary>
+    /// Errors that reproduce identically on retry must propagate so the node fails closed instead
+    /// of spinning through the retry budget against a genuine misconfiguration.
+    /// </summary>
+    /// <param name="errorNumber">The SQL Server error number under test.</param>
+    [Theory]
+    [InlineData(2705)] // duplicate column — unreachable from the race (single CREATE TABLE statement)
+    [InlineData(208)] // invalid object name
+    [InlineData(229)] // permission denied
+    [InlineData(2759)] // "CREATE SCHEMA failed due to previous errors" — retryable only via its companion 2714
+    [InlineData(4060)] // cannot open database
+    [InlineData(18456)] // login failed
+    [InlineData(0)]
+    [InlineData(-2)] // client-side timeout
+    public void Non_race_error_numbers_are_not_retryable(int errorNumber)
+        => Assert.False(SqlServerApiKeyStoreMigrator.IsRetryableMigrationErrorNumber(errorNumber));
+}

ZB.MOM.WW.Auth/tests/ZB.MOM.WW.Auth.ApiKeys.SqlServer.Tests/ZB.MOM.WW.Auth.ApiKeys.SqlServer.Tests.csproj

@@ -0,0 +1,30 @@
+<Project Sdk="Microsoft.NET.Sdk">
+
+  <PropertyGroup>
+    <IsPackable>false</IsPackable>
+  </PropertyGroup>
+
+  <ItemGroup>
+    <PackageReference Include="coverlet.collector" />
+    <PackageReference Include="Microsoft.NET.Test.Sdk" />
+    <PackageReference Include="xunit" />
+    <PackageReference Include="xunit.runner.visualstudio" />
+    <!-- Live tests (Task 3+) skip rather than fail when AUTH_SQLSERVER_CONNSTR is unset. -->
+    <PackageReference Include="Xunit.SkippableFact" />
+    <PackageReference Include="Microsoft.Extensions.DependencyInjection" />
+    <PackageReference Include="Microsoft.Extensions.Configuration" />
+    <PackageReference Include="Microsoft.Extensions.Configuration.Abstractions" />
+    <PackageReference Include="Microsoft.Extensions.Hosting.Abstractions" />
+  </ItemGroup>
+
+  <ItemGroup>
+    <Using Include="Xunit" />
+  </ItemGroup>
+
+  <ItemGroup>
+    <ProjectReference Include="..\..\src\ZB.MOM.WW.Auth.ApiKeys.SqlServer\ZB.MOM.WW.Auth.ApiKeys.SqlServer.csproj" />
+    <ProjectReference Include="..\..\src\ZB.MOM.WW.Auth.ApiKeys\ZB.MOM.WW.Auth.ApiKeys.csproj" />
+    <ProjectReference Include="..\..\src\ZB.MOM.WW.Auth.Abstractions\ZB.MOM.WW.Auth.Abstractions.csproj" />
+  </ItemGroup>
+
+</Project>

ZB.MOM.WW.Auth/tests/ZB.MOM.WW.Auth.ApiKeys.Tests/ApiKeyServiceCollectionExtensionsTests.cs

@@ -5,6 +5,7 @@ using Microsoft.Extensions.Hosting;
 using Microsoft.Extensions.Options;
 using ZB.MOM.WW.Auth.Abstractions.ApiKeys;
 using ZB.MOM.WW.Auth.ApiKeys.DependencyInjection;
+using ZB.MOM.WW.Auth.ApiKeys.Sqlite;
 
 namespace ZB.MOM.WW.Auth.ApiKeys.Tests;
 
@@ -197,4 +198,37 @@ public class ApiKeyServiceCollectionExtensionsTests
                 File.Delete(sqlitePath);
         }
     }
+
+    [Fact]
+    public void AddZbApiKeyAuth_RegistersSqliteMigratorAsIApiKeyStoreMigrator()
+    {
+        IConfiguration config = BuildConfiguration(TempSqlitePath());
+        var services = new ServiceCollection();
+
+        services.AddZbApiKeyAuth(config, ApiKeySection);
+
+        using ServiceProvider sp = services.BuildServiceProvider();
+
+        Assert.IsType<SqliteAuthStoreMigrator>(sp.GetRequiredService<IApiKeyStoreMigrator>());
+        Assert.Same(sp.GetRequiredService<SqliteAuthStoreMigrator>(), sp.GetRequiredService<IApiKeyStoreMigrator>());
+    }
+
+    [Fact]
+    public void AddZbApiKeyAuth_LeavesPreRegisteredMigratorInPlace()
+    {
+        var services = new ServiceCollection();
+        var fake = new FakeMigrator();
+        services.AddSingleton<IApiKeyStoreMigrator>(fake);
+
+        services.AddZbApiKeyAuth(BuildConfiguration(TempSqlitePath()), ApiKeySection);
+
+        using ServiceProvider sp = services.BuildServiceProvider();
+
+        Assert.Same(fake, sp.GetRequiredService<IApiKeyStoreMigrator>());
+    }
+
+    private sealed class FakeMigrator : IApiKeyStoreMigrator
+    {
+        public Task MigrateAsync(CancellationToken ct) => Task.CompletedTask;
+    }
 }

ZB.MOM.WW.Auth/tests/ZB.MOM.WW.Auth.AspNetCore.Tests/ServiceCollectionExtensionsTests.cs

@@ -134,4 +134,30 @@ public class ServiceCollectionExtensionsTests
         await host.StartAsync();
         await host.StopAsync();
     }
+
+    [Fact]
+    public void AddZbLdapAuth_BindsUserBindModeKeys()
+    {
+        IConfiguration config = new ConfigurationBuilder()
+            .AddInMemoryCollection(new Dictionary<string, string?>
+            {
+                [$"{LdapSection}:Server"] = LdapServer,
+                [$"{LdapSection}:SearchBase"] = "dc=example,dc=com",
+                [$"{LdapSection}:Transport"] = nameof(LdapTransport.Ldaps),
+                [$"{LdapSection}:BindMode"] = "User",
+                [$"{LdapSection}:UserBindTemplate"] = "{0}@corp.test",
+                [$"{LdapSection}:AcceptDomainQualifiedNames"] = "false",
+            })
+            .Build();
+        var services = new ServiceCollection();
+
+        services.AddZbLdapAuth(config, LdapSection);
+
+        using ServiceProvider provider = services.BuildServiceProvider();
+        var options = provider.GetRequiredService<IOptions<LdapOptions>>().Value;
+
+        Assert.Equal(LdapBindMode.User, options.BindMode);
+        Assert.Equal("{0}@corp.test", options.UserBindTemplate);
+        Assert.False(options.AcceptDomainQualifiedNames);
+    }
 }

ZB.MOM.WW.Auth/tests/ZB.MOM.WW.Auth.Ldap.Tests/FakeLdapConnection.cs

@@ -16,6 +16,8 @@ internal sealed class FakeLdapConnection : ILdapConnection
     private bool _throwOnConnect;
     private bool _throwOnServiceBind;
     private bool _throwOnUserBind;
+    private int? _firstBindResultCode;
+    private bool _throwOnSearch;
 
     // ---- observation -----
 
@@ -32,13 +34,24 @@ internal sealed class FakeLdapConnection : ILdapConnection
     /// </summary>
     public int BindAttempts { get; private set; }
 
+    /// <summary>The attribute list passed to the most recent <see cref="Search"/> call (the fake still returns every scripted attribute).</summary>
+    public IReadOnlyList<string>? SearchAttributes { get; private set; }
+
+    /// <summary>Every call in order: <c>connect</c>, <c>bind:&lt;dn&gt;</c>, <c>search:&lt;filter&gt;</c>.</summary>
+    public List<string> Calls { get; } = new();
+
     // ---- builder methods -----
 
     /// <summary>
     /// Scripts a user entry that will be returned by the next <see cref="Search"/> call.
-    /// Builds a minimal attribute bag with <c>memberOf</c> and optional <c>displayName</c>.
+    /// Builds a minimal attribute bag with <c>memberOf</c>, optional <c>displayName</c> and any
+    /// <paramref name="extraAttributes"/> (e.g. <c>sAMAccountName</c>) the test needs.
     /// </summary>
-    public FakeLdapConnection WithUserEntry(string dn, string[] memberOf, string? displayName = null)
+    public FakeLdapConnection WithUserEntry(
+        string dn,
+        string[] memberOf,
+        string? displayName = null,
+        IReadOnlyDictionary<string, string[]>? extraAttributes = null)
     {
         var attrs = new Dictionary<string, IReadOnlyList<string>>(StringComparer.OrdinalIgnoreCase)
         {
@@ -46,6 +59,11 @@ internal sealed class FakeLdapConnection : ILdapConnection
         };
         if (displayName is not null)
             attrs["displayName"] = new[] { displayName };
+        if (extraAttributes is not null)
+        {
+            foreach (var (name, values) in extraAttributes)
+                attrs[name] = values;
+        }
 
         _scriptedEntries.Add(new LdapSearchEntry(dn, attrs));
         return this;
@@ -83,6 +101,24 @@ internal sealed class FakeLdapConnection : ILdapConnection
         return this;
     }
 
+    /// <summary>
+    /// Throw <see cref="Novell.Directory.Ldap.LdapException"/> with <paramref name="resultCode"/> on the
+    /// FIRST bind. In user-bind mode the first bind is the user's own, so 49 simulates a wrong
+    /// password and e.g. 52 (Unavailable) a domain controller that cannot answer.
+    /// </summary>
+    public FakeLdapConnection ThrowOnFirstBind(int resultCode)
+    {
+        _firstBindResultCode = resultCode;
+        return this;
+    }
+
+    /// <summary>Throw <see cref="Novell.Directory.Ldap.LdapException"/> from <see cref="Search"/> (unhealthy directory).</summary>
+    public FakeLdapConnection ThrowOnSearch()
+    {
+        _throwOnSearch = true;
+        return this;
+    }
+
     /// <summary>
     /// Throw <see cref="Novell.Directory.Ldap.LdapException"/> from <see cref="Connect"/> to
     /// simulate an unreachable directory (infrastructure failure).
@@ -120,6 +156,7 @@ internal sealed class FakeLdapConnection : ILdapConnection
         int timeoutMs,
         RemoteCertificateValidationCallback? serverCertificateValidationCallback = null)
     {
+        Calls.Add("connect");
         ConnectArgs = (host, port, transport, allowInsecure, timeoutMs);
         ConnectCertCallback = serverCertificateValidationCallback;
         if (_throwOnConnect)
@@ -130,8 +167,12 @@ internal sealed class FakeLdapConnection : ILdapConnection
     public void Bind(string dn, string password)
     {
         BindAttempts++;
+        Calls.Add($"bind:{dn}");
         var isServiceBind = BindAttempts == 1;
 
+        if (isServiceBind && _firstBindResultCode is { } code)
+            throw new Novell.Directory.Ldap.LdapException("Bind refused", code, dn);
+
         if ((_throwOnServiceBind && isServiceBind)
             || (_throwOnUserBind && !isServiceBind)
             || _throwBindDns.Contains(dn))
@@ -147,7 +188,14 @@ internal sealed class FakeLdapConnection : ILdapConnection
         string searchBase,
         string filter,
         IReadOnlyList<string> attributes)
-        => _scriptedEntries.AsReadOnly();
+    {
+        Calls.Add($"search:{filter}");
+        SearchAttributes = attributes;
+        if (_throwOnSearch)
+            throw new Novell.Directory.Ldap.LdapException(
+                "Search failed", Novell.Directory.Ldap.LdapException.Unavailable, searchBase);
+        return _scriptedEntries.AsReadOnly();
+    }
 
     public void Dispose() { /* nothing to clean up */ }
 }

ZB.MOM.WW.Auth/tests/ZB.MOM.WW.Auth.Ldap.Tests/Integration/GLAuthIntegrationTests.cs

@@ -99,4 +99,70 @@ public sealed class GLAuthIntegrationTests
             $"Authentication failed: {result.Failure} (server={server}:{port}, user={user})");
         Assert.NotEmpty(result.Groups);
     }
+
+    /// <summary>
+    /// Failover: primary endpoint points at a closed local port (refused immediately),
+    /// fallback at the real GLAuth. The login must succeed via the fallback, and a second
+    /// login must also succeed (sticky preference — no re-walk through the dead primary).
+    /// Skipped unless <c>ZB_LDAP_IT=1</c> is set; skipped again if the server is unreachable.
+    /// </summary>
+    [SkippableFact]
+    public async Task Authenticate_FailsOverFromDeadPrimary_ToRealGLAuth()
+    {
+        // ------------------------------------------------------------------ opt-in gate
+        Skip.IfNot(
+            Environment.GetEnvironmentVariable("ZB_LDAP_IT") == "1",
+            "Set ZB_LDAP_IT=1 and a reachable GLAuth to run.");
+
+        // ------------------------------------------------------------------ read config
+        var server   = Environment.GetEnvironmentVariable("ZB_LDAP_SERVER")   ?? "localhost";
+        var port     = int.TryParse(Environment.GetEnvironmentVariable("ZB_LDAP_PORT"), out var p) ? p : 3893;
+        var baseDn   = Environment.GetEnvironmentVariable("ZB_LDAP_BASE")     ?? "dc=lmxopcua,dc=local";
+        var svcDn    = Environment.GetEnvironmentVariable("ZB_LDAP_SVC_DN")   ?? "";
+        var svcPw    = Environment.GetEnvironmentVariable("ZB_LDAP_SVC_PW")   ?? "";
+        var user     = Environment.GetEnvironmentVariable("ZB_LDAP_USER")     ?? "";
+        var pw       = Environment.GetEnvironmentVariable("ZB_LDAP_PW")       ?? "";
+        var userAttr = Environment.GetEnvironmentVariable("ZB_LDAP_USERATTR") ?? "cn";
+
+        // ------------------------------------------------------------------ reachability probe
+        try
+        {
+            using var tcp = new TcpClient();
+            // 3-second connect timeout to keep the test suite snappy when the server is absent
+            var connectTask = tcp.ConnectAsync(server, port);
+            if (!connectTask.Wait(TimeSpan.FromSeconds(3)))
+                Skip.If(true, $"GLAuth not reachable at {server}:{port} (connect timed out).");
+        }
+        catch (Exception ex)
+        {
+            Skip.If(true, $"GLAuth not reachable at {server}:{port}: {ex.Message}");
+        }
+
+        // ------------------------------------------------------------------ build options
+        var options = new LdapOptions
+        {
+            Enabled                = true,
+            Server                 = "127.0.0.1",
+            Port                   = 39999,          // closed port -> immediate refusal
+            Transport              = LdapTransport.None,
+            AllowInsecure          = true,
+            SearchBase             = baseDn,
+            ServiceAccountDn       = svcDn,
+            ServiceAccountPassword = svcPw,
+            UserNameAttribute      = userAttr,
+            GroupAttribute         = "memberOf",
+            ConnectionTimeoutMs    = 3000,
+            FallbackServers        = new[] { $"{server}:{port}" },
+        };
+
+        // ------------------------------------------------------------------ exercise the real service
+        var svc = new LdapAuthService(options);
+
+        var first = await svc.AuthenticateAsync(user, pw, CancellationToken.None);
+        Assert.True(first.Succeeded, $"first login failed: {first.Failure}");
+
+        // Second login exercises the sticky preference: it must not re-walk the dead primary.
+        var second = await svc.AuthenticateAsync(user, pw, CancellationToken.None);
+        Assert.True(second.Succeeded, $"second login failed: {second.Failure}");
+    }
 }

ZB.MOM.WW.Auth/tests/ZB.MOM.WW.Auth.Ldap.Tests/LdapAuthServiceFailoverTests.cs

@@ -0,0 +1,297 @@
+using ZB.MOM.WW.Auth.Abstractions.Ldap;
+
+namespace ZB.MOM.WW.Auth.Ldap.Tests;
+
+/// <summary>
+/// Covers the multi-endpoint failover walk: which failures fail over (system-side only),
+/// which are final (any authoritative answer from a healthy directory), how the sticky
+/// endpoint preference moves, and that single-endpoint behaviour is unchanged.
+/// </summary>
+public class LdapAuthServiceFailoverTests
+{
+    private static LdapOptions Opts(params string[] fallbacks) => new()
+    {
+        Enabled = true,
+        Server = "dc1",
+        Port = 636,
+        Transport = LdapTransport.Ldaps,
+        SearchBase = "dc=x",
+        ServiceAccountDn = "cn=svc,dc=x",
+        ServiceAccountPassword = "svcpw",
+        FallbackServers = fallbacks,
+    };
+
+    private static FakeLdapConnection HealthyWithUser() =>
+        new FakeLdapConnection().WithUserEntry("cn=alice,dc=x", new[] { "cn=ops,dc=x" });
+
+    // ---- failover triggers (system-side) ----
+
+    [Fact]
+    public async Task ConnectFailure_FailsOverToFallback_AndSucceeds()
+    {
+        var factory = new SequenceLdapConnectionFactory(
+            new FakeLdapConnection().ThrowOnConnect(), HealthyWithUser());
+        var svc = new LdapAuthService(Opts("dc2:3894"), factory);
+
+        var result = await svc.AuthenticateAsync("alice", "pw", CancellationToken.None);
+
+        Assert.True(result.Succeeded);
+        Assert.Equal(2, factory.Created.Count);
+        Assert.Equal("dc1", factory.Created[0].ConnectArgs!.Value.Host);
+        Assert.Equal("dc2", factory.Created[1].ConnectArgs!.Value.Host);
+        Assert.Equal(3894, factory.Created[1].ConnectArgs!.Value.Port);
+    }
+
+    [Fact]
+    public async Task ServiceBindFailure_FailsOverToFallback()
+    {
+        var factory = new SequenceLdapConnectionFactory(
+            new FakeLdapConnection().ThrowOnServiceBind(), HealthyWithUser());
+        var svc = new LdapAuthService(Opts("dc2"), factory);
+
+        var result = await svc.AuthenticateAsync("alice", "pw", CancellationToken.None);
+
+        Assert.True(result.Succeeded);
+        Assert.Equal(636, factory.Created[1].ConnectArgs!.Value.Port); // bare host adopts Port
+    }
+
+    // ---- user-side results NEVER fail over ----
+
+    [Fact]
+    public async Task BadUserCredentials_DoesNotFailOver()
+    {
+        var factory = new SequenceLdapConnectionFactory(
+            HealthyWithUser().ThrowOnUserBind());
+        var svc = new LdapAuthService(Opts("dc2"), factory);
+
+        var result = await svc.AuthenticateAsync("alice", "wrong", CancellationToken.None);
+
+        Assert.Equal(LdapAuthFailure.BadCredentials, result.Failure);
+        Assert.Single(factory.Created);
+    }
+
+    [Fact]
+    public async Task UserNotFound_DoesNotFailOver()
+    {
+        var factory = new SequenceLdapConnectionFactory(new FakeLdapConnection());
+        var svc = new LdapAuthService(Opts("dc2"), factory);
+
+        var result = await svc.AuthenticateAsync("ghost", "pw", CancellationToken.None);
+
+        Assert.Equal(LdapAuthFailure.UserNotFound, result.Failure);
+        Assert.Single(factory.Created);
+    }
+
+    [Fact]
+    public async Task GroupLookupFailed_DoesNotFailOver()
+    {
+        var factory = new SequenceLdapConnectionFactory(
+            new FakeLdapConnection().WithUserEntry("cn=alice,dc=x", Array.Empty<string>()));
+        var svc = new LdapAuthService(Opts("dc2"), factory);
+
+        var result = await svc.AuthenticateAsync("alice", "pw", CancellationToken.None);
+
+        Assert.Equal(LdapAuthFailure.GroupLookupFailed, result.Failure);
+        Assert.Single(factory.Created);
+    }
+
+    [Fact]
+    public async Task AmbiguousUser_DoesNotFailOver()
+    {
+        var factory = new SequenceLdapConnectionFactory(
+            new FakeLdapConnection().WithDuplicateMatch());
+        var svc = new LdapAuthService(Opts("dc2"), factory);
+
+        var result = await svc.AuthenticateAsync("alice", "pw", CancellationToken.None);
+
+        Assert.Equal(LdapAuthFailure.AmbiguousUser, result.Failure);
+        Assert.Single(factory.Created);
+    }
+
+    // ---- exhaustion ----
+
+    [Fact]
+    public async Task AllEndpointsDown_ReturnsServiceAccountBindFailed()
+    {
+        var factory = new SequenceLdapConnectionFactory(
+            new FakeLdapConnection().ThrowOnConnect(),
+            new FakeLdapConnection().ThrowOnConnect());
+        var svc = new LdapAuthService(Opts("dc2"), factory);
+
+        var result = await svc.AuthenticateAsync("alice", "pw", CancellationToken.None);
+
+        Assert.Equal(LdapAuthFailure.ServiceAccountBindFailed, result.Failure);
+        Assert.Equal(2, factory.Created.Count);
+    }
+
+    // ---- stickiness ----
+
+    [Fact]
+    public async Task StickyPreference_SecondLoginStartsAtLastGoodEndpoint()
+    {
+        var factory = new SequenceLdapConnectionFactory(
+            new FakeLdapConnection().ThrowOnConnect(), // login 1 @ dc1: dead
+            HealthyWithUser(),                          // login 1 @ dc2: answers
+            HealthyWithUser());                         // login 2: should go straight to dc2
+        var svc = new LdapAuthService(Opts("dc2"), factory);
+
+        Assert.True((await svc.AuthenticateAsync("alice", "pw", CancellationToken.None)).Succeeded);
+        Assert.True((await svc.AuthenticateAsync("alice", "pw", CancellationToken.None)).Succeeded);
+
+        Assert.Equal(3, factory.Created.Count);
+        Assert.Equal("dc2", factory.Created[2].ConnectArgs!.Value.Host);
+    }
+
+    [Fact]
+    public async Task StickyPreference_DriftsBackToPrimary_WhenPreferredFails()
+    {
+        var factory = new SequenceLdapConnectionFactory(
+            new FakeLdapConnection().ThrowOnConnect(), // login 1 @ dc1: dead
+            HealthyWithUser(),                          // login 1 @ dc2: answers -> preferred
+            new FakeLdapConnection().ThrowOnConnect(), // login 2 @ dc2: now dead
+            HealthyWithUser(),                          // login 2 @ dc1 (declared order): recovered
+            HealthyWithUser());                         // login 3: should start at dc1 again
+        var svc = new LdapAuthService(Opts("dc2"), factory);
+
+        Assert.True((await svc.AuthenticateAsync("alice", "pw", CancellationToken.None)).Succeeded);
+        Assert.True((await svc.AuthenticateAsync("alice", "pw", CancellationToken.None)).Succeeded);
+        Assert.True((await svc.AuthenticateAsync("alice", "pw", CancellationToken.None)).Succeeded);
+
+        Assert.Equal(5, factory.Created.Count);
+        Assert.Equal("dc1", factory.Created[3].ConnectArgs!.Value.Host);
+        Assert.Equal("dc1", factory.Created[4].ConnectArgs!.Value.Host);
+    }
+
+    [Fact]
+    public async Task StickyPreference_UpdatesOnUserSideAnswer_NotOnlySuccess()
+    {
+        var factory = new SequenceLdapConnectionFactory(
+            new FakeLdapConnection().ThrowOnConnect(), // login 1 @ dc1: dead
+            new FakeLdapConnection(),                   // login 1 @ dc2: answers UserNotFound
+            HealthyWithUser());                         // login 2: should start at dc2
+        var svc = new LdapAuthService(Opts("dc2"), factory);
+
+        var first = await svc.AuthenticateAsync("ghost", "pw", CancellationToken.None);
+        Assert.Equal(LdapAuthFailure.UserNotFound, first.Failure);
+
+        Assert.True((await svc.AuthenticateAsync("alice", "pw", CancellationToken.None)).Succeeded);
+        Assert.Equal("dc2", factory.Created[2].ConnectArgs!.Value.Host);
+    }
+
+    [Fact]
+    public async Task AllEndpointsDown_LeavesPreferenceUnchanged()
+    {
+        // A total outage teaches nothing about which endpoint is healthy, so the walk
+        // must not move the preference — the next login still starts at the primary.
+        var factory = new SequenceLdapConnectionFactory(
+            new FakeLdapConnection().ThrowOnConnect(), // login 1 @ dc1
+            new FakeLdapConnection().ThrowOnConnect(), // login 1 @ dc2
+            HealthyWithUser());                         // login 2: must start at dc1 again
+        var svc = new LdapAuthService(Opts("dc2"), factory);
+
+        Assert.Equal(LdapAuthFailure.ServiceAccountBindFailed,
+            (await svc.AuthenticateAsync("alice", "pw", CancellationToken.None)).Failure);
+        Assert.True((await svc.AuthenticateAsync("alice", "pw", CancellationToken.None)).Succeeded);
+
+        Assert.Equal("dc1", factory.Created[2].ConnectArgs!.Value.Host);
+    }
+
+    // ---- back-compat ----
+
+    [Fact]
+    public async Task NoFallbacks_SingleAttempt_UnchangedBehavior()
+    {
+        var factory = new SequenceLdapConnectionFactory(
+            new FakeLdapConnection().ThrowOnConnect());
+        var svc = new LdapAuthService(Opts(/* no fallbacks */), factory);
+
+        var result = await svc.AuthenticateAsync("alice", "pw", CancellationToken.None);
+
+        Assert.Equal(LdapAuthFailure.ServiceAccountBindFailed, result.Failure);
+        Assert.Single(factory.Created);
+    }
+
+    [Fact]
+    public async Task Disabled_NeverOpensAConnection_EvenWithFallbacks()
+    {
+        var factory = new SequenceLdapConnectionFactory(HealthyWithUser());
+        var svc = new LdapAuthService(Opts("dc2") with { Enabled = false }, factory);
+
+        var result = await svc.AuthenticateAsync("alice", "pw", CancellationToken.None);
+
+        Assert.Equal(LdapAuthFailure.Disabled, result.Failure);
+        Assert.Empty(factory.Created);
+    }
+
+    [Fact]
+    public async Task MalformedFallbackEntry_IsSkippedDefensively()
+    {
+        // The validator rejects this at boot; a service constructed without validation
+        // must degrade to the parseable subset, not throw.
+        var factory = new SequenceLdapConnectionFactory(HealthyWithUser());
+        var svc = new LdapAuthService(Opts(":::"), factory);
+
+        Assert.True((await svc.AuthenticateAsync("alice", "pw", CancellationToken.None)).Succeeded);
+        Assert.Single(factory.Created);
+    }
+
+    // ---- observability ----
+
+    /// <summary>Captures emitted log entries so failover visibility can be asserted.</summary>
+    private sealed class RecordingLogger : Microsoft.Extensions.Logging.ILogger<LdapAuthService>
+    {
+        public List<(Microsoft.Extensions.Logging.LogLevel Level, string Message)> Entries { get; } = new();
+        public IDisposable? BeginScope<TState>(TState state) where TState : notnull => null;
+        public bool IsEnabled(Microsoft.Extensions.Logging.LogLevel logLevel) => true;
+        public void Log<TState>(Microsoft.Extensions.Logging.LogLevel logLevel, Microsoft.Extensions.Logging.EventId eventId,
+            TState state, Exception? exception, Func<TState, Exception?, string> formatter)
+            => Entries.Add((logLevel, formatter(state, exception)));
+    }
+
+    [Fact]
+    public async Task Failover_LogsWarningWithEndpointAndStage_AndInfoOnStickyChange()
+    {
+        var logger = new RecordingLogger();
+        var factory = new SequenceLdapConnectionFactory(
+            new FakeLdapConnection().ThrowOnConnect(), HealthyWithUser());
+        var svc = new LdapAuthService(Opts("dc2:3894"), factory, logger);
+
+        await svc.AuthenticateAsync("alice", "pw", CancellationToken.None);
+
+        Assert.Contains(logger.Entries, e =>
+            e.Level == Microsoft.Extensions.Logging.LogLevel.Warning
+            && e.Message.Contains("dc1:636") && e.Message.Contains("connect"));
+        Assert.Contains(logger.Entries, e =>
+            e.Level == Microsoft.Extensions.Logging.LogLevel.Information
+            && e.Message.Contains("dc2:3894"));
+        // Never log secrets:
+        Assert.DoesNotContain(logger.Entries, e => e.Message.Contains("svcpw") || e.Message.Contains("pw\""));
+    }
+
+    [Fact]
+    public async Task HappyPath_LogsNothing()
+    {
+        var logger = new RecordingLogger();
+        var factory = new SequenceLdapConnectionFactory(HealthyWithUser());
+        var svc = new LdapAuthService(Opts("dc2"), factory, logger);
+
+        await svc.AuthenticateAsync("alice", "pw", CancellationToken.None);
+
+        Assert.Empty(logger.Entries);
+    }
+
+    [Fact]
+    public async Task ServiceBindFailure_LogsTheServiceBindStage_NotConnect()
+    {
+        var logger = new RecordingLogger();
+        var factory = new SequenceLdapConnectionFactory(
+            new FakeLdapConnection().ThrowOnServiceBind(), HealthyWithUser());
+        var svc = new LdapAuthService(Opts("dc2"), factory, logger);
+
+        await svc.AuthenticateAsync("alice", "pw", CancellationToken.None);
+
+        Assert.Contains(logger.Entries, e =>
+            e.Level == Microsoft.Extensions.Logging.LogLevel.Warning
+            && e.Message.Contains("dc1:636") && e.Message.Contains("service-bind"));
+    }
+}

ZB.MOM.WW.Auth/tests/ZB.MOM.WW.Auth.Ldap.Tests/LdapAuthServiceTests.cs

@@ -147,4 +147,45 @@ public class LdapAuthServiceTests
         Assert.True(r.Succeeded);
         Assert.Equal(new[] { "Eng,ineers", "A,B" }, r.Groups);
     }
+
+    [Fact]
+    public async Task Username_IsCanonicalisedFromTheEntry_InServiceAccountMode()
+    {
+        var fake = new FakeLdapConnection().WithUserEntry(
+            "cn=alice,dc=x", new[] { "cn=g,dc=x" }, "Alice",
+            new Dictionary<string, string[]> { ["cn"] = new[] { "alice" } });
+        var svc = new LdapAuthService(Opts(), new FakeLdapConnectionFactory(fake));
+
+        var r = await svc.AuthenticateAsync("ALICE", "pw", default);
+
+        Assert.True(r.Succeeded);
+        Assert.Equal("alice", r.Username);
+    }
+
+    [Fact]
+    public async Task Username_FallsBackToTheTypedName_InServiceAccountMode_WhenAttributeAbsent()
+    {
+        var fake = new FakeLdapConnection().WithUserEntry("cn=alice,dc=x", new[] { "cn=g,dc=x" }, "Alice");
+        var svc = new LdapAuthService(Opts(), new FakeLdapConnectionFactory(fake));
+
+        var r = await svc.AuthenticateAsync("ALICE", "pw", default);
+
+        Assert.True(r.Succeeded);
+        Assert.Equal("ALICE", r.Username);
+    }
+
+    [Theory]
+    [InlineData("")]
+    [InlineData(null)]
+    public async Task EmptyPassword_IsRefused_InServiceAccountMode_WithoutContactingTheDirectory(string? password)
+    {
+        var factory = new SequenceLdapConnectionFactory(
+            new FakeLdapConnection().WithUserEntry("cn=alice,dc=x", new[] { "cn=g,dc=x" }, "Alice"));
+        var svc = new LdapAuthService(Opts(), factory);
+
+        var r = await svc.AuthenticateAsync("alice", password!, default);
+
+        Assert.Equal(LdapAuthFailure.BadCredentials, r.Failure);
+        Assert.Empty(factory.Created);
+    }
 }

ZB.MOM.WW.Auth/tests/ZB.MOM.WW.Auth.Ldap.Tests/LdapAuthServiceUserBindTests.cs

@@ -0,0 +1,222 @@
+using Novell.Directory.Ldap;
+using ZB.MOM.WW.Auth.Abstractions.Ldap;
+
+namespace ZB.MOM.WW.Auth.Ldap.Tests;
+
+/// <summary>
+/// <see cref="LdapBindMode.User"/>: bind as the signing-in user, then read their own entry on the
+/// same connection. The service account in <see cref="Opts"/> is deliberately populated so the
+/// tests prove it is never used.
+/// </summary>
+public class LdapAuthServiceUserBindTests
+{
+    private static LdapOptions Opts(params string[] fallbacks) => new()
+    {
+        Enabled = true,
+        Server = "dc1",
+        Port = 636,
+        Transport = LdapTransport.Ldaps,
+        SearchBase = "dc=x",
+        UserNameAttribute = "sAMAccountName",
+        DisplayNameAttribute = "displayName",
+        GroupAttribute = "memberOf",
+        BindMode = LdapBindMode.User,
+        UserBindTemplate = "{0}@corp.test",
+        ServiceAccountDn = "cn=svc,dc=x",
+        ServiceAccountPassword = "svcpw",
+        FallbackServers = fallbacks,
+    };
+
+    private static FakeLdapConnection Jane() => new FakeLdapConnection()
+        .WithUserEntry("CN=Jane Doe,OU=u,DC=x", new[] { "CN=UG_A,OU=g,DC=x", "CN=UG_B,OU=g,DC=x" }, "Jane Doe");
+
+    private static Task<LdapAuthResult> Login(LdapAuthService svc, string user, string pw = "pw") =>
+        svc.AuthenticateAsync(user, pw, CancellationToken.None);
+
+    [Fact]
+    public async Task Success_BindsAsUser_ThenSearchesOnTheSameConnection()
+    {
+        var factory = new SequenceLdapConnectionFactory(Jane());
+        var result = await Login(new LdapAuthService(Opts(), factory), "jdoe");
+
+        Assert.True(result.Succeeded);
+        Assert.Equal("jdoe", result.Username);
+        Assert.Equal("Jane Doe", result.DisplayName);
+        Assert.Equal(new[] { "UG_A", "UG_B" }, result.Groups);
+        var conn = Assert.Single(factory.Created);
+        Assert.Equal(new[] { "connect", "bind:jdoe@corp.test", "search:(sAMAccountName=jdoe)" }, conn.Calls);
+        Assert.Equal(new[] { "jdoe@corp.test" }, conn.BoundDns);
+    }
+
+    [Theory]
+    [InlineData(@"NAM\jdoe")]
+    [InlineData("jdoe@corp.test")]
+    [InlineData("  jdoe ")]
+    public async Task QualifiedNames_BindAsTheTemplate_AndReturnTheBareName(string typed)
+    {
+        var factory = new SequenceLdapConnectionFactory(Jane());
+        var result = await Login(new LdapAuthService(Opts(), factory), typed);
+
+        Assert.True(result.Succeeded);
+        Assert.Equal("jdoe", result.Username);
+        Assert.Equal(new[] { "jdoe@corp.test" }, factory.Created[0].BoundDns);
+    }
+
+    [Fact]
+    public async Task EmptyPassword_IsRefused_WithoutContactingTheDirectory()
+    {
+        var factory = new SequenceLdapConnectionFactory(Jane());
+        var result = await Login(new LdapAuthService(Opts(), factory), "jdoe", "");
+
+        Assert.Equal(LdapAuthFailure.BadCredentials, result.Failure);
+        Assert.Empty(factory.Created);
+    }
+
+    [Theory]
+    [InlineData("@x")]
+    [InlineData(@"a\b\c")]
+    [InlineData("a@b@c")]
+    public async Task UnusableNames_AreRefused_WithoutContactingTheDirectory(string typed)
+    {
+        var factory = new SequenceLdapConnectionFactory(Jane());
+        var result = await Login(new LdapAuthService(Opts(), factory), typed);
+
+        Assert.Equal(LdapAuthFailure.BadCredentials, result.Failure);
+        Assert.Empty(factory.Created);
+    }
+
+    [Fact]
+    public async Task StrictMode_RefusesQualifiedNames_WithoutContactingTheDirectory()
+    {
+        var factory = new SequenceLdapConnectionFactory(Jane());
+        var svc = new LdapAuthService(Opts() with { AcceptDomainQualifiedNames = false }, factory);
+
+        var result = await Login(svc, @"NAM\jdoe");
+
+        Assert.Equal(LdapAuthFailure.BadCredentials, result.Failure);
+        Assert.Empty(factory.Created);
+    }
+
+    [Fact]
+    public async Task WrongPassword_IsFinal_AndNeverAsksTheFallback()
+    {
+        var factory = new SequenceLdapConnectionFactory(
+            Jane().ThrowOnFirstBind(LdapException.InvalidCredentials), Jane());
+        var result = await Login(new LdapAuthService(Opts("dc2"), factory), "jdoe", "wrong");
+
+        Assert.Equal(LdapAuthFailure.BadCredentials, result.Failure);
+        Assert.Single(factory.Created);
+    }
+
+    [Fact]
+    public async Task NonCredentialBindError_FailsOver()
+    {
+        var factory = new SequenceLdapConnectionFactory(
+            Jane().ThrowOnFirstBind(LdapException.Unavailable), Jane());
+        var result = await Login(new LdapAuthService(Opts("dc2"), factory), "jdoe");
+
+        Assert.True(result.Succeeded);
+        Assert.Equal(2, factory.Created.Count);
+        Assert.Equal("dc2", factory.Created[1].ConnectArgs!.Value.Host);
+    }
+
+    [Fact]
+    public async Task ConnectFailure_FailsOver()
+    {
+        var factory = new SequenceLdapConnectionFactory(new FakeLdapConnection().ThrowOnConnect(), Jane());
+        var result = await Login(new LdapAuthService(Opts("dc2"), factory), "jdoe");
+
+        Assert.True(result.Succeeded);
+        Assert.Equal(2, factory.Created.Count);
+    }
+
+    [Fact]
+    public async Task SearchFailure_FailsOver()
+    {
+        var factory = new SequenceLdapConnectionFactory(Jane().ThrowOnSearch(), Jane());
+        var result = await Login(new LdapAuthService(Opts("dc2"), factory), "jdoe");
+
+        Assert.True(result.Succeeded);
+        Assert.Equal(2, factory.Created.Count);
+    }
+
+    [Fact]
+    public async Task EveryEndpointDown_IsSystemSide()
+    {
+        var factory = new SequenceLdapConnectionFactory(
+            new FakeLdapConnection().ThrowOnConnect(), new FakeLdapConnection().ThrowOnConnect());
+        var result = await Login(new LdapAuthService(Opts("dc2"), factory), "jdoe");
+
+        Assert.Equal(LdapAuthFailure.ServiceAccountBindFailed, result.Failure);
+    }
+
+    [Fact]
+    public async Task NoEntryAfterAGoodBind_IsGroupLookupFailed()
+    {
+        var factory = new SequenceLdapConnectionFactory(new FakeLdapConnection().WithNoMatch());
+        var result = await Login(new LdapAuthService(Opts("dc2"), factory), "jdoe");
+
+        Assert.Equal(LdapAuthFailure.GroupLookupFailed, result.Failure);
+        Assert.Single(factory.Created);
+    }
+
+    [Fact]
+    public async Task TwoEntriesAfterAGoodBind_IsGroupLookupFailed()
+    {
+        var factory = new SequenceLdapConnectionFactory(new FakeLdapConnection().WithDuplicateMatch());
+        var result = await Login(new LdapAuthService(Opts(), factory), "jdoe");
+
+        Assert.Equal(LdapAuthFailure.GroupLookupFailed, result.Failure);
+    }
+
+    [Fact]
+    public async Task NoGroups_IsGroupLookupFailed()
+    {
+        var factory = new SequenceLdapConnectionFactory(
+            new FakeLdapConnection().WithUserEntry("CN=Jane,DC=x", Array.Empty<string>(), "Jane"));
+        var result = await Login(new LdapAuthService(Opts(), factory), "jdoe");
+
+        Assert.Equal(LdapAuthFailure.GroupLookupFailed, result.Failure);
+    }
+
+    [Fact]
+    public async Task Username_IsCanonicalisedFromTheEntry_NotTheTypedCase()
+    {
+        var factory = new SequenceLdapConnectionFactory(new FakeLdapConnection().WithUserEntry(
+            "CN=Jane Doe,OU=u,DC=x", new[] { "CN=UG_A,OU=g,DC=x" }, "Jane Doe",
+            new Dictionary<string, string[]> { ["sAMAccountName"] = new[] { "jdoe" } }));
+        var result = await Login(new LdapAuthService(Opts(), factory), @"NAM\JDOE");
+
+        Assert.True(result.Succeeded);
+        Assert.Equal("jdoe", result.Username);
+    }
+
+    [Fact]
+    public async Task Username_FallsBackToTheTypedName_WhenTheEntryLacksTheAttribute()
+    {
+        var factory = new SequenceLdapConnectionFactory(Jane());
+        var result = await Login(new LdapAuthService(Opts(), factory), "JDOE");
+
+        Assert.True(result.Succeeded);
+        Assert.Equal("JDOE", result.Username);
+    }
+
+    [Fact]
+    public async Task Search_RequestsTheUserNameAttribute()
+    {
+        var factory = new SequenceLdapConnectionFactory(Jane());
+        await Login(new LdapAuthService(Opts(), factory), "jdoe");
+
+        Assert.Equal(new[] { "displayName", "memberOf", "sAMAccountName" }, factory.Created[0].SearchAttributes);
+    }
+
+    [Fact]
+    public async Task Search_DeDuplicatesTheUserNameAttribute_CaseInsensitively()
+    {
+        var factory = new SequenceLdapConnectionFactory(Jane());
+        var svc = new LdapAuthService(Opts() with { DisplayNameAttribute = "SAMACCOUNTNAME" }, factory);
+        await Login(svc, "jdoe");
+
+        Assert.Equal(new[] { "SAMACCOUNTNAME", "memberOf" }, factory.Created[0].SearchAttributes);
+    }
+}

ZB.MOM.WW.Auth/tests/ZB.MOM.WW.Auth.Ldap.Tests/LdapEndpointParserTests.cs

@@ -0,0 +1,35 @@
+using ZB.MOM.WW.Auth.Ldap.Internal;
+
+namespace ZB.MOM.WW.Auth.Ldap.Tests;
+
+public class LdapEndpointParserTests
+{
+    [Theory]
+    [InlineData("dc2.example.com", 636, "dc2.example.com", 636)]      // bare host -> default port
+    [InlineData("dc2.example.com:3894", 636, "dc2.example.com", 3894)] // explicit port wins
+    [InlineData("  dc2.example.com : 3894  ", 636, "dc2.example.com", 3894)] // whitespace tolerated
+    public void TryParse_ValidEntry_Succeeds(string entry, int defaultPort, string host, int port)
+    {
+        Assert.True(LdapEndpointParser.TryParse(entry, defaultPort, out var ep, out _));
+        Assert.Equal(host, ep.Host);
+        Assert.Equal(port, ep.Port);
+    }
+
+    [Theory]
+    [InlineData("")]              // empty
+    [InlineData("   ")]           // whitespace
+    [InlineData(":636")]          // empty host
+    [InlineData("dc2:0")]         // port below range
+    [InlineData("dc2:65536")]     // port above range
+    [InlineData("dc2:abc")]       // non-numeric port
+    [InlineData("2001:db8::1:636")] // IPv6 literal (unsupported, >1 colon)
+    public void TryParse_InvalidEntry_Fails(string entry)
+    {
+        Assert.False(LdapEndpointParser.TryParse(entry, 636, out _, out var error));
+        Assert.NotEmpty(error);
+    }
+
+    [Fact]
+    public void Endpoint_ToString_IsHostColonPort() =>
+        Assert.Equal("dc2:3894", new LdapEndpoint("dc2", 3894).ToString());
+}

ZB.MOM.WW.Auth/tests/ZB.MOM.WW.Auth.Ldap.Tests/LdapOptionsValidatorTests.cs

@@ -88,4 +88,66 @@ public class LdapOptionsValidatorTests
                 ServiceAccountDn = "",
             })
             .Failed);
+
+    [Fact]
+    public void Validator_Succeeds_WithValidFallbackServers() =>
+        Assert.False(new LdapOptionsValidator()
+            .Validate(null, Opts() with { FallbackServers = new[] { "dc2", "dc3:3894" } })
+            .Failed);
+
+    [Theory]
+    [InlineData("")]
+    [InlineData(":636")]
+    [InlineData("dc2:0")]
+    [InlineData("dc2:notaport")]
+    public void Validator_Fails_OnMalformedFallbackServer(string entry)
+    {
+        var result = new LdapOptionsValidator()
+            .Validate(null, Opts() with { FallbackServers = new[] { entry } });
+
+        Assert.True(result.Failed);
+        Assert.Contains(nameof(LdapOptions.FallbackServers), result.FailureMessage);
+    }
+
+    [Fact]
+    public void Validator_Skips_FallbackServers_WhenDisabled() =>
+        // Same inert-when-disabled rule as every other connection field: a disabled
+        // provider never opens a connection, so its fallback list is never walked.
+        Assert.False(new LdapOptionsValidator()
+            .Validate(null, Opts() with { Enabled = false, FallbackServers = new[] { ":::" } })
+            .Failed);
+
+    private static LdapOptions UserOpts() => Opts() with
+    {
+        BindMode = LdapBindMode.User,
+        UserBindTemplate = "{0}@corp.test",
+        ServiceAccountDn = "",
+        ServiceAccountPassword = "",
+    };
+
+    [Fact]
+    public void Defaults_KeepServiceAccountMode()
+    {
+        var o = new LdapOptions();
+        Assert.Equal(LdapBindMode.ServiceAccount, o.BindMode);
+        Assert.Equal("", o.UserBindTemplate);
+        Assert.True(o.AcceptDomainQualifiedNames);
+    }
+
+    [Fact]
+    public void Validator_UserMode_DoesNotRequireServiceAccount() =>
+        Assert.True(new LdapOptionsValidator().Validate(null, UserOpts()).Succeeded);
+
+    [Theory]
+    [InlineData("")]
+    [InlineData("   ")]
+    [InlineData("jdoe@corp.test")]
+    [InlineData("{0}{0}@corp.test")]
+    public void Validator_UserMode_RequiresExactlyOnePlaceholder(string template)
+    {
+        var result = new LdapOptionsValidator().Validate(null, UserOpts() with { UserBindTemplate = template });
+
+        Assert.True(result.Failed);
+        Assert.Contains(nameof(LdapOptions.UserBindTemplate), result.FailureMessage, StringComparison.Ordinal);
+    }
 }

ZB.MOM.WW.Auth/tests/ZB.MOM.WW.Auth.Ldap.Tests/LdapUserNameTests.cs

@@ -0,0 +1,47 @@
+using ZB.MOM.WW.Auth.Ldap.Internal;
+
+namespace ZB.MOM.WW.Auth.Ldap.Tests;
+
+public class LdapUserNameTests
+{
+    [Theory]
+    [InlineData("jdoe")]
+    [InlineData("  jdoe  ")]
+    [InlineData(@"NAM\jdoe")]
+    [InlineData("jdoe@corp.test")]
+    public void Accepting_ReducesToBareName(string input)
+    {
+        Assert.True(LdapUserName.TryNormalize(input, acceptDomainQualified: true, out var name));
+        Assert.Equal("jdoe", name);
+    }
+
+    [Theory]
+    [InlineData(@"NAM\jdoe")]
+    [InlineData("jdoe@corp.test")]
+    public void Strict_RefusesQualifiedNames(string input) =>
+        Assert.False(LdapUserName.TryNormalize(input, acceptDomainQualified: false, out _));
+
+    [Fact]
+    public void Strict_AcceptsBareName()
+    {
+        Assert.True(LdapUserName.TryNormalize(" jdoe ", acceptDomainQualified: false, out var name));
+        Assert.Equal("jdoe", name);
+    }
+
+    [Theory]
+    [InlineData(null)]
+    [InlineData("")]
+    [InlineData("   ")]
+    [InlineData("@x")]
+    [InlineData(@"\jdoe")]
+    [InlineData("jdoe@")]
+    [InlineData(@"NAM\")]
+    [InlineData(@"a\b\c")]
+    [InlineData("a@b@c")]
+    [InlineData(@"NAM\jdoe@corp.test")]
+    public void Unusable_IsRefused(string? input)
+    {
+        Assert.False(LdapUserName.TryNormalize(input, acceptDomainQualified: true, out var name));
+        Assert.Equal("", name);
+    }
+}

ZB.MOM.WW.Auth/tests/ZB.MOM.WW.Auth.Ldap.Tests/NovellSearchReferralTests.cs

@@ -0,0 +1,179 @@
+using Novell.Directory.Ldap;
+using Xunit;
+using ZB.MOM.WW.Auth.Ldap.Internal;
+
+namespace ZB.MOM.WW.Auth.Ldap.Tests;
+
+/// <summary>
+/// Covers <see cref="NovellLdapConnection.ReadEntries"/>' handling of AD continuation
+/// referrals. Active Directory returns one for any search based at a domain root, and
+/// Novell surfaces it by throwing from <c>Next()</c>. Before this was handled, the
+/// exception escaped the search stage and every login against
+/// <c>DC=nam,DC=zimmer,DC=com</c> failed as <c>ServiceAccountBindFailed</c> while the
+/// directory and credentials were healthy.
+/// </summary>
+public sealed class NovellSearchReferralTests
+{
+    [Fact]
+    public void Referral_before_the_only_entry_does_not_lose_it()
+    {
+        var results = new ScriptedResults(
+            Step.Referral(),
+            Step.Entry("CN=dohertj2,OU=Office Users,OU=Warsaw,DC=nam,DC=zimmer,DC=com"));
+
+        var entries = NovellLdapConnection.ReadEntries(results);
+
+        var entry = Assert.Single(entries);
+        Assert.Equal("CN=dohertj2,OU=Office Users,OU=Warsaw,DC=nam,DC=zimmer,DC=com", entry.Dn);
+    }
+
+    [Fact]
+    public void Referral_after_the_entry_does_not_discard_what_was_already_read()
+    {
+        var results = new ScriptedResults(
+            Step.Entry("CN=a,DC=nam,DC=zimmer,DC=com"),
+            Step.Referral());
+
+        var entries = NovellLdapConnection.ReadEntries(results);
+
+        Assert.Equal("CN=a,DC=nam,DC=zimmer,DC=com", Assert.Single(entries).Dn);
+    }
+
+    /// <summary>
+    /// The real shape observed against the NAM domain root: several references interleaved
+    /// with the one user entry. All three of AD's usual continuation points are present.
+    /// </summary>
+    [Fact]
+    public void Multiple_interleaved_referrals_still_yield_every_entry()
+    {
+        var results = new ScriptedResults(
+            Step.Referral(),
+            Step.Entry("CN=a,DC=nam,DC=zimmer,DC=com"),
+            Step.Referral(),
+            Step.Referral(),
+            Step.Entry("CN=b,DC=nam,DC=zimmer,DC=com"));
+
+        var entries = NovellLdapConnection.ReadEntries(results);
+
+        Assert.Equal(["CN=a,DC=nam,DC=zimmer,DC=com", "CN=b,DC=nam,DC=zimmer,DC=com"],
+            entries.Select(e => e.Dn));
+    }
+
+    /// <summary>
+    /// A referral-only result set is a legitimate "nothing here" answer, and must come back
+    /// empty rather than throwing — <c>LdapAuthService</c> reads empty as <c>UserNotFound</c>,
+    /// which is the honest verdict, where an exception would become a system-side failure.
+    /// </summary>
+    [Fact]
+    public void Referrals_only_returns_empty_rather_than_throwing()
+    {
+        var results = new ScriptedResults(Step.Referral(), Step.Referral());
+
+        Assert.Empty(NovellLdapConnection.ReadEntries(results));
+    }
+
+    /// <summary>
+    /// A base <see cref="LdapException"/> carrying the Referral result code is treated the
+    /// same as the derived type — the catch filter must not depend on the exact class.
+    /// </summary>
+    [Fact]
+    public void Referral_result_code_on_a_base_exception_is_also_skipped()
+    {
+        var results = new ScriptedResults(
+            Step.Throwing(new LdapException("referral", LdapException.Referral, null)),
+            Step.Entry("CN=a,DC=nam,DC=zimmer,DC=com"));
+
+        Assert.Single(NovellLdapConnection.ReadEntries(results));
+    }
+
+    /// <summary>
+    /// Only referrals are swallowed. A genuine directory fault must still propagate so the
+    /// caller maps it to a system-side failure — silently returning a short result set would
+    /// turn "the directory is broken" into "no such user", which is the wrong answer to give
+    /// about someone's credentials.
+    /// </summary>
+    [Fact]
+    public void A_non_referral_fault_still_propagates()
+    {
+        var results = new ScriptedResults(
+            Step.Throwing(new LdapException("busy", LdapException.Busy, null)),
+            Step.Entry("CN=a,DC=nam,DC=zimmer,DC=com"));
+
+        var ex = Assert.Throws<LdapException>(() => NovellLdapConnection.ReadEntries(results));
+        Assert.Equal(LdapException.Busy, ex.ResultCode);
+    }
+
+    /// <summary>
+    /// A pathological client that throws a referral without advancing must not spin forever
+    /// inside a fail-closed authentication path.
+    /// </summary>
+    [Fact]
+    public void An_endless_referral_stream_terminates()
+    {
+        var results = new EndlessReferrals();
+
+        var entries = NovellLdapConnection.ReadEntries(results);
+
+        Assert.Empty(entries);
+        Assert.InRange(results.Calls, 1, 2_000);
+    }
+
+    // ---------------------------------------------------------------------
+
+    private sealed record Step(LdapEntry? Value, Exception? Throw)
+    {
+        public static Step Entry(string dn) => new(new LdapEntry(dn, new LdapAttributeSet()), null);
+        public static Step Referral() => new(null, new LdapReferralException("continuation reference"));
+        public static Step Throwing(Exception ex) => new(null, ex);
+    }
+
+    /// <summary>Replays a fixed script of entries and faults through the Novell result seam.</summary>
+    private sealed class ScriptedResults : ILdapSearchResults
+    {
+        private readonly Step[] _steps;
+        private int _index;
+
+        public ScriptedResults(params Step[] steps) => _steps = steps;
+
+        public int Count => _steps.Length;
+
+        public LdapControl[] ResponseControls => [];
+
+        public bool HasMore() => _index < _steps.Length;
+
+        public LdapEntry Next()
+        {
+            var step = _steps[_index++];
+            if (step.Throw is not null) throw step.Throw;
+            return step.Value!;
+        }
+
+        // The production reader drives HasMore()/Next() directly; enumeration is part of the
+        // interface but not part of the path under test.
+        public IEnumerator<LdapEntry> GetEnumerator() => throw new NotSupportedException();
+
+        System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator() => GetEnumerator();
+    }
+
+    /// <summary>Always claims more results and always throws a referral, never advancing.</summary>
+    private sealed class EndlessReferrals : ILdapSearchResults
+    {
+        public int Calls { get; private set; }
+
+        public int Count => int.MaxValue;
+
+        public LdapControl[] ResponseControls => [];
+
+        public bool HasMore() => true;
+
+        public LdapEntry Next()
+        {
+            Calls++;
+            throw new LdapReferralException("continuation reference");
+        }
+
+        public IEnumerator<LdapEntry> GetEnumerator() => throw new NotSupportedException();
+
+        System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator() => GetEnumerator();
+    }
+}

ZB.MOM.WW.Auth/tests/ZB.MOM.WW.Auth.Ldap.Tests/SequenceLdapConnectionFactory.cs

@@ -0,0 +1,51 @@
+using ZB.MOM.WW.Auth.Ldap.Internal;
+
+namespace ZB.MOM.WW.Auth.Ldap.Tests;
+
+/// <summary>
+/// Test double for <see cref="ILdapConnectionFactory"/> that hands out pre-scripted
+/// <see cref="FakeLdapConnection"/> instances in order, one per <see cref="Create"/> call —
+/// exactly matching the failover walk, which opens a fresh connection per endpoint attempt.
+/// Records every created connection for inspection. When the script runs out, returns a
+/// bare unscripted fake (which yields user-not-found) so a miscounted test fails loudly
+/// rather than reusing a stale fake.
+/// </summary>
+internal sealed class SequenceLdapConnectionFactory : ILdapConnectionFactory
+{
+    private readonly Queue<FakeLdapConnection> _script;
+
+    /// <summary>Scripts the connections handed out, in the order the walk will request them.</summary>
+    public SequenceLdapConnectionFactory(params FakeLdapConnection[] fakes)
+        => _script = new Queue<FakeLdapConnection>(fakes);
+
+    /// <summary>Every connection handed out, in creation order (scripted or overflow).</summary>
+    public List<FakeLdapConnection> Created { get; } = new();
+
+    /// <inheritdoc />
+    public ILdapConnection Create()
+    {
+        var fake = _script.Count > 0 ? _script.Dequeue() : new FakeLdapConnection();
+        Created.Add(fake);
+        return fake;
+    }
+}
+
+// ---------------------------------------------------------------------------
+// Smoke test: verifies the sequencing and overflow behaviour the failover tests rely on.
+// ---------------------------------------------------------------------------
+
+public class SequenceLdapConnectionFactorySmokeTests
+{
+    [Fact]
+    public void Create_ReturnsScriptedFakesInOrder_ThenOverflows()
+    {
+        var first = new FakeLdapConnection().ThrowOnConnect();
+        var second = new FakeLdapConnection();
+        var factory = new SequenceLdapConnectionFactory(first, second);
+
+        Assert.Same(first, factory.Create());
+        Assert.Same(second, factory.Create());
+        Assert.NotSame(second, factory.Create()); // overflow: fresh bare fake
+        Assert.Equal(3, factory.Created.Count);
+    }
+}

ZB.MOM.WW.Configuration/CLAUDE.md

@@ -75,3 +75,33 @@ Design documentation:
 - `~/Desktop/scadaproj/components/configuration/spec/SPEC.md` — normalized validation target
 - `~/Desktop/scadaproj/components/configuration/shared-contract/ZB.MOM.WW.Configuration.md` — proposed shared-library API
 - `~/Desktop/scadaproj/components/configuration/current-state/` — per-project current state (code-verified)
+
+## Index history (migrated from the scadaproj root CLAUDE.md)
+
+> Moved verbatim out of the scadaproj root `CLAUDE.md` on 2026-09-03 so it loads only when needed. **This file is now the authoritative place for this history — append new entries here and keep the root index row short.**
+
+### Narrative
+
+The configuration component is fully populated: a normalized [`spec`](components/configuration/spec/SPEC.md), a
+[`shared-contract`](components/configuration/shared-contract/ZB.MOM.WW.Configuration.md), three
+[`current-state`](components/configuration/current-state/) docs, and an adoption [`GAPS`](components/configuration/GAPS.md)
+backlog. Shared = the `IValidateOptions<T>` failure-accumulation base (`OptionsValidatorBase<T>`) +
+reusable rule primitives (`ValidationBuilder`: port / host:port / required / positive-duration / one-of /
+min-count) + `AddValidatedOptions<TOptions,TValidator>()` (bind + validate + `ValidateOnStart`) + the
+pre-host `ConfigPreflight` aggregator (generalizes ScadaBridge's `StartupValidator`, byte-compatible
+message); left per-project = each app's options classes + domain rules, and OtOpcUa's
+draft/generation-content validation (DB-side `sp_ValidateDraft`; its C# `DraftValidator` is dormant).
+
+The shared library is **built and lives in this repo** at [`ZB.MOM.WW.Configuration/`](ZB.MOM.WW.Configuration/)
+(.NET 10; single package `ZB.MOM.WW.Configuration`; 42 tests (verified 2026-07-18); `dotnet pack` → 1 nupkg @ 0.1.0).
+The implementation plan is at
+[`docs/plans/2026-06-01-zb-mom-ww-configuration-shared-library.md`](docs/plans/2026-06-01-zb-mom-ww-configuration-shared-library.md).
+**Adopted across all three apps on 2026-06-01** (OtOpcUa, MxAccessGateway, ScadaBridge) on each repo's
+local default branch (`main`/`master`) — merged, **not yet pushed** to remotes; the package was first
+published to the Gitea feed. Behaviour-preserving onto `OptionsValidatorBase`/`AddValidatedOptions`
+for MxGateway + ScadaBridge (validator messages byte-identical), `StartupValidator` → `ConfigPreflight`
+for ScadaBridge, and net-new `Ldap`/`OpcUa` validators for OtOpcUa. Per-app result tracked in
+[`components/configuration/GAPS.md`](components/configuration/GAPS.md).
+Build/test from `ZB.MOM.WW.Configuration/`: `dotnet test`. Consumer matrix: all three apps consume the
+single package; ScadaBridge is the heaviest adopter (per-module validators + `StartupValidator` →
+`ConfigPreflight`); OtOpcUa adoption is additive (it has no `IValidateOptions` usage today).

ZB.MOM.WW.GalaxyRepository/CLAUDE.md

@@ -0,0 +1,28 @@
+# ZB.MOM.WW.GalaxyRepository
+
+Shared Galaxy object-hierarchy SQL browse + `galaxy_repository.v1` gRPC service for the **ZB.MOM.WW SCADA family** (consumed by HistorianGateway and mxaccessgw). Build/test from this directory: `dotnet test`.
+
+## Index history (migrated from the scadaproj root CLAUDE.md)
+
+> Moved verbatim out of the scadaproj root `CLAUDE.md` on 2026-09-03 so it loads only when needed. **This file is now the authoritative place for this history — append new entries here and keep the root index row short.**
+
+### Narrative
+
+The Galaxy Repository component normalizes the **Galaxy object-hierarchy SQL browse + reusable gRPC service**
+that was previously embedded in `mxaccessgw`. Shared = canonical `galaxy_repository.v1` proto (wire-compatible
+with `mxaccessgw`'s existing contract so OtOpcUa's `GalaxyRepositoryClient` is unaffected), the SQL browse
+provider (`HierarchySql` / `AttributesSql` validated reverse-engineered queries), in-memory hierarchy cache +
+snapshot + deploy-poll refresh `BackgroundService`, `GalaxyHierarchyProjector`, and `AddZbGalaxyRepository` /
+`MapZbGalaxyRepository` DI extension. Left per-consumer = section path, subtree auth filtering, and any
+app-specific paging defaults.
+
+The shared library is **built and lives in this repo** at [`ZB.MOM.WW.GalaxyRepository/`](ZB.MOM.WW.GalaxyRepository/)
+(.NET 10; single package `ZB.MOM.WW.GalaxyRepository`; `dotnet pack` → 1 nupkg @ 0.1.0, **published to
+the Gitea NuGet feed** `gitea.dohertylan.com/api/packages/dohertj2/nuget`). The design doc is at
+[`docs/plans/2026-06-23-historian-gateway-design.md`](docs/plans/2026-06-23-historian-gateway-design.md) (§10, component 1).
+**Consumed by HistorianGateway** as a `PackageReference` from that Gitea feed, pinned at `0.2.0` (originally a
+cross-repo `ProjectReference` to this scadaproj tree; switched to the feed package 2026-06-24; bumped `0.1.0` → `0.2.0` 2026-06-25).
+**mxaccessgw adoption is a tracked follow-on** — once adopted, mxaccessgw's inline Galaxy browse code is replaced
+by the shared lib (the `galaxy_repository.v1` wire contract is already identical, so OtOpcUa and ScadaBridge
+clients are unaffected). Build/test from `ZB.MOM.WW.GalaxyRepository/`: `dotnet test`.
+Consumer matrix: HistorianGateway (initial); mxaccessgw (follow-on adoption).

ZB.MOM.WW.Health/CLAUDE.md

@@ -75,3 +75,30 @@ Design documentation:
 - `~/Desktop/scadaproj/components/health/spec/SPEC.md` — normalized three-tier target
 - `~/Desktop/scadaproj/components/health/shared-contract/ZB.MOM.WW.Health.md` — proposed API (aligned to shipped code)
 - `~/Desktop/scadaproj/components/health/current-state/` — per-project current state (code-verified)
+
+## Index history (migrated from the scadaproj root CLAUDE.md)
+
+> Moved verbatim out of the scadaproj root `CLAUDE.md` on 2026-09-03 so it loads only when needed. **This file is now the authoritative place for this history — append new entries here and keep the root index row short.**
+
+### Component-table row as it read in the index
+
+| Component | Status | Goal | Design | Implementation |
+|---|---|---|---|---|
+| Health (readiness / liveness / active-node) | **Adopted** (feed carries `0.1.0`–`0.3.0`, current lib version **`0.3.0`**; **all 4 apps** reference it + wire `MapZbHealth`). **`0.2.0` (2026-07-24)** added an optional per-entry `data` object to the canonical health JSON and made `AkkaClusterHealthCheck` publish its cluster view (`leader`/`selfAddress`/`selfRoles`/`memberCount`/`unreachableCount`) — additive, so pre-0.2.0 payloads still parse. **`0.2.1` (2026-07-24) is a behaviour fix with a live consumer impact:** the active tier now returns **Unhealthy (503)** on a role-member-but-not-leader node instead of `Degraded`, which the spec maps to 200. The old value made `/health/active` answer 200 on *every* node, so it could not distinguish active from standby over HTTP at all — and OtOpcUa's Traefik routes the admin UI by exactly this probe, so its leader-pinning had silently never worked ([`lmxopcua#494`](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/494)). **`0.3.0` (2026-07-24) finished that fix and ENDED a family-wide duplication.** 0.2.1 made the tier a real 503 but still selected by `RoleLeader`; on a rebuilt rig the two orderings diverge live (akka leader `central-1`, oldest admin `central-2` — the singleton host), so it would have pinned Traefik to the node *not* hosting the work. **Leadership is address-ordered (host, then port) and has no relationship to time; singleton placement is age-ordered.** They agree on a freshly-formed cluster — which is why single-node and happy-path tests never caught it — and diverge permanently after any restart. Both Akka apps had already discovered this independently and each written a private replacement rather than use the shared check (ScadaBridge `OldestNodeActiveHealthCheck`, OtOpcUa `ClusterPrimaryHealthCheck`), so the package's entire active-node surface had **zero consumers — not unused but avoided, twice, for the same reason.** 0.3.0 promotes those two into one `ClusterActiveNode` primitive (oldest `Up` member, optional role scope, plus an ordered role-preference so a fused node answers for the role its singletons are pinned to); `ActiveNodeHealthCheck` and `AkkaActiveNodeGate` both delegate to it, and results now carry `activeRole`/`selfAddress`/`activeNode` in the 0.2.0 `data` object so a **standby reports who IS active**. Two further behaviour changes: a node holding **none** of the preferred roles is now **Unhealthy**, not Healthy — the old "not applicable ⇒ Healthy" is exactly what made the tier answer 200 everywhere, and the case is reachable (OtOpcUa's `RoleParser` admits `dev`-only and `cluster-`only nodes); and identity compares `UniqueAddress`, so a node restarted on the same host:port is correctly a different member. **Both apps migrated and their private copies deleted** — OtOpcUa (`feat/health-0.3.0-active-node` @ `88adec04`) also routes `RedundancyStateActor.SelectOldestUpMemberOfRole` through the shared primitive, so the tier and the OPC UA `ServiceLevel` 250/240 split cannot drift; ScadaBridge (`feat/site-node-health` @ `5d4853a1`) routes `ActiveNodeEvaluator` through it, unifying the S&F delivery gate, the heartbeat `IsActive` stamp, the inbound-API gate and the health tier on one rule. ScadaBridge keeps `SitePairActiveNodeHealthCheck` deliberately — it is a thin adapter over the site's own role-scoped `IClusterNodeProvider` (itself now backed by the shared rule), not a second copy of it. mxgw/HistorianGateway have no Akka cluster and are unaffected. **Both app branches are MERGED and PUSHED** (verified 2026-08-07): OtOpcUa `88adec04` is on `origin/master`; ScadaBridge merged via `b3dc17a5` (2026-07-24) on `origin/main`, private-copy deletions included. | Shared `ZB.MOM.WW.Health` lib | [`components/health/`](components/health/) | [`ZB.MOM.WW.Health/`](ZB.MOM.WW.Health/) |
+
+### Narrative
+
+The health component is fully populated: a normalized [`spec`](components/health/spec/SPEC.md), a
+[`shared-contract`](components/health/shared-contract/ZB.MOM.WW.Health.md), three
+[`current-state`](components/health/current-state/) docs, and an adoption [`GAPS`](components/health/GAPS.md)
+backlog. Shared = three-tier endpoint convention (ready/active/healthz) + canonical JSON writer +
+`IActiveNodeGate` seam + `GrpcDependencyHealthCheck` + `AkkaClusterHealthCheck` + `ActiveNodeHealthCheck`
++ `DatabaseHealthCheck<TContext>`; left per-project = which probes each app registers,
+orchestrator wiring, and ScadaBridge's distributed health-monitoring pipeline.
+
+The shared library is **built and lives in this repo** at [`ZB.MOM.WW.Health/`](ZB.MOM.WW.Health/)
+(.NET 10; 3 packages — `ZB.MOM.WW.Health`, `ZB.MOM.WW.Health.Akka`, `ZB.MOM.WW.Health.EntityFrameworkCore`;
+64 tests (verified 2026-07-18); `dotnet pack` → 3 nupkgs @ 0.1.0).
+**Adopted by all four apps** — verified 2026-07-18 (each references the package and wires `MapZbHealth`); the previous "not yet adopted" claim was stale. Per-app probe coverage vs the spec has NOT been re-verified — see [`components/health/GAPS.md`](components/health/GAPS.md).
+Build/test from `ZB.MOM.WW.Health/`: `dotnet test`. Consumer matrix: MxAccessGateway → core only;
+OtOpcUa & ScadaBridge → all three packages.

ZB.MOM.WW.LocalDb/Directory.Build.props

@@ -5,7 +5,7 @@
     <Nullable>enable</Nullable>
     <ImplicitUsings>enable</ImplicitUsings>
     <LangVersion>latest</LangVersion>
-    <Version>0.1.3</Version>
+    <Version>0.3.4</Version>
     <ManagePackageVersionsCentrally>true</ManagePackageVersionsCentrally>
     <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
     <!-- NU1510 (package-pruning advice): the Replication and Tests projects carry a

ZB.MOM.WW.LocalDb/README.md

@@ -10,16 +10,143 @@ Change capture uses SQLite triggers writing a single HLC-stamped oplog inside th
 consumer's own write transaction (crash-consistent, non-bypassable). Convergence is
 last-writer-wins over a hybrid logical clock. All timestamps are UTC.
 
-## Packages (all `0.1.3`)
+## Packages (all `0.3.4`)
 
 | Package | Contents |
 |---|---|
-| `ZB.MOM.WW.LocalDb` | Embedded DB host: `ILocalDb` SQL surface (`ExecuteAsync` / `QueryAsync` / transactions), `RegisterReplicated(table)` (PK validation + trigger install + oplog schema), `HybridLogicalClock`, `AddZbLocalDb()` DI. |
+| `ZB.MOM.WW.LocalDb` | Embedded DB host: `ILocalDb` SQL surface (`ExecuteAsync` / `QueryAsync` / transactions), `RegisterReplicated(table[, baselineExistingRows])` (PK + BLOB + foreign-key validation, trigger install, oplog schema), `DeregisterReplicated(table)` (trigger removal + oplog/ledger prune), `HybridLogicalClock`, `AddZbLocalDb()` DI. |
 | `ZB.MOM.WW.LocalDb.Contracts` | `localdb_sync.v1` proto + generated gRPC types (packable; a future peer could be non-.NET). |
 | `ZB.MOM.WW.LocalDb.Replication` | gRPC sync engine: passive `LocalDbSyncService`, initiating `SyncBackgroundService`, `MaintenanceBackgroundService`, LWW apply, watermark acks + pruning, snapshot resync, `ISyncStatus`, `LocalDbMetrics`, `AddZbLocalDbReplication()` / `MapZbLocalDbSync()`. |
 
 > Status: **published to the Gitea feed; adopted by ScadaBridge (Phases 1+2) and OtOpcUa (Phase 1).**
 >
+> - `0.3.4` (2026-08-20) — **a converged pair no longer re-streams a full snapshot on every
+>   reconnect once the sender's oplog has been ack-pruned to empty.** Found on the ScadaBridge
+>   docker rig (its register row 106): a rebuilt node (watermark 0) was healed by a snapshot
+>   stamped `as_of_seq = MAX(seq)` — which is 0 on an empty oplog — so its MAX-guarded watermark
+>   stayed 0, and the healthy side's empty-oplog horizon rule (`0.1.2`: streamable from
+>   `last_acked_seq + 1`) saw a gap on every later handshake and re-sent the whole database
+>   (127k rows per restart on the rig) until it happened to write a new row. The as-of is now
+>   `max(MAX(seq), last_acked_seq)`: every delta pruned below `last_acked_seq` is already folded
+>   into the rows the snapshot carries, so that is the floor it may claim. Sender-side only,
+>   wire-compatible, no schema change; pinned by
+>   `WipedPeer_SnapshotLiftsItsWatermark_SoTheNextReconnectSendsNoSecondSnapshot`.
+> - `0.3.3` (2026-08-20) — two latent apply-path hazards from the ScadaBridge adversarial
+>   architecture review (P4-7). Neither had a live trigger in any current consumer; both are the
+>   kind that surface as silent divergence, not as an error. Wire-compatible: no proto change, no
+>   `lib_schema_version` bump, no on-disk schema change.
+>   - **A replicated table may no longer be the parent of a row-mutating `ON DELETE` foreign key.**
+>     The apply path writes remote rows with `INSERT OR REPLACE`, and REPLACE resolves a PK conflict
+>     by DELETING the existing row first. Connections run `PRAGMA foreign_keys=ON`, so that implicit
+>     delete fires the child's `ON DELETE CASCADE` / `SET NULL` / `SET DEFAULT` — while the capture
+>     guard is raised. The child rows change on the applying node ONLY, produce no oplog entry, and
+>     the pair diverges with nothing able to detect it (LWW compares only rows it was told about).
+>     `RegisterReplicated` now inspects every foreign key in the database and refuses such a table
+>     with a `LocalDbRegistrationException` naming the offending child column and action.
+>     `RESTRICT` / `NO ACTION` remain fine — they abort the statement instead of mutating rows
+>     behind capture's back, which the apply path already handles as per-entry poison. Checked at
+>     registration; SQLite cannot add a foreign key to an existing table, so a child table created
+>     later is caught at the next startup's registration rather than at the DDL.
+>   - **An entry's row write and its ledger write are now atomic with each other.** They are two
+>     statements, and SQLite aborts only the failing one — so a statement-level fault between them
+>     (the case the per-entry poison catch exists for) committed a half-applied entry: a tombstone
+>     whose table row was gone while `__localdb_row_version` still called it live, which the snapshot
+>     streamer can only warn about and skip, so the peer never learned the row died; or an upsert
+>     whose new row carried the OLD version and therefore lost the next comparison it should have
+>     won. The pair is wrapped in a per-entry `SAVEPOINT` that is rolled back before dead-lettering,
+>     so a poison entry leaves both writes or neither and the rest of the batch is untouched.
+> - `0.3.2` (2026-08-19) — two Medium findings from the ScadaBridge adversarial architecture review
+>   (P4-1, P4-3). Wire-compatible with every earlier peer: no proto change, no `lib_schema_version`
+>   bump, no on-disk schema change — the keepalive is transport-level and the rest is local policy.
+>   - **A live session now ends itself when a cap prune eats deltas it has not sent.** The pump reads
+>     `seq > sentThruSeq`, which resumes SILENTLY above a pruned hole; the peer acks the later seqs
+>     and the pair is durably divergent for the whole life of that session. `needs_snapshot` (set in
+>     the prune's own transaction) only heals a RECONNECT, and nothing forced one — so on a stable
+>     LAN with a slow-but-alive peer the divergence was unbounded. Every pump iteration now compares
+>     the oplog horizon (`MIN(seq)`, one b-tree seek) against its sent watermark and throws
+>     `LocalDbOplogGapException` when deltas it still owes are gone. The horizon, not the flag,
+>     because the flag is also raised for reasons that do not invalidate the running session.
+>   - **The sync stream has HTTP/2 keepalive, and the server's single-stream guard is newest-wins.**
+>     Neither end ever timed out on silence (a converged pair is legitimately quiet), so a hard crash
+>     or partition of the dialling node left the passive node's handler parked in `MoveNext` on a
+>     half-open stream — holding the guard, reporting itself connected, and rejecting the rebooted
+>     peer's reconnects with `AlreadyExists`, indefinitely on an idle pair. The initiator now dials
+>     with `KeepAlivePingDelay` / `KeepAlivePingTimeout` (new options, 60s/20s, `Always` policy so
+>     idle connections ping too), and a fresh authenticated `Sync` cancels the incumbent and waits
+>     for it to unwind before starting — the guard serializes sessions, it was never a fence, and an
+>     authenticated same-key dialler is by definition the peer. An incumbent that will not yield
+>     within the takeover timeout still gets `AlreadyExists`, because two live sessions over one
+>     database is worse than a refused dial.
+> - `0.3.1` (2026-08-18) — a faulting session flushes its control lane before cancelling its writer,
+>   so a locally-detected handshake mismatch actually reaches the peer instead of leaving it to
+>   reconnect-loop against a generic disconnect.
+> - `0.3.0` (2026-08-15) — **the snapshot-resync flag is per-table.** On-disk bookkeeping schema
+>   goes to **v3** (new `__localdb_snapshot_state`, upgraded in place on open); wire-compatible with
+>   `0.1.x`/`0.2.x` peers, and `lib_schema_version` deliberately stays `1` because it is compared
+>   fail-closed for equality.
+>   - **`RegisterReplicated(table, baselineExistingRows: true)` now flags a resync for THAT TABLE.**
+>     Through `0.2.x` the flag was one bit for the whole database, so baselining one table
+>     re-streamed every registered table in full — and on a pair enabling replication late, both
+>     nodes did that to each other at once. A snapshot now streams only the flagged tables.
+>   - **The database-wide flag survives, unchanged, for the cases that genuinely need it**: an oplog
+>     cap prune or an impossible peer ack loses deltas for the oplog as a whole, so nothing short of
+>     every table is sound. What moved is only where a *baseline* records its debt.
+>   - **Additive wire extension, negotiated by capability, not by version.** `Handshake` gains
+>     `supports_partial_snapshot`; `SnapshotBegin` gains a `tables` scope where **empty means "all
+>     tables"** — which is precisely what a pre-`0.3.0` sender emits and what a pre-`0.3.0` receiver
+>     assumes. A peer that does not advertise the capability is served a **full** snapshot, because a
+>     scoped one would make it advance `last_applied_remote_seq` past deltas it never received.
+>   - **A partial snapshot moves no watermark on either side.** The receiver keeps
+>     `last_applied_remote_seq` (it is one number for the whole peer oplog, and the untouched tables'
+>     deltas still sit below the snapshot's as-of); the sender keeps pumping from where it was. The
+>     overlap re-delivers rows the snapshot already carried, which LWW absorbs.
+>   - **Snapshot debt now clears on acknowledgement, not on enqueue** (new `SnapshotAck`, exchanged
+>     only between `0.3.0` peers). A crash part-way through a snapshot re-streams the same scope on
+>     reconnect instead of losing it. Against a pre-`0.3.0` peer the `0.2.x` clear-on-enqueue
+>     behaviour is kept — there is no ack to wait for.
+> - `0.2.1` (2026-08-14) — two adversarial-review findings against `0.2.0`. Wire-compatible with
+>   `0.1.x`/`0.2.0` peers; no proto change, no `lib_schema_version` bump, no on-disk schema change.
+>   - **`DeregisterReplicated` no longer leaves the HLC without a crash anchor.** Clock recovery on
+>     open is `MAX(hlc_high_water, oplog HLCs, ledger HLCs, last_seen_hlc)`, and `hlc_high_water` was
+>     written only by graceful disposal — while deregistration deletes the table's oplog AND ledger
+>     rows. A crash after a deregister boot could therefore recover the clock below stamps the peer
+>     already holds, and LWW discards those as older. The current clock is now flushed inside the
+>     deregistration transaction, before the prune (and the flush is MAX-guarded, so it can never
+>     regress the anchor).
+>   - **The sync inbox is bounded (64 messages) instead of unbounded**, so inbound memory is capped
+>     at roughly `64 x MaxBatchBytes` rather than "whatever the peer sends" — the bad case being a
+>     mutual full snapshot of a large dataset, buffered whole on both nodes at once. A full inbox now
+>     stops draining the gRPC stream, which closes the peer's flow-control window.
+>   - **A snapshot this node owes is streamed alongside a running receive loop**, not from inside the
+>     handshake. That is what makes bounding safe: previously neither side drained its inbox while
+>     sending, so a mutual snapshot converged only because the inbox was unbounded. Interleaving was
+>     already safe (snapshot rows and deltas both merge through LWW, the watermark write is
+>     MAX-guarded) — only the ordering changed.
+>   - **Documented the cost of late baselining**: the snapshot flag is per-database, so baselining
+>     one table re-streams every registered table, in both directions on a late-enabled pair. Trim
+>     high-volume tables (event logs) before turning replication on late.
+> - `0.2.0` (2026-08-14) — performance + lifecycle work from the ScadaBridge architectural review.
+>   All of it is wire-compatible with `0.1.x` peers (no proto change, no `lib_schema_version` bump);
+>   the on-disk bookkeeping schema goes to **v2** and is upgraded in place on open.
+>   - **`DeregisterReplicated(table)`** — drops a table's capture triggers and prunes its oplog and
+>     row-version rows. A node that does not replicate should not be paying for capture on every
+>     write, and until now there was no way to stop once a file had been registered. Idempotent, and
+>     usable on triggers left behind by an earlier build.
+>   - **`RegisterReplicated(table, baselineExistingRows: true)`** — closes the late-opt-in hole: rows
+>     that predate registration were invisible to the peer forever, because the snapshot streamer
+>     pages from the row-version ledger and CDC never wrote them there. Opting in seeds the ledger at
+>     an LWW-safe floor (HLC `0`, this node's id) and flags a snapshot resync so they actually
+>     travel. Default is still off — the documented CDC contract is unchanged unless you ask.
+>   - **O(1) backlog depth** — `MAX(seq) - MAX(last_acked_seq, MIN(seq) - 1)` replaces a
+>     `COUNT(*)` over the whole unacked backlog that ran once a minute AND on every metrics scrape
+>     (measured 58 VM steps vs 100,020 on a 50k backlog).
+>   - **Byte-budget batching** — `MaxBatchBytes` (default 2 MB) bounds delta and snapshot messages by
+>     summed serialized size, with `MaxBatchSize` demoted to a secondary row cap. Consumers no longer
+>     have to tune the row count against their widest row to stay under the gRPC message limit.
+>     Inbound delta batches and acks already queued are coalesced into one transaction / one ack.
+>   - **`LwwApplier` prepares once per statement** instead of three SQL parses per applied row.
+>   - **Dropped `__localdb_oplog_hlc`** — an index no query ever used, costing a b-tree insert inside
+>     every captured consumer write. Existing files lose it on the next open.
 > - `0.1.3` (2026-07-21) — fixes the other half of the rebuilt-peer story: a rebuilt peer's OWN
 >   writes were silently dropped. `last_applied_remote_seq` is a watermark in the *peer's* seq
 >   space, and a rebuilt peer numbers from 1 again — so the healthy node advertised the old peer's
@@ -107,7 +234,8 @@ node still pushes its own deltas back. Only one side needs `PeerAddress`.
 | `PeerAddress` | `""` | gRPC address to dial. Empty ⇒ passive (initiator idles). |
 | `ApiKey` | `null` | Bearer token the initiator presents on its channel; `null` ⇒ no key header. |
 | `FlushInterval` | `250 ms` | How often the change pump flushes pending oplog deltas. |
-| `MaxBatchSize` | `500` | Max oplog entries per outbound batch (also the snapshot page size). |
+| `MaxBatchBytes` | `2 MB` | **Primary** batch cap: summed serialized bytes per outbound delta/snapshot message. Sized under the 4 MB gRPC default; a single row above the budget is still sent, alone. |
+| `MaxBatchSize` | `500` | **Secondary** row-count cap per outbound batch (also the snapshot DB page size). |
 | `MaxOplogRows` | `1_000_000` | Backlog ceiling; exceeding it flags a snapshot resync and prunes oldest rows. |
 | `MaxOplogAge` | `7 days` | Age ceiling; an unpruned entry older than this flags a snapshot resync. |
 | `TombstoneRetention` | `7 days` | How long deleted-row tombstones are kept before pruning. |
@@ -133,8 +261,15 @@ All options are validated at startup (`ZB.MOM.WW.Configuration` validators).
   delta/snapshot batches), so receiving never blocks on the ability to ack.
 - **Resync** — when a peer's watermark falls below the pruned oplog horizon, it is sent a
   full snapshot of the registered tables (streamed, keyset-paged) and deltas resume above
-  the snapshot's as-of seq. **Each node decides from its OWN state** whether it owes the
-  peer a snapshot.
+  the snapshot's as-of seq. When instead a single table was *baselined*, only that table is
+  streamed and **no watermark moves on either side** — a partial snapshot speaks for nothing
+  outside its own tables, so the deltas it overlaps are still pumped and merged by LWW.
+  **Each node decides from its OWN state** whether it owes the peer a snapshot, and a peer
+  that does not advertise `supports_partial_snapshot` is sent a full one. The snapshot is
+  streamed *after* the handshake, alongside a running receive loop, so a node sending one
+  still applies what the peer sends it — a mutual snapshot exchanges under flow control
+  instead of being buffered whole in memory. Between `0.3.0` peers the sender's flags are
+  cleared by the receiver's `SnapshotAck`, so a crash mid-snapshot re-streams it.
 - **Maintenance** — a background service on **both** roles enforces the oplog row/age caps
   and prunes expired tombstones every `MaintenanceInterval`.
 
@@ -160,14 +295,38 @@ All options are validated at startup (`ZB.MOM.WW.Configuration` validators).
   `RegisterReplicated`, because `json_object` cannot capture it). A *typeless* column can
   still hold a BLOB value at runtime, which then surfaces as a write-time error, not a
   registration error.
+- **A replicated table may not be the parent of a row-mutating `ON DELETE` foreign key**
+  (`CASCADE` / `SET NULL` / `SET DEFAULT` from any other table, or from itself). Applying a
+  remote row uses `INSERT OR REPLACE`, whose implicit delete fires that action with change
+  capture suppressed — the child rows would change on the applying node only and never
+  replicate, diverging the pair undetectably. `RegisterReplicated` refuses such a table
+  (`0.3.3`); use `ON DELETE NO ACTION`/`RESTRICT` and delete children explicitly, or leave
+  the parent unreplicated. The check runs at registration and reads the whole database's
+  foreign keys, so a child table created *after* registration is caught at the next startup,
+  not at the moment of the `CREATE TABLE`.
 - **CDC semantics** — rows that existed **before** `RegisterReplicated` are not captured
-  and are not snapshotted; only changes after registration are replicated. Re-baselining
-  pre-existing rows requires re-writing them.
+  and are not snapshotted; only changes after registration are replicated. **Opt out with
+  `RegisterReplicated(table, baselineExistingRows: true)`**, which seeds the row-version
+  ledger for existing rows at the baseline floor (HLC `0`) and flags a snapshot resync
+  **for that table** so they reach the peer. The floor is below every genuine stamp, so a baselined row loses
+  every comparison against a real remote write and wins only where the peer has no version
+  of that key at all. Two nodes that baseline the same key both hold HLC `0` and resolve by
+  the node-id tie-break — convergent, but arbitrary as to which content survives.
+- **A snapshot still ships whole tables.** The flag is per-table as of `0.3.0`, so baselining
+  one table streams that table and no other — but *that* table travels in full, and two
+  cases still stream **everything**: an oplog cap prune / impossible peer ack (delta
+  continuity is lost for the oplog as a whole), and any peer older than `0.3.0` (which
+  cannot be told a snapshot is partial). So **purge or trim high-volume tables such as event
+  logs before baselining them**, and expect one slow first sync. Bounded either way (batch
+  budget out, bounded inbox in).
 - **REAL ±Infinity** is carried in SQLite's JSON dialect on the wire (fine for the
   SQLite↔SQLite peers this library targets).
 - `AddZbLocalDb` / `AddZbLocalDbReplication` are **first-registration-wins** (idempotent).
-- A **mutual simultaneous snapshot** buffers the peer's inbound snapshot in memory while
-  this node is still sending its own (acceptable for the small datasets this lib targets).
+- **Inbound buffering is bounded, not zero.** The duplex inbox holds up to 64 messages, so
+  peak inbound memory is roughly `64 x MaxBatchBytes` (~128 MB at the 2 MB default, far
+  less in practice — messages are only that large when the sender has that much backlog).
+  Lower `MaxBatchBytes` to lower the ceiling. Beyond the bound the stream stops being
+  drained and the peer is slowed by flow control.
 
 ## Observability
 
@@ -196,8 +355,28 @@ provider is wired or the poll threw, so a DB failure never reads as "0 backlog")
 
 ## Ops notes
 
-- When the oplog row/age caps are exceeded, maintenance prunes to the ceiling and flags
-  `needs_snapshot` (disk safety over delta continuity); the peer then snapshot-resyncs.
+- When the oplog row/age caps are exceeded, maintenance prunes to the ceiling and flags the
+  **database-wide** `needs_snapshot` (disk safety over delta continuity); the peer then gets a
+  full snapshot. Per-table debt lives separately in `__localdb_snapshot_state` and is what
+  baselining writes.
+- **Upgrading a pair to `0.3.0`** needs no coordination: the two nodes negotiate scoped snapshots
+  by capability, so a `0.3.0` node paired with a `0.2.x` one simply keeps sending full snapshots
+  until both are upgraded. Rolling the file forward is one-way — a `0.2.x` build reopening a v3
+  file leaves `__localdb_snapshot_state` in place and ignores it, so any per-table debt recorded
+  by `0.3.0` is invisible to it (roll forward again, or re-baseline).
+- **Turning a table off** — `DeregisterReplicated(table)` drops its capture triggers and prunes
+  its oplog + ledger rows. It must be applied **symmetrically**: the handshake compares
+  registered-table digests fail-closed, so a node that drops a table its peer still
+  replicates stops syncing with a schema-mismatch error. Deregister on both nodes, or on
+  neither. Re-enabling later should pass `baselineExistingRows: true`, or the rows that
+  accumulated while capture was off stay invisible to the peer — but trim high-volume
+  tables first (see Limitations: baselining flags a full, all-tables snapshot). The HLC's
+  durable anchor is flushed in the same transaction, so a crash after a deregister boot
+  cannot bring the clock back below stamps peers already hold.
+- **On-disk schema upgrades** are applied in place when the file is opened, in one
+  transaction, and every step is safe to re-run. `schema_version` in `__localdb_meta` is
+  local-file state only — it is NOT the wire `lib_schema_version`, so a pair mid-upgrade
+  keeps syncing.
 - The **`__localdb_dead_letter`** table holds poison rows (bad JSON, null-row non-tombstone
   entries) and over-drift rejects (when `FailClosedOnDrift`). Inspect it to diagnose
   rejects.
@@ -208,13 +387,17 @@ provider is wired or the poll threw, so a DB failure never reads as "0 backlog")
 
 ## Testing
 
-144 tests, all offline / no live dependencies — unit (HLC ordering + restart
-persistence, trigger SQL generation, LWW decision table, watermark/pruning, digest
-computation), integration over two real SQLite DBs on an in-memory (and real) gRPC pipe
-(bidirectional convergence, delete/update races, offline-accumulate → reconnect,
-prune → snapshot resync), plus seeded randomized property tests (random interleaved op
-sequences → both DBs identical after quiesce). Family conventions apply
-(`TreatWarningsAsErrors`).
+199 tests, all offline / no live dependencies — unit (HLC ordering + restart
+persistence, deregistration's HLC anchor surviving an ungraceful reopen, trigger SQL
+generation, LWW decision table, watermark/pruning, digest computation, schema-version
+upgrade, deregistration idempotency, baseline-floor LWW safety, O(1) depth vs a seeded
+backlog, byte-budget chunking, bounded-inbox backpressure under a slow reader), integration
+over two real SQLite DBs on an in-memory (and real) gRPC pipe (bidirectional convergence,
+delete/update races, offline-accumulate → reconnect, prune → snapshot resync, mutual
+snapshot over a capacity-1 transport, single-table baseline streaming only that table,
+mixed-version interop where a peer without `supports_partial_snapshot` gets a full one,
+snapshot debt surviving until the peer's `SnapshotAck`), plus seeded randomized property tests (random interleaved op sequences → both DBs identical after quiesce).
+Family conventions apply (`TreatWarningsAsErrors`).
 
 Build / test from this directory:
 

ZB.MOM.WW.LocalDb/build/push.sh

@@ -16,9 +16,19 @@ set -euo pipefail
 : "${GITEA_NUGET_SOURCE:?set GITEA_NUGET_SOURCE to your Gitea NuGet feed URL}"
 : "${GITEA_NUGET_KEY:?set GITEA_NUGET_KEY to your Gitea access token}"
 
-dotnet pack -c Release -o ./artifacts
+# Named solution, named output, and a cd to the repo root, all for one reason: `dotnet pack`
+# with neither a project argument nor a fixed working directory packs whatever solution
+# happens to sit in the CALLER's directory — and then pushes it to a live feed under this
+# script's name. Invoked as `bash /path/to/ZB.MOM.WW.LocalDb/build/push.sh` from another
+# repo's checkout, the relative form packed and published 25 unrelated packages (recovered
+# with `dotnet nuget delete`, but only because the token happened to carry delete rights).
+# A publish script must be unambiguous about WHAT it publishes no matter where it is run.
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+cd "$REPO_ROOT"
 
-dotnet nuget push "./artifacts/*.nupkg" \
+dotnet pack ZB.MOM.WW.LocalDb.slnx -c Release -o "$REPO_ROOT/artifacts"
+
+dotnet nuget push "$REPO_ROOT/artifacts/*.nupkg" \
   --source "$GITEA_NUGET_SOURCE" \
   --api-key "$GITEA_NUGET_KEY" \
   --skip-duplicate

ZB.MOM.WW.LocalDb/src/ZB.MOM.WW.LocalDb.Contracts/Protos/localdb_sync.proto

@@ -16,6 +16,7 @@ message SyncMessage {
     SnapshotBegin snapshot_begin = 5; // sender switches to snapshot mode
     SnapshotBatch snapshot_batch = 6;
     SnapshotComplete snapshot_complete = 7;
+    SnapshotAck snapshot_ack = 8;     // 0.3.0+; only sent when the peer advertised support
   }
 }
 
@@ -24,6 +25,13 @@ message Handshake {
   uint32 lib_schema_version = 2;
   repeated TableDigest tables = 3;    // fail-closed on any mismatch
   int64 last_applied_remote_seq = 4;  // highest of YOUR oplog seqs I have applied
+  // 0.3.0+ capability, additive: I understand a snapshot scoped to a SUBSET of the registered
+  // tables (SnapshotBegin.tables) and I will answer a completed snapshot with a SnapshotAck.
+  // A pre-0.3.0 peer leaves this false — proto3 default — and a 0.3.0 sender then promotes any
+  // scoped snapshot to a full one and clears its flags on enqueue, exactly as 0.2.x did. This is
+  // why lib_schema_version stays 1: it is compared fail-closed for EQUALITY, so bumping it would
+  // stop 0.1.x/0.2.x pairs syncing outright.
+  bool supports_partial_snapshot = 5;
 }
 message TableDigest { string table_name = 1; string digest = 2; }
 message HandshakeAck { string node_id = 1; bool snapshot_required = 2; }
@@ -40,7 +48,12 @@ message OplogEntry {
 message DeltaBatch { repeated OplogEntry entries = 1; }
 message DeltaAck { int64 applied_thru_seq = 1; }
 
-message SnapshotBegin { int64 as_of_seq = 1; }
+// tables: the SCOPE of this snapshot (0.3.0+). EMPTY means every registered table — a full
+// snapshot, which is the only thing a pre-0.3.0 sender ever emits and the only thing a
+// pre-0.3.0 receiver can safely assume. A non-empty list means the snapshot is PARTIAL, so the
+// receiver must NOT advance last_applied_remote_seq to as_of_seq (deltas below it for the tables
+// NOT listed have not been delivered); it merges the rows through LWW and leaves the watermark.
+message SnapshotBegin { int64 as_of_seq = 1; repeated string tables = 2; }
 message SnapshotBatch { string table_name = 1; repeated SnapshotRow rows = 2; }
 message SnapshotRow {
   string pk_json = 1;
@@ -50,3 +63,8 @@ message SnapshotRow {
   bool is_tombstone = 5;
 }
 message SnapshotComplete { int64 as_of_seq = 1; }
+// Sent by the RECEIVER once a snapshot has been durably applied, so the sender clears the
+// per-table snapshot flags only after the peer actually holds the rows (a crash mid-snapshot then
+// re-streams the same scope on reconnect). Only exchanged when both peers advertised
+// supports_partial_snapshot; a pre-0.3.0 pair keeps the 0.2.x clear-on-enqueue behaviour.
+message SnapshotAck { int64 as_of_seq = 1; }

ZB.MOM.WW.LocalDb/src/ZB.MOM.WW.LocalDb.Replication/Internal/GrpcSyncDuplex.cs

@@ -12,19 +12,58 @@ namespace ZB.MOM.WW.LocalDb.Replication.Internal;
 internal static class GrpcSyncDuplex
 {
     /// <summary>
-    /// Wires <paramref name="reader"/> into an unbounded inbox channel drained by a background loop,
-    /// and exposes <paramref name="writer"/> as the duplex send. Returns the duplex plus the reader
-    /// loop's task so the caller can observe it. On clean stream end the inbox channel is completed;
-    /// on a reader fault it is completed WITH the exception — the session's handshake and receive
-    /// phases both await the inbox, so terminating it is what unblocks them when the call dies (they
-    /// would otherwise hang on a dead stream).
+    /// Inbound messages buffered between the transport and the session's receive loop.
+    /// </summary>
+    /// <remarks>
+    /// <para>
+    /// <b>Why bounded at all.</b> An unbounded inbox makes this process's memory a function of what
+    /// the PEER decides to send. The worst case is not hypothetical: a full snapshot streams every
+    /// row of every registered table, and both sides of a pair that enabled replication late
+    /// snapshot each other simultaneously — so an unbounded inbox holds an entire peer dataset (an
+    /// event log at its storage cap, say) in memory on both nodes at once. Bounding it converts that
+    /// into backpressure: the pump stops draining the gRPC stream, the HTTP/2 flow-control window
+    /// closes, and the SENDER slows down.
+    /// </para>
+    /// <para>
+    /// <b>Why 64.</b> Memory here is bounded by roughly capacity x
+    /// <see cref="ReplicationOptions.MaxBatchBytes"/> (the sender's own per-message budget), so the
+    /// ceiling is a configured number rather than a peer-controlled one. 64 is four times the
+    /// sender's outbound data-lane depth, so a healthy stream never reaches the bound and pays
+    /// nothing; it engages only when this node is genuinely slower than the peer for a sustained
+    /// burst, which is exactly when backpressure is wanted. Lower <c>MaxBatchBytes</c> to lower the
+    /// ceiling.
+    /// </para>
+    /// <para>
+    /// <b>What makes bounding safe.</b> Backpressure on the inbox deadlocks any design where both
+    /// peers can stop draining their inbox at the same time — mutual snapshots were exactly that
+    /// case while the send ran inside the handshake. <see cref="SyncSession"/> therefore starts its
+    /// receive loop BEFORE streaming a snapshot it owes, so the inbox is drained continuously for
+    /// the whole life of the session; see the remarks on its snapshot phase.
+    /// </para>
+    /// </remarks>
+    internal const int InboxCapacity = 64;
+
+    /// <summary>
+    /// Wires <paramref name="reader"/> into a bounded inbox channel (see
+    /// <see cref="InboxCapacity"/>) drained by a background loop, and exposes
+    /// <paramref name="writer"/> as the duplex send. Returns the duplex plus the reader loop's task
+    /// so the caller can observe it. On clean stream end the inbox channel is completed; on a reader
+    /// fault it is completed WITH the exception — the session's handshake and receive phases both
+    /// await the inbox, so terminating it is what unblocks them when the call dies (they would
+    /// otherwise hang on a dead stream).
     /// </summary>
     public static (SyncDuplex Duplex, Task ReaderTask) Create(
         IAsyncStreamReader<SyncMessage> reader,
         IAsyncStreamWriter<SyncMessage> writer,
         CancellationToken ct)
     {
-        var inbox = Channel.CreateUnbounded<SyncMessage>(new UnboundedChannelOptions { SingleWriter = true });
+        var inbox = Channel.CreateBounded<SyncMessage>(new BoundedChannelOptions(InboxCapacity)
+        {
+            // Wait, never Drop: every inbound message is protocol state (a delta the peer will not
+            // resend in-session, a snapshot frame, an ack). Dropping one is silent divergence.
+            FullMode = BoundedChannelFullMode.Wait,
+            SingleWriter = true,
+        });
         var readerTask = PumpInboundAsync(reader, inbox.Writer, ct);
         var duplex = new SyncDuplex
         {
@@ -44,7 +83,10 @@ internal static class GrpcSyncDuplex
         try
         {
             while (await reader.MoveNext(ct))
-                inbox.TryWrite(reader.Current);
+                // WriteAsync, not TryWrite: on a FULL bounded channel TryWrite returns false and the
+                // message is silently lost. Awaiting here is the backpressure — it stops draining the
+                // gRPC stream, which closes the peer's flow-control window.
+                await inbox.WriteAsync(reader.Current, ct);
         }
         catch (Exception ex)
         {

ZB.MOM.WW.LocalDb/src/ZB.MOM.WW.LocalDb.Replication/Internal/LwwApplier.cs

@@ -12,14 +12,29 @@ internal sealed record ApplyResult(long AppliedThruSeq, int Applied, int Discard
 /// Applies a peer's oplog batch under last-writer-wins in one transaction: the applying guard is
 /// raised so capture triggers skip these writes, per-entry LWW resolves against the local
 /// row-version (hlc, then node_id ordinal tie-break), poison rows are dead-lettered while the
-/// batch continues, and the remote watermark advances in the SAME transaction as the data so a
+/// batch continues — each entry's row write and ledger write sharing a savepoint so a dead-lettered
+/// entry leaves neither — and the remote watermark advances in the SAME transaction as the data so a
 /// crash-replay of the batch is a no-op (LWW's strict-greater compare loses on equal versions).
 /// This never writes <c>__localdb_oplog</c>: the 2-node topology does not re-forward applied deltas.
 /// </summary>
+/// <remarks>
+/// Every statement is issued through a <see cref="PreparedStatements"/> set held for the life of the
+/// batch, so each distinct statement is prepared ONCE and then rebound per row. The previous shape
+/// went through the generic <c>ILocalDbTransaction</c> façade, which builds a fresh
+/// <see cref="SqliteCommand"/> per call — three SQL parses per applied row (version lookup, the
+/// table's delete/upsert, the row-version write), on the path that drains a reconnect backlog.
+/// </remarks>
 internal sealed class LwwApplier(SqliteLocalDb db, Func<DateTimeOffset>? utcNow = null)
 {
     private readonly Func<DateTimeOffset> _utcNow = utcNow ?? (() => DateTimeOffset.UtcNow);
 
+    /// <summary>
+    /// Distinct statements prepared by the most recent <see cref="ApplyBatchAsync"/>. Test-only
+    /// observation: it is the direct evidence that statement count tracks the number of distinct
+    /// STATEMENTS rather than the number of rows.
+    /// </summary>
+    internal int LastBatchStatementCount { get; private set; }
+
     public async Task<ApplyResult> ApplyBatchAsync(IReadOnlyList<OplogEntryRecord> entries, CancellationToken ct)
     {
         if (entries.Count == 0)
@@ -42,13 +57,14 @@ internal sealed class LwwApplier(SqliteLocalDb db, Func<DateTimeOffset>? utcNow
             if (entry.Hlc > maxHlc) maxHlc = entry.Hlc;
         }
 
-        var sqlByTable = new Dictionary<string, (string Delete, string Upsert)>(StringComparer.Ordinal);
         var applied = 0;
         var discarded = 0;
         var deadLettered = 0;
 
-        await using var tx = await db.BeginTransactionAsync(ct);
-        await tx.ExecuteAsync("UPDATE __localdb_applying SET applying = 1 WHERE id = 1", null, ct);
+        await using var tx = await db.BeginPreparedTransactionAsync(ct);
+        using var sql = new PreparedStatements(tx.Connection, tx.Sqlite);
+
+        await sql.SetApplyingAsync(1, ct);
 
         foreach (var entry in entries)
         {
@@ -56,19 +72,16 @@ internal sealed class LwwApplier(SqliteLocalDb db, Func<DateTimeOffset>? utcNow
             // KEY table that silently inserts a phantom auto-rowid row — so it is poison, not a win.
             if (!entry.IsTombstone && entry.RowJson is null)
             {
-                await DeadLetterAsync(tx, entry, "non-tombstone entry with null row_json", utcNow, ct);
+                await sql.DeadLetterAsync(entry, "non-tombstone entry with null row_json", utcNow, ct);
                 deadLettered++;
                 continue;
             }
 
-            var local = await tx.QueryAsync(
-                "SELECT hlc, node_id FROM __localdb_row_version WHERE table_name = @t AND pk_json = @pk",
-                static r => (Hlc: r.GetInt64(0), NodeId: r.GetString(1)),
-                new { t = entry.TableName, pk = entry.PkJson }, ct);
+            var local = await sql.ReadLocalVersionAsync(entry.TableName, entry.PkJson, ct);
 
-            var remoteWins = local.Count == 0
-                || entry.Hlc > local[0].Hlc
-                || (entry.Hlc == local[0].Hlc && string.CompareOrdinal(entry.NodeId, local[0].NodeId) > 0);
+            var remoteWins = local is null
+                || entry.Hlc > local.Value.Hlc
+                || (entry.Hlc == local.Value.Hlc && string.CompareOrdinal(entry.NodeId, local.Value.NodeId) > 0);
 
             if (!remoteWins)
             {
@@ -76,32 +89,28 @@ internal sealed class LwwApplier(SqliteLocalDb db, Func<DateTimeOffset>? utcNow
                 continue;
             }
 
-            if (!sqlByTable.TryGetValue(entry.TableName, out var sql))
-            {
-                sql = BuildTableSql(db.ReplicatedTables[entry.TableName]);
-                sqlByTable[entry.TableName] = sql;
-            }
+            var table = db.ReplicatedTables[entry.TableName];
 
+            // An entry is TWO statements — the row change and its ledger row — and SQLite aborts
+            // only the failing statement, so a fault between them used to commit a half-applied
+            // entry: a tombstone whose table row is gone while __localdb_row_version still calls it
+            // live (which SnapshotStreamer can only warn about and skip, so the peer never learns
+            // the row died), or an upsert whose new row carries the OLD version and therefore loses
+            // the next LWW comparison it should win. A per-entry savepoint makes the pair
+            // all-or-nothing without widening the poison catch to the whole batch.
+            //
+            // Opened OUTSIDE the try on purpose: a SAVEPOINT that failed opened nothing, so the
+            // handler's ROLLBACK TO would name a savepoint that does not exist and fault in turn.
+            await sql.EnterEntryAsync(ct);
             try
             {
                 if (entry.IsTombstone)
-                    await tx.ExecuteAsync(sql.Delete, new { pk = entry.PkJson }, ct);
+                    await sql.DeleteAsync(table, entry.PkJson, ct);
                 else
-                    await tx.ExecuteAsync(sql.Upsert, new { row = entry.RowJson }, ct);
+                    await sql.UpsertAsync(table, entry.RowJson!, ct);
 
-                await tx.ExecuteAsync(
-                    "INSERT OR REPLACE INTO __localdb_row_version " +
-                    "(table_name, pk_json, hlc, node_id, is_tombstone, tombstone_utc) " +
-                    "VALUES (@t, @pk, @hlc, @node, @tomb, @tut)",
-                    new
-                    {
-                        t = entry.TableName,
-                        pk = entry.PkJson,
-                        hlc = entry.Hlc,
-                        node = entry.NodeId,
-                        tomb = entry.IsTombstone ? 1 : 0,
-                        tut = entry.IsTombstone ? utcNow : null,
-                    }, ct);
+                await sql.WriteRowVersionAsync(entry, utcNow, ct);
+                await sql.ReleaseEntryAsync(ct);
                 applied++;
             }
             catch (SqliteException ex)
@@ -109,60 +118,234 @@ internal sealed class LwwApplier(SqliteLocalDb db, Func<DateTimeOffset>? utcNow
                 // Deliberately narrow: only statement-level SQLite errors (e.g. json_extract on
                 // malformed row_json) are treated as per-entry poison — SQLite aborts just the
                 // statement, not the transaction, so the batch continues after dead-lettering.
-                // Any other exception propagates and rolls back the whole batch (fail-closed).
-                await DeadLetterAsync(tx, entry, ex.Message, utcNow, ct);
+                // Any other exception propagates with the savepoint still open, and disposing the
+                // outer transaction unwinds the whole batch (fail-closed).
+                try
+                {
+                    await sql.RollbackEntryAsync(ct);
+                    await sql.DeadLetterAsync(entry, ex.Message, utcNow, ct);
+                }
+                catch (Exception recoveryFailure) when (recoveryFailure is not OperationCanceledException)
+                {
+                    // Recovering from the poison entry itself failed — a wedged connection, an I/O
+                    // error, a dead-letter INSERT that faults. The outcome is unchanged and still
+                    // fail-closed (nothing commits; disposing the batch transaction unwinds it),
+                    // but reporting only the recovery failure would throw away the ORIGINAL cause,
+                    // which is the one that says WHY this entry is poison. Both are carried out:
+                    // the original as InnerException (AggregateException surfaces the first inner
+                    // there), the recovery failure alongside it with its own stack intact.
+                    throw new AggregateException(
+                        $"Failed to dead-letter a poison oplog entry for table '{entry.TableName}' " +
+                        $"(seq {entry.Seq}): {recoveryFailure.Message}. The entry's own fault is the " +
+                        "inner exception. The batch is not committed.",
+                        ex, recoveryFailure);
+                }
+
                 deadLettered++;
             }
         }
 
-        await tx.ExecuteAsync(
-            "UPDATE __localdb_peer_state SET " +
-            "last_applied_remote_seq = MAX(last_applied_remote_seq, @maxSeq), " +
-            "last_seen_hlc = MAX(last_seen_hlc, @maxHlc), last_sync_utc = @utc WHERE id = 1",
-            new { maxSeq, maxHlc, utc = utcNow }, ct);
+        await sql.AdvancePeerStateAsync(maxSeq, maxHlc, utcNow, ct);
+        await sql.SetApplyingAsync(0, ct);
+
+        LastBatchStatementCount = sql.StatementCount;
 
-        await tx.ExecuteAsync("UPDATE __localdb_applying SET applying = 0 WHERE id = 1", null, ct);
         await tx.CommitAsync(ct);
 
         db.Clock.Observe(maxHlc);
         return new ApplyResult(maxSeq, applied, discarded, deadLettered);
     }
 
-    private static Task DeadLetterAsync(
-        ILocalDbTransaction tx, OplogEntryRecord entry, string error, string utcNow, CancellationToken ct) =>
-        tx.ExecuteAsync(
-            "INSERT INTO __localdb_dead_letter " +
-            "(received_utc, table_name, pk_json, row_json, hlc, node_id, error) " +
-            "VALUES (@utc, @t, @pk, @row, @hlc, @node, @err)",
-            new
-            {
-                utc = utcNow,
-                t = entry.TableName,
-                pk = entry.PkJson,
-                row = entry.RowJson,
-                hlc = entry.Hlc,
-                node = entry.NodeId,
-                err = error,
-            }, ct);
-
-    private static (string Delete, string Upsert) BuildTableSql(ReplicatedTable table)
+    /// <summary>
+    /// The statement set one apply-batch runs on: each distinct statement is built and prepared on
+    /// first use, then rebound and re-executed per row. Statements whose SQL depends on the table
+    /// (delete / upsert) are cached per table; the rest are shared across all of them.
+    /// </summary>
+    /// <remarks>
+    /// Reusing the same <see cref="SqliteCommand"/> with unchanged <c>CommandText</c> is what makes
+    /// this prepare-once: Microsoft.Data.Sqlite holds the compiled <c>sqlite3_stmt</c> on the command
+    /// and only recompiles when the text changes. <see cref="SqliteCommand.Prepare"/> is called
+    /// explicitly anyway, so the intent is stated rather than inferred from an implementation detail.
+    /// </remarks>
+    private sealed class PreparedStatements(SqliteConnection connection, SqliteTransaction transaction) : IDisposable
     {
-        var name = Ident(table.Name);
+        private readonly List<SqliteCommand> _owned = [];
+        private readonly Dictionary<string, SqliteCommand> _deletes = new(StringComparer.Ordinal);
+        private readonly Dictionary<string, SqliteCommand> _upserts = new(StringComparer.Ordinal);
 
-        var pkMatch = string.Join(" AND ", table.PkColumns.Select(p => $"{Ident(p)} = json_extract(@pk, {JsonPath(p)})"));
-        var delete = $"DELETE FROM {name} WHERE {pkMatch}";
+        /// <summary>Name of the per-entry savepoint. Fixed: only one is ever open at a time.</summary>
+        private const string EntrySavepoint = "zb_lww_entry";
 
-        var columns = string.Join(", ", table.Columns.Select(c => Ident(c.Name)));
-        var values = string.Join(", ", table.Columns.Select(c => $"json_extract(@row, {JsonPath(c.Name)})"));
-        var upsert = $"INSERT OR REPLACE INTO {name} ({columns}) VALUES ({values})";
+        private SqliteCommand? _applying;
+        private SqliteCommand? _enterEntry;
+        private SqliteCommand? _releaseEntry;
+        private SqliteCommand? _rollbackEntry;
+        private SqliteCommand? _lookup;
+        private SqliteCommand? _rowVersion;
+        private SqliteCommand? _deadLetter;
+        private SqliteCommand? _peerState;
 
-        return (delete, upsert);
+        /// <summary>How many distinct statements this set has prepared.</summary>
+        public int StatementCount => _owned.Count;
+
+        public Task SetApplyingAsync(int value, CancellationToken ct)
+        {
+            var cmd = _applying ??= Create(
+                "UPDATE __localdb_applying SET applying = @applying WHERE id = 1", "@applying");
+            cmd.Parameters["@applying"].Value = value;
+            return cmd.ExecuteNonQueryAsync(ct);
+        }
+
+        /// <summary>Opens the per-entry savepoint inside the batch transaction.</summary>
+        public Task EnterEntryAsync(CancellationToken ct)
+        {
+            var cmd = _enterEntry ??= Create($"SAVEPOINT {EntrySavepoint}");
+            return cmd.ExecuteNonQueryAsync(ct);
+        }
+
+        /// <summary>Commits the entry's work into the batch transaction (pops the savepoint).</summary>
+        public Task ReleaseEntryAsync(CancellationToken ct)
+        {
+            var cmd = _releaseEntry ??= Create($"RELEASE {EntrySavepoint}");
+            return cmd.ExecuteNonQueryAsync(ct);
+        }
+
+        /// <summary>
+        /// Undoes everything the entry did and pops the savepoint, leaving the batch transaction
+        /// open and usable. Two statements, because ROLLBACK TO rewinds to the savepoint but leaves
+        /// it ON the stack — without the RELEASE, every dead-lettered entry would nest another one.
+        /// </summary>
+        public async Task RollbackEntryAsync(CancellationToken ct)
+        {
+            var cmd = _rollbackEntry ??= Create($"ROLLBACK TO {EntrySavepoint}");
+            await cmd.ExecuteNonQueryAsync(ct);
+            await ReleaseEntryAsync(ct);
+        }
+
+        public async Task<(long Hlc, string NodeId)?> ReadLocalVersionAsync(
+            string tableName, string pkJson, CancellationToken ct)
+        {
+            var cmd = _lookup ??= Create(
+                "SELECT hlc, node_id FROM __localdb_row_version WHERE table_name = @t AND pk_json = @pk",
+                "@t", "@pk");
+            cmd.Parameters["@t"].Value = tableName;
+            cmd.Parameters["@pk"].Value = pkJson;
+
+            await using var reader = await cmd.ExecuteReaderAsync(ct);
+            if (!await reader.ReadAsync(ct))
+                return null;
+            return (reader.GetInt64(0), reader.GetString(1));
+        }
+
+        public Task DeleteAsync(ReplicatedTable table, string pkJson, CancellationToken ct)
+        {
+            if (!_deletes.TryGetValue(table.Name, out var cmd))
+                _deletes[table.Name] = cmd = Create(BuildDelete(table), "@pk");
+            cmd.Parameters["@pk"].Value = pkJson;
+            return cmd.ExecuteNonQueryAsync(ct);
+        }
+
+        public Task UpsertAsync(ReplicatedTable table, string rowJson, CancellationToken ct)
+        {
+            if (!_upserts.TryGetValue(table.Name, out var cmd))
+                _upserts[table.Name] = cmd = Create(BuildUpsert(table), "@row");
+            cmd.Parameters["@row"].Value = rowJson;
+            return cmd.ExecuteNonQueryAsync(ct);
+        }
+
+        public Task WriteRowVersionAsync(OplogEntryRecord entry, string utcNow, CancellationToken ct)
+        {
+            var cmd = _rowVersion ??= Create(
+                "INSERT OR REPLACE INTO __localdb_row_version " +
+                "(table_name, pk_json, hlc, node_id, is_tombstone, tombstone_utc) " +
+                "VALUES (@t, @pk, @hlc, @node, @tomb, @tut)",
+                "@t", "@pk", "@hlc", "@node", "@tomb", "@tut");
+            cmd.Parameters["@t"].Value = entry.TableName;
+            cmd.Parameters["@pk"].Value = entry.PkJson;
+            cmd.Parameters["@hlc"].Value = entry.Hlc;
+            cmd.Parameters["@node"].Value = entry.NodeId;
+            cmd.Parameters["@tomb"].Value = entry.IsTombstone ? 1 : 0;
+            cmd.Parameters["@tut"].Value = entry.IsTombstone ? utcNow : DBNull.Value;
+            return cmd.ExecuteNonQueryAsync(ct);
+        }
+
+        public Task DeadLetterAsync(OplogEntryRecord entry, string error, string utcNow, CancellationToken ct)
+        {
+            var cmd = _deadLetter ??= Create(
+                "INSERT INTO __localdb_dead_letter " +
+                "(received_utc, table_name, pk_json, row_json, hlc, node_id, error) " +
+                "VALUES (@utc, @t, @pk, @row, @hlc, @node, @err)",
+                "@utc", "@t", "@pk", "@row", "@hlc", "@node", "@err");
+            cmd.Parameters["@utc"].Value = utcNow;
+            cmd.Parameters["@t"].Value = entry.TableName;
+            cmd.Parameters["@pk"].Value = entry.PkJson;
+            cmd.Parameters["@row"].Value = (object?)entry.RowJson ?? DBNull.Value;
+            cmd.Parameters["@hlc"].Value = entry.Hlc;
+            cmd.Parameters["@node"].Value = entry.NodeId;
+            cmd.Parameters["@err"].Value = error;
+            return cmd.ExecuteNonQueryAsync(ct);
+        }
+
+        public Task AdvancePeerStateAsync(long maxSeq, long maxHlc, string utcNow, CancellationToken ct)
+        {
+            var cmd = _peerState ??= Create(
+                "UPDATE __localdb_peer_state SET " +
+                "last_applied_remote_seq = MAX(last_applied_remote_seq, @maxSeq), " +
+                "last_seen_hlc = MAX(last_seen_hlc, @maxHlc), last_sync_utc = @utc WHERE id = 1",
+                "@maxSeq", "@maxHlc", "@utc");
+            cmd.Parameters["@maxSeq"].Value = maxSeq;
+            cmd.Parameters["@maxHlc"].Value = maxHlc;
+            cmd.Parameters["@utc"].Value = utcNow;
+            return cmd.ExecuteNonQueryAsync(ct);
+        }
+
+        private SqliteCommand Create(string sql, params string[] parameterNames)
+        {
+            var cmd = connection.CreateCommand();
+            try
+            {
+                cmd.Transaction = transaction;
+                cmd.CommandText = sql;
+                foreach (var name in parameterNames)
+                    cmd.Parameters.Add(new SqliteParameter(name, DBNull.Value));
+                cmd.Prepare();
+                _owned.Add(cmd);
+                return cmd;
+            }
+            catch
+            {
+                cmd.Dispose();
+                throw;
+            }
+        }
+
+        public void Dispose()
+        {
+            foreach (var cmd in _owned)
+                cmd.Dispose();
+            _owned.Clear();
+        }
+
+        private static string BuildDelete(ReplicatedTable table)
+        {
+            var pkMatch = string.Join(
+                " AND ", table.PkColumns.Select(p => $"{Ident(p)} = json_extract(@pk, {JsonPath(p)})"));
+            return $"DELETE FROM {Ident(table.Name)} WHERE {pkMatch}";
+        }
+
+        private static string BuildUpsert(ReplicatedTable table)
+        {
+            var columns = string.Join(", ", table.Columns.Select(c => Ident(c.Name)));
+            var values = string.Join(", ", table.Columns.Select(c => $"json_extract(@row, {JsonPath(c.Name)})"));
+            return $"INSERT OR REPLACE INTO {Ident(table.Name)} ({columns}) VALUES ({values})";
+        }
+
+        // Local copy of TriggerSqlGenerator's convention (that helper lives in the core assembly).
+        private static string Ident(string name) => "\"" + name.Replace("\"", "\"\"") + "\"";
+
+        // A single-quoted JSON path with a quoted key member, so a column name with reserved
+        // characters still resolves: $."col". The key's double quotes are backslash-escaped per
+        // JSON path grammar.
+        private static string JsonPath(string column) => "'$.\"" + column.Replace("\"", "\\\"") + "\"'";
     }
-
-    // Local copy of TriggerSqlGenerator's convention (that helper lives in the core assembly).
-    private static string Ident(string name) => "\"" + name.Replace("\"", "\"\"") + "\"";
-
-    // A single-quoted JSON path with a quoted key member, so a column name with reserved characters
-    // still resolves: $."col". The key's double quotes are backslash-escaped per JSON path grammar.
-    private static string JsonPath(string column) => "'$.\"" + column.Replace("\"", "\\\"") + "\"'";
 }

ZB.MOM.WW.LocalDb/src/ZB.MOM.WW.LocalDb.Replication/Internal/OplogStore.cs

@@ -24,8 +24,9 @@ internal sealed class OplogStore(ILocalDb db, ReplicationOptions options, Func<D
 
     /// <summary>
     /// Reads up to <paramref name="maxBatch"/> oplog entries with <c>seq &gt; afterSeq</c> in ascending
-    /// seq order. Caller contract: check <c>needs_snapshot</c> before trusting incremental reads —
-    /// after cap-pruning, reads silently resume past a gap.
+    /// seq order. Caller contract: this read resumes SILENTLY past a cap-pruned gap, so a caller
+    /// streaming incrementally must first confirm its position is still below the horizon — see
+    /// <see cref="GetMinSeqAsync"/>, which <c>SyncSession</c>'s pump checks on every iteration.
     /// </summary>
     public Task<IReadOnlyList<OplogEntryRecord>> ReadBatchAboveAsync(long afterSeq, int maxBatch, CancellationToken ct = default) =>
         db.QueryAsync(
@@ -70,12 +71,42 @@ internal sealed class OplogStore(ILocalDb db, ReplicationOptions options, Func<D
             "DELETE FROM __localdb_row_version WHERE is_tombstone = 1 AND tombstone_utc < @cutoff",
             new { cutoff = FormatUtc(_utcNow() - options.TombstoneRetention) }, ct);
 
-    /// <summary>Counts the unacked backlog: oplog entries above the peer's last-acked watermark.</summary>
+    /// <summary>
+    /// The unacked backlog — oplog entries above the peer's last-acked watermark — in O(1).
+    /// </summary>
+    /// <remarks>
+    /// <para>
+    /// <b>Why not <c>COUNT(*)</c>.</b> The obvious query walks every unacked row, and this value is
+    /// read once a minute by maintenance AND on every metrics scrape. A node whose peer has been
+    /// down for a day therefore pays a full scan of a million-row backlog, repeatedly, precisely
+    /// when it is already under stress.
+    /// </para>
+    /// <para>
+    /// <b>The formula.</b> <c>MAX(seq) - MAX(last_acked_seq, MIN(seq) - 1)</c>, floored at 0. Both
+    /// MIN and MAX over <c>seq</c> are single b-tree seeks (it is the rowid, and SQLite's min/max
+    /// optimization turns each into one seek), and the peer-state read is a one-row lookup — so the
+    /// whole thing is three seeks regardless of backlog size.
+    /// </para>
+    /// <para>
+    /// <b>Why it is exact.</b> <c>seq</c> is AUTOINCREMENT, so retained rows are a contiguous run
+    /// wherever nothing has punched a hole in the middle — and no ordinary path punches holes: ack
+    /// pruning deletes <c>seq &lt;= last_acked_seq</c> (a prefix), the row cap deletes below
+    /// <c>MAX(seq) - keep + 1</c> (a prefix), and the age cap deletes below an HLC cutoff, which is
+    /// a prefix because HLCs are stamped in seq order. Clamping the floor to <c>MIN(seq) - 1</c> is
+    /// what keeps the answer exact after a prune has moved the horizon above the ack watermark;
+    /// without it a cap-pruned oplog would report its whole seq range forever and re-flag the cap on
+    /// every tick.
+    /// </para>
+    /// <para>
+    /// <b>The one inexact case</b> is <c>ILocalDb.DeregisterReplicated</c>, which deletes one
+    /// table's rows from wherever they sit and so can leave holes. The estimate then over-reports by
+    /// at most the number of holes, until the watermark advances past them. Over-reporting is the
+    /// safe direction: it can only trigger a snapshot resync earlier, never suppress one.
+    /// </para>
+    /// </remarks>
     public async Task<long> GetOplogDepthAsync(CancellationToken ct = default)
     {
-        var rows = await db.QueryAsync(
-            "SELECT COUNT(*) FROM __localdb_oplog WHERE seq > (SELECT last_acked_seq FROM __localdb_peer_state WHERE id = 1)",
-            static r => r.GetInt64(0), null, ct);
+        var rows = await db.QueryAsync(DepthSql, static r => r.GetInt64(0), null, ct);
         return rows[0];
     }
 
@@ -84,7 +115,7 @@ internal sealed class OplogStore(ILocalDb db, ReplicationOptions options, Func<D
     /// metric-observation callbacks (gauge scrape / status poll); mirrors
     /// <see cref="GetOplogDepthAsync"/>'s query. Deliberate tradeoff: synchronous I/O on the
     /// metrics-collection thread is acceptable here because WAL-mode readers never block (or are
-    /// blocked by) writers, the COUNT is indexed-cheap, and the worst case is bounded by
+    /// blocked by) writers, the query is three b-tree seeks, and the worst case is bounded by
     /// <c>BusyTimeoutMs</c> — all at scrape frequency, not on any data path. May throw if the db
     /// is disposing — callers on the observation path swallow and drop the sample.
     /// </summary>
@@ -92,11 +123,18 @@ internal sealed class OplogStore(ILocalDb db, ReplicationOptions options, Func<D
     {
         using var conn = db.CreateConnection();
         using var cmd = conn.CreateCommand();
-        cmd.CommandText =
-            "SELECT COUNT(*) FROM __localdb_oplog WHERE seq > (SELECT last_acked_seq FROM __localdb_peer_state WHERE id = 1)";
+        cmd.CommandText = DepthSql;
         return (long)cmd.ExecuteScalar()!;
     }
 
+    // Both depth accessors run exactly this; keeping it in one place is what makes "mirrors the
+    // async query" a fact rather than a comment that can rot.
+    internal const string DepthSql =
+        "SELECT MAX(0, " +
+        "COALESCE((SELECT MAX(seq) FROM __localdb_oplog), 0) - " +
+        "MAX((SELECT last_acked_seq FROM __localdb_peer_state WHERE id = 1), " +
+        "COALESCE((SELECT MIN(seq) FROM __localdb_oplog), 1) - 1))";
+
     /// <summary>
     /// The highest seq this node has ever produced: the oplog's max, or — once acked rows have been
     /// pruned away — the ack watermark that pruned them. Nothing we emit can exceed it, so a peer
@@ -111,6 +149,24 @@ internal sealed class OplogStore(ILocalDb db, ReplicationOptions options, Func<D
         return rows[0];
     }
 
+    /// <summary>
+    /// The oldest oplog seq still present — the delta horizon — or 0 when the oplog is empty.
+    /// </summary>
+    /// <remarks>
+    /// One b-tree seek (<c>seq</c> is the rowid and SQLite optimizes MIN into a seek), which is what
+    /// makes it affordable for the pump to check on every flush interval. Compared against a
+    /// session's sent watermark it answers the only question the pump has: is everything I have not
+    /// yet sent still here? Ack-pruning deletes <c>seq &lt;= last_acked_seq</c> and the peer can only
+    /// ack what it received, so in a healthy session the horizon never climbs past
+    /// <c>sentThruSeq + 1</c>; a higher one means a cap prune ate unsent deltas.
+    /// </remarks>
+    public async Task<long> GetMinSeqAsync(CancellationToken ct = default)
+    {
+        var rows = await db.QueryAsync(
+            "SELECT COALESCE(MIN(seq), 0) FROM __localdb_oplog", static r => r.GetInt64(0), null, ct);
+        return rows[0];
+    }
+
     /// <summary>The highest oplog seq present, or 0 when the oplog is empty.</summary>
     public async Task<long> GetMaxSeqAsync(CancellationToken ct = default)
     {
@@ -133,9 +189,9 @@ internal sealed class OplogStore(ILocalDb db, ReplicationOptions options, Func<D
         // Condition reads and prunes share the transaction so the decision cannot go stale (TOCTOU).
         await using var tx = await db.BeginTransactionAsync(ct);
 
-        var depth = (await tx.QueryAsync(
-            "SELECT COUNT(*) FROM __localdb_oplog WHERE seq > (SELECT last_acked_seq FROM __localdb_peer_state WHERE id = 1)",
-            static r => r.GetInt64(0), null, ct))[0];
+        // Same O(1) expression as GetOplogDepthAsync — the caps check is the per-minute reader that
+        // made the old COUNT(*) expensive in the first place.
+        var depth = (await tx.QueryAsync(DepthSql, static r => r.GetInt64(0), null, ct))[0];
         var overRows = depth > options.MaxOplogRows;
 
         var oldest = await tx.QueryAsync(
@@ -196,10 +252,61 @@ internal sealed class OplogStore(ILocalDb db, ReplicationOptions options, Func<D
         db.ExecuteAsync(
             "UPDATE __localdb_peer_state SET last_applied_remote_seq = 0, last_seen_hlc = 0 WHERE id = 1", null, ct);
 
-    /// <summary>Sets or clears the snapshot-resync-required flag.</summary>
+    /// <summary>
+    /// Sets or clears the DATABASE-WIDE snapshot-resync flag — the one that means "every registered
+    /// table". Set by the oplog-horizon paths (cap prune, impossible peer ack), which invalidate
+    /// delta continuity for all tables at once. Per-table flags live in
+    /// <c>__localdb_snapshot_state</c>; see <see cref="GetTablesNeedingSnapshotAsync"/>.
+    /// </summary>
     public Task SetNeedsSnapshotAsync(bool value, CancellationToken ct = default) =>
         db.ExecuteAsync("UPDATE __localdb_peer_state SET needs_snapshot = @v WHERE id = 1", new { v = value ? 1 : 0 }, ct);
 
+    /// <summary>
+    /// The tables individually flagged as owing the peer a snapshot (schema v3), in ordinal name
+    /// order. Independent of the database-wide flag: baselining one table flags only that table.
+    /// </summary>
+    public Task<IReadOnlyList<string>> GetTablesNeedingSnapshotAsync(CancellationToken ct = default) =>
+        db.QueryAsync(
+            "SELECT table_name FROM __localdb_snapshot_state WHERE needs_snapshot <> 0 ORDER BY table_name",
+            static r => r.GetString(0), null, ct);
+
+    /// <summary>
+    /// Clears the snapshot debt a completed snapshot discharged: the named tables when
+    /// <paramref name="tables"/> is non-empty, otherwise <b>everything</b> — the database-wide flag
+    /// and every per-table row, because a full snapshot subsumes each of them.
+    /// </summary>
+    /// <remarks>
+    /// <para>
+    /// Rows are deleted rather than zeroed: absence is the resting state, and keeping a row per
+    /// table that has ever been baselined would accumulate bookkeeping that means nothing. Runs in
+    /// one transaction so a full snapshot's two clears cannot half-apply and leave the pair
+    /// re-snapshotting on every reconnect.
+    /// </para>
+    /// <para>
+    /// A scoped clear deliberately does NOT touch the database-wide flag. If the cap enforcer
+    /// flagged a whole-database resync while this snapshot was in flight, that debt is real and
+    /// unrelated — the snapshot only carried some of the tables — so it must survive. The full
+    /// clear may take it down because a full snapshot carries every row state as of its as-of seq,
+    /// which is exactly what the pruned deltas would have delivered.
+    /// </para>
+    /// </remarks>
+    public async Task ClearSnapshotDebtAsync(IReadOnlyList<string> tables, CancellationToken ct = default)
+    {
+        await using var tx = await db.BeginTransactionAsync(ct);
+        if (tables.Count == 0)
+        {
+            await tx.ExecuteAsync("UPDATE __localdb_peer_state SET needs_snapshot = 0 WHERE id = 1", null, ct);
+            await tx.ExecuteAsync("DELETE FROM __localdb_snapshot_state", null, ct);
+        }
+        else
+        {
+            foreach (var table in tables)
+                await tx.ExecuteAsync(
+                    "DELETE FROM __localdb_snapshot_state WHERE table_name = @t", new { t = table }, ct);
+        }
+        await tx.CommitAsync(ct);
+    }
+
     /// <summary>
     /// Records the highest remote seq applied locally plus the peer HLC observed with it; also stamps
     /// <c>last_sync_utc</c>. Both watermarks are MAX-guarded (monotonic): a snapshot completing with a

ZB.MOM.WW.LocalDb/src/ZB.MOM.WW.LocalDb.Replication/Internal/SnapshotStreamer.cs

@@ -19,21 +19,66 @@ internal interface ISnapshotApplier
 }
 
 /// <summary>
-/// Streams a full snapshot of every registered table to a peer that has fallen behind the pruned
-/// oplog horizon. Reads the authoritative per-row version from <c>__localdb_row_version</c> joined
+/// Which tables a snapshot covers. An EMPTY <see cref="Tables"/> is the wire's own encoding of
+/// "all registered tables" (see <c>SnapshotBegin.tables</c>) and is what every pre-0.3.0 peer both
+/// sends and expects, so it is deliberately the same value here rather than a separate flag.
+/// </summary>
+/// <remarks>
+/// The distinction is not cosmetic: a FULL snapshot carries every row state as of its
+/// <c>as_of_seq</c>, so the receiver may jump its watermark there and the sender may skip the
+/// deltas it covered. A PARTIAL one carries only the listed tables, so neither side may move a
+/// watermark that also speaks for the tables it left out.
+/// </remarks>
+internal sealed record SnapshotScope(IReadOnlyList<string> Tables)
+{
+    /// <summary>Every registered table — the only scope a pre-0.3.0 peer can be sent.</summary>
+    public static readonly SnapshotScope Full = new([]);
+
+    public bool IsPartial => Tables.Count > 0;
+
+    public bool Includes(string tableName) =>
+        Tables.Count == 0 || Tables.Contains(tableName, StringComparer.Ordinal);
+}
+
+/// <summary>
+/// Streams a snapshot of the tables in the requested <see cref="SnapshotScope"/> — every registered
+/// table for a peer that has fallen behind the pruned oplog horizon, or just the baselined ones for
+/// a scoped resync. Reads the authoritative per-row version from <c>__localdb_row_version</c> joined
 /// to each live table, pages it into <see cref="SnapshotBatch"/> messages through the gated data
 /// lane, and brackets them with SnapshotBegin/SnapshotComplete carrying the max oplog seq as-of.
 /// </summary>
+/// <remarks>
+/// Clearing the snapshot debt is deliberately NOT done here: whether the flags may be cleared at all
+/// depends on the peer's capability (a 0.3.0 peer acknowledges the snapshot, a pre-0.3.0 one cannot),
+/// which is session state. <see cref="SyncSession"/> owns that decision; this type only streams.
+/// </remarks>
 internal sealed class SnapshotStreamer(SqliteLocalDb db, OplogStore store, ReplicationOptions options, ILogger logger)
 {
-    /// <summary>Streams the full snapshot; returns the as-of oplog seq it covers, so the session can skip already-covered tail deltas.</summary>
-    public async Task<long> SendAsync(Func<SyncMessage, CancellationToken, Task> send, CancellationToken ct)
+    /// <summary>
+    /// Streams the snapshot for <paramref name="scope"/>; returns the as-of oplog seq it covers, so
+    /// the session can skip already-covered tail deltas (a FULL snapshot only — a partial one speaks
+    /// for nothing outside its tables).
+    /// </summary>
+    public async Task<long> SendAsync(SnapshotScope scope, Func<SyncMessage, CancellationToken, Task> send, CancellationToken ct)
     {
-        var asOfSeq = await store.GetMaxSeqAsync(ct);
-        await send(new SyncMessage { SnapshotBegin = new SnapshotBegin { AsOfSeq = asOfSeq } }, ct);
+        // The as-of is the highest seq this snapshot can stand in for. MAX(seq) alone is wrong once
+        // ack-pruning has emptied the oplog: it reads 0, and a receiver whose watermark genuinely IS
+        // 0 (a rebuilt node) is then never lifted by the snapshot — its MAX-guarded watermark stays
+        // 0, the empty-oplog horizon rule (streamable from last_acked_seq + 1) sees a gap on the
+        // next handshake, and the whole database is re-streamed on every reconnect until this node
+        // happens to write again. Every delta pruned below last_acked_seq is already folded into the
+        // rows this snapshot carries, so last_acked_seq is the floor the as-of must carry.
+        var asOfSeq = Math.Max(
+            await store.GetMaxSeqAsync(ct),
+            (await store.GetPeerStateAsync(ct)).LastAckedSeq);
+        var begin = new SnapshotBegin { AsOfSeq = asOfSeq };
+        begin.Tables.AddRange(scope.Tables);
+        await send(new SyncMessage { SnapshotBegin = begin }, ct);
 
         var totalRows = 0L;
-        foreach (var table in db.ReplicatedTables.Values.OrderBy(t => t.Name, StringComparer.Ordinal))
+        foreach (var table in db.ReplicatedTables.Values
+                     .Where(t => scope.Includes(t.Name))
+                     .OrderBy(t => t.Name, StringComparer.Ordinal))
         {
             var sql = BuildPageSql(table);
             // Keyset cursor (pk_json > last seen), not LIMIT/OFFSET: concurrent inserts/prunes
@@ -57,7 +102,13 @@ internal sealed class SnapshotStreamer(SqliteLocalDb db, OplogStore store, Repli
                     break;
                 lastPk = rows[^1].PkJson;
 
+                // One DB page may span several messages: the page is bounded by ROW COUNT, but a
+                // message is bounded by BYTES (ReplicationOptions.MaxBatchBytes) — the same budget
+                // the delta pump honours, and for the same reason. Wide rows are exactly why a
+                // snapshot is the easiest way to exceed the transport's message limit: it ships
+                // whole rows, all of them, back to back.
                 var batch = new SnapshotBatch { TableName = table.Name };
+                var bytes = 0L;
                 foreach (var row in rows)
                 {
                     if (!row.IsTombstone && row.RowJson is null)
@@ -79,11 +130,25 @@ internal sealed class SnapshotStreamer(SqliteLocalDb db, OplogStore store, Repli
                     };
                     if (row.RowJson is not null)
                         snapshotRow.RowJson = row.RowJson;
+
+                    var size = RowSize(snapshotRow);
+                    // Flush before adding, never after: a message keeps at least one row however
+                    // large, so an oversized row goes out alone rather than stalling the snapshot.
+                    if (batch.Rows.Count > 0 && bytes + size > options.MaxBatchBytes)
+                    {
+                        await send(new SyncMessage { SnapshotBatch = batch }, ct);
+                        totalRows += batch.Rows.Count;
+                        batch = new SnapshotBatch { TableName = table.Name };
+                        bytes = 0;
+                    }
                     batch.Rows.Add(snapshotRow);
+                    bytes += size;
                 }
                 if (batch.Rows.Count > 0)
+                {
                     await send(new SyncMessage { SnapshotBatch = batch }, ct);
-                totalRows += batch.Rows.Count;
+                    totalRows += batch.Rows.Count;
+                }
 
                 if (rows.Count < options.MaxBatchSize)
                     break;
@@ -91,24 +156,29 @@ internal sealed class SnapshotStreamer(SqliteLocalDb db, OplogStore store, Repli
         }
 
         await send(new SyncMessage { SnapshotComplete = new SnapshotComplete { AsOfSeq = asOfSeq } }, ct);
-        // The sender owed this snapshot (its own needs_snapshot); clearing it here — after the whole
-        // snapshot is on the wire — stops it re-streaming on the next reconnect. Narrow window: the
-        // messages are enqueued on the data lane, not yet transport-delivered, so a crash/drop after
-        // this clear could lose the snapshot with the flag already down. The reconnect handshake's
-        // gap check (peer watermark below the pruned horizon) re-detects that in every case where the
-        // oplog is non-empty; the sole undetectable corner is an EMPTY oplog (nothing to gap-check),
-        // where the peer stays behind until new local writes recreate a horizon. Accepted: closing it
-        // would need transport-level delivery acks for snapshot frames.
-        await store.SetNeedsSnapshotAsync(false, ct);
-        logger.LogInformation("Snapshot sent (as_of_seq {AsOfSeq}, {Rows} rows).", asOfSeq, totalRows);
+        logger.LogInformation(
+            "Snapshot sent (as_of_seq {AsOfSeq}, {Rows} rows, scope {Scope}).",
+            asOfSeq, totalRows, scope.IsPartial ? string.Join(",", scope.Tables) : "all");
         return asOfSeq;
     }
 
+    /// <summary>
+    /// Serialized bytes one row contributes to its batch message: the row's own encoding plus the
+    /// repeated field's tag and length prefix, as protobuf itself computes them.
+    /// </summary>
+    private static long RowSize(SnapshotRow row) =>
+        1 + Google.Protobuf.CodedOutputStream.ComputeMessageSize(row);
+
     // Pages one table's rows by joining the row-version ledger to the live table on the PK columns
     // extracted from pk_json. Rows present in the live table but absent from __localdb_row_version
-    // (data written BEFORE the table was registered for replication) are intentionally NOT snapshotted:
-    // they were never captured, consistent with CDC semantics — a full re-registration is required to
-    // baseline them.
+    // (data written BEFORE the table was registered for replication) are NOT snapshotted: they were
+    // never captured, consistent with CDC semantics.
+    //
+    // That is a real hole for a node that starts replicating LATE — its whole pre-existing dataset is
+    // invisible to the peer, silently and permanently — and the supported way to close it is to
+    // register with `ILocalDb.RegisterReplicated(table, baselineExistingRows: true)`, which seeds the
+    // ledger at an LWW-safe floor and flags the resync that carries the rows here. Nothing changes on
+    // this side: baselined rows are ordinary ledger rows and page like any other.
     private static string BuildPageSql(ReplicatedTable table)
     {
         var name = Ident(table.Name);
@@ -149,6 +219,7 @@ internal sealed class SnapshotApplier : ISnapshotApplier
     private long _asOfSeq;
     private long _rows;
     private long _maxHlc;
+    private bool _partial;
 
     // db is part of the collaborator's canonical shape (mirrors the streamer) even though the merge
     // reaches the database through the applier and store; kept for construction symmetry in the factory.
@@ -165,7 +236,13 @@ internal sealed class SnapshotApplier : ISnapshotApplier
         _asOfSeq = begin.AsOfSeq;
         _rows = 0;
         _maxHlc = 0;
-        _logger.LogInformation("Snapshot receive begun (as_of_seq {AsOfSeq}).", _asOfSeq);
+        // A non-empty table list scopes the snapshot; empty (which is all a pre-0.3.0 sender can
+        // emit) means every registered table. Read here rather than on complete because
+        // SnapshotComplete does not repeat it — the receive loop guarantees a Begin precedes both.
+        _partial = begin.Tables.Count > 0;
+        _logger.LogInformation(
+            "Snapshot receive begun (as_of_seq {AsOfSeq}, scope {Scope}).",
+            _asOfSeq, _partial ? string.Join(",", begin.Tables) : "all");
         return Task.CompletedTask;
     }
 
@@ -187,9 +264,20 @@ internal sealed class SnapshotApplier : ISnapshotApplier
 
     public async Task OnCompleteAsync(SnapshotComplete complete, CancellationToken ct)
     {
+        // A PARTIAL snapshot may not move the inbound seq watermark. That watermark is one number
+        // for the whole peer oplog, and jumping it to as_of_seq would declare every table applied
+        // through that point — including the ones this snapshot deliberately left out, whose deltas
+        // below as_of_seq the sender would then never resend. Only the HLC is carried across (seq 0
+        // is a no-op against the MAX guard), so the clock still cannot regress below what the peer
+        // has stamped. The sender makes the mirror-image concession: it does not advance its own
+        // sent-thru watermark past a partial snapshot, so those deltas are still pumped normally
+        // and merely re-deliver rows the snapshot already carried, which LWW absorbs.
+        var seq = _partial ? 0 : complete.AsOfSeq;
         // SetLastAppliedRemoteSeqAsync is MAX-guarded, so a delta applied mid-snapshot with a seq
         // above the snapshot's as-of (or an HLC above what the snapshot carried) is never regressed.
-        await _store.SetLastAppliedRemoteSeqAsync(complete.AsOfSeq, _maxHlc, ct);
-        _logger.LogInformation("Snapshot receive complete (as_of_seq {AsOfSeq}, {Rows} rows).", complete.AsOfSeq, _rows);
+        await _store.SetLastAppliedRemoteSeqAsync(seq, _maxHlc, ct);
+        _logger.LogInformation(
+            "Snapshot receive complete (as_of_seq {AsOfSeq}, {Rows} rows, partial {Partial}).",
+            complete.AsOfSeq, _rows, _partial);
     }
 }

ZB.MOM.WW.LocalDb/src/ZB.MOM.WW.LocalDb.Replication/Internal/SyncSession.cs

@@ -10,11 +10,13 @@ namespace ZB.MOM.WW.LocalDb.Replication.Internal;
 /// <summary>
 /// The symmetric 2-node sync protocol as one state machine: both the gRPC service and the gRPC
 /// client run this identical logic over a <see cref="SyncDuplex"/>. It performs the fail-closed
-/// handshake, then runs a bidirectional steady state — an outbound change pump and an inbound
-/// apply/ack loop — until the token is cancelled or the stream ends. All outbound traffic flows
-/// through a single writer task fed by two lanes: an unbounded CONTROL lane (handshake messages,
-/// acks) and a bounded DATA lane (delta/snapshot batches). Receiving therefore never blocks on the
-/// ability to write — acks always fit — while the bounded data lane is the pump's backpressure.
+/// handshake, then runs a bidirectional steady state until the token is cancelled or the stream
+/// ends: an outbound phase (the full snapshot this node owes, if any, then the change pump) and an
+/// inbound apply/ack loop, STARTED TOGETHER so the inbox keeps draining even while a snapshot is
+/// being sent. All outbound traffic flows through a single writer task fed by two lanes: an
+/// unbounded CONTROL lane (handshake messages, acks) and a bounded DATA lane (delta/snapshot
+/// batches). Receiving therefore never blocks on the ability to write — acks always fit — while the
+/// bounded data lane is the pump's backpressure.
 /// Local writes never block on the peer; the first fault tears down all loops and propagates.
 /// </summary>
 internal sealed class SyncSession
@@ -25,6 +27,10 @@ internal sealed class SyncSession
     // inbound batch rate (the peer's flow control), so its lane can safely be unbounded.
     private const int DataLaneCapacity = 16;
 
+    // How long a faulting session's teardown lets the writer keep flushing already-enqueued
+    // messages before the backstop cancel; only ever reached when the transport itself stalls.
+    private static readonly TimeSpan FaultFlushGrace = TimeSpan.FromSeconds(2);
+
     private readonly SqliteLocalDb _db;
     private readonly OplogStore _store;
     private readonly LwwApplier _applier;
@@ -45,6 +51,17 @@ internal sealed class SyncSession
     // single receive-loop task, so no synchronization is needed.
     private bool _snapshotReceiving;
 
+    // Whether the peer speaks the 0.3.0 snapshot extensions: scoped snapshots
+    // (SnapshotBegin.tables) and the SnapshotAck that answers a completed one. Written once during
+    // the handshake — which completes before either loop starts — then read-only.
+    private bool _peerSupportsPartialSnapshot;
+
+    // The scope of the snapshot this node is streaming, held until the peer's SnapshotAck discharges
+    // it; null on every session that owes none. Assigned on the pump task strictly BEFORE the
+    // snapshot's first message is enqueued and read on the receive loop when the ack lands, so the
+    // ordering is causal — volatile only to make the cross-thread publication explicit.
+    private SnapshotScope? _sentSnapshotScope;
+
     public SyncSession(
         SqliteLocalDb db, OplogStore store, LwwApplier applier, ReplicationOptions options, ILogger logger,
         Func<DateTimeOffset>? utcNow = null, LocalDbMetrics? metrics = null, SyncStatus? status = null)
@@ -61,11 +78,12 @@ internal sealed class SyncSession
 
     /// <summary>
     /// Invoked when THIS node owes the peer a snapshot (its own snapshot-required computation, not
-    /// the peer's) before deltas resume. Receives a gated send delegate that enqueues on the bounded
-    /// data lane (never the raw transport); returns the as-of oplog seq the snapshot covers so the
-    /// pump can skip already-covered tail deltas. Wired to <see cref="SnapshotStreamer.SendAsync"/>.
+    /// the peer's) before deltas resume. Receives the negotiated <see cref="SnapshotScope"/> and a
+    /// gated send delegate that enqueues on the bounded data lane (never the raw transport); returns
+    /// the as-of oplog seq the snapshot covers so the pump can skip already-covered tail deltas.
+    /// Wired to <see cref="SnapshotStreamer.SendAsync"/>.
     /// </summary>
-    public Func<Func<SyncMessage, CancellationToken, Task>, CancellationToken, Task<long>>? SnapshotSender { get; set; }
+    public Func<SnapshotScope, Func<SyncMessage, CancellationToken, Task>, CancellationToken, Task<long>>? SnapshotSender { get; set; }
 
     /// <summary>
     /// Consumes an inbound snapshot inline on the receive loop: SnapshotBegin opens the receive,
@@ -97,10 +115,19 @@ internal sealed class SyncSession
         var writer = WriterLoopAsync(duplex, control.Reader, data.Reader, linked.Token);
         try
         {
-            await HandshakeAsync(duplex, control.Writer, GatedDataSend, linked.Token);
+            var snapshotScope = await HandshakeAsync(duplex, control.Writer, linked.Token);
 
-            var pump = PumpLoopAsync(data.Writer, linked.Token);
+            // The receive loop starts BEFORE any snapshot this node owes is streamed. That ordering
+            // is load-bearing, not incidental: a snapshot sent from inside the handshake left the
+            // inbox undrained for its whole duration, so a MUTUAL snapshot (both sides flagged —
+            // e.g. both baselined an existing dataset) required an unbounded inbox to avoid
+            // deadlock, and therefore buffered a whole peer dataset in memory. With the receive loop
+            // running throughout, the peer's snapshot is applied as it arrives, the inbox can be
+            // bounded (see GrpcSyncDuplex.InboxCapacity), and mutual snapshots exchange under
+            // ordinary flow control. Interleaving is already safe by design: snapshot rows and
+            // deltas both merge through LWW, and the snapshot's watermark write is MAX-guarded.
             var receive = ReceiveLoopAsync(duplex, control.Writer, linked.Token);
+            var pump = SnapshotThenPumpLoopAsync(snapshotScope, GatedDataSend, data.Writer, linked.Token);
             var loops = new[] { pump, receive, writer };
 
             // The first COMPLETED task is the primary outcome. Cancel the siblings, observe every
@@ -117,15 +144,33 @@ internal sealed class SyncSession
         {
             // Handshake-phase faults (and the primary rethrow above) land here: the writer may
             // still be running, so stop and observe it before the primary exception surfaces.
-            linked.Cancel();
+            // It must be allowed to DRAIN, not be cancelled outright: the Handshake enqueued at
+            // the top of HandshakeAsync can still be sitting in the control lane (the enqueue →
+            // receive → validate → fault path has no yields, while the writer's wakeup waits in
+            // the thread-pool queue), and that message is the only thing that lets the peer
+            // compute its own fail-closed verdict — mismatch detection is symmetric, so this
+            // node's locally-thrown verdict reaches the peer only as the Handshake it transmits.
+            // Cancelling first killed it almost every time the peer's handshake was already
+            // buffered, leaving the healthy peer to misread a schema mismatch as a generic
+            // disconnect and reconnect-loop at the fast backoff. Completing the lanes lets the
+            // writer send what is already enqueued and exit on its own; the deferred cancel is
+            // the backstop that keeps a wedged transport from holding teardown hostage.
+            control.Writer.TryComplete();
+            data.Writer.TryComplete();
+            linked.CancelAfter(FaultFlushGrace);
             await DrainSecondaryAsync(writer);
             throw;
         }
     }
 
-    private async Task HandshakeAsync(
-        SyncDuplex duplex, ChannelWriter<SyncMessage> controlLane,
-        Func<SyncMessage, CancellationToken, Task> dataSend, CancellationToken ct)
+    /// <summary>
+    /// Runs the two-phase fail-closed handshake and returns the snapshot THIS node owes the peer —
+    /// null for none, otherwise the <see cref="SnapshotScope"/> to stream (its own computation from
+    /// its own state; the peer's answer drives the peer's send direction, symmetrically). Sending
+    /// that snapshot is deliberately NOT part of the handshake; see <see cref="RunAsync"/>.
+    /// </summary>
+    private async Task<SnapshotScope?> HandshakeAsync(
+        SyncDuplex duplex, ChannelWriter<SyncMessage> controlLane, CancellationToken ct)
     {
         var peerState = await _store.GetPeerStateAsync(ct);
         var localHandshake = BuildHandshake(peerState);
@@ -142,6 +187,11 @@ internal sealed class SyncSession
 
         ValidateHandshake(localHandshake, peerHandshake);
 
+        // Read before anything can act on it. A pre-0.3.0 peer leaves the field at its proto3
+        // default, so "false" covers both "explicitly unsupported" and "older build" without a
+        // version check — which is the point of advertising a capability rather than a version.
+        _peerSupportsPartialSnapshot = peerHandshake.SupportsPartialSnapshot;
+
         // A different node id on the far end means the peer was replaced, not merely restarted — a
         // rebuilt host, a restored-from-nothing volume. Its oplog numbers from 1 again, so the
         // watermark we hold (in the OLD peer's seq space) would silently swallow its first N
@@ -175,34 +225,78 @@ internal sealed class SyncSession
         _sentThruSeq = peerAppliedThruSeq;
         Interlocked.Exchange(ref _peerAckedSeq, peerAppliedThruSeq);
 
-        var snapshotRequired = await ComputeSnapshotRequiredAsync(peerHandshake, peerState, ct);
+        var snapshotScope = await ComputeSnapshotScopeAsync(peerHandshake, peerState, ct);
         await controlLane.WriteAsync(
-            new SyncMessage { HandshakeAck = new HandshakeAck { NodeId = _db.NodeId, SnapshotRequired = snapshotRequired } }, ct);
+            new SyncMessage { HandshakeAck = new HandshakeAck { NodeId = _db.NodeId, SnapshotRequired = snapshotScope is not null } }, ct);
 
         var peerAckMsg = await ReceiveOneAsync(duplex, ct);
         if (peerAckMsg.MsgCase != SyncMessage.MsgOneofCase.HandshakeAck)
             throw new InvalidOperationException(
                 $"Replication protocol error: expected a HandshakeAck, got {peerAckMsg.MsgCase}.");
 
-        // We stream the snapshot when WE owe the peer one (computed from our pruned horizon /
-        // needs_snapshot), then it lands on the peer's receive loop. The peer's own flag drives ITS
-        // send direction symmetrically; both are safe to run because LWW merges either way.
-        // Memory profile of a MUTUAL snapshot: while we send here (receive loop not yet running),
-        // the peer's inbound frames — potentially its entire snapshot — buffer in the unbounded
-        // duplex inbox until our send completes. Worst case one full peer snapshot held in memory;
-        // acceptable for the small LocalDb datasets this library targets.
-        if (snapshotRequired)
+        return snapshotScope;
+    }
+
+    /// <summary>
+    /// The outbound steady state: stream the full snapshot this node owes the peer (if any), then
+    /// run the incremental delta pump. One task, because the ordering between them is a protocol
+    /// requirement — the pump must not put deltas on the data lane ahead of the snapshot, and it
+    /// starts from the as-of seq the snapshot covered.
+    /// </summary>
+    /// <remarks>
+    /// Split out of the handshake so the receive loop is already draining the inbox while this
+    /// streams (see <see cref="RunAsync"/>). Nothing else about the snapshot changes: it goes
+    /// through the same gated data lane, so it is still bounded by the writer's flow control rather
+    /// than racing the transport.
+    /// </remarks>
+    private async Task SnapshotThenPumpLoopAsync(
+        SnapshotScope? snapshotScope, Func<SyncMessage, CancellationToken, Task> dataSend,
+        ChannelWriter<SyncMessage> dataLane, CancellationToken ct)
+    {
+        if (snapshotScope is not null)
         {
             if (SnapshotSender is null)
                 throw new NotSupportedException("snapshot required but no snapshot sender configured");
-            var snapshotAsOfSeq = await SnapshotSender(dataSend, ct);
+            // Published BEFORE the first frame is enqueued, so the peer's SnapshotAck can never
+            // arrive on the receive loop while this is still null.
+            Volatile.Write(ref _sentSnapshotScope, snapshotScope);
+            var snapshotAsOfSeq = await SnapshotSender(snapshotScope, dataSend, ct);
             _metrics?.RecordSnapshotSent();
             _status?.SetLastSyncUtc(_utcNow());
-            // The snapshot already carries every row state up to as-of; skipping the pump past it
-            // avoids re-sending tail deltas the peer would only discard via LWW.
-            if (snapshotAsOfSeq > _sentThruSeq)
+            // A FULL snapshot carries every row state up to as-of, so skipping the pump past it
+            // avoids re-sending tail deltas the peer would only discard via LWW. A PARTIAL one
+            // speaks only for its own tables, so the pump must still deliver every delta below
+            // as-of — the peer has not moved its watermark either (see SnapshotApplier.OnComplete).
+            if (!snapshotScope.IsPartial && snapshotAsOfSeq > _sentThruSeq)
                 _sentThruSeq = snapshotAsOfSeq;
+
+            // A peer that cannot acknowledge the snapshot leaves only the 0.2.x option: clear once
+            // the frames are on the data lane. The window that opens (enqueued, not yet delivered —
+            // a crash there loses the snapshot with the debt already discharged) is why 0.3.0 peers
+            // wait for the ack instead. The reconnect handshake's gap check re-detects it whenever
+            // the oplog is non-empty; an EMPTY oplog remains the undetectable corner.
+            if (!_peerSupportsPartialSnapshot)
+                await ClearSentSnapshotDebtAsync(ct);
         }
+
+        await PumpLoopAsync(dataLane, ct);
+    }
+
+    /// <summary>
+    /// Discharges the snapshot debt this session's outbound snapshot covered, exactly once.
+    /// </summary>
+    /// <remarks>
+    /// Taking the scope out of the field before clearing makes this idempotent, which matters
+    /// because two paths reach it — the pre-0.3.0 clear-on-enqueue above and the peer's SnapshotAck
+    /// — and a peer that acknowledges anyway (a newer build than its handshake claimed) must not
+    /// clear a second time on top of flags a later baseline may already have re-raised.
+    /// </remarks>
+    private async Task ClearSentSnapshotDebtAsync(CancellationToken ct)
+    {
+        var scope = Interlocked.Exchange(ref _sentSnapshotScope, null);
+        if (scope is null)
+            return;
+        await _store.ClearSnapshotDebtAsync(scope.Tables, ct);
     }
 
     // The ONLY caller of duplex.Send. Control (handshake + acks) always preempts data so the peer's
@@ -246,20 +340,54 @@ internal sealed class SyncSession
         }
     }
 
+    /// <summary>
+    /// Reads oplog deltas above the sent watermark and ships them, splitting each read into as many
+    /// messages as <see cref="ReplicationOptions.MaxBatchBytes"/> requires.
+    /// </summary>
+    /// <remarks>
+    /// Row count alone cannot bound a message: rows are arbitrarily wide, so any row-count cap is
+    /// really a guess about the widest row, and guessing wrong produces a message the transport
+    /// rejects — replication then stops converging with a size error, not a data error. Budgeting by
+    /// summed serialized bytes bounds the thing that is actually limited, and lets the row cap go
+    /// back to meaning what it says.
+    /// </remarks>
     private async Task PumpLoopAsync(ChannelWriter<SyncMessage> dataLane, CancellationToken ct)
     {
         while (true)
         {
             ct.ThrowIfCancellationRequested();
+            // BEFORE the read, because the read itself cannot tell a pruned gap from an idle oplog.
+            ThrowIfPruneGap(await _store.GetMinSeqAsync(ct));
             var batch = await _store.ReadBatchAboveAsync(_sentThruSeq, _options.MaxBatchSize, ct);
             if (batch.Count > 0)
             {
-                var delta = new DeltaBatch();
-                foreach (var entry in batch)
-                    delta.Entries.Add(ToProto(entry));
-                // Blocking here on a full data lane IS the pump's backpressure.
-                await dataLane.WriteAsync(new SyncMessage { DeltaBatch = delta }, ct);
-                _sentThruSeq = batch[^1].Seq;
+                // The exact guard on the rows about to go out, for a prune that landed between the
+                // probe and the read. Free (the seq is already in hand) and strictly tighter.
+                ThrowIfPruneGap(batch[0].Seq);
+                var index = 0;
+                while (index < batch.Count)
+                {
+                    var delta = new DeltaBatch();
+                    var bytes = 0L;
+                    while (index < batch.Count)
+                    {
+                        var proto = ToProto(batch[index]);
+                        var size = EntrySize(proto);
+                        // The first entry of a message is taken however large it is: refusing an
+                        // oversized row would stall the stream on it permanently, whereas sending it
+                        // alone either succeeds or fails loudly at the transport with its real size.
+                        if (delta.Entries.Count > 0 && bytes + size > _options.MaxBatchBytes)
+                            break;
+                        delta.Entries.Add(proto);
+                        bytes += size;
+                        index++;
+                    }
+                    // Blocking here on a full data lane IS the pump's backpressure.
+                    await dataLane.WriteAsync(new SyncMessage { DeltaBatch = delta }, ct);
+                    // Advanced per MESSAGE, not per read: a fault mid-split must not re-send what
+                    // already went out, and must not skip what did not.
+                    _sentThruSeq = batch[index - 1].Seq;
+                }
                 continue;
             }
 
@@ -267,23 +395,77 @@ internal sealed class SyncSession
         }
     }
 
+    /// <summary>
+    /// Faults the session when the oplog horizon has climbed above this session's read position —
+    /// i.e. a cap prune deleted deltas between <see cref="_sentThruSeq"/> and
+    /// <paramref name="oldestAvailableSeq"/>. An empty oplog (0) is not a gap.
+    /// </summary>
+    /// <remarks>
+    /// <para>
+    /// <b>Why a live session has to die for this.</b> <c>EnforceCapsAsync</c> sets
+    /// <c>needs_snapshot</c> in the same transaction as the prune, so a RECONNECT heals — the
+    /// handshake serves a full snapshot instead of deltas. Nothing, however, made the live session
+    /// notice: the pump reads <c>seq &gt; _sentThruSeq</c>, which silently resumes above the hole, and
+    /// the peer acks those later seqs as if the pair were converged. On a stable LAN with a slow but
+    /// alive peer that session can run for days, and the pair is durably divergent for all of it —
+    /// the flag that would fix it is only ever read at handshake time. Tearing down is what turns a
+    /// persisted "needs snapshot" into an actual snapshot.
+    /// </para>
+    /// <para>
+    /// <b>Why the horizon, not the flag.</b> <c>needs_snapshot</c> is also set for reasons that do
+    /// not invalidate THIS session's position (and a full snapshot this session just streamed clears
+    /// it only when the peer's <c>SnapshotAck</c> lands, which is after the pump has started).
+    /// Comparing the horizon against the sent watermark asks the precise question instead, so a
+    /// session is faulted when — and only when — rows it still owes the peer are gone.
+    /// </para>
+    /// </remarks>
+    private void ThrowIfPruneGap(long oldestAvailableSeq)
+    {
+        if (oldestAvailableSeq == 0 || oldestAvailableSeq <= _sentThruSeq + 1)
+            return;
+        throw new LocalDbOplogGapException(
+            $"Oplog deltas above the sent watermark {_sentThruSeq} were pruned (oldest surviving seq " +
+            $"{oldestAvailableSeq}): this session can no longer stream incrementally without skipping " +
+            "them. Ending the session so the reconnect handshake serves a snapshot resync.");
+    }
+
+    /// <summary>
+    /// Serialized bytes one entry contributes to its batch message: the entry's own encoding plus
+    /// the repeated field's tag and length prefix, as protobuf itself computes them.
+    /// </summary>
+    private static long EntrySize(OplogEntry entry) =>
+        1 + Google.Protobuf.CodedOutputStream.ComputeMessageSize(entry);
+
     private async Task ReceiveLoopAsync(
         SyncDuplex duplex, ChannelWriter<SyncMessage> controlLane, CancellationToken ct)
     {
-        while (await duplex.Inbox.WaitToReadAsync(ct))
+        // One-message lookahead. Coalescing means reading ahead until a message turns out NOT to be
+        // of the kind being coalesced, and a Channel cannot un-read — so that message is parked here
+        // and processed first on the next pass. Its presence also keeps the outer loop from waiting
+        // on the inbox when there is already work in hand.
+        SyncMessage? lookahead = null;
+
+        while (lookahead is not null || await duplex.Inbox.WaitToReadAsync(ct))
         {
-            while (duplex.Inbox.TryRead(out var msg))
+            while (true)
             {
+                var msg = lookahead;
+                lookahead = null;
+                if (msg is null)
+                {
+                    if (!duplex.Inbox.TryRead(out var read))
+                        break;
+                    msg = read;
+                }
+
                 switch (msg.MsgCase)
                 {
                     case SyncMessage.MsgOneofCase.DeltaBatch:
                         // Deltas apply normally even mid-snapshot: LWW makes the interleave safe.
-                        await HandleDeltaBatchAsync(controlLane, msg.DeltaBatch, ct);
+                        lookahead = await HandleCoalescedDeltasAsync(duplex, controlLane, msg.DeltaBatch, ct);
                         break;
                     case SyncMessage.MsgOneofCase.DeltaAck:
-                        var acked = msg.DeltaAck.AppliedThruSeq;
-                        InterlockedMax(ref _peerAckedSeq, acked);
-                        await _store.RecordPeerAckAsync(acked, ct);
+                        lookahead = await HandleCoalescedAcksAsync(duplex, msg.DeltaAck, ct);
                         break;
                     case SyncMessage.MsgOneofCase.SnapshotBegin:
                         if (SnapshotApplier is null)
@@ -305,6 +487,20 @@ internal sealed class SyncSession
                         _snapshotReceiving = false;
                         _metrics?.RecordSnapshotReceived();
                         _status?.SetLastSyncUtc(_utcNow());
+                        // Acknowledge only to a peer that asked for it. Sent AFTER the applier's
+                        // durable watermark write, so the ack means "I hold these rows", which is
+                        // what lets the sender drop its per-table flags. On the unbounded control
+                        // lane, like DeltaAck: an ack must never queue behind a data backlog.
+                        if (_peerSupportsPartialSnapshot)
+                            await controlLane.WriteAsync(
+                                new SyncMessage { SnapshotAck = new SnapshotAck { AsOfSeq = msg.SnapshotComplete.AsOfSeq } }, ct);
+                        break;
+                    case SyncMessage.MsgOneofCase.SnapshotAck:
+                        // The peer has durably applied the snapshot THIS node streamed, so the
+                        // tables it covered no longer owe one. Nothing to reconcile against the
+                        // ack's as-of seq: only one snapshot is ever sent per session, so the scope
+                        // held in _sentSnapshotScope is unambiguously the one being acknowledged.
+                        await ClearSentSnapshotDebtAsync(ct);
                         break;
                     default:
                         throw new InvalidOperationException(
@@ -314,19 +510,98 @@ internal sealed class SyncSession
         }
     }
 
-    private async Task HandleDeltaBatchAsync(ChannelWriter<SyncMessage> controlLane, DeltaBatch batch, CancellationToken ct)
+    /// <summary>
+    /// Applies the given delta batch together with any delta batches ALREADY waiting in the inbox,
+    /// as one transaction answered by one ack; returns the non-delta message that ended the run (to
+    /// be processed next), or null if the inbox merely ran dry.
+    /// </summary>
+    /// <remarks>
+    /// <para>
+    /// Byte-budgeting the sender produces more, smaller messages, and one apply transaction plus one
+    /// ack round-trip per message is the cost that would otherwise scale with that. Coalescing pays
+    /// it once per burst instead — and a burst is exactly what a reconnect after an outage delivers.
+    /// </para>
+    /// <para>
+    /// Only messages already queued are absorbed (<c>TryRead</c>, never a wait), so this adds no
+    /// latency: a trickle of one batch at a time coalesces nothing and behaves exactly as before.
+    /// The run stops at the sender's own caps, so the extra memory held is bounded by roughly one
+    /// sender batch beyond the budget — the overshoot of finishing whichever batch crosses it.
+    /// </para>
+    /// </remarks>
+    private async Task<SyncMessage?> HandleCoalescedDeltasAsync(
+        SyncDuplex duplex, ChannelWriter<SyncMessage> controlLane, DeltaBatch first, CancellationToken ct)
     {
-        if (batch.Entries.Count == 0)
+        var entries = new List<OplogEntry>(first.Entries);
+        var bytes = 0L;
+        foreach (var entry in first.Entries)
+            bytes += EntrySize(entry);
+
+        SyncMessage? lookahead = null;
+        while (entries.Count < _options.MaxBatchSize && bytes < _options.MaxBatchBytes
+               && duplex.Inbox.TryRead(out var next))
+        {
+            if (next.MsgCase != SyncMessage.MsgOneofCase.DeltaBatch)
+            {
+                lookahead = next;
+                break;
+            }
+            foreach (var entry in next.DeltaBatch.Entries)
+            {
+                entries.Add(entry);
+                bytes += EntrySize(entry);
+            }
+        }
+
+        await ApplyDeltaEntriesAsync(controlLane, entries, ct);
+        return lookahead;
+    }
+
+    /// <summary>
+    /// Records the given ack together with any acks ALREADY waiting in the inbox, as a single
+    /// watermark write; returns the non-ack message that ended the run, or null if the inbox ran dry.
+    /// </summary>
+    /// <remarks>
+    /// Exactly equivalent to recording each in turn, and strictly cheaper: <c>applied_thru_seq</c> is
+    /// monotonic and <c>RecordPeerAckAsync</c> is MAX-guarded, so every superseded ack would have
+    /// written a watermark that the final one overwrites and run a DELETE below its own horizon.
+    /// Unbounded coalescing is safe here — acks collapse into one number, accumulating nothing.
+    /// </remarks>
+    private async Task<SyncMessage?> HandleCoalescedAcksAsync(
+        SyncDuplex duplex, DeltaAck first, CancellationToken ct)
+    {
+        var acked = first.AppliedThruSeq;
+
+        SyncMessage? lookahead = null;
+        while (duplex.Inbox.TryRead(out var next))
+        {
+            if (next.MsgCase != SyncMessage.MsgOneofCase.DeltaAck)
+            {
+                lookahead = next;
+                break;
+            }
+            if (next.DeltaAck.AppliedThruSeq > acked)
+                acked = next.DeltaAck.AppliedThruSeq;
+        }
+
+        InterlockedMax(ref _peerAckedSeq, acked);
+        await _store.RecordPeerAckAsync(acked, ct);
+        return lookahead;
+    }
+
+    private async Task ApplyDeltaEntriesAsync(
+        ChannelWriter<SyncMessage> controlLane, IReadOnlyList<OplogEntry> batchEntries, CancellationToken ct)
+    {
+        if (batchEntries.Count == 0)
             return;
 
         var now = _utcNow();
         var nowMs = now.ToUnixTimeMilliseconds();
         var driftLimitMs = nowMs + (long)_options.MaxHlcDriftAhead.TotalMilliseconds;
-        var toApply = new List<OplogEntryRecord>(batch.Entries.Count);
+        var toApply = new List<OplogEntryRecord>(batchEntries.Count);
         var maxSeq = 0L;
         var driftDeadLettered = 0;
 
-        foreach (var proto in batch.Entries)
+        foreach (var proto in batchEntries)
         {
             if (proto.Seq > maxSeq) maxSeq = proto.Seq;
             var entry = FromProto(proto);
@@ -407,6 +682,9 @@ internal sealed class SyncSession
             NodeId = _db.NodeId,
             LibSchemaVersion = LibSchemaVersion,
             LastAppliedRemoteSeq = peerState.LastAppliedRemoteSeq,
+            // Unconditional: this build always understands a scoped snapshot and always answers a
+            // completed one. It is a statement about the code, not about configuration or state.
+            SupportsPartialSnapshot = true,
         };
         foreach (var table in _db.ReplicatedTables.Values.OrderBy(t => t.Name, StringComparer.Ordinal))
             handshake.Tables.Add(new TableDigest { TableName = table.Name, Digest = table.Digest });
@@ -437,11 +715,63 @@ internal sealed class SyncSession
             "Replication handshake rejected (fail-closed): " + string.Join("; ", parts) + ".");
     }
 
-    private async Task<bool> ComputeSnapshotRequiredAsync(Handshake peerHandshake, PeerState peerState, CancellationToken ct)
+    /// <summary>
+    /// Decides what snapshot this node owes the peer: null for none, <see cref="SnapshotScope.Full"/>
+    /// for every registered table, or a scoped snapshot naming just the tables that need one.
+    /// </summary>
+    /// <remarks>
+    /// <para>
+    /// The two sources of snapshot debt are genuinely different in reach, which is why they are
+    /// stored separately. An <b>oplog-horizon</b> gap — the peer's watermark below the oldest delta
+    /// still streamable, which is what a cap prune or an impossible peer ack leaves behind — means
+    /// deltas have been lost for the oplog as a whole, so nothing short of every table is sound. A
+    /// <b>baseline</b> means one table's pre-existing rows were never captured, and says nothing at
+    /// all about the others.
+    /// </para>
+    /// <para>
+    /// A scope that happens to name every registered table is returned as Full rather than as a
+    /// list: it covers exactly what a full snapshot covers, so it should also get a full snapshot's
+    /// watermark advance on both sides, and it needs no promotion for a pre-0.3.0 peer.
+    /// </para>
+    /// <para>
+    /// Flags naming tables that are not registered are ignored (they can only stream nothing).
+    /// <c>DeregisterReplicated</c> drops them with the table's ledger rows, so the surviving way to
+    /// hold one is to drop the SQL table without deregistering — which the handshake's fail-closed
+    /// digest comparison rejects for its own reasons.
+    /// </para>
+    /// </remarks>
+    private async Task<SnapshotScope?> ComputeSnapshotScopeAsync(Handshake peerHandshake, PeerState peerState, CancellationToken ct)
     {
-        if (peerState.NeedsSnapshot)
-            return true;
+        if (peerState.NeedsSnapshot || await HasHorizonGapAsync(peerHandshake, peerState, ct))
+            return SnapshotScope.Full;
 
+        var flagged = await _store.GetTablesNeedingSnapshotAsync(ct);
+        if (flagged.Count == 0)
+            return null;
+
+        var scoped = flagged.Where(_db.ReplicatedTables.ContainsKey).ToList();
+        if (scoped.Count == 0)
+            return null;
+        if (scoped.Count >= _db.ReplicatedTables.Count)
+            return SnapshotScope.Full;
+
+        // A peer that would read SnapshotBegin.tables as "all tables" and then advance its watermark
+        // past deltas this snapshot did not carry gets the full snapshot instead. Wider than asked
+        // for, but it is the only shape the older protocol can express correctly — and it is what
+        // 0.2.x did for every baseline.
+        if (!_peerSupportsPartialSnapshot)
+        {
+            _logger.LogInformation(
+                "Peer {PeerNodeId} does not support scoped snapshots; promoting the snapshot of {Tables} to a full one.",
+                PeerNodeId, string.Join(",", scoped));
+            return SnapshotScope.Full;
+        }
+
+        return new SnapshotScope(scoped);
+    }
+
+    private async Task<bool> HasHorizonGapAsync(Handshake peerHandshake, PeerState peerState, CancellationToken ct)
+    {
         var minRows = await _db.QueryAsync(
             "SELECT COALESCE(MIN(seq), 0) FROM __localdb_oplog", static r => r.GetInt64(0), null, ct);
         var minSeq = minRows[0];

ZB.MOM.WW.LocalDb/src/ZB.MOM.WW.LocalDb.Replication/Internal/SyncSessionFactory.cs

@@ -6,7 +6,7 @@ namespace ZB.MOM.WW.LocalDb.Replication.Internal;
 /// <summary>
 /// Shared construction of a <see cref="SyncSession"/> and its dependencies (OplogStore + LwwApplier)
 /// from a <see cref="SqliteLocalDb"/> and the bound options. Both the service and client adapters use
-/// this so they build an identical session; snapshot hooks are left null (Task 12 fills them in).
+/// this so they build an identical session.
 /// </summary>
 internal static class SyncSessionFactory
 {

ZB.MOM.WW.LocalDb/src/ZB.MOM.WW.LocalDb.Replication/LocalDbOplogGapException.cs

@@ -0,0 +1,23 @@
+namespace ZB.MOM.WW.LocalDb.Replication;
+
+/// <summary>
+/// Thrown by a live sync session when the oplog horizon has moved ABOVE the position that session
+/// is streaming from: a cap prune deleted deltas it had not yet sent, so every later delta it could
+/// ship would land on the far side of a hole the peer will never receive.
+/// </summary>
+/// <remarks>
+/// This is not a transport failure and not a data error — it is the session discovering that its own
+/// incremental position has become unsound. Nothing can repair it in-session (the deleted rows are
+/// gone), so the session tears itself down; the same prune set <c>needs_snapshot</c>, which makes the
+/// reconnect handshake serve a snapshot instead of deltas. Catching this and continuing would
+/// reintroduce exactly the silent divergence it exists to stop.
+/// </remarks>
+public sealed class LocalDbOplogGapException : Exception
+{
+    /// <summary>Creates the exception with a message describing the gap.</summary>
+    public LocalDbOplogGapException(string message) : base(message) { }
+
+    /// <summary>Creates the exception with a message and the underlying cause.</summary>
+    public LocalDbOplogGapException(string message, Exception innerException)
+        : base(message, innerException) { }
+}

ZB.MOM.WW.LocalDb/src/ZB.MOM.WW.LocalDb.Replication/LocalDbSyncService.cs

@@ -21,8 +21,26 @@ public sealed class LocalDbSyncService : LocalDbSync.LocalDbSyncBase
     private readonly LocalDbMetrics? _metrics;
     private readonly SyncStatus? _status;
 
-    // 0 = idle, 1 = a stream is active. Interlocked-guarded across concurrent calls (singleton service).
-    private int _active;
+    // The stream currently holding the guard, or null when idle. Guarded by _gate (singleton service).
+    private readonly object _gate = new();
+    private Registration? _active;
+
+    /// <summary>
+    /// How long a displacing call waits for the incumbent it evicted to actually finish before
+    /// giving up and refusing itself. Test seam; the default is generous because eviction only has
+    /// to outrun the operator, not the network.
+    /// </summary>
+    internal TimeSpan TakeoverTimeout { get; set; } = TimeSpan.FromSeconds(10);
+
+    /// <summary>One in-flight <c>Sync</c> handler: its cancellation handle plus a completion signal
+    /// a displacing handler can await, so two sessions never run against the database at once.</summary>
+    private sealed class Registration(CancellationTokenSource cts)
+    {
+        public CancellationTokenSource Cts { get; } = cts;
+
+        public TaskCompletionSource Finished { get; } =
+            new(TaskCreationOptions.RunContinuationsAsynchronously);
+    }
 
     /// <summary>
     /// Creates the passive sync service. <paramref name="status"/> is taken as the public
@@ -41,21 +59,47 @@ public sealed class LocalDbSyncService : LocalDbSync.LocalDbSyncBase
         _status = status as SyncStatus;
     }
 
+    /// <summary>
+    /// Accepts one inbound sync stream, evicting any stream already holding the guard.
+    /// </summary>
+    /// <remarks>
+    /// <para>
+    /// <b>Newest wins.</b> The guard exists to SERIALIZE sessions — a node syncs with exactly one
+    /// peer, and two sessions against one database would fight over the same watermarks — not to
+    /// fence out an impostor; authentication is the interceptor's job, and an authenticated caller
+    /// on this key IS the peer by definition. Refusing the newcomer therefore protected nothing and
+    /// cost everything in the case that actually happens: the dialling node crashes or is
+    /// partitioned, this handler stays parked forever inside <c>MoveNext</c> on a half-open stream
+    /// that will never produce another byte, and every reconnect from the rebooted peer is rejected
+    /// <c>AlreadyExists</c> — potentially for as long as the pair stays idle, while this side keeps
+    /// reporting itself connected. The newest dialler is the one with a live link, so it takes over.
+    /// </para>
+    /// <para>
+    /// Eviction is cancel-then-await, not cancel-and-go: the incumbent's cancellation unwinds its
+    /// session and reader pump, and the newcomer waits for that to complete before starting its own,
+    /// so the "one session at a time" property the guard was built for still holds. An incumbent
+    /// that cannot be evicted within <see cref="TakeoverTimeout"/> is the one case where the old
+    /// answer is still right — the newcomer refuses itself rather than double-running.
+    /// </para>
+    /// </remarks>
     public override async Task Sync(
         IAsyncStreamReader<SyncMessage> requestStream,
         IServerStreamWriter<SyncMessage> responseStream,
         ServerCallContext context)
     {
-        if (Interlocked.CompareExchange(ref _active, 1, 0) != 0)
-            throw new RpcException(new Status(StatusCode.AlreadyExists, "a sync stream is already active"));
+        // The linked source exists so the reader pump is deterministically torn down on EVERY exit
+        // path — including a session fault for a local (non-transport) reason, where the call token
+        // never fires and the pump would otherwise keep reading a stream whose handler has exited.
+        // It is also the handle a later call cancels to displace this one.
+        using var linkedCts = CancellationTokenSource.CreateLinkedTokenSource(context.CancellationToken);
+        var mine = new Registration(linkedCts);
 
         try
         {
-            // The linked source exists so the reader pump is deterministically torn down on EVERY
-            // exit path — including a session fault for a local (non-transport) reason, where the
-            // call token never fires and the pump would otherwise keep reading a stream whose
-            // handler has exited.
-            using var linkedCts = CancellationTokenSource.CreateLinkedTokenSource(context.CancellationToken);
+            // Inside the try so the finally below always releases the guard — including when the
+            // takeover itself is abandoned because THIS caller gave up mid-handover.
+            await TakeOverAsync(mine, context.CancellationToken);
+
             var (duplex, readerTask) = GrpcSyncDuplex.Create(requestStream, responseStream, linkedCts.Token);
             var session = SyncSessionFactory.Create(_db, _options, _loggerFactory, _metrics, _status);
             _status?.SessionStarted();
@@ -81,13 +125,66 @@ public sealed class LocalDbSyncService : LocalDbSync.LocalDbSyncBase
         {
             // Client disconnect / server shutdown: the normal end of a long-lived streaming RPC.
         }
+        catch (OperationCanceledException)
+        {
+            // Displaced by a newer stream: an orderly handover, not a failure of this node.
+            _logger.LogInformation("Sync stream ended: displaced by a newer stream from the peer.");
+        }
         catch (LocalDbSchemaMismatchException ex)
         {
             throw new RpcException(new Status(StatusCode.FailedPrecondition, ex.Message));
         }
         finally
         {
-            Interlocked.Exchange(ref _active, 0);
+            lock (_gate)
+            {
+                if (ReferenceEquals(_active, mine))
+                    _active = null;
+            }
+            mine.Finished.TrySetResult();
+        }
+    }
+
+    /// <summary>
+    /// Claims the single-stream guard for <paramref name="mine"/>, cancelling and awaiting whatever
+    /// held it. Throws <c>AlreadyExists</c> only when the incumbent refuses to end.
+    /// </summary>
+    private async Task TakeOverAsync(Registration mine, CancellationToken ct)
+    {
+        Registration? displaced;
+        lock (_gate)
+        {
+            displaced = _active;
+            _active = mine;
+        }
+        if (displaced is null)
+            return;
+
+        _logger.LogWarning(
+            "A new sync stream arrived while one was already active; displacing the incumbent (newest wins).");
+        try
+        {
+            // The incumbent may already be disposing its own source between its finally and here.
+            displaced.Cts.Cancel();
+        }
+        catch (ObjectDisposedException) { }
+
+        try
+        {
+            await displaced.Finished.Task.WaitAsync(TakeoverTimeout, ct);
+        }
+        catch (TimeoutException)
+        {
+            // Could not evict it: two live sessions over one database is worse than a refused dial,
+            // so fall back to the historical answer — and hand the guard back to the incumbent.
+            lock (_gate)
+            {
+                if (ReferenceEquals(_active, mine))
+                    _active = displaced;
+            }
+            throw new RpcException(new Status(
+                StatusCode.AlreadyExists,
+                "a sync stream is already active and did not yield within the takeover timeout"));
         }
     }
 }

ZB.MOM.WW.LocalDb/src/ZB.MOM.WW.LocalDb.Replication/MaintenanceBackgroundService.cs

@@ -54,7 +54,7 @@ public sealed class MaintenanceBackgroundService : BackgroundService
                 {
                     _snapshotFlagWarned = true;
                     _logger.LogWarning(
-                        "Oplog backlog/age caps exceeded: pruned to the ceiling and flagged needs_snapshot — the peer must snapshot-resync. Not re-logged until a clean tick.");
+                        "Oplog backlog/age caps exceeded: pruned to the ceiling and flagged needs_snapshot — the peer must snapshot-resync. A session streaming from below the new horizon ends itself on its next pump tick so the reconnect serves that snapshot. Not re-logged until a clean tick.");
                 }
                 else if (!snapshotFlagged)
                 {

ZB.MOM.WW.LocalDb/src/ZB.MOM.WW.LocalDb.Replication/ReplicationOptions.cs

@@ -14,9 +14,44 @@ public sealed class ReplicationOptions
     /// <summary>How often the change pump flushes pending oplog deltas to the peer.</summary>
     public TimeSpan FlushInterval { get; set; } = TimeSpan.FromMilliseconds(250);
 
-    /// <summary>Maximum oplog entries sent per outbound batch.</summary>
+    /// <summary>
+    /// Secondary, row-count cap on an outbound batch (and the snapshot streamer's DB page size).
+    /// <see cref="MaxBatchBytes"/> is the primary cap.
+    /// </summary>
+    /// <remarks>
+    /// Historically this was the ONLY cap, which made it a proxy for a byte budget that consumers
+    /// had to tune by hand against their widest row — ScadaBridge pinned it to 16 because a 70 KB
+    /// <c>config_json</c> times the 500 default is ~35 MB against a 4 MB gRPC limit. With
+    /// <see cref="MaxBatchBytes"/> doing that job directly, this is back to being what its name
+    /// says: a bound on how many rows one message may carry.
+    /// </remarks>
     public int MaxBatchSize { get; set; } = 500;
 
+    /// <summary>
+    /// Primary cap on an outbound batch: the summed serialized size of the entries in one
+    /// <c>DeltaBatch</c> / <c>SnapshotBatch</c> message, in bytes. Default 2 MB.
+    /// </summary>
+    /// <remarks>
+    /// <para>
+    /// Sized at half the default 4 MB gRPC receive limit, so the message framing, the oneof wrapper
+    /// and any per-entry overhead the estimate does not model all fit inside the margin.
+    /// </para>
+    /// <para>
+    /// <b>A single row larger than the budget is still sent, alone.</b> Skipping it would stall the
+    /// stream permanently on one oversized row; sending it gives the transport the chance to accept
+    /// it (gRPC limits are configurable) and, if it refuses, a loud transport error naming the size
+    /// rather than silent non-convergence.
+    /// </para>
+    /// <para>
+    /// <b>Wire compatibility:</b> none required. Batch size is a sender-side choice the protocol
+    /// already leaves free — <c>DeltaBatch</c>/<c>SnapshotBatch</c> carry a repeated field with no
+    /// declared count — so a node on this version interoperates with a peer on any earlier one in
+    /// both directions, with no proto change and no <c>lib_schema_version</c> bump. (ScadaBridge
+    /// site pairs stop and start together regardless; this simply does not depend on it.)
+    /// </para>
+    /// </remarks>
+    public int MaxBatchBytes { get; set; } = 2 * 1024 * 1024;
+
     /// <summary>Backlog ceiling: exceeding it flags a snapshot resync and prunes the oldest oplog rows.</summary>
     public long MaxOplogRows { get; set; } = 1_000_000;
 
@@ -29,6 +64,34 @@ public sealed class ReplicationOptions
     /// <summary>Upper bound on the exponential reconnect backoff.</summary>
     public TimeSpan ReconnectBackoffMax { get; set; } = TimeSpan.FromSeconds(60);
 
+    /// <summary>
+    /// How long the initiator's HTTP/2 connection may sit without a keepalive PING before one is
+    /// sent. Default 60s.
+    /// </summary>
+    /// <remarks>
+    /// <para>
+    /// A sync stream is long-lived and, on a converged pair, almost entirely silent — which is
+    /// exactly the shape a half-open TCP connection hides behind. Without pings, a hard crash or a
+    /// partition of one node leaves the other holding a stream that will never produce another byte
+    /// and never fail either: the initiator sits in a session it believes is healthy, and the
+    /// passive node's handler sits inside its single-stream guard, refusing the reconnect that would
+    /// heal the pair. Pings turn that into a detected failure on both ends.
+    /// </para>
+    /// <para>
+    /// Sent with <c>KeepAlivePingPolicy.Always</c>, so they flow on an idle connection too — an
+    /// active-streams-only policy would not fire in precisely the silent-pair case above. Safe over
+    /// h2c: a PING is an HTTP/2 frame, nothing to do with TLS.
+    /// </para>
+    /// </remarks>
+    public TimeSpan KeepAlivePingDelay { get; set; } = TimeSpan.FromSeconds(60);
+
+    /// <summary>
+    /// How long the initiator waits for a keepalive PING ack before tearing the connection down.
+    /// Default 20s. Detection therefore lands within roughly
+    /// <see cref="KeepAlivePingDelay"/> + this.
+    /// </summary>
+    public TimeSpan KeepAlivePingTimeout { get; set; } = TimeSpan.FromSeconds(20);
+
     /// <summary>How often the maintenance service enforces the oplog caps and prunes expired tombstones.</summary>
     public TimeSpan MaintenanceInterval { get; set; } = TimeSpan.FromSeconds(60);
 

ZB.MOM.WW.LocalDb/src/ZB.MOM.WW.LocalDb.Replication/ReplicationOptionsValidator.cs

@@ -28,6 +28,8 @@ public sealed class ReplicationOptionsValidator : IValidateOptions<ReplicationOp
             failures.Add($"LocalDb:Replication:FlushInterval must be greater than zero; got {options.FlushInterval}.");
         if (options.MaxBatchSize <= 0)
             failures.Add($"LocalDb:Replication:MaxBatchSize must be greater than 0; got {options.MaxBatchSize}.");
+        if (options.MaxBatchBytes <= 0)
+            failures.Add($"LocalDb:Replication:MaxBatchBytes must be greater than 0; got {options.MaxBatchBytes}.");
         if (options.MaxOplogRows <= 0)
             failures.Add($"LocalDb:Replication:MaxOplogRows must be greater than 0; got {options.MaxOplogRows}.");
         if (options.MaxOplogAge <= TimeSpan.Zero)
@@ -38,6 +40,10 @@ public sealed class ReplicationOptionsValidator : IValidateOptions<ReplicationOp
             failures.Add($"LocalDb:Replication:ReconnectBackoffMax must be greater than zero; got {options.ReconnectBackoffMax}.");
         else if (options.ReconnectBackoffMax > ReconnectBackoffCeiling)
             failures.Add($"LocalDb:Replication:ReconnectBackoffMax must be at most {ReconnectBackoffCeiling}; got {options.ReconnectBackoffMax}.");
+        if (options.KeepAlivePingDelay <= TimeSpan.Zero)
+            failures.Add($"LocalDb:Replication:KeepAlivePingDelay must be greater than zero; got {options.KeepAlivePingDelay}.");
+        if (options.KeepAlivePingTimeout <= TimeSpan.Zero)
+            failures.Add($"LocalDb:Replication:KeepAlivePingTimeout must be greater than zero; got {options.KeepAlivePingTimeout}.");
         if (options.MaintenanceInterval <= TimeSpan.Zero)
             failures.Add($"LocalDb:Replication:MaintenanceInterval must be greater than zero; got {options.MaintenanceInterval}.");
         if (options.MaxHlcDriftAhead <= TimeSpan.Zero)

ZB.MOM.WW.LocalDb/src/ZB.MOM.WW.LocalDb.Replication/SyncBackgroundService.cs

@@ -1,3 +1,4 @@
+using System.Net.Http;
 using Grpc.Core;
 using Grpc.Net.Client;
 using Microsoft.Extensions.Hosting;
@@ -76,7 +77,7 @@ public sealed class SyncBackgroundService : BackgroundService
             {
                 var channel = ChannelFactory is { } factory
                     ? factory()
-                    : ownedChannel = GrpcChannel.ForAddress(_options.PeerAddress);
+                    : ownedChannel = DialPeer();
 
                 var client = new LocalDbSync.LocalDbSyncClient(channel);
                 var headers = new Metadata();
@@ -152,6 +153,39 @@ public sealed class SyncBackgroundService : BackgroundService
         }
     }
 
+    /// <summary>
+    /// Dials the peer over a channel that sends HTTP/2 keepalive PINGs, so a half-open link is
+    /// detected instead of being waited on forever.
+    /// </summary>
+    /// <remarks>
+    /// <para>
+    /// The bare <c>GrpcChannel.ForAddress(address)</c> this replaces had NO keepalive of any kind,
+    /// and neither end of a sync stream ever times out on silence by design (a converged pair is
+    /// legitimately quiet). A hard crash or partition of the peer therefore left this node inside a
+    /// session that could never make progress and could never fail — and its own half of that stream
+    /// pinned on the peer, where the single-stream guard turns it into a refusal of the very
+    /// reconnect that heals the pair.
+    /// </para>
+    /// <para>
+    /// <c>DisposeHttpClient</c> hands the handler's lifetime to the channel, which this loop already
+    /// disposes on every exit path.
+    /// </para>
+    /// </remarks>
+    private GrpcChannel DialPeer() =>
+        GrpcChannel.ForAddress(_options.PeerAddress, new GrpcChannelOptions
+        {
+            HttpHandler = new SocketsHttpHandler
+            {
+                KeepAlivePingDelay = _options.KeepAlivePingDelay,
+                KeepAlivePingTimeout = _options.KeepAlivePingTimeout,
+                // Always, not WithActiveRequests: the case being closed is a SILENT stream on a dead
+                // link, and an active-requests policy stops pinging once the peer stops responding.
+                KeepAlivePingPolicy = HttpKeepAlivePingPolicy.Always,
+                EnableMultipleHttp2Connections = true,
+            },
+            DisposeHttpClient = true,
+        });
+
     private static bool IsSchemaMismatch(Exception? ex) =>
         ex is LocalDbSchemaMismatchException
         || (ex is RpcException rpc && rpc.StatusCode == StatusCode.FailedPrecondition);

ZB.MOM.WW.LocalDb/src/ZB.MOM.WW.LocalDb/ILocalDb.cs

@@ -38,10 +38,135 @@ public interface ILocalDb
     /// schema digest, and installs the three AFTER triggers that capture every change into the
     /// oplog + row-version tables within the same transaction as the consumer's write. Idempotent
     /// (re-registering regenerates the triggers). Throws <see cref="LocalDbRegistrationException"/>
-    /// if the name is unsafe, the table is missing, or it has no primary key.
+    /// if the name is unsafe, the table is missing, it has no primary key, or it is the parent of a
+    /// foreign key whose <c>ON DELETE</c> action mutates the child (<c>CASCADE</c> / <c>SET NULL</c> /
+    /// <c>SET DEFAULT</c>) — see the remarks for why that last one cannot be replicated.
     /// </summary>
+    /// <remarks>
+    /// <b>Replicated tables may not be the parent of a cascading foreign key.</b> The apply path
+    /// writes remote rows with <c>INSERT OR REPLACE</c>; REPLACE deletes the conflicting row first,
+    /// and with <c>PRAGMA foreign_keys=ON</c> that delete fires the child's <c>ON DELETE</c> action
+    /// while change capture is suppressed. The child rows would be mutated on the applying node only,
+    /// never captured and never shipped, diverging the pair with nothing able to detect it. The check
+    /// runs at registration and inspects the whole database's foreign keys; SQLite cannot add a
+    /// foreign key to an existing table, so a later-created child table is caught on the next
+    /// registration (i.e. the next startup), not at the moment of the DDL.
+    /// </remarks>
     ReplicatedTable RegisterReplicated(string tableName);
 
+    /// <summary>
+    /// As <see cref="RegisterReplicated(string)"/>, additionally baselining rows that already exist
+    /// in the table when <paramref name="baselineExistingRows"/> is true.
+    /// </summary>
+    /// <param name="tableName">The table to opt into replication.</param>
+    /// <param name="baselineExistingRows">
+    /// When true, every row already in the table that has no <c>__localdb_row_version</c> entry is
+    /// given one at the <b>baseline floor</b> — HLC <c>0</c>, stamped with this node's id — and, if
+    /// anything was seeded, a snapshot resync is flagged <b>for this table</b> so the peer actually
+    /// receives them.
+    /// </param>
+    /// <remarks>
+    /// <para>
+    /// <b>The problem this solves.</b> Capture is change-data-capture: without baselining, a row
+    /// written before registration exists in neither the oplog nor the row-version ledger, and the
+    /// snapshot streamer pages from the LEDGER — so a node that starts replicating LATE never sends
+    /// its pre-existing rows, silently and forever. Opting in seeds the ledger so a snapshot can
+    /// carry them.
+    /// </para>
+    /// <para>
+    /// <b>Why the floor is HLC 0, and why that is LWW-safe.</b> A seeded version must never beat a
+    /// genuine write. Every genuine HLC is a UTC millisecond shifted left 16 bits, so every genuine
+    /// HLC is greater than 0: a baselined row therefore LOSES every comparison against a real remote
+    /// write of the same key, and wins only where the peer has no version of that key at all
+    /// (which is precisely the case baselining exists to fix). It cannot perturb the local clock
+    /// either — clock recovery takes a MAX over existing HLCs, and 0 raises nothing.
+    /// </para>
+    /// <para>
+    /// <b>The one genuinely ambiguous case</b> is both nodes baselining the SAME key (e.g. both
+    /// restored the same legacy file). Both sides hold HLC 0, so the LWW tie-break falls to the
+    /// node-id ordinal: deterministic and convergent, but arbitrary as to content. Nothing can do
+    /// better — two rows of unknown, unordered antiquity carry no information about which is newer.
+    /// </para>
+    /// <para>
+    /// <b>Idempotent.</b> Seeding uses <c>ON CONFLICT DO NOTHING</c>, so a row that already has a
+    /// ledger entry — captured normally, or seeded on an earlier boot — keeps its genuine version
+    /// and no snapshot is flagged. Passing true on every boot is therefore free after the first.
+    /// </para>
+    /// <para>
+    /// <b>Cost, and what bounds it (0.3.0).</b> The snapshot flag is <b>per-table</b>
+    /// (<c>__localdb_snapshot_state</c>, on-disk schema v3), so baselining one table asks the peer
+    /// for that table alone — the tables around it are not re-streamed. Through 0.2.x the flag was a
+    /// single per-database bit and baselining anything re-sent everything, in both directions at
+    /// once on a pair enabling replication late.
+    /// </para>
+    /// <para>
+    /// Two cases still stream every registered table, by design. A <b>pre-0.3.0 peer</b> cannot be
+    /// told a snapshot is partial (it would advance its watermark past deltas it never received),
+    /// so a scoped snapshot is promoted to a full one for it. And the <b>oplog-horizon</b> paths —
+    /// a cap prune, an impossible peer ack — invalidate delta continuity for every table at once
+    /// and still set the database-wide flag. So the operational advice survives, narrowed: baseline
+    /// a high-volume table (an event log at its retention cap, say) deliberately, and <b>purge or
+    /// trim it first</b>, because that ONE table's rows still travel in full. The work is bounded
+    /// either way — outbound messages respect <c>MaxBatchBytes</c>, the receive side is bounded by
+    /// the transport's inbox.
+    /// </para>
+    /// <para>
+    /// <b>Crash safety.</b> Between 0.3.0 peers a table's flag is cleared only when the receiver
+    /// acknowledges the completed snapshot, so an ungraceful exit part-way through re-streams the
+    /// same scope on reconnect rather than losing it. Against a pre-0.3.0 peer (which sends no such
+    /// acknowledgement) the 0.2.x behaviour is kept: the flag clears once the snapshot is on the
+    /// wire, and the handshake's horizon gap check is the net.
+    /// </para>
+    /// </remarks>
+    ReplicatedTable RegisterReplicated(string tableName, bool baselineExistingRows);
+
+    /// <summary>
+    /// Removes a table from replication: drops its three capture triggers and prunes its rows from
+    /// <c>__localdb_oplog</c> and <c>__localdb_row_version</c>, in one transaction. Idempotent, and
+    /// deliberately usable on a table this process never registered — cleaning up triggers left
+    /// behind by an EARLIER build is the primary use.
+    /// </summary>
+    /// <param name="tableName">The table to stop capturing. The table's own data is never touched.</param>
+    /// <returns>True if anything was removed (a trigger dropped or a row pruned); false if there was nothing to clean up.</returns>
+    /// <remarks>
+    /// <para>
+    /// <b>Intended caller:</b> an application that has decided this node no longer replicates (or no
+    /// longer replicates this table). Capture is not free — the three triggers run inside every
+    /// consumer write transaction and append to an oplog nothing will ever drain — so a node with no
+    /// peer should not be paying for it.
+    /// </para>
+    /// <para>
+    /// <b>Pruning the row-version ledger is safe with respect to re-registration</b>, because
+    /// <see cref="RegisterReplicated(string, bool)"/> can re-baseline. Without baselining, a
+    /// deregister-then-register cycle would leave the previously-known rows unstreamable — the same
+    /// late-opt-in hole — so the two APIs are designed as a pair.
+    /// </para>
+    /// <para>
+    /// <b>Deregistration must be symmetric across a replicating pair, and the library enforces it
+    /// loudly.</b> The sync handshake compares the two nodes' registered-table digests fail-closed,
+    /// so a node that drops a table its peer still replicates simply stops syncing — with a
+    /// schema-mismatch error, not silent divergence. Deregister on both nodes, or on neither.
+    /// </para>
+    /// <para>
+    /// <b>The HLC's durable crash anchor is flushed as part of the same transaction</b>, before the
+    /// prune. The clock recovers on open as the MAX of <c>hlc_high_water</c>, the oplog's HLCs, the
+    /// row-version ledger's HLCs and the last-seen peer HLC — and <c>hlc_high_water</c> is otherwise
+    /// written only on graceful disposal. Since this API deletes the deregistered table's oplog and
+    /// ledger rows, a file with no other registered table would, after an UNGRACEFUL exit (crash,
+    /// SIGKILL, power loss), have nothing left to recover from: the clock would restart from
+    /// wall-clock time and could re-issue stamps below HLCs peers already hold, which LWW then
+    /// discards as older. Flushing here removes that window.
+    /// </para>
+    /// <para>
+    /// <b>Backlog-depth residual.</b> Pruning one table's rows can leave holes in the oplog's seq
+    /// range, and the O(1) depth estimate assumes the retained range is contiguous, so it can
+    /// over-report until the peer acks past the holes. Over-reporting is conservative — it can only
+    /// trigger an earlier snapshot resync, never suppress one — and on the node this API is for
+    /// (no peer at all) depth is moot.
+    /// </para>
+    /// </remarks>
+    bool DeregisterReplicated(string tableName);
+
     /// <summary>The tables currently opted into replication, keyed by table name.</summary>
     IReadOnlyDictionary<string, ReplicatedTable> ReplicatedTables { get; }
 }

ZB.MOM.WW.LocalDb/src/ZB.MOM.WW.LocalDb/Internal/LocalDbSchema.cs

@@ -1,3 +1,4 @@
+using System.Globalization;
 using Microsoft.Data.Sqlite;
 
 namespace ZB.MOM.WW.LocalDb.Internal;
@@ -7,6 +8,40 @@ namespace ZB.MOM.WW.LocalDb.Internal;
 /// </summary>
 internal static class LocalDbSchema
 {
+    /// <summary>
+    /// The on-disk bookkeeping-schema version this build creates and upgrades to. Bumping it means
+    /// adding a case to <see cref="Upgrade"/>; existing files are upgraded in place on open.
+    /// </summary>
+    /// <remarks>
+    /// Deliberately NOT the same number as <c>SyncSession.LibSchemaVersion</c>, which is the WIRE
+    /// protocol version and is compared fail-closed against the peer's. This one is local-file-only:
+    /// two nodes may briefly sit on different file-schema versions during an upgrade and still sync,
+    /// because nothing about the file layout crosses the wire.
+    /// <list type="bullet">
+    /// <item><description>
+    /// <b>1</b> — the original layout, which also created <c>__localdb_oplog_hlc</c>.
+    /// </description></item>
+    /// <item><description>
+    /// <b>2</b> — drops <c>__localdb_oplog_hlc</c>. That index (<c>table_name, pk_json, hlc</c>) was
+    /// never read by any query: every oplog access path is by <c>seq</c> (the rowid) or a full
+    /// <c>hlc</c>-predicate delete. It cost a b-tree insert on the CAPTURE path — i.e. inside every
+    /// consumer write transaction — to serve nothing.
+    /// </description></item>
+    /// <item><description>
+    /// <b>3</b> — adds <c>__localdb_snapshot_state</c>, the PER-TABLE snapshot-resync flag. Until
+    /// v3 the only flag was <c>__localdb_peer_state.needs_snapshot</c>, one bit for the whole
+    /// database, so baselining a single table re-streamed EVERY registered table in full — in both
+    /// directions at once on a pair enabling replication late. The database-wide bit survives with
+    /// its original meaning (the oplog-horizon and cap-prune paths genuinely invalidate every
+    /// table); the new table narrows the baseline case to the tables that actually need it. The
+    /// transition itself is a version stamp only — <see cref="Ddl"/> creates the table with
+    /// <c>IF NOT EXISTS</c> on every open, so an older file gains it whatever its stamp says, and
+    /// an empty flag table means exactly what it meant before: nothing per-table is owed.
+    /// </description></item>
+    /// </list>
+    /// </remarks>
+    internal const long CurrentSchemaVersion = 3;
+
     private const string Ddl = """
         CREATE TABLE IF NOT EXISTS __localdb_meta (
           id INTEGER PRIMARY KEY CHECK (id = 1),
@@ -25,7 +60,6 @@ internal static class LocalDbSchema
           node_id TEXT NOT NULL,
           is_tombstone INTEGER NOT NULL DEFAULT 0
         );
-        CREATE INDEX IF NOT EXISTS __localdb_oplog_hlc ON __localdb_oplog(table_name, pk_json, hlc);
         CREATE TABLE IF NOT EXISTS __localdb_peer_state (
           id INTEGER PRIMARY KEY CHECK (id = 1),
           peer_node_id TEXT NULL,
@@ -50,6 +84,14 @@ internal static class LocalDbSchema
           id INTEGER PRIMARY KEY CHECK (id = 1),
           applying INTEGER NOT NULL DEFAULT 0
         );
+        -- Per-table snapshot-resync flags (schema v3). A row exists only while that table owes the
+        -- peer a snapshot; the whole-database __localdb_peer_state.needs_snapshot bit still means
+        -- "every registered table" and is what the cap/horizon paths set.
+        -- WITHOUT ROWID: table_name IS the only access path (read as a set, cleared by name).
+        CREATE TABLE IF NOT EXISTS __localdb_snapshot_state (
+          table_name TEXT NOT NULL PRIMARY KEY,
+          needs_snapshot INTEGER NOT NULL DEFAULT 0
+        ) WITHOUT ROWID;
         CREATE TABLE IF NOT EXISTS __localdb_dead_letter (
           id INTEGER PRIMARY KEY AUTOINCREMENT,
           received_utc TEXT NOT NULL,
@@ -80,14 +122,67 @@ internal static class LocalDbSchema
             meta.Transaction = transaction;
             meta.CommandText =
                 "INSERT INTO __localdb_meta (id, node_id, schema_version) " +
-                "SELECT 1, $nodeId, 1 WHERE NOT EXISTS (SELECT 1 FROM __localdb_meta)";
+                "SELECT 1, $nodeId, $version WHERE NOT EXISTS (SELECT 1 FROM __localdb_meta)";
             meta.Parameters.AddWithValue("$nodeId", Guid.NewGuid().ToString("D"));
+            meta.Parameters.AddWithValue("$version", CurrentSchemaVersion);
             meta.ExecuteNonQuery();
         }
 
+        Upgrade(connection, transaction);
+
         transaction.Commit();
     }
 
+    /// <summary>
+    /// Brings a file created by an OLDER build up to <see cref="CurrentSchemaVersion"/>, in the
+    /// caller's transaction. A file already at the current version is untouched.
+    /// </summary>
+    /// <remarks>
+    /// Each step is written to be safe to re-run (IF EXISTS / IF NOT EXISTS), so a crash between the
+    /// DDL and the version stamp costs nothing but a repeat on the next open. Steps run in ascending
+    /// order and the stamp is written once at the end, inside the same transaction as the DDL — a
+    /// file is therefore never observed at the new version with an unapplied step.
+    /// </remarks>
+    private static void Upgrade(SqliteConnection connection, SqliteTransaction transaction)
+    {
+        long current;
+        using (var read = connection.CreateCommand())
+        {
+            read.Transaction = transaction;
+            read.CommandText = "SELECT schema_version FROM __localdb_meta WHERE id = 1";
+            current = Convert.ToInt64(read.ExecuteScalar(), CultureInfo.InvariantCulture);
+        }
+
+        if (current >= CurrentSchemaVersion)
+            return;
+
+        var steps = new List<string>();
+
+        // 1 -> 2: drop the write-amplifying, never-read oplog index (see CurrentSchemaVersion).
+        if (current < 2)
+            steps.Add("DROP INDEX IF EXISTS __localdb_oplog_hlc;");
+
+        // 2 -> 3: __localdb_snapshot_state. No step: the shared Ddl above already ran
+        // CREATE TABLE IF NOT EXISTS for it in this same transaction, and an empty flag table is
+        // the correct v3 state for a file that has never baselined a table. Repeating the DDL here
+        // would be a second source of truth for the same layout. The stamp still moves, because
+        // schema_version records what the file's layout IS, not which statements produced it.
+
+        if (steps.Count > 0)
+        {
+            using var ddl = connection.CreateCommand();
+            ddl.Transaction = transaction;
+            ddl.CommandText = string.Concat(steps);
+            ddl.ExecuteNonQuery();
+        }
+
+        using var stamp = connection.CreateCommand();
+        stamp.Transaction = transaction;
+        stamp.CommandText = "UPDATE __localdb_meta SET schema_version = $version WHERE id = 1";
+        stamp.Parameters.AddWithValue("$version", CurrentSchemaVersion);
+        stamp.ExecuteNonQuery();
+    }
+
     internal static (string NodeId, long HlcHighWater) ReadMeta(SqliteConnection connection)
     {
         using var cmd = connection.CreateCommand();
@@ -99,10 +194,22 @@ internal static class LocalDbSchema
         return (reader.GetString(0), reader.GetInt64(1));
     }
 
-    internal static void FlushHlcHighWater(SqliteConnection connection, long value)
+    /// <summary>
+    /// Writes the clock's durable crash anchor. <paramref name="transaction"/> must be supplied when
+    /// the connection has one open (SQLite commands are rejected otherwise), which is what lets a
+    /// caller flush the anchor atomically with a change that would otherwise erase the clock's other
+    /// recovery evidence — see <see cref="ILocalDb.DeregisterReplicated"/>.
+    /// </summary>
+    /// <remarks>
+    /// MAX-guarded rather than an assignment: every writer of this column is flushing a monotonic
+    /// clock, so a lower value can only be a stale one, and the anchor exists precisely to never
+    /// regress.
+    /// </remarks>
+    internal static void FlushHlcHighWater(SqliteConnection connection, long value, SqliteTransaction? transaction = null)
     {
         using var cmd = connection.CreateCommand();
-        cmd.CommandText = "UPDATE __localdb_meta SET hlc_high_water = $value WHERE id = 1";
+        cmd.Transaction = transaction;
+        cmd.CommandText = "UPDATE __localdb_meta SET hlc_high_water = MAX(hlc_high_water, $value) WHERE id = 1";
         cmd.Parameters.AddWithValue("$value", value);
         cmd.ExecuteNonQuery();
     }

ZB.MOM.WW.LocalDb/src/ZB.MOM.WW.LocalDb/Internal/SqliteLocalDb.cs

@@ -138,7 +138,15 @@ internal sealed class SqliteLocalDb : ILocalDb, IDisposable
         return await ReadAllAsync(cmd, map, ct);
     }
 
-    public async Task<ILocalDbTransaction> BeginTransactionAsync(CancellationToken ct = default)
+    public async Task<ILocalDbTransaction> BeginTransactionAsync(CancellationToken ct = default) =>
+        await BeginPreparedTransactionAsync(ct);
+
+    /// <summary>
+    /// As <see cref="BeginTransactionAsync"/>, but typed so internal callers reach the underlying
+    /// connection and transaction — which is what lets a hot loop hold <see cref="SqliteCommand"/>s
+    /// across many rows and rebind them, instead of re-preparing statements per statement per row.
+    /// </summary>
+    internal async Task<Transaction> BeginPreparedTransactionAsync(CancellationToken ct = default)
     {
         var conn = await CreateConnectionAsync(ct);
         try
@@ -155,8 +163,11 @@ internal sealed class SqliteLocalDb : ILocalDb, IDisposable
 
     public IReadOnlyDictionary<string, ReplicatedTable> ReplicatedTables => _replicatedTables;
 
+    public ReplicatedTable RegisterReplicated(string tableName) =>
+        RegisterReplicated(tableName, baselineExistingRows: false);
+
     // Thread-safe: all master-connection work runs under _masterLock.
-    public ReplicatedTable RegisterReplicated(string tableName)
+    public ReplicatedTable RegisterReplicated(string tableName, bool baselineExistingRows)
     {
         if (string.IsNullOrWhiteSpace(tableName) || !TableNameRegex.IsMatch(tableName))
             throw new LocalDbRegistrationException(
@@ -188,6 +199,8 @@ internal sealed class SqliteLocalDb : ILocalDb, IDisposable
                 throw new LocalDbRegistrationException(
                     $"Cannot register table '{tableName}': replication requires an explicit primary key.");
 
+            EnsureNoInboundActionOnDelete(tableName);
+
             var allColumns = columns.Select(c => (c.Name, c.Type)).ToList();
             var digest = ReplicatedTable.ComputeDigest(tableName, pkColumns, allColumns);
             var table = new ReplicatedTable(tableName, pkColumns, allColumns, digest);
@@ -201,6 +214,10 @@ internal sealed class SqliteLocalDb : ILocalDb, IDisposable
                     cmd.CommandText = script;
                     cmd.ExecuteNonQuery();
                 }
+
+                if (baselineExistingRows)
+                    BaselineExistingRows(tx, table);
+
                 tx.Commit();
             }
 
@@ -209,6 +226,193 @@ internal sealed class SqliteLocalDb : ILocalDb, IDisposable
         }
     }
 
+    // Thread-safe: all master-connection work runs under _masterLock.
+    public bool DeregisterReplicated(string tableName)
+    {
+        if (string.IsNullOrWhiteSpace(tableName) || !TableNameRegex.IsMatch(tableName))
+            throw new LocalDbRegistrationException(
+                $"Cannot deregister table '{tableName}': name must be a plain identifier matching {TableNameRegex}.");
+
+        lock (_masterLock)
+        {
+            using var tx = _master.BeginTransaction();
+
+            // Counted BEFORE the drop: DROP TRIGGER reports no row count, and "did anything change?"
+            // is the caller's cue to log a one-time cleanup rather than a no-op every boot.
+            long triggersPresent;
+            using (var probe = _master.CreateCommand())
+            {
+                probe.Transaction = tx;
+                probe.CommandText =
+                    "SELECT COUNT(*) FROM sqlite_master WHERE type = 'trigger' AND name IN ($ai, $au, $ad)";
+                foreach (var suffix in TriggerSqlGenerator.Suffixes)
+                    probe.Parameters.AddWithValue("$" + suffix, TriggerSqlGenerator.TriggerName(tableName, suffix));
+                triggersPresent = Convert.ToInt64(probe.ExecuteScalar(), CultureInfo.InvariantCulture);
+            }
+
+            using (var drop = _master.CreateCommand())
+            {
+                drop.Transaction = tx;
+                drop.CommandText = TriggerSqlGenerator.BuildUninstallScript(tableName);
+                drop.ExecuteNonQuery();
+            }
+
+            // Flushed BEFORE the prune below, in the SAME transaction, because that prune destroys
+            // the clock's crash evidence. Recovery on open takes a MAX over hlc_high_water, the
+            // oplog's HLCs, the row-version ledger's HLCs and last_seen_hlc — and hlc_high_water is
+            // otherwise written only by Dispose. Deregistering a table deletes that table's oplog
+            // AND ledger rows, so on a file whose registered tables are all deregistered, an
+            // UNGRACEFUL exit (crash, SIGKILL, power loss — no Dispose) afterwards leaves nothing
+            // above zero to recover from: the reopened clock would restart at wall-clock time and
+            // could re-issue HLCs below stamps peers have already observed, which LWW reads as
+            // "older" and discards. One row-level UPDATE on an administrative one-shot buys the
+            // anchor back; it is a separate command from the prune so it cannot inflate prunedRows.
+            LocalDbSchema.FlushHlcHighWater(_master, _clock.Current, tx);
+
+            // Full scan of the oplog by table_name — the index that would have served it is exactly
+            // the one schema v2 drops, and paying a write on every capture to speed up a one-shot
+            // administrative prune is the wrong trade.
+            int prunedRows;
+            using (var prune = _master.CreateCommand())
+            {
+                prune.Transaction = tx;
+                prune.CommandText =
+                    "DELETE FROM __localdb_oplog WHERE table_name = $t;" +
+                    "DELETE FROM __localdb_row_version WHERE table_name = $t;" +
+                    // A pending per-table snapshot flag for a table nothing streams any more would
+                    // never clear — the streamer skips unregistered tables, so the flag would sit
+                    // in the file forever, and re-registering later would find a stale "already
+                    // owed" it did not ask for. Dropped with the ledger it refers to.
+                    "DELETE FROM __localdb_snapshot_state WHERE table_name = $t;";
+                prune.Parameters.AddWithValue("$t", tableName);
+                prunedRows = prune.ExecuteNonQuery();
+            }
+
+            tx.Commit();
+
+            _replicatedTables.TryRemove(tableName, out _);
+            return triggersPresent > 0 || prunedRows > 0;
+        }
+    }
+
+    /// <summary>
+    /// Refuses to register a table that is the PARENT of a foreign key whose <c>ON DELETE</c> action
+    /// mutates the child (<c>CASCADE</c>, <c>SET NULL</c>, <c>SET DEFAULT</c>).
+    /// </summary>
+    /// <remarks>
+    /// <para>
+    /// <b>Why this is a replication hazard, not a schema opinion.</b> The apply path writes a remote
+    /// row with <c>INSERT OR REPLACE</c>, and REPLACE resolves a primary-key conflict by DELETING the
+    /// existing row before inserting the new one. Connections run <c>PRAGMA foreign_keys=ON</c>, and
+    /// that implicit delete fires foreign-key actions (verified behaviour, not inference). It runs
+    /// with the capture guard raised, so the resulting child-row deletes/nulls are invisible to the
+    /// oplog: they happen on this node and are never shipped. The peer, applying the same parent row,
+    /// keeps its children — and the pair has silently diverged with nothing to detect it, since LWW
+    /// only ever compares rows it was told about.
+    /// </para>
+    /// <para>
+    /// <b>Checked at registration only.</b> SQLite cannot add a foreign key to an existing table, so
+    /// a new offender can only arrive as a child table created (or a parent table rebuilt) after this
+    /// call. Consumers register their tables at startup, after schema creation/migration, which is
+    /// where the check belongs; a schema change that adds a cascading child to an already-registered
+    /// parent is caught on the next boot, not at the moment of the DDL.
+    /// </para>
+    /// <para>
+    /// <c>RESTRICT</c> / <c>NO ACTION</c> are deliberately allowed: they abort the offending statement
+    /// with a constraint error rather than mutating rows behind capture's back, which the apply path
+    /// already handles as per-entry poison.
+    /// </para>
+    /// </remarks>
+    private void EnsureNoInboundActionOnDelete(string tableName)
+    {
+        var offenders = new List<string>();
+        using (var cmd = _master.CreateCommand())
+        {
+            // pragma_foreign_key_list is the table-valued form of PRAGMA foreign_key_list, so the
+            // whole database is one query: fk."table" is the PARENT this child points at.
+            cmd.CommandText =
+                "SELECT m.name, fk.\"from\", fk.\"on_delete\" " +
+                "FROM sqlite_master m JOIN pragma_foreign_key_list(m.name) fk " +
+                "WHERE m.type = 'table' AND fk.\"table\" = $t COLLATE NOCASE " +
+                "AND UPPER(fk.\"on_delete\") IN ('CASCADE', 'SET NULL', 'SET DEFAULT') " +
+                "ORDER BY m.name, fk.\"from\"";
+            cmd.Parameters.AddWithValue("$t", tableName);
+            using var reader = cmd.ExecuteReader();
+            while (reader.Read())
+                offenders.Add(
+                    $"{reader.GetString(0)}.{reader.GetString(1)} ON DELETE {reader.GetString(2).ToUpperInvariant()}");
+        }
+
+        if (offenders.Count == 0)
+            return;
+
+        throw new LocalDbRegistrationException(
+            $"Cannot register table '{tableName}': it is the parent of a foreign key with a row-mutating " +
+            $"ON DELETE action ({string.Join(", ", offenders)}). Replication applies remote rows with " +
+            "INSERT OR REPLACE, whose implicit delete fires that action with change capture suppressed — " +
+            "the child rows would be changed on this node only and never replicated, diverging the pair " +
+            "undetectably. Use ON DELETE NO ACTION/RESTRICT and delete children explicitly, or leave this " +
+            "table unreplicated.");
+    }
+
+    /// <summary>
+    /// Seeds <c>__localdb_row_version</c> at the baseline floor for every row of
+    /// <paramref name="table"/> that has no ledger entry, and flags a snapshot resync if it seeded
+    /// anything. Runs in the registration transaction so triggers and baseline commit together.
+    /// </summary>
+    /// <remarks>
+    /// <para>
+    /// The floor is HLC <c>0</c> — see <see cref="ILocalDb.RegisterReplicated(string, bool)"/> for
+    /// why that is the only LWW-safe choice.
+    /// </para>
+    /// <para>
+    /// <b>The <c>needs_snapshot</c> flag is what makes this actually deliver.</b> Seeding the ledger
+    /// alone changes nothing on the wire: a node whose oplog starts at seq 1 presents no gap at the
+    /// handshake, so the peer is told it needs no snapshot and only ever receives post-registration
+    /// deltas. Flagging the resync is what turns the seeded ledger into rows the peer receives.
+    /// Deliberately NOT done by writing oplog entries instead: that would dump the entire table into
+    /// the oplog on the first boot after opting in, likely blowing <c>MaxOplogRows</c> and forcing
+    /// the same snapshot anyway, via a cap-prune.
+    /// </para>
+    /// <para>
+    /// <b>The flag is PER-TABLE</b> (<c>__localdb_snapshot_state</c>, schema v3): baselining one
+    /// table asks for a snapshot of that table alone, not of every registered table. The
+    /// database-wide <c>__localdb_peer_state.needs_snapshot</c> bit is untouched here and keeps its
+    /// original meaning — the oplog-horizon and cap-prune paths invalidate every table's delta
+    /// continuity at once, and only those paths set it. A peer too old to understand a scoped
+    /// snapshot is served a full one instead; the negotiation lives in the replication package.
+    /// </para>
+    /// </remarks>
+    private void BaselineExistingRows(SqliteTransaction tx, ReplicatedTable table)
+    {
+        int seeded;
+        using (var cmd = _master.CreateCommand())
+        {
+            cmd.Transaction = tx;
+            // The trailing WHERE 1 disambiguates the upsert parse on a SELECT source — the same
+            // reason TriggerSqlGenerator's row-version statements carry one.
+            cmd.CommandText =
+                "INSERT INTO __localdb_row_version (table_name, pk_json, hlc, node_id, is_tombstone, tombstone_utc) " +
+                $"SELECT $t, {TriggerSqlGenerator.JsonObject("t", table.PkColumns)}, 0, $node, 0, NULL " +
+                $"FROM {TriggerSqlGenerator.Ident(table.Name)} t WHERE 1 " +
+                "ON CONFLICT(table_name, pk_json) DO NOTHING";
+            cmd.Parameters.AddWithValue("$t", table.Name);
+            cmd.Parameters.AddWithValue("$node", _nodeId);
+            seeded = cmd.ExecuteNonQuery();
+        }
+
+        if (seeded == 0)
+            return;
+
+        using var flag = _master.CreateCommand();
+        flag.Transaction = tx;
+        flag.CommandText =
+            "INSERT INTO __localdb_snapshot_state (table_name, needs_snapshot) VALUES ($t, 1) " +
+            "ON CONFLICT(table_name) DO UPDATE SET needs_snapshot = 1";
+        flag.Parameters.AddWithValue("$t", table.Name);
+        flag.ExecuteNonQuery();
+    }
+
     public void Dispose()
     {
         if (_disposed)
@@ -329,8 +533,14 @@ internal sealed class SqliteLocalDb : ILocalDb, IDisposable
         return Convert.ToInt64(cmd.ExecuteScalar(), CultureInfo.InvariantCulture);
     }
 
-    private sealed class Transaction(SqliteConnection connection, SqliteTransaction transaction) : ILocalDbTransaction
+    internal sealed class Transaction(SqliteConnection connection, SqliteTransaction transaction) : ILocalDbTransaction
     {
+        /// <summary>The pinned connection, for internal callers holding their own commands.</summary>
+        internal SqliteConnection Connection => connection;
+
+        /// <summary>The underlying transaction, to assign to internally-held commands.</summary>
+        internal SqliteTransaction Sqlite => transaction;
+
         public async Task<int> ExecuteAsync(string sql, object? parameters = null, CancellationToken ct = default)
         {
             await using var cmd = connection.CreateCommand();

ZB.MOM.WW.LocalDb/src/ZB.MOM.WW.LocalDb/Registration/TriggerSqlGenerator.cs

@@ -27,14 +27,16 @@ internal static class TriggerSqlGenerator
         "ON CONFLICT(table_name, pk_json) DO UPDATE SET hlc = excluded.hlc, node_id = excluded.node_id, " +
         "is_tombstone = excluded.is_tombstone, tombstone_utc = excluded.tombstone_utc";
 
+    /// <summary>The three trigger suffixes, in the order the install/uninstall scripts emit them.</summary>
+    internal static readonly string[] Suffixes = ["ai", "au", "ad"];
+
     public static string TriggerName(string table, string suffix) => $"__localdb_{table}_{suffix}";
 
     /// <summary>DROP IF EXISTS + CREATE for all three triggers, ready to run in one transaction.</summary>
     public static string BuildInstallScript(ReplicatedTable table)
     {
         var sb = new StringBuilder();
-        foreach (var suffix in new[] { "ai", "au", "ad" })
-            sb.Append("DROP TRIGGER IF EXISTS ").Append(Ident(TriggerName(table.Name, suffix))).Append(";\n");
+        sb.Append(BuildUninstallScript(table.Name));
 
         sb.Append(BuildInsert(table)).Append('\n');
         sb.Append(BuildUpdate(table)).Append('\n');
@@ -42,6 +44,20 @@ internal static class TriggerSqlGenerator
         return sb.ToString();
     }
 
+    /// <summary>
+    /// DROP IF EXISTS for all three triggers of <paramref name="tableName"/>. Takes the bare name
+    /// rather than a <see cref="ReplicatedTable"/> because uninstall must work for a table this
+    /// process never registered — the whole point of the cleanup API is stale triggers left by an
+    /// EARLIER build, whose column list (and therefore digest) is unavailable and irrelevant.
+    /// </summary>
+    public static string BuildUninstallScript(string tableName)
+    {
+        var sb = new StringBuilder();
+        foreach (var suffix in Suffixes)
+            sb.Append("DROP TRIGGER IF EXISTS ").Append(Ident(TriggerName(tableName, suffix))).Append(";\n");
+        return sb.ToString();
+    }
+
     private static string BuildInsert(ReplicatedTable t) =>
         Wrap(t, "ai", "INSERT", NewRowCapture(t));
 
@@ -112,13 +128,16 @@ internal static class TriggerSqlGenerator
     }
 
     // json_object('col', ALIAS."col", ...) — natively handles NULLs and SQLite scalar types.
-    private static string JsonObject(string alias, IEnumerable<string> columns)
+    // Internal rather than private: baseline seeding must build a pk_json identical to what the
+    // capture triggers emit, character for character, or a seeded ledger key would never match the
+    // key a later capture writes for the same row.
+    internal static string JsonObject(string alias, IEnumerable<string> columns)
     {
         var args = columns.Select(c => $"{Literal(c)}, {alias}.{Ident(c)}");
         return $"json_object({string.Join(", ", args)})";
     }
 
-    private static string Ident(string name) => "\"" + name.Replace("\"", "\"\"") + "\"";
+    internal static string Ident(string name) => "\"" + name.Replace("\"", "\"\"") + "\"";
 
     private static string Literal(string value) => "'" + value.Replace("'", "''") + "'";
 }

ZB.MOM.WW.LocalDb/tests/ZB.MOM.WW.LocalDb.Tests/BatchBudgetTests.cs

@@ -0,0 +1,444 @@
+using System.Threading.Channels;
+using Microsoft.Data.Sqlite;
+using Microsoft.Extensions.Logging.Abstractions;
+using ZB.MOM.WW.LocalDb.Contracts;
+using ZB.MOM.WW.LocalDb.Internal;
+using ZB.MOM.WW.LocalDb.Replication;
+using ZB.MOM.WW.LocalDb.Replication.Internal;
+
+namespace ZB.MOM.WW.LocalDb.Tests;
+
+/// <summary>
+/// Outbound batching by summed serialized BYTES (with the row count kept as a secondary cap), and
+/// the inbound coalescing that keeps one transaction + one ack per burst rather than per message.
+/// </summary>
+public sealed class BatchBudgetTests : IDisposable
+{
+    private static readonly TimeSpan RunTimeout = TimeSpan.FromSeconds(15);
+
+    private readonly List<string> _paths = [];
+    private readonly List<IDisposable> _disposables = [];
+
+    public void Dispose()
+    {
+        foreach (var d in _disposables)
+            d.Dispose();
+        SqliteConnection.ClearAllPools();
+        foreach (var p in _paths)
+        {
+            if (File.Exists(p)) File.Delete(p);
+            if (File.Exists(p + "-wal")) File.Delete(p + "-wal");
+            if (File.Exists(p + "-shm")) File.Delete(p + "-shm");
+        }
+    }
+
+    private async Task<SqliteLocalDb> NewOrdersDbAsync()
+    {
+        var path = Path.Combine(Path.GetTempPath(), Guid.NewGuid() + ".db");
+        _paths.Add(path);
+        var db = new SqliteLocalDb(new LocalDbOptions { Path = path });
+        _disposables.Add(db);
+        await db.ExecuteAsync("CREATE TABLE orders (id INTEGER PRIMARY KEY, sku TEXT, qty INTEGER)");
+        db.RegisterReplicated("orders");
+        return db;
+    }
+
+    private static (SyncDuplex Duplex, ChannelWriter<SyncMessage> ToSession, ChannelReader<SyncMessage> FromSession) ScriptedPeer()
+    {
+        var toSession = Channel.CreateUnbounded<SyncMessage>();
+        var fromSession = Channel.CreateUnbounded<SyncMessage>();
+        var duplex = new SyncDuplex
+        {
+            Send = (m, ct) => fromSession.Writer.WriteAsync(m, ct).AsTask(),
+            Inbox = toSession.Reader,
+        };
+        return (duplex, toSession.Writer, fromSession.Reader);
+    }
+
+    private static Handshake MatchingHandshake(SqliteLocalDb sessionDb, string peerNodeId)
+    {
+        var h = new Handshake { NodeId = peerNodeId, LibSchemaVersion = 1, LastAppliedRemoteSeq = 0 };
+        foreach (var t in sessionDb.ReplicatedTables.Values.OrderBy(t => t.Name, StringComparer.Ordinal))
+            h.Tables.Add(new TableDigest { TableName = t.Name, Digest = t.Digest });
+        return h;
+    }
+
+    private static async Task<SyncMessage> NextAsync(ChannelReader<SyncMessage> reader, CancellationToken ct)
+    {
+        await reader.WaitToReadAsync(ct);
+        reader.TryRead(out var msg);
+        return msg!;
+    }
+
+    // Reads the next scripted-peer message while also observing the session task: a session that
+    // faults would otherwise leave this wait to run out the whole test timeout, reporting an
+    // opaque cancellation instead of the session's real exception.
+    private static async Task<SyncMessage> NextAsync(Task run, ChannelReader<SyncMessage> reader, CancellationToken ct)
+    {
+        var next = NextAsync(reader, ct);
+        if (ReferenceEquals(await Task.WhenAny(run, next), run))
+        {
+            await run;
+            throw new InvalidOperationException("The sync session ended before the expected message arrived.");
+        }
+        return await next;
+    }
+
+    private static async Task SwallowAsync(Task task)
+    {
+        try { await task; }
+        catch (OperationCanceledException) { }
+    }
+
+    // Queues a complete peer handshake AHEAD of the session's own, so anything queued after it is
+    // guaranteed to be sitting in the inbox by the time the receive loop starts. That is what makes
+    // the coalescing assertions deterministic rather than a race with the reader.
+    private static async Task PreloadHandshakeAsync(
+        SqliteLocalDb db, ChannelWriter<SyncMessage> toSession, CancellationToken ct)
+    {
+        await toSession.WriteAsync(new SyncMessage { Handshake = MatchingHandshake(db, "peer") }, ct);
+        await toSession.WriteAsync(
+            new SyncMessage { HandshakeAck = new HandshakeAck { NodeId = "peer", SnapshotRequired = false } }, ct);
+    }
+
+    private static long BatchBytes(DeltaBatch batch) =>
+        batch.Entries.Sum(e => 1L + Google.Protobuf.CodedOutputStream.ComputeMessageSize(e));
+
+    private static long BatchBytes(SnapshotBatch batch) =>
+        batch.Rows.Sum(r => 1L + Google.Protobuf.CodedOutputStream.ComputeMessageSize(r));
+
+    private static SyncSession NewSession(SqliteLocalDb db, ReplicationOptions options)
+    {
+        var store = new OplogStore(db, options);
+        var applier = new LwwApplier(db);
+        return new SyncSession(db, store, applier, options, NullLogger.Instance);
+    }
+
+    private static OplogEntry ProtoEntry(long seq, long id, long hlc) =>
+        new()
+        {
+            Seq = seq,
+            TableName = "orders",
+            PkJson = $"{{\"id\":{id}}}",
+            RowJson = $"{{\"id\":{id},\"sku\":\"S\",\"qty\":1}}",
+            Hlc = hlc,
+            NodeId = "peer",
+            IsTombstone = false,
+        };
+
+    // ---- Outbound: delta pump --------------------------------------------------------------
+
+    [Fact]
+    public async Task Pump_SplitsOneReadIntoSeveralMessages_EachWithinTheByteBudget()
+    {
+        const int budget = 4_000;
+        var db = await NewOrdersDbAsync();
+        var wide = new string('x', 900);
+        for (var i = 1; i <= 20; i++)
+            await db.ExecuteAsync(
+                "INSERT INTO orders (id, sku, qty) VALUES (@id, @sku, @id)", new { id = i, sku = wide });
+
+        // Row count alone would put all 20 wide rows (~18 KB) in ONE message; the byte budget is
+        // what actually bounds the message the transport has to carry.
+        var options = new ReplicationOptions
+        {
+            FlushInterval = TimeSpan.FromMilliseconds(20), MaxBatchSize = 500, MaxBatchBytes = budget,
+        };
+        var session = NewSession(db, options);
+        var (duplex, toSession, fromSession) = ScriptedPeer();
+
+        using var cts = new CancellationTokenSource(RunTimeout);
+        var run = session.RunAsync(duplex, cts.Token);
+        await PreloadHandshakeAsync(db, toSession, cts.Token);
+
+        var seen = new List<long>();
+        while (seen.Count < 20)
+        {
+            var msg = await NextAsync(run, fromSession, cts.Token);
+            if (msg.MsgCase != SyncMessage.MsgOneofCase.DeltaBatch)
+                continue;
+            Assert.True(BatchBytes(msg.DeltaBatch) <= budget,
+                $"a {BatchBytes(msg.DeltaBatch)}-byte message exceeded the {budget}-byte budget");
+            Assert.True(msg.DeltaBatch.Entries.Count > 1, "the budget should still fit several of these rows");
+            seen.AddRange(msg.DeltaBatch.Entries.Select(e => e.Seq));
+        }
+
+        // Split, not dropped or duplicated: every captured seq travelled exactly once, in order.
+        Assert.Equal(20, seen.Count);
+        Assert.Equal(seen.OrderBy(x => x).ToList(), seen);
+        Assert.Equal(20, seen.Distinct().Count());
+
+        cts.Cancel();
+        await SwallowAsync(run);
+    }
+
+    [Fact]
+    public async Task Pump_RowLargerThanTheBudget_IsSentAloneRatherThanStallingTheStream()
+    {
+        var db = await NewOrdersDbAsync();
+        await db.ExecuteAsync(
+            "INSERT INTO orders (id, sku, qty) VALUES (1, @sku, 1)", new { sku = new string('x', 5_000) });
+        await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (2, 'small', 2)");
+
+        var options = new ReplicationOptions
+        {
+            FlushInterval = TimeSpan.FromMilliseconds(20), MaxBatchSize = 500, MaxBatchBytes = 100,
+        };
+        var session = NewSession(db, options);
+        var (duplex, toSession, fromSession) = ScriptedPeer();
+
+        using var cts = new CancellationTokenSource(RunTimeout);
+        var run = session.RunAsync(duplex, cts.Token);
+        await PreloadHandshakeAsync(db, toSession, cts.Token);
+
+        var batches = new List<DeltaBatch>();
+        while (batches.Sum(b => b.Entries.Count) < 2)
+        {
+            var msg = await NextAsync(run, fromSession, cts.Token);
+            if (msg.MsgCase == SyncMessage.MsgOneofCase.DeltaBatch)
+                batches.Add(msg.DeltaBatch);
+        }
+
+        // The oversized row goes out on its own — refusing it would wedge the stream on one row
+        // forever, and everything behind it with it.
+        Assert.Equal(2, batches.Count);
+        Assert.Single(batches[0].Entries);
+        Assert.Equal(1, batches[0].Entries[0].Seq);
+        Assert.True(BatchBytes(batches[0]) > options.MaxBatchBytes);
+        Assert.Single(batches[1].Entries);
+        Assert.Equal(2, batches[1].Entries[0].Seq);
+
+        cts.Cancel();
+        await SwallowAsync(run);
+    }
+
+    [Fact]
+    public async Task Pump_RowCountRemainsASecondaryCap()
+    {
+        var db = await NewOrdersDbAsync();
+        for (var i = 1; i <= 12; i++)
+            await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (@id, 'S', @id)", new { id = i });
+
+        // Tiny rows, enormous byte budget: only the row cap can be doing the splitting here.
+        var options = new ReplicationOptions
+        {
+            FlushInterval = TimeSpan.FromMilliseconds(20), MaxBatchSize = 5, MaxBatchBytes = 8 * 1024 * 1024,
+        };
+        var session = NewSession(db, options);
+        var (duplex, toSession, fromSession) = ScriptedPeer();
+
+        using var cts = new CancellationTokenSource(RunTimeout);
+        var run = session.RunAsync(duplex, cts.Token);
+        await PreloadHandshakeAsync(db, toSession, cts.Token);
+
+        var seen = 0;
+        while (seen < 12)
+        {
+            var msg = await NextAsync(run, fromSession, cts.Token);
+            if (msg.MsgCase != SyncMessage.MsgOneofCase.DeltaBatch)
+                continue;
+            Assert.True(msg.DeltaBatch.Entries.Count <= 5);
+            seen += msg.DeltaBatch.Entries.Count;
+        }
+
+        cts.Cancel();
+        await SwallowAsync(run);
+    }
+
+    // ---- Outbound: snapshot --------------------------------------------------------------
+
+    [Fact]
+    public async Task Snapshot_SplitsPagesIntoMessagesWithinTheByteBudget()
+    {
+        const int budget = 4_000;
+        var db = await NewOrdersDbAsync();
+        var wide = new string('x', 900);
+        for (var i = 1; i <= 20; i++)
+            await db.ExecuteAsync(
+                "INSERT INTO orders (id, sku, qty) VALUES (@id, @sku, @id)", new { id = i, sku = wide });
+
+        // A snapshot is the easiest way to blow a message limit: it ships whole rows, all of them,
+        // back to back — so its page size has to be bounded in bytes too, not just in rows.
+        var options = new ReplicationOptions { MaxBatchSize = 500, MaxBatchBytes = budget };
+        var store = new OplogStore(db, options);
+        var streamer = new SnapshotStreamer(db, store, options, NullLogger.Instance);
+
+        var sent = new List<SyncMessage>();
+        await streamer.SendAsync(SnapshotScope.Full, (m, _) => { sent.Add(m); return Task.CompletedTask; }, default);
+
+        var batches = sent.Where(m => m.MsgCase == SyncMessage.MsgOneofCase.SnapshotBatch)
+            .Select(m => m.SnapshotBatch).ToList();
+        Assert.True(batches.Count > 1, "20 wide rows must not have travelled as one message");
+        Assert.All(batches, b => Assert.True(BatchBytes(b) <= budget,
+            $"a {BatchBytes(b)}-byte snapshot message exceeded the {budget}-byte budget"));
+
+        // Bracketed and complete: every row, exactly once.
+        Assert.Equal(SyncMessage.MsgOneofCase.SnapshotBegin, sent[0].MsgCase);
+        Assert.Equal(SyncMessage.MsgOneofCase.SnapshotComplete, sent[^1].MsgCase);
+        var pks = batches.SelectMany(b => b.Rows).Select(r => r.PkJson).ToList();
+        Assert.Equal(20, pks.Count);
+        Assert.Equal(20, pks.Distinct().Count());
+    }
+
+    [Fact]
+    public async Task Snapshot_RowLargerThanTheBudget_IsSentAlone()
+    {
+        var db = await NewOrdersDbAsync();
+        await db.ExecuteAsync(
+            "INSERT INTO orders (id, sku, qty) VALUES (1, @sku, 1)", new { sku = new string('x', 5_000) });
+        await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (2, 'small', 2)");
+
+        var options = new ReplicationOptions { MaxBatchSize = 500, MaxBatchBytes = 100 };
+        var store = new OplogStore(db, options);
+        var streamer = new SnapshotStreamer(db, store, options, NullLogger.Instance);
+
+        var sent = new List<SyncMessage>();
+        await streamer.SendAsync(SnapshotScope.Full, (m, _) => { sent.Add(m); return Task.CompletedTask; }, default);
+
+        var batches = sent.Where(m => m.MsgCase == SyncMessage.MsgOneofCase.SnapshotBatch)
+            .Select(m => m.SnapshotBatch).ToList();
+        Assert.Equal(2, batches.Count);
+        Assert.All(batches, b => Assert.Single(b.Rows));
+        Assert.Equal("{\"id\":1}", batches[0].Rows[0].PkJson);
+        Assert.Equal("{\"id\":2}", batches[1].Rows[0].PkJson);
+    }
+
+    // ---- Inbound: coalescing -------------------------------------------------------------
+
+    [Fact]
+    public async Task Receive_CoalescesQueuedDeltaBatches_IntoOneAck()
+    {
+        var db = await NewOrdersDbAsync();
+        var options = new ReplicationOptions { FlushInterval = TimeSpan.FromSeconds(30) };
+        var session = NewSession(db, options);
+        var (duplex, toSession, fromSession) = ScriptedPeer();
+
+        using var cts = new CancellationTokenSource(RunTimeout);
+        var run = session.RunAsync(duplex, cts.Token);
+
+        // Queued ahead of the receive loop, so all three are demonstrably in the inbox together.
+        await PreloadHandshakeAsync(db, toSession, cts.Token);
+        for (var i = 1; i <= 3; i++)
+        {
+            var batch = new DeltaBatch();
+            batch.Entries.Add(ProtoEntry(i, i, 1_000_000 + i));
+            await toSession.WriteAsync(new SyncMessage { DeltaBatch = batch }, cts.Token);
+        }
+
+        SyncMessage ack;
+        do { ack = await NextAsync(run, fromSession, cts.Token); }
+        while (ack.MsgCase != SyncMessage.MsgOneofCase.DeltaAck);
+
+        // The discriminator: uncoalesced, the FIRST ack could only cover seq 1.
+        Assert.Equal(3, ack.DeltaAck.AppliedThruSeq);
+
+        var rows = await db.QueryAsync("SELECT COUNT(*) FROM orders", r => r.GetInt64(0));
+        Assert.Equal(3, rows[0]);
+
+        cts.Cancel();
+        await SwallowAsync(run);
+    }
+
+    [Fact]
+    public async Task Receive_DeltaCoalescing_DoesNotSwallowTheMessageThatEndsTheRun()
+    {
+        // The risk the one-message lookahead exists to handle: coalescing reads ahead, and whatever
+        // it reads that is NOT a delta must still be processed, not dropped on the floor.
+        var db = await NewOrdersDbAsync();
+        var options = new ReplicationOptions { FlushInterval = TimeSpan.FromSeconds(30) };
+        var session = NewSession(db, options);
+        var captured = new CapturingSnapshotApplier();
+        session.SnapshotApplier = captured;
+        var (duplex, toSession, fromSession) = ScriptedPeer();
+
+        using var cts = new CancellationTokenSource(RunTimeout);
+        var run = session.RunAsync(duplex, cts.Token);
+
+        await PreloadHandshakeAsync(db, toSession, cts.Token);
+        var batch = new DeltaBatch();
+        batch.Entries.Add(ProtoEntry(1, 1, 1_000_001));
+        await toSession.WriteAsync(new SyncMessage { DeltaBatch = batch }, cts.Token);
+        await toSession.WriteAsync(
+            new SyncMessage { SnapshotBegin = new SnapshotBegin { AsOfSeq = 77 } }, cts.Token);
+
+        SyncMessage ack;
+        do { ack = await NextAsync(run, fromSession, cts.Token); }
+        while (ack.MsgCase != SyncMessage.MsgOneofCase.DeltaAck);
+        Assert.Equal(1, ack.DeltaAck.AppliedThruSeq);
+
+        // The SnapshotBegin that terminated the coalescing run still reached its handler.
+        await WaitForAsync(() => captured.BeginAsOfSeq == 77, RunTimeout);
+
+        cts.Cancel();
+        await SwallowAsync(run);
+    }
+
+    [Fact]
+    public async Task Receive_CoalescesQueuedAcks_ToTheHighestWatermark_WithoutLosingTheNextMessage()
+    {
+        var db = await NewOrdersDbAsync();
+        for (var i = 1; i <= 9; i++)
+            await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (@id, 'S', @id)", new { id = i });
+
+        var options = new ReplicationOptions { FlushInterval = TimeSpan.FromSeconds(30) };
+        var session = NewSession(db, options);
+        var captured = new CapturingSnapshotApplier();
+        session.SnapshotApplier = captured;
+        var (duplex, toSession, fromSession) = ScriptedPeer();
+
+        using var cts = new CancellationTokenSource(RunTimeout);
+        var run = session.RunAsync(duplex, cts.Token);
+
+        await PreloadHandshakeAsync(db, toSession, cts.Token);
+        // Out of order on purpose: collapsing acks must take the MAX, never the last.
+        foreach (var seq in new long[] { 4, 9, 2 })
+            await toSession.WriteAsync(
+                new SyncMessage { DeltaAck = new DeltaAck { AppliedThruSeq = seq } }, cts.Token);
+        await toSession.WriteAsync(
+            new SyncMessage { SnapshotBegin = new SnapshotBegin { AsOfSeq = 55 } }, cts.Token);
+
+        await WaitForAsync(async () =>
+        {
+            var acked = await db.QueryAsync(
+                "SELECT last_acked_seq FROM __localdb_peer_state WHERE id = 1", r => r.GetInt64(0));
+            return acked[0] == 9;
+        }, RunTimeout);
+
+        Assert.Equal(9, session.PeerAckedSeq);
+        await WaitForAsync(() => captured.BeginAsOfSeq == 55, RunTimeout);
+
+        cts.Cancel();
+        await SwallowAsync(run);
+    }
+
+    private sealed class CapturingSnapshotApplier : ISnapshotApplier
+    {
+        private long _beginAsOfSeq = -1;
+
+        public long BeginAsOfSeq => Interlocked.Read(ref _beginAsOfSeq);
+
+        public Task OnBeginAsync(SnapshotBegin begin, CancellationToken ct)
+        {
+            Interlocked.Exchange(ref _beginAsOfSeq, begin.AsOfSeq);
+            return Task.CompletedTask;
+        }
+
+        public Task OnBatchAsync(SnapshotBatch batch, CancellationToken ct) => Task.CompletedTask;
+
+        public Task OnCompleteAsync(SnapshotComplete complete, CancellationToken ct) => Task.CompletedTask;
+    }
+
+    private static Task WaitForAsync(Func<bool> predicate, TimeSpan timeout) =>
+        WaitForAsync(() => Task.FromResult(predicate()), timeout);
+
+    private static async Task WaitForAsync(Func<Task<bool>> predicate, TimeSpan timeout)
+    {
+        var deadline = DateTime.UtcNow + timeout;
+        while (DateTime.UtcNow < deadline)
+        {
+            if (await predicate()) return;
+            await Task.Delay(20);
+        }
+        throw new TimeoutException("Condition not reached within " + timeout);
+    }
+}

ZB.MOM.WW.LocalDb/tests/ZB.MOM.WW.LocalDb.Tests/GrpcAdapterTests.cs

@@ -70,8 +70,12 @@ public sealed class GrpcAdapterTests : IAsyncLifetime
     }
 
     [Fact]
-    public async Task SecondConcurrentStream_Rejected()
+    public async Task SecondConcurrentStream_DisplacesTheIncumbent()
     {
+        // Newest wins (0.3.2). The guard serializes sessions; it was never a fence — the auth
+        // interceptor is, and an authenticated same-key dialler IS the peer. Refusing the newcomer
+        // bought nothing and cost the pair its only route back from a half-open link, where the
+        // incumbent handler can hold the guard indefinitely on a dead stream.
         var serverHost = await BuildServerAsync(NewDbPath(), apiKey: null, sink: null);
         var serverDb = serverHost.Services.GetRequiredService<ILocalDb>();
         var handler = serverHost.GetTestServer().CreateHandler();
@@ -85,11 +89,15 @@ public sealed class GrpcAdapterTests : IAsyncLifetime
 
         // Convergence proves the first stream's server handler is active and holding the single-stream guard.
         await WaitForAsync(async () => (await ReadOrders(serverDb)).Count == 1, RunTimeout);
+        var attemptsBefore = bg.ConnectionAttempts;
 
         var client = new LocalDbSync.LocalDbSyncClient(ChannelOver(handler));
         using var call2 = client.Sync(cancellationToken: cts.Token);
-        var ex = await Assert.ThrowsAsync<RpcException>(() => call2.ResponseStream.MoveNext(cts.Token));
-        Assert.Equal(StatusCode.AlreadyExists, ex.StatusCode);
+        Assert.True(await call2.ResponseStream.MoveNext(cts.Token));
+        Assert.Equal(SyncMessage.MsgOneofCase.Handshake, call2.ResponseStream.Current.MsgCase);
+
+        // The incumbent really ended: the displaced client notices and dials again.
+        await WaitForAsync(() => Task.FromResult(bg.ConnectionAttempts > attemptsBefore), RunTimeout);
 
         await bg.StopAsync(CancellationToken.None);
     }
@@ -173,6 +181,71 @@ public sealed class GrpcAdapterTests : IAsyncLifetime
         Assert.True(sw.Elapsed < TimeSpan.FromSeconds(3), $"teardown took {sw.Elapsed}");
     }
 
+    // ---- inbox backpressure ------------------------------------------------------------------
+
+    [Fact]
+    public async Task Inbox_SlowReader_StopsDrainingTheStream_AndLosesNothing()
+    {
+        // The pump must STOP pulling from the gRPC stream once the inbox is full — that stall is
+        // what closes the peer's flow-control window and bounds this process's memory at
+        // InboxCapacity x MaxBatchBytes instead of "whatever the peer decides to send" (a full
+        // snapshot of a large dataset, on both nodes at once, when a pair baselines late).
+        const int capacity = Replication.Internal.GrpcSyncDuplex.InboxCapacity;
+        using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(15));
+        var reader = new CountingReader();
+        var (duplex, readerTask) = Replication.Internal.GrpcSyncDuplex.Create(
+            reader, new HangingWriter(), cts.Token);
+
+        // Nothing reads the inbox: the pump fills it and blocks.
+        await WaitForAsync(() => Task.FromResult(reader.Produced >= capacity), TimeSpan.FromSeconds(5));
+        await Task.Delay(200, cts.Token);
+        // capacity in the channel + at most one in the pump's hand, blocked on WriteAsync.
+        Assert.True(reader.Produced <= capacity + 1,
+            $"the reader pump kept draining an already-full inbox: {reader.Produced} messages read for a {capacity}-message bound");
+
+        // Draining releases it, and every message arrives exactly once in stream order — TryWrite
+        // on a full bounded channel would have dropped them silently instead.
+        for (long expected = 1; expected <= capacity + 1; expected++)
+        {
+            var msg = await ReadInboxAsync(duplex, cts.Token);
+            Assert.Equal(expected, msg.DeltaAck.AppliedThruSeq);
+        }
+        await WaitForAsync(() => Task.FromResult(reader.Produced > capacity + 1), TimeSpan.FromSeconds(5));
+
+        cts.Cancel();
+        try { await readerTask.WaitAsync(TimeSpan.FromSeconds(5)); }
+        catch (OperationCanceledException) { }
+    }
+
+    private static async Task<SyncMessage> ReadInboxAsync(
+        Replication.Internal.SyncDuplex duplex, CancellationToken ct)
+    {
+        await duplex.Inbox.WaitToReadAsync(ct);
+        duplex.Inbox.TryRead(out var message);
+        return message!;
+    }
+
+    // Produces messages as fast as the inbox will take them, numbering each one so the consumer can
+    // prove nothing was dropped or reordered.
+    private sealed class CountingReader : IAsyncStreamReader<SyncMessage>
+    {
+        private long _produced;
+
+        public long Produced => Interlocked.Read(ref _produced);
+
+        public SyncMessage Current { get; private set; } = new();
+
+        public Task<bool> MoveNext(CancellationToken cancellationToken)
+        {
+            cancellationToken.ThrowIfCancellationRequested();
+            Current = new SyncMessage
+            {
+                DeltaAck = new DeltaAck { AppliedThruSeq = Interlocked.Increment(ref _produced) },
+            };
+            return Task.FromResult(true);
+        }
+    }
+
     // ---- validator -------------------------------------------------------------------------
 
     [Fact]
@@ -197,9 +270,13 @@ public sealed class GrpcAdapterTests : IAsyncLifetime
             TombstoneRetention = TimeSpan.Zero,
             ReconnectBackoffMax = TimeSpan.Zero,
             MaxHlcDriftAhead = TimeSpan.Zero,
+            KeepAlivePingDelay = TimeSpan.Zero,
+            KeepAlivePingTimeout = TimeSpan.Zero,
         });
 
         Assert.True(result.Failed);
+        Assert.Contains(result.Failures, f => f.Contains("KeepAlivePingDelay", StringComparison.Ordinal));
+        Assert.Contains(result.Failures, f => f.Contains("KeepAlivePingTimeout", StringComparison.Ordinal));
         Assert.Contains(result.Failures, f => f.Contains("FlushInterval", StringComparison.Ordinal));
         Assert.Contains(result.Failures, f => f.Contains("MaxBatchSize", StringComparison.Ordinal));
         Assert.Contains(result.Failures, f => f.Contains("MaxOplogRows", StringComparison.Ordinal));

ZB.MOM.WW.LocalDb/tests/ZB.MOM.WW.LocalDb.Tests/LocalDbSchemaTests.cs

@@ -51,12 +51,118 @@ public sealed class LocalDbSchemaTests : IDisposable
         foreach (var expected in new[]
                  {
                      "__localdb_meta", "__localdb_oplog", "__localdb_peer_state",
-                     "__localdb_row_version", "__localdb_applying", "__localdb_dead_letter"
+                     "__localdb_row_version", "__localdb_applying", "__localdb_dead_letter",
+                     "__localdb_snapshot_state",
                  })
             Assert.Contains(expected, tables);
 
+        // __localdb_oplog_hlc is deliberately NOT created: schema v2 removed it. It indexed
+        // (table_name, pk_json, hlc), which no query has ever used — every oplog access path is by
+        // seq or a full hlc-predicate delete — while costing a b-tree insert inside every captured
+        // consumer write.
+        Assert.Equal(0, ScalarLong(conn,
+            "SELECT count(*) FROM sqlite_master WHERE type='index' AND name='__localdb_oplog_hlc'"));
+        Assert.Equal(LocalDbSchema.CurrentSchemaVersion,
+            ScalarLong(conn, "SELECT schema_version FROM __localdb_meta WHERE id=1"));
+    }
+
+    [Fact]
+    public void EnsureCreated_UpgradesLegacyFile_DropsOplogHlcIndex()
+    {
+        using var conn = Open();
+
+        // A file exactly as schema v1 left it: the tables, the index, and a v1 stamp.
+        LocalDbSchema.EnsureCreated(conn);
+        Exec(conn, "CREATE INDEX __localdb_oplog_hlc ON __localdb_oplog(table_name, pk_json, hlc);" +
+                   "UPDATE __localdb_meta SET schema_version = 1 WHERE id = 1;");
         Assert.Equal(1, ScalarLong(conn,
             "SELECT count(*) FROM sqlite_master WHERE type='index' AND name='__localdb_oplog_hlc'"));
+
+        LocalDbSchema.EnsureCreated(conn);
+
+        Assert.Equal(0, ScalarLong(conn,
+            "SELECT count(*) FROM sqlite_master WHERE type='index' AND name='__localdb_oplog_hlc'"));
+        Assert.Equal(LocalDbSchema.CurrentSchemaVersion,
+            ScalarLong(conn, "SELECT schema_version FROM __localdb_meta WHERE id=1"));
+    }
+
+    [Fact]
+    public void EnsureCreated_UpgradesV2File_AddsPerTableSnapshotState()
+    {
+        using var conn = Open();
+
+        // A file exactly as schema v2 left it: everything except __localdb_snapshot_state, stamped 2.
+        LocalDbSchema.EnsureCreated(conn);
+        Exec(conn, "DROP TABLE __localdb_snapshot_state;" +
+                   "UPDATE __localdb_meta SET schema_version = 2 WHERE id = 1;");
+
+        LocalDbSchema.EnsureCreated(conn);
+
+        Assert.Equal(1, ScalarLong(conn,
+            "SELECT count(*) FROM sqlite_master WHERE type='table' AND name='__localdb_snapshot_state'"));
+        // Empty is the right v3 state for a file that never baselined anything: no table owes a
+        // snapshot, and the database-wide flag it was carrying keeps whatever value it had.
+        Assert.Equal(0, ScalarLong(conn, "SELECT COUNT(*) FROM __localdb_snapshot_state"));
+        Assert.Equal(LocalDbSchema.CurrentSchemaVersion,
+            ScalarLong(conn, "SELECT schema_version FROM __localdb_meta WHERE id=1"));
+    }
+
+    [Fact]
+    public void EnsureCreated_Upgrade_PreservesTheDatabaseWideSnapshotFlag()
+    {
+        // A v2 file mid-resync (cap prune flagged a full snapshot, the peer was down) must still owe
+        // that snapshot after the upgrade. v3 narrows where NEW debt is recorded; it does not
+        // reinterpret debt already on disk.
+        using var conn = Open();
+        LocalDbSchema.EnsureCreated(conn);
+        Exec(conn, "DROP TABLE __localdb_snapshot_state;" +
+                   "UPDATE __localdb_meta SET schema_version = 2 WHERE id = 1;" +
+                   "UPDATE __localdb_peer_state SET needs_snapshot = 1 WHERE id = 1;");
+
+        LocalDbSchema.EnsureCreated(conn);
+
+        Assert.Equal(1, ScalarLong(conn, "SELECT needs_snapshot FROM __localdb_peer_state WHERE id=1"));
+    }
+
+    [Fact]
+    public void EnsureCreated_Upgrade_PreservesNodeIdentityAndOplogRows()
+    {
+        using var conn = Open();
+        LocalDbSchema.EnsureCreated(conn);
+        var nodeId = ScalarString(conn, "SELECT node_id FROM __localdb_meta WHERE id=1");
+        Exec(conn,
+            "CREATE INDEX __localdb_oplog_hlc ON __localdb_oplog(table_name, pk_json, hlc);" +
+            "UPDATE __localdb_meta SET schema_version = 1 WHERE id = 1;" +
+            "INSERT INTO __localdb_oplog (table_name, pk_json, row_json, hlc, node_id, is_tombstone) " +
+            "VALUES ('orders', '{\"id\":1}', '{\"id\":1}', 42, 'n', 0);" +
+            "UPDATE __localdb_peer_state SET last_acked_seq = 7 WHERE id = 1;");
+
+        LocalDbSchema.EnsureCreated(conn);
+
+        // The upgrade is DDL only — identity, replication watermarks and captured data survive it.
+        Assert.Equal(nodeId, ScalarString(conn, "SELECT node_id FROM __localdb_meta WHERE id=1"));
+        Assert.Equal(1, ScalarLong(conn, "SELECT COUNT(*) FROM __localdb_oplog"));
+        Assert.Equal(42, ScalarLong(conn, "SELECT hlc FROM __localdb_oplog"));
+        Assert.Equal(7, ScalarLong(conn, "SELECT last_acked_seq FROM __localdb_peer_state WHERE id=1"));
+    }
+
+    [Fact]
+    public void EnsureCreated_AlreadyCurrent_LeavesVersionAlone()
+    {
+        using var conn = Open();
+        LocalDbSchema.EnsureCreated(conn);
+        LocalDbSchema.EnsureCreated(conn);
+        LocalDbSchema.EnsureCreated(conn);
+
+        Assert.Equal(LocalDbSchema.CurrentSchemaVersion,
+            ScalarLong(conn, "SELECT schema_version FROM __localdb_meta WHERE id=1"));
+    }
+
+    private static void Exec(SqliteConnection conn, string sql)
+    {
+        using var cmd = conn.CreateCommand();
+        cmd.CommandText = sql;
+        cmd.ExecuteNonQuery();
     }
 
     [Fact]

ZB.MOM.WW.LocalDb/tests/ZB.MOM.WW.LocalDb.Tests/LwwApplierTests.cs

@@ -404,6 +404,180 @@ public sealed class LwwApplierTests : IDisposable
         Assert.Equal(("NEW", (long?)7), await ReadOrder(db, 2));
     }
 
+    [Fact]
+    public async Task Apply_PreparesEachStatementOnce_NotOncePerRow()
+    {
+        // The regression this guards: the old shape went through the generic transaction façade,
+        // which builds a fresh SqliteCommand — and therefore re-parses the SQL — per call. Three
+        // statements per applied row means a 200-row batch parsed 600 statements to run 6 distinct
+        // ones. Statement count must track distinct SQL, not rows.
+        using var db = await NewOrdersDb();
+        await db.ExecuteAsync("CREATE TABLE widgets (id INTEGER PRIMARY KEY, name TEXT)");
+        db.RegisterReplicated("widgets");
+
+        var entries = new List<OplogEntryRecord>();
+        for (var i = 1; i <= 200; i++)
+            entries.Add(Entry(i, $"{{\"id\":{i}}}", $"{{\"id\":{i},\"sku\":\"S\",\"qty\":{i}}}", 1_000_000 + i, "remote"));
+        // A second table and a tombstone, so every statement kind in the set is exercised.
+        entries.Add(new OplogEntryRecord(
+            201, "widgets", "{\"id\":1}", "{\"id\":1,\"name\":\"W\"}", 1_000_300, "remote", false));
+        entries.Add(Entry(202, "{\"id\":1}", null, 1_000_400, "remote", tombstone: true));
+
+        var applier = new LwwApplier(db);
+        var result = await applier.ApplyBatchAsync(entries, default);
+
+        Assert.Equal(202, result.Applied);
+        // Six table-independent statements (applying guard, version lookup, row-version write,
+        // peer state, and the per-entry savepoint's enter/release), plus an upsert per table
+        // touched and a delete for the one table tombstoned. The savepoint's rollback statement is
+        // built on first use like the rest, and nothing in this batch is poison.
+        Assert.Equal(9, applier.LastBatchStatementCount);
+
+        // ...and the batch is still correct, statement reuse notwithstanding.
+        Assert.Null(await ReadOrder(db, 1));
+        Assert.Equal(("S", (long?)200), await ReadOrder(db, 200));
+        var widget = await db.QueryAsync("SELECT name FROM widgets WHERE id = 1", x => x.GetString(0));
+        Assert.Equal("W", widget[0]);
+    }
+
+    [Fact]
+    public async Task Apply_ReusedStatements_SurviveAPoisonRowMidBatch()
+    {
+        // A statement that faults is reset and reused for the following rows; if it were not, one
+        // poison row would silently corrupt every row after it in the batch.
+        using var db = await NewOrdersDb();
+
+        var applier = new LwwApplier(db);
+        var result = await applier.ApplyBatchAsync(
+        [
+            Entry(1, "{\"id\":1}", "{\"id\":1,\"sku\":\"BEFORE\",\"qty\":1}", 1_000_000, "remote"),
+            Entry(2, "{\"id\":2}", "NOT VALID JSON", 1_000_001, "remote"),
+            Entry(3, "{\"id\":3}", "{\"id\":3,\"sku\":\"AFTER\",\"qty\":3}", 1_000_002, "remote"),
+            Entry(4, "{\"id\":4}", "ALSO NOT JSON", 1_000_003, "remote"),
+            Entry(5, "{\"id\":5}", "{\"id\":5,\"sku\":\"LAST\",\"qty\":5}", 1_000_004, "remote"),
+        ], default);
+
+        Assert.Equal(3, result.Applied);
+        Assert.Equal(2, result.DeadLettered);
+        Assert.Equal(("BEFORE", (long?)1), await ReadOrder(db, 1));
+        Assert.Equal(("AFTER", (long?)3), await ReadOrder(db, 3));
+        Assert.Equal(("LAST", (long?)5), await ReadOrder(db, 5));
+    }
+
+    /// <summary>
+    /// Arms a BEFORE INSERT trigger on the row-version ledger that aborts the write for one key —
+    /// a genuine statement-level SQLite fault (SQLITE_CONSTRAINT), landing exactly between an
+    /// entry's table write and its ledger write, with no production test seam.
+    /// </summary>
+    private static Task PoisonRowVersionWriteFor(SqliteLocalDb db, string pkJson) =>
+        db.ExecuteAsync(
+            "CREATE TRIGGER poison_rv BEFORE INSERT ON __localdb_row_version " +
+            $"WHEN NEW.pk_json = '{pkJson}' BEGIN SELECT RAISE(ABORT, 'ledger write failed'); END");
+
+    [Fact]
+    public async Task Apply_TombstoneWhoseLedgerWriteFails_LeavesRowAndLedgerUntouched()
+    {
+        // The hazard: DELETE and the ledger write are two statements, and SQLite aborts only the
+        // failing one. Without a per-entry savepoint the row is gone while __localdb_row_version
+        // still calls it live — SnapshotStreamer can then only warn and skip it, so the peer never
+        // learns the row died and the pair diverges. Both or neither.
+        using var db = await NewOrdersDb();
+        await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1, 'LOCAL', 5)");
+        var localHlc = await RowVersionHlc(db, Pk1);
+        await PoisonRowVersionWriteFor(db, Pk1);
+
+        var result = await new LwwApplier(db).ApplyBatchAsync(
+            [Entry(90, Pk1, null, localHlc + 1000, "remote-node", tombstone: true)], default);
+
+        Assert.Equal(0, result.Applied);
+        Assert.Equal(1, result.DeadLettered);
+        Assert.Equal(("LOCAL", (long?)5), await ReadOrder(db, 1));
+        var rv = await db.QueryAsync(
+            "SELECT hlc, is_tombstone FROM __localdb_row_version WHERE table_name = 'orders' AND pk_json = @pk",
+            x => (Hlc: x.GetInt64(0), Tomb: x.GetInt64(1)), new { pk = Pk1 });
+        Assert.Equal(localHlc, rv[0].Hlc);
+        Assert.Equal(0, rv[0].Tomb);
+    }
+
+    [Fact]
+    public async Task Apply_UpsertWhoseLedgerWriteFails_LeavesRowAndLedgerUntouched()
+    {
+        // Same split, other direction: the new row would sit in the table carrying the OLD version,
+        // so it would lose the very next LWW comparison it should win.
+        using var db = await NewOrdersDb();
+        await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1, 'LOCAL', 5)");
+        var localHlc = await RowVersionHlc(db, Pk1);
+        await PoisonRowVersionWriteFor(db, Pk1);
+
+        var result = await new LwwApplier(db).ApplyBatchAsync(
+            [Entry(91, Pk1, "{\"id\":1,\"sku\":\"REMOTE\",\"qty\":99}", localHlc + 1000, "remote-node")], default);
+
+        Assert.Equal(0, result.Applied);
+        Assert.Equal(1, result.DeadLettered);
+        Assert.Equal(("LOCAL", (long?)5), await ReadOrder(db, 1));
+        Assert.Equal(localHlc, await RowVersionHlc(db, Pk1));
+    }
+
+    [Fact]
+    public async Task Apply_LedgerWriteFailureMidBatch_DoesNotUndoTheRestOfTheBatch()
+    {
+        // The savepoint must rewind ONE entry, not the batch: entries either side of the poison
+        // still apply, and the batch still commits.
+        using var db = await NewOrdersDb();
+        await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1, 'LOCAL', 5)");
+        var localHlc = await RowVersionHlc(db, Pk1);
+        await PoisonRowVersionWriteFor(db, Pk1);
+
+        var result = await new LwwApplier(db).ApplyBatchAsync(
+        [
+            Entry(92, "{\"id\":2}", "{\"id\":2,\"sku\":\"BEFORE\",\"qty\":2}", localHlc + 1000, "remote-node"),
+            Entry(93, Pk1, null, localHlc + 1001, "remote-node", tombstone: true),
+            Entry(94, "{\"id\":3}", "{\"id\":3,\"sku\":\"AFTER\",\"qty\":3}", localHlc + 1002, "remote-node"),
+        ], default);
+
+        Assert.Equal(2, result.Applied);
+        Assert.Equal(1, result.DeadLettered);
+        Assert.Equal(94, result.AppliedThruSeq);
+        Assert.Equal(("BEFORE", (long?)2), await ReadOrder(db, 2));
+        Assert.Equal(("AFTER", (long?)3), await ReadOrder(db, 3));
+        Assert.Equal(("LOCAL", (long?)5), await ReadOrder(db, 1));
+    }
+
+    [Fact]
+    public async Task Apply_DeadLetterFailure_ReportsBothCausesAndCommitsNothing()
+    {
+        // Two faults stacked: the entry is poison (its ledger write aborts) AND recovering from it
+        // fails (the dead-letter INSERT aborts too). Reporting only the second would discard the
+        // one that says WHY the entry was poison, leaving an operator with "dead-letter failed"
+        // and no cause.
+        using var db = await NewOrdersDb();
+        await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1, 'LOCAL', 5)");
+        var localHlc = await RowVersionHlc(db, Pk1);
+        await PoisonRowVersionWriteFor(db, Pk1);
+        await db.ExecuteAsync(
+            "CREATE TRIGGER poison_dl BEFORE INSERT ON __localdb_dead_letter " +
+            "BEGIN SELECT RAISE(ABORT, 'dead-letter write failed'); END");
+
+        var ex = await Assert.ThrowsAsync<AggregateException>(() => new LwwApplier(db).ApplyBatchAsync(
+        [
+            Entry(95, "{\"id\":2}", "{\"id\":2,\"sku\":\"BEFORE\",\"qty\":2}", localHlc + 1000, "remote-node"),
+            Entry(96, Pk1, null, localHlc + 1001, "remote-node", tombstone: true),
+        ], default));
+
+        // The entry's own fault first (InnerException), the recovery fault carried alongside it.
+        Assert.Contains("ledger write failed", ex.InnerException!.Message);
+        Assert.Contains("dead-letter write failed", ex.InnerExceptions[1].Message);
+        Assert.Contains("orders", ex.Message);
+        Assert.Contains("96", ex.Message);
+
+        // Fail-closed: the batch never commits, so even the entry that applied cleanly is gone.
+        Assert.Null(await ReadOrder(db, 2));
+        Assert.Equal(("LOCAL", (long?)5), await ReadOrder(db, 1));
+        var watermark = await db.QueryAsync(
+            "SELECT last_applied_remote_seq FROM __localdb_peer_state WHERE id = 1", x => x.GetInt64(0));
+        Assert.Equal(0, watermark[0]);
+    }
+
     [Fact]
     public async Task Apply_ReplaceExistingRow_DoesNotFireCaptureTriggers()
     {

ZB.MOM.WW.LocalDb/tests/ZB.MOM.WW.LocalDb.Tests/OplogStoreTests.cs

@@ -212,6 +212,137 @@ public sealed class OplogStoreTests : IDisposable
         Assert.Equal("{\"id\":2}", remaining[0].PkJson);
     }
 
+    // ---- O(1) backlog depth --------------------------------------------------------------
+
+    // The definition the O(1) formula has to reproduce. Every scenario below is asserted against
+    // this, not against a hand-computed number, so the two can never drift apart.
+    private static async Task<long> CountingDepth(SqliteLocalDb db) =>
+        (await db.QueryAsync(
+            "SELECT COUNT(*) FROM __localdb_oplog WHERE seq > (SELECT last_acked_seq FROM __localdb_peer_state WHERE id = 1)",
+            r => r.GetInt64(0)))[0];
+
+    private static async Task AssertDepthMatchesCount(SqliteLocalDb db, OplogStore store)
+    {
+        var expected = await CountingDepth(db);
+        Assert.Equal(expected, await store.GetOplogDepthAsync(default));
+        Assert.Equal(expected, store.GetOplogDepthSync());
+    }
+
+    [Fact]
+    public async Task OplogDepth_Empty_IsZero()
+    {
+        using var db = await NewOrdersDb();
+        var store = new OplogStore(db, new ReplicationOptions());
+
+        await AssertDepthMatchesCount(db, store);
+        Assert.Equal(0, await store.GetOplogDepthAsync(default));
+    }
+
+    [Fact]
+    public async Task OplogDepth_AllAcked_EmptiesToZero_NotToTheSeqRange()
+    {
+        using var db = await NewOrdersDb();
+        await WriteRows(db, 5);
+        var store = new OplogStore(db, new ReplicationOptions());
+        await store.RecordPeerAckAsync(await store.GetMaxSeqAsync(default), default);
+
+        await AssertDepthMatchesCount(db, store);
+        Assert.Equal(0, await store.GetOplogDepthAsync(default));
+    }
+
+    [Fact]
+    public async Task OplogDepth_AfterCapPrune_ReportsRetainedRows_NotTheWholeSeqRange()
+    {
+        // The reason the formula clamps its floor to MIN(seq) - 1. A cap prune leaves the retained
+        // rows far above an ack watermark that never moved, so a naive MAX(seq) - last_acked would
+        // report the entire seq range forever and re-flag the cap on every maintenance tick.
+        using var db = await NewOrdersDb();
+        await WriteRows(db, 40);
+        var store = new OplogStore(db, new ReplicationOptions { MaxOplogRows = 5 });
+
+        Assert.True(await store.EnforceCapsAsync(default));
+
+        await AssertDepthMatchesCount(db, store);
+        Assert.Equal(5, await store.GetOplogDepthAsync(default));
+        // And the next tick is clean: depth is not over cap any more.
+        Assert.False(await store.EnforceCapsAsync(default));
+    }
+
+    [Fact]
+    public async Task OplogDepth_AfterAgePrune_MatchesCount()
+    {
+        using var db = await NewOrdersDb();
+        await WriteRows(db, 6);
+        // Backdate the oldest three past the age cap; HLCs rise with seq, so this is a prefix.
+        var expired = DateTimeOffset.UtcNow.AddDays(-30).ToUnixTimeMilliseconds() << 16;
+        await db.ExecuteAsync(
+            "UPDATE __localdb_oplog SET hlc = @h WHERE seq IN (SELECT seq FROM __localdb_oplog ORDER BY seq LIMIT 3)",
+            new { h = expired });
+        var store = new OplogStore(db, new ReplicationOptions());
+
+        Assert.True(await store.EnforceCapsAsync(default));
+
+        await AssertDepthMatchesCount(db, store);
+        Assert.Equal(3, await store.GetOplogDepthAsync(default));
+    }
+
+    [Fact]
+    public async Task OplogDepth_PartialAck_MatchesCount()
+    {
+        using var db = await NewOrdersDb();
+        await WriteRows(db, 9);
+        var store = new OplogStore(db, new ReplicationOptions());
+        var batch = await store.ReadBatchAboveAsync(0, 100, default);
+
+        await store.RecordPeerAckAsync(batch[3].Seq, default);
+
+        await AssertDepthMatchesCount(db, store);
+        Assert.Equal(5, await store.GetOplogDepthAsync(default));
+    }
+
+    [Fact]
+    public async Task OplogDepth_AckPruneThenMoreWrites_MatchesCount()
+    {
+        using var db = await NewOrdersDb();
+        await WriteRows(db, 4);
+        var store = new OplogStore(db, new ReplicationOptions());
+        await store.RecordPeerAckAsync(await store.GetMaxSeqAsync(default), default);
+
+        // The steady state of a converged pair: oplog pruned empty, watermark high, new writes above it.
+        await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (91,'S',1),(92,'S',2)");
+
+        await AssertDepthMatchesCount(db, store);
+        Assert.Equal(2, await store.GetOplogDepthAsync(default));
+    }
+
+    [Fact]
+    public async Task OplogDepth_LargeBacklog_IsConstantTime()
+    {
+        // 200k rows: a COUNT(*) over the backlog runs ~200k VM steps per call (measured: 100,020 at
+        // 50k rows), the seek-based formula runs 58 regardless of size. The loop below costs tens of
+        // milliseconds at seek shape and seconds at scan shape, so the budget separates them by a
+        // wide margin in both directions.
+        using var db = await NewOrdersDb();
+        await db.ExecuteAsync(
+            "INSERT INTO __localdb_oplog (table_name, pk_json, row_json, hlc, node_id, is_tombstone) " +
+            "WITH RECURSIVE c(x) AS (SELECT 1 UNION ALL SELECT x + 1 FROM c WHERE x < 200000) " +
+            "SELECT 'orders', '{\"id\":' || x || '}', '{\"id\":' || x || '}', x << 16, 'n', 0 FROM c");
+        var store = new OplogStore(db, new ReplicationOptions());
+
+        Assert.Equal(200_000, await store.GetOplogDepthAsync(default));
+
+        using var conn = db.CreateConnection();
+        using var cmd = conn.CreateCommand();
+        cmd.CommandText = OplogStore.DepthSql;
+        var sw = System.Diagnostics.Stopwatch.StartNew();
+        for (var i = 0; i < 2_000; i++)
+            Assert.Equal(200_000L, (long)cmd.ExecuteScalar()!);
+        sw.Stop();
+
+        Assert.True(sw.ElapsedMilliseconds < 1_500,
+            $"2000 depth reads over a 200k backlog took {sw.ElapsedMilliseconds} ms — that is scan-shaped, not seek-shaped.");
+    }
+
     [Fact]
     public async Task PeerState_RoundTrips()
     {

ZB.MOM.WW.LocalDb/tests/ZB.MOM.WW.LocalDb.Tests/RegistrationTests.cs

@@ -70,6 +70,109 @@ public sealed class RegistrationTests : IDisposable
         Assert.Contains("BLOB", ex.Message);
     }
 
+    [Fact]
+    public async Task Register_CascadeDeleteParent_Throws()
+    {
+        // The apply path writes remote rows with INSERT OR REPLACE, whose implicit delete fires
+        // ON DELETE CASCADE with capture suppressed — the child rows would die on this node only.
+        using var db = Create();
+        await CreateOrders(db);
+        await db.ExecuteAsync(
+            "CREATE TABLE order_lines (id INTEGER PRIMARY KEY, " +
+            "order_id INTEGER REFERENCES orders(id) ON DELETE CASCADE, sku TEXT)");
+
+        var ex = Assert.Throws<LocalDbRegistrationException>(() => db.RegisterReplicated("orders"));
+        Assert.Contains("orders", ex.Message);
+        Assert.Contains("order_lines.order_id", ex.Message);
+        Assert.Contains("ON DELETE CASCADE", ex.Message);
+
+        // Refused, not half-registered.
+        Assert.Empty(await Triggers(db));
+        Assert.False(db.ReplicatedTables.ContainsKey("orders"));
+    }
+
+    [Fact]
+    public async Task Register_CascadeDeleteParent_IsWhatWouldActuallyDiverge()
+    {
+        // Proves the guard is guarding something real: with the parent NOT registered, applying a
+        // replace to it silently wipes child rows that produce no oplog entry.
+        using var db = Create();
+        await CreateOrders(db);
+        await db.ExecuteAsync(
+            "CREATE TABLE order_lines (id INTEGER PRIMARY KEY, " +
+            "order_id INTEGER REFERENCES orders(id) ON DELETE CASCADE, sku TEXT)");
+        await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1, 'A', 1)");
+        await db.ExecuteAsync("INSERT INTO order_lines (id, order_id, sku) VALUES (10, 1, 'L')");
+
+        await db.ExecuteAsync("INSERT OR REPLACE INTO orders (id, sku, qty) VALUES (1, 'B', 2)");
+
+        var lines = await db.QueryAsync("SELECT COUNT(*) FROM order_lines", r => r.GetInt64(0));
+        Assert.Equal(0, lines[0]);
+    }
+
+    [Theory]
+    [InlineData("SET NULL")]
+    [InlineData("SET DEFAULT")]
+    public async Task Register_RowMutatingOnDeleteParent_Throws(string action)
+    {
+        using var db = Create();
+        await CreateOrders(db);
+        await db.ExecuteAsync(
+            "CREATE TABLE order_lines (id INTEGER PRIMARY KEY, " +
+            $"order_id INTEGER DEFAULT 0 REFERENCES orders(id) ON DELETE {action}, sku TEXT)");
+
+        var ex = Assert.Throws<LocalDbRegistrationException>(() => db.RegisterReplicated("orders"));
+        Assert.Contains($"ON DELETE {action}", ex.Message);
+    }
+
+    [Fact]
+    public async Task Register_NonMutatingOnDeleteParent_IsAllowed()
+    {
+        // RESTRICT/NO ACTION abort the statement instead of changing rows behind capture's back,
+        // which the apply path already handles as per-entry poison.
+        using var db = Create();
+        await CreateOrders(db);
+        await db.ExecuteAsync(
+            "CREATE TABLE order_lines (id INTEGER PRIMARY KEY, " +
+            "order_id INTEGER REFERENCES orders(id) ON DELETE RESTRICT, sku TEXT)");
+
+        db.RegisterReplicated("orders");
+
+        Assert.Equal(3, (await Triggers(db)).Count);
+    }
+
+    [Fact]
+    public async Task Register_SelfReferencingCascade_Throws()
+    {
+        // A table can be its own cascading parent: replacing a row would delete that row's
+        // children in the SAME table, uncaptured. The guard reads every foreign key in the
+        // database, so it does not need a special case — but the README promises this, so it is
+        // pinned.
+        using var db = Create();
+        await db.ExecuteAsync(
+            "CREATE TABLE nodes (id INTEGER PRIMARY KEY, " +
+            "parent_id INTEGER REFERENCES nodes(id) ON DELETE CASCADE, name TEXT)");
+
+        var ex = Assert.Throws<LocalDbRegistrationException>(() => db.RegisterReplicated("nodes"));
+        Assert.Contains("nodes.parent_id", ex.Message);
+        Assert.Contains("ON DELETE CASCADE", ex.Message);
+    }
+
+    [Fact]
+    public async Task Register_CascadeDeleteChild_IsAllowed()
+    {
+        // Only the PARENT side is hazardous: replacing a child row deletes no other table's rows.
+        using var db = Create();
+        await CreateOrders(db);
+        await db.ExecuteAsync(
+            "CREATE TABLE order_lines (id INTEGER PRIMARY KEY, " +
+            "order_id INTEGER REFERENCES orders(id) ON DELETE CASCADE, sku TEXT)");
+
+        var table = db.RegisterReplicated("order_lines");
+
+        Assert.Equal("order_lines", table.Name);
+    }
+
     [Fact]
     public async Task Register_IsIdempotent_RegeneratesTriggers()
     {
@@ -128,4 +231,409 @@ public sealed class RegistrationTests : IDisposable
         Assert.Equal(new[] { "id" }, table.PkColumns);
         Assert.Equal(new[] { "id", "sku", "qty" }, table.Columns.Select(c => c.Name).ToArray());
     }
+
+    // ---- Deregistration ------------------------------------------------------------------
+
+    [Fact]
+    public async Task Deregister_DropsTriggers_PrunesOplogAndLedger_LeavesDataAlone()
+    {
+        using var db = Create();
+        await CreateOrders(db);
+        db.RegisterReplicated("orders");
+        await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1,'A',1),(2,'B',2)");
+
+        var removed = db.DeregisterReplicated("orders");
+
+        Assert.True(removed);
+        Assert.Empty(await Triggers(db));
+        Assert.Equal(0, await CountAsync(db, "SELECT COUNT(*) FROM __localdb_oplog WHERE table_name='orders'"));
+        Assert.Equal(0, await CountAsync(db, "SELECT COUNT(*) FROM __localdb_row_version WHERE table_name='orders'"));
+        Assert.False(db.ReplicatedTables.ContainsKey("orders"));
+
+        // The consumer's own data is never touched — this removes CAPTURE, not rows.
+        Assert.Equal(2, await CountAsync(db, "SELECT COUNT(*) FROM orders"));
+    }
+
+    [Fact]
+    public async Task Deregister_StopsCapture()
+    {
+        using var db = Create();
+        await CreateOrders(db);
+        db.RegisterReplicated("orders");
+        db.DeregisterReplicated("orders");
+
+        await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1,'A',1)");
+        await db.ExecuteAsync("UPDATE orders SET qty = 9 WHERE id = 1");
+        await db.ExecuteAsync("DELETE FROM orders WHERE id = 1");
+
+        Assert.Equal(0, await CountAsync(db, "SELECT COUNT(*) FROM __localdb_oplog"));
+        Assert.Equal(0, await CountAsync(db, "SELECT COUNT(*) FROM __localdb_row_version"));
+    }
+
+    [Fact]
+    public async Task Deregister_IsIdempotent_SecondCallReportsNothingToDo()
+    {
+        using var db = Create();
+        await CreateOrders(db);
+        db.RegisterReplicated("orders");
+        await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1,'A',1)");
+
+        Assert.True(db.DeregisterReplicated("orders"));
+        Assert.False(db.DeregisterReplicated("orders"));
+        Assert.False(db.DeregisterReplicated("orders"));
+        Assert.Empty(await Triggers(db));
+    }
+
+    [Fact]
+    public async Task Deregister_NeverRegisteredTable_IsANoOp()
+    {
+        using var db = Create();
+        await CreateOrders(db);
+
+        Assert.False(db.DeregisterReplicated("orders"));
+        Assert.False(db.DeregisterReplicated("ghost"));
+    }
+
+    [Fact]
+    public async Task Deregister_ClearsTriggersLeftByAnEarlierProcess()
+    {
+        // The primary use: a database file registered by an OLDER build, opened by one that has
+        // decided this node does not replicate. Nothing in memory says the table was ever
+        // registered — the evidence is entirely on disk.
+        using (var first = Create())
+        {
+            await CreateOrders(first);
+            first.RegisterReplicated("orders");
+            await first.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1,'A',1)");
+        }
+        SqliteConnection.ClearAllPools();
+
+        using var second = Create();
+        Assert.Equal(3, (await Triggers(second)).Count);
+        Assert.False(second.ReplicatedTables.ContainsKey("orders"));
+
+        Assert.True(second.DeregisterReplicated("orders"));
+
+        Assert.Empty(await Triggers(second));
+        Assert.Equal(0, await CountAsync(second, "SELECT COUNT(*) FROM __localdb_oplog"));
+    }
+
+    [Fact]
+    public async Task Deregister_LeavesOtherTablesCapturing()
+    {
+        using var db = Create();
+        await CreateOrders(db);
+        await db.ExecuteAsync("CREATE TABLE widgets (id INTEGER PRIMARY KEY, name TEXT)");
+        db.RegisterReplicated("orders");
+        db.RegisterReplicated("widgets");
+        await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1,'A',1)");
+        await db.ExecuteAsync("INSERT INTO widgets (id, name) VALUES (1,'W')");
+
+        db.DeregisterReplicated("orders");
+        await db.ExecuteAsync("INSERT INTO widgets (id, name) VALUES (2,'W2')");
+
+        Assert.Equal(0, await CountAsync(db, "SELECT COUNT(*) FROM __localdb_oplog WHERE table_name='orders'"));
+        Assert.Equal(2, await CountAsync(db, "SELECT COUNT(*) FROM __localdb_oplog WHERE table_name='widgets'"));
+        Assert.True(db.ReplicatedTables.ContainsKey("widgets"));
+    }
+
+    [Fact]
+    public async Task Deregister_ThenRegisterWithBaseline_RestoresTheLedger()
+    {
+        // Pruning the ledger is only safe because re-registration can re-baseline; this is that
+        // round trip.
+        using var db = Create();
+        await CreateOrders(db);
+        db.RegisterReplicated("orders");
+        await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1,'A',1),(2,'B',2)");
+
+        db.DeregisterReplicated("orders");
+        Assert.Equal(0, await CountAsync(db, "SELECT COUNT(*) FROM __localdb_row_version"));
+
+        db.RegisterReplicated("orders", baselineExistingRows: true);
+
+        Assert.Equal(2, await CountAsync(db, "SELECT COUNT(*) FROM __localdb_row_version WHERE table_name='orders'"));
+    }
+
+    // ---- Baselining ----------------------------------------------------------------------
+
+    [Fact]
+    public async Task Register_WithoutBaseline_LeavesPreexistingRowsUnledgered()
+    {
+        // The documented CDC contract, unchanged: opting in is what changes behaviour, not upgrading.
+        using var db = Create();
+        await CreateOrders(db);
+        await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1,'A',1)");
+
+        db.RegisterReplicated("orders");
+
+        Assert.Equal(0, await CountAsync(db, "SELECT COUNT(*) FROM __localdb_row_version"));
+        Assert.False(await NeedsSnapshot(db));
+    }
+
+    [Fact]
+    public async Task Register_WithBaseline_SeedsAtHlcZero_AndFlagsSnapshot()
+    {
+        using var db = Create();
+        await CreateOrders(db);
+        await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1,'A',1),(2,'B',2)");
+
+        db.RegisterReplicated("orders", baselineExistingRows: true);
+
+        var seeded = await db.QueryAsync(
+            "SELECT pk_json, hlc, node_id, is_tombstone FROM __localdb_row_version ORDER BY pk_json",
+            r => (Pk: r.GetString(0), Hlc: r.GetInt64(1), Node: r.GetString(2), Tomb: r.GetInt64(3)));
+
+        Assert.Equal(2, seeded.Count);
+        Assert.Equal(["{\"id\":1}", "{\"id\":2}"], seeded.Select(s => s.Pk));
+        // HLC 0 is the floor: below every genuine stamp (physical ms << 16), so it can never beat one.
+        Assert.All(seeded, s => Assert.Equal(0, s.Hlc));
+        Assert.All(seeded, s => Assert.Equal(db.NodeId, s.Node));
+        Assert.All(seeded, s => Assert.Equal(0, s.Tomb));
+
+        // Without the flag the seeding would be inert: no gap exists at the handshake, so the peer
+        // would be told it needs no snapshot and the baselined rows would never travel.
+        Assert.True(await NeedsSnapshot(db));
+        // ...and it is flagged for THIS TABLE ONLY. Through 0.2.x this set the one database-wide
+        // bit, so baselining `orders` re-streamed every other registered table with it.
+        Assert.False(await NeedsFullSnapshot(db));
+    }
+
+    [Fact]
+    public async Task Register_WithBaseline_FlagsOnlyTheBaselinedTable()
+    {
+        // The per-table flag's reason to exist: opting one table into replication late must not
+        // schedule a full re-stream of its unrelated neighbours.
+        using var db = Create();
+        await CreateOrders(db);
+        await db.ExecuteAsync("CREATE TABLE widgets (id INTEGER PRIMARY KEY, name TEXT)");
+        await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1,'A',1)");
+        await db.ExecuteAsync("INSERT INTO widgets (id, name) VALUES (1,'W')");
+
+        db.RegisterReplicated("widgets");
+        db.RegisterReplicated("orders", baselineExistingRows: true);
+
+        Assert.True(await NeedsSnapshot(db, "orders"));
+        Assert.False(await NeedsSnapshot(db, "widgets"));
+        Assert.False(await NeedsFullSnapshot(db));
+    }
+
+    [Fact]
+    public async Task Deregister_DropsThePendingSnapshotFlag()
+    {
+        // A flag naming a table nothing streams any more can never clear itself — the streamer skips
+        // unregistered tables — so it would sit in the file and re-appear as unasked-for debt if the
+        // table were registered again.
+        using var db = Create();
+        await CreateOrders(db);
+        await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1,'A',1)");
+        db.RegisterReplicated("orders", baselineExistingRows: true);
+        Assert.True(await NeedsSnapshot(db));
+
+        db.DeregisterReplicated("orders");
+
+        Assert.False(await NeedsSnapshot(db));
+    }
+
+    [Fact]
+    public async Task Register_WithBaseline_IsIdempotent_AndDoesNotReflagSnapshot()
+    {
+        using var db = Create();
+        await CreateOrders(db);
+        await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1,'A',1)");
+
+        db.RegisterReplicated("orders", baselineExistingRows: true);
+        // Stand in for the snapshot having been delivered and acknowledged.
+        await db.ExecuteAsync("DELETE FROM __localdb_snapshot_state");
+
+        db.RegisterReplicated("orders", baselineExistingRows: true);
+
+        Assert.Equal(1, await CountAsync(db, "SELECT COUNT(*) FROM __localdb_row_version"));
+        // Nothing new was seeded, so no resync is owed: passing true every boot is free after the first.
+        Assert.False(await NeedsSnapshot(db));
+    }
+
+    [Fact]
+    public async Task Register_WithBaseline_NeverOverwritesACapturedVersion()
+    {
+        // The LWW-safety core: a row with a real, captured version must keep it. A floor that
+        // clobbered genuine versions would make every baselined node look newer than its peer.
+        using var db = Create();
+        await CreateOrders(db);
+        db.RegisterReplicated("orders");
+        await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1,'A',1)");
+        var captured = (await db.QueryAsync(
+            "SELECT hlc FROM __localdb_row_version WHERE pk_json='{\"id\":1}'", r => r.GetInt64(0)))[0];
+        Assert.True(captured > 0);
+
+        // A second, pre-existing row that capture never saw (inserted with triggers removed).
+        db.DeregisterReplicated("orders");
+        await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (2,'B',2)");
+        await db.ExecuteAsync(
+            "INSERT INTO __localdb_row_version (table_name, pk_json, hlc, node_id, is_tombstone) " +
+            "VALUES ('orders', '{\"id\":1}', @h, 'genuine', 0)", new { h = captured });
+
+        db.RegisterReplicated("orders", baselineExistingRows: true);
+
+        var versions = await db.QueryAsync(
+            "SELECT pk_json, hlc, node_id FROM __localdb_row_version ORDER BY pk_json",
+            r => (Pk: r.GetString(0), Hlc: r.GetInt64(1), Node: r.GetString(2)));
+        Assert.Equal(2, versions.Count);
+        Assert.Equal(captured, versions[0].Hlc);
+        Assert.Equal("genuine", versions[0].Node);
+        Assert.Equal(0, versions[1].Hlc);
+    }
+
+    [Fact]
+    public async Task Register_WithBaseline_SeedsPkJsonIdenticalToWhatCaptureWrites()
+    {
+        // A seeded key that differs by so much as a space from the capture triggers' json_object
+        // output would be a DIFFERENT ledger row, and the baseline would silently never merge.
+        using var db = Create();
+        await db.ExecuteAsync("CREATE TABLE kv (a INTEGER, b TEXT, v TEXT, PRIMARY KEY (a, b))");
+        await db.ExecuteAsync("INSERT INTO kv (a, b, v) VALUES (1, 'x', 'seeded')");
+
+        db.RegisterReplicated("kv", baselineExistingRows: true);
+        await db.ExecuteAsync("UPDATE kv SET v = 'captured' WHERE a = 1 AND b = 'x'");
+
+        // One ledger row, not two: the capture upsert landed on the seeded key.
+        var rows = await db.QueryAsync(
+            "SELECT pk_json, hlc FROM __localdb_row_version", r => (Pk: r.GetString(0), Hlc: r.GetInt64(1)));
+        Assert.Single(rows);
+        Assert.Equal("{\"a\":1,\"b\":\"x\"}", rows[0].Pk);
+        Assert.True(rows[0].Hlc > 0, "the genuine capture must have replaced the floor");
+    }
+
+    [Fact]
+    public async Task Register_WithBaseline_FloorLosesToEveryGenuineRemoteWrite()
+    {
+        // Proved through the real LWW path rather than by reasoning about the comparison: an
+        // absurdly small remote HLC of 1 still beats the baseline floor of 0.
+        using var db = Create();
+        await CreateOrders(db);
+        await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1,'BASELINE',1)");
+        db.RegisterReplicated("orders", baselineExistingRows: true);
+
+        var result = await new Replication.Internal.LwwApplier(db).ApplyBatchAsync(
+            [new Replication.Internal.OplogEntryRecord(
+                1, "orders", "{\"id\":1}", "{\"id\":1,\"sku\":\"REMOTE\",\"qty\":9}", 1, "peer", false)],
+            default);
+
+        Assert.Equal(1, result.Applied);
+        var sku = await db.QueryAsync("SELECT sku FROM orders WHERE id = 1", r => r.GetString(0));
+        Assert.Equal("REMOTE", sku[0]);
+    }
+
+    [Fact]
+    public async Task Register_WithBaseline_TieBetweenTwoBaselinedNodes_ResolvesByNodeIdOrdinal()
+    {
+        // Two nodes that baselined the same key both hold HLC 0 — genuinely unordered history. The
+        // node-id tie-break is what keeps that convergent instead of oscillating.
+        using var db = Create();
+        await CreateOrders(db);
+        await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1,'MINE',1)");
+        db.RegisterReplicated("orders", baselineExistingRows: true);
+
+        var applier = new Replication.Internal.LwwApplier(db);
+        var lower = await applier.ApplyBatchAsync(
+            [new Replication.Internal.OplogEntryRecord(
+                1, "orders", "{\"id\":1}", "{\"id\":1,\"sku\":\"LOWER\",\"qty\":9}", 0, "!" + db.NodeId, false)],
+            default);
+        Assert.Equal(1, lower.Discarded);
+
+        var higher = await applier.ApplyBatchAsync(
+            [new Replication.Internal.OplogEntryRecord(
+                2, "orders", "{\"id\":1}", "{\"id\":1,\"sku\":\"HIGHER\",\"qty\":9}", 0, db.NodeId + "z", false)],
+            default);
+        Assert.Equal(1, higher.Applied);
+
+        var sku = await db.QueryAsync("SELECT sku FROM orders WHERE id = 1", r => r.GetString(0));
+        Assert.Equal("HIGHER", sku[0]);
+    }
+
+    [Fact]
+    public async Task Register_WithBaseline_EmptyTable_SeedsNothing_FlagsNothing()
+    {
+        using var db = Create();
+        await CreateOrders(db);
+
+        db.RegisterReplicated("orders", baselineExistingRows: true);
+
+        Assert.Equal(0, await CountAsync(db, "SELECT COUNT(*) FROM __localdb_row_version"));
+        Assert.False(await NeedsSnapshot(db));
+    }
+
+    // ---- Deregistration and the HLC crash anchor ------------------------------------------
+
+    [Fact]
+    public async Task Deregister_FlushesHlcAnchor_SoAnUngracefulReopenCannotRegress()
+    {
+        // The clock recovers as MAX(hlc_high_water, oplog HLCs, ledger HLCs, last_seen_hlc), and
+        // deregistering the only registered table empties both ledgers. Without the in-transaction
+        // flush, an ungraceful exit right afterwards leaves hlc_high_water at 0 and the reopened
+        // clock restarts from wall-clock time — below stamps the peer already holds.
+        var crashed = Create();
+        try
+        {
+            await CreateOrders(crashed);
+            crashed.RegisterReplicated("orders");
+            await crashed.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1,'A',1)");
+
+            // A peer HLC an hour ahead, merged as the applier would merge it: this is what makes the
+            // clock outrun wall-clock time, and therefore what a wall-clock restart would regress.
+            var peerHlc = DateTimeOffset.UtcNow.AddHours(1).ToUnixTimeMilliseconds() << 16;
+            crashed.Clock.Observe(peerHlc);
+            var preDereg = crashed.Clock.Current;
+            Assert.True(preDereg >= peerHlc);
+
+            crashed.DeregisterReplicated("orders");
+
+            Assert.Equal(preDereg, await CountAsync(crashed, "SELECT hlc_high_water FROM __localdb_meta WHERE id = 1"));
+
+            // The evidence the clock would otherwise have recovered from is gone.
+            Assert.Equal(0, await CountAsync(crashed, "SELECT COUNT(*) FROM __localdb_oplog"));
+            Assert.Equal(0, await CountAsync(crashed, "SELECT COUNT(*) FROM __localdb_row_version"));
+            Assert.Equal(0, await CountAsync(crashed, "SELECT last_seen_hlc FROM __localdb_peer_state WHERE id = 1"));
+
+            // Ungraceful reopen: a second instance over the same file while the first was NEVER
+            // disposed, so nothing but the deregistration itself can have written the anchor.
+            using var reopened = Create();
+            Assert.True(reopened.Clock.Current >= preDereg,
+                $"recovered clock {reopened.Clock.Current} regressed below the pre-deregistration stamp {preDereg}");
+            Assert.True(reopened.Clock.Next() > preDereg);
+        }
+        finally
+        {
+            crashed.Dispose();
+        }
+    }
+
+    [Fact]
+    public async Task Deregister_HlcFlush_DoesNotCountAsWorkDone()
+    {
+        // The flush is an UPDATE on every call, including the ones that clean up nothing; it must
+        // not turn the "was there anything to do?" answer into a permanent true.
+        using var db = Create();
+        await CreateOrders(db);
+
+        Assert.False(db.DeregisterReplicated("orders"));
+        Assert.False(db.DeregisterReplicated("ghost"));
+    }
+
+    private static async Task<long> CountAsync(SqliteLocalDb db, string sql) =>
+        (await db.QueryAsync(sql, r => r.GetInt64(0)))[0];
+
+    /// <summary>Whether <paramref name="table"/> individually owes the peer a snapshot (schema v3).</summary>
+    private static async Task<bool> NeedsSnapshot(SqliteLocalDb db, string table = "orders") =>
+        (await db.QueryAsync(
+            "SELECT COUNT(*) FROM __localdb_snapshot_state WHERE table_name = @t AND needs_snapshot <> 0",
+            r => r.GetInt64(0), new { t = table }))[0] != 0;
+
+    /// <summary>
+    /// The DATABASE-WIDE flag — "every registered table". Baselining must never set it: that is the
+    /// whole point of the per-table flags, and setting it would restore the 0.2.x behaviour silently.
+    /// </summary>
+    private static async Task<bool> NeedsFullSnapshot(SqliteLocalDb db) =>
+        (await db.QueryAsync(
+            "SELECT needs_snapshot FROM __localdb_peer_state WHERE id = 1", r => r.GetInt64(0)))[0] != 0;
 }

ZB.MOM.WW.LocalDb/tests/ZB.MOM.WW.LocalDb.Tests/SnapshotResyncTests.cs

@@ -52,14 +52,14 @@ public sealed class SnapshotResyncTests : IDisposable
     // A fresh session stack over an EXISTING database. Reconnecting after a session ends needs a new
     // session (RunAsync is single-use), so any scenario with two connection episodes over one node's
     // data — a peer being replaced between them — goes through here.
-    private static Side NewSessionOver(SqliteLocalDb db, ReplicationOptions? options = null)
+    private static Side NewSessionOver(SqliteLocalDb db, ReplicationOptions? options = null, ILogger? streamerLogger = null)
     {
         options ??= new ReplicationOptions { FlushInterval = TimeSpan.FromMilliseconds(20) };
         var store = new OplogStore(db, options);
         var applier = new LwwApplier(db);
         var session = new SyncSession(db, store, applier, options, NullLogger.Instance);
 
-        var streamer = new SnapshotStreamer(db, store, options, NullLogger.Instance);
+        var streamer = new SnapshotStreamer(db, store, options, streamerLogger ?? NullLogger.Instance);
         var snapshotApplier = new SnapshotApplier(db, applier, store, NullLogger.Instance);
         session.SnapshotSender = streamer.SendAsync;
         session.SnapshotApplier = snapshotApplier;
@@ -67,10 +67,15 @@ public sealed class SnapshotResyncTests : IDisposable
         return new Side(db, store, applier, session);
     }
 
-    private static (SyncDuplex A, SyncDuplex B) DuplexPair()
+    // capacity models a transport that applies backpressure (which the gRPC duplex now is, via its
+    // bounded inbox); null is the historical unbounded pipe.
+    private static (SyncDuplex A, SyncDuplex B) DuplexPair(int? capacity = null)
     {
-        var aToB = Channel.CreateUnbounded<SyncMessage>();
-        var bToA = Channel.CreateUnbounded<SyncMessage>();
+        Channel<SyncMessage> New() => capacity is int c
+            ? Channel.CreateBounded<SyncMessage>(c)
+            : Channel.CreateUnbounded<SyncMessage>();
+        var aToB = New();
+        var bToA = New();
         var a = new SyncDuplex { Send = (m, ct) => aToB.Writer.WriteAsync(m, ct).AsTask(), Inbox = bToA.Reader };
         var b = new SyncDuplex { Send = (m, ct) => bToA.Writer.WriteAsync(m, ct).AsTask(), Inbox = aToB.Reader };
         return (a, b);
@@ -84,9 +89,19 @@ public sealed class SnapshotResyncTests : IDisposable
         return (duplex, toSession.Writer, fromSession.Reader);
     }
 
-    private static Handshake MatchingHandshake(SqliteLocalDb sessionDb, string peerNodeId, long lastAppliedRemoteSeq = 0)
+    // supportsPartialSnapshot defaults to FALSE, which is exactly what a 0.1.x/0.2.x peer puts on
+    // the wire (the field does not exist in its proto, so proto3 decodes the default). Scripted
+    // peers are therefore old peers unless a test says otherwise.
+    private static Handshake MatchingHandshake(
+        SqliteLocalDb sessionDb, string peerNodeId, long lastAppliedRemoteSeq = 0, bool supportsPartialSnapshot = false)
     {
-        var h = new Handshake { NodeId = peerNodeId, LibSchemaVersion = 1, LastAppliedRemoteSeq = lastAppliedRemoteSeq };
+        var h = new Handshake
+        {
+            NodeId = peerNodeId,
+            LibSchemaVersion = 1,
+            LastAppliedRemoteSeq = lastAppliedRemoteSeq,
+            SupportsPartialSnapshot = supportsPartialSnapshot,
+        };
         foreach (var t in sessionDb.ReplicatedTables.Values.OrderBy(t => t.Name, StringComparer.Ordinal))
             h.Tables.Add(new TableDigest { TableName = t.Name, Digest = t.Digest });
         return h;
@@ -109,14 +124,34 @@ public sealed class SnapshotResyncTests : IDisposable
 
     // Completes the two-phase handshake against a real session: the peer never requests a snapshot.
     private static async Task DriveHandshakeAsync(
-        SqliteLocalDb sessionDb, ChannelWriter<SyncMessage> toSession, ChannelReader<SyncMessage> fromSession, CancellationToken ct)
+        SqliteLocalDb sessionDb, ChannelWriter<SyncMessage> toSession, ChannelReader<SyncMessage> fromSession,
+        CancellationToken ct, bool supportsPartialSnapshot = false, long lastAppliedRemoteSeq = 0)
     {
         await NextOfCaseAsync(fromSession, SyncMessage.MsgOneofCase.Handshake, ct);
-        await toSession.WriteAsync(new SyncMessage { Handshake = MatchingHandshake(sessionDb, "peer") }, ct);
+        await toSession.WriteAsync(
+            new SyncMessage
+            {
+                Handshake = MatchingHandshake(
+                    sessionDb, "peer", lastAppliedRemoteSeq, supportsPartialSnapshot),
+            }, ct);
         await NextOfCaseAsync(fromSession, SyncMessage.MsgOneofCase.HandshakeAck, ct);
         await toSession.WriteAsync(new SyncMessage { HandshakeAck = new HandshakeAck { NodeId = "peer", SnapshotRequired = false } }, ct);
     }
 
+    // Reads outbound messages until (and including) the first of the given kind.
+    private static async Task<List<SyncMessage>> DrainThroughAsync(
+        ChannelReader<SyncMessage> reader, SyncMessage.MsgOneofCase last, CancellationToken ct)
+    {
+        var all = new List<SyncMessage>();
+        while (true)
+        {
+            var msg = await NextAsync(reader, ct);
+            all.Add(msg);
+            if (msg.MsgCase == last)
+                return all;
+        }
+    }
+
     private static async Task SwallowAsync(Task task)
     {
         try { await task; }
@@ -154,6 +189,12 @@ public sealed class SnapshotResyncTests : IDisposable
         return r[0];
     }
 
+    /// <summary>The tables individually flagged as owing the peer a snapshot (schema v3).</summary>
+    private static async Task<List<string>> PendingSnapshotTables(SqliteLocalDb db) =>
+        (await db.QueryAsync(
+            "SELECT table_name FROM __localdb_snapshot_state WHERE needs_snapshot <> 0 ORDER BY table_name",
+            x => x.GetString(0))).ToList();
+
     private static async Task<long> OplogCount(SqliteLocalDb db)
     {
         var r = await db.QueryAsync("SELECT COUNT(*) FROM __localdb_oplog", x => x.GetInt64(0));
@@ -170,6 +211,50 @@ public sealed class SnapshotResyncTests : IDisposable
             IsTombstone = false,
         };
 
+    [Fact]
+    public async Task MutualSnapshot_OverABoundedTransport_ConvergesWithoutDeadlock()
+    {
+        // BOTH sides owe a snapshot — the ordinary outcome of enabling replication late on a pair,
+        // since baselining flags needs_snapshot on whichever node baselined and both nodes do.
+        //
+        // The snapshot used to be streamed from INSIDE the handshake, before the receive loop
+        // existed, so neither side drained its inbox while sending. That exchange completed only
+        // because the inbox was unbounded and could hold the peer's ENTIRE snapshot in memory —
+        // which is the memory profile this design gave up on. Over a transport that pushes back
+        // (which the gRPC duplex now is, with its bounded inbox) the old shape deadlocks outright.
+        // Streaming the snapshot alongside an already-running receive loop is what makes both the
+        // bound and this test possible.
+        var options = new ReplicationOptions { MaxBatchSize = 2, FlushInterval = TimeSpan.FromMilliseconds(20) };
+        var a = await NewSideAsync(options);
+        var b = await NewSideAsync(options);
+
+        const int rowsPerSide = 60;
+        for (var i = 1; i <= rowsPerSide; i++)
+        {
+            await a.Db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (@id, 'A', @id)", new { id = i });
+            await b.Db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (@id, 'B', @id)", new { id = 100 + i });
+        }
+        await a.Store.SetNeedsSnapshotAsync(true, default);
+        await b.Store.SetNeedsSnapshotAsync(true, default);
+
+        // Capacity 1 each way: the tightest possible bound, so a side that stops draining its inbox
+        // to send stalls at once instead of getting away with it on buffer slack.
+        var (da, db) = DuplexPair(capacity: 1);
+        using var cts = new CancellationTokenSource(RunTimeout);
+        var runA = a.Session.RunAsync(da, cts.Token);
+        var runB = b.Session.RunAsync(db, cts.Token);
+
+        await WaitForAsync(
+            async () => (await ReadOrders(a.Db)).Count == 2 * rowsPerSide
+                && (await ReadOrders(b.Db)).Count == 2 * rowsPerSide,
+            RunTimeout);
+        Assert.Equal(await ReadOrders(a.Db), await ReadOrders(b.Db));
+
+        cts.Cancel();
+        await SwallowAsync(runA);
+        await SwallowAsync(runB);
+    }
+
     [Fact]
     public async Task WipedPeer_AfterEverythingWasAckedAndPruned_StillGetsSnapshot()
     {
@@ -206,6 +291,74 @@ public sealed class SnapshotResyncTests : IDisposable
         await SwallowAsync(runB);
     }
 
+    [Fact]
+    public async Task WipedPeer_SnapshotLiftsItsWatermark_SoTheNextReconnectSendsNoSecondSnapshot()
+    {
+        // The follow-on to the wiped-peer gap above, found on the ScadaBridge docker rig 2026-08-20
+        // (its deferred-work register row 106): the snapshot DID heal the wiped node, but it was
+        // stamped as_of_seq = MAX(seq) of an oplog that ack-pruning had emptied — i.e. 0 — so the
+        // receiver's (correctly MAX-guarded) watermark stayed at 0. On the next handshake the
+        // healthy side's empty-oplog horizon rule (streamable from last_acked_seq + 1) saw
+        // 0 < last_acked_seq as a gap again and re-streamed the whole database — on EVERY reconnect,
+        // until the healthy side happened to write a new row (127k rows per restart on the rig).
+        // A full snapshot taken now covers every delta ever pruned below last_acked_seq, so that is
+        // the floor the as-of must carry.
+        var a = await NewSideAsync();
+        for (var i = 1; i <= 3; i++)
+            await a.Db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (@id, 'A', @id)", new { id = i });
+        await a.Store.RecordPeerAckAsync(await a.Store.GetMaxSeqAsync());
+        Assert.Equal(0, await OplogCount(a.Db));
+
+        var b = await NewSideAsync();
+
+        // Episode 1: the wiped peer is healed by a full snapshot.
+        var (d1A, d1B) = DuplexPair();
+        using (var cts1 = new CancellationTokenSource(RunTimeout))
+        {
+            var run1A = a.Session.RunAsync(d1A, cts1.Token);
+            var run1B = b.Session.RunAsync(d1B, cts1.Token);
+            await WaitForAsync(async () => (await ReadOrders(b.Db)).Count == 3, RunTimeout);
+            cts1.Cancel();
+            await SwallowAsync(run1A);
+            await SwallowAsync(run1B);
+        }
+
+        // The snapshot must have lifted B's watermark to everything A ever produced (seq 3), not 0.
+        Assert.Equal(3, await LastAppliedRemoteSeq(b.Db));
+
+        // Episode 2: a routine reconnect. A writes one more row so the delta path gives us a
+        // barrier; the snapshot streamer's logger proves whether a snapshot was sent at all.
+        var streamerLog = new CapturingLogger();
+        var a2 = NewSessionOver(a.Db, streamerLogger: streamerLog);
+        var b2 = NewSessionOver(b.Db);
+        var (d2A, d2B) = DuplexPair();
+        using var cts2 = new CancellationTokenSource(RunTimeout);
+        var run2A = a2.Session.RunAsync(d2A, cts2.Token);
+        var run2B = b2.Session.RunAsync(d2B, cts2.Token);
+        await WaitForAsync(() => Task.FromResult(b2.Session.PeerNodeId is not null), RunTimeout);
+        await a.Db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (4, 'DELTA', 4)");
+        await WaitForAsync(async () => (await ReadOrders(b.Db)).Count == 4, RunTimeout);
+
+        Assert.DoesNotContain(streamerLog.Messages, m => m.StartsWith("Snapshot sent", StringComparison.Ordinal));
+        Assert.Equal(4, await LastAppliedRemoteSeq(b.Db));
+
+        cts2.Cancel();
+        await SwallowAsync(run2A);
+        await SwallowAsync(run2B);
+    }
+
+    private sealed class CapturingLogger : ILogger
+    {
+        private readonly List<string> _messages = [];
+        public IReadOnlyList<string> Messages { get { lock (_messages) return _messages.ToList(); } }
+        public IDisposable? BeginScope<TState>(TState state) where TState : notnull => null;
+        public bool IsEnabled(LogLevel logLevel) => true;
+        public void Log<TState>(LogLevel logLevel, EventId eventId, TState state, Exception? exception, Func<TState, Exception?, string> formatter)
+        {
+            lock (_messages) _messages.Add(formatter(state, exception));
+        }
+    }
+
     [Fact]
     public async Task RebuiltPeer_OwnWritesReachTheHealthyNode_DespiteTheOldPeersWatermark()
     {
@@ -292,6 +445,93 @@ public sealed class SnapshotResyncTests : IDisposable
         await SwallowAsync(runB);
     }
 
+    [Fact]
+    public async Task CapPrune_IntoALiveSessionsUnsentRange_FaultsIt_AndTheReconnectSnapshots()
+    {
+        // P4-1. The pump reads `seq > sentThruSeq`, which resumes SILENTLY above a hole a cap prune
+        // punched — so it keeps shipping later deltas, the peer acks them, and the pair is durably
+        // divergent. `needs_snapshot` (set in the prune's own transaction) heals only a RECONNECT,
+        // and nothing forced one: on a stable LAN with a slow-but-alive peer the session, and the
+        // divergence, could last for days. Faulting is what turns the persisted flag into a snapshot.
+        var a = await NewSideAsync();
+        for (var i = 1; i <= 6; i++)
+            await a.Db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (@id, 'A', @id)", new { id = i });
+
+        var (duplex, toSession, fromSession) = ScriptedPeer();
+        using var cts = new CancellationTokenSource(RunTimeout);
+        var run = a.Session.RunAsync(duplex, cts.Token);
+
+        // The peer holds everything so far: no snapshot is owed, and the pump idles at seq 6.
+        await DriveHandshakeAsync(a.Db, toSession, fromSession, cts.Token, lastAppliedRemoteSeq: 6);
+
+        // Write three more rows and prune all but the newest, in ONE transaction — which is what a
+        // cap prune is, and what makes this deterministic: the pump's reader can only ever see the
+        // before state (nothing above seq 6) or the after state (seqs 7 and 8 already gone). It can
+        // never catch rows 7/8 in between and legitimately advance past them.
+        await using (var tx = await a.Db.BeginTransactionAsync(cts.Token))
+        {
+            await tx.ExecuteAsync(
+                "INSERT INTO orders (id, sku, qty) VALUES (7,'PRUNED',7),(8,'PRUNED',8),(9,'KEPT',9)", null, cts.Token);
+            await tx.ExecuteAsync("UPDATE __localdb_peer_state SET needs_snapshot = 1 WHERE id = 1", null, cts.Token);
+            await tx.ExecuteAsync(
+                "DELETE FROM __localdb_oplog WHERE seq < (SELECT MAX(seq) FROM __localdb_oplog)", null, cts.Token);
+            await tx.CommitAsync(cts.Token);
+        }
+
+        var fault = await Assert.ThrowsAsync<LocalDbOplogGapException>(() => run);
+        Assert.Contains("6", fault.Message, StringComparison.Ordinal);
+
+        // And it faulted BEFORE shipping anything from above the hole: a delta for seq 9 would have
+        // moved the peer's watermark past rows 7 and 8, which is the divergence itself.
+        while (fromSession.TryRead(out var stray))
+            Assert.NotEqual(SyncMessage.MsgOneofCase.DeltaBatch, stray.MsgCase);
+
+        // The reconnect is what heals: the prune's needs_snapshot makes the handshake owe a full
+        // snapshot, which carries the pruned rows' current state to a real peer.
+        var a2 = NewSessionOver(a.Db);
+        var b = await NewSideAsync();
+        var (da, db) = DuplexPair();
+        var run2A = a2.Session.RunAsync(da, cts.Token);
+        var run2B = b.Session.RunAsync(db, cts.Token);
+
+        await WaitForAsync(async () => (await ReadOrders(b.Db)).Count == 9, RunTimeout);
+        Assert.Equal(await ReadOrders(a.Db), await ReadOrders(b.Db));
+
+        cts.Cancel();
+        await SwallowAsync(run2A);
+        await SwallowAsync(run2B);
+    }
+
+    [Fact]
+    public async Task AckPruningToEmpty_DoesNotFaultALiveSession()
+    {
+        // The other half of the horizon check: ordinary ack-pruning deletes seq <= last_acked_seq,
+        // which by construction sits at or below the sent watermark, so a converged pair that has
+        // pruned its whole oplog must keep streaming. A gap test written against `needs_snapshot`,
+        // or one that treated any risen horizon as a fault, would kill healthy sessions here.
+        var a = await NewSideAsync();
+        var b = await NewSideAsync();
+        for (var i = 1; i <= 4; i++)
+            await a.Db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (@id, 'A', @id)", new { id = i });
+
+        var (da, db) = DuplexPair();
+        using var cts = new CancellationTokenSource(RunTimeout);
+        var runA = a.Session.RunAsync(da, cts.Token);
+        var runB = b.Session.RunAsync(db, cts.Token);
+
+        await WaitForAsync(async () => (await ReadOrders(b.Db)).Count == 4, RunTimeout);
+        await WaitForAsync(async () => await OplogCount(a.Db) == 0, RunTimeout);
+
+        // Still live, still incremental: a post-prune write reaches B as an ordinary delta.
+        await a.Db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (5, 'POST', 5)");
+        await WaitForAsync(async () => (await ReadOrders(b.Db)).Any(r => r.Id == 5), RunTimeout);
+        Assert.False(runA.IsFaulted, runA.Exception?.ToString());
+
+        cts.Cancel();
+        await SwallowAsync(runA);
+        await SwallowAsync(runB);
+    }
+
     [Fact]
     public async Task Snapshot_MergesNotTruncates()
     {
@@ -536,6 +776,261 @@ public sealed class SnapshotResyncTests : IDisposable
         await SwallowAsync(run);
     }
 
+    // ---- Per-table snapshot scoping (0.3.0) ----------------------------------------------
+
+    private const string WidgetsSql = "CREATE TABLE widgets (id INTEGER PRIMARY KEY, name TEXT)";
+
+    /// <summary>
+    /// A node with two registered tables where exactly one — <c>orders</c> — was baselined.
+    /// <c>widgets</c> is registered BEFORE its rows are written, so ordinary capture ledgers them;
+    /// <c>orders</c> is registered after, so only baselining can. That asymmetry is what makes the
+    /// scope observable: a full snapshot carries both tables' ledgers, a scoped one carries orders
+    /// alone — and both tables hold rows either way, so an empty widgets stream cannot be mistaken
+    /// for "there was nothing to send".
+    /// </summary>
+    private async Task<Side> NewBaselinedSideAsync(ReplicationOptions? options = null)
+    {
+        var path = Path.Combine(Path.GetTempPath(), Guid.NewGuid() + ".db");
+        _paths.Add(path);
+        var db = new SqliteLocalDb(new LocalDbOptions { Path = path });
+        _disposables.Add(db);
+        await db.ExecuteAsync(OrdersSql);
+        await db.ExecuteAsync(WidgetsSql);
+
+        // Predates registration: invisible to CDC, so only baselining puts it in the ledger.
+        await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1,'PRE',1),(2,'PRE',2)");
+
+        db.RegisterReplicated("widgets");
+        await db.ExecuteAsync("INSERT INTO widgets (id, name) VALUES (1,'W1'),(2,'W2')");
+
+        db.RegisterReplicated("orders", baselineExistingRows: true);
+
+        return NewSessionOver(db, options);
+    }
+
+    /// <summary>The peer of <see cref="NewBaselinedSideAsync"/>: the same two tables, empty, owing nothing.</summary>
+    private async Task<Side> NewTwoTableSideAsync(ReplicationOptions? options = null)
+    {
+        var path = Path.Combine(Path.GetTempPath(), Guid.NewGuid() + ".db");
+        _paths.Add(path);
+        var db = new SqliteLocalDb(new LocalDbOptions { Path = path });
+        _disposables.Add(db);
+        await db.ExecuteAsync(OrdersSql);
+        await db.ExecuteAsync(WidgetsSql);
+        db.RegisterReplicated("orders");
+        db.RegisterReplicated("widgets");
+        return NewSessionOver(db, options);
+    }
+
+    private static List<string> SnapshotTablesIn(IEnumerable<SyncMessage> messages) =>
+        messages.Where(m => m.MsgCase == SyncMessage.MsgOneofCase.SnapshotBatch)
+            .Select(m => m.SnapshotBatch.TableName)
+            .Distinct(StringComparer.Ordinal)
+            .Order(StringComparer.Ordinal)
+            .ToList();
+
+    [Fact]
+    public async Task Baselining_OneTable_StreamsOnlyThatTable()
+    {
+        // The whole point of the per-table flag. Through 0.2.x the flag was one bit for the whole
+        // database, so baselining `orders` re-streamed `widgets` (and everything else) in full — in
+        // both directions at once on a pair enabling replication late, which on a database holding
+        // an event log at its retention cap is the largest transfer this library performs.
+        var a = await NewBaselinedSideAsync();
+        Assert.Equal(["orders"], await PendingSnapshotTables(a.Db));
+        Assert.Equal(0, await NeedsSnapshot(a.Db)); // NOT the database-wide flag
+
+        var (duplex, toSession, fromSession) = ScriptedPeer();
+        using var cts = new CancellationTokenSource(RunTimeout);
+        var run = a.Session.RunAsync(duplex, cts.Token);
+
+        await DriveHandshakeAsync(a.Db, toSession, fromSession, cts.Token, supportsPartialSnapshot: true);
+        var frames = await DrainThroughAsync(fromSession, SyncMessage.MsgOneofCase.SnapshotComplete, cts.Token);
+
+        var begin = frames.Single(m => m.MsgCase == SyncMessage.MsgOneofCase.SnapshotBegin).SnapshotBegin;
+        Assert.Equal(["orders"], begin.Tables);
+        Assert.Equal(["orders"], SnapshotTablesIn(frames));
+        // The baselined rows really are in there — the scope narrowed the snapshot, it did not empty it.
+        var pks = frames.Where(m => m.MsgCase == SyncMessage.MsgOneofCase.SnapshotBatch)
+            .SelectMany(m => m.SnapshotBatch.Rows).Select(r => r.PkJson).Order(StringComparer.Ordinal);
+        Assert.Equal(["{\"id\":1}", "{\"id\":2}"], pks);
+
+        cts.Cancel();
+        await SwallowAsync(run);
+    }
+
+    [Fact]
+    public async Task Baselining_AgainstAPeerWithoutPartialSupport_StreamsEveryTable()
+    {
+        // Mixed-version interop, the hard constraint. A pre-0.3.0 peer ignores SnapshotBegin.tables
+        // and would advance last_applied_remote_seq to as_of_seq on complete — which for a SCOPED
+        // snapshot silently discards every delta below as_of for the tables it left out. There is no
+        // way to express "partial" to that peer, so it is served the 0.2.x answer: everything.
+        var a = await NewBaselinedSideAsync();
+        var (duplex, toSession, fromSession) = ScriptedPeer();
+        using var cts = new CancellationTokenSource(RunTimeout);
+        var run = a.Session.RunAsync(duplex, cts.Token);
+
+        // supportsPartialSnapshot omitted == what a 0.1.x/0.2.x handshake decodes to.
+        await DriveHandshakeAsync(a.Db, toSession, fromSession, cts.Token);
+        var frames = await DrainThroughAsync(fromSession, SyncMessage.MsgOneofCase.SnapshotComplete, cts.Token);
+
+        var begin = frames.Single(m => m.MsgCase == SyncMessage.MsgOneofCase.SnapshotBegin).SnapshotBegin;
+        Assert.Empty(begin.Tables); // empty == full, the only scope the old wire can express
+        Assert.Equal(["orders", "widgets"], SnapshotTablesIn(frames));
+
+        cts.Cancel();
+        await SwallowAsync(run);
+    }
+
+    [Fact]
+    public async Task PartialSnapshot_DebtClearsOnlyWhenThePeerAcknowledges()
+    {
+        // Crash safety: an ungraceful exit part-way through a snapshot must re-stream it, so the
+        // flag cannot come down when the frames are merely enqueued. The peer's SnapshotAck is
+        // written after its own durable watermark write, so it means "I hold these rows".
+        var a = await NewBaselinedSideAsync();
+        var (duplex, toSession, fromSession) = ScriptedPeer();
+        using var cts = new CancellationTokenSource(RunTimeout);
+        var run = a.Session.RunAsync(duplex, cts.Token);
+
+        await DriveHandshakeAsync(a.Db, toSession, fromSession, cts.Token, supportsPartialSnapshot: true);
+        var frames = await DrainThroughAsync(fromSession, SyncMessage.MsgOneofCase.SnapshotComplete, cts.Token);
+        var complete = frames[^1].SnapshotComplete;
+
+        // The entire snapshot is on the wire and unacknowledged: the debt must still stand.
+        Assert.Equal(["orders"], await PendingSnapshotTables(a.Db));
+
+        await toSession.WriteAsync(new SyncMessage { SnapshotAck = new SnapshotAck { AsOfSeq = complete.AsOfSeq } }, cts.Token);
+        await WaitForAsync(async () => (await PendingSnapshotTables(a.Db)).Count == 0, RunTimeout);
+
+        cts.Cancel();
+        await SwallowAsync(run);
+    }
+
+    [Fact]
+    public async Task Snapshot_ToAPeerWithoutPartialSupport_ClearsOnEnqueue()
+    {
+        // The 0.2.x behaviour, kept deliberately: a peer that sends no SnapshotAck would otherwise
+        // leave the debt standing forever and re-stream the snapshot on every reconnect.
+        var a = await NewBaselinedSideAsync();
+        var (duplex, toSession, fromSession) = ScriptedPeer();
+        using var cts = new CancellationTokenSource(RunTimeout);
+        var run = a.Session.RunAsync(duplex, cts.Token);
+
+        await DriveHandshakeAsync(a.Db, toSession, fromSession, cts.Token);
+        await DrainThroughAsync(fromSession, SyncMessage.MsgOneofCase.SnapshotComplete, cts.Token);
+
+        await WaitForAsync(async () => (await PendingSnapshotTables(a.Db)).Count == 0, RunTimeout);
+
+        cts.Cancel();
+        await SwallowAsync(run);
+    }
+
+    [Fact]
+    public async Task PartialSnapshot_DoesNotAdvanceTheReceiversWatermark()
+    {
+        // The receiver's half of the same invariant. last_applied_remote_seq is ONE number for the
+        // whole peer oplog; moving it to a partial snapshot's as_of would declare the untouched
+        // tables applied through that point, and their deltas below it would never be resent.
+        var a = await NewSideAsync();
+        await a.Store.SetLastAppliedRemoteSeqAsync(10, 0);
+        var (duplex, toSession, fromSession) = ScriptedPeer();
+
+        using var cts = new CancellationTokenSource(RunTimeout);
+        var run = a.Session.RunAsync(duplex, cts.Token);
+
+        await DriveHandshakeAsync(a.Db, toSession, fromSession, cts.Token, supportsPartialSnapshot: true);
+
+        var begin = new SnapshotBegin { AsOfSeq = 900 };
+        begin.Tables.Add("orders");
+        await toSession.WriteAsync(new SyncMessage { SnapshotBegin = begin }, cts.Token);
+        await toSession.WriteAsync(new SyncMessage
+        {
+            SnapshotBatch = new SnapshotBatch { TableName = "orders", Rows = { SnapRow(1, "SNAP", 6_000_000) } },
+        }, cts.Token);
+        await toSession.WriteAsync(new SyncMessage { SnapshotComplete = new SnapshotComplete { AsOfSeq = 900 } }, cts.Token);
+
+        // The ack proves the complete was processed — and that a 0.3.0 receiver answers one at all.
+        var ack = await NextOfCaseAsync(fromSession, SyncMessage.MsgOneofCase.SnapshotAck, cts.Token);
+        Assert.Equal(900, ack.SnapshotAck.AsOfSeq);
+
+        Assert.Equal(10, await LastAppliedRemoteSeq(a.Db)); // NOT 900
+        Assert.Single(await ReadOrders(a.Db));              // the rows still merged
+
+        cts.Cancel();
+        await SwallowAsync(run);
+    }
+
+    [Fact]
+    public async Task FullSnapshot_ToAnOldPeer_DrawsNoSnapshotAck()
+    {
+        // Symmetry check on the negotiation: a 0.3.0 receiver must not put a message a pre-0.3.0
+        // sender has never heard of on the wire back to it.
+        var a = await NewSideAsync();
+        var (duplex, toSession, fromSession) = ScriptedPeer();
+
+        using var cts = new CancellationTokenSource(RunTimeout);
+        var run = a.Session.RunAsync(duplex, cts.Token);
+
+        await DriveHandshakeAsync(a.Db, toSession, fromSession, cts.Token);
+        await toSession.WriteAsync(new SyncMessage { SnapshotBegin = new SnapshotBegin { AsOfSeq = 300 } }, cts.Token);
+        await toSession.WriteAsync(new SyncMessage
+        {
+            SnapshotBatch = new SnapshotBatch { TableName = "orders", Rows = { SnapRow(1, "SNAP", 6_000_000) } },
+        }, cts.Token);
+        await toSession.WriteAsync(new SyncMessage { SnapshotComplete = new SnapshotComplete { AsOfSeq = 300 } }, cts.Token);
+
+        // A full snapshot from an old peer still advances the watermark — that is the 0.2.x contract.
+        await WaitForAsync(async () => await LastAppliedRemoteSeq(a.Db) == 300, RunTimeout);
+        // Barrier: an ack for this delta proves everything before it was processed, so an absent
+        // SnapshotAck is absent rather than merely late.
+        await toSession.WriteAsync(new SyncMessage
+        {
+            DeltaBatch = new DeltaBatch { Entries = { new OplogEntry
+            {
+                Seq = 301, TableName = "orders", PkJson = "{\"id\":2}",
+                RowJson = "{\"id\":2,\"sku\":\"BARRIER\",\"qty\":1}", Hlc = 6_000_001, NodeId = "peer", IsTombstone = false,
+            } } },
+        }, cts.Token);
+        await NextOfCaseAsync(fromSession, SyncMessage.MsgOneofCase.DeltaAck, cts.Token);
+        Assert.False(fromSession.TryPeek(out var stray) && stray.MsgCase == SyncMessage.MsgOneofCase.SnapshotAck);
+
+        cts.Cancel();
+        await SwallowAsync(run);
+    }
+
+    [Fact]
+    public async Task BaselinedTable_ReachesARealPeer_WithoutRestreamingTheOthers()
+    {
+        // End to end over two real databases: the baselined table's pre-existing rows arrive via the
+        // scoped snapshot, the untouched table's rows arrive as ordinary deltas, and the debt clears.
+        var options = new ReplicationOptions { FlushInterval = TimeSpan.FromMilliseconds(20) };
+        var a = await NewBaselinedSideAsync(options);
+        var b = await NewTwoTableSideAsync(options);
+
+        var (da, db) = DuplexPair();
+        using var cts = new CancellationTokenSource(RunTimeout);
+        var runA = a.Session.RunAsync(da, cts.Token);
+        var runB = b.Session.RunAsync(db, cts.Token);
+
+        async Task<bool> Converged() =>
+            (await ReadOrders(b.Db)).Count == 2
+            && (await b.Db.QueryAsync("SELECT COUNT(*) FROM widgets", x => x.GetInt64(0)))[0] == 2;
+
+        await WaitForAsync(Converged, RunTimeout);
+        Assert.Equal(await ReadOrders(a.Db), await ReadOrders(b.Db));
+        await WaitForAsync(async () => (await PendingSnapshotTables(a.Db)).Count == 0, RunTimeout);
+
+        // A partial snapshot speaks for nothing outside its tables, so neither side may skip the
+        // deltas it covered: B's watermark tracks the oplog, not the snapshot's as-of.
+        Assert.Equal(0, await NeedsSnapshot(a.Db));
+
+        cts.Cancel();
+        await SwallowAsync(runA);
+        await SwallowAsync(runB);
+    }
+
     [Fact]
     public async Task Sender_ClearsNeedsSnapshotFlag()
     {

ZB.MOM.WW.LocalDb/tests/ZB.MOM.WW.LocalDb.Tests/SyncSessionTests.cs

@@ -208,6 +208,50 @@ public sealed class SyncSessionTests : IDisposable
         Assert.Empty(await ReadOrders(b.Db));
     }
 
+    [Fact]
+    public async Task Handshake_FaultTeardown_StillTransmitsTheLocalHandshake()
+    {
+        // Schema-mismatch detection is symmetric: each side computes its verdict FROM the other's
+        // Handshake. So a session that faults on the peer's handshake must still flush its own —
+        // otherwise the healthy peer learns nothing and waits out its receive cap (tests) or
+        // misclassifies the mismatch as a generic disconnect and reconnect-loops (production).
+        //
+        // The race being pinned: with the peer's handshake already buffered, the path from
+        // enqueueing our Handshake on the control lane to the validation throw has no yields,
+        // while the writer's wakeup sits in the thread-pool queue — a teardown that cancels the
+        // writer can therefore kill the Handshake before it ever transmits. Preloading makes the
+        // sprint as tight as possible, and running the scenarios concurrently starves the pool
+        // the way a busy test suite does.
+        const int Iterations = 40;
+        var sides = new List<Side>(Iterations);
+        for (var i = 0; i < Iterations; i++)
+            sides.Add(await NewSideAsync());
+
+        using var cts = new CancellationTokenSource(RunTimeout);
+        var runs = new List<(Task Run, ChannelReader<SyncMessage> FromSession)>(Iterations);
+        foreach (var side in sides)
+        {
+            var (duplex, toSession, fromSession) = ScriptedPeer();
+            var mismatched = MatchingHandshake(side.Db, "peer");
+            mismatched.Tables[0].Digest = "not-the-local-digest";
+            await toSession.WriteAsync(new SyncMessage { Handshake = mismatched }, cts.Token);
+            runs.Add((side.Session.RunAsync(duplex, cts.Token), fromSession));
+        }
+
+        var lost = 0;
+        foreach (var (run, fromSession) in runs)
+        {
+            await Assert.ThrowsAsync<LocalDbSchemaMismatchException>(() => run);
+            var sent = false;
+            while (fromSession.TryRead(out var msg))
+                sent |= msg.MsgCase == SyncMessage.MsgOneofCase.Handshake;
+            if (!sent)
+                lost++;
+        }
+
+        Assert.True(lost == 0, $"{lost}/{Iterations} sessions tore down without transmitting their own Handshake");
+    }
+
     [Fact]
     public async Task Handshake_SchemaVersionMismatch_FailsClosed()
     {
@@ -553,8 +597,9 @@ public sealed class SyncSessionTests : IDisposable
             await a.Store.SetNeedsSnapshotAsync(true, default);
             var (duplex, toSession, fromSession) = ScriptedPeer();
             var sent = new TaskCompletionSource();
-            a.Session.SnapshotSender = async (send, token) =>
+            a.Session.SnapshotSender = async (scope, send, token) =>
             {
+                Assert.False(scope.IsPartial); // the database-wide flag always means every table
                 sent.TrySetResult();
                 await send(new SyncMessage { SnapshotComplete = new SnapshotComplete { AsOfSeq = 7 } }, token);
                 return 7;

ZB.MOM.WW.LocalDb/tests/ZB.MOM.WW.LocalDb.Tests/SyncStreamGuardTests.cs

@@ -0,0 +1,181 @@
+using System.Threading.Channels;
+using Grpc.Core;
+using Microsoft.Extensions.Logging.Abstractions;
+using Microsoft.Extensions.Options;
+using ZB.MOM.WW.LocalDb.Contracts;
+using ZB.MOM.WW.LocalDb.Replication;
+
+namespace ZB.MOM.WW.LocalDb.Tests;
+
+/// <summary>
+/// The passive service's single-stream guard under a HALF-OPEN link: the dialling node is gone, but
+/// nothing on this side knows it, so the handler sits inside <c>MoveNext</c> forever holding the
+/// guard. Through 0.3.1 that made the pair unrecoverable without a restart — every reconnect from
+/// the rebooted peer was refused <c>AlreadyExists</c>, indefinitely on an idle pair, while this node
+/// went on reporting itself connected. The guard is there to serialize sessions, not to fence out a
+/// caller the auth interceptor already vouched for, so the newest dialler wins.
+/// </summary>
+public sealed class SyncStreamGuardTests : IDisposable
+{
+    private static readonly TimeSpan Timeout = TimeSpan.FromSeconds(10);
+
+    private readonly List<string> _paths = [];
+    private readonly List<IDisposable> _disposables = [];
+
+    public void Dispose()
+    {
+        foreach (var d in _disposables)
+            d.Dispose();
+        Microsoft.Data.Sqlite.SqliteConnection.ClearAllPools();
+        foreach (var p in _paths)
+        {
+            if (File.Exists(p)) File.Delete(p);
+            if (File.Exists(p + "-wal")) File.Delete(p + "-wal");
+            if (File.Exists(p + "-shm")) File.Delete(p + "-shm");
+        }
+    }
+
+    private LocalDbSyncService NewService()
+    {
+        var path = Path.Combine(Path.GetTempPath(), Guid.NewGuid() + ".db");
+        _paths.Add(path);
+        var db = new Internal.SqliteLocalDb(new LocalDbOptions { Path = path });
+        _disposables.Add(db);
+        db.ExecuteAsync("CREATE TABLE orders (id INTEGER PRIMARY KEY, sku TEXT)").GetAwaiter().GetResult();
+        db.RegisterReplicated("orders");
+
+        return new LocalDbSyncService(
+            db,
+            Options.Create(new ReplicationOptions { FlushInterval = TimeSpan.FromMilliseconds(20) }),
+            NullLoggerFactory.Instance);
+    }
+
+    private static ServerCallContext ContextFor(CancellationToken ct) => new FakeCallContext(ct);
+
+    [Fact]
+    public async Task WedgedIncumbent_IsDisplacedByANewStream()
+    {
+        var service = NewService();
+        // Eviction is normally generous; the test only needs it to not be the thing under test.
+        service.TakeoverTimeout = Timeout;
+
+        // The incumbent's peer crashed: its stream will never deliver another message, and neither
+        // end times out on silence — a converged pair is legitimately quiet. The handler parks in
+        // the handshake's first read, holding the guard.
+        using var incumbentCts = new CancellationTokenSource();
+        var incumbentOut = new RecordingWriter();
+        var incumbent = service.Sync(new SilentReader(), incumbentOut, ContextFor(incumbentCts.Token));
+
+        // It really is running and really did claim the guard (its handshake is on the wire).
+        await incumbentOut.NextAsync(Timeout);
+        Assert.False(incumbent.IsCompleted);
+
+        // The rebooted peer dials again. Pre-fix this threw AlreadyExists and the pair stayed wedged.
+        using var newcomerCts = new CancellationTokenSource();
+        var newcomerOut = new RecordingWriter();
+        var newcomer = service.Sync(new SilentReader(), newcomerOut, ContextFor(newcomerCts.Token));
+
+        // The incumbent is torn down — gracefully, not as a fault of this node.
+        await incumbent.WaitAsync(Timeout);
+        // ...and the new stream is genuinely served: its own handshake goes out.
+        var handshake = await newcomerOut.NextAsync(Timeout);
+        Assert.Equal(SyncMessage.MsgOneofCase.Handshake, handshake.MsgCase);
+        Assert.False(newcomer.IsCompleted);
+
+        newcomerCts.Cancel();
+        await SwallowAsync(newcomer);
+    }
+
+    [Fact]
+    public async Task DisplacedIncumbent_ReleasesTheGuard_SoAThirdStreamIsAlsoServed()
+    {
+        // Takeover must leave the guard in a state that can be taken over AGAIN — a handover that
+        // leaked the slot would simply move the wedge one reconnect down the line.
+        var service = NewService();
+        service.TakeoverTimeout = Timeout;
+
+        using var firstCts = new CancellationTokenSource();
+        var firstOut = new RecordingWriter();
+        var first = service.Sync(new SilentReader(), firstOut, ContextFor(firstCts.Token));
+        await firstOut.NextAsync(Timeout);
+
+        using var secondCts = new CancellationTokenSource();
+        var secondOut = new RecordingWriter();
+        var second = service.Sync(new SilentReader(), secondOut, ContextFor(secondCts.Token));
+        await first.WaitAsync(Timeout);
+        await secondOut.NextAsync(Timeout);
+
+        using var thirdCts = new CancellationTokenSource();
+        var thirdOut = new RecordingWriter();
+        var third = service.Sync(new SilentReader(), thirdOut, ContextFor(thirdCts.Token));
+        await second.WaitAsync(Timeout);
+        Assert.Equal(SyncMessage.MsgOneofCase.Handshake, (await thirdOut.NextAsync(Timeout)).MsgCase);
+
+        thirdCts.Cancel();
+        await SwallowAsync(third);
+    }
+
+    private static async Task SwallowAsync(Task task)
+    {
+        try { await task.WaitAsync(Timeout); }
+        catch (OperationCanceledException) { }
+        catch (RpcException) { }
+    }
+
+    /// <summary>
+    /// The minimum <see cref="ServerCallContext"/> the handler actually reads: its cancellation
+    /// token. (Grpc.Core.Api ships no test double, and the ASP.NET Core one is not reachable outside
+    /// a live call.)
+    /// </summary>
+    private sealed class FakeCallContext(CancellationToken ct) : ServerCallContext
+    {
+        protected override string MethodCore => "/localdb_sync.v1.LocalDbSync/Sync";
+        protected override string HostCore => "test";
+        protected override string PeerCore => "test";
+        protected override DateTime DeadlineCore => DateTime.MaxValue;
+        protected override Metadata RequestHeadersCore => [];
+        protected override CancellationToken CancellationTokenCore => ct;
+        protected override Metadata ResponseTrailersCore { get; } = [];
+        protected override Status StatusCore { get; set; }
+        protected override WriteOptions? WriteOptionsCore { get; set; }
+        protected override AuthContext AuthContextCore => new(null, new Dictionary<string, List<AuthProperty>>());
+
+        protected override ContextPropagationToken CreatePropagationTokenCore(ContextPropagationOptions? options) =>
+            throw new NotSupportedException();
+
+        protected override Task WriteResponseHeadersAsyncCore(Metadata responseHeaders) => Task.CompletedTask;
+    }
+
+    /// <summary>A stream that never delivers anything and never ends — a half-open link.</summary>
+    private sealed class SilentReader : IAsyncStreamReader<SyncMessage>
+    {
+        public SyncMessage Current => throw new InvalidOperationException("no message");
+
+        public async Task<bool> MoveNext(CancellationToken cancellationToken)
+        {
+            await Task.Delay(System.Threading.Timeout.Infinite, cancellationToken);
+            return false;
+        }
+    }
+
+    private sealed class RecordingWriter : IServerStreamWriter<SyncMessage>
+    {
+        private readonly Channel<SyncMessage> _written = Channel.CreateUnbounded<SyncMessage>();
+
+        public WriteOptions? WriteOptions { get; set; }
+
+        public Task WriteAsync(SyncMessage message) => WriteAsync(message, CancellationToken.None);
+
+        public Task WriteAsync(SyncMessage message, CancellationToken cancellationToken)
+        {
+            _written.Writer.TryWrite(message);
+            return Task.CompletedTask;
+        }
+
+        public async Task<SyncMessage> NextAsync(TimeSpan timeout)
+        {
+            using var cts = new CancellationTokenSource(timeout);
+            return await _written.Reader.ReadAsync(cts.Token);
+        }
+    }
+}

ZB.MOM.WW.Overview/CLAUDE.md

@@ -0,0 +1,65 @@
+# ZB.MOM.WW.Overview
+
+The hosted family dashboard (Blazor Server). Build/test from this directory: `dotnet build ZB.MOM.WW.Overview.slnx`, `dotnet test ZB.MOM.WW.Overview.slnx` (fully offline).
+
+## Index history (migrated from the scadaproj root CLAUDE.md)
+
+> Moved verbatim out of the scadaproj root `CLAUDE.md` on 2026-09-03 so it loads only when needed. **This file is now the authoritative place for this history — append new entries here and keep the root index row short.**
+
+
+Besides the shared libraries, this repo hosts one **application**:
+[`ZB.MOM.WW.Overview/`](ZB.MOM.WW.Overview/) — a Blazor Server single-pane dashboard that polls
+every registered instance's anonymous `/health/ready` + `/health/active`, caches one snapshot
+(pages never poll) and renders per-cluster state including the Akka **leader chip**, which is what
+the `ZB.MOM.WW.Health` `0.2.0`/`0.3.0` per-entry `data` work exists to feed. **No login, strictly
+read-only, holds no credentials** — a requirement, not an omission; it references no Auth/Audit/
+Secrets package. Design + impl plan: `docs/plans/2026-07-22-overview-dashboard-{design,impl-plan}.md`
+(ledger 19/20; the open task is Phase 4 live acceptance, recorded in `41013dc` as 7 of 9 design §9
+checks passing, 2 blocked on an OtOpcUa rig redeploy).
+
+The `Applications` registry ships **empty** on purpose: a deployment supplying none fails startup
+naming the key rather than booting an empty dashboard. Per-deployment overlays are committed —
+`appsettings.Docker.json` (the local rigs, joined via external networks) and
+`appsettings.Wonder.json`.
+
+**Deployed to `wonder-app-vd03` 2026-08-11** (`main @ 4f97e72`) as that box's **fifth** app:
+service `ZbOverview` on **:5320** under the shared NSSM, `ASPNETCORE_ENVIRONMENT=Wonder`,
+5/5 instances Up. Two constraints worth carrying to any future box:
+
+- **Publish self-contained** (`-r win-x64 --self-contained true`). wonder has **no .NET 10 runtime**
+  — every app there carries its own. An FDD publish installs fine and then fails at service start.
+- **Registry `BaseUrl` must not put co-located instances on a shared host name.** Leader-address →
+  instance resolution matches on **host only** (Akka and HTTP ports differ by design), which assumes
+  distinct hosts per instance — true on the rig, false on a single box, where the fleet-wide scan
+  lets the first matching entry win the chip for *every* group. On wonder that made ScadaBridge's
+  **site** card claim `Leader · central`, a different Akka cluster. Fixed by using `127.0.0.1`,
+  which matches nothing, so each chip falls back to the raw Akka address.
+
+**QA runs an instance on EACH pair node — `wonder-app-vq03:5320` and (since 2026-08-14)
+`wonder-app-vq04:5320`** — both watching the same registry, now **10 instances across 4
+applications** after the 2026-08-14 VQ04 singleton clone (MxGateway + HistorianGateway added on
+vq04; see `deployment.md` §7.1). It is the fastest way to read QA's whole topology.
+Its registry uses **real FQDNs**, and that is correct rather than a divergence: the `127.0.0.1`
+rule above exists only because VD03 co-locates every instance on one host. QA is a **2-node pair**
+(`vq03`/`vq04`) with genuinely distinct hosts, so leader resolution works as designed and the chips
+resolve to real names (`node-a (vq03)`). Carry the *reason*, not the literal `127.0.0.1`.
+
+## Commands (from the index)
+
+```bash
+# ZB.MOM.WW.Overview  (~/Desktop/scadaproj/ZB.MOM.WW.Overview — the hosted family dashboard)
+dotnet build ZB.MOM.WW.Overview.slnx
+dotnet test  ZB.MOM.WW.Overview.slnx      # 177 tests, fully offline
+cd docker && docker compose up -d --build # local rigs → http://localhost:5320 (start the rigs first)
+# wonder deploy artifact — SELF-CONTAINED, the box has no .NET 10 runtime:
+dotnet publish src/ZB.MOM.WW.Overview/ZB.MOM.WW.Overview.csproj -c Release -r win-x64 --self-contained true -o <out>
+```
+
+**2026-10-01 — BTDB Manager registered on all three dashboards.** `appsettings.Wonder.json` (vd03,
+committed) gains one instance `wonder` → `http://127.0.0.1:9876` (management link
+`http://10.220.157.247:9876/`); the QA overlay `E:\ApiInstall\Overview\app\appsettings.QA.json` (deployed
+only, identical on vq03/vq04) gains `vq03`/`vq04` → `http://wonder-app-vq0N.zmr.zimmer.com:9876`. All
+`HasActiveRole: false` (single unclustered Blazor app); `/health/ready` carries one `sql` entry (BT
+reachable). The registry is read at startup only, so `ZbOverview` was restarted on each node (previous
+files kept as `*.bak-20261001T02xxxx`). Verified: vd03 6/6 Up, vq03 and vq04 12/12 Up.
+

ZB.MOM.WW.Overview/docker/README.md

@@ -25,11 +25,14 @@ The dev rigs publish only some node ports to the host:
 | OtOpcUa central-1/2 | container `:9000` | **no** — only Traefik `:9200`, load-balanced across the pair |
 | OtOpcUa site-\* (4) | container `:8080` | **no** — not published |
 | HistorianGateway | container `:5220` | yes — `localhost:5220` |
+| BTDB Manager | container `:9877` | yes — `localhost:9877` |
 | MxAccessGateway | windev `:5130` | yes, with the VPN up |
 
 So a host-run dashboard can only see part of the fleet. This stack joins the three rig networks
 (`otopcua-dev_default`, `scadabridge-net`, `zb-historiangw_default`, all declared `external`) and
-addresses every node by container DNS name, which is how it sees all of them.
+addresses every node by container DNS name, which is how it sees all of them. BTDB Manager
+(`~/Desktop/BTDB/docker`) sits on `scadabridge-net` too, because its BT database lives on
+ScadaBridge's infra MSSQL container, so no fourth network is needed for it.
 
 Ports above were established by probing the running rigs on 2026-07-24, **not** read off the
 compose files — the two disagree. In particular OtOpcUa's site nodes serve health on `:8080`, not

ZB.MOM.WW.Overview/src/ZB.MOM.WW.Overview/Components/Widgets/InstanceCard.razor

@@ -42,6 +42,26 @@
             <span class="v @ChecksClass">@ChecksText</span>
         </div>
 
+        @if (SessionText is { Length: > 0 } sessionText)
+        {
+            @* Only the gateway publishes these counts, so the row is absent everywhere else
+               rather than showing an empty or zero value on apps the concept does not apply to. *@
+            <div class="kv" data-testid="mxaccess-sessions">
+                <span class="k">MXAccess</span>
+                <span class="v @SessionClass">@sessionText</span>
+            </div>
+        }
+
+        @if (HistorianPoolText is { Length: > 0 } poolText)
+        {
+            @* "pooled" and not "connections": this is warm pool depth, and an idle pool sitting at
+               0 is normal. Labelling it as connections would read as an outage. *@
+            <div class="kv" data-testid="historian-pools">
+                <span class="k">Sessions pooled</span>
+                <span class="v @HistorianPoolClass" title="Warm pre-authenticated historian sessions per pool">@poolText</span>
+            </div>
+        }
+
         @if (Instance.Error is { Length: > 0 } error)
         {
             <div class="kv">
@@ -260,6 +280,117 @@
         }
     }
 
+    /// <summary>
+    /// The MXAccess session check's entry from the <b>active</b> tier, or null when this instance
+    /// publishes none.
+    /// </summary>
+    /// <remarks>
+    ///     <para>
+    ///         Located by the data it carries rather than by check name, for the same reason
+    ///         <see cref="ClusterDataLine"/> is: a name is a rename away from silently dropping the
+    ///         line, whereas the <c>total</c>/<c>ready</c>/<c>faulted</c> triple is the check's
+    ///         actual contract. All three are required so a future check that happens to publish a
+    ///         lone <c>total</c> cannot be mistaken for this one.
+    ///     </para>
+    ///     <para>
+    ///         It comes from <see cref="InstanceSnapshot.ActiveReport"/>, never the ready tier —
+    ///         the gateway registers this check on <c>active</c> because a gateway with no sessions
+    ///         is still ready to serve.
+    ///     </para>
+    /// </remarks>
+    private ZbHealthEntry? SessionEntry =>
+        Instance.ActiveReport?.Entries.Values.FirstOrDefault(e =>
+            e.DataInt("total") is not null
+            && e.DataInt("ready") is not null
+            && e.DataInt("faulted") is not null);
+
+    /// <summary>The session counts as a short face-level line, or null when there are none to show.</summary>
+    /// <remarks>
+    /// Rendered on the card face rather than inside the collapsed check list: "how many MXAccess
+    /// connections are up" is a number an operator scans for, and a value behind a disclosure
+    /// triangle is one nobody sees. Zero is shown, not hidden — a gateway holding no sessions at
+    /// all is precisely the interesting case, and suppressing it would make the row appear only
+    /// when the news is good.
+    /// </remarks>
+    private string? SessionText
+    {
+        get
+        {
+            var entry = SessionEntry;
+            if (entry is null)
+                return null;
+
+            int total = entry.DataInt("total") ?? 0;
+            if (total == 0)
+                return "0";
+
+            var parts = new List<string> { total.ToString(System.Globalization.CultureInfo.InvariantCulture) };
+            parts.Add($"{entry.DataInt("ready") ?? 0} ready");
+
+            foreach (var (key, label) in new[] { ("faulted", "faulted"), ("starting", "starting") })
+            {
+                if (entry.DataInt(key) is > 0 and var n)
+                    parts.Add($"{n} {label}");
+            }
+
+            return string.Join(" · ", parts);
+        }
+    }
+
+    /// <summary>Highlights the session row only when a session is actually broken.</summary>
+    private string? SessionClass => SessionEntry?.DataInt("faulted") is > 0 ? "bad" : null;
+
+    /// <summary>Pools listed in the order an operator cares about; anything unrecognised follows.</summary>
+    private static readonly string[] PoolOrder = ["read", "event"];
+
+    /// <summary>
+    /// The historian session-pool checks, which publish a <c>warmSessions</c> depth per pool.
+    /// </summary>
+    /// <remarks>
+    ///     <para>
+    ///         These live on the <b>ready</b> tier, not the active one, and carry an entirely
+    ///         different key set from the gateway's session check — so the two rows are separate
+    ///         renderings rather than one abstraction. Forcing them together would mean inventing a
+    ///         shared shape that neither app actually publishes.
+    ///     </para>
+    ///     <para>
+    ///         <b>This is pool depth, not connections in use.</b> A pool reporting 0 is idle, not
+    ///         broken: sessions are leased on demand and the event pool legitimately sits empty until
+    ///         something reads events. The row is labelled "pooled" so nobody reads a 0 as an outage.
+    ///     </para>
+    /// </remarks>
+    private IEnumerable<(string Pool, int Warm, int Failures)> HistorianPools =>
+        (Instance.Report?.Entries.Values ?? Enumerable.Empty<ZbHealthEntry>())
+            .Where(e => e.DataInt("warmSessions") is not null && e.DataString("pool") is { Length: > 0 })
+            .Select(e => (
+                Pool: e.DataString("pool")!,
+                Warm: e.DataInt("warmSessions") ?? 0,
+                Failures: e.DataInt("consecutiveFactoryFailures") ?? 0))
+            .OrderBy(p => Array.IndexOf(PoolOrder, p.Pool) is var i && i >= 0 ? i : int.MaxValue)
+            .ThenBy(p => p.Pool, StringComparer.Ordinal);
+
+    /// <summary>The pool depths as a short face-level line, or null when this app publishes none.</summary>
+    private string? HistorianPoolText
+    {
+        get
+        {
+            var pools = HistorianPools.ToList();
+            if (pools.Count == 0)
+                return null;
+
+            return string.Join(" · ", pools.Select(p =>
+                p.Failures > 0
+                    ? $"{p.Pool} {p.Warm} ({p.Failures} fail)"
+                    : $"{p.Pool} {p.Warm}"));
+        }
+    }
+
+    /// <summary>
+    /// Red only when a pool's session factory is actually failing — never merely because a pool is
+    /// empty, which is the normal idle state.
+    /// </summary>
+    private string? HistorianPoolClass => HistorianPools.Any(p => p.Failures > 0) ? "bad" : null;
+
     private static string DotClass(string? status) => status switch
     {
         "Healthy" => "ok",