docs(index): OtOpcUa address-space availability guarantee; track the OtOpcUa LocalDb design

The #485/#486 fixes changed nothing structural about OtOpcUa — same drivers,
same dual-namespace scheme, same Galaxy flow — so the row is mostly untouched.
What DID change is a fact any OPC UA consumer depends on, and ScadaBridge is
mid-cutover onto exactly this address space: a transient ConfigDb error used to
empty the served address space (a PureRemove of every node, observed live and it
stayed empty), stop every driver and clear every subscription, while still
reporting the deploy Applied. A consumer would have seen all bindings go bad
with nothing upstream admitting a problem. Recorded on the OtOpcUa row.

Also fixes real drift found while checking: the OtOpcUa LocalDb adoption design
was sitting UNTRACKED here and unreferenced by the index, even though the
LocalDb row claims the adoption designs live in this repo. Now committed,
referenced alongside the ScadaBridge one, and its stale header corrected —
it still said Phase 1 was "not yet executed" against lib 0.1.1, when Phase 1
shipped and merged against 0.1.3. Phase 2 designed but not executed, on both
the doc and the row.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW

docs/plans/2026-07-20-otopcua-localdb-design.md

@@ -0,0 +1,204 @@
+# OtOpcUa LocalDb Adoption — Design
+
+**Date:** 2026-07-20
+**Status:** **Phase 1 EXECUTED, merged + pushed** to OtOpcUa `master` 2026-07-21 (merge `c957db52`;
+live gate 8/8, `OtOpcUa/docs/plans/2026-07-20-localdb-phase1-live-gate.md`). Both limitations that gate
+left open were closed the same day (merge `5fdb5a5a`, `…-followups-live-gate.md`), which turned up a
+third library defect reachable only once the first was fixed. **Phase 2 is designed but NOT executed** —
+plan at `OtOpcUa/docs/plans/2026-07-20-localdb-adoption-phase2.md` (9 tasks).
+Implementation plans live in the OtOpcUa repo at
+`docs/plans/2026-07-20-localdb-adoption-phase1.md` and `…-phase2.md`.
+**Library:** `ZB.MOM.WW.LocalDb` / `.Replication` / `.Contracts` — designed against **0.1.1**; Phase 1
+shipped against **0.1.3**, which carries the two rebuilt-peer replication fixes the live gates forced.
+**Predecessor:** ScadaBridge adoption (merged 2026-07-20, PR #23) — this design deliberately reuses its
+wiring pattern and every lesson from its two live gates. See §Lessons appendix.
+**Executor:** plans are written to be executed by Claude Opus agents
+(`claude --model opus` + `superpowers-extended-cc:executing-plans`).
+
+---
+
+## 1. Decisions (with alternatives considered)
+
+Per-decision best-practice fit noted; the recommended option is what this design adopts.
+
+| # | Decision | Chosen | Alternatives considered |
+|---|---|---|---|
+| D1 | **What OtOpcUa caches locally** | **Phase 1: the deployed-configuration artifact** (`Deployment.ArtifactBlob` + current-pointer), giving boot-from-cache when central SQL is unreachable. **Phase 2: the alarm-historian store-and-forward buffer** (today's standalone `alarm-historian.db`). | Secrets store (already has its own Akka replication — leave alone); FasterLog historization outbox (not SQLite, already crash-safe — leave alone); alarm condition state (central SQL by design, survives failover via central — defer). |
+| D2 | **Fate of the dormant LiteDB LocalCache** (`ILocalConfigCache`, `GenerationSealedCache`, `ResilientConfigReader` — built, unit-tested, never registered) | **Delete in Phase 1.** LocalDb supersedes it; keeping two unwired cache designs invites the next reader to wire the wrong one. The useful ideas (fail-closed cache miss, stale-config flag) are re-expressed on LocalDb. | Wire LiteDB as-is (rejected: second embedded engine, no replication, dead since Phase 6.1); back `ILocalConfigCache` with LocalDb behind the existing interface (rejected: the interface models LiteDB generations, not the artifact/pointer shape v3 actually ships). |
+| D3 | **Sync transport** | **Dedicated Kestrel h2c (HTTP/2-only) listener + the library's gRPC stream**, exactly like ScadaBridge's port 8083. New config key `LocalDb:SyncListenPort` (default **0 = no listener**). | Ride the Akka mesh like Secrets replication (rejected: the LocalDb sync engine is gRPC-stream-only; an Akka transport is a library fork, out of scope); multiplex on the existing `:9000` HTTP/1.1 listener (rejected: cleartext `Http1AndHttp2` cannot negotiate h2c without ALPN — prior-knowledge h2c needs an `Http2`-only listener). |
+| D4 | **Rollout gating** | **Storage move ships unconditionally** (LocalDb with replication off is just a local SQLite file); **replication is config-gated default-OFF** (`PeerAddress` unset, `SyncListenPort` 0) and proven on the docker-dev rig's **site-a pair only**, with site-b left unreplicated as the default-OFF pin — the ScadaBridge posture, and consistent with OtOpcUa's existing `Secrets:Replication:Enabled` default-false gate. | Big-bang enable everywhere (rejected: no production KEK-style story yet for sync ApiKey distribution; default-OFF has served the family well). |
+| D5 | **DB layout** | **One consolidated per-node file**, `LocalDb:Path` default `./data/otopcua-localdb.db`. The library is one-DB-per-process (`AddZbLocalDb` first-registration-wins). | Per-concern files (impossible without lib changes). |
+| D6 | **Artifact storage shape** | **Chunked base64 TEXT rows** (`deployment_artifacts`, PK `(deployment_id, chunk_index)`, raw chunk 128 KiB) + a `deployment_pointer` row per cluster with a whole-artifact SHA-256. Reason: `RegisterReplicated` **rejects BLOB columns**, and `MaxBatchSize` batches by **row count** against gRPC's 4 MB default cap — a single unchunked multi-MB artifact row would be undeliverable, full stop (the ScadaBridge "if any single row approaches 4 MB → STOP" rule, designed out from day one). | Single base64 row + low MaxBatchSize (rejected: caps artifact size at ~3 MB forever, silently); store artifact on disk beside the DB and replicate only metadata (rejected: reintroduces the file-vs-row consistency problem replication exists to solve). |
+| D7 | **Which nodes run LocalDb** | **Driver-role nodes only** (`hasDriver`), like the alarm historian. Admin-only nodes have nothing to cache; skipping them avoids the `LocalDb:Path`-required-or-no-boot constraint there. The rig's central pair is `admin,driver`, so it still gets LocalDb. | All nodes (rejected: pointless required-config on admin-only nodes). |
+| D8 | **Phase 2 drain ownership** | **Primary-gated drain**: only the node whose delivered-snapshot role is Primary (via `PrimaryGatePolicy`) drains the replicated alarm S&F buffer to HistorianGateway. Failover semantics: at-least-once (a failover window may re-send events whose delivered-mark hadn't replicated); accepted and documented — same class as ScadaBridge's convergent-not-identical decision. | Both nodes drain with distributed dedup (rejected: complexity; the historian has no idempotence key for `SendEvent`). |
+| D9 | **Bespoke-mechanism deletion** | Phase 2 **rewires `SqliteStoreAndForwardSink` onto `ILocalDb` and deletes its private file/pragma management outright** — cutover in one commit, no dual-mechanism period (the ScadaBridge Phase-2 rule: cutover *is* the test). Public seam (`IAlarmHistorianSink` or equivalent) unchanged. | Parallel-run both sinks (rejected: double-delivery to the historian, and the family precedent is clean cutover). |
+
+## 2. Current state (inventoried 2026-07-20)
+
+| Store | File / location | Mechanism today | Replicated? | Fate |
+|---|---|---|---|---|
+| Deployed config | central SQL `Deployment.ArtifactBlob`; fetched by `DriverHostActor` on `DispatchDeployment` (DPS topic `deployments`); **no local fallback** — node goes `Stale` and ignores deploys when SQL is down | EF Core, push-then-pull | n/a (central) | **Phase 1:** cached per-pair in LocalDb; boot-from-cache when central unreachable |
+| LiteDB config cache | `config_cache.db` (default) — **never created in practice; code never registered** | LiteDB 5, dormant | no | **Phase 1: deleted** |
+| Alarm-historian S&F | `alarm-historian.db` (`AlarmHistorian:DatabasePath`), per-node | bespoke `SqliteStoreAndForwardSink`, own file/pragmas | no | **Phase 2:** table in the consolidated LocalDb, replicated, primary-gated drain |
+| Secrets | `otopcua-secrets.db` per node | `ZB.MOM.WW.Secrets` + optional Akka replication | when enabled | untouched |
+| Continuous-historization outbox | FasterLog dir (`ContinuousHistorization:OutboxPath`) | FasterLog | no | untouched |
+
+## 3. Design
+
+### 3.1 Shape
+
+- Host gains `AddOtOpcUaLocalDb(services, config)` (new `LocalDbRegistration` in
+  `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/`, mirroring `SecretsRegistration`), called from
+  `Program.cs` inside the `hasDriver` branch. It runs `AddZbLocalDb(config, LocalDbSetup.OnReady)` +
+  `AddZbLocalDbReplication(config)` unconditionally (inert without `PeerAddress`).
+- `LocalDbSetup.OnReady` enforces the load-bearing order: **DDL → `RegisterReplicated` → (no migrator in
+  Phase 1; Phase 2 adds one) → first writes**.
+- Sync endpoint: `AddGrpc(o => o.Interceptors.Add<LocalDbSyncAuthInterceptor>())` +
+  `app.MapZbLocalDbSync()`, served on a dedicated `Http2`-only cleartext listener bound iff
+  `LocalDb:SyncListenPort > 0`. **Because explicit `Listen*` calls make Kestrel ignore
+  `ASPNETCORE_URLS`, the existing `:9000` HTTP/1.1 binding must be re-added explicitly in the same
+  `ConfigureKestrel` block** — this is called out as a high-risk task with its own rig verification.
+- Auth: consumer-supplied fail-closed bearer interceptor (fixed-time compare, scoped to
+  `/localdb_sync.v1.LocalDbSync/`), a line-for-line port of ScadaBridge's — the library's passive side
+  verifies nothing. `ApiKey` production value is a `${secret:}` reference (OtOpcUa's pre-host
+  `SecretReferenceExpander` already expands these).
+
+### 3.2 Schema (Phase 1)
+
+```sql
+CREATE TABLE IF NOT EXISTS deployment_artifacts (
+    deployment_id   TEXT    NOT NULL,
+    chunk_index     INTEGER NOT NULL,
+    cluster_id      TEXT    NOT NULL,
+    revision_hash   TEXT    NOT NULL,
+    chunk_count     INTEGER NOT NULL,
+    chunk_base64    TEXT    NOT NULL,
+    cached_at_utc   TEXT    NOT NULL,
+    PRIMARY KEY (deployment_id, chunk_index)
+);
+CREATE TABLE IF NOT EXISTS deployment_pointer (
+    cluster_id        TEXT NOT NULL PRIMARY KEY,
+    deployment_id     TEXT NOT NULL,
+    revision_hash     TEXT NOT NULL,
+    artifact_sha256   TEXT NOT NULL,
+    applied_at_utc    TEXT NOT NULL
+);
+```
+
+Both registered. All PKs are TEXT/composite (no autoincrement — the LWW-overwrite trap). Raw chunk size
+128 KiB (≈ 171 KB base64); with the rig `MaxBatchSize` pin of **16**, worst-case batch ≈ 2.7 MB, under
+the 4 MB gRPC cap. Retention: keep the newest **2** deployments per `cluster_id`; older artifact rows are
+deleted on store (deletes replicate as tombstones — fine). Reassembly verifies `chunk_count` and the
+pointer's `artifact_sha256` before deserializing; any mismatch = cache miss, never a partial apply.
+
+Both pair nodes write the same rows after applying the same deployment; identical-content rows LWW-converge
+(the ScadaBridge dual-migrate precedent — expected, not an anomaly).
+
+### 3.3 Boot-from-cache behavior (Phase 1)
+
+New seam `IDeploymentArtifactCache` (Runtime project; Runtime takes a package ref on the **core**
+`ZB.MOM.WW.LocalDb` only — the ScadaBridge SiteRuntime pattern):
+
+- **Write path:** `DriverHostActor`, immediately after a successful artifact apply, stores the artifact +
+  advances the pointer (fire-and-forget with logging; a cache-write failure never fails the apply).
+- **Read path:** at actor start, when the central fetch of the current deployment fails (SQL unreachable /
+  `Stale`), fall back to `GetCurrentAsync(clusterId)`; on hit, apply the cached artifact and surface a
+  **running-from-cache** signal (log + health degradation), mirroring the dead `StaleConfigFlag` intent.
+  `DispatchDeployment` handling is unchanged — a *new* deployment still requires central (the cache holds
+  the past, not the future). Cache miss keeps today's behavior exactly (Stale, no config).
+- The replication payoff: a standby that never applied deployment N still holds its artifact (replicated
+  from its pair peer) and can boot-from-cache with SQL down.
+
+The exact integration points inside `DriverHostActor` are pinned by a mandatory recon task at the start of
+the Phase 1 plan (with STOP conditions), because that actor's fetch/boot lifecycle was mapped from docs,
+not stepped through.
+
+### 3.4 Phase 2 — alarm-historian S&F onto LocalDb
+
+- `alarm_sf_events` table (TEXT GUID PK; exact columns derived from today's `SqliteStoreAndForwardSink`
+  schema by the Phase 2 recon task; status/attempts/dead-letter semantics preserved as columns).
+- Sink rewired onto `ILocalDb` (public seam unchanged); its private connection/pragma/file management and
+  `AlarmHistorian:DatabasePath` are deleted. `AlarmHistorian:Enabled` still gates the whole feature.
+- One-time migrator: legacy `alarm-historian.db` → consolidated file, run **after** `RegisterReplicated`
+  inside `OnReady`, `INSERT OR IGNORE`, column-intersection copy, deterministic ids for any autoincrement
+  legacy PK (`mig-{node}-{id}`), single transaction, `File.Move(path, path + ".migrated")` only after
+  commit.
+- Drain: primary-gated via the delivered-snapshot role (`PrimaryGatePolicy.ShouldServiceAsPrimary`);
+  delivered/dead-letter marks are row updates that replicate, so the standby's copy tracks drain progress.
+  At-least-once across failover, documented.
+
+### 3.5 Error handling
+
+Library defaults stand: fail-closed schema-digest handshake, per-entry dead-letter for poison rows,
+oplog caps degrade to snapshot resync (never data loss). App-side: cache-write failures are logged and
+dropped (apply already succeeded); cache-read integrity failures are a miss; the interceptor is fail-closed
+(no/empty ApiKey ⇒ `PermissionDenied` for every sync call — a key typo stops convergence, it does not
+degrade to unauthenticated).
+
+### 3.6 Observability & health
+
+- `ISyncStatus` → a `ZB.MOM.WW.Health` check (Degraded when a peer is configured but `Connected` is false,
+  or `OplogBacklog` is null/climbing), registered in the existing `AddOtOpcUaHealth` path.
+- The meter `ZB.MOM.WW.LocalDb.Replication` must be added to OtOpcUa's telemetry meter list **if
+  `AddOtOpcUaObservability` uses an allowlist** — `ZbTelemetryOptions.Meters` is a *silent* allowlist and
+  this exact omission is what the ScadaBridge live gate caught. The plan has an explicit task for it.
+
+### 3.7 Testing & definition of done
+
+- **DI-pin tests over the real `Program.cs`** via `WebApplicationFactory<Program>`
+  (Host.IntegrationTests, xunit.v3): exact replicated-table set (both directions load-bearing),
+  PK pins, `ILocalDb` singleton, default-OFF pin (`ISyncStatus.Connected == false`, `PeerNodeId` null),
+  driver-role gating (admin-only graph has no `ILocalDb`).
+- **Interceptor unit tests** with a hand-rolled fake `ServerCallContext` (`Grpc.Core.Testing` does not
+  exist on grpc-dotnet).
+- **2-node convergence harness** over a real loopback Kestrel h2c transport through the real interceptor,
+  initialized via the **production `LocalDbSetup.OnReady`** (a hand-written schema would only prove the
+  test agrees with itself) — copied from the library's `ConvergenceFixture` / ScadaBridge's
+  `LocalDbSitePairHarness`. DBs registered as pre-constructed singletons so transport teardown doesn't
+  dispose them. Real temp **files**, never `:memory:`; `SqliteConnection.ClearAllPools()` on cleanup.
+- **Positive control (DoD):** with the `RegisterReplicated` calls commented out, the convergence scenarios
+  must go red. Absence assertions get positive controls.
+- **Actor tests** (cache write/read fallback) live in an **xunit2** TestKit project — `Akka.TestKit.Xunit2`
+  cannot be used from the xunit.v3 Host.IntegrationTests project.
+- **Live gate on the docker-dev rig** (site-a pair; adapted from ScadaBridge's 10 checks): artifact
+  chunks + pointer converge byte-identical with identical HLC + origin node id; standby-that-never-fetched
+  boots from cache with SQL stopped; retention prune tombstones on both nodes; oplog rise-and-drain across
+  a transport kill; both-nodes-together restart clean; metrics present on `/metrics`; zero
+  `disk I/O error`/`SQLITE_IOERR` in logs. **All DB sampling via `cp` of the `db`/`-wal`/`-shm` triplet —
+  never host `sqlite3` against a live bind-mounted WAL DB** (the 2026-07-20 incident).
+
+## 4. Operational constraints (inherited, to be documented in OtOpcUa's runbooks)
+
+- A replicated pair must be **stopped/started together** across upgrades (no rolling-upgrade support;
+  the schema-digest handshake fails closed on any drift).
+- A node offline beyond `TombstoneRetention` (default 7 d) can resurrect deleted rows.
+- `MaxBatchSize` is a **row count** against the 4 MB gRPC default cap — the rig pins **16**; chunking keeps
+  the worst-case row ≈ 171 KB.
+- The sync `ApiKey` must be byte-identical on both nodes (fail-closed interceptor).
+- Never inspect a live bind-mounted WAL DB with host `sqlite3`; copy the triplet.
+
+## 5. Out of scope
+
+- Replicating secrets via LocalDb (Secrets keeps its own Akka replicator).
+- The FasterLog continuous-historization outbox.
+- Alarm condition / scripted-alarm state (stays central SQL).
+- N>2 topologies, an Akka transport for LocalDb sync, byte-aware batching (library-side futures).
+- Production enablement (needs a sync-ApiKey distribution decision, like the Secrets KEK one).
+
+## Appendix — ScadaBridge lessons applied
+
+| Lesson (source: ScadaBridge Phases 1–2 + live gates) | Where this design applies it |
+|---|---|
+| DDL → RegisterReplicated → migrate → writes; pre-registration rows are invisible forever, silently | §3.1 `OnReady` ordering; Phase 2 migrator runs last, pinned by an oplog assertion |
+| Autoincrement PKs are an LWW overwrite trap | All new tables TEXT/composite PKs; Phase 2 recon checks the legacy alarm PK, deterministic `mig-` ids if needed |
+| Library passive side verifies no ApiKey; `MapZbLocalDbSync()` can't carry auth on its return value | Consumer interceptor, fail-closed, ported line-for-line (§3.1) |
+| `ZbTelemetryOptions.Meters` is a silent allowlist | Explicit meter-allowlist task (§3.6) |
+| Register in the graph your tests actually build | OtOpcUa's composition root *is* `Program.cs` and `WebApplicationFactory<Program>` builds it — DI pins run against the real graph (§3.7) |
+| Row-count `MaxBatchSize` vs 4 MB gRPC cap; single row near 4 MB = STOP | Chunked artifact storage designed in (D6), rig pin 16 |
+| No rolling upgrades; stop/start pair together | §4, runbook task |
+| Tombstone-retention resurrection | §4 |
+| `CreateConnection()` returns an already-open, UDF-registered connection; no in-memory mode | Plan task notes; test harness uses temp files + `ClearAllPools` |
+| Absence assertions need a positive control; DoD greps must say "no references from *code*" | §3.7 DoD; deletion tasks in both plans phrase greps that way |
+| Never host-`sqlite3` a live WAL DB; `docker exec curl` doesn't exist in `aspnet:10.0` | Live-gate procedure (§3.7, §4); gate uses `cp` triplet + `curlimages/curl` sidecar |
+| `Grpc.Core.Testing` doesn't exist on grpc-dotnet | Interceptor tests hand-roll `ServerCallContext` |
+| Observer-induced failures: check the observer's causal path before blaming the component | Live-gate preamble in both plans |