# OtOpcUa LocalDb Adoption — Design **Date:** 2026-07-20 **Status:** **Phase 1 EXECUTED, merged + pushed** to OtOpcUa `master` 2026-07-21 (merge `c957db52`; live gate 8/8, `OtOpcUa/docs/plans/2026-07-20-localdb-phase1-live-gate.md`). Both limitations that gate left open were closed the same day (merge `5fdb5a5a`, `…-followups-live-gate.md`), which turned up a third library defect reachable only once the first was fixed. **Phase 2 is designed but NOT executed** — plan at `OtOpcUa/docs/plans/2026-07-20-localdb-adoption-phase2.md` (9 tasks). Implementation plans live in the OtOpcUa repo at `docs/plans/2026-07-20-localdb-adoption-phase1.md` and `…-phase2.md`. **Library:** `ZB.MOM.WW.LocalDb` / `.Replication` / `.Contracts` — designed against **0.1.1**; Phase 1 shipped against **0.1.3**, which carries the two rebuilt-peer replication fixes the live gates forced. **Predecessor:** ScadaBridge adoption (merged 2026-07-20, PR #23) — this design deliberately reuses its wiring pattern and every lesson from its two live gates. See §Lessons appendix. **Executor:** plans are written to be executed by Claude Opus agents (`claude --model opus` + `superpowers-extended-cc:executing-plans`). --- ## 1. Decisions (with alternatives considered) Per-decision best-practice fit noted; the recommended option is what this design adopts. | # | Decision | Chosen | Alternatives considered | |---|---|---|---| | D1 | **What OtOpcUa caches locally** | **Phase 1: the deployed-configuration artifact** (`Deployment.ArtifactBlob` + current-pointer), giving boot-from-cache when central SQL is unreachable. **Phase 2: the alarm-historian store-and-forward buffer** (today's standalone `alarm-historian.db`). | Secrets store (already has its own Akka replication — leave alone); FasterLog historization outbox (not SQLite, already crash-safe — leave alone); alarm condition state (central SQL by design, survives failover via central — defer). | | D2 | **Fate of the dormant LiteDB LocalCache** (`ILocalConfigCache`, `GenerationSealedCache`, `ResilientConfigReader` — built, unit-tested, never registered) | **Delete in Phase 1.** LocalDb supersedes it; keeping two unwired cache designs invites the next reader to wire the wrong one. The useful ideas (fail-closed cache miss, stale-config flag) are re-expressed on LocalDb. | Wire LiteDB as-is (rejected: second embedded engine, no replication, dead since Phase 6.1); back `ILocalConfigCache` with LocalDb behind the existing interface (rejected: the interface models LiteDB generations, not the artifact/pointer shape v3 actually ships). | | D3 | **Sync transport** | **Dedicated Kestrel h2c (HTTP/2-only) listener + the library's gRPC stream**, exactly like ScadaBridge's port 8083. New config key `LocalDb:SyncListenPort` (default **0 = no listener**). | Ride the Akka mesh like Secrets replication (rejected: the LocalDb sync engine is gRPC-stream-only; an Akka transport is a library fork, out of scope); multiplex on the existing `:9000` HTTP/1.1 listener (rejected: cleartext `Http1AndHttp2` cannot negotiate h2c without ALPN — prior-knowledge h2c needs an `Http2`-only listener). | | D4 | **Rollout gating** | **Storage move ships unconditionally** (LocalDb with replication off is just a local SQLite file); **replication is config-gated default-OFF** (`PeerAddress` unset, `SyncListenPort` 0) and proven on the docker-dev rig's **site-a pair only**, with site-b left unreplicated as the default-OFF pin — the ScadaBridge posture, and consistent with OtOpcUa's existing `Secrets:Replication:Enabled` default-false gate. | Big-bang enable everywhere (rejected: no production KEK-style story yet for sync ApiKey distribution; default-OFF has served the family well). | | D5 | **DB layout** | **One consolidated per-node file**, `LocalDb:Path` default `./data/otopcua-localdb.db`. The library is one-DB-per-process (`AddZbLocalDb` first-registration-wins). | Per-concern files (impossible without lib changes). | | D6 | **Artifact storage shape** | **Chunked base64 TEXT rows** (`deployment_artifacts`, PK `(deployment_id, chunk_index)`, raw chunk 128 KiB) + a `deployment_pointer` row per cluster with a whole-artifact SHA-256. Reason: `RegisterReplicated` **rejects BLOB columns**, and `MaxBatchSize` batches by **row count** against gRPC's 4 MB default cap — a single unchunked multi-MB artifact row would be undeliverable, full stop (the ScadaBridge "if any single row approaches 4 MB → STOP" rule, designed out from day one). | Single base64 row + low MaxBatchSize (rejected: caps artifact size at ~3 MB forever, silently); store artifact on disk beside the DB and replicate only metadata (rejected: reintroduces the file-vs-row consistency problem replication exists to solve). | | D7 | **Which nodes run LocalDb** | **Driver-role nodes only** (`hasDriver`), like the alarm historian. Admin-only nodes have nothing to cache; skipping them avoids the `LocalDb:Path`-required-or-no-boot constraint there. The rig's central pair is `admin,driver`, so it still gets LocalDb. | All nodes (rejected: pointless required-config on admin-only nodes). | | D8 | **Phase 2 drain ownership** | **Primary-gated drain**: only the node whose delivered-snapshot role is Primary (via `PrimaryGatePolicy`) drains the replicated alarm S&F buffer to HistorianGateway. Failover semantics: at-least-once (a failover window may re-send events whose delivered-mark hadn't replicated); accepted and documented — same class as ScadaBridge's convergent-not-identical decision. | Both nodes drain with distributed dedup (rejected: complexity; the historian has no idempotence key for `SendEvent`). | | D9 | **Bespoke-mechanism deletion** | Phase 2 **rewires `SqliteStoreAndForwardSink` onto `ILocalDb` and deletes its private file/pragma management outright** — cutover in one commit, no dual-mechanism period (the ScadaBridge Phase-2 rule: cutover *is* the test). Public seam (`IAlarmHistorianSink` or equivalent) unchanged. | Parallel-run both sinks (rejected: double-delivery to the historian, and the family precedent is clean cutover). | ## 2. Current state (inventoried 2026-07-20) | Store | File / location | Mechanism today | Replicated? | Fate | |---|---|---|---|---| | Deployed config | central SQL `Deployment.ArtifactBlob`; fetched by `DriverHostActor` on `DispatchDeployment` (DPS topic `deployments`); **no local fallback** — node goes `Stale` and ignores deploys when SQL is down | EF Core, push-then-pull | n/a (central) | **Phase 1:** cached per-pair in LocalDb; boot-from-cache when central unreachable | | LiteDB config cache | `config_cache.db` (default) — **never created in practice; code never registered** | LiteDB 5, dormant | no | **Phase 1: deleted** | | Alarm-historian S&F | `alarm-historian.db` (`AlarmHistorian:DatabasePath`), per-node | bespoke `SqliteStoreAndForwardSink`, own file/pragmas | no | **Phase 2:** table in the consolidated LocalDb, replicated, primary-gated drain | | Secrets | `otopcua-secrets.db` per node | `ZB.MOM.WW.Secrets` + optional Akka replication | when enabled | untouched | | Continuous-historization outbox | FasterLog dir (`ContinuousHistorization:OutboxPath`) | FasterLog | no | untouched | ## 3. Design ### 3.1 Shape - Host gains `AddOtOpcUaLocalDb(services, config)` (new `LocalDbRegistration` in `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/`, mirroring `SecretsRegistration`), called from `Program.cs` inside the `hasDriver` branch. It runs `AddZbLocalDb(config, LocalDbSetup.OnReady)` + `AddZbLocalDbReplication(config)` unconditionally (inert without `PeerAddress`). - `LocalDbSetup.OnReady` enforces the load-bearing order: **DDL → `RegisterReplicated` → (no migrator in Phase 1; Phase 2 adds one) → first writes**. - Sync endpoint: `AddGrpc(o => o.Interceptors.Add())` + `app.MapZbLocalDbSync()`, served on a dedicated `Http2`-only cleartext listener bound iff `LocalDb:SyncListenPort > 0`. **Because explicit `Listen*` calls make Kestrel ignore `ASPNETCORE_URLS`, the existing `:9000` HTTP/1.1 binding must be re-added explicitly in the same `ConfigureKestrel` block** — this is called out as a high-risk task with its own rig verification. - Auth: consumer-supplied fail-closed bearer interceptor (fixed-time compare, scoped to `/localdb_sync.v1.LocalDbSync/`), a line-for-line port of ScadaBridge's — the library's passive side verifies nothing. `ApiKey` production value is a `${secret:}` reference (OtOpcUa's pre-host `SecretReferenceExpander` already expands these). ### 3.2 Schema (Phase 1) ```sql CREATE TABLE IF NOT EXISTS deployment_artifacts ( deployment_id TEXT NOT NULL, chunk_index INTEGER NOT NULL, cluster_id TEXT NOT NULL, revision_hash TEXT NOT NULL, chunk_count INTEGER NOT NULL, chunk_base64 TEXT NOT NULL, cached_at_utc TEXT NOT NULL, PRIMARY KEY (deployment_id, chunk_index) ); CREATE TABLE IF NOT EXISTS deployment_pointer ( cluster_id TEXT NOT NULL PRIMARY KEY, deployment_id TEXT NOT NULL, revision_hash TEXT NOT NULL, artifact_sha256 TEXT NOT NULL, applied_at_utc TEXT NOT NULL ); ``` Both registered. All PKs are TEXT/composite (no autoincrement — the LWW-overwrite trap). Raw chunk size 128 KiB (≈ 171 KB base64); with the rig `MaxBatchSize` pin of **16**, worst-case batch ≈ 2.7 MB, under the 4 MB gRPC cap. Retention: keep the newest **2** deployments per `cluster_id`; older artifact rows are deleted on store (deletes replicate as tombstones — fine). Reassembly verifies `chunk_count` and the pointer's `artifact_sha256` before deserializing; any mismatch = cache miss, never a partial apply. Both pair nodes write the same rows after applying the same deployment; identical-content rows LWW-converge (the ScadaBridge dual-migrate precedent — expected, not an anomaly). ### 3.3 Boot-from-cache behavior (Phase 1) New seam `IDeploymentArtifactCache` (Runtime project; Runtime takes a package ref on the **core** `ZB.MOM.WW.LocalDb` only — the ScadaBridge SiteRuntime pattern): - **Write path:** `DriverHostActor`, immediately after a successful artifact apply, stores the artifact + advances the pointer (fire-and-forget with logging; a cache-write failure never fails the apply). - **Read path:** at actor start, when the central fetch of the current deployment fails (SQL unreachable / `Stale`), fall back to `GetCurrentAsync(clusterId)`; on hit, apply the cached artifact and surface a **running-from-cache** signal (log + health degradation), mirroring the dead `StaleConfigFlag` intent. `DispatchDeployment` handling is unchanged — a *new* deployment still requires central (the cache holds the past, not the future). Cache miss keeps today's behavior exactly (Stale, no config). - The replication payoff: a standby that never applied deployment N still holds its artifact (replicated from its pair peer) and can boot-from-cache with SQL down. The exact integration points inside `DriverHostActor` are pinned by a mandatory recon task at the start of the Phase 1 plan (with STOP conditions), because that actor's fetch/boot lifecycle was mapped from docs, not stepped through. ### 3.4 Phase 2 — alarm-historian S&F onto LocalDb - `alarm_sf_events` table (TEXT GUID PK; exact columns derived from today's `SqliteStoreAndForwardSink` schema by the Phase 2 recon task; status/attempts/dead-letter semantics preserved as columns). - Sink rewired onto `ILocalDb` (public seam unchanged); its private connection/pragma/file management and `AlarmHistorian:DatabasePath` are deleted. `AlarmHistorian:Enabled` still gates the whole feature. - One-time migrator: legacy `alarm-historian.db` → consolidated file, run **after** `RegisterReplicated` inside `OnReady`, `INSERT OR IGNORE`, column-intersection copy, deterministic ids for any autoincrement legacy PK (`mig-{node}-{id}`), single transaction, `File.Move(path, path + ".migrated")` only after commit. - Drain: primary-gated via the delivered-snapshot role (`PrimaryGatePolicy.ShouldServiceAsPrimary`); delivered/dead-letter marks are row updates that replicate, so the standby's copy tracks drain progress. At-least-once across failover, documented. ### 3.5 Error handling Library defaults stand: fail-closed schema-digest handshake, per-entry dead-letter for poison rows, oplog caps degrade to snapshot resync (never data loss). App-side: cache-write failures are logged and dropped (apply already succeeded); cache-read integrity failures are a miss; the interceptor is fail-closed (no/empty ApiKey ⇒ `PermissionDenied` for every sync call — a key typo stops convergence, it does not degrade to unauthenticated). ### 3.6 Observability & health - `ISyncStatus` → a `ZB.MOM.WW.Health` check (Degraded when a peer is configured but `Connected` is false, or `OplogBacklog` is null/climbing), registered in the existing `AddOtOpcUaHealth` path. - The meter `ZB.MOM.WW.LocalDb.Replication` must be added to OtOpcUa's telemetry meter list **if `AddOtOpcUaObservability` uses an allowlist** — `ZbTelemetryOptions.Meters` is a *silent* allowlist and this exact omission is what the ScadaBridge live gate caught. The plan has an explicit task for it. ### 3.7 Testing & definition of done - **DI-pin tests over the real `Program.cs`** via `WebApplicationFactory` (Host.IntegrationTests, xunit.v3): exact replicated-table set (both directions load-bearing), PK pins, `ILocalDb` singleton, default-OFF pin (`ISyncStatus.Connected == false`, `PeerNodeId` null), driver-role gating (admin-only graph has no `ILocalDb`). - **Interceptor unit tests** with a hand-rolled fake `ServerCallContext` (`Grpc.Core.Testing` does not exist on grpc-dotnet). - **2-node convergence harness** over a real loopback Kestrel h2c transport through the real interceptor, initialized via the **production `LocalDbSetup.OnReady`** (a hand-written schema would only prove the test agrees with itself) — copied from the library's `ConvergenceFixture` / ScadaBridge's `LocalDbSitePairHarness`. DBs registered as pre-constructed singletons so transport teardown doesn't dispose them. Real temp **files**, never `:memory:`; `SqliteConnection.ClearAllPools()` on cleanup. - **Positive control (DoD):** with the `RegisterReplicated` calls commented out, the convergence scenarios must go red. Absence assertions get positive controls. - **Actor tests** (cache write/read fallback) live in an **xunit2** TestKit project — `Akka.TestKit.Xunit2` cannot be used from the xunit.v3 Host.IntegrationTests project. - **Live gate on the docker-dev rig** (site-a pair; adapted from ScadaBridge's 10 checks): artifact chunks + pointer converge byte-identical with identical HLC + origin node id; standby-that-never-fetched boots from cache with SQL stopped; retention prune tombstones on both nodes; oplog rise-and-drain across a transport kill; both-nodes-together restart clean; metrics present on `/metrics`; zero `disk I/O error`/`SQLITE_IOERR` in logs. **All DB sampling via `cp` of the `db`/`-wal`/`-shm` triplet — never host `sqlite3` against a live bind-mounted WAL DB** (the 2026-07-20 incident). ## 4. Operational constraints (inherited, to be documented in OtOpcUa's runbooks) - A replicated pair must be **stopped/started together** across upgrades (no rolling-upgrade support; the schema-digest handshake fails closed on any drift). - A node offline beyond `TombstoneRetention` (default 7 d) can resurrect deleted rows. - `MaxBatchSize` is a **row count** against the 4 MB gRPC default cap — the rig pins **16**; chunking keeps the worst-case row ≈ 171 KB. - The sync `ApiKey` must be byte-identical on both nodes (fail-closed interceptor). - Never inspect a live bind-mounted WAL DB with host `sqlite3`; copy the triplet. ## 5. Out of scope - Replicating secrets via LocalDb (Secrets keeps its own Akka replicator). - The FasterLog continuous-historization outbox. - Alarm condition / scripted-alarm state (stays central SQL). - N>2 topologies, an Akka transport for LocalDb sync, byte-aware batching (library-side futures). - Production enablement (needs a sync-ApiKey distribution decision, like the Secrets KEK one). ## Appendix — ScadaBridge lessons applied | Lesson (source: ScadaBridge Phases 1–2 + live gates) | Where this design applies it | |---|---| | DDL → RegisterReplicated → migrate → writes; pre-registration rows are invisible forever, silently | §3.1 `OnReady` ordering; Phase 2 migrator runs last, pinned by an oplog assertion | | Autoincrement PKs are an LWW overwrite trap | All new tables TEXT/composite PKs; Phase 2 recon checks the legacy alarm PK, deterministic `mig-` ids if needed | | Library passive side verifies no ApiKey; `MapZbLocalDbSync()` can't carry auth on its return value | Consumer interceptor, fail-closed, ported line-for-line (§3.1) | | `ZbTelemetryOptions.Meters` is a silent allowlist | Explicit meter-allowlist task (§3.6) | | Register in the graph your tests actually build | OtOpcUa's composition root *is* `Program.cs` and `WebApplicationFactory` builds it — DI pins run against the real graph (§3.7) | | Row-count `MaxBatchSize` vs 4 MB gRPC cap; single row near 4 MB = STOP | Chunked artifact storage designed in (D6), rig pin 16 | | No rolling upgrades; stop/start pair together | §4, runbook task | | Tombstone-retention resurrection | §4 | | `CreateConnection()` returns an already-open, UDF-registered connection; no in-memory mode | Plan task notes; test harness uses temp files + `ClearAllPools` | | Absence assertions need a positive control; DoD greps must say "no references from *code*" | §3.7 DoD; deletion tasks in both plans phrase greps that way | | Never host-`sqlite3` a live WAL DB; `docker exec curl` doesn't exist in `aspnet:10.0` | Live-gate procedure (§3.7, §4); gate uses `cp` triplet + `curlimages/curl` sidecar | | `Grpc.Core.Testing` doesn't exist on grpc-dotnet | Interceptor tests hand-roll `ServerCallContext` | | Observer-induced failures: check the observer's causal path before blaming the component | Live-gate preamble in both plans |