scadaproj/docs/plans/2026-07-20-otopcua-localdb-design.md

# OtOpcUa LocalDb Adoption — Design

**Date:** 2026-07-20
**Status:** **Phase 1 EXECUTED, merged + pushed** to OtOpcUa `master` 2026-07-21 (merge `c957db52`;
live gate 8/8, `OtOpcUa/docs/plans/2026-07-20-localdb-phase1-live-gate.md`). Both limitations that gate
left open were closed the same day (merge `5fdb5a5a`, `…-followups-live-gate.md`), which turned up a
third library defect reachable only once the first was fixed. **Phase 2 EXECUTED — correction 2026-08-07:**
Phase 2 (alarm store-and-forward on LocalDb) was executed, live-gated and **merged to OtOpcUa `master`**
(merge `06b5d81b`, "Merge feat/localdb-phase2: alarm store-and-forward on LocalDb + live-gate fixes";
source: `src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/AlarmSfSchema.cs` + `LocalDbStoreAndForwardSink.cs`).
The "designed but NOT executed" claim that previously stood here was stale. Residual: live-gate checks 3+4
were never re-run on the post-mesh-Phase-6 topology.
Plan at `OtOpcUa/docs/plans/2026-07-20-localdb-adoption-phase2.md` (9 tasks).
Implementation plans live in the OtOpcUa repo at
`docs/plans/2026-07-20-localdb-adoption-phase1.md` and `…-phase2.md`.
**Library:** `ZB.MOM.WW.LocalDb` / `.Replication` / `.Contracts` — designed against **0.1.1**; Phase 1
shipped against **0.1.3**, which carries the two rebuilt-peer replication fixes the live gates forced.
**Predecessor:** ScadaBridge adoption (merged 2026-07-20, PR #23) — this design deliberately reuses its
wiring pattern and every lesson from its two live gates. See §Lessons appendix.
**Executor:** plans are written to be executed by Claude Opus agents
(`claude --model opus` + `superpowers-extended-cc:executing-plans`).

---

## 1. Decisions (with alternatives considered)

Per-decision best-practice fit noted; the recommended option is what this design adopts.

| # | Decision | Chosen | Alternatives considered |
|---|---|---|---|
| D1 | **What OtOpcUa caches locally** | **Phase 1: the deployed-configuration artifact** (`Deployment.ArtifactBlob` + current-pointer), giving boot-from-cache when central SQL is unreachable. **Phase 2: the alarm-historian store-and-forward buffer** (today's standalone `alarm-historian.db`). | Secrets store (already has its own Akka replication — leave alone); FasterLog historization outbox (not SQLite, already crash-safe — leave alone); alarm condition state (central SQL by design, survives failover via central — defer). |
| D2 | **Fate of the dormant LiteDB LocalCache** (`ILocalConfigCache`, `GenerationSealedCache`, `ResilientConfigReader` — built, unit-tested, never registered) | **Delete in Phase 1.** LocalDb supersedes it; keeping two unwired cache designs invites the next reader to wire the wrong one. The useful ideas (fail-closed cache miss, stale-config flag) are re-expressed on LocalDb. | Wire LiteDB as-is (rejected: second embedded engine, no replication, dead since Phase 6.1); back `ILocalConfigCache` with LocalDb behind the existing interface (rejected: the interface models LiteDB generations, not the artifact/pointer shape v3 actually ships). |
| D3 | **Sync transport** | **Dedicated Kestrel h2c (HTTP/2-only) listener + the library's gRPC stream**, exactly like ScadaBridge's port 8083. New config key `LocalDb:SyncListenPort` (default **0 = no listener**). | Ride the Akka mesh like Secrets replication (rejected: the LocalDb sync engine is gRPC-stream-only; an Akka transport is a library fork, out of scope); multiplex on the existing `:9000` HTTP/1.1 listener (rejected: cleartext `Http1AndHttp2` cannot negotiate h2c without ALPN — prior-knowledge h2c needs an `Http2`-only listener). |
| D4 | **Rollout gating** | **Storage move ships unconditionally** (LocalDb with replication off is just a local SQLite file); **replication is config-gated default-OFF** (`PeerAddress` unset, `SyncListenPort` 0) and proven on the docker-dev rig's **site-a pair only**, with site-b left unreplicated as the default-OFF pin — the ScadaBridge posture, and consistent with OtOpcUa's existing `Secrets:Replication:Enabled` default-false gate. | Big-bang enable everywhere (rejected: no production KEK-style story yet for sync ApiKey distribution; default-OFF has served the family well). |
| D5 | **DB layout** | **One consolidated per-node file**, `LocalDb:Path` default `./data/otopcua-localdb.db`. The library is one-DB-per-process (`AddZbLocalDb` first-registration-wins). | Per-concern files (impossible without lib changes). |
| D6 | **Artifact storage shape** | **Chunked base64 TEXT rows** (`deployment_artifacts`, PK `(deployment_id, chunk_index)`, raw chunk 128 KiB) + a `deployment_pointer` row per cluster with a whole-artifact SHA-256. Reason: `RegisterReplicated` **rejects BLOB columns**, and `MaxBatchSize` batches by **row count** against gRPC's 4 MB default cap — a single unchunked multi-MB artifact row would be undeliverable, full stop (the ScadaBridge "if any single row approaches 4 MB → STOP" rule, designed out from day one). | Single base64 row + low MaxBatchSize (rejected: caps artifact size at ~3 MB forever, silently); store artifact on disk beside the DB and replicate only metadata (rejected: reintroduces the file-vs-row consistency problem replication exists to solve). |
| D7 | **Which nodes run LocalDb** | **Driver-role nodes only** (`hasDriver`), like the alarm historian. Admin-only nodes have nothing to cache; skipping them avoids the `LocalDb:Path`-required-or-no-boot constraint there. The rig's central pair is `admin,driver`, so it still gets LocalDb. | All nodes (rejected: pointless required-config on admin-only nodes). |
| D8 | **Phase 2 drain ownership** | **Primary-gated drain**: only the node whose delivered-snapshot role is Primary (via `PrimaryGatePolicy`) drains the replicated alarm S&F buffer to HistorianGateway. Failover semantics: at-least-once (a failover window may re-send events whose delivered-mark hadn't replicated); accepted and documented — same class as ScadaBridge's convergent-not-identical decision. | Both nodes drain with distributed dedup (rejected: complexity; the historian has no idempotence key for `SendEvent`). |
| D9 | **Bespoke-mechanism deletion** | Phase 2 **rewires `SqliteStoreAndForwardSink` onto `ILocalDb` and deletes its private file/pragma management outright** — cutover in one commit, no dual-mechanism period (the ScadaBridge Phase-2 rule: cutover *is* the test). Public seam (`IAlarmHistorianSink` or equivalent) unchanged. | Parallel-run both sinks (rejected: double-delivery to the historian, and the family precedent is clean cutover). |

## 2. Current state (inventoried 2026-07-20)

| Store | File / location | Mechanism today | Replicated? | Fate |
|---|---|---|---|---|
| Deployed config | central SQL `Deployment.ArtifactBlob`; fetched by `DriverHostActor` on `DispatchDeployment` (DPS topic `deployments`); **no local fallback** — node goes `Stale` and ignores deploys when SQL is down | EF Core, push-then-pull | n/a (central) | **Phase 1:** cached per-pair in LocalDb; boot-from-cache when central unreachable |
| LiteDB config cache | `config_cache.db` (default) — **never created in practice; code never registered** | LiteDB 5, dormant | no | **Phase 1: deleted** |
| Alarm-historian S&F | `alarm-historian.db` (`AlarmHistorian:DatabasePath`), per-node | bespoke `SqliteStoreAndForwardSink`, own file/pragmas | no | **Phase 2:** table in the consolidated LocalDb, replicated, primary-gated drain |
| Secrets | `otopcua-secrets.db` per node | `ZB.MOM.WW.Secrets` + optional Akka replication | when enabled | untouched |
| Continuous-historization outbox | FasterLog dir (`ContinuousHistorization:OutboxPath`) | FasterLog | no | untouched |

## 3. Design

### 3.1 Shape

- Host gains `AddOtOpcUaLocalDb(services, config)` (new `LocalDbRegistration` in
  `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/`, mirroring `SecretsRegistration`), called from
  `Program.cs` inside the `hasDriver` branch. It runs `AddZbLocalDb(config, LocalDbSetup.OnReady)` +
  `AddZbLocalDbReplication(config)` unconditionally (inert without `PeerAddress`).
- `LocalDbSetup.OnReady` enforces the load-bearing order: **DDL → `RegisterReplicated` → (no migrator in
  Phase 1; Phase 2 adds one) → first writes**.
- Sync endpoint: `AddGrpc(o => o.Interceptors.Add<LocalDbSyncAuthInterceptor>())` +
  `app.MapZbLocalDbSync()`, served on a dedicated `Http2`-only cleartext listener bound iff
  `LocalDb:SyncListenPort > 0`. **Because explicit `Listen*` calls make Kestrel ignore
  `ASPNETCORE_URLS`, the existing `:9000` HTTP/1.1 binding must be re-added explicitly in the same
  `ConfigureKestrel` block** — this is called out as a high-risk task with its own rig verification.
- Auth: consumer-supplied fail-closed bearer interceptor (fixed-time compare, scoped to
  `/localdb_sync.v1.LocalDbSync/`), a line-for-line port of ScadaBridge's — the library's passive side
  verifies nothing. `ApiKey` production value is a `${secret:}` reference (OtOpcUa's pre-host
  `SecretReferenceExpander` already expands these).

### 3.2 Schema (Phase 1)

```sql
CREATE TABLE IF NOT EXISTS deployment_artifacts (
    deployment_id   TEXT    NOT NULL,
    chunk_index     INTEGER NOT NULL,
    cluster_id      TEXT    NOT NULL,
    revision_hash   TEXT    NOT NULL,
    chunk_count     INTEGER NOT NULL,
    chunk_base64    TEXT    NOT NULL,
    cached_at_utc   TEXT    NOT NULL,
    PRIMARY KEY (deployment_id, chunk_index)
);
CREATE TABLE IF NOT EXISTS deployment_pointer (
    cluster_id        TEXT NOT NULL PRIMARY KEY,
    deployment_id     TEXT NOT NULL,
    revision_hash     TEXT NOT NULL,
    artifact_sha256   TEXT NOT NULL,
    applied_at_utc    TEXT NOT NULL
);
```

Both registered. All PKs are TEXT/composite (no autoincrement — the LWW-overwrite trap). Raw chunk size
128 KiB (≈ 171 KB base64); with the rig `MaxBatchSize` pin of **16**, worst-case batch ≈ 2.7 MB, under
the 4 MB gRPC cap. Retention: keep the newest **2** deployments per `cluster_id`; older artifact rows are
deleted on store (deletes replicate as tombstones — fine). Reassembly verifies `chunk_count` and the
pointer's `artifact_sha256` before deserializing; any mismatch = cache miss, never a partial apply.

Both pair nodes write the same rows after applying the same deployment; identical-content rows LWW-converge
(the ScadaBridge dual-migrate precedent — expected, not an anomaly).

### 3.3 Boot-from-cache behavior (Phase 1)

New seam `IDeploymentArtifactCache` (Runtime project; Runtime takes a package ref on the **core**
`ZB.MOM.WW.LocalDb` only — the ScadaBridge SiteRuntime pattern):

- **Write path:** `DriverHostActor`, immediately after a successful artifact apply, stores the artifact +
  advances the pointer (fire-and-forget with logging; a cache-write failure never fails the apply).
- **Read path:** at actor start, when the central fetch of the current deployment fails (SQL unreachable /
  `Stale`), fall back to `GetCurrentAsync(clusterId)`; on hit, apply the cached artifact and surface a
  **running-from-cache** signal (log + health degradation), mirroring the dead `StaleConfigFlag` intent.
  `DispatchDeployment` handling is unchanged — a *new* deployment still requires central (the cache holds
  the past, not the future). Cache miss keeps today's behavior exactly (Stale, no config).
- The replication payoff: a standby that never applied deployment N still holds its artifact (replicated
  from its pair peer) and can boot-from-cache with SQL down.

The exact integration points inside `DriverHostActor` are pinned by a mandatory recon task at the start of
the Phase 1 plan (with STOP conditions), because that actor's fetch/boot lifecycle was mapped from docs,
not stepped through.

### 3.4 Phase 2 — alarm-historian S&F onto LocalDb

- `alarm_sf_events` table (TEXT GUID PK; exact columns derived from today's `SqliteStoreAndForwardSink`
  schema by the Phase 2 recon task; status/attempts/dead-letter semantics preserved as columns).
- Sink rewired onto `ILocalDb` (public seam unchanged); its private connection/pragma/file management and
  `AlarmHistorian:DatabasePath` are deleted. `AlarmHistorian:Enabled` still gates the whole feature.
- One-time migrator: legacy `alarm-historian.db` → consolidated file, run **after** `RegisterReplicated`
  inside `OnReady`, `INSERT OR IGNORE`, column-intersection copy, deterministic ids for any autoincrement
  legacy PK (`mig-{node}-{id}`), single transaction, `File.Move(path, path + ".migrated")` only after
  commit.
- Drain: primary-gated via the delivered-snapshot role (`PrimaryGatePolicy.ShouldServiceAsPrimary`);
  delivered/dead-letter marks are row updates that replicate, so the standby's copy tracks drain progress.
  At-least-once across failover, documented.

### 3.5 Error handling

Library defaults stand: fail-closed schema-digest handshake, per-entry dead-letter for poison rows,
oplog caps degrade to snapshot resync (never data loss). App-side: cache-write failures are logged and
dropped (apply already succeeded); cache-read integrity failures are a miss; the interceptor is fail-closed
(no/empty ApiKey ⇒ `PermissionDenied` for every sync call — a key typo stops convergence, it does not
degrade to unauthenticated).

### 3.6 Observability & health

- `ISyncStatus` → a `ZB.MOM.WW.Health` check (Degraded when a peer is configured but `Connected` is false,
  or `OplogBacklog` is null/climbing), registered in the existing `AddOtOpcUaHealth` path.
- The meter `ZB.MOM.WW.LocalDb.Replication` must be added to OtOpcUa's telemetry meter list **if
  `AddOtOpcUaObservability` uses an allowlist** — `ZbTelemetryOptions.Meters` is a *silent* allowlist and
  this exact omission is what the ScadaBridge live gate caught. The plan has an explicit task for it.

### 3.7 Testing & definition of done

- **DI-pin tests over the real `Program.cs`** via `WebApplicationFactory<Program>`
  (Host.IntegrationTests, xunit.v3): exact replicated-table set (both directions load-bearing),
  PK pins, `ILocalDb` singleton, default-OFF pin (`ISyncStatus.Connected == false`, `PeerNodeId` null),
  driver-role gating (admin-only graph has no `ILocalDb`).
- **Interceptor unit tests** with a hand-rolled fake `ServerCallContext` (`Grpc.Core.Testing` does not
  exist on grpc-dotnet).
- **2-node convergence harness** over a real loopback Kestrel h2c transport through the real interceptor,
  initialized via the **production `LocalDbSetup.OnReady`** (a hand-written schema would only prove the
  test agrees with itself) — copied from the library's `ConvergenceFixture` / ScadaBridge's
  `LocalDbSitePairHarness`. DBs registered as pre-constructed singletons so transport teardown doesn't
  dispose them. Real temp **files**, never `:memory:`; `SqliteConnection.ClearAllPools()` on cleanup.
- **Positive control (DoD):** with the `RegisterReplicated` calls commented out, the convergence scenarios
  must go red. Absence assertions get positive controls.
- **Actor tests** (cache write/read fallback) live in an **xunit2** TestKit project — `Akka.TestKit.Xunit2`
  cannot be used from the xunit.v3 Host.IntegrationTests project.
- **Live gate on the docker-dev rig** (site-a pair; adapted from ScadaBridge's 10 checks): artifact
  chunks + pointer converge byte-identical with identical HLC + origin node id; standby-that-never-fetched
  boots from cache with SQL stopped; retention prune tombstones on both nodes; oplog rise-and-drain across
  a transport kill; both-nodes-together restart clean; metrics present on `/metrics`; zero
  `disk I/O error`/`SQLITE_IOERR` in logs. **All DB sampling via `cp` of the `db`/`-wal`/`-shm` triplet —
  never host `sqlite3` against a live bind-mounted WAL DB** (the 2026-07-20 incident).

## 4. Operational constraints (inherited, to be documented in OtOpcUa's runbooks)

- A replicated pair must be **stopped/started together** across upgrades (no rolling-upgrade support;
  the schema-digest handshake fails closed on any drift).
- A node offline beyond `TombstoneRetention` (default 7 d) can resurrect deleted rows.
- `MaxBatchSize` is a **row count** against the 4 MB gRPC default cap — the rig pins **16**; chunking keeps
  the worst-case row ≈ 171 KB.
- The sync `ApiKey` must be byte-identical on both nodes (fail-closed interceptor).
- Never inspect a live bind-mounted WAL DB with host `sqlite3`; copy the triplet.

## 5. Out of scope

- Replicating secrets via LocalDb (Secrets keeps its own Akka replicator).
- The FasterLog continuous-historization outbox.
- Alarm condition / scripted-alarm state (stays central SQL).
- N>2 topologies, an Akka transport for LocalDb sync, byte-aware batching (library-side futures).
- Production enablement (needs a sync-ApiKey distribution decision, like the Secrets KEK one).

## Appendix — ScadaBridge lessons applied

| Lesson (source: ScadaBridge Phases 1–2 + live gates) | Where this design applies it |
|---|---|
| DDL → RegisterReplicated → migrate → writes; pre-registration rows are invisible forever, silently | §3.1 `OnReady` ordering; Phase 2 migrator runs last, pinned by an oplog assertion |
| Autoincrement PKs are an LWW overwrite trap | All new tables TEXT/composite PKs; Phase 2 recon checks the legacy alarm PK, deterministic `mig-` ids if needed |
| Library passive side verifies no ApiKey; `MapZbLocalDbSync()` can't carry auth on its return value | Consumer interceptor, fail-closed, ported line-for-line (§3.1) |
| `ZbTelemetryOptions.Meters` is a silent allowlist | Explicit meter-allowlist task (§3.6) |
| Register in the graph your tests actually build | OtOpcUa's composition root *is* `Program.cs` and `WebApplicationFactory<Program>` builds it — DI pins run against the real graph (§3.7) |
| Row-count `MaxBatchSize` vs 4 MB gRPC cap; single row near 4 MB = STOP | Chunked artifact storage designed in (D6), rig pin 16 |
| No rolling upgrades; stop/start pair together | §4, runbook task |
| Tombstone-retention resurrection | §4 |
| `CreateConnection()` returns an already-open, UDF-registered connection; no in-memory mode | Plan task notes; test harness uses temp files + `ClearAllPools` |
| Absence assertions need a positive control; DoD greps must say "no references from *code*" | §3.7 DoD; deletion tasks in both plans phrase greps that way |
| Never host-`sqlite3` a live WAL DB; `docker exec curl` doesn't exist in `aspnet:10.0` | Live-gate procedure (§3.7, §4); gate uses `cp` triplet + `curlimages/curl` sidecar |
| `Grpc.Core.Testing` doesn't exist on grpc-dotnet | Interceptor tests hand-roll `ServerCallContext` |
| Observer-induced failures: check the observer's causal path before blaming the component | Live-gate preamble in both plans |