scadaproj/docs/plans/2026-07-20-otopcua-localdb-design.md

OtOpcUa LocalDb Adoption — Design

Date: 2026-07-20 Status: Phase 1 EXECUTED, merged + pushed to OtOpcUa master 2026-07-21 (merge c957db52; live gate 8/8, OtOpcUa/docs/plans/2026-07-20-localdb-phase1-live-gate.md). Both limitations that gate left open were closed the same day (merge 5fdb5a5a, …-followups-live-gate.md), which turned up a third library defect reachable only once the first was fixed. Phase 2 EXECUTED — correction 2026-08-07: Phase 2 (alarm store-and-forward on LocalDb) was executed, live-gated and merged to OtOpcUa master (merge 06b5d81b, "Merge feat/localdb-phase2: alarm store-and-forward on LocalDb + live-gate fixes"; source: src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/AlarmSfSchema.cs + LocalDbStoreAndForwardSink.cs). The "designed but NOT executed" claim that previously stood here was stale. Residual: live-gate checks 3+4 were never re-run on the post-mesh-Phase-6 topology. Plan at OtOpcUa/docs/plans/2026-07-20-localdb-adoption-phase2.md (9 tasks). Implementation plans live in the OtOpcUa repo at docs/plans/2026-07-20-localdb-adoption-phase1.md and …-phase2.md. Library: ZB.MOM.WW.LocalDb / .Replication / .Contracts — designed against 0.1.1; Phase 1 shipped against 0.1.3, which carries the two rebuilt-peer replication fixes the live gates forced. Predecessor: ScadaBridge adoption (merged 2026-07-20, PR #23) — this design deliberately reuses its wiring pattern and every lesson from its two live gates. See §Lessons appendix. Executor: plans are written to be executed by Claude Opus agents (claude --model opus + superpowers-extended-cc:executing-plans).

---

1. Decisions (with alternatives considered)

Per-decision best-practice fit noted; the recommended option is what this design adopts.

#	Decision	Chosen	Alternatives considered
D1	What OtOpcUa caches locally	Phase 1: the deployed-configuration artifact (Deployment.ArtifactBlob + current-pointer), giving boot-from-cache when central SQL is unreachable. Phase 2: the alarm-historian store-and-forward buffer (today's standalone alarm-historian.db).	Secrets store (already has its own Akka replication — leave alone); FasterLog historization outbox (not SQLite, already crash-safe — leave alone); alarm condition state (central SQL by design, survives failover via central — defer).
D2	Fate of the dormant LiteDB LocalCache (ILocalConfigCache, GenerationSealedCache, ResilientConfigReader — built, unit-tested, never registered)	Delete in Phase 1. LocalDb supersedes it; keeping two unwired cache designs invites the next reader to wire the wrong one. The useful ideas (fail-closed cache miss, stale-config flag) are re-expressed on LocalDb.	Wire LiteDB as-is (rejected: second embedded engine, no replication, dead since Phase 6.1); back ILocalConfigCache with LocalDb behind the existing interface (rejected: the interface models LiteDB generations, not the artifact/pointer shape v3 actually ships).
D3	Sync transport	Dedicated Kestrel h2c (HTTP/2-only) listener + the library's gRPC stream, exactly like ScadaBridge's port 8083. New config key LocalDb:SyncListenPort (default 0 = no listener).	Ride the Akka mesh like Secrets replication (rejected: the LocalDb sync engine is gRPC-stream-only; an Akka transport is a library fork, out of scope); multiplex on the existing :9000 HTTP/1.1 listener (rejected: cleartext Http1AndHttp2 cannot negotiate h2c without ALPN — prior-knowledge h2c needs an Http2-only listener).
D4	Rollout gating	Storage move ships unconditionally (LocalDb with replication off is just a local SQLite file); replication is config-gated default-OFF (PeerAddress unset, SyncListenPort 0) and proven on the docker-dev rig's site-a pair only, with site-b left unreplicated as the default-OFF pin — the ScadaBridge posture, and consistent with OtOpcUa's existing Secrets:Replication:Enabled default-false gate.	Big-bang enable everywhere (rejected: no production KEK-style story yet for sync ApiKey distribution; default-OFF has served the family well).
D5	DB layout	One consolidated per-node file, LocalDb:Path default ./data/otopcua-localdb.db. The library is one-DB-per-process (AddZbLocalDb first-registration-wins).	Per-concern files (impossible without lib changes).
D6	Artifact storage shape	Chunked base64 TEXT rows (deployment_artifacts, PK (deployment_id, chunk_index), raw chunk 128 KiB) + a deployment_pointer row per cluster with a whole-artifact SHA-256. Reason: RegisterReplicated rejects BLOB columns, and MaxBatchSize batches by row count against gRPC's 4 MB default cap — a single unchunked multi-MB artifact row would be undeliverable, full stop (the ScadaBridge "if any single row approaches 4 MB → STOP" rule, designed out from day one).	Single base64 row + low MaxBatchSize (rejected: caps artifact size at ~3 MB forever, silently); store artifact on disk beside the DB and replicate only metadata (rejected: reintroduces the file-vs-row consistency problem replication exists to solve).
D7	Which nodes run LocalDb	Driver-role nodes only (hasDriver), like the alarm historian. Admin-only nodes have nothing to cache; skipping them avoids the LocalDb:Path-required-or-no-boot constraint there. The rig's central pair is admin,driver, so it still gets LocalDb.	All nodes (rejected: pointless required-config on admin-only nodes).
D8	Phase 2 drain ownership	Primary-gated drain: only the node whose delivered-snapshot role is Primary (via PrimaryGatePolicy) drains the replicated alarm S&F buffer to HistorianGateway. Failover semantics: at-least-once (a failover window may re-send events whose delivered-mark hadn't replicated); accepted and documented — same class as ScadaBridge's convergent-not-identical decision.	Both nodes drain with distributed dedup (rejected: complexity; the historian has no idempotence key for SendEvent).
D9	Bespoke-mechanism deletion	Phase 2 rewires SqliteStoreAndForwardSink onto ILocalDb and deletes its private file/pragma management outright — cutover in one commit, no dual-mechanism period (the ScadaBridge Phase-2 rule: cutover is the test). Public seam (IAlarmHistorianSink or equivalent) unchanged.	Parallel-run both sinks (rejected: double-delivery to the historian, and the family precedent is clean cutover).

2. Current state (inventoried 2026-07-20)

Store	File / location	Mechanism today	Replicated?	Fate
Deployed config	central SQL Deployment.ArtifactBlob; fetched by DriverHostActor on DispatchDeployment (DPS topic deployments); no local fallback — node goes Stale and ignores deploys when SQL is down	EF Core, push-then-pull	n/a (central)	Phase 1: cached per-pair in LocalDb; boot-from-cache when central unreachable
LiteDB config cache	config_cache.db (default) — never created in practice; code never registered	LiteDB 5, dormant	no	Phase 1: deleted
Alarm-historian S&F	alarm-historian.db (AlarmHistorian:DatabasePath), per-node	bespoke SqliteStoreAndForwardSink, own file/pragmas	no	Phase 2: table in the consolidated LocalDb, replicated, primary-gated drain
Secrets	otopcua-secrets.db per node	ZB.MOM.WW.Secrets + optional Akka replication	when enabled	untouched
Continuous-historization outbox	FasterLog dir (ContinuousHistorization:OutboxPath)	FasterLog	no	untouched

3. Design

3.1 Shape

• Host gains AddOtOpcUaLocalDb(services, config) (new LocalDbRegistration in src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/, mirroring SecretsRegistration), called from Program.cs inside the hasDriver branch. It runs AddZbLocalDb(config, LocalDbSetup.OnReady) + AddZbLocalDbReplication(config) unconditionally (inert without PeerAddress).
• LocalDbSetup.OnReady enforces the load-bearing order: DDL → RegisterReplicated → (no migrator in Phase 1; Phase 2 adds one) → first writes.
• Sync endpoint: AddGrpc(o => o.Interceptors.Add<LocalDbSyncAuthInterceptor>()) + app.MapZbLocalDbSync(), served on a dedicated Http2-only cleartext listener bound iff LocalDb:SyncListenPort > 0. Because explicit Listen* calls make Kestrel ignore ASPNETCORE_URLS, the existing :9000 HTTP/1.1 binding must be re-added explicitly in the same ConfigureKestrel block — this is called out as a high-risk task with its own rig verification.
• Auth: consumer-supplied fail-closed bearer interceptor (fixed-time compare, scoped to /localdb_sync.v1.LocalDbSync/), a line-for-line port of ScadaBridge's — the library's passive side verifies nothing. ApiKey production value is a ${secret:} reference (OtOpcUa's pre-host SecretReferenceExpander already expands these).

3.2 Schema (Phase 1)

CREATE TABLE IF NOT EXISTS deployment_artifacts (
    deployment_id   TEXT    NOT NULL,
    chunk_index     INTEGER NOT NULL,
    cluster_id      TEXT    NOT NULL,
    revision_hash   TEXT    NOT NULL,
    chunk_count     INTEGER NOT NULL,
    chunk_base64    TEXT    NOT NULL,
    cached_at_utc   TEXT    NOT NULL,
    PRIMARY KEY (deployment_id, chunk_index)
);
CREATE TABLE IF NOT EXISTS deployment_pointer (
    cluster_id        TEXT NOT NULL PRIMARY KEY,
    deployment_id     TEXT NOT NULL,
    revision_hash     TEXT NOT NULL,
    artifact_sha256   TEXT NOT NULL,
    applied_at_utc    TEXT NOT NULL
);

Both registered. All PKs are TEXT/composite (no autoincrement — the LWW-overwrite trap). Raw chunk size 128 KiB (≈ 171 KB base64); with the rig MaxBatchSize pin of 16, worst-case batch ≈ 2.7 MB, under the 4 MB gRPC cap. Retention: keep the newest 2 deployments per cluster_id; older artifact rows are deleted on store (deletes replicate as tombstones — fine). Reassembly verifies chunk_count and the pointer's artifact_sha256 before deserializing; any mismatch = cache miss, never a partial apply.

Both pair nodes write the same rows after applying the same deployment; identical-content rows LWW-converge (the ScadaBridge dual-migrate precedent — expected, not an anomaly).

3.3 Boot-from-cache behavior (Phase 1)

New seam IDeploymentArtifactCache (Runtime project; Runtime takes a package ref on the core ZB.MOM.WW.LocalDb only — the ScadaBridge SiteRuntime pattern):

• Write path: DriverHostActor, immediately after a successful artifact apply, stores the artifact + advances the pointer (fire-and-forget with logging; a cache-write failure never fails the apply).
• Read path: at actor start, when the central fetch of the current deployment fails (SQL unreachable / Stale), fall back to GetCurrentAsync(clusterId); on hit, apply the cached artifact and surface a running-from-cache signal (log + health degradation), mirroring the dead StaleConfigFlag intent. DispatchDeployment handling is unchanged — a new deployment still requires central (the cache holds the past, not the future). Cache miss keeps today's behavior exactly (Stale, no config).
• The replication payoff: a standby that never applied deployment N still holds its artifact (replicated from its pair peer) and can boot-from-cache with SQL down.

The exact integration points inside DriverHostActor are pinned by a mandatory recon task at the start of the Phase 1 plan (with STOP conditions), because that actor's fetch/boot lifecycle was mapped from docs, not stepped through.

3.4 Phase 2 — alarm-historian S&F onto LocalDb

• alarm_sf_events table (TEXT GUID PK; exact columns derived from today's SqliteStoreAndForwardSink schema by the Phase 2 recon task; status/attempts/dead-letter semantics preserved as columns).
• Sink rewired onto ILocalDb (public seam unchanged); its private connection/pragma/file management and AlarmHistorian:DatabasePath are deleted. AlarmHistorian:Enabled still gates the whole feature.
• One-time migrator: legacy alarm-historian.db → consolidated file, run after RegisterReplicated inside OnReady, INSERT OR IGNORE, column-intersection copy, deterministic ids for any autoincrement legacy PK (mig-{node}-{id}), single transaction, File.Move(path, path + ".migrated") only after commit.
• Drain: primary-gated via the delivered-snapshot role (PrimaryGatePolicy.ShouldServiceAsPrimary); delivered/dead-letter marks are row updates that replicate, so the standby's copy tracks drain progress. At-least-once across failover, documented.

3.5 Error handling

Library defaults stand: fail-closed schema-digest handshake, per-entry dead-letter for poison rows, oplog caps degrade to snapshot resync (never data loss). App-side: cache-write failures are logged and dropped (apply already succeeded); cache-read integrity failures are a miss; the interceptor is fail-closed (no/empty ApiKey ⇒ PermissionDenied for every sync call — a key typo stops convergence, it does not degrade to unauthenticated).

3.6 Observability & health

• ISyncStatus → a ZB.MOM.WW.Health check (Degraded when a peer is configured but Connected is false, or OplogBacklog is null/climbing), registered in the existing AddOtOpcUaHealth path.
• The meter ZB.MOM.WW.LocalDb.Replication must be added to OtOpcUa's telemetry meter list if AddOtOpcUaObservability uses an allowlist — ZbTelemetryOptions.Meters is a silent allowlist and this exact omission is what the ScadaBridge live gate caught. The plan has an explicit task for it.

3.7 Testing & definition of done

• DI-pin tests over the real Program.cs via WebApplicationFactory<Program> (Host.IntegrationTests, xunit.v3): exact replicated-table set (both directions load-bearing), PK pins, ILocalDb singleton, default-OFF pin (ISyncStatus.Connected == false, PeerNodeId null), driver-role gating (admin-only graph has no ILocalDb).
• Interceptor unit tests with a hand-rolled fake ServerCallContext (Grpc.Core.Testing does not exist on grpc-dotnet).
• 2-node convergence harness over a real loopback Kestrel h2c transport through the real interceptor, initialized via the production LocalDbSetup.OnReady (a hand-written schema would only prove the test agrees with itself) — copied from the library's ConvergenceFixture / ScadaBridge's LocalDbSitePairHarness. DBs registered as pre-constructed singletons so transport teardown doesn't dispose them. Real temp files, never :memory:; SqliteConnection.ClearAllPools() on cleanup.
• Positive control (DoD): with the RegisterReplicated calls commented out, the convergence scenarios must go red. Absence assertions get positive controls.
• Actor tests (cache write/read fallback) live in an xunit2 TestKit project — Akka.TestKit.Xunit2 cannot be used from the xunit.v3 Host.IntegrationTests project.
• Live gate on the docker-dev rig (site-a pair; adapted from ScadaBridge's 10 checks): artifact chunks + pointer converge byte-identical with identical HLC + origin node id; standby-that-never-fetched boots from cache with SQL stopped; retention prune tombstones on both nodes; oplog rise-and-drain across a transport kill; both-nodes-together restart clean; metrics present on /metrics; zero disk I/O error/SQLITE_IOERR in logs. All DB sampling via cp of the db/-wal/-shm triplet — never host sqlite3 against a live bind-mounted WAL DB (the 2026-07-20 incident).

4. Operational constraints (inherited, to be documented in OtOpcUa's runbooks)

• A replicated pair must be stopped/started together across upgrades (no rolling-upgrade support; the schema-digest handshake fails closed on any drift).
• A node offline beyond TombstoneRetention (default 7 d) can resurrect deleted rows.
• MaxBatchSize is a row count against the 4 MB gRPC default cap — the rig pins 16; chunking keeps the worst-case row ≈ 171 KB.
• The sync ApiKey must be byte-identical on both nodes (fail-closed interceptor).
• Never inspect a live bind-mounted WAL DB with host sqlite3; copy the triplet.

5. Out of scope

• Replicating secrets via LocalDb (Secrets keeps its own Akka replicator).
• The FasterLog continuous-historization outbox.
• Alarm condition / scripted-alarm state (stays central SQL).
• N>2 topologies, an Akka transport for LocalDb sync, byte-aware batching (library-side futures).
• Production enablement (needs a sync-ApiKey distribution decision, like the Secrets KEK one).

Appendix — ScadaBridge lessons applied

Lesson (source: ScadaBridge Phases 1–2 + live gates)	Where this design applies it
DDL → RegisterReplicated → migrate → writes; pre-registration rows are invisible forever, silently	§3.1 OnReady ordering; Phase 2 migrator runs last, pinned by an oplog assertion
Autoincrement PKs are an LWW overwrite trap	All new tables TEXT/composite PKs; Phase 2 recon checks the legacy alarm PK, deterministic mig- ids if needed
Library passive side verifies no ApiKey; MapZbLocalDbSync() can't carry auth on its return value	Consumer interceptor, fail-closed, ported line-for-line (§3.1)
ZbTelemetryOptions.Meters is a silent allowlist	Explicit meter-allowlist task (§3.6)
Register in the graph your tests actually build	OtOpcUa's composition root is Program.cs and WebApplicationFactory<Program> builds it — DI pins run against the real graph (§3.7)
Row-count MaxBatchSize vs 4 MB gRPC cap; single row near 4 MB = STOP	Chunked artifact storage designed in (D6), rig pin 16
No rolling upgrades; stop/start pair together	§4, runbook task
Tombstone-retention resurrection	§4
CreateConnection() returns an already-open, UDF-registered connection; no in-memory mode	Plan task notes; test harness uses temp files + ClearAllPools
Absence assertions need a positive control; DoD greps must say "no references from code"	§3.7 DoD; deletion tasks in both plans phrase greps that way
Never host-sqlite3 a live WAL DB; docker exec curl doesn't exist in aspnet:10.0	Live-gate procedure (§3.7, §4); gate uses cp triplet + curlimages/curl sidecar
Grpc.Core.Testing doesn't exist on grpc-dotnet	Interceptor tests hand-roll ServerCallContext
Observer-induced failures: check the observer's causal path before blaming the component	Live-gate preamble in both plans