docs(localdb): ScadaBridge LocalDb adoption design (approved)
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,143 @@
|
||||
# ScadaBridge — ZB.MOM.WW.LocalDb adoption design
|
||||
|
||||
**Date:** 2026-07-19
|
||||
**Status:** Approved (brainstorming session, all sections user-validated)
|
||||
**Library:** `ZB.MOM.WW.LocalDb` 0.1.0 (feed-published 2026-07-18; no prior app adoption)
|
||||
|
||||
ScadaBridge becomes the first consumer of `ZB.MOM.WW.LocalDb` (embedded SQLite cache +
|
||||
optional 2-node bidirectional gRPC sync, trigger-CDC + HLC last-writer-wins). Target:
|
||||
the 2-node **site cluster pair** (node-a/node-b), which today mixes a bespoke Akka
|
||||
replicator (config + store-and-forward) with entirely un-replicated per-node stores
|
||||
(operation tracking, site events) that lose state on failover.
|
||||
|
||||
## Decisions (user-validated)
|
||||
|
||||
| Question | Decision |
|
||||
|---|---|
|
||||
| Scope | **Both, phased** — Phase 1 adds replication where none exists (tracking + events); Phase 2 migrates the bespoke-replicated DBs (config + store-and-forward) onto LocalDb |
|
||||
| Audit DB | **Out of scope** — `auditlog.db` diverges per-node by design; central pulls the union. Replicating it would double-forward events |
|
||||
| Secrets store | Out of scope — has its own replication answer (`ZB.MOM.WW.Secrets.Replicator.SqlServer` hub mode, wired default-off) |
|
||||
| Central nodes | Out of scope — SQL-Server-first; no LocalDb use case there today |
|
||||
| Rollout | **Config-gated, default OFF** in the product; enabled + live-proven on the local docker cluster as part of the work (Secrets-replication precedent); production enablement is its own later decision |
|
||||
| Phase 2 fate of bespoke replicator | **Replace + delete** — `SiteReplicationActor` + StoreAndForward `ReplicationService` removed outright once live-proven (recoverable in git); no dual-mechanism period |
|
||||
| DB layout | **One consolidated LocalDb file** — the lib is one-DB-per-process (`AddZbLocalDb` first-registration-wins); merge replicated site state into a single file rather than extend the lib to named instances |
|
||||
|
||||
## Current state (inventoried 2026-07-19)
|
||||
|
||||
Site nodes keep six SQLite files under `/app/data`. Replication today:
|
||||
|
||||
| DB | Replicated? | Mechanism |
|
||||
|---|---|---|
|
||||
| `scadabridge.db` (site config: deployed configs, overrides, scripts, external systems, …) | Yes | Bespoke `SiteReplicationActor` (Akka, fire-and-forget LWW notify-and-fetch + chunked anti-entropy resync) |
|
||||
| `store-and-forward.db` (`sf_messages` outbound buffer) | Yes | Bespoke `ReplicationService` op-shipping (Add/Remove/Park/Requeue → newest-wins upserts) via the same actor |
|
||||
| `site-tracking.db` (`OperationTracking`) | **No — diverges** | — |
|
||||
| `site_events.db` (`site_events`) | **No — diverges** | — |
|
||||
| `auditlog.db` | No, by design | Central pulls union over gRPC |
|
||||
| `scadabridge-secrets.db` | No | Secrets G-7 hub mode (separate) |
|
||||
|
||||
Site pair facts that matter: two nodes, active = oldest-Up Akka member, both warm;
|
||||
site nodes already host Kestrel gRPC on **8083**; central and site clusters are separate
|
||||
Akka clusters (irrelevant here — LocalDb sync is intra-site-pair only).
|
||||
|
||||
## Design
|
||||
|
||||
### 1. Shape
|
||||
|
||||
One consolidated LocalDb-managed database per site node — `LocalDb:Path` →
|
||||
`/app/data/site-localdb.db` — replicated between the pair by the library. Adoption
|
||||
follows the Secrets pattern:
|
||||
|
||||
- **Storage move ships unconditionally.** LocalDb with replication off is just a fast
|
||||
local SQLite DB; behavior-equivalent, no flag needed.
|
||||
- **Replication is config-gated, default OFF** (`LocalDb:Replication:PeerAddress` unset
|
||||
⇒ initiator idles; passive endpoint mapped but unused). Enabled in the docker rig as
|
||||
part of the definition of done.
|
||||
|
||||
**Phase 1 — new replication (low risk, pure win):**
|
||||
`OperationTracking` + `site_events` move into the consolidated DB as `RegisterReplicated`
|
||||
tables. `site-tracking.db` and `site_events.db` retire. Failover stops losing the
|
||||
standby's view of op status and site events.
|
||||
|
||||
**Phase 2 — consolidation (starts only after Phase 1 passes the live gate):**
|
||||
`scadabridge.db` config tables + `sf_messages` join the same DB. `SiteReplicationActor`
|
||||
and StoreAndForward `ReplicationService` are **deleted**. CDC triggers replace
|
||||
hand-shipped ops — park/requeue/remove become ordinary captured row updates/deletes —
|
||||
and the lib's snapshot resync replaces the bespoke chunked anti-entropy.
|
||||
|
||||
### 2. Schema + migration
|
||||
|
||||
- **`site_events` PK change (required):** `INTEGER PRIMARY KEY AUTOINCREMENT` → TEXT
|
||||
GUID minted in `SiteEventLogger`. Two nodes minting the same auto-increment id would
|
||||
silently overwrite each other under LWW; GUID rows make events a pure union. Existing
|
||||
indexes (timestamp/type/instance/severity) carry over.
|
||||
- **`OperationTracking`** (TEXT PK `TrackedOperationId`) and **`sf_messages`** (TEXT PK
|
||||
`id`) replicate as-is — no BLOB columns anywhere, so all pass `RegisterReplicated`
|
||||
validation.
|
||||
- **One-time idempotent migrator** at site-node startup: after table creation +
|
||||
`RegisterReplicated` (inside the `AddZbLocalDb` `onReady`), copy rows from legacy DB
|
||||
files via `ILocalDb` inserts — **after** registration, because the lib does not
|
||||
capture or snapshot pre-registration rows; migrated rows must enter the oplog to
|
||||
replicate. Legacy `site_events` rows get fresh GUIDs. Legacy files rename to
|
||||
`*.migrated` on success.
|
||||
- **Store rewiring:** `OperationTrackingStore`, `SiteEventLogger`,
|
||||
`EventLogQueryService` (Phase 2: `SiteStorageService`, `StoreAndForwardStorage`)
|
||||
consume `ILocalDb` so every write goes through the capture path. Public interfaces of
|
||||
these stores are unchanged — callers don't move.
|
||||
- **LWW fit per table:** tracking = status transitions updated by whichever node runs
|
||||
the op, last update wins (correct); events = append-only GUID rows (no conflicts);
|
||||
config tables already use LWW semantics under the bespoke actor; `sf_messages` keeps
|
||||
the same bounded-duplicate race the bespoke replicator documents and accepts (N5).
|
||||
- **Retention/trim jobs** run on both nodes idempotently; concurrent deletes of the
|
||||
same rows converge as tombstones.
|
||||
|
||||
### 3. Wiring, transport, auth
|
||||
|
||||
- `AddZbLocalDb` + `AddZbLocalDbReplication` registered in `SiteServiceRegistration`;
|
||||
`MapZbLocalDbSync()` on the site node's existing Kestrel gRPC host (**port 8083**,
|
||||
alongside site streaming) — no new port.
|
||||
- **Roles:** node-a is the initiator — its per-node appsettings set
|
||||
`LocalDb:Replication:PeerAddress` = node-b:8083. node-b stays passive (no
|
||||
PeerAddress). Both nodes map the passive endpoint, so role assignment doesn't affect
|
||||
correctness, only who dials.
|
||||
- **Auth:** initiator presents `LocalDb:Replication:ApiKey` — a `${secret:}` ref
|
||||
resolved by ScadaBridge's existing Secrets integration. The library's passive side
|
||||
deliberately does not verify keys, so ScadaBridge adds a small fixed-time-compare
|
||||
bearer interceptor on the sync endpoint. h2c in the dev rig; TLS posture matches the
|
||||
existing 8083 endpoint.
|
||||
- **Observability:** the `ZB.MOM.WW.LocalDb.Replication` meter (oplog depth, sync lag,
|
||||
applied/conflict/dead-letter counters) flows through the existing `AddZbTelemetry`
|
||||
Prometheus export. `ISyncStatus` is surfaced as a site health check / dashboard
|
||||
signal.
|
||||
|
||||
### 4. Error handling
|
||||
|
||||
Inherited from the library (all verified in its 144-test suite): fail-closed handshake
|
||||
on schema digest mismatch, capped-backoff reconnect, oplog row/age caps with
|
||||
snapshot-resync fallback (disk safety over delta continuity), per-entry dead-letter for
|
||||
poison rows, HLC drift guard. ScadaBridge-side additions: migrator failure fails
|
||||
startup (no half-migrated state — legacy files only renamed after a committed copy),
|
||||
and the sync-endpoint interceptor rejects unauthenticated streams before the handshake.
|
||||
|
||||
### 5. Testing & definition of done
|
||||
|
||||
- **DI-graph tests over the real Program.cs** (WebApplicationFactory, container-built)
|
||||
— the family's repeated lesson (inert Secrets Akka replicator 0.2.0, DI deadlock
|
||||
0.2.2) is that wiring bugs only show with the real host graph built.
|
||||
- Migration tests: legacy files with data → consolidated DB; idempotent re-run; GUID
|
||||
re-keying; failure leaves legacy files untouched.
|
||||
- 2-node convergence tests in-process (two site hosts, real sync stream): concurrent
|
||||
tracking updates, event union, offline-accumulate → reconnect. Phase 2 adds
|
||||
config-deploy and S&F park/requeue convergence, porting the bespoke replicator's
|
||||
existing test intents before deletion.
|
||||
- **Live gate (docker rig):** replication enabled for one site pair; run
|
||||
`failover-drill.sh`; verify replicated tables converge across the pair and the
|
||||
standby serves complete tracking/event (Phase 2: config + S&F) data after takeover.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Central-node LocalDb use; audit DB replication; secrets store (all covered above).
|
||||
- LocalDb library changes (named instances etc.) — the consolidated-file layout uses
|
||||
0.1.0 as published. If Phase 2 experience shows S&F churn fighting the shared oplog
|
||||
caps, keyed instances are the escape hatch, as its own lib effort.
|
||||
- Production enablement (real key distribution, TLS) — separate decision after the
|
||||
dev-rig proof, per family convention.
|
||||
Reference in New Issue
Block a user