From c834d89ae7d72b3dea05373786a40f0fda27e05c Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Mon, 20 Jul 2026 22:06:58 -0400 Subject: [PATCH] docs(localdb): phase-1 runbook + redundancy/config docs New operations runbook (enable/disable, fail-closed ApiKey rule, stop/start- together, tombstone-retention window, MaxBatchSize row-count-vs-4MB, never sqlite3-a-live-WAL-DB cp-triplet recipe). Configuration.md gains a LocalDb section; Redundancy.md gains a pair-local config cache section (what boot-from- cache does and does not cover); CLAUDE.md gains the Phase 1 scope paragraph. --- CLAUDE.md | 18 +++ docs/Configuration.md | 22 ++++ docs/Redundancy.md | 29 ++++ .../2026-07-20-localdb-pair-replication.md | 124 ++++++++++++++++++ 4 files changed, 193 insertions(+) create mode 100644 docs/operations/2026-07-20-localdb-pair-replication.md diff --git a/CLAUDE.md b/CLAUDE.md index a424866c..0b0d2588 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -219,6 +219,24 @@ The server supports configurable OPC UA transport security via the `OpcUa:Enable The server supports non-transparent warm/hot redundancy via the `Redundancy` section in `appsettings.json`. Two instances share the same Galaxy DB and the same mxaccessgw (under distinct `MxAccess.ClientName` values) but have unique `ApplicationUri` values. Each exposes `RedundancySupport`, `ServerUriArray`, and a dynamic `ServiceLevel` based on role and runtime health. The primary advertises a higher ServiceLevel than the secondary. See `docs/Redundancy.md` for the full guide. +## LocalDb pair-local config cache (Phase 1) + +Every **driver-role** node keeps a consolidated `ZB.MOM.WW.LocalDb` SQLite database (the retired +LiteDB `LocalCache` subsystem was deleted as superseded). Phase-1 scope: it caches the +**deployed-configuration artifact** (chunked, SHA-256-verified, newest-2 retention per cluster) so a +driver node can **boot from cache when central SQL Server is unreachable** — `DriverHostActor` writes +the cache after each successful apply (`IDeploymentArtifactCache` → `LocalDbDeploymentArtifactCache`, +in `Runtime/Deployment/`) and reads it as a boot fallback, flagging running-from-cache. Wired by +`AddOtOpcUaLocalDb` in the `hasDriver` branch only (`Host/Configuration/LocalDbRegistration.cs` + +`LocalDbSetup.OnReady`, whose DDL→`RegisterReplicated` order is load-bearing); **admin-only nodes +never register it.** The cache can optionally replicate to the node's redundant pair peer over the +library's gRPC sync — **default-OFF, fail-closed bearer auth** (`LocalDbSyncAuthInterceptor`), gated +on `LocalDb:SyncListenPort` (a dedicated Kestrel h2c listener) + `LocalDb:Replication:*`. On the +docker-dev rig the **site-a pair** has replication on; central + site-b stay default-OFF (site-b is +the pin). ApiKey via `${secret:}`/env, never committed (the rig's dev key is the committed-dev-secret +exception). See `docs/Configuration.md` (`LocalDb` section), `docs/Redundancy.md` (pair-local config +cache), and the runbook `docs/operations/2026-07-20-localdb-pair-replication.md`. + ## LDAP Authentication The server uses LDAP-based user authentication via the `Security:Ldap` section in `appsettings.json`. When enabled, credentials are validated by LDAP bind against a GLAuth server, and LDAP group membership maps to OPC UA permissions: `ReadOnly` (browse/read), `WriteOperate` (write FreeAccess/Operate attributes), `WriteTune` (write Tune attributes), `WriteConfigure` (write Configure attributes), `AlarmAck` (alarm acknowledgment). `LdapOpcUaUserAuthenticator` (`src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/LdapOpcUaUserAuthenticator.cs`) implements `IOpcUaUserAuthenticator`, delegating the LDAP bind + group lookup to `OtOpcUaLdapAuthService` (`src/Server/ZB.MOM.WW.OtOpcUa.Security/Ldap/OtOpcUaLdapAuthService.cs`, an `ILdapAuthService`). See `docs/security.md` for the full guide. diff --git a/docs/Configuration.md b/docs/Configuration.md index fdcac78c..5f01bfdc 100644 --- a/docs/Configuration.md +++ b/docs/Configuration.md @@ -146,6 +146,28 @@ key must carry the scopes `historian:read`, `historian:write`, `historian:tags:w [`docs/Historian.md`](Historian.md) for the full key reference, the migration note (old Wonderware keys → gateway keys), and the deployment prerequisites. +### `LocalDb` (pair-local config cache — driver role only) + +Driver-role nodes keep a consolidated **`ZB.MOM.WW.LocalDb`** SQLite database that caches the deployed +configuration artifact (chunked) so the node can **boot from cache when central SQL is unreachable**. The +section is bound by `AddOtOpcUaLocalDb` (`Host/Configuration/LocalDbRegistration.cs`) inside the `hasDriver` +branch — **admin-only nodes never register it, and never require `LocalDb:Path`**. Storage is unconditional; +replication stays inert until the sync port + peer are set. + +| Key | Default | Meaning | +|---|---|---| +| `LocalDb:Path` | `./data/otopcua-localdb.db` | SQLite file path. `ValidateOnStart`-required once `AddZbLocalDb` runs (driver nodes only). In containers put it on a durable volume. | +| `LocalDb:SyncListenPort` | `0` | `0` = replication off (no sync listener). A non-zero port binds a dedicated **h2c** sync listener; set the **same** port on both pair nodes. Setting it makes the Host add an explicit Kestrel `Listen*` and re-bind the primary HTTP port (an explicit `Listen*` otherwise discards `ASPNETCORE_URLS`). | +| `LocalDb:Replication:PeerAddress` | `""` | The peer this node **dials** (`http://:`). Set on the initiator only; leave empty on the passive node. The stream is bidirectional. | +| `LocalDb:Replication:ApiKey` | `""` | Bearer token for the sync stream. **Fail-closed:** no key ⇒ the passive endpoint refuses everything; a mismatch ⇒ the pair silently stops converging. **Must be byte-identical on both nodes.** Supply via `${secret:...}` / env — never a cleartext literal in production. | +| `LocalDb:Replication:MaxBatchSize` | (library default) | Rows per replication batch — **row-count-only** against gRPC's 4 MB cap. `16` for the artifact cache (chunk rows ≈171 KB, so 16 × 171 KB ≈ 2.7 MB). | + +Replication is **default-OFF** across the fleet; it is enabled per-pair as an opt-in. See +[`docs/operations/2026-07-20-localdb-pair-replication.md`](operations/2026-07-20-localdb-pair-replication.md) +for the enablement runbook (ApiKey handling, stop/start-together rule, tombstone-retention window, and the +never-`sqlite3`-a-live-WAL-DB inspection rules) and the "pair-local config cache" section of +[`docs/Redundancy.md`](Redundancy.md) for what boot-from-cache does and does not cover. + --- ## Environment variables diff --git a/docs/Redundancy.md b/docs/Redundancy.md index 19fe48b7..6692bed9 100644 --- a/docs/Redundancy.md +++ b/docs/Redundancy.md @@ -226,6 +226,35 @@ Net effect: each alarm transition appears **once** on `/alerts` and would histor See [ScriptedAlarms.md](ScriptedAlarms.md) and [AlarmTracking.md](AlarmTracking.md) for the scripted-alarm engine internals. +## Pair-local config cache (LocalDb — Phase 1) + +Independently of the ServiceLevel machinery above, every **driver-role** node keeps a consolidated +[`ZB.MOM.WW.LocalDb`](../CLAUDE.md) SQLite database that caches the **deployed-configuration +artifact** (chunked). Its job is a single failure mode redundancy does not otherwise cover: a driver +node restarting into a **central-SQL-Server outage** would come up with no configuration at all. With +the cache, it **boots from the last artifact it applied** instead, and logs a running-from-cache +signal. + +- **What replicates.** The cache can *optionally* replicate to the node's redundant pair peer over + the LocalDb library's gRPC sync (HLC-stamped, last-writer-wins). Two tables replicate: + `deployment_artifacts` and `deployment_pointer`. Replication is **default-OFF and fail-closed** — + it stays inert until `LocalDb:SyncListenPort` + a peer are configured, and a missing/mismatched + `LocalDb:Replication:ApiKey` refuses or silently halts sync. It is enabled per-pair as an opt-in. +- **The replicated-cache payoff.** In a pair with replication on, *either* node can be the one that + applied the latest deploy; the peer holds a byte-identical copy. So a node that restarts during a + central outage can boot the current config **even if it never applied that deploy itself** — its + peer did, and the row replicated. +- **What it does NOT cover.** The cache is a **boot fallback for central-SQL outages only**, not a + standby data path and not a replacement for central. A **new deployment still requires central + SQL**; the cache is read only when the central fetch fails at startup, and the node resumes + fresh-config behavior once central returns. It does not replicate live tag values, alarms, or + historian data — only the config artifact. It is orthogonal to the ServiceLevel/primary-gate + logic: a node running from cache still participates in redundancy normally. + +Full detail: the enablement runbook at +[`docs/operations/2026-07-20-localdb-pair-replication.md`](operations/2026-07-20-localdb-pair-replication.md) +and the `LocalDb` section of [`docs/Configuration.md`](Configuration.md). + ## Client-side failover The OtOpcUa Client CLI at `src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI` supports `-F` / `--failover-urls` for automatic client-side failover; for long-running subscriptions the CLI monitors session KeepAlive and reconnects to the next available server, recreating the subscription on the new endpoint. See [`Client.CLI.md`](Client.CLI.md). diff --git a/docs/operations/2026-07-20-localdb-pair-replication.md b/docs/operations/2026-07-20-localdb-pair-replication.md new file mode 100644 index 00000000..5bc1dea5 --- /dev/null +++ b/docs/operations/2026-07-20-localdb-pair-replication.md @@ -0,0 +1,124 @@ +# LocalDb pair replication — operations runbook + +> **Phase 1 scope.** Every driver-role OtOpcUa node keeps a consolidated +> [`ZB.MOM.WW.LocalDb`](../../CLAUDE.md) SQLite database. Today it caches exactly one thing: the +> **deployed-configuration artifact**, chunked, so a node can **boot from cache when central SQL +> Server is unreachable**. That cache can *optionally* replicate to the node's redundant pair peer +> over the library's gRPC sync. **Replication is default-OFF and fail-closed.** This runbook covers +> enabling it, the operational rules that keep a pair converging, and the DB-inspection safety +> rules. + +## What replicates, and what it buys you + +- **Replicated tables:** `deployment_artifacts` (the artifact, split into base64 chunks) and + `deployment_pointer` (one current-deployment pointer per cluster). Registered — in this order, + after the DDL — by `LocalDbSetup.OnReady`. +- **The payoff:** in a redundant driver pair, *either* node can be the one that applied the latest + deploy and wrote the cache. With replication on, the peer holds a byte-identical copy. So a node + that restarts into a central-SQL outage can boot the current config **even if it never applied + that deploy itself** — its peer did, and the row replicated. +- **Convergence model:** last-writer-wins per primary key, HLC-stamped. Both nodes also store + locally after every apply, so the pair converges to identical content regardless of which node + deployed. Retention (newest-2 deployments per cluster) prunes on one node and the deletes + replicate as tombstones. + +## Enabling replication on a pair + +Replication has two knobs, both under `LocalDb`: + +| Key | Role | Notes | +|---|---|---| +| `LocalDb:SyncListenPort` | binds the dedicated **h2c** sync listener | `0` (default) = no listener, replication off. Set the **same** non-zero port on **both** nodes of the pair. | +| `LocalDb:Replication:PeerAddress` | the address this node **dials** | Set on **one** node (the initiator); leave empty on the other (passive). The stream is bidirectional — one dial carries both directions. | +| `LocalDb:Replication:ApiKey` | bearer token for the sync stream | **Must be byte-identical on both nodes.** See fail-closed rule below. Supply via `${secret:...}` / env — never a cleartext literal in production config. | +| `LocalDb:Replication:MaxBatchSize` | rows per replication batch | `16` for the artifact cache. Batching is **row-count-only** against gRPC's 4 MB message cap; artifact chunk rows are ≈171 KB, so 16 × 171 KB ≈ 2.7 MB stays under the cap. Raising it risks tripping the cap. | + +Example (site-a-1 initiator, site-a-2 passive): + +``` +site-a-1: LocalDb__SyncListenPort=9001 + LocalDb__Replication__PeerAddress=http://site-a-2:9001 + LocalDb__Replication__ApiKey=${secret:site-a-localdb-sync-key} + LocalDb__Replication__MaxBatchSize=16 +site-a-2: LocalDb__SyncListenPort=9001 + LocalDb__Replication__ApiKey=${secret:site-a-localdb-sync-key} # no PeerAddress + LocalDb__Replication__MaxBatchSize=16 +``` + +> **Kestrel note.** Setting `SyncListenPort` makes the Host add an explicit `Listen*` for the h2c +> sync listener. An explicit `Listen*` makes Kestrel **ignore `ASPNETCORE_URLS` entirely**, so the +> Host also re-binds the primary HTTP port in the same block. If you change the primary port, +> verify both `:` and `:` appear in the startup "Now listening on" lines. + +### The ApiKey rule (fail-closed) + +The replication library's passive endpoint verifies **no** authentication — the host +`LocalDbSyncAuthInterceptor` is the only gate, and it is **fail-closed**: + +- **No key configured ⇒ every sync call is refused** (`PermissionDenied`). "No key" is never + "no auth required". +- **A key mismatch ⇒ the pair silently stops converging.** The initiator's stream is rejected at + the peer; nothing errors loudly. A typo in one node's key looks exactly like "replication is + broken". If a pair is not converging, **check the keys match first.** + +## Operational rules + +- **Stop / start the pair together where you can.** Each node keeps working (and caching locally) + while its peer is down; the outage is not a data-loss event — the surviving node accumulates + writes and the peer catches up on rejoin. But a long-lived solo node drifts further from its + peer, so avoid leaving a pair split for extended periods. +- **Tombstone-retention resurrection window.** Retention prunes to the newest 2 deployments and + replicates the prune as tombstones. Tombstones are themselves retained only for a bounded window. + If a node is offline **longer than the tombstone-retention window**, a delete that happened during + its outage may no longer be expressible as a tombstone on rejoin — a pruned deployment could + briefly reappear until the next deploy re-prunes it. Keep pair outages well inside that window. +- **What boot-from-cache does NOT cover.** The cache is a *fallback for central-SQL outages at + boot*, not a replacement for central. A **new deployment still requires central SQL** — the cache + is only read when the central fetch fails at startup. When central returns, the node resumes + fresh-config behavior. Boot-from-cache logs a running-from-cache signal; treat a node that stays + on it as a central-connectivity incident, not steady state. + +## Inspecting the LocalDb file — safety rules + +The database runs in **WAL mode**. These rules exist because violating them corrupted a live DB in +the 2026-07-20 ScadaBridge incident: + +- **Never run `sqlite3` on the live file** (host-side, against a bind-mounted or container path). + Opening a live WAL DB from a second process across virtiofs poisons the WAL. **Always copy the + triplet first** and query the copy: + + ```bash + docker cp :/app/data/otopcua-localdb.db /tmp/localdb.db + docker cp :/app/data/otopcua-localdb.db-wal /tmp/localdb.db-wal + docker cp :/app/data/otopcua-localdb.db-shm /tmp/localdb.db-shm + sqlite3 /tmp/localdb.db 'SELECT cluster_id, deployment_id FROM deployment_pointer;' + ``` + +- **Metrics** come from the container, not a host curl (`aspnet:10.0` has no `curl`): + + ```bash + docker run --rm --network container: curlimages/curl:latest -s localhost:/metrics | grep localdb_ + ``` + +- **If an anomaly appears within seconds of your own measurement, suspect the measurement.** + Restart both nodes and re-observe untouched before blaming the library. + +## Useful queries (on a copied triplet) + +```sql +-- What each node thinks the current deployment is +SELECT cluster_id, deployment_id, revision_hash, applied_at_utc FROM deployment_pointer; + +-- Convergence check: the pointer's origin stamp should match on both nodes +SELECT pk_json, hlc, node_id, is_tombstone +FROM __localdb_row_version WHERE table_name = 'deployment_pointer' ORDER BY pk_json; + +-- Replication backlog (should drain to 0 when a pair is caught up) +SELECT COUNT(*) FROM __localdb_oplog; +``` + +## See also + +- [`docs/Redundancy.md`](../Redundancy.md) — the pair-local config cache section. +- [`docs/Configuration.md`](../Configuration.md) — the `LocalDb` appsettings section. +- The two-node convergence harness: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDb/`.