lmxopcua/docs/plans/2026-07-20-localdb-adoption-phase1.md

# OtOpcUa LocalDb Adoption — Phase 1 Implementation Plan

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
>
> **Execution model:** This plan is optimized for **Claude Opus** agents (`claude --model opus`).
> Dispatch implementer subagents on Opus for every task classified `high-risk`; `small`/`trivial`
> tasks may use Sonnet. Work on branch **`feat/localdb-phase1`** in this repo (`~/Desktop/OtOpcUa`,
> remote `lmxopcua`). Do NOT merge to `master` as part of this plan — stop at the DoD task and
> report.
>
> **Design authority:** `~/Desktop/scadaproj/docs/plans/2026-07-20-otopcua-localdb-design.md`.
> Read it before Task 0. The reference adoption is ScadaBridge (merged PR #23) — when in doubt,
> mirror `~/Desktop/ScadaBridge` (files named per task below).

**Goal:** Give every driver-role OtOpcUa node a consolidated `ZB.MOM.WW.LocalDb` database that caches
the deployed-configuration artifact (chunked), boots from that cache when central SQL Server is
unreachable, and optionally replicates it to the node's redundant pair peer over the library's gRPC
sync (default-OFF, fail-closed bearer auth).

**Architecture:** `AddZbLocalDb`/`AddZbLocalDbReplication` wired in a new `LocalDbRegistration`
(mirroring `SecretsRegistration`), DDL + `RegisterReplicated` in `LocalDbSetup.OnReady`, a dedicated
Kestrel h2c listener for `MapZbLocalDbSync` gated on `LocalDb:SyncListenPort`, a consumer-supplied
fail-closed bearer interceptor, and an `IDeploymentArtifactCache` seam written by `DriverHostActor`
after each successful apply and read as a boot fallback. The dormant LiteDB LocalCache subsystem is
deleted as superseded.

**Tech stack:** .NET 10, `ZB.MOM.WW.LocalDb`/`.Replication`/`.Contracts` **0.1.1** (Gitea feed),
`Grpc.AspNetCore` 2.76.0, Akka.NET (existing), xunit.v3 (Host.IntegrationTests) + xunit2 TestKit
(actor tests).

**Hard rules (from the ScadaBridge adoption — violating any of these is a defect):**
1. Order inside `OnReady` is load-bearing: **DDL → `RegisterReplicated` → writes**. Rows written
   before registration are never captured or snapshotted — silently, forever.
2. No autoincrement PKs, no BLOB columns in replicated tables.
3. The library's passive sync endpoint verifies **no** ApiKey — the host interceptor is the only
   auth, and it must be fail-closed (no key configured ⇒ deny everything).
4. `ILocalDb.CreateConnection()` returns an **already-open**, pragma-configured connection with the
   `zb_hlc_next()` UDF. Never call `Open()` on it. There is **no in-memory mode** — tests use temp
   files + `SqliteConnection.ClearAllPools()` before delete.
5. `Grpc.Core.Testing` does not exist on grpc-dotnet — hand-roll fake `ServerCallContext`s.
6. Explicit Kestrel `Listen*` calls make Kestrel **ignore `ASPNETCORE_URLS`** — when adding the sync
   listener you must re-bind the existing HTTP port in the same block, and verify it on the rig.
7. Never run host `sqlite3` against a live bind-mounted WAL DB (copy the `db`/`-wal`/`-shm` triplet
   with `cp` instead). `aspnet:10.0` containers have no `curl` — use a
   `curlimages/curl` sidecar with `--network container:<name>`.

---

### Task 0: Preflight + recon (produces `docs/plans/2026-07-20-localdb-phase1-recon.md`)

**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** none (everything downstream consumes its findings)

**Files:**
- Create: `docs/plans/2026-07-20-localdb-phase1-recon.md`
- Read-only: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs`,
  `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DeploymentArtifact.cs`,
  `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs`,
  `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/SecretsRegistration.cs`,
  `Directory.Packages.props`, `docker-dev/docker-compose.yml`,
  the observability registration (`AddOtOpcUaObservability` implementation),
  `src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/*`

**Step 1: Verify the feed packages restore.**

Run:
```bash
cd ~/Desktop/OtOpcUa && dotnet package search ZB.MOM.WW.LocalDb --source dohertj2-gitea --format json | head -50
```
Expected: `ZB.MOM.WW.LocalDb`, `.Replication`, `.Contracts` at `0.1.1`. If absent → STOP, report.

**Step 2: Map and record, with `file:line` citations, each of:**
1. **Deploy fetch path:** where `DriverHostActor` loads `Deployment.ArtifactBlob` (the
   `_dbFactory.CreateDbContext()` sites, ~L1472/1543), the artifact's runtime type
   (`DeploymentArtifact` — how it's deserialized from the blob), the type/format of
   `DeploymentId` and `RevisionHash`, and where a successful apply completes (the point after
   which the cache write belongs).
2. **Cold-boot path:** what the actor does at startup before any `DispatchDeployment` arrives —
   does it query central SQL for the current deployment? Where does the failure path land
   (the `Stale` state, `DbHealthProbeActor` interaction, L1345 dispatch-ignore)? Identify the
   exact seam where "central fetch failed at boot" is known — that is where the cache fallback goes.
3. **Cluster identity:** how a driver node knows its `ClusterId` (config key / `ClusterNode` row /
   `IClusterRoleInfo`) — the cache is keyed by it.
4. **DriverHostActor construction:** how its dependencies are injected
   (`WithOtOpcUaRuntimeActors`, `Props` wiring) so `IDeploymentArtifactCache` can be threaded in.
   ⚠ `Props.Create` builds an expression tree — adding a parameter silently rebinds
   out-of-position named args at call sites; list every construction site.
5. **Telemetry allowlist:** does `AddOtOpcUaObservability` restrict meters
   (`ZbTelemetryOptions.Meters` or equivalent)? If yes, record where the list lives.
6. **Kestrel/URLs:** confirm the Host binds via `ASPNETCORE_URLS=http://+:9000` with no
   `ConfigureKestrel` calls; record any existing `builder.WebHost` usage.
7. **LiteDB LocalCache:** confirm (grep) that nothing outside
   `src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/` and its tests references
   `ILocalConfigCache`/`GenerationSealedCache`/`ResilientConfigReader`/`LiteDB`.
8. **docker-dev volumes:** whether nodes have a writable data volume; record what `LocalDb:Path`
   should be per node and how env vars are passed (`Cluster__*` style double-underscore).

**Step 3: STOP conditions — halt the plan and report instead of improvising if:**
- the artifact apply path cannot be given a post-apply hook without restructuring the actor;
- `DeploymentId`/cluster identity are not stable strings/GUIDs;
- LiteDB LocalCache turns out to be referenced by live code.

**Step 4: Commit.**
```bash
git add docs/plans/2026-07-20-localdb-phase1-recon.md
git commit -m "docs(localdb): phase-1 recon findings"
```

---

### Task 1: Package references and pins

**Classification:** small
**Estimated implement time:** ~3 min
**Parallelizable with:** none (all code tasks build on it)

**Files:**
- Modify: `Directory.Packages.props`
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj`
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ZB.MOM.WW.OtOpcUa.Runtime.csproj`

**Step 1:** Add to `Directory.Packages.props` (versions exact):
```xml
<PackageVersion Include="ZB.MOM.WW.LocalDb" Version="0.1.1" />
<PackageVersion Include="ZB.MOM.WW.LocalDb.Replication" Version="0.1.1" />
<PackageVersion Include="Grpc.AspNetCore" Version="2.76.0" />
```
Check the existing `Grpc.Net.Client`/`Grpc.Core.Api` are already ≥ 2.76.0 and
`Google.Protobuf` ≥ 3.34.1 (the LocalDb.Replication nuspec floors); raise if not. Confirm a
`SQLitePCLRaw` pin ≥ 2.1.12 exists for the transitive `Microsoft.Data.Sqlite` chain (the family
advisory GHSA-2m69-gcr7-jv3q); Core.AlarmHistorian already pins `bundle_e_sqlite3` 2.1.12 — add
`SQLitePCLRaw.lib.e_sqlite3` 2.1.12 as an explicit reference in the Host if the audit flags it.

**Step 2:** Host csproj: `ZB.MOM.WW.LocalDb`, `ZB.MOM.WW.LocalDb.Replication`, `Grpc.AspNetCore`.
Runtime csproj: `ZB.MOM.WW.LocalDb` only (it needs `ILocalDb`, not the sync engine).

**Step 3:** `dotnet build ZB.MOM.WW.OtOpcUa.slnx` → 0 warnings/errors (repo builds warnings-as-errors).

**Step 4: Commit.** `git commit -m "build(localdb): reference ZB.MOM.WW.LocalDb 0.1.1 + Grpc.AspNetCore"`

---

### Task 2: Schema + `LocalDbSetup.OnReady` + registration tests

**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 3

**Files:**
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/DeploymentCacheSchema.cs`
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbSetup.cs`
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/LocalDbSetupTests.cs` (create the test project
  reference layout to match neighboring Host unit tests; if Host has no unit-test project, place in
  the closest existing one and note the deviation)

**Step 1: Write the failing tests first** — using a real temp-file `ILocalDb` (build it via
`new ServiceCollection().AddZbLocalDb(config, LocalDbSetup.OnReady)` with `LocalDb:Path` pointed at
a temp file; dispose + `SqliteConnection.ClearAllPools()` in cleanup):
- `OnReady_RegistersExactlyTheTwoDeploymentTables` — `db.ReplicatedTables.Keys` ordinal-sorted
  equals `["deployment_artifacts", "deployment_pointer"]` (assert BOTH directions: no fewer, no more).
- `DeploymentArtifacts_PkIsDeploymentIdPlusChunkIndex` and `DeploymentPointer_PkIsClusterId`
  (pin `ReplicatedTable.PkColumns`).
- `RowsWrittenAfterOnReady_EnterTheOplog` — insert a row, assert `SELECT COUNT(*) FROM __localdb_oplog` ≥ 1
  (pins the DDL→register ordering; this is the assertion that catches a silently-broken CDC).

**Step 2: Run tests, watch them fail** (types don't exist yet).

**Step 3: Implement.** `DeploymentCacheSchema` depends only on `Microsoft.Data.Sqlite` (the
ScadaBridge `*Schema.Apply` pattern — self-sufficient for direct construction in tests):

```csharp
namespace ZB.MOM.WW.OtOpcUa.Runtime.Deployment;

public static class DeploymentCacheSchema
{
    public static void Apply(SqliteConnection connection)
    {
        using var cmd = connection.CreateCommand();
        cmd.CommandText = """
            CREATE TABLE IF NOT EXISTS deployment_artifacts (
                deployment_id   TEXT    NOT NULL,
                chunk_index     INTEGER NOT NULL,
                cluster_id      TEXT    NOT NULL,
                revision_hash   TEXT    NOT NULL,
                chunk_count     INTEGER NOT NULL,
                chunk_base64    TEXT    NOT NULL,
                cached_at_utc   TEXT    NOT NULL,
                PRIMARY KEY (deployment_id, chunk_index)
            );
            CREATE TABLE IF NOT EXISTS deployment_pointer (
                cluster_id        TEXT NOT NULL PRIMARY KEY,
                deployment_id     TEXT NOT NULL,
                revision_hash     TEXT NOT NULL,
                artifact_sha256   TEXT NOT NULL,
                applied_at_utc    TEXT NOT NULL
            );
            """;
        cmd.ExecuteNonQuery();
    }
}
```

`LocalDbSetup` (Host):
```csharp
namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;

internal static class LocalDbSetup
{
    // ORDER IS LOAD-BEARING: DDL, then RegisterReplicated, then (nothing else in Phase 1).
    // Rows written before RegisterReplicated are invisible to the peer forever, silently.
    public static void OnReady(ILocalDb db)
    {
        using var connection = db.CreateConnection(); // already open — do NOT call Open()
        DeploymentCacheSchema.Apply(connection);
        db.RegisterReplicated("deployment_artifacts");
        db.RegisterReplicated("deployment_pointer");
    }
}
```

**Step 4:** `dotnet test --filter "FullyQualifiedName~LocalDbSetupTests"` → PASS.

**Step 5: Commit.** `git commit -m "feat(localdb): deployment-cache schema + OnReady registration"`

---

### Task 3: Fail-closed sync auth interceptor + tests

**Classification:** standard
**Estimated implement time:** ~4 min
**Parallelizable with:** Task 2

**Files:**
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbSyncAuthInterceptor.cs`
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/LocalDbSyncAuthInterceptorTests.cs`

Port ScadaBridge's `src/ZB.MOM.WW.ScadaBridge.Host/LocalDbSyncAuthInterceptor.cs` (read it first;
keep behavior identical, adjust namespace only). Semantics to preserve exactly:
- `ServicePrefix = "/localdb_sync.v1.LocalDbSync/"`; any other method path passes through untouched.
- Expected token from `IOptions<ReplicationOptions>.Value.ApiKey`.
- **Fail-closed:** null/empty configured key ⇒ `RpcException(new Status(StatusCode.PermissionDenied, ...))`
  for every sync call. Missing/mismatched bearer ⇒ same.
- `CryptographicOperations.FixedTimeEquals` over UTF-8 bytes; header `authorization`,
  case-insensitive, `"Bearer "` prefix. Override all four server handler kinds.

**Tests** (hand-rolled fake `ServerCallContext` — `Grpc.Core.Testing` does not exist on grpc-dotnet;
copy ScadaBridge's `LocalDbSyncAuthInterceptorTests.cs` fake): non-sync method passes with no key;
sync + no configured key → PermissionDenied; wrong bearer → PermissionDenied; correct bearer → passes.

Write tests first (fail), implement, `dotnet test --filter "FullyQualifiedName~LocalDbSyncAuthInterceptor"`,
commit `feat(localdb): fail-closed bearer interceptor for the sync endpoint`.

---

### Task 4: `LocalDbRegistration` + Program.cs DI wiring + config defaults

**Classification:** high-risk (touches Program.cs / role gating)
**Estimated implement time:** ~5 min
**Parallelizable with:** none (Task 5 edits the same file)

**Files:**
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbRegistration.cs`
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs` (inside the `hasDriver` branch)
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/appsettings.json`

**Step 1:** `LocalDbRegistration`, mirroring `SecretsRegistration`'s shape:
```csharp
namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;

internal static class LocalDbRegistration
{
    /// Driver-role nodes only. Storage is unconditional; replication stays inert until
    /// LocalDb:Replication:PeerAddress (initiator) / LocalDb:SyncListenPort (listener) are set.
    public static IServiceCollection AddOtOpcUaLocalDb(
        this IServiceCollection services, IConfiguration configuration)
    {
        services.AddZbLocalDb(configuration, LocalDbSetup.OnReady);
        services.AddZbLocalDbReplication(configuration);
        return services;
    }

    public static int SyncListenPort(IConfiguration configuration) =>
        configuration.GetValue<int>("LocalDb:SyncListenPort");
}
```

**Step 2:** Program.cs, in the `hasDriver` block (near `AddAlarmHistorian`):
`builder.Services.AddOtOpcUaLocalDb(builder.Configuration);` and
`builder.Services.AddGrpc(o => o.Interceptors.Add<LocalDbSyncAuthInterceptor>());`
(AddGrpc only under `hasDriver` — admin-only nodes expose no sync surface).

**Step 3:** appsettings.json:
```json
"LocalDb": {
  "Path": "./data/otopcua-localdb.db",
  "SyncListenPort": 0,
  "Replication": {}
}
```
`LocalDb:Path` is `ValidateOnStart`-required once `AddZbLocalDb` runs — since registration is
driver-gated, admin-only configs need nothing. Check the role-overlay files
(`appsettings.driver.json` / `appsettings.admin-driver.json`) and any deploy templates for
conflicting `LocalDb` keys.

**Step 4:** Build + full Host unit tests. Verify an admin-only graph doesn't require the key: this
is pinned properly in Task 10's integration tests, but do a quick
`OTOPCUA_ROLES=admin dotnet run` smoke only if cheap; otherwise rely on Task 10.

**Step 5: Commit.** `feat(localdb): wire AddOtOpcUaLocalDb into the driver role`

---

### Task 5: Dedicated h2c sync listener + endpoint mapping (THE Kestrel task)

**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** none (Program.cs)

**Files:**
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs`

**Why high-risk:** the Host today binds exclusively via `ASPNETCORE_URLS=http://+:9000`. Any
explicit `ConfigureKestrel(... Listen*)` makes Kestrel **ignore URLs entirely** (it logs
"Overriding address(es)"), which would silently kill the AdminUI/deploy API behind Traefik.

**Step 1:** Add, before `builder.Build()`:
```csharp
var syncPort = LocalDbRegistration.SyncListenPort(builder.Configuration);
if (hasDriver && syncPort > 0)
{
    // Explicit Listen* replaces ASPNETCORE_URLS wholesale, so re-bind the primary HTTP
    // endpoint here too. Parse it from the configured URLs rather than hard-coding 9000.
    var urls = builder.Configuration["ASPNETCORE_URLS"] ?? builder.Configuration["urls"] ?? "http://+:9000";
    var httpPort = new Uri(urls.Split(';')[0].Replace("+", "localhost").Replace("*", "localhost")).Port;
    builder.WebHost.ConfigureKestrel(k =>
    {
        k.ListenAnyIP(httpPort);                                            // HTTP/1.1 (existing surface)
        k.ListenAnyIP(syncPort, o => o.Protocols = HttpProtocols.Http2);    // h2c, prior-knowledge gRPC
    });
}
```
When `syncPort == 0` (the default) nothing changes — URLs binding stays untouched. Cleartext
`Http1AndHttp2` cannot serve prior-knowledge h2c, hence the dedicated `Http2`-only listener.

**Step 2:** After the app pipeline's other `Map*` calls, driver-gated:
```csharp
if (hasDriver && syncPort > 0)
{
    app.MapZbLocalDbSync();
}
```
(Mapping is harmless when unauthenticated — the interceptor fail-closes — but gating on the port
keeps admin-only and default-OFF graphs entirely free of the endpoint.)

**Step 3: Verify both surfaces.**
```bash
dotnet build ZB.MOM.WW.OtOpcUa.slnx
```
Then a local smoke with the port on:
`ASPNETCORE_URLS=http://+:9000 OTOPCUA_ROLES=driver LocalDb__SyncListenPort=9001 dotnet run --project src/Server/ZB.MOM.WW.OtOpcUa.Host ...`
— expect startup logs to show BOTH `:9000` and `:9001` bound, and `curl -s localhost:9000/healthz`
(or the mapped health route) still answering. If the app needs SQL to boot, defer the smoke to the
Task 12 rig check but say so explicitly in the task notes.

**Step 4: Commit.** `feat(localdb): dedicated h2c listener + MapZbLocalDbSync gated on LocalDb:SyncListenPort`

---

### Task 6: `IDeploymentArtifactCache` + chunked LocalDb implementation + tests

**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 3, Task 5

**Files:**
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/IDeploymentArtifactCache.cs`
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/LocalDbDeploymentArtifactCache.cs`
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Deployment/LocalDbDeploymentArtifactCacheTests.cs`

**Step 1: Failing tests** (real temp-file `ILocalDb` with `LocalDbSetup.OnReady` applied):
- round-trip: store a 300 KiB random artifact → `GetCurrentAsync` returns byte-identical payload
  (forces ≥ 3 chunks at 128 KiB);
- retention: store 3 deployments for one cluster → only the newest 2 remain in
  `deployment_artifacts`; pointer names the newest;
- integrity: corrupt one chunk row via SQL → `GetCurrentAsync` returns null (miss), never a
  truncated artifact;
- missing pointer → null.

**Step 2: Implement:**
```csharp
public interface IDeploymentArtifactCache
{
    Task StoreAsync(string clusterId, string deploymentId, string revisionHash,
                    byte[] artifact, CancellationToken ct = default);
    Task<CachedDeploymentArtifact?> GetCurrentAsync(string clusterId, CancellationToken ct = default);
}

public sealed record CachedDeploymentArtifact(
    string DeploymentId, string RevisionHash, byte[] Artifact, DateTimeOffset AppliedAtUtc);
```
`LocalDbDeploymentArtifactCache(ILocalDb db)`:
- `StoreAsync`: one `ILocalDbTransaction` — delete any existing chunks for this `deployment_id`
  (idempotent re-store), insert chunks (raw 128 * 1024 bytes per chunk, `Convert.ToBase64String`),
  upsert the pointer (`INSERT ... ON CONFLICT(cluster_id) DO UPDATE`), then prune: delete
  `deployment_artifacts` rows whose `deployment_id` is not among the newest 2 `cached_at_utc` for
  this `cluster_id`. SHA-256 over the raw artifact into the pointer. ISO-8601 UTC timestamps.
- `GetCurrentAsync`: read pointer; read chunks `ORDER BY chunk_index`; verify count ==
  `chunk_count` and SHA-256 matches; return null on any mismatch (log a warning naming which check
  failed).
- Use `db.ExecuteAsync`/`QueryAsync` with anonymous-object parameters (`@Name` markers). Remember
  a `Dictionary<string,string>` parameter **throws** — use anonymous objects.

**Step 3:** run tests → PASS. **Step 4: Commit.**
`feat(localdb): chunked deployment-artifact cache over ILocalDb`

---

### Task 7: Cache write path in `DriverHostActor`

**Classification:** high-risk (actor model, `Props` expression-tree trap)
**Estimated implement time:** ~5 min
**Parallelizable with:** none

**Files:** (exact edit sites come from the Task 0 recon doc — cite it in the commit)
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs`
- Modify: the actor-registration site (`WithOtOpcUaRuntimeActors` / DI extension) to provide
  `IDeploymentArtifactCache`
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs` or `LocalDbRegistration` — register
  `IDeploymentArtifactCache` → `LocalDbDeploymentArtifactCache` singleton (driver branch)
- Test: the **xunit2** TestKit project used for existing DriverHostActor tests (recon names it)

**Steps:**
1. **Failing test:** after a successful artifact apply, the cache received `StoreAsync` with the
   applied deployment's id/hash/bytes (inject a recording fake `IDeploymentArtifactCache`).
   Also: a cache that throws on `StoreAsync` does NOT fail the apply (apply result unchanged, error
   logged).
2. Thread the dependency through the actor's constructor. ⚠ `Props.Create` is an expression tree:
   after adding the parameter, re-check **every** construction site for positional/named-arg
   rebinding (the recon lists them) — do not rely on the compiler.
3. Invoke `StoreAsync` fire-and-forget (`PipeTo`-style or a guarded `Task.Run` per the actor's
   existing async conventions — match whatever pattern the actor already uses for side-effect IO)
   at the post-apply point identified in recon. Cluster id from the recon-identified source.
4. Run the actor test suite for this project. Commit:
   `feat(localdb): DriverHostActor stores applied artifacts in the pair-local cache`

---

### Task 8: Boot-from-cache read path + running-from-cache signal

**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** none

**Files:**
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs`
- Test: same xunit2 TestKit project as Task 7

**Steps:**
1. **Failing tests:**
   - central fetch fails at startup AND cache has an artifact → the actor applies the cached
     artifact (assert via whatever observable the apply already exposes) and logs/flags
     running-from-cache;
   - central fetch fails AND cache empty → today's behavior exactly (Stale, no apply);
   - central fetch **succeeds** → cache is NOT consulted (fresh config wins; assert the fake
     cache's `GetCurrentAsync` was never called on the happy path).
2. Implement at the recon-identified boot-failure seam. The signal: a log warning at minimum plus,
   if the actor already publishes health/status (recon says how), a `RunningFromCache` marker on it.
   Do NOT touch `DispatchDeployment` handling — a new deployment still requires central.
3. Run tests; commit `feat(localdb): boot from the pair-local artifact cache when central SQL is unreachable`.

---

### Task 9: Delete the dormant LiteDB LocalCache subsystem

**Classification:** small
**Estimated implement time:** ~3 min
**Parallelizable with:** Task 6, Task 10, Task 11

**Files:**
- Delete: `src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/` (entire directory:
  `ILocalConfigCache`, `LiteDbConfigCache`, `GenerationSealedCache`, `ResilientConfigReader`,
  `GenerationSnapshot`, `StaleConfigFlag`, exceptions)
- Delete: `tests/Core/ZB.MOM.WW.OtOpcUa.Configuration.Tests/GenerationSealedCacheTests.cs`,
  `ResilientConfigReaderTests.cs`
- Modify: `Directory.Packages.props` + the Configuration csproj — remove the `LiteDB` package if
  nothing else references it (grep first)

**Steps:** Confirm the Task 0 recon's "nothing references it" finding still holds
(`grep -rn "ILocalConfigCache\|GenerationSealedCache\|ResilientConfigReader\|LiteDB" src/ tests/`),
delete, build the full solution, run the Configuration test project. DoD phrasing: **no references
from code** — leave any explanatory prose/comments that point readers at LocalDb instead. Add one
line to the recon doc noting LiteDB's removal. Commit:
`refactor(localdb): delete the dormant LiteDB LocalCache (superseded by ZB.MOM.WW.LocalDb)`

---

### Task 10: Health check + telemetry meter allowlist

**Classification:** small
**Estimated implement time:** ~4 min
**Parallelizable with:** Task 9, Task 11

**Files:**
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Health/LocalDbReplicationHealthCheck.cs` (or the
  directory `AddOtOpcUaHealth` uses — recon names it)
- Modify: the `AddOtOpcUaHealth` registration (driver branch)
- Modify: the observability meter list IF Task 0 found an allowlist

**Steps:**
1. Failing unit tests: replication unconfigured (no peer, no listener) → `Healthy` (default-OFF
   must not degrade a plain node); peer configured + `ISyncStatus.Connected == false` → `Degraded`;
   connected + backlog `null` → `Degraded` (unknown backlog is not healthy); connected + backlog
   small → `Healthy`.
2. Implement against `ISyncStatus` + `IOptions<ReplicationOptions>` (peer-configured = non-empty
   `PeerAddress` OR `SyncListenPort > 0` — pass the latter in via options/config).
3. **Meter allowlist:** if `AddOtOpcUaObservability` restricts meters, add
   `"ZB.MOM.WW.LocalDb.Replication"` (use `LocalDbMetrics.MeterName`) — this is a silent allowlist;
   the ScadaBridge live gate is the proof it bites. If there is no allowlist, record that in the
   recon doc and skip.
4. Commit `feat(localdb): replication health check + meter export`.

---

### Task 11: DI-pin integration tests over the real Program.cs

**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 9, Task 10

**Files:**
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDbWiringTests.cs` (xunit.v3,
  `WebApplicationFactory<Program>` — the project already builds the real Program.cs; set
  `OTOPCUA_ROLES` per test case the way neighboring tests do)

**Pins (each its own test):**
1. Driver graph: `ILocalDb` resolves, is a singleton, `ReplicatedTables` ordinal-sorted ==
   `["deployment_artifacts", "deployment_pointer"]` — exact set, both directions load-bearing.
2. Driver graph: `IDeploymentArtifactCache` resolves to `LocalDbDeploymentArtifactCache`.
3. Default-OFF pin: `ISyncStatus` resolves with `Connected == false`, `PeerNodeId == null`.
4. Admin-only graph: `ILocalDb` is NOT registered (`GetService<ILocalDb>()` is null) and boot does
   not demand `LocalDb:Path`.
5. Health: the LocalDb health check is registered in the driver graph.

Point `LocalDb:Path` at a per-test temp file via the factory's config overrides;
`ClearAllPools` + delete in cleanup. These tests exist because DI extensions without a
container-built test have shipped inert three times in this family (Secrets 0.2.0/0.2.2,
ScadaBridge#22). Commit `test(localdb): DI pins over the real host graph`.

---

### Task 12: Two-node convergence harness + tests

**Classification:** high-risk
**Estimated implement time:** ~5 min (harness) + ~4 min (scenarios) — split the commit if needed
**Parallelizable with:** none (depends on Tasks 2, 3, 5, 6)

**Files:**
- Create: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDb/LocalDbPairHarness.cs`
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDb/LocalDbPairConvergenceTests.cs`

**Harness:** copy the pattern from
`~/Desktop/ScadaBridge/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbSitePairHarness.cs` and
the library's `ConvergenceFixture` (`~/Desktop/scadaproj/ZB.MOM.WW.LocalDb/tests/.../Convergence/ConvergenceFixture.cs`):
- Two full stacks over a real loopback Kestrel h2c socket (port 0), **through the real
  `LocalDbSyncAuthInterceptor`** with a shared test key; node A initiator, node B passive.
- Both initialized via the **production `LocalDbSetup.OnReady`** — a hand-written schema proves
  only that the test agrees with itself.
- DBs owned by the harness, registered as pre-constructed singletons (so MS.DI won't dispose them
  on host teardown); `KillTransportAsync`/`RestartTransportAsync`; tight `FlushInterval` (50 ms),
  bounded backoff (2 s); temp files, `ClearAllPools` cleanup; a collection definition to serialize
  these tests.

**Scenarios:**
1. `ArtifactStoredOnA_ConvergesToB_ByteIdentical` — store a multi-chunk artifact via
   `LocalDbDeploymentArtifactCache` on A; B's cache returns byte-identical bytes; both nodes'
   `__localdb_row_version` rows for the pointer carry the **same HLC and origin node id** (B holds
   A's row, not a re-derived one).
2. `RetentionPruneOnA_TombstonesReachB` — third deployment on A prunes the first; B's chunk rows
   for the pruned deployment disappear and `__localdb_row_version WHERE is_tombstone=1` rows exist
   on B.
3. `WritesWhileTransportDown_SurviveRejoin` — kill transport, store on A, restart, converge.
4. `WrongApiKey_NeverConverges` — harness with mismatched keys: assert **no** convergence after a
   bounded wait AND (positive control) that the same scenario with matching keys converges — an
   absence assertion without a positive control passed vacuously in ScadaBridge.

**DoD within this task:** temporarily comment out the two `RegisterReplicated` calls and confirm
scenarios 1–3 go red (run locally, do not commit the red state); restore. Record the red/green
evidence in the task notes. Commit `test(localdb): 2-node convergence harness + scenarios`.

---

### Task 13: docker-dev rig configuration

**Classification:** small
**Estimated implement time:** ~4 min
**Parallelizable with:** Task 14

**Files:**
- Modify: `docker-dev/docker-compose.yml`

**Steps** (adjust to the recon's findings on volumes/env style):
1. Every driver node: a named volume (or existing data mount) backing `/app/data`, and
   `LocalDb__Path=/app/data/otopcua-localdb.db`.
2. **site-a pair only** (mirror the ScadaBridge default-OFF posture — site-b stays unreplicated as
   the pin):
   - site-a-1: `LocalDb__SyncListenPort=9001`, `LocalDb__Replication__PeerAddress=http://site-a-2:9001`,
     `LocalDb__Replication__ApiKey=dev-site-a-localdb-sync-key`, `LocalDb__Replication__MaxBatchSize=16`
   - site-a-2: `LocalDb__SyncListenPort=9001`, same `ApiKey`, same `MaxBatchSize`, **no PeerAddress**
     (passive; the stream is bidirectional).
   The key must be byte-identical on both — the interceptor fail-closes on any mismatch and the
   pair silently stops converging.
3. `MaxBatchSize=16` because batching is row-count-only against gRPC's 4 MB cap and artifact chunk
   rows are ≈ 171 KB (16 × 171 KB ≈ 2.7 MB worst case).
4. `docker compose config` to validate; do NOT bring the rig up in this task (the live gate task
   owns rig runs).
5. Commit `chore(localdb): rig config — consolidated path everywhere, replication on the site-a pair`.

---

### Task 14: Documentation

**Classification:** small
**Estimated implement time:** ~4 min
**Parallelizable with:** Task 13

**Files:**
- Create: `docs/operations/2026-07-20-localdb-pair-replication.md` (runbook: enable/disable,
  ApiKey handling via `${secret:}`, stop/start-together rule, tombstone-retention resurrection
  window, MaxBatchSize row-count-vs-4MB, never host-`sqlite3` a live WAL DB / cp-triplet recipe)
- Modify: `docs/Redundancy.md` (a short "pair-local config cache" section: what replicates, what
  boot-from-cache does and does not cover)
- Modify: `CLAUDE.md` (this repo): LocalDb adoption row/paragraph — state Phase 1 scope, default-OFF,
  rig-only enablement
- Modify: `docs/Configuration.md` if it documents config keys (add the `LocalDb` section)

Commit `docs(localdb): phase-1 runbook + redundancy/config docs`.

---

### Task 15: DoD sweep (offline)

**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** none (final offline task)

**Steps:**
1. `dotnet build ZB.MOM.WW.OtOpcUa.slnx` → **0 warnings** (warnings-as-errors).
2. `dotnet test ZB.MOM.WW.OtOpcUa.slnx` → full suite green; record counts. Baseline any
   pre-existing failures BEFORE this branch (`git stash` → run → unstash) rather than assuming;
   report deltas only.
3. Greps (phrase as "no references from code"; explanatory comments may remain):
   `grep -rn "LiteDB\|ILocalConfigCache\|GenerationSealedCache" src/ tests/` → no code references.
4. Confirm the positive-control evidence from Task 12 is recorded.
5. Update `docs/plans/2026-07-20-localdb-adoption-phase1.md.tasks.json` statuses.
6. Commit `chore(localdb): phase-1 DoD sweep` and STOP. Report: branch name, commit list, test
   counts, and that the live gate (Task 16) needs the rig + operator go-ahead.

---

### Task 16: Live gate on the docker-dev rig (run only with explicit user go-ahead)

**Classification:** high-risk (rig)
**Estimated implement time:** ~30 min wall-clock (mostly waiting)
**Parallelizable with:** none

**Preamble:** All DB inspection via `cp` of the `db`/`-wal`/`-shm` triplet out of the container
(`docker cp`) and querying the copy — never host `sqlite3` on the live file (it poisons the WAL
across virtiofs; root cause of the 2026-07-20 ScadaBridge incident). Metrics via
`docker run --rm --network container:<node> curlimages/curl:latest -s localhost:<port>/metrics`.
If an anomaly appears within seconds of one of your own measurements, suspect the measurement
first — restart both nodes and re-run untouched before blaming the library.

**Checks (all must PASS; record evidence in `docs/plans/2026-07-20-localdb-phase1-live-gate.md`):**
1. Rig up, site-a pair healthy, `/metrics` on both shows `localdb_` series (proves the meter
   allowlist work).
2. Deploy a config through the normal deploy API → both site-a nodes' `deployment_pointer` +
   `deployment_artifacts` rows are byte-identical with **identical HLC + origin node id** (one
   node's row replicated, not two independent derivations — both nodes also locally store after
   apply, so expect LWW-converged identical content either way; assert convergence, and note which
   origin won).
3. **Boot-from-cache:** stop the SQL container; restart site-a-2; it comes up serving the cached
   config with the running-from-cache signal in its log; start SQL again; node recovers fresh
   behavior.
4. **Replicated-cache payoff:** wipe site-a-2's local DB file (container stopped), restart it with
   SQL **up**, let replication repopulate the cache from site-a-1, then repeat check 3's SQL-down
   restart — it boots from a cache it never wrote itself.
5. Transport kill (pause site-a-2 container): store/deploy on a-1, oplog depth rises; unpause;
   drains to 0; converged.
6. Retention: deploy a 3rd config; pruned deployment's chunks gone on BOTH nodes with tombstone
   rows present on both.
7. Both-nodes-together restart: clean rejoin, identical counts, zero
   `disk I/O error`/`SQLITE_IOERR`/`corrupt` in either log.
8. site-b pair (default-OFF pin): LocalDb file exists and works locally, `ISyncStatus` health
   Healthy, no sync connections, no listener on 9001.

Record PASS/FAIL per check with the actual evidence (row counts, md5s, log lines). Commit the gate
doc. Do not merge — report.

---

## Task persistence

Tasks file: `docs/plans/2026-07-20-localdb-adoption-phase1.md.tasks.json` (same directory).
Update statuses as tasks complete; any deviation from this plan gets a `deviation` note on the task
(the ScadaBridge tasks.json convention — the record of *why* is what makes the plan auditable).