dohertj2/lmxopcua
1
0
Fork 0
Code Issues 8 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
d8a85c3d890412e2a212e5e4e95ba7b426b5a2aa
lmxopcua/docs/plans/2026-07-20-localdb-adoption-phase1.md
T
Joseph Doherty 42f8550716 docs(localdb): phase-1 recon findings
2026-07-20 10:18:01 -04:00

35 KiB
Raw Blame History

OtOpcUa LocalDb Adoption — Phase 1 Implementation Plan

For Claude: REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.

Execution model: This plan is optimized for Claude Opus agents (claude --model opus). Dispatch implementer subagents on Opus for every task classified high-risk; small/trivial tasks may use Sonnet. Work on branch feat/localdb-phase1 in this repo (~/Desktop/OtOpcUa, remote lmxopcua). Do NOT merge to master as part of this plan — stop at the DoD task and report.

Design authority: ~/Desktop/scadaproj/docs/plans/2026-07-20-otopcua-localdb-design.md. Read it before Task 0. The reference adoption is ScadaBridge (merged PR #23) — when in doubt, mirror ~/Desktop/ScadaBridge (files named per task below).

Goal: Give every driver-role OtOpcUa node a consolidated ZB.MOM.WW.LocalDb database that caches the deployed-configuration artifact (chunked), boots from that cache when central SQL Server is unreachable, and optionally replicates it to the node's redundant pair peer over the library's gRPC sync (default-OFF, fail-closed bearer auth).

Architecture: AddZbLocalDb/AddZbLocalDbReplication wired in a new LocalDbRegistration (mirroring SecretsRegistration), DDL + RegisterReplicated in LocalDbSetup.OnReady, a dedicated Kestrel h2c listener for MapZbLocalDbSync gated on LocalDb:SyncListenPort, a consumer-supplied fail-closed bearer interceptor, and an IDeploymentArtifactCache seam written by DriverHostActor after each successful apply and read as a boot fallback. The dormant LiteDB LocalCache subsystem is deleted as superseded.

Tech stack: .NET 10, ZB.MOM.WW.LocalDb/.Replication/.Contracts 0.1.1 (Gitea feed), Grpc.AspNetCore 2.76.0, Akka.NET (existing), xunit.v3 (Host.IntegrationTests) + xunit2 TestKit (actor tests).

Hard rules (from the ScadaBridge adoption — violating any of these is a defect):

  1. Order inside OnReady is load-bearing: DDL → RegisterReplicated → writes. Rows written before registration are never captured or snapshotted — silently, forever.
  2. No autoincrement PKs, no BLOB columns in replicated tables.
  3. The library's passive sync endpoint verifies no ApiKey — the host interceptor is the only auth, and it must be fail-closed (no key configured ⇒ deny everything).
  4. ILocalDb.CreateConnection() returns an already-open, pragma-configured connection with the zb_hlc_next() UDF. Never call Open() on it. There is no in-memory mode — tests use temp files + SqliteConnection.ClearAllPools() before delete.
  5. Grpc.Core.Testing does not exist on grpc-dotnet — hand-roll fake ServerCallContexts.
  6. Explicit Kestrel Listen* calls make Kestrel ignore ASPNETCORE_URLS — when adding the sync listener you must re-bind the existing HTTP port in the same block, and verify it on the rig.
  7. Never run host sqlite3 against a live bind-mounted WAL DB (copy the db/-wal/-shm triplet with cp instead). aspnet:10.0 containers have no curl — use a curlimages/curl sidecar with --network container:<name>.

Task 0: Preflight + recon (produces docs/plans/2026-07-20-localdb-phase1-recon.md)

Classification: standard Estimated implement time: ~5 min Parallelizable with: none (everything downstream consumes its findings)

Files:

  • Create: docs/plans/2026-07-20-localdb-phase1-recon.md
  • Read-only: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs, src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DeploymentArtifact.cs, src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs, src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/SecretsRegistration.cs, Directory.Packages.props, docker-dev/docker-compose.yml, the observability registration (AddOtOpcUaObservability implementation), src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/*

Step 1: Verify the feed packages restore.

Run:

cd ~/Desktop/OtOpcUa && dotnet package search ZB.MOM.WW.LocalDb --source dohertj2-gitea --format json | head -50

Expected: ZB.MOM.WW.LocalDb, .Replication, .Contracts at 0.1.1. If absent → STOP, report.

Step 2: Map and record, with file:line citations, each of:

  1. Deploy fetch path: where DriverHostActor loads Deployment.ArtifactBlob (the _dbFactory.CreateDbContext() sites, ~L1472/1543), the artifact's runtime type (DeploymentArtifact — how it's deserialized from the blob), the type/format of DeploymentId and RevisionHash, and where a successful apply completes (the point after which the cache write belongs).
  2. Cold-boot path: what the actor does at startup before any DispatchDeployment arrives — does it query central SQL for the current deployment? Where does the failure path land (the Stale state, DbHealthProbeActor interaction, L1345 dispatch-ignore)? Identify the exact seam where "central fetch failed at boot" is known — that is where the cache fallback goes.
  3. Cluster identity: how a driver node knows its ClusterId (config key / ClusterNode row / IClusterRoleInfo) — the cache is keyed by it.
  4. DriverHostActor construction: how its dependencies are injected (WithOtOpcUaRuntimeActors, Props wiring) so IDeploymentArtifactCache can be threaded in. ⚠ Props.Create builds an expression tree — adding a parameter silently rebinds out-of-position named args at call sites; list every construction site.
  5. Telemetry allowlist: does AddOtOpcUaObservability restrict meters (ZbTelemetryOptions.Meters or equivalent)? If yes, record where the list lives.
  6. Kestrel/URLs: confirm the Host binds via ASPNETCORE_URLS=http://+:9000 with no ConfigureKestrel calls; record any existing builder.WebHost usage.
  7. LiteDB LocalCache: confirm (grep) that nothing outside src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/ and its tests references ILocalConfigCache/GenerationSealedCache/ResilientConfigReader/LiteDB.
  8. docker-dev volumes: whether nodes have a writable data volume; record what LocalDb:Path should be per node and how env vars are passed (Cluster__* style double-underscore).

Step 3: STOP conditions — halt the plan and report instead of improvising if:

  • the artifact apply path cannot be given a post-apply hook without restructuring the actor;
  • DeploymentId/cluster identity are not stable strings/GUIDs;
  • LiteDB LocalCache turns out to be referenced by live code.

Step 4: Commit.

git add docs/plans/2026-07-20-localdb-phase1-recon.md
git commit -m "docs(localdb): phase-1 recon findings"

Task 1: Package references and pins

Classification: small Estimated implement time: ~3 min Parallelizable with: none (all code tasks build on it)

Files:

  • Modify: Directory.Packages.props
  • Modify: src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj
  • Modify: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ZB.MOM.WW.OtOpcUa.Runtime.csproj

Step 1: Add to Directory.Packages.props (versions exact):

<PackageVersion Include="ZB.MOM.WW.LocalDb" Version="0.1.1" />
<PackageVersion Include="ZB.MOM.WW.LocalDb.Replication" Version="0.1.1" />
<PackageVersion Include="Grpc.AspNetCore" Version="2.76.0" />

Check the existing Grpc.Net.Client/Grpc.Core.Api are already ≥ 2.76.0 and Google.Protobuf ≥ 3.34.1 (the LocalDb.Replication nuspec floors); raise if not. Confirm a SQLitePCLRaw pin ≥ 2.1.12 exists for the transitive Microsoft.Data.Sqlite chain (the family advisory GHSA-2m69-gcr7-jv3q); Core.AlarmHistorian already pins bundle_e_sqlite3 2.1.12 — add SQLitePCLRaw.lib.e_sqlite3 2.1.12 as an explicit reference in the Host if the audit flags it.

Step 2: Host csproj: ZB.MOM.WW.LocalDb, ZB.MOM.WW.LocalDb.Replication, Grpc.AspNetCore. Runtime csproj: ZB.MOM.WW.LocalDb only (it needs ILocalDb, not the sync engine).

Step 3: dotnet build ZB.MOM.WW.OtOpcUa.slnx → 0 warnings/errors (repo builds warnings-as-errors).

Step 4: Commit. git commit -m "build(localdb): reference ZB.MOM.WW.LocalDb 0.1.1 + Grpc.AspNetCore"

Task 2: Schema + LocalDbSetup.OnReady + registration tests

Classification: standard Estimated implement time: ~5 min Parallelizable with: Task 3

Files:

  • Create: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/DeploymentCacheSchema.cs
  • Create: src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbSetup.cs
  • Test: tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/LocalDbSetupTests.cs (create the test project reference layout to match neighboring Host unit tests; if Host has no unit-test project, place in the closest existing one and note the deviation)

Step 1: Write the failing tests first — using a real temp-file ILocalDb (build it via new ServiceCollection().AddZbLocalDb(config, LocalDbSetup.OnReady) with LocalDb:Path pointed at a temp file; dispose + SqliteConnection.ClearAllPools() in cleanup):

  • OnReady_RegistersExactlyTheTwoDeploymentTables — db.ReplicatedTables.Keys ordinal-sorted equals ["deployment_artifacts", "deployment_pointer"] (assert BOTH directions: no fewer, no more).
  • DeploymentArtifacts_PkIsDeploymentIdPlusChunkIndex and DeploymentPointer_PkIsClusterId (pin ReplicatedTable.PkColumns).
  • RowsWrittenAfterOnReady_EnterTheOplog — insert a row, assert SELECT COUNT(*) FROM __localdb_oplog ≥ 1 (pins the DDL→register ordering; this is the assertion that catches a silently-broken CDC).

Step 2: Run tests, watch them fail (types don't exist yet).

Step 3: Implement. DeploymentCacheSchema depends only on Microsoft.Data.Sqlite (the ScadaBridge *Schema.Apply pattern — self-sufficient for direct construction in tests):

namespace ZB.MOM.WW.OtOpcUa.Runtime.Deployment;

public static class DeploymentCacheSchema
{
    public static void Apply(SqliteConnection connection)
    {
        using var cmd = connection.CreateCommand();
        cmd.CommandText = """
            CREATE TABLE IF NOT EXISTS deployment_artifacts (
                deployment_id   TEXT    NOT NULL,
                chunk_index     INTEGER NOT NULL,
                cluster_id      TEXT    NOT NULL,
                revision_hash   TEXT    NOT NULL,
                chunk_count     INTEGER NOT NULL,
                chunk_base64    TEXT    NOT NULL,
                cached_at_utc   TEXT    NOT NULL,
                PRIMARY KEY (deployment_id, chunk_index)
            );
            CREATE TABLE IF NOT EXISTS deployment_pointer (
                cluster_id        TEXT NOT NULL PRIMARY KEY,
                deployment_id     TEXT NOT NULL,
                revision_hash     TEXT NOT NULL,
                artifact_sha256   TEXT NOT NULL,
                applied_at_utc    TEXT NOT NULL
            );
            """;
        cmd.ExecuteNonQuery();
    }
}

LocalDbSetup (Host):

namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;

internal static class LocalDbSetup
{
    // ORDER IS LOAD-BEARING: DDL, then RegisterReplicated, then (nothing else in Phase 1).
    // Rows written before RegisterReplicated are invisible to the peer forever, silently.
    public static void OnReady(ILocalDb db)
    {
        using var connection = db.CreateConnection(); // already open — do NOT call Open()
        DeploymentCacheSchema.Apply(connection);
        db.RegisterReplicated("deployment_artifacts");
        db.RegisterReplicated("deployment_pointer");
    }
}

Step 4: dotnet test --filter "FullyQualifiedName~LocalDbSetupTests" → PASS.

Step 5: Commit. git commit -m "feat(localdb): deployment-cache schema + OnReady registration"

Task 3: Fail-closed sync auth interceptor + tests

Classification: standard Estimated implement time: ~4 min Parallelizable with: Task 2

Files:

  • Create: src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbSyncAuthInterceptor.cs
  • Test: tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/LocalDbSyncAuthInterceptorTests.cs

Port ScadaBridge's src/ZB.MOM.WW.ScadaBridge.Host/LocalDbSyncAuthInterceptor.cs (read it first; keep behavior identical, adjust namespace only). Semantics to preserve exactly:

  • ServicePrefix = "/localdb_sync.v1.LocalDbSync/"; any other method path passes through untouched.
  • Expected token from IOptions<ReplicationOptions>.Value.ApiKey.
  • Fail-closed: null/empty configured key ⇒ RpcException(new Status(StatusCode.PermissionDenied, ...)) for every sync call. Missing/mismatched bearer ⇒ same.
  • CryptographicOperations.FixedTimeEquals over UTF-8 bytes; header authorization, case-insensitive, "Bearer " prefix. Override all four server handler kinds.

Tests (hand-rolled fake ServerCallContext — Grpc.Core.Testing does not exist on grpc-dotnet; copy ScadaBridge's LocalDbSyncAuthInterceptorTests.cs fake): non-sync method passes with no key; sync + no configured key → PermissionDenied; wrong bearer → PermissionDenied; correct bearer → passes.

Write tests first (fail), implement, dotnet test --filter "FullyQualifiedName~LocalDbSyncAuthInterceptor", commit feat(localdb): fail-closed bearer interceptor for the sync endpoint.

Task 4: LocalDbRegistration + Program.cs DI wiring + config defaults

Classification: high-risk (touches Program.cs / role gating) Estimated implement time: ~5 min Parallelizable with: none (Task 5 edits the same file)

Files:

  • Create: src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbRegistration.cs
  • Modify: src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs (inside the hasDriver branch)
  • Modify: src/Server/ZB.MOM.WW.OtOpcUa.Host/appsettings.json

Step 1: LocalDbRegistration, mirroring SecretsRegistration's shape:

namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;

internal static class LocalDbRegistration
{
    /// Driver-role nodes only. Storage is unconditional; replication stays inert until
    /// LocalDb:Replication:PeerAddress (initiator) / LocalDb:SyncListenPort (listener) are set.
    public static IServiceCollection AddOtOpcUaLocalDb(
        this IServiceCollection services, IConfiguration configuration)
    {
        services.AddZbLocalDb(configuration, LocalDbSetup.OnReady);
        services.AddZbLocalDbReplication(configuration);
        return services;
    }

    public static int SyncListenPort(IConfiguration configuration) =>
        configuration.GetValue<int>("LocalDb:SyncListenPort");
}

Step 2: Program.cs, in the hasDriver block (near AddAlarmHistorian): builder.Services.AddOtOpcUaLocalDb(builder.Configuration); and builder.Services.AddGrpc(o => o.Interceptors.Add<LocalDbSyncAuthInterceptor>()); (AddGrpc only under hasDriver — admin-only nodes expose no sync surface).

Step 3: appsettings.json:

"LocalDb": {
  "Path": "./data/otopcua-localdb.db",
  "SyncListenPort": 0,
  "Replication": {}
}

LocalDb:Path is ValidateOnStart-required once AddZbLocalDb runs — since registration is driver-gated, admin-only configs need nothing. Check the role-overlay files (appsettings.driver.json / appsettings.admin-driver.json) and any deploy templates for conflicting LocalDb keys.

Step 4: Build + full Host unit tests. Verify an admin-only graph doesn't require the key: this is pinned properly in Task 10's integration tests, but do a quick OTOPCUA_ROLES=admin dotnet run smoke only if cheap; otherwise rely on Task 10.

Step 5: Commit. feat(localdb): wire AddOtOpcUaLocalDb into the driver role

Task 5: Dedicated h2c sync listener + endpoint mapping (THE Kestrel task)

Classification: high-risk Estimated implement time: ~5 min Parallelizable with: none (Program.cs)

Files:

  • Modify: src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs

Why high-risk: the Host today binds exclusively via ASPNETCORE_URLS=http://+:9000. Any explicit ConfigureKestrel(... Listen*) makes Kestrel ignore URLs entirely (it logs "Overriding address(es)"), which would silently kill the AdminUI/deploy API behind Traefik.

Step 1: Add, before builder.Build():

var syncPort = LocalDbRegistration.SyncListenPort(builder.Configuration);
if (hasDriver && syncPort > 0)
{
    // Explicit Listen* replaces ASPNETCORE_URLS wholesale, so re-bind the primary HTTP
    // endpoint here too. Parse it from the configured URLs rather than hard-coding 9000.
    var urls = builder.Configuration["ASPNETCORE_URLS"] ?? builder.Configuration["urls"] ?? "http://+:9000";
    var httpPort = new Uri(urls.Split(';')[0].Replace("+", "localhost").Replace("*", "localhost")).Port;
    builder.WebHost.ConfigureKestrel(k =>
    {
        k.ListenAnyIP(httpPort);                                            // HTTP/1.1 (existing surface)
        k.ListenAnyIP(syncPort, o => o.Protocols = HttpProtocols.Http2);    // h2c, prior-knowledge gRPC
    });
}

When syncPort == 0 (the default) nothing changes — URLs binding stays untouched. Cleartext Http1AndHttp2 cannot serve prior-knowledge h2c, hence the dedicated Http2-only listener.

Step 2: After the app pipeline's other Map* calls, driver-gated:

if (hasDriver && syncPort > 0)
{
    app.MapZbLocalDbSync();
}

(Mapping is harmless when unauthenticated — the interceptor fail-closes — but gating on the port keeps admin-only and default-OFF graphs entirely free of the endpoint.)

Step 3: Verify both surfaces.

dotnet build ZB.MOM.WW.OtOpcUa.slnx

Then a local smoke with the port on: ASPNETCORE_URLS=http://+:9000 OTOPCUA_ROLES=driver LocalDb__SyncListenPort=9001 dotnet run --project src/Server/ZB.MOM.WW.OtOpcUa.Host ... — expect startup logs to show BOTH :9000 and :9001 bound, and curl -s localhost:9000/healthz (or the mapped health route) still answering. If the app needs SQL to boot, defer the smoke to the Task 12 rig check but say so explicitly in the task notes.

Step 4: Commit. feat(localdb): dedicated h2c listener + MapZbLocalDbSync gated on LocalDb:SyncListenPort

Task 6: IDeploymentArtifactCache + chunked LocalDb implementation + tests

Classification: standard Estimated implement time: ~5 min Parallelizable with: Task 3, Task 5

Files:

  • Create: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/IDeploymentArtifactCache.cs
  • Create: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/LocalDbDeploymentArtifactCache.cs
  • Test: tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Deployment/LocalDbDeploymentArtifactCacheTests.cs

Step 1: Failing tests (real temp-file ILocalDb with LocalDbSetup.OnReady applied):

  • round-trip: store a 300 KiB random artifact → GetCurrentAsync returns byte-identical payload (forces ≥ 3 chunks at 128 KiB);
  • retention: store 3 deployments for one cluster → only the newest 2 remain in deployment_artifacts; pointer names the newest;
  • integrity: corrupt one chunk row via SQL → GetCurrentAsync returns null (miss), never a truncated artifact;
  • missing pointer → null.

Step 2: Implement:

public interface IDeploymentArtifactCache
{
    Task StoreAsync(string clusterId, string deploymentId, string revisionHash,
                    byte[] artifact, CancellationToken ct = default);
    Task<CachedDeploymentArtifact?> GetCurrentAsync(string clusterId, CancellationToken ct = default);
}

public sealed record CachedDeploymentArtifact(
    string DeploymentId, string RevisionHash, byte[] Artifact, DateTimeOffset AppliedAtUtc);

LocalDbDeploymentArtifactCache(ILocalDb db):

  • StoreAsync: one ILocalDbTransaction — delete any existing chunks for this deployment_id (idempotent re-store), insert chunks (raw 128 * 1024 bytes per chunk, Convert.ToBase64String), upsert the pointer (INSERT ... ON CONFLICT(cluster_id) DO UPDATE), then prune: delete deployment_artifacts rows whose deployment_id is not among the newest 2 cached_at_utc for this cluster_id. SHA-256 over the raw artifact into the pointer. ISO-8601 UTC timestamps.
  • GetCurrentAsync: read pointer; read chunks ORDER BY chunk_index; verify count == chunk_count and SHA-256 matches; return null on any mismatch (log a warning naming which check failed).
  • Use db.ExecuteAsync/QueryAsync with anonymous-object parameters (@Name markers). Remember a Dictionary<string,string> parameter throws — use anonymous objects.

Step 3: run tests → PASS. Step 4: Commit. feat(localdb): chunked deployment-artifact cache over ILocalDb

Task 7: Cache write path in DriverHostActor

Classification: high-risk (actor model, Props expression-tree trap) Estimated implement time: ~5 min Parallelizable with: none

Files: (exact edit sites come from the Task 0 recon doc — cite it in the commit)

  • Modify: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs
  • Modify: the actor-registration site (WithOtOpcUaRuntimeActors / DI extension) to provide IDeploymentArtifactCache
  • Modify: src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs or LocalDbRegistration — register IDeploymentArtifactCache → LocalDbDeploymentArtifactCache singleton (driver branch)
  • Test: the xunit2 TestKit project used for existing DriverHostActor tests (recon names it)

Steps:

  1. Failing test: after a successful artifact apply, the cache received StoreAsync with the applied deployment's id/hash/bytes (inject a recording fake IDeploymentArtifactCache). Also: a cache that throws on StoreAsync does NOT fail the apply (apply result unchanged, error logged).
  2. Thread the dependency through the actor's constructor. ⚠ Props.Create is an expression tree: after adding the parameter, re-check every construction site for positional/named-arg rebinding (the recon lists them) — do not rely on the compiler.
  3. Invoke StoreAsync fire-and-forget (PipeTo-style or a guarded Task.Run per the actor's existing async conventions — match whatever pattern the actor already uses for side-effect IO) at the post-apply point identified in recon. Cluster id from the recon-identified source.
  4. Run the actor test suite for this project. Commit: feat(localdb): DriverHostActor stores applied artifacts in the pair-local cache

Task 8: Boot-from-cache read path + running-from-cache signal

Classification: high-risk Estimated implement time: ~5 min Parallelizable with: none

Files:

  • Modify: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs
  • Test: same xunit2 TestKit project as Task 7

Steps:

  1. Failing tests:
    • central fetch fails at startup AND cache has an artifact → the actor applies the cached artifact (assert via whatever observable the apply already exposes) and logs/flags running-from-cache;
    • central fetch fails AND cache empty → today's behavior exactly (Stale, no apply);
    • central fetch succeeds → cache is NOT consulted (fresh config wins; assert the fake cache's GetCurrentAsync was never called on the happy path).
  2. Implement at the recon-identified boot-failure seam. The signal: a log warning at minimum plus, if the actor already publishes health/status (recon says how), a RunningFromCache marker on it. Do NOT touch DispatchDeployment handling — a new deployment still requires central.
  3. Run tests; commit feat(localdb): boot from the pair-local artifact cache when central SQL is unreachable.

Task 9: Delete the dormant LiteDB LocalCache subsystem

Classification: small Estimated implement time: ~3 min Parallelizable with: Task 6, Task 10, Task 11

Files:

  • Delete: src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/ (entire directory: ILocalConfigCache, LiteDbConfigCache, GenerationSealedCache, ResilientConfigReader, GenerationSnapshot, StaleConfigFlag, exceptions)
  • Delete: tests/Core/ZB.MOM.WW.OtOpcUa.Configuration.Tests/GenerationSealedCacheTests.cs, ResilientConfigReaderTests.cs
  • Modify: Directory.Packages.props + the Configuration csproj — remove the LiteDB package if nothing else references it (grep first)

Steps: Confirm the Task 0 recon's "nothing references it" finding still holds (grep -rn "ILocalConfigCache\|GenerationSealedCache\|ResilientConfigReader\|LiteDB" src/ tests/), delete, build the full solution, run the Configuration test project. DoD phrasing: no references from code — leave any explanatory prose/comments that point readers at LocalDb instead. Add one line to the recon doc noting LiteDB's removal. Commit: refactor(localdb): delete the dormant LiteDB LocalCache (superseded by ZB.MOM.WW.LocalDb)

Task 10: Health check + telemetry meter allowlist

Classification: small Estimated implement time: ~4 min Parallelizable with: Task 9, Task 11

Files:

  • Create: src/Server/ZB.MOM.WW.OtOpcUa.Host/Health/LocalDbReplicationHealthCheck.cs (or the directory AddOtOpcUaHealth uses — recon names it)
  • Modify: the AddOtOpcUaHealth registration (driver branch)
  • Modify: the observability meter list IF Task 0 found an allowlist

Steps:

  1. Failing unit tests: replication unconfigured (no peer, no listener) → Healthy (default-OFF must not degrade a plain node); peer configured + ISyncStatus.Connected == false → Degraded; connected + backlog null → Degraded (unknown backlog is not healthy); connected + backlog small → Healthy.
  2. Implement against ISyncStatus + IOptions<ReplicationOptions> (peer-configured = non-empty PeerAddress OR SyncListenPort > 0 — pass the latter in via options/config).
  3. Meter allowlist: if AddOtOpcUaObservability restricts meters, add "ZB.MOM.WW.LocalDb.Replication" (use LocalDbMetrics.MeterName) — this is a silent allowlist; the ScadaBridge live gate is the proof it bites. If there is no allowlist, record that in the recon doc and skip.
  4. Commit feat(localdb): replication health check + meter export.

Task 11: DI-pin integration tests over the real Program.cs

Classification: standard Estimated implement time: ~5 min Parallelizable with: Task 9, Task 10

Files:

  • Test: tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDbWiringTests.cs (xunit.v3, WebApplicationFactory<Program> — the project already builds the real Program.cs; set OTOPCUA_ROLES per test case the way neighboring tests do)

Pins (each its own test):

  1. Driver graph: ILocalDb resolves, is a singleton, ReplicatedTables ordinal-sorted == ["deployment_artifacts", "deployment_pointer"] — exact set, both directions load-bearing.
  2. Driver graph: IDeploymentArtifactCache resolves to LocalDbDeploymentArtifactCache.
  3. Default-OFF pin: ISyncStatus resolves with Connected == false, PeerNodeId == null.
  4. Admin-only graph: ILocalDb is NOT registered (GetService<ILocalDb>() is null) and boot does not demand LocalDb:Path.
  5. Health: the LocalDb health check is registered in the driver graph.

Point LocalDb:Path at a per-test temp file via the factory's config overrides; ClearAllPools + delete in cleanup. These tests exist because DI extensions without a container-built test have shipped inert three times in this family (Secrets 0.2.0/0.2.2, ScadaBridge#22). Commit test(localdb): DI pins over the real host graph.

Task 12: Two-node convergence harness + tests

Classification: high-risk Estimated implement time: ~5 min (harness) + ~4 min (scenarios) — split the commit if needed Parallelizable with: none (depends on Tasks 2, 3, 5, 6)

Files:

  • Create: tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDb/LocalDbPairHarness.cs
  • Test: tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDb/LocalDbPairConvergenceTests.cs

Harness: copy the pattern from ~/Desktop/ScadaBridge/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbSitePairHarness.cs and the library's ConvergenceFixture (~/Desktop/scadaproj/ZB.MOM.WW.LocalDb/tests/.../Convergence/ConvergenceFixture.cs):

  • Two full stacks over a real loopback Kestrel h2c socket (port 0), through the real LocalDbSyncAuthInterceptor with a shared test key; node A initiator, node B passive.
  • Both initialized via the production LocalDbSetup.OnReady — a hand-written schema proves only that the test agrees with itself.
  • DBs owned by the harness, registered as pre-constructed singletons (so MS.DI won't dispose them on host teardown); KillTransportAsync/RestartTransportAsync; tight FlushInterval (50 ms), bounded backoff (2 s); temp files, ClearAllPools cleanup; a collection definition to serialize these tests.

Scenarios:

  1. ArtifactStoredOnA_ConvergesToB_ByteIdentical — store a multi-chunk artifact via LocalDbDeploymentArtifactCache on A; B's cache returns byte-identical bytes; both nodes' __localdb_row_version rows for the pointer carry the same HLC and origin node id (B holds A's row, not a re-derived one).
  2. RetentionPruneOnA_TombstonesReachB — third deployment on A prunes the first; B's chunk rows for the pruned deployment disappear and __localdb_row_version WHERE is_tombstone=1 rows exist on B.
  3. WritesWhileTransportDown_SurviveRejoin — kill transport, store on A, restart, converge.
  4. WrongApiKey_NeverConverges — harness with mismatched keys: assert no convergence after a bounded wait AND (positive control) that the same scenario with matching keys converges — an absence assertion without a positive control passed vacuously in ScadaBridge.

DoD within this task: temporarily comment out the two RegisterReplicated calls and confirm scenarios 1–3 go red (run locally, do not commit the red state); restore. Record the red/green evidence in the task notes. Commit test(localdb): 2-node convergence harness + scenarios.

Task 13: docker-dev rig configuration

Classification: small Estimated implement time: ~4 min Parallelizable with: Task 14

Files:

  • Modify: docker-dev/docker-compose.yml

Steps (adjust to the recon's findings on volumes/env style):

  1. Every driver node: a named volume (or existing data mount) backing /app/data, and LocalDb__Path=/app/data/otopcua-localdb.db.
  2. site-a pair only (mirror the ScadaBridge default-OFF posture — site-b stays unreplicated as the pin):
    • site-a-1: LocalDb__SyncListenPort=9001, LocalDb__Replication__PeerAddress=http://site-a-2:9001, LocalDb__Replication__ApiKey=dev-site-a-localdb-sync-key, LocalDb__Replication__MaxBatchSize=16
    • site-a-2: LocalDb__SyncListenPort=9001, same ApiKey, same MaxBatchSize, no PeerAddress (passive; the stream is bidirectional). The key must be byte-identical on both — the interceptor fail-closes on any mismatch and the pair silently stops converging.
  3. MaxBatchSize=16 because batching is row-count-only against gRPC's 4 MB cap and artifact chunk rows are ≈ 171 KB (16 × 171 KB ≈ 2.7 MB worst case).
  4. docker compose config to validate; do NOT bring the rig up in this task (the live gate task owns rig runs).
  5. Commit chore(localdb): rig config — consolidated path everywhere, replication on the site-a pair.

Task 14: Documentation

Classification: small Estimated implement time: ~4 min Parallelizable with: Task 13

Files:

  • Create: docs/operations/2026-07-20-localdb-pair-replication.md (runbook: enable/disable, ApiKey handling via ${secret:}, stop/start-together rule, tombstone-retention resurrection window, MaxBatchSize row-count-vs-4MB, never host-sqlite3 a live WAL DB / cp-triplet recipe)
  • Modify: docs/Redundancy.md (a short "pair-local config cache" section: what replicates, what boot-from-cache does and does not cover)
  • Modify: CLAUDE.md (this repo): LocalDb adoption row/paragraph — state Phase 1 scope, default-OFF, rig-only enablement
  • Modify: docs/Configuration.md if it documents config keys (add the LocalDb section)

Commit docs(localdb): phase-1 runbook + redundancy/config docs.

Task 15: DoD sweep (offline)

Classification: standard Estimated implement time: ~5 min Parallelizable with: none (final offline task)

Steps:

  1. dotnet build ZB.MOM.WW.OtOpcUa.slnx → 0 warnings (warnings-as-errors).
  2. dotnet test ZB.MOM.WW.OtOpcUa.slnx → full suite green; record counts. Baseline any pre-existing failures BEFORE this branch (git stash → run → unstash) rather than assuming; report deltas only.
  3. Greps (phrase as "no references from code"; explanatory comments may remain): grep -rn "LiteDB\|ILocalConfigCache\|GenerationSealedCache" src/ tests/ → no code references.
  4. Confirm the positive-control evidence from Task 12 is recorded.
  5. Update docs/plans/2026-07-20-localdb-adoption-phase1.md.tasks.json statuses.
  6. Commit chore(localdb): phase-1 DoD sweep and STOP. Report: branch name, commit list, test counts, and that the live gate (Task 16) needs the rig + operator go-ahead.

Task 16: Live gate on the docker-dev rig (run only with explicit user go-ahead)

Classification: high-risk (rig) Estimated implement time: ~30 min wall-clock (mostly waiting) Parallelizable with: none

Preamble: All DB inspection via cp of the db/-wal/-shm triplet out of the container (docker cp) and querying the copy — never host sqlite3 on the live file (it poisons the WAL across virtiofs; root cause of the 2026-07-20 ScadaBridge incident). Metrics via docker run --rm --network container:<node> curlimages/curl:latest -s localhost:<port>/metrics. If an anomaly appears within seconds of one of your own measurements, suspect the measurement first — restart both nodes and re-run untouched before blaming the library.

Checks (all must PASS; record evidence in docs/plans/2026-07-20-localdb-phase1-live-gate.md):

  1. Rig up, site-a pair healthy, /metrics on both shows localdb_ series (proves the meter allowlist work).
  2. Deploy a config through the normal deploy API → both site-a nodes' deployment_pointer + deployment_artifacts rows are byte-identical with identical HLC + origin node id (one node's row replicated, not two independent derivations — both nodes also locally store after apply, so expect LWW-converged identical content either way; assert convergence, and note which origin won).
  3. Boot-from-cache: stop the SQL container; restart site-a-2; it comes up serving the cached config with the running-from-cache signal in its log; start SQL again; node recovers fresh behavior.
  4. Replicated-cache payoff: wipe site-a-2's local DB file (container stopped), restart it with SQL up, let replication repopulate the cache from site-a-1, then repeat check 3's SQL-down restart — it boots from a cache it never wrote itself.
  5. Transport kill (pause site-a-2 container): store/deploy on a-1, oplog depth rises; unpause; drains to 0; converged.
  6. Retention: deploy a 3rd config; pruned deployment's chunks gone on BOTH nodes with tombstone rows present on both.
  7. Both-nodes-together restart: clean rejoin, identical counts, zero disk I/O error/SQLITE_IOERR/corrupt in either log.
  8. site-b pair (default-OFF pin): LocalDb file exists and works locally, ISyncStatus health Healthy, no sync connections, no listener on 9001.

Record PASS/FAIL per check with the actual evidence (row counts, md5s, log lines). Commit the gate doc. Do not merge — report.

Task persistence

Tasks file: docs/plans/2026-07-20-localdb-adoption-phase1.md.tasks.json (same directory). Update statuses as tasks complete; any deviation from this plan gets a deviation note on the task (the ScadaBridge tasks.json convention — the record of why is what makes the plan auditable).

Reference in New Issue View Git Blame Copy Permalink