lmxopcua/docs/plans/2026-07-20-localdb-phase1-recon.md

LocalDb Phase 1 — Task 0 Recon Findings

Date: 2026-07-20 Branch: feat/localdb-phase1 (based on master) Plan: docs/plans/2026-07-20-localdb-adoption-phase1.md Design authority: ~/Desktop/scadaproj/docs/plans/2026-07-20-otopcua-localdb-design.md

All file:line citations are against the branch base commit.

---

Step 1 — Feed verification: PASS

$ dotnet package search ZB.MOM.WW.LocalDb --source dohertj2-gitea --format json
ZB.MOM.WW.LocalDb              0.1.1
ZB.MOM.WW.LocalDb.Contracts    0.1.1
ZB.MOM.WW.LocalDb.Replication  0.1.1

All three present at 0.1.1. No STOP. (Note: the local NuGet cache holds both 0.1.0 and 0.1.1 — pin exactly 0.1.1 so a stale cache cannot win.)

---

Step 3 — STOP conditions: NONE TRIGGERED

STOP condition	Verdict
Artifact apply path cannot take a post-apply hook without restructuring the actor	Clear — ApplyAndAck has a clean insertion point (§2.2)
DeploymentId / cluster identity are not stable strings/GUIDs	Clear — both are stable strings, but cluster identity is not resolvable at boot; see D-1
LiteDB LocalCache is referenced by live code	Clear — zero live-code references (§7)

Three STOP conditions clear. Eight deviations from the plan's stated assumptions are recorded in §9 — four of them (D-1, D-3, D-5, D-6) change the work materially.

---

1. Deploy fetch path

1.1 Blob-load sites — there are five CreateDbContext() calls, not two

Line	Method	Purpose
DriverHostActor.cs:513	Bootstrap()	NodeDeploymentStates + Deployments at PreStart
DriverHostActor.cs:1472	ReconcileDrivers(DeploymentId)	loads ArtifactBlob
DriverHostActor.cs:1543	PushDesiredSubscriptions(DeploymentId)	loads ArtifactBlob again
DriverHostActor.cs:2000	TryRecoverFromStale()	latest Sealed deployment (id + hash only, no blob)
DriverHostActor.cs:2029	UpsertNodeDeploymentState(...)	writes NodeDeploymentState

DriverHostActor.cs:1472-1476:

using var db = _dbFactory.CreateDbContext();
blob = db.Deployments.AsNoTracking()
    .Where(d => d.DeploymentId == deploymentId.Value)
    .Select(d => d.ArtifactBlob)
    .FirstOrDefault() ?? Array.Empty<byte>();

:1543-1547 is byte-identical apart from the local. The same blob is read twice per apply.

1.2 Artifact runtime type — there is no instance type

DeploymentArtifact (DeploymentArtifact.cs:48) is a static parser class over ReadOnlySpan<byte>. The blob is never materialised into an object graph.

• Storage: byte[] — Deployment.cs:29 public byte[] ArtifactBlob { get; init; } = Array.Empty<byte>();
• Format: UTF-8 JSON, JsonDocument.Parse (DeploymentArtifact.cs:66, :490). Not MessagePack.
• Leniency contract: empty/malformed blob → empty result, never throws (:62, :537-541).

Consequence for the cache: we cache raw byte[]. No serializer coupling. Good.

1.3 Identity types

Type	Definition	Shape
DeploymentId	Commons/Types/DeploymentId.cs:3 — readonly record struct DeploymentId(Guid Value)	ToString() → Value.ToString("N"), 32 hex chars, no hyphens (:10)
RevisionHash	Commons/Types/RevisionHash.cs:8 — readonly record struct RevisionHash(string Value)	SHA-256 hex, lowercase 64 chars, no 0x; empty invalid (:3-6)

Both are stable, filesystem/SQL-safe strings. Use DeploymentId.ToString() (the "N" form) as the deployment_id TEXT column value.

---

2. Post-apply point

2.1 ApplyAndAck control flow — DriverHostActor.cs:1413-1458

1415  _applyingDeploymentId = deploymentId;
1416  Become(Applying);
1425  UpsertNodeDeploymentState(..., Applying, null)
1427  try {
1429      ReconcileDrivers(deploymentId);            // blob fetch #1 — SWALLOWS DB errors
1430      _currentRevision = revision;
1431      UpsertNodeDeploymentState(..., Applied, null);
1432      SendAck(..., ApplyAckOutcome.Applied, null, correlation);
1436      _opcUaPublishActor?.Tell(RebuildAddressSpace(...));
1439      PushDesiredSubscriptions(deploymentId);    // blob fetch #2 — SWALLOWS DB errors
1440      OtOpcUaTelemetry.DeploymentApplied.Add(1, "outcome"="ack");
1441      _log.Info("applied deployment ...");
1443  }
1444  catch (Exception ex) { ...Failed ACK... }
1452  finally { _applyingDeploymentId = null; Become(Steady); }

2.2 Where the cache write belongs

Immediately after DriverHostActor.cs:1439, before :1440 — but wrapped in its own try/catch. Reason: :1432 already sent an Applied ACK to the coordinator. An unguarded cache write that throws would fall into the catch at :1444 and send a second, contradictory Failed ACK for a deployment the fleet already believes is applied.

2.3 Paths that must NOT write to the cache

1. catch at :1444 — any apply failure.

2. Idempotent short-circuit at :1402-1409 — HandleDispatchFromSteady returns early with an Applied ACK when _currentRevision == msg.RevisionHash. ApplyAndAck is never entered and no blob is read.

3. ⚠ THE SILENT-DEGRADATION TRAP. ReconcileDrivers:1478-1483 and PushDesiredSubscriptions:1549-1553 catch DB failures, log a warning, and return without rethrowing. ApplyAndAck therefore reaches :1440 and declares success having applied zero drivers when the blob load failed. Caching "whatever we read" would persist an empty artifact as a successful apply — poisoning the boot-from-cache path with a config that starts no drivers.

   Mitigation (load-bearing): the cache write must be gated on the blob having actually loaded non-empty. Have ReconcileDrivers surface the loaded blob upward (a field or an out-param) rather than re-reading it a third time, and skip the cache write when it is Array.Empty<byte>(). This also removes the duplicate read noted in §1.1.

4. RestoreApplied (:1514-1528) — the bootstrap replay path. Restoring from central; caching here is harmless and arguably desirable, but it is a distinct seam. Phase 1 scope: skip it.

5. TryRecoverFromStale (:1996-2022) — never reads the artifact at all (see §3.4).

---

3. Cold-boot path

3.1 PreStart — DriverHostActor.cs:415-436

Ctor already did Become(Steady) at :411. PreStart subscribes to topics, spawns the virtual-tag and scripted-alarm hosts, then calls Bootstrap() at :435.

3.2 Bootstrap() — DriverHostActor.cs:506-565 — yes, it queries central SQL at boot

Two queries inside one try: latest NodeDeploymentState for self (:514-518), then the matching Deployment row (:527-532). Four outcomes: no prior deployments → Steady; Applied → RestoreApplied (:544); Applying orphan → ApplyAndAck replay (:549); Failed → Steady.

3.3 THE SEAM — DriverHostActor.cs:560-564

catch (Exception ex)
{
    _log.Warning(ex, "DriverHost {Node}: ConfigDb unreachable on bootstrap; entering Stale", _localNode);
    Become(Stale);
}

This is the exact point where "central fetch failed at boot" is known. The cache fallback goes between :562 and :563: attempt a local-cache load, and only Become(Stale) if the cache is also a miss.

⚠ The catch also fires if the second query throws after the first succeeded, so latest is not in scope-valid state from the catch. The cache must be self-describing — it must carry its own DeploymentId + RevisionHash, because at this seam the actor has no idea what it should be restoring. The planned deployment_pointer shape already satisfies this.

3.4 The Stale state — DriverHostActor.cs:1343-1379

ReconnectInterval = 30s (:55), retry timer started at :1378.

The dispatch-ignore at :1345-1348 discards DispatchDeployment entirely — no ACK, no buffering (contrast Applying:599 which does Self.Forward(msg)). A Stale node looks like a timeout to the coordinator. Phase 1 does not change this (per design §3.3: a new deployment still requires central).

RouteNodeWrite fast-fails at :1359-1360 with "driver host stale (config DB unreachable)".

3.5 Pre-existing gap worth recording (not Phase 1 scope)

TryRecoverFromStale (:1996-2022) sets _currentRevision and marks Applied without calling ReconcileDrivers or PushDesiredSubscriptions — documented at :1692-1695 and :1706. A Stale-recovered node reports Applied while running zero drivers, and HandleDispatchFromSteady:1402 then short-circuits on revision match. Boot-from-cache largely neutralises this in practice, but the bookkeeping remains optimistic. Logged as a follow-up, not fixed here.

3.6 DbHealthProbeActor

No interaction with DriverHostActor whatsoever. Spawned as a sibling (ServiceCollectionExtensions.cs:243-246), consumed only by OpcUaPublishActor (OpcUaPublishActor.cs:105, :256, :528, :543-544). DriverHostActor runs its own independent 30s retry-db timer. If running-from-cache should later influence OPC UA ServiceLevel, OpcUaPublishActor's DB-health seam is the precedent to mirror.

---

4. Cluster identity — the one real design problem (see D-1)

There is no ClusterId field on DriverHostActor and no accessor for it. The actor knows only _localNode (DriverHostActor.cs:61, type Commons.Types.NodeId, aliased at :28).

ClusterId is resolved per-artifact, by DeploymentArtifact.ResolveClusterScope(blob, nodeId) (DeploymentArtifact.cs:485-516) — it scans the artifact's Nodes[] array for a matching NodeId and reads that element's ClusterId. Types: ClusterScope (:46), ClusterFilterMode (:28).

IClusterRoleInfo (Commons/Interfaces/IClusterRoleInfo.cs:12) exposes LocalNode, LocalRoles, HasRole, MembersWithRole, RoleLeader, RoleLeaderChanged — no ClusterId. docker-dev/docker-compose.yml has Cluster__Hostname/Port/Roles/SeedNodes — no Cluster__ClusterId. The ClusterNode entity (Configuration/Entities/ClusterNode.cs:7,10) carries both, but lives in central SQL — exactly what is unreachable at the seam that needs it.

The bind: the design keys the cache by ClusterId so a replicated pair shares one cache entry. But ClusterId is only discoverable from a blob you already have or from central SQL. At the §3.3 boot-failure seam we have neither.

Resolution adopted (D-1): keep cluster_id as the deployment_pointer PK — pair sharing is the whole point of replication and must not be given up. Resolve it at each end as follows:

• Write path (ApplyAndAck, §2.2): call DeploymentArtifact.ResolveClusterScope(blob, nodeId) on the blob just applied. ClusterFilterMode.ScopeTo → use its ClusterId. None (single-cluster artifact) or Suppress → fall back to the literal "__single" sentinel, matching the actor's own existing degenerate-case convention at :1827-1830.
• Read path (boot seam, §3.3): we cannot compute the key, so do not key the read. Select the single deployment_pointer row, ordered by applied_at_utc DESC, LIMIT 1. A node belongs to exactly one cluster and its pair peer replicates the same cluster's row, so the table holds one row in every real topology. If more than one row is present (a node was re-homed between clusters), take the newest and log a warning naming both cluster ids — an operator-visible signal rather than a silent wrong-config boot.
• Optional escape hatch: honour a Cluster:ClusterId config key when present. Deferred — not needed for the rig, and adding a required key contradicts the design's "storage ships unconditionally" posture.

This preserves the replication payoff (design §3.3) and the schema exactly as specified, and confines the change to the read query's WHERE clause.

---

5. DriverHostActor construction — the Props expression tree

5.1 The factory — DriverHostActor.cs:326-346

static Props(...) takes 16 parameters, 14 optional, and forwards them fully positionally into Akka.Actor.Props.Create(() => new DriverHostActor(...)) at :343-346. Ctor at :375-412, assignments :393-408.

⚠ Six IActorRef? parameters and three interface-typed parameters mean many mis-bindings are type-compatible and therefore compile clean. Callers use named arguments (safe); the internal forwarding at :343-346 is positional (unsafe).

Rule for Task 7: append the new parameter LAST in the Props signature, LAST in the ctor signature, and LAST in the positional forwarding list at :346. Change nothing else.

5.2 Construction sites — 27 total (1 production + 26 test)

Production (1): Runtime/ServiceCollectionExtensions.cs:332, inside WithOtOpcUaRuntimeActors (:193); registered as DriverHostActorKey (:344). All args named except the first three. Does not pass virtualTagHostOverride, scriptedAlarmHostOverride, or driverMemberCountProvider.

WithOtOpcUaRuntimeActors callers: Host/Program.cs:331, Host.IntegrationTests/TwoNodeClusterHarness.cs:398, ServiceCollectionExtensionsTests.cs:42,144.

Tests (26), all in tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/:

File	Lines
Drivers/DriverHostActorTests.cs	28, 60, 81, 118, 143, 189
Drivers/DriverHostActorReconcileTests.cs	37, 59, 90, 119, 156, 193
Drivers/DriverHostActorDiscoveryTests.cs	385, 671, 748, 1023
Drivers/DriverHostActorWriteRoutingTests.cs	148, 208
Drivers/DriverHostActorPrimaryGateTests.cs	228, 247
Drivers/DriverHostActorVirtualTagTests.cs	44, 76
Drivers/DriverHostActorNativeAlarmTests.cs	350
Drivers/DriverHostActorNativeAlarmAckRoutingTests.cs	126
Drivers/DriverHostActorProbeResultDropTests.cs	66
Drivers/DriverHostActorHistoryWriterTests.cs	59
Drivers/DiscoveryInjectionEndToEndTests.cs	219
VirtualTags/DependencyMuxActorTests.cs	139

No site calls Props.Create<DriverHostActor>() or new DriverHostActor(...) directly — the only new DriverHostActor in the tree is inside the factory at :343. Because the new parameter is optional and appended last, no test site should need editing.

5.3 Status object for the running-from-cache signal

NodeDiagnosticsSnapshot (Commons/Interfaces/NodeDiagnosticsSnapshot.cs:17-22) — built in HandleGetDiagnostics (DriverHostActor.cs:1381-1398), a positional record of (NodeId, RevisionHash? CurrentRevision, IReadOnlyList<DriverInstanceDiagnostics> Drivers, DateTime AsOfUtc). GetDiagnostics is handled in all three states (Steady:570, Applying:601, Stale:1349), so a flag here is observable regardless of state. This is the natural home for RunningFromCache (append last to the record). Consumer: IFleetDiagnosticsClient → AdminUI.

IDriverHealthPublisher (:73, defaulted :402) is not a fit — the host never calls it; it only passes it down to DriverInstanceActor children (:1845, :1859). Per-driver, not per-node.

5.4 Async / side-effect IO convention

The actor is overwhelmingly synchronous-blocking on DB IO — all five CreateDbContext() sites use using var db = ... with synchronous LINQ on the actor thread. No async/await in the file, no Task.Run.

The single async pattern is ContinueWith(..., TaskScheduler.Default).PipeTo(replyTo) in HandleRouteNodeWrite (:1188-1200), with an explicit Sender capture before the continuation.

Recommendation for Task 7: a synchronous cache write at the §2.2 point is consistent with the file's existing style and is correct there — the actor is mid-apply, holds Applying, and every other DB call on that path already blocks. A fire-and-forget PipeTo(Self) would need a handler registered in Steady, Applying and Stale, or it dead-letters after the Become(Steady) at :1456. Prefer synchronous + try/catch. (This deviates from the plan's "fire-and-forget"; see D-4.)

IWithTimers is already implemented (:50, :282); "retry-db" key in use at :1378/:2009.

---

6. Host composition root

6.1 Role flag — Program.cs:47-50

var roles = RoleParser.Parse(Environment.GetEnvironmentVariable("OTOPCUA_ROLES"));
var hasAdmin = roles.Contains("admin");
var hasDriver = roles.Contains("driver");

Allowed roles: admin, driver, dev (Cluster/RoleParser.cs:5-8); unknown throws (:25-27).

6.2 hasDriver branch — Program.cs:131-310

Anchor for the new call: Program.cs:173-175 builder.Services.AddAlarmHistorian(...), inside the config-gated cluster at :154-195. Must land before AddAkka at :314.

6.3 Pipeline — Program.cs:368-404

:383 MapStaticAssets · :385-396 if (hasAdmin) { ... } · :398 MapOtOpcUaHealth() · :399 MapOtOpcUaMetrics() · :404 RunAsync().

⚠ There is no hasDriver block in the pipeline half — only hasAdmin is branched on. A driver-gated app.Map* is a new construct; it goes between :396 and :398.

Program.cs:406-410 re-exports public partial class Program.

6.4 Kestrel / URLs — CONFIRMED: no Kestrel configuration exists

• Program.cs:57 builder.WebHost.UseStaticWebAssets(); — the only builder.WebHost call.
• Zero ConfigureKestrel / UseUrls / ListenAnyIP / UseKestrel anywhere in src/.
• Binding is exclusively via ASPNETCORE_URLS: docker-compose.yml:173 (central-1) and :240 (central-2) set http://+:9000; launchSettings.json:7 sets http://localhost:9000.

Two traps for Task 5:

1. ⚠ scripts/install/Install-Services.ps1:186 ($hostEnv += "ASPNETCORE_URLS=...") is inside if ($hasAdmin) at :185. A driver-only Windows-service node gets no ASPNETCORE_URLS at all and binds the ASP.NET default. The plan's parse-URLs-then-rebind snippet would re-bind http://+:9000 there — a port that node never listened on. The ?? "http://+:9000" fallback in the plan's snippet is therefore wrong for driver-only nodes; see D-6.
2. ⚠ Host.IntegrationTests/TwoNodeClusterHarness.cs:331 already calls builder.WebHost.UseKestrel(o => o.Listen(IPAddress.Parse(LoopbackHost), 0));. Adding a production ConfigureKestrel gives two competing explicit configurations in integration tests — last-one-wins. The syncPort > 0 gate keeps this dormant by default (harness sets no sync port), but any harness that does set one will fight the port-0 binding.

Health route for smoke tests: /healthz — liveness, runs no checks, always 200 while the process is up, AllowAnonymous. (Health/HealthEndpoints.cs:48 → MapZbHealth(); routes from the ZB.MOM.WW.Health package.) Also /health/ready, /health/active, and /metrics (Program.cs:399).

6.5 SecretsRegistration template — Host/Configuration/SecretsRegistration.cs

Shape for LocalDbRegistration to mirror: public static class in ZB.MOM.WW.OtOpcUa.Host.Configuration; public const string section/key path constants; a public static bool IsXEnabled(IConfiguration) predicate using GetValue(key, defaultValue: false) (default-deny) that Program.cs can also call; AddOtOpcUaX(this IServiceCollection, IConfiguration) returning IServiceCollection, both args ArgumentNullException.ThrowIfNull-guarded; XML docs with <remarks><para> explaining why the gate is default-deny.

Call-site precedent: Program.cs:363 AddOtOpcUaSecrets(...); predicate reused inside the AddAkka lambda at Program.cs:323.

6.6 Telemetry meter allowlist — EXISTS. Task 10 step 3 is REQUIRED, not conditional.

Host/Observability/ObservabilityExtensions.cs:25-38:

return services.AddZbTelemetry(o =>
{
    o.ServiceName = "otopcua";
    o.Meters = [OtOpcUaTelemetry.MeterName];          // <-- ObservabilityExtensions.cs:30
    o.ActivitySources = [OtOpcUaTelemetry.ActivitySourceName];

o.Meters is a strict allowlist — ZbTelemetryExtensions.cs:51-54 iterates it calling metrics.AddMeter(name); no wildcard support. Anything unlisted is silently not exported. Single element today: OtOpcUaTelemetry.MeterName = "ZB.MOM.WW.OtOpcUa" (Commons/Observability/OtOpcUaTelemetry.cs:19).

Action: add LocalDbMetrics.MeterName at ObservabilityExtensions.cs:30. This is exactly the omission the ScadaBridge live gate caught.

6.7 Health registration — Host/Health/HealthEndpoints.cs:19-41

AddOtOpcUaHealth() takes no parameters and is called unconditionally at Program.cs:365, outside both role blocks. Three .AddTypeActivatedCheck<> calls (configdb, akka, admin-leader); no registration-time role gating exists — ActiveNodeHealthCheck does runtime role scoping instead (returns Healthy when the node lacks the role).

⚠ There are zero IHealthCheck implementations in this repo — all three come from the ZB.MOM.WW.Health.* packages (Directory.Packages.props:124-126, v0.1.0). A LocalDb check would be the first health-check class in the tree; Host/Health/ holds only HealthEndpoints.cs today.

Consequence for Task 10: driver-gating the new check requires changing AddOtOpcUaHealth()'s signature (add bool hasDriver or IConfiguration) and its Program.cs:365 call site. Preferred alternative: follow the ActiveNodeHealthCheck precedent — register unconditionally and have the check return Healthy when LocalDb is not registered. That keeps the signature and matches the "default-OFF must not degrade a plain node" requirement for free. See D-7.

6.8 Config files

File	Top-level sections
appsettings.json	Serilog, Security, Secrets, ServerHistorian, ContinuousHistorization, AlarmHistorian, Deployment
appsettings.driver.json / .admin.json / .admin-driver.json / .Development.json	Serilog, Security only

Overlay selection: Program.cs:67 sorts roles ordinally and joins with - → admin,driver becomes appsettings.admin-driver.json, added optional: true at :70; env vars (:77) and CLI args (:78) are re-appended after, so deployment overrides outrank the overlay.

LocalDb key search: CONFIRMED ABSENT. Case-insensitive localdb across all *.json, *.yml, *.ps1, *.csproj, *.props matches only the two Phase-1/Phase-2 planning artifacts. No conflicting keys in any overlay or in Install-Services.ps1.

Placement convention: each feature section opens with a "_comment" string then "Enabled": false; per-field caveats use sibling "_<Field>Comment" keys (appsettings.json:34, :45). Place LocalDb between AlarmHistorian (ends :61) and Deployment (:62).

---

7. LiteDB LocalCache — dormant, safe to delete

7.1 Files (6) under src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/

ILocalConfigCache.cs, GenerationSnapshot.cs, StaleConfigFlag.cs, LiteDbConfigCache.cs (also declares LocalConfigCacheCorruptException at :128), GenerationSealedCache.cs, ResilientConfigReader.cs. Single namespace ...Configuration.LocalCache. The only two using LiteDB; statements in src/ are LiteDbConfigCache.cs:1 and GenerationSealedCache.cs:1.

7.2 Reference classification — no live-code references. NO STOP.

• (c) live code elsewhere: ZERO. Also confirms full dormancy: SealedBootstrap (referenced by docs/v2/v2-release-readiness.md:46) has zero hits in src/ and tests/ — it no longer exists.
• Two benign near-misses:
  • Configuration/Services/ILdapGroupRoleMappingService.cs:14 — the string ResilientConfigReader inside an XML doc comment. Prose only. Needs a comment edit or the doc goes stale.
  • Configuration/ZB.MOM.WW.OtOpcUa.Configuration.csproj:29 — <PackageReference Include="LiteDB"/>, the only one in the repo.
• Dozens of LocalConfigCacheCorruptException hits under src/**/bin/**/*.xml are generated doc build artifacts (GenerateDocumentationFile=true), not source.

7.3 ⚠ Test-file drift — the plan's delete list is incomplete

Plan …phase1.md:484-485 names only GenerationSealedCacheTests.cs and ResilientConfigReaderTests.cs. Actual test files in tests/Core/ZB.MOM.WW.OtOpcUa.Configuration.Tests/:

File	Status
GenerationSealedCacheTests.cs	in plan ✅
ResilientConfigReaderTests.cs	in plan ✅ — note StaleConfigFlagTests is co-located here at :323
LiteDbConfigCacheTests.cs	MISSING from the plan — will fail to compile if the source is deleted (:125 calls new LiteDB.LiteDatabase(...) directly)

See D-3.

7.3b Outcome (Task 9, executed 2026-07-20)

Deleted: the 6 LocalCache/ sources and all three test files (including LiteDbConfigCacheTests.cs, which the plan omitted — D-3). LiteDB removed from both Configuration.csproj and Directory.Packages.props. The stale XML-doc reference at ILdapGroupRoleMappingService.cs:14 was rewritten to state the present truth (no fallback exists; reviving it means an admin-side cache on LocalDb, not restoring the old pipeline) rather than deleted, per the DoD's "explanatory prose may remain". Configuration tests: 92/92 green.

Follow-up found while executing: Polly.Core in Configuration.csproj is now orphaned too — ResilientConfigReader was its only real consumer in that project (remaining Polly mentions are comments). Left in place deliberately: removing it is outside Task 9's scope and could break a consumer relying on the transitive flow. Worth a separate cleanup.

7.4 Package removal

• PackageReference: Configuration.csproj:29 — only consumer.
• PackageVersion: Directory.Packages.props:35 — LiteDB 5.0.21.

Both safe to delete; no transitive consumer.

---

8. Test projects, rig, packages

8.1 Test project layout

Question	Answer
tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/	DOES NOT EXIST — there is no Host unit-test project at all
Closest Host project	Host.IntegrationTests — despite the name it carries plenty of pure unit tests (LdapOptionsValidatorTests, ServerHistorianOptionsValidatorTests, ResilienceInvokerFactoryRegistrationTests, …)
tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/	EXISTS
DriverHostActor tests	Runtime.Tests/Drivers/ — 11 DriverHostActor*Tests.cs files. xunit v2 (xunit 2.9.3), references Akka.TestKit.Xunit2 ✅
Host.IntegrationTests xunit version	xunit.v3 1.1.0; SDK Microsoft.NET.Sdk.Web

Directory.Packages.props documents that AdminUI.Tests / ControlPlane.Tests / Runtime.Tests are the three xunit-v2 holdouts, held there by Akka.TestKit.Xunit2 (verified 2026-07-13: no xunit.v3 Akka TestKit exists). This matches the design's "actor tests live in an xunit2 TestKit project".

8.2 ⚠ WebApplicationFactory<Program> is deliberately not used — Task 11 must be rewritten

The only mention of the type in the entire tests/ tree is a comment explaining why it is avoided:

Host.IntegrationTests/TwoNodeClusterHarness.cs:48

/// <para>Why not <c>WebApplicationFactory&lt;Program&gt;</c>? Program.cs reads <c>OTOPCUA_ROLES</c> ...

The actual pattern builds the host directly — TwoNodeClusterHarness.cs:329:

var builder = WebApplication.CreateBuilder(new WebApplicationOptions { Args = [] });

Microsoft.AspNetCore.Mvc.Testing is still referenced (csproj :15) but only for transitive test-host plumbing. Copy TwoNodeClusterHarness, not WebApplicationFactory. See D-5.

Host.IntegrationTests also has its own docker-compose.yml (sql 14331, ldap 3894) gated by DockerFixtureAvailability.cs.

8.3 docker-dev rig topology — plan's site-a/site-b assumption is correct

Single Akka mesh (otopcua), seeded by central-1. Tenant separation is by ServerCluster.ClusterId rows (MAIN / SITE-A / SITE-B) inside one shared ConfigDb — not separate meshes.

Service	Roles	Cluster
central-1, central-2	admin,driver	MAIN
site-a-1, site-a-2	driver	SITE-A
site-b-1, site-b-2	driver	SITE-B
sql, migrator, cluster-seed, traefik	—	infra

There is no admin-only service — the "admin" nodes are fused admin,driver. All six host nodes share the &otopcua-host YAML anchor.

⚠ NO HOST NODE HAS A WRITABLE VOLUME. The only volumes: in the file are sql → otopcua-mssql-data, plus two read-only binds (./seed:ro, ./traefik-dynamic.yml:ro). All six nodes write to the container's ephemeral layer, destroyed on recreate. This is already an accepted pattern (per-container Secrets:SqlitePath=otopcua-secrets.db, replicated over Akka rather than persisted). Task 13 must add six named volumes — there is nothing to piggyback on. See D-8.

Env style: confirmed double-underscore Section__Key, with array indices and dictionary keys as a third segment (Cluster__SeedNodes__0, Security__Ldap__GroupToRole__ReadOnly). Exceptions that are plain SCREAMING_SNAKE (read directly, not config-bound): OTOPCUA_ROLES, ZB_SECRETS_MASTER_KEY, GALAXY_MXGW_API_KEY, OTOPCUA_CONFIG_CONNECTION, ASPNETCORE_URLS.

Ports. Host-published: 14330→sql, 4840-4845→the six nodes' OPC UA, 9200→traefik web, 8089→traefik dashboard. Container-internal on every node: 4053 (Akka remoting), 9000 (Kestrel), 4840 (OPC UA). Also reserved on this machine: 14331/3894 (Host.IntegrationTests compose), 80/8080 (sibling scadabridge/scadalink stack), 10.100.0.35:3893 (shared GLAuth).

9001 (the plan's choice) is free container-internal and need not be published. ✅

8.4 Package versions — Directory.Packages.props

Package	Version	Line
Grpc.Core.Api	2.76.0	32
Grpc.Net.Client	2.76.0	33
Grpc.AspNetCore	NOT PRESENT	—
Google.Protobuf	3.34.1	31
Microsoft.Data.Sqlite	10.0.7	~55
SQLitePCLRaw.bundle_e_sqlite3	2.1.12	~108
LiteDB	5.0.21	35

Both Grpc floors and the Protobuf floor are already satisfied — no bumps needed, only the new Grpc.AspNetCore entry (use 2.76.0 to match the train; watch for NU1605/NU1608 if it wants a Protobuf newer than the pinned 3.34.1).

⚠ SQLitePCLRaw is a surgical direct pin, not a transitive one. CentralPackageTransitivePinningEnabled is deliberately off (it breaks the Roslyn version split). The 2.1.12 pin is promoted by a direct PackageReference in Core.AlarmHistorian (Core.AlarmHistorian.csproj:15 + :21) — today the only Sqlite consumer in src/.

Any new project that pulls Microsoft.Data.Sqlite MUST also take a direct <PackageReference Include="SQLitePCLRaw.bundle_e_sqlite3"/>, or it silently resolves the vulnerable 2.1.11 native bundle. This applies to Host and Runtime (both gain ZB.MOM.WW.LocalDb, which depends on Microsoft.Data.Sqlite) and to every test project that builds a real ILocalDb. Copy the Core.AlarmHistorian.csproj:15,21 pair exactly. The plan's Task 1 mentions SQLitePCLRaw.lib.e_sqlite3 "if the audit flags it" — the correct package is bundle_e_sqlite3, and it is required, not conditional.

---

9. Deviations from the plan

Recorded per the tasks.json deviation convention. D-1, D-3, D-5, D-6 change the work materially.

#	Plan says	Reality	Resolution
D-1	Cache keyed by ClusterId; "cluster identity" is a recon item with a STOP condition	ClusterId is stable, but not resolvable at the boot seam — it lives inside the artifact or in the unreachable central DB (§4)	Keep cluster_id as PK (pair sharing is the point). Write: resolve via DeploymentArtifact.ResolveClusterScope, "__single" sentinel for the degenerate case. Read: unkeyed ORDER BY applied_at_utc DESC LIMIT 1, warn if >1 row. No STOP.
D-2	Cache write "fire-and-forget (PipeTo-style or guarded Task.Run)"	The actor has no async DB IO at all; a PipeTo(Self) needs handlers in all three states or it dead-letters after Become(Steady) (§5.4)	Synchronous write in its own try/catch at :1439. Matches the file's style; failure still cannot fail the apply.
D-3	Task 9 deletes 2 test files	3 test files reference the subsystem — LiteDbConfigCacheTests.cs is missing from the list (§7.3)	Delete all three. Also fix the stale XML-doc mention at ILdapGroupRoleMappingService.cs:14.
D-4	Cache the artifact after a successful apply	ReconcileDrivers/PushDesiredSubscriptions swallow DB errors, so a "successful" apply can have loaded an empty blob and started zero drivers (§2.3)	Gate the write on a non-empty blob; surface the blob from ReconcileDrivers instead of a third read. Load-bearing — without it the cache can be poisoned with an empty config.
D-5	Task 11 uses WebApplicationFactory<Program>	Deliberately not used in this repo; TwoNodeClusterHarness.cs:48 documents why (Program.cs reads OTOPCUA_ROLES from the environment) (§8.2)	Rewrite Task 11 against the TwoNodeClusterHarness direct-host-build pattern.
D-6	Task 5 re-binds ASPNETCORE_URLS ?? "http://+:9000"	Install-Services.ps1:185-186 sets ASPNETCORE_URLS only when hasAdmin — the fallback would bind a driver-only node to a port it never served (§6.4)	Re-bind only when a URL is actually configured; when absent, add only the sync listener. Also note TwoNodeClusterHarness.cs:331 already calls UseKestrel(Listen(…, 0)).
D-7	Task 10 driver-gates the health check	AddOtOpcUaHealth() takes no args and is called unconditionally at Program.cs:365; no registration-time gating exists anywhere (§6.7)	Register unconditionally; return Healthy when LocalDb is absent — the ActiveNodeHealthCheck precedent. Avoids a signature change and satisfies "default-OFF must not degrade a plain node".
D-8	Task 13 "existing data mount"	No host node has any writable volume (§8.3)	Add six named volumes. Note: 9001 is free container-internal. Meter allowlist (§6.6) is required, not conditional.

Follow-ups logged, not fixed here

1. TryRecoverFromStale marks Applied without reconciling drivers (§3.5) — pre-existing, documented in-code at :1692-1695. Out of Phase 1 scope.
2. Duplicate blob read per apply (:1472 and :1543, §1.1) — D-4's fix removes one of them as a side effect.
3. Install-Services.ps1 gives driver-only nodes no ASPNETCORE_URLS (§6.4) — pre-existing; D-6 works around it rather than fixing it.