lmxopcua/docs/plans/2026-07-20-localdb-phase2-live-gate.md

LocalDb Phase 2 — live gate (docker-dev rig)

Task 8 of 2026-07-20-localdb-adoption-phase2.md. Evidence log. Rig: local docker-dev/docker-compose.yml — 2 central + site-a pair (replication ON) + site-b pair (default-OFF), one Akka cluster, one shared SQL.

Safety rules followed (same as the Phase 1 gate): all DB inspection via docker cp of the db/-wal/-shm triplet out of the container, querying the copy — never host sqlite3 on the live WAL file.

Headline

The gate did not get past its own first boot before finding real defects. Four production bugs, all fixed on-branch with regression tests, plus a fifth finding in the test suite itself. Three of the four crash-looped every driver node; the fourth silently stopped alarm history everywhere.

Checks 1, 2, 5 and 6 pass. Checks 3 and 4 are NOT satisfied — see "The blocker" below; they cannot be run as written on a rig that has no per-pair redundancy roles, and the defect they were meant to confirm turned out to be the opposite of what the plan assumed.

Seeding the migration

The rig had no pre-consolidation alarm-historian.db, so one was synthesised per node and docker cp-ed to /app/alarm-historian.db (the WORKDIR-relative default) into created-but-not-started containers — the exact upgrade shape.

Deliberately overlapping content, to test D-6 in production: 6 rows on site-a-1, 5 on site-a-2, 2 of them byte-identical payloads (the warm-pair duplication HistorianAdapterActor produces in every boot window). 11 legacy rows, 9 distinct.

Defects found and fixed

1. Empty ServerHistorian:ApiKey crash-loops the host (was classified "degrades")

AlarmHistorian:Enabled=true requires the gateway alarm writer, which requires a key. HistorianGatewayClientOptions.Validate() throws The gateway API key must not be empty inside the client factory during Akka startup, so every driver node crash-looped.

ServerHistorianOptionsValidator exists precisely to convert that class of failure into a named OptionsValidationException — but its documented fail tier explicitly excluded ApiKey, reasoning that a keyless client "degrades rather than crashes (the gateway rejects calls)". That premise is false: the client validates its own options at construction, so the process dies before any call. Fixed; the key is never echoed in the message.

2. UseTls / endpoint-scheme mismatch crash-loops the host

Immediately after fixing #1, the rig crash-looped again: UseTls defaults to true, the rig endpoint is http://, and the client throws UseTls requires an https gateway endpoint. The inverse (https:// + UseTls=false) throws too — both strings confirmed in the shipped client assembly. Setting an http endpoint without clearing UseTls is an ordinary migration slip that takes the host down. Now validated in both directions.

3. Plaintext h2c was unreachable — every http:// deployment crashed

Third crash-loop, and a genuine product bug rather than operator config. HistorianGatewayClientAdapter.Create forwarded the TLS-only options unconditionally, and AllowUntrustedServerCertificate defaults to false, so it always sent RequireCertificateValidation=true — which the client rejects outright when UseTls=false (RequireCertificateValidation is a TLS-only option and requires UseTls=true).

CLAUDE.md and docs/Historian.md both document http:// as the supported way to select h2c, but no h2c configuration could start, and the only workaround was to claim AllowUntrustedServerCertificate=true — a certificate posture for a connection with no certificate. Fixed: TLS-only options stay at their defaults when UseTls=false.

4. THE BLOCKER — the drain gate deferred to a Primary that cannot deliver

With the rig finally up, every driver node logged Historian drain suspended. Nothing drained anywhere — including the two site-b nodes, which have no redundancy peer and no replication at all. Before Phase 2 this deployment drained fine, because the drain was ungated.

Root cause, established from the rig rather than guessed:

• RedundancyStateActor derives roles from Akka's cluster-wide RoleLeader("driver").
• central-1/central-2 carry both admin and driver Akka roles (OTOPCUA_ROLES=admin,driver).
• So the elected driver Primary is central-1 — a node that replicates nobody's LocalDb and does not even run the alarm historian.
• Every site node was Secondary, saw that a Primary existed, and stood down in its favour.

The redundancy role is a cluster-wide election; the alarm queue is pair-local. Deferring on "somebody is Primary" is therefore wrong in principle: in a multi-pair fleet the Primary is usually in another pair and holds none of this node's rows. The consequence is not a duplicate — it is the buffer growing on every node until the capacity wall evicts the oldest events, i.e. silent permanent loss of the audit trail the queue exists to protect.

Fixed in three layers, each with a positive control:

1. PrimaryGatePolicy.ShouldDrainAlarmHistory(role, queueSharingPeerIsPrimary) — a separate policy from ShouldServiceAsPrimary. Unknown role ⇒ drain. A node stands down only for a Primary that holds its rows. The two gates now deliberately disagree, and a test pins that disagreement so nobody "harmonises" them back.
2. DriverHostActor publishes that verdict, matching the snapshot's Primary against this node's configured LocalDb replication peer host.
3. AddAlarmHistorian short-circuits the gate entirely when replication is not configured — a node whose rows exist nowhere else must never defer. The predicate tests both Replication:PeerAddress and SyncListenPort, because only the dialing half sets the former and both halves share the queue.

The asymmetry of the error costs is what drives every one of these: a false allow costs a duplicate history row, which the design already accepts (at-least-once across a failover, by design, and payload-hash ids collapse identical events); a false deny loses data silently.

5. A third vacuous test — AwaitAssert on the seeded value

Deleting the guard (publishing the old write-gate verdict) left DriverHostActorRoleViewTests green. AwaitAssert polls until an assertion passes, and the assertion was "the view reads open" — which is also the value the view is seeded with, so it succeeded at the first poll, before the actor had processed anything. It could not distinguish a correct publish from no publish at all.

Fixed by asserting on the sequence of published values via a recording view, never on current state. Re-run under the control: red, for exactly the cases that matter. This is the same "absence-assertion passes by race" trap recorded for #485/#486, in a new disguise.

Checks

#	Check	Result	Evidence
1	Migration ran; .migrated sidecar on both site-a nodes; row counts match	✅ PASS	alarm-historian.db.migrated present on both. 11 legacy rows across the two files → exactly 9 rows on each node, id sets byte-identical — the 2 shared payloads collapsed, proving D-6's payload-hash identity in production. Each node holds the other's rows (legacy/a1-only-* present on a-2 and vice versa). Legacy attempt_count distinction survived the copy (the 3-attempt row still leads the 0-attempt rows), last_error carried, 0 dead-lettered
2	Alarm burst converges: identical rowsets, oplog drains to 0, dead letters 0	✅ PASS	Identical id sets on both nodes; __localdb_oplog depth 0 on both; 0 dead-lettered. Drain visibly live against the deliberately unresolvable gateway — attempt_count climbing 33 → 42 → 168 with last_error='retry-please', which is exactly the buffering behaviour the queue exists for
3	Only the primary drains	❌ NOT SATISFIED — see defect 4	As written the check assumes a per-pair Primary exists. On this rig the elected Primary is a central node outside both pairs, so the honest post-fix behaviour is that all four nodes drain. No node can currently identify a queue-sharing Primary: site-b is unpaired, and of the site-a pair only the dialing half knows its peer. Safe (no loss, duplicates only), but the check's intent is unproven
4	Failover: standby resumes draining; count double-delivered events	⛔ NOT RUN	Superseded by defect 4. A meaningful failover test needs a rig where the pair itself elects a Primary; measuring "double delivery" is meaningless while both halves drain unconditionally
5	Both nodes restarted together: clean rejoin, identical counts, no I/O errors	✅ PASS	docker compose restart site-a-1 site-a-2 → 0 disk I/O error / SQLITE_IOERR / corrupt lines and 0 fatals in either log; both back to 9 rows / 0 dead / oplog 0, still identical
6	site-b default-OFF pin: sink works locally, no sync traffic	✅ PASS	alarm_sf_events created and registered on both site-b nodes; port 9001 not bound (/proc/net/tcp), no sync listener, no replication config; both drain their own queue (defect 4's third fix) with 0 fatals

The open design question

Update 2026-08-07 (truth sweep): resolved — recommendation (C) shipped as mesh Phase 6 (merged 2026-07-24, tip 2839deb5): RedundancyStateActor is now a cluster-{ClusterId}-scoped singleton, so the election is per application Cluster exactly as recommended below. What honestly remains open: gate checks 3 and 4 (only-the-primary-drains; failover double-delivery count) were never re-run on the post-Phase-6 three-mesh topology, so the drain gate's intent is still unproven live.

The gate leaves one question that is a design fork, not a bug to patch:

A pair cannot currently identify its own Primary. Redundancy roles are elected cluster-wide, but the queue is pair-local. Only the dialing half of a pair knows its peer (Replication:PeerAddress); the listening half sets SyncListenPort only. So the drain gate can never fully engage, and both halves of a replicated pair drain — safe, but duplicated.

Follow-up analysis — the root cause is wider than the drain gate

Two things assumed while writing the section above turned out to be wrong, and correcting them changes the recommendation.

The rig is not misconfigured. central-1/central-2 carry admin,driver deliberately — the compose says so in as many words ("central is admin,driver"), because they are themselves a redundant pair that also runs the MAIN cluster's drivers. Stripping the driver role would change what the rig models, not fix anything.

A "pair" is not a missing concept — it already exists, as the application Cluster. OtOpcUa's Cluster/ClusterId (the config-DB entity that owns drivers, devices and ClusterNode rows) is exactly the unit a redundant pair belongs to. Phase 1 already relies on this: the deployment-artifact cache is keyed by ClusterId so a pair shares one entry, and the rig's three application clusters (MAIN, SITE-A, SITE-B) are precisely its three pairs.

So the real defect is that RedundancyStateActor elects one Primary per Akka cluster, while every meaningful unit of redundancy is an application cluster. The rig runs six driver nodes across three application clusters in one Akka cluster — a topology the product otherwise supports throughout — and exactly one of those six can ever be Primary.

That mis-scoping is not confined to the alarm drain. Every consumer of the role inherits it: ServiceLevel, the alerts emit gate, and the inbound device-write gate. On a multi-cluster fleet today, only one driver node cluster-wide will service Primary-gated work at all.

Revised recommendation

(C), as a separate change with its own live gate — and explicitly not (A).

• (C) is no longer "invent a pair concept"; it is "elect per ClusterId instead of per Akka cluster", using a first-class entity that already exists. It is feasible where it needs to be: RedundancyStateActor has no DB access today, but it is an admin-role singleton in the same assembly as AdminOperationsActor, which already reads and groups ClusterNodes by cluster.
• (A) is now actively undesirable. Deriving pair identity from LocalDb:Replication:PeerAddress would introduce a second, parallel notion of "who is my partner" that (C) immediately obsoletes — and it cannot be done cleanly anyway: the library exposes no initiator/passive flag, so making both halves declare a peer means both halves dial, opening two duplex sessions purely to obtain a configuration value.
• (C) touches the inbound device-write gate, which is safety-critical (the archreview 03/S4 dual-primary fix). It therefore belongs in its own change, with its own live gate, rather than bolted onto Phase 2.

Until then this branch is safe in every topology — no configuration loses data — so what remains is reclaiming the de-duplication benefit, not an outstanding hazard.

Not merged

Per the plan, nothing on this branch is merged.

Update 2026-08-07 (truth sweep): since merged — 06b5d81b (Merge feat/localdb-phase2: alarm store-and-forward on LocalDb + live-gate fixes).