Deploy sealed green with 4 DB-less site nodes acking (central persists acks);
ServiceLevel held 240 on a site node with central SQL stopped (survive-alone);
restarted site node booted last-known-good from the LocalDb pointer, no ConfigDb
read; alarm_condition_state table live + replicated. Full record in
2026-07-23-mesh-phase4-live-gate.md. Task 9 (drop dead table) deferred.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Task 4+6 code review re-surfaced the OpcUaPublishActor raw-sink wipe on a
null-factory node (the exact #485/#486 defect Task 3 fixes) — Task 3 must land
before any node flips to FetchAndCache. Ordering already enforces this
(3 → 7 → 10 → 11).
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Nullable ConfigDb factory; UpsertNodeDeploymentState no-ops when absent
(central persists acks from the ApplyAck); scripted-alarm condition state
served from the LocalDb store instead of the ConfigDb-backed Ef store.
Combines Phase-4 Tasks 4 + 6. Removes the interim dbFactory! cast.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Task 2 code review surfaced two unguarded ConfigDb sites reachable on a
driver-only node once Tasks 0+1 landed (SpawnScriptedAlarmHost's Ef store,
unconditional UpsertNodeDeploymentState). Task 4 brief now names them + the
dbFactory! removal; Task 6 re-homes the alarm store to LocalDb.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 4 of the per-cluster mesh program. Removes the four remaining driver-side
ConfigDb consumers + the registration itself so a driver-only node boots with no
ConfigDb connection string:
- ConfigDb registered iff hasAdmin (Program.cs); driver-only requires none
- driver-only ⇒ FetchAndCache mandatory (validator)
- DbHealthProbeActor not spawned on driver-only; DbReachable=true constant
(client-visible ServiceLevel change — a healthy DB-less node publishes 240/250
with central down, per user decision 2026-07-23)
- DriverHostActor: nullable factory, drop redundant SQL ack-writes (central's
PersistNodeAck is the ack system of record)
- EfAlarmConditionStateStore → LocalDbAlarmConditionStateStore (new replicated
alarm_condition_state table)
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 3 Task 9. docker-dev rig: central-1/2 carry ConfigServe (:4055 h2c + committed
dev key) and stay Direct; the four site nodes carry ConfigSource (both central
endpoints + matching key) defaulting to Direct. Flip only the site nodes with
OTOPCUA_CONFIG_MODE=FetchAndCache at 'docker compose up' — central keeps SQL and models
the target topology. Binding :4055 activates the dedicated-h2c Kestrel takeover on
central (re-binds :9000 too). Docs: new docs/Configuration.md ConfigSource/ConfigServe
section; docs/Redundancy.md 'config bytes travel out-of-band' note; the program plan
Phase 3 section + tracking row flipped to CODE-COMPLETE; a CLAUDE.md 'Config source'
section.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Eleven tasks. User decisions baked in: gRPC fetch RPC (not token-gated HTTP),
shared node bearer key (not a per-deployment token, so no migration and
DispatchDeployment is unchanged), a ConfigSource:Mode dark switch (Direct
default), both pair nodes fetch (no Primary gating). Header carries the five
recon facts and the Phase 3/4 boundary (config READS only; the NodeDeploymentState
write, DbHealthProbe and EfAlarmConditionStateStore stay for Phase 4).
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Flipped the docker-dev rig end-to-end to MeshTransport:Mode=ClusterClient
(OTOPCUA_MESH_MODE, no rebuild) and ran the gate. PASSED for the transport:
deploy seals on ClusterClient, a stopped node fails the deploy naming it, the
MaintenanceMode hatch shrinks the DB-sourced contact set live (6->5->6 on both
centrals independently), and the frame-size canary stayed silent.
Four findings, recorded in 2026-07-22-mesh-phase2-live-gate.md:
A. A 10-30s post-restart delivery window exists, but the DPS control fails
identically -- it is a cluster-convergence property, not a Phase 2
regression. Running the control is the only reason this is a note and not
a false attribution.
B. buffer-size=0's stated rationale was OVERSTATED. A node applies a TimedOut
deployment anyway -- live-observed at +44s -- via DriverHostActor.Bootstrap
replaying the orphan Applying row on restart. Zero buffering removes the
silent, timer-driven replay, not the boot-time DB one. Corrected in the
plan decisions table, Configuration.md, and CLAUDE.md; the decision stands,
the justification narrowed.
C. Stopping BOTH centrals strands the non-seed nodes at the Akka membership
layer (frozen mid-Exiting, not Unreachable, so auto-down does not help).
Transport-independent; a Phase 7 downing/rejoin drill item.
D. The AdminUI driver Restart/Reconnect buttons (DriverStatusPanel) are
rendered by no page -- orphaned since v3 Batch 2 retired /clusters/{id}/
drivers. So gate steps 6/7 could not run; the driver-control node-side leg
is proven live only because DispatchDeployment shares its EventStream path.
Step 8's planned log-signal does not exist (HandleApplyAck is silent on
success); proven instead by both centrals independently logging per-instance
ClusterClient creation, which a singleton cannot.
Also adds a repo-root .dockerignore: docker-dev/Dockerfile does COPY . ., which
was pulling .claude/worktrees (19GB of agent checkouts with their own bin/obj)
into the build context and exhausting the Docker VM disk mid-COPY.
Rig restored to Mode=Dps, six members, no leftover maintenance flags.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Rig: all six docker-dev nodes carry MeshTransport__Mode plus both central
contact points. Mode is "${OTOPCUA_MESH_MODE:-Dps}", so the rig still comes up
on the transport it has always used and the live gate can flip the whole fleet
at `docker compose up` without a compose edit or a rebuild.
Docs: a MeshTransport section in Configuration.md (including why contact points
are addresses only, why the contact set does not scope delivery, and why
buffer-size=0 is a behaviour decision rather than a tuning knob), a "Command
transport" section in Redundancy.md contrasting the two modes, and a summary
beside the Redundancy notes in CLAUDE.md.
Corrects the design doc's claim that ScadaBridge replies with a typed failure to
every unhandled message. Neither of their comm actors has a ReceiveAny or an
Unhandled override; the idiom is per message type and fires on a missing
registration, and an unknown type dead-letters there as it does anywhere else.
Records the exit-gate deviation in the program plan: "an Ask timing out cleanly
against a stopped node" cannot be run, because no Ask crosses this boundary. The
honest equivalents are a stopped node failing the deploy at the apply deadline
and an unreachable contact dropping the command with a Warning.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 2 of the per-cluster mesh program: move the three central->node command
channels (deployments, driver-control, alarm-commands) and the deployment-acks
reply channel off DistributedPubSub onto Akka ClusterClient, behind a config
flag.
Recon of both repos turned up three corrections to the design doc's account of
the sister project, all folded into the plan:
- Design doc S2 claims "every unhandled message replies with a typed failure".
ScadaBridge has no ReceiveAny/Unhandled override in either comm actor; the
typed-failure idiom fires only for a MISSING REGISTRATION, per message type.
- "No central buffering" was never a setting they wrote. They have zero
akka.cluster.client HOCON and run the defaults (buffer-size 1000,
reconnect-timeout off). We choose buffer-size = 0 deliberately.
- Publish -> Send is the wrong substitution. Today's deploy notify is a
broadcast every DriverHostActor receives (no ClusterId filter exists on the
node side), so it must become SendToAll. Send would deploy to exactly one
node of the fleet and seal green on partial acks.
Also records the single-mesh duplicate-delivery trap (one ClusterClient in
Phase 2, not one per Cluster -- a receptionist serves its whole mesh) and one
deviation from the program plan's exit gate: there is no cross-boundary Ask in
Phase 2, because every migrated command is fire-and-forget with a local reply,
so "an Ask timing out cleanly" cannot be run as written.
Marks the program tracking tables current: prereq + Phase 1 are DONE (they
still read "not executed"), Phase 2 in progress.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 1 gate step 4 failed: setting Enabled = 0 to take a node out of service
returns 422 ClusterEnabledNodeCountMismatch, because
DraftValidator.ValidateClusterTopology requires the enabled-node count to equal
ServerCluster.NodeCount. On a Warm/Hot pair — every cluster on the rig, and
every cluster in the target topology — Enabled can therefore never be the
maintenance hatch. The plan called step 4 "the check that makes step 3
acceptable to ship", so Task 3's behaviour change was not shippable as it stood:
a node down for maintenance would block every deployment to its cluster.
The two rules were each reasonable and contradictory together — the validator
reads Enabled as "part of the declared topology", Phase 1 additionally read it
as "expect an ack". Split the meanings rather than weaken either rule:
Enabled part of the declared topology (validator, untouched)
MaintenanceMode expected to participate now (coordinator + reconciler)
Rejected alternatives: counting configured rather than enabled nodes (drops the
guard against booting a pair into InvalidTopology); downgrading the rule to a
warning (weakens a deploy gate for everyone); shipping with no hatch.
Sabotage: dropping !n.MaintenanceMode from the coordinator query turns the new
test red. It also asserts both nodes remain Enabled, so the fix cannot quietly
regress to disabling the row after all.
Configuration.Tests 95/95, ControlPlane.Tests 101/101, solution builds clean.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
- docker-dev seed: AkkaPort = 4053 on all six ClusterNode rows, so a freshly
seeded rig matches a migrated one. GrpcPort left null.
- config-db-schema.md: both columns, why AkkaPort is NOT NULL/4053 and GrpcPort
is nullable, the unenforced duplication + its reconciler, and Enabled's new
second meaning as the deploy path's expected-ack set.
- Configuration.md: Cluster:Port / PublicHostname now flag that they are stored
twice, with the "update the row too" instruction and why the drift is silent.
- design doc §7: Phase 1 marked done, plus a "Phase 1 as shipped" note recording
both deviations rather than leaving the sketch reading as what happened.
- program plan: Phase 1 marked done; AdminUI node edit explicitly deferred.
- CLAUDE.md: the deploy-path behaviour change and its three consequences.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
ClusterNode.AkkaPort and the node's own Cluster:Port are the same fact in two
places and nothing made them agree. Phase 2 dials the row instead of gossiping,
so a node binding 4054 while its row says 4053 becomes unreachable from central
— and the symptom is a silent absence of acks rather than an error. That is the
shape of the Modbus/ModbusTcp and TwinCat/Focas drifts already in this repo.
Since Phase 1 also made the rows the deploy path's expected-ack set, drift
already costs a failed deployment today.
Implemented as the plan's preferred option: an admin-role singleton comparing
rows against the membership an admin node can already see, rather than each
driver node asserting its own row — Phase 4 removes the driver nodes' ConfigDb
connection, so a self-assertion written there would have to be deleted again.
Three shapes, split by severity: a row whose dial target disagrees with its own
NodeId and a running node with no row are Errors; an enabled row with no
matching member is a Warning, because a node down for maintenance is a
legitimate state. Findings are logged only when the set changes — a check that
reprints the same warning every sweep trains operators to filter it out.
Documented limitation: Phase 2 must revisit this. Once the fleet splits into
one mesh per cluster an admin node cannot see site members, and every site row
would report EnabledRowNotInCluster forever.
Sabotage: removing the change-detection guard turns the repeat test red.
ControlPlane.Tests 100/100.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Per-cluster mesh Phase 1 groundwork. Once the meshes split (Phase 2) central
can no longer see a site node's appsettings, so the transport ports it must
dial have to live in a row central can read.
AkkaPort is non-nullable with a 4053 default — every node listens on a
remoting port, so 0 is never a truthful value and pre-existing rows must
migrate to something real. GrpcPort is nullable with no default: nothing
listens on it until Phase 5, and a non-null default would assert a port that
does not exist.
Both are documented as central's dial targets rather than the node's own
binding config; the duplication against Cluster:Port is reconciled in Task 4.
The new SchemaCompliance test is red until the migration lands (Task 2).
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
docs/Redundancy.md's bootstrap section is rewritten around the mechanism that is
actually in force: which of Akka's two bootstrap processes runs is decided by
whether seed-nodes[0] is this node's own address, so the ORDER of Cluster:SeedNodes
is the fix, not a timer. Adds the per-process table, the shipped self-first
example, why the validator is conditional (site nodes are not seeds) and why it
matches PublicHostname rather than the 0.0.0.0 bind address.
The retirement is documented rather than erased: a new subsection explains that a
watchdog outside the join handshake cannot tell "no seed answered" from "a seed
answered and the join is in flight", and the ea45ace1 live-gate finding is kept
verbatim as the evidence that produced that conclusion (InitJoinAck at 10:49:05,
old incarnation retired 10:49:06, watchdog fired 10:49:15, second cluster). The
watchdog's own earlier PASS is kept too — it did pass what it was gated on.
Also: a live gate for the NEW mechanism is registered as outstanding (not
re-drilled on the docker-dev rig since the swap), and the plan doc that shipped
the watchdog gets a superseded banner rather than an edit, so the execution record
and its Task 8 finding stay readable.
Ripple: CLAUDE.md cluster section, docs/Configuration.md + docs/v2/Cluster.md +
docs/ServiceHosting.md SeedNodes entries, the per-cluster-mesh design doc's two
"CLOSED by SelfFormAfter" notes, and mesh-program Phase 6 (which had this
convergence queued — now marked done early) + Phase 7's live-gate name.
Corrects the plan's behavior table (the mid-join-handshake row was FALSE) and
notes the mesh-program Phase 6 convergence on ScadaBridge's self-first seed
ordering, which retires the watchdog + guard entirely.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
The live gate for the manual-failover control caught the self-form fallback
forming a SECOND cluster. A node bounced by a failover restarted, received
InitJoinAck from its live peer — the join was in flight and healthy — but did
not get the Welcome inside the 10s window, because the peer's ring still held
the node's previous incarnation (Exiting -> Down -> Removed). The fallback
fired on the timer and the node islanded itself until an operator restarted it.
The original design assumed Cluster.Join(SelfAddress) would be ignored
mid-handshake. It is not — it wins. And since manual failover deliberately
produces that restart, every failover could island the node it bounced.
Before self-forming, TCP-probe the other seed addresses; a reachable peer means
wait another window instead. That is also what the fallback claims to detect:
'no seed answered InitJoin (peer down at boot)'.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
The design doc still described the downing gap and the RoleLeader derivation as
open. Both shipped earlier today, so 6.2 and 4 now carry the fix and its pin
instead of the diagnosis, and 7 marks 0a/0b done. 0a's live gate stays open and
is explicitly deferred to phase 7: the failure only appears in a 1-vs-1 split and
docker-dev is a single six-node mesh, so there is no rig on which to drill it
until the meshes are actually split.
The phase 1 plan lands with one finding that changes its shape. The design pairs
"ClusterNode gains address columns" with "coordinator sources its expected-ack set
from the DB" as though they were one change; they are independent, and the second
is far smaller than it reads. ClusterNode.NodeId is ALREADY in the coordinator's
identifier space — the rig seeds central-1:4053 and so on, because
NodeDeploymentState.NodeId is FK-bound to it and the coordinator's
membership-derived ids already satisfy that FK. That the deploy path works today
is standing proof the two sets agree.
What is not small is the semantics. Membership-derived and DB-derived sets differ
on a configured-but-stopped node: today the deployment seals green without it —
telling the operator the fleet is deployed when it is not — and afterwards it
would fail at the apply deadline. Failing is the better behaviour and is required
once central cannot see cluster membership, but it needs an escape hatch or a node
down for maintenance blocks every deploy. ClusterNode.Enabled already is that
hatch, so the plan filters on it and gates the phase on a live check that proves
both halves: a stopped node fails the deploy legibly, and disabling it lets the
deploy seal.
The plan also adds a task the design does not have: AkkaPort duplicates the node's
own Cluster:Port with nothing making them agree, and an unenforced duplicated
address is the same shape as the Modbus/ModbusTcp and TwinCat/Focas drifts already
recorded here.
Phases 2-7 remain unstarted. They change the running data path and rewrite the
rig, and each needs its own plan and live gate.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
RedundancyStateActor derived the driver Primary from ClusterState.RoleLeader("driver").
Akka offers two different notions of a "first" member and they are not the same
one: role leader is the lowest-ADDRESSED Up member (host, then port), while
ClusterSingletonManager places singletons on the OLDEST — lowest up-number.
They agree on a freshly-formed cluster, which is why every existing test passed.
They diverge after any restart: the restarted node re-joins as the youngest while
keeping its address, so if it holds the lower address it becomes role leader while
the singletons stay put. The snapshot would then name a Primary that is not
hosting the work, and every Primary-gated surface follows it — inbound device
writes, native-alarm acks, the fleet-wide alerts emit, and the alarm-history
drain would all enable on the wrong node while the node actually running the
singletons stayed gated off.
BuildSnapshot now selects the oldest Up member carrying the driver role, matching
singleton placement. Leaving members are excluded: a node handing its singletons
over must not be named Primary.
NodeRedundancyState.IsRoleLeaderForDriver is renamed IsDriverPrimary, and
NodeHealthInputs.IsDriverRoleLeader likewise. Keeping the old names would have
left the wire contract asserting a derivation the code no longer uses — the same
drift that made this defect invisible.
Proven by a real two-node cluster rather than a mock. RedundancyPrimaryElectionTests
binds the first-joining node to the HIGHER port, so oldest and lowest-address name
different nodes, and includes a fixture assertion that the divergence actually
occurred — without it the real assertion could pass for the wrong reason. Positive
control: restoring the RoleLeader derivation turns exactly the two election tests
red while the fixture check stays green.
Runtime.Tests 440 passed, ControlPlane.Tests 82, Cluster.Tests 36.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Ports the sister project's live-proven fix (ScadaBridge cf3bd52f). OtOpcUa ran
the identical configuration it indicts: keep-oldest with down-if-alone = on.
Akka.NET 1.5.62's KeepOldest.OldestDecision only lets the down-if-alone branch
rescue a side holding >= 2 members. A two-node cluster losing a peer is always a
1-vs-1 split, so the branch never fires, the lone survivor falls through to
DownReachable and downs ITSELF, and run-coordinated-shutdown-when-down then
terminates it. down-if-alone is a 3+-node feature and does not do what its name
suggests for a pair: a crash of the oldest node is a total outage, which is the
exact failure the redundancy pair exists to absorb.
Cluster:SplitBrainResolverStrategy now selects the provider, defaulting to
auto-down: Akka's AutoDowning with auto-down-unreachable-after = 15s, so the
leader among the reachable members downs the unreachable peer and a crash of
either node fails over in place. keep-oldest remains available for deployments
that would rather take an outage than ever run dual-active during a real
partition. An unrecognised value fails the host at startup rather than falling
through to the fatal default.
The tests assert the EFFECTIVE configuration — they start a real host through
WithOtOpcUaClusterBootstrap and read akka.cluster.downing-provider-class back off
the running ActorSystem. This is not incidental. The first draft inlined the
three calls the bootstrap makes instead of calling it, which pinned the test's
own wiring: sabotaging production's HOCON precedence left all 36 green. The
prior file had the same shape at a smaller scale, asserting only that
BuildClusterOptions returned a KeepOldestOption — true, and true of a
configuration that cannot fail over. Positive control: making BuildDowningHocon
emit nothing for auto-down turns exactly the two effective-config guards red.
Measured while verifying rather than assumed: HoconAddMode.Append also wins here,
purely because it is added last, so the mode name is not the guarantee. The
comment now says so instead of asserting a precedence rule that does not hold.
Not yet live-drilled on OtOpcUa. The docker-dev rig is a single six-node mesh
where the 1-vs-1 pathology cannot occur; the kill-the-oldest drill belongs with
the per-cluster mesh work that makes every mesh exactly two nodes (design doc
6.2 / Phase 0a). Recorded as an outstanding gate in docs/Redundancy.md.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
I reported this gap as still open, citing a requirements doc line and the PR that
made the failover drill honest about it. Both were stale: ScadaBridge closed it
three hours earlier in cf3bd52f by switching to auto-down. My check missed it
because I compared only one direction (main..origin/main) and never asked whether
local main was AHEAD of origin, so two unpushed commits were invisible to me.
The substance matters more than the correction. Their commit message records a
live-proven finding: Akka.NET 1.5.62 KeepOldest.OldestDecision only lets
down-if-alone rescue a side with >= 2 members, so in a 1-vs-1 split the survivor
takes DownReachable and downs ITSELF. OtOpcUa runs precisely that configuration —
akka.conf active-strategy keep-oldest with down-if-alone on, plus the matching
typed KeepOldestOption — which means a two-node pair cannot fail over when the
oldest node crashes. That is a total-outage defect inside the redundancy feature,
independent of the mesh design, and it is now Phase 0a: most urgent, and needing a
crash-the-oldest live gate because the failure only appears 1-vs-1.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Section 5 still listed "site-local-primary storage" among the things deliberately
NOT copied from ScadaBridge — which §6.1 now reverses, since driver nodes stop
connecting to the ConfigDb and read their configuration from LocalDb. Replaced
with an explicit copied/not-copied split, and narrowed the not-copied item to what
it should have said all along: their transient-only central alarm cache, which
OtOpcUa has no reason to adopt because it persists alarm history through the
historian and Phase 2's replicated buffer already survives a failover.
Also corrected a renumbering artefact — the risk about rewriting the docker-dev
rig cited Phase 4, but the mesh partition moved to Phase 6 when the two data-path
phases were inserted.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
(1) One central DB, driver nodes disconnected from it. Verified in ScadaBridge
before adopting: AddConfigurationDatabase is called at Program.cs:264, inside the
Central branch (89-478); the Site branch starts at 479, so site nodes never
register the central database at all. There is still exactly one authoritative
configuration database — it is central-only, and sites fetch from central and
cache locally.
The consequence worth naming: this promotes LocalDb from an outage cache to the
driver node's steady-state configuration store. Boot-from-cache stops being the
exceptional path and becomes the normal one. Phase 1's chunking, SHA-256
verification, retention and pair replication all carry over — but a cache defect
that used to be a degraded-mode bug becomes a total-availability one, so the #485
unreadable-artifact class now has a larger blast radius.
Audited the five driver-side ConfigDb consumers that must be re-homed. Two are
more than mechanical: EfAlarmConditionStateStore holds pair-local Part 9 condition
state and should follow the alarm S&F buffer into LocalDb, and DbHealthProbeActor
feeds ServiceLevel — a driver node with no DB to probe changes what a
client-visible value means.
(2) The keep-oldest gap: corrected the status rather than the conclusion. What was
resolved is the documentation and the drill, not the hole. Gitea PR #12 (closed)
carried T1 badc97af, which made the drill "measure the registered keep-oldest
outage instead of pretending recovery", and T2 d5364506, which dropped "the
undeliverable ~25s active-crash promise". The gap itself is still the registered
deferred keep-oldest topology decision on the 2026-07-08 master tracker, with no
open issue. Accepted here with the same posture, recorded as a known risk.
(3) Auth matches ScadaBridge for now — unauthenticated inter-cluster transports on
a trusted network, recorded as an accepted risk with the fail-closed interceptor
already in our tree noted as the template for whenever it is revisited.
Sequencing grows from six phases to eight; the two that change a running system's
data path rather than its wiring each get their own live gate.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Design for review, mirroring the ScadaBridge hub-and-spoke shape. Researched from
the ScadaBridge tree directly rather than from its CLAUDE.md, which understates
the boundary in five ways: there are THREE transports (ClusterClient, gRPC, plus
token-gated HTTP), the gRPC direction is inverted (central dials INTO each site),
active node is the OLDEST Up member and explicitly never the cluster leader, all
clusters share one ActorSystem name, and site nodes carry two roles.
Records a defect worth fixing regardless of the topology decision: OtOpcUa uses
two different node-selection rules at once. SBR is keep-oldest and singletons are
placed on the oldest member, but RedundancyStateActor elects Primary from
RoleLeader("driver") — lowest address. Those diverge permanently after a
restart-and-rejoin, so the node the SBR protects and hosts every singleton on need
not be the node the data-plane gates consider Primary. ScadaBridge forbids that
rule by name, for the reason it gives: both sides claim leadership during a
partition, which is the dual-primary shape archreview 03/S4 exists to prevent.
Three open questions are left explicitly undecided: whether to adopt their
autonomous-site data architecture or keep the shared ConfigDb; what to do about
the registered two-node keep-oldest total-outage gap they acknowledge but have not
closed; and whether to authenticate transports they left unauthenticated.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Corrects two claims made while writing the Phase 2 gate record.
The docker-dev rig is NOT misconfigured: central-1/central-2 carry admin,driver
deliberately — the compose says so — because they are themselves a pair that runs
the MAIN cluster's drivers.
And a redundant pair is not a missing concept. The application Cluster (ClusterId)
already is one: it owns drivers, devices and ClusterNode rows, and Phase 1's
deployment-artifact cache is keyed by it precisely so a pair shares one entry.
So the root cause is wider than the alarm drain. RedundancyStateActor elects one
Primary per AKKA cluster while every real unit of redundancy is an APPLICATION
cluster, and ServiceLevel, the alerts emit gate and the inbound device-write gate
all inherit that mis-scoping. On a fleet of pairs in one Akka cluster, exactly one
driver node cluster-wide services Primary-gated work.
Recommendation revised in the gate doc: fix the election scope (its own change,
its own gate, because it touches the safety-critical write gate), and explicitly
do NOT derive pair identity from LocalDb:Replication:PeerAddress — that would add
a second parallel notion of 'my partner' that the real fix obsoletes, and the
library has no initiator flag, so both halves would have to dial just to obtain a
config value.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Gate record: docs/plans/2026-07-20-localdb-phase2-live-gate.md.
Checks 1, 2, 5, 6 pass. Checks 3 and 4 are NOT satisfied — the defect they
were meant to confirm turned out to be the opposite of what the plan assumed.
Three defects crash-looped every driver node before check 1 could even run:
1. An empty ServerHistorian:ApiKey kills the host. ServerHistorianOptions-
Validator exists to turn exactly that class of failure into a named
OptionsValidationException, but its documented fail tier explicitly excluded
ApiKey on the reasoning that a keyless client "degrades — the gateway rejects
calls". It does not: the client validates its own options at construction, so
the process dies during Akka startup and never makes a call.
2. UseTls disagreeing with the endpoint scheme kills the host too, in both
directions (both messages confirmed in the shipped client assembly). Moving an
endpoint from https to http without clearing UseTls is an ordinary migration
slip.
3. Plaintext h2c was UNREACHABLE. HistorianGatewayClientAdapter forwarded the
TLS-only options unconditionally, and AllowUntrustedServerCertificate defaults
to false, so it always sent RequireCertificateValidation=true — which the
client rejects outright when UseTls=false. Every http:// deployment crashed,
though the scheme is documented as the supported way to select h2c, and the
only workaround was to assert a certificate posture for a connection that has
no certificate.
The fourth was the blocker, and it is Phase 2's own:
4. The drain gate deferred to a Primary that cannot deliver. Redundancy roles are
elected CLUSTER-WIDE; the alarm queue is PAIR-LOCAL. On the rig the elected
driver Primary is central-1 — it carries the driver Akka role, replicates
nobody's LocalDb and does not even run the alarm historian — so every driver
node logged "Historian drain suspended", including the two site-b nodes that
have no peer at all. Nothing drained anywhere, where before Phase 2 it drained
fine. The cost is not a duplicate; it is the buffer growing to the capacity
wall and evicting the audit trail it exists to protect.
Fixed in three layers: a separate ShouldDrainAlarmHistory policy (unknown role
drains; the two gates now deliberately disagree, and a test pins that); peer-
host matching in DriverHostActor so a node stands down only for a Primary
holding its rows; and AddAlarmHistorian short-circuiting the gate when
replication is unconfigured — testing BOTH Replication:PeerAddress and
SyncListenPort, since only the dialing half sets the former while both halves
share the queue.
Every one of these follows from the asymmetry: a false allow costs a duplicate
row, which at-least-once delivery already accepts and payload-hash ids
collapse; a false deny loses data silently.
A third vacuous test, caught by the same delete-the-guard discipline: the
role-view tests stayed green with the guard removed, because AwaitAssert polls
until an assertion passes and the assertion was "reads open" — which is the
SEEDED value, satisfied at the first poll before the actor processed anything.
They now assert the sequence of published values through a recording view; the
control then goes red for exactly the cases that matter.
Migration evidence: 11 legacy rows across two deliberately overlapping files
converged to exactly 9 identical rows on both nodes, proving D-6's payload-hash
identity on real nodes rather than in a fixture.
Open design fork, recorded in the gate doc rather than decided here: a pair
cannot currently identify its own Primary, so both halves drain. Safe in every
topology — nothing loses data — but the gate's de-duplication benefit is
unrealised until roles are scoped per pair.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Joseph Doherty