Per-plan copy-paste command (subagent-driven-development in a git worktree),
plus the slash-command and executing-plans/resume alternatives, and a note that
the four independent plans can run in concurrent worktrees.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Add a living progress tracker (docs/plans/2026-07-24-driver-expansion-tracking.md)
for the driver-expansion program's Waves 0-2, linked from the program design doc's
document map. One row per deliverable pointing at design + implementation plan + status.
Add executable implementation plans (plan .md + co-located .tasks.json) for the four
Wave 1/2 drivers, each authored by writing-plans methodology off its design doc and
mirroring the relevant existing driver:
- Modbus RTU 11 tasks (extends Driver.Modbus; RTU-over-TCP, direct-serial descoped)
- SQL poll 22 tasks (ISqlDialect seam, SQL Server only in v1; SQLite + central-SQL fixtures)
- MTConnect Agent 23 tasks (browse free via Wave-0 seam; TrakHound-vs-hand-rolled = Task 0)
- MQTT/Sparkplug B 27 tasks (P1 plain MQTT 0-14 then P2 Sparkplug 15-26; MQTTnet-5 pinning spike = Task 0)
Wave 0 (universal browser) remains merged (056887d6) with its live gate open (#468).
All four new plans are design-only -> plan-ready; nothing built yet.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
The simultaneous-cold-start split-brain, carried as a Phase 7 "candidate follow-up,
not shipped", is now closed by the opt-in Cluster:BootstrapGuard (279d1d0f). Documents
the guard in CLAUDE.md's bootstrap section and marks the Phase 7 finding closed.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Closes the residual Phase-7 gap: when BOTH nodes of a 2-node pair cold-start at the
same instant, each self-first seed runs FirstSeedNodeProcess, times out waiting for
the other, and forms its own 1-node cluster — two Primaries in one pair (the Phase 6
live gate reproduced this reliably on docker). The prior mitigation was operational
(staggered start / compose depends_on), which does not exist on production hardware.
The guard (dark switch Cluster:BootstrapGuard:Enabled, default OFF) prevents the split
without giving up cold-start-alone:
- The lower-address node is the preferred founder: self-first, forms immediately.
- The higher node probes its partner's Akka port (TCP connect) up to PartnerProbeSeconds:
reachable => peer-first (join the founder, never race it); unreachable => self-first
(partner is dead, form alone). The order is decided BEFORE the single JoinSeedNodes,
from an explicit reachability signal — never re-formed mid-handshake (the retired
SelfFormAfter failure mode). When on, Akka gets no config seeds (BuildClusterOptions)
and ClusterBootstrapCoordinator drives the join.
Review-driven hardening: case-insensitive tie-break (a hostname-casing mismatch would
reopen the split); fail-fast validation of the timing knobs; the residual "founder dies
in the probe->join window" hang is documented and made operator-visible (warning + a
restart recovers). Real-ActorSystem coordinator tests cover the load-bearing higher-node
cold-start-alone case; 147/147 Cluster tests pass.
Live-gated on docker-dev (site-a = enablement demo, serialization removed, guard on;
site-b keeps depends_on-serialization + guard off as the A/B control): simultaneous
start -> site-a-1 founds, site-a-2 probes-reachable-joins -> 250/240, NO split;
higher-node-alone -> probes dead founder, self-forms -> 250; founder rejoin -> 240.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
All 6 failover drills pass on the docker-dev three-mesh rig:
- D1 MAIN graceful manual-failover via the AdminUI "Trigger failover" button —
Primary leaves, peer→250, restarts and rejoins with no split.
- D2 SITE-A auto-down failover, both directions; D3 SITE-B crash-the-oldest.
- D4 auto-down 1-vs-1 survival — every survivor stayed Up (closes the gate
deferred since Phase 0a).
- D5 self-first cold-start-alone — a node boots with its partner down and forms
its own mesh (gate b).
- D6 recovery — restarted nodes rejoin as Secondary (240).
Ships a one-page co-located operator runbook (OtOpcUa + ScadaBridge on shared
site VMs). Manual-failover button confirmed MAIN-only by construction; site
pairs (driver-only, no UI) fail over via auto-down. Carried Phase-6 finding —
simultaneous cold-start of both pair VMs can split — mitigated operationally in
the runbook (staggered start); a product-level join guard is a candidate
follow-up, not shipped.
The per-cluster mesh program (Phases 0a–7) is COMPLETE.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Records the full live-gate run against master on the docker-dev three-mesh rig.
All 8 legs pass: three isolated 2-node meshes, one Primary per pair (250/240),
deploy sealed acks=6 over per-cluster ClusterClient, telemetry gRPC :4056 up +
reconnect, reconciler own-cluster-scoped, secrets pair-local, split validator
refuses Dps, and per-pair failover with auto-down 1-vs-1 survival (also closes the
deferred Phase 0a gate). Three defects were caught and fixed forward (3a4ed8dd,
792f28ec) — see the live-gate doc's Findings section. Task 9 marked completed.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Two docker-dev-only defects the Phase 6 live gate surfaced (both invisible to
`docker compose config`, which validates the file but never boots the app):
1. Driver-only site nodes crashed at boot with OptionsValidationException — LDAP
transport None + AllowInsecure false. Only central carried the Security__Ldap__*
block, but the site nodes serve OPC UA endpoints and validate LDAP options too, and
they override `environment` wholesale (YAML `<<:` doesn't deep-merge it). Extracted
the block to a shared &ldap-env anchor merged into every host node.
2. Site pairs split-brained at simultaneous startup (both nodes "JOINING itself",
250/250 instead of 250/240). Both are self-first seeds, so both run Akka's
FirstSeedNodeProcess; the partner didn't depend on its pair founder, so they raced
and each formed a 1-node cluster. Added an &akka-founder-healthcheck (bash /dev/tcp
probe of Akka 4053 — no nc/curl in the image) to site-a-1/site-b-1 and switched the
partners to depends_on service_healthy, so the founder forms first and the partner
JOINS it. (Carry the underlying simultaneous-cold-start property to Phase 7 — it is
inherent to symmetric self-first seeding, present for central since #459.)
Live gate record: docs/plans/2026-07-24-mesh-phase6-live-gate.md (legs 1-2 PASS —
three isolated meshes, one Primary per pair at 250/240).
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Redundancy.md: KNOWN-LIMITATION (per-Akka-cluster election) marked RESOLVED;
new 'Per-cluster meshes (Phase 6)' section + pair-local secrets note; manual-
failover mesh-scope caveat removed; auto-down 1-vs-1 note updated (every mesh is
now two nodes, drill deferred to Phase 7). IManualFailoverService + ClusterRedundancy
razor caveats updated to pair-local reality. Configuration.md documents the
cluster-{ClusterId} role, per-pair self-first seeds, SplitTopologyTransportValidator,
and the intentional Dps default. Program + design status tables + CLAUDE.md marked
Phase 6 DONE.
(Task 8 subagent completed the edits but died on an API error before committing;
reviewed and committed by the controller.)
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 6 splits the fleet into one mesh per Cluster, so an admin (central)
node's Cluster.State.Members shows only its own pair — it no longer sees
site members via gossip. The ClusterNodeAddressReconciler singleton compared
live membership against EVERY ClusterNode row fleet-wide, so post-split every
foreign-cluster row would log EnabledRowNotInCluster forever.
Scope the actor's two DB queries to rows whose ClusterId matches the admin
node's own (IClusterRoleInfo.ClusterId, sourced at registration). A legacy
admin node with no cluster role (null ClusterId) still reconciles the whole
fleet — it genuinely sees every member via gossip. The pure Reconcile
function and AddressMismatchKind semantics are unchanged; scoping lives
entirely in the actor's queries.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Plan + task ledger for splitting the single fleet mesh into three 2-node
meshes (central + per-site pairs), cluster-scoped roles + redundancy
singleton, per-cluster ClusterClient, own-cluster reconciler, split-topology
transport validator + flipped defaults, pair-local secrets, and the docker-dev
rig rewrite. Decisions settled with the user 2026-07-24.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Full 12-stream telemetry mesh formed in Grpc mode, AdminUI pill live (data path
proven), kill-and-reconnect recovered a node's stream in ~5s; Dps baseline dials
nothing. Surfaced+validated the ClusterNode.GrpcPort upgrade-backfill gotcha.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Deploy sealed green with 4 DB-less site nodes acking (central persists acks);
ServiceLevel held 240 on a site node with central SQL stopped (survive-alone);
restarted site node booted last-known-good from the LocalDb pointer, no ConfigDb
read; alarm_condition_state table live + replicated. Full record in
2026-07-23-mesh-phase4-live-gate.md. Task 9 (drop dead table) deferred.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Task 4+6 code review re-surfaced the OpcUaPublishActor raw-sink wipe on a
null-factory node (the exact #485/#486 defect Task 3 fixes) — Task 3 must land
before any node flips to FetchAndCache. Ordering already enforces this
(3 → 7 → 10 → 11).
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Nullable ConfigDb factory; UpsertNodeDeploymentState no-ops when absent
(central persists acks from the ApplyAck); scripted-alarm condition state
served from the LocalDb store instead of the ConfigDb-backed Ef store.
Combines Phase-4 Tasks 4 + 6. Removes the interim dbFactory! cast.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Task 2 code review surfaced two unguarded ConfigDb sites reachable on a
driver-only node once Tasks 0+1 landed (SpawnScriptedAlarmHost's Ef store,
unconditional UpsertNodeDeploymentState). Task 4 brief now names them + the
dbFactory! removal; Task 6 re-homes the alarm store to LocalDb.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 4 of the per-cluster mesh program. Removes the four remaining driver-side
ConfigDb consumers + the registration itself so a driver-only node boots with no
ConfigDb connection string:
- ConfigDb registered iff hasAdmin (Program.cs); driver-only requires none
- driver-only ⇒ FetchAndCache mandatory (validator)
- DbHealthProbeActor not spawned on driver-only; DbReachable=true constant
(client-visible ServiceLevel change — a healthy DB-less node publishes 240/250
with central down, per user decision 2026-07-23)
- DriverHostActor: nullable factory, drop redundant SQL ack-writes (central's
PersistNodeAck is the ack system of record)
- EfAlarmConditionStateStore → LocalDbAlarmConditionStateStore (new replicated
alarm_condition_state table)
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 3 Task 9. docker-dev rig: central-1/2 carry ConfigServe (:4055 h2c + committed
dev key) and stay Direct; the four site nodes carry ConfigSource (both central
endpoints + matching key) defaulting to Direct. Flip only the site nodes with
OTOPCUA_CONFIG_MODE=FetchAndCache at 'docker compose up' — central keeps SQL and models
the target topology. Binding :4055 activates the dedicated-h2c Kestrel takeover on
central (re-binds :9000 too). Docs: new docs/Configuration.md ConfigSource/ConfigServe
section; docs/Redundancy.md 'config bytes travel out-of-band' note; the program plan
Phase 3 section + tracking row flipped to CODE-COMPLETE; a CLAUDE.md 'Config source'
section.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Eleven tasks. User decisions baked in: gRPC fetch RPC (not token-gated HTTP),
shared node bearer key (not a per-deployment token, so no migration and
DispatchDeployment is unchanged), a ConfigSource:Mode dark switch (Direct
default), both pair nodes fetch (no Primary gating). Header carries the five
recon facts and the Phase 3/4 boundary (config READS only; the NodeDeploymentState
write, DbHealthProbe and EfAlarmConditionStateStore stay for Phase 4).
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Flipped the docker-dev rig end-to-end to MeshTransport:Mode=ClusterClient
(OTOPCUA_MESH_MODE, no rebuild) and ran the gate. PASSED for the transport:
deploy seals on ClusterClient, a stopped node fails the deploy naming it, the
MaintenanceMode hatch shrinks the DB-sourced contact set live (6->5->6 on both
centrals independently), and the frame-size canary stayed silent.
Four findings, recorded in 2026-07-22-mesh-phase2-live-gate.md:
A. A 10-30s post-restart delivery window exists, but the DPS control fails
identically -- it is a cluster-convergence property, not a Phase 2
regression. Running the control is the only reason this is a note and not
a false attribution.
B. buffer-size=0's stated rationale was OVERSTATED. A node applies a TimedOut
deployment anyway -- live-observed at +44s -- via DriverHostActor.Bootstrap
replaying the orphan Applying row on restart. Zero buffering removes the
silent, timer-driven replay, not the boot-time DB one. Corrected in the
plan decisions table, Configuration.md, and CLAUDE.md; the decision stands,
the justification narrowed.
C. Stopping BOTH centrals strands the non-seed nodes at the Akka membership
layer (frozen mid-Exiting, not Unreachable, so auto-down does not help).
Transport-independent; a Phase 7 downing/rejoin drill item.
D. The AdminUI driver Restart/Reconnect buttons (DriverStatusPanel) are
rendered by no page -- orphaned since v3 Batch 2 retired /clusters/{id}/
drivers. So gate steps 6/7 could not run; the driver-control node-side leg
is proven live only because DispatchDeployment shares its EventStream path.
Step 8's planned log-signal does not exist (HandleApplyAck is silent on
success); proven instead by both centrals independently logging per-instance
ClusterClient creation, which a singleton cannot.
Also adds a repo-root .dockerignore: docker-dev/Dockerfile does COPY . ., which
was pulling .claude/worktrees (19GB of agent checkouts with their own bin/obj)
into the build context and exhausting the Docker VM disk mid-COPY.
Rig restored to Mode=Dps, six members, no leftover maintenance flags.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Rig: all six docker-dev nodes carry MeshTransport__Mode plus both central
contact points. Mode is "${OTOPCUA_MESH_MODE:-Dps}", so the rig still comes up
on the transport it has always used and the live gate can flip the whole fleet
at `docker compose up` without a compose edit or a rebuild.
Docs: a MeshTransport section in Configuration.md (including why contact points
are addresses only, why the contact set does not scope delivery, and why
buffer-size=0 is a behaviour decision rather than a tuning knob), a "Command
transport" section in Redundancy.md contrasting the two modes, and a summary
beside the Redundancy notes in CLAUDE.md.
Corrects the design doc's claim that ScadaBridge replies with a typed failure to
every unhandled message. Neither of their comm actors has a ReceiveAny or an
Unhandled override; the idiom is per message type and fires on a missing
registration, and an unknown type dead-letters there as it does anywhere else.
Records the exit-gate deviation in the program plan: "an Ask timing out cleanly
against a stopped node" cannot be run, because no Ask crosses this boundary. The
honest equivalents are a stopped node failing the deploy at the apply deadline
and an unreachable contact dropping the command with a Warning.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 2 of the per-cluster mesh program: move the three central->node command
channels (deployments, driver-control, alarm-commands) and the deployment-acks
reply channel off DistributedPubSub onto Akka ClusterClient, behind a config
flag.
Recon of both repos turned up three corrections to the design doc's account of
the sister project, all folded into the plan:
- Design doc S2 claims "every unhandled message replies with a typed failure".
ScadaBridge has no ReceiveAny/Unhandled override in either comm actor; the
typed-failure idiom fires only for a MISSING REGISTRATION, per message type.
- "No central buffering" was never a setting they wrote. They have zero
akka.cluster.client HOCON and run the defaults (buffer-size 1000,
reconnect-timeout off). We choose buffer-size = 0 deliberately.
- Publish -> Send is the wrong substitution. Today's deploy notify is a
broadcast every DriverHostActor receives (no ClusterId filter exists on the
node side), so it must become SendToAll. Send would deploy to exactly one
node of the fleet and seal green on partial acks.
Also records the single-mesh duplicate-delivery trap (one ClusterClient in
Phase 2, not one per Cluster -- a receptionist serves its whole mesh) and one
deviation from the program plan's exit gate: there is no cross-boundary Ask in
Phase 2, because every migrated command is fire-and-forget with a local reply,
so "an Ask timing out cleanly" cannot be run as written.
Marks the program tracking tables current: prereq + Phase 1 are DONE (they
still read "not executed"), Phase 2 in progress.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 1 gate step 4 failed: setting Enabled = 0 to take a node out of service
returns 422 ClusterEnabledNodeCountMismatch, because
DraftValidator.ValidateClusterTopology requires the enabled-node count to equal
ServerCluster.NodeCount. On a Warm/Hot pair — every cluster on the rig, and
every cluster in the target topology — Enabled can therefore never be the
maintenance hatch. The plan called step 4 "the check that makes step 3
acceptable to ship", so Task 3's behaviour change was not shippable as it stood:
a node down for maintenance would block every deployment to its cluster.
The two rules were each reasonable and contradictory together — the validator
reads Enabled as "part of the declared topology", Phase 1 additionally read it
as "expect an ack". Split the meanings rather than weaken either rule:
Enabled part of the declared topology (validator, untouched)
MaintenanceMode expected to participate now (coordinator + reconciler)
Rejected alternatives: counting configured rather than enabled nodes (drops the
guard against booting a pair into InvalidTopology); downgrading the rule to a
warning (weakens a deploy gate for everyone); shipping with no hatch.
Sabotage: dropping !n.MaintenanceMode from the coordinator query turns the new
test red. It also asserts both nodes remain Enabled, so the fix cannot quietly
regress to disabling the row after all.
Configuration.Tests 95/95, ControlPlane.Tests 101/101, solution builds clean.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
- docker-dev seed: AkkaPort = 4053 on all six ClusterNode rows, so a freshly
seeded rig matches a migrated one. GrpcPort left null.
- config-db-schema.md: both columns, why AkkaPort is NOT NULL/4053 and GrpcPort
is nullable, the unenforced duplication + its reconciler, and Enabled's new
second meaning as the deploy path's expected-ack set.
- Configuration.md: Cluster:Port / PublicHostname now flag that they are stored
twice, with the "update the row too" instruction and why the drift is silent.
- design doc §7: Phase 1 marked done, plus a "Phase 1 as shipped" note recording
both deviations rather than leaving the sketch reading as what happened.
- program plan: Phase 1 marked done; AdminUI node edit explicitly deferred.
- CLAUDE.md: the deploy-path behaviour change and its three consequences.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
ClusterNode.AkkaPort and the node's own Cluster:Port are the same fact in two
places and nothing made them agree. Phase 2 dials the row instead of gossiping,
so a node binding 4054 while its row says 4053 becomes unreachable from central
— and the symptom is a silent absence of acks rather than an error. That is the
shape of the Modbus/ModbusTcp and TwinCat/Focas drifts already in this repo.
Since Phase 1 also made the rows the deploy path's expected-ack set, drift
already costs a failed deployment today.
Implemented as the plan's preferred option: an admin-role singleton comparing
rows against the membership an admin node can already see, rather than each
driver node asserting its own row — Phase 4 removes the driver nodes' ConfigDb
connection, so a self-assertion written there would have to be deleted again.
Three shapes, split by severity: a row whose dial target disagrees with its own
NodeId and a running node with no row are Errors; an enabled row with no
matching member is a Warning, because a node down for maintenance is a
legitimate state. Findings are logged only when the set changes — a check that
reprints the same warning every sweep trains operators to filter it out.
Documented limitation: Phase 2 must revisit this. Once the fleet splits into
one mesh per cluster an admin node cannot see site members, and every site row
would report EnabledRowNotInCluster forever.
Sabotage: removing the change-detection guard turns the repeat test red.
ControlPlane.Tests 100/100.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Per-cluster mesh Phase 1 groundwork. Once the meshes split (Phase 2) central
can no longer see a site node's appsettings, so the transport ports it must
dial have to live in a row central can read.
AkkaPort is non-nullable with a 4053 default — every node listens on a
remoting port, so 0 is never a truthful value and pre-existing rows must
migrate to something real. GrpcPort is nullable with no default: nothing
listens on it until Phase 5, and a non-null default would assert a port that
does not exist.
Both are documented as central's dial targets rather than the node's own
binding config; the duplication against Cluster:Port is reconciled in Task 4.
The new SchemaCompliance test is red until the migration lands (Task 2).
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Joseph Doherty