lmxopcua/docs/plans/2026-07-22-mesh-phase1-live-gate.md

Per-cluster mesh Phase 1 — live gate record

Date: 2026-07-22 · Rig: docker-dev (six-node single mesh) · Branch: feat/mesh-phase1 Plan: 2026-07-21-per-cluster-mesh-phase1.md

Result: PASSED, after the gate found a blocker that changed the design (step 4).

Offline tests cannot cover the substitution's real risk — that the DB set and the live set disagree on a running fleet. This is that check.

Setup

Migration applied by the rig's migrator service against the shared OtOpcUa database, then all six host nodes restarted on the rebuilt image. All six pre-existing ClusterNode rows inherited AkkaPort = 4053 from the migration's DEFAULT 4053 — the row-level evidence that the default does its job for rows predating the column:

central-1:4053|central-1|4053|NULL|1
central-2:4053|central-2|4053|NULL|1
site-a-1:4053 |site-a-1 |4053|NULL|1
site-a-2:4053 |site-a-2 |4053|NULL|1
site-b-1:4053 |site-b-1 |4053|NULL|1
site-b-2:4053 |site-b-2 |4053|NULL|1
   NodeId     |  Host   |Akka|Grpc|En

Step 1 — deploy with all six nodes up → PASS

POST /api/deployments → 202 Accepted, deployment 38f83b37.

Deployment.Status = 2 (Sealed). Six NodeDeploymentState rows, every one Status = 1 (Applied). No FK 547 — the DB-derived NodeIds satisfy the NodeDeploymentState → ClusterNode foreign key, which is the standing proof that the DB set and the membership set agree on this rig.

Step 2 — dropped, not skipped

The plan called for a cluster-scoped deploy. There is no such thing. Deployment has no ClusterId, ConfigComposer.SnapshotAndFlattenAsync always snapshots the whole DB, and DeploymentArtifact.ResolveClusterScope is node-side self-scoping of a fleet-wide artifact. Removed from the plan rather than faked green.

Step 3 — deploy with one node stopped → PASS (the new behaviour)

docker stop otopcua-dev-site-b-2-1, then deploy 37d72301.

Deployment.Status = 4 (TimedOut) after the 2-minute apply deadline. Five nodes Applied, site-b-2:4053 still Applying. Under the membership rule this deployment would have sealed green with five nodes and never mentioned the sixth.

The operator-facing log, which the plan required to be legible:

[12:52:02 WRN] Deployment 37d723018e374c0eb04c7b6beccc3b3e timed out after 00:02:00 (5/6 acks
landed). No ack from: site-b-2:4053. Each is an enabled ClusterNode row — start the node, or set
ClusterNode.Enabled = 0 if it is down for maintenance, then redeploy

(The remedy in that message was corrected to MaintenanceMode = 1 — see step 4.)

Step 4 — the escape hatch → FAILED, then fixed and re-run → PASS

First attempt: the hatch did not exist

UPDATE ClusterNode SET Enabled = 0 WHERE NodeId = 'site-b-2:4053', then deploy:

HTTP 422
[ClusterEnabledNodeCountMismatch] Cluster 'SITE-B' declares NodeCount=2 but has 1 Enabled nodes.
Toggle the missing node(s) back on or change RedundancyMode/NodeCount to match.

DraftValidator.ValidateClusterTopology rejects the deployment before it dispatches. So Enabled = 0 cannot be used on a node of a 2-node cluster — and every cluster on this rig, and every cluster in the program plan's target topology, is a 2-node pair.

This was a blocker, not a wrinkle. The plan called step 4 "the check that makes step 3 acceptable to ship". Without it, a node down for maintenance blocks every deployment to its cluster, with no remedy short of editing NodeCount / RedundancyMode — which changes the runtime redundancy posture to work around a deploy gate.

The two rules were each reasonable and contradictory together: the validator reads Enabled as "part of the declared topology", Phase 1 additionally read it as "expect an ack".

Fix (Task 7): split the meanings

New ClusterNode.MaintenanceMode bit NOT NULL DEFAULT 0. Enabled keeps its topology meaning and the validator is untouched; the coordinator's expected-ack set and the address reconciler both filter on Enabled && !MaintenanceMode.

Re-run: PASS

docker stop site-b-2 + MaintenanceMode = 1, then deploy 82c5e678:

• 202 Accepted — the topology gate passes, because the node is still Enabled (1|1).
• Deployment.Status = 2 (Sealed).
• Five NodeDeploymentState rows, all Applied. site-b-2:4053 has no row at all — excluded from the expected set rather than waited for and forgiven.

Task 4 (reconciler) — live behaviour, not a planned gate step

Observed unprompted during the rolling restart, which is better evidence than a staged check:

[12:44:49 WRN] ClusterNode address check [EnabledRowNotInCluster] site-a-1:4053: ...
[12:44:49 WRN] ... site-a-2:4053 ... site-b-1:4053 ... site-b-2:4053
[12:44:56 INF] ClusterNode addresses reconcile with cluster membership (6 driver members)

It warned per row while the site nodes were still starting, then converged to a single clean line — and logged nothing further, confirming the change-detection guard suppresses repeats on the 5-minute sweep.

Maintenance interaction, from the step 4 re-run:

[13:06:02 WRN] ... site-b-2:4053 ...            ← node dropped, flag not yet set
[13:06:05 INF] ... reconcile with cluster membership (5 driver members)   ← flag set: silent

The transient warning is honest — for those three seconds the row genuinely was enabled, present in the expected-ack set, and absent from the cluster.

What this gate did not cover

• Phase 2's premise. AkkaPort / GrpcPort are groundwork; nothing dials them yet, so "the value is correct" is only checked by the reconciler comparing it to gossip. The first real exercise is Phase 2.
• The reconciler under split meshes. On the current single mesh an admin node sees every driver member. After Phase 6 it will not, and every site row would report EnabledRowNotInCluster permanently. Recorded as a known limitation on the class and in the design doc.
• Enabled = 0 on a single-node (RedundancyMode.None) cluster. Untested — the rig has no such cluster. It would fail the same topology rule (0 enabled vs NodeCount = 1), so MaintenanceMode is the hatch there too.