Joseph Doherty
ee69caa270
Phase 1 gate step 4 failed: setting Enabled = 0 to take a node out of service returns 422 ClusterEnabledNodeCountMismatch, because DraftValidator.ValidateClusterTopology requires the enabled-node count to equal ServerCluster.NodeCount. On a Warm/Hot pair — every cluster on the rig, and every cluster in the target topology — Enabled can therefore never be the maintenance hatch. The plan called step 4 "the check that makes step 3 acceptable to ship", so Task 3's behaviour change was not shippable as it stood: a node down for maintenance would block every deployment to its cluster. The two rules were each reasonable and contradictory together — the validator reads Enabled as "part of the declared topology", Phase 1 additionally read it as "expect an ack". Split the meanings rather than weaken either rule: Enabled part of the declared topology (validator, untouched) MaintenanceMode expected to participate now (coordinator + reconciler) Rejected alternatives: counting configured rather than enabled nodes (drops the guard against booting a pair into InvalidTopology); downgrading the rule to a warning (weakens a deploy gate for everyone); shipping with no hatch. Sabotage: dropping !n.MaintenanceMode from the coordinator query turns the new test red. It also asserts both nodes remain Enabled, so the fix cannot quietly regress to disabling the row after all. Configuration.Tests 95/95, ControlPlane.Tests 101/101, solution builds clean. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
135 lines
6.1 KiB
Markdown
135 lines
6.1 KiB
Markdown
# Per-cluster mesh Phase 1 — live gate record
|
|
|
|
**Date:** 2026-07-22 · **Rig:** `docker-dev` (six-node single mesh) · **Branch:** `feat/mesh-phase1`
|
|
**Plan:** [`2026-07-21-per-cluster-mesh-phase1.md`](2026-07-21-per-cluster-mesh-phase1.md)
|
|
|
|
**Result: PASSED**, after the gate found a blocker that changed the design (step 4).
|
|
|
|
Offline tests cannot cover the substitution's real risk — that the DB set and the live set disagree on
|
|
a running fleet. This is that check.
|
|
|
|
## Setup
|
|
|
|
Migration applied by the rig's `migrator` service against the shared `OtOpcUa` database, then all six
|
|
host nodes restarted on the rebuilt image. **All six pre-existing `ClusterNode` rows inherited
|
|
`AkkaPort = 4053` from the migration's `DEFAULT 4053`** — the row-level evidence that the default does
|
|
its job for rows predating the column:
|
|
|
|
```
|
|
central-1:4053|central-1|4053|NULL|1
|
|
central-2:4053|central-2|4053|NULL|1
|
|
site-a-1:4053 |site-a-1 |4053|NULL|1
|
|
site-a-2:4053 |site-a-2 |4053|NULL|1
|
|
site-b-1:4053 |site-b-1 |4053|NULL|1
|
|
site-b-2:4053 |site-b-2 |4053|NULL|1
|
|
NodeId | Host |Akka|Grpc|En
|
|
```
|
|
|
|
## Step 1 — deploy with all six nodes up → **PASS**
|
|
|
|
`POST /api/deployments` → `202 Accepted`, deployment `38f83b37`.
|
|
|
|
`Deployment.Status = 2` (Sealed). Six `NodeDeploymentState` rows, every one `Status = 1` (Applied).
|
|
**No FK 547** — the DB-derived NodeIds satisfy the `NodeDeploymentState → ClusterNode` foreign key,
|
|
which is the standing proof that the DB set and the membership set agree on this rig.
|
|
|
|
## Step 2 — *dropped, not skipped*
|
|
|
|
The plan called for a cluster-scoped deploy. **There is no such thing.** `Deployment` has no
|
|
`ClusterId`, `ConfigComposer.SnapshotAndFlattenAsync` always snapshots the whole DB, and
|
|
`DeploymentArtifact.ResolveClusterScope` is *node-side* self-scoping of a fleet-wide artifact. Removed
|
|
from the plan rather than faked green.
|
|
|
|
## Step 3 — deploy with one node stopped → **PASS** (the new behaviour)
|
|
|
|
`docker stop otopcua-dev-site-b-2-1`, then deploy `37d72301`.
|
|
|
|
`Deployment.Status = 4` (TimedOut) after the 2-minute apply deadline. Five nodes `Applied`,
|
|
`site-b-2:4053` still `Applying`. Under the membership rule this deployment would have **sealed
|
|
green** with five nodes and never mentioned the sixth.
|
|
|
|
The operator-facing log, which the plan required to be legible:
|
|
|
|
```
|
|
[12:52:02 WRN] Deployment 37d723018e374c0eb04c7b6beccc3b3e timed out after 00:02:00 (5/6 acks
|
|
landed). No ack from: site-b-2:4053. Each is an enabled ClusterNode row — start the node, or set
|
|
ClusterNode.Enabled = 0 if it is down for maintenance, then redeploy
|
|
```
|
|
|
|
(The remedy in that message was corrected to `MaintenanceMode = 1` — see step 4.)
|
|
|
|
## Step 4 — the escape hatch → **FAILED, then fixed and re-run → PASS**
|
|
|
|
### First attempt: the hatch did not exist
|
|
|
|
`UPDATE ClusterNode SET Enabled = 0 WHERE NodeId = 'site-b-2:4053'`, then deploy:
|
|
|
|
```
|
|
HTTP 422
|
|
[ClusterEnabledNodeCountMismatch] Cluster 'SITE-B' declares NodeCount=2 but has 1 Enabled nodes.
|
|
Toggle the missing node(s) back on or change RedundancyMode/NodeCount to match.
|
|
```
|
|
|
|
`DraftValidator.ValidateClusterTopology` rejects the deployment **before it dispatches**. So
|
|
`Enabled = 0` cannot be used on a node of a 2-node cluster — and every cluster on this rig, and every
|
|
cluster in the program plan's target topology, is a 2-node pair.
|
|
|
|
**This was a blocker, not a wrinkle.** The plan called step 4 *"the check that makes step 3
|
|
acceptable to ship"*. Without it, a node down for maintenance blocks every deployment to its cluster,
|
|
with no remedy short of editing `NodeCount` / `RedundancyMode` — which changes the runtime redundancy
|
|
posture to work around a deploy gate.
|
|
|
|
The two rules were each reasonable and contradictory together: the validator reads `Enabled` as *"part
|
|
of the declared topology"*, Phase 1 additionally read it as *"expect an ack"*.
|
|
|
|
### Fix (Task 7): split the meanings
|
|
|
|
New `ClusterNode.MaintenanceMode bit NOT NULL DEFAULT 0`. `Enabled` keeps its topology meaning and the
|
|
validator is untouched; the coordinator's expected-ack set and the address reconciler both filter on
|
|
`Enabled && !MaintenanceMode`.
|
|
|
|
### Re-run: PASS
|
|
|
|
`docker stop` `site-b-2` + `MaintenanceMode = 1`, then deploy `82c5e678`:
|
|
|
|
- `202 Accepted` — the topology gate passes, because the node is still `Enabled` (`1|1`).
|
|
- `Deployment.Status = 2` (Sealed).
|
|
- **Five** `NodeDeploymentState` rows, all `Applied`. `site-b-2:4053` has no row at all — excluded
|
|
from the expected set rather than waited for and forgiven.
|
|
|
|
## Task 4 (reconciler) — live behaviour, not a planned gate step
|
|
|
|
Observed unprompted during the rolling restart, which is better evidence than a staged check:
|
|
|
|
```
|
|
[12:44:49 WRN] ClusterNode address check [EnabledRowNotInCluster] site-a-1:4053: ...
|
|
[12:44:49 WRN] ... site-a-2:4053 ... site-b-1:4053 ... site-b-2:4053
|
|
[12:44:56 INF] ClusterNode addresses reconcile with cluster membership (6 driver members)
|
|
```
|
|
|
|
It warned per row while the site nodes were still starting, then converged to a single clean line —
|
|
and **logged nothing further**, confirming the change-detection guard suppresses repeats on the
|
|
5-minute sweep.
|
|
|
|
Maintenance interaction, from the step 4 re-run:
|
|
|
|
```
|
|
[13:06:02 WRN] ... site-b-2:4053 ... ← node dropped, flag not yet set
|
|
[13:06:05 INF] ... reconcile with cluster membership (5 driver members) ← flag set: silent
|
|
```
|
|
|
|
The transient warning is honest — for those three seconds the row genuinely was enabled, present in
|
|
the expected-ack set, and absent from the cluster.
|
|
|
|
## What this gate did not cover
|
|
|
|
- **Phase 2's premise.** `AkkaPort` / `GrpcPort` are groundwork; nothing dials them yet, so "the value
|
|
is correct" is only checked by the reconciler comparing it to gossip. The first real exercise is
|
|
Phase 2.
|
|
- **The reconciler under split meshes.** On the current single mesh an admin node sees every driver
|
|
member. After Phase 6 it will not, and every site row would report `EnabledRowNotInCluster`
|
|
permanently. Recorded as a known limitation on the class and in the design doc.
|
|
- **`Enabled = 0` on a single-node (`RedundancyMode.None`) cluster.** Untested — the rig has no such
|
|
cluster. It would fail the same topology rule (0 enabled vs `NodeCount = 1`), so `MaintenanceMode`
|
|
is the hatch there too.
|