feat(config): add ClusterNode.MaintenanceMode — the hatch the live gate proved missing
Phase 1 gate step 4 failed: setting Enabled = 0 to take a node out of service returns 422 ClusterEnabledNodeCountMismatch, because DraftValidator.ValidateClusterTopology requires the enabled-node count to equal ServerCluster.NodeCount. On a Warm/Hot pair — every cluster on the rig, and every cluster in the target topology — Enabled can therefore never be the maintenance hatch. The plan called step 4 "the check that makes step 3 acceptable to ship", so Task 3's behaviour change was not shippable as it stood: a node down for maintenance would block every deployment to its cluster. The two rules were each reasonable and contradictory together — the validator reads Enabled as "part of the declared topology", Phase 1 additionally read it as "expect an ack". Split the meanings rather than weaken either rule: Enabled part of the declared topology (validator, untouched) MaintenanceMode expected to participate now (coordinator + reconciler) Rejected alternatives: counting configured rather than enabled nodes (drops the guard against booting a pair into InvalidTopology); downgrading the rule to a warning (weakens a deploy gate for everyone); shipping with no hatch. Sabotage: dropping !n.MaintenanceMode from the coordinator query turns the new test red. It also asserts both nodes remain Enabled, so the fix cannot quietly regress to disabling the row after all. Configuration.Tests 95/95, ControlPlane.Tests 101/101, solution builds clean. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
@@ -12,8 +12,17 @@ sourcing its expected-ack set from those rows.
|
||||
**Architecture:** Additive schema change plus one substitution inside the coordinator. Nothing else
|
||||
moves; the deploy channel stays on DistributedPubSub until Phase 2.
|
||||
|
||||
**Status:** NOT STARTED. Read §"The decision this phase forces" before executing — it changes
|
||||
deploy-seal semantics and should be confirmed, not assumed.
|
||||
**Status: DONE 2026-07-22** (branch `feat/mesh-phase1`). Live gate:
|
||||
[`2026-07-22-mesh-phase1-live-gate.md`](2026-07-22-mesh-phase1-live-gate.md) — **PASSED**, after gate
|
||||
step 4 found that the escape hatch this phase depends on did not exist. Three deviations from the
|
||||
plan below, all decided before implementing or forced by the gate:
|
||||
|
||||
1. **Task 3's cluster-scope filtering was dropped** — there is no cluster-scoped deployment to filter
|
||||
on. See the gate doc, step 2.
|
||||
2. **No per-node role column.** Every `ClusterNode` row is now *defined* to be a driver node.
|
||||
3. **Task 7 added:** `ClusterNode.MaintenanceMode`. `Enabled = 0` is rejected by
|
||||
`DraftValidator.ValidateClusterTopology` on any 2-node pair, so it could never be the maintenance
|
||||
hatch — which the plan called "the check that makes step 3 acceptable to ship".
|
||||
|
||||
---
|
||||
|
||||
@@ -54,7 +63,7 @@ But it is a behaviour change, and it needs an operator escape hatch or a node ta
|
||||
maintenance blocks every deployment. `ClusterNode.Enabled` already exists and is exactly that hatch,
|
||||
so the query filters on it: **`Enabled = 1` rows are expected to ack; disabled rows are not.**
|
||||
|
||||
**Confirm this before executing Task 3.** If the preference is to keep sealing green past a down
|
||||
**Confirmed 2026-07-22 before executing Task 3** — DB-derived with the hatch. (The hatch turned out to be `MaintenanceMode`, not `Enabled`; see Task 7.) If the preference is to keep sealing green past a down
|
||||
node, the alternative is to intersect the DB set with current membership, which preserves today's
|
||||
behaviour but does not survive the mesh split — it would have to be revisited in Phase 2 anyway.
|
||||
|
||||
@@ -189,3 +198,25 @@ disagree on the rig. Run on `docker-dev`:
|
||||
hatch, and is the check that makes step 3 acceptable to ship.
|
||||
|
||||
Record results in a gate doc beside this plan, per the Phase 1/Phase 2 precedent.
|
||||
|
||||
## Task 7: `MaintenanceMode` — the escape hatch step 4 proved missing (ADDED 2026-07-22)
|
||||
|
||||
**Classification:** high-risk — deploy path + schema
|
||||
**Why:** gate step 4 returned `422 ClusterEnabledNodeCountMismatch`. `Enabled = 0` on one node of a
|
||||
Warm/Hot pair fails `DraftValidator.ValidateClusterTopology` (enabled count must equal `NodeCount`),
|
||||
so the hatch Task 3 relies on cannot be used on any 2-node cluster — i.e. every cluster in the target
|
||||
topology. Task 3's behaviour change is only shippable with a hatch that works.
|
||||
|
||||
**Decision:** split the meanings rather than weaken either rule.
|
||||
|
||||
| Flag | Question | Enforced by |
|
||||
|---|---|---|
|
||||
| `Enabled` | part of the declared topology? | `DraftValidator` vs `ServerCluster.NodeCount` |
|
||||
| `MaintenanceMode` | expected to participate right now? | `ConfigPublishCoordinator`, `ClusterNodeAddressReconciler` |
|
||||
|
||||
Rejected: counting configured rather than enabled nodes (removes the InvalidTopology guard);
|
||||
downgrading the rule to a warning (weakens a deploy gate for everyone); shipping without a hatch.
|
||||
|
||||
**Done:** column + migration `AddClusterNodeMaintenanceMode`; both consumers filter on
|
||||
`Enabled && !MaintenanceMode`; deadline + reconciler messages name the right flag; docs; gate step 4
|
||||
re-run green.
|
||||
|
||||
@@ -0,0 +1,134 @@
|
||||
# Per-cluster mesh Phase 1 — live gate record
|
||||
|
||||
**Date:** 2026-07-22 · **Rig:** `docker-dev` (six-node single mesh) · **Branch:** `feat/mesh-phase1`
|
||||
**Plan:** [`2026-07-21-per-cluster-mesh-phase1.md`](2026-07-21-per-cluster-mesh-phase1.md)
|
||||
|
||||
**Result: PASSED**, after the gate found a blocker that changed the design (step 4).
|
||||
|
||||
Offline tests cannot cover the substitution's real risk — that the DB set and the live set disagree on
|
||||
a running fleet. This is that check.
|
||||
|
||||
## Setup
|
||||
|
||||
Migration applied by the rig's `migrator` service against the shared `OtOpcUa` database, then all six
|
||||
host nodes restarted on the rebuilt image. **All six pre-existing `ClusterNode` rows inherited
|
||||
`AkkaPort = 4053` from the migration's `DEFAULT 4053`** — the row-level evidence that the default does
|
||||
its job for rows predating the column:
|
||||
|
||||
```
|
||||
central-1:4053|central-1|4053|NULL|1
|
||||
central-2:4053|central-2|4053|NULL|1
|
||||
site-a-1:4053 |site-a-1 |4053|NULL|1
|
||||
site-a-2:4053 |site-a-2 |4053|NULL|1
|
||||
site-b-1:4053 |site-b-1 |4053|NULL|1
|
||||
site-b-2:4053 |site-b-2 |4053|NULL|1
|
||||
NodeId | Host |Akka|Grpc|En
|
||||
```
|
||||
|
||||
## Step 1 — deploy with all six nodes up → **PASS**
|
||||
|
||||
`POST /api/deployments` → `202 Accepted`, deployment `38f83b37`.
|
||||
|
||||
`Deployment.Status = 2` (Sealed). Six `NodeDeploymentState` rows, every one `Status = 1` (Applied).
|
||||
**No FK 547** — the DB-derived NodeIds satisfy the `NodeDeploymentState → ClusterNode` foreign key,
|
||||
which is the standing proof that the DB set and the membership set agree on this rig.
|
||||
|
||||
## Step 2 — *dropped, not skipped*
|
||||
|
||||
The plan called for a cluster-scoped deploy. **There is no such thing.** `Deployment` has no
|
||||
`ClusterId`, `ConfigComposer.SnapshotAndFlattenAsync` always snapshots the whole DB, and
|
||||
`DeploymentArtifact.ResolveClusterScope` is *node-side* self-scoping of a fleet-wide artifact. Removed
|
||||
from the plan rather than faked green.
|
||||
|
||||
## Step 3 — deploy with one node stopped → **PASS** (the new behaviour)
|
||||
|
||||
`docker stop otopcua-dev-site-b-2-1`, then deploy `37d72301`.
|
||||
|
||||
`Deployment.Status = 4` (TimedOut) after the 2-minute apply deadline. Five nodes `Applied`,
|
||||
`site-b-2:4053` still `Applying`. Under the membership rule this deployment would have **sealed
|
||||
green** with five nodes and never mentioned the sixth.
|
||||
|
||||
The operator-facing log, which the plan required to be legible:
|
||||
|
||||
```
|
||||
[12:52:02 WRN] Deployment 37d723018e374c0eb04c7b6beccc3b3e timed out after 00:02:00 (5/6 acks
|
||||
landed). No ack from: site-b-2:4053. Each is an enabled ClusterNode row — start the node, or set
|
||||
ClusterNode.Enabled = 0 if it is down for maintenance, then redeploy
|
||||
```
|
||||
|
||||
(The remedy in that message was corrected to `MaintenanceMode = 1` — see step 4.)
|
||||
|
||||
## Step 4 — the escape hatch → **FAILED, then fixed and re-run → PASS**
|
||||
|
||||
### First attempt: the hatch did not exist
|
||||
|
||||
`UPDATE ClusterNode SET Enabled = 0 WHERE NodeId = 'site-b-2:4053'`, then deploy:
|
||||
|
||||
```
|
||||
HTTP 422
|
||||
[ClusterEnabledNodeCountMismatch] Cluster 'SITE-B' declares NodeCount=2 but has 1 Enabled nodes.
|
||||
Toggle the missing node(s) back on or change RedundancyMode/NodeCount to match.
|
||||
```
|
||||
|
||||
`DraftValidator.ValidateClusterTopology` rejects the deployment **before it dispatches**. So
|
||||
`Enabled = 0` cannot be used on a node of a 2-node cluster — and every cluster on this rig, and every
|
||||
cluster in the program plan's target topology, is a 2-node pair.
|
||||
|
||||
**This was a blocker, not a wrinkle.** The plan called step 4 *"the check that makes step 3
|
||||
acceptable to ship"*. Without it, a node down for maintenance blocks every deployment to its cluster,
|
||||
with no remedy short of editing `NodeCount` / `RedundancyMode` — which changes the runtime redundancy
|
||||
posture to work around a deploy gate.
|
||||
|
||||
The two rules were each reasonable and contradictory together: the validator reads `Enabled` as *"part
|
||||
of the declared topology"*, Phase 1 additionally read it as *"expect an ack"*.
|
||||
|
||||
### Fix (Task 7): split the meanings
|
||||
|
||||
New `ClusterNode.MaintenanceMode bit NOT NULL DEFAULT 0`. `Enabled` keeps its topology meaning and the
|
||||
validator is untouched; the coordinator's expected-ack set and the address reconciler both filter on
|
||||
`Enabled && !MaintenanceMode`.
|
||||
|
||||
### Re-run: PASS
|
||||
|
||||
`docker stop` `site-b-2` + `MaintenanceMode = 1`, then deploy `82c5e678`:
|
||||
|
||||
- `202 Accepted` — the topology gate passes, because the node is still `Enabled` (`1|1`).
|
||||
- `Deployment.Status = 2` (Sealed).
|
||||
- **Five** `NodeDeploymentState` rows, all `Applied`. `site-b-2:4053` has no row at all — excluded
|
||||
from the expected set rather than waited for and forgiven.
|
||||
|
||||
## Task 4 (reconciler) — live behaviour, not a planned gate step
|
||||
|
||||
Observed unprompted during the rolling restart, which is better evidence than a staged check:
|
||||
|
||||
```
|
||||
[12:44:49 WRN] ClusterNode address check [EnabledRowNotInCluster] site-a-1:4053: ...
|
||||
[12:44:49 WRN] ... site-a-2:4053 ... site-b-1:4053 ... site-b-2:4053
|
||||
[12:44:56 INF] ClusterNode addresses reconcile with cluster membership (6 driver members)
|
||||
```
|
||||
|
||||
It warned per row while the site nodes were still starting, then converged to a single clean line —
|
||||
and **logged nothing further**, confirming the change-detection guard suppresses repeats on the
|
||||
5-minute sweep.
|
||||
|
||||
Maintenance interaction, from the step 4 re-run:
|
||||
|
||||
```
|
||||
[13:06:02 WRN] ... site-b-2:4053 ... ← node dropped, flag not yet set
|
||||
[13:06:05 INF] ... reconcile with cluster membership (5 driver members) ← flag set: silent
|
||||
```
|
||||
|
||||
The transient warning is honest — for those three seconds the row genuinely was enabled, present in
|
||||
the expected-ack set, and absent from the cluster.
|
||||
|
||||
## What this gate did not cover
|
||||
|
||||
- **Phase 2's premise.** `AkkaPort` / `GrpcPort` are groundwork; nothing dials them yet, so "the value
|
||||
is correct" is only checked by the reconciler comparing it to gossip. The first real exercise is
|
||||
Phase 2.
|
||||
- **The reconciler under split meshes.** On the current single mesh an admin node sees every driver
|
||||
member. After Phase 6 it will not, and every site row would report `EnabledRowNotInCluster`
|
||||
permanently. Recorded as a known limitation on the class and in the design doc.
|
||||
- **`Enabled = 0` on a single-node (`RedundancyMode.None`) cluster.** Untested — the rig has no such
|
||||
cluster. It would fail the same topology rule (0 enabled vs `NodeCount = 1`), so `MaintenanceMode`
|
||||
is the hatch there too.
|
||||
@@ -154,6 +154,7 @@ CREATE TABLE dbo.ClusterNode (
|
||||
ServiceLevelBase tinyint NOT NULL DEFAULT 200,
|
||||
DriverConfigOverridesJson nvarchar(max) NULL CHECK (DriverConfigOverridesJson IS NULL OR ISJSON(DriverConfigOverridesJson) = 1),
|
||||
Enabled bit NOT NULL DEFAULT 1,
|
||||
MaintenanceMode bit NOT NULL DEFAULT 0,
|
||||
LastSeenAt datetime2(3) NULL,
|
||||
CreatedAt datetime2(3) NOT NULL DEFAULT SYSUTCDATETIME(),
|
||||
CreatedBy nvarchar(128) NOT NULL
|
||||
@@ -190,10 +191,27 @@ rows against observed cluster membership and logs an Error on mismatch. It reads
|
||||
than having each driver node assert its own row, because Phase 4 removes the driver nodes' ConfigDb
|
||||
connection entirely.
|
||||
|
||||
`Enabled` also gained a second meaning in Phase 1: it is the **expected-ack set** for a deployment.
|
||||
`ConfigPublishCoordinator` no longer derives that set from cluster membership, so an enabled row whose
|
||||
node is down now fails the deployment at the apply deadline instead of letting it seal green without
|
||||
that node. Set `Enabled = 0` for a node taken down for maintenance.
|
||||
#### `Enabled` vs `MaintenanceMode` — two flags, two questions
|
||||
|
||||
Phase 1 made the rows the deploy path's **expected-ack set**: `ConfigPublishCoordinator` no longer
|
||||
derives it from cluster membership, so a row whose node is down now fails the deployment at the apply
|
||||
deadline instead of letting it seal green without that node.
|
||||
|
||||
The escape hatch is **`MaintenanceMode = 1`**, not `Enabled = 0`:
|
||||
|
||||
| Flag | Question it answers | Checked by |
|
||||
|---|---|---|
|
||||
| `Enabled` | Is this node part of the cluster's declared topology? | `DraftValidator.ValidateClusterTopology` — the enabled-node count must equal `ServerCluster.NodeCount` |
|
||||
| `MaintenanceMode` | Should we expect it to participate right now? | `ConfigPublishCoordinator` (expected-ack set) and `ClusterNodeAddressReconciler` (absence warnings) |
|
||||
|
||||
`Enabled = 0` on one node of a Warm/Hot pair is **rejected at the deploy gate** with
|
||||
`ClusterEnabledNodeCountMismatch` — the enabled count drops to 1 while `NodeCount` stays 2. Since
|
||||
every cluster in the target topology is a 2-node pair, `Enabled` cannot serve as the maintenance
|
||||
hatch at all. The Phase 1 live gate found this; the two rules were independently reasonable and
|
||||
contradictory together, so the meanings were split rather than either rule being weakened.
|
||||
|
||||
A node in maintenance is still enabled, so the topology gate still passes, and the runtime redundancy
|
||||
posture (`NodeCount` / `RedundancyMode`) is unchanged.
|
||||
|
||||
`DriverConfigOverridesJson` shape:
|
||||
|
||||
|
||||
Reference in New Issue
Block a user