feat(config): add ClusterNode.MaintenanceMode — the hatch the live gate proved missing
Phase 1 gate step 4 failed: setting Enabled = 0 to take a node out of service returns 422 ClusterEnabledNodeCountMismatch, because DraftValidator.ValidateClusterTopology requires the enabled-node count to equal ServerCluster.NodeCount. On a Warm/Hot pair — every cluster on the rig, and every cluster in the target topology — Enabled can therefore never be the maintenance hatch. The plan called step 4 "the check that makes step 3 acceptable to ship", so Task 3's behaviour change was not shippable as it stood: a node down for maintenance would block every deployment to its cluster. The two rules were each reasonable and contradictory together — the validator reads Enabled as "part of the declared topology", Phase 1 additionally read it as "expect an ack". Split the meanings rather than weaken either rule: Enabled part of the declared topology (validator, untouched) MaintenanceMode expected to participate now (coordinator + reconciler) Rejected alternatives: counting configured rather than enabled nodes (drops the guard against booting a pair into InvalidTopology); downgrading the rule to a warning (weakens a deploy gate for everyone); shipping with no hatch. Sabotage: dropping !n.MaintenanceMode from the coordinator query turns the new test red. It also asserts both nodes remain Enabled, so the fix cannot quietly regress to disabling the row after all. Configuration.Tests 95/95, ControlPlane.Tests 101/101, solution builds clean. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
@@ -12,8 +12,17 @@ sourcing its expected-ack set from those rows.
|
||||
**Architecture:** Additive schema change plus one substitution inside the coordinator. Nothing else
|
||||
moves; the deploy channel stays on DistributedPubSub until Phase 2.
|
||||
|
||||
**Status:** NOT STARTED. Read §"The decision this phase forces" before executing — it changes
|
||||
deploy-seal semantics and should be confirmed, not assumed.
|
||||
**Status: DONE 2026-07-22** (branch `feat/mesh-phase1`). Live gate:
|
||||
[`2026-07-22-mesh-phase1-live-gate.md`](2026-07-22-mesh-phase1-live-gate.md) — **PASSED**, after gate
|
||||
step 4 found that the escape hatch this phase depends on did not exist. Three deviations from the
|
||||
plan below, all decided before implementing or forced by the gate:
|
||||
|
||||
1. **Task 3's cluster-scope filtering was dropped** — there is no cluster-scoped deployment to filter
|
||||
on. See the gate doc, step 2.
|
||||
2. **No per-node role column.** Every `ClusterNode` row is now *defined* to be a driver node.
|
||||
3. **Task 7 added:** `ClusterNode.MaintenanceMode`. `Enabled = 0` is rejected by
|
||||
`DraftValidator.ValidateClusterTopology` on any 2-node pair, so it could never be the maintenance
|
||||
hatch — which the plan called "the check that makes step 3 acceptable to ship".
|
||||
|
||||
---
|
||||
|
||||
@@ -54,7 +63,7 @@ But it is a behaviour change, and it needs an operator escape hatch or a node ta
|
||||
maintenance blocks every deployment. `ClusterNode.Enabled` already exists and is exactly that hatch,
|
||||
so the query filters on it: **`Enabled = 1` rows are expected to ack; disabled rows are not.**
|
||||
|
||||
**Confirm this before executing Task 3.** If the preference is to keep sealing green past a down
|
||||
**Confirmed 2026-07-22 before executing Task 3** — DB-derived with the hatch. (The hatch turned out to be `MaintenanceMode`, not `Enabled`; see Task 7.) If the preference is to keep sealing green past a down
|
||||
node, the alternative is to intersect the DB set with current membership, which preserves today's
|
||||
behaviour but does not survive the mesh split — it would have to be revisited in Phase 2 anyway.
|
||||
|
||||
@@ -189,3 +198,25 @@ disagree on the rig. Run on `docker-dev`:
|
||||
hatch, and is the check that makes step 3 acceptable to ship.
|
||||
|
||||
Record results in a gate doc beside this plan, per the Phase 1/Phase 2 precedent.
|
||||
|
||||
## Task 7: `MaintenanceMode` — the escape hatch step 4 proved missing (ADDED 2026-07-22)
|
||||
|
||||
**Classification:** high-risk — deploy path + schema
|
||||
**Why:** gate step 4 returned `422 ClusterEnabledNodeCountMismatch`. `Enabled = 0` on one node of a
|
||||
Warm/Hot pair fails `DraftValidator.ValidateClusterTopology` (enabled count must equal `NodeCount`),
|
||||
so the hatch Task 3 relies on cannot be used on any 2-node cluster — i.e. every cluster in the target
|
||||
topology. Task 3's behaviour change is only shippable with a hatch that works.
|
||||
|
||||
**Decision:** split the meanings rather than weaken either rule.
|
||||
|
||||
| Flag | Question | Enforced by |
|
||||
|---|---|---|
|
||||
| `Enabled` | part of the declared topology? | `DraftValidator` vs `ServerCluster.NodeCount` |
|
||||
| `MaintenanceMode` | expected to participate right now? | `ConfigPublishCoordinator`, `ClusterNodeAddressReconciler` |
|
||||
|
||||
Rejected: counting configured rather than enabled nodes (removes the InvalidTopology guard);
|
||||
downgrading the rule to a warning (weakens a deploy gate for everyone); shipping without a hatch.
|
||||
|
||||
**Done:** column + migration `AddClusterNodeMaintenanceMode`; both consumers filter on
|
||||
`Enabled && !MaintenanceMode`; deadline + reconciler messages name the right flag; docs; gate step 4
|
||||
re-run green.
|
||||
|
||||
Reference in New Issue
Block a user