feat(config): add ClusterNode.MaintenanceMode — the hatch the live gate proved missing

Phase 1 gate step 4 failed: setting Enabled = 0 to take a node out of service
returns 422 ClusterEnabledNodeCountMismatch, because
DraftValidator.ValidateClusterTopology requires the enabled-node count to equal
ServerCluster.NodeCount. On a Warm/Hot pair — every cluster on the rig, and
every cluster in the target topology — Enabled can therefore never be the
maintenance hatch. The plan called step 4 "the check that makes step 3
acceptable to ship", so Task 3's behaviour change was not shippable as it stood:
a node down for maintenance would block every deployment to its cluster.

The two rules were each reasonable and contradictory together — the validator
reads Enabled as "part of the declared topology", Phase 1 additionally read it
as "expect an ack". Split the meanings rather than weaken either rule:

  Enabled          part of the declared topology  (validator, untouched)
  MaintenanceMode  expected to participate now    (coordinator + reconciler)

Rejected alternatives: counting configured rather than enabled nodes (drops the
guard against booting a pair into InvalidTopology); downgrading the rule to a
warning (weakens a deploy gate for everyone); shipping with no hatch.

Sabotage: dropping !n.MaintenanceMode from the coordinator query turns the new
test red. It also asserts both nodes remain Enabled, so the fix cannot quietly
regress to disabling the row after all.

Configuration.Tests 95/95, ControlPlane.Tests 101/101, solution builds clean.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
2026-07-22 09:07:29 -04:00
parent 57e1d01766
commit ee69caa270
12 changed files with 2039 additions and 32 deletions
@@ -12,8 +12,17 @@ sourcing its expected-ack set from those rows.
**Architecture:** Additive schema change plus one substitution inside the coordinator. Nothing else
moves; the deploy channel stays on DistributedPubSub until Phase 2.
**Status:** NOT STARTED. Read §"The decision this phase forces" before executing — it changes
deploy-seal semantics and should be confirmed, not assumed.
**Status: DONE 2026-07-22** (branch `feat/mesh-phase1`). Live gate:
[`2026-07-22-mesh-phase1-live-gate.md`](2026-07-22-mesh-phase1-live-gate.md) — **PASSED**, after gate
step 4 found that the escape hatch this phase depends on did not exist. Three deviations from the
plan below, all decided before implementing or forced by the gate:
1. **Task 3's cluster-scope filtering was dropped** — there is no cluster-scoped deployment to filter
on. See the gate doc, step 2.
2. **No per-node role column.** Every `ClusterNode` row is now *defined* to be a driver node.
3. **Task 7 added:** `ClusterNode.MaintenanceMode`. `Enabled = 0` is rejected by
`DraftValidator.ValidateClusterTopology` on any 2-node pair, so it could never be the maintenance
hatch — which the plan called "the check that makes step 3 acceptable to ship".
---
@@ -54,7 +63,7 @@ But it is a behaviour change, and it needs an operator escape hatch or a node ta
maintenance blocks every deployment. `ClusterNode.Enabled` already exists and is exactly that hatch,
so the query filters on it: **`Enabled = 1` rows are expected to ack; disabled rows are not.**
**Confirm this before executing Task 3.** If the preference is to keep sealing green past a down
**Confirmed 2026-07-22 before executing Task 3** — DB-derived with the hatch. (The hatch turned out to be `MaintenanceMode`, not `Enabled`; see Task 7.) If the preference is to keep sealing green past a down
node, the alternative is to intersect the DB set with current membership, which preserves today's
behaviour but does not survive the mesh split — it would have to be revisited in Phase 2 anyway.
@@ -189,3 +198,25 @@ disagree on the rig. Run on `docker-dev`:
hatch, and is the check that makes step 3 acceptable to ship.
Record results in a gate doc beside this plan, per the Phase 1/Phase 2 precedent.
## Task 7: `MaintenanceMode` — the escape hatch step 4 proved missing (ADDED 2026-07-22)
**Classification:** high-risk — deploy path + schema
**Why:** gate step 4 returned `422 ClusterEnabledNodeCountMismatch`. `Enabled = 0` on one node of a
Warm/Hot pair fails `DraftValidator.ValidateClusterTopology` (enabled count must equal `NodeCount`),
so the hatch Task 3 relies on cannot be used on any 2-node cluster — i.e. every cluster in the target
topology. Task 3's behaviour change is only shippable with a hatch that works.
**Decision:** split the meanings rather than weaken either rule.
| Flag | Question | Enforced by |
|---|---|---|
| `Enabled` | part of the declared topology? | `DraftValidator` vs `ServerCluster.NodeCount` |
| `MaintenanceMode` | expected to participate right now? | `ConfigPublishCoordinator`, `ClusterNodeAddressReconciler` |
Rejected: counting configured rather than enabled nodes (removes the InvalidTopology guard);
downgrading the rule to a warning (weakens a deploy gate for everyone); shipping without a hatch.
**Done:** column + migration `AddClusterNodeMaintenanceMode`; both consumers filter on
`Enabled && !MaintenanceMode`; deadline + reconciler messages name the right flag; docs; gate step 4
re-run green.