feat(config): add ClusterNode.MaintenanceMode — the hatch the live gate proved missing

Phase 1 gate step 4 failed: setting Enabled = 0 to take a node out of service
returns 422 ClusterEnabledNodeCountMismatch, because
DraftValidator.ValidateClusterTopology requires the enabled-node count to equal
ServerCluster.NodeCount. On a Warm/Hot pair — every cluster on the rig, and
every cluster in the target topology — Enabled can therefore never be the
maintenance hatch. The plan called step 4 "the check that makes step 3
acceptable to ship", so Task 3's behaviour change was not shippable as it stood:
a node down for maintenance would block every deployment to its cluster.

The two rules were each reasonable and contradictory together — the validator
reads Enabled as "part of the declared topology", Phase 1 additionally read it
as "expect an ack". Split the meanings rather than weaken either rule:

  Enabled          part of the declared topology  (validator, untouched)
  MaintenanceMode  expected to participate now    (coordinator + reconciler)

Rejected alternatives: counting configured rather than enabled nodes (drops the
guard against booting a pair into InvalidTopology); downgrading the rule to a
warning (weakens a deploy gate for everyone); shipping with no hatch.

Sabotage: dropping !n.MaintenanceMode from the coordinator query turns the new
test red. It also asserts both nodes remain Enabled, so the fix cannot quietly
regress to disabling the row after all.

Configuration.Tests 95/95, ControlPlane.Tests 101/101, solution builds clean.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
2026-07-22 09:07:29 -04:00
parent 57e1d01766
commit ee69caa270
12 changed files with 2039 additions and 32 deletions
@@ -63,8 +63,37 @@ public sealed class ClusterNode
public string? DriverConfigOverridesJson { get; set; }
/// <summary>Gets or sets a value indicating whether this node is enabled.</summary>
/// <remarks>
/// <b>Part of the declared topology.</b> <c>DraftValidator.ValidateClusterTopology</c> requires
/// the enabled-node count to equal <see cref="ServerCluster.NodeCount"/>, so disabling one node
/// of a Warm/Hot pair is a deploy-blocking validation error by design — it would boot the
/// runtime into the InvalidTopology band. To take a node out of service temporarily, use
/// <see cref="MaintenanceMode"/> instead.
/// </remarks>
public bool Enabled { get; set; } = true;
/// <summary>
/// Node is temporarily out of service: still part of the cluster's declared topology, but not
/// expected to participate. A deployment does not wait for its ack, and the address reconciler
/// does not warn that it is absent.
/// </summary>
/// <remarks>
/// <para>
/// Separate from <see cref="Enabled"/> because the two answer different questions.
/// <see cref="Enabled"/> means "this node is part of the declared topology" and is checked
/// against <see cref="ServerCluster.NodeCount"/>; this flag means "do not expect it right
/// now". Per-cluster mesh Phase 1 made an enabled row's ack mandatory, so without this a
/// node down for maintenance would block every deployment to its cluster — and clearing
/// <see cref="Enabled"/> to escape that is itself rejected by the topology gate on any
/// 2-node pair, which is every cluster in the target deployment.
/// </para>
/// <para>
/// Found by the Phase 1 live gate: the two rules were independently reasonable and
/// contradictory together.
/// </para>
/// </remarks>
public bool MaintenanceMode { get; set; }
/// <summary>Gets or sets the timestamp when this node was last seen.</summary>
public DateTime? LastSeenAt { get; set; }