feat(config): add ClusterNode.MaintenanceMode — the hatch the live gate proved missing

Phase 1 gate step 4 failed: setting Enabled = 0 to take a node out of service
returns 422 ClusterEnabledNodeCountMismatch, because
DraftValidator.ValidateClusterTopology requires the enabled-node count to equal
ServerCluster.NodeCount. On a Warm/Hot pair — every cluster on the rig, and
every cluster in the target topology — Enabled can therefore never be the
maintenance hatch. The plan called step 4 "the check that makes step 3
acceptable to ship", so Task 3's behaviour change was not shippable as it stood:
a node down for maintenance would block every deployment to its cluster.

The two rules were each reasonable and contradictory together — the validator
reads Enabled as "part of the declared topology", Phase 1 additionally read it
as "expect an ack". Split the meanings rather than weaken either rule:

  Enabled          part of the declared topology  (validator, untouched)
  MaintenanceMode  expected to participate now    (coordinator + reconciler)

Rejected alternatives: counting configured rather than enabled nodes (drops the
guard against booting a pair into InvalidTopology); downgrading the rule to a
warning (weakens a deploy gate for everyone); shipping with no hatch.

Sabotage: dropping !n.MaintenanceMode from the coordinator query turns the new
test red. It also asserts both nodes remain Enabled, so the fix cannot quietly
regress to disabling the row after all.

Configuration.Tests 95/95, ControlPlane.Tests 101/101, solution builds clean.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
2026-07-22 09:07:29 -04:00
parent 57e1d01766
commit ee69caa270
12 changed files with 2039 additions and 32 deletions
+1 -1
View File
@@ -232,7 +232,7 @@ Assert the **effective** cluster config off a running `ActorSystem` (as `SplitBr
**Phase 1 landed 2026-07-22 and changed a deploy-path behaviour.** `ConfigPublishCoordinator` sources its expected-ack set from **enabled `ClusterNode` rows**, not from `Akka.Cluster.State.Members` filtered by the `driver` role — central must be able to name a deployment's nodes without sharing a gossip ring with them. Consequences worth knowing before touching deploys: **Phase 1 landed 2026-07-22 and changed a deploy-path behaviour.** `ConfigPublishCoordinator` sources its expected-ack set from **enabled `ClusterNode` rows**, not from `Akka.Cluster.State.Members` filtered by the `driver` role — central must be able to name a deployment's nodes without sharing a gossip ring with them. Consequences worth knowing before touching deploys:
- **A configured node that is switched off now FAILS the deployment** at the apply deadline instead of letting it seal green without that node. `ClusterNode.Enabled = 0` is the maintenance hatch; the deadline log names the silent nodes. - **A configured node that is switched off now FAILS the deployment** at the apply deadline instead of letting it seal green without that node. The maintenance hatch is the **new `ClusterNode.MaintenanceMode` column**, NOT `Enabled = 0`: the live gate showed `Enabled = 0` is rejected outright by `DraftValidator.ValidateClusterTopology` (`ClusterEnabledNodeCountMismatch` — enabled count must equal `ServerCluster.NodeCount`), so on a 2-node pair — every cluster in the target topology — it can never be the hatch. `Enabled` = part of the declared topology; `MaintenanceMode` = do not expect it now. The deadline log names the silent nodes and points at the right flag.
- **Every `ClusterNode` row is now *defined* to be a driver node.** The DB has no per-node role column and deliberately did not gain one (it would drift from `Cluster:Roles`). Giving an admin-only node a `ClusterNode` row makes every deploy time out. - **Every `ClusterNode` row is now *defined* to be a driver node.** The DB has no per-node role column and deliberately did not gain one (it would drift from `Cluster:Roles`). Giving an admin-only node a `ClusterNode` row makes every deploy time out.
- **There is no such thing as a cluster-scoped deployment.** `Deployment` has no `ClusterId`, `ConfigComposer` always snapshots the whole DB, and `DeploymentArtifact.ResolveClusterScope` is *node-side* self-scoping of a fleet-wide artifact. The phase-1 plan asked for cluster-scope filtering of the ack set; it was dropped as describing a feature that does not exist. - **There is no such thing as a cluster-scoped deployment.** `Deployment` has no `ClusterId`, `ConfigComposer` always snapshots the whole DB, and `DeploymentArtifact.ResolveClusterScope` is *node-side* self-scoping of a fleet-wide artifact. The phase-1 plan asked for cluster-scope filtering of the ack set; it was dropped as describing a feature that does not exist.
- `ClusterNode` gained `AkkaPort` (NOT NULL, 4053) + `GrpcPort` (nullable) as **central's dial targets** for Phases 2/5. They duplicate the node's own `Cluster:Port` / `Cluster:PublicHostname` with no schema-level enforcement, so `ClusterNodeAddressReconcilerActor` (admin singleton) logs an Error on drift. **Phase 2 must revisit it** — once the meshes split, an admin node cannot see site members and every site row would report `EnabledRowNotInCluster` forever. - `ClusterNode` gained `AkkaPort` (NOT NULL, 4053) + `GrpcPort` (nullable) as **central's dial targets** for Phases 2/5. They duplicate the node's own `Cluster:Port` / `Cluster:PublicHostname` with no schema-level enforcement, so `ClusterNodeAddressReconcilerActor` (admin singleton) logs an Error on drift. **Phase 2 must revisit it** — once the meshes split, an admin node cannot see site members and every site row would report `EnabledRowNotInCluster` forever.
@@ -12,8 +12,17 @@ sourcing its expected-ack set from those rows.
**Architecture:** Additive schema change plus one substitution inside the coordinator. Nothing else **Architecture:** Additive schema change plus one substitution inside the coordinator. Nothing else
moves; the deploy channel stays on DistributedPubSub until Phase 2. moves; the deploy channel stays on DistributedPubSub until Phase 2.
**Status:** NOT STARTED. Read §"The decision this phase forces" before executing — it changes **Status: DONE 2026-07-22** (branch `feat/mesh-phase1`). Live gate:
deploy-seal semantics and should be confirmed, not assumed. [`2026-07-22-mesh-phase1-live-gate.md`](2026-07-22-mesh-phase1-live-gate.md) — **PASSED**, after gate
step 4 found that the escape hatch this phase depends on did not exist. Three deviations from the
plan below, all decided before implementing or forced by the gate:
1. **Task 3's cluster-scope filtering was dropped** — there is no cluster-scoped deployment to filter
on. See the gate doc, step 2.
2. **No per-node role column.** Every `ClusterNode` row is now *defined* to be a driver node.
3. **Task 7 added:** `ClusterNode.MaintenanceMode`. `Enabled = 0` is rejected by
`DraftValidator.ValidateClusterTopology` on any 2-node pair, so it could never be the maintenance
hatch — which the plan called "the check that makes step 3 acceptable to ship".
--- ---
@@ -54,7 +63,7 @@ But it is a behaviour change, and it needs an operator escape hatch or a node ta
maintenance blocks every deployment. `ClusterNode.Enabled` already exists and is exactly that hatch, maintenance blocks every deployment. `ClusterNode.Enabled` already exists and is exactly that hatch,
so the query filters on it: **`Enabled = 1` rows are expected to ack; disabled rows are not.** so the query filters on it: **`Enabled = 1` rows are expected to ack; disabled rows are not.**
**Confirm this before executing Task 3.** If the preference is to keep sealing green past a down **Confirmed 2026-07-22 before executing Task 3** — DB-derived with the hatch. (The hatch turned out to be `MaintenanceMode`, not `Enabled`; see Task 7.) If the preference is to keep sealing green past a down
node, the alternative is to intersect the DB set with current membership, which preserves today's node, the alternative is to intersect the DB set with current membership, which preserves today's
behaviour but does not survive the mesh split — it would have to be revisited in Phase 2 anyway. behaviour but does not survive the mesh split — it would have to be revisited in Phase 2 anyway.
@@ -189,3 +198,25 @@ disagree on the rig. Run on `docker-dev`:
hatch, and is the check that makes step 3 acceptable to ship. hatch, and is the check that makes step 3 acceptable to ship.
Record results in a gate doc beside this plan, per the Phase 1/Phase 2 precedent. Record results in a gate doc beside this plan, per the Phase 1/Phase 2 precedent.
## Task 7: `MaintenanceMode` — the escape hatch step 4 proved missing (ADDED 2026-07-22)
**Classification:** high-risk — deploy path + schema
**Why:** gate step 4 returned `422 ClusterEnabledNodeCountMismatch`. `Enabled = 0` on one node of a
Warm/Hot pair fails `DraftValidator.ValidateClusterTopology` (enabled count must equal `NodeCount`),
so the hatch Task 3 relies on cannot be used on any 2-node cluster — i.e. every cluster in the target
topology. Task 3's behaviour change is only shippable with a hatch that works.
**Decision:** split the meanings rather than weaken either rule.
| Flag | Question | Enforced by |
|---|---|---|
| `Enabled` | part of the declared topology? | `DraftValidator` vs `ServerCluster.NodeCount` |
| `MaintenanceMode` | expected to participate right now? | `ConfigPublishCoordinator`, `ClusterNodeAddressReconciler` |
Rejected: counting configured rather than enabled nodes (removes the InvalidTopology guard);
downgrading the rule to a warning (weakens a deploy gate for everyone); shipping without a hatch.
**Done:** column + migration `AddClusterNodeMaintenanceMode`; both consumers filter on
`Enabled && !MaintenanceMode`; deadline + reconciler messages name the right flag; docs; gate step 4
re-run green.
@@ -0,0 +1,134 @@
# Per-cluster mesh Phase 1 — live gate record
**Date:** 2026-07-22 · **Rig:** `docker-dev` (six-node single mesh) · **Branch:** `feat/mesh-phase1`
**Plan:** [`2026-07-21-per-cluster-mesh-phase1.md`](2026-07-21-per-cluster-mesh-phase1.md)
**Result: PASSED**, after the gate found a blocker that changed the design (step 4).
Offline tests cannot cover the substitution's real risk — that the DB set and the live set disagree on
a running fleet. This is that check.
## Setup
Migration applied by the rig's `migrator` service against the shared `OtOpcUa` database, then all six
host nodes restarted on the rebuilt image. **All six pre-existing `ClusterNode` rows inherited
`AkkaPort = 4053` from the migration's `DEFAULT 4053`** — the row-level evidence that the default does
its job for rows predating the column:
```
central-1:4053|central-1|4053|NULL|1
central-2:4053|central-2|4053|NULL|1
site-a-1:4053 |site-a-1 |4053|NULL|1
site-a-2:4053 |site-a-2 |4053|NULL|1
site-b-1:4053 |site-b-1 |4053|NULL|1
site-b-2:4053 |site-b-2 |4053|NULL|1
NodeId | Host |Akka|Grpc|En
```
## Step 1 — deploy with all six nodes up → **PASS**
`POST /api/deployments``202 Accepted`, deployment `38f83b37`.
`Deployment.Status = 2` (Sealed). Six `NodeDeploymentState` rows, every one `Status = 1` (Applied).
**No FK 547** — the DB-derived NodeIds satisfy the `NodeDeploymentState → ClusterNode` foreign key,
which is the standing proof that the DB set and the membership set agree on this rig.
## Step 2 — *dropped, not skipped*
The plan called for a cluster-scoped deploy. **There is no such thing.** `Deployment` has no
`ClusterId`, `ConfigComposer.SnapshotAndFlattenAsync` always snapshots the whole DB, and
`DeploymentArtifact.ResolveClusterScope` is *node-side* self-scoping of a fleet-wide artifact. Removed
from the plan rather than faked green.
## Step 3 — deploy with one node stopped → **PASS** (the new behaviour)
`docker stop otopcua-dev-site-b-2-1`, then deploy `37d72301`.
`Deployment.Status = 4` (TimedOut) after the 2-minute apply deadline. Five nodes `Applied`,
`site-b-2:4053` still `Applying`. Under the membership rule this deployment would have **sealed
green** with five nodes and never mentioned the sixth.
The operator-facing log, which the plan required to be legible:
```
[12:52:02 WRN] Deployment 37d723018e374c0eb04c7b6beccc3b3e timed out after 00:02:00 (5/6 acks
landed). No ack from: site-b-2:4053. Each is an enabled ClusterNode row — start the node, or set
ClusterNode.Enabled = 0 if it is down for maintenance, then redeploy
```
(The remedy in that message was corrected to `MaintenanceMode = 1` — see step 4.)
## Step 4 — the escape hatch → **FAILED, then fixed and re-run → PASS**
### First attempt: the hatch did not exist
`UPDATE ClusterNode SET Enabled = 0 WHERE NodeId = 'site-b-2:4053'`, then deploy:
```
HTTP 422
[ClusterEnabledNodeCountMismatch] Cluster 'SITE-B' declares NodeCount=2 but has 1 Enabled nodes.
Toggle the missing node(s) back on or change RedundancyMode/NodeCount to match.
```
`DraftValidator.ValidateClusterTopology` rejects the deployment **before it dispatches**. So
`Enabled = 0` cannot be used on a node of a 2-node cluster — and every cluster on this rig, and every
cluster in the program plan's target topology, is a 2-node pair.
**This was a blocker, not a wrinkle.** The plan called step 4 *"the check that makes step 3
acceptable to ship"*. Without it, a node down for maintenance blocks every deployment to its cluster,
with no remedy short of editing `NodeCount` / `RedundancyMode` — which changes the runtime redundancy
posture to work around a deploy gate.
The two rules were each reasonable and contradictory together: the validator reads `Enabled` as *"part
of the declared topology"*, Phase 1 additionally read it as *"expect an ack"*.
### Fix (Task 7): split the meanings
New `ClusterNode.MaintenanceMode bit NOT NULL DEFAULT 0`. `Enabled` keeps its topology meaning and the
validator is untouched; the coordinator's expected-ack set and the address reconciler both filter on
`Enabled && !MaintenanceMode`.
### Re-run: PASS
`docker stop` `site-b-2` + `MaintenanceMode = 1`, then deploy `82c5e678`:
- `202 Accepted` — the topology gate passes, because the node is still `Enabled` (`1|1`).
- `Deployment.Status = 2` (Sealed).
- **Five** `NodeDeploymentState` rows, all `Applied`. `site-b-2:4053` has no row at all — excluded
from the expected set rather than waited for and forgiven.
## Task 4 (reconciler) — live behaviour, not a planned gate step
Observed unprompted during the rolling restart, which is better evidence than a staged check:
```
[12:44:49 WRN] ClusterNode address check [EnabledRowNotInCluster] site-a-1:4053: ...
[12:44:49 WRN] ... site-a-2:4053 ... site-b-1:4053 ... site-b-2:4053
[12:44:56 INF] ClusterNode addresses reconcile with cluster membership (6 driver members)
```
It warned per row while the site nodes were still starting, then converged to a single clean line —
and **logged nothing further**, confirming the change-detection guard suppresses repeats on the
5-minute sweep.
Maintenance interaction, from the step 4 re-run:
```
[13:06:02 WRN] ... site-b-2:4053 ... ← node dropped, flag not yet set
[13:06:05 INF] ... reconcile with cluster membership (5 driver members) ← flag set: silent
```
The transient warning is honest — for those three seconds the row genuinely was enabled, present in
the expected-ack set, and absent from the cluster.
## What this gate did not cover
- **Phase 2's premise.** `AkkaPort` / `GrpcPort` are groundwork; nothing dials them yet, so "the value
is correct" is only checked by the reconciler comparing it to gossip. The first real exercise is
Phase 2.
- **The reconciler under split meshes.** On the current single mesh an admin node sees every driver
member. After Phase 6 it will not, and every site row would report `EnabledRowNotInCluster`
permanently. Recorded as a known limitation on the class and in the design doc.
- **`Enabled = 0` on a single-node (`RedundancyMode.None`) cluster.** Untested — the rig has no such
cluster. It would fail the same topology rule (0 enabled vs `NodeCount = 1`), so `MaintenanceMode`
is the hatch there too.
+22 -4
View File
@@ -154,6 +154,7 @@ CREATE TABLE dbo.ClusterNode (
ServiceLevelBase tinyint NOT NULL DEFAULT 200, ServiceLevelBase tinyint NOT NULL DEFAULT 200,
DriverConfigOverridesJson nvarchar(max) NULL CHECK (DriverConfigOverridesJson IS NULL OR ISJSON(DriverConfigOverridesJson) = 1), DriverConfigOverridesJson nvarchar(max) NULL CHECK (DriverConfigOverridesJson IS NULL OR ISJSON(DriverConfigOverridesJson) = 1),
Enabled bit NOT NULL DEFAULT 1, Enabled bit NOT NULL DEFAULT 1,
MaintenanceMode bit NOT NULL DEFAULT 0,
LastSeenAt datetime2(3) NULL, LastSeenAt datetime2(3) NULL,
CreatedAt datetime2(3) NOT NULL DEFAULT SYSUTCDATETIME(), CreatedAt datetime2(3) NOT NULL DEFAULT SYSUTCDATETIME(),
CreatedBy nvarchar(128) NOT NULL CreatedBy nvarchar(128) NOT NULL
@@ -190,10 +191,27 @@ rows against observed cluster membership and logs an Error on mismatch. It reads
than having each driver node assert its own row, because Phase 4 removes the driver nodes' ConfigDb than having each driver node assert its own row, because Phase 4 removes the driver nodes' ConfigDb
connection entirely. connection entirely.
`Enabled` also gained a second meaning in Phase 1: it is the **expected-ack set** for a deployment. #### `Enabled` vs `MaintenanceMode` — two flags, two questions
`ConfigPublishCoordinator` no longer derives that set from cluster membership, so an enabled row whose
node is down now fails the deployment at the apply deadline instead of letting it seal green without Phase 1 made the rows the deploy path's **expected-ack set**: `ConfigPublishCoordinator` no longer
that node. Set `Enabled = 0` for a node taken down for maintenance. derives it from cluster membership, so a row whose node is down now fails the deployment at the apply
deadline instead of letting it seal green without that node.
The escape hatch is **`MaintenanceMode = 1`**, not `Enabled = 0`:
| Flag | Question it answers | Checked by |
|---|---|---|
| `Enabled` | Is this node part of the cluster's declared topology? | `DraftValidator.ValidateClusterTopology` — the enabled-node count must equal `ServerCluster.NodeCount` |
| `MaintenanceMode` | Should we expect it to participate right now? | `ConfigPublishCoordinator` (expected-ack set) and `ClusterNodeAddressReconciler` (absence warnings) |
`Enabled = 0` on one node of a Warm/Hot pair is **rejected at the deploy gate** with
`ClusterEnabledNodeCountMismatch` — the enabled count drops to 1 while `NodeCount` stays 2. Since
every cluster in the target topology is a 2-node pair, `Enabled` cannot serve as the maintenance
hatch at all. The Phase 1 live gate found this; the two rules were independently reasonable and
contradictory together, so the meanings were split rather than either rule being weakened.
A node in maintenance is still enabled, so the topology gate still passes, and the runtime redundancy
posture (`NodeCount` / `RedundancyMode`) is unchanged.
`DriverConfigOverridesJson` shape: `DriverConfigOverridesJson` shape:
@@ -63,8 +63,37 @@ public sealed class ClusterNode
public string? DriverConfigOverridesJson { get; set; } public string? DriverConfigOverridesJson { get; set; }
/// <summary>Gets or sets a value indicating whether this node is enabled.</summary> /// <summary>Gets or sets a value indicating whether this node is enabled.</summary>
/// <remarks>
/// <b>Part of the declared topology.</b> <c>DraftValidator.ValidateClusterTopology</c> requires
/// the enabled-node count to equal <see cref="ServerCluster.NodeCount"/>, so disabling one node
/// of a Warm/Hot pair is a deploy-blocking validation error by design — it would boot the
/// runtime into the InvalidTopology band. To take a node out of service temporarily, use
/// <see cref="MaintenanceMode"/> instead.
/// </remarks>
public bool Enabled { get; set; } = true; public bool Enabled { get; set; } = true;
/// <summary>
/// Node is temporarily out of service: still part of the cluster's declared topology, but not
/// expected to participate. A deployment does not wait for its ack, and the address reconciler
/// does not warn that it is absent.
/// </summary>
/// <remarks>
/// <para>
/// Separate from <see cref="Enabled"/> because the two answer different questions.
/// <see cref="Enabled"/> means "this node is part of the declared topology" and is checked
/// against <see cref="ServerCluster.NodeCount"/>; this flag means "do not expect it right
/// now". Per-cluster mesh Phase 1 made an enabled row's ack mandatory, so without this a
/// node down for maintenance would block every deployment to its cluster — and clearing
/// <see cref="Enabled"/> to escape that is itself rejected by the topology gate on any
/// 2-node pair, which is every cluster in the target deployment.
/// </para>
/// <para>
/// Found by the Phase 1 live gate: the two rules were independently reasonable and
/// contradictory together.
/// </para>
/// </remarks>
public bool MaintenanceMode { get; set; }
/// <summary>Gets or sets the timestamp when this node was last seen.</summary> /// <summary>Gets or sets the timestamp when this node was last seen.</summary>
public DateTime? LastSeenAt { get; set; } public DateTime? LastSeenAt { get; set; }
@@ -0,0 +1,29 @@
using Microsoft.EntityFrameworkCore.Migrations;
#nullable disable
namespace ZB.MOM.WW.OtOpcUa.Configuration.Migrations
{
/// <inheritdoc />
public partial class AddClusterNodeMaintenanceMode : Migration
{
/// <inheritdoc />
protected override void Up(MigrationBuilder migrationBuilder)
{
migrationBuilder.AddColumn<bool>(
name: "MaintenanceMode",
table: "ClusterNode",
type: "bit",
nullable: false,
defaultValue: false);
}
/// <inheritdoc />
protected override void Down(MigrationBuilder migrationBuilder)
{
migrationBuilder.DropColumn(
name: "MaintenanceMode",
table: "ClusterNode");
}
}
}
@@ -92,6 +92,9 @@ namespace ZB.MOM.WW.OtOpcUa.Configuration.Migrations
b.Property<DateTime?>("LastSeenAt") b.Property<DateTime?>("LastSeenAt")
.HasColumnType("datetime2(3)"); .HasColumnType("datetime2(3)");
b.Property<bool>("MaintenanceMode")
.HasColumnType("bit");
b.Property<int>("OpcUaPort") b.Property<int>("OpcUaPort")
.HasColumnType("int"); .HasColumnType("int");
@@ -27,8 +27,16 @@ namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Coordinators;
/// <b>Behaviour change:</b> a configured node that is switched off is now <i>expected</i>, so a /// <b>Behaviour change:</b> a configured node that is switched off is now <i>expected</i>, so a
/// deployment dispatched while it is down fails at the apply deadline instead of sealing green /// deployment dispatched while it is down fails at the apply deadline instead of sealing green
/// without it. That is deliberate — under the membership rule the operator was told the fleet was /// without it. That is deliberate — under the membership rule the operator was told the fleet was
/// deployed when it was not. <c>ClusterNode.Enabled = 0</c> is the escape hatch for a node taken /// deployed when it was not. <c>ClusterNode.MaintenanceMode = 1</c> is the escape hatch for a node
/// down for maintenance. /// taken down for maintenance.
/// </para>
/// <para>
/// <b>Not <c>Enabled = 0</c>.</b> That was the intended hatch until the Phase 1 live gate found it
/// unusable: <c>DraftValidator.ValidateClusterTopology</c> requires the enabled-node count to equal
/// <c>ServerCluster.NodeCount</c>, so clearing <c>Enabled</c> on either node of a Warm/Hot pair
/// rejects the deployment outright — and every cluster in the target topology is a pair. The two
/// rules were independently reasonable and contradictory together, so the meanings were split:
/// <c>Enabled</c> = part of the declared topology, <c>MaintenanceMode</c> = do not expect it now.
/// </para> /// </para>
/// <para> /// <para>
/// <b>Every <c>ClusterNode</c> row is assumed to be a driver node.</b> The membership rule filtered /// <b>Every <c>ClusterNode</c> row is assumed to be a driver node.</b> The membership rule filtered
@@ -271,8 +279,8 @@ public sealed class ConfigPublishCoordinator : ReceiveActor, IWithTimers
var missing = _expectedAcks.Where(n => !_acks.ContainsKey(n)).Select(n => n.Value).Order().ToList(); var missing = _expectedAcks.Where(n => !_acks.ContainsKey(n)).Select(n => n.Value).Order().ToList();
_log.Warning( _log.Warning(
"Deployment {Id} timed out after {Deadline} ({Acked}/{Total} acks landed). " + "Deployment {Id} timed out after {Deadline} ({Acked}/{Total} acks landed). " +
"No ack from: {MissingNodes}. Each is an enabled ClusterNode row — start the node, or " + "No ack from: {MissingNodes}. Each is an enabled, non-maintenance ClusterNode row — start " +
"set ClusterNode.Enabled = 0 if it is down for maintenance, then redeploy", "the node, or set ClusterNode.MaintenanceMode = 1 if it is down for maintenance, then redeploy",
_current.Value, _applyDeadline, _acks.Count, _expectedAcks.Count, string.Join(", ", missing)); _current.Value, _applyDeadline, _acks.Count, _expectedAcks.Count, string.Join(", ", missing));
ResetForNext(); ResetForNext();
} }
@@ -316,7 +324,7 @@ public sealed class ConfigPublishCoordinator : ReceiveActor, IWithTimers
private static HashSet<NodeId> DiscoverDriverNodes(OtOpcUaConfigDbContext db) => private static HashSet<NodeId> DiscoverDriverNodes(OtOpcUaConfigDbContext db) =>
db.ClusterNodes db.ClusterNodes
.AsNoTracking() .AsNoTracking()
.Where(n => n.Enabled) .Where(n => n.Enabled && !n.MaintenanceMode)
.Select(n => n.NodeId) .Select(n => n.NodeId)
.ToList() .ToList()
.Select(NodeId.Parse) .Select(NodeId.Parse)
@@ -75,21 +75,22 @@ public static class ClusterNodeAddressReconciler
/// them would report a mismatch for correct configuration. /// them would report a mismatch for correct configuration.
/// </param> /// </param>
/// <param name="rows">Every <c>ClusterNode</c> row, enabled or not.</param> /// <param name="rows">Every <c>ClusterNode</c> row, enabled or not.</param>
/// <param name="enabledNodeIds"> /// <param name="expectedPresentNodeIds">
/// The subset of <paramref name="rows"/> that is enabled. Disabled rows are deliberately /// The subset of <paramref name="rows"/> that should currently be in the cluster: enabled and
/// still matched against membership (a disabled row for a running node is worth knowing) but /// not in maintenance. Excluded rows are deliberately still matched against membership (a
/// never reported as <see cref="AddressMismatchKind.EnabledRowNotInCluster"/> — being absent /// disabled row for a running node is worth knowing) but never reported as
/// is the whole point of disabling one. /// <see cref="AddressMismatchKind.EnabledRowNotInCluster"/> — being absent is the whole point
/// of disabling one or putting it in maintenance.
/// </param> /// </param>
/// <returns>Every disagreement found, ordered by node id for stable logging.</returns> /// <returns>Every disagreement found, ordered by node id for stable logging.</returns>
public static IReadOnlyList<ClusterNodeAddressMismatch> Reconcile( public static IReadOnlyList<ClusterNodeAddressMismatch> Reconcile(
IReadOnlyCollection<(string Host, int Port)> observedDriverMembers, IReadOnlyCollection<(string Host, int Port)> observedDriverMembers,
IReadOnlyCollection<ClusterNodeAddress> rows, IReadOnlyCollection<ClusterNodeAddress> rows,
IReadOnlyCollection<string> enabledNodeIds) IReadOnlyCollection<string> expectedPresentNodeIds)
{ {
var byNodeId = new Dictionary<string, ClusterNodeAddress>(StringComparer.OrdinalIgnoreCase); var byNodeId = new Dictionary<string, ClusterNodeAddress>(StringComparer.OrdinalIgnoreCase);
foreach (var row in rows) byNodeId[row.NodeId] = row; foreach (var row in rows) byNodeId[row.NodeId] = row;
var enabled = new HashSet<string>(enabledNodeIds, StringComparer.OrdinalIgnoreCase); var expectedPresent = new HashSet<string>(expectedPresentNodeIds, StringComparer.OrdinalIgnoreCase);
var findings = new List<ClusterNodeAddressMismatch>(); var findings = new List<ClusterNodeAddressMismatch>();
var seen = new HashSet<string>(StringComparer.OrdinalIgnoreCase); var seen = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
@@ -121,12 +122,12 @@ public static class ClusterNodeAddressReconciler
foreach (var row in rows) foreach (var row in rows)
{ {
if (seen.Contains(row.NodeId) || !enabled.Contains(row.NodeId)) continue; if (seen.Contains(row.NodeId) || !expectedPresent.Contains(row.NodeId)) continue;
findings.Add(new ClusterNodeAddressMismatch( findings.Add(new ClusterNodeAddressMismatch(
AddressMismatchKind.EnabledRowNotInCluster, row.NodeId, AddressMismatchKind.EnabledRowNotInCluster, row.NodeId,
$"enabled ClusterNode row {row.NodeId} has no matching driver member; the next " + $"enabled ClusterNode row {row.NodeId} has no matching driver member; the next " +
"deployment will fail at the apply deadline waiting for it. Start the node, or set " + "deployment will fail at the apply deadline waiting for it. Start the node, or set " +
"ClusterNode.Enabled = 0 while it is down for maintenance")); "ClusterNode.MaintenanceMode = 1 while it is down for maintenance"));
} }
return findings return findings
@@ -89,8 +89,11 @@ public sealed class ClusterNodeAddressReconcilerActor : ReceiveActor, IWithTimer
rows = db.ClusterNodes.AsNoTracking() rows = db.ClusterNodes.AsNoTracking()
.Select(n => new ClusterNodeAddress(n.NodeId, n.Host, n.AkkaPort)) .Select(n => new ClusterNodeAddress(n.NodeId, n.Host, n.AkkaPort))
.ToList(); .ToList();
// Same predicate the coordinator's expected-ack set uses — a node the deploy path will
// not wait for must not be reported as missing either, or the maintenance hatch trades a
// failed deployment for a permanent warning.
enabled = db.ClusterNodes.AsNoTracking() enabled = db.ClusterNodes.AsNoTracking()
.Where(n => n.Enabled).Select(n => n.NodeId).ToList(); .Where(n => n.Enabled && !n.MaintenanceMode).Select(n => n.NodeId).ToList();
} }
catch (Exception ex) catch (Exception ex)
{ {
@@ -48,7 +48,7 @@ public sealed class ConfigPublishCoordinatorDbSourcedAcksTests : ControlPlaneAct
{ {
var dbFactory = NewInMemoryDbFactory(); var dbFactory = NewInMemoryDbFactory();
var deploymentId = SeedDispatchingDeployment(dbFactory); var deploymentId = SeedDispatchingDeployment(dbFactory);
SeedNodes(dbFactory, ("site-a-1:4053", true)); SeedNodes(dbFactory, ("site-a-1:4053", true, false));
var actor = Sys.ActorOf(ConfigPublishCoordinator.Props(dbFactory, NoTimeout)); var actor = Sys.ActorOf(ConfigPublishCoordinator.Props(dbFactory, NoTimeout));
actor.Tell(new DispatchDeployment(deploymentId, TestRevision, CorrelationId.NewId())); actor.Tell(new DispatchDeployment(deploymentId, TestRevision, CorrelationId.NewId()));
@@ -73,16 +73,18 @@ public sealed class ConfigPublishCoordinatorDbSourcedAcksTests : ControlPlaneAct
} }
/// <summary> /// <summary>
/// The <c>Enabled = 0</c> escape hatch: a disabled row is NOT expected, so a deployment seals /// A disabled row is NOT expected, so a deployment seals without it — for a node genuinely
/// without it. This is what makes the "a stopped node now fails the deploy" behaviour change /// decommissioned rather than temporarily down. <b>This is not the maintenance hatch</b>: the
/// acceptable to ship — without it, a node down for maintenance blocks every deployment. /// topology gate rejects a deployment whose enabled-node count differs from the cluster's
/// <c>NodeCount</c>, so on any 2-node pair clearing <c>Enabled</c> never reaches this code.
/// See <see cref="MaintenanceMode_rows_are_not_expected_to_ack_while_staying_enabled"/>.
/// </summary> /// </summary>
[Fact] [Fact]
public void Disabled_ClusterNode_rows_are_not_expected_to_ack() public void Disabled_ClusterNode_rows_are_not_expected_to_ack()
{ {
var dbFactory = NewInMemoryDbFactory(); var dbFactory = NewInMemoryDbFactory();
var deploymentId = SeedDispatchingDeployment(dbFactory); var deploymentId = SeedDispatchingDeployment(dbFactory);
SeedNodes(dbFactory, ("site-a-1:4053", true), ("site-a-2:4053", false)); SeedNodes(dbFactory, ("site-a-1:4053", true, false), ("site-a-2:4053", false, false));
var actor = Sys.ActorOf(ConfigPublishCoordinator.Props(dbFactory, NoTimeout)); var actor = Sys.ActorOf(ConfigPublishCoordinator.Props(dbFactory, NoTimeout));
actor.Tell(new DispatchDeployment(deploymentId, TestRevision, CorrelationId.NewId())); actor.Tell(new DispatchDeployment(deploymentId, TestRevision, CorrelationId.NewId()));
@@ -116,7 +118,7 @@ public sealed class ConfigPublishCoordinatorDbSourcedAcksTests : ControlPlaneAct
{ {
var dbFactory = NewInMemoryDbFactory(); var dbFactory = NewInMemoryDbFactory();
var deploymentId = SeedDispatchingDeployment(dbFactory); var deploymentId = SeedDispatchingDeployment(dbFactory);
SeedNodes(dbFactory, ("site-a-1:4053", true)); SeedNodes(dbFactory, ("site-a-1:4053", true, false));
var actor = Sys.ActorOf(ConfigPublishCoordinator.Props(dbFactory, NoTimeout)); var actor = Sys.ActorOf(ConfigPublishCoordinator.Props(dbFactory, NoTimeout));
actor.Tell(new DispatchDeployment(deploymentId, TestRevision, CorrelationId.NewId())); actor.Tell(new DispatchDeployment(deploymentId, TestRevision, CorrelationId.NewId()));
@@ -158,7 +160,7 @@ public sealed class ConfigPublishCoordinatorDbSourcedAcksTests : ControlPlaneAct
{ {
var dbFactory = NewInMemoryDbFactory(); var dbFactory = NewInMemoryDbFactory();
var deploymentId = SeedDispatchingDeployment(dbFactory); var deploymentId = SeedDispatchingDeployment(dbFactory);
SeedNodes(dbFactory, ("site-a-1:4053", false), ("site-a-2:4053", false)); SeedNodes(dbFactory, ("site-a-1:4053", false, false), ("site-a-2:4053", false, false));
var actor = Sys.ActorOf(ConfigPublishCoordinator.Props(dbFactory, NoTimeout)); var actor = Sys.ActorOf(ConfigPublishCoordinator.Props(dbFactory, NoTimeout));
actor.Tell(new DispatchDeployment(deploymentId, TestRevision, CorrelationId.NewId())); actor.Tell(new DispatchDeployment(deploymentId, TestRevision, CorrelationId.NewId()));
@@ -171,9 +173,50 @@ public sealed class ConfigPublishCoordinatorDbSourcedAcksTests : ControlPlaneAct
}, duration: TimeSpan.FromSeconds(3)); }, duration: TimeSpan.FromSeconds(3));
} }
/// <summary>
/// The maintenance hatch, and the reason it is a separate flag. The Phase 1 live gate set
/// <c>Enabled = 0</c> on one node of a 2-node pair and the deployment was rejected outright by
/// <c>DraftValidator.ValidateClusterTopology</c> (enabled-node count must equal
/// <c>NodeCount</c>) — so the intended escape hatch could not be used on any pair, which is
/// every cluster in the target topology. <c>MaintenanceMode</c> excludes a node from the
/// expected-ack set while leaving it enabled, so the topology gate still passes.
/// </summary>
[Fact]
public void MaintenanceMode_rows_are_not_expected_to_ack_while_staying_enabled()
{
var dbFactory = NewInMemoryDbFactory();
var deploymentId = SeedDispatchingDeployment(dbFactory);
SeedNodes(dbFactory, ("site-a-1:4053", true, false), ("site-a-2:4053", true, true));
// The topology gate counts Enabled, so both nodes still count — this is what Enabled = 0
// broke. Assert it here so the fix cannot be "quietly disable it after all".
using (var check = dbFactory.CreateDbContext())
{
check.ClusterNodes.Count(n => n.Enabled).ShouldBe(2);
}
var actor = Sys.ActorOf(ConfigPublishCoordinator.Props(dbFactory, NoTimeout));
actor.Tell(new DispatchDeployment(deploymentId, TestRevision, CorrelationId.NewId()));
AwaitAssert(() =>
{
using var db = dbFactory.CreateDbContext();
db.NodeDeploymentStates.Select(s => s.NodeId).ShouldBe(new[] { "site-a-1:4053" });
}, duration: TimeSpan.FromSeconds(3));
actor.Tell(new ApplyAck(deploymentId, NodeId.Parse("site-a-1:4053"),
ApplyAckOutcome.Applied, null, CorrelationId.NewId()));
AwaitAssert(() =>
{
using var db = dbFactory.CreateDbContext();
db.Deployments.Single().Status.ShouldBe(DeploymentStatus.Sealed);
}, duration: TimeSpan.FromSeconds(3));
}
private static void SeedNodes( private static void SeedNodes(
IDbContextFactory<OtOpcUaConfigDbContext> dbFactory, IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
params (string NodeId, bool Enabled)[] nodes) params (string NodeId, bool Enabled, bool Maintenance)[] nodes)
{ {
using var db = dbFactory.CreateDbContext(); using var db = dbFactory.CreateDbContext();
db.ServerClusters.Add(new ServerCluster db.ServerClusters.Add(new ServerCluster
@@ -186,7 +229,7 @@ public sealed class ConfigPublishCoordinatorDbSourcedAcksTests : ControlPlaneAct
RedundancyMode = RedundancyMode.Warm, RedundancyMode = RedundancyMode.Warm,
CreatedBy = "test", CreatedBy = "test",
}); });
foreach (var (nodeId, enabled) in nodes) foreach (var (nodeId, enabled, maintenance) in nodes)
{ {
db.ClusterNodes.Add(new ClusterNode db.ClusterNodes.Add(new ClusterNode
{ {
@@ -195,6 +238,7 @@ public sealed class ConfigPublishCoordinatorDbSourcedAcksTests : ControlPlaneAct
Host = nodeId.Split(':')[0], Host = nodeId.Split(':')[0],
ApplicationUri = $"urn:OtOpcUa:{nodeId}", ApplicationUri = $"urn:OtOpcUa:{nodeId}",
Enabled = enabled, Enabled = enabled,
MaintenanceMode = maintenance,
CreatedBy = "test", CreatedBy = "test",
}); });
} }