# Per-cluster mesh Phase 1 — live gate record

**Date:** 2026-07-22 · **Rig:** `docker-dev` (six-node single mesh) · **Branch:** `feat/mesh-phase1`
**Plan:** [`2026-07-21-per-cluster-mesh-phase1.md`](2026-07-21-per-cluster-mesh-phase1.md)

**Result: PASSED**, after the gate found a blocker that changed the design (step 4).

Offline tests cannot cover the substitution's real risk — that the DB set and the live set disagree on
a running fleet. This is that check.

## Setup

Migration applied by the rig's `migrator` service against the shared `OtOpcUa` database, then all six
host nodes restarted on the rebuilt image. **All six pre-existing `ClusterNode` rows inherited
`AkkaPort = 4053` from the migration's `DEFAULT 4053`** — the row-level evidence that the default does
its job for rows predating the column:

```
central-1:4053|central-1|4053|NULL|1
central-2:4053|central-2|4053|NULL|1
site-a-1:4053 |site-a-1 |4053|NULL|1
site-a-2:4053 |site-a-2 |4053|NULL|1
site-b-1:4053 |site-b-1 |4053|NULL|1
site-b-2:4053 |site-b-2 |4053|NULL|1
   NodeId     |  Host   |Akka|Grpc|En
```

## Step 1 — deploy with all six nodes up → **PASS**

`POST /api/deployments` → `202 Accepted`, deployment `38f83b37`.

`Deployment.Status = 2` (Sealed). Six `NodeDeploymentState` rows, every one `Status = 1` (Applied).
**No FK 547** — the DB-derived NodeIds satisfy the `NodeDeploymentState → ClusterNode` foreign key,
which is the standing proof that the DB set and the membership set agree on this rig.

## Step 2 — *dropped, not skipped*

The plan called for a cluster-scoped deploy. **There is no such thing.** `Deployment` has no
`ClusterId`, `ConfigComposer.SnapshotAndFlattenAsync` always snapshots the whole DB, and
`DeploymentArtifact.ResolveClusterScope` is *node-side* self-scoping of a fleet-wide artifact. Removed
from the plan rather than faked green.

## Step 3 — deploy with one node stopped → **PASS** (the new behaviour)

`docker stop otopcua-dev-site-b-2-1`, then deploy `37d72301`.

`Deployment.Status = 4` (TimedOut) after the 2-minute apply deadline. Five nodes `Applied`,
`site-b-2:4053` still `Applying`. Under the membership rule this deployment would have **sealed
green** with five nodes and never mentioned the sixth.

The operator-facing log, which the plan required to be legible:

```
[12:52:02 WRN] Deployment 37d723018e374c0eb04c7b6beccc3b3e timed out after 00:02:00 (5/6 acks
landed). No ack from: site-b-2:4053. Each is an enabled ClusterNode row — start the node, or set
ClusterNode.Enabled = 0 if it is down for maintenance, then redeploy
```

(The remedy in that message was corrected to `MaintenanceMode = 1` — see step 4.)

## Step 4 — the escape hatch → **FAILED, then fixed and re-run → PASS**

### First attempt: the hatch did not exist

`UPDATE ClusterNode SET Enabled = 0 WHERE NodeId = 'site-b-2:4053'`, then deploy:

```
HTTP 422
[ClusterEnabledNodeCountMismatch] Cluster 'SITE-B' declares NodeCount=2 but has 1 Enabled nodes.
Toggle the missing node(s) back on or change RedundancyMode/NodeCount to match.
```

`DraftValidator.ValidateClusterTopology` rejects the deployment **before it dispatches**. So
`Enabled = 0` cannot be used on a node of a 2-node cluster — and every cluster on this rig, and every
cluster in the program plan's target topology, is a 2-node pair.

**This was a blocker, not a wrinkle.** The plan called step 4 *"the check that makes step 3
acceptable to ship"*. Without it, a node down for maintenance blocks every deployment to its cluster,
with no remedy short of editing `NodeCount` / `RedundancyMode` — which changes the runtime redundancy
posture to work around a deploy gate.

The two rules were each reasonable and contradictory together: the validator reads `Enabled` as *"part
of the declared topology"*, Phase 1 additionally read it as *"expect an ack"*.

### Fix (Task 7): split the meanings

New `ClusterNode.MaintenanceMode bit NOT NULL DEFAULT 0`. `Enabled` keeps its topology meaning and the
validator is untouched; the coordinator's expected-ack set and the address reconciler both filter on
`Enabled && !MaintenanceMode`.

### Re-run: PASS

`docker stop` `site-b-2` + `MaintenanceMode = 1`, then deploy `82c5e678`:

- `202 Accepted` — the topology gate passes, because the node is still `Enabled` (`1|1`).
- `Deployment.Status = 2` (Sealed).
- **Five** `NodeDeploymentState` rows, all `Applied`. `site-b-2:4053` has no row at all — excluded
  from the expected set rather than waited for and forgiven.

## Task 4 (reconciler) — live behaviour, not a planned gate step

Observed unprompted during the rolling restart, which is better evidence than a staged check:

```
[12:44:49 WRN] ClusterNode address check [EnabledRowNotInCluster] site-a-1:4053: ...
[12:44:49 WRN] ... site-a-2:4053 ... site-b-1:4053 ... site-b-2:4053
[12:44:56 INF] ClusterNode addresses reconcile with cluster membership (6 driver members)
```

It warned per row while the site nodes were still starting, then converged to a single clean line —
and **logged nothing further**, confirming the change-detection guard suppresses repeats on the
5-minute sweep.

Maintenance interaction, from the step 4 re-run:

```
[13:06:02 WRN] ... site-b-2:4053 ...            ← node dropped, flag not yet set
[13:06:05 INF] ... reconcile with cluster membership (5 driver members)   ← flag set: silent
```

The transient warning is honest — for those three seconds the row genuinely was enabled, present in
the expected-ack set, and absent from the cluster.

## What this gate did not cover

- **Phase 2's premise.** `AkkaPort` / `GrpcPort` are groundwork; nothing dials them yet, so "the value
  is correct" is only checked by the reconciler comparing it to gossip. The first real exercise is
  Phase 2.
- **The reconciler under split meshes.** On the current single mesh an admin node sees every driver
  member. After Phase 6 it will not, and every site row would report `EnabledRowNotInCluster`
  permanently. Recorded as a known limitation on the class and in the design doc.
- **`Enabled = 0` on a single-node (`RedundancyMode.None`) cluster.** Untested — the rig has no such
  cluster. It would fail the same topology rule (0 enabled vs `NodeCount = 1`), so `MaintenanceMode`
  is the hatch there too.