Compare commits

..

10 Commits

Author SHA1 Message Date
Joseph Doherty 7654f24dab test(harness): serialize Host.IntegrationTests; drop the reconciler's unused singleton proxy
v2-ci / build (push) Successful in 3m51s
v2-ci / unit-tests (push) Failing after 13m29s
Closes the last branch-only failures. Measured across five full-suite runs on
this branch versus two on master in a clean worktree: master failed only AbCip
in this assembly (2/2), while the branch additionally failed one or two E2E
deploy tests, rotating between DeployHappyPath, DriverReconnect and
EquipmentNamespaceMaterialization (4/4). Each passed in isolation, and the
assembly on its own was 195/201 throughout — the failures only appeared under
full-suite CPU contention.

Cause is contention, not correctness. Every test here builds a real two-node
Akka cluster — two Kestrel hosts, two ActorSystems with remoting, six admin
singletons, an EF context, a deploy pipeline — and xUnit ran them concurrently
with each other and with every other assembly. That was survivable while the
coordinator sealed instantly on an empty expected-ack set; now that it waits for
a real ack from every configured node, these tests measure an end-to-end
round-trip and starve.

Serialising this one assembly makes them deterministic: 195/201 with only
AbCip_Green_AgainstSim, and the full-solution failure set is now IDENTICAL to
master's — 7 shared, zero branch-only. Cost is wall-clock for this assembly,
40s -> 3m35s; it is a handful of heavyweight E2E tests, not a broad unit suite,
and a widened timeout alone did not fix it (tried first, at 45s).

Also drops createProxyToo for the address reconciler. Nothing resolves
ClusterNodeAddressReconcilerKey from the registry — the actor only reads cluster
state and logs — so the proxy was a second actor per node hunting for a
singleton nobody addresses. Kept because it removes dead machinery, not because
it fixed the flakiness; measured on its own, it did not.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-22 09:51:10 -04:00
Joseph Doherty 0b7c53f64f test(harness): widen the deploy-seal timeout to match a real two-node deploy
Comparing full-suite runs on this branch against master in a clean worktree:
both fail 11 tests, 9 identical. Master's two extras are load-flaky unit tests
(AbCip Probe_loops, Galaxy EventPumpBoundedChannel); this branch's two extras
were both Host.IntegrationTests deploy-path tests — the area this change
re-times — so they are attributable here rather than to background flakiness.

Cause is margin, not correctness. These waits used to observe a coordinator
that sealed instantly on an empty expected-ack set; they now observe a real
ApplyAck round-trip from every configured node. 15s was enormous margin against
"instant" and thin against the real thing, so they failed only under full-suite
CPU contention.

Hoisted to TwoNodeClusterHarness.DeploySealTimeout (45s) so the reason is
recorded once rather than as four unexplained numbers.

DriverReconnectE2eTests is deliberately left alone: it seeds both ClusterNode
rows itself and unconditionally, so its expected-ack set is identical before and
after this change. Widening its timeout would be papering over a flake this
change did not cause.

Host.IntegrationTests 195/201; sole failure AbCip_Green_AgainstSim, verified
failing on master.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-22 09:28:32 -04:00
Joseph Doherty 14746f2995 test(harness): seed ClusterNode rows in in-memory mode; retarget the failover deploy test
The full-suite run surfaced two load-dependent failures that were green in
isolation. Root cause: SeedDefaultClusterAsync no-opped unless
OTOPCUA_HARNESS_USE_SQL=1, on the reasoning that "the in-memory provider ignores
FK constraints, so deploy E2E tests pass without it". Phase 1 invalidated that.

With no ClusterNode rows the coordinator's expected-ack set is empty, so it
seals IMMEDIATELY. "Wait for Sealed" therefore stopped implying "every node has
applied" — and because DriverHostActor.UpsertNodeDeploymentState writes each
node's row on its own schedule, every test counting those rows after a seal
became a race. Reproducibly green alone, red under suite load, and it surfaced
in a different test each run, which is what made it look like flakiness rather
than a contract change.

Seeding in both modes restores the invariant those tests were written against
and is closer to production either way: a real fleet always has these rows,
because the FK requires them.

Deployment_started_with_node_b_down_seals_with_one_node_state asserted the
behaviour Phase 1 deliberately removed — its own comment documented membership
snapshotting, i.e. sealing green while a configured node never received the
deployment. Replaced by two tests covering the new contract: a stopped node is
still expected (both state rows exist, B still Applying, deployment does not
seal), and MaintenanceMode is what makes it seal with one.

The new "does not seal" test asserts both rows exist rather than only the
absence of a seal, so it cannot pass against a coordinator that simply died.

Host.IntegrationTests 195/201, sole remaining failure AbCip_Green_AgainstSim —
verified failing on master in a clean worktree, so pre-existing fixture
baseline, not a regression.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-22 09:19:53 -04:00
Joseph Doherty e27b7f43f5 chore(mesh): mark Phase 1 tasks complete
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-22 09:07:48 -04:00
Joseph Doherty ee69caa270 feat(config): add ClusterNode.MaintenanceMode — the hatch the live gate proved missing
Phase 1 gate step 4 failed: setting Enabled = 0 to take a node out of service
returns 422 ClusterEnabledNodeCountMismatch, because
DraftValidator.ValidateClusterTopology requires the enabled-node count to equal
ServerCluster.NodeCount. On a Warm/Hot pair — every cluster on the rig, and
every cluster in the target topology — Enabled can therefore never be the
maintenance hatch. The plan called step 4 "the check that makes step 3
acceptable to ship", so Task 3's behaviour change was not shippable as it stood:
a node down for maintenance would block every deployment to its cluster.

The two rules were each reasonable and contradictory together — the validator
reads Enabled as "part of the declared topology", Phase 1 additionally read it
as "expect an ack". Split the meanings rather than weaken either rule:

  Enabled          part of the declared topology  (validator, untouched)
  MaintenanceMode  expected to participate now    (coordinator + reconciler)

Rejected alternatives: counting configured rather than enabled nodes (drops the
guard against booting a pair into InvalidTopology); downgrading the rule to a
warning (weakens a deploy gate for everyone); shipping with no hatch.

Sabotage: dropping !n.MaintenanceMode from the coordinator query turns the new
test red. It also asserts both nodes remain Enabled, so the fix cannot quietly
regress to disabling the row after all.

Configuration.Tests 95/95, ControlPlane.Tests 101/101, solution builds clean.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-22 09:07:29 -04:00
Joseph Doherty 57e1d01766 docs(mesh): rig seed + Phase 1 documentation
- docker-dev seed: AkkaPort = 4053 on all six ClusterNode rows, so a freshly
  seeded rig matches a migrated one. GrpcPort left null.
- config-db-schema.md: both columns, why AkkaPort is NOT NULL/4053 and GrpcPort
  is nullable, the unenforced duplication + its reconciler, and Enabled's new
  second meaning as the deploy path's expected-ack set.
- Configuration.md: Cluster:Port / PublicHostname now flag that they are stored
  twice, with the "update the row too" instruction and why the drift is silent.
- design doc §7: Phase 1 marked done, plus a "Phase 1 as shipped" note recording
  both deviations rather than leaving the sketch reading as what happened.
- program plan: Phase 1 marked done; AdminUI node edit explicitly deferred.
- CLAUDE.md: the deploy-path behaviour change and its three consequences.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-22 08:38:45 -04:00
Joseph Doherty 90c1302f71 feat(fleet): reconcile ClusterNode dial targets against live membership
ClusterNode.AkkaPort and the node's own Cluster:Port are the same fact in two
places and nothing made them agree. Phase 2 dials the row instead of gossiping,
so a node binding 4054 while its row says 4053 becomes unreachable from central
— and the symptom is a silent absence of acks rather than an error. That is the
shape of the Modbus/ModbusTcp and TwinCat/Focas drifts already in this repo.
Since Phase 1 also made the rows the deploy path's expected-ack set, drift
already costs a failed deployment today.

Implemented as the plan's preferred option: an admin-role singleton comparing
rows against the membership an admin node can already see, rather than each
driver node asserting its own row — Phase 4 removes the driver nodes' ConfigDb
connection, so a self-assertion written there would have to be deleted again.

Three shapes, split by severity: a row whose dial target disagrees with its own
NodeId and a running node with no row are Errors; an enabled row with no
matching member is a Warning, because a node down for maintenance is a
legitimate state. Findings are logged only when the set changes — a check that
reprints the same warning every sweep trains operators to filter it out.

Documented limitation: Phase 2 must revisit this. Once the fleet splits into
one mesh per cluster an admin node cannot see site members, and every site row
would report EnabledRowNotInCluster forever.

Sabotage: removing the change-detection guard turns the repeat test red.
ControlPlane.Tests 100/100.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-22 08:36:44 -04:00
Joseph Doherty d88e245503 feat(deploy): coordinator sources its expected-ack set from ClusterNode rows
Cuts ConfigPublishCoordinator's last genuinely mesh-bound dependency: central
must be able to name the nodes a deployment is for without sharing a gossip
ring with them, because Phase 2 splits the fleet into one mesh per Cluster.

Behaviour change, confirmed before implementing: a configured node that is
switched off is now expected, so deploying while it is down fails at the apply
deadline instead of sealing green without it. Under the membership rule the
operator was told the fleet was deployed when it was not. ClusterNode.Enabled
= 0 is the maintenance hatch, and the deadline log now names the silent nodes
and points at that hatch — "4/5 acks landed" would leave an operator reading
logs on five machines.

Two assumptions are now documented on the class rather than left implicit:
every ClusterNode row is a driver node (the DB has no role column and
deliberately does not gain one — it would drift from Cluster:Roles), and
ServerCluster.Enabled is not consulted because nothing else consults it.

Dropped from the plan: cluster-scope filtering of the expected set. There is
no cluster-scoped deployment to filter on — Deployment has no ClusterId,
ConfigComposer always snapshots the whole DB, and ResolveClusterScope is
node-side self-scoping of a fleet-wide artifact.

Positive control: reverting DiscoverDriverNodes to the membership scan turns
three of the four new tests red. The fourth pins the seal-empty branch and is
documented as derivation-insensitive rather than left to look like coverage.

ControlPlane.Tests 90/90.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-22 08:30:46 -04:00
Joseph Doherty da74ebd696 feat(config): migration AddClusterNodeTransportPorts
Additive only — two AddColumn ops, no AlterColumn, so no accumulated model
drift is riding along with this change:

  ALTER TABLE [ClusterNode] ADD [AkkaPort] int NOT NULL DEFAULT 4053;
  ALTER TABLE [ClusterNode] ADD [GrpcPort] int NULL;

Adds a third test covering the DB-side default specifically. The entity
round-trip cannot see it: ClusterNode.AkkaPort's CLR initializer is also 4053,
so that assertion passes with the mapping default deleted. The new test inserts
through raw SQL with the column omitted, so only the schema can supply the
value — verified by setting defaultValue: 0, which turns exactly that one test
red and leaves the other two green.

Configuration.Tests 95/95.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-22 08:26:39 -04:00
Joseph Doherty 2bbb02713c feat(config): add ClusterNode.AkkaPort/GrpcPort — central's dial targets
Per-cluster mesh Phase 1 groundwork. Once the meshes split (Phase 2) central
can no longer see a site node's appsettings, so the transport ports it must
dial have to live in a row central can read.

AkkaPort is non-nullable with a 4053 default — every node listens on a
remoting port, so 0 is never a truthful value and pre-existing rows must
migrate to something real. GrpcPort is nullable with no default: nothing
listens on it until Phase 5, and a non-null default would assert a port that
does not exist.

Both are documented as central's dial targets rather than the node's own
binding config; the duplication against Cluster:Port is reconciled in Task 4.

The new SchemaCompliance test is red until the migration lands (Task 2).

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-22 08:22:46 -04:00
30 changed files with 5029 additions and 68 deletions
+8 -1
View File
@@ -228,7 +228,14 @@ The server supports non-transparent warm/hot redundancy via the `Redundancy` sec
Assert the **effective** cluster config off a running `ActorSystem` (as `SplitBrainResolverActivationTests` does), never the typed options or the akka.conf text — several HOCON fragments compete and the losing one still reads correctly in the file. Assert the **effective** cluster config off a running `ActorSystem` (as `SplitBrainResolverActivationTests` does), never the typed options or the akka.conf text — several HOCON fragments compete and the losing one still reads correctly in the file.
**Still per-Akka-cluster, not per application `Cluster`.** One Primary is elected across the whole Akka mesh, so a fleet running several application clusters in one mesh (what docker-dev models) gets one Primary among all driver nodes. The fix is the per-cluster mesh workstream: `docs/plans/2026-07-21-per-cluster-mesh-design.md` (Phases 0a/0b done; 17 not started, Phase 1 planned in `…-phase1.md`). **Still per-Akka-cluster, not per application `Cluster`.** One Primary is elected across the whole Akka mesh, so a fleet running several application clusters in one mesh (what docker-dev models) gets one Primary among all driver nodes. The fix is the per-cluster mesh workstream: `docs/plans/2026-07-21-per-cluster-mesh-design.md` (Phases 0a/0b/1 done; 27 not started).
**Phase 1 landed 2026-07-22 and changed a deploy-path behaviour.** `ConfigPublishCoordinator` sources its expected-ack set from **enabled `ClusterNode` rows**, not from `Akka.Cluster.State.Members` filtered by the `driver` role — central must be able to name a deployment's nodes without sharing a gossip ring with them. Consequences worth knowing before touching deploys:
- **A configured node that is switched off now FAILS the deployment** at the apply deadline instead of letting it seal green without that node. The maintenance hatch is the **new `ClusterNode.MaintenanceMode` column**, NOT `Enabled = 0`: the live gate showed `Enabled = 0` is rejected outright by `DraftValidator.ValidateClusterTopology` (`ClusterEnabledNodeCountMismatch` — enabled count must equal `ServerCluster.NodeCount`), so on a 2-node pair — every cluster in the target topology — it can never be the hatch. `Enabled` = part of the declared topology; `MaintenanceMode` = do not expect it now. The deadline log names the silent nodes and points at the right flag.
- **Every `ClusterNode` row is now *defined* to be a driver node.** The DB has no per-node role column and deliberately did not gain one (it would drift from `Cluster:Roles`). Giving an admin-only node a `ClusterNode` row makes every deploy time out.
- **There is no such thing as a cluster-scoped deployment.** `Deployment` has no `ClusterId`, `ConfigComposer` always snapshots the whole DB, and `DeploymentArtifact.ResolveClusterScope` is *node-side* self-scoping of a fleet-wide artifact. The phase-1 plan asked for cluster-scope filtering of the ack set; it was dropped as describing a feature that does not exist.
- `ClusterNode` gained `AkkaPort` (NOT NULL, 4053) + `GrpcPort` (nullable) as **central's dial targets** for Phases 2/5. They duplicate the node's own `Cluster:Port` / `Cluster:PublicHostname` with no schema-level enforcement, so `ClusterNodeAddressReconcilerActor` (admin singleton) logs an Error on drift. **Phase 2 must revisit it** — once the meshes split, an admin node cannot see site members and every site row would report `EnabledRowNotInCluster` forever.
## LocalDb pair-local store (Phases 1 + 2) ## LocalDb pair-local store (Phases 1 + 2)
+12 -12
View File
@@ -64,13 +64,13 @@ IF NOT EXISTS (SELECT 1 FROM dbo.ServerCluster WHERE ClusterId = 'SITE-B')
IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'central-1:4053') IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'central-1:4053')
INSERT INTO dbo.ClusterNode INSERT INTO dbo.ClusterNode
(NodeId, ClusterId, Host, OpcUaPort, DashboardPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy) (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
VALUES ('central-1:4053', 'MAIN', 'central-1', 4840, 8081, 'urn:OtOpcUa:central-1', 200, 1, 'docker-dev-seed'); VALUES ('central-1:4053', 'MAIN', 'central-1', 4840, 8081, 4053, 'urn:OtOpcUa:central-1', 200, 1, 'docker-dev-seed');
IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'central-2:4053') IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'central-2:4053')
INSERT INTO dbo.ClusterNode INSERT INTO dbo.ClusterNode
(NodeId, ClusterId, Host, OpcUaPort, DashboardPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy) (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
VALUES ('central-2:4053', 'MAIN', 'central-2', 4840, 8081, 'urn:OtOpcUa:central-2', 150, 1, 'docker-dev-seed'); VALUES ('central-2:4053', 'MAIN', 'central-2', 4840, 8081, 4053, 'urn:OtOpcUa:central-2', 150, 1, 'docker-dev-seed');
------------------------------------------------------------------------------ ------------------------------------------------------------------------------
-- ClusterNode — site A -- ClusterNode — site A
@@ -78,13 +78,13 @@ IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'central-2:4053')
IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'site-a-1:4053') IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'site-a-1:4053')
INSERT INTO dbo.ClusterNode INSERT INTO dbo.ClusterNode
(NodeId, ClusterId, Host, OpcUaPort, DashboardPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy) (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
VALUES ('site-a-1:4053', 'SITE-A', 'site-a-1', 4840, 8081, 'urn:OtOpcUa:site-a-1', 200, 1, 'docker-dev-seed'); VALUES ('site-a-1:4053', 'SITE-A', 'site-a-1', 4840, 8081, 4053, 'urn:OtOpcUa:site-a-1', 200, 1, 'docker-dev-seed');
IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'site-a-2:4053') IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'site-a-2:4053')
INSERT INTO dbo.ClusterNode INSERT INTO dbo.ClusterNode
(NodeId, ClusterId, Host, OpcUaPort, DashboardPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy) (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
VALUES ('site-a-2:4053', 'SITE-A', 'site-a-2', 4840, 8081, 'urn:OtOpcUa:site-a-2', 150, 1, 'docker-dev-seed'); VALUES ('site-a-2:4053', 'SITE-A', 'site-a-2', 4840, 8081, 4053, 'urn:OtOpcUa:site-a-2', 150, 1, 'docker-dev-seed');
------------------------------------------------------------------------------ ------------------------------------------------------------------------------
-- ClusterNode — site B -- ClusterNode — site B
@@ -92,13 +92,13 @@ IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'site-a-2:4053')
IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'site-b-1:4053') IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'site-b-1:4053')
INSERT INTO dbo.ClusterNode INSERT INTO dbo.ClusterNode
(NodeId, ClusterId, Host, OpcUaPort, DashboardPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy) (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
VALUES ('site-b-1:4053', 'SITE-B', 'site-b-1', 4840, 8081, 'urn:OtOpcUa:site-b-1', 200, 1, 'docker-dev-seed'); VALUES ('site-b-1:4053', 'SITE-B', 'site-b-1', 4840, 8081, 4053, 'urn:OtOpcUa:site-b-1', 200, 1, 'docker-dev-seed');
IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'site-b-2:4053') IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'site-b-2:4053')
INSERT INTO dbo.ClusterNode INSERT INTO dbo.ClusterNode
(NodeId, ClusterId, Host, OpcUaPort, DashboardPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy) (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
VALUES ('site-b-2:4053', 'SITE-B', 'site-b-2', 4840, 8081, 'urn:OtOpcUa:site-b-2', 150, 1, 'docker-dev-seed'); VALUES ('site-b-2:4053', 'SITE-B', 'site-b-2', 4840, 8081, 4053, 'urn:OtOpcUa:site-b-2', 150, 1, 'docker-dev-seed');
------------------------------------------------------------------------------ ------------------------------------------------------------------------------
-- Galaxy MxAccess gateway — INTENTIONALLY NOT SEEDED (retired 2026-06-15) -- Galaxy MxAccess gateway — INTENTIONALLY NOT SEEDED (retired 2026-06-15)
+4 -2
View File
@@ -101,13 +101,15 @@ The checked-in `appsettings*.json` files are deliberately thin: they carry only
|---|---|---|---| |---|---|---|---|
| `SystemName` | string | `otopcua` | Akka actor-system name. | | `SystemName` | string | `otopcua` | Akka actor-system name. |
| `Hostname` | string | `0.0.0.0` | Bind hostname. | | `Hostname` | string | `0.0.0.0` | Bind hostname. |
| `Port` | int | `4053` | Cluster transport port. | | `Port` | int | `4053` | Cluster transport port. **Duplicated as `ClusterNode.AkkaPort` in the Config DB** — see the note below. |
| `PublicHostname` | string | `127.0.0.1` | Hostname advertised in cluster gossip; must be reachable by peers. | | `PublicHostname` | string | `127.0.0.1` | Hostname advertised in cluster gossip; must be reachable by peers. **Duplicated as `ClusterNode.Host`** — see the note below. |
| `SeedNodes` | string[] | `[]` | Seed nodes for bootstrapping. **Ordered:** a node that appears in its own list must be entry 0 (only `seed-nodes[0]` can form a new cluster), or the host refuses to start — see [Redundancy.md § Bootstrap: self-first seed ordering](Redundancy.md#bootstrap-self-first-seed-ordering). | | `SeedNodes` | string[] | `[]` | Seed nodes for bootstrapping. **Ordered:** a node that appears in its own list must be entry 0 (only `seed-nodes[0]` can form a new cluster), or the host refuses to start — see [Redundancy.md § Bootstrap: self-first seed ordering](Redundancy.md#bootstrap-self-first-seed-ordering). |
| `Roles` | string[] | `[]` | Cluster roles for this node. When empty, falls back to `OTOPCUA_ROLES`. Allowed values: `admin`, `driver`, `dev`. | | `Roles` | string[] | `[]` | Cluster roles for this node. When empty, falls back to `OTOPCUA_ROLES`. Allowed values: `admin`, `driver`, `dev`. |
> The full redundancy model (ServiceLevel tiers, split-brain, peer discovery) is in [`Redundancy.md`](Redundancy.md). The OPC UA peer-URI advertising lives in the `OpcUa:PeerApplicationUris` key above. > The full redundancy model (ServiceLevel tiers, split-brain, peer discovery) is in [`Redundancy.md`](Redundancy.md). The OPC UA peer-URI advertising lives in the `OpcUa:PeerApplicationUris` key above.
> **`Port` / `PublicHostname` are stored twice.** The node binds from these keys; the fleet's `ClusterNode` row records the same address as `AkkaPort` / `Host` so that **central can dial the node without sharing a gossip ring with it** — which is what [per-cluster mesh Phase 2](plans/2026-07-21-per-cluster-mesh-design.md) needs. Nothing in the schema makes the two agree, so `ClusterNodeAddressReconcilerActor` (admin-role singleton) compares them against live membership and logs an Error on mismatch. If you change `Cluster:Port` or `Cluster:PublicHostname` on a node, **update its `ClusterNode` row too** — a node binding 4054 with a row saying 4053 still gossips fine today, and fails silently in Phase 2. The row's `NodeId` is `host:port` and is also the deploy path's ack identity, so a drifted node's deployment acks stop matching as well. See [`config-db-schema.md` § `ClusterNode`](v2/config-db-schema.md#clusternode).
### `ConnectionStrings` → `ConfigDb` ### `ConnectionStrings` → `ConfigDb`
- **Purpose:** the central Config DB connection string. **Required for every role**`Program.cs` calls `AddOtOpcUaConfigDb` unconditionally. - **Purpose:** the central Config DB connection string. **Required for every role**`Program.cs` calls `AddOtOpcUaConfigDb` unconditionally.
@@ -287,7 +287,7 @@ Deliberately not a task plan — per-phase plans follow, one at a time.
|---|---|---| |---|---|---|
| 0a | Downing strategy: keep-oldest cannot survive an oldest-node crash in a 2-node cluster (§6.2) | **Yes** — **DONE 2026-07-21**, live gate deferred to Phase 7 | | 0a | Downing strategy: keep-oldest cannot survive an oldest-node crash in a 2-node cluster (§6.2) | **Yes** — **DONE 2026-07-21**, live gate deferred to Phase 7 |
| 0b | Oldest-Up role derivation (§4) | **Yes** — **DONE 2026-07-21** | | 0b | Oldest-Up role derivation (§4) | **Yes** — **DONE 2026-07-21** |
| 1 | `ClusterNode` gains Akka + gRPC address columns; coordinator sources its expected-ack set from the DB | Yes | | 1 | `ClusterNode` gains Akka + gRPC address columns; coordinator sources its expected-ack set from the DB | **Yes** — **DONE 2026-07-22** (see below) |
| 2 | Comm actors + receptionist registration; ClusterClient transport; deploy notify + acks across the boundary | No | | 2 | Comm actors + receptionist registration; ClusterClient transport; deploy notify + acks across the boundary | No |
| 3 | Config fetch-and-cache: artifact served by central, driver nodes read LocalDb (§6.1) | No | | 3 | Config fetch-and-cache: artifact served by central, driver nodes read LocalDb (§6.1) | No |
| 4 | Cut the driver-side ConfigDb connection: re-home `EfAlarmConditionStateStore`, resolve the `DbHealthProbeActor` ServiceLevel input, audit `OpcUaPublishActor` | No | | 4 | Cut the driver-side ConfigDb connection: re-home `EfAlarmConditionStateStore`, resolve the `DbHealthProbeActor` ServiceLevel input, audit `OpcUaPublishActor` | No |
@@ -298,6 +298,25 @@ Deliberately not a task plan — per-phase plans follow, one at a time.
Phases 3 and 4 are the ones that change a running system's data path rather than its wiring, and each Phases 3 and 4 are the ones that change a running system's data path rather than its wiring, and each
deserves its own live gate. deserves its own live gate.
### Phase 1 as shipped (2026-07-22)
Plan: `2026-07-21-per-cluster-mesh-phase1.md`. Two deviations from the sketch above, both decided
before implementing:
- **No cluster-scope filtering of the expected-ack set.** The plan called for it, but there is no
cluster-scoped deployment to filter on: `Deployment` has no `ClusterId`,
`ConfigComposer.SnapshotAndFlattenAsync` always snapshots the whole DB, and
`DeploymentArtifact.ResolveClusterScope` is *node-side* self-scoping of a fleet-wide artifact. The
expected set is every enabled `ClusterNode` row, which is what the membership rule produced too.
- **No per-node role column.** The membership rule filtered on the `driver` role and the DB has none.
Rather than add one — a second declaration of node roles, free to drift from `Cluster:Roles` — every
`ClusterNode` row is now *defined* to be a driver node. An admin-only node must not be given one.
Also shipped beyond the sketch: `ClusterNodeAddressReconciler` (Task 4), an admin singleton that
catches `AkkaPort` drifting from the node's own `Cluster:Port`. **Phase 2 must revisit it** — once the
meshes split, an admin node cannot see site members and every site row would report
`EnabledRowNotInCluster` forever.
## 8. Risks ## 8. Risks
- **LocalDb becomes load-bearing for configuration, not just resilient.** Phase 1 built it as a - **LocalDb becomes load-bearing for configuration, not just resilient.** Phase 1 built it as a
@@ -12,8 +12,17 @@ sourcing its expected-ack set from those rows.
**Architecture:** Additive schema change plus one substitution inside the coordinator. Nothing else **Architecture:** Additive schema change plus one substitution inside the coordinator. Nothing else
moves; the deploy channel stays on DistributedPubSub until Phase 2. moves; the deploy channel stays on DistributedPubSub until Phase 2.
**Status:** NOT STARTED. Read §"The decision this phase forces" before executing — it changes **Status: DONE 2026-07-22** (branch `feat/mesh-phase1`). Live gate:
deploy-seal semantics and should be confirmed, not assumed. [`2026-07-22-mesh-phase1-live-gate.md`](2026-07-22-mesh-phase1-live-gate.md) — **PASSED**, after gate
step 4 found that the escape hatch this phase depends on did not exist. Three deviations from the
plan below, all decided before implementing or forced by the gate:
1. **Task 3's cluster-scope filtering was dropped** — there is no cluster-scoped deployment to filter
on. See the gate doc, step 2.
2. **No per-node role column.** Every `ClusterNode` row is now *defined* to be a driver node.
3. **Task 7 added:** `ClusterNode.MaintenanceMode`. `Enabled = 0` is rejected by
`DraftValidator.ValidateClusterTopology` on any 2-node pair, so it could never be the maintenance
hatch — which the plan called "the check that makes step 3 acceptable to ship".
--- ---
@@ -54,7 +63,7 @@ But it is a behaviour change, and it needs an operator escape hatch or a node ta
maintenance blocks every deployment. `ClusterNode.Enabled` already exists and is exactly that hatch, maintenance blocks every deployment. `ClusterNode.Enabled` already exists and is exactly that hatch,
so the query filters on it: **`Enabled = 1` rows are expected to ack; disabled rows are not.** so the query filters on it: **`Enabled = 1` rows are expected to ack; disabled rows are not.**
**Confirm this before executing Task 3.** If the preference is to keep sealing green past a down **Confirmed 2026-07-22 before executing Task 3** — DB-derived with the hatch. (The hatch turned out to be `MaintenanceMode`, not `Enabled`; see Task 7.) If the preference is to keep sealing green past a down
node, the alternative is to intersect the DB set with current membership, which preserves today's node, the alternative is to intersect the DB set with current membership, which preserves today's
behaviour but does not survive the mesh split — it would have to be revisited in Phase 2 anyway. behaviour but does not survive the mesh split — it would have to be revisited in Phase 2 anyway.
@@ -189,3 +198,25 @@ disagree on the rig. Run on `docker-dev`:
hatch, and is the check that makes step 3 acceptable to ship. hatch, and is the check that makes step 3 acceptable to ship.
Record results in a gate doc beside this plan, per the Phase 1/Phase 2 precedent. Record results in a gate doc beside this plan, per the Phase 1/Phase 2 precedent.
## Task 7: `MaintenanceMode` — the escape hatch step 4 proved missing (ADDED 2026-07-22)
**Classification:** high-risk — deploy path + schema
**Why:** gate step 4 returned `422 ClusterEnabledNodeCountMismatch`. `Enabled = 0` on one node of a
Warm/Hot pair fails `DraftValidator.ValidateClusterTopology` (enabled count must equal `NodeCount`),
so the hatch Task 3 relies on cannot be used on any 2-node cluster — i.e. every cluster in the target
topology. Task 3's behaviour change is only shippable with a hatch that works.
**Decision:** split the meanings rather than weaken either rule.
| Flag | Question | Enforced by |
|---|---|---|
| `Enabled` | part of the declared topology? | `DraftValidator` vs `ServerCluster.NodeCount` |
| `MaintenanceMode` | expected to participate right now? | `ConfigPublishCoordinator`, `ClusterNodeAddressReconciler` |
Rejected: counting configured rather than enabled nodes (removes the InvalidTopology guard);
downgrading the rule to a warning (weakens a deploy gate for everyone); shipping without a hatch.
**Done:** column + migration `AddClusterNodeMaintenanceMode`; both consumers filter on
`Enabled && !MaintenanceMode`; deadline + reconciler messages name the right flag; docs; gate step 4
re-run green.
@@ -0,0 +1,76 @@
{
"plan": "docs/plans/2026-07-21-per-cluster-mesh-phase1.md",
"branch": "feat/mesh-phase1",
"decisions": {
"seal-semantics": "DB-derived + Enabled hatch (confirmed 2026-07-22)",
"cluster-scope": "Dropped \u2014 no cluster-scoped deployment exists; expected set = all enabled ClusterNode rows",
"node-roles": "Every ClusterNode row is a driver node; documented + pinned by test, no Roles column",
"adminui-edit": "Deferred to Phase 2 (nothing reads the columns until then)",
"maintenance-hatch": "MaintenanceMode column, not Enabled=0 (blocked by DraftValidator on any 2-node pair)"
},
"tasks": [
{
"id": 1,
"subject": "Address columns on ClusterNode",
"status": "completed",
"commit": "2bbb0271",
"blockedBy": []
},
{
"id": 2,
"subject": "EF migration AddClusterNodeTransportPorts",
"status": "completed",
"commit": "da74ebd6",
"blockedBy": [
1
]
},
{
"id": 3,
"subject": "Coordinator sources expected-ack set from ClusterNode",
"status": "completed",
"commit": "d88e2455",
"blockedBy": [
2
]
},
{
"id": 4,
"subject": "Reconcile duplicated AkkaPort with the node's own Cluster:Port",
"status": "completed",
"blockedBy": [
2
],
"commit": "90c1302f"
},
{
"id": 5,
"subject": "Rig seed + docs",
"status": "completed",
"blockedBy": [
3,
4
],
"commit": "57e1d017"
},
{
"id": 6,
"subject": "Live gate on docker-dev",
"status": "completed",
"blockedBy": [
5
],
"result": "live-gate PASSED (2026-07-22-mesh-phase1-live-gate.md)"
},
{
"id": 7,
"subject": "MaintenanceMode \u2014 escape hatch the live gate proved missing",
"status": "completed",
"commit": "ee69caa2",
"blockedBy": [
6
],
"origin": "added from gate step 4 failure"
}
]
}
@@ -0,0 +1,134 @@
# Per-cluster mesh Phase 1 — live gate record
**Date:** 2026-07-22 · **Rig:** `docker-dev` (six-node single mesh) · **Branch:** `feat/mesh-phase1`
**Plan:** [`2026-07-21-per-cluster-mesh-phase1.md`](2026-07-21-per-cluster-mesh-phase1.md)
**Result: PASSED**, after the gate found a blocker that changed the design (step 4).
Offline tests cannot cover the substitution's real risk — that the DB set and the live set disagree on
a running fleet. This is that check.
## Setup
Migration applied by the rig's `migrator` service against the shared `OtOpcUa` database, then all six
host nodes restarted on the rebuilt image. **All six pre-existing `ClusterNode` rows inherited
`AkkaPort = 4053` from the migration's `DEFAULT 4053`** — the row-level evidence that the default does
its job for rows predating the column:
```
central-1:4053|central-1|4053|NULL|1
central-2:4053|central-2|4053|NULL|1
site-a-1:4053 |site-a-1 |4053|NULL|1
site-a-2:4053 |site-a-2 |4053|NULL|1
site-b-1:4053 |site-b-1 |4053|NULL|1
site-b-2:4053 |site-b-2 |4053|NULL|1
NodeId | Host |Akka|Grpc|En
```
## Step 1 — deploy with all six nodes up → **PASS**
`POST /api/deployments``202 Accepted`, deployment `38f83b37`.
`Deployment.Status = 2` (Sealed). Six `NodeDeploymentState` rows, every one `Status = 1` (Applied).
**No FK 547** — the DB-derived NodeIds satisfy the `NodeDeploymentState → ClusterNode` foreign key,
which is the standing proof that the DB set and the membership set agree on this rig.
## Step 2 — *dropped, not skipped*
The plan called for a cluster-scoped deploy. **There is no such thing.** `Deployment` has no
`ClusterId`, `ConfigComposer.SnapshotAndFlattenAsync` always snapshots the whole DB, and
`DeploymentArtifact.ResolveClusterScope` is *node-side* self-scoping of a fleet-wide artifact. Removed
from the plan rather than faked green.
## Step 3 — deploy with one node stopped → **PASS** (the new behaviour)
`docker stop otopcua-dev-site-b-2-1`, then deploy `37d72301`.
`Deployment.Status = 4` (TimedOut) after the 2-minute apply deadline. Five nodes `Applied`,
`site-b-2:4053` still `Applying`. Under the membership rule this deployment would have **sealed
green** with five nodes and never mentioned the sixth.
The operator-facing log, which the plan required to be legible:
```
[12:52:02 WRN] Deployment 37d723018e374c0eb04c7b6beccc3b3e timed out after 00:02:00 (5/6 acks
landed). No ack from: site-b-2:4053. Each is an enabled ClusterNode row — start the node, or set
ClusterNode.Enabled = 0 if it is down for maintenance, then redeploy
```
(The remedy in that message was corrected to `MaintenanceMode = 1` — see step 4.)
## Step 4 — the escape hatch → **FAILED, then fixed and re-run → PASS**
### First attempt: the hatch did not exist
`UPDATE ClusterNode SET Enabled = 0 WHERE NodeId = 'site-b-2:4053'`, then deploy:
```
HTTP 422
[ClusterEnabledNodeCountMismatch] Cluster 'SITE-B' declares NodeCount=2 but has 1 Enabled nodes.
Toggle the missing node(s) back on or change RedundancyMode/NodeCount to match.
```
`DraftValidator.ValidateClusterTopology` rejects the deployment **before it dispatches**. So
`Enabled = 0` cannot be used on a node of a 2-node cluster — and every cluster on this rig, and every
cluster in the program plan's target topology, is a 2-node pair.
**This was a blocker, not a wrinkle.** The plan called step 4 *"the check that makes step 3
acceptable to ship"*. Without it, a node down for maintenance blocks every deployment to its cluster,
with no remedy short of editing `NodeCount` / `RedundancyMode` — which changes the runtime redundancy
posture to work around a deploy gate.
The two rules were each reasonable and contradictory together: the validator reads `Enabled` as *"part
of the declared topology"*, Phase 1 additionally read it as *"expect an ack"*.
### Fix (Task 7): split the meanings
New `ClusterNode.MaintenanceMode bit NOT NULL DEFAULT 0`. `Enabled` keeps its topology meaning and the
validator is untouched; the coordinator's expected-ack set and the address reconciler both filter on
`Enabled && !MaintenanceMode`.
### Re-run: PASS
`docker stop` `site-b-2` + `MaintenanceMode = 1`, then deploy `82c5e678`:
- `202 Accepted` — the topology gate passes, because the node is still `Enabled` (`1|1`).
- `Deployment.Status = 2` (Sealed).
- **Five** `NodeDeploymentState` rows, all `Applied`. `site-b-2:4053` has no row at all — excluded
from the expected set rather than waited for and forgiven.
## Task 4 (reconciler) — live behaviour, not a planned gate step
Observed unprompted during the rolling restart, which is better evidence than a staged check:
```
[12:44:49 WRN] ClusterNode address check [EnabledRowNotInCluster] site-a-1:4053: ...
[12:44:49 WRN] ... site-a-2:4053 ... site-b-1:4053 ... site-b-2:4053
[12:44:56 INF] ClusterNode addresses reconcile with cluster membership (6 driver members)
```
It warned per row while the site nodes were still starting, then converged to a single clean line —
and **logged nothing further**, confirming the change-detection guard suppresses repeats on the
5-minute sweep.
Maintenance interaction, from the step 4 re-run:
```
[13:06:02 WRN] ... site-b-2:4053 ... ← node dropped, flag not yet set
[13:06:05 INF] ... reconcile with cluster membership (5 driver members) ← flag set: silent
```
The transient warning is honest — for those three seconds the row genuinely was enabled, present in
the expected-ack set, and absent from the cluster.
## What this gate did not cover
- **Phase 2's premise.** `AkkaPort` / `GrpcPort` are groundwork; nothing dials them yet, so "the value
is correct" is only checked by the reconciler comparing it to gossip. The first real exercise is
Phase 2.
- **The reconciler under split meshes.** On the current single mesh an admin node sees every driver
member. After Phase 6 it will not, and every site row would report `EnabledRowNotInCluster`
permanently. Recorded as a known limitation on the class and in the design doc.
- **`Enabled = 0` on a single-node (`RedundancyMode.None`) cluster.** Untested — the rig has no such
cluster. It would fail the same topology rule (0 enabled vs `NodeCount = 1`), so `MaintenanceMode`
is the hatch there too.
@@ -82,7 +82,13 @@ per phase:** (1) invoke writing-plans in this repo to produce
references, exploring current code first; (2) execute it task-by-task; (3) run the phase's exit references, exploring current code first; (2) execute it task-by-task; (3) run the phase's exit
gate; (4) update this file's status column and the design doc's §7 table. gate; (4) update this file's status column and the design doc's §7 table.
### Phase 1 — `ClusterNode` address columns + DB-sourced ack set ### Phase 1 — `ClusterNode` address columns + DB-sourced ack set — **DONE 2026-07-22**
Shipped per `2026-07-21-per-cluster-mesh-phase1.md`; see that plan's Task 6 gate record and the
design doc's "Phase 1 as shipped" note for the two scope deviations (no cluster-scope filtering — no
such deployment exists; no per-node role column) and the one addition (`ClusterNodeAddressReconciler`).
**AdminUI node edit was deferred to Phase 2** — nothing reads the columns until then, and the
migration default plus the rig seed cover every node today.
**Scope:** `ClusterNode` gains Akka + gRPC address columns (mirroring ScadaBridge's `Site` **Scope:** `ClusterNode` gains Akka + gRPC address columns (mirroring ScadaBridge's `Site`
entity `NodeAAddress`/`GrpcNodeAAddress` pattern, but per-node rows); EF migration; AdminUI node entity `NodeAAddress`/`GrpcNodeAAddress` pattern, but per-node rows); EF migration; AdminUI node
edit surfaces the fields; `ConfigPublishCoordinator` derives its expected-ack set from edit surfaces the fields; `ConfigPublishCoordinator` derives its expected-ack set from
+46
View File
@@ -148,10 +148,13 @@ CREATE TABLE dbo.ClusterNode (
Host nvarchar(255) NOT NULL, Host nvarchar(255) NOT NULL,
OpcUaPort int NOT NULL DEFAULT 4840, OpcUaPort int NOT NULL DEFAULT 4840,
DashboardPort int NOT NULL DEFAULT 8081, DashboardPort int NOT NULL DEFAULT 8081,
AkkaPort int NOT NULL DEFAULT 4053,
GrpcPort int NULL,
ApplicationUri nvarchar(256) NOT NULL, ApplicationUri nvarchar(256) NOT NULL,
ServiceLevelBase tinyint NOT NULL DEFAULT 200, ServiceLevelBase tinyint NOT NULL DEFAULT 200,
DriverConfigOverridesJson nvarchar(max) NULL CHECK (DriverConfigOverridesJson IS NULL OR ISJSON(DriverConfigOverridesJson) = 1), DriverConfigOverridesJson nvarchar(max) NULL CHECK (DriverConfigOverridesJson IS NULL OR ISJSON(DriverConfigOverridesJson) = 1),
Enabled bit NOT NULL DEFAULT 1, Enabled bit NOT NULL DEFAULT 1,
MaintenanceMode bit NOT NULL DEFAULT 0,
LastSeenAt datetime2(3) NULL, LastSeenAt datetime2(3) NULL,
CreatedAt datetime2(3) NOT NULL DEFAULT SYSUTCDATETIME(), CreatedAt datetime2(3) NOT NULL DEFAULT SYSUTCDATETIME(),
CreatedBy nvarchar(128) NOT NULL CreatedBy nvarchar(128) NOT NULL
@@ -167,6 +170,49 @@ CREATE UNIQUE INDEX UX_ClusterNode_Primary_Per_Cluster
WHERE RedundancyRole = 'Primary'; WHERE RedundancyRole = 'Primary';
``` ```
#### `AkkaPort` / `GrpcPort` — central's dial targets (per-cluster mesh Phase 1)
These are **the addresses central dials**, not the node's own binding configuration. The node binds
from `Cluster:Port` in its own appsettings; these columns duplicate that value for a reader that
cannot see the node's config. Today central shares a gossip ring with every node and does not need
them — [per-cluster mesh Phase 2](../plans/2026-07-21-per-cluster-mesh-design.md) splits the fleet
into one mesh per `Cluster`, after which `Host` + `AkkaPort` is how central builds its ClusterClient
contact points, and `GrpcPort` is the Phase 5 telemetry stream.
`AkkaPort` is `NOT NULL DEFAULT 4053` because every node listens on a remoting port — `0` is never a
truthful value, and rows predating the column must migrate to something real. `GrpcPort` is nullable
with **no** default because nothing listens on it until Phase 5, and a non-null default would assert
a port that does not exist.
**The duplication against `Cluster:Port` is not enforced by the schema.** A node that binds 4054 while
its row says 4053 is unreachable from central in Phase 2, and the symptom there is a silent absence of
acks rather than an error. `ClusterNodeAddressReconcilerActor` (an admin-role singleton) compares the
rows against observed cluster membership and logs an Error on mismatch. It reads membership rather
than having each driver node assert its own row, because Phase 4 removes the driver nodes' ConfigDb
connection entirely.
#### `Enabled` vs `MaintenanceMode` — two flags, two questions
Phase 1 made the rows the deploy path's **expected-ack set**: `ConfigPublishCoordinator` no longer
derives it from cluster membership, so a row whose node is down now fails the deployment at the apply
deadline instead of letting it seal green without that node.
The escape hatch is **`MaintenanceMode = 1`**, not `Enabled = 0`:
| Flag | Question it answers | Checked by |
|---|---|---|
| `Enabled` | Is this node part of the cluster's declared topology? | `DraftValidator.ValidateClusterTopology` — the enabled-node count must equal `ServerCluster.NodeCount` |
| `MaintenanceMode` | Should we expect it to participate right now? | `ConfigPublishCoordinator` (expected-ack set) and `ClusterNodeAddressReconciler` (absence warnings) |
`Enabled = 0` on one node of a Warm/Hot pair is **rejected at the deploy gate** with
`ClusterEnabledNodeCountMismatch` — the enabled count drops to 1 while `NodeCount` stays 2. Since
every cluster in the target topology is a 2-node pair, `Enabled` cannot serve as the maintenance
hatch at all. The Phase 1 live gate found this; the two rules were independently reasonable and
contradictory together, so the meanings were split rather than either rule being weakened.
A node in maintenance is still enabled, so the topology gate still passes, and the runtime redundancy
posture (`NodeCount` / `RedundancyMode`) is unchanged.
`DriverConfigOverridesJson` shape: `DriverConfigOverridesJson` shape:
```jsonc ```jsonc
@@ -18,6 +18,33 @@ public sealed class ClusterNode
/// <summary>The dashboard HTTP port (default 8081).</summary> /// <summary>The dashboard HTTP port (default 8081).</summary>
public int DashboardPort { get; set; } = 8081; public int DashboardPort { get; set; } = 8081;
/// <summary>
/// Akka remoting port (default 4053). <b>This is central's dial target, not the node's own
/// binding configuration</b> — the node binds from <c>Cluster:Port</c> in its own appsettings,
/// and this column duplicates that value for a reader that cannot see the node's config.
/// Per-cluster mesh Phase 2 builds its ClusterClient contact points from
/// <see cref="Host"/> + this port, at which point central and the site node no longer share a
/// gossip ring and central has no other way to learn it.
/// </summary>
/// <remarks>
/// The duplication is real and unenforced by the schema. <c>ClusterNodeAddressReconciler</c>
/// on the admin node compares this row against observed cluster membership and logs an Error
/// on mismatch — a node that binds 4054 while its row says 4053 is unreachable from central in
/// Phase 2, and the symptom there is a silent absence of acks rather than an error.
/// </remarks>
public int AkkaPort { get; set; } = 4053;
/// <summary>
/// gRPC port central dials for the Phase 5 telemetry stream, or <see langword="null"/> when the
/// node exposes none. Also central's dial target rather than the node's binding config.
/// </summary>
/// <remarks>
/// Nullable by intent: nothing listens on this port yet, and a non-null default would assert a
/// port that does not exist. Phase 5 populates it; until then <see langword="null"/> is the
/// honest value.
/// </remarks>
public int? GrpcPort { get; set; }
/// <summary> /// <summary>
/// OPC UA <c>ApplicationUri</c> — MUST be unique per node per OPC UA spec. Clients pin trust here. /// OPC UA <c>ApplicationUri</c> — MUST be unique per node per OPC UA spec. Clients pin trust here.
/// Fleet-wide unique index enforces no two nodes share a value. /// Fleet-wide unique index enforces no two nodes share a value.
@@ -36,8 +63,37 @@ public sealed class ClusterNode
public string? DriverConfigOverridesJson { get; set; } public string? DriverConfigOverridesJson { get; set; }
/// <summary>Gets or sets a value indicating whether this node is enabled.</summary> /// <summary>Gets or sets a value indicating whether this node is enabled.</summary>
/// <remarks>
/// <b>Part of the declared topology.</b> <c>DraftValidator.ValidateClusterTopology</c> requires
/// the enabled-node count to equal <see cref="ServerCluster.NodeCount"/>, so disabling one node
/// of a Warm/Hot pair is a deploy-blocking validation error by design — it would boot the
/// runtime into the InvalidTopology band. To take a node out of service temporarily, use
/// <see cref="MaintenanceMode"/> instead.
/// </remarks>
public bool Enabled { get; set; } = true; public bool Enabled { get; set; } = true;
/// <summary>
/// Node is temporarily out of service: still part of the cluster's declared topology, but not
/// expected to participate. A deployment does not wait for its ack, and the address reconciler
/// does not warn that it is absent.
/// </summary>
/// <remarks>
/// <para>
/// Separate from <see cref="Enabled"/> because the two answer different questions.
/// <see cref="Enabled"/> means "this node is part of the declared topology" and is checked
/// against <see cref="ServerCluster.NodeCount"/>; this flag means "do not expect it right
/// now". Per-cluster mesh Phase 1 made an enabled row's ack mandatory, so without this a
/// node down for maintenance would block every deployment to its cluster — and clearing
/// <see cref="Enabled"/> to escape that is itself rejected by the topology gate on any
/// 2-node pair, which is every cluster in the target deployment.
/// </para>
/// <para>
/// Found by the Phase 1 live gate: the two rules were independently reasonable and
/// contradictory together.
/// </para>
/// </remarks>
public bool MaintenanceMode { get; set; }
/// <summary>Gets or sets the timestamp when this node was last seen.</summary> /// <summary>Gets or sets the timestamp when this node was last seen.</summary>
public DateTime? LastSeenAt { get; set; } public DateTime? LastSeenAt { get; set; }
@@ -0,0 +1,39 @@
using Microsoft.EntityFrameworkCore.Migrations;
#nullable disable
namespace ZB.MOM.WW.OtOpcUa.Configuration.Migrations
{
/// <inheritdoc />
public partial class AddClusterNodeTransportPorts : Migration
{
/// <inheritdoc />
protected override void Up(MigrationBuilder migrationBuilder)
{
migrationBuilder.AddColumn<int>(
name: "AkkaPort",
table: "ClusterNode",
type: "int",
nullable: false,
defaultValue: 4053);
migrationBuilder.AddColumn<int>(
name: "GrpcPort",
table: "ClusterNode",
type: "int",
nullable: true);
}
/// <inheritdoc />
protected override void Down(MigrationBuilder migrationBuilder)
{
migrationBuilder.DropColumn(
name: "AkkaPort",
table: "ClusterNode");
migrationBuilder.DropColumn(
name: "GrpcPort",
table: "ClusterNode");
}
}
}
@@ -0,0 +1,29 @@
using Microsoft.EntityFrameworkCore.Migrations;
#nullable disable
namespace ZB.MOM.WW.OtOpcUa.Configuration.Migrations
{
/// <inheritdoc />
public partial class AddClusterNodeMaintenanceMode : Migration
{
/// <inheritdoc />
protected override void Up(MigrationBuilder migrationBuilder)
{
migrationBuilder.AddColumn<bool>(
name: "MaintenanceMode",
table: "ClusterNode",
type: "bit",
nullable: false,
defaultValue: false);
}
/// <inheritdoc />
protected override void Down(MigrationBuilder migrationBuilder)
{
migrationBuilder.DropColumn(
name: "MaintenanceMode",
table: "ClusterNode");
}
}
}
@@ -47,6 +47,11 @@ namespace ZB.MOM.WW.OtOpcUa.Configuration.Migrations
.HasMaxLength(64) .HasMaxLength(64)
.HasColumnType("nvarchar(64)"); .HasColumnType("nvarchar(64)");
b.Property<int>("AkkaPort")
.ValueGeneratedOnAdd()
.HasColumnType("int")
.HasDefaultValue(4053);
b.Property<string>("ApplicationUri") b.Property<string>("ApplicationUri")
.IsRequired() .IsRequired()
.HasMaxLength(256) .HasMaxLength(256)
@@ -76,6 +81,9 @@ namespace ZB.MOM.WW.OtOpcUa.Configuration.Migrations
b.Property<bool>("Enabled") b.Property<bool>("Enabled")
.HasColumnType("bit"); .HasColumnType("bit");
b.Property<int?>("GrpcPort")
.HasColumnType("int");
b.Property<string>("Host") b.Property<string>("Host")
.IsRequired() .IsRequired()
.HasMaxLength(255) .HasMaxLength(255)
@@ -84,6 +92,9 @@ namespace ZB.MOM.WW.OtOpcUa.Configuration.Migrations
b.Property<DateTime?>("LastSeenAt") b.Property<DateTime?>("LastSeenAt")
.HasColumnType("datetime2(3)"); .HasColumnType("datetime2(3)");
b.Property<bool>("MaintenanceMode")
.HasColumnType("bit");
b.Property<int>("OpcUaPort") b.Property<int>("OpcUaPort")
.HasColumnType("int"); .HasColumnType("int");
@@ -142,6 +142,11 @@ public sealed class OtOpcUaConfigDbContext(DbContextOptions<OtOpcUaConfigDbConte
e.Property(x => x.Host).HasMaxLength(255); e.Property(x => x.Host).HasMaxLength(255);
e.Property(x => x.ApplicationUri).HasMaxLength(256); e.Property(x => x.ApplicationUri).HasMaxLength(256);
e.Property(x => x.DriverConfigOverridesJson).HasColumnType("nvarchar(max)"); e.Property(x => x.DriverConfigOverridesJson).HasColumnType("nvarchar(max)");
// Central's dial targets (per-cluster mesh Phase 1). AkkaPort carries a DB-side default so
// rows migrated from before the column existed come out at 4053 rather than 0 — every node
// in the fleet does listen on a remoting port, so 0 is never a truthful value for it.
// GrpcPort has no default: nothing listens on it until Phase 5.
e.Property(x => x.AkkaPort).HasDefaultValue(4053);
e.Property(x => x.LastSeenAt).HasColumnType("datetime2(3)"); e.Property(x => x.LastSeenAt).HasColumnType("datetime2(3)");
e.Property(x => x.CreatedAt).HasColumnType("datetime2(3)").HasDefaultValueSql("SYSUTCDATETIME()"); e.Property(x => x.CreatedAt).HasColumnType("datetime2(3)").HasDefaultValueSql("SYSUTCDATETIME()");
e.Property(x => x.CreatedBy).HasMaxLength(128); e.Property(x => x.CreatedBy).HasMaxLength(128);
@@ -1,5 +1,4 @@
using Akka.Actor; using Akka.Actor;
using Akka.Cluster;
using Akka.Cluster.Tools.PublishSubscribe; using Akka.Cluster.Tools.PublishSubscribe;
using Akka.Event; using Akka.Event;
using Microsoft.EntityFrameworkCore; using Microsoft.EntityFrameworkCore;
@@ -18,8 +17,40 @@ namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Coordinators;
/// has acked Applied. Per-node ACKs are persisted in <c>NodeDeploymentState</c> so a failover of /// has acked Applied. Per-node ACKs are persisted in <c>NodeDeploymentState</c> so a failover of
/// this singleton can recover in-flight state from the DB. /// this singleton can recover in-flight state from the DB.
/// ///
/// Discovery of the "expected ACK set" comes from <c>Akka.Cluster.State.Members</c> filtered by /// Discovery of the "expected ACK set" comes from the <b>enabled <c>ClusterNode</c> rows</b>, not
/// the <c>driver</c> role — the DB does not own per-node role assignment. /// from <c>Akka.Cluster.State.Members</c> (per-cluster mesh Phase 1). Central must be able to name
/// the nodes a deployment is for without sharing a gossip ring with them — Phase 2 splits the fleet
/// into one mesh per <c>Cluster</c>, after which central can no longer see a site node's membership
/// at all. This was the coordinator's last genuinely mesh-bound dependency.
///
/// <para>
/// <b>Behaviour change:</b> a configured node that is switched off is now <i>expected</i>, so a
/// deployment dispatched while it is down fails at the apply deadline instead of sealing green
/// without it. That is deliberate — under the membership rule the operator was told the fleet was
/// deployed when it was not. <c>ClusterNode.MaintenanceMode = 1</c> is the escape hatch for a node
/// taken down for maintenance.
/// </para>
/// <para>
/// <b>Not <c>Enabled = 0</c>.</b> That was the intended hatch until the Phase 1 live gate found it
/// unusable: <c>DraftValidator.ValidateClusterTopology</c> requires the enabled-node count to equal
/// <c>ServerCluster.NodeCount</c>, so clearing <c>Enabled</c> on either node of a Warm/Hot pair
/// rejects the deployment outright — and every cluster in the target topology is a pair. The two
/// rules were independently reasonable and contradictory together, so the meanings were split:
/// <c>Enabled</c> = part of the declared topology, <c>MaintenanceMode</c> = do not expect it now.
/// </para>
/// <para>
/// <b>Every <c>ClusterNode</c> row is assumed to be a driver node.</b> The membership rule filtered
/// on the <c>driver</c> role; the DB has no per-node role column, and deliberately does not gain one
/// here (a second declaration of node roles would drift from <c>Cluster:Roles</c> in the node's own
/// appsettings). The assumption holds because the entity's whole shape — <c>Host</c>,
/// <c>OpcUaPort</c>, <c>ApplicationUri</c>, <c>ServiceLevelBase</c> — describes an OPC UA server
/// node. An admin-only node must not be given a <c>ClusterNode</c> row: it would be expected to ack
/// a deployment it has no <c>DriverHostActor</c> to apply, and every deploy would time out.
/// </para>
/// <para>
/// <c>ServerCluster.Enabled</c> is <b>not</b> consulted — nothing else in the codebase consults it,
/// so honouring it here would invent semantics. Disable the nodes, not the cluster row.
/// </para>
/// </summary> /// </summary>
public sealed class ConfigPublishCoordinator : ReceiveActor, IWithTimers public sealed class ConfigPublishCoordinator : ReceiveActor, IWithTimers
{ {
@@ -119,11 +150,15 @@ public sealed class ConfigPublishCoordinator : ReceiveActor, IWithTimers
{ {
_current = msg.DeploymentId; _current = msg.DeploymentId;
_acks.Clear(); _acks.Clear();
_expectedAcks = DiscoverDriverNodes();
// Seed NodeDeploymentState rows so a failover knows which nodes were expected to ack. // Seed NodeDeploymentState rows so a failover knows which nodes were expected to ack.
using (var db = _dbFactory.CreateDbContext()) using (var db = _dbFactory.CreateDbContext())
{ {
// Discovery reuses this context rather than opening a second one — it reads the same
// ClusterNode table the NodeDeploymentState rows below are FK-bound to, so a single
// context keeps the expected set and the seeded rows consistent by construction.
_expectedAcks = DiscoverDriverNodes(db);
foreach (var node in _expectedAcks) foreach (var node in _expectedAcks)
{ {
db.NodeDeploymentStates.Add(new NodeDeploymentState db.NodeDeploymentStates.Add(new NodeDeploymentState
@@ -142,9 +177,14 @@ public sealed class ConfigPublishCoordinator : ReceiveActor, IWithTimers
if (_expectedAcks.Count == 0) if (_expectedAcks.Count == 0)
{ {
// No driver-role members. Seal immediately — the alternative is hanging forever // No enabled ClusterNode rows. Seal immediately — the alternative is hanging forever
// waiting for ACKs that will never come. // waiting for ACKs that will never come. Note this is now a *configuration* error
_log.Warning("DispatchDeployment {Id}: no driver-role members in cluster; sealing empty", // rather than a topology observation: under the old membership rule an empty set meant
// "no driver nodes have joined yet", which could resolve itself; an empty set here means
// the fleet has no enabled nodes registered, which cannot.
_log.Warning(
"DispatchDeployment {Id}: no enabled ClusterNode rows registered; sealing empty. " +
"Nothing will receive this deployment — check the fleet's node configuration",
msg.DeploymentId); msg.DeploymentId);
SealDeployment(); SealDeployment();
} }
@@ -233,8 +273,15 @@ public sealed class ConfigPublishCoordinator : ReceiveActor, IWithTimers
using var db = _dbFactory.CreateDbContext(); using var db = _dbFactory.CreateDbContext();
UpdateDeploymentStatus(db, _current.Value, DeploymentStatus.TimedOut); UpdateDeploymentStatus(db, _current.Value, DeploymentStatus.TimedOut);
db.SaveChanges(); db.SaveChanges();
_log.Warning("Deployment {Id} timed out after {Deadline} ({Acked}/{Total} acks landed)", // Name the silent nodes, not just the counts. Since the expected set became DB-derived this
_current.Value, _applyDeadline, _acks.Count, _expectedAcks.Count); // is the failure an operator meets after deploying while a configured node is switched off,
// so it has to say which node — "4/5 acks landed" leaves them reading logs on five machines.
var missing = _expectedAcks.Where(n => !_acks.ContainsKey(n)).Select(n => n.Value).Order().ToList();
_log.Warning(
"Deployment {Id} timed out after {Deadline} ({Acked}/{Total} acks landed). " +
"No ack from: {MissingNodes}. Each is an enabled, non-maintenance ClusterNode row — start " +
"the node, or set ClusterNode.MaintenanceMode = 1 if it is down for maintenance, then redeploy",
_current.Value, _applyDeadline, _acks.Count, _expectedAcks.Count, string.Join(", ", missing));
ResetForNext(); ResetForNext();
} }
@@ -259,22 +306,29 @@ public sealed class ConfigPublishCoordinator : ReceiveActor, IWithTimers
if (sealNow) d.SealedAtUtc = DateTime.UtcNow; if (sealNow) d.SealedAtUtc = DateTime.UtcNow;
} }
private HashSet<NodeId> DiscoverDriverNodes() /// <summary>
{ /// The set of nodes expected to ack this deployment: every <b>enabled</b> <c>ClusterNode</c>
var cluster = Akka.Cluster.Cluster.Get(Context.System); /// row. See the class remarks for why this is DB-derived rather than membership-derived, and
var nodes = new HashSet<NodeId>(NodeIdComparer); /// for the two assumptions it rests on (every row is a driver node; <c>Enabled</c> is the
foreach (var member in cluster.State.Members) /// maintenance hatch).
{ /// </summary>
if (member.Status is not (MemberStatus.Up or MemberStatus.Joining)) continue; /// <remarks>
if (!member.Roles.Contains("driver")) continue; /// <c>ClusterNode.NodeId</c> is already in the <c>host:port</c> identifier space the
var host = member.Address.Host; /// membership rule derived — the rig seeds <c>central-1:4053</c> and so on, and
if (string.IsNullOrWhiteSpace(host)) continue; /// <c>NodeDeploymentState.NodeId</c> is FK-bound to this column, so the fact that the
// Match ClusterRoleInfo's NodeId derivation (host:port) so DriverHostActor's /// membership-derived rows have always satisfied that FK is standing proof the two sets
// self-identification and the coordinator's expected-ack set agree. /// agree. It also has to stay that way: <c>DriverHostActor</c> self-identifies from
nodes.Add(NodeId.Parse($"{host}:{member.Address.Port ?? 0}")); /// <c>ClusterRoleInfo</c>'s <c>host:port</c>, so a row whose NodeId is anything else is a
} /// node whose acks will never match its expected-ack entry.
return nodes; /// </remarks>
} private static HashSet<NodeId> DiscoverDriverNodes(OtOpcUaConfigDbContext db) =>
db.ClusterNodes
.AsNoTracking()
.Where(n => n.Enabled && !n.MaintenanceMode)
.Select(n => n.NodeId)
.ToList()
.Select(NodeId.Parse)
.ToHashSet(NodeIdComparer);
/// <summary>Case-insensitive <see cref="NodeId"/> equality (by <see cref="NodeId.Value"/>), /// <summary>Case-insensitive <see cref="NodeId"/> equality (by <see cref="NodeId.Value"/>),
/// matching the case-insensitive scoping in <c>DeploymentArtifact.ResolveClusterScope</c> so the /// matching the case-insensitive scoping in <c>DeploymentArtifact.ResolveClusterScope</c> so the
@@ -0,0 +1,138 @@
namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Fleet;
/// <summary>What kind of disagreement was found between a node's real address and its row.</summary>
public enum AddressMismatchKind
{
/// <summary>
/// The row was matched to a live member by <c>NodeId</c>, but its <c>Host</c> / <c>AkkaPort</c>
/// columns name a different address. Phase 2 dials those columns, so central would dial
/// somewhere the node is not — and the symptom there is a silent absence of acks.
/// </summary>
RowDialTargetDisagrees,
/// <summary>
/// A driver-role member is in the cluster with no enabled <c>ClusterNode</c> row at its
/// address. Its deployment acks are discarded (see <c>ConfigPublishCoordinator</c>), so it
/// silently receives deployments it is never credited for.
/// </summary>
RunningNodeHasNoRow,
/// <summary>
/// An enabled row names a node that is not currently a driver-role member. Either it is down
/// — in which case the next deployment fails at the apply deadline — or it binds an address
/// other than the one its row claims.
/// </summary>
EnabledRowNotInCluster,
}
/// <summary>One disagreement, with enough detail to act on without opening the DB.</summary>
/// <param name="Kind">Which disagreement this is.</param>
/// <param name="NodeId">The <c>host:port</c> identity the finding is about.</param>
/// <param name="Detail">Human-readable specifics, including what to do about it.</param>
public sealed record ClusterNodeAddressMismatch(AddressMismatchKind Kind, string NodeId, string Detail);
/// <summary>A <c>ClusterNode</c> row reduced to the address facts this check cares about.</summary>
/// <param name="NodeId">The row's primary key, in the <c>host:port</c> space.</param>
/// <param name="Host">The host central will dial in Phase 2.</param>
/// <param name="AkkaPort">The remoting port central will dial in Phase 2.</param>
public sealed record ClusterNodeAddress(string NodeId, string Host, int AkkaPort);
/// <summary>
/// Compares the addresses driver nodes actually occupy against the <c>ClusterNode</c> rows that
/// claim to describe them.
/// </summary>
/// <remarks>
/// <para>
/// <b>Why this exists.</b> <c>ClusterNode.AkkaPort</c> and the node's own <c>Cluster:Port</c>
/// are the same fact stored in two places, and nothing makes them agree. A node that binds
/// 4054 while its row says 4053 is unreachable from central once Phase 2 dials the row
/// instead of gossiping — and an unenforced duplicated address is the exact shape of the
/// <c>Modbus</c>/<c>ModbusTcp</c> and <c>TwinCat</c>/<c>Focas</c> drifts already recorded in
/// this repo. Since Phase 1 also made the rows the deploy path's expected-ack set, a drifted
/// row already costs a failed deployment today.
/// </para>
/// <para>
/// <b>It runs on an admin node</b>, comparing rows against the membership an admin node can
/// already see, rather than having each driver node assert its own row — Phase 4 removes the
/// driver nodes' ConfigDb connection entirely, so a self-assertion written there would have
/// to be deleted again.
/// </para>
/// <para>
/// <b>Phase 2 must revisit this.</b> Once the fleet splits into one mesh per cluster, an
/// admin node no longer sees site members at all, and every site row would report
/// <see cref="AddressMismatchKind.EnabledRowNotInCluster"/> forever. The check then has to
/// move to whatever transport replaces gossip — or be scoped to the central mesh.
/// </para>
/// </remarks>
public static class ClusterNodeAddressReconciler
{
/// <summary>
/// Reconciles observed driver-role member addresses against the configured rows.
/// </summary>
/// <param name="observedDriverMembers">
/// <c>host</c>/<c>port</c> of every Up member carrying the <c>driver</c> role. Admin-only
/// members must be excluded — they legitimately have no <c>ClusterNode</c> row, and including
/// them would report a mismatch for correct configuration.
/// </param>
/// <param name="rows">Every <c>ClusterNode</c> row, enabled or not.</param>
/// <param name="expectedPresentNodeIds">
/// The subset of <paramref name="rows"/> that should currently be in the cluster: enabled and
/// not in maintenance. Excluded rows are deliberately still matched against membership (a
/// disabled row for a running node is worth knowing) but never reported as
/// <see cref="AddressMismatchKind.EnabledRowNotInCluster"/> — being absent is the whole point
/// of disabling one or putting it in maintenance.
/// </param>
/// <returns>Every disagreement found, ordered by node id for stable logging.</returns>
public static IReadOnlyList<ClusterNodeAddressMismatch> Reconcile(
IReadOnlyCollection<(string Host, int Port)> observedDriverMembers,
IReadOnlyCollection<ClusterNodeAddress> rows,
IReadOnlyCollection<string> expectedPresentNodeIds)
{
var byNodeId = new Dictionary<string, ClusterNodeAddress>(StringComparer.OrdinalIgnoreCase);
foreach (var row in rows) byNodeId[row.NodeId] = row;
var expectedPresent = new HashSet<string>(expectedPresentNodeIds, StringComparer.OrdinalIgnoreCase);
var findings = new List<ClusterNodeAddressMismatch>();
var seen = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
foreach (var (host, port) in observedDriverMembers)
{
var nodeId = $"{host}:{port}";
seen.Add(nodeId);
if (!byNodeId.TryGetValue(nodeId, out var row))
{
findings.Add(new ClusterNodeAddressMismatch(
AddressMismatchKind.RunningNodeHasNoRow, nodeId,
$"driver node is in the cluster at {nodeId} but no ClusterNode row has that NodeId; " +
"its deployment acks are discarded. Either register the row or correct the node's " +
"Cluster:PublicHostname / Cluster:Port to match an existing one"));
continue;
}
if (!string.Equals(row.Host, host, StringComparison.OrdinalIgnoreCase) || row.AkkaPort != port)
{
findings.Add(new ClusterNodeAddressMismatch(
AddressMismatchKind.RowDialTargetDisagrees, nodeId,
$"node is at {nodeId} but its row's dial target is {row.Host}:{row.AkkaPort}; " +
"central will dial the wrong address once Phase 2 stops using gossip. " +
"Correct ClusterNode.Host / ClusterNode.AkkaPort"));
}
}
foreach (var row in rows)
{
if (seen.Contains(row.NodeId) || !expectedPresent.Contains(row.NodeId)) continue;
findings.Add(new ClusterNodeAddressMismatch(
AddressMismatchKind.EnabledRowNotInCluster, row.NodeId,
$"enabled ClusterNode row {row.NodeId} has no matching driver member; the next " +
"deployment will fail at the apply deadline waiting for it. Start the node, or set " +
"ClusterNode.MaintenanceMode = 1 while it is down for maintenance"));
}
return findings
.OrderBy(f => f.NodeId, StringComparer.OrdinalIgnoreCase)
.ThenBy(f => f.Kind)
.ToList();
}
}
@@ -0,0 +1,136 @@
using Akka.Actor;
using Akka.Cluster;
using Akka.Event;
using Microsoft.EntityFrameworkCore;
using ZB.MOM.WW.OtOpcUa.Configuration;
namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Fleet;
/// <summary>
/// Admin-role cluster singleton that runs <see cref="ClusterNodeAddressReconciler"/> against live
/// membership and logs what it finds. A singleton rather than a per-admin-node actor so a
/// two-admin fleet reports each disagreement once.
/// </summary>
/// <remarks>
/// Recomputes on membership change (debounced) and on a slow periodic sweep, and <b>logs only
/// when the finding set changes</b>. A node legitimately down would otherwise reprint the same
/// warning every sweep until someone silences it by ignoring the log, which is the failure mode
/// this check exists to avoid.
/// </remarks>
public sealed class ClusterNodeAddressReconcilerActor : ReceiveActor, IWithTimers
{
/// <summary>Driver role name — only members carrying it are expected to have a ClusterNode row.</summary>
public const string DriverRole = "driver";
/// <summary>Collapses a burst of membership events (a rolling restart) into one reconcile.</summary>
public static readonly TimeSpan DebounceWindow = TimeSpan.FromSeconds(2);
/// <summary>Periodic sweep, so a row edited in the AdminUI is checked without a topology change.</summary>
public static readonly TimeSpan DefaultSweepInterval = TimeSpan.FromMinutes(5);
private readonly IDbContextFactory<OtOpcUaConfigDbContext> _dbFactory;
private readonly Akka.Cluster.Cluster _cluster;
private readonly TimeSpan _sweepInterval;
private readonly ILoggingAdapter _log = Context.GetLogger();
private IReadOnlyList<ClusterNodeAddressMismatch>? _lastReported;
/// <summary>Gets the timer scheduler for this actor.</summary>
public ITimerScheduler Timers { get; set; } = null!;
/// <summary>Creates Props for the reconciler.</summary>
/// <param name="dbFactory">Factory for the config database context.</param>
/// <param name="sweepInterval">Periodic sweep interval; defaults to <see cref="DefaultSweepInterval"/>.</param>
/// <returns>Props for actor creation.</returns>
public static Props Props(
IDbContextFactory<OtOpcUaConfigDbContext> dbFactory, TimeSpan? sweepInterval = null) =>
Akka.Actor.Props.Create(() => new ClusterNodeAddressReconcilerActor(dbFactory, sweepInterval));
/// <summary>Initializes a new instance of the <see cref="ClusterNodeAddressReconcilerActor"/> class.</summary>
/// <param name="dbFactory">Factory for the config database context.</param>
/// <param name="sweepInterval">Periodic sweep interval; defaults to <see cref="DefaultSweepInterval"/>.</param>
public ClusterNodeAddressReconcilerActor(
IDbContextFactory<OtOpcUaConfigDbContext> dbFactory, TimeSpan? sweepInterval = null)
{
_dbFactory = dbFactory;
_cluster = Akka.Cluster.Cluster.Get(Context.System);
_sweepInterval = sweepInterval ?? DefaultSweepInterval;
Receive<ClusterEvent.IMemberEvent>(_ => ScheduleReconcile());
Receive<ClusterEvent.CurrentClusterState>(_ => ScheduleReconcile());
Receive<ReconcileNow>(_ => Reconcile());
}
/// <inheritdoc />
protected override void PreStart()
{
_cluster.Subscribe(Self, ClusterEvent.InitialStateAsEvents, typeof(ClusterEvent.IMemberEvent));
Timers.StartPeriodicTimer("sweep", ReconcileNow.Instance, _sweepInterval, _sweepInterval);
}
/// <inheritdoc />
protected override void PostStop() => _cluster.Unsubscribe(Self);
private void ScheduleReconcile() =>
Timers.StartSingleTimer("debounce", ReconcileNow.Instance, DebounceWindow);
private void Reconcile()
{
var observed = _cluster.State.Members
.Where(m => m.Status == MemberStatus.Up && m.Roles.Contains(DriverRole))
.Select(m => (Host: m.Address.Host ?? string.Empty, Port: m.Address.Port ?? 0))
.Where(a => !string.IsNullOrWhiteSpace(a.Host) && a.Port > 0)
.ToList();
List<ClusterNodeAddress> rows;
List<string> enabled;
try
{
using var db = _dbFactory.CreateDbContext();
rows = db.ClusterNodes.AsNoTracking()
.Select(n => new ClusterNodeAddress(n.NodeId, n.Host, n.AkkaPort))
.ToList();
// Same predicate the coordinator's expected-ack set uses — a node the deploy path will
// not wait for must not be reported as missing either, or the maintenance hatch trades a
// failed deployment for a permanent warning.
enabled = db.ClusterNodes.AsNoTracking()
.Where(n => n.Enabled && !n.MaintenanceMode).Select(n => n.NodeId).ToList();
}
catch (Exception ex)
{
// A consistency check must never take the admin node down with it. Central SQL being
// briefly unreachable is an operational fact, not a reason to restart this singleton.
_log.Warning(ex, "ClusterNode address reconcile skipped — config database unreachable");
return;
}
var findings = ClusterNodeAddressReconciler.Reconcile(observed, rows, enabled);
if (_lastReported is not null && findings.SequenceEqual(_lastReported)) return;
_lastReported = findings;
if (findings.Count == 0)
{
_log.Info("ClusterNode addresses reconcile with cluster membership ({Count} driver members)",
observed.Count);
return;
}
foreach (var f in findings)
{
// Error for the two shapes that are always a misconfiguration; Warning for a row whose
// node is merely absent, which is a legitimate state during maintenance.
if (f.Kind == AddressMismatchKind.EnabledRowNotInCluster)
_log.Warning("ClusterNode address check [{Kind}] {NodeId}: {Detail}", f.Kind, f.NodeId, f.Detail);
else
_log.Error("ClusterNode address check [{Kind}] {NodeId}: {Detail}", f.Kind, f.NodeId, f.Detail);
}
}
/// <summary>Self-message: run a reconcile pass now.</summary>
public sealed class ReconcileNow
{
/// <summary>The singleton instance.</summary>
public static readonly ReconcileNow Instance = new();
private ReconcileNow() { }
}
}
@@ -23,9 +23,11 @@ public static class ServiceCollectionExtensions
public const string AuditWriterSingletonName = "audit-writer"; public const string AuditWriterSingletonName = "audit-writer";
public const string FleetStatusSingletonName = "fleet-status"; public const string FleetStatusSingletonName = "fleet-status";
public const string RedundancyStateSingletonName = "redundancy-state"; public const string RedundancyStateSingletonName = "redundancy-state";
public const string ClusterNodeAddressReconcilerSingletonName = "cluster-node-address-reconciler";
/// <summary> /// <summary>
/// Registers all five admin-role cluster singletons + their proxies on the AkkaConfigurationBuilder. /// Registers the admin-role cluster singletons on the AkkaConfigurationBuilder — five with
/// registry proxies, plus the address reconciler, which is proxy-less because nothing addresses it.
/// Must be called against the same builder used by <c>AkkaHostedService</c> so the singletons /// Must be called against the same builder used by <c>AkkaHostedService</c> so the singletons
/// share the host's ActorSystem. /// share the host's ActorSystem.
/// </summary> /// </summary>
@@ -90,6 +92,26 @@ public static class ServiceCollectionExtensions
(system, registry, resolver) => RedundancyStateActor.Props(), (system, registry, resolver) => RedundancyStateActor.Props(),
singletonOptions); singletonOptions);
// Per-cluster mesh Phase 1: ClusterNode.AkkaPort duplicates the node's own Cluster:Port and
// nothing makes them agree. Runs here rather than on each driver node because Phase 4 removes
// the driver nodes' ConfigDb connection entirely.
//
// createProxyToo: false — unlike the five singletons above, nothing ever resolves
// ClusterNodeAddressReconcilerKey from the registry. This actor only reads cluster state and
// logs; no one sends it messages. A proxy would be a second actor per node whose only job is
// to hunt for a singleton nobody addresses — visible as the "ClusterSingletonProxy failed to
// find an associated singleton after 30 seconds" startup warning, and as extra cluster
// chatter in every two-node test harness the integration suite builds.
builder.WithSingleton<ClusterNodeAddressReconcilerKey>(
ClusterNodeAddressReconcilerSingletonName,
(system, registry, resolver) =>
{
var dbFactory = resolver.GetService<IDbContextFactory<OtOpcUaConfigDbContext>>();
return ClusterNodeAddressReconcilerActor.Props(dbFactory);
},
singletonOptions,
createProxyToo: false);
return builder; return builder;
} }
} }
@@ -100,3 +122,4 @@ public sealed class AdminOperationsActorKey { }
public sealed class AuditWriterActorKey { } public sealed class AuditWriterActorKey { }
public sealed class FleetStatusBroadcasterKey { } public sealed class FleetStatusBroadcasterKey { }
public sealed class RedundancyStateActorKey { } public sealed class RedundancyStateActorKey { }
public sealed class ClusterNodeAddressReconcilerKey { }
@@ -0,0 +1,149 @@
using Microsoft.EntityFrameworkCore;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
using ZB.MOM.WW.OtOpcUa.Configuration.Enums;
namespace ZB.MOM.WW.OtOpcUa.Configuration.Tests;
/// <summary>
/// Round-trips <see cref="ClusterNode.AkkaPort"/> / <see cref="ClusterNode.GrpcPort"/> through
/// the real schema (per-cluster mesh Phase 1). These are <b>central's dial targets</b>: Phase 2
/// builds its ClusterClient contact points from <c>Host</c> + <c>AkkaPort</c>, and Phase 5 dials
/// <c>GrpcPort</c> for the telemetry stream. Central cannot see a site node's appsettings once
/// the meshes split, so the value has to live in a row central can read.
/// </summary>
[Trait("Category", "SchemaCompliance")]
[Collection(nameof(SchemaComplianceCollection))]
public sealed class ClusterNodeTransportPortsTests(SchemaComplianceFixture fixture)
{
/// <summary>Verifies both ports survive a DB round-trip and that AkkaPort defaults to 4053.</summary>
[Fact]
public async Task Transport_ports_round_trip_and_AkkaPort_defaults_to_4053()
{
await using var ctx = NewContext();
var clusterId = await SeedClusterAsync(ctx);
// Explicit non-default values on one node — proves the columns are mapped and persisted
// rather than silently dropped (an unmapped property round-trips fine in memory).
ctx.ClusterNodes.Add(NewNode(clusterId, "explicit", n =>
{
n.AkkaPort = 4055;
n.GrpcPort = 5223;
}));
// Nothing set on the other — AkkaPort must land on 4053 and GrpcPort must stay null.
// GrpcPort is deliberately nullable: nothing listens on it until Phase 5, and a non-null
// default would assert a port that does not exist.
ctx.ClusterNodes.Add(NewNode(clusterId, "defaulted"));
await ctx.SaveChangesAsync();
await using var read = NewContext();
var explicitNode = await read.ClusterNodes.AsNoTracking()
.SingleAsync(n => n.NodeId == $"{clusterId}-explicit:4053");
explicitNode.AkkaPort.ShouldBe(4055);
explicitNode.GrpcPort.ShouldBe(5223);
var defaulted = await read.ClusterNodes.AsNoTracking()
.SingleAsync(n => n.NodeId == $"{clusterId}-defaulted:4053");
defaulted.AkkaPort.ShouldBe(4053);
defaulted.GrpcPort.ShouldBeNull();
}
/// <summary>
/// Verifies the <b>DB-side</b> default on <c>AkkaPort</c> — the migration's
/// <c>defaultValue: 4053</c>, which is what rows created before the column existed inherit.
/// The entity round-trip above cannot see this: <see cref="ClusterNode.AkkaPort"/>'s CLR
/// initializer is also 4053, so that assertion passes with the mapping default deleted. This
/// one inserts through raw SQL with the column omitted, so only the schema can supply the value.
/// </summary>
[Fact]
public async Task AkkaPort_has_a_DB_side_default_of_4053()
{
await using var ctx = NewContext();
var clusterId = await SeedClusterAsync(ctx);
await using (var conn = fixture.OpenConnection())
{
var cmd = conn.CreateCommand();
cmd.CommandText = """
INSERT INTO dbo.ClusterNode (NodeId, ClusterId, Host, OpcUaPort, DashboardPort,
ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
VALUES (@nodeId, @clusterId, @host, 4840, 8081, @uri, 200, 1, 'raw-sql');
""";
cmd.Parameters.AddWithValue("@nodeId", $"{clusterId}-raw:4053");
cmd.Parameters.AddWithValue("@clusterId", clusterId);
cmd.Parameters.AddWithValue("@host", $"{clusterId}-raw");
cmd.Parameters.AddWithValue("@uri", $"urn:OtOpcUa:{clusterId}-raw");
await cmd.ExecuteNonQueryAsync();
}
await using var read = NewContext();
var row = await read.ClusterNodes.AsNoTracking()
.SingleAsync(n => n.NodeId == $"{clusterId}-raw:4053");
row.AkkaPort.ShouldBe(4053);
row.GrpcPort.ShouldBeNull();
}
/// <summary>
/// Verifies the columns are queryable server-side — Phase 2's contact-point refresh selects
/// them in SQL rather than materialising every node, so a client-side-evaluation regression
/// (or an unmapped property) has to fail here.
/// </summary>
[Fact]
public async Task Transport_ports_are_queryable_server_side()
{
await using var ctx = NewContext();
var clusterId = await SeedClusterAsync(ctx);
ctx.ClusterNodes.Add(NewNode(clusterId, "a", n => n.GrpcPort = 5223));
ctx.ClusterNodes.Add(NewNode(clusterId, "b"));
await ctx.SaveChangesAsync();
await using var read = NewContext();
var contacts = await read.ClusterNodes.AsNoTracking()
.Where(n => n.ClusterId == clusterId && n.GrpcPort != null)
.Select(n => new { n.Host, n.AkkaPort, n.GrpcPort })
.ToListAsync();
contacts.Count.ShouldBe(1);
contacts[0].AkkaPort.ShouldBe(4053);
contacts[0].GrpcPort.ShouldBe(5223);
}
private static ClusterNode NewNode(string clusterId, string suffix, Action<ClusterNode>? configure = null)
{
var node = new ClusterNode
{
NodeId = $"{clusterId}-{suffix}:4053",
ClusterId = clusterId,
Host = $"{clusterId}-{suffix}",
ApplicationUri = $"urn:OtOpcUa:{clusterId}-{suffix}",
CreatedBy = nameof(ClusterNodeTransportPortsTests),
};
configure?.Invoke(node);
return node;
}
private static async Task<string> SeedClusterAsync(OtOpcUaConfigDbContext ctx)
{
// Unique per test: the collection shares one database, and ApplicationUri is fleet-wide unique.
var clusterId = $"PORTS-{Guid.NewGuid():N}"[..24];
ctx.ServerClusters.Add(new ServerCluster
{
ClusterId = clusterId,
Name = clusterId,
Enterprise = "zb",
Site = "ports-test",
// CK_ServerCluster_RedundancyMode_NodeCount: Warm/Hot require exactly 2 nodes.
RedundancyMode = RedundancyMode.Warm,
NodeCount = 2,
CreatedBy = nameof(ClusterNodeTransportPortsTests),
});
await ctx.SaveChangesAsync();
return clusterId;
}
private OtOpcUaConfigDbContext NewContext() =>
new(new DbContextOptionsBuilder<OtOpcUaConfigDbContext>()
.UseSqlServer(fixture.ConnectionString)
.Options);
}
@@ -0,0 +1,98 @@
using Akka.Actor;
using Microsoft.EntityFrameworkCore;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Configuration;
using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
using ZB.MOM.WW.OtOpcUa.Configuration.Enums;
using ZB.MOM.WW.OtOpcUa.ControlPlane.Fleet;
using ZB.MOM.WW.OtOpcUa.ControlPlane.Tests.Harness;
namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Tests;
/// <summary>
/// Proves the reconciler is actually wired to something — subscribes to membership, reads the DB,
/// and emits. <see cref="ClusterNodeAddressReconcilerTests"/> covers the comparison logic, which a
/// dormant actor would leave perfectly correct and completely inert.
/// </summary>
public sealed class ClusterNodeAddressReconcilerActorTests : ControlPlaneActorTestBase
{
/// <summary>
/// An enabled row with no matching driver member is warned about. The harness ActorSystem
/// joins as <c>admin</c> with no driver members, so the seeded row is unmatched by
/// construction.
/// </summary>
[Fact]
public void Enabled_row_with_no_matching_member_is_logged()
{
var dbFactory = NewInMemoryDbFactory();
SeedNode(dbFactory, "site-a-1:4053", enabled: true);
EventFilter.Warning(contains: "site-a-1:4053").ExpectOne(TimeSpan.FromSeconds(10), () =>
Sys.ActorOf(ClusterNodeAddressReconcilerActor.Props(dbFactory)));
}
/// <summary>
/// The finding set is logged once, not once per sweep. A check that reprints the same warning
/// every five minutes trains operators to filter it out, which costs more than it catches —
/// so this runs a deliberately fast sweep and asserts the count stays at one.
/// </summary>
[Fact]
public void Unchanged_findings_are_not_repeated_on_every_sweep()
{
var dbFactory = NewInMemoryDbFactory();
SeedNode(dbFactory, "site-a-1:4053", enabled: true);
// ~6 sweeps inside the window; without the change-detection guard this logs 6 times.
EventFilter.Warning(contains: "site-a-1:4053").Expect(1, TimeSpan.FromSeconds(3), () =>
{
Sys.ActorOf(ClusterNodeAddressReconcilerActor.Props(
dbFactory, sweepInterval: TimeSpan.FromMilliseconds(400)));
// Hold the filter open past several sweeps rather than returning immediately.
ExpectNoMsg(TimeSpan.FromSeconds(2.5));
});
}
/// <summary>
/// A disabled row is silence — the maintenance hatch must not trade a failed deployment for a
/// permanent warning. Positive control for the test above: same setup, one flag flipped.
/// </summary>
[Fact]
public void Disabled_row_produces_no_warning()
{
var dbFactory = NewInMemoryDbFactory();
SeedNode(dbFactory, "site-a-1:4053", enabled: false);
EventFilter.Warning(contains: "site-a-1:4053").Expect(0, TimeSpan.FromSeconds(2), () =>
{
Sys.ActorOf(ClusterNodeAddressReconcilerActor.Props(
dbFactory, sweepInterval: TimeSpan.FromMilliseconds(400)));
ExpectNoMsg(TimeSpan.FromSeconds(1.5));
});
}
private static void SeedNode(
IDbContextFactory<OtOpcUaConfigDbContext> dbFactory, string nodeId, bool enabled)
{
using var db = dbFactory.CreateDbContext();
db.ServerClusters.Add(new ServerCluster
{
ClusterId = "SITE-A",
Name = "Site A",
Enterprise = "zb",
Site = "site-a",
NodeCount = 2,
RedundancyMode = RedundancyMode.Warm,
CreatedBy = "test",
});
db.ClusterNodes.Add(new ClusterNode
{
NodeId = nodeId,
ClusterId = "SITE-A",
Host = nodeId.Split(':')[0],
ApplicationUri = $"urn:OtOpcUa:{nodeId}",
Enabled = enabled,
CreatedBy = "test",
});
db.SaveChanges();
}
}
@@ -0,0 +1,126 @@
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.ControlPlane.Fleet;
namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Tests;
/// <summary>
/// Per-cluster mesh Phase 1 Task 4: <c>ClusterNode.AkkaPort</c> and the node's own
/// <c>Cluster:Port</c> are the same fact in two places, and nothing makes them agree. These pin
/// the shapes that drift takes.
/// </summary>
public sealed class ClusterNodeAddressReconcilerTests
{
private static ClusterNodeAddress Row(string host, int akkaPort, string? nodeId = null) =>
new(nodeId ?? $"{host}:{akkaPort}", host, akkaPort);
/// <summary>A fleet whose rows match membership reports nothing.</summary>
[Fact]
public void Matching_rows_and_membership_produce_no_findings()
{
var findings = ClusterNodeAddressReconciler.Reconcile(
[("site-a-1", 4053), ("site-a-2", 4053)],
[Row("site-a-1", 4053), Row("site-a-2", 4053)],
["site-a-1:4053", "site-a-2:4053"]);
findings.ShouldBeEmpty();
}
/// <summary>
/// The drift the plan names: a node binds 4054 while its row says 4053. Membership shows it
/// at an address no row claims, and the row it should have matched looks absent — both halves
/// are reported, because either alone reads as a different problem.
/// </summary>
[Fact]
public void Node_bound_to_the_wrong_port_is_reported_from_both_sides()
{
var findings = ClusterNodeAddressReconciler.Reconcile(
[("site-a-1", 4054)],
[Row("site-a-1", 4053)],
["site-a-1:4053"]);
findings.Count.ShouldBe(2);
findings.ShouldContain(f =>
f.Kind == AddressMismatchKind.RunningNodeHasNoRow && f.NodeId == "site-a-1:4054");
findings.ShouldContain(f =>
f.Kind == AddressMismatchKind.EnabledRowNotInCluster && f.NodeId == "site-a-1:4053");
}
/// <summary>
/// The Phase 2 shape: the row's NodeId still matches membership, but its <c>Host</c> /
/// <c>AkkaPort</c> columns — the values central will actually dial — have been edited away
/// from it. Gossip still works today, so nothing else would notice.
/// </summary>
[Fact]
public void Row_whose_dial_target_disagrees_with_its_own_NodeId_is_reported()
{
var findings = ClusterNodeAddressReconciler.Reconcile(
[("site-a-1", 4053)],
[Row("site-a-1", 4054, nodeId: "site-a-1:4053")],
["site-a-1:4053"]);
var only = findings.ShouldHaveSingleItem();
only.Kind.ShouldBe(AddressMismatchKind.RowDialTargetDisagrees);
only.NodeId.ShouldBe("site-a-1:4053");
only.Detail.ShouldContain("site-a-1:4054");
}
/// <summary>A running driver node nobody registered — the coordinator discards its acks.</summary>
[Fact]
public void Running_node_with_no_row_is_reported()
{
var findings = ClusterNodeAddressReconciler.Reconcile(
[("ghost-9", 4053)], [], []);
findings.ShouldHaveSingleItem().Kind.ShouldBe(AddressMismatchKind.RunningNodeHasNoRow);
}
/// <summary>
/// A disabled row for an absent node is silence, not a finding — being absent is the entire
/// point of disabling it. Without this the maintenance hatch would trade a failed deploy for
/// a permanent warning.
/// </summary>
[Fact]
public void Disabled_row_for_an_absent_node_is_not_reported()
{
var findings = ClusterNodeAddressReconciler.Reconcile(
[("site-a-1", 4053)],
[Row("site-a-1", 4053), Row("site-a-2", 4053)],
["site-a-1:4053"]);
findings.ShouldBeEmpty();
}
/// <summary>
/// An admin-only member has no <c>ClusterNode</c> row by design, so the caller filters
/// membership to driver-role members. Passing an unfiltered list would report correct
/// configuration as broken — pinned here because the filter lives at the call site.
/// </summary>
[Fact]
public void Admin_only_members_are_the_callers_responsibility_to_exclude()
{
// What the actor passes: driver members only.
ClusterNodeAddressReconciler.Reconcile(
[("central-1", 4053)], [Row("central-1", 4053)], ["central-1:4053"])
.ShouldBeEmpty();
// What an unfiltered call would produce — the false positive this guards against.
ClusterNodeAddressReconciler.Reconcile(
[("central-1", 4053), ("admin-only-1", 4053)],
[Row("central-1", 4053)],
["central-1:4053"])
.ShouldHaveSingleItem().Kind.ShouldBe(AddressMismatchKind.RunningNodeHasNoRow);
}
/// <summary>
/// Host matching is case-insensitive, matching the coordinator's <c>NodeIdComparer</c> — DNS
/// and SQL collation are both case-insensitive, so the same node surfaces with either casing.
/// </summary>
[Fact]
public void Host_matching_is_case_insensitive()
{
ClusterNodeAddressReconciler.Reconcile(
[("SITE-A-1", 4053)], [Row("site-a-1", 4053)], ["site-a-1:4053"])
.ShouldBeEmpty();
}
}
@@ -0,0 +1,263 @@
using Akka.Actor;
using Microsoft.EntityFrameworkCore;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy;
using ZB.MOM.WW.OtOpcUa.Commons.Types;
using ZB.MOM.WW.OtOpcUa.Configuration;
using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
using ZB.MOM.WW.OtOpcUa.Configuration.Enums;
using ZB.MOM.WW.OtOpcUa.ControlPlane.Coordinators;
using ZB.MOM.WW.OtOpcUa.ControlPlane.Tests.Harness;
namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Tests;
/// <summary>
/// Per-cluster mesh Phase 1: the coordinator's expected-ack set comes from enabled
/// <c>ClusterNode</c> rows, not from <c>Akka.Cluster.State.Members</c> filtered by role.
/// </summary>
/// <remarks>
/// <para>
/// <b>Why every test here seeds ClusterNode rows and asserts a non-empty expected set.</b>
/// The harness's ActorSystem self-joins with role <c>admin</c> and no <c>driver</c> members,
/// so the OLD derivation returned an empty set for every one of these scenarios and sealed
/// immediately. Any test that merely observes "the deployment did not seal" would therefore
/// have passed against the old code too. Each test below distinguishes the two by requiring
/// the coordinator to WAIT for a specific node and then seal when exactly that node acks —
/// behaviour the membership rule cannot produce in this harness at all.
/// </para>
/// <para>
/// Verified by reverting <c>DiscoverDriverNodes</c> to the membership scan: the first three
/// go red. <see cref="No_enabled_rows_seals_empty"/> stays green under both derivations and
/// is <b>not</b> a test of the derivation — the harness has no driver members AND no enabled
/// rows, so both rules produce an empty set. It pins the seal-empty branch and its reworded
/// reason, nothing more.
/// </para>
/// </remarks>
public sealed class ConfigPublishCoordinatorDbSourcedAcksTests : ControlPlaneActorTestBase
{
private static readonly RevisionHash TestRevision = RevisionHash.Parse(new string('b', 64));
private static readonly TimeSpan NoTimeout = TimeSpan.FromMinutes(5);
/// <summary>
/// An enabled row is expected to ack; the deployment waits for it and seals when it arrives.
/// This is the DB-derived set doing something the membership set demonstrably cannot here.
/// </summary>
[Fact]
public void Enabled_ClusterNode_rows_are_the_expected_ack_set()
{
var dbFactory = NewInMemoryDbFactory();
var deploymentId = SeedDispatchingDeployment(dbFactory);
SeedNodes(dbFactory, ("site-a-1:4053", true, false));
var actor = Sys.ActorOf(ConfigPublishCoordinator.Props(dbFactory, NoTimeout));
actor.Tell(new DispatchDeployment(deploymentId, TestRevision, CorrelationId.NewId()));
// The seeded row must have produced a NodeDeploymentState row — proof the coordinator
// expected it, independent of whether it later sealed.
AwaitAssert(() =>
{
using var db = dbFactory.CreateDbContext();
db.NodeDeploymentStates.Select(s => s.NodeId).ShouldBe(new[] { "site-a-1:4053" });
db.Deployments.Single().Status.ShouldBe(DeploymentStatus.AwaitingApplyAcks);
}, duration: TimeSpan.FromSeconds(3));
actor.Tell(new ApplyAck(deploymentId, NodeId.Parse("site-a-1:4053"),
ApplyAckOutcome.Applied, null, CorrelationId.NewId()));
AwaitAssert(() =>
{
using var db = dbFactory.CreateDbContext();
db.Deployments.Single().Status.ShouldBe(DeploymentStatus.Sealed);
}, duration: TimeSpan.FromSeconds(3));
}
/// <summary>
/// A disabled row is NOT expected, so a deployment seals without it — for a node genuinely
/// decommissioned rather than temporarily down. <b>This is not the maintenance hatch</b>: the
/// topology gate rejects a deployment whose enabled-node count differs from the cluster's
/// <c>NodeCount</c>, so on any 2-node pair clearing <c>Enabled</c> never reaches this code.
/// See <see cref="MaintenanceMode_rows_are_not_expected_to_ack_while_staying_enabled"/>.
/// </summary>
[Fact]
public void Disabled_ClusterNode_rows_are_not_expected_to_ack()
{
var dbFactory = NewInMemoryDbFactory();
var deploymentId = SeedDispatchingDeployment(dbFactory);
SeedNodes(dbFactory, ("site-a-1:4053", true, false), ("site-a-2:4053", false, false));
var actor = Sys.ActorOf(ConfigPublishCoordinator.Props(dbFactory, NoTimeout));
actor.Tell(new DispatchDeployment(deploymentId, TestRevision, CorrelationId.NewId()));
// Only the enabled node is expected — the disabled one gets no NodeDeploymentState row.
AwaitAssert(() =>
{
using var db = dbFactory.CreateDbContext();
db.NodeDeploymentStates.Select(s => s.NodeId).ShouldBe(new[] { "site-a-1:4053" });
}, duration: TimeSpan.FromSeconds(3));
// ...and the enabled node's ack alone seals it. If the disabled row were expected this would
// sit at AwaitingApplyAcks until the deadline.
actor.Tell(new ApplyAck(deploymentId, NodeId.Parse("site-a-1:4053"),
ApplyAckOutcome.Applied, null, CorrelationId.NewId()));
AwaitAssert(() =>
{
using var db = dbFactory.CreateDbContext();
db.Deployments.Single().Status.ShouldBe(DeploymentStatus.Sealed);
}, duration: TimeSpan.FromSeconds(3));
}
/// <summary>
/// An ack from a node with no <c>ClusterNode</c> row is discarded — it is not in the expected
/// set, so it must not count toward the seal. Under the membership rule a running-but-
/// unregistered node WAS expected, and then threw FK 547 when its state row was written.
/// </summary>
[Fact]
public void Ack_from_a_node_absent_from_ClusterNode_is_discarded()
{
var dbFactory = NewInMemoryDbFactory();
var deploymentId = SeedDispatchingDeployment(dbFactory);
SeedNodes(dbFactory, ("site-a-1:4053", true, false));
var actor = Sys.ActorOf(ConfigPublishCoordinator.Props(dbFactory, NoTimeout));
actor.Tell(new DispatchDeployment(deploymentId, TestRevision, CorrelationId.NewId()));
AwaitAssert(() =>
{
using var db = dbFactory.CreateDbContext();
db.Deployments.Single().Status.ShouldBe(DeploymentStatus.AwaitingApplyAcks);
}, duration: TimeSpan.FromSeconds(3));
// A node nobody registered acks. One expected node, one ack — if it were counted, the
// coordinator would seal here.
actor.Tell(new ApplyAck(deploymentId, NodeId.Parse("ghost-9:4053"),
ApplyAckOutcome.Applied, null, CorrelationId.NewId()));
ExpectNoMsg(TimeSpan.FromMilliseconds(400));
using (var db = dbFactory.CreateDbContext())
{
db.Deployments.Single().Status.ShouldBe(DeploymentStatus.AwaitingApplyAcks);
}
// Positive control on the wait itself: the real node's ack DOES seal it, so the assertion
// above measured the discard rather than a coordinator that had simply stopped responding.
actor.Tell(new ApplyAck(deploymentId, NodeId.Parse("site-a-1:4053"),
ApplyAckOutcome.Applied, null, CorrelationId.NewId()));
AwaitAssert(() =>
{
using var db = dbFactory.CreateDbContext();
db.Deployments.Single().Status.ShouldBe(DeploymentStatus.Sealed);
}, duration: TimeSpan.FromSeconds(3));
}
/// <summary>
/// No enabled rows at all seals empty. Same outcome as the old empty-membership branch, but
/// it now means "the fleet has no enabled nodes registered" — a configuration error that
/// cannot resolve itself — rather than "no driver node has joined yet".
/// </summary>
[Fact]
public void No_enabled_rows_seals_empty()
{
var dbFactory = NewInMemoryDbFactory();
var deploymentId = SeedDispatchingDeployment(dbFactory);
SeedNodes(dbFactory, ("site-a-1:4053", false, false), ("site-a-2:4053", false, false));
var actor = Sys.ActorOf(ConfigPublishCoordinator.Props(dbFactory, NoTimeout));
actor.Tell(new DispatchDeployment(deploymentId, TestRevision, CorrelationId.NewId()));
AwaitAssert(() =>
{
using var db = dbFactory.CreateDbContext();
db.Deployments.Single().Status.ShouldBe(DeploymentStatus.Sealed);
db.NodeDeploymentStates.ShouldBeEmpty();
}, duration: TimeSpan.FromSeconds(3));
}
/// <summary>
/// The maintenance hatch, and the reason it is a separate flag. The Phase 1 live gate set
/// <c>Enabled = 0</c> on one node of a 2-node pair and the deployment was rejected outright by
/// <c>DraftValidator.ValidateClusterTopology</c> (enabled-node count must equal
/// <c>NodeCount</c>) — so the intended escape hatch could not be used on any pair, which is
/// every cluster in the target topology. <c>MaintenanceMode</c> excludes a node from the
/// expected-ack set while leaving it enabled, so the topology gate still passes.
/// </summary>
[Fact]
public void MaintenanceMode_rows_are_not_expected_to_ack_while_staying_enabled()
{
var dbFactory = NewInMemoryDbFactory();
var deploymentId = SeedDispatchingDeployment(dbFactory);
SeedNodes(dbFactory, ("site-a-1:4053", true, false), ("site-a-2:4053", true, true));
// The topology gate counts Enabled, so both nodes still count — this is what Enabled = 0
// broke. Assert it here so the fix cannot be "quietly disable it after all".
using (var check = dbFactory.CreateDbContext())
{
check.ClusterNodes.Count(n => n.Enabled).ShouldBe(2);
}
var actor = Sys.ActorOf(ConfigPublishCoordinator.Props(dbFactory, NoTimeout));
actor.Tell(new DispatchDeployment(deploymentId, TestRevision, CorrelationId.NewId()));
AwaitAssert(() =>
{
using var db = dbFactory.CreateDbContext();
db.NodeDeploymentStates.Select(s => s.NodeId).ShouldBe(new[] { "site-a-1:4053" });
}, duration: TimeSpan.FromSeconds(3));
actor.Tell(new ApplyAck(deploymentId, NodeId.Parse("site-a-1:4053"),
ApplyAckOutcome.Applied, null, CorrelationId.NewId()));
AwaitAssert(() =>
{
using var db = dbFactory.CreateDbContext();
db.Deployments.Single().Status.ShouldBe(DeploymentStatus.Sealed);
}, duration: TimeSpan.FromSeconds(3));
}
private static void SeedNodes(
IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
params (string NodeId, bool Enabled, bool Maintenance)[] nodes)
{
using var db = dbFactory.CreateDbContext();
db.ServerClusters.Add(new ServerCluster
{
ClusterId = "SITE-A",
Name = "Site A",
Enterprise = "zb",
Site = "site-a",
NodeCount = 2,
RedundancyMode = RedundancyMode.Warm,
CreatedBy = "test",
});
foreach (var (nodeId, enabled, maintenance) in nodes)
{
db.ClusterNodes.Add(new ClusterNode
{
NodeId = nodeId,
ClusterId = "SITE-A",
Host = nodeId.Split(':')[0],
ApplicationUri = $"urn:OtOpcUa:{nodeId}",
Enabled = enabled,
MaintenanceMode = maintenance,
CreatedBy = "test",
});
}
db.SaveChanges();
}
private static DeploymentId SeedDispatchingDeployment(
IDbContextFactory<OtOpcUaConfigDbContext> dbFactory)
{
var id = DeploymentId.NewId();
using var db = dbFactory.CreateDbContext();
db.Deployments.Add(new Deployment
{
DeploymentId = id.Value,
RevisionHash = TestRevision.Value,
Status = DeploymentStatus.Dispatching,
CreatedBy = "test",
});
db.SaveChanges();
return id;
}
}
@@ -0,0 +1,14 @@
using Xunit;
// Every test in this assembly builds a real two-node Akka cluster: two in-process Kestrel hosts,
// two ActorSystems with remoting on ephemeral ports, six admin-role cluster singletons, an EF
// context and a deploy pipeline. Running several of those concurrently — on top of the other test
// assemblies dotnet test runs in parallel — starves the cluster-formation and ApplyAck timers these
// tests measure, so failures appear under full-suite load and vanish in isolation.
//
// That was tolerable while the deploy path sealed immediately on an empty expected-ack set. Since
// per-cluster mesh Phase 1 the coordinator waits for a real ack from every configured node, so these
// tests now measure an end-to-end round-trip and are correspondingly more sensitive to CPU
// starvation. Serialising this assembly trades wall-clock for determinism; the assembly is a handful
// of heavyweight E2E tests, not a broad unit suite.
[assembly: CollectionBehavior(DisableTestParallelization = true)]
@@ -40,7 +40,7 @@ public sealed class DeployHappyPathTests
await using var db = await CreateDbAsync(harness); await using var db = await CreateDbAsync(harness);
var d = await db.Deployments.AsNoTracking().FirstOrDefaultAsync(d => d.DeploymentId == deploymentId, Ct); var d = await db.Deployments.AsNoTracking().FirstOrDefaultAsync(d => d.DeploymentId == deploymentId, Ct);
return d?.Status == DeploymentStatus.Sealed; return d?.Status == DeploymentStatus.Sealed;
}, TimeSpan.FromSeconds(15)); }, TwoNodeClusterHarness.DeploySealTimeout);
await using var db = await CreateDbAsync(harness); await using var db = await CreateDbAsync(harness);
var deployment = await db.Deployments.AsNoTracking() var deployment = await db.Deployments.AsNoTracking()
@@ -74,7 +74,7 @@ public sealed class DeployHappyPathTests
var d = await db.Deployments.AsNoTracking() var d = await db.Deployments.AsNoTracking()
.FirstOrDefaultAsync(d => d.DeploymentId == first.DeploymentId!.Value.Value, Ct); .FirstOrDefaultAsync(d => d.DeploymentId == first.DeploymentId!.Value.Value, Ct);
return d?.Status == DeploymentStatus.Sealed; return d?.Status == DeploymentStatus.Sealed;
}, TimeSpan.FromSeconds(15)); }, TwoNodeClusterHarness.DeploySealTimeout);
// Same DB state → same revision hash → AdminOperations short-circuits with NoChanges, // Same DB state → same revision hash → AdminOperations short-circuits with NoChanges,
// OR (if it accepts) the second dispatch is idempotent (DriverHostActor recognises // OR (if it accepts) the second dispatch is idempotent (DriverHostActor recognises
@@ -90,7 +90,7 @@ public sealed class DeployHappyPathTests
var d = await db.Deployments.AsNoTracking() var d = await db.Deployments.AsNoTracking()
.FirstOrDefaultAsync(d => d.DeploymentId == second.DeploymentId!.Value.Value, Ct); .FirstOrDefaultAsync(d => d.DeploymentId == second.DeploymentId!.Value.Value, Ct);
return d?.Status == DeploymentStatus.Sealed; return d?.Status == DeploymentStatus.Sealed;
}, TimeSpan.FromSeconds(15)); }, TwoNodeClusterHarness.DeploySealTimeout);
second.RevisionHash!.Value.Value.ShouldBe(first.RevisionHash!.Value.Value); second.RevisionHash!.Value.Value.ShouldBe(first.RevisionHash!.Value.Value);
} }
@@ -85,7 +85,7 @@ public sealed class EquipmentNamespaceMaterializationTests
return true; return true;
} }
return false; return false;
}, TimeSpan.FromSeconds(15)); }, TwoNodeClusterHarness.DeploySealTimeout);
var composition = DeploymentArtifact.ParseComposition(artifact); var composition = DeploymentArtifact.ParseComposition(artifact);
@@ -182,7 +182,7 @@ public sealed class EquipmentNamespaceMaterializationTests
return true; return true;
} }
return false; return false;
}, TimeSpan.FromSeconds(20)); }, TwoNodeClusterHarness.DeploySealTimeout);
await using (var db = await CreateDbAsync(harness)) await using (var db = await CreateDbAsync(harness))
{ {
@@ -50,13 +50,22 @@ public sealed class FailoverDuringDeployTests
.Count(m => m.Status == MemberStatus.Up).ShouldBe(2); .Count(m => m.Status == MemberStatus.Up).ShouldBe(2);
} }
/// <summary>Verifies that a deployment started with node B down seals with one-node state.</summary> /// <summary>
/// A deployment started with node B down no longer seals without it — B's <c>ClusterNode</c>
/// row is enabled, so it is expected to ack and the deployment waits.
/// </summary>
/// <remarks>
/// <b>This test asserted the opposite until per-cluster mesh Phase 1.</b> It was named
/// <c>Deployment_started_with_node_b_down_seals_with_one_node_state</c> and documented that
/// "<c>DiscoverDriverNodes</c> snapshots membership at dispatch time — when only node A is Up,
/// only one ApplyAck is expected and the deployment seals without B ever participating". That
/// is exactly the behaviour Phase 1 removed: it told the operator the fleet was deployed while
/// a configured node had not received it. The expected-ack set now comes from enabled
/// <c>ClusterNode</c> rows, so a node that is merely switched off is still expected.
/// </remarks>
[Fact] [Fact]
public async Task Deployment_started_with_node_b_down_seals_with_one_node_state() public async Task Deployment_started_with_node_b_down_does_not_seal_without_it()
{ {
// Establishes that ConfigPublishCoordinator.DiscoverDriverNodes snapshots membership at
// dispatch time — when only node A is Up, only one ApplyAck is expected and the
// deployment seals without B ever participating.
await using var harness = await TwoNodeClusterHarness.StartAsync(); await using var harness = await TwoNodeClusterHarness.StartAsync();
await harness.SeedDefaultClusterAsync(); await harness.SeedDefaultClusterAsync();
@@ -70,19 +79,78 @@ public sealed class FailoverDuringDeployTests
result.Outcome.ShouldBe(StartDeploymentOutcome.Accepted); result.Outcome.ShouldBe(StartDeploymentOutcome.Accepted);
var deploymentId = result.DeploymentId!.Value.Value; var deploymentId = result.DeploymentId!.Value.Value;
// Positive evidence that B was EXPECTED, not merely slow: the coordinator seeds a row per
// expected node at dispatch, so both rows must exist with B still Applying. Asserting only
// "it didn't seal" would pass just as well against a coordinator that had died.
await WaitForAsync(async () => await WaitForAsync(async () =>
{ {
await using var db = await CreateDbAsync(harness); await using var pollDb = await CreateDbAsync(harness);
var d = await db.Deployments.AsNoTracking() return await pollDb.NodeDeploymentStates.AsNoTracking()
.FirstOrDefaultAsync(d => d.DeploymentId == deploymentId, Ct); .CountAsync(s => s.DeploymentId == deploymentId, Ct) == 2;
return d?.Status == DeploymentStatus.Sealed; }, TwoNodeClusterHarness.DeploySealTimeout);
}, TimeSpan.FromSeconds(15));
await using var db = await CreateDbAsync(harness); await using var db = await CreateDbAsync(harness);
var nodeStates = await db.NodeDeploymentStates.AsNoTracking() var nodeStates = await db.NodeDeploymentStates.AsNoTracking()
.Where(s => s.DeploymentId == deploymentId) .Where(s => s.DeploymentId == deploymentId)
.ToListAsync(Ct); .ToListAsync(Ct);
nodeStates.Count.ShouldBe(1); nodeStates.Count.ShouldBe(2, "both configured nodes are expected to ack");
nodeStates.Count(s => s.Status == NodeDeploymentStatus.Applied)
.ShouldBe(1, "only the running node applied");
nodeStates.ShouldContain(s => s.Status == NodeDeploymentStatus.Applying,
"the stopped node's ack is still outstanding");
var deployment = await db.Deployments.AsNoTracking()
.FirstAsync(d => d.DeploymentId == deploymentId, Ct);
deployment.Status.ShouldNotBe(DeploymentStatus.Sealed,
"a deployment must not seal green while a configured node has not received it");
}
/// <summary>
/// The maintenance hatch, end-to-end: a node down <i>and</i> flagged
/// <c>MaintenanceMode</c> is not expected, so the deployment seals with one node state — the
/// behaviour the test above used to assert unconditionally, now something an operator has to
/// ask for.
/// </summary>
[Fact]
public async Task Deployment_seals_without_a_node_flagged_for_maintenance()
{
await using var harness = await TwoNodeClusterHarness.StartAsync();
await harness.SeedDefaultClusterAsync();
await harness.StopNodeBAsync();
await harness.WaitForClusterSizeAsync(1, TimeSpan.FromSeconds(20));
await using (var setup = await CreateDbAsync(harness))
{
var nodeB = await setup.ClusterNodes.FirstAsync(n => n.NodeId == harness.NodeBNodeId, Ct);
nodeB.MaintenanceMode = true;
await setup.SaveChangesAsync(Ct);
// Still Enabled — DraftValidator.ValidateClusterTopology requires the enabled-node count
// to equal ServerCluster.NodeCount, which is why MaintenanceMode exists as its own flag.
nodeB.Enabled.ShouldBeTrue();
}
await using var scope = harness.NodeA.Services.CreateAsyncScope();
var client = scope.ServiceProvider.GetRequiredService<IAdminOperationsClient>();
var result = await client.StartDeploymentAsync(createdBy: "alice@test", Ct);
result.Outcome.ShouldBe(StartDeploymentOutcome.Accepted, $"Deploy not accepted: {result.Message}");
var deploymentId = result.DeploymentId!.Value.Value;
await WaitForAsync(async () =>
{
await using var pollDb = await CreateDbAsync(harness);
var d = await pollDb.Deployments.AsNoTracking()
.FirstOrDefaultAsync(d => d.DeploymentId == deploymentId, Ct);
return d?.Status == DeploymentStatus.Sealed;
}, TwoNodeClusterHarness.DeploySealTimeout);
await using var db = await CreateDbAsync(harness);
var nodeStates = await db.NodeDeploymentStates.AsNoTracking()
.Where(s => s.DeploymentId == deploymentId)
.ToListAsync(Ct);
nodeStates.Count.ShouldBe(1, "the maintenance node is not expected to ack");
nodeStates[0].NodeId.ShouldBe(harness.NodeANodeId);
nodeStates[0].Status.ShouldBe(NodeDeploymentStatus.Applied); nodeStates[0].Status.ShouldBe(NodeDeploymentStatus.Applied);
} }
@@ -67,7 +67,7 @@ public sealed class FleetDiagnosticsRoundTripTests
{ {
var snap = await diagnostics.GetDiagnosticsAsync(nodeId, Ct); var snap = await diagnostics.GetDiagnosticsAsync(nodeId, Ct);
return snap.CurrentRevision == expectedRev; return snap.CurrentRevision == expectedRev;
}, TimeSpan.FromSeconds(15)); }, TwoNodeClusterHarness.DeploySealTimeout);
} }
} }
@@ -96,6 +96,16 @@ public sealed class TwoNodeClusterHarness : IAsyncDisposable
// host:port into NodeId so the cluster membership stays distinct on different ports. // host:port into NodeId so the cluster membership stays distinct on different ports.
public const string LoopbackHost = "127.0.0.1"; public const string LoopbackHost = "127.0.0.1";
/// <summary>
/// How long a deploy-seal assertion waits. Generous on purpose: since per-cluster mesh
/// Phase 1 the coordinator waits for a real ApplyAck from every configured node before
/// sealing, so this measures an end-to-end two-node deploy rather than — as it did when the
/// expected-ack set came up empty — an immediate seal. The old 15s had enormous margin
/// against "instant" and thin margin against the real thing, which showed up as deploy
/// tests failing only under full-suite CPU contention.
/// </summary>
public static readonly TimeSpan DeploySealTimeout = TimeSpan.FromSeconds(45);
/// <summary>Gets the Akka ActorSystem for node A.</summary> /// <summary>Gets the Akka ActorSystem for node A.</summary>
public ActorSystem NodeASystem => NodeA.Services.GetRequiredService<ActorSystem>(); public ActorSystem NodeASystem => NodeA.Services.GetRequiredService<ActorSystem>();
/// <summary>Gets the Akka ActorSystem for node B.</summary> /// <summary>Gets the Akka ActorSystem for node B.</summary>
@@ -122,16 +132,26 @@ public sealed class TwoNodeClusterHarness : IAsyncDisposable
/// Seeds a default <see cref="ServerCluster"/> plus a <see cref="ClusterNode"/> row for BOTH /// Seeds a default <see cref="ServerCluster"/> plus a <see cref="ClusterNode"/> row for BOTH
/// harness nodes (<see cref="NodeANodeId"/> / <see cref="NodeBNodeId"/>) so the real-SQL FK /// harness nodes (<see cref="NodeANodeId"/> / <see cref="NodeBNodeId"/>) so the real-SQL FK
/// constraint <c>FK_NodeDeploymentState_ClusterNode_NodeId</c> is satisfied when a deployment /// constraint <c>FK_NodeDeploymentState_ClusterNode_NodeId</c> is satisfied when a deployment
/// records per-node state. The EF in-memory provider ignores FK constraints, so deploy E2E /// records per-node state. Against SQL Server each node's <c>NodeDeploymentState</c> INSERT
/// tests pass without this; against SQL Server each node's <c>NodeDeploymentState</c> INSERT
/// fails without its parent <see cref="ClusterNode"/> row and the deployment never seals. /// fails without its parent <see cref="ClusterNode"/> row and the deployment never seals.
/// No-op unless <c>OTOPCUA_HARNESS_USE_SQL=1</c> (in-memory needs no seeding). Call once, before /// Call once, before <c>StartDeploymentAsync</c>, in tests that don't already seed their own
/// <c>StartDeploymentAsync</c>, in tests that don't already seed their own cluster + both nodes. /// cluster + both nodes.
/// </summary> /// </summary>
/// <remarks>
/// <b>This used to no-op in in-memory mode</b> ("the in-memory provider ignores FK
/// constraints, so deploy E2E tests pass without it"). That stopped being true in per-cluster
/// mesh Phase 1: <c>ConfigPublishCoordinator</c> now derives its expected-ack set from these
/// rows rather than from cluster membership, so with none seeded it seals <i>immediately</i>
/// with an empty set. "Wait for Sealed" then no longer implies "every node has applied", and
/// since <c>DriverHostActor.UpsertNodeDeploymentState</c> writes each node's row on its own
/// schedule, any test counting those rows after a seal became a race — reproducibly green
/// alone and red under suite load. Seeding in both modes restores the invariant the tests
/// were written against, and is closer to production either way: a real fleet always has
/// these rows, because the FK requires them.
/// </remarks>
/// <param name="clusterId">Cluster id for the seeded rows. Defaults to <c>MAIN</c>.</param> /// <param name="clusterId">Cluster id for the seeded rows. Defaults to <c>MAIN</c>.</param>
public async Task SeedDefaultClusterAsync(string clusterId = "MAIN") public async Task SeedDefaultClusterAsync(string clusterId = "MAIN")
{ {
if (!Mode.UseSqlServer) return;
await using var db = await CreateConfigDbContextAsync(); await using var db = await CreateConfigDbContextAsync();
db.ServerClusters.Add(new ServerCluster db.ServerClusters.Add(new ServerCluster
{ {