Compare commits
10 Commits
e7311f11a6
...
7654f24dab
| Author | SHA1 | Date | |
|---|---|---|---|
| 7654f24dab | |||
| 0b7c53f64f | |||
| 14746f2995 | |||
| e27b7f43f5 | |||
| ee69caa270 | |||
| 57e1d01766 | |||
| 90c1302f71 | |||
| d88e245503 | |||
| da74ebd696 | |||
| 2bbb02713c |
@@ -228,7 +228,14 @@ The server supports non-transparent warm/hot redundancy via the `Redundancy` sec
|
|||||||
|
|
||||||
Assert the **effective** cluster config off a running `ActorSystem` (as `SplitBrainResolverActivationTests` does), never the typed options or the akka.conf text — several HOCON fragments compete and the losing one still reads correctly in the file.
|
Assert the **effective** cluster config off a running `ActorSystem` (as `SplitBrainResolverActivationTests` does), never the typed options or the akka.conf text — several HOCON fragments compete and the losing one still reads correctly in the file.
|
||||||
|
|
||||||
**Still per-Akka-cluster, not per application `Cluster`.** One Primary is elected across the whole Akka mesh, so a fleet running several application clusters in one mesh (what docker-dev models) gets one Primary among all driver nodes. The fix is the per-cluster mesh workstream: `docs/plans/2026-07-21-per-cluster-mesh-design.md` (Phases 0a/0b done; 1–7 not started, Phase 1 planned in `…-phase1.md`).
|
**Still per-Akka-cluster, not per application `Cluster`.** One Primary is elected across the whole Akka mesh, so a fleet running several application clusters in one mesh (what docker-dev models) gets one Primary among all driver nodes. The fix is the per-cluster mesh workstream: `docs/plans/2026-07-21-per-cluster-mesh-design.md` (Phases 0a/0b/1 done; 2–7 not started).
|
||||||
|
|
||||||
|
**Phase 1 landed 2026-07-22 and changed a deploy-path behaviour.** `ConfigPublishCoordinator` sources its expected-ack set from **enabled `ClusterNode` rows**, not from `Akka.Cluster.State.Members` filtered by the `driver` role — central must be able to name a deployment's nodes without sharing a gossip ring with them. Consequences worth knowing before touching deploys:
|
||||||
|
|
||||||
|
- **A configured node that is switched off now FAILS the deployment** at the apply deadline instead of letting it seal green without that node. The maintenance hatch is the **new `ClusterNode.MaintenanceMode` column**, NOT `Enabled = 0`: the live gate showed `Enabled = 0` is rejected outright by `DraftValidator.ValidateClusterTopology` (`ClusterEnabledNodeCountMismatch` — enabled count must equal `ServerCluster.NodeCount`), so on a 2-node pair — every cluster in the target topology — it can never be the hatch. `Enabled` = part of the declared topology; `MaintenanceMode` = do not expect it now. The deadline log names the silent nodes and points at the right flag.
|
||||||
|
- **Every `ClusterNode` row is now *defined* to be a driver node.** The DB has no per-node role column and deliberately did not gain one (it would drift from `Cluster:Roles`). Giving an admin-only node a `ClusterNode` row makes every deploy time out.
|
||||||
|
- **There is no such thing as a cluster-scoped deployment.** `Deployment` has no `ClusterId`, `ConfigComposer` always snapshots the whole DB, and `DeploymentArtifact.ResolveClusterScope` is *node-side* self-scoping of a fleet-wide artifact. The phase-1 plan asked for cluster-scope filtering of the ack set; it was dropped as describing a feature that does not exist.
|
||||||
|
- `ClusterNode` gained `AkkaPort` (NOT NULL, 4053) + `GrpcPort` (nullable) as **central's dial targets** for Phases 2/5. They duplicate the node's own `Cluster:Port` / `Cluster:PublicHostname` with no schema-level enforcement, so `ClusterNodeAddressReconcilerActor` (admin singleton) logs an Error on drift. **Phase 2 must revisit it** — once the meshes split, an admin node cannot see site members and every site row would report `EnabledRowNotInCluster` forever.
|
||||||
|
|
||||||
## LocalDb pair-local store (Phases 1 + 2)
|
## LocalDb pair-local store (Phases 1 + 2)
|
||||||
|
|
||||||
|
|||||||
@@ -64,13 +64,13 @@ IF NOT EXISTS (SELECT 1 FROM dbo.ServerCluster WHERE ClusterId = 'SITE-B')
|
|||||||
|
|
||||||
IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'central-1:4053')
|
IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'central-1:4053')
|
||||||
INSERT INTO dbo.ClusterNode
|
INSERT INTO dbo.ClusterNode
|
||||||
(NodeId, ClusterId, Host, OpcUaPort, DashboardPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
|
(NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
|
||||||
VALUES ('central-1:4053', 'MAIN', 'central-1', 4840, 8081, 'urn:OtOpcUa:central-1', 200, 1, 'docker-dev-seed');
|
VALUES ('central-1:4053', 'MAIN', 'central-1', 4840, 8081, 4053, 'urn:OtOpcUa:central-1', 200, 1, 'docker-dev-seed');
|
||||||
|
|
||||||
IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'central-2:4053')
|
IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'central-2:4053')
|
||||||
INSERT INTO dbo.ClusterNode
|
INSERT INTO dbo.ClusterNode
|
||||||
(NodeId, ClusterId, Host, OpcUaPort, DashboardPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
|
(NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
|
||||||
VALUES ('central-2:4053', 'MAIN', 'central-2', 4840, 8081, 'urn:OtOpcUa:central-2', 150, 1, 'docker-dev-seed');
|
VALUES ('central-2:4053', 'MAIN', 'central-2', 4840, 8081, 4053, 'urn:OtOpcUa:central-2', 150, 1, 'docker-dev-seed');
|
||||||
|
|
||||||
------------------------------------------------------------------------------
|
------------------------------------------------------------------------------
|
||||||
-- ClusterNode — site A
|
-- ClusterNode — site A
|
||||||
@@ -78,13 +78,13 @@ IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'central-2:4053')
|
|||||||
|
|
||||||
IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'site-a-1:4053')
|
IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'site-a-1:4053')
|
||||||
INSERT INTO dbo.ClusterNode
|
INSERT INTO dbo.ClusterNode
|
||||||
(NodeId, ClusterId, Host, OpcUaPort, DashboardPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
|
(NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
|
||||||
VALUES ('site-a-1:4053', 'SITE-A', 'site-a-1', 4840, 8081, 'urn:OtOpcUa:site-a-1', 200, 1, 'docker-dev-seed');
|
VALUES ('site-a-1:4053', 'SITE-A', 'site-a-1', 4840, 8081, 4053, 'urn:OtOpcUa:site-a-1', 200, 1, 'docker-dev-seed');
|
||||||
|
|
||||||
IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'site-a-2:4053')
|
IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'site-a-2:4053')
|
||||||
INSERT INTO dbo.ClusterNode
|
INSERT INTO dbo.ClusterNode
|
||||||
(NodeId, ClusterId, Host, OpcUaPort, DashboardPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
|
(NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
|
||||||
VALUES ('site-a-2:4053', 'SITE-A', 'site-a-2', 4840, 8081, 'urn:OtOpcUa:site-a-2', 150, 1, 'docker-dev-seed');
|
VALUES ('site-a-2:4053', 'SITE-A', 'site-a-2', 4840, 8081, 4053, 'urn:OtOpcUa:site-a-2', 150, 1, 'docker-dev-seed');
|
||||||
|
|
||||||
------------------------------------------------------------------------------
|
------------------------------------------------------------------------------
|
||||||
-- ClusterNode — site B
|
-- ClusterNode — site B
|
||||||
@@ -92,13 +92,13 @@ IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'site-a-2:4053')
|
|||||||
|
|
||||||
IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'site-b-1:4053')
|
IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'site-b-1:4053')
|
||||||
INSERT INTO dbo.ClusterNode
|
INSERT INTO dbo.ClusterNode
|
||||||
(NodeId, ClusterId, Host, OpcUaPort, DashboardPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
|
(NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
|
||||||
VALUES ('site-b-1:4053', 'SITE-B', 'site-b-1', 4840, 8081, 'urn:OtOpcUa:site-b-1', 200, 1, 'docker-dev-seed');
|
VALUES ('site-b-1:4053', 'SITE-B', 'site-b-1', 4840, 8081, 4053, 'urn:OtOpcUa:site-b-1', 200, 1, 'docker-dev-seed');
|
||||||
|
|
||||||
IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'site-b-2:4053')
|
IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'site-b-2:4053')
|
||||||
INSERT INTO dbo.ClusterNode
|
INSERT INTO dbo.ClusterNode
|
||||||
(NodeId, ClusterId, Host, OpcUaPort, DashboardPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
|
(NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
|
||||||
VALUES ('site-b-2:4053', 'SITE-B', 'site-b-2', 4840, 8081, 'urn:OtOpcUa:site-b-2', 150, 1, 'docker-dev-seed');
|
VALUES ('site-b-2:4053', 'SITE-B', 'site-b-2', 4840, 8081, 4053, 'urn:OtOpcUa:site-b-2', 150, 1, 'docker-dev-seed');
|
||||||
|
|
||||||
------------------------------------------------------------------------------
|
------------------------------------------------------------------------------
|
||||||
-- Galaxy MxAccess gateway — INTENTIONALLY NOT SEEDED (retired 2026-06-15)
|
-- Galaxy MxAccess gateway — INTENTIONALLY NOT SEEDED (retired 2026-06-15)
|
||||||
|
|||||||
@@ -101,13 +101,15 @@ The checked-in `appsettings*.json` files are deliberately thin: they carry only
|
|||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| `SystemName` | string | `otopcua` | Akka actor-system name. |
|
| `SystemName` | string | `otopcua` | Akka actor-system name. |
|
||||||
| `Hostname` | string | `0.0.0.0` | Bind hostname. |
|
| `Hostname` | string | `0.0.0.0` | Bind hostname. |
|
||||||
| `Port` | int | `4053` | Cluster transport port. |
|
| `Port` | int | `4053` | Cluster transport port. **Duplicated as `ClusterNode.AkkaPort` in the Config DB** — see the note below. |
|
||||||
| `PublicHostname` | string | `127.0.0.1` | Hostname advertised in cluster gossip; must be reachable by peers. |
|
| `PublicHostname` | string | `127.0.0.1` | Hostname advertised in cluster gossip; must be reachable by peers. **Duplicated as `ClusterNode.Host`** — see the note below. |
|
||||||
| `SeedNodes` | string[] | `[]` | Seed nodes for bootstrapping. **Ordered:** a node that appears in its own list must be entry 0 (only `seed-nodes[0]` can form a new cluster), or the host refuses to start — see [Redundancy.md § Bootstrap: self-first seed ordering](Redundancy.md#bootstrap-self-first-seed-ordering). |
|
| `SeedNodes` | string[] | `[]` | Seed nodes for bootstrapping. **Ordered:** a node that appears in its own list must be entry 0 (only `seed-nodes[0]` can form a new cluster), or the host refuses to start — see [Redundancy.md § Bootstrap: self-first seed ordering](Redundancy.md#bootstrap-self-first-seed-ordering). |
|
||||||
| `Roles` | string[] | `[]` | Cluster roles for this node. When empty, falls back to `OTOPCUA_ROLES`. Allowed values: `admin`, `driver`, `dev`. |
|
| `Roles` | string[] | `[]` | Cluster roles for this node. When empty, falls back to `OTOPCUA_ROLES`. Allowed values: `admin`, `driver`, `dev`. |
|
||||||
|
|
||||||
> The full redundancy model (ServiceLevel tiers, split-brain, peer discovery) is in [`Redundancy.md`](Redundancy.md). The OPC UA peer-URI advertising lives in the `OpcUa:PeerApplicationUris` key above.
|
> The full redundancy model (ServiceLevel tiers, split-brain, peer discovery) is in [`Redundancy.md`](Redundancy.md). The OPC UA peer-URI advertising lives in the `OpcUa:PeerApplicationUris` key above.
|
||||||
|
|
||||||
|
> **`Port` / `PublicHostname` are stored twice.** The node binds from these keys; the fleet's `ClusterNode` row records the same address as `AkkaPort` / `Host` so that **central can dial the node without sharing a gossip ring with it** — which is what [per-cluster mesh Phase 2](plans/2026-07-21-per-cluster-mesh-design.md) needs. Nothing in the schema makes the two agree, so `ClusterNodeAddressReconcilerActor` (admin-role singleton) compares them against live membership and logs an Error on mismatch. If you change `Cluster:Port` or `Cluster:PublicHostname` on a node, **update its `ClusterNode` row too** — a node binding 4054 with a row saying 4053 still gossips fine today, and fails silently in Phase 2. The row's `NodeId` is `host:port` and is also the deploy path's ack identity, so a drifted node's deployment acks stop matching as well. See [`config-db-schema.md` § `ClusterNode`](v2/config-db-schema.md#clusternode).
|
||||||
|
|
||||||
### `ConnectionStrings` → `ConfigDb`
|
### `ConnectionStrings` → `ConfigDb`
|
||||||
|
|
||||||
- **Purpose:** the central Config DB connection string. **Required for every role** — `Program.cs` calls `AddOtOpcUaConfigDb` unconditionally.
|
- **Purpose:** the central Config DB connection string. **Required for every role** — `Program.cs` calls `AddOtOpcUaConfigDb` unconditionally.
|
||||||
|
|||||||
@@ -287,7 +287,7 @@ Deliberately not a task plan — per-phase plans follow, one at a time.
|
|||||||
|---|---|---|
|
|---|---|---|
|
||||||
| 0a | Downing strategy: keep-oldest cannot survive an oldest-node crash in a 2-node cluster (§6.2) | **Yes** — **DONE 2026-07-21**, live gate deferred to Phase 7 |
|
| 0a | Downing strategy: keep-oldest cannot survive an oldest-node crash in a 2-node cluster (§6.2) | **Yes** — **DONE 2026-07-21**, live gate deferred to Phase 7 |
|
||||||
| 0b | Oldest-Up role derivation (§4) | **Yes** — **DONE 2026-07-21** |
|
| 0b | Oldest-Up role derivation (§4) | **Yes** — **DONE 2026-07-21** |
|
||||||
| 1 | `ClusterNode` gains Akka + gRPC address columns; coordinator sources its expected-ack set from the DB | Yes |
|
| 1 | `ClusterNode` gains Akka + gRPC address columns; coordinator sources its expected-ack set from the DB | **Yes** — **DONE 2026-07-22** (see below) |
|
||||||
| 2 | Comm actors + receptionist registration; ClusterClient transport; deploy notify + acks across the boundary | No |
|
| 2 | Comm actors + receptionist registration; ClusterClient transport; deploy notify + acks across the boundary | No |
|
||||||
| 3 | Config fetch-and-cache: artifact served by central, driver nodes read LocalDb (§6.1) | No |
|
| 3 | Config fetch-and-cache: artifact served by central, driver nodes read LocalDb (§6.1) | No |
|
||||||
| 4 | Cut the driver-side ConfigDb connection: re-home `EfAlarmConditionStateStore`, resolve the `DbHealthProbeActor` ServiceLevel input, audit `OpcUaPublishActor` | No |
|
| 4 | Cut the driver-side ConfigDb connection: re-home `EfAlarmConditionStateStore`, resolve the `DbHealthProbeActor` ServiceLevel input, audit `OpcUaPublishActor` | No |
|
||||||
@@ -298,6 +298,25 @@ Deliberately not a task plan — per-phase plans follow, one at a time.
|
|||||||
Phases 3 and 4 are the ones that change a running system's data path rather than its wiring, and each
|
Phases 3 and 4 are the ones that change a running system's data path rather than its wiring, and each
|
||||||
deserves its own live gate.
|
deserves its own live gate.
|
||||||
|
|
||||||
|
### Phase 1 as shipped (2026-07-22)
|
||||||
|
|
||||||
|
Plan: `2026-07-21-per-cluster-mesh-phase1.md`. Two deviations from the sketch above, both decided
|
||||||
|
before implementing:
|
||||||
|
|
||||||
|
- **No cluster-scope filtering of the expected-ack set.** The plan called for it, but there is no
|
||||||
|
cluster-scoped deployment to filter on: `Deployment` has no `ClusterId`,
|
||||||
|
`ConfigComposer.SnapshotAndFlattenAsync` always snapshots the whole DB, and
|
||||||
|
`DeploymentArtifact.ResolveClusterScope` is *node-side* self-scoping of a fleet-wide artifact. The
|
||||||
|
expected set is every enabled `ClusterNode` row, which is what the membership rule produced too.
|
||||||
|
- **No per-node role column.** The membership rule filtered on the `driver` role and the DB has none.
|
||||||
|
Rather than add one — a second declaration of node roles, free to drift from `Cluster:Roles` — every
|
||||||
|
`ClusterNode` row is now *defined* to be a driver node. An admin-only node must not be given one.
|
||||||
|
|
||||||
|
Also shipped beyond the sketch: `ClusterNodeAddressReconciler` (Task 4), an admin singleton that
|
||||||
|
catches `AkkaPort` drifting from the node's own `Cluster:Port`. **Phase 2 must revisit it** — once the
|
||||||
|
meshes split, an admin node cannot see site members and every site row would report
|
||||||
|
`EnabledRowNotInCluster` forever.
|
||||||
|
|
||||||
## 8. Risks
|
## 8. Risks
|
||||||
|
|
||||||
- **LocalDb becomes load-bearing for configuration, not just resilient.** Phase 1 built it as a
|
- **LocalDb becomes load-bearing for configuration, not just resilient.** Phase 1 built it as a
|
||||||
|
|||||||
@@ -12,8 +12,17 @@ sourcing its expected-ack set from those rows.
|
|||||||
**Architecture:** Additive schema change plus one substitution inside the coordinator. Nothing else
|
**Architecture:** Additive schema change plus one substitution inside the coordinator. Nothing else
|
||||||
moves; the deploy channel stays on DistributedPubSub until Phase 2.
|
moves; the deploy channel stays on DistributedPubSub until Phase 2.
|
||||||
|
|
||||||
**Status:** NOT STARTED. Read §"The decision this phase forces" before executing — it changes
|
**Status: DONE 2026-07-22** (branch `feat/mesh-phase1`). Live gate:
|
||||||
deploy-seal semantics and should be confirmed, not assumed.
|
[`2026-07-22-mesh-phase1-live-gate.md`](2026-07-22-mesh-phase1-live-gate.md) — **PASSED**, after gate
|
||||||
|
step 4 found that the escape hatch this phase depends on did not exist. Three deviations from the
|
||||||
|
plan below, all decided before implementing or forced by the gate:
|
||||||
|
|
||||||
|
1. **Task 3's cluster-scope filtering was dropped** — there is no cluster-scoped deployment to filter
|
||||||
|
on. See the gate doc, step 2.
|
||||||
|
2. **No per-node role column.** Every `ClusterNode` row is now *defined* to be a driver node.
|
||||||
|
3. **Task 7 added:** `ClusterNode.MaintenanceMode`. `Enabled = 0` is rejected by
|
||||||
|
`DraftValidator.ValidateClusterTopology` on any 2-node pair, so it could never be the maintenance
|
||||||
|
hatch — which the plan called "the check that makes step 3 acceptable to ship".
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -54,7 +63,7 @@ But it is a behaviour change, and it needs an operator escape hatch or a node ta
|
|||||||
maintenance blocks every deployment. `ClusterNode.Enabled` already exists and is exactly that hatch,
|
maintenance blocks every deployment. `ClusterNode.Enabled` already exists and is exactly that hatch,
|
||||||
so the query filters on it: **`Enabled = 1` rows are expected to ack; disabled rows are not.**
|
so the query filters on it: **`Enabled = 1` rows are expected to ack; disabled rows are not.**
|
||||||
|
|
||||||
**Confirm this before executing Task 3.** If the preference is to keep sealing green past a down
|
**Confirmed 2026-07-22 before executing Task 3** — DB-derived with the hatch. (The hatch turned out to be `MaintenanceMode`, not `Enabled`; see Task 7.) If the preference is to keep sealing green past a down
|
||||||
node, the alternative is to intersect the DB set with current membership, which preserves today's
|
node, the alternative is to intersect the DB set with current membership, which preserves today's
|
||||||
behaviour but does not survive the mesh split — it would have to be revisited in Phase 2 anyway.
|
behaviour but does not survive the mesh split — it would have to be revisited in Phase 2 anyway.
|
||||||
|
|
||||||
@@ -189,3 +198,25 @@ disagree on the rig. Run on `docker-dev`:
|
|||||||
hatch, and is the check that makes step 3 acceptable to ship.
|
hatch, and is the check that makes step 3 acceptable to ship.
|
||||||
|
|
||||||
Record results in a gate doc beside this plan, per the Phase 1/Phase 2 precedent.
|
Record results in a gate doc beside this plan, per the Phase 1/Phase 2 precedent.
|
||||||
|
|
||||||
|
## Task 7: `MaintenanceMode` — the escape hatch step 4 proved missing (ADDED 2026-07-22)
|
||||||
|
|
||||||
|
**Classification:** high-risk — deploy path + schema
|
||||||
|
**Why:** gate step 4 returned `422 ClusterEnabledNodeCountMismatch`. `Enabled = 0` on one node of a
|
||||||
|
Warm/Hot pair fails `DraftValidator.ValidateClusterTopology` (enabled count must equal `NodeCount`),
|
||||||
|
so the hatch Task 3 relies on cannot be used on any 2-node cluster — i.e. every cluster in the target
|
||||||
|
topology. Task 3's behaviour change is only shippable with a hatch that works.
|
||||||
|
|
||||||
|
**Decision:** split the meanings rather than weaken either rule.
|
||||||
|
|
||||||
|
| Flag | Question | Enforced by |
|
||||||
|
|---|---|---|
|
||||||
|
| `Enabled` | part of the declared topology? | `DraftValidator` vs `ServerCluster.NodeCount` |
|
||||||
|
| `MaintenanceMode` | expected to participate right now? | `ConfigPublishCoordinator`, `ClusterNodeAddressReconciler` |
|
||||||
|
|
||||||
|
Rejected: counting configured rather than enabled nodes (removes the InvalidTopology guard);
|
||||||
|
downgrading the rule to a warning (weakens a deploy gate for everyone); shipping without a hatch.
|
||||||
|
|
||||||
|
**Done:** column + migration `AddClusterNodeMaintenanceMode`; both consumers filter on
|
||||||
|
`Enabled && !MaintenanceMode`; deadline + reconciler messages name the right flag; docs; gate step 4
|
||||||
|
re-run green.
|
||||||
|
|||||||
@@ -0,0 +1,76 @@
|
|||||||
|
{
|
||||||
|
"plan": "docs/plans/2026-07-21-per-cluster-mesh-phase1.md",
|
||||||
|
"branch": "feat/mesh-phase1",
|
||||||
|
"decisions": {
|
||||||
|
"seal-semantics": "DB-derived + Enabled hatch (confirmed 2026-07-22)",
|
||||||
|
"cluster-scope": "Dropped \u2014 no cluster-scoped deployment exists; expected set = all enabled ClusterNode rows",
|
||||||
|
"node-roles": "Every ClusterNode row is a driver node; documented + pinned by test, no Roles column",
|
||||||
|
"adminui-edit": "Deferred to Phase 2 (nothing reads the columns until then)",
|
||||||
|
"maintenance-hatch": "MaintenanceMode column, not Enabled=0 (blocked by DraftValidator on any 2-node pair)"
|
||||||
|
},
|
||||||
|
"tasks": [
|
||||||
|
{
|
||||||
|
"id": 1,
|
||||||
|
"subject": "Address columns on ClusterNode",
|
||||||
|
"status": "completed",
|
||||||
|
"commit": "2bbb0271",
|
||||||
|
"blockedBy": []
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": 2,
|
||||||
|
"subject": "EF migration AddClusterNodeTransportPorts",
|
||||||
|
"status": "completed",
|
||||||
|
"commit": "da74ebd6",
|
||||||
|
"blockedBy": [
|
||||||
|
1
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": 3,
|
||||||
|
"subject": "Coordinator sources expected-ack set from ClusterNode",
|
||||||
|
"status": "completed",
|
||||||
|
"commit": "d88e2455",
|
||||||
|
"blockedBy": [
|
||||||
|
2
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": 4,
|
||||||
|
"subject": "Reconcile duplicated AkkaPort with the node's own Cluster:Port",
|
||||||
|
"status": "completed",
|
||||||
|
"blockedBy": [
|
||||||
|
2
|
||||||
|
],
|
||||||
|
"commit": "90c1302f"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": 5,
|
||||||
|
"subject": "Rig seed + docs",
|
||||||
|
"status": "completed",
|
||||||
|
"blockedBy": [
|
||||||
|
3,
|
||||||
|
4
|
||||||
|
],
|
||||||
|
"commit": "57e1d017"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": 6,
|
||||||
|
"subject": "Live gate on docker-dev",
|
||||||
|
"status": "completed",
|
||||||
|
"blockedBy": [
|
||||||
|
5
|
||||||
|
],
|
||||||
|
"result": "live-gate PASSED (2026-07-22-mesh-phase1-live-gate.md)"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": 7,
|
||||||
|
"subject": "MaintenanceMode \u2014 escape hatch the live gate proved missing",
|
||||||
|
"status": "completed",
|
||||||
|
"commit": "ee69caa2",
|
||||||
|
"blockedBy": [
|
||||||
|
6
|
||||||
|
],
|
||||||
|
"origin": "added from gate step 4 failure"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
@@ -0,0 +1,134 @@
|
|||||||
|
# Per-cluster mesh Phase 1 — live gate record
|
||||||
|
|
||||||
|
**Date:** 2026-07-22 · **Rig:** `docker-dev` (six-node single mesh) · **Branch:** `feat/mesh-phase1`
|
||||||
|
**Plan:** [`2026-07-21-per-cluster-mesh-phase1.md`](2026-07-21-per-cluster-mesh-phase1.md)
|
||||||
|
|
||||||
|
**Result: PASSED**, after the gate found a blocker that changed the design (step 4).
|
||||||
|
|
||||||
|
Offline tests cannot cover the substitution's real risk — that the DB set and the live set disagree on
|
||||||
|
a running fleet. This is that check.
|
||||||
|
|
||||||
|
## Setup
|
||||||
|
|
||||||
|
Migration applied by the rig's `migrator` service against the shared `OtOpcUa` database, then all six
|
||||||
|
host nodes restarted on the rebuilt image. **All six pre-existing `ClusterNode` rows inherited
|
||||||
|
`AkkaPort = 4053` from the migration's `DEFAULT 4053`** — the row-level evidence that the default does
|
||||||
|
its job for rows predating the column:
|
||||||
|
|
||||||
|
```
|
||||||
|
central-1:4053|central-1|4053|NULL|1
|
||||||
|
central-2:4053|central-2|4053|NULL|1
|
||||||
|
site-a-1:4053 |site-a-1 |4053|NULL|1
|
||||||
|
site-a-2:4053 |site-a-2 |4053|NULL|1
|
||||||
|
site-b-1:4053 |site-b-1 |4053|NULL|1
|
||||||
|
site-b-2:4053 |site-b-2 |4053|NULL|1
|
||||||
|
NodeId | Host |Akka|Grpc|En
|
||||||
|
```
|
||||||
|
|
||||||
|
## Step 1 — deploy with all six nodes up → **PASS**
|
||||||
|
|
||||||
|
`POST /api/deployments` → `202 Accepted`, deployment `38f83b37`.
|
||||||
|
|
||||||
|
`Deployment.Status = 2` (Sealed). Six `NodeDeploymentState` rows, every one `Status = 1` (Applied).
|
||||||
|
**No FK 547** — the DB-derived NodeIds satisfy the `NodeDeploymentState → ClusterNode` foreign key,
|
||||||
|
which is the standing proof that the DB set and the membership set agree on this rig.
|
||||||
|
|
||||||
|
## Step 2 — *dropped, not skipped*
|
||||||
|
|
||||||
|
The plan called for a cluster-scoped deploy. **There is no such thing.** `Deployment` has no
|
||||||
|
`ClusterId`, `ConfigComposer.SnapshotAndFlattenAsync` always snapshots the whole DB, and
|
||||||
|
`DeploymentArtifact.ResolveClusterScope` is *node-side* self-scoping of a fleet-wide artifact. Removed
|
||||||
|
from the plan rather than faked green.
|
||||||
|
|
||||||
|
## Step 3 — deploy with one node stopped → **PASS** (the new behaviour)
|
||||||
|
|
||||||
|
`docker stop otopcua-dev-site-b-2-1`, then deploy `37d72301`.
|
||||||
|
|
||||||
|
`Deployment.Status = 4` (TimedOut) after the 2-minute apply deadline. Five nodes `Applied`,
|
||||||
|
`site-b-2:4053` still `Applying`. Under the membership rule this deployment would have **sealed
|
||||||
|
green** with five nodes and never mentioned the sixth.
|
||||||
|
|
||||||
|
The operator-facing log, which the plan required to be legible:
|
||||||
|
|
||||||
|
```
|
||||||
|
[12:52:02 WRN] Deployment 37d723018e374c0eb04c7b6beccc3b3e timed out after 00:02:00 (5/6 acks
|
||||||
|
landed). No ack from: site-b-2:4053. Each is an enabled ClusterNode row — start the node, or set
|
||||||
|
ClusterNode.Enabled = 0 if it is down for maintenance, then redeploy
|
||||||
|
```
|
||||||
|
|
||||||
|
(The remedy in that message was corrected to `MaintenanceMode = 1` — see step 4.)
|
||||||
|
|
||||||
|
## Step 4 — the escape hatch → **FAILED, then fixed and re-run → PASS**
|
||||||
|
|
||||||
|
### First attempt: the hatch did not exist
|
||||||
|
|
||||||
|
`UPDATE ClusterNode SET Enabled = 0 WHERE NodeId = 'site-b-2:4053'`, then deploy:
|
||||||
|
|
||||||
|
```
|
||||||
|
HTTP 422
|
||||||
|
[ClusterEnabledNodeCountMismatch] Cluster 'SITE-B' declares NodeCount=2 but has 1 Enabled nodes.
|
||||||
|
Toggle the missing node(s) back on or change RedundancyMode/NodeCount to match.
|
||||||
|
```
|
||||||
|
|
||||||
|
`DraftValidator.ValidateClusterTopology` rejects the deployment **before it dispatches**. So
|
||||||
|
`Enabled = 0` cannot be used on a node of a 2-node cluster — and every cluster on this rig, and every
|
||||||
|
cluster in the program plan's target topology, is a 2-node pair.
|
||||||
|
|
||||||
|
**This was a blocker, not a wrinkle.** The plan called step 4 *"the check that makes step 3
|
||||||
|
acceptable to ship"*. Without it, a node down for maintenance blocks every deployment to its cluster,
|
||||||
|
with no remedy short of editing `NodeCount` / `RedundancyMode` — which changes the runtime redundancy
|
||||||
|
posture to work around a deploy gate.
|
||||||
|
|
||||||
|
The two rules were each reasonable and contradictory together: the validator reads `Enabled` as *"part
|
||||||
|
of the declared topology"*, Phase 1 additionally read it as *"expect an ack"*.
|
||||||
|
|
||||||
|
### Fix (Task 7): split the meanings
|
||||||
|
|
||||||
|
New `ClusterNode.MaintenanceMode bit NOT NULL DEFAULT 0`. `Enabled` keeps its topology meaning and the
|
||||||
|
validator is untouched; the coordinator's expected-ack set and the address reconciler both filter on
|
||||||
|
`Enabled && !MaintenanceMode`.
|
||||||
|
|
||||||
|
### Re-run: PASS
|
||||||
|
|
||||||
|
`docker stop` `site-b-2` + `MaintenanceMode = 1`, then deploy `82c5e678`:
|
||||||
|
|
||||||
|
- `202 Accepted` — the topology gate passes, because the node is still `Enabled` (`1|1`).
|
||||||
|
- `Deployment.Status = 2` (Sealed).
|
||||||
|
- **Five** `NodeDeploymentState` rows, all `Applied`. `site-b-2:4053` has no row at all — excluded
|
||||||
|
from the expected set rather than waited for and forgiven.
|
||||||
|
|
||||||
|
## Task 4 (reconciler) — live behaviour, not a planned gate step
|
||||||
|
|
||||||
|
Observed unprompted during the rolling restart, which is better evidence than a staged check:
|
||||||
|
|
||||||
|
```
|
||||||
|
[12:44:49 WRN] ClusterNode address check [EnabledRowNotInCluster] site-a-1:4053: ...
|
||||||
|
[12:44:49 WRN] ... site-a-2:4053 ... site-b-1:4053 ... site-b-2:4053
|
||||||
|
[12:44:56 INF] ClusterNode addresses reconcile with cluster membership (6 driver members)
|
||||||
|
```
|
||||||
|
|
||||||
|
It warned per row while the site nodes were still starting, then converged to a single clean line —
|
||||||
|
and **logged nothing further**, confirming the change-detection guard suppresses repeats on the
|
||||||
|
5-minute sweep.
|
||||||
|
|
||||||
|
Maintenance interaction, from the step 4 re-run:
|
||||||
|
|
||||||
|
```
|
||||||
|
[13:06:02 WRN] ... site-b-2:4053 ... ← node dropped, flag not yet set
|
||||||
|
[13:06:05 INF] ... reconcile with cluster membership (5 driver members) ← flag set: silent
|
||||||
|
```
|
||||||
|
|
||||||
|
The transient warning is honest — for those three seconds the row genuinely was enabled, present in
|
||||||
|
the expected-ack set, and absent from the cluster.
|
||||||
|
|
||||||
|
## What this gate did not cover
|
||||||
|
|
||||||
|
- **Phase 2's premise.** `AkkaPort` / `GrpcPort` are groundwork; nothing dials them yet, so "the value
|
||||||
|
is correct" is only checked by the reconciler comparing it to gossip. The first real exercise is
|
||||||
|
Phase 2.
|
||||||
|
- **The reconciler under split meshes.** On the current single mesh an admin node sees every driver
|
||||||
|
member. After Phase 6 it will not, and every site row would report `EnabledRowNotInCluster`
|
||||||
|
permanently. Recorded as a known limitation on the class and in the design doc.
|
||||||
|
- **`Enabled = 0` on a single-node (`RedundancyMode.None`) cluster.** Untested — the rig has no such
|
||||||
|
cluster. It would fail the same topology rule (0 enabled vs `NodeCount = 1`), so `MaintenanceMode`
|
||||||
|
is the hatch there too.
|
||||||
@@ -82,7 +82,13 @@ per phase:** (1) invoke writing-plans in this repo to produce
|
|||||||
references, exploring current code first; (2) execute it task-by-task; (3) run the phase's exit
|
references, exploring current code first; (2) execute it task-by-task; (3) run the phase's exit
|
||||||
gate; (4) update this file's status column and the design doc's §7 table.
|
gate; (4) update this file's status column and the design doc's §7 table.
|
||||||
|
|
||||||
### Phase 1 — `ClusterNode` address columns + DB-sourced ack set
|
### Phase 1 — `ClusterNode` address columns + DB-sourced ack set — **DONE 2026-07-22**
|
||||||
|
Shipped per `2026-07-21-per-cluster-mesh-phase1.md`; see that plan's Task 6 gate record and the
|
||||||
|
design doc's "Phase 1 as shipped" note for the two scope deviations (no cluster-scope filtering — no
|
||||||
|
such deployment exists; no per-node role column) and the one addition (`ClusterNodeAddressReconciler`).
|
||||||
|
**AdminUI node edit was deferred to Phase 2** — nothing reads the columns until then, and the
|
||||||
|
migration default plus the rig seed cover every node today.
|
||||||
|
|
||||||
**Scope:** `ClusterNode` gains Akka + gRPC address columns (mirroring ScadaBridge's `Site`
|
**Scope:** `ClusterNode` gains Akka + gRPC address columns (mirroring ScadaBridge's `Site`
|
||||||
entity `NodeAAddress`/`GrpcNodeAAddress` pattern, but per-node rows); EF migration; AdminUI node
|
entity `NodeAAddress`/`GrpcNodeAAddress` pattern, but per-node rows); EF migration; AdminUI node
|
||||||
edit surfaces the fields; `ConfigPublishCoordinator` derives its expected-ack set from
|
edit surfaces the fields; `ConfigPublishCoordinator` derives its expected-ack set from
|
||||||
|
|||||||
@@ -148,10 +148,13 @@ CREATE TABLE dbo.ClusterNode (
|
|||||||
Host nvarchar(255) NOT NULL,
|
Host nvarchar(255) NOT NULL,
|
||||||
OpcUaPort int NOT NULL DEFAULT 4840,
|
OpcUaPort int NOT NULL DEFAULT 4840,
|
||||||
DashboardPort int NOT NULL DEFAULT 8081,
|
DashboardPort int NOT NULL DEFAULT 8081,
|
||||||
|
AkkaPort int NOT NULL DEFAULT 4053,
|
||||||
|
GrpcPort int NULL,
|
||||||
ApplicationUri nvarchar(256) NOT NULL,
|
ApplicationUri nvarchar(256) NOT NULL,
|
||||||
ServiceLevelBase tinyint NOT NULL DEFAULT 200,
|
ServiceLevelBase tinyint NOT NULL DEFAULT 200,
|
||||||
DriverConfigOverridesJson nvarchar(max) NULL CHECK (DriverConfigOverridesJson IS NULL OR ISJSON(DriverConfigOverridesJson) = 1),
|
DriverConfigOverridesJson nvarchar(max) NULL CHECK (DriverConfigOverridesJson IS NULL OR ISJSON(DriverConfigOverridesJson) = 1),
|
||||||
Enabled bit NOT NULL DEFAULT 1,
|
Enabled bit NOT NULL DEFAULT 1,
|
||||||
|
MaintenanceMode bit NOT NULL DEFAULT 0,
|
||||||
LastSeenAt datetime2(3) NULL,
|
LastSeenAt datetime2(3) NULL,
|
||||||
CreatedAt datetime2(3) NOT NULL DEFAULT SYSUTCDATETIME(),
|
CreatedAt datetime2(3) NOT NULL DEFAULT SYSUTCDATETIME(),
|
||||||
CreatedBy nvarchar(128) NOT NULL
|
CreatedBy nvarchar(128) NOT NULL
|
||||||
@@ -167,6 +170,49 @@ CREATE UNIQUE INDEX UX_ClusterNode_Primary_Per_Cluster
|
|||||||
WHERE RedundancyRole = 'Primary';
|
WHERE RedundancyRole = 'Primary';
|
||||||
```
|
```
|
||||||
|
|
||||||
|
#### `AkkaPort` / `GrpcPort` — central's dial targets (per-cluster mesh Phase 1)
|
||||||
|
|
||||||
|
These are **the addresses central dials**, not the node's own binding configuration. The node binds
|
||||||
|
from `Cluster:Port` in its own appsettings; these columns duplicate that value for a reader that
|
||||||
|
cannot see the node's config. Today central shares a gossip ring with every node and does not need
|
||||||
|
them — [per-cluster mesh Phase 2](../plans/2026-07-21-per-cluster-mesh-design.md) splits the fleet
|
||||||
|
into one mesh per `Cluster`, after which `Host` + `AkkaPort` is how central builds its ClusterClient
|
||||||
|
contact points, and `GrpcPort` is the Phase 5 telemetry stream.
|
||||||
|
|
||||||
|
`AkkaPort` is `NOT NULL DEFAULT 4053` because every node listens on a remoting port — `0` is never a
|
||||||
|
truthful value, and rows predating the column must migrate to something real. `GrpcPort` is nullable
|
||||||
|
with **no** default because nothing listens on it until Phase 5, and a non-null default would assert
|
||||||
|
a port that does not exist.
|
||||||
|
|
||||||
|
**The duplication against `Cluster:Port` is not enforced by the schema.** A node that binds 4054 while
|
||||||
|
its row says 4053 is unreachable from central in Phase 2, and the symptom there is a silent absence of
|
||||||
|
acks rather than an error. `ClusterNodeAddressReconcilerActor` (an admin-role singleton) compares the
|
||||||
|
rows against observed cluster membership and logs an Error on mismatch. It reads membership rather
|
||||||
|
than having each driver node assert its own row, because Phase 4 removes the driver nodes' ConfigDb
|
||||||
|
connection entirely.
|
||||||
|
|
||||||
|
#### `Enabled` vs `MaintenanceMode` — two flags, two questions
|
||||||
|
|
||||||
|
Phase 1 made the rows the deploy path's **expected-ack set**: `ConfigPublishCoordinator` no longer
|
||||||
|
derives it from cluster membership, so a row whose node is down now fails the deployment at the apply
|
||||||
|
deadline instead of letting it seal green without that node.
|
||||||
|
|
||||||
|
The escape hatch is **`MaintenanceMode = 1`**, not `Enabled = 0`:
|
||||||
|
|
||||||
|
| Flag | Question it answers | Checked by |
|
||||||
|
|---|---|---|
|
||||||
|
| `Enabled` | Is this node part of the cluster's declared topology? | `DraftValidator.ValidateClusterTopology` — the enabled-node count must equal `ServerCluster.NodeCount` |
|
||||||
|
| `MaintenanceMode` | Should we expect it to participate right now? | `ConfigPublishCoordinator` (expected-ack set) and `ClusterNodeAddressReconciler` (absence warnings) |
|
||||||
|
|
||||||
|
`Enabled = 0` on one node of a Warm/Hot pair is **rejected at the deploy gate** with
|
||||||
|
`ClusterEnabledNodeCountMismatch` — the enabled count drops to 1 while `NodeCount` stays 2. Since
|
||||||
|
every cluster in the target topology is a 2-node pair, `Enabled` cannot serve as the maintenance
|
||||||
|
hatch at all. The Phase 1 live gate found this; the two rules were independently reasonable and
|
||||||
|
contradictory together, so the meanings were split rather than either rule being weakened.
|
||||||
|
|
||||||
|
A node in maintenance is still enabled, so the topology gate still passes, and the runtime redundancy
|
||||||
|
posture (`NodeCount` / `RedundancyMode`) is unchanged.
|
||||||
|
|
||||||
`DriverConfigOverridesJson` shape:
|
`DriverConfigOverridesJson` shape:
|
||||||
|
|
||||||
```jsonc
|
```jsonc
|
||||||
|
|||||||
@@ -18,6 +18,33 @@ public sealed class ClusterNode
|
|||||||
/// <summary>The dashboard HTTP port (default 8081).</summary>
|
/// <summary>The dashboard HTTP port (default 8081).</summary>
|
||||||
public int DashboardPort { get; set; } = 8081;
|
public int DashboardPort { get; set; } = 8081;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Akka remoting port (default 4053). <b>This is central's dial target, not the node's own
|
||||||
|
/// binding configuration</b> — the node binds from <c>Cluster:Port</c> in its own appsettings,
|
||||||
|
/// and this column duplicates that value for a reader that cannot see the node's config.
|
||||||
|
/// Per-cluster mesh Phase 2 builds its ClusterClient contact points from
|
||||||
|
/// <see cref="Host"/> + this port, at which point central and the site node no longer share a
|
||||||
|
/// gossip ring and central has no other way to learn it.
|
||||||
|
/// </summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// The duplication is real and unenforced by the schema. <c>ClusterNodeAddressReconciler</c>
|
||||||
|
/// on the admin node compares this row against observed cluster membership and logs an Error
|
||||||
|
/// on mismatch — a node that binds 4054 while its row says 4053 is unreachable from central in
|
||||||
|
/// Phase 2, and the symptom there is a silent absence of acks rather than an error.
|
||||||
|
/// </remarks>
|
||||||
|
public int AkkaPort { get; set; } = 4053;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// gRPC port central dials for the Phase 5 telemetry stream, or <see langword="null"/> when the
|
||||||
|
/// node exposes none. Also central's dial target rather than the node's binding config.
|
||||||
|
/// </summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// Nullable by intent: nothing listens on this port yet, and a non-null default would assert a
|
||||||
|
/// port that does not exist. Phase 5 populates it; until then <see langword="null"/> is the
|
||||||
|
/// honest value.
|
||||||
|
/// </remarks>
|
||||||
|
public int? GrpcPort { get; set; }
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// OPC UA <c>ApplicationUri</c> — MUST be unique per node per OPC UA spec. Clients pin trust here.
|
/// OPC UA <c>ApplicationUri</c> — MUST be unique per node per OPC UA spec. Clients pin trust here.
|
||||||
/// Fleet-wide unique index enforces no two nodes share a value.
|
/// Fleet-wide unique index enforces no two nodes share a value.
|
||||||
@@ -36,8 +63,37 @@ public sealed class ClusterNode
|
|||||||
public string? DriverConfigOverridesJson { get; set; }
|
public string? DriverConfigOverridesJson { get; set; }
|
||||||
|
|
||||||
/// <summary>Gets or sets a value indicating whether this node is enabled.</summary>
|
/// <summary>Gets or sets a value indicating whether this node is enabled.</summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// <b>Part of the declared topology.</b> <c>DraftValidator.ValidateClusterTopology</c> requires
|
||||||
|
/// the enabled-node count to equal <see cref="ServerCluster.NodeCount"/>, so disabling one node
|
||||||
|
/// of a Warm/Hot pair is a deploy-blocking validation error by design — it would boot the
|
||||||
|
/// runtime into the InvalidTopology band. To take a node out of service temporarily, use
|
||||||
|
/// <see cref="MaintenanceMode"/> instead.
|
||||||
|
/// </remarks>
|
||||||
public bool Enabled { get; set; } = true;
|
public bool Enabled { get; set; } = true;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Node is temporarily out of service: still part of the cluster's declared topology, but not
|
||||||
|
/// expected to participate. A deployment does not wait for its ack, and the address reconciler
|
||||||
|
/// does not warn that it is absent.
|
||||||
|
/// </summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// <para>
|
||||||
|
/// Separate from <see cref="Enabled"/> because the two answer different questions.
|
||||||
|
/// <see cref="Enabled"/> means "this node is part of the declared topology" and is checked
|
||||||
|
/// against <see cref="ServerCluster.NodeCount"/>; this flag means "do not expect it right
|
||||||
|
/// now". Per-cluster mesh Phase 1 made an enabled row's ack mandatory, so without this a
|
||||||
|
/// node down for maintenance would block every deployment to its cluster — and clearing
|
||||||
|
/// <see cref="Enabled"/> to escape that is itself rejected by the topology gate on any
|
||||||
|
/// 2-node pair, which is every cluster in the target deployment.
|
||||||
|
/// </para>
|
||||||
|
/// <para>
|
||||||
|
/// Found by the Phase 1 live gate: the two rules were independently reasonable and
|
||||||
|
/// contradictory together.
|
||||||
|
/// </para>
|
||||||
|
/// </remarks>
|
||||||
|
public bool MaintenanceMode { get; set; }
|
||||||
|
|
||||||
/// <summary>Gets or sets the timestamp when this node was last seen.</summary>
|
/// <summary>Gets or sets the timestamp when this node was last seen.</summary>
|
||||||
public DateTime? LastSeenAt { get; set; }
|
public DateTime? LastSeenAt { get; set; }
|
||||||
|
|
||||||
|
|||||||
+1704
File diff suppressed because it is too large
Load Diff
+39
@@ -0,0 +1,39 @@
|
|||||||
|
using Microsoft.EntityFrameworkCore.Migrations;
|
||||||
|
|
||||||
|
#nullable disable
|
||||||
|
|
||||||
|
namespace ZB.MOM.WW.OtOpcUa.Configuration.Migrations
|
||||||
|
{
|
||||||
|
/// <inheritdoc />
|
||||||
|
public partial class AddClusterNodeTransportPorts : Migration
|
||||||
|
{
|
||||||
|
/// <inheritdoc />
|
||||||
|
protected override void Up(MigrationBuilder migrationBuilder)
|
||||||
|
{
|
||||||
|
migrationBuilder.AddColumn<int>(
|
||||||
|
name: "AkkaPort",
|
||||||
|
table: "ClusterNode",
|
||||||
|
type: "int",
|
||||||
|
nullable: false,
|
||||||
|
defaultValue: 4053);
|
||||||
|
|
||||||
|
migrationBuilder.AddColumn<int>(
|
||||||
|
name: "GrpcPort",
|
||||||
|
table: "ClusterNode",
|
||||||
|
type: "int",
|
||||||
|
nullable: true);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
protected override void Down(MigrationBuilder migrationBuilder)
|
||||||
|
{
|
||||||
|
migrationBuilder.DropColumn(
|
||||||
|
name: "AkkaPort",
|
||||||
|
table: "ClusterNode");
|
||||||
|
|
||||||
|
migrationBuilder.DropColumn(
|
||||||
|
name: "GrpcPort",
|
||||||
|
table: "ClusterNode");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
+1707
File diff suppressed because it is too large
Load Diff
+29
@@ -0,0 +1,29 @@
|
|||||||
|
using Microsoft.EntityFrameworkCore.Migrations;
|
||||||
|
|
||||||
|
#nullable disable
|
||||||
|
|
||||||
|
namespace ZB.MOM.WW.OtOpcUa.Configuration.Migrations
|
||||||
|
{
|
||||||
|
/// <inheritdoc />
|
||||||
|
public partial class AddClusterNodeMaintenanceMode : Migration
|
||||||
|
{
|
||||||
|
/// <inheritdoc />
|
||||||
|
protected override void Up(MigrationBuilder migrationBuilder)
|
||||||
|
{
|
||||||
|
migrationBuilder.AddColumn<bool>(
|
||||||
|
name: "MaintenanceMode",
|
||||||
|
table: "ClusterNode",
|
||||||
|
type: "bit",
|
||||||
|
nullable: false,
|
||||||
|
defaultValue: false);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
protected override void Down(MigrationBuilder migrationBuilder)
|
||||||
|
{
|
||||||
|
migrationBuilder.DropColumn(
|
||||||
|
name: "MaintenanceMode",
|
||||||
|
table: "ClusterNode");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
+11
@@ -47,6 +47,11 @@ namespace ZB.MOM.WW.OtOpcUa.Configuration.Migrations
|
|||||||
.HasMaxLength(64)
|
.HasMaxLength(64)
|
||||||
.HasColumnType("nvarchar(64)");
|
.HasColumnType("nvarchar(64)");
|
||||||
|
|
||||||
|
b.Property<int>("AkkaPort")
|
||||||
|
.ValueGeneratedOnAdd()
|
||||||
|
.HasColumnType("int")
|
||||||
|
.HasDefaultValue(4053);
|
||||||
|
|
||||||
b.Property<string>("ApplicationUri")
|
b.Property<string>("ApplicationUri")
|
||||||
.IsRequired()
|
.IsRequired()
|
||||||
.HasMaxLength(256)
|
.HasMaxLength(256)
|
||||||
@@ -76,6 +81,9 @@ namespace ZB.MOM.WW.OtOpcUa.Configuration.Migrations
|
|||||||
b.Property<bool>("Enabled")
|
b.Property<bool>("Enabled")
|
||||||
.HasColumnType("bit");
|
.HasColumnType("bit");
|
||||||
|
|
||||||
|
b.Property<int?>("GrpcPort")
|
||||||
|
.HasColumnType("int");
|
||||||
|
|
||||||
b.Property<string>("Host")
|
b.Property<string>("Host")
|
||||||
.IsRequired()
|
.IsRequired()
|
||||||
.HasMaxLength(255)
|
.HasMaxLength(255)
|
||||||
@@ -84,6 +92,9 @@ namespace ZB.MOM.WW.OtOpcUa.Configuration.Migrations
|
|||||||
b.Property<DateTime?>("LastSeenAt")
|
b.Property<DateTime?>("LastSeenAt")
|
||||||
.HasColumnType("datetime2(3)");
|
.HasColumnType("datetime2(3)");
|
||||||
|
|
||||||
|
b.Property<bool>("MaintenanceMode")
|
||||||
|
.HasColumnType("bit");
|
||||||
|
|
||||||
b.Property<int>("OpcUaPort")
|
b.Property<int>("OpcUaPort")
|
||||||
.HasColumnType("int");
|
.HasColumnType("int");
|
||||||
|
|
||||||
|
|||||||
@@ -142,6 +142,11 @@ public sealed class OtOpcUaConfigDbContext(DbContextOptions<OtOpcUaConfigDbConte
|
|||||||
e.Property(x => x.Host).HasMaxLength(255);
|
e.Property(x => x.Host).HasMaxLength(255);
|
||||||
e.Property(x => x.ApplicationUri).HasMaxLength(256);
|
e.Property(x => x.ApplicationUri).HasMaxLength(256);
|
||||||
e.Property(x => x.DriverConfigOverridesJson).HasColumnType("nvarchar(max)");
|
e.Property(x => x.DriverConfigOverridesJson).HasColumnType("nvarchar(max)");
|
||||||
|
// Central's dial targets (per-cluster mesh Phase 1). AkkaPort carries a DB-side default so
|
||||||
|
// rows migrated from before the column existed come out at 4053 rather than 0 — every node
|
||||||
|
// in the fleet does listen on a remoting port, so 0 is never a truthful value for it.
|
||||||
|
// GrpcPort has no default: nothing listens on it until Phase 5.
|
||||||
|
e.Property(x => x.AkkaPort).HasDefaultValue(4053);
|
||||||
e.Property(x => x.LastSeenAt).HasColumnType("datetime2(3)");
|
e.Property(x => x.LastSeenAt).HasColumnType("datetime2(3)");
|
||||||
e.Property(x => x.CreatedAt).HasColumnType("datetime2(3)").HasDefaultValueSql("SYSUTCDATETIME()");
|
e.Property(x => x.CreatedAt).HasColumnType("datetime2(3)").HasDefaultValueSql("SYSUTCDATETIME()");
|
||||||
e.Property(x => x.CreatedBy).HasMaxLength(128);
|
e.Property(x => x.CreatedBy).HasMaxLength(128);
|
||||||
|
|||||||
+79
-25
@@ -1,5 +1,4 @@
|
|||||||
using Akka.Actor;
|
using Akka.Actor;
|
||||||
using Akka.Cluster;
|
|
||||||
using Akka.Cluster.Tools.PublishSubscribe;
|
using Akka.Cluster.Tools.PublishSubscribe;
|
||||||
using Akka.Event;
|
using Akka.Event;
|
||||||
using Microsoft.EntityFrameworkCore;
|
using Microsoft.EntityFrameworkCore;
|
||||||
@@ -18,8 +17,40 @@ namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Coordinators;
|
|||||||
/// has acked Applied. Per-node ACKs are persisted in <c>NodeDeploymentState</c> so a failover of
|
/// has acked Applied. Per-node ACKs are persisted in <c>NodeDeploymentState</c> so a failover of
|
||||||
/// this singleton can recover in-flight state from the DB.
|
/// this singleton can recover in-flight state from the DB.
|
||||||
///
|
///
|
||||||
/// Discovery of the "expected ACK set" comes from <c>Akka.Cluster.State.Members</c> filtered by
|
/// Discovery of the "expected ACK set" comes from the <b>enabled <c>ClusterNode</c> rows</b>, not
|
||||||
/// the <c>driver</c> role — the DB does not own per-node role assignment.
|
/// from <c>Akka.Cluster.State.Members</c> (per-cluster mesh Phase 1). Central must be able to name
|
||||||
|
/// the nodes a deployment is for without sharing a gossip ring with them — Phase 2 splits the fleet
|
||||||
|
/// into one mesh per <c>Cluster</c>, after which central can no longer see a site node's membership
|
||||||
|
/// at all. This was the coordinator's last genuinely mesh-bound dependency.
|
||||||
|
///
|
||||||
|
/// <para>
|
||||||
|
/// <b>Behaviour change:</b> a configured node that is switched off is now <i>expected</i>, so a
|
||||||
|
/// deployment dispatched while it is down fails at the apply deadline instead of sealing green
|
||||||
|
/// without it. That is deliberate — under the membership rule the operator was told the fleet was
|
||||||
|
/// deployed when it was not. <c>ClusterNode.MaintenanceMode = 1</c> is the escape hatch for a node
|
||||||
|
/// taken down for maintenance.
|
||||||
|
/// </para>
|
||||||
|
/// <para>
|
||||||
|
/// <b>Not <c>Enabled = 0</c>.</b> That was the intended hatch until the Phase 1 live gate found it
|
||||||
|
/// unusable: <c>DraftValidator.ValidateClusterTopology</c> requires the enabled-node count to equal
|
||||||
|
/// <c>ServerCluster.NodeCount</c>, so clearing <c>Enabled</c> on either node of a Warm/Hot pair
|
||||||
|
/// rejects the deployment outright — and every cluster in the target topology is a pair. The two
|
||||||
|
/// rules were independently reasonable and contradictory together, so the meanings were split:
|
||||||
|
/// <c>Enabled</c> = part of the declared topology, <c>MaintenanceMode</c> = do not expect it now.
|
||||||
|
/// </para>
|
||||||
|
/// <para>
|
||||||
|
/// <b>Every <c>ClusterNode</c> row is assumed to be a driver node.</b> The membership rule filtered
|
||||||
|
/// on the <c>driver</c> role; the DB has no per-node role column, and deliberately does not gain one
|
||||||
|
/// here (a second declaration of node roles would drift from <c>Cluster:Roles</c> in the node's own
|
||||||
|
/// appsettings). The assumption holds because the entity's whole shape — <c>Host</c>,
|
||||||
|
/// <c>OpcUaPort</c>, <c>ApplicationUri</c>, <c>ServiceLevelBase</c> — describes an OPC UA server
|
||||||
|
/// node. An admin-only node must not be given a <c>ClusterNode</c> row: it would be expected to ack
|
||||||
|
/// a deployment it has no <c>DriverHostActor</c> to apply, and every deploy would time out.
|
||||||
|
/// </para>
|
||||||
|
/// <para>
|
||||||
|
/// <c>ServerCluster.Enabled</c> is <b>not</b> consulted — nothing else in the codebase consults it,
|
||||||
|
/// so honouring it here would invent semantics. Disable the nodes, not the cluster row.
|
||||||
|
/// </para>
|
||||||
/// </summary>
|
/// </summary>
|
||||||
public sealed class ConfigPublishCoordinator : ReceiveActor, IWithTimers
|
public sealed class ConfigPublishCoordinator : ReceiveActor, IWithTimers
|
||||||
{
|
{
|
||||||
@@ -119,11 +150,15 @@ public sealed class ConfigPublishCoordinator : ReceiveActor, IWithTimers
|
|||||||
{
|
{
|
||||||
_current = msg.DeploymentId;
|
_current = msg.DeploymentId;
|
||||||
_acks.Clear();
|
_acks.Clear();
|
||||||
_expectedAcks = DiscoverDriverNodes();
|
|
||||||
|
|
||||||
// Seed NodeDeploymentState rows so a failover knows which nodes were expected to ack.
|
// Seed NodeDeploymentState rows so a failover knows which nodes were expected to ack.
|
||||||
using (var db = _dbFactory.CreateDbContext())
|
using (var db = _dbFactory.CreateDbContext())
|
||||||
{
|
{
|
||||||
|
// Discovery reuses this context rather than opening a second one — it reads the same
|
||||||
|
// ClusterNode table the NodeDeploymentState rows below are FK-bound to, so a single
|
||||||
|
// context keeps the expected set and the seeded rows consistent by construction.
|
||||||
|
_expectedAcks = DiscoverDriverNodes(db);
|
||||||
|
|
||||||
foreach (var node in _expectedAcks)
|
foreach (var node in _expectedAcks)
|
||||||
{
|
{
|
||||||
db.NodeDeploymentStates.Add(new NodeDeploymentState
|
db.NodeDeploymentStates.Add(new NodeDeploymentState
|
||||||
@@ -142,9 +177,14 @@ public sealed class ConfigPublishCoordinator : ReceiveActor, IWithTimers
|
|||||||
|
|
||||||
if (_expectedAcks.Count == 0)
|
if (_expectedAcks.Count == 0)
|
||||||
{
|
{
|
||||||
// No driver-role members. Seal immediately — the alternative is hanging forever
|
// No enabled ClusterNode rows. Seal immediately — the alternative is hanging forever
|
||||||
// waiting for ACKs that will never come.
|
// waiting for ACKs that will never come. Note this is now a *configuration* error
|
||||||
_log.Warning("DispatchDeployment {Id}: no driver-role members in cluster; sealing empty",
|
// rather than a topology observation: under the old membership rule an empty set meant
|
||||||
|
// "no driver nodes have joined yet", which could resolve itself; an empty set here means
|
||||||
|
// the fleet has no enabled nodes registered, which cannot.
|
||||||
|
_log.Warning(
|
||||||
|
"DispatchDeployment {Id}: no enabled ClusterNode rows registered; sealing empty. " +
|
||||||
|
"Nothing will receive this deployment — check the fleet's node configuration",
|
||||||
msg.DeploymentId);
|
msg.DeploymentId);
|
||||||
SealDeployment();
|
SealDeployment();
|
||||||
}
|
}
|
||||||
@@ -233,8 +273,15 @@ public sealed class ConfigPublishCoordinator : ReceiveActor, IWithTimers
|
|||||||
using var db = _dbFactory.CreateDbContext();
|
using var db = _dbFactory.CreateDbContext();
|
||||||
UpdateDeploymentStatus(db, _current.Value, DeploymentStatus.TimedOut);
|
UpdateDeploymentStatus(db, _current.Value, DeploymentStatus.TimedOut);
|
||||||
db.SaveChanges();
|
db.SaveChanges();
|
||||||
_log.Warning("Deployment {Id} timed out after {Deadline} ({Acked}/{Total} acks landed)",
|
// Name the silent nodes, not just the counts. Since the expected set became DB-derived this
|
||||||
_current.Value, _applyDeadline, _acks.Count, _expectedAcks.Count);
|
// is the failure an operator meets after deploying while a configured node is switched off,
|
||||||
|
// so it has to say which node — "4/5 acks landed" leaves them reading logs on five machines.
|
||||||
|
var missing = _expectedAcks.Where(n => !_acks.ContainsKey(n)).Select(n => n.Value).Order().ToList();
|
||||||
|
_log.Warning(
|
||||||
|
"Deployment {Id} timed out after {Deadline} ({Acked}/{Total} acks landed). " +
|
||||||
|
"No ack from: {MissingNodes}. Each is an enabled, non-maintenance ClusterNode row — start " +
|
||||||
|
"the node, or set ClusterNode.MaintenanceMode = 1 if it is down for maintenance, then redeploy",
|
||||||
|
_current.Value, _applyDeadline, _acks.Count, _expectedAcks.Count, string.Join(", ", missing));
|
||||||
ResetForNext();
|
ResetForNext();
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -259,22 +306,29 @@ public sealed class ConfigPublishCoordinator : ReceiveActor, IWithTimers
|
|||||||
if (sealNow) d.SealedAtUtc = DateTime.UtcNow;
|
if (sealNow) d.SealedAtUtc = DateTime.UtcNow;
|
||||||
}
|
}
|
||||||
|
|
||||||
private HashSet<NodeId> DiscoverDriverNodes()
|
/// <summary>
|
||||||
{
|
/// The set of nodes expected to ack this deployment: every <b>enabled</b> <c>ClusterNode</c>
|
||||||
var cluster = Akka.Cluster.Cluster.Get(Context.System);
|
/// row. See the class remarks for why this is DB-derived rather than membership-derived, and
|
||||||
var nodes = new HashSet<NodeId>(NodeIdComparer);
|
/// for the two assumptions it rests on (every row is a driver node; <c>Enabled</c> is the
|
||||||
foreach (var member in cluster.State.Members)
|
/// maintenance hatch).
|
||||||
{
|
/// </summary>
|
||||||
if (member.Status is not (MemberStatus.Up or MemberStatus.Joining)) continue;
|
/// <remarks>
|
||||||
if (!member.Roles.Contains("driver")) continue;
|
/// <c>ClusterNode.NodeId</c> is already in the <c>host:port</c> identifier space the
|
||||||
var host = member.Address.Host;
|
/// membership rule derived — the rig seeds <c>central-1:4053</c> and so on, and
|
||||||
if (string.IsNullOrWhiteSpace(host)) continue;
|
/// <c>NodeDeploymentState.NodeId</c> is FK-bound to this column, so the fact that the
|
||||||
// Match ClusterRoleInfo's NodeId derivation (host:port) so DriverHostActor's
|
/// membership-derived rows have always satisfied that FK is standing proof the two sets
|
||||||
// self-identification and the coordinator's expected-ack set agree.
|
/// agree. It also has to stay that way: <c>DriverHostActor</c> self-identifies from
|
||||||
nodes.Add(NodeId.Parse($"{host}:{member.Address.Port ?? 0}"));
|
/// <c>ClusterRoleInfo</c>'s <c>host:port</c>, so a row whose NodeId is anything else is a
|
||||||
}
|
/// node whose acks will never match its expected-ack entry.
|
||||||
return nodes;
|
/// </remarks>
|
||||||
}
|
private static HashSet<NodeId> DiscoverDriverNodes(OtOpcUaConfigDbContext db) =>
|
||||||
|
db.ClusterNodes
|
||||||
|
.AsNoTracking()
|
||||||
|
.Where(n => n.Enabled && !n.MaintenanceMode)
|
||||||
|
.Select(n => n.NodeId)
|
||||||
|
.ToList()
|
||||||
|
.Select(NodeId.Parse)
|
||||||
|
.ToHashSet(NodeIdComparer);
|
||||||
|
|
||||||
/// <summary>Case-insensitive <see cref="NodeId"/> equality (by <see cref="NodeId.Value"/>),
|
/// <summary>Case-insensitive <see cref="NodeId"/> equality (by <see cref="NodeId.Value"/>),
|
||||||
/// matching the case-insensitive scoping in <c>DeploymentArtifact.ResolveClusterScope</c> so the
|
/// matching the case-insensitive scoping in <c>DeploymentArtifact.ResolveClusterScope</c> so the
|
||||||
|
|||||||
@@ -0,0 +1,138 @@
|
|||||||
|
namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Fleet;
|
||||||
|
|
||||||
|
/// <summary>What kind of disagreement was found between a node's real address and its row.</summary>
|
||||||
|
public enum AddressMismatchKind
|
||||||
|
{
|
||||||
|
/// <summary>
|
||||||
|
/// The row was matched to a live member by <c>NodeId</c>, but its <c>Host</c> / <c>AkkaPort</c>
|
||||||
|
/// columns name a different address. Phase 2 dials those columns, so central would dial
|
||||||
|
/// somewhere the node is not — and the symptom there is a silent absence of acks.
|
||||||
|
/// </summary>
|
||||||
|
RowDialTargetDisagrees,
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// A driver-role member is in the cluster with no enabled <c>ClusterNode</c> row at its
|
||||||
|
/// address. Its deployment acks are discarded (see <c>ConfigPublishCoordinator</c>), so it
|
||||||
|
/// silently receives deployments it is never credited for.
|
||||||
|
/// </summary>
|
||||||
|
RunningNodeHasNoRow,
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// An enabled row names a node that is not currently a driver-role member. Either it is down
|
||||||
|
/// — in which case the next deployment fails at the apply deadline — or it binds an address
|
||||||
|
/// other than the one its row claims.
|
||||||
|
/// </summary>
|
||||||
|
EnabledRowNotInCluster,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>One disagreement, with enough detail to act on without opening the DB.</summary>
|
||||||
|
/// <param name="Kind">Which disagreement this is.</param>
|
||||||
|
/// <param name="NodeId">The <c>host:port</c> identity the finding is about.</param>
|
||||||
|
/// <param name="Detail">Human-readable specifics, including what to do about it.</param>
|
||||||
|
public sealed record ClusterNodeAddressMismatch(AddressMismatchKind Kind, string NodeId, string Detail);
|
||||||
|
|
||||||
|
/// <summary>A <c>ClusterNode</c> row reduced to the address facts this check cares about.</summary>
|
||||||
|
/// <param name="NodeId">The row's primary key, in the <c>host:port</c> space.</param>
|
||||||
|
/// <param name="Host">The host central will dial in Phase 2.</param>
|
||||||
|
/// <param name="AkkaPort">The remoting port central will dial in Phase 2.</param>
|
||||||
|
public sealed record ClusterNodeAddress(string NodeId, string Host, int AkkaPort);
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Compares the addresses driver nodes actually occupy against the <c>ClusterNode</c> rows that
|
||||||
|
/// claim to describe them.
|
||||||
|
/// </summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// <para>
|
||||||
|
/// <b>Why this exists.</b> <c>ClusterNode.AkkaPort</c> and the node's own <c>Cluster:Port</c>
|
||||||
|
/// are the same fact stored in two places, and nothing makes them agree. A node that binds
|
||||||
|
/// 4054 while its row says 4053 is unreachable from central once Phase 2 dials the row
|
||||||
|
/// instead of gossiping — and an unenforced duplicated address is the exact shape of the
|
||||||
|
/// <c>Modbus</c>/<c>ModbusTcp</c> and <c>TwinCat</c>/<c>Focas</c> drifts already recorded in
|
||||||
|
/// this repo. Since Phase 1 also made the rows the deploy path's expected-ack set, a drifted
|
||||||
|
/// row already costs a failed deployment today.
|
||||||
|
/// </para>
|
||||||
|
/// <para>
|
||||||
|
/// <b>It runs on an admin node</b>, comparing rows against the membership an admin node can
|
||||||
|
/// already see, rather than having each driver node assert its own row — Phase 4 removes the
|
||||||
|
/// driver nodes' ConfigDb connection entirely, so a self-assertion written there would have
|
||||||
|
/// to be deleted again.
|
||||||
|
/// </para>
|
||||||
|
/// <para>
|
||||||
|
/// <b>Phase 2 must revisit this.</b> Once the fleet splits into one mesh per cluster, an
|
||||||
|
/// admin node no longer sees site members at all, and every site row would report
|
||||||
|
/// <see cref="AddressMismatchKind.EnabledRowNotInCluster"/> forever. The check then has to
|
||||||
|
/// move to whatever transport replaces gossip — or be scoped to the central mesh.
|
||||||
|
/// </para>
|
||||||
|
/// </remarks>
|
||||||
|
public static class ClusterNodeAddressReconciler
|
||||||
|
{
|
||||||
|
/// <summary>
|
||||||
|
/// Reconciles observed driver-role member addresses against the configured rows.
|
||||||
|
/// </summary>
|
||||||
|
/// <param name="observedDriverMembers">
|
||||||
|
/// <c>host</c>/<c>port</c> of every Up member carrying the <c>driver</c> role. Admin-only
|
||||||
|
/// members must be excluded — they legitimately have no <c>ClusterNode</c> row, and including
|
||||||
|
/// them would report a mismatch for correct configuration.
|
||||||
|
/// </param>
|
||||||
|
/// <param name="rows">Every <c>ClusterNode</c> row, enabled or not.</param>
|
||||||
|
/// <param name="expectedPresentNodeIds">
|
||||||
|
/// The subset of <paramref name="rows"/> that should currently be in the cluster: enabled and
|
||||||
|
/// not in maintenance. Excluded rows are deliberately still matched against membership (a
|
||||||
|
/// disabled row for a running node is worth knowing) but never reported as
|
||||||
|
/// <see cref="AddressMismatchKind.EnabledRowNotInCluster"/> — being absent is the whole point
|
||||||
|
/// of disabling one or putting it in maintenance.
|
||||||
|
/// </param>
|
||||||
|
/// <returns>Every disagreement found, ordered by node id for stable logging.</returns>
|
||||||
|
public static IReadOnlyList<ClusterNodeAddressMismatch> Reconcile(
|
||||||
|
IReadOnlyCollection<(string Host, int Port)> observedDriverMembers,
|
||||||
|
IReadOnlyCollection<ClusterNodeAddress> rows,
|
||||||
|
IReadOnlyCollection<string> expectedPresentNodeIds)
|
||||||
|
{
|
||||||
|
var byNodeId = new Dictionary<string, ClusterNodeAddress>(StringComparer.OrdinalIgnoreCase);
|
||||||
|
foreach (var row in rows) byNodeId[row.NodeId] = row;
|
||||||
|
var expectedPresent = new HashSet<string>(expectedPresentNodeIds, StringComparer.OrdinalIgnoreCase);
|
||||||
|
|
||||||
|
var findings = new List<ClusterNodeAddressMismatch>();
|
||||||
|
var seen = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
|
||||||
|
|
||||||
|
foreach (var (host, port) in observedDriverMembers)
|
||||||
|
{
|
||||||
|
var nodeId = $"{host}:{port}";
|
||||||
|
seen.Add(nodeId);
|
||||||
|
|
||||||
|
if (!byNodeId.TryGetValue(nodeId, out var row))
|
||||||
|
{
|
||||||
|
findings.Add(new ClusterNodeAddressMismatch(
|
||||||
|
AddressMismatchKind.RunningNodeHasNoRow, nodeId,
|
||||||
|
$"driver node is in the cluster at {nodeId} but no ClusterNode row has that NodeId; " +
|
||||||
|
"its deployment acks are discarded. Either register the row or correct the node's " +
|
||||||
|
"Cluster:PublicHostname / Cluster:Port to match an existing one"));
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!string.Equals(row.Host, host, StringComparison.OrdinalIgnoreCase) || row.AkkaPort != port)
|
||||||
|
{
|
||||||
|
findings.Add(new ClusterNodeAddressMismatch(
|
||||||
|
AddressMismatchKind.RowDialTargetDisagrees, nodeId,
|
||||||
|
$"node is at {nodeId} but its row's dial target is {row.Host}:{row.AkkaPort}; " +
|
||||||
|
"central will dial the wrong address once Phase 2 stops using gossip. " +
|
||||||
|
"Correct ClusterNode.Host / ClusterNode.AkkaPort"));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
foreach (var row in rows)
|
||||||
|
{
|
||||||
|
if (seen.Contains(row.NodeId) || !expectedPresent.Contains(row.NodeId)) continue;
|
||||||
|
findings.Add(new ClusterNodeAddressMismatch(
|
||||||
|
AddressMismatchKind.EnabledRowNotInCluster, row.NodeId,
|
||||||
|
$"enabled ClusterNode row {row.NodeId} has no matching driver member; the next " +
|
||||||
|
"deployment will fail at the apply deadline waiting for it. Start the node, or set " +
|
||||||
|
"ClusterNode.MaintenanceMode = 1 while it is down for maintenance"));
|
||||||
|
}
|
||||||
|
|
||||||
|
return findings
|
||||||
|
.OrderBy(f => f.NodeId, StringComparer.OrdinalIgnoreCase)
|
||||||
|
.ThenBy(f => f.Kind)
|
||||||
|
.ToList();
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,136 @@
|
|||||||
|
using Akka.Actor;
|
||||||
|
using Akka.Cluster;
|
||||||
|
using Akka.Event;
|
||||||
|
using Microsoft.EntityFrameworkCore;
|
||||||
|
using ZB.MOM.WW.OtOpcUa.Configuration;
|
||||||
|
|
||||||
|
namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Fleet;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Admin-role cluster singleton that runs <see cref="ClusterNodeAddressReconciler"/> against live
|
||||||
|
/// membership and logs what it finds. A singleton rather than a per-admin-node actor so a
|
||||||
|
/// two-admin fleet reports each disagreement once.
|
||||||
|
/// </summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// Recomputes on membership change (debounced) and on a slow periodic sweep, and <b>logs only
|
||||||
|
/// when the finding set changes</b>. A node legitimately down would otherwise reprint the same
|
||||||
|
/// warning every sweep until someone silences it by ignoring the log, which is the failure mode
|
||||||
|
/// this check exists to avoid.
|
||||||
|
/// </remarks>
|
||||||
|
public sealed class ClusterNodeAddressReconcilerActor : ReceiveActor, IWithTimers
|
||||||
|
{
|
||||||
|
/// <summary>Driver role name — only members carrying it are expected to have a ClusterNode row.</summary>
|
||||||
|
public const string DriverRole = "driver";
|
||||||
|
|
||||||
|
/// <summary>Collapses a burst of membership events (a rolling restart) into one reconcile.</summary>
|
||||||
|
public static readonly TimeSpan DebounceWindow = TimeSpan.FromSeconds(2);
|
||||||
|
|
||||||
|
/// <summary>Periodic sweep, so a row edited in the AdminUI is checked without a topology change.</summary>
|
||||||
|
public static readonly TimeSpan DefaultSweepInterval = TimeSpan.FromMinutes(5);
|
||||||
|
|
||||||
|
private readonly IDbContextFactory<OtOpcUaConfigDbContext> _dbFactory;
|
||||||
|
private readonly Akka.Cluster.Cluster _cluster;
|
||||||
|
private readonly TimeSpan _sweepInterval;
|
||||||
|
private readonly ILoggingAdapter _log = Context.GetLogger();
|
||||||
|
private IReadOnlyList<ClusterNodeAddressMismatch>? _lastReported;
|
||||||
|
|
||||||
|
/// <summary>Gets the timer scheduler for this actor.</summary>
|
||||||
|
public ITimerScheduler Timers { get; set; } = null!;
|
||||||
|
|
||||||
|
/// <summary>Creates Props for the reconciler.</summary>
|
||||||
|
/// <param name="dbFactory">Factory for the config database context.</param>
|
||||||
|
/// <param name="sweepInterval">Periodic sweep interval; defaults to <see cref="DefaultSweepInterval"/>.</param>
|
||||||
|
/// <returns>Props for actor creation.</returns>
|
||||||
|
public static Props Props(
|
||||||
|
IDbContextFactory<OtOpcUaConfigDbContext> dbFactory, TimeSpan? sweepInterval = null) =>
|
||||||
|
Akka.Actor.Props.Create(() => new ClusterNodeAddressReconcilerActor(dbFactory, sweepInterval));
|
||||||
|
|
||||||
|
/// <summary>Initializes a new instance of the <see cref="ClusterNodeAddressReconcilerActor"/> class.</summary>
|
||||||
|
/// <param name="dbFactory">Factory for the config database context.</param>
|
||||||
|
/// <param name="sweepInterval">Periodic sweep interval; defaults to <see cref="DefaultSweepInterval"/>.</param>
|
||||||
|
public ClusterNodeAddressReconcilerActor(
|
||||||
|
IDbContextFactory<OtOpcUaConfigDbContext> dbFactory, TimeSpan? sweepInterval = null)
|
||||||
|
{
|
||||||
|
_dbFactory = dbFactory;
|
||||||
|
_cluster = Akka.Cluster.Cluster.Get(Context.System);
|
||||||
|
_sweepInterval = sweepInterval ?? DefaultSweepInterval;
|
||||||
|
|
||||||
|
Receive<ClusterEvent.IMemberEvent>(_ => ScheduleReconcile());
|
||||||
|
Receive<ClusterEvent.CurrentClusterState>(_ => ScheduleReconcile());
|
||||||
|
Receive<ReconcileNow>(_ => Reconcile());
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
protected override void PreStart()
|
||||||
|
{
|
||||||
|
_cluster.Subscribe(Self, ClusterEvent.InitialStateAsEvents, typeof(ClusterEvent.IMemberEvent));
|
||||||
|
Timers.StartPeriodicTimer("sweep", ReconcileNow.Instance, _sweepInterval, _sweepInterval);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
protected override void PostStop() => _cluster.Unsubscribe(Self);
|
||||||
|
|
||||||
|
private void ScheduleReconcile() =>
|
||||||
|
Timers.StartSingleTimer("debounce", ReconcileNow.Instance, DebounceWindow);
|
||||||
|
|
||||||
|
private void Reconcile()
|
||||||
|
{
|
||||||
|
var observed = _cluster.State.Members
|
||||||
|
.Where(m => m.Status == MemberStatus.Up && m.Roles.Contains(DriverRole))
|
||||||
|
.Select(m => (Host: m.Address.Host ?? string.Empty, Port: m.Address.Port ?? 0))
|
||||||
|
.Where(a => !string.IsNullOrWhiteSpace(a.Host) && a.Port > 0)
|
||||||
|
.ToList();
|
||||||
|
|
||||||
|
List<ClusterNodeAddress> rows;
|
||||||
|
List<string> enabled;
|
||||||
|
try
|
||||||
|
{
|
||||||
|
using var db = _dbFactory.CreateDbContext();
|
||||||
|
rows = db.ClusterNodes.AsNoTracking()
|
||||||
|
.Select(n => new ClusterNodeAddress(n.NodeId, n.Host, n.AkkaPort))
|
||||||
|
.ToList();
|
||||||
|
// Same predicate the coordinator's expected-ack set uses — a node the deploy path will
|
||||||
|
// not wait for must not be reported as missing either, or the maintenance hatch trades a
|
||||||
|
// failed deployment for a permanent warning.
|
||||||
|
enabled = db.ClusterNodes.AsNoTracking()
|
||||||
|
.Where(n => n.Enabled && !n.MaintenanceMode).Select(n => n.NodeId).ToList();
|
||||||
|
}
|
||||||
|
catch (Exception ex)
|
||||||
|
{
|
||||||
|
// A consistency check must never take the admin node down with it. Central SQL being
|
||||||
|
// briefly unreachable is an operational fact, not a reason to restart this singleton.
|
||||||
|
_log.Warning(ex, "ClusterNode address reconcile skipped — config database unreachable");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
var findings = ClusterNodeAddressReconciler.Reconcile(observed, rows, enabled);
|
||||||
|
|
||||||
|
if (_lastReported is not null && findings.SequenceEqual(_lastReported)) return;
|
||||||
|
_lastReported = findings;
|
||||||
|
|
||||||
|
if (findings.Count == 0)
|
||||||
|
{
|
||||||
|
_log.Info("ClusterNode addresses reconcile with cluster membership ({Count} driver members)",
|
||||||
|
observed.Count);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
foreach (var f in findings)
|
||||||
|
{
|
||||||
|
// Error for the two shapes that are always a misconfiguration; Warning for a row whose
|
||||||
|
// node is merely absent, which is a legitimate state during maintenance.
|
||||||
|
if (f.Kind == AddressMismatchKind.EnabledRowNotInCluster)
|
||||||
|
_log.Warning("ClusterNode address check [{Kind}] {NodeId}: {Detail}", f.Kind, f.NodeId, f.Detail);
|
||||||
|
else
|
||||||
|
_log.Error("ClusterNode address check [{Kind}] {NodeId}: {Detail}", f.Kind, f.NodeId, f.Detail);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>Self-message: run a reconcile pass now.</summary>
|
||||||
|
public sealed class ReconcileNow
|
||||||
|
{
|
||||||
|
/// <summary>The singleton instance.</summary>
|
||||||
|
public static readonly ReconcileNow Instance = new();
|
||||||
|
private ReconcileNow() { }
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -23,9 +23,11 @@ public static class ServiceCollectionExtensions
|
|||||||
public const string AuditWriterSingletonName = "audit-writer";
|
public const string AuditWriterSingletonName = "audit-writer";
|
||||||
public const string FleetStatusSingletonName = "fleet-status";
|
public const string FleetStatusSingletonName = "fleet-status";
|
||||||
public const string RedundancyStateSingletonName = "redundancy-state";
|
public const string RedundancyStateSingletonName = "redundancy-state";
|
||||||
|
public const string ClusterNodeAddressReconcilerSingletonName = "cluster-node-address-reconciler";
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// Registers all five admin-role cluster singletons + their proxies on the AkkaConfigurationBuilder.
|
/// Registers the admin-role cluster singletons on the AkkaConfigurationBuilder — five with
|
||||||
|
/// registry proxies, plus the address reconciler, which is proxy-less because nothing addresses it.
|
||||||
/// Must be called against the same builder used by <c>AkkaHostedService</c> so the singletons
|
/// Must be called against the same builder used by <c>AkkaHostedService</c> so the singletons
|
||||||
/// share the host's ActorSystem.
|
/// share the host's ActorSystem.
|
||||||
/// </summary>
|
/// </summary>
|
||||||
@@ -90,6 +92,26 @@ public static class ServiceCollectionExtensions
|
|||||||
(system, registry, resolver) => RedundancyStateActor.Props(),
|
(system, registry, resolver) => RedundancyStateActor.Props(),
|
||||||
singletonOptions);
|
singletonOptions);
|
||||||
|
|
||||||
|
// Per-cluster mesh Phase 1: ClusterNode.AkkaPort duplicates the node's own Cluster:Port and
|
||||||
|
// nothing makes them agree. Runs here rather than on each driver node because Phase 4 removes
|
||||||
|
// the driver nodes' ConfigDb connection entirely.
|
||||||
|
//
|
||||||
|
// createProxyToo: false — unlike the five singletons above, nothing ever resolves
|
||||||
|
// ClusterNodeAddressReconcilerKey from the registry. This actor only reads cluster state and
|
||||||
|
// logs; no one sends it messages. A proxy would be a second actor per node whose only job is
|
||||||
|
// to hunt for a singleton nobody addresses — visible as the "ClusterSingletonProxy failed to
|
||||||
|
// find an associated singleton after 30 seconds" startup warning, and as extra cluster
|
||||||
|
// chatter in every two-node test harness the integration suite builds.
|
||||||
|
builder.WithSingleton<ClusterNodeAddressReconcilerKey>(
|
||||||
|
ClusterNodeAddressReconcilerSingletonName,
|
||||||
|
(system, registry, resolver) =>
|
||||||
|
{
|
||||||
|
var dbFactory = resolver.GetService<IDbContextFactory<OtOpcUaConfigDbContext>>();
|
||||||
|
return ClusterNodeAddressReconcilerActor.Props(dbFactory);
|
||||||
|
},
|
||||||
|
singletonOptions,
|
||||||
|
createProxyToo: false);
|
||||||
|
|
||||||
return builder;
|
return builder;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -100,3 +122,4 @@ public sealed class AdminOperationsActorKey { }
|
|||||||
public sealed class AuditWriterActorKey { }
|
public sealed class AuditWriterActorKey { }
|
||||||
public sealed class FleetStatusBroadcasterKey { }
|
public sealed class FleetStatusBroadcasterKey { }
|
||||||
public sealed class RedundancyStateActorKey { }
|
public sealed class RedundancyStateActorKey { }
|
||||||
|
public sealed class ClusterNodeAddressReconcilerKey { }
|
||||||
|
|||||||
@@ -0,0 +1,149 @@
|
|||||||
|
using Microsoft.EntityFrameworkCore;
|
||||||
|
using Shouldly;
|
||||||
|
using Xunit;
|
||||||
|
using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
|
||||||
|
using ZB.MOM.WW.OtOpcUa.Configuration.Enums;
|
||||||
|
|
||||||
|
namespace ZB.MOM.WW.OtOpcUa.Configuration.Tests;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Round-trips <see cref="ClusterNode.AkkaPort"/> / <see cref="ClusterNode.GrpcPort"/> through
|
||||||
|
/// the real schema (per-cluster mesh Phase 1). These are <b>central's dial targets</b>: Phase 2
|
||||||
|
/// builds its ClusterClient contact points from <c>Host</c> + <c>AkkaPort</c>, and Phase 5 dials
|
||||||
|
/// <c>GrpcPort</c> for the telemetry stream. Central cannot see a site node's appsettings once
|
||||||
|
/// the meshes split, so the value has to live in a row central can read.
|
||||||
|
/// </summary>
|
||||||
|
[Trait("Category", "SchemaCompliance")]
|
||||||
|
[Collection(nameof(SchemaComplianceCollection))]
|
||||||
|
public sealed class ClusterNodeTransportPortsTests(SchemaComplianceFixture fixture)
|
||||||
|
{
|
||||||
|
/// <summary>Verifies both ports survive a DB round-trip and that AkkaPort defaults to 4053.</summary>
|
||||||
|
[Fact]
|
||||||
|
public async Task Transport_ports_round_trip_and_AkkaPort_defaults_to_4053()
|
||||||
|
{
|
||||||
|
await using var ctx = NewContext();
|
||||||
|
var clusterId = await SeedClusterAsync(ctx);
|
||||||
|
|
||||||
|
// Explicit non-default values on one node — proves the columns are mapped and persisted
|
||||||
|
// rather than silently dropped (an unmapped property round-trips fine in memory).
|
||||||
|
ctx.ClusterNodes.Add(NewNode(clusterId, "explicit", n =>
|
||||||
|
{
|
||||||
|
n.AkkaPort = 4055;
|
||||||
|
n.GrpcPort = 5223;
|
||||||
|
}));
|
||||||
|
// Nothing set on the other — AkkaPort must land on 4053 and GrpcPort must stay null.
|
||||||
|
// GrpcPort is deliberately nullable: nothing listens on it until Phase 5, and a non-null
|
||||||
|
// default would assert a port that does not exist.
|
||||||
|
ctx.ClusterNodes.Add(NewNode(clusterId, "defaulted"));
|
||||||
|
await ctx.SaveChangesAsync();
|
||||||
|
|
||||||
|
await using var read = NewContext();
|
||||||
|
var explicitNode = await read.ClusterNodes.AsNoTracking()
|
||||||
|
.SingleAsync(n => n.NodeId == $"{clusterId}-explicit:4053");
|
||||||
|
explicitNode.AkkaPort.ShouldBe(4055);
|
||||||
|
explicitNode.GrpcPort.ShouldBe(5223);
|
||||||
|
|
||||||
|
var defaulted = await read.ClusterNodes.AsNoTracking()
|
||||||
|
.SingleAsync(n => n.NodeId == $"{clusterId}-defaulted:4053");
|
||||||
|
defaulted.AkkaPort.ShouldBe(4053);
|
||||||
|
defaulted.GrpcPort.ShouldBeNull();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Verifies the <b>DB-side</b> default on <c>AkkaPort</c> — the migration's
|
||||||
|
/// <c>defaultValue: 4053</c>, which is what rows created before the column existed inherit.
|
||||||
|
/// The entity round-trip above cannot see this: <see cref="ClusterNode.AkkaPort"/>'s CLR
|
||||||
|
/// initializer is also 4053, so that assertion passes with the mapping default deleted. This
|
||||||
|
/// one inserts through raw SQL with the column omitted, so only the schema can supply the value.
|
||||||
|
/// </summary>
|
||||||
|
[Fact]
|
||||||
|
public async Task AkkaPort_has_a_DB_side_default_of_4053()
|
||||||
|
{
|
||||||
|
await using var ctx = NewContext();
|
||||||
|
var clusterId = await SeedClusterAsync(ctx);
|
||||||
|
|
||||||
|
await using (var conn = fixture.OpenConnection())
|
||||||
|
{
|
||||||
|
var cmd = conn.CreateCommand();
|
||||||
|
cmd.CommandText = """
|
||||||
|
INSERT INTO dbo.ClusterNode (NodeId, ClusterId, Host, OpcUaPort, DashboardPort,
|
||||||
|
ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
|
||||||
|
VALUES (@nodeId, @clusterId, @host, 4840, 8081, @uri, 200, 1, 'raw-sql');
|
||||||
|
""";
|
||||||
|
cmd.Parameters.AddWithValue("@nodeId", $"{clusterId}-raw:4053");
|
||||||
|
cmd.Parameters.AddWithValue("@clusterId", clusterId);
|
||||||
|
cmd.Parameters.AddWithValue("@host", $"{clusterId}-raw");
|
||||||
|
cmd.Parameters.AddWithValue("@uri", $"urn:OtOpcUa:{clusterId}-raw");
|
||||||
|
await cmd.ExecuteNonQueryAsync();
|
||||||
|
}
|
||||||
|
|
||||||
|
await using var read = NewContext();
|
||||||
|
var row = await read.ClusterNodes.AsNoTracking()
|
||||||
|
.SingleAsync(n => n.NodeId == $"{clusterId}-raw:4053");
|
||||||
|
row.AkkaPort.ShouldBe(4053);
|
||||||
|
row.GrpcPort.ShouldBeNull();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Verifies the columns are queryable server-side — Phase 2's contact-point refresh selects
|
||||||
|
/// them in SQL rather than materialising every node, so a client-side-evaluation regression
|
||||||
|
/// (or an unmapped property) has to fail here.
|
||||||
|
/// </summary>
|
||||||
|
[Fact]
|
||||||
|
public async Task Transport_ports_are_queryable_server_side()
|
||||||
|
{
|
||||||
|
await using var ctx = NewContext();
|
||||||
|
var clusterId = await SeedClusterAsync(ctx);
|
||||||
|
ctx.ClusterNodes.Add(NewNode(clusterId, "a", n => n.GrpcPort = 5223));
|
||||||
|
ctx.ClusterNodes.Add(NewNode(clusterId, "b"));
|
||||||
|
await ctx.SaveChangesAsync();
|
||||||
|
|
||||||
|
await using var read = NewContext();
|
||||||
|
var contacts = await read.ClusterNodes.AsNoTracking()
|
||||||
|
.Where(n => n.ClusterId == clusterId && n.GrpcPort != null)
|
||||||
|
.Select(n => new { n.Host, n.AkkaPort, n.GrpcPort })
|
||||||
|
.ToListAsync();
|
||||||
|
|
||||||
|
contacts.Count.ShouldBe(1);
|
||||||
|
contacts[0].AkkaPort.ShouldBe(4053);
|
||||||
|
contacts[0].GrpcPort.ShouldBe(5223);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static ClusterNode NewNode(string clusterId, string suffix, Action<ClusterNode>? configure = null)
|
||||||
|
{
|
||||||
|
var node = new ClusterNode
|
||||||
|
{
|
||||||
|
NodeId = $"{clusterId}-{suffix}:4053",
|
||||||
|
ClusterId = clusterId,
|
||||||
|
Host = $"{clusterId}-{suffix}",
|
||||||
|
ApplicationUri = $"urn:OtOpcUa:{clusterId}-{suffix}",
|
||||||
|
CreatedBy = nameof(ClusterNodeTransportPortsTests),
|
||||||
|
};
|
||||||
|
configure?.Invoke(node);
|
||||||
|
return node;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static async Task<string> SeedClusterAsync(OtOpcUaConfigDbContext ctx)
|
||||||
|
{
|
||||||
|
// Unique per test: the collection shares one database, and ApplicationUri is fleet-wide unique.
|
||||||
|
var clusterId = $"PORTS-{Guid.NewGuid():N}"[..24];
|
||||||
|
ctx.ServerClusters.Add(new ServerCluster
|
||||||
|
{
|
||||||
|
ClusterId = clusterId,
|
||||||
|
Name = clusterId,
|
||||||
|
Enterprise = "zb",
|
||||||
|
Site = "ports-test",
|
||||||
|
// CK_ServerCluster_RedundancyMode_NodeCount: Warm/Hot require exactly 2 nodes.
|
||||||
|
RedundancyMode = RedundancyMode.Warm,
|
||||||
|
NodeCount = 2,
|
||||||
|
CreatedBy = nameof(ClusterNodeTransportPortsTests),
|
||||||
|
});
|
||||||
|
await ctx.SaveChangesAsync();
|
||||||
|
return clusterId;
|
||||||
|
}
|
||||||
|
|
||||||
|
private OtOpcUaConfigDbContext NewContext() =>
|
||||||
|
new(new DbContextOptionsBuilder<OtOpcUaConfigDbContext>()
|
||||||
|
.UseSqlServer(fixture.ConnectionString)
|
||||||
|
.Options);
|
||||||
|
}
|
||||||
+98
@@ -0,0 +1,98 @@
|
|||||||
|
using Akka.Actor;
|
||||||
|
using Microsoft.EntityFrameworkCore;
|
||||||
|
using Xunit;
|
||||||
|
using ZB.MOM.WW.OtOpcUa.Configuration;
|
||||||
|
using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
|
||||||
|
using ZB.MOM.WW.OtOpcUa.Configuration.Enums;
|
||||||
|
using ZB.MOM.WW.OtOpcUa.ControlPlane.Fleet;
|
||||||
|
using ZB.MOM.WW.OtOpcUa.ControlPlane.Tests.Harness;
|
||||||
|
|
||||||
|
namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Tests;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Proves the reconciler is actually wired to something — subscribes to membership, reads the DB,
|
||||||
|
/// and emits. <see cref="ClusterNodeAddressReconcilerTests"/> covers the comparison logic, which a
|
||||||
|
/// dormant actor would leave perfectly correct and completely inert.
|
||||||
|
/// </summary>
|
||||||
|
public sealed class ClusterNodeAddressReconcilerActorTests : ControlPlaneActorTestBase
|
||||||
|
{
|
||||||
|
/// <summary>
|
||||||
|
/// An enabled row with no matching driver member is warned about. The harness ActorSystem
|
||||||
|
/// joins as <c>admin</c> with no driver members, so the seeded row is unmatched by
|
||||||
|
/// construction.
|
||||||
|
/// </summary>
|
||||||
|
[Fact]
|
||||||
|
public void Enabled_row_with_no_matching_member_is_logged()
|
||||||
|
{
|
||||||
|
var dbFactory = NewInMemoryDbFactory();
|
||||||
|
SeedNode(dbFactory, "site-a-1:4053", enabled: true);
|
||||||
|
|
||||||
|
EventFilter.Warning(contains: "site-a-1:4053").ExpectOne(TimeSpan.FromSeconds(10), () =>
|
||||||
|
Sys.ActorOf(ClusterNodeAddressReconcilerActor.Props(dbFactory)));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// The finding set is logged once, not once per sweep. A check that reprints the same warning
|
||||||
|
/// every five minutes trains operators to filter it out, which costs more than it catches —
|
||||||
|
/// so this runs a deliberately fast sweep and asserts the count stays at one.
|
||||||
|
/// </summary>
|
||||||
|
[Fact]
|
||||||
|
public void Unchanged_findings_are_not_repeated_on_every_sweep()
|
||||||
|
{
|
||||||
|
var dbFactory = NewInMemoryDbFactory();
|
||||||
|
SeedNode(dbFactory, "site-a-1:4053", enabled: true);
|
||||||
|
|
||||||
|
// ~6 sweeps inside the window; without the change-detection guard this logs 6 times.
|
||||||
|
EventFilter.Warning(contains: "site-a-1:4053").Expect(1, TimeSpan.FromSeconds(3), () =>
|
||||||
|
{
|
||||||
|
Sys.ActorOf(ClusterNodeAddressReconcilerActor.Props(
|
||||||
|
dbFactory, sweepInterval: TimeSpan.FromMilliseconds(400)));
|
||||||
|
// Hold the filter open past several sweeps rather than returning immediately.
|
||||||
|
ExpectNoMsg(TimeSpan.FromSeconds(2.5));
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// A disabled row is silence — the maintenance hatch must not trade a failed deployment for a
|
||||||
|
/// permanent warning. Positive control for the test above: same setup, one flag flipped.
|
||||||
|
/// </summary>
|
||||||
|
[Fact]
|
||||||
|
public void Disabled_row_produces_no_warning()
|
||||||
|
{
|
||||||
|
var dbFactory = NewInMemoryDbFactory();
|
||||||
|
SeedNode(dbFactory, "site-a-1:4053", enabled: false);
|
||||||
|
|
||||||
|
EventFilter.Warning(contains: "site-a-1:4053").Expect(0, TimeSpan.FromSeconds(2), () =>
|
||||||
|
{
|
||||||
|
Sys.ActorOf(ClusterNodeAddressReconcilerActor.Props(
|
||||||
|
dbFactory, sweepInterval: TimeSpan.FromMilliseconds(400)));
|
||||||
|
ExpectNoMsg(TimeSpan.FromSeconds(1.5));
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
private static void SeedNode(
|
||||||
|
IDbContextFactory<OtOpcUaConfigDbContext> dbFactory, string nodeId, bool enabled)
|
||||||
|
{
|
||||||
|
using var db = dbFactory.CreateDbContext();
|
||||||
|
db.ServerClusters.Add(new ServerCluster
|
||||||
|
{
|
||||||
|
ClusterId = "SITE-A",
|
||||||
|
Name = "Site A",
|
||||||
|
Enterprise = "zb",
|
||||||
|
Site = "site-a",
|
||||||
|
NodeCount = 2,
|
||||||
|
RedundancyMode = RedundancyMode.Warm,
|
||||||
|
CreatedBy = "test",
|
||||||
|
});
|
||||||
|
db.ClusterNodes.Add(new ClusterNode
|
||||||
|
{
|
||||||
|
NodeId = nodeId,
|
||||||
|
ClusterId = "SITE-A",
|
||||||
|
Host = nodeId.Split(':')[0],
|
||||||
|
ApplicationUri = $"urn:OtOpcUa:{nodeId}",
|
||||||
|
Enabled = enabled,
|
||||||
|
CreatedBy = "test",
|
||||||
|
});
|
||||||
|
db.SaveChanges();
|
||||||
|
}
|
||||||
|
}
|
||||||
+126
@@ -0,0 +1,126 @@
|
|||||||
|
using Shouldly;
|
||||||
|
using Xunit;
|
||||||
|
using ZB.MOM.WW.OtOpcUa.ControlPlane.Fleet;
|
||||||
|
|
||||||
|
namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Tests;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Per-cluster mesh Phase 1 Task 4: <c>ClusterNode.AkkaPort</c> and the node's own
|
||||||
|
/// <c>Cluster:Port</c> are the same fact in two places, and nothing makes them agree. These pin
|
||||||
|
/// the shapes that drift takes.
|
||||||
|
/// </summary>
|
||||||
|
public sealed class ClusterNodeAddressReconcilerTests
|
||||||
|
{
|
||||||
|
private static ClusterNodeAddress Row(string host, int akkaPort, string? nodeId = null) =>
|
||||||
|
new(nodeId ?? $"{host}:{akkaPort}", host, akkaPort);
|
||||||
|
|
||||||
|
/// <summary>A fleet whose rows match membership reports nothing.</summary>
|
||||||
|
[Fact]
|
||||||
|
public void Matching_rows_and_membership_produce_no_findings()
|
||||||
|
{
|
||||||
|
var findings = ClusterNodeAddressReconciler.Reconcile(
|
||||||
|
[("site-a-1", 4053), ("site-a-2", 4053)],
|
||||||
|
[Row("site-a-1", 4053), Row("site-a-2", 4053)],
|
||||||
|
["site-a-1:4053", "site-a-2:4053"]);
|
||||||
|
|
||||||
|
findings.ShouldBeEmpty();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// The drift the plan names: a node binds 4054 while its row says 4053. Membership shows it
|
||||||
|
/// at an address no row claims, and the row it should have matched looks absent — both halves
|
||||||
|
/// are reported, because either alone reads as a different problem.
|
||||||
|
/// </summary>
|
||||||
|
[Fact]
|
||||||
|
public void Node_bound_to_the_wrong_port_is_reported_from_both_sides()
|
||||||
|
{
|
||||||
|
var findings = ClusterNodeAddressReconciler.Reconcile(
|
||||||
|
[("site-a-1", 4054)],
|
||||||
|
[Row("site-a-1", 4053)],
|
||||||
|
["site-a-1:4053"]);
|
||||||
|
|
||||||
|
findings.Count.ShouldBe(2);
|
||||||
|
findings.ShouldContain(f =>
|
||||||
|
f.Kind == AddressMismatchKind.RunningNodeHasNoRow && f.NodeId == "site-a-1:4054");
|
||||||
|
findings.ShouldContain(f =>
|
||||||
|
f.Kind == AddressMismatchKind.EnabledRowNotInCluster && f.NodeId == "site-a-1:4053");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// The Phase 2 shape: the row's NodeId still matches membership, but its <c>Host</c> /
|
||||||
|
/// <c>AkkaPort</c> columns — the values central will actually dial — have been edited away
|
||||||
|
/// from it. Gossip still works today, so nothing else would notice.
|
||||||
|
/// </summary>
|
||||||
|
[Fact]
|
||||||
|
public void Row_whose_dial_target_disagrees_with_its_own_NodeId_is_reported()
|
||||||
|
{
|
||||||
|
var findings = ClusterNodeAddressReconciler.Reconcile(
|
||||||
|
[("site-a-1", 4053)],
|
||||||
|
[Row("site-a-1", 4054, nodeId: "site-a-1:4053")],
|
||||||
|
["site-a-1:4053"]);
|
||||||
|
|
||||||
|
var only = findings.ShouldHaveSingleItem();
|
||||||
|
only.Kind.ShouldBe(AddressMismatchKind.RowDialTargetDisagrees);
|
||||||
|
only.NodeId.ShouldBe("site-a-1:4053");
|
||||||
|
only.Detail.ShouldContain("site-a-1:4054");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>A running driver node nobody registered — the coordinator discards its acks.</summary>
|
||||||
|
[Fact]
|
||||||
|
public void Running_node_with_no_row_is_reported()
|
||||||
|
{
|
||||||
|
var findings = ClusterNodeAddressReconciler.Reconcile(
|
||||||
|
[("ghost-9", 4053)], [], []);
|
||||||
|
|
||||||
|
findings.ShouldHaveSingleItem().Kind.ShouldBe(AddressMismatchKind.RunningNodeHasNoRow);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// A disabled row for an absent node is silence, not a finding — being absent is the entire
|
||||||
|
/// point of disabling it. Without this the maintenance hatch would trade a failed deploy for
|
||||||
|
/// a permanent warning.
|
||||||
|
/// </summary>
|
||||||
|
[Fact]
|
||||||
|
public void Disabled_row_for_an_absent_node_is_not_reported()
|
||||||
|
{
|
||||||
|
var findings = ClusterNodeAddressReconciler.Reconcile(
|
||||||
|
[("site-a-1", 4053)],
|
||||||
|
[Row("site-a-1", 4053), Row("site-a-2", 4053)],
|
||||||
|
["site-a-1:4053"]);
|
||||||
|
|
||||||
|
findings.ShouldBeEmpty();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// An admin-only member has no <c>ClusterNode</c> row by design, so the caller filters
|
||||||
|
/// membership to driver-role members. Passing an unfiltered list would report correct
|
||||||
|
/// configuration as broken — pinned here because the filter lives at the call site.
|
||||||
|
/// </summary>
|
||||||
|
[Fact]
|
||||||
|
public void Admin_only_members_are_the_callers_responsibility_to_exclude()
|
||||||
|
{
|
||||||
|
// What the actor passes: driver members only.
|
||||||
|
ClusterNodeAddressReconciler.Reconcile(
|
||||||
|
[("central-1", 4053)], [Row("central-1", 4053)], ["central-1:4053"])
|
||||||
|
.ShouldBeEmpty();
|
||||||
|
|
||||||
|
// What an unfiltered call would produce — the false positive this guards against.
|
||||||
|
ClusterNodeAddressReconciler.Reconcile(
|
||||||
|
[("central-1", 4053), ("admin-only-1", 4053)],
|
||||||
|
[Row("central-1", 4053)],
|
||||||
|
["central-1:4053"])
|
||||||
|
.ShouldHaveSingleItem().Kind.ShouldBe(AddressMismatchKind.RunningNodeHasNoRow);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Host matching is case-insensitive, matching the coordinator's <c>NodeIdComparer</c> — DNS
|
||||||
|
/// and SQL collation are both case-insensitive, so the same node surfaces with either casing.
|
||||||
|
/// </summary>
|
||||||
|
[Fact]
|
||||||
|
public void Host_matching_is_case_insensitive()
|
||||||
|
{
|
||||||
|
ClusterNodeAddressReconciler.Reconcile(
|
||||||
|
[("SITE-A-1", 4053)], [Row("site-a-1", 4053)], ["site-a-1:4053"])
|
||||||
|
.ShouldBeEmpty();
|
||||||
|
}
|
||||||
|
}
|
||||||
+263
@@ -0,0 +1,263 @@
|
|||||||
|
using Akka.Actor;
|
||||||
|
using Microsoft.EntityFrameworkCore;
|
||||||
|
using Shouldly;
|
||||||
|
using Xunit;
|
||||||
|
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy;
|
||||||
|
using ZB.MOM.WW.OtOpcUa.Commons.Types;
|
||||||
|
using ZB.MOM.WW.OtOpcUa.Configuration;
|
||||||
|
using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
|
||||||
|
using ZB.MOM.WW.OtOpcUa.Configuration.Enums;
|
||||||
|
using ZB.MOM.WW.OtOpcUa.ControlPlane.Coordinators;
|
||||||
|
using ZB.MOM.WW.OtOpcUa.ControlPlane.Tests.Harness;
|
||||||
|
|
||||||
|
namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Tests;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Per-cluster mesh Phase 1: the coordinator's expected-ack set comes from enabled
|
||||||
|
/// <c>ClusterNode</c> rows, not from <c>Akka.Cluster.State.Members</c> filtered by role.
|
||||||
|
/// </summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// <para>
|
||||||
|
/// <b>Why every test here seeds ClusterNode rows and asserts a non-empty expected set.</b>
|
||||||
|
/// The harness's ActorSystem self-joins with role <c>admin</c> and no <c>driver</c> members,
|
||||||
|
/// so the OLD derivation returned an empty set for every one of these scenarios and sealed
|
||||||
|
/// immediately. Any test that merely observes "the deployment did not seal" would therefore
|
||||||
|
/// have passed against the old code too. Each test below distinguishes the two by requiring
|
||||||
|
/// the coordinator to WAIT for a specific node and then seal when exactly that node acks —
|
||||||
|
/// behaviour the membership rule cannot produce in this harness at all.
|
||||||
|
/// </para>
|
||||||
|
/// <para>
|
||||||
|
/// Verified by reverting <c>DiscoverDriverNodes</c> to the membership scan: the first three
|
||||||
|
/// go red. <see cref="No_enabled_rows_seals_empty"/> stays green under both derivations and
|
||||||
|
/// is <b>not</b> a test of the derivation — the harness has no driver members AND no enabled
|
||||||
|
/// rows, so both rules produce an empty set. It pins the seal-empty branch and its reworded
|
||||||
|
/// reason, nothing more.
|
||||||
|
/// </para>
|
||||||
|
/// </remarks>
|
||||||
|
public sealed class ConfigPublishCoordinatorDbSourcedAcksTests : ControlPlaneActorTestBase
|
||||||
|
{
|
||||||
|
private static readonly RevisionHash TestRevision = RevisionHash.Parse(new string('b', 64));
|
||||||
|
private static readonly TimeSpan NoTimeout = TimeSpan.FromMinutes(5);
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// An enabled row is expected to ack; the deployment waits for it and seals when it arrives.
|
||||||
|
/// This is the DB-derived set doing something the membership set demonstrably cannot here.
|
||||||
|
/// </summary>
|
||||||
|
[Fact]
|
||||||
|
public void Enabled_ClusterNode_rows_are_the_expected_ack_set()
|
||||||
|
{
|
||||||
|
var dbFactory = NewInMemoryDbFactory();
|
||||||
|
var deploymentId = SeedDispatchingDeployment(dbFactory);
|
||||||
|
SeedNodes(dbFactory, ("site-a-1:4053", true, false));
|
||||||
|
|
||||||
|
var actor = Sys.ActorOf(ConfigPublishCoordinator.Props(dbFactory, NoTimeout));
|
||||||
|
actor.Tell(new DispatchDeployment(deploymentId, TestRevision, CorrelationId.NewId()));
|
||||||
|
|
||||||
|
// The seeded row must have produced a NodeDeploymentState row — proof the coordinator
|
||||||
|
// expected it, independent of whether it later sealed.
|
||||||
|
AwaitAssert(() =>
|
||||||
|
{
|
||||||
|
using var db = dbFactory.CreateDbContext();
|
||||||
|
db.NodeDeploymentStates.Select(s => s.NodeId).ShouldBe(new[] { "site-a-1:4053" });
|
||||||
|
db.Deployments.Single().Status.ShouldBe(DeploymentStatus.AwaitingApplyAcks);
|
||||||
|
}, duration: TimeSpan.FromSeconds(3));
|
||||||
|
|
||||||
|
actor.Tell(new ApplyAck(deploymentId, NodeId.Parse("site-a-1:4053"),
|
||||||
|
ApplyAckOutcome.Applied, null, CorrelationId.NewId()));
|
||||||
|
|
||||||
|
AwaitAssert(() =>
|
||||||
|
{
|
||||||
|
using var db = dbFactory.CreateDbContext();
|
||||||
|
db.Deployments.Single().Status.ShouldBe(DeploymentStatus.Sealed);
|
||||||
|
}, duration: TimeSpan.FromSeconds(3));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// A disabled row is NOT expected, so a deployment seals without it — for a node genuinely
|
||||||
|
/// decommissioned rather than temporarily down. <b>This is not the maintenance hatch</b>: the
|
||||||
|
/// topology gate rejects a deployment whose enabled-node count differs from the cluster's
|
||||||
|
/// <c>NodeCount</c>, so on any 2-node pair clearing <c>Enabled</c> never reaches this code.
|
||||||
|
/// See <see cref="MaintenanceMode_rows_are_not_expected_to_ack_while_staying_enabled"/>.
|
||||||
|
/// </summary>
|
||||||
|
[Fact]
|
||||||
|
public void Disabled_ClusterNode_rows_are_not_expected_to_ack()
|
||||||
|
{
|
||||||
|
var dbFactory = NewInMemoryDbFactory();
|
||||||
|
var deploymentId = SeedDispatchingDeployment(dbFactory);
|
||||||
|
SeedNodes(dbFactory, ("site-a-1:4053", true, false), ("site-a-2:4053", false, false));
|
||||||
|
|
||||||
|
var actor = Sys.ActorOf(ConfigPublishCoordinator.Props(dbFactory, NoTimeout));
|
||||||
|
actor.Tell(new DispatchDeployment(deploymentId, TestRevision, CorrelationId.NewId()));
|
||||||
|
|
||||||
|
// Only the enabled node is expected — the disabled one gets no NodeDeploymentState row.
|
||||||
|
AwaitAssert(() =>
|
||||||
|
{
|
||||||
|
using var db = dbFactory.CreateDbContext();
|
||||||
|
db.NodeDeploymentStates.Select(s => s.NodeId).ShouldBe(new[] { "site-a-1:4053" });
|
||||||
|
}, duration: TimeSpan.FromSeconds(3));
|
||||||
|
|
||||||
|
// ...and the enabled node's ack alone seals it. If the disabled row were expected this would
|
||||||
|
// sit at AwaitingApplyAcks until the deadline.
|
||||||
|
actor.Tell(new ApplyAck(deploymentId, NodeId.Parse("site-a-1:4053"),
|
||||||
|
ApplyAckOutcome.Applied, null, CorrelationId.NewId()));
|
||||||
|
|
||||||
|
AwaitAssert(() =>
|
||||||
|
{
|
||||||
|
using var db = dbFactory.CreateDbContext();
|
||||||
|
db.Deployments.Single().Status.ShouldBe(DeploymentStatus.Sealed);
|
||||||
|
}, duration: TimeSpan.FromSeconds(3));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// An ack from a node with no <c>ClusterNode</c> row is discarded — it is not in the expected
|
||||||
|
/// set, so it must not count toward the seal. Under the membership rule a running-but-
|
||||||
|
/// unregistered node WAS expected, and then threw FK 547 when its state row was written.
|
||||||
|
/// </summary>
|
||||||
|
[Fact]
|
||||||
|
public void Ack_from_a_node_absent_from_ClusterNode_is_discarded()
|
||||||
|
{
|
||||||
|
var dbFactory = NewInMemoryDbFactory();
|
||||||
|
var deploymentId = SeedDispatchingDeployment(dbFactory);
|
||||||
|
SeedNodes(dbFactory, ("site-a-1:4053", true, false));
|
||||||
|
|
||||||
|
var actor = Sys.ActorOf(ConfigPublishCoordinator.Props(dbFactory, NoTimeout));
|
||||||
|
actor.Tell(new DispatchDeployment(deploymentId, TestRevision, CorrelationId.NewId()));
|
||||||
|
AwaitAssert(() =>
|
||||||
|
{
|
||||||
|
using var db = dbFactory.CreateDbContext();
|
||||||
|
db.Deployments.Single().Status.ShouldBe(DeploymentStatus.AwaitingApplyAcks);
|
||||||
|
}, duration: TimeSpan.FromSeconds(3));
|
||||||
|
|
||||||
|
// A node nobody registered acks. One expected node, one ack — if it were counted, the
|
||||||
|
// coordinator would seal here.
|
||||||
|
actor.Tell(new ApplyAck(deploymentId, NodeId.Parse("ghost-9:4053"),
|
||||||
|
ApplyAckOutcome.Applied, null, CorrelationId.NewId()));
|
||||||
|
ExpectNoMsg(TimeSpan.FromMilliseconds(400));
|
||||||
|
|
||||||
|
using (var db = dbFactory.CreateDbContext())
|
||||||
|
{
|
||||||
|
db.Deployments.Single().Status.ShouldBe(DeploymentStatus.AwaitingApplyAcks);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Positive control on the wait itself: the real node's ack DOES seal it, so the assertion
|
||||||
|
// above measured the discard rather than a coordinator that had simply stopped responding.
|
||||||
|
actor.Tell(new ApplyAck(deploymentId, NodeId.Parse("site-a-1:4053"),
|
||||||
|
ApplyAckOutcome.Applied, null, CorrelationId.NewId()));
|
||||||
|
AwaitAssert(() =>
|
||||||
|
{
|
||||||
|
using var db = dbFactory.CreateDbContext();
|
||||||
|
db.Deployments.Single().Status.ShouldBe(DeploymentStatus.Sealed);
|
||||||
|
}, duration: TimeSpan.FromSeconds(3));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// No enabled rows at all seals empty. Same outcome as the old empty-membership branch, but
|
||||||
|
/// it now means "the fleet has no enabled nodes registered" — a configuration error that
|
||||||
|
/// cannot resolve itself — rather than "no driver node has joined yet".
|
||||||
|
/// </summary>
|
||||||
|
[Fact]
|
||||||
|
public void No_enabled_rows_seals_empty()
|
||||||
|
{
|
||||||
|
var dbFactory = NewInMemoryDbFactory();
|
||||||
|
var deploymentId = SeedDispatchingDeployment(dbFactory);
|
||||||
|
SeedNodes(dbFactory, ("site-a-1:4053", false, false), ("site-a-2:4053", false, false));
|
||||||
|
|
||||||
|
var actor = Sys.ActorOf(ConfigPublishCoordinator.Props(dbFactory, NoTimeout));
|
||||||
|
actor.Tell(new DispatchDeployment(deploymentId, TestRevision, CorrelationId.NewId()));
|
||||||
|
|
||||||
|
AwaitAssert(() =>
|
||||||
|
{
|
||||||
|
using var db = dbFactory.CreateDbContext();
|
||||||
|
db.Deployments.Single().Status.ShouldBe(DeploymentStatus.Sealed);
|
||||||
|
db.NodeDeploymentStates.ShouldBeEmpty();
|
||||||
|
}, duration: TimeSpan.FromSeconds(3));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// The maintenance hatch, and the reason it is a separate flag. The Phase 1 live gate set
|
||||||
|
/// <c>Enabled = 0</c> on one node of a 2-node pair and the deployment was rejected outright by
|
||||||
|
/// <c>DraftValidator.ValidateClusterTopology</c> (enabled-node count must equal
|
||||||
|
/// <c>NodeCount</c>) — so the intended escape hatch could not be used on any pair, which is
|
||||||
|
/// every cluster in the target topology. <c>MaintenanceMode</c> excludes a node from the
|
||||||
|
/// expected-ack set while leaving it enabled, so the topology gate still passes.
|
||||||
|
/// </summary>
|
||||||
|
[Fact]
|
||||||
|
public void MaintenanceMode_rows_are_not_expected_to_ack_while_staying_enabled()
|
||||||
|
{
|
||||||
|
var dbFactory = NewInMemoryDbFactory();
|
||||||
|
var deploymentId = SeedDispatchingDeployment(dbFactory);
|
||||||
|
SeedNodes(dbFactory, ("site-a-1:4053", true, false), ("site-a-2:4053", true, true));
|
||||||
|
|
||||||
|
// The topology gate counts Enabled, so both nodes still count — this is what Enabled = 0
|
||||||
|
// broke. Assert it here so the fix cannot be "quietly disable it after all".
|
||||||
|
using (var check = dbFactory.CreateDbContext())
|
||||||
|
{
|
||||||
|
check.ClusterNodes.Count(n => n.Enabled).ShouldBe(2);
|
||||||
|
}
|
||||||
|
|
||||||
|
var actor = Sys.ActorOf(ConfigPublishCoordinator.Props(dbFactory, NoTimeout));
|
||||||
|
actor.Tell(new DispatchDeployment(deploymentId, TestRevision, CorrelationId.NewId()));
|
||||||
|
|
||||||
|
AwaitAssert(() =>
|
||||||
|
{
|
||||||
|
using var db = dbFactory.CreateDbContext();
|
||||||
|
db.NodeDeploymentStates.Select(s => s.NodeId).ShouldBe(new[] { "site-a-1:4053" });
|
||||||
|
}, duration: TimeSpan.FromSeconds(3));
|
||||||
|
|
||||||
|
actor.Tell(new ApplyAck(deploymentId, NodeId.Parse("site-a-1:4053"),
|
||||||
|
ApplyAckOutcome.Applied, null, CorrelationId.NewId()));
|
||||||
|
|
||||||
|
AwaitAssert(() =>
|
||||||
|
{
|
||||||
|
using var db = dbFactory.CreateDbContext();
|
||||||
|
db.Deployments.Single().Status.ShouldBe(DeploymentStatus.Sealed);
|
||||||
|
}, duration: TimeSpan.FromSeconds(3));
|
||||||
|
}
|
||||||
|
|
||||||
|
private static void SeedNodes(
|
||||||
|
IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
|
||||||
|
params (string NodeId, bool Enabled, bool Maintenance)[] nodes)
|
||||||
|
{
|
||||||
|
using var db = dbFactory.CreateDbContext();
|
||||||
|
db.ServerClusters.Add(new ServerCluster
|
||||||
|
{
|
||||||
|
ClusterId = "SITE-A",
|
||||||
|
Name = "Site A",
|
||||||
|
Enterprise = "zb",
|
||||||
|
Site = "site-a",
|
||||||
|
NodeCount = 2,
|
||||||
|
RedundancyMode = RedundancyMode.Warm,
|
||||||
|
CreatedBy = "test",
|
||||||
|
});
|
||||||
|
foreach (var (nodeId, enabled, maintenance) in nodes)
|
||||||
|
{
|
||||||
|
db.ClusterNodes.Add(new ClusterNode
|
||||||
|
{
|
||||||
|
NodeId = nodeId,
|
||||||
|
ClusterId = "SITE-A",
|
||||||
|
Host = nodeId.Split(':')[0],
|
||||||
|
ApplicationUri = $"urn:OtOpcUa:{nodeId}",
|
||||||
|
Enabled = enabled,
|
||||||
|
MaintenanceMode = maintenance,
|
||||||
|
CreatedBy = "test",
|
||||||
|
});
|
||||||
|
}
|
||||||
|
db.SaveChanges();
|
||||||
|
}
|
||||||
|
|
||||||
|
private static DeploymentId SeedDispatchingDeployment(
|
||||||
|
IDbContextFactory<OtOpcUaConfigDbContext> dbFactory)
|
||||||
|
{
|
||||||
|
var id = DeploymentId.NewId();
|
||||||
|
using var db = dbFactory.CreateDbContext();
|
||||||
|
db.Deployments.Add(new Deployment
|
||||||
|
{
|
||||||
|
DeploymentId = id.Value,
|
||||||
|
RevisionHash = TestRevision.Value,
|
||||||
|
Status = DeploymentStatus.Dispatching,
|
||||||
|
CreatedBy = "test",
|
||||||
|
});
|
||||||
|
db.SaveChanges();
|
||||||
|
return id;
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,14 @@
|
|||||||
|
using Xunit;
|
||||||
|
|
||||||
|
// Every test in this assembly builds a real two-node Akka cluster: two in-process Kestrel hosts,
|
||||||
|
// two ActorSystems with remoting on ephemeral ports, six admin-role cluster singletons, an EF
|
||||||
|
// context and a deploy pipeline. Running several of those concurrently — on top of the other test
|
||||||
|
// assemblies dotnet test runs in parallel — starves the cluster-formation and ApplyAck timers these
|
||||||
|
// tests measure, so failures appear under full-suite load and vanish in isolation.
|
||||||
|
//
|
||||||
|
// That was tolerable while the deploy path sealed immediately on an empty expected-ack set. Since
|
||||||
|
// per-cluster mesh Phase 1 the coordinator waits for a real ack from every configured node, so these
|
||||||
|
// tests now measure an end-to-end round-trip and are correspondingly more sensitive to CPU
|
||||||
|
// starvation. Serialising this assembly trades wall-clock for determinism; the assembly is a handful
|
||||||
|
// of heavyweight E2E tests, not a broad unit suite.
|
||||||
|
[assembly: CollectionBehavior(DisableTestParallelization = true)]
|
||||||
@@ -40,7 +40,7 @@ public sealed class DeployHappyPathTests
|
|||||||
await using var db = await CreateDbAsync(harness);
|
await using var db = await CreateDbAsync(harness);
|
||||||
var d = await db.Deployments.AsNoTracking().FirstOrDefaultAsync(d => d.DeploymentId == deploymentId, Ct);
|
var d = await db.Deployments.AsNoTracking().FirstOrDefaultAsync(d => d.DeploymentId == deploymentId, Ct);
|
||||||
return d?.Status == DeploymentStatus.Sealed;
|
return d?.Status == DeploymentStatus.Sealed;
|
||||||
}, TimeSpan.FromSeconds(15));
|
}, TwoNodeClusterHarness.DeploySealTimeout);
|
||||||
|
|
||||||
await using var db = await CreateDbAsync(harness);
|
await using var db = await CreateDbAsync(harness);
|
||||||
var deployment = await db.Deployments.AsNoTracking()
|
var deployment = await db.Deployments.AsNoTracking()
|
||||||
@@ -74,7 +74,7 @@ public sealed class DeployHappyPathTests
|
|||||||
var d = await db.Deployments.AsNoTracking()
|
var d = await db.Deployments.AsNoTracking()
|
||||||
.FirstOrDefaultAsync(d => d.DeploymentId == first.DeploymentId!.Value.Value, Ct);
|
.FirstOrDefaultAsync(d => d.DeploymentId == first.DeploymentId!.Value.Value, Ct);
|
||||||
return d?.Status == DeploymentStatus.Sealed;
|
return d?.Status == DeploymentStatus.Sealed;
|
||||||
}, TimeSpan.FromSeconds(15));
|
}, TwoNodeClusterHarness.DeploySealTimeout);
|
||||||
|
|
||||||
// Same DB state → same revision hash → AdminOperations short-circuits with NoChanges,
|
// Same DB state → same revision hash → AdminOperations short-circuits with NoChanges,
|
||||||
// OR (if it accepts) the second dispatch is idempotent (DriverHostActor recognises
|
// OR (if it accepts) the second dispatch is idempotent (DriverHostActor recognises
|
||||||
@@ -90,7 +90,7 @@ public sealed class DeployHappyPathTests
|
|||||||
var d = await db.Deployments.AsNoTracking()
|
var d = await db.Deployments.AsNoTracking()
|
||||||
.FirstOrDefaultAsync(d => d.DeploymentId == second.DeploymentId!.Value.Value, Ct);
|
.FirstOrDefaultAsync(d => d.DeploymentId == second.DeploymentId!.Value.Value, Ct);
|
||||||
return d?.Status == DeploymentStatus.Sealed;
|
return d?.Status == DeploymentStatus.Sealed;
|
||||||
}, TimeSpan.FromSeconds(15));
|
}, TwoNodeClusterHarness.DeploySealTimeout);
|
||||||
|
|
||||||
second.RevisionHash!.Value.Value.ShouldBe(first.RevisionHash!.Value.Value);
|
second.RevisionHash!.Value.Value.ShouldBe(first.RevisionHash!.Value.Value);
|
||||||
}
|
}
|
||||||
|
|||||||
+2
-2
@@ -85,7 +85,7 @@ public sealed class EquipmentNamespaceMaterializationTests
|
|||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
return false;
|
return false;
|
||||||
}, TimeSpan.FromSeconds(15));
|
}, TwoNodeClusterHarness.DeploySealTimeout);
|
||||||
|
|
||||||
var composition = DeploymentArtifact.ParseComposition(artifact);
|
var composition = DeploymentArtifact.ParseComposition(artifact);
|
||||||
|
|
||||||
@@ -182,7 +182,7 @@ public sealed class EquipmentNamespaceMaterializationTests
|
|||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
return false;
|
return false;
|
||||||
}, TimeSpan.FromSeconds(20));
|
}, TwoNodeClusterHarness.DeploySealTimeout);
|
||||||
|
|
||||||
await using (var db = await CreateDbAsync(harness))
|
await using (var db = await CreateDbAsync(harness))
|
||||||
{
|
{
|
||||||
|
|||||||
@@ -50,13 +50,22 @@ public sealed class FailoverDuringDeployTests
|
|||||||
.Count(m => m.Status == MemberStatus.Up).ShouldBe(2);
|
.Count(m => m.Status == MemberStatus.Up).ShouldBe(2);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// <summary>Verifies that a deployment started with node B down seals with one-node state.</summary>
|
/// <summary>
|
||||||
|
/// A deployment started with node B down no longer seals without it — B's <c>ClusterNode</c>
|
||||||
|
/// row is enabled, so it is expected to ack and the deployment waits.
|
||||||
|
/// </summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// <b>This test asserted the opposite until per-cluster mesh Phase 1.</b> It was named
|
||||||
|
/// <c>Deployment_started_with_node_b_down_seals_with_one_node_state</c> and documented that
|
||||||
|
/// "<c>DiscoverDriverNodes</c> snapshots membership at dispatch time — when only node A is Up,
|
||||||
|
/// only one ApplyAck is expected and the deployment seals without B ever participating". That
|
||||||
|
/// is exactly the behaviour Phase 1 removed: it told the operator the fleet was deployed while
|
||||||
|
/// a configured node had not received it. The expected-ack set now comes from enabled
|
||||||
|
/// <c>ClusterNode</c> rows, so a node that is merely switched off is still expected.
|
||||||
|
/// </remarks>
|
||||||
[Fact]
|
[Fact]
|
||||||
public async Task Deployment_started_with_node_b_down_seals_with_one_node_state()
|
public async Task Deployment_started_with_node_b_down_does_not_seal_without_it()
|
||||||
{
|
{
|
||||||
// Establishes that ConfigPublishCoordinator.DiscoverDriverNodes snapshots membership at
|
|
||||||
// dispatch time — when only node A is Up, only one ApplyAck is expected and the
|
|
||||||
// deployment seals without B ever participating.
|
|
||||||
await using var harness = await TwoNodeClusterHarness.StartAsync();
|
await using var harness = await TwoNodeClusterHarness.StartAsync();
|
||||||
await harness.SeedDefaultClusterAsync();
|
await harness.SeedDefaultClusterAsync();
|
||||||
|
|
||||||
@@ -70,19 +79,78 @@ public sealed class FailoverDuringDeployTests
|
|||||||
result.Outcome.ShouldBe(StartDeploymentOutcome.Accepted);
|
result.Outcome.ShouldBe(StartDeploymentOutcome.Accepted);
|
||||||
var deploymentId = result.DeploymentId!.Value.Value;
|
var deploymentId = result.DeploymentId!.Value.Value;
|
||||||
|
|
||||||
|
// Positive evidence that B was EXPECTED, not merely slow: the coordinator seeds a row per
|
||||||
|
// expected node at dispatch, so both rows must exist with B still Applying. Asserting only
|
||||||
|
// "it didn't seal" would pass just as well against a coordinator that had died.
|
||||||
await WaitForAsync(async () =>
|
await WaitForAsync(async () =>
|
||||||
{
|
{
|
||||||
await using var db = await CreateDbAsync(harness);
|
await using var pollDb = await CreateDbAsync(harness);
|
||||||
var d = await db.Deployments.AsNoTracking()
|
return await pollDb.NodeDeploymentStates.AsNoTracking()
|
||||||
.FirstOrDefaultAsync(d => d.DeploymentId == deploymentId, Ct);
|
.CountAsync(s => s.DeploymentId == deploymentId, Ct) == 2;
|
||||||
return d?.Status == DeploymentStatus.Sealed;
|
}, TwoNodeClusterHarness.DeploySealTimeout);
|
||||||
}, TimeSpan.FromSeconds(15));
|
|
||||||
|
|
||||||
await using var db = await CreateDbAsync(harness);
|
await using var db = await CreateDbAsync(harness);
|
||||||
var nodeStates = await db.NodeDeploymentStates.AsNoTracking()
|
var nodeStates = await db.NodeDeploymentStates.AsNoTracking()
|
||||||
.Where(s => s.DeploymentId == deploymentId)
|
.Where(s => s.DeploymentId == deploymentId)
|
||||||
.ToListAsync(Ct);
|
.ToListAsync(Ct);
|
||||||
nodeStates.Count.ShouldBe(1);
|
nodeStates.Count.ShouldBe(2, "both configured nodes are expected to ack");
|
||||||
|
nodeStates.Count(s => s.Status == NodeDeploymentStatus.Applied)
|
||||||
|
.ShouldBe(1, "only the running node applied");
|
||||||
|
nodeStates.ShouldContain(s => s.Status == NodeDeploymentStatus.Applying,
|
||||||
|
"the stopped node's ack is still outstanding");
|
||||||
|
|
||||||
|
var deployment = await db.Deployments.AsNoTracking()
|
||||||
|
.FirstAsync(d => d.DeploymentId == deploymentId, Ct);
|
||||||
|
deployment.Status.ShouldNotBe(DeploymentStatus.Sealed,
|
||||||
|
"a deployment must not seal green while a configured node has not received it");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// The maintenance hatch, end-to-end: a node down <i>and</i> flagged
|
||||||
|
/// <c>MaintenanceMode</c> is not expected, so the deployment seals with one node state — the
|
||||||
|
/// behaviour the test above used to assert unconditionally, now something an operator has to
|
||||||
|
/// ask for.
|
||||||
|
/// </summary>
|
||||||
|
[Fact]
|
||||||
|
public async Task Deployment_seals_without_a_node_flagged_for_maintenance()
|
||||||
|
{
|
||||||
|
await using var harness = await TwoNodeClusterHarness.StartAsync();
|
||||||
|
await harness.SeedDefaultClusterAsync();
|
||||||
|
|
||||||
|
await harness.StopNodeBAsync();
|
||||||
|
await harness.WaitForClusterSizeAsync(1, TimeSpan.FromSeconds(20));
|
||||||
|
|
||||||
|
await using (var setup = await CreateDbAsync(harness))
|
||||||
|
{
|
||||||
|
var nodeB = await setup.ClusterNodes.FirstAsync(n => n.NodeId == harness.NodeBNodeId, Ct);
|
||||||
|
nodeB.MaintenanceMode = true;
|
||||||
|
await setup.SaveChangesAsync(Ct);
|
||||||
|
// Still Enabled — DraftValidator.ValidateClusterTopology requires the enabled-node count
|
||||||
|
// to equal ServerCluster.NodeCount, which is why MaintenanceMode exists as its own flag.
|
||||||
|
nodeB.Enabled.ShouldBeTrue();
|
||||||
|
}
|
||||||
|
|
||||||
|
await using var scope = harness.NodeA.Services.CreateAsyncScope();
|
||||||
|
var client = scope.ServiceProvider.GetRequiredService<IAdminOperationsClient>();
|
||||||
|
|
||||||
|
var result = await client.StartDeploymentAsync(createdBy: "alice@test", Ct);
|
||||||
|
result.Outcome.ShouldBe(StartDeploymentOutcome.Accepted, $"Deploy not accepted: {result.Message}");
|
||||||
|
var deploymentId = result.DeploymentId!.Value.Value;
|
||||||
|
|
||||||
|
await WaitForAsync(async () =>
|
||||||
|
{
|
||||||
|
await using var pollDb = await CreateDbAsync(harness);
|
||||||
|
var d = await pollDb.Deployments.AsNoTracking()
|
||||||
|
.FirstOrDefaultAsync(d => d.DeploymentId == deploymentId, Ct);
|
||||||
|
return d?.Status == DeploymentStatus.Sealed;
|
||||||
|
}, TwoNodeClusterHarness.DeploySealTimeout);
|
||||||
|
|
||||||
|
await using var db = await CreateDbAsync(harness);
|
||||||
|
var nodeStates = await db.NodeDeploymentStates.AsNoTracking()
|
||||||
|
.Where(s => s.DeploymentId == deploymentId)
|
||||||
|
.ToListAsync(Ct);
|
||||||
|
nodeStates.Count.ShouldBe(1, "the maintenance node is not expected to ack");
|
||||||
|
nodeStates[0].NodeId.ShouldBe(harness.NodeANodeId);
|
||||||
nodeStates[0].Status.ShouldBe(NodeDeploymentStatus.Applied);
|
nodeStates[0].Status.ShouldBe(NodeDeploymentStatus.Applied);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
+1
-1
@@ -67,7 +67,7 @@ public sealed class FleetDiagnosticsRoundTripTests
|
|||||||
{
|
{
|
||||||
var snap = await diagnostics.GetDiagnosticsAsync(nodeId, Ct);
|
var snap = await diagnostics.GetDiagnosticsAsync(nodeId, Ct);
|
||||||
return snap.CurrentRevision == expectedRev;
|
return snap.CurrentRevision == expectedRev;
|
||||||
}, TimeSpan.FromSeconds(15));
|
}, TwoNodeClusterHarness.DeploySealTimeout);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -96,6 +96,16 @@ public sealed class TwoNodeClusterHarness : IAsyncDisposable
|
|||||||
// host:port into NodeId so the cluster membership stays distinct on different ports.
|
// host:port into NodeId so the cluster membership stays distinct on different ports.
|
||||||
public const string LoopbackHost = "127.0.0.1";
|
public const string LoopbackHost = "127.0.0.1";
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// How long a deploy-seal assertion waits. Generous on purpose: since per-cluster mesh
|
||||||
|
/// Phase 1 the coordinator waits for a real ApplyAck from every configured node before
|
||||||
|
/// sealing, so this measures an end-to-end two-node deploy rather than — as it did when the
|
||||||
|
/// expected-ack set came up empty — an immediate seal. The old 15s had enormous margin
|
||||||
|
/// against "instant" and thin margin against the real thing, which showed up as deploy
|
||||||
|
/// tests failing only under full-suite CPU contention.
|
||||||
|
/// </summary>
|
||||||
|
public static readonly TimeSpan DeploySealTimeout = TimeSpan.FromSeconds(45);
|
||||||
|
|
||||||
/// <summary>Gets the Akka ActorSystem for node A.</summary>
|
/// <summary>Gets the Akka ActorSystem for node A.</summary>
|
||||||
public ActorSystem NodeASystem => NodeA.Services.GetRequiredService<ActorSystem>();
|
public ActorSystem NodeASystem => NodeA.Services.GetRequiredService<ActorSystem>();
|
||||||
/// <summary>Gets the Akka ActorSystem for node B.</summary>
|
/// <summary>Gets the Akka ActorSystem for node B.</summary>
|
||||||
@@ -122,16 +132,26 @@ public sealed class TwoNodeClusterHarness : IAsyncDisposable
|
|||||||
/// Seeds a default <see cref="ServerCluster"/> plus a <see cref="ClusterNode"/> row for BOTH
|
/// Seeds a default <see cref="ServerCluster"/> plus a <see cref="ClusterNode"/> row for BOTH
|
||||||
/// harness nodes (<see cref="NodeANodeId"/> / <see cref="NodeBNodeId"/>) so the real-SQL FK
|
/// harness nodes (<see cref="NodeANodeId"/> / <see cref="NodeBNodeId"/>) so the real-SQL FK
|
||||||
/// constraint <c>FK_NodeDeploymentState_ClusterNode_NodeId</c> is satisfied when a deployment
|
/// constraint <c>FK_NodeDeploymentState_ClusterNode_NodeId</c> is satisfied when a deployment
|
||||||
/// records per-node state. The EF in-memory provider ignores FK constraints, so deploy E2E
|
/// records per-node state. Against SQL Server each node's <c>NodeDeploymentState</c> INSERT
|
||||||
/// tests pass without this; against SQL Server each node's <c>NodeDeploymentState</c> INSERT
|
|
||||||
/// fails without its parent <see cref="ClusterNode"/> row and the deployment never seals.
|
/// fails without its parent <see cref="ClusterNode"/> row and the deployment never seals.
|
||||||
/// No-op unless <c>OTOPCUA_HARNESS_USE_SQL=1</c> (in-memory needs no seeding). Call once, before
|
/// Call once, before <c>StartDeploymentAsync</c>, in tests that don't already seed their own
|
||||||
/// <c>StartDeploymentAsync</c>, in tests that don't already seed their own cluster + both nodes.
|
/// cluster + both nodes.
|
||||||
/// </summary>
|
/// </summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// <b>This used to no-op in in-memory mode</b> ("the in-memory provider ignores FK
|
||||||
|
/// constraints, so deploy E2E tests pass without it"). That stopped being true in per-cluster
|
||||||
|
/// mesh Phase 1: <c>ConfigPublishCoordinator</c> now derives its expected-ack set from these
|
||||||
|
/// rows rather than from cluster membership, so with none seeded it seals <i>immediately</i>
|
||||||
|
/// with an empty set. "Wait for Sealed" then no longer implies "every node has applied", and
|
||||||
|
/// since <c>DriverHostActor.UpsertNodeDeploymentState</c> writes each node's row on its own
|
||||||
|
/// schedule, any test counting those rows after a seal became a race — reproducibly green
|
||||||
|
/// alone and red under suite load. Seeding in both modes restores the invariant the tests
|
||||||
|
/// were written against, and is closer to production either way: a real fleet always has
|
||||||
|
/// these rows, because the FK requires them.
|
||||||
|
/// </remarks>
|
||||||
/// <param name="clusterId">Cluster id for the seeded rows. Defaults to <c>MAIN</c>.</param>
|
/// <param name="clusterId">Cluster id for the seeded rows. Defaults to <c>MAIN</c>.</param>
|
||||||
public async Task SeedDefaultClusterAsync(string clusterId = "MAIN")
|
public async Task SeedDefaultClusterAsync(string clusterId = "MAIN")
|
||||||
{
|
{
|
||||||
if (!Mode.UseSqlServer) return;
|
|
||||||
await using var db = await CreateConfigDbContextAsync();
|
await using var db = await CreateConfigDbContextAsync();
|
||||||
db.ServerClusters.Add(new ServerCluster
|
db.ServerClusters.Add(new ServerCluster
|
||||||
{
|
{
|
||||||
|
|||||||
Reference in New Issue
Block a user