docs(design): per-ClusterId scoping for hub-and-spoke single mesh
Central cluster (2 fused admin+driver nodes) hosts the only UI + deploy singleton; site clusters (2 driver-only nodes each) join the central mesh and are logically separated by ClusterId. Each node applies only its own cluster's drivers + address space on a global deploy. Approved design; next step is the implementation plan.
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,193 @@
|
|||||||
|
# Per-ClusterId Scoping (hub-and-spoke single mesh) — Design
|
||||||
|
|
||||||
|
**Date:** 2026-06-07
|
||||||
|
**Status:** Approved (brainstorming complete; next step: writing-plans)
|
||||||
|
**Branch:** `feat/per-cluster-scoping`
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
Let one **central** cluster's Admin UI manage and deploy to multiple
|
||||||
|
logically-separate clusters that share a single Akka mesh. The central cluster
|
||||||
|
runs 2 fused `admin,driver` nodes (the only UI + the only deploy singleton);
|
||||||
|
each site cluster runs 2 `driver`-only nodes. A single global deploy from the
|
||||||
|
central UI reaches every node, and **each node applies only the slice of the
|
||||||
|
configuration that belongs to its own `ClusterId`** — its drivers and its OPC UA
|
||||||
|
address space. Ship global deploy first; per-cluster deploy is a later follow-up.
|
||||||
|
|
||||||
|
## Why this needs runtime work
|
||||||
|
|
||||||
|
The deploy channel is **in-mesh**: AdminUI → `admin-operations` singleton →
|
||||||
|
`ConfigPublishCoordinator` → DistributedPubSub → driver nodes. DistributedPubSub
|
||||||
|
does not cross Akka mesh boundaries, so for the central UI to deploy to site
|
||||||
|
servers the site nodes **must join the central mesh**. But the runtime currently
|
||||||
|
assumes **one Akka mesh == one logical cluster**:
|
||||||
|
|
||||||
|
- `DriverHostActor.ReconcileDrivers` spawns **every** `DriverInstance` in the
|
||||||
|
artifact with no cluster filter (`DriverHostActor.cs:367`). The `ClusterId` on
|
||||||
|
a spec is used only to *label* health snapshots.
|
||||||
|
- `ConfigPublishCoordinator.DiscoverDriverNodes` broadcasts to **every** driver
|
||||||
|
member of the mesh, no `ClusterId` filter (`ConfigPublishCoordinator.cs:248`).
|
||||||
|
- `ConfigComposer.SnapshotAndFlattenAsync` snapshots **all** clusters' rows into
|
||||||
|
one flat artifact; the address space is built from the whole thing.
|
||||||
|
|
||||||
|
Consequence today: put MAIN + SITE-A + SITE-B nodes in one mesh and every node
|
||||||
|
spawns every cluster's drivers (Galaxy auto-stubs on Linux, so it *would* start)
|
||||||
|
and serves a **merged** address space of all three clusters. That is why the
|
||||||
|
existing docker-dev rig uses three isolated meshes.
|
||||||
|
|
||||||
|
This design adds the missing per-`ClusterId` scoping so a shared mesh behaves as
|
||||||
|
distinct logical clusters.
|
||||||
|
|
||||||
|
## Approach (chosen: A — node-side, parse-time filter, ClusterId from the artifact)
|
||||||
|
|
||||||
|
Each node resolves *its own* `ClusterId` by finding its `NodeId`
|
||||||
|
(`_localNode.Value`, format `"host:port"`, e.g. `central-1:4053`) in the
|
||||||
|
artifact's `ClusterNode` rows, then filters both the driver specs and the
|
||||||
|
address-space composition to that cluster.
|
||||||
|
|
||||||
|
The artifact is a self-contained, consistent snapshot that already includes
|
||||||
|
`ClusterNode` + `DriverInstance` + `Namespace` + `UnsArea` (all carrying
|
||||||
|
`ClusterId`), so resolution needs **no extra DB query** and has no
|
||||||
|
seal-vs-apply inconsistency window. The coordinator stays a **single broadcast**;
|
||||||
|
every node just applies its own slice.
|
||||||
|
|
||||||
|
### Alternatives considered
|
||||||
|
|
||||||
|
- **B — control-plane per-node artifact slices.** `ConfigComposer` emits a
|
||||||
|
filtered artifact per cluster and the coordinator dispatches the right slice to
|
||||||
|
each node. Rejected: turns one broadcast into per-cluster dispatch (a large
|
||||||
|
change to the deploy/ack model), contradicts "ship global first," and still
|
||||||
|
needs the same transitive `ClusterId` resolution.
|
||||||
|
- **C — runtime DB lookup for ClusterId.** Node queries `ClusterNode` by its
|
||||||
|
address at apply time, then filters post-parse. Rejected: extra DB round-trip
|
||||||
|
per node per deploy and a seal-vs-apply inconsistency window; the artifact
|
||||||
|
already contains everything A needs.
|
||||||
|
|
||||||
|
## Components
|
||||||
|
|
||||||
|
### 1. Self-`ClusterId` resolution
|
||||||
|
|
||||||
|
New helper `DeploymentArtifact.ParseClusterScope(blob, nodeId)` returning
|
||||||
|
`(string? ClusterId, int ClusterCount)`:
|
||||||
|
- `ClusterId` = the `ClusterNode` row whose `NodeId == nodeId`, else `null`.
|
||||||
|
- `ClusterCount` = number of `ServerCluster` rows in the artifact.
|
||||||
|
|
||||||
|
Both `DriverHostActor` and `OpcUaPublishActor` call it with `_localNode.Value`.
|
||||||
|
|
||||||
|
**Fallback rule (single source of truth for every filter site):**
|
||||||
|
|
||||||
|
| Condition | Behavior |
|
||||||
|
|---|---|
|
||||||
|
| `ClusterCount ≤ 1` | **Lenient — no filter** (legacy single-cluster meshes + the entire existing test suite behave exactly as today). |
|
||||||
|
| `ClusterCount > 1` and `ClusterId` resolved | **Filter to my cluster.** |
|
||||||
|
| `ClusterCount > 1` and `ClusterId` unresolved | **Apply nothing + log error** (a node in a multi-cluster mesh with no `ClusterNode` row is misconfigured; serving everything would leak other clusters' data). |
|
||||||
|
|
||||||
|
The `ClusterCount ≤ 1` lenient branch is what protects the existing ~210 v2
|
||||||
|
tests and any single-cluster deployment from any behavior change.
|
||||||
|
|
||||||
|
### 2. Driver-spawn filter — `DriverHostActor`
|
||||||
|
|
||||||
|
`DriverInstanceSpec` already carries `ClusterId`, so in `ReconcileDrivers` (and
|
||||||
|
the restart `RestoreServedState` path) apply a one-line predicate over the parsed
|
||||||
|
specs using the fallback rule. In multi-cluster mode, specs with a `null`
|
||||||
|
`ClusterId` are excluded + logged (should never occur — `ConfigComposer` always
|
||||||
|
serializes the column).
|
||||||
|
|
||||||
|
### 3. Address-space filter — `ParseComposition` + `OpcUaPublishActor`
|
||||||
|
|
||||||
|
Add `DeploymentArtifact.ParseComposition(blob, clusterId)`. At parse time the raw
|
||||||
|
artifact entities still carry `ClusterId` / `NamespaceId` / `UnsAreaId` /
|
||||||
|
`DriverInstanceId`, so build in-cluster id sets from the artifact and filter every
|
||||||
|
projection:
|
||||||
|
|
||||||
|
| Projection | Filter predicate |
|
||||||
|
|---|---|
|
||||||
|
| `UnsAreas` | `ClusterId == mine` (direct) |
|
||||||
|
| `UnsLines` | `UnsAreaId ∈ myAreas` |
|
||||||
|
| `EquipmentNodes` | `DriverInstanceId ∈ myDrivers` |
|
||||||
|
| `DriverInstancePlans` | `DriverInstanceId ∈ myDrivers` |
|
||||||
|
| `GalaxyTags` / `EquipmentTags` | `DriverInstanceId ∈ myDrivers` |
|
||||||
|
| `ScriptedAlarmPlans` | `EquipmentId ∈ myEquipment` |
|
||||||
|
|
||||||
|
`OpcUaPublishActor.HandleRebuild` resolves `myClusterId` and calls the filtered
|
||||||
|
parse before `Phase7Planner.Compute`. `_lastApplied` becomes the filtered
|
||||||
|
composition, so the incremental diff stays correct across redeploys. The no-arg
|
||||||
|
`ParseComposition(blob)` is left untouched (legacy / single-cluster path).
|
||||||
|
|
||||||
|
### 4. Deploy ack / convergence
|
||||||
|
|
||||||
|
`ConfigPublishCoordinator` keeps broadcasting to all driver members and waiting
|
||||||
|
for all acks (in the new rig all 6 nodes are driver-role). Each node applies its
|
||||||
|
slice and acks — **including a node whose cluster has an empty slice**. The one
|
||||||
|
risk: the ack must fire even when the node's plan is empty. Implementation will
|
||||||
|
**verify the ack is unconditional** and add a small fix if it is currently gated
|
||||||
|
on a non-empty change set. No change to `DiscoverDriverNodes`.
|
||||||
|
|
||||||
|
### 5. docker-dev compose + seed rewrite
|
||||||
|
|
||||||
|
- **compose:** remove `admin-a` / `admin-b` / `driver-a` / `driver-b`; add
|
||||||
|
`central-1` / `central-2` (`OTOPCUA_ROLES=admin,driver`, seed = `central-1`,
|
||||||
|
OPC UA `4840` / `4841`, ASPNETCORE UI on `:9000`). `site-a-1/2`, `site-b-1/2`
|
||||||
|
become `driver`-only (`OTOPCUA_ROLES=driver`, `Cluster__Roles__0=driver`, seed
|
||||||
|
→ `central-1`, OPC UA `4842`–`4845`), dropping their UI / Jwt / Ldap /
|
||||||
|
DeployApiKey env + Traefik exposure. All nodes share the one ConfigDb.
|
||||||
|
- **traefik:** single `PathPrefix(/)` router → `central-1` / `central-2`
|
||||||
|
(sticky cookie); drop the two site routers + services in both
|
||||||
|
`docker-compose.yml` and `traefik-dynamic.yml`.
|
||||||
|
- **seed SQL (`seed/seed-clusters.sql`):** MAIN `ClusterNode` rows become
|
||||||
|
`central-1:4053` / `central-2:4053` (replacing `driver-a` / `driver-b`);
|
||||||
|
SITE-A / SITE-B keep their `ServerCluster` + 2 `ClusterNode` rows but **no
|
||||||
|
drivers/tags** (empty sites). Update the `Notes` columns + the file header
|
||||||
|
comments. The Galaxy namespace / driver / tags stay on MAIN (they run on the
|
||||||
|
central fused nodes).
|
||||||
|
- **compose header + comment blocks:** rewrite the topology description (single
|
||||||
|
mesh, hub-and-spoke, central-only UI).
|
||||||
|
|
||||||
|
## Data flow (after the change)
|
||||||
|
|
||||||
|
1. Operator clicks **Deploy** in the central UI (or `POST /api/deployments`).
|
||||||
|
2. `admin-operations` singleton (on a central node) → `ConfigComposer` snapshots
|
||||||
|
**all** clusters' rows into one artifact → `ConfigPublishCoordinator`
|
||||||
|
broadcasts `DispatchDeployment` to **all** driver members.
|
||||||
|
3. Each node resolves its own `ClusterId` from the artifact's `ClusterNode` rows.
|
||||||
|
4. `DriverHostActor` spawns only its cluster's `DriverInstance`s.
|
||||||
|
5. `OpcUaPublishActor` materialises only its cluster's address space.
|
||||||
|
6. Every node acks; the coordinator seals the deployment when all acks arrive.
|
||||||
|
7. Result: central `:4840`/`:4841` serve MAIN's Galaxy tree; site
|
||||||
|
`:4842`–`:4845` serve only their own (empty until configured) trees.
|
||||||
|
|
||||||
|
## Error handling
|
||||||
|
|
||||||
|
- **Misconfigured node** (multi-cluster mesh, no matching `ClusterNode` row):
|
||||||
|
applies nothing, logs an error, still acks (so the deploy converges rather than
|
||||||
|
hanging). Surfaced for the operator to add the missing `ClusterNode` row.
|
||||||
|
- **Pre-PR / single-cluster artifacts:** `ClusterCount ≤ 1` → lenient no-filter,
|
||||||
|
identical to current behavior.
|
||||||
|
- **Empty cluster slice:** node applies an empty plan and acks normally.
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
- **Unit:** `ParseClusterScope` (match / miss / count); `ParseComposition(blob,
|
||||||
|
clusterId)` (cross-cluster projections excluded; transitive resolution for
|
||||||
|
UnsLine / Equipment / Tag / ScriptedAlarm); the driver-spec filter predicate
|
||||||
|
(lenient / strict / unresolved-strict).
|
||||||
|
- **Integration:** a 2-cluster scoping test on the in-process harness — two
|
||||||
|
driver nodes assigned to different `ClusterId`s, one deploy, assert each spawns
|
||||||
|
only its cluster's drivers and materialises only its cluster's tree.
|
||||||
|
- **Backward-compat:** the existing single-cluster suites must stay green (the
|
||||||
|
`ClusterCount ≤ 1` lenient branch guarantees this).
|
||||||
|
- **Live (docker-dev rig):** bring the rig up, sign into the central UI, confirm
|
||||||
|
3 clusters listed, deploy, confirm `:4840` shows the Galaxy tree and
|
||||||
|
`:4842`/`:4844` are empty (not the merged tree).
|
||||||
|
|
||||||
|
## Classification
|
||||||
|
|
||||||
|
High-risk — touches the actor model, the Phase7 data contract, and the deploy
|
||||||
|
path. The implementation plan will be TDD'd section by section.
|
||||||
|
|
||||||
|
## Out of scope (follow-ups)
|
||||||
|
|
||||||
|
- **Per-cluster deploy** (deploy just SITE-A from the UI) — global deploy ships
|
||||||
|
first; per-cluster targeting is a later coordinator + UI enhancement.
|
||||||
|
- **Seeding demo drivers on the sites** — sites start empty; drivers are added
|
||||||
|
via the central UI.
|
||||||
Reference in New Issue
Block a user