# Per-ClusterId Scoping (hub-and-spoke single mesh) — Design

**Date:** 2026-06-07
**Status:** Approved (brainstorming complete; next step: writing-plans)
**Branch:** `feat/per-cluster-scoping`

## Goal

Let one **central** cluster's Admin UI manage and deploy to multiple
logically-separate clusters that share a single Akka mesh. The central cluster
runs 2 fused `admin,driver` nodes (the only UI + the only deploy singleton);
each site cluster runs 2 `driver`-only nodes. A single global deploy from the
central UI reaches every node, and **each node applies only the slice of the
configuration that belongs to its own `ClusterId`** — its drivers and its OPC UA
address space. Ship global deploy first; per-cluster deploy is a later follow-up.

## Why this needs runtime work

The deploy channel is **in-mesh**: AdminUI → `admin-operations` singleton →
`ConfigPublishCoordinator` → DistributedPubSub → driver nodes. DistributedPubSub
does not cross Akka mesh boundaries, so for the central UI to deploy to site
servers the site nodes **must join the central mesh**. But the runtime currently
assumes **one Akka mesh == one logical cluster**:

- `DriverHostActor.ReconcileDrivers` spawns **every** `DriverInstance` in the
  artifact with no cluster filter (`DriverHostActor.cs:367`). The `ClusterId` on
  a spec is used only to *label* health snapshots.
- `ConfigPublishCoordinator.DiscoverDriverNodes` broadcasts to **every** driver
  member of the mesh, no `ClusterId` filter (`ConfigPublishCoordinator.cs:248`).
- `ConfigComposer.SnapshotAndFlattenAsync` snapshots **all** clusters' rows into
  one flat artifact; the address space is built from the whole thing.

Consequence today: put MAIN + SITE-A + SITE-B nodes in one mesh and every node
spawns every cluster's drivers (Galaxy auto-stubs on Linux, so it *would* start)
and serves a **merged** address space of all three clusters. That is why the
existing docker-dev rig uses three isolated meshes.

This design adds the missing per-`ClusterId` scoping so a shared mesh behaves as
distinct logical clusters.

## Approach (chosen: A — node-side, parse-time filter, ClusterId from the artifact)

Each node resolves *its own* `ClusterId` by finding its `NodeId`
(`_localNode.Value`, format `"host:port"`, e.g. `central-1:4053`) in the
artifact's `ClusterNode` rows, then filters both the driver specs and the
address-space composition to that cluster.

The artifact is a self-contained, consistent snapshot that already includes
`ClusterNode` + `DriverInstance` + `Namespace` + `UnsArea` (all carrying
`ClusterId`), so resolution needs **no extra DB query** and has no
seal-vs-apply inconsistency window. The coordinator stays a **single broadcast**;
every node just applies its own slice.

### Alternatives considered

- **B — control-plane per-node artifact slices.** `ConfigComposer` emits a
  filtered artifact per cluster and the coordinator dispatches the right slice to
  each node. Rejected: turns one broadcast into per-cluster dispatch (a large
  change to the deploy/ack model), contradicts "ship global first," and still
  needs the same transitive `ClusterId` resolution.
- **C — runtime DB lookup for ClusterId.** Node queries `ClusterNode` by its
  address at apply time, then filters post-parse. Rejected: extra DB round-trip
  per node per deploy and a seal-vs-apply inconsistency window; the artifact
  already contains everything A needs.

## Components

### 1. Self-`ClusterId` resolution

New helper `DeploymentArtifact.ParseClusterScope(blob, nodeId)` returning
`(string? ClusterId, int ClusterCount)`:
- `ClusterId` = the `ClusterNode` row whose `NodeId == nodeId`, else `null`.
- `ClusterCount` = number of `ServerCluster` rows in the artifact.

Both `DriverHostActor` and `OpcUaPublishActor` call it with `_localNode.Value`.

**Fallback rule (single source of truth for every filter site):**

| Condition | Behavior |
|---|---|
| `ClusterCount ≤ 1` | **Lenient — no filter** (legacy single-cluster meshes + the entire existing test suite behave exactly as today). |
| `ClusterCount > 1` and `ClusterId` resolved | **Filter to my cluster.** |
| `ClusterCount > 1` and `ClusterId` unresolved | **Apply nothing + log error** (a node in a multi-cluster mesh with no `ClusterNode` row is misconfigured; serving everything would leak other clusters' data). |

The `ClusterCount ≤ 1` lenient branch is what protects the existing ~210 v2
tests and any single-cluster deployment from any behavior change.

### 2. Driver-spawn filter — `DriverHostActor`

`DriverInstanceSpec` already carries `ClusterId`, so in `ReconcileDrivers` (and
the restart `RestoreServedState` path) apply a one-line predicate over the parsed
specs using the fallback rule. In multi-cluster mode, specs with a `null`
`ClusterId` are excluded + logged (should never occur — `ConfigComposer` always
serializes the column).

### 3. Address-space filter — `ParseComposition` + `OpcUaPublishActor`

Add `DeploymentArtifact.ParseComposition(blob, clusterId)`. At parse time the raw
artifact entities still carry `ClusterId` / `NamespaceId` / `UnsAreaId` /
`DriverInstanceId`, so build in-cluster id sets from the artifact and filter every
projection:

| Projection | Filter predicate |
|---|---|
| `UnsAreas` | `ClusterId == mine` (direct) |
| `UnsLines` | `UnsAreaId ∈ myAreas` |
| `EquipmentNodes` | `DriverInstanceId ∈ myDrivers` |
| `DriverInstancePlans` | `DriverInstanceId ∈ myDrivers` |
| `GalaxyTags` / `EquipmentTags` | `DriverInstanceId ∈ myDrivers` |
| `ScriptedAlarmPlans` | `EquipmentId ∈ myEquipment` |

`OpcUaPublishActor.HandleRebuild` resolves `myClusterId` and calls the filtered
parse before `Phase7Planner.Compute`. `_lastApplied` becomes the filtered
composition, so the incremental diff stays correct across redeploys. The no-arg
`ParseComposition(blob)` is left untouched (legacy / single-cluster path).

### 4. Deploy ack / convergence

`ConfigPublishCoordinator` keeps broadcasting to all driver members and waiting
for all acks (in the new rig all 6 nodes are driver-role). Each node applies its
slice and acks — **including a node whose cluster has an empty slice**. The one
risk: the ack must fire even when the node's plan is empty. Implementation will
**verify the ack is unconditional** and add a small fix if it is currently gated
on a non-empty change set. No change to `DiscoverDriverNodes`.

### 5. docker-dev compose + seed rewrite

- **compose:** remove `admin-a` / `admin-b` / `driver-a` / `driver-b`; add
  `central-1` / `central-2` (`OTOPCUA_ROLES=admin,driver`, seed = `central-1`,
  OPC UA `4840` / `4841`, ASPNETCORE UI on `:9000`). `site-a-1/2`, `site-b-1/2`
  become `driver`-only (`OTOPCUA_ROLES=driver`, `Cluster__Roles__0=driver`, seed
  → `central-1`, OPC UA `4842`–`4845`), dropping their UI / Jwt / Ldap /
  DeployApiKey env + Traefik exposure. All nodes share the one ConfigDb.
- **traefik:** single `PathPrefix(/)` router → `central-1` / `central-2`
  (sticky cookie); drop the two site routers + services in both
  `docker-compose.yml` and `traefik-dynamic.yml`.
- **seed SQL (`seed/seed-clusters.sql`):** MAIN `ClusterNode` rows become
  `central-1:4053` / `central-2:4053` (replacing `driver-a` / `driver-b`);
  SITE-A / SITE-B keep their `ServerCluster` + 2 `ClusterNode` rows but **no
  drivers/tags** (empty sites). Update the `Notes` columns + the file header
  comments. The Galaxy namespace / driver / tags stay on MAIN (they run on the
  central fused nodes).
- **compose header + comment blocks:** rewrite the topology description (single
  mesh, hub-and-spoke, central-only UI).

## Data flow (after the change)

1. Operator clicks **Deploy** in the central UI (or `POST /api/deployments`).
2. `admin-operations` singleton (on a central node) → `ConfigComposer` snapshots
   **all** clusters' rows into one artifact → `ConfigPublishCoordinator`
   broadcasts `DispatchDeployment` to **all** driver members.
3. Each node resolves its own `ClusterId` from the artifact's `ClusterNode` rows.
4. `DriverHostActor` spawns only its cluster's `DriverInstance`s.
5. `OpcUaPublishActor` materialises only its cluster's address space.
6. Every node acks; the coordinator seals the deployment when all acks arrive.
7. Result: central `:4840`/`:4841` serve MAIN's Galaxy tree; site
   `:4842`–`:4845` serve only their own (empty until configured) trees.

## Error handling

- **Misconfigured node** (multi-cluster mesh, no matching `ClusterNode` row):
  applies nothing, logs an error, still acks (so the deploy converges rather than
  hanging). Surfaced for the operator to add the missing `ClusterNode` row.
- **Pre-PR / single-cluster artifacts:** `ClusterCount ≤ 1` → lenient no-filter,
  identical to current behavior.
- **Empty cluster slice:** node applies an empty plan and acks normally.

## Testing

- **Unit:** `ParseClusterScope` (match / miss / count); `ParseComposition(blob,
  clusterId)` (cross-cluster projections excluded; transitive resolution for
  UnsLine / Equipment / Tag / ScriptedAlarm); the driver-spec filter predicate
  (lenient / strict / unresolved-strict).
- **Integration:** a 2-cluster scoping test on the in-process harness — two
  driver nodes assigned to different `ClusterId`s, one deploy, assert each spawns
  only its cluster's drivers and materialises only its cluster's tree.
- **Backward-compat:** the existing single-cluster suites must stay green (the
  `ClusterCount ≤ 1` lenient branch guarantees this).
- **Live (docker-dev rig):** bring the rig up, sign into the central UI, confirm
  3 clusters listed, deploy, confirm `:4840` shows the Galaxy tree and
  `:4842`/`:4844` are empty (not the merged tree).

## Classification

High-risk — touches the actor model, the Phase7 data contract, and the deploy
path. The implementation plan will be TDD'd section by section.

## Out of scope (follow-ups)

- **Per-cluster deploy** (deploy just SITE-A from the UI) — global deploy ships
  first; per-cluster targeting is a later coordinator + UI enhancement.
- **Seeding demo drivers on the sites** — sites start empty; drivers are added
  via the central UI.