5db01053e8
v2-ci / unit-tests (push) Waiting to run
Bounded, docs-only part of #717. The standard non-transparent metadata itself is NOT implemented here: the owner-selected contract is Warm with an explicit degraded standby (and None standalone), today's Secondary serves Good data at ServiceLevel 240, and the design/spec/plan for that standby behaviour is still pending on the issue - advertising Warm now would misdescribe the server. - docs/Redundancy.md: the "deferred pending an SDK object-type upgrade" claim was wrong; the pinned SDK 1.5.378.106 can build the graph (CreateMainNodeManagerFactory + AddBehaviourToPredefinedNode). Records what is and is not advertised (i=3709 SDK default, no i=11314) and the blocker. - .claude/rules/cluster-topology.md: said each node exposes RedundancySupport and ServerUriArray; it does not. - docs/OpcUaServer.md: same false claim about the SDK exposing them. Interop gates A4/A5b stay unmet; no product code changed. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
426 lines
43 KiB
Markdown
426 lines
43 KiB
Markdown
---
|
||
paths:
|
||
- "src/Core/ZB.MOM.WW.OtOpcUa.Cluster/**"
|
||
- "src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/**"
|
||
- "src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Redundancy/**"
|
||
- "src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Telemetry/**"
|
||
- "src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/**"
|
||
- "src/Server/ZB.MOM.WW.OtOpcUa.Host/**"
|
||
- "docker-dev/**"
|
||
- "docs/Redundancy.md"
|
||
- "docs/Configuration.md"
|
||
- "docs/Telemetry.md"
|
||
---
|
||
|
||
# Cluster topology: redundancy, role authority, per-cluster meshes, MeshTransport, ConfigSource/ConfigServe, telemetry transport, LocalDb, DB-less driver nodes
|
||
|
||
> Moved verbatim from the root `CLAUDE.md` (2026-10-13) so it loads only when working in this area.
|
||
> Root `CLAUDE.md` § "Subsystem guides" indexes it. Edit it here, not in the root file.
|
||
|
||
## Redundancy
|
||
|
||
The server supports non-transparent warm/hot redundancy via the `Redundancy` section in `appsettings.json`. Two instances share the same Galaxy DB and the same mxaccessgw (under distinct `MxAccess.ClientName` values) but have unique `ApplicationUri` values. Each exposes a dynamic `ServiceLevel` based on role and runtime health and publishes self + peer URIs in `Server.ServerArray` (`i=2254`); the primary advertises a higher ServiceLevel than the secondary. ⚠️ It does **not** yet set `ServerRedundancy.RedundancySupport` (`i=3709`, reads the SDK default) or expose `ServerUriArray` (`i=11314`) — #717: feasible on the pinned SDK without an upgrade, but blocked on the owner-selected "Warm with a degraded standby" contract (today's standby serves Good data at 240), so do not "just set it to Warm". See `docs/Redundancy.md` for the full guide.
|
||
|
||
**Two corrections landed 2026-07-21 — both were total-outage or wrong-node defects, and both are worth knowing before touching this area.**
|
||
|
||
- **Downing is `auto-down`, not `keep-oldest`.** `Cluster:SplitBrainResolverStrategy` (default `auto-down`) selects the provider via `ServiceCollectionExtensions.BuildDowningHocon`; an unknown value fails host start. A two-node pair running the previous `keep-oldest` + `down-if-alone` **could not survive a crash of the oldest node**: Akka's `KeepOldest.OldestDecision` only lets `down-if-alone` rescue a side holding ≥ 2 members, so the 1-vs-1 survivor downs *itself* and exits. `down-if-alone` is a 3+-node feature. The accepted trade for `auto-down` is dual-active during a genuine network partition. **Its live gate is CLOSED** — Phase 7 drill D4 (2026-07-24) ran the 1-vs-1 crash-the-oldest survival drill on the three-mesh docker-dev rig and every survivor stayed Up, never self-downing (`c50ebcf7`; `docs/plans/2026-07-24-mesh-phase7-failover-drills.md`).
|
||
- **The Primary is the oldest Up `driver` member, not the role leader.** `RedundancyStateActor` previously used `ClusterState.RoleLeader("driver")` (lowest *address*), while `ClusterSingletonManager` places singletons on the *oldest*. They agree only on a freshly-formed cluster and diverge after any restart, so the Primary-gated data plane (writes, alarm acks, alerts emit, alarm-history drain) could enable on a node not hosting the singletons. `NodeRedundancyState.IsRoleLeaderForDriver` was renamed **`IsDriverPrimary`**.
|
||
|
||
**A failed OPC UA endpoint steps the Primary down (lmxopcua#762).** `IOpcUaEndpointState` now lives in `Commons.OpcUa` so the Runtime can read it: a `Failed` SDK start feeds `EndpointListening=false` into `ServiceLevelCalculator` (→ 0, beats the +10 Primary bonus), and `OpcUaEndpointStepDownActor` makes a failed **Primary** gracefully `Leave` its mesh (the manual-failover mechanism) once another Up driver member has reported its own endpoint listening on the `opcua-endpoint-state` DPS topic. Do **not** "fix" this by making the election skip the node — the Primary must stay where the singletons are. The peer-report guard is what stops a both-endpoints-failed pair from bouncing the Primary (and restarting nodes) forever. See `docs/Redundancy.md` §"Endpoint start failure: Primary step-down".
|
||
|
||
**A third bootstrap gap closed 2026-07-22 — downing was never the whole story, and the fix changed twice.** Even under `auto-down`, a node cold-starting while its peer was dead **never came Up**: Akka runs `FirstSeedNodeProcess` — the only bootstrap path that can form a *new* cluster when no peer answers `InitJoin` — exclusively when `seed-nodes[0]` is the node’s own address; every other node runs `JoinSeedNodeProcess` and retries `InitJoin` forever. **`Cluster:SeedNodes` is therefore ORDERED: every node that is one of its own seeds lists ITSELF first, partner second** (docker-dev `central-2` was swapped; site nodes list only `central-1` and are legitimately not seeds). `AkkaClusterOptionsValidator` (`AddValidatedOptions`/`ValidateOnStart` in `AddOtOpcUaCluster`) enforces it at boot — **conditionally** (self must be entry 0 only *if* self is in the list, or every site node would refuse to start), matching on `PublicHostname`+`Port`, never the `0.0.0.0` bind address. The earlier fix — a `Cluster:SelfFormAfter` timer calling `Cluster.Join(SelfAddress)` — is **deleted**: it sat outside Akka’s join handshake, so it could not tell "no seed answered" from "a seed answered and the join is in flight", and it islanded a manually-failed-over node live before a TCP reachability guard patched that one shape. Pinned by `SelfFirstSeedBootstrapTests` (real in-process clusters, incl. a falsifiability control proving peer-first ordering never forms) + `AkkaClusterOptionsValidatorTests`. See `docs/Redundancy.md` §"Bootstrap: self-first seed ordering".
|
||
|
||
**Simultaneous cold-start splits — the bootstrap guard closes it (opt-in, `Cluster:BootstrapGuard:Enabled`, default OFF).** Self-first on BOTH nodes means when both cold-start at the same instant, each runs `FirstSeedNodeProcess`, times out, and forms its OWN 1-node cluster → two Primaries in one pair (the Phase 6 live gate reproduced this on docker; two separate clusters do NOT auto-merge). The guard prevents it without losing cold-start-alone: the lexicographically **lower** `host:port` node is the preferred founder (self-first, forms immediately); the **higher** node TCP-probes the partner's Akka port up to `PartnerProbeSeconds` (25s) — **reachable ⇒ peer-first (join the founder)**, **unreachable ⇒ self-first (partner dead, form alone)**. The order is chosen BEFORE the single `JoinSeedNodes`, from an explicit reachability signal — it never re-forms mid-handshake (the `SelfFormAfter` failure mode). When on, Akka gets NO config seeds (`BuildClusterOptions` empties `ClusterOptions.SeedNodes`) and `ClusterBootstrapCoordinator` drives the join. Residual trade (accepted, operator-visible via a warning + restart-recovers): the higher node hangs if the founder dies in the probe→join window. docker-dev: **site-a = guard demo** (guard on, serialization removed); **site-b = A/B control** (guard off, `depends_on: service_healthy` serialization). The guard is the production-faithful fix (compose `depends_on` doesn't exist on real VMs). Pinned by `ClusterBootstrapGuardTests` + `ClusterBootstrapCoordinatorTests` (real 2-node clusters, incl. the higher-node-cold-start-alone case). See `docs/Redundancy.md` §"Bootstrap guard".
|
||
|
||
Assert the **effective** cluster config off a running `ActorSystem` (as `SplitBrainResolverActivationTests` does), never the typed options or the akka.conf text — several HOCON fragments compete and the losing one still reads correctly in the file.
|
||
|
||
**RESOLVED by per-cluster mesh Phase 6 (2026-07-24) — no longer per-Akka-cluster.** The "one Primary across the whole Akka mesh" limitation described above is gone; see the Phase 6 paragraph immediately below.
|
||
|
||
**Per-cluster mesh Phase 6 (2026-07-24) — the fleet is now three independent 2-node meshes, not one.** The single fleet-wide Akka mesh was split into one mesh per application `Cluster`: the central pair (`admin,driver,cluster-MAIN`), site-a (`driver,cluster-SITE-A`), site-b (`driver,cluster-SITE-B`) — same `ActorSystem` name, separated purely by per-pair self-first seeds (each node's `Cluster:SeedNodes` lists itself first, its own pair partner second, never a node from another cluster's pair). Consequences:
|
||
|
||
- **The redundancy singleton is cluster-scoped and spawned per driver node.** `RedundancyStateActor` is now a `cluster-{ClusterId}`-scoped singleton (falls back to plain `driver` for a legacy node), so each pair elects and publishes its **own** Primary on its own mesh's DPS. **Two or more Primaries exist fleet-wide, one per pair — by design, not a bug.** ⚠️ **Wiring landmine (fixed `3a4ed8dd`, caught by the live gate):** the singleton's cluster-role *scope* is computed at `AddAkka`-configurator time, so it must come from `AkkaClusterOptions` (config) — **never** by resolving `IClusterRoleInfo` from the SP there. `ClusterRoleInfo` depends on the `ActorSystem`, and that lambda runs *while the ActorSystem is being built*, so resolving it recurses into ActorSystem construction and **stack-overflows every node at boot** (compiles clean; the rehome tests passed a hand-built fake straight into the extension, mocking around the composition-root line). Keep `WithOtOpcUaClusterRedundancySingleton`/`BuildClusterRedundancySingletonOptions` taking `AkkaClusterOptions`.
|
||
- `ClusterNodeAddressReconciler` is **own-cluster-scoped** — an admin node can only reconcile `ClusterNode` rows in its own cluster; it has no gossip visibility into another mesh.
|
||
- `CentralCommunicationActor` dials each node directly over gRPC — one `NodeCommandService.Execute` call per enabled `ClusterNode` row carrying a `GrpcPort` — rather than fanning through a client/receptionist object. **Superseded by the [ClusterClient→gRPC migration](docs/plans/2026-08-06-clusterclient-to-grpc-design.md):** the Akka `ClusterClient` this bullet originally described (one per application `Cluster`, closing the `TODO(Phase 6)` Phase 2 left behind) is deleted; the per-node gRPC dial is its replacement, not an evolution of it.
|
||
- **`SplitTopologyTransportValidator` fails host start** on a node carrying a `cluster-{ClusterId}` role unless it also runs the split-safe transports: `MeshTransport:Mode=Grpc`, `Telemetry:Mode=Grpc` (driver), `TelemetryDial:Mode=Grpc` (admin). A DPS transport on a cluster-role node would try to gossip across a partition that no longer exists.
|
||
- **The compiled transport-mode defaults were deliberately NOT flipped** — `MeshTransport:Mode`/`Telemetry:Mode`/`TelemetryDial:Mode` all still default to `Dps`; a blast-radius assessment showed flipping the default breaks every integration test and appsettings profile that leaves the key unset. The validator + explicit per-deployment config is the guardrail, not a default flip.
|
||
- **Secrets replication is pair-local** — each pair replicates its own Akka DPS secrets within itself; there is no fleet-wide/cross-cluster sync after the split (sites have no SQL, so no SQL-hub, consistent with Phase 4).
|
||
|
||
See `docs/Redundancy.md` §"Per-cluster meshes (Phase 6)", `docs/Configuration.md` §"Per-cluster mesh split (Phase 6)", and `docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md`.
|
||
|
||
## Role authority (`EffectiveRoles`)
|
||
|
||
**A node's roles are resolved in exactly one place** (archreview R3-04, closing 01/S-12 = 03/C5 and
|
||
01/C-9): `EffectiveRoles.Resolve` (`ZB.MOM.WW.OtOpcUa.Cluster`), with one precedence —
|
||
`Cluster:Roles` when non-empty, else `OTOPCUA_ROLES`, else empty — and two-layer normalization: a
|
||
plain role (`admin`/`driver`/`dev`) is trimmed and lower-cased, but a cluster role's ClusterId suffix
|
||
(`cluster-SITE-A`) is kept verbatim, because lower-casing it would rename the Akka role of a running
|
||
deployment. An unknown role from **either** source fails host start. `Program.cs` resolves one
|
||
`EffectiveRoles` instance at the top of the composition root and registers it as a singleton; that
|
||
SAME instance drives both the conditional host wiring (`hasAdmin`/`hasDriver`) and
|
||
`AkkaClusterOptions.Roles` (what Akka gossips) — there is no second derivation for the two to
|
||
disagree about, which is why no runtime "do the views agree" assert exists. Before this, `Program.cs`
|
||
read `OTOPCUA_ROLES` only, so a node configured solely through `Cluster:Roles` gossiped the right
|
||
Akka roles while `Program.cs`'s own wiring mounted none of them — the inverse of the pre-#532 field
|
||
incident (`docs/deployments/wonder-app-vd03-makino-z-34184.md`).
|
||
|
||
- **A role overlay may not itself set `Cluster:Roles`.** Roles are resolved *before* the per-role
|
||
appsettings overlay loads (the overlay's filename depends on the answer), so a later layer changing
|
||
`Cluster:Roles` would leave the bound options disagreeing with the registered instance.
|
||
`AkkaClusterOptionsValidator` fails host start if this ever happens, naming both views — a no-op
|
||
today since none of the four shipped overlays carries a `Cluster` section.
|
||
- **Both sources set to different normalized role sets logs a Warning** at boot (`Cluster:Roles`
|
||
still wins outright) — the boot banner also names which source won (`source=Configured` /
|
||
`Environment` / `None`).
|
||
- **`RoleParser.GossipedRolesHave`** (case-insensitive) is defence in depth at the handful of
|
||
production sites that read LIVE GOSSIP directly (`Member.Roles`) rather than through
|
||
`EffectiveRoles` — a peer running an older, pre-R3-04 build can still gossip a miscased plain role
|
||
during a rolling upgrade. ⚠️ **Accepted residual (#620), not closed by R3-04:**
|
||
`RedundancyStateActor`'s election delegates to `ZB.MOM.WW.Health.Akka`'s `ClusterActiveNode.OldestUpMember`,
|
||
which matches via Akka's OWN ordinal `Member.HasRole`. Deliberately **not** patched — neither in the
|
||
library nor with a local case-insensitive pre-filter, which would split the Primary election from
|
||
`ClusterSingletonManager` placement (also ordinal). It only bites a mixed fixed/pre-fix rolling-upgrade
|
||
window; finish the upgrade. Full rationale: `docs/Redundancy.md` ("Accepted residual (#620)").
|
||
- **`Program.cs` wraps the composition root** (#618): a start-up failure logs Fatal, writes `FATAL:` to
|
||
stderr (the bootstrap logger is still silent before `AddZbSerilog`), flushes, and `Environment.Exit(1)`s
|
||
— leftover Akka/Kestrel threads would otherwise keep the process alive. `HostAbortedException`
|
||
(design-time tooling) passes through untouched.
|
||
- **`RoleAuthorityGuardTests`** (`tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/`) is the enforcement:
|
||
a source scan failing on any production file outside `EffectiveRoles.cs`/`RoleParser.cs` that reads
|
||
`OTOPCUA_ROLES`, `Cluster:Roles` as code, or declares a bare `"admin"`/`"driver"`/`"dev"` literal —
|
||
with a small, reasoned allow-list for the handful of genuinely unrelated domain uses of those same
|
||
English words (a raw-tree node-kind label, a Sql driver's own JSON marker, an LDAP-derived AdminUI
|
||
RBAC role).
|
||
|
||
See `docs/Configuration.md` §`Roles`, `docs/ServiceHosting.md` §"Role gating", and
|
||
`archreview/plans/R3-04-one-role-authority-plan.md`.
|
||
|
||
## Mesh command transport (`MeshTransport`)
|
||
|
||
**The Akka `ClusterClient` transport described in this section's earlier history is DELETED**
|
||
(branch `feat/grpc-phase3-deletion`, [ClusterClient→gRPC migration](docs/plans/2026-08-06-clusterclient-to-grpc-design.md),
|
||
Phase 3) — code, the `akka.cluster.client` HOCON block, the receptionist extension,
|
||
`MeshTransport:CentralContactPoints`, and the `ClusterClient` mode value itself, which now **fails
|
||
validation loudly** at boot (`MeshTransportOptionsValidator` names the key and the allowed values)
|
||
rather than silently falling back to `Dps`. What remains is exactly two transports, selected by
|
||
`MeshTransport:Mode`:
|
||
|
||
- **`Dps`** (default) — DistributedPubSub, the path this repo has always used. **Pair-local only**:
|
||
it works only when central and the node share a gossip ring, which after Phase 6 means within one
|
||
2-node mesh.
|
||
- **`Grpc`** — unary gRPC calls on the `mesh_transport.v1` contract. **Required** on any node
|
||
carrying a `cluster-{ClusterId}` role; `SplitTopologyTransportValidator` refuses to boot such a
|
||
node on anything else.
|
||
|
||
Central→node commands (`deployments`, `driver-control`, `alarm-commands`) ride `NodeCommandService.Execute`,
|
||
served on each node's driver-role telemetry h2c listener (`Telemetry:GrpcListenPort`, the docker-dev
|
||
rig's `:4056` — it shares that listener with `TelemetryStreamGrpcService` rather than claiming a port
|
||
of its own). `CentralCommunicationActor` dials it **directly, one call per node** — the target set is
|
||
every enabled, non-maintenance `ClusterNode` row carrying a `GrpcPort`, dialled at `Host`+`GrpcPort`
|
||
and refreshed every `MeshTransport:ContactRefreshSeconds`. There is no client/receptionist object and
|
||
no rotation: each command goes to the specific node dialled, which is what makes per-node delivery
|
||
provable — the old `ClusterClient.SendToAll`/`Send` distinction this section used to explain no longer
|
||
applies to either direction.
|
||
|
||
Node→central deploy acks ride `CentralCommandService.ReportApply`, served on central's admin-role
|
||
config-serve h2c listener (`ConfigServe:GrpcListenPort`, the rig's `:4055` — again sharing rather than
|
||
claiming a new port). `GrpcApplyAckSender` walks `MeshTransport:CentralGrpcEndpoints` in order and
|
||
stops at the first success.
|
||
|
||
Auth is a **per-cluster PSK bearer**, both directions: `MeshTransport:GrpcPsk` (node-side, one key per
|
||
application `Cluster`) and `MeshTransport:ClusterPsks` (central-side map, consulted before the Secrets
|
||
store). `MeshCommandAuthInterceptor` (node) and `CentralCommandAuthInterceptor` (central) fail closed
|
||
on every unresolvable case — constant-time compare, no key material logged. Production keys are
|
||
`${secret:OT-MESH-PSK-<clusterId>}`; the docker-dev rig supplies three literal dev keys via env
|
||
(`dev-mesh-psk-MAIN`/`-SITE-A`/`-SITE-B`).
|
||
|
||
Things that carried over unchanged from the `ClusterClient` era, because they were never
|
||
`ClusterClient`-specific to begin with:
|
||
|
||
- **Fire-and-forget, no retry, on both transports.** A failed gRPC dispatch is logged (naming the node
|
||
and the gRPC status) and dropped in the dispatch second. No queue, no ordering guarantee across
|
||
nodes — the deploy coordinator's apply deadline and a node's boot-time replay are the only recovery.
|
||
The old `buffer-size = 0` decision this section used to describe is gone with the HOCON block that
|
||
set it; the gRPC transport has nothing to buffer in the first place — a dial either lands or doesn't.
|
||
- **Inbound commands re-emit on the node's `EventStream`, not back onto DPS.** DPS is mesh-wide, so a
|
||
DPS re-publish would deliver N copies to every subscriber. Node-side subscribers (`DriverHostActor`,
|
||
`ScriptedAlarmHostActor`) accept a command from either transport identically.
|
||
- **Do not over-claim what fire-and-forget buys.** The Phase 2 live gate observed a node applying a
|
||
deployment **44 s after it was sealed `TimedOut`** anyway, via `DriverHostActor.Bootstrap()`
|
||
replaying the orphan `Applying` row on restart. That backstop is unaffected by the ClusterClient
|
||
deletion — it was always about the deploy path, not the transport.
|
||
|
||
`redundancy-state` stays on DPS in both modes (bidirectional, and built from `Cluster.State`, which
|
||
makes it genuinely mesh-bound). See `docs/Configuration.md` §`MeshTransport`, `docs/Redundancy.md`
|
||
§"Command transport", and the migration's [design doc](docs/plans/2026-08-06-clusterclient-to-grpc-design.md) /
|
||
[plan](docs/plans/2026-08-06-clusterclient-to-grpc.md) / [tracking doc](docs/plans/2026-08-06-clusterclient-to-grpc-tracking.md) /
|
||
[live gate](docs/plans/2026-08-06-clusterclient-to-grpc-live-gate.md) for the phase-by-phase history —
|
||
those are historical records, not updated further by this sweep. `docs/plans/2026-07-22-mesh-phase2-clusterclient-transport.md`
|
||
is the original Phase 2 design doc and is likewise left as a historical record of the transport this
|
||
section now describes as deleted.
|
||
|
||
**Phase 1 landed 2026-07-22 and changed a deploy-path behaviour.** `ConfigPublishCoordinator` sources its expected-ack set from **enabled `ClusterNode` rows**, not from `Akka.Cluster.State.Members` filtered by the `driver` role — central must be able to name a deployment's nodes without sharing a gossip ring with them. Consequences worth knowing before touching deploys:
|
||
|
||
- **A configured node that is switched off now FAILS the deployment** at the apply deadline instead of letting it seal green without that node. The maintenance hatch is the **new `ClusterNode.MaintenanceMode` column**, NOT `Enabled = 0`: the live gate showed `Enabled = 0` is rejected outright by `DraftValidator.ValidateClusterTopology` (`ClusterEnabledNodeCountMismatch` — enabled count must equal `ServerCluster.NodeCount`), so on a 2-node pair — every cluster in the target topology — it can never be the hatch. `Enabled` = part of the declared topology; `MaintenanceMode` = do not expect it now. The deadline log names the silent nodes and points at the right flag.
|
||
- **Every `ClusterNode` row is now *defined* to be a driver node.** The DB has no per-node role column and deliberately did not gain one (it would drift from `Cluster:Roles`). Giving an admin-only node a `ClusterNode` row makes every deploy time out.
|
||
- **There is no such thing as a cluster-scoped deployment.** `Deployment` has no `ClusterId`, `ConfigComposer` always snapshots the whole DB, and `DeploymentArtifact.ResolveClusterScope` is *node-side* self-scoping of a fleet-wide artifact. The phase-1 plan asked for cluster-scope filtering of the ack set; it was dropped as describing a feature that does not exist.
|
||
- `ClusterNode` gained `AkkaPort` (NOT NULL, 4053) + `GrpcPort` (nullable) as **central's dial targets** for Phases 2/5. They duplicate the node's own `Cluster:Port` / `Cluster:PublicHostname` with no schema-level enforcement, so `ClusterNodeAddressReconcilerActor` (admin singleton) logs an Error on drift. **Phase 2 must revisit it** — once the meshes split, an admin node cannot see site members and every site row would report `EnabledRowNotInCluster` forever. **Resolved, and not the way this bullet originally expected:** the [ClusterClient→gRPC migration](docs/plans/2026-08-06-clusterclient-to-grpc-design.md) deleted the Akka `ClusterClient` transport that would have dialled `Host`+`AkkaPort`; the Phase 2 command dial now uses `Host`+`GrpcPort`, the same column Phase 5 telemetry already dialled — `AkkaPort` is Akka remoting only again.
|
||
|
||
## Config source (`ConfigSource` / `ConfigServe`)
|
||
|
||
Per-cluster mesh **Phase 3** (config fetch-and-cache) added a second way a **driver** node obtains its
|
||
deployed configuration, selected by `ConfigSource:Mode`: `Direct` (read `Deployment.ArtifactBlob` from
|
||
central SQL — **the default**, byte-for-byte today's behaviour) or `FetchAndCache` (fetch the artifact
|
||
bytes from central over a dedicated-h2c **gRPC stream**, verify `SHA-256 == RevisionHash`, cache in
|
||
LocalDb, and read **only** the cache — no central-SQL config read). Central (admin role) serves via
|
||
`DeploymentArtifactService` behind `ConfigServeAuthInterceptor` on `ConfigServe:GrpcListenPort`, gated by
|
||
a **shared node bearer key** (`ConfigServe:ApiKey == ConfigSource:ApiKey`, `FixedTimeEquals`,
|
||
fail-closed). Things worth knowing before touching it:
|
||
|
||
- **The serve side is wired in both modes** (mapped whenever `ConfigServe:GrpcListenPort > 0` on an
|
||
admin node), so flipping a node to `FetchAndCache` is a config change, not a redeploy — the same
|
||
dark-switch discipline as [MeshTransport](#mesh-command-transport-meshtransport).
|
||
- **`RevisionHash == SHA-256(artifact bytes)` is load-bearing.** `ConfigComposer` computes the revision
|
||
hash as the lowercase-hex SHA-256 of the exact blob, so the node verifies a fetch against the
|
||
`RevisionHash` the dispatch already carries — **no hash rides in the proto, and `DispatchDeployment`
|
||
stays payload-free.** A future change to how the revision hash is derived silently breaks every fetch.
|
||
**The blob is gzip-compressed since 2026-08-15, and the hash covers the COMPRESSED bytes** —
|
||
`ConfigComposer.Pack` gzips (`CompressionLevel.Fastest`) immediately after `SerializeToUtf8Bytes` and
|
||
hashes the result, so the invariant holds unchanged and nothing between the composer and
|
||
`DeploymentArtifact.Open` (SQL `ArtifactBlob`, the LocalDb cache chunks, the gRPC fetch stream) ever
|
||
looks inside. `Open` sniffs the gzip magic `1F 8B` — valid JSON can never start with `0x1F` — so
|
||
pre-upgrade uncompressed rows keep parsing; inflation runs under a 512 MiB guard and every
|
||
decompression failure surfaces as `JsonException`, the same family the malformed-JSON paths already
|
||
handle. Two operational consequences in `docs/Configuration.md` §`ConfigSource`: the first deploy after
|
||
upgrade **re-hashes every revision** (a one-time full re-dispatch, not a no-op), and **nodes must be
|
||
upgraded BEFORE central starts emitting compressed blobs.**
|
||
- **A null fetch is "no answer," never an empty config (#485).** All endpoints down / `NotFound` / SHA
|
||
mismatch / zero-length ⇒ the fetcher returns null ⇒ the apply FAILS, the revision does not advance,
|
||
and the node keeps serving last-known-good. It never tears down to an empty address space. At boot a
|
||
`FetchAndCache` node restores served state from the LocalDb pointer with **no** central-SQL read; an
|
||
empty/unreadable cache lands Steady-with-no-revision (the first dispatch fetches), **never Stale**.
|
||
**Malformed non-empty bytes are "no answer" too** — on cache boot, the `FetchAndCache` boot reconcile and
|
||
the subscription push alike (the lenient swallow leg is closed). A cache boot from malformed bytes does
|
||
**not** adopt the revision (`Direct` ⇒ Stale; `FetchAndCache` ⇒ no revision, the re-dispatch fetches).
|
||
- **Both pair nodes fetch independently** — no Primary gating of the fetch (same shape as both reading
|
||
SQL today; `StoreAsync` is idempotent and the cache replicates).
|
||
- **The dedicated h2c listener coexists with the LocalDb-sync listener.** A fused admin+driver node with
|
||
both ports set re-binds its existing HTTP surface exactly **once** and adds both dedicated HTTP/2-only
|
||
ports — getting this wrong either double-binds (crash) or silently unbinds the AdminUI (the Kestrel
|
||
`Listen*`-discards-`ASPNETCORE_URLS` trap the LocalDb block already warned about).
|
||
- **Phase boundary: config *reads* only.** A `FetchAndCache` node still writes its `NodeDeploymentState`
|
||
ack row to central SQL; `DbHealthProbe` and `EfAlarmConditionStateStore` still touch SQL. Cutting those
|
||
is Phase 4 — resist folding it in, or the "no SQL" scope becomes unprovable-in-parts.
|
||
|
||
On the docker-dev rig the central pair carries `ConfigServe` (`:4055`, committed dev key) and stays
|
||
hardcoded `Direct` via its admin role; the four site nodes carry `ConfigSource` (endpoints + matching
|
||
key) hardcoded `FetchAndCache` — mandatory for driver-only nodes since Phase 4, not a flip. There is
|
||
no `OTOPCUA_CONFIG_MODE` env var in the compose file. See
|
||
`docs/Configuration.md` §`ConfigSource` / `ConfigServe`, `docs/Redundancy.md` §"Command transport", and
|
||
`docs/plans/2026-07-22-mesh-phase3-config-fetch-and-cache.md`.
|
||
|
||
## Live telemetry transport (`Telemetry` / `TelemetryDial`)
|
||
|
||
Per-cluster mesh **Phase 5** (merged `35552a96`; live gate PASSED `f134935f`) added a second
|
||
transport for the four live-observability channels the AdminUI's `/alerts`, `/script-log`, and
|
||
`/hosts` panels feed on, selected by `Telemetry:Mode` (node/serve side) and `TelemetryDial:Mode`
|
||
(central/dial side): `Dps` — the four existing DPS SignalR bridges, **the default**, today's
|
||
behaviour — or `Grpc`, one gRPC server-streaming contract carrying all four channels as `oneof`
|
||
event kinds. **The direction is inverted from `ConfigSource`/`MeshTransport`: the driver node hosts
|
||
the gRPC server (dedicated Kestrel h2c port), and central is the client that dials in** — mirroring
|
||
ScadaBridge's `SiteStreamService` shape, and required so telemetry survives once the meshes split
|
||
(Phase 6) and central no longer shares a gossip ring with a site node. Things worth knowing before
|
||
touching it:
|
||
|
||
- **Migrated (4 channels):** `alerts` (`AlarmTransitionEvent`), `script-logs` (`ScriptLogEntry`),
|
||
`driver-health` (`DriverHealthChanged`), `driver-resilience-status`
|
||
(`DriverResilienceStatusChanged`) — feeding the exact same central sinks
|
||
(`IInProcessBroadcaster<...>`, `IDriverStatusSnapshotStore`, `IDriverResilienceStatusStore`) in
|
||
either mode.
|
||
- **Deferred, with reasons (do not read "seven observability topics" from the program sketch as the
|
||
delivered scope):** `redundancy-state` stays on DPS in every mode (pair-local, built from
|
||
`Cluster.State`, drives ServiceLevel + the Primary gate — genuinely mesh-bound by design, not an
|
||
observability panel); `deployment-acks` already rides the Phase 2 gRPC command transport
|
||
(`MeshTransport:Mode=Grpc`, `CentralCommandService.ReportApply`) as a command-plane reply, not
|
||
telemetry. A third, `fleet-status`, was **DELETED** rather than deferred (2026-08-06, Gitea #542):
|
||
`FleetStatusBroadcaster` → DPS → SignalR bridge → `FleetStatusHub` had **zero clients end to end**
|
||
(no browser JS, no `HubConnection` — `/fleet` polls `NodeDeploymentState` in the Config DB), its
|
||
`DriverHostStatusHeartbeat` input had no production sender, and Phase 6 left its "fleet" as just
|
||
central's own pair. Fleet visibility is `/fleet` + the four per-node telemetry dials above.
|
||
- **The node ALWAYS hosts and ALWAYS double-publishes, in both modes.** Every publish seam emits
|
||
into a node-local `ITelemetryLocalHub` **and** still publishes DPS, unconditionally; the gRPC
|
||
server binds whenever `Telemetry:GrpcListenPort > 0`, independent of `Telemetry:Mode`. Only
|
||
`TelemetryDial:Mode` on central actually switches anything: which of the two always-available
|
||
sources (DPS bridges vs. the `TelemetryDialSupervisor` dialer) feeds the sinks. This is what
|
||
makes the flip a config change, not a coordinated redeploy — same dark-switch discipline as
|
||
[MeshTransport](#mesh-command-transport-meshtransport) and [ConfigSource](#config-source-configsource--configserve).
|
||
- **Authenticated from day one, fail-closed** — a shared node bearer key (`Telemetry:ApiKey` ==
|
||
`TelemetryDial:ApiKey`), gated by `TelemetryStreamAuthInterceptor`
|
||
(`CryptographicOperations.FixedTimeEquals`, `PermissionDenied` on mismatch, an unset key rejects
|
||
every call). This **supersedes** design doc §6.3's "match ScadaBridge's unauthenticated posture
|
||
for now" — ScadaBridge itself has since closed that gap with this identical interceptor pattern.
|
||
- **Reconnect story:** central runs one reconnecting dialer per driver node (discovered from
|
||
`ClusterNode.Host`/`GrpcPort`, refreshed every `TelemetryDial:ContactRefreshSeconds`), immediate
|
||
first retry then ~5s fixed backoff, generation-stamped so a superseded stream's late error/event
|
||
is ignored, and it never permanently gives up (observability, not data plane). `TelemetryLocalHub.Evict` drops a removed driver's snapshots (`StopChild`, via `plan.ToStop`; central-side
|
||
stores are not evicted). The node-local hub
|
||
replays last-value snapshots for `driver-health`/`driver-resilience-status` on every (re)subscribe
|
||
so those stores re-prime immediately after a reconnect; `alerts`/`script-logs` are append logs and
|
||
tolerate the gap. The `/alerts`/`/script-log` pill reflects aggregate stream health (live when ≥1
|
||
node stream is up) — the same semantics the DPS pill has always had.
|
||
|
||
See `docs/Telemetry.md` for the full architecture, `docs/Configuration.md`
|
||
§`Telemetry`/`TelemetryDial` for the config keys, and
|
||
`docs/plans/2026-07-23-mesh-phase5-grpc-telemetry-stream.md` for the implementation plan and live
|
||
gate procedure.
|
||
|
||
## LocalDb pair-local store (Phases 1 + 2)
|
||
|
||
Every **driver-role** node keeps a consolidated `ZB.MOM.WW.LocalDb` SQLite database (the retired
|
||
LiteDB `LocalCache` subsystem was deleted as superseded). It holds **three replicated tables**:
|
||
`deployment_artifacts` + `deployment_pointer` (Phase 1) and `alarm_sf_events` (Phase 2).
|
||
|
||
**Phase 1** caches the
|
||
**deployed-configuration artifact** (chunked, SHA-256-verified, newest-2 retention per cluster) so a
|
||
driver node can **boot from cache when central SQL Server is unreachable** — `DriverHostActor` writes
|
||
the cache after each successful apply (`IDeploymentArtifactCache` → `LocalDbDeploymentArtifactCache`,
|
||
in `Runtime/Deployment/`) and reads it as a boot fallback, flagging running-from-cache. Wired by
|
||
`AddOtOpcUaLocalDb` in the `hasDriver` branch only (`Host/Configuration/LocalDbRegistration.cs` +
|
||
`LocalDbSetup.OnReady`, whose DDL→`RegisterReplicated` order is load-bearing); **admin-only nodes
|
||
never register it.** The cache can optionally replicate to the node's redundant pair peer over the
|
||
library's gRPC sync — **default-OFF, fail-closed bearer auth** (`LocalDbSyncAuthInterceptor`), gated
|
||
on `LocalDb:SyncListenPort` (a dedicated Kestrel h2c listener) + `LocalDb:Replication:*`. On the
|
||
docker-dev rig the **site-a pair** has replication on; central + site-b stay default-OFF (site-b is
|
||
the pin). ApiKey via `${secret:}`/env, never committed (the rig's dev key is the committed-dev-secret
|
||
exception).
|
||
|
||
**Phase 2** moved the **alarm store-and-forward buffer** off its standalone `alarm-historian.db`
|
||
and into the same database as `alarm_sf_events`, so a node that dies holding undelivered alarm
|
||
history no longer takes it to the grave. `SqliteStoreAndForwardSink` became
|
||
`LocalDbStoreAndForwardSink` (`Core.AlarmHistorian`), and the breaking config key
|
||
`AlarmHistorian:DatabasePath` was **removed** — it is still read, by `AlarmSfLegacyMigrator`, purely
|
||
to find a pre-consolidation file to copy across on first boot (renamed `.migrated` after the copy
|
||
commits). Because the buffer replicates, **only the node holding the Primary role drains it**: the
|
||
sink takes a `drainGate` delegate, Runtime supplies it from the `IRedundancyRoleView` singleton that
|
||
`DriverHostActor` publishes its `PrimaryGatePolicy` verdict to, and a gated-off node reports the new
|
||
`HistorianDrainState.NotPrimary`. Delivery is therefore **at-least-once across a failover, by
|
||
design** (no dedup layer). Row ids are a SHA-256 of the event payload rather than a GUID, so the
|
||
same event accepted on both nodes — which happens in every boot window, since
|
||
`HistorianAdapterActor` default-writes while its role is unknown — converges to one row. On the
|
||
docker-dev rig the site-a pair and site-b both run `AlarmHistorian__Enabled=true` against a
|
||
deliberately unresolvable gateway endpoint, so the buffer is always a live, non-draining queue.
|
||
|
||
**The library pin is `0.3.4`** (`Directory.Packages.props`, both `ZB.MOM.WW.LocalDb` and
|
||
`.Replication`), bumped from `0.2.1` on 2026-09-03 (was itself bumped from `0.1.3` on 2026-08-15).
|
||
The 0.2.1→0.3.4 bump is wire-compatible with 0.2.x, and the on-disk bookkeeping schema v2 → v3
|
||
upgrades in place on open, so no rig or volume action was needed. `RegisterReplicated(string)` is
|
||
unchanged and everything 0.2.x/0.3.x added is additive, **except** that `RegisterReplicated` (0.3.3)
|
||
now throws `LocalDbRegistrationException` for a table that is the parent of a row-mutating ON
|
||
DELETE FK — none of this repo's four replicated tables have such an FK, so this did not fire.
|
||
0.3.0 made the snapshot-resync flag **per-table** (see the ⚠️ below, now corrected); 0.3.2 added
|
||
validated config `Replication:KeepAlivePingDelay`/`KeepAlivePingTimeout` (defaults 60s/20s, left
|
||
unset here) and a public `LocalDbOplogGapException`, neither yet wired to a call site. **Its two
|
||
new APIs — `DeregisterReplicated` and `RegisterReplicated(table, baselineExistingRows:)` — are now
|
||
both used** (2026-08-18); the bullets below are the shape that landed, not the bump itself. Two
|
||
things are worth knowing:
|
||
|
||
- **`MaxBatchSize` no longer means what the rig's `16` was chosen to mean.** 0.2.0 made batching
|
||
**byte-budgeted** — `LocalDb:Replication:MaxBatchBytes` (2 MB default, left unset here, sized
|
||
under the 4 MB gRPC cap) bounds the wire message. `MaxBatchSize` is demoted to a secondary row
|
||
cap **and** the DB read-page size in `OplogStore.ReadBatchAboveAsync`/`SnapshotStreamer`, which
|
||
materialise a whole page in memory *before* the byte split runs. So the site-a pair keeps
|
||
`16` — it now bounds that transient allocation (≈2.7 MB at ≈171 KB artifact chunk rows), not the
|
||
message size. Every comment and doc row that called it "row-count-only against gRPC's 4 MB cap"
|
||
was corrected with the bump; do not restore that rationale.
|
||
- **Registration FOLLOWS the replication config — it is no longer unconditional (2026-08-18).**
|
||
`LocalDbSetup.OnReady` reads one predicate, `LocalDbReplicationPolicy.CapturesChanges`
|
||
(via `LocalDbSetup.ReplicationIsConfigured`):
|
||
**`LocalDb:Replication:PeerAddress` OR `LocalDb:Replication:ApiKey` is non-blank.** The **OR is
|
||
not belt-and-braces** — only the dialling half of a pair sets `PeerAddress`, so keying on it
|
||
alone would strip capture from every *passive* node (site-a-2 carries `ApiKey` +
|
||
`LocalDb:SyncListenPort` and nothing else) and the pair would converge in one direction only,
|
||
silently. ⚠️ **`SyncListenPort` deliberately does NOT join the OR:** the passive endpoint is
|
||
fail-closed on the key, so a port without an `ApiKey` rejects every inbound stream and has
|
||
nothing to capture for.
|
||
- ⚠️ **The alarm DRAIN gate uses a different predicate** — `LocalDbReplicationPolicy.SharesRowsWithPeer`:
|
||
`ApiKey` **and** (`PeerAddress` **or** `LocalDb:SyncListenPort > 0`), i.e. "a peer actually holds these
|
||
rows". Capture must be wider than the drain gate (a dialling-only node still captures); a node with no
|
||
peer must still drain, so the two predicates are deliberately not the same.
|
||
The **passive** half (no `PeerAddress`) cannot name its peer, so it stands down for its **mesh partner**
|
||
(the other member of its `cluster-{ClusterId}` role); a legacy passive half with no mesh role still
|
||
drains alongside its partner (duplicate historian rows). ⚠️ Before a node has joined its mesh the
|
||
published Primary verdict (and the live data-plane gate, which must stay the same decision) **allows** —
|
||
for seconds, up to ~25 s under the bootstrap guard — so a cold-starting node can briefly co-hold the
|
||
Sparkplug primary-host claim with its Primary peer.
|
||
- **Configured ⇒ `RegisterReplicated(table, baselineExistingRows: true)`**, per table, in the
|
||
same load-bearing DDL→register order. Baselining is what makes enabling replication on a node
|
||
that has been running *without* it actually converge: capture is CDC and the snapshot streamer
|
||
pages from `__localdb_row_version`, so pre-registration rows were in neither and a late opt-in
|
||
used to replicate only writes made after the restart. It seeds the ledger at the LWW floor
|
||
(HLC `0`, this node's id) and flags a resync — a baselined row loses to any genuine remote
|
||
write of the same key and wins only where the peer has no version at all. It is idempotent, so
|
||
it is free on every boot after the first. As of `0.3.0` (this repo pins `0.3.4`) the snapshot
|
||
flag is **per-table**, not per-database — baselining one table no longer streams every
|
||
registered table in both directions. (Prior to `0.3.0`, on the `0.2.1` pin this repo carried
|
||
until 2026-09-03, the flag was per-database and baselining anything streamed *every*
|
||
registered table in full; the old advice to trim `alarm_sf_events` before enabling replication
|
||
no longer applies at the current pin.)
|
||
- **Not configured ⇒ `DeregisterReplicated(table)` per table**, an idempotent boot-time
|
||
self-heal: it drops the capture triggers an earlier (unconditional) build left in the file and
|
||
prunes that table's oplog + row-version rows. Merely declining to register would leave every
|
||
pre-2026-08-18 file capturing forever — two INSERTs plus a `json_object` of the whole row
|
||
inside every write transaction — on precisely the node that has no peer to capture for. The
|
||
call returns `true` when it removed something, so the Information line fires **once, and only
|
||
when it actually cleaned**; a default-OFF node's ordinary boot stays silent. The logger is
|
||
Serilog's static `Log.Logger` (this runs inside the `AddZbLocalDb` singleton factory, where
|
||
resolving one from DI would nest a resolution inside a singleton construction; `Program.cs`
|
||
assigns it before anything resolves `ILocalDb`), with an optional parameter so tests observe
|
||
the line rather than infer it.
|
||
- ⚠️ **PAIR SYMMETRY: flipping registration is a BOTH-NODES-TOGETHER change.** The sync handshake
|
||
compares registered-table digests **fail-closed**, so a node that deregisters a table its peer
|
||
still replicates stops syncing with a schema mismatch. Enabling replication on an established
|
||
pair means setting the config and **restarting both nodes** — do it on both, or on neither.
|
||
- **Rollout note:** the only replicating pair in the fleet is docker-dev **site-a**, and it was
|
||
enabled from a fresh pair, so the impact of the old gap was low — central and site-b are
|
||
default-OFF and simply shed their unused triggers on their next boot. `LocalDbSetupTests`
|
||
pins both branches (registered-table set on a configured node, empty set + zero triggers +
|
||
the HLC-0 baseline ledger entry on the other) and the DDL→register ordering, unchanged.
|
||
|
||
See `docs/Configuration.md` (`LocalDb` section), `docs/Redundancy.md` (pair-local config
|
||
cache), `docs/AlarmHistorian.md`, and the runbook
|
||
`docs/operations/2026-07-20-localdb-pair-replication.md`.
|
||
|
||
## Phase 4 — cut the driver-side ConfigDb connection
|
||
|
||
Per-cluster mesh **Phase 4** finished the job Phase 3 started: a **driver-only** node (`Cluster:Roles`
|
||
has `driver`, not `admin`) now boots with **no `ConfigDb` connection string at all**. `Program.cs`
|
||
gates `AddOtOpcUaConfigDb` (and its health check) on `hasAdmin`; a fused `admin,driver` node is
|
||
unaffected (it keeps ConfigDb via its admin role). Consequences:
|
||
|
||
- **`ConfigSource:Mode` must be `FetchAndCache` on a driver-only node** — `ConfigSourceOptionsValidator`
|
||
fails host start if such a node is left on `Direct` (there is nothing to read Direct from). A fused
|
||
node may stay `Direct`.
|
||
- **ServiceLevel semantics change — client-visible.** With no ConfigDb, `DbHealthProbeActor` is not
|
||
spawned (`ServiceCollectionExtensions.WithOtOpcUaRuntimeActors` skips it when the resolved db
|
||
factory is null), and `OpcUaPublishActor` runs `dbLess`: `DbReachable` is fed constant `true` and
|
||
`Stale` comes from the redundancy-snapshot age alone. A healthy driver-only node therefore publishes
|
||
**240** (or **250** as driver Primary) even with central SQL unreachable — the survive-alone posture
|
||
— where a `Direct`/DB-backed node in the same state drops to 100/0. `ServiceLevelCalculator` itself
|
||
is unchanged; only the inputs a DB-less node feeds it changed. See `docs/Redundancy.md` §"DB-less
|
||
(driver-only) nodes".
|
||
- **Scripted-alarm Part 9 condition state moved to LocalDb.** `LocalDbAlarmConditionStateStore`
|
||
(replicated `alarm_condition_state` table, pair-local like the Phase-2 alarm S&F buffer) replaces
|
||
`EfAlarmConditionStateStore` on every driver-role node, fused central included. The DB-backed
|
||
`ScriptedAlarmState` table **was dropped** (Task 9, `3a590a0c`, migration
|
||
`20260723183050_DropScriptedAlarmStateTable`); `EfAlarmConditionStateStore` is deleted.
|
||
- **Central persists deploy acks.** `DriverHostActor` no longer writes `NodeDeploymentState` to SQL on
|
||
a driver-only node — `ConfigPublishCoordinator.PersistNodeAck` (already fed by every ApplyAck) is
|
||
the system of record.
|
||
- **LDAP group→role DB grants don't reach driver-only nodes.** They register the empty
|
||
`NullLdapGroupRoleMappingService` (no ConfigDb to read `LdapGroupRoleMapping` from), so
|
||
`OtOpcUaGroupRoleMapper` falls back to the `Security:Ldap:GroupToRole` appsettings baseline only —
|
||
the same rows a driver node never received via config either, so this is not a regression.
|
||
|
||
Merged to master (`26fad75c`); **live gate PASSED** (`1281aebf` — driver nodes run ConfigDb-free) and
|
||
Task 9 landed (`3a590a0c`). See `docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md` and
|
||
`docs/plans/2026-07-22-per-cluster-mesh-program.md` §Phase 4.
|