lmxopcua/.claude/rules/cluster-topology.md

paths

paths
src/Core/ZB.MOM.WW.OtOpcUa.Cluster/** src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/** src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Redundancy/** src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Telemetry/** src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/** src/Server/ZB.MOM.WW.OtOpcUa.Host/** docker-dev/** docs/Redundancy.md docs/Configuration.md docs/Telemetry.md

Cluster topology: redundancy, role authority, per-cluster meshes, MeshTransport, ConfigSource/ConfigServe, telemetry transport, LocalDb, DB-less driver nodes

Moved verbatim from the root CLAUDE.md (2026-10-13) so it loads only when working in this area. Root CLAUDE.md § "Subsystem guides" indexes it. Edit it here, not in the root file.

Redundancy

The server supports non-transparent warm/hot redundancy via the Redundancy section in appsettings.json. Two instances share the same Galaxy DB and the same mxaccessgw (under distinct MxAccess.ClientName values) but have unique ApplicationUri values. Each exposes RedundancySupport, ServerUriArray, and a dynamic ServiceLevel based on role and runtime health. The primary advertises a higher ServiceLevel than the secondary. See docs/Redundancy.md for the full guide.

Two corrections landed 2026-07-21 — both were total-outage or wrong-node defects, and both are worth knowing before touching this area.

• Downing is auto-down, not keep-oldest. Cluster:SplitBrainResolverStrategy (default auto-down) selects the provider via ServiceCollectionExtensions.BuildDowningHocon; an unknown value fails host start. A two-node pair running the previous keep-oldest + down-if-alone could not survive a crash of the oldest node: Akka's KeepOldest.OldestDecision only lets down-if-alone rescue a side holding ≥ 2 members, so the 1-vs-1 survivor downs itself and exits. down-if-alone is a 3+-node feature. The accepted trade for auto-down is dual-active during a genuine network partition. Its live gate is CLOSED — Phase 7 drill D4 (2026-07-24) ran the 1-vs-1 crash-the-oldest survival drill on the three-mesh docker-dev rig and every survivor stayed Up, never self-downing (c50ebcf7; docs/plans/2026-07-24-mesh-phase7-failover-drills.md).
• The Primary is the oldest Up driver member, not the role leader. RedundancyStateActor previously used ClusterState.RoleLeader("driver") (lowest address), while ClusterSingletonManager places singletons on the oldest. They agree only on a freshly-formed cluster and diverge after any restart, so the Primary-gated data plane (writes, alarm acks, alerts emit, alarm-history drain) could enable on a node not hosting the singletons. NodeRedundancyState.IsRoleLeaderForDriver was renamed IsDriverPrimary.

A third bootstrap gap closed 2026-07-22 — downing was never the whole story, and the fix changed twice. Even under auto-down, a node cold-starting while its peer was dead never came Up: Akka runs FirstSeedNodeProcess — the only bootstrap path that can form a new cluster when no peer answers InitJoin — exclusively when seed-nodes[0] is the node’s own address; every other node runs JoinSeedNodeProcess and retries InitJoin forever. Cluster:SeedNodes is therefore ORDERED: every node that is one of its own seeds lists ITSELF first, partner second (docker-dev central-2 was swapped; site nodes list only central-1 and are legitimately not seeds). AkkaClusterOptionsValidator (AddValidatedOptions/ValidateOnStart in AddOtOpcUaCluster) enforces it at boot — conditionally (self must be entry 0 only if self is in the list, or every site node would refuse to start), matching on PublicHostname+Port, never the 0.0.0.0 bind address. The earlier fix — a Cluster:SelfFormAfter timer calling Cluster.Join(SelfAddress) — is deleted: it sat outside Akka’s join handshake, so it could not tell "no seed answered" from "a seed answered and the join is in flight", and it islanded a manually-failed-over node live before a TCP reachability guard patched that one shape. Pinned by SelfFirstSeedBootstrapTests (real in-process clusters, incl. a falsifiability control proving peer-first ordering never forms) + AkkaClusterOptionsValidatorTests. See docs/Redundancy.md §"Bootstrap: self-first seed ordering".

Simultaneous cold-start splits — the bootstrap guard closes it (opt-in, Cluster:BootstrapGuard:Enabled, default OFF). Self-first on BOTH nodes means when both cold-start at the same instant, each runs FirstSeedNodeProcess, times out, and forms its OWN 1-node cluster → two Primaries in one pair (the Phase 6 live gate reproduced this on docker; two separate clusters do NOT auto-merge). The guard prevents it without losing cold-start-alone: the lexicographically lower host:port node is the preferred founder (self-first, forms immediately); the higher node TCP-probes the partner's Akka port up to PartnerProbeSeconds (25s) — reachable ⇒ peer-first (join the founder), unreachable ⇒ self-first (partner dead, form alone). The order is chosen BEFORE the single JoinSeedNodes, from an explicit reachability signal — it never re-forms mid-handshake (the SelfFormAfter failure mode). When on, Akka gets NO config seeds (BuildClusterOptions empties ClusterOptions.SeedNodes) and ClusterBootstrapCoordinator drives the join. Residual trade (accepted, operator-visible via a warning + restart-recovers): the higher node hangs if the founder dies in the probe→join window. docker-dev: site-a = guard demo (guard on, serialization removed); site-b = A/B control (guard off, depends_on: service_healthy serialization). The guard is the production-faithful fix (compose depends_on doesn't exist on real VMs). Pinned by ClusterBootstrapGuardTests + ClusterBootstrapCoordinatorTests (real 2-node clusters, incl. the higher-node-cold-start-alone case). See docs/Redundancy.md §"Bootstrap guard".

Assert the effective cluster config off a running ActorSystem (as SplitBrainResolverActivationTests does), never the typed options or the akka.conf text — several HOCON fragments compete and the losing one still reads correctly in the file.

RESOLVED by per-cluster mesh Phase 6 (2026-07-24) — no longer per-Akka-cluster. The "one Primary across the whole Akka mesh" limitation described above is gone; see the Phase 6 paragraph immediately below.

Per-cluster mesh Phase 6 (2026-07-24) — the fleet is now three independent 2-node meshes, not one. The single fleet-wide Akka mesh was split into one mesh per application Cluster: the central pair (admin,driver,cluster-MAIN), site-a (driver,cluster-SITE-A), site-b (driver,cluster-SITE-B) — same ActorSystem name, separated purely by per-pair self-first seeds (each node's Cluster:SeedNodes lists itself first, its own pair partner second, never a node from another cluster's pair). Consequences:

• The redundancy singleton is cluster-scoped and spawned per driver node. RedundancyStateActor is now a cluster-{ClusterId}-scoped singleton (falls back to plain driver for a legacy node), so each pair elects and publishes its own Primary on its own mesh's DPS. Two or more Primaries exist fleet-wide, one per pair — by design, not a bug. ⚠️ Wiring landmine (fixed 3a4ed8dd, caught by the live gate): the singleton's cluster-role scope is computed at AddAkka-configurator time, so it must come from AkkaClusterOptions (config) — never by resolving IClusterRoleInfo from the SP there. ClusterRoleInfo depends on the ActorSystem, and that lambda runs while the ActorSystem is being built, so resolving it recurses into ActorSystem construction and stack-overflows every node at boot (compiles clean; the rehome tests passed a hand-built fake straight into the extension, mocking around the composition-root line). Keep WithOtOpcUaClusterRedundancySingleton/BuildClusterRedundancySingletonOptions taking AkkaClusterOptions.
• ClusterNodeAddressReconciler is own-cluster-scoped — an admin node can only reconcile ClusterNode rows in its own cluster; it has no gossip visibility into another mesh.
• CentralCommunicationActor dials each node directly over gRPC — one NodeCommandService.Execute call per enabled ClusterNode row carrying a GrpcPort — rather than fanning through a client/receptionist object. Superseded by the ClusterClient→gRPC migration: the Akka ClusterClient this bullet originally described (one per application Cluster, closing the TODO(Phase 6) Phase 2 left behind) is deleted; the per-node gRPC dial is its replacement, not an evolution of it.
• SplitTopologyTransportValidator fails host start on a node carrying a cluster-{ClusterId} role unless it also runs the split-safe transports: MeshTransport:Mode=Grpc, Telemetry:Mode=Grpc (driver), TelemetryDial:Mode=Grpc (admin). A DPS transport on a cluster-role node would try to gossip across a partition that no longer exists.
• The compiled transport-mode defaults were deliberately NOT flipped — MeshTransport:Mode/Telemetry:Mode/TelemetryDial:Mode all still default to Dps; a blast-radius assessment showed flipping the default breaks every integration test and appsettings profile that leaves the key unset. The validator + explicit per-deployment config is the guardrail, not a default flip.
• Secrets replication is pair-local — each pair replicates its own Akka DPS secrets within itself; there is no fleet-wide/cross-cluster sync after the split (sites have no SQL, so no SQL-hub, consistent with Phase 4).

See docs/Redundancy.md §"Per-cluster meshes (Phase 6)", docs/Configuration.md §"Per-cluster mesh split (Phase 6)", and docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md.

Role authority (EffectiveRoles)

A node's roles are resolved in exactly one place (archreview R3-04, closing 01/S-12 = 03/C5 and 01/C-9): EffectiveRoles.Resolve (ZB.MOM.WW.OtOpcUa.Cluster), with one precedence — Cluster:Roles when non-empty, else OTOPCUA_ROLES, else empty — and two-layer normalization: a plain role (admin/driver/dev) is trimmed and lower-cased, but a cluster role's ClusterId suffix (cluster-SITE-A) is kept verbatim, because lower-casing it would rename the Akka role of a running deployment. An unknown role from either source fails host start. Program.cs resolves one EffectiveRoles instance at the top of the composition root and registers it as a singleton; that SAME instance drives both the conditional host wiring (hasAdmin/hasDriver) and AkkaClusterOptions.Roles (what Akka gossips) — there is no second derivation for the two to disagree about, which is why no runtime "do the views agree" assert exists. Before this, Program.cs read OTOPCUA_ROLES only, so a node configured solely through Cluster:Roles gossiped the right Akka roles while Program.cs's own wiring mounted none of them — the inverse of the pre-#532 field incident (docs/deployments/wonder-app-vd03-makino-z-34184.md).

• A role overlay may not itself set Cluster:Roles. Roles are resolved before the per-role appsettings overlay loads (the overlay's filename depends on the answer), so a later layer changing Cluster:Roles would leave the bound options disagreeing with the registered instance. AkkaClusterOptionsValidator fails host start if this ever happens, naming both views — a no-op today since none of the four shipped overlays carries a Cluster section.
• Both sources set to different normalized role sets logs a Warning at boot (Cluster:Roles still wins outright) — the boot banner also names which source won (source=Configured / Environment / None).
• RoleParser.GossipedRolesHave (case-insensitive) is defence in depth at the handful of production sites that read LIVE GOSSIP directly (Member.Roles) rather than through EffectiveRoles — a peer running an older, pre-R3-04 build can still gossip a miscased plain role during a rolling upgrade. ⚠️ Accepted residual (#620), not closed by R3-04: RedundancyStateActor's election delegates to ZB.MOM.WW.Health.Akka's ClusterActiveNode.OldestUpMember, which matches via Akka's OWN ordinal Member.HasRole. Deliberately not patched — neither in the library nor with a local case-insensitive pre-filter, which would split the Primary election from ClusterSingletonManager placement (also ordinal). It only bites a mixed fixed/pre-fix rolling-upgrade window; finish the upgrade. Full rationale: docs/Redundancy.md ("Accepted residual (#620)").
• Program.cs wraps the composition root (#618): a start-up failure logs Fatal, writes FATAL: to stderr (the bootstrap logger is still silent before AddZbSerilog), flushes, and Environment.Exit(1)s — leftover Akka/Kestrel threads would otherwise keep the process alive. HostAbortedException (design-time tooling) passes through untouched.
• RoleAuthorityGuardTests (tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/) is the enforcement: a source scan failing on any production file outside EffectiveRoles.cs/RoleParser.cs that reads OTOPCUA_ROLES, Cluster:Roles as code, or declares a bare "admin"/"driver"/"dev" literal — with a small, reasoned allow-list for the handful of genuinely unrelated domain uses of those same English words (a raw-tree node-kind label, a Sql driver's own JSON marker, an LDAP-derived AdminUI RBAC role).

See docs/Configuration.md §Roles, docs/ServiceHosting.md §"Role gating", and archreview/plans/R3-04-one-role-authority-plan.md.

Mesh command transport (MeshTransport)

The Akka ClusterClient transport described in this section's earlier history is DELETED (branch feat/grpc-phase3-deletion, ClusterClient→gRPC migration, Phase 3) — code, the akka.cluster.client HOCON block, the receptionist extension, MeshTransport:CentralContactPoints, and the ClusterClient mode value itself, which now fails validation loudly at boot (MeshTransportOptionsValidator names the key and the allowed values) rather than silently falling back to Dps. What remains is exactly two transports, selected by MeshTransport:Mode:

• Dps (default) — DistributedPubSub, the path this repo has always used. Pair-local only: it works only when central and the node share a gossip ring, which after Phase 6 means within one 2-node mesh.
• Grpc — unary gRPC calls on the mesh_transport.v1 contract. Required on any node carrying a cluster-{ClusterId} role; SplitTopologyTransportValidator refuses to boot such a node on anything else.

Central→node commands (deployments, driver-control, alarm-commands) ride NodeCommandService.Execute, served on each node's driver-role telemetry h2c listener (Telemetry:GrpcListenPort, the docker-dev rig's :4056 — it shares that listener with TelemetryStreamGrpcService rather than claiming a port of its own). CentralCommunicationActor dials it directly, one call per node — the target set is every enabled, non-maintenance ClusterNode row carrying a GrpcPort, dialled at Host+GrpcPort and refreshed every MeshTransport:ContactRefreshSeconds. There is no client/receptionist object and no rotation: each command goes to the specific node dialled, which is what makes per-node delivery provable — the old ClusterClient.SendToAll/Send distinction this section used to explain no longer applies to either direction.

Node→central deploy acks ride CentralCommandService.ReportApply, served on central's admin-role config-serve h2c listener (ConfigServe:GrpcListenPort, the rig's :4055 — again sharing rather than claiming a new port). GrpcApplyAckSender walks MeshTransport:CentralGrpcEndpoints in order and stops at the first success.

Auth is a per-cluster PSK bearer, both directions: MeshTransport:GrpcPsk (node-side, one key per application Cluster) and MeshTransport:ClusterPsks (central-side map, consulted before the Secrets store). MeshCommandAuthInterceptor (node) and CentralCommandAuthInterceptor (central) fail closed on every unresolvable case — constant-time compare, no key material logged. Production keys are ${secret:OT-MESH-PSK-<clusterId>}; the docker-dev rig supplies three literal dev keys via env (dev-mesh-psk-MAIN/-SITE-A/-SITE-B).

Things that carried over unchanged from the ClusterClient era, because they were never ClusterClient-specific to begin with:

• Fire-and-forget, no retry, on both transports. A failed gRPC dispatch is logged (naming the node and the gRPC status) and dropped in the dispatch second. No queue, no ordering guarantee across nodes — the deploy coordinator's apply deadline and a node's boot-time replay are the only recovery. The old buffer-size = 0 decision this section used to describe is gone with the HOCON block that set it; the gRPC transport has nothing to buffer in the first place — a dial either lands or doesn't.
• Inbound commands re-emit on the node's EventStream, not back onto DPS. DPS is mesh-wide, so a DPS re-publish would deliver N copies to every subscriber. Node-side subscribers (DriverHostActor, ScriptedAlarmHostActor) accept a command from either transport identically.
• Do not over-claim what fire-and-forget buys. The Phase 2 live gate observed a node applying a deployment 44 s after it was sealed TimedOut anyway, via DriverHostActor.Bootstrap() replaying the orphan Applying row on restart. That backstop is unaffected by the ClusterClient deletion — it was always about the deploy path, not the transport.

redundancy-state stays on DPS in both modes (bidirectional, and built from Cluster.State, which makes it genuinely mesh-bound). See docs/Configuration.md §MeshTransport, docs/Redundancy.md §"Command transport", and the migration's design doc / plan / tracking doc / live gate for the phase-by-phase history — those are historical records, not updated further by this sweep. docs/plans/2026-07-22-mesh-phase2-clusterclient-transport.md is the original Phase 2 design doc and is likewise left as a historical record of the transport this section now describes as deleted.

Phase 1 landed 2026-07-22 and changed a deploy-path behaviour. ConfigPublishCoordinator sources its expected-ack set from enabled ClusterNode rows, not from Akka.Cluster.State.Members filtered by the driver role — central must be able to name a deployment's nodes without sharing a gossip ring with them. Consequences worth knowing before touching deploys:

• A configured node that is switched off now FAILS the deployment at the apply deadline instead of letting it seal green without that node. The maintenance hatch is the new ClusterNode.MaintenanceMode column, NOT Enabled = 0: the live gate showed Enabled = 0 is rejected outright by DraftValidator.ValidateClusterTopology (ClusterEnabledNodeCountMismatch — enabled count must equal ServerCluster.NodeCount), so on a 2-node pair — every cluster in the target topology — it can never be the hatch. Enabled = part of the declared topology; MaintenanceMode = do not expect it now. The deadline log names the silent nodes and points at the right flag.
• Every ClusterNode row is now defined to be a driver node. The DB has no per-node role column and deliberately did not gain one (it would drift from Cluster:Roles). Giving an admin-only node a ClusterNode row makes every deploy time out.
• There is no such thing as a cluster-scoped deployment. Deployment has no ClusterId, ConfigComposer always snapshots the whole DB, and DeploymentArtifact.ResolveClusterScope is node-side self-scoping of a fleet-wide artifact. The phase-1 plan asked for cluster-scope filtering of the ack set; it was dropped as describing a feature that does not exist.
• ClusterNode gained AkkaPort (NOT NULL, 4053) + GrpcPort (nullable) as central's dial targets for Phases 2/5. They duplicate the node's own Cluster:Port / Cluster:PublicHostname with no schema-level enforcement, so ClusterNodeAddressReconcilerActor (admin singleton) logs an Error on drift. Phase 2 must revisit it — once the meshes split, an admin node cannot see site members and every site row would report EnabledRowNotInCluster forever. Resolved, and not the way this bullet originally expected: the ClusterClient→gRPC migration deleted the Akka ClusterClient transport that would have dialled Host+AkkaPort; the Phase 2 command dial now uses Host+GrpcPort, the same column Phase 5 telemetry already dialled — AkkaPort is Akka remoting only again.

Config source (ConfigSource / ConfigServe)

Per-cluster mesh Phase 3 (config fetch-and-cache) added a second way a driver node obtains its deployed configuration, selected by ConfigSource:Mode: Direct (read Deployment.ArtifactBlob from central SQL — the default, byte-for-byte today's behaviour) or FetchAndCache (fetch the artifact bytes from central over a dedicated-h2c gRPC stream, verify SHA-256 == RevisionHash, cache in LocalDb, and read only the cache — no central-SQL config read). Central (admin role) serves via DeploymentArtifactService behind ConfigServeAuthInterceptor on ConfigServe:GrpcListenPort, gated by a shared node bearer key (ConfigServe:ApiKey == ConfigSource:ApiKey, FixedTimeEquals, fail-closed). Things worth knowing before touching it:

• The serve side is wired in both modes (mapped whenever ConfigServe:GrpcListenPort > 0 on an admin node), so flipping a node to FetchAndCache is a config change, not a redeploy — the same dark-switch discipline as MeshTransport.
• RevisionHash == SHA-256(artifact bytes) is load-bearing. ConfigComposer computes the revision hash as the lowercase-hex SHA-256 of the exact blob, so the node verifies a fetch against the RevisionHash the dispatch already carries — no hash rides in the proto, and DispatchDeployment stays payload-free. A future change to how the revision hash is derived silently breaks every fetch. The blob is gzip-compressed since 2026-08-15, and the hash covers the COMPRESSED bytes — ConfigComposer.Pack gzips (CompressionLevel.Fastest) immediately after SerializeToUtf8Bytes and hashes the result, so the invariant holds unchanged and nothing between the composer and DeploymentArtifact.Open (SQL ArtifactBlob, the LocalDb cache chunks, the gRPC fetch stream) ever looks inside. Open sniffs the gzip magic 1F 8B — valid JSON can never start with 0x1F — so pre-upgrade uncompressed rows keep parsing; inflation runs under a 512 MiB guard and every decompression failure surfaces as JsonException, the same family the malformed-JSON paths already handle. Two operational consequences in docs/Configuration.md §ConfigSource: the first deploy after upgrade re-hashes every revision (a one-time full re-dispatch, not a no-op), and nodes must be upgraded BEFORE central starts emitting compressed blobs.
• A null fetch is "no answer," never an empty config (#485). All endpoints down / NotFound / SHA mismatch / zero-length ⇒ the fetcher returns null ⇒ the apply FAILS, the revision does not advance, and the node keeps serving last-known-good. It never tears down to an empty address space. At boot a FetchAndCache node restores served state from the LocalDb pointer with no central-SQL read; an empty/unreadable cache lands Steady-with-no-revision (the first dispatch fetches), never Stale. Malformed non-empty bytes are "no answer" too — on cache boot, the FetchAndCache boot reconcile and the subscription push alike (the lenient swallow leg is closed). A cache boot from malformed bytes does not adopt the revision (Direct ⇒ Stale; FetchAndCache ⇒ no revision, the re-dispatch fetches).
• Both pair nodes fetch independently — no Primary gating of the fetch (same shape as both reading SQL today; StoreAsync is idempotent and the cache replicates).
• The dedicated h2c listener coexists with the LocalDb-sync listener. A fused admin+driver node with both ports set re-binds its existing HTTP surface exactly once and adds both dedicated HTTP/2-only ports — getting this wrong either double-binds (crash) or silently unbinds the AdminUI (the Kestrel Listen*-discards-ASPNETCORE_URLS trap the LocalDb block already warned about).
• Phase boundary: config reads only. A FetchAndCache node still writes its NodeDeploymentState ack row to central SQL; DbHealthProbe and EfAlarmConditionStateStore still touch SQL. Cutting those is Phase 4 — resist folding it in, or the "no SQL" scope becomes unprovable-in-parts.

On the docker-dev rig the central pair carries ConfigServe (:4055, committed dev key) and stays hardcoded Direct via its admin role; the four site nodes carry ConfigSource (endpoints + matching key) hardcoded FetchAndCache — mandatory for driver-only nodes since Phase 4, not a flip. There is no OTOPCUA_CONFIG_MODE env var in the compose file. See docs/Configuration.md §ConfigSource / ConfigServe, docs/Redundancy.md §"Command transport", and docs/plans/2026-07-22-mesh-phase3-config-fetch-and-cache.md.

Live telemetry transport (Telemetry / TelemetryDial)

Per-cluster mesh Phase 5 (merged 35552a96; live gate PASSED f134935f) added a second transport for the four live-observability channels the AdminUI's /alerts, /script-log, and /hosts panels feed on, selected by Telemetry:Mode (node/serve side) and TelemetryDial:Mode (central/dial side): Dps — the four existing DPS SignalR bridges, the default, today's behaviour — or Grpc, one gRPC server-streaming contract carrying all four channels as oneof event kinds. The direction is inverted from ConfigSource/MeshTransport: the driver node hosts the gRPC server (dedicated Kestrel h2c port), and central is the client that dials in — mirroring ScadaBridge's SiteStreamService shape, and required so telemetry survives once the meshes split (Phase 6) and central no longer shares a gossip ring with a site node. Things worth knowing before touching it:

• Migrated (4 channels): alerts (AlarmTransitionEvent), script-logs (ScriptLogEntry), driver-health (DriverHealthChanged), driver-resilience-status (DriverResilienceStatusChanged) — feeding the exact same central sinks (IInProcessBroadcaster<...>, IDriverStatusSnapshotStore, IDriverResilienceStatusStore) in either mode.
• Deferred, with reasons (do not read "seven observability topics" from the program sketch as the delivered scope): redundancy-state stays on DPS in every mode (pair-local, built from Cluster.State, drives ServiceLevel + the Primary gate — genuinely mesh-bound by design, not an observability panel); deployment-acks already rides the Phase 2 gRPC command transport (MeshTransport:Mode=Grpc, CentralCommandService.ReportApply) as a command-plane reply, not telemetry. A third, fleet-status, was DELETED rather than deferred (2026-08-06, Gitea #542): FleetStatusBroadcaster → DPS → SignalR bridge → FleetStatusHub had zero clients end to end (no browser JS, no HubConnection — /fleet polls NodeDeploymentState in the Config DB), its DriverHostStatusHeartbeat input had no production sender, and Phase 6 left its "fleet" as just central's own pair. Fleet visibility is /fleet + the four per-node telemetry dials above.
• The node ALWAYS hosts and ALWAYS double-publishes, in both modes. Every publish seam emits into a node-local ITelemetryLocalHub and still publishes DPS, unconditionally; the gRPC server binds whenever Telemetry:GrpcListenPort > 0, independent of Telemetry:Mode. Only TelemetryDial:Mode on central actually switches anything: which of the two always-available sources (DPS bridges vs. the TelemetryDialSupervisor dialer) feeds the sinks. This is what makes the flip a config change, not a coordinated redeploy — same dark-switch discipline as MeshTransport and ConfigSource.
• Authenticated from day one, fail-closed — a shared node bearer key (Telemetry:ApiKey == TelemetryDial:ApiKey), gated by TelemetryStreamAuthInterceptor (CryptographicOperations.FixedTimeEquals, PermissionDenied on mismatch, an unset key rejects every call). This supersedes design doc §6.3's "match ScadaBridge's unauthenticated posture for now" — ScadaBridge itself has since closed that gap with this identical interceptor pattern.
• Reconnect story: central runs one reconnecting dialer per driver node (discovered from ClusterNode.Host/GrpcPort, refreshed every TelemetryDial:ContactRefreshSeconds), immediate first retry then ~5s fixed backoff, generation-stamped so a superseded stream's late error/event is ignored, and it never permanently gives up (observability, not data plane). TelemetryLocalHub.Evict drops a removed driver's snapshots (StopChild, via plan.ToStop; central-side stores are not evicted). The node-local hub replays last-value snapshots for driver-health/driver-resilience-status on every (re)subscribe so those stores re-prime immediately after a reconnect; alerts/script-logs are append logs and tolerate the gap. The /alerts//script-log pill reflects aggregate stream health (live when ≥1 node stream is up) — the same semantics the DPS pill has always had.

See docs/Telemetry.md for the full architecture, docs/Configuration.md §Telemetry/TelemetryDial for the config keys, and docs/plans/2026-07-23-mesh-phase5-grpc-telemetry-stream.md for the implementation plan and live gate procedure.

LocalDb pair-local store (Phases 1 + 2)

Every driver-role node keeps a consolidated ZB.MOM.WW.LocalDb SQLite database (the retired LiteDB LocalCache subsystem was deleted as superseded). It holds three replicated tables: deployment_artifacts + deployment_pointer (Phase 1) and alarm_sf_events (Phase 2).

Phase 1 caches the deployed-configuration artifact (chunked, SHA-256-verified, newest-2 retention per cluster) so a driver node can boot from cache when central SQL Server is unreachable — DriverHostActor writes the cache after each successful apply (IDeploymentArtifactCache → LocalDbDeploymentArtifactCache, in Runtime/Deployment/) and reads it as a boot fallback, flagging running-from-cache. Wired by AddOtOpcUaLocalDb in the hasDriver branch only (Host/Configuration/LocalDbRegistration.cs + LocalDbSetup.OnReady, whose DDL→RegisterReplicated order is load-bearing); admin-only nodes never register it. The cache can optionally replicate to the node's redundant pair peer over the library's gRPC sync — default-OFF, fail-closed bearer auth (LocalDbSyncAuthInterceptor), gated on LocalDb:SyncListenPort (a dedicated Kestrel h2c listener) + LocalDb:Replication:*. On the docker-dev rig the site-a pair has replication on; central + site-b stay default-OFF (site-b is the pin). ApiKey via ${secret:}/env, never committed (the rig's dev key is the committed-dev-secret exception).

Phase 2 moved the alarm store-and-forward buffer off its standalone alarm-historian.db and into the same database as alarm_sf_events, so a node that dies holding undelivered alarm history no longer takes it to the grave. SqliteStoreAndForwardSink became LocalDbStoreAndForwardSink (Core.AlarmHistorian), and the breaking config key AlarmHistorian:DatabasePath was removed — it is still read, by AlarmSfLegacyMigrator, purely to find a pre-consolidation file to copy across on first boot (renamed .migrated after the copy commits). Because the buffer replicates, only the node holding the Primary role drains it: the sink takes a drainGate delegate, Runtime supplies it from the IRedundancyRoleView singleton that DriverHostActor publishes its PrimaryGatePolicy verdict to, and a gated-off node reports the new HistorianDrainState.NotPrimary. Delivery is therefore at-least-once across a failover, by design (no dedup layer). Row ids are a SHA-256 of the event payload rather than a GUID, so the same event accepted on both nodes — which happens in every boot window, since HistorianAdapterActor default-writes while its role is unknown — converges to one row. On the docker-dev rig the site-a pair and site-b both run AlarmHistorian__Enabled=true against a deliberately unresolvable gateway endpoint, so the buffer is always a live, non-draining queue.

The library pin is 0.3.4 (Directory.Packages.props, both ZB.MOM.WW.LocalDb and .Replication), bumped from 0.2.1 on 2026-09-03 (was itself bumped from 0.1.3 on 2026-08-15). The 0.2.1→0.3.4 bump is wire-compatible with 0.2.x, and the on-disk bookkeeping schema v2 → v3 upgrades in place on open, so no rig or volume action was needed. RegisterReplicated(string) is unchanged and everything 0.2.x/0.3.x added is additive, except that RegisterReplicated (0.3.3) now throws LocalDbRegistrationException for a table that is the parent of a row-mutating ON DELETE FK — none of this repo's four replicated tables have such an FK, so this did not fire. 0.3.0 made the snapshot-resync flag per-table (see the ⚠️ below, now corrected); 0.3.2 added validated config Replication:KeepAlivePingDelay/KeepAlivePingTimeout (defaults 60s/20s, left unset here) and a public LocalDbOplogGapException, neither yet wired to a call site. Its two new APIs — DeregisterReplicated and RegisterReplicated(table, baselineExistingRows:) — are now both used (2026-08-18); the bullets below are the shape that landed, not the bump itself. Two things are worth knowing:

• MaxBatchSize no longer means what the rig's 16 was chosen to mean. 0.2.0 made batching byte-budgeted — LocalDb:Replication:MaxBatchBytes (2 MB default, left unset here, sized under the 4 MB gRPC cap) bounds the wire message. MaxBatchSize is demoted to a secondary row cap and the DB read-page size in OplogStore.ReadBatchAboveAsync/SnapshotStreamer, which materialise a whole page in memory before the byte split runs. So the site-a pair keeps 16 — it now bounds that transient allocation (≈2.7 MB at ≈171 KB artifact chunk rows), not the message size. Every comment and doc row that called it "row-count-only against gRPC's 4 MB cap" was corrected with the bump; do not restore that rationale.
• Registration FOLLOWS the replication config — it is no longer unconditional (2026-08-18). LocalDbSetup.OnReady reads one predicate, LocalDbReplicationPolicy.CapturesChanges (via LocalDbSetup.ReplicationIsConfigured): LocalDb:Replication:PeerAddress OR LocalDb:Replication:ApiKey is non-blank. The OR is not belt-and-braces — only the dialling half of a pair sets PeerAddress, so keying on it alone would strip capture from every passive node (site-a-2 carries ApiKey + LocalDb:SyncListenPort and nothing else) and the pair would converge in one direction only, silently. ⚠️ SyncListenPort deliberately does NOT join the OR: the passive endpoint is fail-closed on the key, so a port without an ApiKey rejects every inbound stream and has nothing to capture for.
  • ⚠️ The alarm DRAIN gate uses a different predicate — LocalDbReplicationPolicy.SharesRowsWithPeer: ApiKey and (PeerAddress or LocalDb:SyncListenPort > 0), i.e. "a peer actually holds these rows". Capture must be wider than the drain gate (a dialling-only node still captures); a node with no peer must still drain, so the two predicates are deliberately not the same. The passive half (no PeerAddress) cannot name its peer, so it stands down for its mesh partner (the other member of its cluster-{ClusterId} role); a legacy passive half with no mesh role still drains alongside its partner (duplicate historian rows). ⚠️ Before a node has joined its mesh the published Primary verdict (and the live data-plane gate, which must stay the same decision) allows — for seconds, up to ~25 s under the bootstrap guard — so a cold-starting node can briefly co-hold the Sparkplug primary-host claim with its Primary peer.
  • Configured ⇒ RegisterReplicated(table, baselineExistingRows: true), per table, in the same load-bearing DDL→register order. Baselining is what makes enabling replication on a node that has been running without it actually converge: capture is CDC and the snapshot streamer pages from __localdb_row_version, so pre-registration rows were in neither and a late opt-in used to replicate only writes made after the restart. It seeds the ledger at the LWW floor (HLC 0, this node's id) and flags a resync — a baselined row loses to any genuine remote write of the same key and wins only where the peer has no version at all. It is idempotent, so it is free on every boot after the first. As of 0.3.0 (this repo pins 0.3.4) the snapshot flag is per-table, not per-database — baselining one table no longer streams every registered table in both directions. (Prior to 0.3.0, on the 0.2.1 pin this repo carried until 2026-09-03, the flag was per-database and baselining anything streamed every registered table in full; the old advice to trim alarm_sf_events before enabling replication no longer applies at the current pin.)
  • Not configured ⇒ DeregisterReplicated(table) per table, an idempotent boot-time self-heal: it drops the capture triggers an earlier (unconditional) build left in the file and prunes that table's oplog + row-version rows. Merely declining to register would leave every pre-2026-08-18 file capturing forever — two INSERTs plus a json_object of the whole row inside every write transaction — on precisely the node that has no peer to capture for. The call returns true when it removed something, so the Information line fires once, and only when it actually cleaned; a default-OFF node's ordinary boot stays silent. The logger is Serilog's static Log.Logger (this runs inside the AddZbLocalDb singleton factory, where resolving one from DI would nest a resolution inside a singleton construction; Program.cs assigns it before anything resolves ILocalDb), with an optional parameter so tests observe the line rather than infer it.
  • ⚠️ PAIR SYMMETRY: flipping registration is a BOTH-NODES-TOGETHER change. The sync handshake compares registered-table digests fail-closed, so a node that deregisters a table its peer still replicates stops syncing with a schema mismatch. Enabling replication on an established pair means setting the config and restarting both nodes — do it on both, or on neither.
  • Rollout note: the only replicating pair in the fleet is docker-dev site-a, and it was enabled from a fresh pair, so the impact of the old gap was low — central and site-b are default-OFF and simply shed their unused triggers on their next boot. LocalDbSetupTests pins both branches (registered-table set on a configured node, empty set + zero triggers + the HLC-0 baseline ledger entry on the other) and the DDL→register ordering, unchanged.

See docs/Configuration.md (LocalDb section), docs/Redundancy.md (pair-local config cache), docs/AlarmHistorian.md, and the runbook docs/operations/2026-07-20-localdb-pair-replication.md.

Phase 4 — cut the driver-side ConfigDb connection

Per-cluster mesh Phase 4 finished the job Phase 3 started: a driver-only node (Cluster:Roles has driver, not admin) now boots with no ConfigDb connection string at all. Program.cs gates AddOtOpcUaConfigDb (and its health check) on hasAdmin; a fused admin,driver node is unaffected (it keeps ConfigDb via its admin role). Consequences:

• ConfigSource:Mode must be FetchAndCache on a driver-only node — ConfigSourceOptionsValidator fails host start if such a node is left on Direct (there is nothing to read Direct from). A fused node may stay Direct.
• ServiceLevel semantics change — client-visible. With no ConfigDb, DbHealthProbeActor is not spawned (ServiceCollectionExtensions.WithOtOpcUaRuntimeActors skips it when the resolved db factory is null), and OpcUaPublishActor runs dbLess: DbReachable is fed constant true and Stale comes from the redundancy-snapshot age alone. A healthy driver-only node therefore publishes 240 (or 250 as driver Primary) even with central SQL unreachable — the survive-alone posture — where a Direct/DB-backed node in the same state drops to 100/0. ServiceLevelCalculator itself is unchanged; only the inputs a DB-less node feeds it changed. See docs/Redundancy.md §"DB-less (driver-only) nodes".
• Scripted-alarm Part 9 condition state moved to LocalDb. LocalDbAlarmConditionStateStore (replicated alarm_condition_state table, pair-local like the Phase-2 alarm S&F buffer) replaces EfAlarmConditionStateStore on every driver-role node, fused central included. The DB-backed ScriptedAlarmState table was dropped (Task 9, 3a590a0c, migration 20260723183050_DropScriptedAlarmStateTable); EfAlarmConditionStateStore is deleted.
• Central persists deploy acks. DriverHostActor no longer writes NodeDeploymentState to SQL on a driver-only node — ConfigPublishCoordinator.PersistNodeAck (already fed by every ApplyAck) is the system of record.
• LDAP group→role DB grants don't reach driver-only nodes. They register the empty NullLdapGroupRoleMappingService (no ConfigDb to read LdapGroupRoleMapping from), so OtOpcUaGroupRoleMapper falls back to the Security:Ldap:GroupToRole appsettings baseline only — the same rows a driver node never received via config either, so this is not a regression.

Merged to master (26fad75c); live gate PASSED (1281aebf — driver nodes run ConfigDb-free) and Task 9 landed (3a590a0c). See docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md and docs/plans/2026-07-22-per-cluster-mesh-program.md §Phase 4.