Merge branch 'fix/cached-telemetry-drain-hot-loop'

refactor(comm): retire ClusterClient naming after the gRPC cutover
Phase 4 of the ClusterClient -> gRPC migration deleted the Akka transports but left the naming behind: `ClusterClientSiteAuditClient` was transport- agnostic and worked unchanged, so it survived the deletion under a name that now describes a transport the repo no longer has. Same for a scatter of doc-comments still framing gRPC as "the new transport" beside an Akka one that is gone. Renames it to `SiteCommunicationAuditClient` (and its test file) and rewrites the stale comments to describe the single transport that exists. Also tightens CLAUDE.md: drops the self-describing directory listing and the 27-component enumeration in favour of the non-obvious parts only. Behaviour-neutral: names and prose only. Recorded as the Phase 4 follow-up in docs/plans/2026-07-22-clusterclient-to-grpc-plan.md.
2026-07-27 15:50:42 -04:00 · 2026-07-27 15:40:01 -04:00 · 2026-07-24 13:35:27 -04:00 · 2026-07-24 13:35:24 -04:00 · 2026-07-24 13:21:58 -04:00 · 2026-07-24 08:57:44 -04:00
89 changed files with 2537 additions and 353 deletions
@@ -8,29 +8,12 @@ When a change is requested, the default assumption is: update the design doc *an

 ### Top-level directories

- `src/` — C#/.NET implementation, one project per component (e.g. `ZB.MOM.WW.ScadaBridge.AuditLog`, `ZB.MOM.WW.ScadaBridge.NotificationOutbox`, `ZB.MOM.WW.ScadaBridge.SiteCallAudit`, `ZB.MOM.WW.ScadaBridge.CentralUI`, `ZB.MOM.WW.ScadaBridge.Host`, …). Solution file: `ZB.MOM.WW.ScadaBridge.slnx`.
- `tests/` — Test projects (unit + integration).
- `docs/` — Design documentation: `docs/requirements/` (high-level + per-component specs), `docs/test_infra/` (test infrastructure), `docs/plans/` (design-decision and implementation-plan docs). The spec the code implements.
- `docker/` — 8-node cluster topology (2 central + 3 sites), `deploy.sh`, per-node `appsettings.*.json`. See [`docker/README.md`](docker/README.md) for setup, ports, and management commands. Rebuild + redeploy with `bash docker/deploy.sh`.
- `docker-env2/` — Minimal second cluster topology (2 central + 1 site × 2 nodes), runs concurrently with `docker/` on host ports 91XX. Built specifically for testing the Transport (#24) feature with two real environments. See [`docker-env2/README.md`](docker-env2/README.md). Rebuild + redeploy with `bash docker-env2/deploy.sh`.
- `infra/` — Docker Compose for local test services (MS SQL, OPC UA, SMTP, REST API, Traefik). **LDAP is no longer started here** — dev/test LDAP is the shared GLAuth on `10.100.0.35:3893` (source of truth: `scadaproj/infra/glauth/`).
- `deploy/` — Production/on-host deployment artifacts (e.g. `deploy/wonder-app-vd03/`: `appsettings.Central.json`, `appsettings.Site.json`, `install.ps1`/`uninstall.ps1`, `RUNBOOK.md`).
- `deployments/` — Deployment topology notes (`docker-cluster.md`, `docker-cluster-env2.md`, `README.md`).
- `code-reviews/` — Per-component code-review notes (one folder per component, plus `_template`).
- `tools/` — Repo maintenance/utility scripts (e.g. `rename-to-scadabridge.sh`).
- `AkkaDotNet/` — Akka.NET reference documentation and best-practices notes.
- `deprecated/` — Retired docs/notes kept for reference.
- `logs/` — Local runtime log output.
- `vendor/` — Vendored third-party assets (currently an empty placeholder).
- `.claude/` — Claude Code project config (settings, skills, agents).
+Layout is self-describing (`ls`, plus each directory's own README). Only the non-obvious parts are recorded here:

-### Key documents
-
- `README.md` — Master index with component table and architecture diagrams.
- `docs/requirements/HighLevelReqs.md` — Complete high-level requirements covering all functional areas.
- `docs/requirements/Component-*.md` — Individual component design documents (one per component) — the spec the code implements.
- `docs/test_infra/test_infra.md` — Master test infrastructure doc (OPC UA, MS SQL, SMTP, REST API, Traefik). LDAP is the shared GLAuth on `10.100.0.35:3893` (not a local infra container; see `scadaproj/infra/glauth/`).
- `docs/plans/` — Design decision and implementation-plan documents from refinement sessions.
+- `docker/` — the primary 8-node cluster topology (2 central + 3 sites). Rebuild + redeploy with `bash docker/deploy.sh`.
+- `docker-env2/` — minimal second cluster (2 central + 1 site × 2 nodes) that runs **concurrently** with `docker/` on host ports 91XX; built for testing Transport (#24) against two real environments. Rebuild with `bash docker-env2/deploy.sh`.
+- `infra/` — Docker Compose for local test services (MS SQL, OPC UA, SMTP, REST API, Traefik). **LDAP is no longer started here** — dev/test LDAP is the shared GLAuth on `10.100.0.35:3893` (source of truth: `scadaproj/infra/glauth/`), and the same applies to `docs/test_infra/test_infra.md`, which still reads as if it were a local container.
+- `deprecated/` — retired docs/notes kept only for reference; not current.

 ## Sister Projects

@@ -70,35 +53,11 @@ Other peers in the `scadaproj` family (see `scadaproj/CLAUDE.md` for details): `
 - Run tests with `dotnet test ZB.MOM.WW.ScadaBridge.slnx`. `ZB.MOM.WW.ScadaBridge.CLI.Tests` is now a member of the slnx (added 2026-07; arch-review 08 §2.4) — the old "silently skipped" gotcha no longer applies, so a solution-level test run exercises the CLI suite too.
 - **Propagate cross-repo changes to the umbrella index.** This repo is indexed by the parent workspace `~/Desktop/scadaproj`. When a fact its index records changes here — remote/push status, stack/component summary, cross-project wire relationships (OPC UA → OtOpcUa, gRPC → mxaccessgw), or the solution/namespace shape — update the **ScadaBridge entry in [`../scadaproj/CLAUDE.md`](../scadaproj/CLAUDE.md)** in the same change so the umbrella index never drifts from this repo. (Mirrors the same rule in the peer repos, e.g. `MxAccessGateway`/`HistorianGateway`.)

-## Current Component List (27 components)
+## Component List

-1. Template Engine — Template modeling, inheritance, composition, validation, flattening, diffs.
-2. Deployment Manager — Central-side deployment pipeline, system-wide artifact deployment, instance lifecycle.
-3. Site Runtime — Site-side actor hierarchy (Deployment Manager singleton, Instance/Script/Alarm Actors), script compilation, Akka stream.
-4. Data Connection Layer — Protocol abstraction (OPC UA, custom), subscription management, clean data pipe.
-5. Central–Site Communication — Akka.NET ClusterClient (command/control) + gRPC server-streaming (real-time data), message patterns, debug streaming.
-6. Store-and-Forward Engine — Buffering, fixed-interval retry, parking, SQLite persistence, replication.
-7. External System Gateway — External system definitions, API method invocation, database connections.
-8. Notification Service — Central-only notification-list and SMTP definitions, per-type delivery adapters (sites no longer deliver notifications).
-9. Central UI — Web-based management interface, all workflows.
-10. Security & Auth — LDAP/AD authentication, role-based authorization, site-scoped permissions.
-11. Health Monitoring — Site health metrics collection and central reporting.
-12. Site Event Logging — Local operational event logs at sites with central query access.
-13. Cluster Infrastructure — Akka.NET cluster setup, active/standby failover, singleton support.
-14. Inbound API — Web API for external systems, API key auth, script-based implementations.
-15. Host — Single deployable binary, role-based component registration, Akka.NET bootstrap.
-16. Commons — Shared types, POCO entity classes, repository interfaces, message contracts.
-17. Configuration Database — EF Core data access layer, repositories, unit-of-work, audit logging (IAuditService), migrations.
-18. Management Service — Akka.NET actor providing programmatic access to all admin operations, ClusterClientReceptionist registration.
-19. CLI — Command-line tool using HTTP Management API, System.CommandLine, JSON/table output.
-20. Traefik Proxy — Reverse proxy/load balancer fronting central cluster, active node routing via `/health/active`, automatic failover.
-21. Notification Outbox — Central component ingesting store-and-forwarded notifications, `Notifications` audit table, dispatcher loop, retry/parking, delivery KPIs.
-22. Site Call Audit — Central component auditing site cached calls (`CachedCall`/`CachedWrite`); `SiteCalls` audit table, telemetry ingest, reconciliation, KPIs, central→site Retry/Discard relay; sites remain the source of truth.
-23. Audit Log — Central append-only AuditLog table spanning every script-trust-boundary action (outbound API sync+cached, outbound DB sync+cached, notifications, inbound API). Site SQLite hot-path + gRPC telemetry + reconciliation; combined telemetry with Site Call Audit; central direct-write for Notification Outbox dispatch + Inbound API; monthly partitioning, 365-day retention.
-24. Transport — File-based, encrypted bundle export/import via Central UI. Templates (with **all** child collections — attributes/alarms/scripts/compositions/**native-alarm-sources**, field-complete incl. `LockedInDerived` + script cadence/timeout), system artifacts, central-only configuration, plus site/instance-scoped config (`Site`s, site `DataConnection`s, `Instance`s + real `AreaName` by name) reconciled across environments by a `BundleNameMap` name-mapping subsystem. Per-conflict resolution with a per-line Myers diff. Runs the **script trust gate** at import review (5th `ScriptTrustValidator` call site — forbidden-API scripts rejected pre-runtime; covers template/shared/ApiMethod bodies, template script + alarm Expression-trigger bodies, **and instance alarm-override trigger expressions** — all hard-block; template-script name-resolution findings are advisory warnings, ApiMethod findings hard-block). Correlated audit via `BundleImportId`. Never touches site runtime nodes (imported instances land `NotDeployed`).
-25. Script Analysis — Shared authoritative script-trust analyzer: unified forbidden-API deny-list (`ScriptTrustPolicy`), fused semantic+syntactic validator (`ScriptTrustValidator`), Roslyn compile wrapper (`RoslynScriptCompiler`), and compile-only globals stubs (`ScriptCompileSurface`/`TriggerCompileSurface`); consumed by Template Engine, Site Runtime, Inbound API, and Central UI.
-26. KPI History — Reusable central KPI-history backbone: tall/EAV `KpiSample` store in central MS SQL, `KpiHistoryRecorderActor` cluster singleton (`kpi-history-recorder`, not readiness-gated) sampling DI-registered `IKpiSampleSource`s every minute, bucketed query (`GetRawSeriesAsync` + `KpiSeriesBucketer`) + scoped `KpiHistoryQueryService`, and a reusable custom-SVG `KpiTrendChart`; ships trends for Notification Outbox, Site Call Audit, Audit Log, and Site Health.
-27. DelmiaNotifier — Standalone external client tool (NOT a cluster component, NOT in the Host): a compact Native-AOT (`win-x64`) console app (`WWNotifier.exe`) that DELMIA Apriso shells out to per recipe download. POSTs to the Inbound API `DelmiaRecipeDownload` method (`X-API-Key`, key from `SCADABRIDGE_API_KEY`), with connect-failure-only failover across a comma-list of base URLs, and reports the legacy `YES`/`NO` + exit-code stdout contract — a drop-in replacement for the legacy `WWNotifier` (see `docs/former-api-specs/dnc/`). Zero-dependency BCL-only, `System.Text.Json` source-gen. Project README: `src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/README.md`; design: `docs/plans/2026-06-26-delmia-recipe-notifier-design.md`.
+27 components. The catalog — what each one is and its key design commitments — lives in the
+`scadabridge-components` skill (`.claude/skills/scadabridge-components/SKILL.md`); the authoritative
+spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carries the component table.

 ## Key Design Decisions (for context across sessions)

@@ -120,29 +79,26 @@ Other peers in the `scadaproj` family (see `scadaproj/CLAUDE.md` for details): `
 - DCL write failures returned synchronously to calling script.
 - Tag path resolution retried periodically for devices still booting.
 - Static attribute writes persisted to local SQLite (survive restart/failover, reset on redeployment).
- **Consolidated site database (LocalDb Phase 1 + 2, complete 2026-07-20).** **Ten** tables now live in ONE `ZB.MOM.WW.LocalDb`-managed SQLite file — the Phase 1 pair (`OperationTracking`, `site_events`) plus Phase 2's `sf_messages` and the seven site config tables (`deployed_configurations`, `static_attribute_overrides`, `shared_scripts`, `external_systems`, `database_connections`, `data_connection_definitions`, `native_alarm_state`), configured by the **required** `LocalDb:Path` (`/app/data/site-localdb.db` on the rig; validated with `ValidateOnStart`, so a site config missing it fails to boot). Both are `RegisterReplicated` tables. Consequences worth knowing:
+- **Consolidated site database (LocalDb Phase 1 + 2, complete 2026-07-20).** Ten tables now live in ONE `ZB.MOM.WW.LocalDb`-managed SQLite file (enumerate them with `grep -r RegisterReplicated src/`), configured by the **required** `LocalDb:Path` (`/app/data/site-localdb.db` on the rig; validated with `ValidateOnStart`, so a site config missing it fails to boot). Consequences worth knowing:
  - `site_events.id` changed from autoincrement INTEGER to an application-minted **GUID**. Last-writer-wins keys on the primary key, so two nodes independently minting `id=1,2,3…` would destroy each other's events rather than merge them. The event-log read path uses a composite `(timestamp, id)` keyset cursor with an **opaque string** continuation token; `EventLogEntry.Id` and both `ContinuationToken`s are `string`/`string?` on the site↔central Akka DTOs.
  - `ScadaBridge:OperationTracking:ConnectionString`, `ScadaBridge:SiteEventLog:DatabasePath` and — as of Phase 2 — `ScadaBridge:StoreAndForward:SqliteDbPath` + `ScadaBridge:Database:SiteDbPath` are all **migration-only** — nothing reads them but `SiteLocalDbLegacyMigrator`, which copies a pre-Phase-1 file in once (deterministic `mig-{NodeName}-{legacyId}` event ids, `INSERT OR IGNORE`, runs AFTER `RegisterReplicated` so migrated rows replicate) and renames it `.migrated`. Delete the keys once a node has migrated.
  - This incidentally fixes a data-loss bug: both legacy databases defaulted to CWD-relative paths **outside** the mounted volume and were discarded on every container recreate.
  - **Replication is default-OFF and opt-in** via `LocalDb:Replication:PeerAddress` + a matching `ApiKey` on both nodes. `LocalDbSyncAuthInterceptor` is **fail-closed**: no configured key means no sync stream is accepted at all, so a key typo does not degrade to unauthenticated replication — the pair simply stops converging. The sync endpoint shares the existing site gRPC h2c listener (8083); no new port. Rig posture: **site-a replicated, site-b/site-c deliberately not**, so both states are proven side-by-side. Status surfaces on the site health report as `LocalDbReplicationConnected` / `LocalDbOplogBacklog` (both nullable — null means "no data", NOT "disconnected with an empty backlog") and as `localdb_*` Prometheus series. Note `ZbTelemetryOptions.Meters` is an **allowlist** (`SiteServiceRegistration.ObservedMeters`); an unlisted meter exports nothing, silently.
-  - Design: **scadaproj** `docs/plans/2026-07-19-scadabridge-localdb-design.md` (that doc lives in the umbrella repo, not here); Phase 1 plan + Phase 2 gate are here under `docs/plans/`. **Phase 2 is COMPLETE** (branch `feat/localdb-phase2`, live gate PASS 2026-07-20 — all 10 checks, evidence in `docs/plans/2026-07-19-localdb-phase2-live-gate.md`). It moved the config tables + `sf_messages` in and **deleted** `SiteReplicationActor`, its `ReplicationMessages`, StoreAndForward's `ReplicationService`, and `StoreAndForwardStorage.ReplaceAllAsync`. What those did, and why nothing replaced them:
+  - Design: **scadaproj** `docs/plans/2026-07-19-scadabridge-localdb-design.md` (that doc lives in the umbrella repo, not here); Phase 1 plan + Phase 2 gate are here under `docs/plans/`. Phase 2 **deleted** `SiteReplicationActor`, its `ReplicationMessages`, StoreAndForward's `ReplicationService`, and `StoreAndForwardStorage.ReplaceAllAsync` — do not reintroduce them:

-    - `SiteReplicationActor` pushed config deploys to the peer and ran a **notify-and-fetch** exchange (tell the standby a deploy happened; it HTTP-fetches the config itself, with retries and a superseded-404 path). Config rows now simply replicate — **the standby makes no fetch at all** during a deploy (verified live). `SiteReconciliationActor` still fetches at node STARTUP when central reports gaps; that path survives and is a different thing.
-    - `ReplicationService` fanned each buffer mutation (add/remove/park/requeue) to the standby by hand. CDC triggers on `sf_messages` do it now.
-    - `ReplaceAllAsync` was a destructive delete-all-then-insert-all resync. It was not merely unused after the cutover but **unsafe to keep**: a mass DELETE on a now-replicated table would be captured and shipped to the peer. LocalDb's snapshot resync merges per row under LWW and never deletes, which is also why the old N1 directional-authority guard is gone — there is no wipe left to gate.
+    - CDC replication does all three jobs now: config deploys reach the standby as ordinary row changes — **the standby makes no fetch at all** during a deploy (`SiteReconciliationActor`'s node-STARTUP fetch when central reports gaps is a different, surviving path) — and buffer mutations replicate via triggers on `sf_messages`. `ReplaceAllAsync` was a destructive delete-all-then-insert-all resync and is **unsafe to reintroduce**: a mass DELETE on a replicated table would be captured and shipped to the peer. LocalDb's snapshot resync merges per row under LWW and never deletes, which is also why the old N1 directional-authority guard is gone — there is no wipe left to gate.
    - **`notification_lists` and `smtp_configurations` are created but deliberately NOT registered.** They are permanently empty on a site (no writer since 2026-07-10, the migrator skips them, the active-node purge keeps them empty), and registering them would open a standing replication channel whose only historical payload was plaintext SMTP passwords. Pinned by a security-named test, and verified live: those two tables have **no CDC triggers** on either rig node.
    - **Operational constraints (read before upgrading a site pair):** stop and start both nodes TOGETHER — rolling one at a time is no longer supported, since the legacy `SfBufferSnapshot` compatibility handler went with the replicator. And a node offline longer than `LocalDb:Replication:TombstoneRetention` (default 7 days) can resurrect deleted rows on rejoin. See `docs/deployment/topology-guide.md`.
    - `LocalDb:Replication:MaxBatchSize` batches by ROW COUNT, not bytes, against a 4 MB gRPC cap — the rig pins it to **16** (~70 KB worst-case `config_json` x 16 ~= 1.1 MB). The 500 default would allow ~35 MB.
 - All timestamps are UTC throughout the system.
 - Inter-cluster communication uses **three** transports, not two — **all cross-cluster command/control and data now rides gRPC** after the ClusterClient→gRPC migration's Phase 4 (`docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`) deleted Akka `ClusterClient`/`ClusterClientReceptionist`: (1) **gRPC command/control** — site→central over the central-hosted `CentralControlService` (`GrpcCentralTransport`, sticky central-a→central-b channel pair; deployments/notifications/health/heartbeat/audit-ingest/reconcile), and central→site over the site-hosted `SiteCommandService` (`GrpcSiteTransport`, per-site NodeA→NodeB channel pair; the 28 lifecycle/OPC-UA/query/parked/route/failover commands); (2) **gRPC** server-streaming for real-time data (attribute values, alarm states, `SiteStreamService`); and (3) **plain token-gated HTTP** for the deployment config itself — notify-and-fetch, the site pulls the config from `DeploymentConfigEndpoints` (`ManagementService/DeploymentConfigEndpoints.cs`) with an `X-Deployment-Token` header, `AllowAnonymous` with the per-deployment token as the entire security boundary. The gRPC boundary is per-site PSK-authenticated (`ControlPlaneAuthInterceptor`, unchanged). There is **no receptionist registration** — discovery is by dialling configured endpoints; central builds one `SitePairChannelProvider` per site (addresses from `Site.GrpcNodeAAddress`/`GrpcNodeBAddress`, refreshed from the DB every 60s and on admin changes), sites dial `ScadaBridge:Communication:CentralGrpcEndpoints` (both central nodes, h2c on `CentralGrpcPort` 8083, **NOT** via Traefik). **Discovery is asymmetric by design:** central discovers site gRPC addresses from the *database* (refreshable at runtime), sites discover central from *appsettings* (`CentralGrpcEndpoints`, static — restart required; `StartupValidator` requires a Site node to list at least one). `Akka.Cluster.Tools` stays for ClusterSingleton; only the ClusterClient part is gone. **Central never buffers for an unreachable site** — the send fails with the caller's Ask/deadline timing out; a `ConnectionStateChanged` mechanism built for this was deleted as dead code.
- **All clusters share ONE ActorSystem name**, `"scadabridge"` — hardcoded at `AkkaHostedService.cs:191`. Central and each site are separate clusters *only* by seed-node partitioning. This is required, not incidental: Akka.Remote address matching means a ClusterClient could not reach a differently-named system.
+- **All clusters share ONE ActorSystem name**, `"scadabridge"` — hardcoded in `AkkaHostedService` at the `ActorSystem.Create` call. Central and each site are separate clusters *only* by seed-node partitioning. The constraint originated with ClusterClient (Akka.Remote address matching meant it could not reach a differently-named system); whether it is still load-bearing after the gRPC migration has **not** been re-verified, so treat the name as fixed until someone checks.
 - **`ActiveNodeEvaluator.SelfIsOldestUp` is THE single definition of "active node"** (`Communication/ClusterState/ActiveNodeEvaluator.cs`) — the **oldest Up member** in a role scope, and explicitly **never `cluster.State.Leader`**: leadership (lowest address) is an Akka-internal concept that diverges from singleton placement permanently once the original first node restarts and rejoins, and both sides claim it during a partition. The equivalence *oldest-Up == where `ClusterSingletonManager` places singletons* **is** the design. `ClusterActivityEvaluator.SelfIsOldest`, the S&F delivery gate, `/health/active` and the heartbeat `IsActive` stamp all delegate here.
- Site nodes carry **two Akka roles**: the base `Site` plus a site-specific `site-{SiteId}` (`BuildRoles`, `AkkaHostedService.cs:386`). Singletons scope to the **site-specific** role.
- **The gRPC boundary is authenticated (PSK) as of 2026-07-22; Akka remoting still is not, and nothing is encrypted.** Akka remoting sets no `enable-ssl`, no secure cookie, no `trusted-selection-paths` — so the ClusterClient command/control path remains open to anyone who can reach the remoting port, and the boundary still assumes a trusted network. The gRPC listener stays **h2c**, but `SiteStreamService` is no longer open: `ControlPlaneAuthInterceptor` (`Host/ControlPlaneAuthInterceptor.cs`) gates `/sitestream.SiteStreamService/` — including the `PullAuditEvents`/`PullSiteCalls` RPCs that return audit rows — against a **per-site preshared key**, fail-closed, constant-time compared, alongside the separate `LocalDbSyncAuthInterceptor` on `/localdb_sync.v1.LocalDbSync/` with its own separate key. **Site side:** `ScadaBridge:Communication:GrpcPsk`, in production `${secret:SB-GRPC-PSK-<siteId>}`, and **`StartupValidator` refuses to boot a site node without it** (an unset key would leave the node healthy-looking but serving nothing). **Central side:** `SitePskProvider` resolves `SB-GRPC-PSK-{siteId}` from the secrets store at channel-build time (sites are added at runtime, so no boot-time expansion is possible), with `ScadaBridge:Communication:SitePsks:{siteId}` as an override for hosts running without a master key — the docker rig uses the latter. One key per site, never fleet-wide. A bearer token over h2c is readable and replayable on-path; TLS is the follow-on hardening and needs no change to this design. Introduced by Phase 0 of the ClusterClient→gRPC migration (`docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`).
- **Akka frame size is the default 128 KB with `log-frame-size-exceeding` off**, and no custom serializer is configured — so payload carried over ClusterClient is JSON-escaped a second time by the default Newtonsoft serializer, roughly doubling it. Over the limit the transport drops **that one message** without tearing down the association (heartbeats keep flowing, the site still reports healthy) and the central Ask simply times out. See `docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md`; `DeployArtifactsCommand` still carries payload and remains exposed.
- gRPC streaming channel — **note the direction is inverted from the data flow**: data moves site→central, but each **site node hosts the gRPC server** (`SiteStreamGrpcServer`, Kestrel h2c, port 8083, mapped **only in the Site branch** of `Program.cs`) and **central is the client**, dialling in. There is **no gRPC server on central at all**, which is why the two `Ingest*` unary RPCs — documented as a "central-side ingest surface" — are dead in practice (acknowledged in `AkkaHostedService.cs:510-519`); sites reach central over ClusterClient instead. Central creates per-site `SiteStreamGrpcClient` via `SiteStreamGrpcClientFactory`, keyed **`(siteId, endpoint)`** — the key was widened from site-only to fix an arch-review High where one session's NodeA→NodeB flip disposed a channel another session was still using. Site entity has GrpcNodeAAddress/GrpcNodeBAddress fields. Proto: `sitestream.proto`, **6 RPCs** (2 server-streaming: `SubscribeInstance`, `SubscribeSite` (site-wide, alarm-only); 4 unary: `IngestAuditEvents`, `IngestCachedTelemetry`, `PullAuditEvents`, `PullSiteCalls`), `SiteStreamEvent` (oneof: AttributeValueUpdate, AlarmStateUpdate). Field numbers are never reused; evolution is additive only (`AlarmStateUpdate` grew 7→23 fields for the native-alarm mirror). Generated C# is **vendored** under `Communication/SiteStreamGrpc/` with the `<Protobuf>` include commented out — regeneration is a manual toggle-build-copy-untoggle. DebugStreamEvent message removed (no longer flows through ClusterClient).
- Native alarms: a read-only mirror of native alarms from OPC UA Alarms & Conditions servers and the MxAccess Gateway, unified onto an A&C-style condition model (`AlarmConditionState`: orthogonal Active/Acked/Confirmed/Shelved/Suppressed + 0–1000 severity) plus an `AlarmKind` discriminator (Computed/NativeOpcUa/NativeMxAccess). New DCL capability seam `IAlarmSubscribableConnection` (implemented by the OPC UA and MxGateway adapters); the `DataConnectionActor` opens ONE alarm feed per connection and routes transitions to instances by source-object reference. A `NativeAlarmActor` (peer to the computed `AlarmActor` under `InstanceActor`) mirrors one source binding: snapshot atomic-swap on (re)subscribe, retention (drops once inactive+acked), per-source cap, and site SQLite persistence (`native_alarm_state`, survives failover, cleared on redeploy/undeploy — mirrors static overrides). State streams to central over the additively-enriched gRPC `AlarmStateUpdate` (the existing computed `AlarmStateChanged` was enriched additively) and seeds via the DebugView snapshot. Authoring: `TemplateNativeAlarmSource` / `InstanceNativeAlarmSourceOverride` entities flatten to `ResolvedNativeAlarmSource` (inherit/compose/override); management commands + ManagementActor handlers + CLI (`template/instance native-alarm-source`) + Central UI (template editor tab + instance override panel) + enriched DebugView alarm table. Read-only — no ack-back; no central tables.
- OPC UA / MxGateway UX (M7): operator **Alarm Summary** page (`/monitoring/alarms`, RequireDeployment, read-only) fans out the existing per-instance `DebugViewSnapshot` Ask (SemaphoreSlim-capped, partial-results tolerant) and aggregates client-side — no central alarm store; shared `AlarmStateBadges` component. **Aggregated live stream shipped 2026-07-10** (`docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.md`): a transient in-memory per-site central live cache (`ISiteAlarmLiveCache`) fed by a site-wide, alarm-only `SubscribeSite` gRPC stream (seed-then-stream), pushing near-real-time deltas to the page over the Blazor circuit with the 15s poll kept as fallback + NotReporting authority — still no persisted central alarm store. OPC UA node browser gains `BrowseNext` continuation paging ("Load more"), a bounded recursive address-space **search** (`IAddressSpaceSearchable` seam; depth + result caps; substring on DisplayName/path), and **type-info** (DataType/ValueRank/Writable on `BrowseNode` for Variables). Attribute-override **CSV bulk import** (`OverrideCsvParser`, all-or-nothing) via InstanceConfigure `InputFile` + CLI `instance import-overrides --file` (native-alarm-source-override CSV deferred). **Verify-endpoint** probe (temporary `RealOpcUaClient`, short timeout, captures an untrusted server cert but NEVER trusts it) + **site-local cert trust**: per-node `CertStoreActor` (runs on every site node, not a singleton) writing the `.der` into the node's OPC UA trusted-peer PKI store; DeploymentManager broadcasts `TrustServerCertCommand`/`RemoveServerCertCommand` to BOTH site nodes so PKI stores stay consistent across failover; Admin-gated cert-management UI (`/design/connections/{id}/certificates`). No central persistence of cert trust (follow-up).
+- Site nodes carry **two Akka roles**: the base `Site` plus a site-specific `site-{SiteId}` (`AkkaHostedService.BuildRoles`). Singletons scope to the **site-specific** role.
+- **The gRPC boundary is authenticated (PSK) as of 2026-07-22; Akka remoting still is not, and nothing is encrypted.** Akka remoting sets no `enable-ssl`, no secure cookie, no `trusted-selection-paths` — so intra-cluster Akka remoting remains open to anyone who can reach the remoting port, and that boundary still assumes a trusted network. The gRPC listener stays **h2c**, but `SiteStreamService` is no longer open: `ControlPlaneAuthInterceptor` (`Host/ControlPlaneAuthInterceptor.cs`) gates `/sitestream.SiteStreamService/` — including the `PullAuditEvents`/`PullSiteCalls` RPCs that return audit rows — against a **per-site preshared key**, fail-closed, constant-time compared, alongside the separate `LocalDbSyncAuthInterceptor` on `/localdb_sync.v1.LocalDbSync/` with its own separate key. **Site side:** `ScadaBridge:Communication:GrpcPsk`, in production `${secret:SB-GRPC-PSK-<siteId>}`, and **`StartupValidator` refuses to boot a site node without it** (an unset key would leave the node healthy-looking but serving nothing). **Central side:** `SitePskProvider` resolves `SB-GRPC-PSK-{siteId}` from the secrets store at channel-build time (sites are added at runtime, so no boot-time expansion is possible), with `ScadaBridge:Communication:SitePsks:{siteId}` as an override for hosts running without a master key — the docker rig uses the latter. One key per site, never fleet-wide. A bearer token over h2c is readable and replayable on-path; TLS is the follow-on hardening and needs no change to this design. Introduced by Phase 0 of the ClusterClient→gRPC migration (`docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`).
+- gRPC streaming channel — **note the direction is inverted from the data flow**: data moves site→central, but each **site node hosts the gRPC server** (`SiteStreamGrpcServer`, Kestrel h2c, port 8083, mapped **only in the Site branch** of `Program.cs`) and **central is the client**, dialling in. Central creates per-site `SiteStreamGrpcClient` via `SiteStreamGrpcClientFactory`, keyed **`(siteId, endpoint)`** — the key was widened from site-only to fix an arch-review High where one session's NodeA→NodeB flip disposed a channel another session was still using. Proto evolution is **additive only** and field numbers are never reused (`AlarmStateUpdate` grew 7→23 fields for the native-alarm mirror). Generated C# is **vendored** under `Communication/SiteStreamGrpc/` with the `<Protobuf>` include commented out — regeneration is a manual toggle-build-copy-untoggle.
+- Native alarms are a **read-only** mirror of OPC UA Alarms & Conditions and MxAccess Gateway alarms — **no ack-back, no central tables**; state lives in the site's `native_alarm_state`, survives failover, and is cleared on redeploy/undeploy (mirrors static overrides). Central's per-site live alarm cache (`ISiteAlarmLiveCache`) is **transient in-memory only** — there is deliberately no persisted central alarm store, so the 15s poll remains the NotReporting authority behind the live stream. See `Component-DataConnectionLayer.md` / `Component-CentralUI.md` for the model and the authoring surface.
+- OPC UA cert trust is **site-local and not persisted centrally** (follow-up): the verify-endpoint probe captures an untrusted server cert but **NEVER trusts it**, and DeploymentManager broadcasts `TrustServerCertCommand`/`RemoveServerCertCommand` to **BOTH** site nodes — `CertStoreActor` runs on every site node, not as a singleton, so PKI stores stay consistent across failover.

 ### External Integrations
 - External System Gateway: HTTP/REST only, JSON serialization, API key + Basic Auth.
@@ -218,13 +174,14 @@ Other peers in the `scadaproj` family (see `scadaproj/CLAUDE.md` for details): `
 - Failure detection: 2s heartbeat, 10s threshold. Total failover ~25s (drill-measured 2026-07-21 under auto-down: active-crash TAKEOVER in 28s, standby-crash removal in 27s with 0 routing blips — `docker/failover-drill.sh`).
 - CoordinatedShutdown for graceful singleton handover.
 - Automatic dual-node recovery from persistent storage.
- **Active/standby is decided by `ActiveNodeEvaluator.SelfIsOldestUp`, never by cluster leadership** — see the Architecture note above. `/health/active` is **central-only** (site nodes map no `/health/*` at all) and backs both Traefik's active-node routing and `IActiveNodeGate`, so the proxy and the Inbound API always agree on which node is active. Central never needs to know which *site* node is active: ClusterClient contact rotation reaches either receptionist and the site-internal `ClusterSingletonProxy` lands the work on the active node for free. The **exception is gRPC**, which picks `GrpcNodeAAddress`/`GrpcNodeBAddress` explicitly and flips on error.
+- **Active/standby is decided by `ActiveNodeEvaluator.SelfIsOldestUp`, never by cluster leadership** — see the Architecture note above. `/health/active` is **central-only** (site nodes map no `/health/*` at all) and backs both Traefik's active-node routing and `IActiveNodeGate`, so the proxy and the Inbound API always agree on which node is active. Central reaches a site by dialling `GrpcNodeAAddress`/`GrpcNodeBAddress` explicitly and flipping on error; whichever node answers, the site-internal `ClusterSingletonProxy` lands the work on the active node for free, so central still never needs to track which *site* node is active.
 - **Seed-node ordering: every node lists ITSELF first (decision 2026-07-22) — the boot-alone gap is CLOSED.** Only `seed-nodes[0]` may self-join to form a new cluster (Akka runs `FirstSeedNodeProcess` for it, `JoinSeedNodeProcess` — which can never form one — for everyone else). All 14 shipped node appsettings now lead with the node's own address, so any node can cold-start alone and become operational unattended (~5s, `seed-node-timeout`); `StartupValidator` fails the boot if the ordering is broken (compares host AND port; Akka does no DNS canonicalisation). Two nodes cold-starting together while mutually reachable converge on ONE cluster via the `InitJoin` handshake — they split only under a genuine boot-time partition, the same class auto-down accepts. **An external self-form timer (`Cluster.Join(SelfAddress)` after a window) was implemented and REJECTED:** it sits outside the join handshake, so on a routine standby restart — where the peer is alive but the join is stalled behind removal of the node's own stale incarnation — it fires mid-join and permanently splits the pair (measured: still split after 90s). Regression tests: `SelfFirstSeedBootstrapTests`. The keep-oldest active-crash total outage was separately closed by the auto-down decision. See `docs/requirements/Component-ClusterInfrastructure.md` → Seed Node Ordering.
+- **Simultaneous-cold-start split-brain guard (opt-in, Gitea #33) — the residual self-first cost, closed.** Self-first-on-both means a *truly simultaneous* cold start (shared power/hypervisor event) races `FirstSeedNodeProcess` on BOTH nodes → two 1-node clusters that never merge (in-process loopback tests converge and hide it; real parallel VM starts don't — OtOpcUa reproduced it live). Ported from OtOpcUa (`lmxopcua` `d1dac87f`): a **dark switch `ScadaBridge:Cluster:BootstrapGuard:Enabled` (default OFF, guard-off byte-identical)**. On: `BuildHocon` emits an EMPTY seed list (Akka does not auto-join) and `ClusterBootstrapCoordinator` (`IHostedService`, registered in BOTH the Central and Site composition roots) issues ONE reachability-gated `JoinSeedNodes` — the pure `ClusterBootstrapGuard` core makes the lower canonical `host:port` the **founder** (self-first, forms immediately), the higher node TCP-probes the founder up to `PartnerProbeSeconds` (25s) and joins **peer-first** if reachable else **self-first** (cold-start-alone preserved). Decided BEFORE a single join, **never re-forms mid-handshake** (that was the rejected self-form-timer's flaw); case-insensitive founder tie-break; probe timings validated `>0` when enabled. Residual accepted trade: founder dies in the probe→join window → higher node hangs unjoined, coordinator WARNs, a restart recovers it. Tests: `ClusterBootstrapGuardTests` (pure) + `ClusterBootstrapCoordinatorTests` (real-ActorSystem, incl. both-cold-start-together-form-one-cluster). See `docs/requirements/Component-ClusterInfrastructure.md` → Simultaneous cold start.

 ### UI & Monitoring
 - Central UI: Blazor Server (ASP.NET Core + SignalR) with Bootstrap CSS. No third-party component frameworks (no Blazorise, MudBlazor, Radzen, etc.). Build custom Blazor components for tables, grids, forms, etc.
 - UI design: Clean, corporate, internal-use aesthetic. Not flashy. Use the `frontend-design` skill when designing UI pages/components.
- Debug view: real-time streaming via DebugStreamBridgeActor + gRPC (events via SiteStreamGrpcClient, snapshot via ClusterClient). Health dashboard: 10s polling timer. Deployment status: real-time push via SignalR.
+- Debug view: real-time streaming via DebugStreamBridgeActor + gRPC (events via `SiteStreamGrpcClient`, snapshot via the `SiteCommandService` query surface — `QueryReply.DebugViewSnapshot`). Health dashboard: 10s polling timer. Deployment status: real-time push via SignalR.
 - Health reports: 30s interval, 60s offline threshold, monotonic sequence numbers, raw error counts per interval.
 - Dead letter monitoring as a health metric.
 - Site Event Logging: 30-day retention, 1GB storage cap, daily purge, paginated queries with keyword search.
@@ -253,14 +210,9 @@ Other peers in the `scadaproj` family (see `scadaproj/CLAUDE.md` for details): `

 ## Tool Usage

- When consulting with the Codex MCP tool, use model `gpt-5.4`.
 - When a task requires setting up or controlling system state (sites, templates, instances, data connections, deployments, security, etc.) and the Central UI is not needed, prefer the ScadaBridge CLI over manual DB edits or UI navigation. See [`src/ZB.MOM.WW.ScadaBridge.CLI/README.md`](src/ZB.MOM.WW.ScadaBridge.CLI/README.md) for the full command reference.

 ### CLI Quick Reference (Docker / OrbStack)

- **Management URL**: `http://localhost:9000` — the CLI connects via the Traefik load balancer, which routes to the active central node. Direct access: central-a on port 9001, central-b on port 9002.
- **Test user**: `--username multi-role --password password` — has Admin, Design, and Deployment roles. The `admin` user only has the Admin role and cannot create templates, data connections, or deploy.
- **Config file**: `~/.scadabridge/config.json` — stores `managementUrl` and default format. See `docker/README.md` for a ready-to-use test config.
- **Rebuild cluster**: `bash docker/deploy.sh` — builds the `scadabridge:latest` image and recreates all containers. Run this after code changes to ManagementActor, Host, or any server-side component.
- **Infrastructure services**: `cd infra && docker compose up -d` — starts MS SQL, OPC UA, SMTP, and REST API. These are separate from the cluster containers in `docker/`. **LDAP is NOT started here** — it is the shared GLAuth on `10.100.0.35:3893` (dc=zb,dc=local); source of truth and config: `scadaproj/infra/glauth/`.
- **All test LDAP passwords**: `password` (see `scadaproj/infra/glauth/config.toml` for users and groups; canonical cross-app login: `multi-role`).
+Management URL, test credentials, rebuild/redeploy commands and the infra services are in the
+`scadabridge-cluster-ops` skill (`.claude/skills/scadabridge-cluster-ops/SKILL.md`).
@@ -90,9 +90,9 @@
      to mark tests as Skipped (not silently Passed) when MSSQL is unreachable.
    -->
    <PackageVersion Include="Xunit.SkippableFact" Version="1.5.61" />
-    <PackageVersion Include="ZB.MOM.WW.Health" Version="0.1.0" />
-    <PackageVersion Include="ZB.MOM.WW.Health.Akka" Version="0.1.0" />
-    <PackageVersion Include="ZB.MOM.WW.Health.EntityFrameworkCore" Version="0.1.0" />
+    <PackageVersion Include="ZB.MOM.WW.Health" Version="0.3.0" />
+    <PackageVersion Include="ZB.MOM.WW.Health.Akka" Version="0.3.0" />
+    <PackageVersion Include="ZB.MOM.WW.Health.EntityFrameworkCore" Version="0.3.0" />
    <PackageVersion Include="ZB.MOM.WW.Telemetry" Version="0.1.0" />
    <PackageVersion Include="ZB.MOM.WW.Telemetry.Serilog" Version="0.1.0" />
    <PackageVersion Include="ZB.MOM.WW.MxGateway.Client" Version="0.1.1" />
@@ -1,8 +1,55 @@
 # Cached-telemetry drain hot-loops forever on a row whose tracking snapshot is gone

-**Date:** 2026-07-20 · **Status:** OPEN · **Severity:** Medium (log flood + wasted I/O; no data loss)
+**Date:** 2026-07-20 · **Status:** RESOLVED (2026-07-20) · **Severity:** High — revised up from Medium on investigation (permanent stall of the cached-telemetry path, not just a log flood)
 · **Area:** AuditLog / Site Telemetry

+## Resolution (2026-07-20)
+
+Fixed by letting an unresolvable row LEAVE the queue.
+`SiteAuditTelemetryActor.OnCachedDrainAsync` now abandons the operational half of a cached row
+whose tracking snapshot is still unresolvable after a grace period
+(`SiteAuditTelemetryOptions.CachedTrackingGraceSeconds`, default **300 s**) and marks it
+`Forwarded`. A row with no `CorrelationId` at all is abandoned immediately — it can never resolve.
+
+**Marking `Forwarded` does not drop the audit data.** That state's role in this machine is "no
+longer owed by the drain, still eligible for reconciliation", which is exactly the situation:
+`ISiteAuditQueue.ReadPendingSinceAsync` covers `Forwarded` rows as well as `Pending` ones and
+central dedups on `EventId`, so the reconciliation pull still delivers them. What is genuinely lost
+is the operational (`SiteCalls`) half — unrecoverable regardless, since the tracking row it would
+have been built from no longer exists.
+
+Three further behaviours, each deliberate:
+
+- **Inside the grace window the row is still retried.** A missing snapshot is normally a brief
+  write race (the audit row lands microseconds before the tracking row); abandoning on the first
+  failed lookup would discard the operational half of every cached call that lost that race.
+- **A tracking-store THROW never abandons.** A throw is a store fault (locked, corrupt,
+  mid-restore), not a verdict about the row. Those rows stay `Pending` however old they are.
+- **Logging is per drain pass, not per row.** One summary line each for abandoned, lookup-failed
+  (with the first exception attached) and deferred rows. The deferred case dropped to Debug — being
+  inside the grace window is normal operation, not a warning.
+
+### Severity was revised UP during the fix
+
+The original write-up called this a log flood with "no data loss", which understated it. The queue
+is read **oldest-first with a fixed `BatchSize` (default 256)**, so once a batch's worth of
+permanently-unresolvable rows collects at the head, every drain re-reads exactly those rows, fails
+identically, and **never reaches the newer rows behind them**. The cached-telemetry path stalls
+permanently. The audit half still reached central by reconciliation, so "no data loss" held — but
+"Medium" did not.
+
+**Fixed in:** `SiteAuditTelemetryActor.cs` (`AbandonUnresolvableAsync` + the rewritten skip
+branches), `SiteAuditTelemetryOptions.cs` (`CachedTrackingGraceSeconds`).
+**Tests:** `SiteAuditTelemetryActorTests` — `CachedDrain_OrphanRow_PastGrace_IsAbandoned_...`,
+`..._InsideGrace_IsRetried_NotAbandoned`, `..._FullBatchOfUnresolvableRows_DoesNotStarveTheQueue`,
+`..._TrackingStoreThrow_DoesNotAbandonTheRow`. The pre-existing
+`CachedDrain_OrphanRow_NoTrackingSnapshot_IsSkipped_DoesNotCrash` was **retargeted, not deleted** —
+it had pinned the defect ("skipped and stays Pending"), so its orphan assertion is inverted while
+the half that still holds is kept verbatim. Verified non-vacuous: with abandonment disabled the two
+abandonment tests go red; the two must-NOT-abandon guards stay green in both directions.
+
+The diagnosis below is retained as the record of how it was found.
+
 ## Summary

 `SiteAuditTelemetryActor`'s cached-telemetry drain reads Pending audit rows, looks up each row's
@@ -0,0 +1,64 @@
+# OtOpcUa v3 native-alarm B/C live-gate (Gitea #14) — PASS
+
+**Date:** 2026-07-23 · **Scope:** items **B** (native-alarm dedup) + **C** (UNS-bound alarm routing) of #14,
+against a UNS structure + scripted Part-9 alarm authored on the live `otopcua-dev` v3 cluster (see
+`2026-07-23-otopcua-v3-raw-path-live-gate.md` for the connection + `SiteAOnly` substrate). Instrument:
+ScadaBridge itself (its DCL A&C client is the thing under test).
+
+## Headline
+
+**Both B and C PASS, and item C's original premise turned out to be already-solved code.** No ScadaBridge
+code change is needed for B/C. The scope-doc worry — "a UNS-bound source won't `StartsWith`-match a
+condition whose `SourceName` is the RawPath/ScriptedAlarmId" — is **obsolete**: ScadaBridge routes native
+alarms by the **subscribed node reference**, not the event `SourceName` (deliberate fix, Gitea #17). The
+only remaining #14 work is the item-A data re-author (greenfield, operational).
+
+## Substrate (on OtOpcUa, SITE-A cluster)
+
+- UNS: `area-1/line-1/pump01` → signal `SiteAOnly` (equipment folder node `ns=3;s=EQ-54f7711e160d`,
+  which has **`EventNotifier=1`**; signal `ns=3;s=area-1/line-1/pump01/SiteAOnly` mirrors the raw value).
+- Scripted Part-9 condition `ns=3;s=SA-siteahigh` (a `HasComponent` child of the equipment folder),
+  `AlarmConditionType`, severity 700, predicate driven by `SiteAOnly` (set always-active for the gate).
+- `Server` (i=2253) `EventNotifier=1` (the connection-wide aggregate).
+
+## What ScadaBridge did (bindings on template 2148, instance 98, connection `OtOpcUa-v3-raw`)
+
+Two `TemplateNativeAlarmSource` bindings, deliberately overlapping to stress dedup:
+- `ServerAlarms` → `--source-ref i=2253` (Server aggregate)
+- `UnsEquipAlarms` → `--source-ref nsu=https://zb.com/otopcua/uns;s=EQ-54f7711e160d` (UNS equipment node)
+
+## Findings / checks
+
+| # | Check | Result |
+|---|-------|--------|
+| C1 | A **UNS-node-scoped** binding receives + routes the native alarm | **PASS** — `UnsEquipAlarms` mirrored `SA-siteahigh.SiteA High` active=True, sev 700, `kind=NativeOpcUa` |
+| C2 | The condition's `SourceName` = **ScriptedAlarmId** (`SA-siteahigh`), NOT RawPath | **Confirmed** (captured live) — and it does **not** break routing (routing keys off the subscribed node, Gitea #17), only feeds the alarm's identity (`alarmName = SA-siteahigh.SiteA High`) |
+| C3 | Item C's original "SourceName mismatch drops UNS-bound transitions" premise | **Obsolete** — `OpcUaAlarmMapper.BuildIdentity` stamps `SourceObjectReference` = the subscription ref; `DataConnectionActor` routes on that, not `SourceName` |
+| B1 | Fanned condition (Server aggregate + equipment-folder notifier) mirrored **once**, not duplicated | **PASS** — active condition appears exactly once (on the more-specific equipment feed); the Server binding stays an inactive placeholder. Stable across repeated snapshots |
+| B2 | Mechanism | ScadaBridge opens **one** OPC UA `Subscription` per connection with one `MonitoredItem` per source-ref; `ConditionRefresh` delivers each retained condition **once per subscription**, to the most-specific monitored notifier (the equipment folder) — so overlapping bindings do not double-mirror the fan-out |
+| B3 | Residual double-mirror risk | Latent, **not triggered** here: it needs two bindings whose `SourceReference` strings are literal prefixes of one another AND a delivery matching both (`DataConnectionActor` has no cross-binding, per-ConditionId dedup — `InstanceActor._latestAlarmEvents` is last-writer-wins by `alarmName`). Disjoint node refs (as here) don't collide |
+
+## Captured real payload (what §5 phase 3 wanted)
+
+`kind=NativeOpcUa`, `alarmName='SA-siteahigh.SiteA High'` (`SourceName`=`SA-siteahigh` = ScriptedAlarmId,
+`.` + `ConditionName`=`SiteA High`), `alarmTypeName='AlarmConditionType'`, `severity=700`,
+`condition.active=True`. A&C state is delivered via events/`ConditionRefresh` (ScadaBridge triggers a
+refresh on every subscribe), not a plain attribute read.
+
+## Bottom line for #14
+
+- **A** — raw/UNS binding re-author: validated end-to-end (raw-path gate proved `nsu=` bind→read; this gate
+  proved UNS bind→alarm). Remaining work is data re-authoring (greenfield), **no code**.
+- **B** — dedup: **PASS**, no double-mirror across the v3 fan-out (one subscription + `ConditionRefresh`).
+- **C** — UNS-bound routing: **PASS**; the original concern was already solved by Gitea #17.
+- **D/E/F** — shipped in phase 2 (`2026-07-23-otopcua-v3-nsu-hardening-and-browse-ux.md`).
+
+**#14 needs no further ScadaBridge code for B/C.** The only shipped code from this whole effort was the
+phase-2 `nsu=` hardening + the `RealOpcUaClient` host-rewrite fix (surfaced by the raw gate). #14 can close
+once item A's re-author is done in the target environment(s).
+
+## Rig artifacts (removable on next `docker/deploy.sh`)
+
+OtOpcUa ConfigDb: `UnsArea SITEA-area1` / `UnsLine SITEA-line1` / `Equipment EQ-54f7711e160d` /
+`UnsTagReference UTR-siteapump01-siteaonly` / `Script SC-siteahigh` / `ScriptedAlarm SA-siteahigh`.
+ScadaBridge: template 2148 native-alarm-sources `ServerAlarms` + `UnsEquipAlarms`; instance 98; connection 3044.
@@ -0,0 +1,79 @@
+# OtOpcUa v3 raw-path live-gate (Gitea #14) — PASS
+
+**Date:** 2026-07-23 · **Scope:** the raw-subtree slice of #14 (decision: "raw gate now, flag B/C").
+**Rig:** ScadaBridge docker rig (`scadabridge:latest`, rebuilt with the host-rewrite fix) against the
+**live `otopcua-dev` v3 cluster** (`otopcua-host:dev`, built 2026-07-23, branch `feat/mesh-phase5`,
+past the v3 merge `ec6598ce`). Related: scope doc `2026-07-23-otopcua-v3-dual-namespace-cutover-scope.md`,
+phase-2 plan `2026-07-23-otopcua-v3-nsu-hardening-and-browse-ux.md`.
+
+## Headline
+
+ScadaBridge is confirmed **structurally v3-safe against a real dual-namespace server** — durable
+`nsu=` references are emitted by browse, stored on bindings, and resolved live against the server's
+`NamespaceArray` to Good-quality reads. The gate also **surfaced and fixed a real DCL robustness
+bug** (couldn't dial a server advertising `0.0.0.0`). Items **B/C remain deferred** — the dev
+cluster has no UNS nodes and no native alarms to gate them against.
+
+## Environment findings
+
+- v3 cluster publishes both namespaces: `ns=2 = https://zb.com/otopcua/raw`,
+  `ns=3 = https://zb.com/otopcua/uns`. **Raw lands at ns=2 — the exact index collision scope §4
+  item A predicted** (v2's sole namespace and v3 raw both at ns=2).
+- Raw tree is **sparsely populated**: one materialized live Variable,
+  `nsu=https://zb.com/otopcua/raw;s=sa-modbus2/plc/SiteAOnly` (Int16, AccessLevel=1 → **read-only**).
+  Other drivers (calc1, opcua1, pymodbus, gate-modbus, Area1) expose folders but no live leaf tags.
+- **UNS (ns=3) is empty** — `ns=3;s=OtOpcUa` → `BadNodeIdUnknown`; no Equipment authored.
+- **No native alarms** — zero `HasCondition`/`HasEventSource` refs anywhere in the raw tree.
+
+## The fix this gate produced
+
+`RealOpcUaClient` took the server-advertised `EndpointDescription.EndpointUrl` verbatim for the
+session channel. OtOpcUa advertises `opc.tcp://0.0.0.0:4840/OtOpcUa` (its
+`OpcUaApplicationHostOptions.PublicHostname` defaults to `0.0.0.0`), and dialling `0.0.0.0` from
+inside the client container hits its own loopback → connect fails. Added
+`RealOpcUaClient.RewriteEndpointHostForReachability` (swaps the advertised authority for the
+reachable discovery host/port, preserves scheme+path, no-op when already reachable), called in both
+`ConnectAsync` and `VerifyEndpointAsync`. Mirrors `CoreClientUtils.SelectEndpoint` and OtOpcUa's own
+client (`IEndpointDiscovery`). Commit `a1abbff7` on `fix/opcua-discovery-host-rewrite`; 9 unit tests
+(`RealOpcUaClientEndpointRewriteTests`).
+
+## Wiring (how the gate was run)
+
+1. Rebuilt `scadabridge:latest` with the fix (`bash docker/deploy.sh`).
+2. `docker network connect otopcua-dev_default scadabridge-site-a-a` (and `-b`) — bridged the two
+   docker networks so site-a can resolve `otopcua-dev-site-a-1-1`.
+3. `data-connection create --site-id 1 --name OtOpcUa-v3-raw --protocol OpcUa` →
+   `endpointUrl=opc.tcp://otopcua-dev-site-a-1-1:4840/OtOpcUa`, securityMode none (id **3044**).
+4. `deploy artifacts --site-id 1` — pushed the def to site-a's LocalDb + made it live in the DCL.
+5. Template **2148** (`OtOpcUaV3Probe`) + attribute `SiteA` (Int32,
+   `--data-source nsu=https://zb.com/otopcua/raw;s=sa-modbus2/plc/SiteAOnly`); instance **98**
+   (`otopcua-v3-probe-1`) on site-a; `set-bindings [["SiteA",3044]]`; `deploy instance`.
+
+## Checks
+
+| # | Check | Result |
+|---|-------|--------|
+| 1 | Connect via `ConnectAsync` (through the rewrite fix) | **PASS** — `data-connection browse` returned the address space |
+| 2 | Connect via `VerifyEndpointAsync` (second rewrite call site) | **PASS** — `verify-endpoint` → `{"success":true}` |
+| 3 | Browse emits durable `nsu=` throughout | **PASS** — root `nsu=…/raw;s=OtOpcUa`, drivers + `…;s=sa-modbus2/plc/SiteAOnly` |
+| 4 | Both namespaces visible / discoverable | **PASS (partial)** — raw browsable; uns registered but empty (nothing to browse) |
+| 5 | Durable `nsu=` binding resolves live | **PASS** — attribute stores `nsu=…`, subscription resolves it against the live `NamespaceArray` |
+| 6 | Read round-trip, Good quality, live | **PASS** — `SiteA` Good, values changing 59689→59806→59822→59834 across snapshots |
+| 7 | Write round-trip | **N/A** — the only materialized tag is read-only (AccessLevel=1); no writable v3 tag to gate |
+
+## Deferred (unchanged — need OtOpcUa-side authoring on the dev cluster)
+
+- **Item A** — re-author legacy `ns=2` bindings to `nsu=` (data; the collision is confirmed real here).
+- **Item B** — native-alarm dedup across raw+uns fan-out: **no native alarms exist** to drive it.
+- **Item C** — UNS-bound → RawPath alarm routing: **no UNS nodes and no alarms exist** to drive it.
+
+To gate B/C, the OtOpcUa dev cluster needs UNS Equipment authored (populate ns=3, referencing raw
+tags) and at least one alarm-configured tag raising a Part-9 condition that fans to raw + equipment
+notifiers.
+
+## Rig artifacts created (non-default rig state)
+
+- ScadaBridge site-a-a / site-a-b attached to `otopcua-dev_default` (dropped on next `docker/deploy.sh`).
+- Site-a config: DataConnection **3044** `OtOpcUa-v3-raw`, template **2148**, instance **98** (deployed).
+  Harmless; remove with `instance delete 98` / `template delete 2148` / `data-connection delete 3044`
+  if a clean rig is wanted.
@@ -137,6 +137,27 @@ Behavior, covered by `SelfFirstSeedBootstrapTests` (real in-process clusters bui

 The docker failover drill (`docker/failover-drill.sh`) proves both directions: `standby` mode kills the younger node (active untouched, zero routing blips); `active` mode kills the active/oldest node and asserts the survivor **takes over while the victim is still down**.

+### Simultaneous cold start — the bootstrap guard (opt-in, Gitea #33)
+
+Self-first-on-both has a residual cost. When **both** nodes cold-start at the same instant, each is `seed-nodes[0]` for itself, so **each runs `FirstSeedNodeProcess`**, times out waiting for the other, and forms its **own** single-node cluster — a split brain (two oldest members, two singletons, dual-active) that does **not** auto-merge and persists until an operator restarts one side. In-process loopback tests usually converge (the `InitJoin` handshake resolves before either self-join deadline, so row 3 above passes), which is exactly why the split hides until a real deployment powers up both VMs truly in parallel — a shared power event or a hypervisor host reboot. The sister project **OtOpcUa** (same `ZB.MOM.WW.*` Akka topology) reproduced it reliably on its docker rig.
+
+The **bootstrap guard** eliminates the split without giving up cold-start-alone. It is an **opt-in dark switch** — `ScadaBridge:Cluster:BootstrapGuard:Enabled`, **default off**, so a node with the guard disabled keeps Akka's config-driven self-first auto-join byte-identical. When enabled, the node starts with **no config seed nodes** (`AkkaHostedService.BuildHocon` emits an empty seed list, so Akka does not auto-join) and a coordinator (`ClusterBootstrapCoordinator`, an `IHostedService` registered in both the Central and Site composition roots) picks the join order after a reachability probe:
+
+- The node with the lexicographically **lower** canonical `host:port` is the **preferred founder**: it joins self-first and forms immediately if no peer answers — no probe, no delay. The tie-break is **case-insensitive** (a hostname-casing mismatch between the two configs must never make both think they are the founder and re-open the split).
+- The **higher** node TCP-probes its partner's Akka port (a node binds its port well before it forms) up to `PartnerProbeSeconds`: **reachable ⇒ peer-first** (join the founder, never race it); **unreachable after the window ⇒ self-first** (the partner is genuinely down, so cold-start-alone is preserved for the higher node too).
+- The order is decided **before** a single `Cluster.JoinSeedNodes`, from an explicit reachability signal. The coordinator **never re-forms mid-handshake** — that was the failure mode of the rejected self-form timer above (a `Join(self)` on a bare timeout is not ignored mid-handshake; it wins and islands a node).
+
+**Residual trade-off (accepted, operator-visible).** Once the higher node commits peer-first it cannot self-form (`JoinSeedNodeProcess` retries forever). If the founder dies in the small probe→join window, the higher node hangs unjoined; the coordinator logs a WARNING after a bounded grace, and a **restart** recovers it (the guard re-runs, finds the founder down, and forms alone). The guard deliberately does not re-decide mid-handshake.
+
+| Key | Default | Meaning |
+|---|---|---|
+| `BootstrapGuard:Enabled` | `false` | Dark switch. Only meaningful on a node that is one of its own two pair seeds; inert elsewhere (single-node install, self absent, 3+ seeds). |
+| `BootstrapGuard:PartnerProbeSeconds` | `25` | Higher node's probe window. Must comfortably exceed the founder's process-start-to-Akka-bind time, or a slow-but-alive founder is mistaken for dead and the split re-opens. Validated `> 0` at boot when the guard is on. |
+| `BootstrapGuard:PartnerProbeIntervalMs` | `500` | Interval between probes. Validated `> 0` when enabled. |
+| `BootstrapGuard:ProbeConnectTimeoutMs` | `1000` | Per-probe TCP connect timeout. Validated `> 0` when enabled. |
+
+Decision core (`ClusterBootstrapGuard`) is a pure, fully unit-tested function; the probe + `JoinSeedNodes` runtime (`ClusterBootstrapCoordinator`) is covered by real-ActorSystem tests including the load-bearing higher-node-cold-start-alone case and the headline both-cold-start-together-form-one-cluster case (`ClusterBootstrapCoordinatorTests`). The alternative is purely operational (stagger the two VMs' service-manager start / start the founder first); the docker rig's compose `depends_on` does that today, but that does not exist on the real co-located VMs — the guard is the production-faithful fix. Ported from OtOpcUa (`lmxopcua` commit `d1dac87f`); implementing it in both products keeps their failure/recovery model identical, since a shared power event hits both pairs at once.
+
 ### Manual Failover (admin-triggered)

 An Administrator can swap the central pair's roles deliberately — for planned maintenance on the active node, or to move singletons off a node behaving badly without waiting for a crash. The control is the **Trigger failover** button on the central-cluster card of the Health dashboard (`/monitoring/health`).
@@ -191,7 +212,7 @@ These values balance failover speed with stability — fast enough that data col

 If both nodes in a cluster fail simultaneously (e.g., site power outage):

-1. **No manual intervention required.** Since both nodes are configured as seed nodes, whichever node starts first forms a new cluster. The second node joins when it starts.
+1. **No manual intervention required.** Since both nodes are configured as seed nodes, whichever node starts first forms a new cluster (self-first ordering) and the second node joins when it starts. If both power up truly simultaneously, each self-first seed can run `FirstSeedNodeProcess` and form its own 1-node cluster — a split brain; enable the opt-in **bootstrap guard** (Gitea #33, see *Simultaneous cold start* above) on co-located VM pairs that share a power domain to make them converge on one cluster instead.
 2. **State recovery** (each node has its own local copy of all required data):
   - **Site clusters**: The Deployment Manager singleton reads deployed configurations from local SQLite and re-creates the full Instance Actor hierarchy. Store-and-forward buffers are already persisted locally. Alarm states re-evaluate from incoming data values.
   - **Central cluster**: All state is in MS SQL (configuration database). The active node resumes normal operation.
@@ -0,0 +1,169 @@
+# Environment Variables
+
+A deep-scan inventory of every environment variable ScadaBridge reads — the Host, the CLI, the
+DelmiaNotifier client, EF design-time tooling, and the test/CI harness — plus the supporting infra
+containers. Compiled by scanning `Environment.GetEnvironmentVariable` call sites, the Host's
+`AddEnvironmentVariables()` config source, `docker/` + `docker-env2/` compose files,
+`deploy/wonder-app-vd03/install.ps1`, and every `appsettings*.json`.
+
+**Scan date:** 2026-07-24 · **Branch:** `main`
+
+Two columns qualify each variable:
+
+- **Scope** — where it is read: **Runtime** (the running product — Host / CLI / DelmiaNotifier),
+  **Design-time** (`dotnet ef` tooling), **Test** (unit / integration / E2E / perf harness only, never
+  the shipped product), or **Infra** (a sidecar container, not ScadaBridge code).
+- **In appsettings?** — whether a shipped `appsettings*.json` already carries the same config key
+  (so the env var is an *override* or *secret substitute*), versus being env-only. `${secret:...}`
+  means the key is present but its value is a secret token, not a literal.
+
+## How configuration resolves (read this first)
+
+The Host builds its configuration in `src/ZB.MOM.WW.ScadaBridge.Host/Program.cs` in this precedence
+order (later overrides earlier):
+
+1. `appsettings.json`
+2. `appsettings.{SCADABRIDGE_CONFIG}.json` (role file — `Central` or `Site`)
+3. **`.AddEnvironmentVariables()`** ← every variable below in the "config-override" table
+4. `.AddCommandLine(args)`
+
+Then a **pre-host secrets expander** resolves any `${secret:NAME}` tokens left in the merged config,
+reading the encrypted secrets store whose master key comes from **`ZB_SECRETS_MASTER_KEY`**. Because
+step 3 runs *before* expansion, a whole-key env override (e.g. `ScadaBridge__Database__ConfigurationDb`)
+wins over — and short-circuits — the corresponding `${secret:...}` token. That is the deliberate
+"rollback / dev" path the docker rigs use.
+
+**Config-key → env-var mapping:** replace each `:` in a config path with `__` (double underscore).
+So `ScadaBridge:Communication:GrpcPsk` is set via `ScadaBridge__Communication__GrpcPsk`. Any config
+key in the app can be overridden this way; the table below lists only the ones with an established
+purpose (secrets kept off disk, or dev-rig wiring).
+
+---
+
+## 1. Role & host environment selection
+
+| Variable | Scope | In appsettings? | Consumed by | Purpose | Potential values | Required |
+|---|---|---|---|---|---|---|
+| `SCADABRIDGE_CONFIG` | Runtime | No (it *selects* the `appsettings.{value}.json` file) | Host (`Program.cs:42`) | Selects the role-specific overlay. **Primary role switch.** | `Central`, `Site` | Effectively yes (falls back to `DOTNET_ENVIRONMENT`, then `Production`) |
+| `DOTNET_ENVIRONMENT` | Runtime | No | Host (`Program.cs:43`) | Fallback role selector when `SCADABRIDGE_CONFIG` is unset; generic .NET environment name. | `Central`, `Site`, `Development`, `Production` | No |
+| `ASPNETCORE_ENVIRONMENT` | Runtime | No | ASP.NET Core + `DisableLoginGuard` (`Program.cs:438`) | Standard ASP.NET environment. `DisableLogin` is refused unless this is `Development`. | `Development`, `Production` | No (defaults to `Production` behavior) |
+| `ASPNETCORE_URLS` | Runtime | No (Kestrel binding; not in these appsettings) | Kestrel | HTTP bind addresses. Docker sets `http://+:5000`; install.ps1 sets `http://+:{WebPort}` (8085). | e.g. `http://+:5000`, `http://+:8085` | No |
+
+> `SCADABRIDGE_CONFIG` and `ASPNETCORE_ENVIRONMENT` are independent. The docker rig runs
+> `SCADABRIDGE_CONFIG=Central` **and** `ASPNETCORE_ENVIRONMENT=Development` simultaneously.
+
+## 2. Secrets store master key (KEK)
+
+| Variable | Scope | In appsettings? | Consumed by | Purpose | Potential values | Required |
+|---|---|---|---|---|---|---|
+| `ZB_SECRETS_MASTER_KEY` | Runtime | Name only (`Secrets:MasterKey:EnvVarName` in `appsettings.json`; the value is never committed) | Host secrets expander | The KEK that decrypts the `ZB.MOM.WW.Secrets` store, from which every `${secret:...}` token resolves. A missing reference fails closed (`SecretNotFoundException`) before any SQL/LDAP/cluster wiring. | an opaque base64/hex master key | Yes on any node whose appsettings uses `${secret:...}` (shipped Central + Site do). Docker rigs avoid it with whole-key overrides. |
+
+The variable *name* is itself configurable via `Secrets:MasterKey:EnvVarName` (defaults to
+`ZB_SECRETS_MASTER_KEY`). See `docs/operations/2026-07-16-secrets-clustered-master-key.md`.
+
+## 3. Config-override env vars (`ScadaBridge__*` whole-key)
+
+Map onto config keys via the `:`→`__` rule. Keep secrets off disk (production, via a secret manager)
+or wire the dev rigs without a KEK. The `__site-x` / `__site-a` leaf is the literal `siteId`.
+All are **Runtime** scope (the Host reads them; Host integration-test fixtures set the same names, but
+that is the same variable, not a separate one).
+
+| Variable | In appsettings? | Config key | Purpose | Potential values | Required |
+|---|---|---|---|---|---|
+| `ScadaBridge__Database__ConfigurationDb` | Yes — `${secret:sql/scadabridge/configdb-connection}` in `appsettings.Central.json` | `ScadaBridge:Database:ConfigurationDb` | Central config DB connection string (MS SQL). Central-only. | `Server=host,1433;Database=ScadaBridgeConfig;User Id=...;Password=...;TrustServerCertificate=true` | Yes on Central — via env **or** the `${secret}` |
+| `ScadaBridge__Database__MachineDataDb` | No in shipped src appsettings; present in the deploy overlay (`deploy/wonder-app-vd03/appsettings.Central.json`, as `${...}`). install.ps1 derives it as `<DbName>MachineData`. | `ScadaBridge:Database:MachineDataDb` | Central machine-data DB connection string. Central-only. | connection string | Yes on Central |
+| `ScadaBridge__Security__JwtSigningKey` | Yes — `${secret:security/scadabridge/jwt-signing-key}` in `appsettings.Central.json` | `ScadaBridge:Security:JwtSigningKey` | HMAC-SHA256 signing key for the cookie-embedded session JWT. | ≥32-char secret | Yes on Central |
+| `ScadaBridge__Security__Ldap__ServiceAccountPassword` | Yes — `${secret:ldap/scadabridge/service-account-password}` in `appsettings.Central.json` | `ScadaBridge:Security:Ldap:ServiceAccountPassword` | LDAP bind-account password. (Renamed from the old flat `...__LdapServiceAccountPassword`.) | the bind password | Yes on Central if LDAP bind is used |
+| `ScadaBridge__InboundApi__ApiKeyPepper` | No — env/secret only (not in any shipped appsettings) | `ScadaBridge:InboundApi:ApiKeyPepper` | Per-environment pepper for the inbound API-key peppered-HMAC verifier. Hard Central startup requirement since 2026-06-03. | ≥16-char secret, unique per environment | Yes on Central |
+| `ScadaBridge__Communication__GrpcPsk` | Yes — `${secret:SB-GRPC-PSK-site-1}` in `appsettings.Site.json` | `ScadaBridge:Communication:GrpcPsk` | **Site side** of the per-site gRPC control-plane PSK. StartupValidator refuses to boot a Site node without it; both pair nodes carry the same value. | per-site secret (e.g. `dev-grpc-psk-docker-site-a`); prod `${secret:SB-GRPC-PSK-<siteId>}` | Yes on Site |
+| `ScadaBridge__Communication__SitePsks__<siteId>` | No in shipped src appsettings (central normally resolves the `SB-GRPC-PSK-<siteId>` secret); present only in the docker-compose env blocks | `ScadaBridge:Communication:SitePsks:<siteId>` | **Central side** of the same key: the value central presents when dialling that site. Must equal the site's `GrpcPsk`. Override for hosts without a master secret store. | matches the site's `GrpcPsk` | On Central, either this **or** the `SB-GRPC-PSK-<siteId>` store secret |
+
+Docker dev values (dev-only-insecure, committed in `docker/docker-compose.yml`):
+`ScadaBridge__Communication__SitePsks__site-a = dev-grpc-psk-docker-site-a` (also `site-b`, `site-c`);
+`docker-env2` uses `SitePsks__site-x = dev-grpc-psk-docker-env2-site-x`.
+
+> Any other config key is overridable the same way (e.g. `ScadaBridge__Cluster__SplitBrainResolverStrategy`,
+> `ScadaBridge__Logging__MinimumLevel`) — these are the standard .NET config binder, not "special" env
+> vars. Prefer `appsettings` files for non-secret values.
+
+### `${secret:...}` tokens (resolved from the store, NOT env vars)
+
+Not environment variables — config values resolved by the secrets expander using `ZB_SECRETS_MASTER_KEY`.
+All are **Runtime** scope and, by definition, present **In appsettings** as tokens.
+
+| Token (in `appsettings`) | Meaning |
+|---|---|
+| `${secret:sql/scadabridge/configdb-connection}` | Central config DB connection string |
+| `${secret:ldap/scadabridge/service-account-password}` | LDAP bind password |
+| `${secret:security/scadabridge/jwt-signing-key}` | JWT signing key |
+| `${secret:SB-GRPC-PSK-<siteId>}` | Per-site gRPC PSK (site side); central resolves `SB-GRPC-PSK-{siteId}` at channel-build time |
+
+## 4. CLI (`scadabridge` / `ZB.MOM.WW.ScadaBridge.CLI`)
+
+Resolution order per value: command-line flag → env var → `~/.scadabridge/config.json`. The CLI does
+**not** read `appsettings.json` — its config file is `~/.scadabridge/config.json` (a separate schema).
+
+| Variable | Scope | In appsettings? | Purpose | Potential values | Required |
+|---|---|---|---|---|---|
+| `SCADABRIDGE_MANAGEMENT_URL` | Runtime (CLI) | No — CLI config file has `managementUrl` instead | Management API base URL. Overridden by `--url`; overrides the config file. Missing all three → exit `1`, `NO_URL`. | e.g. `http://localhost:9000` | One of flag/env/config |
+| `SCADABRIDGE_FORMAT` | Runtime (CLI) | No — CLI config file has `defaultFormat` instead | Default output format. Overridden by `--format`. | `json`, `table` | No (defaults to `json`) |
+| `SCADABRIDGE_USERNAME` | Runtime (CLI) | No | LDAP username fallback when `--username` absent. | e.g. `multi-role` | No (some commands need auth) |
+| `SCADABRIDGE_PASSWORD` | Runtime (CLI) | No | LDAP password fallback when `--password` absent. **Preferred over `--password`** — avoids leaking into process listings / shell history. | the user's password | No (some commands need auth) |
+
+## 5. DelmiaNotifier client (`WWNotifier.exe`)
+
+| Variable | Scope | In appsettings? | Purpose | Potential values | Required |
+|---|---|---|---|---|---|
+| `SCADABRIDGE_API_KEY` | Runtime (client) | No — the notifier's `appsettings.json` holds only `BaseUrls`/`TimeoutSeconds`/`LogPath`; the key is **env-only by design** | Inbound API key sent as `X-API-Key`. Read only from the environment, never a file. Missing/empty → prints `API key not configured (SCADABRIDGE_API_KEY)`, exit `-1`, no HTTP attempt. | e.g. `sbk_...` | Yes |
+
+> This is also why `System.Environment` is wholly forbidden in the script trust model — it would let a
+> site/inbound script exfiltrate `SCADABRIDGE_API_KEY` and other host env secrets.
+
+## 6. EF Core design-time tooling
+
+| Variable | Scope | In appsettings? | Purpose | Potential values | Required |
+|---|---|---|---|---|---|
+| `SCADABRIDGE_DESIGNTIME_CONNECTIONSTRING` | Design-time | Fallback for `ScadaBridge:Database:ConfigurationDb` — the factory reads that appsettings key first, then this env var | Connection string for `dotnet ef` migrations tooling when the Host's appsettings isn't the source. No hardcoded fallback — missing → actionable failure. | a config DB connection string | Only for `dotnet ef` when appsettings is unavailable |
+
+## 7. Test / CI-only
+
+Not read by the running product — the test harness only. Most default gracefully (skip, or use a
+local docker value) when unset.
+
+| Variable | Scope | In appsettings? | Consumed by | Purpose | Potential values |
+|---|---|---|---|---|---|
+| `SCADABRIDGE_MSSQL_TEST_CONN` | Test | No | `MsSqlMigrationFixture`, AuditLog migration tests | Admin (sa) connection to a running MS SQL; absent → local docker sa default; drives `Skip.If`. | sa connection string |
+| `SCADABRIDGE_PLAYWRIGHT_DB` | Test | No | `PlaywrightDbConnection` | DB connection for Playwright E2E seeding; absent → `localhost:1433`. | connection string |
+| `SCADABRIDGE_CLI_DLL` | Test | No | `CliRunner` | Absolute path to the built CLI dll for Playwright when auto-probing fails. | path to `...CLI.dll` |
+| `SCADABRIDGE_AUDIT_FILTER_4KB_P95_US` | Test | No | `HotPathLatencyTests` | Override P95 latency threshold (µs) for the 4 KB audit-filter perf test. | positive number (default 5) |
+| `SCADABRIDGE_AUDIT_FILTER_RAW_P95_US` | Test | No | `HotPathLatencyTests` | Override P95 threshold for the raw-capture perf test. | positive number |
+| `SCADABRIDGE_AUDIT_FILTER_NOOP_P95_US` | Test | No | `HotPathLatencyTests` | Override P95 threshold for the no-op fast-path perf test. | positive number (default 5) |
+| `HOME` / `USERPROFILE` | Test | No | `CliConfig` tests | Redirected to locate `~/.scadabridge/config.json` under test. | temp dir path |
+
+The Host test fixtures (`CentralDbTestEnvironment`, `ScadaBridgeWebApplicationFactory`,
+`StartupValidationTests`) set the **section-3 config-override vars** (`ScadaBridge__Database__*`,
+`ScadaBridge__InboundApi__ApiKeyPepper`, etc.) and `DOTNET_ENVIRONMENT=Central` at runtime; those are
+the same variables already documented above, not new ones.
+
+## 8. Supporting infrastructure containers (not the app)
+
+Set on the sidecar/infra containers in `infra/docker-compose.yml`, consumed by those images, not by
+ScadaBridge code — included for completeness when standing up a rig.
+
+| Variable | Scope | In appsettings? | Container | Purpose | Value used |
+|---|---|---|---|---|---|
+| `MSSQL_SA_PASSWORD` | Infra | No | `scadabridge-mssql` | SQL Server `sa` password. | `ScadaBridge_Dev1#` (dev-only) |
+| `MSSQL_PID` | Infra | No | `scadabridge-mssql` | SQL Server edition. | `Developer` |
+
+---
+
+## Quick reference: minimum sets to boot a real deploy
+
+**Central node** (via env, or the `${secret:...}` equivalents + `ZB_SECRETS_MASTER_KEY`):
+`SCADABRIDGE_CONFIG=Central`, `ScadaBridge__Database__ConfigurationDb`,
+`ScadaBridge__Database__MachineDataDb`, `ScadaBridge__Security__JwtSigningKey`,
+`ScadaBridge__Security__Ldap__ServiceAccountPassword`, `ScadaBridge__InboundApi__ApiKeyPepper`, and
+for each managed site either the `SB-GRPC-PSK-<siteId>` secret or `ScadaBridge__Communication__SitePsks__<siteId>`.
+
+**Site node:** `SCADABRIDGE_CONFIG=Site` and `ScadaBridge__Communication__GrpcPsk` (or its
+`${secret:SB-GRPC-PSK-<siteId>}` form). StartupValidator hard-fails without the PSK.
@@ -144,8 +144,8 @@ public static class ServiceCollectionExtensions

        // ISiteStreamAuditClient: NoOp default. This binding remains correct for
        // central/test composition roots that have no SiteCommunicationActor.
-        // The real implementation is ClusterClientSiteAuditClient, which pushes
-        // audit telemetry to central over Akka ClusterClient via the site's
+        // The real implementation is SiteCommunicationAuditClient, which pushes
+        // audit telemetry to central via the site's
        // SiteCommunicationActor — the Host wires it directly into the
        // SiteAuditTelemetryActor's Props.Create call for site roles (it cannot
        // be a DI singleton because it needs the SiteCommunicationActor IActorRef,
@@ -37,7 +37,7 @@ namespace ZB.MOM.WW.ScadaBridge.AuditLog.Site.Telemetry;
 /// <b>Local-write only — the wire push is the drain actor's job.</b> This
 /// forwarder is deliberately synchronous against the two site-local SQLite
 /// stores and never pushes to central itself. The site→central transport is
-/// now live: <c>ClusterClientSiteAuditClient</c> is the production binding of
+/// now live: <c>SiteCommunicationAuditClient</c> is the production binding of
 /// <see cref="ISiteStreamAuditClient"/> on site roles (with
 /// <c>NoOpSiteStreamAuditClient</c> retained only for central/test composition
 /// roots). The push happens out-of-band: <see cref="SiteAuditTelemetryActor"/>
@@ -6,9 +6,9 @@ namespace ZB.MOM.WW.ScadaBridge.AuditLog.Site.Telemetry;
 /// Mockable abstraction over the central site-audit push surface that
 /// <see cref="SiteAuditTelemetryActor"/> uses to forward <see cref="AuditEventBatch"/>
 /// payloads. The production implementation is
-/// <see cref="ClusterClientSiteAuditClient"/> — a ClusterClient-based client,
-/// wired in the Host for site roles, that forwards batches to central via the
-/// site's <c>SiteCommunicationActor</c>. Unit tests substitute via NSubstitute
+/// <see cref="SiteCommunicationAuditClient"/> — wired in the Host for site
+/// roles, it forwards batches to central via the site's
+/// <c>SiteCommunicationActor</c>. Unit tests substitute via NSubstitute
 /// against this interface so the actor never needs a live transport.
 /// </summary>
 public interface ISiteStreamAuditClient
@@ -36,8 +36,8 @@ public interface ISiteStreamAuditClient
    /// once central has acknowledged them.
    /// </summary>
    /// <remarks>
-    /// The production <see cref="ClusterClientSiteAuditClient"/> forwards over
-    /// the ClusterClient transport; the <see cref="NoOpSiteStreamAuditClient"/>
+    /// The production <see cref="SiteCommunicationAuditClient"/> forwards over
+    /// the site's central transport; the <see cref="NoOpSiteStreamAuditClient"/>
    /// DI default (used by central and test composition roots) returns an empty
    /// ack so no rows are flipped.
    /// </remarks>
@@ -7,9 +7,8 @@ namespace ZB.MOM.WW.ScadaBridge.AuditLog.Site.Telemetry;
 /// <see cref="ZB.MOM.WW.ScadaBridge.AuditLog.ServiceCollectionExtensions.AddAuditLog"/>.
 /// It is a no-op binding for composition roots that have no
 /// <c>SiteCommunicationActor</c> — central and test roots. Site roles override
-/// it in the Host with the ClusterClient-based
-/// <see cref="ClusterClientSiteAuditClient"/>, which actually forwards audit
-/// telemetry to central.
+/// it in the Host with <see cref="SiteCommunicationAuditClient"/>, which
+/// actually forwards audit telemetry to central.
 /// </summary>
 /// <remarks>
 /// <para>
@@ -33,7 +32,7 @@ public sealed class NoOpSiteStreamAuditClient : ISiteStreamAuditClient
    {
        ArgumentNullException.ThrowIfNull(batch);
        // Empty ack — no EventIds will be flipped to Forwarded, so rows stay
-        // Pending until the real ClusterClientSiteAuditClient (or a test stub)
+        // Pending until the real SiteCommunicationAuditClient (or a test stub)
        // takes over.
        return Task.FromResult(EmptyAck);
    }
@@ -45,7 +44,7 @@ public sealed class NoOpSiteStreamAuditClient : ISiteStreamAuditClient
        // Empty ack — same rationale as IngestAuditEventsAsync. The site still
        // writes the audit + tracking rows to its SQLite stores authoritatively;
        // central-side state only materialises once the real
-        // ClusterClientSiteAuditClient (or a test stub) is wired in.
+        // SiteCommunicationAuditClient (or a test stub) is wired in.
        return Task.FromResult(EmptyAck);
    }
 }
@@ -261,6 +261,18 @@ public class SiteAuditTelemetryActor : ReceiveActor

            var batch = new CachedTelemetryBatch();
            var emittedEventIds = new List<Guid>(pending.Count);
+            // Rows whose operational half can never be built. They are marked
+            // Forwarded so they LEAVE this queue — see AbandonUnresolvableAsync
+            // for why that is safe and why leaving them Pending is not.
+            var abandonedEventIds = new List<Guid>();
+            // Counted, not logged per row: a whole batch can fail identically
+            // (tracking store down, tracking retention elapsed), and one line
+            // per row per drain is what turned this into a log flood.
+            var deferredNoSnapshot = 0;
+            var lookupFailures = 0;
+            Exception? firstLookupFailure = null;
+            var graceSeconds = Math.Max(0, _options.CachedTrackingGraceSeconds);
+            var abandonBefore = DateTime.UtcNow - TimeSpan.FromSeconds(graceSeconds);

            foreach (var auditRow in pending)
            {
@@ -268,13 +280,9 @@ public class SiteAuditTelemetryActor : ReceiveActor
                {
                    // CorrelationId carries the TrackedOperationId for cached
                    // rows — see CachedCallLifecycleBridge.BuildPacket. Without
-                    // it we can't look up the tracking row; log + skip so the
-                    // bad row doesn't block the rest of the batch. The audit
-                    // row stays Pending (still not in emittedEventIds) and
-                    // central reconciliation will pick it up.
-                    _logger.LogWarning(
-                        "Cached-telemetry drain: audit row {EventId} ({Action}) has no CorrelationId; skipping.",
-                        auditRow.EventId, auditRow.Action);
+                    // it there is nothing to look up, ever, so this row is
+                    // abandoned immediately rather than after the grace period.
+                    abandonedEventIds.Add(auditRow.EventId);
                    continue;
                }

@@ -288,24 +296,32 @@ public class SiteAuditTelemetryActor : ReceiveActor
                catch (Exception ex)
                {
                    // A tracking-store throw must NOT abort the rest of the
-                    // batch — the audit half is best-effort. Log and skip
-                    // this row; it stays Pending for the next drain.
-                    _logger.LogWarning(ex,
-                        "Cached-telemetry drain: tracking lookup threw for {EventId} (TrackedOperationId {Tid}, sqlite {SqliteError}); skipping.",
-                        auditRow.EventId, auditRow.CorrelationId, SqliteErrorCodes.Describe(ex));
+                    // batch — the audit half is best-effort. A throw is a
+                    // STORE fault (locked, corrupt) rather than a verdict about
+                    // this row, so the row is never abandoned on this path: it
+                    // stays Pending and retries once the store recovers.
+                    lookupFailures++;
+                    firstLookupFailure ??= ex;
                    continue;
                }

                if (snapshot is null)
                {
-                    // No tracking row — possible if the audit row is older
-                    // than the tracking retention window, or the tracking
-                    // store was reset. The audit half remains valid and will
-                    // be picked up by central reconciliation; skip the
-                    // combined push for this row.
-                    _logger.LogWarning(
-                        "Cached-telemetry drain: no tracking snapshot for {EventId} (TrackedOperationId {Tid}); skipping.",
-                        auditRow.EventId, auditRow.CorrelationId);
+                    // No tracking row. Within the grace window this is the
+                    // ordinary write race (the audit row landed first), so
+                    // retry. Past it the snapshot is gone for good — tracking
+                    // retention elapsed, or the two stores were reset
+                    // independently — and retrying forever would wedge the
+                    // queue behind these rows.
+                    if (auditRow.OccurredAtUtc <= abandonBefore)
+                    {
+                        abandonedEventIds.Add(auditRow.EventId);
+                    }
+                    else
+                    {
+                        deferredNoSnapshot++;
+                    }
+
                    continue;
                }

@@ -314,11 +330,33 @@ public class SiteAuditTelemetryActor : ReceiveActor
                emittedEventIds.Add(auditRow.EventId);
            }

+            if (lookupFailures > 0)
+            {
+                _logger.LogWarning(firstLookupFailure,
+                    "Cached-telemetry drain: tracking lookup failed for {Count} of {Total} rows " +
+                    "(sqlite {SqliteError}); they stay Pending and retry next drain. First failure attached.",
+                    lookupFailures, pending.Count, SqliteErrorCodes.Describe(firstLookupFailure!));
+            }
+
+            if (deferredNoSnapshot > 0)
+            {
+                _logger.LogDebug(
+                    "Cached-telemetry drain: {Count} row(s) have no tracking snapshot yet and are inside the " +
+                    "{Grace}s grace window; retrying next drain.",
+                    deferredNoSnapshot, graceSeconds);
+            }
+
+            if (abandonedEventIds.Count > 0)
+            {
+                await AbandonUnresolvableAsync(abandonedEventIds, graceSeconds, ct)
+                    .ConfigureAwait(false);
+            }
+
            if (batch.Packets.Count == 0)
            {
-                // Every row in this read was skipped (no CorrelationId / no
-                // tracking snapshot). Leave them Pending and try again next
-                // drain — the underlying race normally resolves on its own.
+                // Nothing resolvable in this read. Any permanently-unresolvable
+                // rows have just been marked Forwarded above, so the next drain
+                // sees past them rather than re-reading the same head of queue.
                return;
            }

@@ -358,6 +396,59 @@ public class SiteAuditTelemetryActor : ReceiveActor
        }
    }

+    /// <summary>
+    /// Marks cached rows whose operational half can never be built as
+    /// Forwarded, so they leave the cached-drain queue.
+    /// </summary>
+    /// <remarks>
+    /// <para>
+    /// <b>Why they must leave.</b> Leaving an unresolvable row Pending — the
+    /// previous behaviour — is not a harmless skip. The queue is read
+    /// oldest-first with a fixed <c>BatchSize</c>, so once a batch's worth of
+    /// unresolvable rows collects at the head, every drain re-reads exactly
+    /// those rows, fails identically, and never sees the newer rows behind
+    /// them. The cached-telemetry path stalls permanently, and each pass logs
+    /// once per row (measured at ~2 800 warnings/minute on a rig).
+    /// </para>
+    /// <para>
+    /// <b>Why Forwarded is the honest state.</b> Its role in this state machine
+    /// is "no longer owed by the drain, still eligible for reconciliation" —
+    /// which is exactly the situation here. The audit half is NOT dropped:
+    /// <c>ReadPendingSinceAsync</c> covers Forwarded rows as well as Pending
+    /// ones and central dedups on EventId, so the reconciliation pull still
+    /// delivers them. What is genuinely lost is the operational
+    /// (<c>SiteCalls</c>) half — and that is unrecoverable regardless, because
+    /// the tracking row it would have been built from no longer exists.
+    /// </para>
+    /// <para>
+    /// A failure to mark is swallowed: the rows simply stay Pending and the
+    /// next drain retries. Escalating here would take down the audit drain over
+    /// a best-effort cleanup.
+    /// </para>
+    /// </remarks>
+    private async Task AbandonUnresolvableAsync(
+        IReadOnlyList<Guid> eventIds, int graceSeconds, CancellationToken ct)
+    {
+        try
+        {
+            await _queue.MarkForwardedAsync(eventIds, ct).ConfigureAwait(false);
+            _logger.LogWarning(
+                "Cached-telemetry drain: abandoned the operational half of {Count} row(s) with no " +
+                "resolvable tracking snapshot after {Grace}s; marked Forwarded so they no longer block " +
+                "the queue. The audit half still reaches central via the reconciliation pull. This is " +
+                "expected after tracking retention elapses or the tracking store is reset independently " +
+                "of the audit store.",
+                eventIds.Count, graceSeconds);
+        }
+        catch (Exception ex)
+        {
+            _logger.LogWarning(ex,
+                "Cached-telemetry drain: could not mark {Count} unresolvable row(s) Forwarded " +
+                "(sqlite {SqliteError}); they stay Pending and will be retried.",
+                eventIds.Count, SqliteErrorCodes.Describe(ex));
+        }
+    }
+
    private static AuditEventBatch BuildBatch(IReadOnlyList<AuditEvent> events)
    {
        var batch = new AuditEventBatch();
@@ -25,4 +25,33 @@ public sealed class SiteAuditTelemetryOptions
    /// Longer interval avoids hammering an idle SQLite + gRPC channel.
    /// </summary>
    public int IdleIntervalSeconds { get; set; } = 30;
+
+    /// <summary>
+    /// How long a cached-telemetry audit row may go without a resolvable
+    /// <c>OperationTracking</c> snapshot before the drain ABANDONS its
+    /// operational half and marks the row Forwarded. Default: 300 s.
+    /// </summary>
+    /// <remarks>
+    /// <para>
+    /// A missing tracking snapshot is normally a brief write race — the audit
+    /// row lands microseconds before the tracking row — so the drain retries
+    /// within this grace period. Past it the snapshot is almost certainly gone
+    /// for good (tracking retention elapsed while central was unreachable, or
+    /// the two stores were reset independently), and retrying forever is
+    /// actively harmful: the queue is read oldest-first with a fixed
+    /// <see cref="BatchSize"/>, so a batch's worth of permanently-unresolvable
+    /// rows at the head STARVES every newer cached row behind them.
+    /// </para>
+    /// <para>
+    /// Abandoning costs only the operational (<c>SiteCalls</c>) half, which is
+    /// unrecoverable anyway once the tracking row is gone. The audit half is
+    /// NOT lost: the reconciliation pull covers Forwarded rows as well as
+    /// Pending ones, and central dedups on EventId.
+    /// </para>
+    /// <para>
+    /// Set to 0 to abandon on the first failed lookup. Negative values are
+    /// treated as 0.
+    /// </para>
+    /// </remarks>
+    public int CachedTrackingGraceSeconds { get; set; } = 300;
 }
@@ -7,11 +7,11 @@ namespace ZB.MOM.WW.ScadaBridge.AuditLog.Site.Telemetry;

 /// <summary>
 /// Production <see cref="ISiteStreamAuditClient"/> binding for site composition
-/// roots: pushes audit telemetry to central over Akka <c>ClusterClient</c> via
-/// the site's <c>SiteCommunicationActor</c>. The actor forwards the command to
-/// <c>/user/central-communication</c> and the central
-/// <c>CentralCommunicationActor</c> Asks the <c>AuditLogIngestActor</c> proxy —
-/// the same command/control transport notifications already use. Wired by the
+/// roots: pushes audit telemetry to central via the site's
+/// <c>SiteCommunicationActor</c>, which owns the transport choice (today
+/// <c>GrpcCentralTransport</c>, over the central-hosted <c>CentralControlService</c>).
+/// The central <c>CentralCommunicationActor</c> Asks the <c>AuditLogIngestActor</c>
+/// proxy — the same command/control path notifications already use. Wired by the
 /// Host for site roles; central and test composition roots keep the
 /// <see cref="NoOpSiteStreamAuditClient"/> DI default (they have no
 /// <c>SiteCommunicationActor</c>).
@@ -34,12 +34,12 @@ namespace ZB.MOM.WW.ScadaBridge.AuditLog.Site.Telemetry;
 /// <see cref="CachedTelemetryBatch"/>) because the
 /// <see cref="SiteAuditTelemetryActor"/> builds them with
 /// <see cref="AuditEventDtoMapper.ToDto"/>. This client converts them back into
-/// the <see cref="AuditEvent"/> / <see cref="SiteCall"/> entities the Akka
-/// command messages carry — the same DTO→entity translation the
+/// the <see cref="AuditEvent"/> / <see cref="SiteCall"/> entities the command
+/// messages carry — the same DTO→entity translation the
 /// <c>SiteStreamGrpcServer</c> performs for the gRPC reconciliation path.
 /// </para>
 /// </remarks>
-public sealed class ClusterClientSiteAuditClient : ISiteStreamAuditClient
+public sealed class SiteCommunicationAuditClient : ISiteStreamAuditClient
 {
    private readonly IActorRef _siteCommunicationActor;
    private readonly TimeSpan _askTimeout;
@@ -49,15 +49,15 @@ public sealed class ClusterClientSiteAuditClient : ISiteStreamAuditClient
    /// </summary>
    /// <param name="siteCommunicationActor">
    /// The site's <c>SiteCommunicationActor</c> — it forwards the ingest command
-    /// over the registered central ClusterClient and routes the reply back to
-    /// this client's Ask.
+    /// to central over its configured <c>ICentralTransport</c> and routes the
+    /// reply back to this client's Ask.
    /// </param>
    /// <param name="askTimeout">
    /// Ask timeout for the round-trip to central. On expiry the Ask throws
    /// <see cref="Akka.Actor.AskTimeoutException"/>, which the drain loop treats
    /// as transient (rows stay <c>Pending</c>).
    /// </param>
-    public ClusterClientSiteAuditClient(IActorRef siteCommunicationActor, TimeSpan askTimeout)
+    public SiteCommunicationAuditClient(IActorRef siteCommunicationActor, TimeSpan askTimeout)
    {
        ArgumentNullException.ThrowIfNull(siteCommunicationActor);
        _siteCommunicationActor = siteCommunicationActor;
@@ -0,0 +1,155 @@
+using System.Text.RegularExpressions;
+
+namespace ZB.MOM.WW.ScadaBridge.ClusterInfrastructure;
+
+/// <summary>
+/// Pure decision logic for the simultaneous-cold-start split-brain guard (Gitea #33).
+/// </summary>
+/// <remarks>
+///     <para>
+///         Both nodes of a 2-node pair are self-first seeds so <b>either</b> can cold-start alone when
+///         its partner is dead (see <see cref="ClusterOptions.SeedNodes"/>). The cost is that when
+///         BOTH cold-start at the same instant, each runs Akka's <c>FirstSeedNodeProcess</c>, times out
+///         waiting for the other, and forms its OWN single-node cluster — a split brain (two oldest
+///         nodes, two singletons, dual-active). Two independent clusters do not auto-merge, so the
+///         split persists until an operator restarts one side. It bites on any event that powers up
+///         both site VMs together (site power restoration, hypervisor host reboot).
+///     </para>
+///     <para>
+///         This guard breaks the symmetry deterministically without giving up cold-start-alone. The
+///         node with the lexicographically <b>lower</b> canonical address is the <i>preferred
+///         founder</i>: it always uses self-first order and forms immediately. The <b>higher</b> node
+///         first probes whether its partner's Akka endpoint is reachable (see
+///         <see cref="ClusterBootstrapCoordinator"/>): reachable ⇒ peer-first order so it JOINS the
+///         founder rather than racing it; unreachable after the probe window ⇒ the partner is genuinely
+///         down, so it falls back to self-first and forms alone. Exactly one node founds when both are
+///         present; the lower node always founds when it starts alone.
+///     </para>
+///     <para>
+///         <b>Residual trade-off (accepted).</b> Once the higher node observes the partner reachable it
+///         commits to peer-first — Akka's <c>JoinSeedNodeProcess</c> retries <c>InitJoin</c> forever and
+///         never self-forms. If the founder dies in the small window between the probe succeeding and
+///         the join completing, the higher node hangs unjoined until it is restarted (a restart re-runs
+///         the guard, finds the founder unreachable, and self-forms). We deliberately do NOT re-decide
+///         mid-handshake — that is exactly the retired <c>SelfFormAfter</c> failure mode. Instead
+///         <see cref="ClusterBootstrapCoordinator"/> makes the hang operator-visible with a warning if
+///         the node has not come Up within a bounded time after committing peer-first.
+///     </para>
+///     <para>
+///         This decides the join order BEFORE issuing a single <c>JoinSeedNodes</c>, from an explicit
+///         reachability signal — unlike a timer-based watchdog, which fires a <c>Join(self)</c>
+///         mid-handshake on a bare timeout and could not tell "no seed answered" from "a join is in
+///         flight", islanding a node a failover had just bounced (the rejected self-form-watchdog
+///         design; see <c>SelfFirstSeedBootstrapTests</c>).
+///     </para>
+/// </remarks>
+public static class ClusterBootstrapGuard
+{
+    private static readonly Regex SeedPattern =
+        new(@"^akka(?:\.[a-z0-9]+)?://[^@/]+@(?<host>[^:/]+):(?<port>\d+)", RegexOptions.IgnoreCase | RegexOptions.Compiled);
+
+    /// <summary>The analyzed bootstrap role of this node within its seed list.</summary>
+    /// <param name="Applies">True only when the guard engages: exactly two seeds, one of them this node.</param>
+    /// <param name="IsFounder">True when this node is the preferred founder (lower address) — form immediately, no probe.</param>
+    /// <param name="PartnerHost">The partner's advertised host (to probe when this node is the higher one); null when not applicable.</param>
+    /// <param name="PartnerPort">The partner's Akka port.</param>
+    /// <param name="SelfFirstOrder"><c>[self, partner]</c> — self-first; forms a new cluster when no peer answers.</param>
+    /// <param name="PeerFirstOrder"><c>[partner, self]</c> — peer-first; joins the partner's cluster, never self-forms while it is up.</param>
+    public sealed record BootstrapRole(
+        bool Applies,
+        bool IsFounder,
+        string? PartnerHost,
+        int PartnerPort,
+        string[] SelfFirstOrder,
+        string[] PeerFirstOrder);
+
+    /// <summary>
+    /// Analyzes the configured seed list to decide this node's bootstrap role. The guard engages only
+    /// for the pair shape (exactly two seeds, one of them this node); any other shape
+    /// (single-seed / single-node install, three+ seeds, self absent) returns
+    /// <see cref="BootstrapRole.Applies"/> = false and the caller joins the configured seeds unchanged.
+    /// </summary>
+    /// <param name="seeds">The configured seed URIs (self-first by convention, but order is not relied on here).</param>
+    /// <param name="selfHost">This node's advertised host (<c>NodeOptions.NodeHostname</c>).</param>
+    /// <param name="selfPort">This node's Akka remoting port (<c>NodeOptions.RemotingPort</c>).</param>
+    /// <returns>The analyzed role; never null.</returns>
+    public static BootstrapRole Analyze(IReadOnlyList<string> seeds, string selfHost, int selfPort)
+    {
+        ArgumentNullException.ThrowIfNull(seeds);
+        ArgumentException.ThrowIfNullOrWhiteSpace(selfHost);
+
+        var na = new BootstrapRole(false, false, null, 0, Array.Empty<string>(), Array.Empty<string>());
+        if (seeds.Count != 2) return na;
+
+        var selfKey = Key(selfHost, selfPort);
+        string? self = null, partner = null;
+        string? partnerHost = null;
+        var partnerPort = 0;
+
+        foreach (var seed in seeds)
+        {
+            if (!TryParse(seed, out var host, out var port)) return na;
+            if (string.Equals(Key(host, port), selfKey, StringComparison.OrdinalIgnoreCase))
+                self = seed;
+            else
+            {
+                partner = seed;
+                partnerHost = host;
+                partnerPort = port;
+            }
+        }
+
+        // Self must be exactly one of the two, and the other must be a distinct partner.
+        if (self is null || partner is null || partnerHost is null) return na;
+
+        var selfFirst = new[] { self, partner };
+        var peerFirst = new[] { partner, self };
+
+        // Deterministic tie-break: the lower canonical address is the preferred founder. Both nodes
+        // compute the same comparison (same two seeds), so exactly one is the founder. Case-INSENSITIVE
+        // to match how self/partner are classified above (and StartupValidator.SeedNodeIsSelf):
+        // container/DNS hostnames are conventionally case-insensitive, and a casing difference between
+        // the two sides' seed config must NOT make both think they are the founder (which would reopen
+        // the very split this guard closes).
+        var isFounder = string.Compare(
+            Key(selfHost, selfPort),
+            Key(partnerHost, partnerPort),
+            StringComparison.OrdinalIgnoreCase) < 0;
+
+        return new BootstrapRole(true, isFounder, partnerHost, partnerPort, selfFirst, peerFirst);
+    }
+
+    /// <summary>
+    /// The order the higher (non-founder) node uses once it has learned whether its partner is
+    /// reachable: peer-first when the founder is up (join it), self-first when the founder is absent
+    /// (form alone — cold-start-alone preserved).
+    /// </summary>
+    /// <param name="role">The analyzed role (must be applicable and NOT the founder).</param>
+    /// <param name="partnerReachable">Whether the partner's Akka endpoint became reachable within the probe window.</param>
+    /// <returns>The seed order to pass to <c>Cluster.JoinSeedNodes</c>.</returns>
+    public static string[] HigherNodeOrder(BootstrapRole role, bool partnerReachable)
+    {
+        ArgumentNullException.ThrowIfNull(role);
+        return partnerReachable ? role.PeerFirstOrder : role.SelfFirstOrder;
+    }
+
+    /// <summary>Parses <c>akka.tcp://system@host:port[/...]</c> into host + port.</summary>
+    /// <param name="seed">The seed URI.</param>
+    /// <param name="host">The parsed advertised host.</param>
+    /// <param name="port">The parsed Akka port.</param>
+    /// <returns>True when the seed matched the expected shape.</returns>
+    public static bool TryParse(string? seed, out string host, out int port)
+    {
+        host = string.Empty;
+        port = 0;
+        if (string.IsNullOrWhiteSpace(seed)) return false;
+
+        var m = SeedPattern.Match(seed.Trim());
+        if (!m.Success) return false;
+        if (!int.TryParse(m.Groups["port"].Value, out port)) return false;
+        host = m.Groups["host"].Value;
+        return true;
+    }
+
+    private static string Key(string host, int port) => $"{host}:{port}";
+}
@@ -114,4 +114,50 @@ public class ClusterOptions
    /// dials forever — log noise and a defeated validation intent (review 01).
    /// </summary>
    public bool AllowSingleNodeCluster { get; set; }
+
+    /// <summary>
+    /// The simultaneous-cold-start split-brain guard (<c>ScadaBridge:Cluster:BootstrapGuard</c>).
+    /// Default OFF — a dark switch, so existing deployments and tests keep Akka's config-driven
+    /// self-first auto-join byte-identical. See <see cref="ClusterBootstrapGuard"/> for the
+    /// decision logic and <c>ClusterBootstrapCoordinator</c> for the runtime (Gitea #33).
+    /// </summary>
+    public ClusterBootstrapGuardOptions BootstrapGuard { get; set; } = new();
+}
+
+/// <summary>
+/// Configuration for the simultaneous-cold-start split-brain guard. Both nodes of a pair are
+/// self-first seeds so <b>either</b> can cold-start alone when its partner is dead — but the cost
+/// is that when BOTH cold-start at the same instant each runs Akka's <c>FirstSeedNodeProcess</c>,
+/// times out waiting for the other, and forms its own single-node cluster (a split brain that does
+/// not auto-merge). When <see cref="Enabled"/>, the node does NOT auto-join from its config seeds
+/// (<c>AkkaHostedService.BuildHocon</c> emits an empty seed list); a coordinator picks the join
+/// order — founder self-first / joiner peer-first — after a reachability probe. See
+/// <see cref="ClusterBootstrapGuard"/>.
+/// </summary>
+public sealed class ClusterBootstrapGuardOptions
+{
+    /// <summary>
+    /// Whether the guard is active. Default <see langword="false"/> — Akka auto-joins from the
+    /// config seed list exactly as before. Only meaningful on a node that is one of its own two
+    /// pair seeds; inert everywhere else (single-seed site node, self absent, 3+ seeds).
+    /// </summary>
+    public bool Enabled { get; set; }
+
+    /// <summary>
+    /// How long the higher-address node probes its partner's Akka endpoint before concluding the
+    /// partner is dead and forming alone. Must comfortably exceed the partner's worst-case
+    /// process-start-to-Akka-bind time so a slow-but-alive partner is never mistaken for a dead one
+    /// (which would re-open the split). Default 25s. Validated <c>&gt; 0</c> at boot when the guard is on.
+    /// </summary>
+    public int PartnerProbeSeconds { get; set; } = 25;
+
+    /// <summary>
+    /// The interval between partner reachability probes, in milliseconds. Default 500ms.
+    /// </summary>
+    public int PartnerProbeIntervalMs { get; set; } = 500;
+
+    /// <summary>
+    /// The per-probe TCP connect timeout, in milliseconds. Default 1000ms.
+    /// </summary>
+    public int ProbeConnectTimeoutMs { get; set; } = 1000;
 }
@@ -30,6 +30,8 @@ public sealed class ClusterOptionsValidator : OptionsValidatorBase<ClusterOption
    /// <inheritdoc />
    protected override void Validate(ValidationBuilder builder, ClusterOptions options)
    {
+        ValidateBootstrapGuard(builder, options.BootstrapGuard);
+
        // The design doc states "both nodes are seed nodes — each node lists
        // both itself and its partner" so a properly-configured deployment lists
        // two. Accepting a single-seed configuration silently defeats the
@@ -78,4 +80,33 @@ public sealed class ClusterOptionsValidator : OptionsValidatorBase<ClusterOption
            + "oldest node can run as an isolated single-node cluster during a partition while the "
            + "younger node forms its own, producing two live clusters.");
    }
+
+    /// <summary>
+    /// When the bootstrap guard is enabled (Gitea #33), its timing knobs must be positive. A
+    /// zero/negative <see cref="ClusterBootstrapGuardOptions.PartnerProbeSeconds"/> in particular
+    /// silently degrades the guard to "never wait, always conclude the partner is dead" — the
+    /// higher node would form alone immediately and re-open the very split the guard exists to
+    /// close. Fail fast at boot rather than producing that silent degradation. Nothing is checked
+    /// when the guard is off (the knobs are inert), so a disabled guard never blocks a boot.
+    /// </summary>
+    private static void ValidateBootstrapGuard(ValidationBuilder builder, ClusterBootstrapGuardOptions? guard)
+    {
+        if (guard is null || !guard.Enabled)
+        {
+            return;
+        }
+
+        builder.RequireThat(guard.PartnerProbeSeconds > 0,
+            $"ClusterOptions.BootstrapGuard.PartnerProbeSeconds must be > 0 when the guard is enabled "
+            + $"(was {guard.PartnerProbeSeconds}); a non-positive value makes the higher node conclude its "
+            + "partner is dead without waiting and form alone, re-opening the split the guard prevents.");
+
+        builder.RequireThat(guard.PartnerProbeIntervalMs > 0,
+            $"ClusterOptions.BootstrapGuard.PartnerProbeIntervalMs must be > 0 when the guard is enabled "
+            + $"(was {guard.PartnerProbeIntervalMs}).");
+
+        builder.RequireThat(guard.ProbeConnectTimeoutMs > 0,
+            $"ClusterOptions.BootstrapGuard.ProbeConnectTimeoutMs must be > 0 when the guard is enabled "
+            + $"(was {guard.ProbeConnectTimeoutMs}).");
+    }
 }
@@ -10,9 +10,9 @@ public class Site
    public string SiteIdentifier { get; set; }
    /// <summary>Optional description of the site.</summary>
    public string? Description { get; set; }
-    /// <summary>Akka remote address for site node A (ClusterClient contact point).</summary>
+    /// <summary>Akka remote address for site node A.</summary>
    public string? NodeAAddress { get; set; }
-    /// <summary>Akka remote address for site node B (ClusterClient contact point).</summary>
+    /// <summary>Akka remote address for site node B.</summary>
    public string? NodeBAddress { get; set; }
    /// <summary>gRPC endpoint for site node A used by the central SiteStreamGrpcClient.</summary>
    public string? GrpcNodeAAddress { get; set; }
@@ -28,7 +28,7 @@ public enum SiteCallRelayOutcome
    NotParked,

    /// <summary>
-    /// The owning site could not be reached (offline / no ClusterClient route /
+    /// The owning site could not be reached (offline / no route to the site /
    /// relay timed out). The action was NOT applied; the operator may retry once
    /// the site is back online.
    /// </summary>
@@ -19,8 +19,8 @@ namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.DebugView;
 /// </para>
 /// <para>
 /// A new dedicated message type (<c>DebugViewInstanceNotFound</c>) was
-/// considered but rejected: the ClusterClient / ClusterClientReceptionist
-/// channel is typed on the request side and the bridge actor is already
+/// considered but rejected: the site↔central channel is typed on the request
+/// side and the bridge actor is already
 /// pattern-matching on <c>DebugViewSnapshot</c> for the initial-snapshot TCS
 /// in <c>DebugStreamService</c>.  Introducing a second reply type would require
 /// every consumer to handle an additional <c>Ask</c> result union — more change
@@ -1,7 +1,7 @@
 namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;

 /// <summary>
-/// Site→central (over ClusterClient) on node startup: the node's local deployed inventory,
+/// Site→central on node startup: the node's local deployed inventory,
 /// so central can reply with fetch tokens for whatever the node is missing or stale
 /// (self-heal a node that was down during a deploy).
 /// </summary>
@@ -108,8 +108,8 @@ public record SiteHealthReport(

 /// <summary>
 /// Broadcast wrapper used between central nodes to keep per-node
-/// CentralHealthAggregator state in sync. ClusterClient load-balances each
-/// incoming SiteHealthReport to one central node; that node re-publishes
+/// CentralHealthAggregator state in sync. A site delivers each
+/// SiteHealthReport to ONE central node; that node re-publishes
 /// this wrapper on a DistributedPubSub topic so the peer node's aggregator
 /// also processes the report (idempotently — sequence numbers guard against
 /// double-counting).
@@ -9,7 +9,7 @@ namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.Instance;
 /// <b>Site-local only.</b> The optional <see cref="Predicate"/> is a non-serializable
 /// in-process delegate, so this message MUST flow only within a single site node's
 /// actor system (script execution → Instance Actor). It is never sent across the
-/// ClusterClient / gRPC boundary. The value-equality form (<see cref="TargetValueEncoded"/>)
+/// site↔central gRPC boundary. The value-equality form (<see cref="TargetValueEncoded"/>)
 /// would serialize, but the routed/inbound variant is deliberately out of scope here.
 /// </para>
 /// </summary>
@@ -6,7 +6,7 @@ namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.Management;
 /// Bidirectional name registry for management command records. The registry contains
 /// exactly the non-abstract <c>*Command</c> types declared in the
 /// <c>ZB.MOM.WW.ScadaBridge.Commons.Messages.Management</c> namespace; these are the commands that
-/// travel over the HTTP / ClusterClient management boundary.
+/// travel over the HTTP management boundary.
 /// </summary>
 /// <remarks>
 /// <see cref="Resolve"/> and <see cref="GetCommandName"/> are symmetric:
@@ -2,7 +2,7 @@ namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.Management;

 // Schema-library authoring commands. The reusable named JSON-Schema
 // library (the SharedSchema entity + ISharedSchemaRepository) gains its CRUD
-// surface here. These records travel the same HTTP / ClusterClient management
+// surface here. These records travel the same HTTP management
 // boundary as every other *Command and are auto-discovered by reflection in
 // ManagementCommandRegistry (no manual registry entry needed).
 //
@@ -14,7 +14,7 @@ namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.RemoteQuery;
 /// </para>
 /// <para>
 /// Either site node may receive this: <c>SiteCommunicationActor</c> is registered per node
-/// (not as a singleton), and ClusterClient contact rotation reaches whichever answers. That is
+/// (not as a singleton), and central's gRPC dial reaches whichever node answers. That is
 /// fine — <c>Cluster.Leave(address)</c> is valid from any member, and the target is resolved
 /// from cluster state rather than from who received the message.
 /// </para>
@@ -89,7 +89,7 @@ public class CentralCommunicationActor : ReceiveActor
    /// Default Ask timeout for routing audit ingest commands to the
    /// Effective Ask timeout for audit ingest routing. Defaults to
    /// <see cref="Grpc.SiteStreamGrpcServer.AuditIngestAskTimeout"/> (30 s) — the two
-    /// audit-ingest transports (gRPC vs ClusterClient) now share one source of truth
+    /// audit-ingest entry points (the site stream server and the control plane) share one source of truth
    /// for the timeout. Overridable via the constructor so tests can exercise the
    /// timeout/fault path without waiting 30 s. When the window is exceeded the Ask
    /// faults and that fault is piped back to the caller as a
@@ -167,8 +167,8 @@ public class CentralCommunicationActor : ReceiveActor
        });

        // Notification Outbox ingest: a site forwards a buffered NotificationSubmit to the
-        // central cluster via ClusterClient. Forward to the outbox proxy so the original
-        // Sender (the site's ClusterClient path) is preserved and the NotificationSubmitAck
+        // central cluster. Forward to the outbox proxy so the original
+        // Sender (the site's transport path) is preserved and the NotificationSubmitAck
        // routes straight back to the site.
        Receive<NotificationSubmit>(HandleNotificationSubmit);

@@ -186,9 +186,9 @@ public class CentralCommunicationActor : ReceiveActor
        });

        // Audit Log site→central ingest: a site forwards a batch of audit
-        // events to the central cluster via ClusterClient. Ask the ingest proxy
+        // events to the central cluster. Ask the ingest proxy
        // and pipe the IngestAuditEventsReply back to the original Sender (the
-        // site's ClusterClient path) so the site can flip its rows to Forwarded.
+        // site's transport path) so the site can flip its rows to Forwarded.
        Receive<IngestAuditEventsCommand>(HandleIngestAuditEvents);

        // Audit Log combined-telemetry ingest: routes to the same proxy
@@ -196,9 +196,9 @@ public class CentralCommunicationActor : ReceiveActor
        Receive<IngestCachedTelemetryCommand>(HandleIngestCachedTelemetry);

        // Startup reconciliation: a site node forwards its local deployed inventory on
-        // startup via ClusterClient. Resolve the scoped ReconcileService, diff the
+        // startup. Resolve the scoped ReconcileService, diff the
        // inventory against central's expected set, and pipe the ReconcileSiteResponse
-        // (gap fetch tokens + orphans) straight back to the site node's ClusterClient.
+        // (gap fetch tokens + orphans) straight back to the site node.
        Receive<ReconcileSiteRequest>(HandleReconcileSiteRequest);
    }

@@ -254,7 +254,7 @@ public class CentralCommunicationActor : ReceiveActor
        }

        // Capture Sender before the async/PipeTo — Akka resets Sender between
-        // dispatches. The reply is piped straight back to the site's ClusterClient.
+        // dispatches. The reply is piped straight back to the calling site node.
        // On an Ask timeout or a faulted reply, PipeTo delivers a Status.Failure to
        // replyTo: the fault propagates to the caller rather than being swallowed.
        // The site's own Ask through this path then faults, and the site drain loop
@@ -285,10 +285,10 @@ public class CentralCommunicationActor : ReceiveActor
    }

    /// <summary>
-    /// Startup reconciliation (site→central over ClusterClient): resolve the scoped
+    /// Startup reconciliation (site→central): resolve the scoped
    /// <see cref="ReconcileService"/> in a DI scope, diff the node's reported inventory
    /// against central's expected set, and pipe the <see cref="ReconcileSiteResponse"/>
-    /// back to the site node's ClusterClient path. The actor stays thin — all the diff
+    /// back to the site node's transport path. The actor stays thin — all the diff
    /// and staging logic lives in the service. Mirrors the DB-access pattern used by
    /// <see cref="LoadSiteAddressesFromDb"/> (Task.Run + CreateScope + PipeTo) and the
    /// Sender-preservation pattern of <see cref="HandleIngestAuditEvents"/>.
@@ -331,7 +331,7 @@ public class CentralCommunicationActor : ReceiveActor
        MarkHeartbeatLocally(heartbeat);

        // Fan the heartbeat out to the peer central node so BOTH aggregators mark
-        // it, regardless of which central node the site's ClusterClient delivered
+        // it, regardless of which central node the site delivered
        // to. Without this, a heartbeat that only ever reaches one node leaves the
        // other node's aggregator blind to that site's liveness after a failover
        // (arch review 02, Low). MarkHeartbeat is idempotent (timestamp overwrite),
@@ -358,7 +358,7 @@ public class CentralCommunicationActor : ReceiveActor
    }

    /// <summary>
-    /// Handles a report delivered directly from a site (via ClusterClient):
+    /// Handles a report delivered directly from a site:
    /// process locally, then fan out to the peer central node so its
    /// aggregator stays in sync.
    /// </summary>
@@ -407,7 +407,7 @@ public class CentralCommunicationActor : ReceiveActor
    // HandleConnectionStateChanged removed — no production
    // caller emitted ConnectionStateChanged, so the workflow ran only in tests.
    // Disconnect detection is owned by the transport layers (gRPC keepalive +
-    // ClusterClient/Ask timeout).
+    // Ask timeout).

    private void HandleSiteEnvelope(SiteEnvelope envelope)
    {
@@ -491,8 +491,8 @@ public class CentralCommunicationActor : ReceiveActor

    private void HandleSiteAddressCacheLoaded(SiteAddressCacheLoaded msg)
    {
-        // Per-transport per-site resource reconciliation (create/stop ClusterClients, or
-        // build/drop gRPC channel pairs). Runs on the actor thread each refresh tick.
+        // Per-site transport resource reconciliation (build/drop gRPC channel
+        // pairs). Runs on the actor thread each refresh tick.
        _transport.ReconcileSites(msg);

        // Self-healing eviction: a site deleted from configuration would otherwise
@@ -525,7 +525,7 @@ public class CentralCommunicationActor : ReceiveActor

        // Subscribe to the peer-replication topic so we receive health reports
        // delivered to the other central node and keep our local aggregator
-        // in sync (ClusterClient load-balances reports across nodes).
+        // in sync (a site may deliver its report to either central node).
        // Tolerant of non-clustered hosts (TestKit) where the extension is absent.
        try
        {
@@ -581,7 +581,7 @@ public record RefreshSiteAddresses;
 /// discipline. The producer wraps the constructed buckets with
 /// <c>List&lt;T&gt;.AsReadOnly()</c> before piping to Self.
 /// </summary>
-/// <param name="SiteContacts">Akka ClusterClient contact addresses per site (from NodeA/NodeBAddress).</param>
+/// <param name="SiteContacts">Akka remote addresses per site (from NodeA/NodeBAddress).</param>
 /// <param name="KnownSiteIds">Every configured site id, address-bearing or not, for aggregator pruning.</param>
 /// <param name="GrpcContacts">
 /// gRPC endpoint pairs per site (from GrpcNodeA/GrpcNodeBAddress) — the streaming path's columns,
@@ -603,8 +603,8 @@ public readonly record struct SiteGrpcEndpoints(string? NodeA, string? NodeB);
 /// <summary>
 /// Peer-replication envelope for a site heartbeat, fanned out over the same
 /// DistributedPubSub topic as <see cref="SiteHealthReportReplica"/> so both
-/// central aggregators mark heartbeats regardless of which node the site's
-/// ClusterClient delivered to. Only ever travels central↔central (same assembly
+/// central aggregators mark heartbeats regardless of which node the site
+/// delivered to. Only ever travels central↔central (same assembly
 /// version, rolled together), so the default reflective serializer is safe.
 /// </summary>
 public sealed record SiteHeartbeatReplica(HeartbeatMessage Heartbeat);
@@ -94,10 +94,10 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers
    /// <summary>
    /// Initializes the debug stream bridge actor and registers message handlers.
    /// </summary>
-    /// <param name="siteIdentifier">Site identifier for targeting ClusterClient messages and logging.</param>
+    /// <param name="siteIdentifier">Site identifier for targeting site-addressed messages and logging.</param>
    /// <param name="instanceUniqueName">Unique name of the instance whose debug stream is being bridged.</param>
    /// <param name="correlationId">Correlation id for the debug session.</param>
-    /// <param name="centralCommunicationActor">Actor used to forward ClusterClient messages to the site.</param>
+    /// <param name="centralCommunicationActor">Actor used to forward site-addressed messages to the site.</param>
    /// <param name="onEvent">Callback invoked on each received debug event.</param>
    /// <param name="onTerminated">Callback invoked when the stream terminates.</param>
    /// <param name="grpcFactory">Factory for creating gRPC streaming clients.</param>
@@ -124,7 +124,7 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers
        _grpcNodeAAddress = grpcNodeAAddress;
        _grpcNodeBAddress = grpcNodeBAddress;

-        // Initial snapshot response from the site (via ClusterClient).
+        // Initial snapshot response from the site.
        // If the site reports InstanceNotFound=true the instance is not
        // deployed there. Under the stream-first lifecycle the gRPC stream
        // was already opened in PreStart, so the not-found path must tear it down
@@ -4,13 +4,11 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Actors;

 /// <summary>
 /// The central→site command-send seam, injected into <see cref="CentralCommunicationActor"/>
-/// below the <see cref="SiteEnvelope"/> handler. Exactly one implementation is active per node,
-/// chosen by <c>ScadaBridge:Communication:SiteTransport</c>:
-/// <list type="bullet">
-///   <item><see cref="AkkaSiteTransport"/> — today's per-site <c>ClusterClient</c> path (default).</item>
-///   <item><see cref="ZB.MOM.WW.ScadaBridge.Communication.Grpc.GrpcSiteTransport"/> — the site
-///     <c>SiteCommandService</c> gRPC plane.</item>
-/// </list>
+/// below the <see cref="SiteEnvelope"/> handler. The sole implementation is
+/// <see cref="ZB.MOM.WW.ScadaBridge.Communication.Grpc.GrpcSiteTransport"/>, over the site
+/// <c>SiteCommandService</c> gRPC plane. The seam predates that: it was introduced so the Akka
+/// <c>ClusterClient</c> path and the gRPC plane could run side by side during the migration, and
+/// is kept because it is the natural substitution point for tests.
 /// The producers above the seam (<c>CommunicationService</c>'s 27 commands, <c>SiteCallAuditActor</c>'s
 /// 2 parked relays, and <c>DebugStreamBridgeActor</c>'s subscribe/unsubscribe) are unchanged — they
 /// still <c>Ask</c>/<c>Tell</c> a <see cref="SiteEnvelope"/> to the actor, which delegates here.
@@ -27,14 +25,14 @@ public interface ISiteCommandTransport
    /// delivered to <paramref name="replyTo"/> — for an <c>Ask</c> that is the temporary ask actor
    /// (completing the caller's task); for a <c>Tell</c>-with-sender (the debug bridge) that is the
    /// originating actor. A message with no route (an unknown site) is warned and dropped so the
-    /// caller's <c>Ask</c> times out, exactly as today's ClusterClient path behaves.
+    /// caller's <c>Ask</c> times out — central never buffers for an unreachable site.
    /// </summary>
    /// <param name="envelope">The site-addressed command envelope.</param>
    /// <param name="replyTo">Where a reply (or a <see cref="Status.Failure"/>) is delivered.</param>
    void Send(SiteEnvelope envelope, IActorRef replyTo);

    /// <summary>
-    /// Reconciles per-site transport resources (ClusterClients for Akka, channel pairs for gRPC)
+    /// Reconciles per-site transport resources (gRPC channel pairs)
    /// against the freshly loaded site set. Called once per DB refresh tick with the same cache
    /// message the actor already receives — the ONE DB-poll loop feeds both transports.
    /// </summary>
@@ -536,7 +536,7 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
        if (_retryCount > MaxRetries)
        {
            // Give up the stream, but do NOT stop the aggregator: the periodic reconcile
-            // still refreshes the cache from ClusterClient snapshots, so the page keeps a
+            // still refreshes the cache from site snapshots, so the page keeps a
            // (slower) live-ish view rather than going dark. A later reconcile-triggered
            // reconnect is not attempted here; the stream is simply left down.
            _log.Error("Site-alarm gRPC stream for {0} exceeded max retries ({1}); leaving stream down, " +
@@ -21,8 +21,8 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Actors;
 /// </summary>
 /// <remarks>
 /// <para>
-/// Two transports call this one unit: the Akka <see cref="SiteCommunicationActor"/>
-/// (ClusterClient) and the new <c>SiteCommandGrpcService</c> (gRPC). Centralising the
+/// Two entry points call this one unit: the site-side <see cref="SiteCommunicationActor"/>
+/// and <c>SiteCommandGrpcService</c>. Centralising the
 /// table means the two can never drift on where a command goes. Each transport keeps
 /// its own send mechanics — the actor <c>Forward</c>s (preserving the Ask sender), the
 /// gRPC service <c>Ask</c>s and encodes the reply — but both read the same
@@ -219,9 +219,9 @@ public sealed class SiteCommandDispatcher
    private static Route Immediate(object reply) => new(RouteDisposition.ImmediateReply, null, reply);

    /// <summary>
-    /// The actor's failover path: resolve the standby AND issue the leave in one step (today's
-    /// coupled behaviour over ClusterClient, where <c>Tell</c> merely enqueues the ack so leave
-    /// order is immaterial), then return the ack.
+    /// The actor's failover path: resolve the standby AND issue the leave in one step (coupled,
+    /// because <c>Tell</c> merely enqueues the ack so leave order is immaterial here), then
+    /// return the ack.
    /// </summary>
    /// <param name="msg">The failover command.</param>
    /// <returns>The ack to send back.</returns>
@@ -283,8 +283,8 @@ public sealed class SiteCommandDispatcher
        }
        catch (Exception ex)
        {
-            // A fault must be reported to the operator, never thrown into supervision (over
-            // ClusterClient) or surfaced as a broken stream (over gRPC).
+            // A fault must be reported to the operator, never thrown into supervision or
+            // surfaced as a broken stream (over gRPC).
            return new FailoverOutcome(
                new SiteFailoverAck(
                    msg.CorrelationId, Accepted: false, TargetAddress: null, ErrorMessage: ex.Message),
@@ -16,9 +16,10 @@ using ZB.MOM.WW.ScadaBridge.Commons.Messages.RemoteQuery;
 namespace ZB.MOM.WW.ScadaBridge.Communication.Actors;

 /// <summary>
-/// Site-side actor that receives messages from central via ClusterClient and routes
-/// them to the appropriate local actors. Also sends heartbeats and health reports
-/// to central via the registered ClusterClient.
+/// Site-side actor that receives messages from central over the site-hosted
+/// <c>SiteCommandService</c> gRPC plane and routes them to the appropriate local actors.
+/// Also sends heartbeats and health reports to central over its
+/// <see cref="ICentralTransport"/> (<see cref="Grpc.GrpcCentralTransport"/> in production).
 ///
 /// Routes all 8 message patterns to local handlers.
 /// </summary>
@@ -157,15 +158,15 @@ public class SiteCommunicationActor : ReceiveActor, IWithTimers
        // the SITE-SPECIFIC role, because that is what site singletons (the Deployment
        // Manager) are placed on. Either node may receive this (the actor is per-node, not a
        // singleton, and contact rotation picks whichever answers); the target is resolved
-        // from cluster state, not from who received the message. Over ClusterClient the ack
-        // Tell merely enqueues, so the dispatcher resolves-and-leaves in one step (the gRPC
-        // transport defers the leave to keep ack-before-Leave — see PrepareFailover).
+        // from cluster state, not from who received the message. The gRPC transport defers the
+        // leave to keep ack-before-Leave — see PrepareFailover. (Under the removed ClusterClient
+        // path the ack Tell merely enqueued, so the dispatcher resolved-and-left in one step.)
        Receive<TriggerSiteFailover>(msg => Sender.Tell(_dispatcher.HandleFailover(msg)));

-        // The seven site→central sends now delegate to the injected transport (ClusterClient by
-        // default, gRPC when configured). Each handler captures the current Sender as the reply
-        // target so central's reply routes straight back to the waiting Ask, not through this
-        // actor — the exact sender-forwarding the ClusterClient path relied on. The per-message
+        // The seven site→central sends delegate to the injected transport (GrpcCentralTransport in
+        // production). Each handler captures the current Sender as the reply target so central's
+        // reply routes straight back to the waiting Ask, not through this actor — preserving the
+        // sender-forwarding the original ClusterClient path relied on. The per-message
        // "no transport / not-accepted" fallbacks live inside the transport now.

        // Notification Outbox: forward a buffered notification (S&F forwarder's Ask → ack back).
@@ -229,9 +230,9 @@ public class SiteCommunicationActor : ReceiveActor, IWithTimers
    /// Executes a resolved command route within the actor: Forward to the target (preserving the
    /// central Ask sender so the reply routes straight back to the waiting Ask), or — when a
    /// null-guarded handler is unregistered — Tell the caller the dispatcher's synthetic reply.
-    /// The fire-and-forget disposition (UnsubscribeDebugView) is a plain Forward here, exactly as
-    /// before: over ClusterClient the site never acked it, so the synthetic ack is a gRPC-only
-    /// concern.
+    /// The fire-and-forget disposition (UnsubscribeDebugView) is a plain Forward here: the site
+    /// never acked it under the original ClusterClient path either, so the synthetic ack is a
+    /// gRPC-only concern.
    /// </summary>
    /// <param name="command">The migrated central→site command to route.</param>
    private void DispatchCommand(object command)
@@ -13,7 +13,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
  ///
  /// Direction: SITE is the client, CENTRAL is the server — the inverse of
  /// SiteStreamService, where central dials the site. That asymmetry is deliberate
-  /// and mirrors the direction the Akka ClusterClient traffic flows today: these
+  /// and mirrors the direction the Akka ClusterClient traffic used to flow: these
  /// seven calls are exactly the seven messages SiteCommunicationActor sends to
  /// /user/central-communication.
  ///
@@ -1,4 +1,5 @@
 using Akka.Cluster;
+using ZB.MOM.WW.Health.Akka;

 namespace ZB.MOM.WW.ScadaBridge.Communication.ClusterState;

@@ -32,17 +33,13 @@ public static class ActiveNodeEvaluator
    /// <param name="cluster">The Akka cluster to evaluate.</param>
    /// <param name="role">Optional role scope; when set, only members with this role are considered.</param>
    /// <returns><c>true</c> when self is Up and the oldest Up member in the role scope.</returns>
-    public static bool SelfIsOldestUp(Cluster cluster, string? role = null)
-    {
-        var self = cluster.SelfMember;
-        if (self.Status != MemberStatus.Up)
-            return false;
-        if (role != null && !self.HasRole(role))
-            return false;
-
-        return cluster.State.Members
-            .Where(m => m.Status == MemberStatus.Up)
-            .Where(m => role == null || m.HasRole(role))
-            .All(m => m.UniqueAddress.Equals(self.UniqueAddress) || self.IsOlderThan(m));
-    }
+    /// <remarks>
+    /// Delegates to the shared <see cref="ClusterActiveNode.SelfIsActive"/>
+    /// (<c>ZB.MOM.WW.Health.Akka</c> 0.3.0), which is this evaluator promoted into the shared library
+    /// after OtOpcUa independently needed the same rule and wrote its own third copy. The rule now has
+    /// one implementation family-wide; this method survives as Communication's entry point into it, so
+    /// the delivery gate and heartbeat keep their existing call sites and layering.
+    /// </remarks>
+    public static bool SelfIsOldestUp(Cluster cluster, string? role = null) =>
+        ClusterActiveNode.SelfIsActive(cluster, role);
 }
@@ -139,7 +139,7 @@ public class CommunicationService
    /// Queries a site for the currently-applied deployment
    /// identity of a single instance. Used by the Deployment Manager before a
    /// re-deploy to reconcile against the site's actual state. Sent over the
-    /// existing ClusterClient command/control transport; the Ask times out (no
+    /// gRPC command/control transport; the Ask times out (no
    /// central buffering) if the site is unreachable, and the caller falls
    /// through to a normal deploy.
    /// </summary>
@@ -16,8 +16,8 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
 /// <para>
 /// The seven pairs mirror, one for one, the seven messages
 /// <c>SiteCommunicationActor</c> forwards to <c>/user/central-communication</c> over
-/// Akka <c>ClusterClient</c> today. Both transports carry the SAME message types
-/// end-to-end — central's handlers are untouched by the migration — so this mapper is
+/// the control plane. The in-process and wire representations carry the SAME message types
+/// end-to-end — central's handlers were untouched by the migration — so this mapper is
 /// the only place the two representations meet, and a field that does not survive a
 /// round-trip here is a field the gRPC transport silently drops.
 /// </para>
@@ -14,7 +14,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
 /// <summary>
 /// Central-hosted gRPC face of the seven site→central control messages
 /// (<c>Protos/central_control.proto</c>). Decodes each request onto the SAME in-process
-/// message type the Akka <c>ClusterClient</c> path already carries, <c>Ask</c>s
+/// message type the removed Akka <c>ClusterClient</c> path used to carry, <c>Ask</c>s
 /// <see cref="Actors.CentralCommunicationActor"/>, and encodes the reply back.
 /// </summary>
 /// <remarks>
@@ -26,8 +26,8 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
 /// </para>
 /// <para>
 /// <b>Zero handler logic lives here.</b> Every RPC lands on a receive
-/// <c>CentralCommunicationActor</c> already implements for the ClusterClient path, so the two
-/// transports cannot drift in behaviour: the actor is the single implementation, and this class
+/// <c>CentralCommunicationActor</c> already implements, so transport and handler cannot drift
+/// in behaviour: the actor is the single implementation, and this class
 /// is a codec plus an <c>Ask</c>. That is also why the service takes the actor through
 /// <see cref="SetReady"/> rather than resolving anything from DI — the actor is created by the
 /// host's Akka bootstrap, not by the container.
@@ -36,8 +36,8 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
 /// <b>Fault semantics deliberately differ from <see cref="SiteStreamGrpcServer"/>'s ingest
 /// RPCs.</b> That server answers a failed audit ingest with an EMPTY <c>IngestAck</c>; this one
 /// fails the call with a non-OK status. Both leave the site's rows <c>Pending</c> for the next
-/// drain, so the outcome is the same — but this service replaces the ClusterClient path, whose
-/// documented behaviour is to propagate the fault (<c>CentralCommunicationActor</c>'s
+/// drain, so the outcome is the same — but this service replaced the ClusterClient path, whose
+/// documented behaviour was to propagate the fault (<c>CentralCommunicationActor</c>'s
 /// <c>HandleIngestAuditEvents</c> pipes a <c>Status.Failure</c> back), and preserving that keeps
 /// a lost batch visible as a failure rather than as a successful call that acked nothing.
 /// </para>
@@ -94,8 +94,8 @@ public sealed class CentralControlGrpcService : CentralControlService.CentralCon
    /// actor exists and can receive, NOT that every downstream singleton proxy
    /// (<c>notification-outbox</c>, <c>audit-log-ingest</c>) has registered itself yet. Those
    /// register moments later in the same startup path, and the actor already answers a call
-    /// that beats them with the same "not available, retry" reply it gives on the ClusterClient
-    /// path — so gating readiness on them would add nothing but a longer window in which sites
+    /// that beats them with a "not available, retry" reply
+    /// — so gating readiness on them would add nothing but a longer window in which sites
    /// see <see cref="StatusCode.Unavailable"/>.
    /// </remarks>
    /// <param name="centralCommunicationActor">The central communication actor.</param>
@@ -11,8 +11,8 @@ using AkkaStatus = Akka.Actor.Status;
 namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;

 /// <summary>
-/// The <see cref="ICentralTransport"/> that carries the seven site→central sends over gRPC — the
-/// migration target for the Akka <c>ClusterClient</c> path. Each method encodes the message with
+/// The <see cref="ICentralTransport"/> that carries the seven site→central sends over gRPC —
+/// the replacement for the removed Akka <c>ClusterClient</c> path. Each method encodes the message with
 /// <see cref="CentralControlDtoMapper"/>, dials <c>CentralControlService</c> through the sticky
 /// <see cref="CentralChannelProvider"/>, and delivers the decoded reply (or a transient-failure
 /// signal) to the waiting Ask.
@@ -81,7 +81,7 @@ public sealed class GrpcSiteTransport : ISiteCommandTransport
        }
        catch (SiteChannelUnavailableException)
        {
-            // Parity with the Akka "no ClusterClient for site" path: warn and drop, so the caller's
+            // Parity with the removed Akka "no ClusterClient for site" path: warn and drop, so the caller's
            // Ask times out. Central never buffers.
            _logger.LogWarning(
                "No gRPC channel for site {SiteId}; dropping {Message} (caller's Ask will time out)",
@@ -22,7 +22,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
 /// </para>
 /// <para>
 /// <b>Coexistence — server-side only.</b> This makes the site ALSO listen on gRPC for commands;
-/// nothing central flips to gRPC here (that is T1B.3). Central still dials sites via ClusterClient.
+/// this was the server half, landed before central was flipped to dial it (T1B.3).
 /// </para>
 /// <para>
 /// <b>Fire-and-forget.</b> The Akka path never acks <c>UnsubscribeDebugView</c>, but a unary RPC
@@ -9,7 +9,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
 /// <summary>
 /// Raised when a site has no usable gRPC channel — an unknown site, or a site with neither
 /// <c>GrpcNodeAAddress</c> nor <c>GrpcNodeBAddress</c> configured. The gRPC transport treats this
-/// the way the Akka path treats "no ClusterClient for site": warn and drop, so the caller's Ask
+/// the way the removed Akka path treated "no ClusterClient for site": warn and drop, so the caller's Ask
 /// times out (central never buffers).
 /// </summary>
 public sealed class SiteChannelUnavailableException(string siteId)
@@ -17,7 +17,7 @@ import "Protos/sitestream.proto";
 //
 // Direction: SITE is the client, CENTRAL is the server — the inverse of
 // SiteStreamService, where central dials the site. That asymmetry is deliberate
-// and mirrors the direction the Akka ClusterClient traffic flows today: these
+// and mirrors the direction the Akka ClusterClient traffic used to flow: these
 // seven calls are exactly the seven messages SiteCommunicationActor sends to
 // /user/central-communication.
 //
@@ -8,8 +8,8 @@ namespace ZB.MOM.WW.ScadaBridge.Communication;

 /// <summary>
 /// Central-side startup-reconciliation handler. A site node, on startup, reports its
-/// local deployed inventory via <see cref="ReconcileSiteRequest"/> (delivered over
-/// ClusterClient to <see cref="Actors.CentralCommunicationActor"/>); this service diffs
+/// local deployed inventory via <see cref="ReconcileSiteRequest"/> (delivered over the
+/// control plane to <see cref="Actors.CentralCommunicationActor"/>); this service diffs
 /// it against central's expected deployed set and replies with fresh fetch tokens for the
 /// gap — instances the node is missing or has at a stale revision — plus the orphan names
 /// (present locally but no longer deployed centrally, which the node only logs).
@@ -28,6 +28,13 @@
    <PackageReference Include="Grpc.Net.Client" />
    <PackageReference Include="Grpc.Tools" PrivateAssets="All" />
    <PackageReference Include="ZB.MOM.WW.Configuration" />
+    <!-- For ClusterActiveNode — the shared "oldest Up member" primitive that ActiveNodeEvaluator was
+         promoted into. The store-and-forward delivery gate, the heartbeat IsActive stamp and the
+         /health/active tier must all name the SAME active node; delegating to one implementation is
+         what makes that structural. It lives in Health.Akka because that is the family's Akka-cluster
+         utility package (it already hosts AkkaClusterStatusPolicy and an endpoint gate); a dedicated
+         cluster package would be a better home if a third such primitive appears. -->
+    <PackageReference Include="ZB.MOM.WW.Health.Akka" />
  </ItemGroup>

  <ItemGroup>
@@ -123,6 +123,11 @@ public class RealOpcUaClient : IOpcUaClient
            endpoint = new EndpointDescription(endpointUrl);
        }

+        // The server advertises its own base address, which is frequently unroutable from here
+        // (a 0.0.0.0 wildcard bind, or an internal container/NAT hostname). Swap it back to the
+        // host the operator actually reached, or the session channel dials an unreachable address.
+        RewriteEndpointHostForReachability(endpoint, endpointUrl);
+
        var endpointConfig = EndpointConfiguration.Create(appConfig);
        var configuredEndpoint = new ConfiguredEndpoint(null, endpoint, endpointConfig);

@@ -157,6 +162,52 @@ public class RealOpcUaClient : IOpcUaClient
        await _subscription.CreateAsync(cancellationToken);
    }

+    /// <summary>
+    /// Rewrites a discovered endpoint's host/port to match the reachable URL the operator
+    /// configured, when the server advertised a different (unroutable) one.
+    /// </summary>
+    /// <remarks>
+    /// An OPC UA server advertises its base address from its OWN configuration, not the route the
+    /// client took to reach it — so <c>GetEndpoints</c> routinely returns a wildcard bind
+    /// (<c>opc.tcp://0.0.0.0:4840/…</c>) or an internal container/NAT hostname the client cannot
+    /// dial. The OPC Foundation session opens its channel to <see cref="EndpointDescription.EndpointUrl"/>
+    /// verbatim, so a 0.0.0.0 advertisement resolves to the client's OWN loopback and the connect
+    /// fails. This swaps only the authority (host + port) to the reachable one, preserving the
+    /// advertised scheme and path (e.g. <c>/OtOpcUa</c>). Mirrors <c>CoreClientUtils.SelectEndpoint</c>.
+    /// A no-op when the advertised host/port is already reachable (so an opc-plc server that
+    /// advertises its real container name is left untouched).
+    /// </remarks>
+    internal static void RewriteEndpointHostForReachability(EndpointDescription? endpoint, string discoveryUrl)
+    {
+        if (endpoint is null || string.IsNullOrWhiteSpace(endpoint.EndpointUrl))
+            return;
+        if (!Uri.TryCreate(discoveryUrl, UriKind.Absolute, out var reachable))
+            return;
+        if (!Uri.TryCreate(endpoint.EndpointUrl, UriKind.Absolute, out var advertised))
+            return;
+
+        // Already reachable — leave it exactly as advertised.
+        if (string.Equals(advertised.Host, reachable.Host, StringComparison.OrdinalIgnoreCase)
+            && advertised.Port == reachable.Port)
+            return;
+
+        // An IPv6 literal must keep its brackets, or the reconstructed authority (`::1:4840`) is
+        // ambiguous and unparseable. Uri.Host may or may not include them depending on the scheme,
+        // so bracket only when missing (never double-wrap).
+        var host = reachable.Host;
+        if (reachable.HostNameType == UriHostNameType.IPv6 && !host.StartsWith('['))
+            host = $"[{host}]";
+        // opc.tcp has no default port known to Uri, so Uri.Port is -1 when the text omits one.
+        // Prefer the reachable URL's port, fall back to the advertised one, and emit no ":port"
+        // at all when neither carries one (rather than a malformed ":-1").
+        var port = reachable.Port >= 0 ? reachable.Port : advertised.Port;
+        var authority = port >= 0 ? $"{host}:{port}" : host;
+        // Authority-only advertisements parse to AbsolutePath "/" — drop it so we don't append a
+        // spurious trailing slash the server never advertised.
+        var path = advertised.AbsolutePath == "/" ? string.Empty : advertised.AbsolutePath;
+        endpoint.EndpointUrl = $"{advertised.Scheme}://{authority}{path}";
+    }
+
    /// <summary>
    /// Probes an OPC UA endpoint configuration WITHOUT persisting it or creating a
    /// long-lived connection — connect, capture the server certificate if it is untrusted,
@@ -258,6 +309,10 @@ public class RealOpcUaClient : IOpcUaClient
                endpoint = new EndpointDescription(endpointUrl);
            }

+            // Same reachability rewrite as ConnectAsync — a probe against a server advertising a
+            // 0.0.0.0 / NAT base address must dial the reachable host, not the advertised one.
+            RewriteEndpointHostForReachability(endpoint, endpointUrl);
+
            var endpointConfig = EndpointConfiguration.Create(appConfig);
            var configuredEndpoint = new ConfiguredEndpoint(null, endpoint, endpointConfig);

@@ -4,7 +4,7 @@ namespace ZB.MOM.WW.ScadaBridge.HealthMonitoring;

 /// <summary>
 /// Abstraction for sending health reports to central.
-/// In production, implemented over Akka ClusterClient with an acked round-trip
+/// In production, implemented over the gRPC central transport with an acked round-trip
 /// (review 01 [Medium]): the transport Asks and awaits a
 /// <see cref="SiteHealthReportAck"/>, so a lost/rejected report surfaces as a
 /// fault the sender can observe (and its per-interval counter-restore fires).
@@ -261,8 +261,16 @@ public class AkkaHostedService : IHostedService
        TimeSpan transportHeartbeat,
        TimeSpan transportFailure)
    {
-        var seedNodesStr = string.Join(",",
-            clusterOptions.SeedNodes.Select(QuoteHocon));
+        // Simultaneous-cold-start split-brain guard (Gitea #33): when enabled the node must NOT
+        // auto-join from its config seeds — Akka's FirstSeedNodeProcess is exactly what races the
+        // peer and forms two 1-node clusters. Emit an EMPTY seed list so the ActorSystem comes up
+        // unjoined, and ClusterBootstrapCoordinator issues the single reachability-gated
+        // JoinSeedNodes (founder self-first / joiner peer-first). Default OFF, so guard-off configs
+        // render byte-identical to before.
+        var effectiveSeeds = clusterOptions.BootstrapGuard.Enabled
+            ? Enumerable.Empty<string>()
+            : clusterOptions.SeedNodes;
+        var seedNodesStr = string.Join(",", effectiveSeeds.Select(QuoteHocon));
        var rolesStr = string.Join(",", roles.Select(QuoteHocon));

        // auto-down (default): AutoDowning provider — the leader among the reachable
@@ -548,7 +556,7 @@ akka {{
            _logger);

        // Hand the audit-ingest proxy to the CentralCommunicationActor so audit
-        // ingest commands forwarded by sites over ClusterClient are routed to the
+        // ingest commands forwarded by sites are routed to the
        // singleton. Mirrors the RegisterNotificationOutbox wiring above.
        centralCommActor.Tell(new RegisterAuditIngest(auditIngest.Proxy));

@@ -620,7 +628,7 @@ akka {{

        // Hand the CentralCommunicationActor to the SiteCallAudit
        // actor so it can relay operator Retry/Discard on parked cached calls to
-        // the owning site (over the per-site ClusterClient via SiteEnvelope).
+        // the owning site (over the per-site gRPC transport via SiteEnvelope).
        // Mirrors the RegisterAuditIngest / RegisterNotificationOutbox wiring;
        // the message is sent to the singleton proxy so it reaches whichever
        // central node currently hosts the singleton.
@@ -888,7 +896,7 @@ akka {{
        // Event log handler — cluster singleton so queries always reach the
        // active node. The event log is node-local SQLite and is not
        // replicated; only the active node records events. A per-node handler
-        // would let a ClusterClient query land on the standby and find nothing.
+        // would let a central query land on the standby and find nothing.
        var eventLogQueryService = _serviceProvider.GetService<SiteEventLogging.IEventLogQueryService>();
        if (eventLogQueryService != null)
        {
@@ -1040,23 +1048,23 @@ akka {{
        // collaborators through its constructor, so we resolve them from DI
        // and pass them in via Props.Create rather than relying on a future
        // FactoryProvider. The real site→central client is constructed and
-        // wired immediately below: a ClusterClientSiteAuditClient (ClusterClient
-        // transport, not gRPC) replaces the DI-default NoOpSiteStreamAuditClient
+        // wired immediately below: a SiteCommunicationAuditClient (routing through
+        // the SiteCommunicationActor's central transport) replaces the DI-default NoOpSiteStreamAuditClient
        // for site roles, without disturbing the rest of this wiring.
        var siteAuditOptions = _serviceProvider
            .GetRequiredService<IOptions<ZB.MOM.WW.ScadaBridge.AuditLog.Site.Telemetry.SiteAuditTelemetryOptions>>();
        var siteAuditQueue = _serviceProvider
            .GetRequiredService<ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services.ISiteAuditQueue>();
        // Audit Log follow-up: the production site→central audit
-        // push uses the ClusterClient transport via the SiteCommunicationActor,
+        // push routes through the SiteCommunicationActor's central transport,
        // not the DI-resolved NoOpSiteStreamAuditClient. The NoOp default stays
        // correct for central/test composition roots (no SiteCommunicationActor);
-        // a site role wires the real ClusterClient-based client here so the
+        // a site role wires the real client here so the
        // SQLite Pending backlog actually drains to central. The forward Ask
        // reuses NotificationForwardTimeout — the same site→central command
        // forward bound notifications already use over this transport.
        ZB.MOM.WW.ScadaBridge.AuditLog.Site.Telemetry.ISiteStreamAuditClient siteAuditClient =
-            new ZB.MOM.WW.ScadaBridge.AuditLog.Site.Telemetry.ClusterClientSiteAuditClient(
+            new ZB.MOM.WW.ScadaBridge.AuditLog.Site.Telemetry.SiteCommunicationAuditClient(
                siteCommActor,
                _communicationOptions.NotificationForwardTimeout);
        var siteAuditLogger = _serviceProvider.GetRequiredService<ILoggerFactory>()
@@ -1092,10 +1100,10 @@ akka {{
        // SetReady asserts a deliberately narrow contract. By this point the
        // actor system exists, SiteStreamManager.Initialize has run, and every
        // role actor (SiteCommunicationActor, deployment-manager singleton,
-        // the ClusterClient) has been created with ActorOf —
+        // the central transport) has been created with ActorOf —
        // creation and the registration Tells are synchronous and strictly ordered.
        // What is NOT guaranteed is completion of each actor's PreStart or the
-        // ClusterClient's initial-contact handshake with central: those are
+        // the central transport's initial connection to central: those are
        // intentionally asynchronous. Gating readiness on the central handshake would
        // be wrong — a site must come up and stream locally even while central is
        // briefly unreachable. gRPC readiness therefore guarantees "the site actor
@@ -0,0 +1,211 @@
+using System.Diagnostics;
+using System.Net.Sockets;
+using Akka.Actor;
+using Akka.Cluster;
+using Microsoft.Extensions.Options;
+using ZB.MOM.WW.ScadaBridge.ClusterInfrastructure;
+
+namespace ZB.MOM.WW.ScadaBridge.Host.Actors;
+
+/// <summary>
+/// Runs the simultaneous-cold-start split-brain guard (Gitea #33): when
+/// <c>ScadaBridge:Cluster:BootstrapGuard:Enabled</c>, the node starts with NO config seed nodes
+/// (<c>AkkaHostedService.BuildHocon</c> emits an empty seed list), and this coordinator picks the join
+/// order and issues the single <see cref="Akka.Cluster.Cluster.JoinSeedNodes"/> once the ActorSystem is
+/// up. See <see cref="ClusterBootstrapGuard"/> for the decision logic and the split-brain rationale.
+/// </summary>
+/// <remarks>
+///     <para>
+///         The founder (lower canonical address) joins its self-first order immediately. The higher
+///         node probes its partner's Akka endpoint (a plain TCP connect — the partner has bound its
+///         port well before it forms) up to <see cref="ClusterBootstrapGuardOptions.PartnerProbeSeconds"/>:
+///         reachable ⇒ peer-first (join the founder); unreachable ⇒ self-first (partner is dead, form
+///         alone). The join runs on a background task so it never blocks host startup, and it is issued
+///         exactly once — the coordinator never re-forms a node mid-handshake, the failure mode of the
+///         rejected self-form-watchdog design (see <c>SelfFirstSeedBootstrapTests</c>).
+///     </para>
+///     <para>
+///         Registered unconditionally in both the Central and Site composition roots but a no-op unless
+///         the guard is enabled, so guard-off deployments keep Akka's config-driven self-first
+///         auto-join byte-identical.
+///     </para>
+/// </remarks>
+public sealed class ClusterBootstrapCoordinator : IHostedService
+{
+    private readonly Func<ActorSystem> _system;
+    private readonly ClusterOptions _clusterOptions;
+    private readonly NodeOptions _nodeOptions;
+    private readonly ILogger<ClusterBootstrapCoordinator> _log;
+    private readonly CancellationTokenSource _cts = new();
+    private Task? _joinTask;
+
+    /// <summary>Creates the coordinator.</summary>
+    /// <param name="system">Lazy accessor for the node's ActorSystem (resolved after the Akka hosted service creates it).</param>
+    /// <param name="clusterOptions">The bound cluster options (seed list + guard settings).</param>
+    /// <param name="nodeOptions">The bound node identity (advertised hostname + remoting port).</param>
+    /// <param name="log">Logger for the bootstrap decision.</param>
+    public ClusterBootstrapCoordinator(
+        Func<ActorSystem> system,
+        IOptions<ClusterOptions> clusterOptions,
+        IOptions<NodeOptions> nodeOptions,
+        ILogger<ClusterBootstrapCoordinator> log)
+    {
+        _system = system ?? throw new ArgumentNullException(nameof(system));
+        _clusterOptions = clusterOptions?.Value ?? throw new ArgumentNullException(nameof(clusterOptions));
+        _nodeOptions = nodeOptions?.Value ?? throw new ArgumentNullException(nameof(nodeOptions));
+        _log = log ?? throw new ArgumentNullException(nameof(log));
+    }
+
+    /// <inheritdoc/>
+    public Task StartAsync(CancellationToken cancellationToken)
+    {
+        if (!_clusterOptions.BootstrapGuard.Enabled)
+            return Task.CompletedTask; // dark switch off — Akka auto-joined from config seeds already.
+
+        // Fire-and-forget so a higher node with a dead partner (which waits the full probe window)
+        // never blocks host startup. Exceptions are logged; a failed join leaves the node unjoined,
+        // which is visible and recoverable, never a silent split.
+        _joinTask = Task.Run(() => RunAsync(_cts.Token), _cts.Token);
+        return Task.CompletedTask;
+    }
+
+    /// <inheritdoc/>
+    public async Task StopAsync(CancellationToken cancellationToken)
+    {
+        await _cts.CancelAsync().ConfigureAwait(false);
+        if (_joinTask is not null)
+        {
+            try { await _joinTask.ConfigureAwait(false); }
+            catch (OperationCanceledException) { /* expected on shutdown */ }
+        }
+    }
+
+    private async Task RunAsync(CancellationToken ct)
+    {
+        try
+        {
+            var seeds = _clusterOptions.SeedNodes ?? new List<string>();
+            var role = ClusterBootstrapGuard.Analyze(seeds, _nodeOptions.NodeHostname, _nodeOptions.RemotingPort);
+
+            string[] order;
+            var committedPeerFirst = false;
+            if (!role.Applies)
+            {
+                // Not a pair seed (single-node install, self absent, malformed, or 3+ seeds): join the
+                // configured seeds unchanged — the guard arbitrates only the 2-node pair race.
+                order = seeds.ToArray();
+                _log.LogInformation(
+                    "Bootstrap guard: not a pair seed ({SeedCount} seed(s)); joining configured seeds unchanged.",
+                    seeds.Count);
+            }
+            else if (role.IsFounder)
+            {
+                order = role.SelfFirstOrder;
+                _log.LogInformation(
+                    "Bootstrap guard: this node is the preferred founder (lower address); joining self-first, forming immediately if no peer answers.");
+            }
+            else
+            {
+                var reachable = await ProbePartnerAsync(role.PartnerHost!, role.PartnerPort, ct).ConfigureAwait(false);
+                order = ClusterBootstrapGuard.HigherNodeOrder(role, reachable);
+                committedPeerFirst = reachable;
+                _log.LogInformation(
+                    "Bootstrap guard: this node is the higher address; partner {PartnerHost}:{PartnerPort} was {Reachability} within the probe window; joining {Order}.",
+                    role.PartnerHost, role.PartnerPort, reachable ? "REACHABLE" : "UNREACHABLE",
+                    reachable ? "peer-first (join the founder)" : "self-first (partner down, forming alone)");
+            }
+
+            if (order.Length == 0)
+            {
+                _log.LogWarning("Bootstrap guard: no seed nodes to join; node stays unjoined until a seed is configured.");
+                return;
+            }
+
+            var cluster = Akka.Cluster.Cluster.Get(_system());
+            var addresses = order.Select(Address.Parse).ToArray();
+            cluster.JoinSeedNodes(addresses);
+
+            // Peer-first is a one-way commitment: JoinSeedNodeProcess never self-forms. If the founder
+            // died in the probe→join window, this node hangs unjoined forever. We do NOT re-form here
+            // (the rejected mid-handshake failure mode) — we make the hang operator-visible so a
+            // restart, which re-runs the guard against the now-dead founder and self-forms, recovers
+            // it. Nothing is done on the founder / self-first paths, which always self-form on their own.
+            if (committedPeerFirst)
+                await WarnIfNotUpAsync(cluster, ct).ConfigureAwait(false);
+        }
+        catch (OperationCanceledException)
+        {
+            // Host shutting down before the join completed — nothing to do.
+        }
+        catch (Exception ex)
+        {
+            _log.LogError(ex, "Bootstrap guard: failed to issue JoinSeedNodes; node remains unjoined.");
+        }
+    }
+
+    /// <summary>
+    /// After committing peer-first, waits a bounded time for this node to reach <c>Up</c> and logs a
+    /// clear warning if it does not — the founder must have died in the probe→join window, leaving this
+    /// node hung in <c>JoinSeedNodeProcess</c>. Read-only: it never re-forms or re-joins (that is the
+    /// rejected mid-handshake failure mode); it only surfaces the condition so an operator restarts the node.
+    /// </summary>
+    private async Task WarnIfNotUpAsync(Akka.Cluster.Cluster cluster, CancellationToken ct)
+    {
+        // Give the join a generous grace: the founder's own self-form (seed-node-timeout) plus this
+        // node's join round-trip. Two probe windows is comfortably beyond both.
+        var grace = TimeSpan.FromSeconds(Math.Max(10, _clusterOptions.BootstrapGuard.PartnerProbeSeconds * 2));
+        var sw = Stopwatch.StartNew();
+        while (!ct.IsCancellationRequested && sw.Elapsed < grace)
+        {
+            if (cluster.SelfMember.Status == MemberStatus.Up) return; // joined — all good
+            try { await Task.Delay(TimeSpan.FromSeconds(1), ct).ConfigureAwait(false); }
+            catch (OperationCanceledException) { return; }
+        }
+
+        if (!ct.IsCancellationRequested && cluster.SelfMember.Status != MemberStatus.Up)
+            _log.LogWarning(
+                "Bootstrap guard: committed peer-first but this node is still {Status} after {Grace}s — its founder likely died in the probe→join window. This node will NOT self-form on its own (peer-first never does). RESTART it to recover: the guard will re-run, find the founder down, and form alone.",
+                cluster.SelfMember.Status, (int)grace.TotalSeconds);
+    }
+
+    /// <summary>
+    /// Polls the partner's Akka endpoint with a plain TCP connect until it is reachable or the probe
+    /// window elapses. Reachable-at-TCP is a sound proxy for "the partner is coming up": a node binds
+    /// its Akka port near the start of startup, well before it forms or joins a cluster.
+    /// </summary>
+    private async Task<bool> ProbePartnerAsync(string host, int port, CancellationToken ct)
+    {
+        var window = TimeSpan.FromSeconds(Math.Max(0, _clusterOptions.BootstrapGuard.PartnerProbeSeconds));
+        var interval = TimeSpan.FromMilliseconds(Math.Max(50, _clusterOptions.BootstrapGuard.PartnerProbeIntervalMs));
+        var sw = Stopwatch.StartNew();
+
+        while (!ct.IsCancellationRequested)
+        {
+            if (await TryConnectAsync(host, port, ct).ConfigureAwait(false)) return true;
+            if (sw.Elapsed >= window) return false;
+            try { await Task.Delay(interval, ct).ConfigureAwait(false); }
+            catch (OperationCanceledException) { return false; }
+        }
+        return false;
+    }
+
+    private async Task<bool> TryConnectAsync(string host, int port, CancellationToken ct)
+    {
+        try
+        {
+            using var client = new TcpClient();
+            using var cts = CancellationTokenSource.CreateLinkedTokenSource(ct);
+            cts.CancelAfter(Math.Max(100, _clusterOptions.BootstrapGuard.ProbeConnectTimeoutMs));
+            await client.ConnectAsync(host, port, cts.Token).ConfigureAwait(false);
+            return client.Connected;
+        }
+        catch (OperationCanceledException) when (ct.IsCancellationRequested)
+        {
+            throw; // host shutdown — propagate
+        }
+        catch
+        {
+            return false; // connect refused / timed out / DNS not resolvable yet — partner not up
+        }
+    }
+}
@@ -14,7 +14,8 @@ namespace ZB.MOM.WW.ScadaBridge.Host.Health;
 /// <see cref="ClusterActivityEvaluator.SelfIsOldest(Cluster, string?)"/> — NOT
 /// the cluster leader, whose address-ordered definition diverges from singleton
 /// placement after a restart (review 01 [High]). This is the same evaluator
-/// backing <see cref="OldestNodeActiveHealthCheck"/>, so
+/// backing the shared <c>ActiveNodeHealthCheck</c> registered on the active tier
+/// — both route to <c>ClusterActiveNode</c> — so
 /// <see cref="InboundApiEndpointFilter"/> can return HTTP 503 on a standby.
 ///
 /// Registered only in the Central-role branch of <c>Program.cs</c>. The gate
@@ -1,4 +1,5 @@
 using Akka.Cluster;
+using ZB.MOM.WW.Health.Akka;

 namespace ZB.MOM.WW.ScadaBridge.Host.Health;

@@ -27,14 +28,10 @@ public static class ClusterActivityEvaluator
    /// <param name="cluster">The Akka cluster to evaluate.</param>
    /// <param name="role">Optional role scope; when set, only members with this role are considered.</param>
    /// <returns>The oldest Up member in the role scope, or null when none is Up.</returns>
-    public static Member? OldestUpMember(Cluster cluster, string? role = null)
-    {
-        Member? oldest = null;
-        foreach (var m in cluster.State.Members.Where(m => m.Status == MemberStatus.Up))
-        {
-            if (role != null && !m.HasRole(role)) continue;
-            if (oldest == null || m.IsOlderThan(oldest)) oldest = m;
-        }
-        return oldest;
-    }
+    /// <remarks>
+    /// Delegates to the shared <see cref="ClusterActiveNode.OldestUpMember(Cluster, string?)"/>, so
+    /// the label a node shows and the answer <see cref="SelfIsOldest"/> gates on come from one rule.
+    /// </remarks>
+    public static Member? OldestUpMember(Cluster cluster, string? role = null) =>
+        ClusterActiveNode.OldestUpMember(cluster, role);
 }
@@ -1,34 +0,0 @@
-using Akka.Actor;
-using Microsoft.Extensions.Diagnostics.HealthChecks;
-
-namespace ZB.MOM.WW.ScadaBridge.Host.Health;
-
-/// <summary>
-/// /health/active check: Healthy only on the oldest Up member — the node that
-/// hosts the cluster singletons. Replaces the shared package's leader-based
-/// ActiveNodeHealthCheck (review 01 [High]): leadership is address-ordered and
-/// diverges from singleton placement after a restart, and during a partition
-/// BOTH nodes compute themselves leader, so Traefik served both. Oldest-member
-/// semantics keep exactly one active node through a partition (the non-oldest
-/// side still sees the old oldest member until SBR resolves membership).
-/// </summary>
-public sealed class OldestNodeActiveHealthCheck : IHealthCheck
-{
-    private readonly ActorSystem _system;
-
-    /// <summary>Initializes a new <see cref="OldestNodeActiveHealthCheck"/> bound to the running actor system.</summary>
-    /// <param name="system">The live cluster actor system.</param>
-    public OldestNodeActiveHealthCheck(ActorSystem system) => _system = system;
-
-    /// <summary>Reports Healthy only when this node is the oldest Up cluster member (the singleton host); otherwise reports Unhealthy.</summary>
-    /// <param name="context">The health check context (unused; the result depends only on cluster membership).</param>
-    /// <param name="ct">Cancellation token.</param>
-    /// <returns>A task that resolves to the health check result.</returns>
-    public Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context, CancellationToken ct = default)
-    {
-        var cluster = Akka.Cluster.Cluster.Get(_system);
-        return Task.FromResult(ClusterActivityEvaluator.SelfIsOldest(cluster)
-            ? HealthCheckResult.Healthy("oldest Up member (singleton host)")
-            : HealthCheckResult.Unhealthy("not the oldest Up member (standby)"));
-    }
-}
@@ -0,0 +1,77 @@
+using Microsoft.Extensions.Diagnostics.HealthChecks;
+using ZB.MOM.WW.LocalDb;
+using ZB.MOM.WW.LocalDb.Replication;
+
+namespace ZB.MOM.WW.ScadaBridge.Host.Health;
+
+/// <summary>
+/// /health/ready check for the consolidated site database (LocalDb). Site nodes have no EF
+/// context — central's <c>DatabaseHealthCheck&lt;ScadaBridgeDbContext&gt;</c> is central-only — so
+/// the site store is probed directly through the registered <see cref="ILocalDb"/> with a trivial
+/// <c>SELECT 1</c>. That is the honest readiness question for a site node: everything it persists
+/// (OperationTracking, site events, store-and-forward, the replicated config tables) lives here.
+/// </summary>
+/// <remarks>
+/// Replication state is reported as <c>data</c> ENRICHMENT ONLY and never affects the status.
+/// Replication is default-OFF by design, so a node with no peer reports <c>connected=false</c> with
+/// a real 0 backlog — the healthy steady state of an unreplicated node — and failing the check on
+/// it would mark every correctly-configured single node unready.
+/// </remarks>
+public sealed class SiteLocalDbHealthCheck : IHealthCheck
+{
+    private readonly ILocalDb _localDb;
+    private readonly ISyncStatus? _syncStatus;
+
+    /// <summary>Initializes a new <see cref="SiteLocalDbHealthCheck"/>.</summary>
+    /// <param name="localDb">The consolidated site database registered by <c>AddZbLocalDb</c>.</param>
+    /// <param name="syncStatus">
+    /// The replication status view, when the replication engine is registered. Optional so the check
+    /// still works in a composition that wires storage without replication.
+    /// </param>
+    public SiteLocalDbHealthCheck(ILocalDb localDb, ISyncStatus? syncStatus = null)
+    {
+        _localDb = localDb ?? throw new ArgumentNullException(nameof(localDb));
+        _syncStatus = syncStatus;
+    }
+
+    /// <summary>Probes the site database and reports its reachability.</summary>
+    /// <param name="context">The health check context (unused; the verdict depends only on the probe).</param>
+    /// <param name="cancellationToken">Cancellation token.</param>
+    /// <returns>Healthy when the probe query succeeds; Unhealthy with the fault otherwise.</returns>
+    public async Task<HealthCheckResult> CheckHealthAsync(
+        HealthCheckContext context,
+        CancellationToken cancellationToken = default)
+    {
+        try
+        {
+            await _localDb.QueryAsync("SELECT 1;", static r => r.GetInt32(0), ct: cancellationToken)
+                .ConfigureAwait(false);
+        }
+        catch (Exception ex) when (ex is not OperationCanceledException)
+        {
+            return HealthCheckResult.Unhealthy("Site LocalDb is not reachable.", ex);
+        }
+
+        return HealthCheckResult.Healthy("Site LocalDb reachable.", BuildData());
+    }
+
+    private IReadOnlyDictionary<string, object> BuildData()
+    {
+        var data = new Dictionary<string, object>(StringComparer.Ordinal);
+        if (_syncStatus is null)
+            return data;
+
+        data["replicationConnected"] = _syncStatus.Connected;
+
+        // Nullable on purpose upstream: null means "backlog unknown" (no provider wired or the poll
+        // failed), which must not be flattened to 0 — that would render a pair which cannot read its
+        // own oplog as perfectly caught up. Omit the key instead.
+        if (_syncStatus.OplogBacklog is { } backlog)
+            data["replicationOplogBacklog"] = backlog;
+
+        if (_syncStatus.PeerNodeId is { } peer)
+            data["replicationPeerNodeId"] = peer;
+
+        return data;
+    }
+}
@@ -0,0 +1,39 @@
+using Microsoft.Extensions.Diagnostics.HealthChecks;
+using ZB.MOM.WW.ScadaBridge.HealthMonitoring;
+
+namespace ZB.MOM.WW.ScadaBridge.Host.Health;
+
+/// <summary>
+/// /health/active check for a SITE pair: Healthy only on the pair's primary — the oldest Up member
+/// carrying the <c>site-{SiteId}</c> role. Same active/standby semantics central publishes, so a
+/// consumer reads one contract fleet-wide (200 = Active, 503 = Standby).
+/// </summary>
+/// <remarks>
+/// Deliberately NOT central's unscoped shared <c>ActiveNodeHealthCheck</c>: that one calls
+/// <c>ClusterActivityEvaluator.SelfIsOldest(cluster)</c> with no role argument, so on a mesh that
+/// carries more than one site's members it computes "oldest" across the wrong member set and a site
+/// node could report Active because it happens to be the oldest member overall. Delegating to
+/// <see cref="IClusterNodeProvider.SelfIsPrimary"/> reuses the role-scoped evaluation the site
+/// already trusts for singleton placement and event-log purge — one primary definition per site,
+/// not two that can disagree.
+/// </remarks>
+public sealed class SitePairActiveNodeHealthCheck : IHealthCheck
+{
+    private readonly IClusterNodeProvider _nodeProvider;
+
+    /// <summary>Initializes a new <see cref="SitePairActiveNodeHealthCheck"/>.</summary>
+    /// <param name="nodeProvider">The role-scoped site cluster node provider.</param>
+    public SitePairActiveNodeHealthCheck(IClusterNodeProvider nodeProvider) =>
+        _nodeProvider = nodeProvider ?? throw new ArgumentNullException(nameof(nodeProvider));
+
+    /// <summary>Reports Healthy on the site pair's primary and Unhealthy on the standby.</summary>
+    /// <param name="context">The health check context (unused; the verdict depends only on cluster membership).</param>
+    /// <param name="cancellationToken">Cancellation token.</param>
+    /// <returns>Healthy on the primary; Unhealthy (503 = Standby) otherwise.</returns>
+    public Task<HealthCheckResult> CheckHealthAsync(
+        HealthCheckContext context,
+        CancellationToken cancellationToken = default) =>
+        Task.FromResult(_nodeProvider.SelfIsPrimary
+            ? HealthCheckResult.Healthy("site pair primary (oldest Up member of the site role)")
+            : HealthCheckResult.Unhealthy("not the site pair primary (standby)"));
+}
@@ -160,7 +160,7 @@ try
        // which would let DI hand the instance back and bypass Grpc.AspNetCore's own activation
        // path (the shape that once hid a two-public-constructor defect until the rig caught it).
        // CentralControlGrpcService decodes each request onto the same in-process message the
-        // ClusterClient path carries and Asks CentralCommunicationActor — zero handler logic here.
+        // removed ClusterClient path used to carry, and Asks CentralCommunicationActor — zero handler logic here.
        builder.Services.AddGrpc(options =>
        {
            options.Interceptors.Add<CentralControlAuthInterceptor>();
@@ -344,9 +344,10 @@ try
        // Ready tier (/health/ready): database + akka-cluster + required-singletons.
        // Active tier (/health/active): active-node = oldest Up member (singleton host) AND
        // database reachable (review 01 [High]) — a DB-dead active node drops from Traefik
-        // rotation, and the active-node check is the host-local OldestNodeActiveHealthCheck,
-        // NOT the shared leader-based ActiveNodeHealthCheck (leadership diverges from singleton
-        // placement after a restart and both sides claim leadership during a partition).
+        // rotation. The active-node check is the shared ActiveNodeHealthCheck, which selects by age
+        // as of Health 0.3.0; through 0.2.1 it selected by leadership, which diverges from singleton
+        // placement after a restart and names both sides during a partition, so this host kept a
+        // private copy until the shared one was fixed.
        // The Akka checks resolve ActorSystem from DI via the singleton bridge registered below;
        // the DatabaseHealthCheck<TContext> resolves a scoped ScadaBridgeDbContext (no factory).
        builder.Services.AddHealthChecks()
@@ -371,9 +372,12 @@ try
                "required-singletons",
                failureStatus: null,
                tags: new[] { ZbHealthTags.Ready })
-            // Host-local oldest-member check (review 01 [High]) — see OldestNodeActiveHealthCheck.
-            // Resolves ActorSystem from DI, like the akka-cluster check above.
-            .AddTypeActivatedCheck<OldestNodeActiveHealthCheck>(
+            // Oldest-Up-member check (review 01 [High]), unscoped: every central member competes for
+            // the one active slot. Since ZB.MOM.WW.Health 0.3.0 this IS the shared check — it selects
+            // by age, so the host-local OldestNodeActiveHealthCheck that existed only to avoid the old
+            // leader-based selection is gone. Resolves ActorSystem from DI, like the akka-cluster
+            // check above.
+            .AddTypeActivatedCheck<ActiveNodeHealthCheck>(
                "active-node",
                failureStatus: null,
                tags: new[] { ZbHealthTags.Active });
@@ -394,14 +398,28 @@ try
        builder.Services.AddSingleton<Akka.Actor.ActorSystem>(sp =>
            sp.GetRequiredService<AkkaHostedService>().GetOrCreateActorSystem());

+        // Simultaneous-cold-start split-brain guard (Gitea #33, dark switch
+        // ScadaBridge:Cluster:BootstrapGuard:Enabled, default off). Registered AFTER the
+        // AkkaHostedService/ActorSystem bridge so its StartAsync resolves the system the main path
+        // built; when the guard is off it no-ops (Akka auto-joined from config seeds), so registering
+        // it unconditionally is safe. When on, the node started unjoined (empty HOCON seed list, see
+        // AkkaHostedService.BuildHocon) and this coordinator issues the single reachability-gated
+        // JoinSeedNodes. It resolves the ActorSystem lazily via GetOrCreateActorSystem so it never
+        // races startup or caches a null.
+        builder.Services.AddHostedService(sp => new ClusterBootstrapCoordinator(
+            () => sp.GetRequiredService<AkkaHostedService>().GetOrCreateActorSystem(),
+            sp.GetRequiredService<IOptions<ClusterOptions>>(),
+            sp.GetRequiredService<IOptions<NodeOptions>>(),
+            sp.GetRequiredService<ILogger<ClusterBootstrapCoordinator>>()));
+
        // Register the production IActiveNodeGate implementation so
        // standby-node gating is actually enforced (the InboundApiEndpointFilter
        // consults IActiveNodeGate and defaults to "allow" when none is registered,
        // which leaves the design's "central cluster only (active node)" guarantee
        // unenforced in deployed binaries). The gate is backed by the same oldest-member
-        // evaluator (ClusterActivityEvaluator) as OldestNodeActiveHealthCheck above, so the
-        // inbound API and the /health/active endpoint Traefik routes against agree on
-        // which node is active.
+        // evaluator (ClusterActivityEvaluator, which delegates to the shared ClusterActiveNode) as
+        // the active-node check above, so the inbound API and the /health/active endpoint Traefik
+        // routes against agree on which node is active.
        builder.Services.AddSingleton<ZB.MOM.WW.ScadaBridge.InboundAPI.IActiveNodeGate, ActiveNodeGate>();

        // Admin-triggered manual failover of the central pair (Health page control,
@@ -620,7 +638,7 @@ try
        builder.Services.AddSingleton<ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamGrpcServer>();
        // Site command plane (central→site) — the gRPC peer of SiteStreamGrpcServer. Both share
        // the h2c listener and the ControlPlaneAuthInterceptor PSK gate; both are readiness-gated
-        // (SetReady, below). Central still dials over ClusterClient until T1B.3.
+        // (SetReady, below).
        builder.Services.AddSingleton<ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteCommandGrpcService>();

        // Existing site service registrations (this is also where LocalDb and its
@@ -634,6 +652,20 @@ try
        // must be present before the Map* calls on the site role.
        app.UseRouting();

+        // Map the canonical three-tier health endpoints on the site node:
+        //   /health/ready  — akka-cluster (site-pair membership, and the cluster-view data the
+        //                    family overview dashboard reads) + localdb (the consolidated site
+        //                    store — the only database a site node has).
+        //   /health/active — active-node = this site pair's primary; 200 on the primary, 503 on
+        //                    the standby, the same contract central publishes.
+        //   /healthz       — bare process liveness.
+        // Checks are registered in SiteServiceRegistration.Configure. All three endpoints are
+        // anonymous and use the canonical ZbHealthWriter JSON; the site pipeline runs no
+        // authentication middleware and has no FallbackPolicy, so nothing extra is needed to keep
+        // them reachable. They are served on the HTTP/1.1 listener (metricsPort, default :8084)
+        // alongside /metrics — the gRPC listener is HTTP/2-only and unusable by a plain HTTP client.
+        app.MapZbHealth();
+
        // Observability — mount the always-on Prometheus /metrics scrape endpoint.
        // AddZbTelemetry (in SiteServiceRegistration.Configure → BindSharedOptions)
        // wires the OTel Resource + standard instrumentation + Prometheus exporter;
@@ -1,5 +1,7 @@
 using Microsoft.Extensions.DependencyInjection.Extensions;
 using Microsoft.Extensions.Options;
+using ZB.MOM.WW.Health;
+using ZB.MOM.WW.Health.Akka;
 using ZB.MOM.WW.ScadaBridge.AuditLog;
 using ZB.MOM.WW.ScadaBridge.ClusterInfrastructure;
 using ZB.MOM.WW.ScadaBridge.Communication;
@@ -168,6 +170,19 @@ public static class SiteServiceRegistration
        services.AddSingleton<Akka.Actor.ActorSystem>(sp =>
            sp.GetRequiredService<AkkaHostedService>().GetOrCreateActorSystem());

+        // Simultaneous-cold-start split-brain guard (Gitea #33, dark switch
+        // ScadaBridge:Cluster:BootstrapGuard:Enabled, default off). This is the case the guard was
+        // built for — a shared power event that powers up both site VMs together. Registered AFTER
+        // the AkkaHostedService/ActorSystem bridge (mirrors the Central composition root in
+        // Program.cs); it no-ops when the guard is off, so registering it unconditionally is safe.
+        // When on, the node started unjoined (empty HOCON seed list) and this coordinator issues the
+        // single reachability-gated JoinSeedNodes, resolving the ActorSystem lazily.
+        services.AddHostedService(sp => new ClusterBootstrapCoordinator(
+            () => sp.GetRequiredService<AkkaHostedService>().GetOrCreateActorSystem(),
+            sp.GetRequiredService<IOptions<ClusterOptions>>(),
+            sp.GetRequiredService<IOptions<NodeOptions>>(),
+            sp.GetRequiredService<ILogger<ClusterBootstrapCoordinator>>()));
+
        // Cluster node status provider for health reports
        services.AddSingleton<IClusterNodeProvider>(sp =>
        {
@@ -190,6 +205,39 @@ public static class SiteServiceRegistration
            return () => nodeProvider.SelfIsPrimary;
        });

+        // Health checks — the shared ZB.MOM.WW.Health probes, mapped by MapZbHealth on the site's
+        // HTTP/1.1 listener (Program.cs). Site nodes served no health endpoints before this; the
+        // family overview dashboard probes every node the same way, so a site node has to answer
+        // the same two questions central does.
+        //
+        //   Ready  (/health/ready)  akka-cluster — this node's site-pair membership. It also carries
+        //                           the cluster-view data (leader/memberCount/...) that the
+        //                           dashboard reads, free with ZB.MOM.WW.Health 0.2.0.
+        //                           localdb      — the consolidated site store, which is the only
+        //                           database a site node has (central's EF DatabaseHealthCheck is
+        //                           central-only and does not apply here).
+        //   Active (/health/active) active-node   — site-pair primary vs standby.
+        //
+        // Registered HERE rather than in Program.cs, like the LocalDb registrations above, so the
+        // composition-root tests that build this graph actually cover them. The Akka check resolves
+        // ActorSystem from DI per probe via the singleton bridge registered above.
+        services.AddHealthChecks()
+            .AddTypeActivatedCheck<AkkaClusterHealthCheck>(
+                "akka-cluster",
+                failureStatus: null,
+                tags: new[] { ZbHealthTags.Ready },
+                args: AkkaClusterStatusPolicy.Default)
+            .AddTypeActivatedCheck<SiteLocalDbHealthCheck>(
+                "localdb",
+                failureStatus: null,
+                tags: new[] { ZbHealthTags.Ready })
+            // NOT central's unscoped shared ActiveNodeHealthCheck — that one is role-less; see
+            // SitePairActiveNodeHealthCheck for why a site pair needs the role-scoped answer.
+            .AddTypeActivatedCheck<SitePairActiveNodeHealthCheck>(
+                "active-node",
+                failureStatus: null,
+                tags: new[] { ZbHealthTags.Active });
+
        // Options binding
        BindSharedOptions(services, config);

@@ -17,7 +17,14 @@
      "StableAfter": "00:00:15",
      "HeartbeatInterval": "00:00:02",
      "FailureDetectionThreshold": "00:00:10",
-      "MinNrOfMembers": 1
+      "MinNrOfMembers": 1,
+      "_bootstrapGuard": "Gitea #33 simultaneous-cold-start split-brain guard. Dark switch, default off (Akka's config-driven self-first auto-join, unchanged). Enable ONLY where both pair VMs can power up together (shared power/hypervisor event) with no start-order serialization: then the lower host:port founds self-first and the higher probes-then-joins, so the pair converges to one cluster instead of two 1-node clusters. Timings validated > 0 when enabled.",
+      "BootstrapGuard": {
+        "Enabled": false,
+        "PartnerProbeSeconds": 25,
+        "PartnerProbeIntervalMs": 500,
+        "ProbeConnectTimeoutMs": 1000
+      }
    },
    "_secrets": "Host-003: Secrets are NOT committed in this file. The ${secret:...} references below are resolved at startup by the pre-host secrets expander (before StartupValidator runs) from the encrypted secrets store (SQLite, seeded via the ZB.MOM.WW.Secrets store/CLI/UI). The KEK is supplied out-of-band via the ZB_SECRETS_MASTER_KEY environment variable and never committed; an unseeded reference fails closed (SecretNotFoundException) before any SQL/LDAP/cluster wiring. ROLLBACK: the whole-key environment override still wins — set ScadaBridge__Database__ConfigurationDb / ScadaBridge__Security__Ldap__ServiceAccountPassword / ScadaBridge__Security__JwtSigningKey and .AddEnvironmentVariables() overlays the concrete value before expansion runs, so the ${secret:...} token is never evaluated. NOTE (Task 1.4): the LDAP settings moved into the nested Security:Ldap sub-section (bound to the shared ZB.MOM.WW.Auth LdapOptions) — the service-account-password env var is ScadaBridge__Security__Ldap__ServiceAccountPassword (was ScadaBridge__Security__LdapServiceAccountPassword).",
    "Database": {
@@ -20,7 +20,14 @@
      "StableAfter": "00:00:15",
      "HeartbeatInterval": "00:00:02",
      "FailureDetectionThreshold": "00:00:10",
-      "MinNrOfMembers": 1
+      "MinNrOfMembers": 1,
+      "_bootstrapGuard": "Gitea #33 simultaneous-cold-start split-brain guard. Dark switch, default off (Akka's config-driven self-first auto-join, unchanged). Enable ONLY where both pair VMs can power up together (shared power/hypervisor event) with no start-order serialization: then the lower host:port founds self-first and the higher probes-then-joins, so the pair converges to one cluster instead of two 1-node clusters. Timings validated > 0 when enabled.",
+      "BootstrapGuard": {
+        "Enabled": false,
+        "PartnerProbeSeconds": 25,
+        "PartnerProbeIntervalMs": 500,
+        "ProbeConnectTimeoutMs": 1000
+      }
    },
    "Database": {
      // Migration-only as of LocalDb Phase 2. The site config tables now live in the
@@ -388,7 +388,7 @@ public class RouteTarget

    /// <summary>
    /// N3: awaits a routed Ask and TYPES unreachability. AskTimeoutException means
-    /// the site never answered (no ClusterClient contact, site down, partition) —
+    /// the site never answered (no route to the site, site down, partition) —
    /// the communication layer's own typed signal (CentralCommunicationActor drops
    /// envelopes for contactless sites so the Ask expires caller-side) — and
    /// surfaces as SiteUnreachableException so the executor maps it to the spec's
@@ -38,7 +38,7 @@ using ZB.MOM.WW.ScadaBridge.TemplateEngine.Services;
 namespace ZB.MOM.WW.ScadaBridge.ManagementService;

 /// <summary>
-/// Central actor that handles all management commands from the CLI (via ClusterClient).
+/// Central actor that handles all management commands from the CLI (via the HTTP /management endpoint).
 /// Receives ManagementEnvelope messages, authorizes based on roles, then delegates to
 /// the appropriate service or repository using scoped DI.
 /// </summary>
@@ -477,7 +477,7 @@ public class ManagementActor : ReceiveActor
            // NOTE: ResolveRolesCommand is intentionally NOT dispatched. The two-step
            // "ResolveRoles + command" flow is retired — the HTTP endpoint performs LDAP
            // auth and role resolution itself before sending a single envelope. Leaving a
-            // handler would expose role-mapping data to any raw ClusterClient sender with
+            // handler would expose role-mapping data to any raw in-cluster sender with
            // no role requirement; the command now falls through to the default below.
            _ => throw new NotSupportedException($"Unknown command type: {command.GetType().Name}")
        };
@@ -164,7 +164,7 @@ public class SiteCallAuditActor : ReceiveActor
    /// <summary>
    /// The central→site command transport — the
    /// <c>CentralCommunicationActor</c>, which owns the per-site
-    /// <c>ClusterClient</c> map and routes a <see cref="SiteEnvelope"/> to the
+    /// transport map and routes a <see cref="SiteEnvelope"/> to the
    /// owning site. Set via <see cref="RegisterCentralCommunication"/> by the
    /// Host after both actors exist (this actor is a cluster singleton; the
    /// transport actor is created separately). Null until registration
@@ -1079,7 +1079,7 @@ public class SiteCallAuditActor : ReceiveActor
    /// <c>SiteCalls</c> mirror row. It wraps a <see cref="RetryParkedOperation"/>
    /// in a <see cref="SiteEnvelope"/> addressed to <c>SourceSite</c>, Asks the
    /// <c>CentralCommunicationActor</c> (which routes it over the per-site
-    /// <c>ClusterClient</c>), and maps the site's
+    /// gRPC transport), and maps the site's
    /// <see cref="ParkedOperationActionAck"/> — or an Ask timeout — onto a
    /// <see cref="RetrySiteCallResponse"/>. A timeout / no-route is reported as
    /// the distinct <see cref="SiteCallRelayOutcome.SiteUnreachable"/> outcome,
@@ -30,7 +30,7 @@ public class SiteCallAuditOptions
    /// Ask timeout for the central→site Retry/Discard relay. When
    /// the owning site does not ack a <c>RetryParkedOperation</c> /
    /// <c>DiscardParkedOperation</c> within this window — site offline, no
-    /// ClusterClient route, or central buffering deliberately absent — the relay
+    /// route to the site, or central buffering deliberately absent — the relay
    /// reports a <c>SiteUnreachable</c> outcome. Default 10 seconds: long enough
    /// to absorb a healthy cross-cluster round-trip, short enough that an
    /// operator clicking Retry on an offline site gets a fast, honest answer.
@@ -27,8 +27,8 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors;
 /// crashes on these failures.
 /// </para>
 /// <para>
-/// The pass runs after a small startup delay (so the central ClusterClient has time to
-/// register) and is driven entirely off the actor thread: the Ask + fetch + write happen in
+/// The pass runs after a small startup delay (so the site's central transport has time to
+/// connect) and is driven entirely off the actor thread: the Ask + fetch + write happen in
 /// an awaited continuation whose summary is captured in an internal message
 /// <see cref="ReconcilePassResult"/> piped back to <c>Self</c>. The actor thread never blocks.
 /// </para>
@@ -60,14 +60,14 @@ public sealed class SiteReconciliationActor : ReceiveActor, IWithTimers
    /// <param name="configFetcher">Fetches a deployment's flattened config JSON from central over HTTP.</param>
    /// <param name="siteCommunicationActor">
    /// The site's <c>SiteCommunicationActor</c>; it forwards the
-    /// <see cref="ReconcileSiteRequest"/> over the registered central ClusterClient and routes
+    /// <see cref="ReconcileSiteRequest"/> over the site's central transport and routes
    /// the <see cref="ReconcileSiteResponse"/> back to this actor's Ask.
    /// </param>
    /// <param name="siteIdentifier">This node's site identifier (resolved by central).</param>
    /// <param name="nodeId">This node's semantic id (e.g. <c>node-a</c>/<c>node-b</c>), for logging/diagnostics.</param>
    /// <param name="logger">Logger.</param>
    /// <param name="initialDelay">
-    /// Delay before the single startup pass, giving the central ClusterClient time to register.
+    /// Delay before the single startup pass, giving the site's central transport time to connect.
    /// Defaults to 5 seconds.
    /// </param>
    /// <param name="askTimeout">Round-trip timeout for the reconcile Ask to central. Defaults to 30 seconds.</param>
@@ -105,7 +105,7 @@ public sealed class SiteReconciliationActor : ReceiveActor, IWithTimers
    protected override void PreStart()
    {
        base.PreStart();
-        // One-shot pass after a small delay so the central ClusterClient can register first.
+        // One-shot pass after a small delay so the site's central transport can connect first.
        // Non-blocking: the timer fires RunReconcile back onto this actor's mailbox.
        Timers.StartSingleTimer(StartupTimerKey, RunReconcile.Instance, _initialDelay);
        _logger.LogInformation(
@@ -74,7 +74,7 @@ public class ScriptRuntimeContext

    /// <summary>
    /// Notification Outbox: the site communication actor that <c>Notify.Status</c>
-    /// queries central through (via the ClusterClient command/control transport).
+    /// queries central through (via the gRPC command/control transport).
    /// </summary>
    private readonly ICanTell? _siteCommunicationActor;

@@ -175,7 +175,7 @@ public class ScriptRuntimeContext
    /// <param name="externalSystemClient">Optional client for external system API calls.</param>
    /// <param name="databaseGateway">Optional gateway for database connection and cached write access.</param>
    /// <param name="storeAndForward">Optional store-and-forward service for notification delivery.</param>
-    /// <param name="siteCommunicationActor">Optional actor for site-to-central communication (ClusterClient).</param>
+    /// <param name="siteCommunicationActor">Optional actor for site-to-central communication.</param>
    /// <param name="siteId">Identifier of the site where this instance is running.</param>
    /// <param name="sourceScript">Optional name of the source script for audit trail identification.</param>
    /// <param name="auditWriter">Optional writer for audit log entries.</param>
@@ -26,10 +26,10 @@ namespace ZB.MOM.WW.ScadaBridge.StoreAndForward;
 ///   for operator forensics.</description></item>
 /// </list>
 ///
-/// The forward travels over the ClusterClient command/control transport: the handler
+/// The forward travels over the gRPC command/control transport: the handler
 /// <see cref="ActorRefImplicitSenderExtensions.Ask{T}(ICanTell, object, TimeSpan?)">Asks</see>
 /// the site communication actor, which wraps the message in a
-/// <c>ClusterClient.Send("/user/central-communication", …)</c> and routes central's
+/// <c>SubmitNotification</c> call on <c>CentralControlService</c> and routes central's
 /// reply straight back to this Ask.
 /// </summary>
 public sealed class NotificationForwarder
@@ -45,7 +45,7 @@ public sealed class NotificationForwarder
    /// </summary>
    /// <param name="siteCommunicationActor">
    /// The site communication actor. It forwards a <see cref="NotificationSubmit"/> to
-    /// central via the registered ClusterClient and replies with the
+    /// central over the site's central transport and replies with the
    /// <see cref="NotificationSubmitAck"/>.
    /// </param>
    /// <param name="sourceSiteId">This site's identifier, stamped on every submit.</param>
@@ -362,11 +362,22 @@ public class SiteAuditTelemetryActorTests : TestKit
    }

    [Fact]
-    public async Task CachedDrain_OrphanRow_NoTrackingSnapshot_IsSkipped_DoesNotCrash()
+    public async Task CachedDrain_OrphanRow_PastGrace_IsAbandoned_AndTheValidRowStillFlows()
    {
+        // RETARGETED, not rewritten from scratch: this test previously asserted
+        // the orphan "is skipped and stays Pending", which is precisely the
+        // defect fixed here (known-issues/2026-07-20-cached-telemetry-drain-
+        // hot-loop.md) — a Pending row that can never resolve is re-read every
+        // drain forever and starves the queue behind it. The half of the test
+        // that still holds (the valid row flows, the drain does not crash) is
+        // kept verbatim; only the orphan's fate is inverted.
+        //
+        // NewCachedEvent stamps occurredAtUtc in the past, well beyond the
+        // default 300 s grace, so this orphan is past the abandon threshold.
+        //
        // Arrange — two cached audit rows: one with a tracking snapshot, one
        // orphaned (the tracking store returns null). The orphaned row must be
-        // skipped without aborting the batch — the valid row still flows.
+        // abandoned without aborting the batch — the valid row still flows.
        var orphan = NewCachedEvent(AuditKind.CachedSubmit);
        var valid = NewCachedEvent(AuditKind.CachedResolve);

@@ -403,8 +414,8 @@ public class SiteAuditTelemetryActorTests : TestKit
        // Act
        CreateActorWithCachedDrain();

-        // Assert — exactly one push containing ONLY the valid row; the orphan
-        // is skipped and stays Pending (not in MarkForwardedAsync's id list).
+        // Assert — exactly one push containing ONLY the valid row: the orphan
+        // has no operational half to send.
        await AwaitAssertAsync(async () =>
        {
            await _client.Received(1).IngestCachedTelemetryAsync(
@@ -415,9 +426,131 @@ public class SiteAuditTelemetryActorTests : TestKit
        Assert.Single(capturedBatch!.Packets);
        Assert.Equal(valid.EventId.ToString(), capturedBatch.Packets[0].AuditEvent.EventId);

+        // The valid row is marked Forwarded because central ack'd it...
        await _queue.Received(1).MarkForwardedAsync(
            Arg.Is<IReadOnlyList<Guid>>(g => g.Count == 1 && g[0] == valid.EventId),
            Arg.Any<CancellationToken>());
+
+        // ...and the orphan is marked Forwarded too, in its own call, so it
+        // LEAVES the cached queue. Its audit half is still delivered by the
+        // reconciliation pull, which covers Forwarded rows as well as Pending.
+        await _queue.Received(1).MarkForwardedAsync(
+            Arg.Is<IReadOnlyList<Guid>>(g => g.Count == 1 && g[0] == orphan.EventId),
+            Arg.Any<CancellationToken>());
+    }
+
+    [Fact]
+    public async Task CachedDrain_OrphanRow_InsideGrace_IsRetried_NotAbandoned()
+    {
+        // The other side of the grace window, and the reason abandonment is not
+        // immediate: a missing snapshot is normally a brief write race (the
+        // audit row lands microseconds before the tracking row). Abandoning on
+        // the first failed lookup would throw away the operational half of
+        // every cached call that happened to lose that race.
+        var fresh = ScadaBridgeAuditEventFactory.Create(
+            channel: AuditChannel.ApiOutbound,
+            kind: AuditKind.CachedSubmit,
+            status: AuditStatus.Submitted,
+            eventId: Guid.NewGuid(),
+            occurredAtUtc: DateTime.UtcNow,          // inside the grace window
+            target: "ERP.GetOrder",
+            sourceSiteId: "site-1",
+            correlationId: Guid.NewGuid());
+
+        _queue.ReadPendingAsync(Arg.Any<int>(), Arg.Any<CancellationToken>())
+            .Returns(Task.FromResult<IReadOnlyList<AuditEvent>>(Array.Empty<AuditEvent>()));
+        _queue.ReadPendingCachedTelemetryAsync(Arg.Any<int>(), Arg.Any<CancellationToken>())
+            .Returns(Task.FromResult<IReadOnlyList<AuditEvent>>(new[] { fresh }));
+        _trackingStore.GetStatusAsync(
+                new TrackedOperationId(fresh.CorrelationId!.Value), Arg.Any<CancellationToken>())
+            .Returns(Task.FromResult<TrackingStatusSnapshot?>(null));
+
+        CreateActorWithCachedDrain();
+
+        // Give the drain several ticks to prove it keeps retrying rather than
+        // abandoning. A positive control guards against vacuity: the row must
+        // actually have been READ more than once, otherwise "never marked
+        // Forwarded" would also be true of a drain that never ran at all.
+        await AwaitAssertAsync(async () =>
+        {
+            await _queue.Received(Quantity.Within(2, int.MaxValue))
+                .ReadPendingCachedTelemetryAsync(Arg.Any<int>(), Arg.Any<CancellationToken>());
+        }, TimeSpan.FromSeconds(6));
+
+        await _queue.DidNotReceive().MarkForwardedAsync(
+            Arg.Any<IReadOnlyList<Guid>>(), Arg.Any<CancellationToken>());
+    }
+
+    [Fact]
+    public async Task CachedDrain_FullBatchOfUnresolvableRows_DoesNotStarveTheQueue()
+    {
+        // The actual regression. The queue is read oldest-first with a fixed
+        // BatchSize, so before the fix a batch's worth of permanently
+        // unresolvable rows at the head meant every subsequent drain re-read
+        // exactly those rows, failed identically, and never reached the newer
+        // rows behind them — a permanent stall of the cached-telemetry path,
+        // plus one log line per row per pass (~2,800/min measured on a rig).
+        var stuck = Enumerable.Range(0, 4).Select(_ => NewCachedEvent()).ToList();
+
+        _queue.ReadPendingAsync(Arg.Any<int>(), Arg.Any<CancellationToken>())
+            .Returns(Task.FromResult<IReadOnlyList<AuditEvent>>(Array.Empty<AuditEvent>()));
+        _queue.ReadPendingCachedTelemetryAsync(Arg.Any<int>(), Arg.Any<CancellationToken>())
+            .Returns(Task.FromResult<IReadOnlyList<AuditEvent>>(stuck));
+        foreach (var row in stuck)
+        {
+            _trackingStore.GetStatusAsync(
+                    new TrackedOperationId(row.CorrelationId!.Value), Arg.Any<CancellationToken>())
+                .Returns(Task.FromResult<TrackingStatusSnapshot?>(null));
+        }
+
+        CreateActorWithCachedDrain();
+
+        // Every row is abandoned in ONE call, so the head of the queue clears
+        // and the next drain can see past it.
+        await AwaitAssertAsync(async () =>
+        {
+            await _queue.Received(1).MarkForwardedAsync(
+                Arg.Is<IReadOnlyList<Guid>>(g => g.Count == stuck.Count),
+                Arg.Any<CancellationToken>());
+        }, TimeSpan.FromSeconds(5));
+
+        // Nothing was pushed — there was no operational half to build for any
+        // of them — but the drain still made progress rather than spinning.
+        await _client.DidNotReceive().IngestCachedTelemetryAsync(
+            Arg.Any<CachedTelemetryBatch>(), Arg.Any<CancellationToken>());
+    }
+
+    [Fact]
+    public async Task CachedDrain_TrackingStoreThrow_DoesNotAbandonTheRow()
+    {
+        // A throw is a STORE fault (locked, corrupt, mid-restore), not a
+        // verdict about this row: the snapshot may well exist and be readable
+        // a second later. Abandoning here would discard the operational half
+        // of every in-flight cached call during a transient SQLite lock — so
+        // the throw path deliberately never abandons, however old the row is.
+        var row = NewCachedEvent();   // deliberately older than the grace window
+
+        _queue.ReadPendingAsync(Arg.Any<int>(), Arg.Any<CancellationToken>())
+            .Returns(Task.FromResult<IReadOnlyList<AuditEvent>>(Array.Empty<AuditEvent>()));
+        _queue.ReadPendingCachedTelemetryAsync(Arg.Any<int>(), Arg.Any<CancellationToken>())
+            .Returns(Task.FromResult<IReadOnlyList<AuditEvent>>(new[] { row }));
+        _trackingStore.GetStatusAsync(
+                new TrackedOperationId(row.CorrelationId!.Value), Arg.Any<CancellationToken>())
+            .ThrowsAsync(new InvalidOperationException("tracking store unavailable"));
+
+        CreateActorWithCachedDrain();
+
+        // Positive control again: prove the drain actually ran repeatedly, so
+        // "never abandoned" is a real observation rather than an artefact of a
+        // drain that never executed.
+        await AwaitAssertAsync(async () =>
+        {
+            await _queue.Received(Quantity.Within(2, int.MaxValue))
+                .ReadPendingCachedTelemetryAsync(Arg.Any<int>(), Arg.Any<CancellationToken>());
+        }, TimeSpan.FromSeconds(6));
+
+        await _queue.DidNotReceive().MarkForwardedAsync(
+            Arg.Any<IReadOnlyList<Guid>>(), Arg.Any<CancellationToken>());
    }

    [Fact]
@@ -11,13 +11,13 @@ using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
 namespace ZB.MOM.WW.ScadaBridge.AuditLog.Tests.Site.Telemetry;

 /// <summary>
-/// Tests for <see cref="ClusterClientSiteAuditClient"/> — the production
+/// Tests for <see cref="SiteCommunicationAuditClient"/> — the production
 /// <see cref="ISiteStreamAuditClient"/> binding wired by the Host for site
 /// roles. The client maps the proto-DTO batches produced by
 /// <see cref="SiteAuditTelemetryActor"/> into the Akka
 /// <see cref="IngestAuditEventsCommand"/> / <see cref="IngestCachedTelemetryCommand"/>
-/// messages, Asks the site's <c>SiteCommunicationActor</c> (which forwards over
-/// ClusterClient to central), and maps the reply back into an
+/// messages, Asks the site's <c>SiteCommunicationActor</c> (which forwards to
+/// central over its central transport), and maps the reply back into an
 /// <see cref="IngestAck"/>.
 /// </summary>
 /// <remarks>
@@ -25,7 +25,7 @@ namespace ZB.MOM.WW.ScadaBridge.AuditLog.Tests.Site.Telemetry;
 /// it lets the tests assert the exact command shape AND drive the reply (or
 /// withhold one to exercise the Ask-timeout path).
 /// </remarks>
-public class ClusterClientSiteAuditClientTests : TestKit
+public class SiteCommunicationAuditClientTests : TestKit
 {
    /// <summary>Short Ask timeout so the timeout test completes quickly.</summary>
    private static readonly TimeSpan AskTimeout = TimeSpan.FromMilliseconds(500);
@@ -65,7 +65,7 @@ public class ClusterClientSiteAuditClientTests : TestKit
    public async Task IngestAuditEventsAsync_FullAck_MapsAllAcceptedIdsOntoAck()
    {
        var probe = CreateTestProbe();
-        var sut = new ClusterClientSiteAuditClient(probe.Ref, AskTimeout);
+        var sut = new SiteCommunicationAuditClient(probe.Ref, AskTimeout);

        var events = Enumerable.Range(0, 3).Select(_ => NewEvent()).ToList();
        var batch = BatchOf(events);
@@ -92,7 +92,7 @@ public class ClusterClientSiteAuditClientTests : TestKit
    public async Task IngestAuditEventsAsync_PartialAck_OnlyAcceptedIdsAppearOnAck()
    {
        var probe = CreateTestProbe();
-        var sut = new ClusterClientSiteAuditClient(probe.Ref, AskTimeout);
+        var sut = new SiteCommunicationAuditClient(probe.Ref, AskTimeout);

        var events = Enumerable.Range(0, 5).Select(_ => NewEvent()).ToList();
        var accepted = events.Take(3).Select(e => e.EventId).ToList();
@@ -114,7 +114,7 @@ public class ClusterClientSiteAuditClientTests : TestKit
    public async Task IngestAuditEventsAsync_AskTimeout_Throws_SoDrainLoopKeepsRowsPending()
    {
        var probe = CreateTestProbe();
-        var sut = new ClusterClientSiteAuditClient(probe.Ref, AskTimeout);
+        var sut = new SiteCommunicationAuditClient(probe.Ref, AskTimeout);

        var batch = BatchOf(new[] { NewEvent() });

@@ -131,7 +131,7 @@ public class ClusterClientSiteAuditClientTests : TestKit
    public async Task IngestAuditEventsAsync_FaultedReply_Throws()
    {
        var probe = CreateTestProbe();
-        var sut = new ClusterClientSiteAuditClient(probe.Ref, AskTimeout);
+        var sut = new SiteCommunicationAuditClient(probe.Ref, AskTimeout);

        var task = sut.IngestAuditEventsAsync(BatchOf(new[] { NewEvent() }), CancellationToken.None);
        probe.ExpectMsg<IngestAuditEventsCommand>(TimeSpan.FromSeconds(3));
@@ -147,7 +147,7 @@ public class ClusterClientSiteAuditClientTests : TestKit
    public async Task IngestCachedTelemetryAsync_RoutesCommand_AndMapsReply()
    {
        var probe = CreateTestProbe();
-        var sut = new ClusterClientSiteAuditClient(probe.Ref, AskTimeout);
+        var sut = new SiteCommunicationAuditClient(probe.Ref, AskTimeout);

        var events = Enumerable.Range(0, 2).Select(_ => NewEvent()).ToList();
        var batch = new CachedTelemetryBatch();
@@ -182,7 +182,7 @@ public class ClusterClientSiteAuditClientTests : TestKit
    public async Task IngestCachedTelemetryAsync_AskTimeout_Throws()
    {
        var probe = CreateTestProbe();
-        var sut = new ClusterClientSiteAuditClient(probe.Ref, AskTimeout);
+        var sut = new SiteCommunicationAuditClient(probe.Ref, AskTimeout);

        var batch = new CachedTelemetryBatch();
        batch.Packets.Add(new CachedTelemetryPacket
@@ -28,7 +28,7 @@ namespace ZB.MOM.WW.ScadaBridge.CentralUI.PlaywrightTests.Deployment;
 /// <b>Tolerant terminal-state assertion (validation-behavior protocol):</b> whether
 /// <c>Connect</c> reaches a <b>Live</b> badge depends on the owning site returning a
 /// snapshot for the instance. Reading <c>DebugView.razor::Connect</c>: it awaits
-/// <c>DebugStreamService.StartStreamAsync</c> (a ClusterClient snapshot round-trip);
+/// <c>DebugStreamService.StartStreamAsync</c> (a site snapshot round-trip);
 /// on success it flips <c>_connected = true</c> (Live badge + success toast), and on
 /// any exception it surfaces a <c>Connect failed: …</c> error <c>.toast</c>. Either
 /// way <c>_connecting</c> resets, so the click always resolves — it never hangs. The
@@ -19,13 +19,13 @@ namespace ZB.MOM.WW.ScadaBridge.CentralUI.PlaywrightTests.Deployment;
 /// and would never appear, so the test mints an instance via the shared
 /// <see cref="DeploymentFixture"/> and drives it to <c>Enabled</c> over the CLI
 /// (<c>instance deploy</c>) before selecting it. The fixture provisions onto the
-/// real, running <c>site-a</c> so the ClusterClient snapshot round-trip resolves
-/// fast (an unknown site has no registered ClusterClient and only times out).
+/// real, running <c>site-a</c> so the snapshot round-trip resolves
+/// fast (an unknown site has no reachable gRPC endpoint and only times out).
 /// </para>
 ///
 /// <para>
 /// <b>Tolerant terminal-state wait, then the tabs/trees assertions</b> — Connect
-/// awaits a ClusterClient snapshot round-trip; on success the page flips to the
+/// awaits a site snapshot round-trip; on success the page flips to the
 /// Live badge and renders the tabbed card, on failure it surfaces a
 /// <c>Connect failed: …</c> <c>.toast</c> (see <c>DebugView.razor::Connect</c>).
 /// The test first waits on a TERMINAL state (Live badge OR error toast) within a
@@ -11,8 +11,8 @@ namespace ZB.MOM.WW.ScadaBridge.CentralUI.PlaywrightTests.Deployment;
 ///
 /// <para>
 /// <b>Why <c>site-a</c> and not a throwaway site:</b> the deploy / enable / disable
-/// actions relay to the owning site over the Akka <c>ClusterClient</c>. An unknown
-/// site identifier has no registered <c>ClusterClient</c>, so the relay only
+/// actions relay to the owning site over the gRPC site command plane. An unknown
+/// site identifier has no reachable gRPC endpoint, so the relay only
 /// resolves on a slow 10-second timeout (and never surfaces a fast failure toast).
 /// To exercise the real action path, the ephemeral instances must therefore live
 /// on a <em>real, running</em> site — <c>site-a</c> — rather than a fixture-created
@@ -282,8 +282,8 @@ public class SiteCallsPageTests
            // be relayed to the owning site. Unlike the display-only tests above,
            // this one actually relays to the owning site, so the SourceSite must
            // be a *real* site identifier from the running cluster (site-a) and
-            // not the cosmetic "plant-a" label: an unknown site has no registered
-            // ClusterClient, so CentralCommunicationActor drops the envelope
+            // not the cosmetic "plant-a" label: an unknown site has no reachable
+            // gRPC endpoint, so CentralCommunicationActor drops the envelope
            // without replying and the relay only resolves on the 10s inner Ask
            // timeout — too slow for the toast assertion below. Relayed to a live
            // site, the site finds no parked S&F message for this freshly-seeded
@@ -319,7 +319,7 @@ public class SiteCallsPageTests
            // We only assert that an outcome toast appears (exactly one — the
            // single-toast contract), not which one, since the live cluster
            // state determines the outcome. The wait is generous (15s): the
-            // relay round-trips to the site over ClusterClient, and a worst-case
+            // relay round-trips to the site over gRPC, and a worst-case
            // path can sit on the 10s inner relay timeout before the response —
            // and the toast itself auto-dismisses 5s after it appears, so the
            // assertion must catch it inside that window.
@@ -42,7 +42,9 @@ public class MonitoringAuthorizationTests
    public void HealthDashboard_IsIntentionallyAllAuthenticatedRoles()
    {
        // Health Dashboard stays all-roles (no policy) per the design doc.
-        var attr = AuthorizeOf<Health>();
+        // Fully qualified: the Communication.Health namespace otherwise makes
+        // the bare `Health` page type ambiguous.
+        var attr = AuthorizeOf<CentralUI.Components.Pages.Monitoring.Health>();

        Assert.NotNull(attr);
        Assert.Null(attr!.Policy);
@@ -0,0 +1,147 @@
+namespace ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests;
+
+/// <summary>
+/// Unit tests for <see cref="ClusterBootstrapGuard"/> — the pure decision core of the
+/// simultaneous-cold-start split-brain guard (Gitea #33). The runtime probe + JoinSeedNodes wiring
+/// lives in <c>ClusterBootstrapCoordinator</c> and is covered by the real-ActorSystem
+/// <c>ClusterBootstrapCoordinatorTests</c> in the integration suite.
+/// </summary>
+public class ClusterBootstrapGuardTests
+{
+    private const string A = "akka.tcp://scadabridge@scadabridge-site-a-a:8082";
+    private const string B = "akka.tcp://scadabridge@scadabridge-site-a-b:8082";
+
+    [Fact]
+    public void Lower_address_node_is_the_founder_and_needs_no_probe()
+    {
+        // scadabridge-site-a-a < scadabridge-site-a-b, so on node-a the guard makes it the founder.
+        var role = ClusterBootstrapGuard.Analyze(new[] { A, B }, "scadabridge-site-a-a", 8082);
+
+        Assert.True(role.Applies);
+        Assert.True(role.IsFounder);
+        Assert.Equal(new[] { A, B }, role.SelfFirstOrder);
+    }
+
+    [Fact]
+    public void Higher_address_node_is_not_the_founder_and_targets_the_partner_for_probing()
+    {
+        // On node-b the partner is node-a (the lower/founder).
+        var role = ClusterBootstrapGuard.Analyze(new[] { B, A }, "scadabridge-site-a-b", 8082);
+
+        Assert.True(role.Applies);
+        Assert.False(role.IsFounder);
+        Assert.Equal("scadabridge-site-a-a", role.PartnerHost);
+        Assert.Equal(8082, role.PartnerPort);
+    }
+
+    [Fact]
+    public void Tie_break_is_symmetric_regardless_of_seed_order()
+    {
+        // Both nodes must agree on who founds no matter how each lists its seeds.
+        var onLower = ClusterBootstrapGuard.Analyze(new[] { A, B }, "scadabridge-site-a-a", 8082);
+        var onHigher = ClusterBootstrapGuard.Analyze(new[] { B, A }, "scadabridge-site-a-b", 8082);
+
+        Assert.True(onLower.IsFounder);
+        Assert.False(onHigher.IsFounder);
+        // Exactly one founder.
+        Assert.True(onLower.IsFounder ^ onHigher.IsFounder);
+    }
+
+    [Fact]
+    public void Tie_break_is_case_insensitive_so_a_casing_mismatch_cannot_make_both_founders()
+    {
+        // The two sides' seed config differ only in host casing (review note 1): if the tie-break
+        // were case-sensitive, both could conclude they are the founder and re-open the split.
+        var lowerSelf = "akka.tcp://scadabridge@SITE-A-A:8082";
+        var higherSelf = "akka.tcp://scadabridge@site-a-b:8082";
+
+        var onLower = ClusterBootstrapGuard.Analyze(new[] { lowerSelf, higherSelf }, "SITE-A-A", 8082);
+        var onHigher = ClusterBootstrapGuard.Analyze(new[] { higherSelf, lowerSelf }, "site-a-b", 8082);
+
+        Assert.True(onLower.IsFounder ^ onHigher.IsFounder);
+        Assert.True(onLower.IsFounder); // "site-a-a" < "site-a-b" ordinal-ignore-case
+    }
+
+    [Fact]
+    public void Higher_node_joins_the_founder_when_partner_is_reachable()
+    {
+        var role = ClusterBootstrapGuard.Analyze(new[] { B, A }, "scadabridge-site-a-b", 8082);
+
+        // Reachable ⇒ peer-first ⇒ JoinSeedNodeProcess ⇒ joins the founder, never self-forms.
+        Assert.Equal(new[] { A, B }, ClusterBootstrapGuard.HigherNodeOrder(role, partnerReachable: true)); // partner (node-a) first
+    }
+
+    [Fact]
+    public void Higher_node_forms_alone_when_partner_is_unreachable()
+    {
+        var role = ClusterBootstrapGuard.Analyze(new[] { B, A }, "scadabridge-site-a-b", 8082);
+
+        // Unreachable after the window ⇒ partner is dead ⇒ self-first ⇒ cold-start-alone preserved.
+        Assert.Equal(new[] { B, A }, ClusterBootstrapGuard.HigherNodeOrder(role, partnerReachable: false)); // self (node-b) first
+    }
+
+    [Fact]
+    public void Guard_does_not_apply_to_a_single_seed_node()
+    {
+        // A single-node install lists exactly one seed (itself) — not a pair, guard is inert.
+        var role = ClusterBootstrapGuard.Analyze(new[] { "akka.tcp://scadabridge@lone:8081" }, "lone", 8081);
+
+        Assert.False(role.Applies);
+    }
+
+    [Fact]
+    public void Guard_does_not_apply_when_self_is_absent_from_the_two_seeds()
+    {
+        var role = ClusterBootstrapGuard.Analyze(new[] { A, B }, "scadabridge-central-a", 8081);
+
+        Assert.False(role.Applies);
+    }
+
+    [Fact]
+    public void Guard_does_not_apply_to_three_or_more_seeds()
+    {
+        var role = ClusterBootstrapGuard.Analyze(
+            new[] { A, B, "akka.tcp://scadabridge@scadabridge-site-a-c:8082" }, "scadabridge-site-a-a", 8082);
+
+        Assert.False(role.Applies);
+    }
+
+    [Fact]
+    public void Guard_does_not_apply_when_a_seed_is_malformed()
+    {
+        var role = ClusterBootstrapGuard.Analyze(new[] { A, "not-a-seed" }, "scadabridge-site-a-a", 8082);
+
+        Assert.False(role.Applies);
+    }
+
+    [Theory]
+    [InlineData("akka.tcp://scadabridge@scadabridge-site-a-a:8082", "scadabridge-site-a-a", 8082)]
+    [InlineData("akka.tcp://scadabridge@10.0.0.5:8082/", "10.0.0.5", 8082)]
+    [InlineData("  akka.tcp://scadabridge@host.lan:1234  ", "host.lan", 1234)]
+    public void TryParse_extracts_host_and_port(string seed, string expectedHost, int expectedPort)
+    {
+        Assert.True(ClusterBootstrapGuard.TryParse(seed, out var host, out var port));
+        Assert.Equal(expectedHost, host);
+        Assert.Equal(expectedPort, port);
+    }
+
+    [Theory]
+    [InlineData("")]
+    [InlineData("garbage")]
+    [InlineData("akka.tcp://scadabridge@host")] // no port
+    public void TryParse_rejects_malformed_seeds(string seed)
+    {
+        Assert.False(ClusterBootstrapGuard.TryParse(seed, out _, out _));
+    }
+
+    [Fact]
+    public void Port_participates_in_the_tie_break_when_hosts_are_equal()
+    {
+        // Shared-host loopback pair (test rig): same host, different ports — port breaks the tie.
+        const string low = "akka.tcp://scadabridge@127.0.0.1:8081";
+        const string high = "akka.tcp://scadabridge@127.0.0.1:8082";
+
+        Assert.True(ClusterBootstrapGuard.Analyze(new[] { low, high }, "127.0.0.1", 8081).IsFounder);
+        Assert.False(ClusterBootstrapGuard.Analyze(new[] { high, low }, "127.0.0.1", 8082).IsFounder);
+    }
+}
@@ -215,4 +215,68 @@ public class ClusterOptionsValidatorTests
        Assert.Contains("StableAfter", result.FailureMessage);
        Assert.Contains("HeartbeatInterval", result.FailureMessage);
    }
+
+    // ---- BootstrapGuard timing validation (Gitea #33) ----
+
+    [Fact]
+    public void BootstrapGuard_disabled_with_zero_timings_passes_validation()
+    {
+        // The knobs are inert when the guard is off, so a disabled guard must never block a boot —
+        // even if the timing values are nonsense (0 here).
+        var options = ValidOptions();
+        options.BootstrapGuard = new ClusterBootstrapGuardOptions
+        {
+            Enabled = false,
+            PartnerProbeSeconds = 0,
+            PartnerProbeIntervalMs = 0,
+            ProbeConnectTimeoutMs = 0,
+        };
+
+        var result = new ClusterOptionsValidator().Validate(null, options);
+
+        Assert.True(result.Succeeded, result.FailureMessage);
+    }
+
+    [Fact]
+    public void BootstrapGuard_enabled_with_default_timings_passes_validation()
+    {
+        var options = ValidOptions();
+        options.BootstrapGuard = new ClusterBootstrapGuardOptions { Enabled = true };
+
+        var result = new ClusterOptionsValidator().Validate(null, options);
+
+        Assert.True(result.Succeeded, result.FailureMessage);
+    }
+
+    [Fact]
+    public void BootstrapGuard_enabled_with_nonpositive_probe_seconds_fails_validation()
+    {
+        // A zero PartnerProbeSeconds silently degrades the guard to "never wait, always conclude the
+        // partner is dead" — the higher node forms alone immediately and re-opens the split. Reject it.
+        var options = ValidOptions();
+        options.BootstrapGuard = new ClusterBootstrapGuardOptions { Enabled = true, PartnerProbeSeconds = 0 };
+
+        var result = new ClusterOptionsValidator().Validate(null, options);
+
+        Assert.True(result.Failed);
+        Assert.Contains("PartnerProbeSeconds", result.FailureMessage);
+    }
+
+    [Fact]
+    public void BootstrapGuard_enabled_with_nonpositive_interval_or_timeout_fails_validation()
+    {
+        var options = ValidOptions();
+        options.BootstrapGuard = new ClusterBootstrapGuardOptions
+        {
+            Enabled = true,
+            PartnerProbeIntervalMs = 0,
+            ProbeConnectTimeoutMs = -1,
+        };
+
+        var result = new ClusterOptionsValidator().Validate(null, options);
+
+        Assert.True(result.Failed);
+        Assert.Contains("PartnerProbeIntervalMs", result.FailureMessage);
+        Assert.Contains("ProbeConnectTimeoutMs", result.FailureMessage);
+    }
 }
@@ -8,10 +8,10 @@ using ZB.MOM.WW.ScadaBridge.Commons.Messages.Streaming;
 namespace ZB.MOM.WW.ScadaBridge.Commons.Tests.Messages;

 /// <summary>
-/// Contract-lock tests for the highest-traffic ClusterClient record contracts.
-/// ClusterClient traffic uses Akka.NET's default Newtonsoft-JSON serializer and the
-/// design supports system-wide artifact version skew across sites, so these records
-/// must obey additive-only evolution:
+/// Contract-lock tests for the highest-traffic cross-boundary record contracts.
+/// These predate the ClusterClient→gRPC migration, when this traffic rode Akka.NET's
+/// default Newtonsoft-JSON serializer; the design still supports system-wide artifact
+/// version skew across sites, so these records must still obey additive-only evolution:
 ///   (a) an old-shape payload (missing newer trailing optional fields) still
 ///       deserializes, with the missing fields taking their constructor defaults;
 ///   (b) an unknown field from a *newer* peer is ignored (never throws);
@@ -94,7 +94,7 @@ public class ManagementCommandRegistryTests

    /// <summary>
    /// M9-T32c: the schema-library CRUD commands must auto-register by reflection
-    /// (no manual registry entry) so the HTTP / ClusterClient boundary can route them.
+    /// (no manual registry entry) so the HTTP management boundary can route them.
    /// </summary>
    [Fact]
    public void Resolve_SchemaLibraryCommands_AllRegistered()
@@ -104,7 +104,7 @@ public class SiteCommunicationActorTransportTests : TestKit
        // The seam preserves the reply-routing the S&F layer depends on: when the transport
        // answers the captured replyTo with a Status.Failure (its transient-failure signal on a
        // non-OK status), that failure reaches the original sender — here the test actor — so the
-        // waiting Ask faults, exactly as it did on the ClusterClient path.
+        // waiting Ask faults, exactly as it did on the removed ClusterClient path.
        var transport = Substitute.For<ICentralTransport>();
        transport
            .When(t => t.SubmitNotification(Arg.Any<NotificationSubmit>(), Arg.Any<IActorRef>()))
@@ -15,10 +15,10 @@ using ZB.MOM.WW.ScadaBridge.Communication.Actors;
 namespace ZB.MOM.WW.ScadaBridge.Communication.Tests;

 /// <summary>
-/// Tests for the Audit Log (#23) site→central ClusterClient ingest routing on
-/// <see cref="CentralCommunicationActor"/>. A site ClusterClient delivers
+/// Tests for the Audit Log (#23) site→central ingest routing on
+/// <see cref="CentralCommunicationActor"/>. A site delivers
 /// <see cref="IngestAuditEventsCommand"/> / <see cref="IngestCachedTelemetryCommand"/>
-/// to the receptionist-registered actor, which forwards to the registered
+/// to the actor, which forwards to the registered
 /// <c>AuditLogIngestActor</c> proxy and routes the reply back to the site.
 /// Mirrors the NotificationSubmit / RegisterNotificationOutbox pattern.
 /// </summary>
@@ -17,7 +17,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Tests;
 /// Tests that <see cref="CentralCommunicationActor"/> routes a site→central
 /// <see cref="ReconcileSiteRequest"/> through the scoped <see cref="ReconcileService"/>
 /// and pipes the resulting <see cref="ReconcileSiteResponse"/> back to the original
-/// sender (the site's ClusterClient path). Mirrors the audit-ingest routing tests.
+/// sender (the site's transport path). Mirrors the audit-ingest routing tests.
 /// </summary>
 public class CentralCommunicationActorReconcileTests : TestKit
 {
@@ -293,7 +293,7 @@ public class DebugStreamBridgeActorTests : TestKit

        ctx.BridgeActor.Tell(new StopDebugStream());

-        // Should send ClusterClient unsubscribe
+        // Should send the site-addressed unsubscribe
        var envelope = ctx.CommProbe.ExpectMsg<SiteEnvelope>();
        Assert.IsType<UnsubscribeDebugViewRequest>(envelope.Message);

@@ -60,7 +60,7 @@ public class SiteCommunicationActorTests : TestKit
    public void DeploymentStateQuery_ForwardedToDeploymentManager()
    {
        // DeploymentManager-006: the site-before-redeploy query travels over the
-        // ClusterClient command/control transport and is routed to the local
+        // gRPC command/control transport and is routed to the local
        // Deployment Manager, which owns the deployed-config store.
        var dmProbe = CreateTestProbe();
        var siteActor = Sys.ActorOf(Props.Create(() =>
@@ -0,0 +1,102 @@
+using Opc.Ua;
+using ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Adapters;
+
+namespace ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests.Adapters;
+
+/// <summary>
+/// Covers <see cref="RealOpcUaClient.RewriteEndpointHostForReachability"/> — the fix for OPC UA
+/// servers (e.g. OtOpcUa v3) that advertise an unroutable base address (a 0.0.0.0 wildcard bind or
+/// an internal container/NAT hostname). The session must dial the host the operator reached, not
+/// the advertised one, while preserving the advertised scheme + path.
+/// </summary>
+public class RealOpcUaClientEndpointRewriteTests
+{
+    private static string? Rewrite(string advertised, string discovery)
+    {
+        var ep = new EndpointDescription(advertised);
+        RealOpcUaClient.RewriteEndpointHostForReachability(ep, discovery);
+        return ep.EndpointUrl;
+    }
+
+    [Fact]
+    public void Rewrites_0_0_0_0_advertisement_to_the_reachable_host_preserving_path()
+    {
+        Assert.Equal(
+            "opc.tcp://otopcua-dev-site-a-1-1:4840/OtOpcUa",
+            Rewrite("opc.tcp://0.0.0.0:4840/OtOpcUa", "opc.tcp://otopcua-dev-site-a-1-1:4840/OtOpcUa"));
+    }
+
+    [Fact]
+    public void Rewrites_internal_nat_hostname_to_the_reachable_host()
+    {
+        Assert.Equal(
+            "opc.tcp://plc.example.com:4840/OtOpcUa",
+            Rewrite("opc.tcp://be5667fab9da:4840/OtOpcUa", "opc.tcp://plc.example.com:4840/OtOpcUa"));
+    }
+
+    [Fact]
+    public void Rewrites_port_too_when_the_advertised_port_differs()
+    {
+        Assert.Equal(
+            "opc.tcp://gateway:51210/OtOpcUa",
+            Rewrite("opc.tcp://0.0.0.0:4840/OtOpcUa", "opc.tcp://gateway:51210/OtOpcUa"));
+    }
+
+    [Fact]
+    public void Leaves_an_already_reachable_advertisement_untouched()
+    {
+        // opc-plc advertises its real container name — must not be rewritten.
+        Assert.Equal(
+            "opc.tcp://scadabridge-opcua:50000",
+            Rewrite("opc.tcp://scadabridge-opcua:50000", "opc.tcp://scadabridge-opcua:50000"));
+    }
+
+    [Fact]
+    public void Handles_authority_only_advertisement_without_appending_a_trailing_slash()
+    {
+        Assert.Equal(
+            "opc.tcp://host-b:4840",
+            Rewrite("opc.tcp://0.0.0.0:4840", "opc.tcp://host-b:4840"));
+    }
+
+    [Fact]
+    public void Re_wraps_an_ipv6_reachable_host_in_brackets()
+    {
+        Assert.Equal(
+            "opc.tcp://[::1]:4840/OtOpcUa",
+            Rewrite("opc.tcp://0.0.0.0:4840/OtOpcUa", "opc.tcp://[::1]:4840/OtOpcUa"));
+    }
+
+    [Fact]
+    public void Emits_no_port_when_neither_url_carries_one()
+    {
+        // opc.tcp has no default port, so Uri.Port is -1 on both — must not emit ":-1".
+        Assert.Equal(
+            "opc.tcp://myhost/UaServer",
+            Rewrite("opc.tcp://0.0.0.0/UaServer", "opc.tcp://myhost/UaServer"));
+    }
+
+    [Fact]
+    public void Uses_the_reachable_port_when_the_advertisement_omits_one()
+    {
+        Assert.Equal(
+            "opc.tcp://myhost:4840/UaServer",
+            Rewrite("opc.tcp://0.0.0.0/UaServer", "opc.tcp://myhost:4840/UaServer"));
+    }
+
+    [Theory]
+    [InlineData("")]
+    [InlineData("   ")]
+    [InlineData("not-a-uri")]
+    public void Leaves_endpoint_unchanged_when_discovery_url_is_unusable(string discovery)
+    {
+        Assert.Equal("opc.tcp://0.0.0.0:4840/OtOpcUa", Rewrite("opc.tcp://0.0.0.0:4840/OtOpcUa", discovery));
+    }
+
+    [Fact]
+    public void Null_endpoint_is_a_no_op()
+    {
+        // Must not throw.
+        RealOpcUaClient.RewriteEndpointHostForReachability(null, "opc.tcp://host:4840");
+    }
+}
@@ -5,6 +5,7 @@ using Microsoft.Extensions.DependencyInjection;
 using Microsoft.Extensions.Diagnostics.HealthChecks;
 using Microsoft.Extensions.Options;
 using ZB.MOM.WW.Health;
+using ZB.MOM.WW.Health.Akka;
 using ZB.MOM.WW.ScadaBridge.Host.Health;

 namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
@@ -207,11 +208,13 @@ public class HealthCheckTests : IDisposable
    }

    [Fact]
-    public void ActiveNodeCheck_IsHostLocalOldestNodeCheck()
+    public void ActiveNodeCheck_IsTheSharedOldestUpMemberCheck()
    {
-        // The shared package's ActiveNodeHealthCheck is LEADER-based (review 01 [High]):
-        // during a partition both nodes think they lead and Traefik serves both. The
-        // active-node check must be the host-local oldest-member check instead.
+        // The property under test is that the active tier selects by AGE, not by leadership: during a
+        // partition both sides think they lead and Traefik would serve both (review 01 [High]).
+        // Through ZB.MOM.WW.Health 0.2.1 the shared check was leader-based, so this host carried a
+        // private OldestNodeActiveHealthCheck; 0.3.0 made the shared check age-based, so the private
+        // copy is gone and the shared type is now the correct answer here.
        var previousEnv = Environment.GetEnvironmentVariable("DOTNET_ENVIRONMENT");
        try
        {
@@ -222,7 +225,7 @@ public class HealthCheckTests : IDisposable
            Assert.Contains(ZbHealthTags.Active, active.Tags);

            var check = active.Factory(factory.Services);
-            Assert.IsType<OldestNodeActiveHealthCheck>(check);
+            Assert.IsType<ActiveNodeHealthCheck>(check);
        }
        finally
        {
@@ -0,0 +1,239 @@
+using Microsoft.AspNetCore.Builder;
+using Microsoft.Extensions.Configuration;
+using Microsoft.Extensions.DependencyInjection;
+using Microsoft.Extensions.Diagnostics.HealthChecks;
+using Microsoft.Extensions.Options;
+using ZB.MOM.WW.Health;
+using ZB.MOM.WW.Health.Akka;
+using ZB.MOM.WW.LocalDb;
+using ZB.MOM.WW.LocalDb.Registration;
+using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
+using ZB.MOM.WW.ScadaBridge.HealthMonitoring;
+using ZB.MOM.WW.ScadaBridge.Host.Health;
+
+namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
+
+/// <summary>
+/// Site-node health endpoints. Site nodes previously served no health surface at all — only gRPC on
+/// the HTTP/2 listener and <c>/metrics</c> on the HTTP/1.1 one — so the family overview dashboard
+/// had no uniform way to probe them. These tests pin the three checks the site now registers and
+/// the tier each one lands in.
+/// <para>
+/// The registration assertions build the REAL container from
+/// <see cref="SiteServiceRegistration.Configure"/> rather than a hand-assembled
+/// <c>ServiceCollection</c>, for the reason spelled out in <see cref="SiteLocalDbWiringTests"/>:
+/// this family's wiring defects have consistently been ones that only a container-built graph
+/// catches. Each registration is additionally ACTIVATED through its factory, so a check that is
+/// registered but cannot resolve its dependencies fails here instead of at the first live probe.
+/// </para>
+/// </summary>
+public class SiteHealthCheckTests : IDisposable
+{
+    private readonly WebApplication _host;
+    private readonly string _tempDbPath;
+    private readonly string _tempSiteDbPath;
+    private readonly string _tempTrackingDbPath;
+
+    public SiteHealthCheckTests()
+    {
+        var stamp = Guid.NewGuid();
+        _tempDbPath = Path.Combine(Path.GetTempPath(), $"scadabridge_health_localdb_{stamp}.db");
+        _tempSiteDbPath = Path.Combine(Path.GetTempPath(), $"scadabridge_health_site_{stamp}.db");
+        _tempTrackingDbPath = Path.Combine(Path.GetTempPath(), $"scadabridge_health_track_{stamp}.db");
+
+        var builder = WebApplication.CreateBuilder();
+        builder.Configuration.Sources.Clear();
+        builder.Configuration.AddInMemoryCollection(new Dictionary<string, string?>
+        {
+            ["ScadaBridge:Node:Role"] = "Site",
+            ["ScadaBridge:Node:NodeName"] = "node-a",
+            ["ScadaBridge:Node:NodeHostname"] = "test-site",
+            ["ScadaBridge:Node:SiteId"] = "TestSite",
+            ["ScadaBridge:Node:RemotingPort"] = "0",
+            ["ScadaBridge:Node:GrpcPort"] = "0",
+            ["ScadaBridge:Database:SiteDbPath"] = _tempSiteDbPath,
+            ["ScadaBridge:OperationTracking:ConnectionString"] = $"Data Source={_tempTrackingDbPath}",
+            ["ScadaBridge:Cluster:SeedNodes:0"] = "akka.tcp://scadabridge@localhost:2551",
+            ["ScadaBridge:Cluster:SeedNodes:1"] = "akka.tcp://scadabridge@localhost:2552",
+            ["LocalDb:Path"] = _tempDbPath,
+        });
+
+        builder.Services.AddGrpc();
+        builder.Services.AddSingleton<ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamGrpcServer>();
+
+        SiteServiceRegistration.Configure(builder.Services, builder.Configuration);
+
+        AkkaHostedServiceRemover.RemoveAkkaHostedServiceOnly(builder.Services);
+
+        _host = builder.Build();
+    }
+
+    public void Dispose()
+    {
+        (_host as IDisposable)?.Dispose();
+        foreach (var path in new[] { _tempDbPath, _tempSiteDbPath, _tempTrackingDbPath })
+        {
+            try { File.Delete(path); } catch { /* best effort */ }
+        }
+
+        GC.SuppressFinalize(this);
+    }
+
+    private IReadOnlyList<HealthCheckRegistration> Registrations =>
+        _host.Services.GetRequiredService<IOptions<HealthCheckServiceOptions>>().Value.Registrations.ToList();
+
+    [Fact]
+    public void Site_RegistersExactlyTheThreeSiteChecks()
+    {
+        // Exact-set, not "contains": an extra check silently changes what /health/ready means for
+        // every consumer, and a site node inheriting a central-only probe is precisely the mistake
+        // this asserts against.
+        Assert.Equal(
+            new[] { "active-node", "akka-cluster", "localdb" },
+            Registrations.Select(r => r.Name).OrderBy(n => n, StringComparer.Ordinal).ToArray());
+    }
+
+    [Theory]
+    [InlineData("akka-cluster", ZbHealthTags.Ready)]
+    [InlineData("localdb", ZbHealthTags.Ready)]
+    [InlineData("active-node", ZbHealthTags.Active)]
+    public void Site_ChecksCarryTheirIntendedTier(string name, string expectedTag)
+    {
+        var registration = Assert.Single(Registrations, r => r.Name == name);
+
+        // Exactly one tier tag each: MapZbHealth selects by tag, so a check tagged both Ready and
+        // Active would put standby-ness into readiness and drop the standby out of readiness too.
+        Assert.Equal(new[] { expectedTag }, registration.Tags.ToArray());
+    }
+
+    [Theory]
+    [InlineData("akka-cluster", typeof(AkkaClusterHealthCheck))]
+    [InlineData("localdb", typeof(SiteLocalDbHealthCheck))]
+    [InlineData("active-node", typeof(SitePairActiveNodeHealthCheck))]
+    public void Site_ChecksActivateToTheIntendedType(string name, Type expected)
+    {
+        var registration = Assert.Single(Registrations, r => r.Name == name);
+
+        // Running the factory is the point: a type-activated check whose constructor dependencies
+        // are not registered on site would throw here rather than on the first probe in production.
+        var check = registration.Factory(_host.Services);
+
+        Assert.IsType(expected, check);
+    }
+
+    [Fact]
+    public void Site_ActiveNodeCheck_IsRoleScoped_NotTheUnscopedCentralCheck()
+    {
+        // Central registers the shared ActiveNodeHealthCheck UNSCOPED — oldest Up member of the whole
+        // cluster. On a mesh carrying more than one site's members that computes "oldest" across the
+        // wrong member set, so a site node could report Active purely for being the oldest member
+        // overall. A site must answer for its own site-{SiteId} role, which SitePairActiveNodeHealthCheck
+        // does by delegating to the site's IClusterNodeProvider (itself now backed by the shared
+        // ClusterActiveNode, so the rule is shared even though the scoping is local).
+        var check = Assert.Single(Registrations, r => r.Name == "active-node").Factory(_host.Services);
+
+        Assert.IsType<SitePairActiveNodeHealthCheck>(check);
+        Assert.IsNotType<ActiveNodeHealthCheck>(check);
+    }
+
+    [Theory]
+    [InlineData("database")]
+    [InlineData("required-singletons")]
+    public void Site_DoesNotRegisterCentralOnlyChecks(string centralOnlyName)
+    {
+        // Positive control first: if the harness ever stopped registering health checks at all,
+        // every "is absent" assertion below would pass vacuously and this test would be worthless.
+        Assert.Contains(Registrations, r => r.Name == "akka-cluster");
+
+        Assert.DoesNotContain(Registrations, r => r.Name == centralOnlyName);
+    }
+
+    [Fact]
+    public async Task Site_LocalDbCheck_ReportsHealthyAgainstTheRealSiteDatabase()
+    {
+        // Behaviour, not wiring: the probe runs against the consolidated site database this
+        // composition actually built, so a check that resolves but cannot query fails here.
+        var check = (SiteLocalDbHealthCheck)Assert.Single(Registrations, r => r.Name == "localdb")
+            .Factory(_host.Services);
+
+        var result = await check.CheckHealthAsync(NewContext("localdb", check));
+
+        Assert.Equal(HealthStatus.Healthy, result.Status);
+    }
+
+    [Fact]
+    public async Task Site_LocalDbCheck_EnrichesWithReplicationStateWithoutFailingOnDefaultOff()
+    {
+        // Replication ships default-OFF, so the unreplicated steady state must stay Healthy while
+        // still surfacing the state as data — enrichment must never become a verdict.
+        var check = (SiteLocalDbHealthCheck)Assert.Single(Registrations, r => r.Name == "localdb")
+            .Factory(_host.Services);
+
+        var result = await check.CheckHealthAsync(NewContext("localdb", check));
+
+        Assert.Equal(HealthStatus.Healthy, result.Status);
+        Assert.False(Assert.IsType<bool>(result.Data["replicationConnected"]));
+        Assert.Equal(0L, Assert.IsType<long>(result.Data["replicationOplogBacklog"]));
+    }
+
+    [Fact]
+    public async Task Site_LocalDbCheck_ReportsUnhealthyWhenTheStoreIsUnreachable()
+    {
+        var check = new SiteLocalDbHealthCheck(new ThrowingLocalDb());
+
+        var result = await check.CheckHealthAsync(NewContext("localdb", check));
+
+        Assert.Equal(HealthStatus.Unhealthy, result.Status);
+    }
+
+    [Theory]
+    [InlineData(true, HealthStatus.Healthy)]
+    [InlineData(false, HealthStatus.Unhealthy)]
+    public async Task Site_ActiveNodeCheck_SplitsPrimaryFromStandby(bool selfIsPrimary, HealthStatus expected)
+    {
+        // 200 on the primary / 503 on the standby is the same contract central publishes, so a
+        // consumer reads one active-node meaning fleet-wide.
+        var check = new SitePairActiveNodeHealthCheck(new StubNodeProvider(selfIsPrimary));
+
+        var result = await check.CheckHealthAsync(NewContext("active-node", check));
+
+        Assert.Equal(expected, result.Status);
+    }
+
+    private static HealthCheckContext NewContext(string name, IHealthCheck check) => new()
+    {
+        Registration = new HealthCheckRegistration(name, check, HealthStatus.Unhealthy, tags: null),
+    };
+
+    private sealed class StubNodeProvider(bool selfIsPrimary) : IClusterNodeProvider
+    {
+        public bool SelfIsPrimary { get; } = selfIsPrimary;
+
+        public IReadOnlyList<NodeStatus> GetClusterNodes() => [];
+    }
+
+    private sealed class ThrowingLocalDb : ILocalDb
+    {
+        public Task<int> ExecuteAsync(string sql, object? parameters = null, CancellationToken ct = default) =>
+            throw new InvalidOperationException("store unreachable");
+
+        public Task<IReadOnlyList<T>> QueryAsync<T>(
+            string sql,
+            Func<Microsoft.Data.Sqlite.SqliteDataReader, T> map,
+            object? parameters = null,
+            CancellationToken ct = default) =>
+            throw new InvalidOperationException("store unreachable");
+
+        public Task<ILocalDbTransaction> BeginTransactionAsync(CancellationToken ct = default) =>
+            throw new InvalidOperationException("store unreachable");
+
+        public Microsoft.Data.Sqlite.SqliteConnection CreateConnection() =>
+            throw new InvalidOperationException("store unreachable");
+
+        public ReplicatedTable RegisterReplicated(string tableName) =>
+            throw new InvalidOperationException("store unreachable");
+
+        public IReadOnlyDictionary<string, ReplicatedTable> ReplicatedTables =>
+            throw new InvalidOperationException("store unreachable");
+    }
+}
@@ -0,0 +1,156 @@
+using System.Net;
+using System.Text.Json;
+using Microsoft.AspNetCore.Hosting;
+using Microsoft.AspNetCore.Mvc.Testing;
+using Microsoft.AspNetCore.TestHost;
+using Microsoft.Extensions.DependencyInjection;
+
+namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
+
+/// <summary>
+/// Proves the site pipeline actually MAPS the health endpoints — the half
+/// <see cref="SiteHealthCheckTests"/> cannot cover, since that fixture builds the site container
+/// from <see cref="SiteServiceRegistration"/> and never runs <c>Program</c>'s endpoint wiring.
+/// Without this, deleting <c>app.MapZbHealth()</c> from the site branch would leave every
+/// registration test green while site nodes served 404 to the overview dashboard.
+/// <para>
+/// Boots the real <c>Program</c> in the Site role. <c>Program</c> builds its own
+/// <c>ConfigurationBuilder</c> before the web host exists, so its role and settings come from
+/// PROCESS-WIDE environment variables (<c>SCADABRIDGE_CONFIG</c> selects appsettings.Site.json)
+/// rather than from <c>WithWebHostBuilder</c> — which is why this fixture joins
+/// <see cref="HostBootCollection"/> and restores every var it sets.
+/// </para>
+/// </summary>
+[Collection(HostBootCollection.Name)]
+public class SiteHealthEndpointTests : IDisposable
+{
+    private readonly List<IDisposable> _disposables = new();
+    private readonly Dictionary<string, string?> _previousEnv = new(StringComparer.Ordinal);
+    private readonly string _tempDbPath;
+
+    public SiteHealthEndpointTests()
+    {
+        _tempDbPath = Path.Combine(Path.GetTempPath(), $"scadabridge_health_ep_{Guid.NewGuid()}.db");
+
+        // Whole-key env overrides, the sanctioned path: supplying GrpcPsk concretely makes the
+        // pre-host secrets expander skip appsettings.Site.json's ${secret:SB-GRPC-PSK-site-1}
+        // token, so this test needs no seeded secrets store.
+        //
+        // The remoting port is moved off the site default (8082) so this fixture cannot collide
+        // with a locally running node, but it must be a REAL port and seed-nodes[0] must be this
+        // node itself — StartupValidator enforces both before the host is built, and it runs
+        // whether or not the Akka hosted service is later removed. Nothing binds it: the hosted
+        // service is removed below.
+        SetEnv("SCADABRIDGE_CONFIG", "Site");
+        SetEnv("ScadaBridge__Node__Role", "Site");
+        SetEnv("ScadaBridge__Node__SiteId", "TestSite");
+        SetEnv("ScadaBridge__Node__NodeHostname", "localhost");
+        SetEnv("ScadaBridge__Node__RemotingPort", "18082");
+        SetEnv("ScadaBridge__Cluster__SeedNodes__0", "akka.tcp://scadabridge@localhost:18082");
+        SetEnv("ScadaBridge__Cluster__SeedNodes__1", "akka.tcp://scadabridge@localhost:18085");
+        SetEnv("ScadaBridge__Communication__GrpcPsk", "test-psk-0123456789");
+        SetEnv("LocalDb__Path", _tempDbPath);
+    }
+
+    private void SetEnv(string key, string? value)
+    {
+        _previousEnv[key] = Environment.GetEnvironmentVariable(key);
+        Environment.SetEnvironmentVariable(key, value);
+    }
+
+    public void Dispose()
+    {
+        foreach (var d in _disposables)
+        {
+            try { d.Dispose(); } catch { /* best effort */ }
+        }
+
+        foreach (var (key, value) in _previousEnv)
+        {
+            Environment.SetEnvironmentVariable(key, value);
+        }
+
+        try { File.Delete(_tempDbPath); } catch { /* best effort */ }
+
+        GC.SuppressFinalize(this);
+    }
+
+    private HttpClient CreateSiteClient()
+    {
+        var factory = new WebApplicationFactory<Program>()
+            .WithWebHostBuilder(builder =>
+            {
+                // WebApplicationFactory defaults the environment to Development, which turns on
+                // ValidateOnBuild/ValidateScopes. The site graph carries scoped registrations whose
+                // dependencies are central-only (ReconcileService → IDeploymentManagerRepository,
+                // AuditLogKpiSampleSource → IAuditLogRepository) and are never resolved on a site
+                // node, so eager validation fails on a composition that runs fine in production.
+                // That is a pre-existing property of the site container, not of the health wiring
+                // under test — run this fixture the way a site node actually runs.
+                builder.UseEnvironment("Production");
+
+                // ConfigureTestServices runs after the app's own registrations, so this drops the
+                // hosted service that would form a real cluster while leaving the AkkaHostedService
+                // singleton resolvable (the site pipeline resolves it for shutdown ordering).
+                builder.ConfigureTestServices(AkkaHostedServiceRemover.RemoveAkkaHostedServiceOnly);
+            });
+        _disposables.Add(factory);
+
+        var client = factory.CreateClient();
+        _disposables.Add(client);
+        return client;
+    }
+
+    [Fact]
+    public async Task Site_MapsHealthReady_WithTheCanonicalJsonBody()
+    {
+        var response = await CreateSiteClient().GetAsync("/health/ready");
+
+        // Never 404. 200 vs 503 depends on cluster state this fixture does not form, so both are
+        // accepted — what is asserted is that the tier is mapped and answers in canonical shape.
+        Assert.NotEqual(HttpStatusCode.NotFound, response.StatusCode);
+        Assert.True(
+            response.StatusCode is HttpStatusCode.OK or HttpStatusCode.ServiceUnavailable,
+            $"Expected 200 or 503, got {(int)response.StatusCode}");
+
+        using var doc = JsonDocument.Parse(await response.Content.ReadAsStringAsync());
+        var entries = doc.RootElement.GetProperty("entries");
+
+        Assert.True(entries.TryGetProperty("akka-cluster", out _), "ready tier must carry akka-cluster");
+        Assert.True(entries.TryGetProperty("localdb", out _), "ready tier must carry localdb");
+        // Readiness must NOT depend on being the primary, or a healthy standby would read unready.
+        Assert.False(entries.TryGetProperty("active-node", out _), "active-node belongs to the Active tier");
+    }
+
+    [Fact]
+    public async Task Site_MapsHealthActive_CarryingOnlyTheActiveNodeCheck()
+    {
+        var response = await CreateSiteClient().GetAsync("/health/active");
+
+        Assert.NotEqual(HttpStatusCode.NotFound, response.StatusCode);
+
+        using var doc = JsonDocument.Parse(await response.Content.ReadAsStringAsync());
+        var entries = doc.RootElement.GetProperty("entries");
+
+        Assert.Equal(
+            new[] { "active-node" },
+            entries.EnumerateObject().Select(p => p.Name).ToArray());
+    }
+
+    [Fact]
+    public async Task Site_HealthEndpointsAreAnonymous()
+    {
+        // The dashboard authenticates nowhere. The site pipeline runs no authentication middleware
+        // today, so this is a regression pin: adding one later must not silently close these.
+        var client = CreateSiteClient();
+
+        foreach (var path in new[] { "/health/ready", "/health/active", "/healthz" })
+        {
+            var response = await client.GetAsync(path);
+
+            Assert.NotEqual(HttpStatusCode.Unauthorized, response.StatusCode);
+            Assert.NotEqual(HttpStatusCode.Forbidden, response.StatusCode);
+            Assert.NotEqual(HttpStatusCode.NotFound, response.StatusCode);
+        }
+    }
+}
@@ -24,14 +24,14 @@ namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.AuditLog;

 /// <summary>
 /// End-to-end integration test for the Audit Log (#23) site→central push path
-/// introduced by the "real ClusterClient-based site audit push client" follow-up.
+/// introduced by the "real site audit push client" follow-up.
 /// </summary>
 /// <remarks>
 /// <para>
 /// Exercises the full production chain in one actor system: the real
 /// <see cref="SqliteAuditWriter"/> site SQLite hot-path, the real
 /// <see cref="SiteAuditTelemetryActor"/> drain loop, the real
-/// <see cref="ClusterClientSiteAuditClient"/>, the real
+/// <see cref="SiteCommunicationAuditClient"/>, the real
 /// <see cref="SiteCommunicationActor"/> forward, the real
 /// <see cref="CentralCommunicationActor"/> routing, and the real
 /// <c>AuditLogIngestActor</c> ingest — only the cross-cluster gRPC transport itself is
@@ -180,7 +180,7 @@ public class SiteAuditPushFlowTests : TestKit
            new BridgeCentralTransport(centralCommActor))));

        // The production site audit push client — the unit under integration.
-        var auditClient = new ClusterClientSiteAuditClient(
+        var auditClient = new SiteCommunicationAuditClient(
            siteCommActor, TimeSpan.FromSeconds(5));

        // Real SiteAuditTelemetryActor drains the writer's Pending queue and
@@ -0,0 +1,204 @@
+using Akka.Actor;
+using Akka.Cluster;
+using Akka.Configuration;
+using Microsoft.Extensions.Logging.Abstractions;
+using Microsoft.Extensions.Options;
+using ZB.MOM.WW.ScadaBridge.ClusterInfrastructure;
+using ZB.MOM.WW.ScadaBridge.Host;
+using ZB.MOM.WW.ScadaBridge.Host.Actors;
+
+namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster;
+
+/// <summary>
+/// Real-ActorSystem tests for the simultaneous-cold-start split-brain guard (Gitea #33):
+/// <see cref="ClusterBootstrapCoordinator"/> + <see cref="ClusterBootstrapGuard"/>, driven exactly as
+/// production drives them — the node's ActorSystem is built from the production <c>BuildHocon</c> output
+/// with the guard enabled (which emits an EMPTY seed list, so Akka does not auto-join), then the real
+/// coordinator issues the reachability-gated <c>JoinSeedNodes</c>. This subsystem has a history of bugs
+/// invisible to pure unit tests, so the load-bearing correctness properties are asserted against running
+/// nodes, mirroring <see cref="SelfFirstSeedBootstrapTests"/>.
+/// </summary>
+/// <remarks>
+/// Ephemeral loopback ports share host <c>127.0.0.1</c>, so the port breaks the guard's ordinal
+/// <c>host:port</c> tie — <c>Min(port)</c> is the founder, <c>Max(port)</c> is the higher (probing) node.
+/// </remarks>
+public sealed class ClusterBootstrapCoordinatorTests : IAsyncLifetime
+{
+    private readonly List<ActorSystem> _systems = new();
+    private readonly List<ClusterBootstrapCoordinator> _coordinators = new();
+
+    /// <summary>Starts a node with the bootstrap guard ENABLED, wired exactly as the composition roots
+    /// do: the ActorSystem is built with an empty HOCON seed list (guard on) and the real coordinator
+    /// issues the reachability-gated join. Seeds are listed self-first (as every shipped config is); the
+    /// guard, not the order, decides founder vs. joiner.</summary>
+    private async Task<ActorSystem> StartGuardedNodeAsync(int selfPort, int peerPort, int probeSeconds = 4)
+    {
+        var self = $"akka.tcp://scadabridge@127.0.0.1:{selfPort}";
+        var peer = $"akka.tcp://scadabridge@127.0.0.1:{peerPort}";
+        var nodeOptions = new NodeOptions { Role = "Central", NodeHostname = "127.0.0.1", RemotingPort = selfPort };
+        var clusterOptions = new ClusterOptions
+        {
+            SeedNodes = new List<string> { self, peer },
+            BootstrapGuard = new ClusterBootstrapGuardOptions
+            {
+                Enabled = true,
+                PartnerProbeSeconds = probeSeconds,
+                PartnerProbeIntervalMs = 250,
+                ProbeConnectTimeoutMs = 500,
+            },
+            StableAfter = TimeSpan.FromSeconds(3),
+            HeartbeatInterval = TimeSpan.FromMilliseconds(500),
+            FailureDetectionThreshold = TimeSpan.FromSeconds(2),
+            MinNrOfMembers = 1,
+        };
+
+        var hocon = AkkaHostedService.BuildHocon(
+            nodeOptions, clusterOptions, new[] { "Central" },
+            TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(3));
+        var system = ActorSystem.Create("scadabridge", ConfigurationFactory.ParseString(hocon));
+        _systems.Add(system);
+
+        var coordinator = new ClusterBootstrapCoordinator(
+            () => system, Options.Create(clusterOptions), Options.Create(nodeOptions),
+            NullLogger<ClusterBootstrapCoordinator>.Instance);
+        _coordinators.Add(coordinator);
+        await coordinator.StartAsync(CancellationToken.None);
+        return system;
+    }
+
+    private static async Task<bool> WaitForUpAsync(ActorSystem system, int expected, TimeSpan timeout)
+    {
+        var cluster = Akka.Cluster.Cluster.Get(system);
+        var deadline = DateTime.UtcNow + timeout;
+        while (DateTime.UtcNow < deadline)
+        {
+            if (cluster.State.Members.Count(m => m.Status == MemberStatus.Up) >= expected) return true;
+            await Task.Delay(200);
+        }
+        return cluster.State.Members.Count(m => m.Status == MemberStatus.Up) >= expected;
+    }
+
+    /// <summary>
+    /// The FOUNDER (lower address) with the guard on forms a cluster alone when its partner is dead —
+    /// exactly as un-guarded self-first does. The guard must not regress the preferred founder.
+    /// </summary>
+    [Fact]
+    public async Task Founder_forms_alone_when_partner_is_dead()
+    {
+        var low = TwoNodeClusterFixture.GetFreeTcpPort();
+        var high = TwoNodeClusterFixture.GetFreeTcpPort();
+        var founderPort = Math.Min(low, high);
+        var deadPeerPort = Math.Max(low, high); // nothing listening
+
+        var system = await StartGuardedNodeAsync(founderPort, deadPeerPort);
+
+        Assert.True(await WaitForUpAsync(system, 1, TimeSpan.FromSeconds(30)),
+            "the founder (lower address) must self-form immediately when its partner is down");
+    }
+
+    /// <summary>
+    /// THE LOAD-BEARING CASE. The HIGHER node with the guard on must still cold-start ALONE when its
+    /// partner is genuinely dead: it probes the (dead) founder, times out, and falls back to self-first.
+    /// If the guard's peer-first logic were wrong, this node would hang in JoinSeedNodeProcess forever —
+    /// the very regression a naive "always let the lower node found" rule would introduce.
+    /// </summary>
+    [Fact]
+    public async Task Higher_node_forms_alone_when_partner_is_dead_after_probing()
+    {
+        var low = TwoNodeClusterFixture.GetFreeTcpPort();
+        var high = TwoNodeClusterFixture.GetFreeTcpPort();
+        var higherPort = Math.Max(low, high);
+        var deadFounderPort = Math.Min(low, high); // the founder is dead
+
+        var system = await StartGuardedNodeAsync(higherPort, deadFounderPort, probeSeconds: 3);
+
+        // Must come Up AFTER the probe window (~3s) expires and it falls back to self-first.
+        Assert.True(await WaitForUpAsync(system, 1, TimeSpan.FromSeconds(30)),
+            "the higher node must fall back to self-first and form alone once the dead partner never answers the probe");
+    }
+
+    /// <summary>
+    /// FALSIFIABILITY CONTROL for the test above: the higher node does NOT form alone during the probe
+    /// window — it is genuinely waiting/probing, not self-forming immediately (which would mean the
+    /// guard is inert). If this ever starts seeing a member before the window, the probe is not gating.
+    /// </summary>
+    [Fact]
+    public async Task Higher_node_does_not_form_before_the_probe_window_expires()
+    {
+        var low = TwoNodeClusterFixture.GetFreeTcpPort();
+        var high = TwoNodeClusterFixture.GetFreeTcpPort();
+        var higherPort = Math.Max(low, high);
+        var deadFounderPort = Math.Min(low, high);
+
+        var system = await StartGuardedNodeAsync(higherPort, deadFounderPort, probeSeconds: 10);
+
+        // Well within the 10s probe window: still no cluster, because it is probing the dead founder.
+        Assert.False(await WaitForUpAsync(system, 1, TimeSpan.FromSeconds(3)),
+            "the higher node must probe for the founder, not self-form immediately");
+    }
+
+    /// <summary>
+    /// The higher node JOINS a live founder rather than forming a second cluster: start the founder,
+    /// then the higher node — its probe finds the founder reachable, it commits peer-first, and the
+    /// pair converges to ONE cluster of two.
+    /// </summary>
+    [Fact]
+    public async Task Higher_node_joins_a_live_founder()
+    {
+        var low = TwoNodeClusterFixture.GetFreeTcpPort();
+        var high = TwoNodeClusterFixture.GetFreeTcpPort();
+        var founderPort = Math.Min(low, high);
+        var higherPort = Math.Max(low, high);
+
+        var founder = await StartGuardedNodeAsync(founderPort, higherPort);
+        Assert.True(await WaitForUpAsync(founder, 1, TimeSpan.FromSeconds(30)),
+            "the founder must be Up before the higher node probes it");
+
+        var higher = await StartGuardedNodeAsync(higherPort, founderPort);
+
+        Assert.True(await WaitForUpAsync(higher, 2, TimeSpan.FromSeconds(60)),
+            "the higher node must join the founder, forming one cluster of two");
+        Assert.True(await WaitForUpAsync(founder, 2, TimeSpan.FromSeconds(60)),
+            "the founder must see the higher node as a member of ITS cluster");
+    }
+
+    /// <summary>
+    /// THE HEADLINE #33 CASE. Both nodes cold-start truly simultaneously with the guard on. Without the
+    /// guard each would run FirstSeedNodeProcess, race, and form its own 1-node cluster (split brain).
+    /// With it, the founder forms and the higher node probes-then-joins, so the pair converges to
+    /// EXACTLY ONE cluster of two — no two oldest, no dual singletons.
+    /// </summary>
+    [Fact]
+    public async Task Both_nodes_cold_starting_together_form_exactly_one_cluster()
+    {
+        var low = TwoNodeClusterFixture.GetFreeTcpPort();
+        var high = TwoNodeClusterFixture.GetFreeTcpPort();
+        var founderPort = Math.Min(low, high);
+        var higherPort = Math.Max(low, high);
+
+        // Start both without serialization — as a shared power event does.
+        var founderTask = StartGuardedNodeAsync(founderPort, higherPort);
+        var higherTask = StartGuardedNodeAsync(higherPort, founderPort);
+        var founder = await founderTask;
+        var higher = await higherTask;
+
+        Assert.True(await WaitForUpAsync(founder, 2, TimeSpan.FromSeconds(60)),
+            "the pair must converge to one 2-member cluster, not two 1-member clusters");
+        Assert.True(await WaitForUpAsync(higher, 2, TimeSpan.FromSeconds(60)),
+            "the higher node must be in the SAME cluster as the founder");
+    }
+
+    public Task InitializeAsync() => Task.CompletedTask;
+
+    public async Task DisposeAsync()
+    {
+        foreach (var c in _coordinators)
+        {
+            try { await c.StopAsync(CancellationToken.None); } catch { /* teardown */ }
+        }
+        foreach (var s in _systems)
+        {
+            try { await s.Terminate().WaitAsync(TimeSpan.FromSeconds(10)); } catch { /* teardown */ }
+        }
+    }
+}
@@ -47,7 +47,7 @@ public class DispatchCoverageTests : TestKit, IDisposable
    {
        // The two-step "ResolveRoles + command" flow is retired — the HTTP endpoint
        // resolves roles itself and sends a single envelope. Leaving a handler would
-        // expose role-mapping data to a raw ClusterClient sender with no role gate.
+        // expose role-mapping data to a raw in-cluster sender with no role gate.
        // See ManagementActor.DispatchCommand (the comment above the default arm, ~line 472).
        typeof(ResolveRolesCommand),

@@ -1089,7 +1089,7 @@ public class ManagementActorTests : TestKit, IDisposable
    //
    // The two-step ResolveRoles + command flow is retired: the HTTP endpoint does
    // LDAP auth and role resolution itself. The actor must no longer dispatch
-    // ResolveRolesCommand — a stray ClusterClient sender hitting it gets a uniform
+    // ResolveRolesCommand — a stray in-cluster sender hitting it gets a uniform
    // ManagementError rather than an unauthenticated role-mapping enumeration.
    // ========================================================================

@@ -164,7 +164,7 @@ public class SiteCallRelayTests : TestKit
    public void RetrySiteCall_SiteNeverReplies_ReportsSiteUnreachable()
    {
        // A central comm probe that silently drops the relay — models an offline
-        // site / no ClusterClient route: the Ask times out.
+        // site / no route to the site: the Ask times out.
        var central = CreateTestProbe();
        var actor = CreateActor(central.Ref);
Author	SHA1	Message	Date
Joseph Doherty	663d03e89c	Merge branch 'fix/cached-telemetry-drain-hot-loop'	2026-07-27 15:50:42 -04:00
Joseph Doherty	63c16d6912	refactor(comm): retire ClusterClient naming after the gRPC cutover Phase 4 of the ClusterClient -> gRPC migration deleted the Akka transports but left the naming behind: `ClusterClientSiteAuditClient` was transport- agnostic and worked unchanged, so it survived the deletion under a name that now describes a transport the repo no longer has. Same for a scatter of doc-comments still framing gRPC as "the new transport" beside an Akka one that is gone. Renames it to `SiteCommunicationAuditClient` (and its test file) and rewrites the stale comments to describe the single transport that exists. Also tightens CLAUDE.md: drops the self-describing directory listing and the 27-component enumeration in favour of the non-obvious parts only. Behaviour-neutral: names and prose only. Recorded as the Phase 4 follow-up in docs/plans/2026-07-22-clusterclient-to-grpc-plan.md.	2026-07-27 15:40:01 -04:00
Joseph Doherty	b3dc17a5fe	Merge feat/site-node-health: serve health on site nodes + shared active-node check (Health 0.3.0)	2026-07-24 13:35:27 -04:00
Joseph Doherty	1c9159a7ce	fix(tests): qualify Health page type — Communication.Health namespace shadowed it	2026-07-24 13:35:24 -04:00
Joseph Doherty	5d4853a1f0	refactor(health): adopt the shared active-node check (Health 0.3.0) OldestNodeActiveHealthCheck existed because the shared ActiveNodeHealthCheck selected by cluster leadership (review 01 [High]): leadership is address-ordered and diverges from singleton placement after a restart, and during a partition both sides compute themselves leader so Traefik served both. Health 0.3.0 makes the shared check age-based — it is now this host's own rule, promoted — so the private copy is deleted and central registers the shared type. ActiveNodeEvaluator, the "THE single definition of active node" this repo already maintained, now delegates to the shared ClusterActiveNode rather than re-implementing it. That keeps the delivery gate, the heartbeat IsActive stamp, the inbound-API gate and the /health/active tier on one rule, and it means the rule is shared with OtOpcUa, which had independently written a third copy. Communication takes a ZB.MOM.WW.Health.Akka reference for it; the layering trade-off is recorded at the PackageReference. SitePairActiveNodeHealthCheck is deliberately KEPT. It is not a duplicate of the rule — it is a thin adapter over the site's own IClusterNodeProvider, which scopes to site-{SiteId} and is itself now backed by ClusterActiveNode. Registering the shared check directly would have to re-derive the site role and would lose the property that the tier and singleton placement come from the same provider. Central stays unscoped: every central member competes for one active slot. No behaviour change on any node — same rule, one implementation instead of two. Verified: Host.Tests 439/439, Communication ActiveNode 2/2.	2026-07-24 13:21:58 -04:00
Joseph Doherty	c1216044a4	docs(cluster): document the #33 bootstrap guard; disabled BootstrapGuard block in Host appsettings Component-ClusterInfrastructure.md gains a 'Simultaneous cold start — the bootstrap guard' subsection (dark switch, founder/probe semantics, config table, accepted trade) plus a forward reference from Dual-Node Recovery. Host appsettings.Central/Site.json carry a disabled BootstrapGuard block with a _bootstrapGuard note for operator discoverability. CLAUDE.md Cluster & Failover note added.	2026-07-24 08:57:44 -04:00
Joseph Doherty	248676ed16	feat(cluster): simultaneous-cold-start split-brain guard (#33 ) Port OtOpcUa's bootstrap guard (lmxopcua d1dac87f) to ScadaBridge's BuildHocon bootstrap. Both pair nodes are self-first seeds, so a true simultaneous cold start (shared site power event) races FirstSeedNodeProcess on both and forms two 1-node clusters that never merge. Opt-in dark switch ScadaBridge:Cluster:BootstrapGuard:Enabled (default off, guard-off behavior byte-identical). When on: BuildHocon emits an empty seed list so Akka does not auto-join, and ClusterBootstrapCoordinator (IHostedService, registered in both the Central and Site composition roots) picks the join order from ClusterBootstrapGuard's pure decision core — the lower canonical host:port is the founder (self-first, forms immediately); the higher node TCP-probes the founder up to PartnerProbeSeconds and joins peer-first if reachable, else self-first (cold-start-alone preserved). Decides before a single JoinSeedNodes, never re-forms mid-handshake. Review notes carried over: case-insensitive founder tie-break; fail-fast validation of probe timings when enabled; higher-node-cold-start-alone covered by a real-ActorSystem test. 21 unit + 5 real-cluster tests (incl. the headline both-cold-start-together-form-one-cluster).	2026-07-24 08:55:48 -04:00
Joseph Doherty	47850c0f53	feat(health): serve MapZbHealth on site nodes Site nodes served no health surface at all — gRPC on the HTTP/2-only listener and /metrics on the HTTP/1.1 one — so nothing outside the cluster could ask a site node whether it was ready or which half of the pair was active. The family overview dashboard probes every instance the same way, and this is the one gap. Three checks, registered in SiteServiceRegistration.Configure (not Program.cs, so the composition-root tests that build this graph actually cover them) and mapped by app.MapZbHealth() on the site's HTTP/1.1 listener (default :8084) alongside /metrics: akka-cluster [Ready] the shared AkkaClusterHealthCheck. Also carries the cluster-view data (leader/memberCount/...) the dashboard reads, free with ZB.MOM.WW.Health 0.2.0. localdb [Ready] NEW SiteLocalDbHealthCheck — SELECT 1 through the registered ILocalDb. Site has no EF context; central's DatabaseHealthCheck<ScadaBridgeDbContext> is central-only. Replication state rides along as data ENRICHMENT only: replication is default-OFF, so failing on it would mark every correctly-configured node unready. active-node [Active] NEW SitePairActiveNodeHealthCheck, delegating to IClusterNodeProvider.SelfIsPrimary. Deliberately NOT central's OldestNodeActiveHealthCheck: that one calls SelfIsOldest(cluster) with no role argument and would compute "oldest" across the wrong member set on a mesh carrying more than one site. Anonymous, as central's are: the site pipeline runs no authentication middleware and has no FallbackPolicy, so nothing extra was needed. Also bumps ZB.MOM.WW.Health* 0.1.0 -> 0.2.0 (central gains data.leader for free). Tests: SiteHealthCheckTests builds the REAL site container and activates every registration through its factory (exact-set names, one tier tag each, resolved types, central-only checks absent behind a positive control) plus behaviour for both new checks. SiteHealthEndpointTests boots the real site Program over WebApplicationFactory and proves the endpoints are MAPPED — without it, deleting MapZbHealth would leave every registration test green while site nodes 404'd. Prereq for scadaproj docs/plans/2026-07-22-overview-dashboard-impl-plan.md Phase 1.	2026-07-24 05:54:30 -04:00
Joseph Doherty	bb138d5254	docs: add env_variables.md — full environment-variable inventory Deep-scan catalog of every environment variable ScadaBridge reads (Host, CLI, DelmiaNotifier, EF design-time tooling, test/CI harness) plus supporting infra containers. Each entry carries scope (Runtime/Design-time/Test/Infra), whether a shipped appsettings key already exists for it, consumer, purpose, potential values, and required-ness. Compiled from GetEnvironmentVariable call sites, the Host AddEnvironmentVariables() source, docker/docker-env2 compose, install.ps1, and all appsettings*.json.	2026-07-24 05:02:24 -04:00
Joseph Doherty	9a12826172	docs(plans): OtOpcUa v3 native-alarm B/C live-gate PASS (#14 ) — no code needed; item C premise obsolete (Gitea #17 )	2026-07-24 04:33:20 -04:00
Joseph Doherty	86d129de6c	fix(dcl): harden endpoint host-rewrite — IPv6 brackets + portless URLs Code review of `a1abbff7` found two edge cases: a discovery/advertised URL with no explicit port emitted a malformed ':-1' (opc.tcp has no Uri default port), and an IPv6 literal host could lose/double its brackets. Omit the port when neither URL carries one; bracket an IPv6 host only when missing. +4 tests (12 total).	2026-07-23 16:58:44 -04:00
Joseph Doherty	dbec3ee263	docs(plans): OtOpcUa v3 raw-path live-gate PASS (#14 ) — fix validated live, B/C deferred	2026-07-23 16:54:48 -04:00
Joseph Doherty	a1abbff760	fix(dcl): rewrite advertised OPC UA endpoint host to the reachable one An OPC UA server advertises its base address from its own config, not the route the client took — commonly a 0.0.0.0 wildcard bind or an internal container/NAT hostname. The OPC Foundation session dials EndpointDescription.EndpointUrl verbatim, so a 0.0.0.0 advertisement resolved to the client's own loopback and the connect failed. RealOpcUaClient now swaps the advertised authority for the reachable host/port (ConnectAsync + VerifyEndpointAsync), preserving scheme+path; no-op when already reachable. Surfaced live-gating OtOpcUa v3 (#14).	2026-07-23 16:43:45 -04:00
Joseph Doherty	62cddcfa56	fix(audit): let an unresolvable cached row leave the drain queue A cached-telemetry audit row whose OperationTracking snapshot could not be resolved was skipped and left Pending. Nothing ever removed it, so the next drain re-read it, failed identically, and logged again — forever. The severity was worse than the original write-up said. The queue is read oldest-first with a fixed BatchSize (default 256), so once a batch's worth of permanently-unresolvable rows collected at the head, every drain re-read exactly those rows and NEVER REACHED the newer rows behind them. That is a permanent stall of the cached-telemetry path, not the "log flood, no data loss" the issue was first filed as. Measured on the rig at ~2,800 warnings/minute, surviving both a process restart and a container restart. Fix: after a grace period (CachedTrackingGraceSeconds, default 300 s) the operational half is abandoned and the row marked Forwarded. A row with no CorrelationId is abandoned immediately — it can never resolve. Marking Forwarded does not drop audit data. That state means "no longer owed by the drain, still eligible for reconciliation", which is exactly this situation: ReadPendingSinceAsync covers Forwarded as well as Pending and central dedups on EventId, so the reconciliation pull still delivers the audit half. What is lost is the operational (SiteCalls) half, which is unrecoverable anyway once the tracking row is gone. Three deliberate boundaries: - Inside the grace window the row is still retried — a missing snapshot is normally a brief write race, and abandoning on the first failed lookup would discard the operational half of every cached call that lost that race. - A tracking-store THROW never abandons, however old the row: a throw is a store fault (locked, corrupt, mid-restore), not a verdict about the row. - Logging is per drain pass, not per row. Deferred rows dropped to Debug — being inside the grace window is normal operation, not a warning. CachedDrain_OrphanRow_NoTrackingSnapshot_IsSkipped_DoesNotCrash was RETARGETED rather than deleted: it pinned the defect ("skipped and stays Pending"), so its orphan assertion is inverted and the half that still holds is kept verbatim. Three tests added, including the starvation regression. Both "must NOT abandon" tests carry a positive control on the read count, so they cannot pass by virtue of a drain that never ran. Verified non-vacuous: with abandonment disabled the two abandonment tests go red and the two guards stay green. Build 0 warnings; AuditLog 358, Host 330, SiteRuntime 512, StoreAndForward 130, Communication 312, Commons 684, Integration 94 — all pass. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts	2026-07-20 06:29:22 -04:00