Merge feat/site-node-health: serve health on site nodes + shared active-node check (Health 0.3.0)

fix(tests): qualify Health page type — Communication.Health namespace shadowed it
refactor(health): adopt the shared active-node check (Health 0.3.0)
2026-07-24 13:35:27 -04:00 · 2026-07-24 13:35:24 -04:00 · 2026-07-24 13:21:58 -04:00 · 2026-07-24 08:57:44 -04:00 · 2026-07-24 08:55:48 -04:00 · 2026-07-24 05:54:30 -04:00
31 changed files with 2016 additions and 81 deletions
@@ -220,6 +220,7 @@ Other peers in the `scadaproj` family (see `scadaproj/CLAUDE.md` for details): `
 - Automatic dual-node recovery from persistent storage.
 - **Active/standby is decided by `ActiveNodeEvaluator.SelfIsOldestUp`, never by cluster leadership** — see the Architecture note above. `/health/active` is **central-only** (site nodes map no `/health/*` at all) and backs both Traefik's active-node routing and `IActiveNodeGate`, so the proxy and the Inbound API always agree on which node is active. Central never needs to know which *site* node is active: ClusterClient contact rotation reaches either receptionist and the site-internal `ClusterSingletonProxy` lands the work on the active node for free. The **exception is gRPC**, which picks `GrpcNodeAAddress`/`GrpcNodeBAddress` explicitly and flips on error.
 - **Seed-node ordering: every node lists ITSELF first (decision 2026-07-22) — the boot-alone gap is CLOSED.** Only `seed-nodes[0]` may self-join to form a new cluster (Akka runs `FirstSeedNodeProcess` for it, `JoinSeedNodeProcess` — which can never form one — for everyone else). All 14 shipped node appsettings now lead with the node's own address, so any node can cold-start alone and become operational unattended (~5s, `seed-node-timeout`); `StartupValidator` fails the boot if the ordering is broken (compares host AND port; Akka does no DNS canonicalisation). Two nodes cold-starting together while mutually reachable converge on ONE cluster via the `InitJoin` handshake — they split only under a genuine boot-time partition, the same class auto-down accepts. **An external self-form timer (`Cluster.Join(SelfAddress)` after a window) was implemented and REJECTED:** it sits outside the join handshake, so on a routine standby restart — where the peer is alive but the join is stalled behind removal of the node's own stale incarnation — it fires mid-join and permanently splits the pair (measured: still split after 90s). Regression tests: `SelfFirstSeedBootstrapTests`. The keep-oldest active-crash total outage was separately closed by the auto-down decision. See `docs/requirements/Component-ClusterInfrastructure.md` → Seed Node Ordering.
 - **Simultaneous-cold-start split-brain guard (opt-in, Gitea #33) — the residual self-first cost, closed.** Self-first-on-both means a *truly simultaneous* cold start (shared power/hypervisor event) races `FirstSeedNodeProcess` on BOTH nodes → two 1-node clusters that never merge (in-process loopback tests converge and hide it; real parallel VM starts don't — OtOpcUa reproduced it live). Ported from OtOpcUa (`lmxopcua` `d1dac87f`): a **dark switch `ScadaBridge:Cluster:BootstrapGuard:Enabled` (default OFF, guard-off byte-identical)**. On: `BuildHocon` emits an EMPTY seed list (Akka does not auto-join) and `ClusterBootstrapCoordinator` (`IHostedService`, registered in BOTH the Central and Site composition roots) issues ONE reachability-gated `JoinSeedNodes` — the pure `ClusterBootstrapGuard` core makes the lower canonical `host:port` the **founder** (self-first, forms immediately), the higher node TCP-probes the founder up to `PartnerProbeSeconds` (25s) and joins **peer-first** if reachable else **self-first** (cold-start-alone preserved). Decided BEFORE a single join, **never re-forms mid-handshake** (that was the rejected self-form-timer's flaw); case-insensitive founder tie-break; probe timings validated `>0` when enabled. Residual accepted trade: founder dies in the probe→join window → higher node hangs unjoined, coordinator WARNs, a restart recovers it. Tests: `ClusterBootstrapGuardTests` (pure) + `ClusterBootstrapCoordinatorTests` (real-ActorSystem, incl. both-cold-start-together-form-one-cluster). See `docs/requirements/Component-ClusterInfrastructure.md` → Simultaneous cold start.
 ### UI & Monitoring
 - Central UI: Blazor Server (ASP.NET Core + SignalR) with Bootstrap CSS. No third-party component frameworks (no Blazorise, MudBlazor, Radzen, etc.). Build custom Blazor components for tables, grids, forms, etc.
@@ -90,9 +90,9 @@
      to mark tests as Skipped (not silently Passed) when MSSQL is unreachable.
    -->
    <PackageVersion Include="Xunit.SkippableFact" Version="1.5.61" />
-    <PackageVersion Include="ZB.MOM.WW.Health" Version="0.1.0" />
+    <PackageVersion Include="ZB.MOM.WW.Health" Version="0.3.0" />
-    <PackageVersion Include="ZB.MOM.WW.Health.Akka" Version="0.1.0" />
+    <PackageVersion Include="ZB.MOM.WW.Health.Akka" Version="0.3.0" />
-    <PackageVersion Include="ZB.MOM.WW.Health.EntityFrameworkCore" Version="0.1.0" />
+    <PackageVersion Include="ZB.MOM.WW.Health.EntityFrameworkCore" Version="0.3.0" />
    <PackageVersion Include="ZB.MOM.WW.Telemetry" Version="0.1.0" />
    <PackageVersion Include="ZB.MOM.WW.Telemetry.Serilog" Version="0.1.0" />
    <PackageVersion Include="ZB.MOM.WW.MxGateway.Client" Version="0.1.1" />
@@ -0,0 +1,64 @@
 # OtOpcUa v3 native-alarm B/C live-gate (Gitea #14) — PASS
 **Date:** 2026-07-23 · **Scope:** items **B** (native-alarm dedup) + **C** (UNS-bound alarm routing) of #14,
 against a UNS structure + scripted Part-9 alarm authored on the live `otopcua-dev` v3 cluster (see
 `2026-07-23-otopcua-v3-raw-path-live-gate.md` for the connection + `SiteAOnly` substrate). Instrument:
 ScadaBridge itself (its DCL A&C client is the thing under test).
 ## Headline
 **Both B and C PASS, and item C's original premise turned out to be already-solved code.** No ScadaBridge
 code change is needed for B/C. The scope-doc worry — "a UNS-bound source won't `StartsWith`-match a
 condition whose `SourceName` is the RawPath/ScriptedAlarmId" — is **obsolete**: ScadaBridge routes native
 alarms by the **subscribed node reference**, not the event `SourceName` (deliberate fix, Gitea #17). The
 only remaining #14 work is the item-A data re-author (greenfield, operational).
 ## Substrate (on OtOpcUa, SITE-A cluster)
 - UNS: `area-1/line-1/pump01` → signal `SiteAOnly` (equipment folder node `ns=3;s=EQ-54f7711e160d`,
  which has **`EventNotifier=1`**; signal `ns=3;s=area-1/line-1/pump01/SiteAOnly` mirrors the raw value).
 - Scripted Part-9 condition `ns=3;s=SA-siteahigh` (a `HasComponent` child of the equipment folder),
  `AlarmConditionType`, severity 700, predicate driven by `SiteAOnly` (set always-active for the gate).
 - `Server` (i=2253) `EventNotifier=1` (the connection-wide aggregate).
 ## What ScadaBridge did (bindings on template 2148, instance 98, connection `OtOpcUa-v3-raw`)
 Two `TemplateNativeAlarmSource` bindings, deliberately overlapping to stress dedup:
 - `ServerAlarms` → `--source-ref i=2253` (Server aggregate)
 - `UnsEquipAlarms` → `--source-ref nsu=https://zb.com/otopcua/uns;s=EQ-54f7711e160d` (UNS equipment node)
 ## Findings / checks
 | # | Check | Result |
 |---|-------|--------|
 | C1 | A **UNS-node-scoped** binding receives + routes the native alarm | **PASS** — `UnsEquipAlarms` mirrored `SA-siteahigh.SiteA High` active=True, sev 700, `kind=NativeOpcUa` |
 | C2 | The condition's `SourceName` = **ScriptedAlarmId** (`SA-siteahigh`), NOT RawPath | **Confirmed** (captured live) — and it does **not** break routing (routing keys off the subscribed node, Gitea #17), only feeds the alarm's identity (`alarmName = SA-siteahigh.SiteA High`) |
 | C3 | Item C's original "SourceName mismatch drops UNS-bound transitions" premise | **Obsolete** — `OpcUaAlarmMapper.BuildIdentity` stamps `SourceObjectReference` = the subscription ref; `DataConnectionActor` routes on that, not `SourceName` |
 | B1 | Fanned condition (Server aggregate + equipment-folder notifier) mirrored **once**, not duplicated | **PASS** — active condition appears exactly once (on the more-specific equipment feed); the Server binding stays an inactive placeholder. Stable across repeated snapshots |
 | B2 | Mechanism | ScadaBridge opens **one** OPC UA `Subscription` per connection with one `MonitoredItem` per source-ref; `ConditionRefresh` delivers each retained condition **once per subscription**, to the most-specific monitored notifier (the equipment folder) — so overlapping bindings do not double-mirror the fan-out |
 | B3 | Residual double-mirror risk | Latent, **not triggered** here: it needs two bindings whose `SourceReference` strings are literal prefixes of one another AND a delivery matching both (`DataConnectionActor` has no cross-binding, per-ConditionId dedup — `InstanceActor._latestAlarmEvents` is last-writer-wins by `alarmName`). Disjoint node refs (as here) don't collide |
 ## Captured real payload (what §5 phase 3 wanted)
 `kind=NativeOpcUa`, `alarmName='SA-siteahigh.SiteA High'` (`SourceName`=`SA-siteahigh` = ScriptedAlarmId,
 `.` + `ConditionName`=`SiteA High`), `alarmTypeName='AlarmConditionType'`, `severity=700`,
 `condition.active=True`. A&C state is delivered via events/`ConditionRefresh` (ScadaBridge triggers a
 refresh on every subscribe), not a plain attribute read.
 ## Bottom line for #14
 - **A** — raw/UNS binding re-author: validated end-to-end (raw-path gate proved `nsu=` bind→read; this gate
  proved UNS bind→alarm). Remaining work is data re-authoring (greenfield), **no code**.
 - **B** — dedup: **PASS**, no double-mirror across the v3 fan-out (one subscription + `ConditionRefresh`).
 - **C** — UNS-bound routing: **PASS**; the original concern was already solved by Gitea #17.
 - **D/E/F** — shipped in phase 2 (`2026-07-23-otopcua-v3-nsu-hardening-and-browse-ux.md`).
 **#14 needs no further ScadaBridge code for B/C.** The only shipped code from this whole effort was the
 phase-2 `nsu=` hardening + the `RealOpcUaClient` host-rewrite fix (surfaced by the raw gate). #14 can close
 once item A's re-author is done in the target environment(s).
 ## Rig artifacts (removable on next `docker/deploy.sh`)
 OtOpcUa ConfigDb: `UnsArea SITEA-area1` / `UnsLine SITEA-line1` / `Equipment EQ-54f7711e160d` /
 `UnsTagReference UTR-siteapump01-siteaonly` / `Script SC-siteahigh` / `ScriptedAlarm SA-siteahigh`.
 ScadaBridge: template 2148 native-alarm-sources `ServerAlarms` + `UnsEquipAlarms`; instance 98; connection 3044.
@@ -0,0 +1,79 @@
 # OtOpcUa v3 raw-path live-gate (Gitea #14) — PASS
 **Date:** 2026-07-23 · **Scope:** the raw-subtree slice of #14 (decision: "raw gate now, flag B/C").
 **Rig:** ScadaBridge docker rig (`scadabridge:latest`, rebuilt with the host-rewrite fix) against the
 **live `otopcua-dev` v3 cluster** (`otopcua-host:dev`, built 2026-07-23, branch `feat/mesh-phase5`,
 past the v3 merge `ec6598ce`). Related: scope doc `2026-07-23-otopcua-v3-dual-namespace-cutover-scope.md`,
 phase-2 plan `2026-07-23-otopcua-v3-nsu-hardening-and-browse-ux.md`.
 ## Headline
 ScadaBridge is confirmed **structurally v3-safe against a real dual-namespace server** — durable
 `nsu=` references are emitted by browse, stored on bindings, and resolved live against the server's
 `NamespaceArray` to Good-quality reads. The gate also **surfaced and fixed a real DCL robustness
 bug** (couldn't dial a server advertising `0.0.0.0`). Items **B/C remain deferred** — the dev
 cluster has no UNS nodes and no native alarms to gate them against.
 ## Environment findings
 - v3 cluster publishes both namespaces: `ns=2 = https://zb.com/otopcua/raw`,
  `ns=3 = https://zb.com/otopcua/uns`. **Raw lands at ns=2 — the exact index collision scope §4
  item A predicted** (v2's sole namespace and v3 raw both at ns=2).
 - Raw tree is **sparsely populated**: one materialized live Variable,
  `nsu=https://zb.com/otopcua/raw;s=sa-modbus2/plc/SiteAOnly` (Int16, AccessLevel=1 → **read-only**).
  Other drivers (calc1, opcua1, pymodbus, gate-modbus, Area1) expose folders but no live leaf tags.
 - **UNS (ns=3) is empty** — `ns=3;s=OtOpcUa` → `BadNodeIdUnknown`; no Equipment authored.
 - **No native alarms** — zero `HasCondition`/`HasEventSource` refs anywhere in the raw tree.
 ## The fix this gate produced
 `RealOpcUaClient` took the server-advertised `EndpointDescription.EndpointUrl` verbatim for the
 session channel. OtOpcUa advertises `opc.tcp://0.0.0.0:4840/OtOpcUa` (its
 `OpcUaApplicationHostOptions.PublicHostname` defaults to `0.0.0.0`), and dialling `0.0.0.0` from
 inside the client container hits its own loopback → connect fails. Added
 `RealOpcUaClient.RewriteEndpointHostForReachability` (swaps the advertised authority for the
 reachable discovery host/port, preserves scheme+path, no-op when already reachable), called in both
 `ConnectAsync` and `VerifyEndpointAsync`. Mirrors `CoreClientUtils.SelectEndpoint` and OtOpcUa's own
 client (`IEndpointDiscovery`). Commit `a1abbff7` on `fix/opcua-discovery-host-rewrite`; 9 unit tests
 (`RealOpcUaClientEndpointRewriteTests`).
 ## Wiring (how the gate was run)
 1. Rebuilt `scadabridge:latest` with the fix (`bash docker/deploy.sh`).
 2. `docker network connect otopcua-dev_default scadabridge-site-a-a` (and `-b`) — bridged the two
   docker networks so site-a can resolve `otopcua-dev-site-a-1-1`.
 3. `data-connection create --site-id 1 --name OtOpcUa-v3-raw --protocol OpcUa` →
   `endpointUrl=opc.tcp://otopcua-dev-site-a-1-1:4840/OtOpcUa`, securityMode none (id **3044**).
 4. `deploy artifacts --site-id 1` — pushed the def to site-a's LocalDb + made it live in the DCL.
 5. Template **2148** (`OtOpcUaV3Probe`) + attribute `SiteA` (Int32,
   `--data-source nsu=https://zb.com/otopcua/raw;s=sa-modbus2/plc/SiteAOnly`); instance **98**
   (`otopcua-v3-probe-1`) on site-a; `set-bindings [["SiteA",3044]]`; `deploy instance`.
 ## Checks
 | # | Check | Result |
 |---|-------|--------|
 | 1 | Connect via `ConnectAsync` (through the rewrite fix) | **PASS** — `data-connection browse` returned the address space |
 | 2 | Connect via `VerifyEndpointAsync` (second rewrite call site) | **PASS** — `verify-endpoint` → `{"success":true}` |
 | 3 | Browse emits durable `nsu=` throughout | **PASS** — root `nsu=…/raw;s=OtOpcUa`, drivers + `…;s=sa-modbus2/plc/SiteAOnly` |
 | 4 | Both namespaces visible / discoverable | **PASS (partial)** — raw browsable; uns registered but empty (nothing to browse) |
 | 5 | Durable `nsu=` binding resolves live | **PASS** — attribute stores `nsu=…`, subscription resolves it against the live `NamespaceArray` |
 | 6 | Read round-trip, Good quality, live | **PASS** — `SiteA` Good, values changing 59689→59806→59822→59834 across snapshots |
 | 7 | Write round-trip | **N/A** — the only materialized tag is read-only (AccessLevel=1); no writable v3 tag to gate |
 ## Deferred (unchanged — need OtOpcUa-side authoring on the dev cluster)
 - **Item A** — re-author legacy `ns=2` bindings to `nsu=` (data; the collision is confirmed real here).
 - **Item B** — native-alarm dedup across raw+uns fan-out: **no native alarms exist** to drive it.
 - **Item C** — UNS-bound → RawPath alarm routing: **no UNS nodes and no alarms exist** to drive it.
 To gate B/C, the OtOpcUa dev cluster needs UNS Equipment authored (populate ns=3, referencing raw
 tags) and at least one alarm-configured tag raising a Part-9 condition that fans to raw + equipment
 notifiers.
 ## Rig artifacts created (non-default rig state)
 - ScadaBridge site-a-a / site-a-b attached to `otopcua-dev_default` (dropped on next `docker/deploy.sh`).
 - Site-a config: DataConnection **3044** `OtOpcUa-v3-raw`, template **2148**, instance **98** (deployed).
  Harmless; remove with `instance delete 98` / `template delete 2148` / `data-connection delete 3044`
  if a clean rig is wanted.
@@ -137,6 +137,27 @@ Behavior, covered by `SelfFirstSeedBootstrapTests` (real in-process clusters bui
 The docker failover drill (`docker/failover-drill.sh`) proves both directions: `standby` mode kills the younger node (active untouched, zero routing blips); `active` mode kills the active/oldest node and asserts the survivor **takes over while the victim is still down**.
 ### Simultaneous cold start — the bootstrap guard (opt-in, Gitea #33)
 Self-first-on-both has a residual cost. When **both** nodes cold-start at the same instant, each is `seed-nodes[0]` for itself, so **each runs `FirstSeedNodeProcess`**, times out waiting for the other, and forms its **own** single-node cluster — a split brain (two oldest members, two singletons, dual-active) that does **not** auto-merge and persists until an operator restarts one side. In-process loopback tests usually converge (the `InitJoin` handshake resolves before either self-join deadline, so row 3 above passes), which is exactly why the split hides until a real deployment powers up both VMs truly in parallel — a shared power event or a hypervisor host reboot. The sister project **OtOpcUa** (same `ZB.MOM.WW.*` Akka topology) reproduced it reliably on its docker rig.
 The **bootstrap guard** eliminates the split without giving up cold-start-alone. It is an **opt-in dark switch** — `ScadaBridge:Cluster:BootstrapGuard:Enabled`, **default off**, so a node with the guard disabled keeps Akka's config-driven self-first auto-join byte-identical. When enabled, the node starts with **no config seed nodes** (`AkkaHostedService.BuildHocon` emits an empty seed list, so Akka does not auto-join) and a coordinator (`ClusterBootstrapCoordinator`, an `IHostedService` registered in both the Central and Site composition roots) picks the join order after a reachability probe:
 - The node with the lexicographically **lower** canonical `host:port` is the **preferred founder**: it joins self-first and forms immediately if no peer answers — no probe, no delay. The tie-break is **case-insensitive** (a hostname-casing mismatch between the two configs must never make both think they are the founder and re-open the split).
 - The **higher** node TCP-probes its partner's Akka port (a node binds its port well before it forms) up to `PartnerProbeSeconds`: **reachable ⇒ peer-first** (join the founder, never race it); **unreachable after the window ⇒ self-first** (the partner is genuinely down, so cold-start-alone is preserved for the higher node too).
 - The order is decided **before** a single `Cluster.JoinSeedNodes`, from an explicit reachability signal. The coordinator **never re-forms mid-handshake** — that was the failure mode of the rejected self-form timer above (a `Join(self)` on a bare timeout is not ignored mid-handshake; it wins and islands a node).
 **Residual trade-off (accepted, operator-visible).** Once the higher node commits peer-first it cannot self-form (`JoinSeedNodeProcess` retries forever). If the founder dies in the small probe→join window, the higher node hangs unjoined; the coordinator logs a WARNING after a bounded grace, and a **restart** recovers it (the guard re-runs, finds the founder down, and forms alone). The guard deliberately does not re-decide mid-handshake.
 | Key | Default | Meaning |
 |---|---|---|
 | `BootstrapGuard:Enabled` | `false` | Dark switch. Only meaningful on a node that is one of its own two pair seeds; inert elsewhere (single-node install, self absent, 3+ seeds). |
 | `BootstrapGuard:PartnerProbeSeconds` | `25` | Higher node's probe window. Must comfortably exceed the founder's process-start-to-Akka-bind time, or a slow-but-alive founder is mistaken for dead and the split re-opens. Validated `> 0` at boot when the guard is on. |
 | `BootstrapGuard:PartnerProbeIntervalMs` | `500` | Interval between probes. Validated `> 0` when enabled. |
 | `BootstrapGuard:ProbeConnectTimeoutMs` | `1000` | Per-probe TCP connect timeout. Validated `> 0` when enabled. |
 Decision core (`ClusterBootstrapGuard`) is a pure, fully unit-tested function; the probe + `JoinSeedNodes` runtime (`ClusterBootstrapCoordinator`) is covered by real-ActorSystem tests including the load-bearing higher-node-cold-start-alone case and the headline both-cold-start-together-form-one-cluster case (`ClusterBootstrapCoordinatorTests`). The alternative is purely operational (stagger the two VMs' service-manager start / start the founder first); the docker rig's compose `depends_on` does that today, but that does not exist on the real co-located VMs — the guard is the production-faithful fix. Ported from OtOpcUa (`lmxopcua` commit `d1dac87f`); implementing it in both products keeps their failure/recovery model identical, since a shared power event hits both pairs at once.
 ### Manual Failover (admin-triggered)
 An Administrator can swap the central pair's roles deliberately — for planned maintenance on the active node, or to move singletons off a node behaving badly without waiting for a crash. The control is the **Trigger failover** button on the central-cluster card of the Health dashboard (`/monitoring/health`).
@@ -191,7 +212,7 @@ These values balance failover speed with stability — fast enough that data col
 If both nodes in a cluster fail simultaneously (e.g., site power outage):
-1. **No manual intervention required.** Since both nodes are configured as seed nodes, whichever node starts first forms a new cluster. The second node joins when it starts.
+1. **No manual intervention required.** Since both nodes are configured as seed nodes, whichever node starts first forms a new cluster (self-first ordering) and the second node joins when it starts. If both power up truly simultaneously, each self-first seed can run `FirstSeedNodeProcess` and form its own 1-node cluster — a split brain; enable the opt-in **bootstrap guard** (Gitea #33, see *Simultaneous cold start* above) on co-located VM pairs that share a power domain to make them converge on one cluster instead.
 2. **State recovery** (each node has its own local copy of all required data):
   - **Site clusters**: The Deployment Manager singleton reads deployed configurations from local SQLite and re-creates the full Instance Actor hierarchy. Store-and-forward buffers are already persisted locally. Alarm states re-evaluate from incoming data values.
   - **Central cluster**: All state is in MS SQL (configuration database). The active node resumes normal operation.
@@ -0,0 +1,169 @@
 # Environment Variables
 A deep-scan inventory of every environment variable ScadaBridge reads — the Host, the CLI, the
 DelmiaNotifier client, EF design-time tooling, and the test/CI harness — plus the supporting infra
 containers. Compiled by scanning `Environment.GetEnvironmentVariable` call sites, the Host's
 `AddEnvironmentVariables()` config source, `docker/` + `docker-env2/` compose files,
 `deploy/wonder-app-vd03/install.ps1`, and every `appsettings*.json`.
 **Scan date:** 2026-07-24 · **Branch:** `main`
 Two columns qualify each variable:
 - **Scope** — where it is read: **Runtime** (the running product — Host / CLI / DelmiaNotifier),
  **Design-time** (`dotnet ef` tooling), **Test** (unit / integration / E2E / perf harness only, never
  the shipped product), or **Infra** (a sidecar container, not ScadaBridge code).
 - **In appsettings?** — whether a shipped `appsettings*.json` already carries the same config key
  (so the env var is an *override* or *secret substitute*), versus being env-only. `${secret:...}`
  means the key is present but its value is a secret token, not a literal.
 ## How configuration resolves (read this first)
 The Host builds its configuration in `src/ZB.MOM.WW.ScadaBridge.Host/Program.cs` in this precedence
 order (later overrides earlier):
 1. `appsettings.json`
 2. `appsettings.{SCADABRIDGE_CONFIG}.json` (role file — `Central` or `Site`)
 3. **`.AddEnvironmentVariables()`** ← every variable below in the "config-override" table
 4. `.AddCommandLine(args)`
 Then a **pre-host secrets expander** resolves any `${secret:NAME}` tokens left in the merged config,
 reading the encrypted secrets store whose master key comes from **`ZB_SECRETS_MASTER_KEY`**. Because
 step 3 runs *before* expansion, a whole-key env override (e.g. `ScadaBridge__Database__ConfigurationDb`)
 wins over — and short-circuits — the corresponding `${secret:...}` token. That is the deliberate
 "rollback / dev" path the docker rigs use.
 **Config-key → env-var mapping:** replace each `:` in a config path with `__` (double underscore).
 So `ScadaBridge:Communication:GrpcPsk` is set via `ScadaBridge__Communication__GrpcPsk`. Any config
 key in the app can be overridden this way; the table below lists only the ones with an established
 purpose (secrets kept off disk, or dev-rig wiring).
 ---
 ## 1. Role & host environment selection
 | Variable | Scope | In appsettings? | Consumed by | Purpose | Potential values | Required |
 |---|---|---|---|---|---|---|
 | `SCADABRIDGE_CONFIG` | Runtime | No (it *selects* the `appsettings.{value}.json` file) | Host (`Program.cs:42`) | Selects the role-specific overlay. **Primary role switch.** | `Central`, `Site` | Effectively yes (falls back to `DOTNET_ENVIRONMENT`, then `Production`) |
 | `DOTNET_ENVIRONMENT` | Runtime | No | Host (`Program.cs:43`) | Fallback role selector when `SCADABRIDGE_CONFIG` is unset; generic .NET environment name. | `Central`, `Site`, `Development`, `Production` | No |
 | `ASPNETCORE_ENVIRONMENT` | Runtime | No | ASP.NET Core + `DisableLoginGuard` (`Program.cs:438`) | Standard ASP.NET environment. `DisableLogin` is refused unless this is `Development`. | `Development`, `Production` | No (defaults to `Production` behavior) |
 | `ASPNETCORE_URLS` | Runtime | No (Kestrel binding; not in these appsettings) | Kestrel | HTTP bind addresses. Docker sets `http://+:5000`; install.ps1 sets `http://+:{WebPort}` (8085). | e.g. `http://+:5000`, `http://+:8085` | No |
 > `SCADABRIDGE_CONFIG` and `ASPNETCORE_ENVIRONMENT` are independent. The docker rig runs
 > `SCADABRIDGE_CONFIG=Central` **and** `ASPNETCORE_ENVIRONMENT=Development` simultaneously.
 ## 2. Secrets store master key (KEK)
 | Variable | Scope | In appsettings? | Consumed by | Purpose | Potential values | Required |
 |---|---|---|---|---|---|---|
 | `ZB_SECRETS_MASTER_KEY` | Runtime | Name only (`Secrets:MasterKey:EnvVarName` in `appsettings.json`; the value is never committed) | Host secrets expander | The KEK that decrypts the `ZB.MOM.WW.Secrets` store, from which every `${secret:...}` token resolves. A missing reference fails closed (`SecretNotFoundException`) before any SQL/LDAP/cluster wiring. | an opaque base64/hex master key | Yes on any node whose appsettings uses `${secret:...}` (shipped Central + Site do). Docker rigs avoid it with whole-key overrides. |
 The variable *name* is itself configurable via `Secrets:MasterKey:EnvVarName` (defaults to
 `ZB_SECRETS_MASTER_KEY`). See `docs/operations/2026-07-16-secrets-clustered-master-key.md`.
 ## 3. Config-override env vars (`ScadaBridge__*` whole-key)
 Map onto config keys via the `:`→`__` rule. Keep secrets off disk (production, via a secret manager)
 or wire the dev rigs without a KEK. The `__site-x` / `__site-a` leaf is the literal `siteId`.
 All are **Runtime** scope (the Host reads them; Host integration-test fixtures set the same names, but
 that is the same variable, not a separate one).
 | Variable | In appsettings? | Config key | Purpose | Potential values | Required |
 |---|---|---|---|---|---|
 | `ScadaBridge__Database__ConfigurationDb` | Yes — `${secret:sql/scadabridge/configdb-connection}` in `appsettings.Central.json` | `ScadaBridge:Database:ConfigurationDb` | Central config DB connection string (MS SQL). Central-only. | `Server=host,1433;Database=ScadaBridgeConfig;User Id=...;Password=...;TrustServerCertificate=true` | Yes on Central — via env **or** the `${secret}` |
 | `ScadaBridge__Database__MachineDataDb` | No in shipped src appsettings; present in the deploy overlay (`deploy/wonder-app-vd03/appsettings.Central.json`, as `${...}`). install.ps1 derives it as `<DbName>MachineData`. | `ScadaBridge:Database:MachineDataDb` | Central machine-data DB connection string. Central-only. | connection string | Yes on Central |
 | `ScadaBridge__Security__JwtSigningKey` | Yes — `${secret:security/scadabridge/jwt-signing-key}` in `appsettings.Central.json` | `ScadaBridge:Security:JwtSigningKey` | HMAC-SHA256 signing key for the cookie-embedded session JWT. | ≥32-char secret | Yes on Central |
 | `ScadaBridge__Security__Ldap__ServiceAccountPassword` | Yes — `${secret:ldap/scadabridge/service-account-password}` in `appsettings.Central.json` | `ScadaBridge:Security:Ldap:ServiceAccountPassword` | LDAP bind-account password. (Renamed from the old flat `...__LdapServiceAccountPassword`.) | the bind password | Yes on Central if LDAP bind is used |
 | `ScadaBridge__InboundApi__ApiKeyPepper` | No — env/secret only (not in any shipped appsettings) | `ScadaBridge:InboundApi:ApiKeyPepper` | Per-environment pepper for the inbound API-key peppered-HMAC verifier. Hard Central startup requirement since 2026-06-03. | ≥16-char secret, unique per environment | Yes on Central |
 | `ScadaBridge__Communication__GrpcPsk` | Yes — `${secret:SB-GRPC-PSK-site-1}` in `appsettings.Site.json` | `ScadaBridge:Communication:GrpcPsk` | **Site side** of the per-site gRPC control-plane PSK. StartupValidator refuses to boot a Site node without it; both pair nodes carry the same value. | per-site secret (e.g. `dev-grpc-psk-docker-site-a`); prod `${secret:SB-GRPC-PSK-<siteId>}` | Yes on Site |
 | `ScadaBridge__Communication__SitePsks__<siteId>` | No in shipped src appsettings (central normally resolves the `SB-GRPC-PSK-<siteId>` secret); present only in the docker-compose env blocks | `ScadaBridge:Communication:SitePsks:<siteId>` | **Central side** of the same key: the value central presents when dialling that site. Must equal the site's `GrpcPsk`. Override for hosts without a master secret store. | matches the site's `GrpcPsk` | On Central, either this **or** the `SB-GRPC-PSK-<siteId>` store secret |
 Docker dev values (dev-only-insecure, committed in `docker/docker-compose.yml`):
 `ScadaBridge__Communication__SitePsks__site-a = dev-grpc-psk-docker-site-a` (also `site-b`, `site-c`);
 `docker-env2` uses `SitePsks__site-x = dev-grpc-psk-docker-env2-site-x`.
 > Any other config key is overridable the same way (e.g. `ScadaBridge__Cluster__SplitBrainResolverStrategy`,
 > `ScadaBridge__Logging__MinimumLevel`) — these are the standard .NET config binder, not "special" env
 > vars. Prefer `appsettings` files for non-secret values.
 ### `${secret:...}` tokens (resolved from the store, NOT env vars)
 Not environment variables — config values resolved by the secrets expander using `ZB_SECRETS_MASTER_KEY`.
 All are **Runtime** scope and, by definition, present **In appsettings** as tokens.
 | Token (in `appsettings`) | Meaning |
 |---|---|
 | `${secret:sql/scadabridge/configdb-connection}` | Central config DB connection string |
 | `${secret:ldap/scadabridge/service-account-password}` | LDAP bind password |
 | `${secret:security/scadabridge/jwt-signing-key}` | JWT signing key |
 | `${secret:SB-GRPC-PSK-<siteId>}` | Per-site gRPC PSK (site side); central resolves `SB-GRPC-PSK-{siteId}` at channel-build time |
 ## 4. CLI (`scadabridge` / `ZB.MOM.WW.ScadaBridge.CLI`)
 Resolution order per value: command-line flag → env var → `~/.scadabridge/config.json`. The CLI does
 **not** read `appsettings.json` — its config file is `~/.scadabridge/config.json` (a separate schema).
 | Variable | Scope | In appsettings? | Purpose | Potential values | Required |
 |---|---|---|---|---|---|
 | `SCADABRIDGE_MANAGEMENT_URL` | Runtime (CLI) | No — CLI config file has `managementUrl` instead | Management API base URL. Overridden by `--url`; overrides the config file. Missing all three → exit `1`, `NO_URL`. | e.g. `http://localhost:9000` | One of flag/env/config |
 | `SCADABRIDGE_FORMAT` | Runtime (CLI) | No — CLI config file has `defaultFormat` instead | Default output format. Overridden by `--format`. | `json`, `table` | No (defaults to `json`) |
 | `SCADABRIDGE_USERNAME` | Runtime (CLI) | No | LDAP username fallback when `--username` absent. | e.g. `multi-role` | No (some commands need auth) |
 | `SCADABRIDGE_PASSWORD` | Runtime (CLI) | No | LDAP password fallback when `--password` absent. **Preferred over `--password`** — avoids leaking into process listings / shell history. | the user's password | No (some commands need auth) |
 ## 5. DelmiaNotifier client (`WWNotifier.exe`)
 | Variable | Scope | In appsettings? | Purpose | Potential values | Required |
 |---|---|---|---|---|---|
 | `SCADABRIDGE_API_KEY` | Runtime (client) | No — the notifier's `appsettings.json` holds only `BaseUrls`/`TimeoutSeconds`/`LogPath`; the key is **env-only by design** | Inbound API key sent as `X-API-Key`. Read only from the environment, never a file. Missing/empty → prints `API key not configured (SCADABRIDGE_API_KEY)`, exit `-1`, no HTTP attempt. | e.g. `sbk_...` | Yes |
 > This is also why `System.Environment` is wholly forbidden in the script trust model — it would let a
 > site/inbound script exfiltrate `SCADABRIDGE_API_KEY` and other host env secrets.
 ## 6. EF Core design-time tooling
 | Variable | Scope | In appsettings? | Purpose | Potential values | Required |
 |---|---|---|---|---|---|
 | `SCADABRIDGE_DESIGNTIME_CONNECTIONSTRING` | Design-time | Fallback for `ScadaBridge:Database:ConfigurationDb` — the factory reads that appsettings key first, then this env var | Connection string for `dotnet ef` migrations tooling when the Host's appsettings isn't the source. No hardcoded fallback — missing → actionable failure. | a config DB connection string | Only for `dotnet ef` when appsettings is unavailable |
 ## 7. Test / CI-only
 Not read by the running product — the test harness only. Most default gracefully (skip, or use a
 local docker value) when unset.
 | Variable | Scope | In appsettings? | Consumed by | Purpose | Potential values |
 |---|---|---|---|---|---|
 | `SCADABRIDGE_MSSQL_TEST_CONN` | Test | No | `MsSqlMigrationFixture`, AuditLog migration tests | Admin (sa) connection to a running MS SQL; absent → local docker sa default; drives `Skip.If`. | sa connection string |
 | `SCADABRIDGE_PLAYWRIGHT_DB` | Test | No | `PlaywrightDbConnection` | DB connection for Playwright E2E seeding; absent → `localhost:1433`. | connection string |
 | `SCADABRIDGE_CLI_DLL` | Test | No | `CliRunner` | Absolute path to the built CLI dll for Playwright when auto-probing fails. | path to `...CLI.dll` |
 | `SCADABRIDGE_AUDIT_FILTER_4KB_P95_US` | Test | No | `HotPathLatencyTests` | Override P95 latency threshold (µs) for the 4 KB audit-filter perf test. | positive number (default 5) |
 | `SCADABRIDGE_AUDIT_FILTER_RAW_P95_US` | Test | No | `HotPathLatencyTests` | Override P95 threshold for the raw-capture perf test. | positive number |
 | `SCADABRIDGE_AUDIT_FILTER_NOOP_P95_US` | Test | No | `HotPathLatencyTests` | Override P95 threshold for the no-op fast-path perf test. | positive number (default 5) |
 | `HOME` / `USERPROFILE` | Test | No | `CliConfig` tests | Redirected to locate `~/.scadabridge/config.json` under test. | temp dir path |
 The Host test fixtures (`CentralDbTestEnvironment`, `ScadaBridgeWebApplicationFactory`,
 `StartupValidationTests`) set the **section-3 config-override vars** (`ScadaBridge__Database__*`,
 `ScadaBridge__InboundApi__ApiKeyPepper`, etc.) and `DOTNET_ENVIRONMENT=Central` at runtime; those are
 the same variables already documented above, not new ones.
 ## 8. Supporting infrastructure containers (not the app)
 Set on the sidecar/infra containers in `infra/docker-compose.yml`, consumed by those images, not by
 ScadaBridge code — included for completeness when standing up a rig.
 | Variable | Scope | In appsettings? | Container | Purpose | Value used |
 |---|---|---|---|---|---|
 | `MSSQL_SA_PASSWORD` | Infra | No | `scadabridge-mssql` | SQL Server `sa` password. | `ScadaBridge_Dev1#` (dev-only) |
 | `MSSQL_PID` | Infra | No | `scadabridge-mssql` | SQL Server edition. | `Developer` |
 ---
 ## Quick reference: minimum sets to boot a real deploy
 **Central node** (via env, or the `${secret:...}` equivalents + `ZB_SECRETS_MASTER_KEY`):
 `SCADABRIDGE_CONFIG=Central`, `ScadaBridge__Database__ConfigurationDb`,
 `ScadaBridge__Database__MachineDataDb`, `ScadaBridge__Security__JwtSigningKey`,
 `ScadaBridge__Security__Ldap__ServiceAccountPassword`, `ScadaBridge__InboundApi__ApiKeyPepper`, and
 for each managed site either the `SB-GRPC-PSK-<siteId>` secret or `ScadaBridge__Communication__SitePsks__<siteId>`.
 **Site node:** `SCADABRIDGE_CONFIG=Site` and `ScadaBridge__Communication__GrpcPsk` (or its
 `${secret:SB-GRPC-PSK-<siteId>}` form). StartupValidator hard-fails without the PSK.
@@ -0,0 +1,155 @@
 using System.Text.RegularExpressions;
 namespace ZB.MOM.WW.ScadaBridge.ClusterInfrastructure;
 /// <summary>
 /// Pure decision logic for the simultaneous-cold-start split-brain guard (Gitea #33).
 /// </summary>
 /// <remarks>
 ///     <para>
 ///         Both nodes of a 2-node pair are self-first seeds so <b>either</b> can cold-start alone when
 ///         its partner is dead (see <see cref="ClusterOptions.SeedNodes"/>). The cost is that when
 ///         BOTH cold-start at the same instant, each runs Akka's <c>FirstSeedNodeProcess</c>, times out
 ///         waiting for the other, and forms its OWN single-node cluster — a split brain (two oldest
 ///         nodes, two singletons, dual-active). Two independent clusters do not auto-merge, so the
 ///         split persists until an operator restarts one side. It bites on any event that powers up
 ///         both site VMs together (site power restoration, hypervisor host reboot).
 ///     </para>
 ///     <para>
 ///         This guard breaks the symmetry deterministically without giving up cold-start-alone. The
 ///         node with the lexicographically <b>lower</b> canonical address is the <i>preferred
 ///         founder</i>: it always uses self-first order and forms immediately. The <b>higher</b> node
 ///         first probes whether its partner's Akka endpoint is reachable (see
 ///         <see cref="ClusterBootstrapCoordinator"/>): reachable ⇒ peer-first order so it JOINS the
 ///         founder rather than racing it; unreachable after the probe window ⇒ the partner is genuinely
 ///         down, so it falls back to self-first and forms alone. Exactly one node founds when both are
 ///         present; the lower node always founds when it starts alone.
 ///     </para>
 ///     <para>
 ///         <b>Residual trade-off (accepted).</b> Once the higher node observes the partner reachable it
 ///         commits to peer-first — Akka's <c>JoinSeedNodeProcess</c> retries <c>InitJoin</c> forever and
 ///         never self-forms. If the founder dies in the small window between the probe succeeding and
 ///         the join completing, the higher node hangs unjoined until it is restarted (a restart re-runs
 ///         the guard, finds the founder unreachable, and self-forms). We deliberately do NOT re-decide
 ///         mid-handshake — that is exactly the retired <c>SelfFormAfter</c> failure mode. Instead
 ///         <see cref="ClusterBootstrapCoordinator"/> makes the hang operator-visible with a warning if
 ///         the node has not come Up within a bounded time after committing peer-first.
 ///     </para>
 ///     <para>
 ///         This decides the join order BEFORE issuing a single <c>JoinSeedNodes</c>, from an explicit
 ///         reachability signal — unlike a timer-based watchdog, which fires a <c>Join(self)</c>
 ///         mid-handshake on a bare timeout and could not tell "no seed answered" from "a join is in
 ///         flight", islanding a node a failover had just bounced (the rejected self-form-watchdog
 ///         design; see <c>SelfFirstSeedBootstrapTests</c>).
 ///     </para>
 /// </remarks>
 public static class ClusterBootstrapGuard
 {
    private static readonly Regex SeedPattern =
        new(@"^akka(?:\.[a-z0-9]+)?://[^@/]+@(?<host>[^:/]+):(?<port>\d+)", RegexOptions.IgnoreCase | RegexOptions.Compiled);
    /// <summary>The analyzed bootstrap role of this node within its seed list.</summary>
    /// <param name="Applies">True only when the guard engages: exactly two seeds, one of them this node.</param>
    /// <param name="IsFounder">True when this node is the preferred founder (lower address) — form immediately, no probe.</param>
    /// <param name="PartnerHost">The partner's advertised host (to probe when this node is the higher one); null when not applicable.</param>
    /// <param name="PartnerPort">The partner's Akka port.</param>
    /// <param name="SelfFirstOrder"><c>[self, partner]</c> — self-first; forms a new cluster when no peer answers.</param>
    /// <param name="PeerFirstOrder"><c>[partner, self]</c> — peer-first; joins the partner's cluster, never self-forms while it is up.</param>
    public sealed record BootstrapRole(
        bool Applies,
        bool IsFounder,
        string? PartnerHost,
        int PartnerPort,
        string[] SelfFirstOrder,
        string[] PeerFirstOrder);
    /// <summary>
    /// Analyzes the configured seed list to decide this node's bootstrap role. The guard engages only
    /// for the pair shape (exactly two seeds, one of them this node); any other shape
    /// (single-seed / single-node install, three+ seeds, self absent) returns
    /// <see cref="BootstrapRole.Applies"/> = false and the caller joins the configured seeds unchanged.
    /// </summary>
    /// <param name="seeds">The configured seed URIs (self-first by convention, but order is not relied on here).</param>
    /// <param name="selfHost">This node's advertised host (<c>NodeOptions.NodeHostname</c>).</param>
    /// <param name="selfPort">This node's Akka remoting port (<c>NodeOptions.RemotingPort</c>).</param>
    /// <returns>The analyzed role; never null.</returns>
    public static BootstrapRole Analyze(IReadOnlyList<string> seeds, string selfHost, int selfPort)
    {
        ArgumentNullException.ThrowIfNull(seeds);
        ArgumentException.ThrowIfNullOrWhiteSpace(selfHost);
        var na = new BootstrapRole(false, false, null, 0, Array.Empty<string>(), Array.Empty<string>());
        if (seeds.Count != 2) return na;
        var selfKey = Key(selfHost, selfPort);
        string? self = null, partner = null;
        string? partnerHost = null;
        var partnerPort = 0;
        foreach (var seed in seeds)
        {
            if (!TryParse(seed, out var host, out var port)) return na;
            if (string.Equals(Key(host, port), selfKey, StringComparison.OrdinalIgnoreCase))
                self = seed;
            else
            {
                partner = seed;
                partnerHost = host;
                partnerPort = port;
            }
        }
        // Self must be exactly one of the two, and the other must be a distinct partner.
        if (self is null || partner is null || partnerHost is null) return na;
        var selfFirst = new[] { self, partner };
        var peerFirst = new[] { partner, self };
        // Deterministic tie-break: the lower canonical address is the preferred founder. Both nodes
        // compute the same comparison (same two seeds), so exactly one is the founder. Case-INSENSITIVE
        // to match how self/partner are classified above (and StartupValidator.SeedNodeIsSelf):
        // container/DNS hostnames are conventionally case-insensitive, and a casing difference between
        // the two sides' seed config must NOT make both think they are the founder (which would reopen
        // the very split this guard closes).
        var isFounder = string.Compare(
            Key(selfHost, selfPort),
            Key(partnerHost, partnerPort),
            StringComparison.OrdinalIgnoreCase) < 0;
        return new BootstrapRole(true, isFounder, partnerHost, partnerPort, selfFirst, peerFirst);
    }
    /// <summary>
    /// The order the higher (non-founder) node uses once it has learned whether its partner is
    /// reachable: peer-first when the founder is up (join it), self-first when the founder is absent
    /// (form alone — cold-start-alone preserved).
    /// </summary>
    /// <param name="role">The analyzed role (must be applicable and NOT the founder).</param>
    /// <param name="partnerReachable">Whether the partner's Akka endpoint became reachable within the probe window.</param>
    /// <returns>The seed order to pass to <c>Cluster.JoinSeedNodes</c>.</returns>
    public static string[] HigherNodeOrder(BootstrapRole role, bool partnerReachable)
    {
        ArgumentNullException.ThrowIfNull(role);
        return partnerReachable ? role.PeerFirstOrder : role.SelfFirstOrder;
    }
    /// <summary>Parses <c>akka.tcp://system@host:port[/...]</c> into host + port.</summary>
    /// <param name="seed">The seed URI.</param>
    /// <param name="host">The parsed advertised host.</param>
    /// <param name="port">The parsed Akka port.</param>
    /// <returns>True when the seed matched the expected shape.</returns>
    public static bool TryParse(string? seed, out string host, out int port)
    {
        host = string.Empty;
        port = 0;
        if (string.IsNullOrWhiteSpace(seed)) return false;
        var m = SeedPattern.Match(seed.Trim());
        if (!m.Success) return false;
        if (!int.TryParse(m.Groups["port"].Value, out port)) return false;
        host = m.Groups["host"].Value;
        return true;
    }
    private static string Key(string host, int port) => $"{host}:{port}";
 }
@@ -114,4 +114,50 @@ public class ClusterOptions
    /// dials forever — log noise and a defeated validation intent (review 01).
    /// </summary>
    public bool AllowSingleNodeCluster { get; set; }
    /// <summary>
    /// The simultaneous-cold-start split-brain guard (<c>ScadaBridge:Cluster:BootstrapGuard</c>).
    /// Default OFF — a dark switch, so existing deployments and tests keep Akka's config-driven
    /// self-first auto-join byte-identical. See <see cref="ClusterBootstrapGuard"/> for the
    /// decision logic and <c>ClusterBootstrapCoordinator</c> for the runtime (Gitea #33).
    /// </summary>
    public ClusterBootstrapGuardOptions BootstrapGuard { get; set; } = new();
 }
 /// <summary>
 /// Configuration for the simultaneous-cold-start split-brain guard. Both nodes of a pair are
 /// self-first seeds so <b>either</b> can cold-start alone when its partner is dead — but the cost
 /// is that when BOTH cold-start at the same instant each runs Akka's <c>FirstSeedNodeProcess</c>,
 /// times out waiting for the other, and forms its own single-node cluster (a split brain that does
 /// not auto-merge). When <see cref="Enabled"/>, the node does NOT auto-join from its config seeds
 /// (<c>AkkaHostedService.BuildHocon</c> emits an empty seed list); a coordinator picks the join
 /// order — founder self-first / joiner peer-first — after a reachability probe. See
 /// <see cref="ClusterBootstrapGuard"/>.
 /// </summary>
 public sealed class ClusterBootstrapGuardOptions
 {
    /// <summary>
    /// Whether the guard is active. Default <see langword="false"/> — Akka auto-joins from the
    /// config seed list exactly as before. Only meaningful on a node that is one of its own two
    /// pair seeds; inert everywhere else (single-seed site node, self absent, 3+ seeds).
    /// </summary>
    public bool Enabled { get; set; }
    /// <summary>
    /// How long the higher-address node probes its partner's Akka endpoint before concluding the
    /// partner is dead and forming alone. Must comfortably exceed the partner's worst-case
    /// process-start-to-Akka-bind time so a slow-but-alive partner is never mistaken for a dead one
    /// (which would re-open the split). Default 25s. Validated <c>&gt; 0</c> at boot when the guard is on.
    /// </summary>
    public int PartnerProbeSeconds { get; set; } = 25;
    /// <summary>
    /// The interval between partner reachability probes, in milliseconds. Default 500ms.
    /// </summary>
    public int PartnerProbeIntervalMs { get; set; } = 500;
    /// <summary>
    /// The per-probe TCP connect timeout, in milliseconds. Default 1000ms.
    /// </summary>
    public int ProbeConnectTimeoutMs { get; set; } = 1000;
 }
@@ -30,6 +30,8 @@ public sealed class ClusterOptionsValidator : OptionsValidatorBase<ClusterOption
    /// <inheritdoc />
    protected override void Validate(ValidationBuilder builder, ClusterOptions options)
    {
        ValidateBootstrapGuard(builder, options.BootstrapGuard);
        // The design doc states "both nodes are seed nodes — each node lists
        // both itself and its partner" so a properly-configured deployment lists
        // two. Accepting a single-seed configuration silently defeats the
@@ -78,4 +80,33 @@ public sealed class ClusterOptionsValidator : OptionsValidatorBase<ClusterOption
            + "oldest node can run as an isolated single-node cluster during a partition while the "
            + "younger node forms its own, producing two live clusters.");
    }
    /// <summary>
    /// When the bootstrap guard is enabled (Gitea #33), its timing knobs must be positive. A
    /// zero/negative <see cref="ClusterBootstrapGuardOptions.PartnerProbeSeconds"/> in particular
    /// silently degrades the guard to "never wait, always conclude the partner is dead" — the
    /// higher node would form alone immediately and re-open the very split the guard exists to
    /// close. Fail fast at boot rather than producing that silent degradation. Nothing is checked
    /// when the guard is off (the knobs are inert), so a disabled guard never blocks a boot.
    /// </summary>
    private static void ValidateBootstrapGuard(ValidationBuilder builder, ClusterBootstrapGuardOptions? guard)
    {
        if (guard is null || !guard.Enabled)
        {
            return;
        }
        builder.RequireThat(guard.PartnerProbeSeconds > 0,
            $"ClusterOptions.BootstrapGuard.PartnerProbeSeconds must be > 0 when the guard is enabled "
            + $"(was {guard.PartnerProbeSeconds}); a non-positive value makes the higher node conclude its "
            + "partner is dead without waiting and form alone, re-opening the split the guard prevents.");
        builder.RequireThat(guard.PartnerProbeIntervalMs > 0,
            $"ClusterOptions.BootstrapGuard.PartnerProbeIntervalMs must be > 0 when the guard is enabled "
            + $"(was {guard.PartnerProbeIntervalMs}).");
        builder.RequireThat(guard.ProbeConnectTimeoutMs > 0,
            $"ClusterOptions.BootstrapGuard.ProbeConnectTimeoutMs must be > 0 when the guard is enabled "
            + $"(was {guard.ProbeConnectTimeoutMs}).");
    }
 }
@@ -1,4 +1,5 @@
 using Akka.Cluster;
 using ZB.MOM.WW.Health.Akka;
 namespace ZB.MOM.WW.ScadaBridge.Communication.ClusterState;
@@ -32,17 +33,13 @@ public static class ActiveNodeEvaluator
    /// <param name="cluster">The Akka cluster to evaluate.</param>
    /// <param name="role">Optional role scope; when set, only members with this role are considered.</param>
    /// <returns><c>true</c> when self is Up and the oldest Up member in the role scope.</returns>
-    public static bool SelfIsOldestUp(Cluster cluster, string? role = null)
+    /// <remarks>
-    {
+    /// Delegates to the shared <see cref="ClusterActiveNode.SelfIsActive"/>
-        var self = cluster.SelfMember;
+    /// (<c>ZB.MOM.WW.Health.Akka</c> 0.3.0), which is this evaluator promoted into the shared library
-        if (self.Status != MemberStatus.Up)
+    /// after OtOpcUa independently needed the same rule and wrote its own third copy. The rule now has
-            return false;
+    /// one implementation family-wide; this method survives as Communication's entry point into it, so
-        if (role != null && !self.HasRole(role))
+    /// the delivery gate and heartbeat keep their existing call sites and layering.
-            return false;
+    /// </remarks>
-
+    public static bool SelfIsOldestUp(Cluster cluster, string? role = null) =>
-        return cluster.State.Members
+        ClusterActiveNode.SelfIsActive(cluster, role);
            .Where(m => m.Status == MemberStatus.Up)
            .Where(m => role == null || m.HasRole(role))
            .All(m => m.UniqueAddress.Equals(self.UniqueAddress) || self.IsOlderThan(m));
    }
 }
@@ -28,6 +28,13 @@
    <PackageReference Include="Grpc.Net.Client" />
    <PackageReference Include="Grpc.Tools" PrivateAssets="All" />
    <PackageReference Include="ZB.MOM.WW.Configuration" />
    <!-- For ClusterActiveNode — the shared "oldest Up member" primitive that ActiveNodeEvaluator was
         promoted into. The store-and-forward delivery gate, the heartbeat IsActive stamp and the
         /health/active tier must all name the SAME active node; delegating to one implementation is
         what makes that structural. It lives in Health.Akka because that is the family's Akka-cluster
         utility package (it already hosts AkkaClusterStatusPolicy and an endpoint gate); a dedicated
         cluster package would be a better home if a third such primitive appears. -->
    <PackageReference Include="ZB.MOM.WW.Health.Akka" />
  </ItemGroup>
  <ItemGroup>
@@ -123,6 +123,11 @@ public class RealOpcUaClient : IOpcUaClient
            endpoint = new EndpointDescription(endpointUrl);
        }
        // The server advertises its own base address, which is frequently unroutable from here
        // (a 0.0.0.0 wildcard bind, or an internal container/NAT hostname). Swap it back to the
        // host the operator actually reached, or the session channel dials an unreachable address.
        RewriteEndpointHostForReachability(endpoint, endpointUrl);
        var endpointConfig = EndpointConfiguration.Create(appConfig);
        var configuredEndpoint = new ConfiguredEndpoint(null, endpoint, endpointConfig);
@@ -157,6 +162,52 @@ public class RealOpcUaClient : IOpcUaClient
        await _subscription.CreateAsync(cancellationToken);
    }
    /// <summary>
    /// Rewrites a discovered endpoint's host/port to match the reachable URL the operator
    /// configured, when the server advertised a different (unroutable) one.
    /// </summary>
    /// <remarks>
    /// An OPC UA server advertises its base address from its OWN configuration, not the route the
    /// client took to reach it — so <c>GetEndpoints</c> routinely returns a wildcard bind
    /// (<c>opc.tcp://0.0.0.0:4840/…</c>) or an internal container/NAT hostname the client cannot
    /// dial. The OPC Foundation session opens its channel to <see cref="EndpointDescription.EndpointUrl"/>
    /// verbatim, so a 0.0.0.0 advertisement resolves to the client's OWN loopback and the connect
    /// fails. This swaps only the authority (host + port) to the reachable one, preserving the
    /// advertised scheme and path (e.g. <c>/OtOpcUa</c>). Mirrors <c>CoreClientUtils.SelectEndpoint</c>.
    /// A no-op when the advertised host/port is already reachable (so an opc-plc server that
    /// advertises its real container name is left untouched).
    /// </remarks>
    internal static void RewriteEndpointHostForReachability(EndpointDescription? endpoint, string discoveryUrl)
    {
        if (endpoint is null || string.IsNullOrWhiteSpace(endpoint.EndpointUrl))
            return;
        if (!Uri.TryCreate(discoveryUrl, UriKind.Absolute, out var reachable))
            return;
        if (!Uri.TryCreate(endpoint.EndpointUrl, UriKind.Absolute, out var advertised))
            return;
        // Already reachable — leave it exactly as advertised.
        if (string.Equals(advertised.Host, reachable.Host, StringComparison.OrdinalIgnoreCase)
            && advertised.Port == reachable.Port)
            return;
        // An IPv6 literal must keep its brackets, or the reconstructed authority (`::1:4840`) is
        // ambiguous and unparseable. Uri.Host may or may not include them depending on the scheme,
        // so bracket only when missing (never double-wrap).
        var host = reachable.Host;
        if (reachable.HostNameType == UriHostNameType.IPv6 && !host.StartsWith('['))
            host = $"[{host}]";
        // opc.tcp has no default port known to Uri, so Uri.Port is -1 when the text omits one.
        // Prefer the reachable URL's port, fall back to the advertised one, and emit no ":port"
        // at all when neither carries one (rather than a malformed ":-1").
        var port = reachable.Port >= 0 ? reachable.Port : advertised.Port;
        var authority = port >= 0 ? $"{host}:{port}" : host;
        // Authority-only advertisements parse to AbsolutePath "/" — drop it so we don't append a
        // spurious trailing slash the server never advertised.
        var path = advertised.AbsolutePath == "/" ? string.Empty : advertised.AbsolutePath;
        endpoint.EndpointUrl = $"{advertised.Scheme}://{authority}{path}";
    }
    /// <summary>
    /// Probes an OPC UA endpoint configuration WITHOUT persisting it or creating a
    /// long-lived connection — connect, capture the server certificate if it is untrusted,
@@ -258,6 +309,10 @@ public class RealOpcUaClient : IOpcUaClient
                endpoint = new EndpointDescription(endpointUrl);
            }
            // Same reachability rewrite as ConnectAsync — a probe against a server advertising a
            // 0.0.0.0 / NAT base address must dial the reachable host, not the advertised one.
            RewriteEndpointHostForReachability(endpoint, endpointUrl);
            var endpointConfig = EndpointConfiguration.Create(appConfig);
            var configuredEndpoint = new ConfiguredEndpoint(null, endpoint, endpointConfig);
@@ -261,8 +261,16 @@ public class AkkaHostedService : IHostedService
        TimeSpan transportHeartbeat,
        TimeSpan transportFailure)
    {
-        var seedNodesStr = string.Join(",",
+        // Simultaneous-cold-start split-brain guard (Gitea #33): when enabled the node must NOT
-            clusterOptions.SeedNodes.Select(QuoteHocon));
+        // auto-join from its config seeds — Akka's FirstSeedNodeProcess is exactly what races the
        // peer and forms two 1-node clusters. Emit an EMPTY seed list so the ActorSystem comes up
        // unjoined, and ClusterBootstrapCoordinator issues the single reachability-gated
        // JoinSeedNodes (founder self-first / joiner peer-first). Default OFF, so guard-off configs
        // render byte-identical to before.
        var effectiveSeeds = clusterOptions.BootstrapGuard.Enabled
            ? Enumerable.Empty<string>()
            : clusterOptions.SeedNodes;
        var seedNodesStr = string.Join(",", effectiveSeeds.Select(QuoteHocon));
        var rolesStr = string.Join(",", roles.Select(QuoteHocon));
        // auto-down (default): AutoDowning provider — the leader among the reachable
@@ -0,0 +1,211 @@
 using System.Diagnostics;
 using System.Net.Sockets;
 using Akka.Actor;
 using Akka.Cluster;
 using Microsoft.Extensions.Options;
 using ZB.MOM.WW.ScadaBridge.ClusterInfrastructure;
 namespace ZB.MOM.WW.ScadaBridge.Host.Actors;
 /// <summary>
 /// Runs the simultaneous-cold-start split-brain guard (Gitea #33): when
 /// <c>ScadaBridge:Cluster:BootstrapGuard:Enabled</c>, the node starts with NO config seed nodes
 /// (<c>AkkaHostedService.BuildHocon</c> emits an empty seed list), and this coordinator picks the join
 /// order and issues the single <see cref="Akka.Cluster.Cluster.JoinSeedNodes"/> once the ActorSystem is
 /// up. See <see cref="ClusterBootstrapGuard"/> for the decision logic and the split-brain rationale.
 /// </summary>
 /// <remarks>
 ///     <para>
 ///         The founder (lower canonical address) joins its self-first order immediately. The higher
 ///         node probes its partner's Akka endpoint (a plain TCP connect — the partner has bound its
 ///         port well before it forms) up to <see cref="ClusterBootstrapGuardOptions.PartnerProbeSeconds"/>:
 ///         reachable ⇒ peer-first (join the founder); unreachable ⇒ self-first (partner is dead, form
 ///         alone). The join runs on a background task so it never blocks host startup, and it is issued
 ///         exactly once — the coordinator never re-forms a node mid-handshake, the failure mode of the
 ///         rejected self-form-watchdog design (see <c>SelfFirstSeedBootstrapTests</c>).
 ///     </para>
 ///     <para>
 ///         Registered unconditionally in both the Central and Site composition roots but a no-op unless
 ///         the guard is enabled, so guard-off deployments keep Akka's config-driven self-first
 ///         auto-join byte-identical.
 ///     </para>
 /// </remarks>
 public sealed class ClusterBootstrapCoordinator : IHostedService
 {
    private readonly Func<ActorSystem> _system;
    private readonly ClusterOptions _clusterOptions;
    private readonly NodeOptions _nodeOptions;
    private readonly ILogger<ClusterBootstrapCoordinator> _log;
    private readonly CancellationTokenSource _cts = new();
    private Task? _joinTask;
    /// <summary>Creates the coordinator.</summary>
    /// <param name="system">Lazy accessor for the node's ActorSystem (resolved after the Akka hosted service creates it).</param>
    /// <param name="clusterOptions">The bound cluster options (seed list + guard settings).</param>
    /// <param name="nodeOptions">The bound node identity (advertised hostname + remoting port).</param>
    /// <param name="log">Logger for the bootstrap decision.</param>
    public ClusterBootstrapCoordinator(
        Func<ActorSystem> system,
        IOptions<ClusterOptions> clusterOptions,
        IOptions<NodeOptions> nodeOptions,
        ILogger<ClusterBootstrapCoordinator> log)
    {
        _system = system ?? throw new ArgumentNullException(nameof(system));
        _clusterOptions = clusterOptions?.Value ?? throw new ArgumentNullException(nameof(clusterOptions));
        _nodeOptions = nodeOptions?.Value ?? throw new ArgumentNullException(nameof(nodeOptions));
        _log = log ?? throw new ArgumentNullException(nameof(log));
    }
    /// <inheritdoc/>
    public Task StartAsync(CancellationToken cancellationToken)
    {
        if (!_clusterOptions.BootstrapGuard.Enabled)
            return Task.CompletedTask; // dark switch off — Akka auto-joined from config seeds already.
        // Fire-and-forget so a higher node with a dead partner (which waits the full probe window)
        // never blocks host startup. Exceptions are logged; a failed join leaves the node unjoined,
        // which is visible and recoverable, never a silent split.
        _joinTask = Task.Run(() => RunAsync(_cts.Token), _cts.Token);
        return Task.CompletedTask;
    }
    /// <inheritdoc/>
    public async Task StopAsync(CancellationToken cancellationToken)
    {
        await _cts.CancelAsync().ConfigureAwait(false);
        if (_joinTask is not null)
        {
            try { await _joinTask.ConfigureAwait(false); }
            catch (OperationCanceledException) { /* expected on shutdown */ }
        }
    }
    private async Task RunAsync(CancellationToken ct)
    {
        try
        {
            var seeds = _clusterOptions.SeedNodes ?? new List<string>();
            var role = ClusterBootstrapGuard.Analyze(seeds, _nodeOptions.NodeHostname, _nodeOptions.RemotingPort);
            string[] order;
            var committedPeerFirst = false;
            if (!role.Applies)
            {
                // Not a pair seed (single-node install, self absent, malformed, or 3+ seeds): join the
                // configured seeds unchanged — the guard arbitrates only the 2-node pair race.
                order = seeds.ToArray();
                _log.LogInformation(
                    "Bootstrap guard: not a pair seed ({SeedCount} seed(s)); joining configured seeds unchanged.",
                    seeds.Count);
            }
            else if (role.IsFounder)
            {
                order = role.SelfFirstOrder;
                _log.LogInformation(
                    "Bootstrap guard: this node is the preferred founder (lower address); joining self-first, forming immediately if no peer answers.");
            }
            else
            {
                var reachable = await ProbePartnerAsync(role.PartnerHost!, role.PartnerPort, ct).ConfigureAwait(false);
                order = ClusterBootstrapGuard.HigherNodeOrder(role, reachable);
                committedPeerFirst = reachable;
                _log.LogInformation(
                    "Bootstrap guard: this node is the higher address; partner {PartnerHost}:{PartnerPort} was {Reachability} within the probe window; joining {Order}.",
                    role.PartnerHost, role.PartnerPort, reachable ? "REACHABLE" : "UNREACHABLE",
                    reachable ? "peer-first (join the founder)" : "self-first (partner down, forming alone)");
            }
            if (order.Length == 0)
            {
                _log.LogWarning("Bootstrap guard: no seed nodes to join; node stays unjoined until a seed is configured.");
                return;
            }
            var cluster = Akka.Cluster.Cluster.Get(_system());
            var addresses = order.Select(Address.Parse).ToArray();
            cluster.JoinSeedNodes(addresses);
            // Peer-first is a one-way commitment: JoinSeedNodeProcess never self-forms. If the founder
            // died in the probe→join window, this node hangs unjoined forever. We do NOT re-form here
            // (the rejected mid-handshake failure mode) — we make the hang operator-visible so a
            // restart, which re-runs the guard against the now-dead founder and self-forms, recovers
            // it. Nothing is done on the founder / self-first paths, which always self-form on their own.
            if (committedPeerFirst)
                await WarnIfNotUpAsync(cluster, ct).ConfigureAwait(false);
        }
        catch (OperationCanceledException)
        {
            // Host shutting down before the join completed — nothing to do.
        }
        catch (Exception ex)
        {
            _log.LogError(ex, "Bootstrap guard: failed to issue JoinSeedNodes; node remains unjoined.");
        }
    }
    /// <summary>
    /// After committing peer-first, waits a bounded time for this node to reach <c>Up</c> and logs a
    /// clear warning if it does not — the founder must have died in the probe→join window, leaving this
    /// node hung in <c>JoinSeedNodeProcess</c>. Read-only: it never re-forms or re-joins (that is the
    /// rejected mid-handshake failure mode); it only surfaces the condition so an operator restarts the node.
    /// </summary>
    private async Task WarnIfNotUpAsync(Akka.Cluster.Cluster cluster, CancellationToken ct)
    {
        // Give the join a generous grace: the founder's own self-form (seed-node-timeout) plus this
        // node's join round-trip. Two probe windows is comfortably beyond both.
        var grace = TimeSpan.FromSeconds(Math.Max(10, _clusterOptions.BootstrapGuard.PartnerProbeSeconds * 2));
        var sw = Stopwatch.StartNew();
        while (!ct.IsCancellationRequested && sw.Elapsed < grace)
        {
            if (cluster.SelfMember.Status == MemberStatus.Up) return; // joined — all good
            try { await Task.Delay(TimeSpan.FromSeconds(1), ct).ConfigureAwait(false); }
            catch (OperationCanceledException) { return; }
        }
        if (!ct.IsCancellationRequested && cluster.SelfMember.Status != MemberStatus.Up)
            _log.LogWarning(
                "Bootstrap guard: committed peer-first but this node is still {Status} after {Grace}s — its founder likely died in the probe→join window. This node will NOT self-form on its own (peer-first never does). RESTART it to recover: the guard will re-run, find the founder down, and form alone.",
                cluster.SelfMember.Status, (int)grace.TotalSeconds);
    }
    /// <summary>
    /// Polls the partner's Akka endpoint with a plain TCP connect until it is reachable or the probe
    /// window elapses. Reachable-at-TCP is a sound proxy for "the partner is coming up": a node binds
    /// its Akka port near the start of startup, well before it forms or joins a cluster.
    /// </summary>
    private async Task<bool> ProbePartnerAsync(string host, int port, CancellationToken ct)
    {
        var window = TimeSpan.FromSeconds(Math.Max(0, _clusterOptions.BootstrapGuard.PartnerProbeSeconds));
        var interval = TimeSpan.FromMilliseconds(Math.Max(50, _clusterOptions.BootstrapGuard.PartnerProbeIntervalMs));
        var sw = Stopwatch.StartNew();
        while (!ct.IsCancellationRequested)
        {
            if (await TryConnectAsync(host, port, ct).ConfigureAwait(false)) return true;
            if (sw.Elapsed >= window) return false;
            try { await Task.Delay(interval, ct).ConfigureAwait(false); }
            catch (OperationCanceledException) { return false; }
        }
        return false;
    }
    private async Task<bool> TryConnectAsync(string host, int port, CancellationToken ct)
    {
        try
        {
            using var client = new TcpClient();
            using var cts = CancellationTokenSource.CreateLinkedTokenSource(ct);
            cts.CancelAfter(Math.Max(100, _clusterOptions.BootstrapGuard.ProbeConnectTimeoutMs));
            await client.ConnectAsync(host, port, cts.Token).ConfigureAwait(false);
            return client.Connected;
        }
        catch (OperationCanceledException) when (ct.IsCancellationRequested)
        {
            throw; // host shutdown — propagate
        }
        catch
        {
            return false; // connect refused / timed out / DNS not resolvable yet — partner not up
        }
    }
 }
@@ -14,7 +14,8 @@ namespace ZB.MOM.WW.ScadaBridge.Host.Health;
 /// <see cref="ClusterActivityEvaluator.SelfIsOldest(Cluster, string?)"/> — NOT
 /// the cluster leader, whose address-ordered definition diverges from singleton
 /// placement after a restart (review 01 [High]). This is the same evaluator
-/// backing <see cref="OldestNodeActiveHealthCheck"/>, so
+/// backing the shared <c>ActiveNodeHealthCheck</c> registered on the active tier
 /// — both route to <c>ClusterActiveNode</c> — so
 /// <see cref="InboundApiEndpointFilter"/> can return HTTP 503 on a standby.
 ///
 /// Registered only in the Central-role branch of <c>Program.cs</c>. The gate
@@ -1,4 +1,5 @@
 using Akka.Cluster;
 using ZB.MOM.WW.Health.Akka;
 namespace ZB.MOM.WW.ScadaBridge.Host.Health;
@@ -27,14 +28,10 @@ public static class ClusterActivityEvaluator
    /// <param name="cluster">The Akka cluster to evaluate.</param>
    /// <param name="role">Optional role scope; when set, only members with this role are considered.</param>
    /// <returns>The oldest Up member in the role scope, or null when none is Up.</returns>
-    public static Member? OldestUpMember(Cluster cluster, string? role = null)
+    /// <remarks>
-    {
+    /// Delegates to the shared <see cref="ClusterActiveNode.OldestUpMember(Cluster, string?)"/>, so
-        Member? oldest = null;
+    /// the label a node shows and the answer <see cref="SelfIsOldest"/> gates on come from one rule.
-        foreach (var m in cluster.State.Members.Where(m => m.Status == MemberStatus.Up))
+    /// </remarks>
-        {
+    public static Member? OldestUpMember(Cluster cluster, string? role = null) =>
-            if (role != null && !m.HasRole(role)) continue;
+        ClusterActiveNode.OldestUpMember(cluster, role);
            if (oldest == null || m.IsOlderThan(oldest)) oldest = m;
        }
        return oldest;
    }
 }
@@ -1,34 +0,0 @@
 using Akka.Actor;
 using Microsoft.Extensions.Diagnostics.HealthChecks;
 namespace ZB.MOM.WW.ScadaBridge.Host.Health;
 /// <summary>
 /// /health/active check: Healthy only on the oldest Up member — the node that
 /// hosts the cluster singletons. Replaces the shared package's leader-based
 /// ActiveNodeHealthCheck (review 01 [High]): leadership is address-ordered and
 /// diverges from singleton placement after a restart, and during a partition
 /// BOTH nodes compute themselves leader, so Traefik served both. Oldest-member
 /// semantics keep exactly one active node through a partition (the non-oldest
 /// side still sees the old oldest member until SBR resolves membership).
 /// </summary>
 public sealed class OldestNodeActiveHealthCheck : IHealthCheck
 {
    private readonly ActorSystem _system;
    /// <summary>Initializes a new <see cref="OldestNodeActiveHealthCheck"/> bound to the running actor system.</summary>
    /// <param name="system">The live cluster actor system.</param>
    public OldestNodeActiveHealthCheck(ActorSystem system) => _system = system;
    /// <summary>Reports Healthy only when this node is the oldest Up cluster member (the singleton host); otherwise reports Unhealthy.</summary>
    /// <param name="context">The health check context (unused; the result depends only on cluster membership).</param>
    /// <param name="ct">Cancellation token.</param>
    /// <returns>A task that resolves to the health check result.</returns>
    public Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context, CancellationToken ct = default)
    {
        var cluster = Akka.Cluster.Cluster.Get(_system);
        return Task.FromResult(ClusterActivityEvaluator.SelfIsOldest(cluster)
            ? HealthCheckResult.Healthy("oldest Up member (singleton host)")
            : HealthCheckResult.Unhealthy("not the oldest Up member (standby)"));
    }
 }
@@ -0,0 +1,77 @@
 using Microsoft.Extensions.Diagnostics.HealthChecks;
 using ZB.MOM.WW.LocalDb;
 using ZB.MOM.WW.LocalDb.Replication;
 namespace ZB.MOM.WW.ScadaBridge.Host.Health;
 /// <summary>
 /// /health/ready check for the consolidated site database (LocalDb). Site nodes have no EF
 /// context — central's <c>DatabaseHealthCheck&lt;ScadaBridgeDbContext&gt;</c> is central-only — so
 /// the site store is probed directly through the registered <see cref="ILocalDb"/> with a trivial
 /// <c>SELECT 1</c>. That is the honest readiness question for a site node: everything it persists
 /// (OperationTracking, site events, store-and-forward, the replicated config tables) lives here.
 /// </summary>
 /// <remarks>
 /// Replication state is reported as <c>data</c> ENRICHMENT ONLY and never affects the status.
 /// Replication is default-OFF by design, so a node with no peer reports <c>connected=false</c> with
 /// a real 0 backlog — the healthy steady state of an unreplicated node — and failing the check on
 /// it would mark every correctly-configured single node unready.
 /// </remarks>
 public sealed class SiteLocalDbHealthCheck : IHealthCheck
 {
    private readonly ILocalDb _localDb;
    private readonly ISyncStatus? _syncStatus;
    /// <summary>Initializes a new <see cref="SiteLocalDbHealthCheck"/>.</summary>
    /// <param name="localDb">The consolidated site database registered by <c>AddZbLocalDb</c>.</param>
    /// <param name="syncStatus">
    /// The replication status view, when the replication engine is registered. Optional so the check
    /// still works in a composition that wires storage without replication.
    /// </param>
    public SiteLocalDbHealthCheck(ILocalDb localDb, ISyncStatus? syncStatus = null)
    {
        _localDb = localDb ?? throw new ArgumentNullException(nameof(localDb));
        _syncStatus = syncStatus;
    }
    /// <summary>Probes the site database and reports its reachability.</summary>
    /// <param name="context">The health check context (unused; the verdict depends only on the probe).</param>
    /// <param name="cancellationToken">Cancellation token.</param>
    /// <returns>Healthy when the probe query succeeds; Unhealthy with the fault otherwise.</returns>
    public async Task<HealthCheckResult> CheckHealthAsync(
        HealthCheckContext context,
        CancellationToken cancellationToken = default)
    {
        try
        {
            await _localDb.QueryAsync("SELECT 1;", static r => r.GetInt32(0), ct: cancellationToken)
                .ConfigureAwait(false);
        }
        catch (Exception ex) when (ex is not OperationCanceledException)
        {
            return HealthCheckResult.Unhealthy("Site LocalDb is not reachable.", ex);
        }
        return HealthCheckResult.Healthy("Site LocalDb reachable.", BuildData());
    }
    private IReadOnlyDictionary<string, object> BuildData()
    {
        var data = new Dictionary<string, object>(StringComparer.Ordinal);
        if (_syncStatus is null)
            return data;
        data["replicationConnected"] = _syncStatus.Connected;
        // Nullable on purpose upstream: null means "backlog unknown" (no provider wired or the poll
        // failed), which must not be flattened to 0 — that would render a pair which cannot read its
        // own oplog as perfectly caught up. Omit the key instead.
        if (_syncStatus.OplogBacklog is { } backlog)
            data["replicationOplogBacklog"] = backlog;
        if (_syncStatus.PeerNodeId is { } peer)
            data["replicationPeerNodeId"] = peer;
        return data;
    }
 }
@@ -0,0 +1,39 @@
 using Microsoft.Extensions.Diagnostics.HealthChecks;
 using ZB.MOM.WW.ScadaBridge.HealthMonitoring;
 namespace ZB.MOM.WW.ScadaBridge.Host.Health;
 /// <summary>
 /// /health/active check for a SITE pair: Healthy only on the pair's primary — the oldest Up member
 /// carrying the <c>site-{SiteId}</c> role. Same active/standby semantics central publishes, so a
 /// consumer reads one contract fleet-wide (200 = Active, 503 = Standby).
 /// </summary>
 /// <remarks>
 /// Deliberately NOT central's unscoped shared <c>ActiveNodeHealthCheck</c>: that one calls
 /// <c>ClusterActivityEvaluator.SelfIsOldest(cluster)</c> with no role argument, so on a mesh that
 /// carries more than one site's members it computes "oldest" across the wrong member set and a site
 /// node could report Active because it happens to be the oldest member overall. Delegating to
 /// <see cref="IClusterNodeProvider.SelfIsPrimary"/> reuses the role-scoped evaluation the site
 /// already trusts for singleton placement and event-log purge — one primary definition per site,
 /// not two that can disagree.
 /// </remarks>
 public sealed class SitePairActiveNodeHealthCheck : IHealthCheck
 {
    private readonly IClusterNodeProvider _nodeProvider;
    /// <summary>Initializes a new <see cref="SitePairActiveNodeHealthCheck"/>.</summary>
    /// <param name="nodeProvider">The role-scoped site cluster node provider.</param>
    public SitePairActiveNodeHealthCheck(IClusterNodeProvider nodeProvider) =>
        _nodeProvider = nodeProvider ?? throw new ArgumentNullException(nameof(nodeProvider));
    /// <summary>Reports Healthy on the site pair's primary and Unhealthy on the standby.</summary>
    /// <param name="context">The health check context (unused; the verdict depends only on cluster membership).</param>
    /// <param name="cancellationToken">Cancellation token.</param>
    /// <returns>Healthy on the primary; Unhealthy (503 = Standby) otherwise.</returns>
    public Task<HealthCheckResult> CheckHealthAsync(
        HealthCheckContext context,
        CancellationToken cancellationToken = default) =>
        Task.FromResult(_nodeProvider.SelfIsPrimary
            ? HealthCheckResult.Healthy("site pair primary (oldest Up member of the site role)")
            : HealthCheckResult.Unhealthy("not the site pair primary (standby)"));
 }
@@ -344,9 +344,10 @@ try
        // Ready tier (/health/ready): database + akka-cluster + required-singletons.
        // Active tier (/health/active): active-node = oldest Up member (singleton host) AND
        // database reachable (review 01 [High]) — a DB-dead active node drops from Traefik
-        // rotation, and the active-node check is the host-local OldestNodeActiveHealthCheck,
+        // rotation. The active-node check is the shared ActiveNodeHealthCheck, which selects by age
-        // NOT the shared leader-based ActiveNodeHealthCheck (leadership diverges from singleton
+        // as of Health 0.3.0; through 0.2.1 it selected by leadership, which diverges from singleton
-        // placement after a restart and both sides claim leadership during a partition).
+        // placement after a restart and names both sides during a partition, so this host kept a
        // private copy until the shared one was fixed.
        // The Akka checks resolve ActorSystem from DI via the singleton bridge registered below;
        // the DatabaseHealthCheck<TContext> resolves a scoped ScadaBridgeDbContext (no factory).
        builder.Services.AddHealthChecks()
@@ -371,9 +372,12 @@ try
                "required-singletons",
                failureStatus: null,
                tags: new[] { ZbHealthTags.Ready })
-            // Host-local oldest-member check (review 01 [High]) — see OldestNodeActiveHealthCheck.
+            // Oldest-Up-member check (review 01 [High]), unscoped: every central member competes for
-            // Resolves ActorSystem from DI, like the akka-cluster check above.
+            // the one active slot. Since ZB.MOM.WW.Health 0.3.0 this IS the shared check — it selects
-            .AddTypeActivatedCheck<OldestNodeActiveHealthCheck>(
+            // by age, so the host-local OldestNodeActiveHealthCheck that existed only to avoid the old
            // leader-based selection is gone. Resolves ActorSystem from DI, like the akka-cluster
            // check above.
            .AddTypeActivatedCheck<ActiveNodeHealthCheck>(
                "active-node",
                failureStatus: null,
                tags: new[] { ZbHealthTags.Active });
@@ -394,14 +398,28 @@ try
        builder.Services.AddSingleton<Akka.Actor.ActorSystem>(sp =>
            sp.GetRequiredService<AkkaHostedService>().GetOrCreateActorSystem());
        // Simultaneous-cold-start split-brain guard (Gitea #33, dark switch
        // ScadaBridge:Cluster:BootstrapGuard:Enabled, default off). Registered AFTER the
        // AkkaHostedService/ActorSystem bridge so its StartAsync resolves the system the main path
        // built; when the guard is off it no-ops (Akka auto-joined from config seeds), so registering
        // it unconditionally is safe. When on, the node started unjoined (empty HOCON seed list, see
        // AkkaHostedService.BuildHocon) and this coordinator issues the single reachability-gated
        // JoinSeedNodes. It resolves the ActorSystem lazily via GetOrCreateActorSystem so it never
        // races startup or caches a null.
        builder.Services.AddHostedService(sp => new ClusterBootstrapCoordinator(
            () => sp.GetRequiredService<AkkaHostedService>().GetOrCreateActorSystem(),
            sp.GetRequiredService<IOptions<ClusterOptions>>(),
            sp.GetRequiredService<IOptions<NodeOptions>>(),
            sp.GetRequiredService<ILogger<ClusterBootstrapCoordinator>>()));
        // Register the production IActiveNodeGate implementation so
        // standby-node gating is actually enforced (the InboundApiEndpointFilter
        // consults IActiveNodeGate and defaults to "allow" when none is registered,
        // which leaves the design's "central cluster only (active node)" guarantee
        // unenforced in deployed binaries). The gate is backed by the same oldest-member
-        // evaluator (ClusterActivityEvaluator) as OldestNodeActiveHealthCheck above, so the
+        // evaluator (ClusterActivityEvaluator, which delegates to the shared ClusterActiveNode) as
-        // inbound API and the /health/active endpoint Traefik routes against agree on
+        // the active-node check above, so the inbound API and the /health/active endpoint Traefik
-        // which node is active.
+        // routes against agree on which node is active.
        builder.Services.AddSingleton<ZB.MOM.WW.ScadaBridge.InboundAPI.IActiveNodeGate, ActiveNodeGate>();
        // Admin-triggered manual failover of the central pair (Health page control,
@@ -634,6 +652,20 @@ try
        // must be present before the Map* calls on the site role.
        app.UseRouting();
        // Map the canonical three-tier health endpoints on the site node:
        //   /health/ready  — akka-cluster (site-pair membership, and the cluster-view data the
        //                    family overview dashboard reads) + localdb (the consolidated site
        //                    store — the only database a site node has).
        //   /health/active — active-node = this site pair's primary; 200 on the primary, 503 on
        //                    the standby, the same contract central publishes.
        //   /healthz       — bare process liveness.
        // Checks are registered in SiteServiceRegistration.Configure. All three endpoints are
        // anonymous and use the canonical ZbHealthWriter JSON; the site pipeline runs no
        // authentication middleware and has no FallbackPolicy, so nothing extra is needed to keep
        // them reachable. They are served on the HTTP/1.1 listener (metricsPort, default :8084)
        // alongside /metrics — the gRPC listener is HTTP/2-only and unusable by a plain HTTP client.
        app.MapZbHealth();
        // Observability — mount the always-on Prometheus /metrics scrape endpoint.
        // AddZbTelemetry (in SiteServiceRegistration.Configure → BindSharedOptions)
        // wires the OTel Resource + standard instrumentation + Prometheus exporter;
@@ -1,5 +1,7 @@
 using Microsoft.Extensions.DependencyInjection.Extensions;
 using Microsoft.Extensions.Options;
 using ZB.MOM.WW.Health;
 using ZB.MOM.WW.Health.Akka;
 using ZB.MOM.WW.ScadaBridge.AuditLog;
 using ZB.MOM.WW.ScadaBridge.ClusterInfrastructure;
 using ZB.MOM.WW.ScadaBridge.Communication;
@@ -168,6 +170,19 @@ public static class SiteServiceRegistration
        services.AddSingleton<Akka.Actor.ActorSystem>(sp =>
            sp.GetRequiredService<AkkaHostedService>().GetOrCreateActorSystem());
        // Simultaneous-cold-start split-brain guard (Gitea #33, dark switch
        // ScadaBridge:Cluster:BootstrapGuard:Enabled, default off). This is the case the guard was
        // built for — a shared power event that powers up both site VMs together. Registered AFTER
        // the AkkaHostedService/ActorSystem bridge (mirrors the Central composition root in
        // Program.cs); it no-ops when the guard is off, so registering it unconditionally is safe.
        // When on, the node started unjoined (empty HOCON seed list) and this coordinator issues the
        // single reachability-gated JoinSeedNodes, resolving the ActorSystem lazily.
        services.AddHostedService(sp => new ClusterBootstrapCoordinator(
            () => sp.GetRequiredService<AkkaHostedService>().GetOrCreateActorSystem(),
            sp.GetRequiredService<IOptions<ClusterOptions>>(),
            sp.GetRequiredService<IOptions<NodeOptions>>(),
            sp.GetRequiredService<ILogger<ClusterBootstrapCoordinator>>()));
        // Cluster node status provider for health reports
        services.AddSingleton<IClusterNodeProvider>(sp =>
        {
@@ -190,6 +205,39 @@ public static class SiteServiceRegistration
            return () => nodeProvider.SelfIsPrimary;
        });
        // Health checks — the shared ZB.MOM.WW.Health probes, mapped by MapZbHealth on the site's
        // HTTP/1.1 listener (Program.cs). Site nodes served no health endpoints before this; the
        // family overview dashboard probes every node the same way, so a site node has to answer
        // the same two questions central does.
        //
        //   Ready  (/health/ready)  akka-cluster — this node's site-pair membership. It also carries
        //                           the cluster-view data (leader/memberCount/...) that the
        //                           dashboard reads, free with ZB.MOM.WW.Health 0.2.0.
        //                           localdb      — the consolidated site store, which is the only
        //                           database a site node has (central's EF DatabaseHealthCheck is
        //                           central-only and does not apply here).
        //   Active (/health/active) active-node   — site-pair primary vs standby.
        //
        // Registered HERE rather than in Program.cs, like the LocalDb registrations above, so the
        // composition-root tests that build this graph actually cover them. The Akka check resolves
        // ActorSystem from DI per probe via the singleton bridge registered above.
        services.AddHealthChecks()
            .AddTypeActivatedCheck<AkkaClusterHealthCheck>(
                "akka-cluster",
                failureStatus: null,
                tags: new[] { ZbHealthTags.Ready },
                args: AkkaClusterStatusPolicy.Default)
            .AddTypeActivatedCheck<SiteLocalDbHealthCheck>(
                "localdb",
                failureStatus: null,
                tags: new[] { ZbHealthTags.Ready })
            // NOT central's unscoped shared ActiveNodeHealthCheck — that one is role-less; see
            // SitePairActiveNodeHealthCheck for why a site pair needs the role-scoped answer.
            .AddTypeActivatedCheck<SitePairActiveNodeHealthCheck>(
                "active-node",
                failureStatus: null,
                tags: new[] { ZbHealthTags.Active });
        // Options binding
        BindSharedOptions(services, config);
@@ -17,7 +17,14 @@
      "StableAfter": "00:00:15",
      "HeartbeatInterval": "00:00:02",
      "FailureDetectionThreshold": "00:00:10",
-      "MinNrOfMembers": 1
+      "MinNrOfMembers": 1,
      "_bootstrapGuard": "Gitea #33 simultaneous-cold-start split-brain guard. Dark switch, default off (Akka's config-driven self-first auto-join, unchanged). Enable ONLY where both pair VMs can power up together (shared power/hypervisor event) with no start-order serialization: then the lower host:port founds self-first and the higher probes-then-joins, so the pair converges to one cluster instead of two 1-node clusters. Timings validated > 0 when enabled.",
      "BootstrapGuard": {
        "Enabled": false,
        "PartnerProbeSeconds": 25,
        "PartnerProbeIntervalMs": 500,
        "ProbeConnectTimeoutMs": 1000
      }
    },
    "_secrets": "Host-003: Secrets are NOT committed in this file. The ${secret:...} references below are resolved at startup by the pre-host secrets expander (before StartupValidator runs) from the encrypted secrets store (SQLite, seeded via the ZB.MOM.WW.Secrets store/CLI/UI). The KEK is supplied out-of-band via the ZB_SECRETS_MASTER_KEY environment variable and never committed; an unseeded reference fails closed (SecretNotFoundException) before any SQL/LDAP/cluster wiring. ROLLBACK: the whole-key environment override still wins — set ScadaBridge__Database__ConfigurationDb / ScadaBridge__Security__Ldap__ServiceAccountPassword / ScadaBridge__Security__JwtSigningKey and .AddEnvironmentVariables() overlays the concrete value before expansion runs, so the ${secret:...} token is never evaluated. NOTE (Task 1.4): the LDAP settings moved into the nested Security:Ldap sub-section (bound to the shared ZB.MOM.WW.Auth LdapOptions) — the service-account-password env var is ScadaBridge__Security__Ldap__ServiceAccountPassword (was ScadaBridge__Security__LdapServiceAccountPassword).",
    "Database": {
@@ -20,7 +20,14 @@
      "StableAfter": "00:00:15",
      "HeartbeatInterval": "00:00:02",
      "FailureDetectionThreshold": "00:00:10",
-      "MinNrOfMembers": 1
+      "MinNrOfMembers": 1,
      "_bootstrapGuard": "Gitea #33 simultaneous-cold-start split-brain guard. Dark switch, default off (Akka's config-driven self-first auto-join, unchanged). Enable ONLY where both pair VMs can power up together (shared power/hypervisor event) with no start-order serialization: then the lower host:port founds self-first and the higher probes-then-joins, so the pair converges to one cluster instead of two 1-node clusters. Timings validated > 0 when enabled.",
      "BootstrapGuard": {
        "Enabled": false,
        "PartnerProbeSeconds": 25,
        "PartnerProbeIntervalMs": 500,
        "ProbeConnectTimeoutMs": 1000
      }
    },
    "Database": {
      // Migration-only as of LocalDb Phase 2. The site config tables now live in the
@@ -42,7 +42,9 @@ public class MonitoringAuthorizationTests
    public void HealthDashboard_IsIntentionallyAllAuthenticatedRoles()
    {
        // Health Dashboard stays all-roles (no policy) per the design doc.
-        var attr = AuthorizeOf<Health>();
+        // Fully qualified: the Communication.Health namespace otherwise makes
        // the bare `Health` page type ambiguous.
        var attr = AuthorizeOf<CentralUI.Components.Pages.Monitoring.Health>();
        Assert.NotNull(attr);
        Assert.Null(attr!.Policy);
@@ -0,0 +1,147 @@
 namespace ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests;
 /// <summary>
 /// Unit tests for <see cref="ClusterBootstrapGuard"/> — the pure decision core of the
 /// simultaneous-cold-start split-brain guard (Gitea #33). The runtime probe + JoinSeedNodes wiring
 /// lives in <c>ClusterBootstrapCoordinator</c> and is covered by the real-ActorSystem
 /// <c>ClusterBootstrapCoordinatorTests</c> in the integration suite.
 /// </summary>
 public class ClusterBootstrapGuardTests
 {
    private const string A = "akka.tcp://scadabridge@scadabridge-site-a-a:8082";
    private const string B = "akka.tcp://scadabridge@scadabridge-site-a-b:8082";
    [Fact]
    public void Lower_address_node_is_the_founder_and_needs_no_probe()
    {
        // scadabridge-site-a-a < scadabridge-site-a-b, so on node-a the guard makes it the founder.
        var role = ClusterBootstrapGuard.Analyze(new[] { A, B }, "scadabridge-site-a-a", 8082);
        Assert.True(role.Applies);
        Assert.True(role.IsFounder);
        Assert.Equal(new[] { A, B }, role.SelfFirstOrder);
    }
    [Fact]
    public void Higher_address_node_is_not_the_founder_and_targets_the_partner_for_probing()
    {
        // On node-b the partner is node-a (the lower/founder).
        var role = ClusterBootstrapGuard.Analyze(new[] { B, A }, "scadabridge-site-a-b", 8082);
        Assert.True(role.Applies);
        Assert.False(role.IsFounder);
        Assert.Equal("scadabridge-site-a-a", role.PartnerHost);
        Assert.Equal(8082, role.PartnerPort);
    }
    [Fact]
    public void Tie_break_is_symmetric_regardless_of_seed_order()
    {
        // Both nodes must agree on who founds no matter how each lists its seeds.
        var onLower = ClusterBootstrapGuard.Analyze(new[] { A, B }, "scadabridge-site-a-a", 8082);
        var onHigher = ClusterBootstrapGuard.Analyze(new[] { B, A }, "scadabridge-site-a-b", 8082);
        Assert.True(onLower.IsFounder);
        Assert.False(onHigher.IsFounder);
        // Exactly one founder.
        Assert.True(onLower.IsFounder ^ onHigher.IsFounder);
    }
    [Fact]
    public void Tie_break_is_case_insensitive_so_a_casing_mismatch_cannot_make_both_founders()
    {
        // The two sides' seed config differ only in host casing (review note 1): if the tie-break
        // were case-sensitive, both could conclude they are the founder and re-open the split.
        var lowerSelf = "akka.tcp://scadabridge@SITE-A-A:8082";
        var higherSelf = "akka.tcp://scadabridge@site-a-b:8082";
        var onLower = ClusterBootstrapGuard.Analyze(new[] { lowerSelf, higherSelf }, "SITE-A-A", 8082);
        var onHigher = ClusterBootstrapGuard.Analyze(new[] { higherSelf, lowerSelf }, "site-a-b", 8082);
        Assert.True(onLower.IsFounder ^ onHigher.IsFounder);
        Assert.True(onLower.IsFounder); // "site-a-a" < "site-a-b" ordinal-ignore-case
    }
    [Fact]
    public void Higher_node_joins_the_founder_when_partner_is_reachable()
    {
        var role = ClusterBootstrapGuard.Analyze(new[] { B, A }, "scadabridge-site-a-b", 8082);
        // Reachable ⇒ peer-first ⇒ JoinSeedNodeProcess ⇒ joins the founder, never self-forms.
        Assert.Equal(new[] { A, B }, ClusterBootstrapGuard.HigherNodeOrder(role, partnerReachable: true)); // partner (node-a) first
    }
    [Fact]
    public void Higher_node_forms_alone_when_partner_is_unreachable()
    {
        var role = ClusterBootstrapGuard.Analyze(new[] { B, A }, "scadabridge-site-a-b", 8082);
        // Unreachable after the window ⇒ partner is dead ⇒ self-first ⇒ cold-start-alone preserved.
        Assert.Equal(new[] { B, A }, ClusterBootstrapGuard.HigherNodeOrder(role, partnerReachable: false)); // self (node-b) first
    }
    [Fact]
    public void Guard_does_not_apply_to_a_single_seed_node()
    {
        // A single-node install lists exactly one seed (itself) — not a pair, guard is inert.
        var role = ClusterBootstrapGuard.Analyze(new[] { "akka.tcp://scadabridge@lone:8081" }, "lone", 8081);
        Assert.False(role.Applies);
    }
    [Fact]
    public void Guard_does_not_apply_when_self_is_absent_from_the_two_seeds()
    {
        var role = ClusterBootstrapGuard.Analyze(new[] { A, B }, "scadabridge-central-a", 8081);
        Assert.False(role.Applies);
    }
    [Fact]
    public void Guard_does_not_apply_to_three_or_more_seeds()
    {
        var role = ClusterBootstrapGuard.Analyze(
            new[] { A, B, "akka.tcp://scadabridge@scadabridge-site-a-c:8082" }, "scadabridge-site-a-a", 8082);
        Assert.False(role.Applies);
    }
    [Fact]
    public void Guard_does_not_apply_when_a_seed_is_malformed()
    {
        var role = ClusterBootstrapGuard.Analyze(new[] { A, "not-a-seed" }, "scadabridge-site-a-a", 8082);
        Assert.False(role.Applies);
    }
    [Theory]
    [InlineData("akka.tcp://scadabridge@scadabridge-site-a-a:8082", "scadabridge-site-a-a", 8082)]
    [InlineData("akka.tcp://scadabridge@10.0.0.5:8082/", "10.0.0.5", 8082)]
    [InlineData("  akka.tcp://scadabridge@host.lan:1234  ", "host.lan", 1234)]
    public void TryParse_extracts_host_and_port(string seed, string expectedHost, int expectedPort)
    {
        Assert.True(ClusterBootstrapGuard.TryParse(seed, out var host, out var port));
        Assert.Equal(expectedHost, host);
        Assert.Equal(expectedPort, port);
    }
    [Theory]
    [InlineData("")]
    [InlineData("garbage")]
    [InlineData("akka.tcp://scadabridge@host")] // no port
    public void TryParse_rejects_malformed_seeds(string seed)
    {
        Assert.False(ClusterBootstrapGuard.TryParse(seed, out _, out _));
    }
    [Fact]
    public void Port_participates_in_the_tie_break_when_hosts_are_equal()
    {
        // Shared-host loopback pair (test rig): same host, different ports — port breaks the tie.
        const string low = "akka.tcp://scadabridge@127.0.0.1:8081";
        const string high = "akka.tcp://scadabridge@127.0.0.1:8082";
        Assert.True(ClusterBootstrapGuard.Analyze(new[] { low, high }, "127.0.0.1", 8081).IsFounder);
        Assert.False(ClusterBootstrapGuard.Analyze(new[] { high, low }, "127.0.0.1", 8082).IsFounder);
    }
 }
@@ -215,4 +215,68 @@ public class ClusterOptionsValidatorTests
        Assert.Contains("StableAfter", result.FailureMessage);
        Assert.Contains("HeartbeatInterval", result.FailureMessage);
    }
    // ---- BootstrapGuard timing validation (Gitea #33) ----
    [Fact]
    public void BootstrapGuard_disabled_with_zero_timings_passes_validation()
    {
        // The knobs are inert when the guard is off, so a disabled guard must never block a boot —
        // even if the timing values are nonsense (0 here).
        var options = ValidOptions();
        options.BootstrapGuard = new ClusterBootstrapGuardOptions
        {
            Enabled = false,
            PartnerProbeSeconds = 0,
            PartnerProbeIntervalMs = 0,
            ProbeConnectTimeoutMs = 0,
        };
        var result = new ClusterOptionsValidator().Validate(null, options);
        Assert.True(result.Succeeded, result.FailureMessage);
    }
    [Fact]
    public void BootstrapGuard_enabled_with_default_timings_passes_validation()
    {
        var options = ValidOptions();
        options.BootstrapGuard = new ClusterBootstrapGuardOptions { Enabled = true };
        var result = new ClusterOptionsValidator().Validate(null, options);
        Assert.True(result.Succeeded, result.FailureMessage);
    }
    [Fact]
    public void BootstrapGuard_enabled_with_nonpositive_probe_seconds_fails_validation()
    {
        // A zero PartnerProbeSeconds silently degrades the guard to "never wait, always conclude the
        // partner is dead" — the higher node forms alone immediately and re-opens the split. Reject it.
        var options = ValidOptions();
        options.BootstrapGuard = new ClusterBootstrapGuardOptions { Enabled = true, PartnerProbeSeconds = 0 };
        var result = new ClusterOptionsValidator().Validate(null, options);
        Assert.True(result.Failed);
        Assert.Contains("PartnerProbeSeconds", result.FailureMessage);
    }
    [Fact]
    public void BootstrapGuard_enabled_with_nonpositive_interval_or_timeout_fails_validation()
    {
        var options = ValidOptions();
        options.BootstrapGuard = new ClusterBootstrapGuardOptions
        {
            Enabled = true,
            PartnerProbeIntervalMs = 0,
            ProbeConnectTimeoutMs = -1,
        };
        var result = new ClusterOptionsValidator().Validate(null, options);
        Assert.True(result.Failed);
        Assert.Contains("PartnerProbeIntervalMs", result.FailureMessage);
        Assert.Contains("ProbeConnectTimeoutMs", result.FailureMessage);
    }
 }
@@ -0,0 +1,102 @@
 using Opc.Ua;
 using ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Adapters;
 namespace ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests.Adapters;
 /// <summary>
 /// Covers <see cref="RealOpcUaClient.RewriteEndpointHostForReachability"/> — the fix for OPC UA
 /// servers (e.g. OtOpcUa v3) that advertise an unroutable base address (a 0.0.0.0 wildcard bind or
 /// an internal container/NAT hostname). The session must dial the host the operator reached, not
 /// the advertised one, while preserving the advertised scheme + path.
 /// </summary>
 public class RealOpcUaClientEndpointRewriteTests
 {
    private static string? Rewrite(string advertised, string discovery)
    {
        var ep = new EndpointDescription(advertised);
        RealOpcUaClient.RewriteEndpointHostForReachability(ep, discovery);
        return ep.EndpointUrl;
    }
    [Fact]
    public void Rewrites_0_0_0_0_advertisement_to_the_reachable_host_preserving_path()
    {
        Assert.Equal(
            "opc.tcp://otopcua-dev-site-a-1-1:4840/OtOpcUa",
            Rewrite("opc.tcp://0.0.0.0:4840/OtOpcUa", "opc.tcp://otopcua-dev-site-a-1-1:4840/OtOpcUa"));
    }
    [Fact]
    public void Rewrites_internal_nat_hostname_to_the_reachable_host()
    {
        Assert.Equal(
            "opc.tcp://plc.example.com:4840/OtOpcUa",
            Rewrite("opc.tcp://be5667fab9da:4840/OtOpcUa", "opc.tcp://plc.example.com:4840/OtOpcUa"));
    }
    [Fact]
    public void Rewrites_port_too_when_the_advertised_port_differs()
    {
        Assert.Equal(
            "opc.tcp://gateway:51210/OtOpcUa",
            Rewrite("opc.tcp://0.0.0.0:4840/OtOpcUa", "opc.tcp://gateway:51210/OtOpcUa"));
    }
    [Fact]
    public void Leaves_an_already_reachable_advertisement_untouched()
    {
        // opc-plc advertises its real container name — must not be rewritten.
        Assert.Equal(
            "opc.tcp://scadabridge-opcua:50000",
            Rewrite("opc.tcp://scadabridge-opcua:50000", "opc.tcp://scadabridge-opcua:50000"));
    }
    [Fact]
    public void Handles_authority_only_advertisement_without_appending_a_trailing_slash()
    {
        Assert.Equal(
            "opc.tcp://host-b:4840",
            Rewrite("opc.tcp://0.0.0.0:4840", "opc.tcp://host-b:4840"));
    }
    [Fact]
    public void Re_wraps_an_ipv6_reachable_host_in_brackets()
    {
        Assert.Equal(
            "opc.tcp://[::1]:4840/OtOpcUa",
            Rewrite("opc.tcp://0.0.0.0:4840/OtOpcUa", "opc.tcp://[::1]:4840/OtOpcUa"));
    }
    [Fact]
    public void Emits_no_port_when_neither_url_carries_one()
    {
        // opc.tcp has no default port, so Uri.Port is -1 on both — must not emit ":-1".
        Assert.Equal(
            "opc.tcp://myhost/UaServer",
            Rewrite("opc.tcp://0.0.0.0/UaServer", "opc.tcp://myhost/UaServer"));
    }
    [Fact]
    public void Uses_the_reachable_port_when_the_advertisement_omits_one()
    {
        Assert.Equal(
            "opc.tcp://myhost:4840/UaServer",
            Rewrite("opc.tcp://0.0.0.0/UaServer", "opc.tcp://myhost:4840/UaServer"));
    }
    [Theory]
    [InlineData("")]
    [InlineData("   ")]
    [InlineData("not-a-uri")]
    public void Leaves_endpoint_unchanged_when_discovery_url_is_unusable(string discovery)
    {
        Assert.Equal("opc.tcp://0.0.0.0:4840/OtOpcUa", Rewrite("opc.tcp://0.0.0.0:4840/OtOpcUa", discovery));
    }
    [Fact]
    public void Null_endpoint_is_a_no_op()
    {
        // Must not throw.
        RealOpcUaClient.RewriteEndpointHostForReachability(null, "opc.tcp://host:4840");
    }
 }
@@ -5,6 +5,7 @@ using Microsoft.Extensions.DependencyInjection;
 using Microsoft.Extensions.Diagnostics.HealthChecks;
 using Microsoft.Extensions.Options;
 using ZB.MOM.WW.Health;
 using ZB.MOM.WW.Health.Akka;
 using ZB.MOM.WW.ScadaBridge.Host.Health;
 namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
@@ -207,11 +208,13 @@ public class HealthCheckTests : IDisposable
    }
    [Fact]
-    public void ActiveNodeCheck_IsHostLocalOldestNodeCheck()
+    public void ActiveNodeCheck_IsTheSharedOldestUpMemberCheck()
    {
-        // The shared package's ActiveNodeHealthCheck is LEADER-based (review 01 [High]):
+        // The property under test is that the active tier selects by AGE, not by leadership: during a
-        // during a partition both nodes think they lead and Traefik serves both. The
+        // partition both sides think they lead and Traefik would serve both (review 01 [High]).
-        // active-node check must be the host-local oldest-member check instead.
+        // Through ZB.MOM.WW.Health 0.2.1 the shared check was leader-based, so this host carried a
        // private OldestNodeActiveHealthCheck; 0.3.0 made the shared check age-based, so the private
        // copy is gone and the shared type is now the correct answer here.
        var previousEnv = Environment.GetEnvironmentVariable("DOTNET_ENVIRONMENT");
        try
        {
@@ -222,7 +225,7 @@ public class HealthCheckTests : IDisposable
            Assert.Contains(ZbHealthTags.Active, active.Tags);
            var check = active.Factory(factory.Services);
-            Assert.IsType<OldestNodeActiveHealthCheck>(check);
+            Assert.IsType<ActiveNodeHealthCheck>(check);
        }
        finally
        {
@@ -0,0 +1,239 @@
 using Microsoft.AspNetCore.Builder;
 using Microsoft.Extensions.Configuration;
 using Microsoft.Extensions.DependencyInjection;
 using Microsoft.Extensions.Diagnostics.HealthChecks;
 using Microsoft.Extensions.Options;
 using ZB.MOM.WW.Health;
 using ZB.MOM.WW.Health.Akka;
 using ZB.MOM.WW.LocalDb;
 using ZB.MOM.WW.LocalDb.Registration;
 using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
 using ZB.MOM.WW.ScadaBridge.HealthMonitoring;
 using ZB.MOM.WW.ScadaBridge.Host.Health;
 namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
 /// <summary>
 /// Site-node health endpoints. Site nodes previously served no health surface at all — only gRPC on
 /// the HTTP/2 listener and <c>/metrics</c> on the HTTP/1.1 one — so the family overview dashboard
 /// had no uniform way to probe them. These tests pin the three checks the site now registers and
 /// the tier each one lands in.
 /// <para>
 /// The registration assertions build the REAL container from
 /// <see cref="SiteServiceRegistration.Configure"/> rather than a hand-assembled
 /// <c>ServiceCollection</c>, for the reason spelled out in <see cref="SiteLocalDbWiringTests"/>:
 /// this family's wiring defects have consistently been ones that only a container-built graph
 /// catches. Each registration is additionally ACTIVATED through its factory, so a check that is
 /// registered but cannot resolve its dependencies fails here instead of at the first live probe.
 /// </para>
 /// </summary>
 public class SiteHealthCheckTests : IDisposable
 {
    private readonly WebApplication _host;
    private readonly string _tempDbPath;
    private readonly string _tempSiteDbPath;
    private readonly string _tempTrackingDbPath;
    public SiteHealthCheckTests()
    {
        var stamp = Guid.NewGuid();
        _tempDbPath = Path.Combine(Path.GetTempPath(), $"scadabridge_health_localdb_{stamp}.db");
        _tempSiteDbPath = Path.Combine(Path.GetTempPath(), $"scadabridge_health_site_{stamp}.db");
        _tempTrackingDbPath = Path.Combine(Path.GetTempPath(), $"scadabridge_health_track_{stamp}.db");
        var builder = WebApplication.CreateBuilder();
        builder.Configuration.Sources.Clear();
        builder.Configuration.AddInMemoryCollection(new Dictionary<string, string?>
        {
            ["ScadaBridge:Node:Role"] = "Site",
            ["ScadaBridge:Node:NodeName"] = "node-a",
            ["ScadaBridge:Node:NodeHostname"] = "test-site",
            ["ScadaBridge:Node:SiteId"] = "TestSite",
            ["ScadaBridge:Node:RemotingPort"] = "0",
            ["ScadaBridge:Node:GrpcPort"] = "0",
            ["ScadaBridge:Database:SiteDbPath"] = _tempSiteDbPath,
            ["ScadaBridge:OperationTracking:ConnectionString"] = $"Data Source={_tempTrackingDbPath}",
            ["ScadaBridge:Cluster:SeedNodes:0"] = "akka.tcp://scadabridge@localhost:2551",
            ["ScadaBridge:Cluster:SeedNodes:1"] = "akka.tcp://scadabridge@localhost:2552",
            ["LocalDb:Path"] = _tempDbPath,
        });
        builder.Services.AddGrpc();
        builder.Services.AddSingleton<ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamGrpcServer>();
        SiteServiceRegistration.Configure(builder.Services, builder.Configuration);
        AkkaHostedServiceRemover.RemoveAkkaHostedServiceOnly(builder.Services);
        _host = builder.Build();
    }
    public void Dispose()
    {
        (_host as IDisposable)?.Dispose();
        foreach (var path in new[] { _tempDbPath, _tempSiteDbPath, _tempTrackingDbPath })
        {
            try { File.Delete(path); } catch { /* best effort */ }
        }
        GC.SuppressFinalize(this);
    }
    private IReadOnlyList<HealthCheckRegistration> Registrations =>
        _host.Services.GetRequiredService<IOptions<HealthCheckServiceOptions>>().Value.Registrations.ToList();
    [Fact]
    public void Site_RegistersExactlyTheThreeSiteChecks()
    {
        // Exact-set, not "contains": an extra check silently changes what /health/ready means for
        // every consumer, and a site node inheriting a central-only probe is precisely the mistake
        // this asserts against.
        Assert.Equal(
            new[] { "active-node", "akka-cluster", "localdb" },
            Registrations.Select(r => r.Name).OrderBy(n => n, StringComparer.Ordinal).ToArray());
    }
    [Theory]
    [InlineData("akka-cluster", ZbHealthTags.Ready)]
    [InlineData("localdb", ZbHealthTags.Ready)]
    [InlineData("active-node", ZbHealthTags.Active)]
    public void Site_ChecksCarryTheirIntendedTier(string name, string expectedTag)
    {
        var registration = Assert.Single(Registrations, r => r.Name == name);
        // Exactly one tier tag each: MapZbHealth selects by tag, so a check tagged both Ready and
        // Active would put standby-ness into readiness and drop the standby out of readiness too.
        Assert.Equal(new[] { expectedTag }, registration.Tags.ToArray());
    }
    [Theory]
    [InlineData("akka-cluster", typeof(AkkaClusterHealthCheck))]
    [InlineData("localdb", typeof(SiteLocalDbHealthCheck))]
    [InlineData("active-node", typeof(SitePairActiveNodeHealthCheck))]
    public void Site_ChecksActivateToTheIntendedType(string name, Type expected)
    {
        var registration = Assert.Single(Registrations, r => r.Name == name);
        // Running the factory is the point: a type-activated check whose constructor dependencies
        // are not registered on site would throw here rather than on the first probe in production.
        var check = registration.Factory(_host.Services);
        Assert.IsType(expected, check);
    }
    [Fact]
    public void Site_ActiveNodeCheck_IsRoleScoped_NotTheUnscopedCentralCheck()
    {
        // Central registers the shared ActiveNodeHealthCheck UNSCOPED — oldest Up member of the whole
        // cluster. On a mesh carrying more than one site's members that computes "oldest" across the
        // wrong member set, so a site node could report Active purely for being the oldest member
        // overall. A site must answer for its own site-{SiteId} role, which SitePairActiveNodeHealthCheck
        // does by delegating to the site's IClusterNodeProvider (itself now backed by the shared
        // ClusterActiveNode, so the rule is shared even though the scoping is local).
        var check = Assert.Single(Registrations, r => r.Name == "active-node").Factory(_host.Services);
        Assert.IsType<SitePairActiveNodeHealthCheck>(check);
        Assert.IsNotType<ActiveNodeHealthCheck>(check);
    }
    [Theory]
    [InlineData("database")]
    [InlineData("required-singletons")]
    public void Site_DoesNotRegisterCentralOnlyChecks(string centralOnlyName)
    {
        // Positive control first: if the harness ever stopped registering health checks at all,
        // every "is absent" assertion below would pass vacuously and this test would be worthless.
        Assert.Contains(Registrations, r => r.Name == "akka-cluster");
        Assert.DoesNotContain(Registrations, r => r.Name == centralOnlyName);
    }
    [Fact]
    public async Task Site_LocalDbCheck_ReportsHealthyAgainstTheRealSiteDatabase()
    {
        // Behaviour, not wiring: the probe runs against the consolidated site database this
        // composition actually built, so a check that resolves but cannot query fails here.
        var check = (SiteLocalDbHealthCheck)Assert.Single(Registrations, r => r.Name == "localdb")
            .Factory(_host.Services);
        var result = await check.CheckHealthAsync(NewContext("localdb", check));
        Assert.Equal(HealthStatus.Healthy, result.Status);
    }
    [Fact]
    public async Task Site_LocalDbCheck_EnrichesWithReplicationStateWithoutFailingOnDefaultOff()
    {
        // Replication ships default-OFF, so the unreplicated steady state must stay Healthy while
        // still surfacing the state as data — enrichment must never become a verdict.
        var check = (SiteLocalDbHealthCheck)Assert.Single(Registrations, r => r.Name == "localdb")
            .Factory(_host.Services);
        var result = await check.CheckHealthAsync(NewContext("localdb", check));
        Assert.Equal(HealthStatus.Healthy, result.Status);
        Assert.False(Assert.IsType<bool>(result.Data["replicationConnected"]));
        Assert.Equal(0L, Assert.IsType<long>(result.Data["replicationOplogBacklog"]));
    }
    [Fact]
    public async Task Site_LocalDbCheck_ReportsUnhealthyWhenTheStoreIsUnreachable()
    {
        var check = new SiteLocalDbHealthCheck(new ThrowingLocalDb());
        var result = await check.CheckHealthAsync(NewContext("localdb", check));
        Assert.Equal(HealthStatus.Unhealthy, result.Status);
    }
    [Theory]
    [InlineData(true, HealthStatus.Healthy)]
    [InlineData(false, HealthStatus.Unhealthy)]
    public async Task Site_ActiveNodeCheck_SplitsPrimaryFromStandby(bool selfIsPrimary, HealthStatus expected)
    {
        // 200 on the primary / 503 on the standby is the same contract central publishes, so a
        // consumer reads one active-node meaning fleet-wide.
        var check = new SitePairActiveNodeHealthCheck(new StubNodeProvider(selfIsPrimary));
        var result = await check.CheckHealthAsync(NewContext("active-node", check));
        Assert.Equal(expected, result.Status);
    }
    private static HealthCheckContext NewContext(string name, IHealthCheck check) => new()
    {
        Registration = new HealthCheckRegistration(name, check, HealthStatus.Unhealthy, tags: null),
    };
    private sealed class StubNodeProvider(bool selfIsPrimary) : IClusterNodeProvider
    {
        public bool SelfIsPrimary { get; } = selfIsPrimary;
        public IReadOnlyList<NodeStatus> GetClusterNodes() => [];
    }
    private sealed class ThrowingLocalDb : ILocalDb
    {
        public Task<int> ExecuteAsync(string sql, object? parameters = null, CancellationToken ct = default) =>
            throw new InvalidOperationException("store unreachable");
        public Task<IReadOnlyList<T>> QueryAsync<T>(
            string sql,
            Func<Microsoft.Data.Sqlite.SqliteDataReader, T> map,
            object? parameters = null,
            CancellationToken ct = default) =>
            throw new InvalidOperationException("store unreachable");
        public Task<ILocalDbTransaction> BeginTransactionAsync(CancellationToken ct = default) =>
            throw new InvalidOperationException("store unreachable");
        public Microsoft.Data.Sqlite.SqliteConnection CreateConnection() =>
            throw new InvalidOperationException("store unreachable");
        public ReplicatedTable RegisterReplicated(string tableName) =>
            throw new InvalidOperationException("store unreachable");
        public IReadOnlyDictionary<string, ReplicatedTable> ReplicatedTables =>
            throw new InvalidOperationException("store unreachable");
    }
 }
@@ -0,0 +1,156 @@
 using System.Net;
 using System.Text.Json;
 using Microsoft.AspNetCore.Hosting;
 using Microsoft.AspNetCore.Mvc.Testing;
 using Microsoft.AspNetCore.TestHost;
 using Microsoft.Extensions.DependencyInjection;
 namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
 /// <summary>
 /// Proves the site pipeline actually MAPS the health endpoints — the half
 /// <see cref="SiteHealthCheckTests"/> cannot cover, since that fixture builds the site container
 /// from <see cref="SiteServiceRegistration"/> and never runs <c>Program</c>'s endpoint wiring.
 /// Without this, deleting <c>app.MapZbHealth()</c> from the site branch would leave every
 /// registration test green while site nodes served 404 to the overview dashboard.
 /// <para>
 /// Boots the real <c>Program</c> in the Site role. <c>Program</c> builds its own
 /// <c>ConfigurationBuilder</c> before the web host exists, so its role and settings come from
 /// PROCESS-WIDE environment variables (<c>SCADABRIDGE_CONFIG</c> selects appsettings.Site.json)
 /// rather than from <c>WithWebHostBuilder</c> — which is why this fixture joins
 /// <see cref="HostBootCollection"/> and restores every var it sets.
 /// </para>
 /// </summary>
 [Collection(HostBootCollection.Name)]
 public class SiteHealthEndpointTests : IDisposable
 {
    private readonly List<IDisposable> _disposables = new();
    private readonly Dictionary<string, string?> _previousEnv = new(StringComparer.Ordinal);
    private readonly string _tempDbPath;
    public SiteHealthEndpointTests()
    {
        _tempDbPath = Path.Combine(Path.GetTempPath(), $"scadabridge_health_ep_{Guid.NewGuid()}.db");
        // Whole-key env overrides, the sanctioned path: supplying GrpcPsk concretely makes the
        // pre-host secrets expander skip appsettings.Site.json's ${secret:SB-GRPC-PSK-site-1}
        // token, so this test needs no seeded secrets store.
        //
        // The remoting port is moved off the site default (8082) so this fixture cannot collide
        // with a locally running node, but it must be a REAL port and seed-nodes[0] must be this
        // node itself — StartupValidator enforces both before the host is built, and it runs
        // whether or not the Akka hosted service is later removed. Nothing binds it: the hosted
        // service is removed below.
        SetEnv("SCADABRIDGE_CONFIG", "Site");
        SetEnv("ScadaBridge__Node__Role", "Site");
        SetEnv("ScadaBridge__Node__SiteId", "TestSite");
        SetEnv("ScadaBridge__Node__NodeHostname", "localhost");
        SetEnv("ScadaBridge__Node__RemotingPort", "18082");
        SetEnv("ScadaBridge__Cluster__SeedNodes__0", "akka.tcp://scadabridge@localhost:18082");
        SetEnv("ScadaBridge__Cluster__SeedNodes__1", "akka.tcp://scadabridge@localhost:18085");
        SetEnv("ScadaBridge__Communication__GrpcPsk", "test-psk-0123456789");
        SetEnv("LocalDb__Path", _tempDbPath);
    }
    private void SetEnv(string key, string? value)
    {
        _previousEnv[key] = Environment.GetEnvironmentVariable(key);
        Environment.SetEnvironmentVariable(key, value);
    }
    public void Dispose()
    {
        foreach (var d in _disposables)
        {
            try { d.Dispose(); } catch { /* best effort */ }
        }
        foreach (var (key, value) in _previousEnv)
        {
            Environment.SetEnvironmentVariable(key, value);
        }
        try { File.Delete(_tempDbPath); } catch { /* best effort */ }
        GC.SuppressFinalize(this);
    }
    private HttpClient CreateSiteClient()
    {
        var factory = new WebApplicationFactory<Program>()
            .WithWebHostBuilder(builder =>
            {
                // WebApplicationFactory defaults the environment to Development, which turns on
                // ValidateOnBuild/ValidateScopes. The site graph carries scoped registrations whose
                // dependencies are central-only (ReconcileService → IDeploymentManagerRepository,
                // AuditLogKpiSampleSource → IAuditLogRepository) and are never resolved on a site
                // node, so eager validation fails on a composition that runs fine in production.
                // That is a pre-existing property of the site container, not of the health wiring
                // under test — run this fixture the way a site node actually runs.
                builder.UseEnvironment("Production");
                // ConfigureTestServices runs after the app's own registrations, so this drops the
                // hosted service that would form a real cluster while leaving the AkkaHostedService
                // singleton resolvable (the site pipeline resolves it for shutdown ordering).
                builder.ConfigureTestServices(AkkaHostedServiceRemover.RemoveAkkaHostedServiceOnly);
            });
        _disposables.Add(factory);
        var client = factory.CreateClient();
        _disposables.Add(client);
        return client;
    }
    [Fact]
    public async Task Site_MapsHealthReady_WithTheCanonicalJsonBody()
    {
        var response = await CreateSiteClient().GetAsync("/health/ready");
        // Never 404. 200 vs 503 depends on cluster state this fixture does not form, so both are
        // accepted — what is asserted is that the tier is mapped and answers in canonical shape.
        Assert.NotEqual(HttpStatusCode.NotFound, response.StatusCode);
        Assert.True(
            response.StatusCode is HttpStatusCode.OK or HttpStatusCode.ServiceUnavailable,
            $"Expected 200 or 503, got {(int)response.StatusCode}");
        using var doc = JsonDocument.Parse(await response.Content.ReadAsStringAsync());
        var entries = doc.RootElement.GetProperty("entries");
        Assert.True(entries.TryGetProperty("akka-cluster", out _), "ready tier must carry akka-cluster");
        Assert.True(entries.TryGetProperty("localdb", out _), "ready tier must carry localdb");
        // Readiness must NOT depend on being the primary, or a healthy standby would read unready.
        Assert.False(entries.TryGetProperty("active-node", out _), "active-node belongs to the Active tier");
    }
    [Fact]
    public async Task Site_MapsHealthActive_CarryingOnlyTheActiveNodeCheck()
    {
        var response = await CreateSiteClient().GetAsync("/health/active");
        Assert.NotEqual(HttpStatusCode.NotFound, response.StatusCode);
        using var doc = JsonDocument.Parse(await response.Content.ReadAsStringAsync());
        var entries = doc.RootElement.GetProperty("entries");
        Assert.Equal(
            new[] { "active-node" },
            entries.EnumerateObject().Select(p => p.Name).ToArray());
    }
    [Fact]
    public async Task Site_HealthEndpointsAreAnonymous()
    {
        // The dashboard authenticates nowhere. The site pipeline runs no authentication middleware
        // today, so this is a regression pin: adding one later must not silently close these.
        var client = CreateSiteClient();
        foreach (var path in new[] { "/health/ready", "/health/active", "/healthz" })
        {
            var response = await client.GetAsync(path);
            Assert.NotEqual(HttpStatusCode.Unauthorized, response.StatusCode);
            Assert.NotEqual(HttpStatusCode.Forbidden, response.StatusCode);
            Assert.NotEqual(HttpStatusCode.NotFound, response.StatusCode);
        }
    }
 }
@@ -0,0 +1,204 @@
 using Akka.Actor;
 using Akka.Cluster;
 using Akka.Configuration;
 using Microsoft.Extensions.Logging.Abstractions;
 using Microsoft.Extensions.Options;
 using ZB.MOM.WW.ScadaBridge.ClusterInfrastructure;
 using ZB.MOM.WW.ScadaBridge.Host;
 using ZB.MOM.WW.ScadaBridge.Host.Actors;
 namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster;
 /// <summary>
 /// Real-ActorSystem tests for the simultaneous-cold-start split-brain guard (Gitea #33):
 /// <see cref="ClusterBootstrapCoordinator"/> + <see cref="ClusterBootstrapGuard"/>, driven exactly as
 /// production drives them — the node's ActorSystem is built from the production <c>BuildHocon</c> output
 /// with the guard enabled (which emits an EMPTY seed list, so Akka does not auto-join), then the real
 /// coordinator issues the reachability-gated <c>JoinSeedNodes</c>. This subsystem has a history of bugs
 /// invisible to pure unit tests, so the load-bearing correctness properties are asserted against running
 /// nodes, mirroring <see cref="SelfFirstSeedBootstrapTests"/>.
 /// </summary>
 /// <remarks>
 /// Ephemeral loopback ports share host <c>127.0.0.1</c>, so the port breaks the guard's ordinal
 /// <c>host:port</c> tie — <c>Min(port)</c> is the founder, <c>Max(port)</c> is the higher (probing) node.
 /// </remarks>
 public sealed class ClusterBootstrapCoordinatorTests : IAsyncLifetime
 {
    private readonly List<ActorSystem> _systems = new();
    private readonly List<ClusterBootstrapCoordinator> _coordinators = new();
    /// <summary>Starts a node with the bootstrap guard ENABLED, wired exactly as the composition roots
    /// do: the ActorSystem is built with an empty HOCON seed list (guard on) and the real coordinator
    /// issues the reachability-gated join. Seeds are listed self-first (as every shipped config is); the
    /// guard, not the order, decides founder vs. joiner.</summary>
    private async Task<ActorSystem> StartGuardedNodeAsync(int selfPort, int peerPort, int probeSeconds = 4)
    {
        var self = $"akka.tcp://scadabridge@127.0.0.1:{selfPort}";
        var peer = $"akka.tcp://scadabridge@127.0.0.1:{peerPort}";
        var nodeOptions = new NodeOptions { Role = "Central", NodeHostname = "127.0.0.1", RemotingPort = selfPort };
        var clusterOptions = new ClusterOptions
        {
            SeedNodes = new List<string> { self, peer },
            BootstrapGuard = new ClusterBootstrapGuardOptions
            {
                Enabled = true,
                PartnerProbeSeconds = probeSeconds,
                PartnerProbeIntervalMs = 250,
                ProbeConnectTimeoutMs = 500,
            },
            StableAfter = TimeSpan.FromSeconds(3),
            HeartbeatInterval = TimeSpan.FromMilliseconds(500),
            FailureDetectionThreshold = TimeSpan.FromSeconds(2),
            MinNrOfMembers = 1,
        };
        var hocon = AkkaHostedService.BuildHocon(
            nodeOptions, clusterOptions, new[] { "Central" },
            TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(3));
        var system = ActorSystem.Create("scadabridge", ConfigurationFactory.ParseString(hocon));
        _systems.Add(system);
        var coordinator = new ClusterBootstrapCoordinator(
            () => system, Options.Create(clusterOptions), Options.Create(nodeOptions),
            NullLogger<ClusterBootstrapCoordinator>.Instance);
        _coordinators.Add(coordinator);
        await coordinator.StartAsync(CancellationToken.None);
        return system;
    }
    private static async Task<bool> WaitForUpAsync(ActorSystem system, int expected, TimeSpan timeout)
    {
        var cluster = Akka.Cluster.Cluster.Get(system);
        var deadline = DateTime.UtcNow + timeout;
        while (DateTime.UtcNow < deadline)
        {
            if (cluster.State.Members.Count(m => m.Status == MemberStatus.Up) >= expected) return true;
            await Task.Delay(200);
        }
        return cluster.State.Members.Count(m => m.Status == MemberStatus.Up) >= expected;
    }
    /// <summary>
    /// The FOUNDER (lower address) with the guard on forms a cluster alone when its partner is dead —
    /// exactly as un-guarded self-first does. The guard must not regress the preferred founder.
    /// </summary>
    [Fact]
    public async Task Founder_forms_alone_when_partner_is_dead()
    {
        var low = TwoNodeClusterFixture.GetFreeTcpPort();
        var high = TwoNodeClusterFixture.GetFreeTcpPort();
        var founderPort = Math.Min(low, high);
        var deadPeerPort = Math.Max(low, high); // nothing listening
        var system = await StartGuardedNodeAsync(founderPort, deadPeerPort);
        Assert.True(await WaitForUpAsync(system, 1, TimeSpan.FromSeconds(30)),
            "the founder (lower address) must self-form immediately when its partner is down");
    }
    /// <summary>
    /// THE LOAD-BEARING CASE. The HIGHER node with the guard on must still cold-start ALONE when its
    /// partner is genuinely dead: it probes the (dead) founder, times out, and falls back to self-first.
    /// If the guard's peer-first logic were wrong, this node would hang in JoinSeedNodeProcess forever —
    /// the very regression a naive "always let the lower node found" rule would introduce.
    /// </summary>
    [Fact]
    public async Task Higher_node_forms_alone_when_partner_is_dead_after_probing()
    {
        var low = TwoNodeClusterFixture.GetFreeTcpPort();
        var high = TwoNodeClusterFixture.GetFreeTcpPort();
        var higherPort = Math.Max(low, high);
        var deadFounderPort = Math.Min(low, high); // the founder is dead
        var system = await StartGuardedNodeAsync(higherPort, deadFounderPort, probeSeconds: 3);
        // Must come Up AFTER the probe window (~3s) expires and it falls back to self-first.
        Assert.True(await WaitForUpAsync(system, 1, TimeSpan.FromSeconds(30)),
            "the higher node must fall back to self-first and form alone once the dead partner never answers the probe");
    }
    /// <summary>
    /// FALSIFIABILITY CONTROL for the test above: the higher node does NOT form alone during the probe
    /// window — it is genuinely waiting/probing, not self-forming immediately (which would mean the
    /// guard is inert). If this ever starts seeing a member before the window, the probe is not gating.
    /// </summary>
    [Fact]
    public async Task Higher_node_does_not_form_before_the_probe_window_expires()
    {
        var low = TwoNodeClusterFixture.GetFreeTcpPort();
        var high = TwoNodeClusterFixture.GetFreeTcpPort();
        var higherPort = Math.Max(low, high);
        var deadFounderPort = Math.Min(low, high);
        var system = await StartGuardedNodeAsync(higherPort, deadFounderPort, probeSeconds: 10);
        // Well within the 10s probe window: still no cluster, because it is probing the dead founder.
        Assert.False(await WaitForUpAsync(system, 1, TimeSpan.FromSeconds(3)),
            "the higher node must probe for the founder, not self-form immediately");
    }
    /// <summary>
    /// The higher node JOINS a live founder rather than forming a second cluster: start the founder,
    /// then the higher node — its probe finds the founder reachable, it commits peer-first, and the
    /// pair converges to ONE cluster of two.
    /// </summary>
    [Fact]
    public async Task Higher_node_joins_a_live_founder()
    {
        var low = TwoNodeClusterFixture.GetFreeTcpPort();
        var high = TwoNodeClusterFixture.GetFreeTcpPort();
        var founderPort = Math.Min(low, high);
        var higherPort = Math.Max(low, high);
        var founder = await StartGuardedNodeAsync(founderPort, higherPort);
        Assert.True(await WaitForUpAsync(founder, 1, TimeSpan.FromSeconds(30)),
            "the founder must be Up before the higher node probes it");
        var higher = await StartGuardedNodeAsync(higherPort, founderPort);
        Assert.True(await WaitForUpAsync(higher, 2, TimeSpan.FromSeconds(60)),
            "the higher node must join the founder, forming one cluster of two");
        Assert.True(await WaitForUpAsync(founder, 2, TimeSpan.FromSeconds(60)),
            "the founder must see the higher node as a member of ITS cluster");
    }
    /// <summary>
    /// THE HEADLINE #33 CASE. Both nodes cold-start truly simultaneously with the guard on. Without the
    /// guard each would run FirstSeedNodeProcess, race, and form its own 1-node cluster (split brain).
    /// With it, the founder forms and the higher node probes-then-joins, so the pair converges to
    /// EXACTLY ONE cluster of two — no two oldest, no dual singletons.
    /// </summary>
    [Fact]
    public async Task Both_nodes_cold_starting_together_form_exactly_one_cluster()
    {
        var low = TwoNodeClusterFixture.GetFreeTcpPort();
        var high = TwoNodeClusterFixture.GetFreeTcpPort();
        var founderPort = Math.Min(low, high);
        var higherPort = Math.Max(low, high);
        // Start both without serialization — as a shared power event does.
        var founderTask = StartGuardedNodeAsync(founderPort, higherPort);
        var higherTask = StartGuardedNodeAsync(higherPort, founderPort);
        var founder = await founderTask;
        var higher = await higherTask;
        Assert.True(await WaitForUpAsync(founder, 2, TimeSpan.FromSeconds(60)),
            "the pair must converge to one 2-member cluster, not two 1-member clusters");
        Assert.True(await WaitForUpAsync(higher, 2, TimeSpan.FromSeconds(60)),
            "the higher node must be in the SAME cluster as the founder");
    }
    public Task InitializeAsync() => Task.CompletedTask;
    public async Task DisposeAsync()
    {
        foreach (var c in _coordinators)
        {
            try { await c.StopAsync(CancellationToken.None); } catch { /* teardown */ }
        }
        foreach (var s in _systems)
        {
            try { await s.Terminate().WaitAsync(TimeSpan.FromSeconds(10)); } catch { /* teardown */ }
        }
    }
 }
Author	SHA1	Message	Date
Joseph Doherty	b3dc17a5fe	Merge feat/site-node-health: serve health on site nodes + shared active-node check (Health 0.3.0)	2026-07-24 13:35:27 -04:00
Joseph Doherty	1c9159a7ce	fix(tests): qualify Health page type — Communication.Health namespace shadowed it	2026-07-24 13:35:24 -04:00
Joseph Doherty	5d4853a1f0	refactor(health): adopt the shared active-node check (Health 0.3.0) OldestNodeActiveHealthCheck existed because the shared ActiveNodeHealthCheck selected by cluster leadership (review 01 [High]): leadership is address-ordered and diverges from singleton placement after a restart, and during a partition both sides compute themselves leader so Traefik served both. Health 0.3.0 makes the shared check age-based — it is now this host's own rule, promoted — so the private copy is deleted and central registers the shared type. ActiveNodeEvaluator, the "THE single definition of active node" this repo already maintained, now delegates to the shared ClusterActiveNode rather than re-implementing it. That keeps the delivery gate, the heartbeat IsActive stamp, the inbound-API gate and the /health/active tier on one rule, and it means the rule is shared with OtOpcUa, which had independently written a third copy. Communication takes a ZB.MOM.WW.Health.Akka reference for it; the layering trade-off is recorded at the PackageReference. SitePairActiveNodeHealthCheck is deliberately KEPT. It is not a duplicate of the rule — it is a thin adapter over the site's own IClusterNodeProvider, which scopes to site-{SiteId} and is itself now backed by ClusterActiveNode. Registering the shared check directly would have to re-derive the site role and would lose the property that the tier and singleton placement come from the same provider. Central stays unscoped: every central member competes for one active slot. No behaviour change on any node — same rule, one implementation instead of two. Verified: Host.Tests 439/439, Communication ActiveNode 2/2.	2026-07-24 13:21:58 -04:00
Joseph Doherty	c1216044a4	docs(cluster): document the #33 bootstrap guard; disabled BootstrapGuard block in Host appsettings Component-ClusterInfrastructure.md gains a 'Simultaneous cold start — the bootstrap guard' subsection (dark switch, founder/probe semantics, config table, accepted trade) plus a forward reference from Dual-Node Recovery. Host appsettings.Central/Site.json carry a disabled BootstrapGuard block with a _bootstrapGuard note for operator discoverability. CLAUDE.md Cluster & Failover note added.	2026-07-24 08:57:44 -04:00
Joseph Doherty	248676ed16	feat(cluster): simultaneous-cold-start split-brain guard (#33 ) Port OtOpcUa's bootstrap guard (lmxopcua d1dac87f) to ScadaBridge's BuildHocon bootstrap. Both pair nodes are self-first seeds, so a true simultaneous cold start (shared site power event) races FirstSeedNodeProcess on both and forms two 1-node clusters that never merge. Opt-in dark switch ScadaBridge:Cluster:BootstrapGuard:Enabled (default off, guard-off behavior byte-identical). When on: BuildHocon emits an empty seed list so Akka does not auto-join, and ClusterBootstrapCoordinator (IHostedService, registered in both the Central and Site composition roots) picks the join order from ClusterBootstrapGuard's pure decision core — the lower canonical host:port is the founder (self-first, forms immediately); the higher node TCP-probes the founder up to PartnerProbeSeconds and joins peer-first if reachable, else self-first (cold-start-alone preserved). Decides before a single JoinSeedNodes, never re-forms mid-handshake. Review notes carried over: case-insensitive founder tie-break; fail-fast validation of probe timings when enabled; higher-node-cold-start-alone covered by a real-ActorSystem test. 21 unit + 5 real-cluster tests (incl. the headline both-cold-start-together-form-one-cluster).	2026-07-24 08:55:48 -04:00
Joseph Doherty	47850c0f53	feat(health): serve MapZbHealth on site nodes Site nodes served no health surface at all — gRPC on the HTTP/2-only listener and /metrics on the HTTP/1.1 one — so nothing outside the cluster could ask a site node whether it was ready or which half of the pair was active. The family overview dashboard probes every instance the same way, and this is the one gap. Three checks, registered in SiteServiceRegistration.Configure (not Program.cs, so the composition-root tests that build this graph actually cover them) and mapped by app.MapZbHealth() on the site's HTTP/1.1 listener (default :8084) alongside /metrics: akka-cluster [Ready] the shared AkkaClusterHealthCheck. Also carries the cluster-view data (leader/memberCount/...) the dashboard reads, free with ZB.MOM.WW.Health 0.2.0. localdb [Ready] NEW SiteLocalDbHealthCheck — SELECT 1 through the registered ILocalDb. Site has no EF context; central's DatabaseHealthCheck<ScadaBridgeDbContext> is central-only. Replication state rides along as data ENRICHMENT only: replication is default-OFF, so failing on it would mark every correctly-configured node unready. active-node [Active] NEW SitePairActiveNodeHealthCheck, delegating to IClusterNodeProvider.SelfIsPrimary. Deliberately NOT central's OldestNodeActiveHealthCheck: that one calls SelfIsOldest(cluster) with no role argument and would compute "oldest" across the wrong member set on a mesh carrying more than one site. Anonymous, as central's are: the site pipeline runs no authentication middleware and has no FallbackPolicy, so nothing extra was needed. Also bumps ZB.MOM.WW.Health* 0.1.0 -> 0.2.0 (central gains data.leader for free). Tests: SiteHealthCheckTests builds the REAL site container and activates every registration through its factory (exact-set names, one tier tag each, resolved types, central-only checks absent behind a positive control) plus behaviour for both new checks. SiteHealthEndpointTests boots the real site Program over WebApplicationFactory and proves the endpoints are MAPPED — without it, deleting MapZbHealth would leave every registration test green while site nodes 404'd. Prereq for scadaproj docs/plans/2026-07-22-overview-dashboard-impl-plan.md Phase 1.	2026-07-24 05:54:30 -04:00
Joseph Doherty	bb138d5254	docs: add env_variables.md — full environment-variable inventory Deep-scan catalog of every environment variable ScadaBridge reads (Host, CLI, DelmiaNotifier, EF design-time tooling, test/CI harness) plus supporting infra containers. Each entry carries scope (Runtime/Design-time/Test/Infra), whether a shipped appsettings key already exists for it, consumer, purpose, potential values, and required-ness. Compiled from GetEnvironmentVariable call sites, the Host AddEnvironmentVariables() source, docker/docker-env2 compose, install.ps1, and all appsettings*.json.	2026-07-24 05:02:24 -04:00
Joseph Doherty	9a12826172	docs(plans): OtOpcUa v3 native-alarm B/C live-gate PASS (#14 ) — no code needed; item C premise obsolete (Gitea #17 )	2026-07-24 04:33:20 -04:00
Joseph Doherty	86d129de6c	fix(dcl): harden endpoint host-rewrite — IPv6 brackets + portless URLs Code review of `a1abbff7` found two edge cases: a discovery/advertised URL with no explicit port emitted a malformed ':-1' (opc.tcp has no Uri default port), and an IPv6 literal host could lose/double its brackets. Omit the port when neither URL carries one; bracket an IPv6 host only when missing. +4 tests (12 total).	2026-07-23 16:58:44 -04:00
Joseph Doherty	dbec3ee263	docs(plans): OtOpcUa v3 raw-path live-gate PASS (#14 ) — fix validated live, B/C deferred	2026-07-23 16:54:48 -04:00
Joseph Doherty	a1abbff760	fix(dcl): rewrite advertised OPC UA endpoint host to the reachable one An OPC UA server advertises its base address from its own config, not the route the client took — commonly a 0.0.0.0 wildcard bind or an internal container/NAT hostname. The OPC Foundation session dials EndpointDescription.EndpointUrl verbatim, so a 0.0.0.0 advertisement resolved to the client's own loopback and the connect failed. RealOpcUaClient now swaps the advertised authority for the reachable host/port (ConnectAsync + VerifyEndpointAsync), preserving scheme+path; no-op when already reachable. Surfaced live-gating OtOpcUa v3 (#14).	2026-07-23 16:43:45 -04:00