3 Commits

Author SHA1 Message Date
Joseph Doherty c1216044a4 docs(cluster): document the #33 bootstrap guard; disabled BootstrapGuard block in Host appsettings
Component-ClusterInfrastructure.md gains a 'Simultaneous cold start — the
bootstrap guard' subsection (dark switch, founder/probe semantics, config table,
accepted trade) plus a forward reference from Dual-Node Recovery. Host
appsettings.Central/Site.json carry a disabled BootstrapGuard block with a
_bootstrapGuard note for operator discoverability. CLAUDE.md Cluster & Failover
note added.
2026-07-24 08:57:44 -04:00
Joseph Doherty 248676ed16 feat(cluster): simultaneous-cold-start split-brain guard (#33)
Port OtOpcUa's bootstrap guard (lmxopcua d1dac87f) to ScadaBridge's
BuildHocon bootstrap. Both pair nodes are self-first seeds, so a true
simultaneous cold start (shared site power event) races FirstSeedNodeProcess
on both and forms two 1-node clusters that never merge.

Opt-in dark switch ScadaBridge:Cluster:BootstrapGuard:Enabled (default off,
guard-off behavior byte-identical). When on: BuildHocon emits an empty seed
list so Akka does not auto-join, and ClusterBootstrapCoordinator (IHostedService,
registered in both the Central and Site composition roots) picks the join order
from ClusterBootstrapGuard's pure decision core — the lower canonical host:port
is the founder (self-first, forms immediately); the higher node TCP-probes the
founder up to PartnerProbeSeconds and joins peer-first if reachable, else
self-first (cold-start-alone preserved). Decides before a single JoinSeedNodes,
never re-forms mid-handshake.

Review notes carried over: case-insensitive founder tie-break; fail-fast
validation of probe timings when enabled; higher-node-cold-start-alone covered
by a real-ActorSystem test. 21 unit + 5 real-cluster tests (incl. the headline
both-cold-start-together-form-one-cluster).
2026-07-24 08:55:48 -04:00
Joseph Doherty bb138d5254 docs: add env_variables.md — full environment-variable inventory
Deep-scan catalog of every environment variable ScadaBridge reads (Host, CLI,
DelmiaNotifier, EF design-time tooling, test/CI harness) plus supporting infra
containers. Each entry carries scope (Runtime/Design-time/Test/Infra), whether a
shipped appsettings key already exists for it, consumer, purpose, potential
values, and required-ness. Compiled from GetEnvironmentVariable call sites, the
Host AddEnvironmentVariables() source, docker/docker-env2 compose, install.ps1,
and all appsettings*.json.
2026-07-24 05:02:24 -04:00
15 changed files with 1103 additions and 5 deletions
+1
View File
@@ -220,6 +220,7 @@ Other peers in the `scadaproj` family (see `scadaproj/CLAUDE.md` for details): `
- Automatic dual-node recovery from persistent storage.
- **Active/standby is decided by `ActiveNodeEvaluator.SelfIsOldestUp`, never by cluster leadership** — see the Architecture note above. `/health/active` is **central-only** (site nodes map no `/health/*` at all) and backs both Traefik's active-node routing and `IActiveNodeGate`, so the proxy and the Inbound API always agree on which node is active. Central never needs to know which *site* node is active: ClusterClient contact rotation reaches either receptionist and the site-internal `ClusterSingletonProxy` lands the work on the active node for free. The **exception is gRPC**, which picks `GrpcNodeAAddress`/`GrpcNodeBAddress` explicitly and flips on error.
- **Seed-node ordering: every node lists ITSELF first (decision 2026-07-22) — the boot-alone gap is CLOSED.** Only `seed-nodes[0]` may self-join to form a new cluster (Akka runs `FirstSeedNodeProcess` for it, `JoinSeedNodeProcess` — which can never form one — for everyone else). All 14 shipped node appsettings now lead with the node's own address, so any node can cold-start alone and become operational unattended (~5s, `seed-node-timeout`); `StartupValidator` fails the boot if the ordering is broken (compares host AND port; Akka does no DNS canonicalisation). Two nodes cold-starting together while mutually reachable converge on ONE cluster via the `InitJoin` handshake — they split only under a genuine boot-time partition, the same class auto-down accepts. **An external self-form timer (`Cluster.Join(SelfAddress)` after a window) was implemented and REJECTED:** it sits outside the join handshake, so on a routine standby restart — where the peer is alive but the join is stalled behind removal of the node's own stale incarnation — it fires mid-join and permanently splits the pair (measured: still split after 90s). Regression tests: `SelfFirstSeedBootstrapTests`. The keep-oldest active-crash total outage was separately closed by the auto-down decision. See `docs/requirements/Component-ClusterInfrastructure.md` → Seed Node Ordering.
- **Simultaneous-cold-start split-brain guard (opt-in, Gitea #33) — the residual self-first cost, closed.** Self-first-on-both means a *truly simultaneous* cold start (shared power/hypervisor event) races `FirstSeedNodeProcess` on BOTH nodes → two 1-node clusters that never merge (in-process loopback tests converge and hide it; real parallel VM starts don't — OtOpcUa reproduced it live). Ported from OtOpcUa (`lmxopcua` `d1dac87f`): a **dark switch `ScadaBridge:Cluster:BootstrapGuard:Enabled` (default OFF, guard-off byte-identical)**. On: `BuildHocon` emits an EMPTY seed list (Akka does not auto-join) and `ClusterBootstrapCoordinator` (`IHostedService`, registered in BOTH the Central and Site composition roots) issues ONE reachability-gated `JoinSeedNodes` — the pure `ClusterBootstrapGuard` core makes the lower canonical `host:port` the **founder** (self-first, forms immediately), the higher node TCP-probes the founder up to `PartnerProbeSeconds` (25s) and joins **peer-first** if reachable else **self-first** (cold-start-alone preserved). Decided BEFORE a single join, **never re-forms mid-handshake** (that was the rejected self-form-timer's flaw); case-insensitive founder tie-break; probe timings validated `>0` when enabled. Residual accepted trade: founder dies in the probe→join window → higher node hangs unjoined, coordinator WARNs, a restart recovers it. Tests: `ClusterBootstrapGuardTests` (pure) + `ClusterBootstrapCoordinatorTests` (real-ActorSystem, incl. both-cold-start-together-form-one-cluster). See `docs/requirements/Component-ClusterInfrastructure.md` → Simultaneous cold start.
### UI & Monitoring
- Central UI: Blazor Server (ASP.NET Core + SignalR) with Bootstrap CSS. No third-party component frameworks (no Blazorise, MudBlazor, Radzen, etc.). Build custom Blazor components for tables, grids, forms, etc.
@@ -137,6 +137,27 @@ Behavior, covered by `SelfFirstSeedBootstrapTests` (real in-process clusters bui
The docker failover drill (`docker/failover-drill.sh`) proves both directions: `standby` mode kills the younger node (active untouched, zero routing blips); `active` mode kills the active/oldest node and asserts the survivor **takes over while the victim is still down**.
### Simultaneous cold start — the bootstrap guard (opt-in, Gitea #33)
Self-first-on-both has a residual cost. When **both** nodes cold-start at the same instant, each is `seed-nodes[0]` for itself, so **each runs `FirstSeedNodeProcess`**, times out waiting for the other, and forms its **own** single-node cluster — a split brain (two oldest members, two singletons, dual-active) that does **not** auto-merge and persists until an operator restarts one side. In-process loopback tests usually converge (the `InitJoin` handshake resolves before either self-join deadline, so row 3 above passes), which is exactly why the split hides until a real deployment powers up both VMs truly in parallel — a shared power event or a hypervisor host reboot. The sister project **OtOpcUa** (same `ZB.MOM.WW.*` Akka topology) reproduced it reliably on its docker rig.
The **bootstrap guard** eliminates the split without giving up cold-start-alone. It is an **opt-in dark switch**`ScadaBridge:Cluster:BootstrapGuard:Enabled`, **default off**, so a node with the guard disabled keeps Akka's config-driven self-first auto-join byte-identical. When enabled, the node starts with **no config seed nodes** (`AkkaHostedService.BuildHocon` emits an empty seed list, so Akka does not auto-join) and a coordinator (`ClusterBootstrapCoordinator`, an `IHostedService` registered in both the Central and Site composition roots) picks the join order after a reachability probe:
- The node with the lexicographically **lower** canonical `host:port` is the **preferred founder**: it joins self-first and forms immediately if no peer answers — no probe, no delay. The tie-break is **case-insensitive** (a hostname-casing mismatch between the two configs must never make both think they are the founder and re-open the split).
- The **higher** node TCP-probes its partner's Akka port (a node binds its port well before it forms) up to `PartnerProbeSeconds`: **reachable ⇒ peer-first** (join the founder, never race it); **unreachable after the window ⇒ self-first** (the partner is genuinely down, so cold-start-alone is preserved for the higher node too).
- The order is decided **before** a single `Cluster.JoinSeedNodes`, from an explicit reachability signal. The coordinator **never re-forms mid-handshake** — that was the failure mode of the rejected self-form timer above (a `Join(self)` on a bare timeout is not ignored mid-handshake; it wins and islands a node).
**Residual trade-off (accepted, operator-visible).** Once the higher node commits peer-first it cannot self-form (`JoinSeedNodeProcess` retries forever). If the founder dies in the small probe→join window, the higher node hangs unjoined; the coordinator logs a WARNING after a bounded grace, and a **restart** recovers it (the guard re-runs, finds the founder down, and forms alone). The guard deliberately does not re-decide mid-handshake.
| Key | Default | Meaning |
|---|---|---|
| `BootstrapGuard:Enabled` | `false` | Dark switch. Only meaningful on a node that is one of its own two pair seeds; inert elsewhere (single-node install, self absent, 3+ seeds). |
| `BootstrapGuard:PartnerProbeSeconds` | `25` | Higher node's probe window. Must comfortably exceed the founder's process-start-to-Akka-bind time, or a slow-but-alive founder is mistaken for dead and the split re-opens. Validated `> 0` at boot when the guard is on. |
| `BootstrapGuard:PartnerProbeIntervalMs` | `500` | Interval between probes. Validated `> 0` when enabled. |
| `BootstrapGuard:ProbeConnectTimeoutMs` | `1000` | Per-probe TCP connect timeout. Validated `> 0` when enabled. |
Decision core (`ClusterBootstrapGuard`) is a pure, fully unit-tested function; the probe + `JoinSeedNodes` runtime (`ClusterBootstrapCoordinator`) is covered by real-ActorSystem tests including the load-bearing higher-node-cold-start-alone case and the headline both-cold-start-together-form-one-cluster case (`ClusterBootstrapCoordinatorTests`). The alternative is purely operational (stagger the two VMs' service-manager start / start the founder first); the docker rig's compose `depends_on` does that today, but that does not exist on the real co-located VMs — the guard is the production-faithful fix. Ported from OtOpcUa (`lmxopcua` commit `d1dac87f`); implementing it in both products keeps their failure/recovery model identical, since a shared power event hits both pairs at once.
### Manual Failover (admin-triggered)
An Administrator can swap the central pair's roles deliberately — for planned maintenance on the active node, or to move singletons off a node behaving badly without waiting for a crash. The control is the **Trigger failover** button on the central-cluster card of the Health dashboard (`/monitoring/health`).
@@ -191,7 +212,7 @@ These values balance failover speed with stability — fast enough that data col
If both nodes in a cluster fail simultaneously (e.g., site power outage):
1. **No manual intervention required.** Since both nodes are configured as seed nodes, whichever node starts first forms a new cluster. The second node joins when it starts.
1. **No manual intervention required.** Since both nodes are configured as seed nodes, whichever node starts first forms a new cluster (self-first ordering) and the second node joins when it starts. If both power up truly simultaneously, each self-first seed can run `FirstSeedNodeProcess` and form its own 1-node cluster — a split brain; enable the opt-in **bootstrap guard** (Gitea #33, see *Simultaneous cold start* above) on co-located VM pairs that share a power domain to make them converge on one cluster instead.
2. **State recovery** (each node has its own local copy of all required data):
- **Site clusters**: The Deployment Manager singleton reads deployed configurations from local SQLite and re-creates the full Instance Actor hierarchy. Store-and-forward buffers are already persisted locally. Alarm states re-evaluate from incoming data values.
- **Central cluster**: All state is in MS SQL (configuration database). The active node resumes normal operation.
+169
View File
@@ -0,0 +1,169 @@
# Environment Variables
A deep-scan inventory of every environment variable ScadaBridge reads — the Host, the CLI, the
DelmiaNotifier client, EF design-time tooling, and the test/CI harness — plus the supporting infra
containers. Compiled by scanning `Environment.GetEnvironmentVariable` call sites, the Host's
`AddEnvironmentVariables()` config source, `docker/` + `docker-env2/` compose files,
`deploy/wonder-app-vd03/install.ps1`, and every `appsettings*.json`.
**Scan date:** 2026-07-24 · **Branch:** `main`
Two columns qualify each variable:
- **Scope** — where it is read: **Runtime** (the running product — Host / CLI / DelmiaNotifier),
**Design-time** (`dotnet ef` tooling), **Test** (unit / integration / E2E / perf harness only, never
the shipped product), or **Infra** (a sidecar container, not ScadaBridge code).
- **In appsettings?** — whether a shipped `appsettings*.json` already carries the same config key
(so the env var is an *override* or *secret substitute*), versus being env-only. `${secret:...}`
means the key is present but its value is a secret token, not a literal.
## How configuration resolves (read this first)
The Host builds its configuration in `src/ZB.MOM.WW.ScadaBridge.Host/Program.cs` in this precedence
order (later overrides earlier):
1. `appsettings.json`
2. `appsettings.{SCADABRIDGE_CONFIG}.json` (role file — `Central` or `Site`)
3. **`.AddEnvironmentVariables()`** ← every variable below in the "config-override" table
4. `.AddCommandLine(args)`
Then a **pre-host secrets expander** resolves any `${secret:NAME}` tokens left in the merged config,
reading the encrypted secrets store whose master key comes from **`ZB_SECRETS_MASTER_KEY`**. Because
step 3 runs *before* expansion, a whole-key env override (e.g. `ScadaBridge__Database__ConfigurationDb`)
wins over — and short-circuits — the corresponding `${secret:...}` token. That is the deliberate
"rollback / dev" path the docker rigs use.
**Config-key → env-var mapping:** replace each `:` in a config path with `__` (double underscore).
So `ScadaBridge:Communication:GrpcPsk` is set via `ScadaBridge__Communication__GrpcPsk`. Any config
key in the app can be overridden this way; the table below lists only the ones with an established
purpose (secrets kept off disk, or dev-rig wiring).
---
## 1. Role & host environment selection
| Variable | Scope | In appsettings? | Consumed by | Purpose | Potential values | Required |
|---|---|---|---|---|---|---|
| `SCADABRIDGE_CONFIG` | Runtime | No (it *selects* the `appsettings.{value}.json` file) | Host (`Program.cs:42`) | Selects the role-specific overlay. **Primary role switch.** | `Central`, `Site` | Effectively yes (falls back to `DOTNET_ENVIRONMENT`, then `Production`) |
| `DOTNET_ENVIRONMENT` | Runtime | No | Host (`Program.cs:43`) | Fallback role selector when `SCADABRIDGE_CONFIG` is unset; generic .NET environment name. | `Central`, `Site`, `Development`, `Production` | No |
| `ASPNETCORE_ENVIRONMENT` | Runtime | No | ASP.NET Core + `DisableLoginGuard` (`Program.cs:438`) | Standard ASP.NET environment. `DisableLogin` is refused unless this is `Development`. | `Development`, `Production` | No (defaults to `Production` behavior) |
| `ASPNETCORE_URLS` | Runtime | No (Kestrel binding; not in these appsettings) | Kestrel | HTTP bind addresses. Docker sets `http://+:5000`; install.ps1 sets `http://+:{WebPort}` (8085). | e.g. `http://+:5000`, `http://+:8085` | No |
> `SCADABRIDGE_CONFIG` and `ASPNETCORE_ENVIRONMENT` are independent. The docker rig runs
> `SCADABRIDGE_CONFIG=Central` **and** `ASPNETCORE_ENVIRONMENT=Development` simultaneously.
## 2. Secrets store master key (KEK)
| Variable | Scope | In appsettings? | Consumed by | Purpose | Potential values | Required |
|---|---|---|---|---|---|---|
| `ZB_SECRETS_MASTER_KEY` | Runtime | Name only (`Secrets:MasterKey:EnvVarName` in `appsettings.json`; the value is never committed) | Host secrets expander | The KEK that decrypts the `ZB.MOM.WW.Secrets` store, from which every `${secret:...}` token resolves. A missing reference fails closed (`SecretNotFoundException`) before any SQL/LDAP/cluster wiring. | an opaque base64/hex master key | Yes on any node whose appsettings uses `${secret:...}` (shipped Central + Site do). Docker rigs avoid it with whole-key overrides. |
The variable *name* is itself configurable via `Secrets:MasterKey:EnvVarName` (defaults to
`ZB_SECRETS_MASTER_KEY`). See `docs/operations/2026-07-16-secrets-clustered-master-key.md`.
## 3. Config-override env vars (`ScadaBridge__*` whole-key)
Map onto config keys via the `:``__` rule. Keep secrets off disk (production, via a secret manager)
or wire the dev rigs without a KEK. The `__site-x` / `__site-a` leaf is the literal `siteId`.
All are **Runtime** scope (the Host reads them; Host integration-test fixtures set the same names, but
that is the same variable, not a separate one).
| Variable | In appsettings? | Config key | Purpose | Potential values | Required |
|---|---|---|---|---|---|
| `ScadaBridge__Database__ConfigurationDb` | Yes — `${secret:sql/scadabridge/configdb-connection}` in `appsettings.Central.json` | `ScadaBridge:Database:ConfigurationDb` | Central config DB connection string (MS SQL). Central-only. | `Server=host,1433;Database=ScadaBridgeConfig;User Id=...;Password=...;TrustServerCertificate=true` | Yes on Central — via env **or** the `${secret}` |
| `ScadaBridge__Database__MachineDataDb` | No in shipped src appsettings; present in the deploy overlay (`deploy/wonder-app-vd03/appsettings.Central.json`, as `${...}`). install.ps1 derives it as `<DbName>MachineData`. | `ScadaBridge:Database:MachineDataDb` | Central machine-data DB connection string. Central-only. | connection string | Yes on Central |
| `ScadaBridge__Security__JwtSigningKey` | Yes — `${secret:security/scadabridge/jwt-signing-key}` in `appsettings.Central.json` | `ScadaBridge:Security:JwtSigningKey` | HMAC-SHA256 signing key for the cookie-embedded session JWT. | ≥32-char secret | Yes on Central |
| `ScadaBridge__Security__Ldap__ServiceAccountPassword` | Yes — `${secret:ldap/scadabridge/service-account-password}` in `appsettings.Central.json` | `ScadaBridge:Security:Ldap:ServiceAccountPassword` | LDAP bind-account password. (Renamed from the old flat `...__LdapServiceAccountPassword`.) | the bind password | Yes on Central if LDAP bind is used |
| `ScadaBridge__InboundApi__ApiKeyPepper` | No — env/secret only (not in any shipped appsettings) | `ScadaBridge:InboundApi:ApiKeyPepper` | Per-environment pepper for the inbound API-key peppered-HMAC verifier. Hard Central startup requirement since 2026-06-03. | ≥16-char secret, unique per environment | Yes on Central |
| `ScadaBridge__Communication__GrpcPsk` | Yes — `${secret:SB-GRPC-PSK-site-1}` in `appsettings.Site.json` | `ScadaBridge:Communication:GrpcPsk` | **Site side** of the per-site gRPC control-plane PSK. StartupValidator refuses to boot a Site node without it; both pair nodes carry the same value. | per-site secret (e.g. `dev-grpc-psk-docker-site-a`); prod `${secret:SB-GRPC-PSK-<siteId>}` | Yes on Site |
| `ScadaBridge__Communication__SitePsks__<siteId>` | No in shipped src appsettings (central normally resolves the `SB-GRPC-PSK-<siteId>` secret); present only in the docker-compose env blocks | `ScadaBridge:Communication:SitePsks:<siteId>` | **Central side** of the same key: the value central presents when dialling that site. Must equal the site's `GrpcPsk`. Override for hosts without a master secret store. | matches the site's `GrpcPsk` | On Central, either this **or** the `SB-GRPC-PSK-<siteId>` store secret |
Docker dev values (dev-only-insecure, committed in `docker/docker-compose.yml`):
`ScadaBridge__Communication__SitePsks__site-a = dev-grpc-psk-docker-site-a` (also `site-b`, `site-c`);
`docker-env2` uses `SitePsks__site-x = dev-grpc-psk-docker-env2-site-x`.
> Any other config key is overridable the same way (e.g. `ScadaBridge__Cluster__SplitBrainResolverStrategy`,
> `ScadaBridge__Logging__MinimumLevel`) — these are the standard .NET config binder, not "special" env
> vars. Prefer `appsettings` files for non-secret values.
### `${secret:...}` tokens (resolved from the store, NOT env vars)
Not environment variables — config values resolved by the secrets expander using `ZB_SECRETS_MASTER_KEY`.
All are **Runtime** scope and, by definition, present **In appsettings** as tokens.
| Token (in `appsettings`) | Meaning |
|---|---|
| `${secret:sql/scadabridge/configdb-connection}` | Central config DB connection string |
| `${secret:ldap/scadabridge/service-account-password}` | LDAP bind password |
| `${secret:security/scadabridge/jwt-signing-key}` | JWT signing key |
| `${secret:SB-GRPC-PSK-<siteId>}` | Per-site gRPC PSK (site side); central resolves `SB-GRPC-PSK-{siteId}` at channel-build time |
## 4. CLI (`scadabridge` / `ZB.MOM.WW.ScadaBridge.CLI`)
Resolution order per value: command-line flag → env var → `~/.scadabridge/config.json`. The CLI does
**not** read `appsettings.json` — its config file is `~/.scadabridge/config.json` (a separate schema).
| Variable | Scope | In appsettings? | Purpose | Potential values | Required |
|---|---|---|---|---|---|
| `SCADABRIDGE_MANAGEMENT_URL` | Runtime (CLI) | No — CLI config file has `managementUrl` instead | Management API base URL. Overridden by `--url`; overrides the config file. Missing all three → exit `1`, `NO_URL`. | e.g. `http://localhost:9000` | One of flag/env/config |
| `SCADABRIDGE_FORMAT` | Runtime (CLI) | No — CLI config file has `defaultFormat` instead | Default output format. Overridden by `--format`. | `json`, `table` | No (defaults to `json`) |
| `SCADABRIDGE_USERNAME` | Runtime (CLI) | No | LDAP username fallback when `--username` absent. | e.g. `multi-role` | No (some commands need auth) |
| `SCADABRIDGE_PASSWORD` | Runtime (CLI) | No | LDAP password fallback when `--password` absent. **Preferred over `--password`** — avoids leaking into process listings / shell history. | the user's password | No (some commands need auth) |
## 5. DelmiaNotifier client (`WWNotifier.exe`)
| Variable | Scope | In appsettings? | Purpose | Potential values | Required |
|---|---|---|---|---|---|
| `SCADABRIDGE_API_KEY` | Runtime (client) | No — the notifier's `appsettings.json` holds only `BaseUrls`/`TimeoutSeconds`/`LogPath`; the key is **env-only by design** | Inbound API key sent as `X-API-Key`. Read only from the environment, never a file. Missing/empty → prints `API key not configured (SCADABRIDGE_API_KEY)`, exit `-1`, no HTTP attempt. | e.g. `sbk_...` | Yes |
> This is also why `System.Environment` is wholly forbidden in the script trust model — it would let a
> site/inbound script exfiltrate `SCADABRIDGE_API_KEY` and other host env secrets.
## 6. EF Core design-time tooling
| Variable | Scope | In appsettings? | Purpose | Potential values | Required |
|---|---|---|---|---|---|
| `SCADABRIDGE_DESIGNTIME_CONNECTIONSTRING` | Design-time | Fallback for `ScadaBridge:Database:ConfigurationDb` — the factory reads that appsettings key first, then this env var | Connection string for `dotnet ef` migrations tooling when the Host's appsettings isn't the source. No hardcoded fallback — missing → actionable failure. | a config DB connection string | Only for `dotnet ef` when appsettings is unavailable |
## 7. Test / CI-only
Not read by the running product — the test harness only. Most default gracefully (skip, or use a
local docker value) when unset.
| Variable | Scope | In appsettings? | Consumed by | Purpose | Potential values |
|---|---|---|---|---|---|
| `SCADABRIDGE_MSSQL_TEST_CONN` | Test | No | `MsSqlMigrationFixture`, AuditLog migration tests | Admin (sa) connection to a running MS SQL; absent → local docker sa default; drives `Skip.If`. | sa connection string |
| `SCADABRIDGE_PLAYWRIGHT_DB` | Test | No | `PlaywrightDbConnection` | DB connection for Playwright E2E seeding; absent → `localhost:1433`. | connection string |
| `SCADABRIDGE_CLI_DLL` | Test | No | `CliRunner` | Absolute path to the built CLI dll for Playwright when auto-probing fails. | path to `...CLI.dll` |
| `SCADABRIDGE_AUDIT_FILTER_4KB_P95_US` | Test | No | `HotPathLatencyTests` | Override P95 latency threshold (µs) for the 4 KB audit-filter perf test. | positive number (default 5) |
| `SCADABRIDGE_AUDIT_FILTER_RAW_P95_US` | Test | No | `HotPathLatencyTests` | Override P95 threshold for the raw-capture perf test. | positive number |
| `SCADABRIDGE_AUDIT_FILTER_NOOP_P95_US` | Test | No | `HotPathLatencyTests` | Override P95 threshold for the no-op fast-path perf test. | positive number (default 5) |
| `HOME` / `USERPROFILE` | Test | No | `CliConfig` tests | Redirected to locate `~/.scadabridge/config.json` under test. | temp dir path |
The Host test fixtures (`CentralDbTestEnvironment`, `ScadaBridgeWebApplicationFactory`,
`StartupValidationTests`) set the **section-3 config-override vars** (`ScadaBridge__Database__*`,
`ScadaBridge__InboundApi__ApiKeyPepper`, etc.) and `DOTNET_ENVIRONMENT=Central` at runtime; those are
the same variables already documented above, not new ones.
## 8. Supporting infrastructure containers (not the app)
Set on the sidecar/infra containers in `infra/docker-compose.yml`, consumed by those images, not by
ScadaBridge code — included for completeness when standing up a rig.
| Variable | Scope | In appsettings? | Container | Purpose | Value used |
|---|---|---|---|---|---|
| `MSSQL_SA_PASSWORD` | Infra | No | `scadabridge-mssql` | SQL Server `sa` password. | `ScadaBridge_Dev1#` (dev-only) |
| `MSSQL_PID` | Infra | No | `scadabridge-mssql` | SQL Server edition. | `Developer` |
---
## Quick reference: minimum sets to boot a real deploy
**Central node** (via env, or the `${secret:...}` equivalents + `ZB_SECRETS_MASTER_KEY`):
`SCADABRIDGE_CONFIG=Central`, `ScadaBridge__Database__ConfigurationDb`,
`ScadaBridge__Database__MachineDataDb`, `ScadaBridge__Security__JwtSigningKey`,
`ScadaBridge__Security__Ldap__ServiceAccountPassword`, `ScadaBridge__InboundApi__ApiKeyPepper`, and
for each managed site either the `SB-GRPC-PSK-<siteId>` secret or `ScadaBridge__Communication__SitePsks__<siteId>`.
**Site node:** `SCADABRIDGE_CONFIG=Site` and `ScadaBridge__Communication__GrpcPsk` (or its
`${secret:SB-GRPC-PSK-<siteId>}` form). StartupValidator hard-fails without the PSK.
@@ -0,0 +1,155 @@
using System.Text.RegularExpressions;
namespace ZB.MOM.WW.ScadaBridge.ClusterInfrastructure;
/// <summary>
/// Pure decision logic for the simultaneous-cold-start split-brain guard (Gitea #33).
/// </summary>
/// <remarks>
/// <para>
/// Both nodes of a 2-node pair are self-first seeds so <b>either</b> can cold-start alone when
/// its partner is dead (see <see cref="ClusterOptions.SeedNodes"/>). The cost is that when
/// BOTH cold-start at the same instant, each runs Akka's <c>FirstSeedNodeProcess</c>, times out
/// waiting for the other, and forms its OWN single-node cluster — a split brain (two oldest
/// nodes, two singletons, dual-active). Two independent clusters do not auto-merge, so the
/// split persists until an operator restarts one side. It bites on any event that powers up
/// both site VMs together (site power restoration, hypervisor host reboot).
/// </para>
/// <para>
/// This guard breaks the symmetry deterministically without giving up cold-start-alone. The
/// node with the lexicographically <b>lower</b> canonical address is the <i>preferred
/// founder</i>: it always uses self-first order and forms immediately. The <b>higher</b> node
/// first probes whether its partner's Akka endpoint is reachable (see
/// <see cref="ClusterBootstrapCoordinator"/>): reachable ⇒ peer-first order so it JOINS the
/// founder rather than racing it; unreachable after the probe window ⇒ the partner is genuinely
/// down, so it falls back to self-first and forms alone. Exactly one node founds when both are
/// present; the lower node always founds when it starts alone.
/// </para>
/// <para>
/// <b>Residual trade-off (accepted).</b> Once the higher node observes the partner reachable it
/// commits to peer-first — Akka's <c>JoinSeedNodeProcess</c> retries <c>InitJoin</c> forever and
/// never self-forms. If the founder dies in the small window between the probe succeeding and
/// the join completing, the higher node hangs unjoined until it is restarted (a restart re-runs
/// the guard, finds the founder unreachable, and self-forms). We deliberately do NOT re-decide
/// mid-handshake — that is exactly the retired <c>SelfFormAfter</c> failure mode. Instead
/// <see cref="ClusterBootstrapCoordinator"/> makes the hang operator-visible with a warning if
/// the node has not come Up within a bounded time after committing peer-first.
/// </para>
/// <para>
/// This decides the join order BEFORE issuing a single <c>JoinSeedNodes</c>, from an explicit
/// reachability signal — unlike a timer-based watchdog, which fires a <c>Join(self)</c>
/// mid-handshake on a bare timeout and could not tell "no seed answered" from "a join is in
/// flight", islanding a node a failover had just bounced (the rejected self-form-watchdog
/// design; see <c>SelfFirstSeedBootstrapTests</c>).
/// </para>
/// </remarks>
public static class ClusterBootstrapGuard
{
private static readonly Regex SeedPattern =
new(@"^akka(?:\.[a-z0-9]+)?://[^@/]+@(?<host>[^:/]+):(?<port>\d+)", RegexOptions.IgnoreCase | RegexOptions.Compiled);
/// <summary>The analyzed bootstrap role of this node within its seed list.</summary>
/// <param name="Applies">True only when the guard engages: exactly two seeds, one of them this node.</param>
/// <param name="IsFounder">True when this node is the preferred founder (lower address) — form immediately, no probe.</param>
/// <param name="PartnerHost">The partner's advertised host (to probe when this node is the higher one); null when not applicable.</param>
/// <param name="PartnerPort">The partner's Akka port.</param>
/// <param name="SelfFirstOrder"><c>[self, partner]</c> — self-first; forms a new cluster when no peer answers.</param>
/// <param name="PeerFirstOrder"><c>[partner, self]</c> — peer-first; joins the partner's cluster, never self-forms while it is up.</param>
public sealed record BootstrapRole(
bool Applies,
bool IsFounder,
string? PartnerHost,
int PartnerPort,
string[] SelfFirstOrder,
string[] PeerFirstOrder);
/// <summary>
/// Analyzes the configured seed list to decide this node's bootstrap role. The guard engages only
/// for the pair shape (exactly two seeds, one of them this node); any other shape
/// (single-seed / single-node install, three+ seeds, self absent) returns
/// <see cref="BootstrapRole.Applies"/> = false and the caller joins the configured seeds unchanged.
/// </summary>
/// <param name="seeds">The configured seed URIs (self-first by convention, but order is not relied on here).</param>
/// <param name="selfHost">This node's advertised host (<c>NodeOptions.NodeHostname</c>).</param>
/// <param name="selfPort">This node's Akka remoting port (<c>NodeOptions.RemotingPort</c>).</param>
/// <returns>The analyzed role; never null.</returns>
public static BootstrapRole Analyze(IReadOnlyList<string> seeds, string selfHost, int selfPort)
{
ArgumentNullException.ThrowIfNull(seeds);
ArgumentException.ThrowIfNullOrWhiteSpace(selfHost);
var na = new BootstrapRole(false, false, null, 0, Array.Empty<string>(), Array.Empty<string>());
if (seeds.Count != 2) return na;
var selfKey = Key(selfHost, selfPort);
string? self = null, partner = null;
string? partnerHost = null;
var partnerPort = 0;
foreach (var seed in seeds)
{
if (!TryParse(seed, out var host, out var port)) return na;
if (string.Equals(Key(host, port), selfKey, StringComparison.OrdinalIgnoreCase))
self = seed;
else
{
partner = seed;
partnerHost = host;
partnerPort = port;
}
}
// Self must be exactly one of the two, and the other must be a distinct partner.
if (self is null || partner is null || partnerHost is null) return na;
var selfFirst = new[] { self, partner };
var peerFirst = new[] { partner, self };
// Deterministic tie-break: the lower canonical address is the preferred founder. Both nodes
// compute the same comparison (same two seeds), so exactly one is the founder. Case-INSENSITIVE
// to match how self/partner are classified above (and StartupValidator.SeedNodeIsSelf):
// container/DNS hostnames are conventionally case-insensitive, and a casing difference between
// the two sides' seed config must NOT make both think they are the founder (which would reopen
// the very split this guard closes).
var isFounder = string.Compare(
Key(selfHost, selfPort),
Key(partnerHost, partnerPort),
StringComparison.OrdinalIgnoreCase) < 0;
return new BootstrapRole(true, isFounder, partnerHost, partnerPort, selfFirst, peerFirst);
}
/// <summary>
/// The order the higher (non-founder) node uses once it has learned whether its partner is
/// reachable: peer-first when the founder is up (join it), self-first when the founder is absent
/// (form alone — cold-start-alone preserved).
/// </summary>
/// <param name="role">The analyzed role (must be applicable and NOT the founder).</param>
/// <param name="partnerReachable">Whether the partner's Akka endpoint became reachable within the probe window.</param>
/// <returns>The seed order to pass to <c>Cluster.JoinSeedNodes</c>.</returns>
public static string[] HigherNodeOrder(BootstrapRole role, bool partnerReachable)
{
ArgumentNullException.ThrowIfNull(role);
return partnerReachable ? role.PeerFirstOrder : role.SelfFirstOrder;
}
/// <summary>Parses <c>akka.tcp://system@host:port[/...]</c> into host + port.</summary>
/// <param name="seed">The seed URI.</param>
/// <param name="host">The parsed advertised host.</param>
/// <param name="port">The parsed Akka port.</param>
/// <returns>True when the seed matched the expected shape.</returns>
public static bool TryParse(string? seed, out string host, out int port)
{
host = string.Empty;
port = 0;
if (string.IsNullOrWhiteSpace(seed)) return false;
var m = SeedPattern.Match(seed.Trim());
if (!m.Success) return false;
if (!int.TryParse(m.Groups["port"].Value, out port)) return false;
host = m.Groups["host"].Value;
return true;
}
private static string Key(string host, int port) => $"{host}:{port}";
}
@@ -114,4 +114,50 @@ public class ClusterOptions
/// dials forever — log noise and a defeated validation intent (review 01).
/// </summary>
public bool AllowSingleNodeCluster { get; set; }
/// <summary>
/// The simultaneous-cold-start split-brain guard (<c>ScadaBridge:Cluster:BootstrapGuard</c>).
/// Default OFF — a dark switch, so existing deployments and tests keep Akka's config-driven
/// self-first auto-join byte-identical. See <see cref="ClusterBootstrapGuard"/> for the
/// decision logic and <c>ClusterBootstrapCoordinator</c> for the runtime (Gitea #33).
/// </summary>
public ClusterBootstrapGuardOptions BootstrapGuard { get; set; } = new();
}
/// <summary>
/// Configuration for the simultaneous-cold-start split-brain guard. Both nodes of a pair are
/// self-first seeds so <b>either</b> can cold-start alone when its partner is dead — but the cost
/// is that when BOTH cold-start at the same instant each runs Akka's <c>FirstSeedNodeProcess</c>,
/// times out waiting for the other, and forms its own single-node cluster (a split brain that does
/// not auto-merge). When <see cref="Enabled"/>, the node does NOT auto-join from its config seeds
/// (<c>AkkaHostedService.BuildHocon</c> emits an empty seed list); a coordinator picks the join
/// order — founder self-first / joiner peer-first — after a reachability probe. See
/// <see cref="ClusterBootstrapGuard"/>.
/// </summary>
public sealed class ClusterBootstrapGuardOptions
{
/// <summary>
/// Whether the guard is active. Default <see langword="false"/> — Akka auto-joins from the
/// config seed list exactly as before. Only meaningful on a node that is one of its own two
/// pair seeds; inert everywhere else (single-seed site node, self absent, 3+ seeds).
/// </summary>
public bool Enabled { get; set; }
/// <summary>
/// How long the higher-address node probes its partner's Akka endpoint before concluding the
/// partner is dead and forming alone. Must comfortably exceed the partner's worst-case
/// process-start-to-Akka-bind time so a slow-but-alive partner is never mistaken for a dead one
/// (which would re-open the split). Default 25s. Validated <c>&gt; 0</c> at boot when the guard is on.
/// </summary>
public int PartnerProbeSeconds { get; set; } = 25;
/// <summary>
/// The interval between partner reachability probes, in milliseconds. Default 500ms.
/// </summary>
public int PartnerProbeIntervalMs { get; set; } = 500;
/// <summary>
/// The per-probe TCP connect timeout, in milliseconds. Default 1000ms.
/// </summary>
public int ProbeConnectTimeoutMs { get; set; } = 1000;
}
@@ -30,6 +30,8 @@ public sealed class ClusterOptionsValidator : OptionsValidatorBase<ClusterOption
/// <inheritdoc />
protected override void Validate(ValidationBuilder builder, ClusterOptions options)
{
ValidateBootstrapGuard(builder, options.BootstrapGuard);
// The design doc states "both nodes are seed nodes — each node lists
// both itself and its partner" so a properly-configured deployment lists
// two. Accepting a single-seed configuration silently defeats the
@@ -78,4 +80,33 @@ public sealed class ClusterOptionsValidator : OptionsValidatorBase<ClusterOption
+ "oldest node can run as an isolated single-node cluster during a partition while the "
+ "younger node forms its own, producing two live clusters.");
}
/// <summary>
/// When the bootstrap guard is enabled (Gitea #33), its timing knobs must be positive. A
/// zero/negative <see cref="ClusterBootstrapGuardOptions.PartnerProbeSeconds"/> in particular
/// silently degrades the guard to "never wait, always conclude the partner is dead" — the
/// higher node would form alone immediately and re-open the very split the guard exists to
/// close. Fail fast at boot rather than producing that silent degradation. Nothing is checked
/// when the guard is off (the knobs are inert), so a disabled guard never blocks a boot.
/// </summary>
private static void ValidateBootstrapGuard(ValidationBuilder builder, ClusterBootstrapGuardOptions? guard)
{
if (guard is null || !guard.Enabled)
{
return;
}
builder.RequireThat(guard.PartnerProbeSeconds > 0,
$"ClusterOptions.BootstrapGuard.PartnerProbeSeconds must be > 0 when the guard is enabled "
+ $"(was {guard.PartnerProbeSeconds}); a non-positive value makes the higher node conclude its "
+ "partner is dead without waiting and form alone, re-opening the split the guard prevents.");
builder.RequireThat(guard.PartnerProbeIntervalMs > 0,
$"ClusterOptions.BootstrapGuard.PartnerProbeIntervalMs must be > 0 when the guard is enabled "
+ $"(was {guard.PartnerProbeIntervalMs}).");
builder.RequireThat(guard.ProbeConnectTimeoutMs > 0,
$"ClusterOptions.BootstrapGuard.ProbeConnectTimeoutMs must be > 0 when the guard is enabled "
+ $"(was {guard.ProbeConnectTimeoutMs}).");
}
}
@@ -261,8 +261,16 @@ public class AkkaHostedService : IHostedService
TimeSpan transportHeartbeat,
TimeSpan transportFailure)
{
var seedNodesStr = string.Join(",",
clusterOptions.SeedNodes.Select(QuoteHocon));
// Simultaneous-cold-start split-brain guard (Gitea #33): when enabled the node must NOT
// auto-join from its config seeds — Akka's FirstSeedNodeProcess is exactly what races the
// peer and forms two 1-node clusters. Emit an EMPTY seed list so the ActorSystem comes up
// unjoined, and ClusterBootstrapCoordinator issues the single reachability-gated
// JoinSeedNodes (founder self-first / joiner peer-first). Default OFF, so guard-off configs
// render byte-identical to before.
var effectiveSeeds = clusterOptions.BootstrapGuard.Enabled
? Enumerable.Empty<string>()
: clusterOptions.SeedNodes;
var seedNodesStr = string.Join(",", effectiveSeeds.Select(QuoteHocon));
var rolesStr = string.Join(",", roles.Select(QuoteHocon));
// auto-down (default): AutoDowning provider — the leader among the reachable
@@ -0,0 +1,211 @@
using System.Diagnostics;
using System.Net.Sockets;
using Akka.Actor;
using Akka.Cluster;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.ScadaBridge.ClusterInfrastructure;
namespace ZB.MOM.WW.ScadaBridge.Host.Actors;
/// <summary>
/// Runs the simultaneous-cold-start split-brain guard (Gitea #33): when
/// <c>ScadaBridge:Cluster:BootstrapGuard:Enabled</c>, the node starts with NO config seed nodes
/// (<c>AkkaHostedService.BuildHocon</c> emits an empty seed list), and this coordinator picks the join
/// order and issues the single <see cref="Akka.Cluster.Cluster.JoinSeedNodes"/> once the ActorSystem is
/// up. See <see cref="ClusterBootstrapGuard"/> for the decision logic and the split-brain rationale.
/// </summary>
/// <remarks>
/// <para>
/// The founder (lower canonical address) joins its self-first order immediately. The higher
/// node probes its partner's Akka endpoint (a plain TCP connect — the partner has bound its
/// port well before it forms) up to <see cref="ClusterBootstrapGuardOptions.PartnerProbeSeconds"/>:
/// reachable ⇒ peer-first (join the founder); unreachable ⇒ self-first (partner is dead, form
/// alone). The join runs on a background task so it never blocks host startup, and it is issued
/// exactly once — the coordinator never re-forms a node mid-handshake, the failure mode of the
/// rejected self-form-watchdog design (see <c>SelfFirstSeedBootstrapTests</c>).
/// </para>
/// <para>
/// Registered unconditionally in both the Central and Site composition roots but a no-op unless
/// the guard is enabled, so guard-off deployments keep Akka's config-driven self-first
/// auto-join byte-identical.
/// </para>
/// </remarks>
public sealed class ClusterBootstrapCoordinator : IHostedService
{
private readonly Func<ActorSystem> _system;
private readonly ClusterOptions _clusterOptions;
private readonly NodeOptions _nodeOptions;
private readonly ILogger<ClusterBootstrapCoordinator> _log;
private readonly CancellationTokenSource _cts = new();
private Task? _joinTask;
/// <summary>Creates the coordinator.</summary>
/// <param name="system">Lazy accessor for the node's ActorSystem (resolved after the Akka hosted service creates it).</param>
/// <param name="clusterOptions">The bound cluster options (seed list + guard settings).</param>
/// <param name="nodeOptions">The bound node identity (advertised hostname + remoting port).</param>
/// <param name="log">Logger for the bootstrap decision.</param>
public ClusterBootstrapCoordinator(
Func<ActorSystem> system,
IOptions<ClusterOptions> clusterOptions,
IOptions<NodeOptions> nodeOptions,
ILogger<ClusterBootstrapCoordinator> log)
{
_system = system ?? throw new ArgumentNullException(nameof(system));
_clusterOptions = clusterOptions?.Value ?? throw new ArgumentNullException(nameof(clusterOptions));
_nodeOptions = nodeOptions?.Value ?? throw new ArgumentNullException(nameof(nodeOptions));
_log = log ?? throw new ArgumentNullException(nameof(log));
}
/// <inheritdoc/>
public Task StartAsync(CancellationToken cancellationToken)
{
if (!_clusterOptions.BootstrapGuard.Enabled)
return Task.CompletedTask; // dark switch off — Akka auto-joined from config seeds already.
// Fire-and-forget so a higher node with a dead partner (which waits the full probe window)
// never blocks host startup. Exceptions are logged; a failed join leaves the node unjoined,
// which is visible and recoverable, never a silent split.
_joinTask = Task.Run(() => RunAsync(_cts.Token), _cts.Token);
return Task.CompletedTask;
}
/// <inheritdoc/>
public async Task StopAsync(CancellationToken cancellationToken)
{
await _cts.CancelAsync().ConfigureAwait(false);
if (_joinTask is not null)
{
try { await _joinTask.ConfigureAwait(false); }
catch (OperationCanceledException) { /* expected on shutdown */ }
}
}
private async Task RunAsync(CancellationToken ct)
{
try
{
var seeds = _clusterOptions.SeedNodes ?? new List<string>();
var role = ClusterBootstrapGuard.Analyze(seeds, _nodeOptions.NodeHostname, _nodeOptions.RemotingPort);
string[] order;
var committedPeerFirst = false;
if (!role.Applies)
{
// Not a pair seed (single-node install, self absent, malformed, or 3+ seeds): join the
// configured seeds unchanged — the guard arbitrates only the 2-node pair race.
order = seeds.ToArray();
_log.LogInformation(
"Bootstrap guard: not a pair seed ({SeedCount} seed(s)); joining configured seeds unchanged.",
seeds.Count);
}
else if (role.IsFounder)
{
order = role.SelfFirstOrder;
_log.LogInformation(
"Bootstrap guard: this node is the preferred founder (lower address); joining self-first, forming immediately if no peer answers.");
}
else
{
var reachable = await ProbePartnerAsync(role.PartnerHost!, role.PartnerPort, ct).ConfigureAwait(false);
order = ClusterBootstrapGuard.HigherNodeOrder(role, reachable);
committedPeerFirst = reachable;
_log.LogInformation(
"Bootstrap guard: this node is the higher address; partner {PartnerHost}:{PartnerPort} was {Reachability} within the probe window; joining {Order}.",
role.PartnerHost, role.PartnerPort, reachable ? "REACHABLE" : "UNREACHABLE",
reachable ? "peer-first (join the founder)" : "self-first (partner down, forming alone)");
}
if (order.Length == 0)
{
_log.LogWarning("Bootstrap guard: no seed nodes to join; node stays unjoined until a seed is configured.");
return;
}
var cluster = Akka.Cluster.Cluster.Get(_system());
var addresses = order.Select(Address.Parse).ToArray();
cluster.JoinSeedNodes(addresses);
// Peer-first is a one-way commitment: JoinSeedNodeProcess never self-forms. If the founder
// died in the probe→join window, this node hangs unjoined forever. We do NOT re-form here
// (the rejected mid-handshake failure mode) — we make the hang operator-visible so a
// restart, which re-runs the guard against the now-dead founder and self-forms, recovers
// it. Nothing is done on the founder / self-first paths, which always self-form on their own.
if (committedPeerFirst)
await WarnIfNotUpAsync(cluster, ct).ConfigureAwait(false);
}
catch (OperationCanceledException)
{
// Host shutting down before the join completed — nothing to do.
}
catch (Exception ex)
{
_log.LogError(ex, "Bootstrap guard: failed to issue JoinSeedNodes; node remains unjoined.");
}
}
/// <summary>
/// After committing peer-first, waits a bounded time for this node to reach <c>Up</c> and logs a
/// clear warning if it does not — the founder must have died in the probe→join window, leaving this
/// node hung in <c>JoinSeedNodeProcess</c>. Read-only: it never re-forms or re-joins (that is the
/// rejected mid-handshake failure mode); it only surfaces the condition so an operator restarts the node.
/// </summary>
private async Task WarnIfNotUpAsync(Akka.Cluster.Cluster cluster, CancellationToken ct)
{
// Give the join a generous grace: the founder's own self-form (seed-node-timeout) plus this
// node's join round-trip. Two probe windows is comfortably beyond both.
var grace = TimeSpan.FromSeconds(Math.Max(10, _clusterOptions.BootstrapGuard.PartnerProbeSeconds * 2));
var sw = Stopwatch.StartNew();
while (!ct.IsCancellationRequested && sw.Elapsed < grace)
{
if (cluster.SelfMember.Status == MemberStatus.Up) return; // joined — all good
try { await Task.Delay(TimeSpan.FromSeconds(1), ct).ConfigureAwait(false); }
catch (OperationCanceledException) { return; }
}
if (!ct.IsCancellationRequested && cluster.SelfMember.Status != MemberStatus.Up)
_log.LogWarning(
"Bootstrap guard: committed peer-first but this node is still {Status} after {Grace}s — its founder likely died in the probe→join window. This node will NOT self-form on its own (peer-first never does). RESTART it to recover: the guard will re-run, find the founder down, and form alone.",
cluster.SelfMember.Status, (int)grace.TotalSeconds);
}
/// <summary>
/// Polls the partner's Akka endpoint with a plain TCP connect until it is reachable or the probe
/// window elapses. Reachable-at-TCP is a sound proxy for "the partner is coming up": a node binds
/// its Akka port near the start of startup, well before it forms or joins a cluster.
/// </summary>
private async Task<bool> ProbePartnerAsync(string host, int port, CancellationToken ct)
{
var window = TimeSpan.FromSeconds(Math.Max(0, _clusterOptions.BootstrapGuard.PartnerProbeSeconds));
var interval = TimeSpan.FromMilliseconds(Math.Max(50, _clusterOptions.BootstrapGuard.PartnerProbeIntervalMs));
var sw = Stopwatch.StartNew();
while (!ct.IsCancellationRequested)
{
if (await TryConnectAsync(host, port, ct).ConfigureAwait(false)) return true;
if (sw.Elapsed >= window) return false;
try { await Task.Delay(interval, ct).ConfigureAwait(false); }
catch (OperationCanceledException) { return false; }
}
return false;
}
private async Task<bool> TryConnectAsync(string host, int port, CancellationToken ct)
{
try
{
using var client = new TcpClient();
using var cts = CancellationTokenSource.CreateLinkedTokenSource(ct);
cts.CancelAfter(Math.Max(100, _clusterOptions.BootstrapGuard.ProbeConnectTimeoutMs));
await client.ConnectAsync(host, port, cts.Token).ConfigureAwait(false);
return client.Connected;
}
catch (OperationCanceledException) when (ct.IsCancellationRequested)
{
throw; // host shutdown — propagate
}
catch
{
return false; // connect refused / timed out / DNS not resolvable yet — partner not up
}
}
}
+14
View File
@@ -394,6 +394,20 @@ try
builder.Services.AddSingleton<Akka.Actor.ActorSystem>(sp =>
sp.GetRequiredService<AkkaHostedService>().GetOrCreateActorSystem());
// Simultaneous-cold-start split-brain guard (Gitea #33, dark switch
// ScadaBridge:Cluster:BootstrapGuard:Enabled, default off). Registered AFTER the
// AkkaHostedService/ActorSystem bridge so its StartAsync resolves the system the main path
// built; when the guard is off it no-ops (Akka auto-joined from config seeds), so registering
// it unconditionally is safe. When on, the node started unjoined (empty HOCON seed list, see
// AkkaHostedService.BuildHocon) and this coordinator issues the single reachability-gated
// JoinSeedNodes. It resolves the ActorSystem lazily via GetOrCreateActorSystem so it never
// races startup or caches a null.
builder.Services.AddHostedService(sp => new ClusterBootstrapCoordinator(
() => sp.GetRequiredService<AkkaHostedService>().GetOrCreateActorSystem(),
sp.GetRequiredService<IOptions<ClusterOptions>>(),
sp.GetRequiredService<IOptions<NodeOptions>>(),
sp.GetRequiredService<ILogger<ClusterBootstrapCoordinator>>()));
// Register the production IActiveNodeGate implementation so
// standby-node gating is actually enforced (the InboundApiEndpointFilter
// consults IActiveNodeGate and defaults to "allow" when none is registered,
@@ -168,6 +168,19 @@ public static class SiteServiceRegistration
services.AddSingleton<Akka.Actor.ActorSystem>(sp =>
sp.GetRequiredService<AkkaHostedService>().GetOrCreateActorSystem());
// Simultaneous-cold-start split-brain guard (Gitea #33, dark switch
// ScadaBridge:Cluster:BootstrapGuard:Enabled, default off). This is the case the guard was
// built for — a shared power event that powers up both site VMs together. Registered AFTER
// the AkkaHostedService/ActorSystem bridge (mirrors the Central composition root in
// Program.cs); it no-ops when the guard is off, so registering it unconditionally is safe.
// When on, the node started unjoined (empty HOCON seed list) and this coordinator issues the
// single reachability-gated JoinSeedNodes, resolving the ActorSystem lazily.
services.AddHostedService(sp => new ClusterBootstrapCoordinator(
() => sp.GetRequiredService<AkkaHostedService>().GetOrCreateActorSystem(),
sp.GetRequiredService<IOptions<ClusterOptions>>(),
sp.GetRequiredService<IOptions<NodeOptions>>(),
sp.GetRequiredService<ILogger<ClusterBootstrapCoordinator>>()));
// Cluster node status provider for health reports
services.AddSingleton<IClusterNodeProvider>(sp =>
{
@@ -17,7 +17,14 @@
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
"MinNrOfMembers": 1
"MinNrOfMembers": 1,
"_bootstrapGuard": "Gitea #33 simultaneous-cold-start split-brain guard. Dark switch, default off (Akka's config-driven self-first auto-join, unchanged). Enable ONLY where both pair VMs can power up together (shared power/hypervisor event) with no start-order serialization: then the lower host:port founds self-first and the higher probes-then-joins, so the pair converges to one cluster instead of two 1-node clusters. Timings validated > 0 when enabled.",
"BootstrapGuard": {
"Enabled": false,
"PartnerProbeSeconds": 25,
"PartnerProbeIntervalMs": 500,
"ProbeConnectTimeoutMs": 1000
}
},
"_secrets": "Host-003: Secrets are NOT committed in this file. The ${secret:...} references below are resolved at startup by the pre-host secrets expander (before StartupValidator runs) from the encrypted secrets store (SQLite, seeded via the ZB.MOM.WW.Secrets store/CLI/UI). The KEK is supplied out-of-band via the ZB_SECRETS_MASTER_KEY environment variable and never committed; an unseeded reference fails closed (SecretNotFoundException) before any SQL/LDAP/cluster wiring. ROLLBACK: the whole-key environment override still wins — set ScadaBridge__Database__ConfigurationDb / ScadaBridge__Security__Ldap__ServiceAccountPassword / ScadaBridge__Security__JwtSigningKey and .AddEnvironmentVariables() overlays the concrete value before expansion runs, so the ${secret:...} token is never evaluated. NOTE (Task 1.4): the LDAP settings moved into the nested Security:Ldap sub-section (bound to the shared ZB.MOM.WW.Auth LdapOptions) — the service-account-password env var is ScadaBridge__Security__Ldap__ServiceAccountPassword (was ScadaBridge__Security__LdapServiceAccountPassword).",
"Database": {
@@ -20,7 +20,14 @@
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
"MinNrOfMembers": 1
"MinNrOfMembers": 1,
"_bootstrapGuard": "Gitea #33 simultaneous-cold-start split-brain guard. Dark switch, default off (Akka's config-driven self-first auto-join, unchanged). Enable ONLY where both pair VMs can power up together (shared power/hypervisor event) with no start-order serialization: then the lower host:port founds self-first and the higher probes-then-joins, so the pair converges to one cluster instead of two 1-node clusters. Timings validated > 0 when enabled.",
"BootstrapGuard": {
"Enabled": false,
"PartnerProbeSeconds": 25,
"PartnerProbeIntervalMs": 500,
"ProbeConnectTimeoutMs": 1000
}
},
"Database": {
// Migration-only as of LocalDb Phase 2. The site config tables now live in the
@@ -0,0 +1,147 @@
namespace ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests;
/// <summary>
/// Unit tests for <see cref="ClusterBootstrapGuard"/> — the pure decision core of the
/// simultaneous-cold-start split-brain guard (Gitea #33). The runtime probe + JoinSeedNodes wiring
/// lives in <c>ClusterBootstrapCoordinator</c> and is covered by the real-ActorSystem
/// <c>ClusterBootstrapCoordinatorTests</c> in the integration suite.
/// </summary>
public class ClusterBootstrapGuardTests
{
private const string A = "akka.tcp://scadabridge@scadabridge-site-a-a:8082";
private const string B = "akka.tcp://scadabridge@scadabridge-site-a-b:8082";
[Fact]
public void Lower_address_node_is_the_founder_and_needs_no_probe()
{
// scadabridge-site-a-a < scadabridge-site-a-b, so on node-a the guard makes it the founder.
var role = ClusterBootstrapGuard.Analyze(new[] { A, B }, "scadabridge-site-a-a", 8082);
Assert.True(role.Applies);
Assert.True(role.IsFounder);
Assert.Equal(new[] { A, B }, role.SelfFirstOrder);
}
[Fact]
public void Higher_address_node_is_not_the_founder_and_targets_the_partner_for_probing()
{
// On node-b the partner is node-a (the lower/founder).
var role = ClusterBootstrapGuard.Analyze(new[] { B, A }, "scadabridge-site-a-b", 8082);
Assert.True(role.Applies);
Assert.False(role.IsFounder);
Assert.Equal("scadabridge-site-a-a", role.PartnerHost);
Assert.Equal(8082, role.PartnerPort);
}
[Fact]
public void Tie_break_is_symmetric_regardless_of_seed_order()
{
// Both nodes must agree on who founds no matter how each lists its seeds.
var onLower = ClusterBootstrapGuard.Analyze(new[] { A, B }, "scadabridge-site-a-a", 8082);
var onHigher = ClusterBootstrapGuard.Analyze(new[] { B, A }, "scadabridge-site-a-b", 8082);
Assert.True(onLower.IsFounder);
Assert.False(onHigher.IsFounder);
// Exactly one founder.
Assert.True(onLower.IsFounder ^ onHigher.IsFounder);
}
[Fact]
public void Tie_break_is_case_insensitive_so_a_casing_mismatch_cannot_make_both_founders()
{
// The two sides' seed config differ only in host casing (review note 1): if the tie-break
// were case-sensitive, both could conclude they are the founder and re-open the split.
var lowerSelf = "akka.tcp://scadabridge@SITE-A-A:8082";
var higherSelf = "akka.tcp://scadabridge@site-a-b:8082";
var onLower = ClusterBootstrapGuard.Analyze(new[] { lowerSelf, higherSelf }, "SITE-A-A", 8082);
var onHigher = ClusterBootstrapGuard.Analyze(new[] { higherSelf, lowerSelf }, "site-a-b", 8082);
Assert.True(onLower.IsFounder ^ onHigher.IsFounder);
Assert.True(onLower.IsFounder); // "site-a-a" < "site-a-b" ordinal-ignore-case
}
[Fact]
public void Higher_node_joins_the_founder_when_partner_is_reachable()
{
var role = ClusterBootstrapGuard.Analyze(new[] { B, A }, "scadabridge-site-a-b", 8082);
// Reachable ⇒ peer-first ⇒ JoinSeedNodeProcess ⇒ joins the founder, never self-forms.
Assert.Equal(new[] { A, B }, ClusterBootstrapGuard.HigherNodeOrder(role, partnerReachable: true)); // partner (node-a) first
}
[Fact]
public void Higher_node_forms_alone_when_partner_is_unreachable()
{
var role = ClusterBootstrapGuard.Analyze(new[] { B, A }, "scadabridge-site-a-b", 8082);
// Unreachable after the window ⇒ partner is dead ⇒ self-first ⇒ cold-start-alone preserved.
Assert.Equal(new[] { B, A }, ClusterBootstrapGuard.HigherNodeOrder(role, partnerReachable: false)); // self (node-b) first
}
[Fact]
public void Guard_does_not_apply_to_a_single_seed_node()
{
// A single-node install lists exactly one seed (itself) — not a pair, guard is inert.
var role = ClusterBootstrapGuard.Analyze(new[] { "akka.tcp://scadabridge@lone:8081" }, "lone", 8081);
Assert.False(role.Applies);
}
[Fact]
public void Guard_does_not_apply_when_self_is_absent_from_the_two_seeds()
{
var role = ClusterBootstrapGuard.Analyze(new[] { A, B }, "scadabridge-central-a", 8081);
Assert.False(role.Applies);
}
[Fact]
public void Guard_does_not_apply_to_three_or_more_seeds()
{
var role = ClusterBootstrapGuard.Analyze(
new[] { A, B, "akka.tcp://scadabridge@scadabridge-site-a-c:8082" }, "scadabridge-site-a-a", 8082);
Assert.False(role.Applies);
}
[Fact]
public void Guard_does_not_apply_when_a_seed_is_malformed()
{
var role = ClusterBootstrapGuard.Analyze(new[] { A, "not-a-seed" }, "scadabridge-site-a-a", 8082);
Assert.False(role.Applies);
}
[Theory]
[InlineData("akka.tcp://scadabridge@scadabridge-site-a-a:8082", "scadabridge-site-a-a", 8082)]
[InlineData("akka.tcp://scadabridge@10.0.0.5:8082/", "10.0.0.5", 8082)]
[InlineData(" akka.tcp://scadabridge@host.lan:1234 ", "host.lan", 1234)]
public void TryParse_extracts_host_and_port(string seed, string expectedHost, int expectedPort)
{
Assert.True(ClusterBootstrapGuard.TryParse(seed, out var host, out var port));
Assert.Equal(expectedHost, host);
Assert.Equal(expectedPort, port);
}
[Theory]
[InlineData("")]
[InlineData("garbage")]
[InlineData("akka.tcp://scadabridge@host")] // no port
public void TryParse_rejects_malformed_seeds(string seed)
{
Assert.False(ClusterBootstrapGuard.TryParse(seed, out _, out _));
}
[Fact]
public void Port_participates_in_the_tie_break_when_hosts_are_equal()
{
// Shared-host loopback pair (test rig): same host, different ports — port breaks the tie.
const string low = "akka.tcp://scadabridge@127.0.0.1:8081";
const string high = "akka.tcp://scadabridge@127.0.0.1:8082";
Assert.True(ClusterBootstrapGuard.Analyze(new[] { low, high }, "127.0.0.1", 8081).IsFounder);
Assert.False(ClusterBootstrapGuard.Analyze(new[] { high, low }, "127.0.0.1", 8082).IsFounder);
}
}
@@ -215,4 +215,68 @@ public class ClusterOptionsValidatorTests
Assert.Contains("StableAfter", result.FailureMessage);
Assert.Contains("HeartbeatInterval", result.FailureMessage);
}
// ---- BootstrapGuard timing validation (Gitea #33) ----
[Fact]
public void BootstrapGuard_disabled_with_zero_timings_passes_validation()
{
// The knobs are inert when the guard is off, so a disabled guard must never block a boot —
// even if the timing values are nonsense (0 here).
var options = ValidOptions();
options.BootstrapGuard = new ClusterBootstrapGuardOptions
{
Enabled = false,
PartnerProbeSeconds = 0,
PartnerProbeIntervalMs = 0,
ProbeConnectTimeoutMs = 0,
};
var result = new ClusterOptionsValidator().Validate(null, options);
Assert.True(result.Succeeded, result.FailureMessage);
}
[Fact]
public void BootstrapGuard_enabled_with_default_timings_passes_validation()
{
var options = ValidOptions();
options.BootstrapGuard = new ClusterBootstrapGuardOptions { Enabled = true };
var result = new ClusterOptionsValidator().Validate(null, options);
Assert.True(result.Succeeded, result.FailureMessage);
}
[Fact]
public void BootstrapGuard_enabled_with_nonpositive_probe_seconds_fails_validation()
{
// A zero PartnerProbeSeconds silently degrades the guard to "never wait, always conclude the
// partner is dead" — the higher node forms alone immediately and re-opens the split. Reject it.
var options = ValidOptions();
options.BootstrapGuard = new ClusterBootstrapGuardOptions { Enabled = true, PartnerProbeSeconds = 0 };
var result = new ClusterOptionsValidator().Validate(null, options);
Assert.True(result.Failed);
Assert.Contains("PartnerProbeSeconds", result.FailureMessage);
}
[Fact]
public void BootstrapGuard_enabled_with_nonpositive_interval_or_timeout_fails_validation()
{
var options = ValidOptions();
options.BootstrapGuard = new ClusterBootstrapGuardOptions
{
Enabled = true,
PartnerProbeIntervalMs = 0,
ProbeConnectTimeoutMs = -1,
};
var result = new ClusterOptionsValidator().Validate(null, options);
Assert.True(result.Failed);
Assert.Contains("PartnerProbeIntervalMs", result.FailureMessage);
Assert.Contains("ProbeConnectTimeoutMs", result.FailureMessage);
}
}
@@ -0,0 +1,204 @@
using Akka.Actor;
using Akka.Cluster;
using Akka.Configuration;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.ScadaBridge.ClusterInfrastructure;
using ZB.MOM.WW.ScadaBridge.Host;
using ZB.MOM.WW.ScadaBridge.Host.Actors;
namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster;
/// <summary>
/// Real-ActorSystem tests for the simultaneous-cold-start split-brain guard (Gitea #33):
/// <see cref="ClusterBootstrapCoordinator"/> + <see cref="ClusterBootstrapGuard"/>, driven exactly as
/// production drives them — the node's ActorSystem is built from the production <c>BuildHocon</c> output
/// with the guard enabled (which emits an EMPTY seed list, so Akka does not auto-join), then the real
/// coordinator issues the reachability-gated <c>JoinSeedNodes</c>. This subsystem has a history of bugs
/// invisible to pure unit tests, so the load-bearing correctness properties are asserted against running
/// nodes, mirroring <see cref="SelfFirstSeedBootstrapTests"/>.
/// </summary>
/// <remarks>
/// Ephemeral loopback ports share host <c>127.0.0.1</c>, so the port breaks the guard's ordinal
/// <c>host:port</c> tie — <c>Min(port)</c> is the founder, <c>Max(port)</c> is the higher (probing) node.
/// </remarks>
public sealed class ClusterBootstrapCoordinatorTests : IAsyncLifetime
{
private readonly List<ActorSystem> _systems = new();
private readonly List<ClusterBootstrapCoordinator> _coordinators = new();
/// <summary>Starts a node with the bootstrap guard ENABLED, wired exactly as the composition roots
/// do: the ActorSystem is built with an empty HOCON seed list (guard on) and the real coordinator
/// issues the reachability-gated join. Seeds are listed self-first (as every shipped config is); the
/// guard, not the order, decides founder vs. joiner.</summary>
private async Task<ActorSystem> StartGuardedNodeAsync(int selfPort, int peerPort, int probeSeconds = 4)
{
var self = $"akka.tcp://scadabridge@127.0.0.1:{selfPort}";
var peer = $"akka.tcp://scadabridge@127.0.0.1:{peerPort}";
var nodeOptions = new NodeOptions { Role = "Central", NodeHostname = "127.0.0.1", RemotingPort = selfPort };
var clusterOptions = new ClusterOptions
{
SeedNodes = new List<string> { self, peer },
BootstrapGuard = new ClusterBootstrapGuardOptions
{
Enabled = true,
PartnerProbeSeconds = probeSeconds,
PartnerProbeIntervalMs = 250,
ProbeConnectTimeoutMs = 500,
},
StableAfter = TimeSpan.FromSeconds(3),
HeartbeatInterval = TimeSpan.FromMilliseconds(500),
FailureDetectionThreshold = TimeSpan.FromSeconds(2),
MinNrOfMembers = 1,
};
var hocon = AkkaHostedService.BuildHocon(
nodeOptions, clusterOptions, new[] { "Central" },
TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(3));
var system = ActorSystem.Create("scadabridge", ConfigurationFactory.ParseString(hocon));
_systems.Add(system);
var coordinator = new ClusterBootstrapCoordinator(
() => system, Options.Create(clusterOptions), Options.Create(nodeOptions),
NullLogger<ClusterBootstrapCoordinator>.Instance);
_coordinators.Add(coordinator);
await coordinator.StartAsync(CancellationToken.None);
return system;
}
private static async Task<bool> WaitForUpAsync(ActorSystem system, int expected, TimeSpan timeout)
{
var cluster = Akka.Cluster.Cluster.Get(system);
var deadline = DateTime.UtcNow + timeout;
while (DateTime.UtcNow < deadline)
{
if (cluster.State.Members.Count(m => m.Status == MemberStatus.Up) >= expected) return true;
await Task.Delay(200);
}
return cluster.State.Members.Count(m => m.Status == MemberStatus.Up) >= expected;
}
/// <summary>
/// The FOUNDER (lower address) with the guard on forms a cluster alone when its partner is dead —
/// exactly as un-guarded self-first does. The guard must not regress the preferred founder.
/// </summary>
[Fact]
public async Task Founder_forms_alone_when_partner_is_dead()
{
var low = TwoNodeClusterFixture.GetFreeTcpPort();
var high = TwoNodeClusterFixture.GetFreeTcpPort();
var founderPort = Math.Min(low, high);
var deadPeerPort = Math.Max(low, high); // nothing listening
var system = await StartGuardedNodeAsync(founderPort, deadPeerPort);
Assert.True(await WaitForUpAsync(system, 1, TimeSpan.FromSeconds(30)),
"the founder (lower address) must self-form immediately when its partner is down");
}
/// <summary>
/// THE LOAD-BEARING CASE. The HIGHER node with the guard on must still cold-start ALONE when its
/// partner is genuinely dead: it probes the (dead) founder, times out, and falls back to self-first.
/// If the guard's peer-first logic were wrong, this node would hang in JoinSeedNodeProcess forever —
/// the very regression a naive "always let the lower node found" rule would introduce.
/// </summary>
[Fact]
public async Task Higher_node_forms_alone_when_partner_is_dead_after_probing()
{
var low = TwoNodeClusterFixture.GetFreeTcpPort();
var high = TwoNodeClusterFixture.GetFreeTcpPort();
var higherPort = Math.Max(low, high);
var deadFounderPort = Math.Min(low, high); // the founder is dead
var system = await StartGuardedNodeAsync(higherPort, deadFounderPort, probeSeconds: 3);
// Must come Up AFTER the probe window (~3s) expires and it falls back to self-first.
Assert.True(await WaitForUpAsync(system, 1, TimeSpan.FromSeconds(30)),
"the higher node must fall back to self-first and form alone once the dead partner never answers the probe");
}
/// <summary>
/// FALSIFIABILITY CONTROL for the test above: the higher node does NOT form alone during the probe
/// window — it is genuinely waiting/probing, not self-forming immediately (which would mean the
/// guard is inert). If this ever starts seeing a member before the window, the probe is not gating.
/// </summary>
[Fact]
public async Task Higher_node_does_not_form_before_the_probe_window_expires()
{
var low = TwoNodeClusterFixture.GetFreeTcpPort();
var high = TwoNodeClusterFixture.GetFreeTcpPort();
var higherPort = Math.Max(low, high);
var deadFounderPort = Math.Min(low, high);
var system = await StartGuardedNodeAsync(higherPort, deadFounderPort, probeSeconds: 10);
// Well within the 10s probe window: still no cluster, because it is probing the dead founder.
Assert.False(await WaitForUpAsync(system, 1, TimeSpan.FromSeconds(3)),
"the higher node must probe for the founder, not self-form immediately");
}
/// <summary>
/// The higher node JOINS a live founder rather than forming a second cluster: start the founder,
/// then the higher node — its probe finds the founder reachable, it commits peer-first, and the
/// pair converges to ONE cluster of two.
/// </summary>
[Fact]
public async Task Higher_node_joins_a_live_founder()
{
var low = TwoNodeClusterFixture.GetFreeTcpPort();
var high = TwoNodeClusterFixture.GetFreeTcpPort();
var founderPort = Math.Min(low, high);
var higherPort = Math.Max(low, high);
var founder = await StartGuardedNodeAsync(founderPort, higherPort);
Assert.True(await WaitForUpAsync(founder, 1, TimeSpan.FromSeconds(30)),
"the founder must be Up before the higher node probes it");
var higher = await StartGuardedNodeAsync(higherPort, founderPort);
Assert.True(await WaitForUpAsync(higher, 2, TimeSpan.FromSeconds(60)),
"the higher node must join the founder, forming one cluster of two");
Assert.True(await WaitForUpAsync(founder, 2, TimeSpan.FromSeconds(60)),
"the founder must see the higher node as a member of ITS cluster");
}
/// <summary>
/// THE HEADLINE #33 CASE. Both nodes cold-start truly simultaneously with the guard on. Without the
/// guard each would run FirstSeedNodeProcess, race, and form its own 1-node cluster (split brain).
/// With it, the founder forms and the higher node probes-then-joins, so the pair converges to
/// EXACTLY ONE cluster of two — no two oldest, no dual singletons.
/// </summary>
[Fact]
public async Task Both_nodes_cold_starting_together_form_exactly_one_cluster()
{
var low = TwoNodeClusterFixture.GetFreeTcpPort();
var high = TwoNodeClusterFixture.GetFreeTcpPort();
var founderPort = Math.Min(low, high);
var higherPort = Math.Max(low, high);
// Start both without serialization — as a shared power event does.
var founderTask = StartGuardedNodeAsync(founderPort, higherPort);
var higherTask = StartGuardedNodeAsync(higherPort, founderPort);
var founder = await founderTask;
var higher = await higherTask;
Assert.True(await WaitForUpAsync(founder, 2, TimeSpan.FromSeconds(60)),
"the pair must converge to one 2-member cluster, not two 1-member clusters");
Assert.True(await WaitForUpAsync(higher, 2, TimeSpan.FromSeconds(60)),
"the higher node must be in the SAME cluster as the founder");
}
public Task InitializeAsync() => Task.CompletedTask;
public async Task DisposeAsync()
{
foreach (var c in _coordinators)
{
try { await c.StopAsync(CancellationToken.None); } catch { /* teardown */ }
}
foreach (var s in _systems)
{
try { await s.Terminate().WaitAsync(TimeSpan.FromSeconds(10)); } catch { /* teardown */ }
}
}
}