Every node now lists ITSELF as seed-nodes[0] and its partner second. Akka runs
FirstSeedNodeProcess -- the only bootstrap path that can form a NEW cluster when
no peer answers InitJoin -- exclusively for seed-nodes[0]; every other node runs
JoinSeedNodeProcess and retries InitJoin forever. That is why a lone cold-starting
central-b never came Up (the "registered outage gap"), and self-first ordering
closes it using Akka's own protocol.
- 6 node appsettings swapped (the *-node-b configs; the -a nodes were already
self-first). All 14 shipped node configs now satisfy the invariant.
- StartupValidator enforces it at boot, comparing host AND port -- the invariant
fails silently when broken, so it is enforced loudly. NOTE: the gitignored
deploy/wonder-app-vd03/ overlay must be reordered before its next deploy or
that node will refuse to boot.
- SelfFirstSeedBootstrapTests: real in-process clusters at production
failure-detection timings, incl. a falsifiability control proving the OLD
peer-first ordering never forms.
Rejected alternative (implemented, measured, discarded): an external self-form
timer calling Cluster.Join(SelfAddress) after a window. It sits outside Akka's
join handshake and so cannot tell "no seed answered" from "a seed answered and
the join is in flight". On a routine standby restart the peer is alive but the
join stalls behind removal of the node's own stale incarnation; a Join(self)
during TryingToJoin abandons the in-flight join and forms a second cluster at
the same address -- still split after 90s. Docs that claimed self-first ordering
was unsafe for simultaneous cold start are corrected: while mutually reachable
the InitJoin handshake converges them to one cluster (measured).
Two-node keep-oldest could NEVER survive a crash of the oldest/active node:
Akka.NET 1.5.62 KeepOldest.OldestDecision only lets down-if-alone rescue a
side with >= 2 members, so the 1-vs-1 survivor takes DownReachable and downs
ITSELF — proven live on the rig ('SBR took decision ... including myself')
before this change. static-quorum(1) is worse (IsTooManyMembers -> DownAll);
keep-majority just re-keys the fatal crash to the lowest address.
SplitBrainResolverStrategy gains 'auto-down' (new default): BuildHocon emits
Akka's AutoDowning provider with auto-down-unreachable-after = StableAfter.
The leader among the REACHABLE members downs the unreachable peer, so the
survivor takes over singletons and /health/active in ~25s regardless of which
node died. Accepted trade (explicit owner decision): a real network partition
runs dual-active until an operator restarts one side. keep-oldest remains
supported; DownIfAlone validation is now scoped to it.
Live drill on the rebuilt rig: active-crash TAKEOVER in 28s (victim still
down; all 7 singletons Younger->Oldest), standby-crash removal 27s with 0
routing blips; victims rejoin as standby in 2s. New real-cluster tests pin
both directions (SbrFailoverTests.AutoDown_*); TwoNodeClusterFixture gains a
strategy knob. All 16 appsettings flipped (src, docker, docker-env2, and the
gitignored wonder-app-vd03 overlay on disk — owner must sync to the host).
Docs: decision record docs/plans/2026-07-21-auto-down-availability-decision.md,
Component-ClusterInfrastructure downing section rewritten, drill + README
reworked (active mode now asserts takeover), deferred-work SBR row resolved.
Populates ScadaBridge:Communication:CentralFetchBaseUrl in every central
appsettings for the notify-and-fetch deploy flow. An empty value now
causes a fail-fast on deploy; this config prevents that regression.
- docker/: http://scadabridge-traefik (in-network Traefik LB, port 80)
- docker-env2/: http://scadabridge-env2-traefik (env2 Traefik LB, port 80)
- src/Host base: http://localhost:5000 (ASP.NET Core default for single-host dev)
- deploy/wonder-app-vd03: http://localhost:8085 (gitignored; edited in main repo)
RUNBOOK Upgrading section updated with note on this setting.
Wire the M6 KPI History recorder into the central composition path:
- Program.cs: call services.AddKpiHistory(configuration) on the central-only
branch alongside AddNotificationOutbox/AddAuditLog/AddSiteCallAudit.
- AkkaHostedService.cs: register KpiHistoryRecorderActor as a central,
non-role-scoped ClusterSingletonManager + ClusterSingletonProxy + a
PhaseClusterLeave CoordinatedShutdown graceful-stop drain (singleton name
'kpi-history-recorder'), copied/adapted from the audit-log-purge block.
- appsettings.Central.json (Host + docker + docker-env2 central nodes): add a
ScadaBridge:KpiHistory section (SampleInterval 00:01:00, RetentionDays 90,
PurgeInterval 1.00:00:00, DefaultMaxSeriesPoints 200).
KPI history is observability/best-effort and MUST NOT gate readiness: the
recorder is deliberately NOT added to RequiredSingletonsHealthCheck or any
other readiness gate.
Adds a "Dev Disable-Login Flag" subsection to Component-Security.md covering
ScadaBridge:Security:Auth:DisableLogin / User, the AutoLoginAuthenticationHandler
mechanism, and the no-environment-guard / startup-warning production risk.
Ships DisableLogin: false under ScadaBridge → Security → Auth in:
- src/.../Host/appsettings.json (canonical default)
- docker/central-node-a/appsettings.Central.json
- docker/central-node-b/appsettings.Central.json
Also records DL-3 commit SHAs in the plan tasks file.
Both :9000 (docker) and :9100 (docker-env2) central nodes now bind the shared dev
GLAuth (scadaproj/infra/glauth/, dc=zb,dc=local) via the cn=serviceaccount search
account instead of the bundled scadabridge-ldap container (now commented out in
infra/docker-compose.yml, kept for rollback). Verified: multi-role -> all 4 roles
on both clusters with scadabridge-ldap stopped.
Replace dc=scadabridge,dc=local with dc=zb,dc=local in all dev/test LDAP
references — app config, docker test-cluster node configs (docker/ and
docker-env2/), GLAuth fixture, dev tooling, Host.Tests fixtures,
IntegrationTests factory, and operational test_infra docs. OU structure
(ou=SCADA-Admins,ou=users,etc.) preserved throughout. Email domains
(@scadabridge.local), hostnames, and container names are untouched.
Historical plan docs (2026-05-24-second-environment.md,
2026-05-31-folder-repo-rename-scadabridge-design.md) excluded as
point-in-time records. No synthetic dc=example,dc=com placeholders touched.
- NavMenu: move Import Bundle out of the nested RequireDesign/RequireAdmin
double-gate into the top-level Admin section so an Admin-only user sees it
without needing the Design role; Export Bundle stays in the Design section.
- TransportImport: inject IAuditService + ScadaLinkDbContext; emit a
BundleImportUnlockFailed audit row (best-effort, swallowed on failure) on
every wrong-passphrase attempt in SubmitPassphraseAsync, with attempt
number and error reason in afterState.
- docker central-node-a/b appsettings: add ScadaLink:Transport section with
SourceEnvironment = "docker-cluster" so the importer picks up a non-null
environment name in the audit trail.
- CentralUI.Tests: register IAuditService mock + SQLite in-memory
ScadaLinkDbContext in TransportImportPageTests to satisfy the two new injects.
Adds "NodeName" to the ScadaLink:Node section of each per-node
appsettings:
- central-a, central-b for the two central nodes
- node-a, node-b under each of the three sites (site-a, site-b, site-c)
After this commit + a redeploy, every fresh AuditLog / Notifications /
SiteCalls row gets stamped with the originating node's role name via
INodeIdentityProvider, satisfying the design's SourceNode invariant
end-to-end.
Both central nodes ran on the NotificationOutboxOptions code defaults
(100 / 10s = 600/min) because the mounted per-node appsettings.Central.json
had no ScadaLink:NotificationOutbox section. Add the section with
DispatchBatchSize 1000 + DispatchInterval 5s — measured ~6,000/min after
restart (sweep duration becomes the binding constraint, which is fine:
the no-overlap guard self-regulates). Dev-cluster tuning only.
The cookie SecurePolicy was hard-coded to Always, so the auth cookie was always
marked Secure and the browser never sent it over plain HTTP — making login
impossible on the HTTP-only Docker dev cluster (login succeeded server-side but
every following request was unauthenticated). Add SecurityOptions.RequireHttps-
Cookie (default true — production stays HTTPS-only); when false the cookie uses
SameAsRequest. The docker/ central nodes set it false.
Multi-stage Dockerfile with NuGet restore layer caching, per-node appsettings
with Docker hostnames, shared bridge network with infra services, and
build/deploy/teardown scripts. Ports use 90xx block to avoid conflicts.
Joseph Doherty