Joseph Doherty
b0eb653bad
Corrects the #459 finding. 2-node keep-oldest recovery works fine (the ScadaBridge sister project proves it); OtOpcUa was missing the supervision pieces that make it automatic, and docs/Redundancy.md wrongly claimed in-place oldest-crash failover. Mechanism (confirmed on a 2-container rig + by decompiling Akka KeepOldest.OldestDecision): on an OLDEST-node crash keep-oldest downs the LONE survivor (DownReachable including myself) — down-if-alone can't rescue a lone survivor (its branch needs >=2 survivors). Recovery is exit-and-rejoin: run-coordinated-shutdown-when-down terminates the node and the service supervisor restarts it. My earlier 'total outage' was a docker-dev artifact (no restart policy); production Install-Services.ps1 already has sc.exe failure restart. Changes (ScadaBridge parity): - ActorSystemTerminationWatchdog (Host, registered after AddAkka): watches ActorSystem.WhenTerminated and on an unexpected self-down calls StopApplication so the process exits (supervisor restarts it) instead of idling with a dead actor system. Distinguishes graceful shutdown via _stopRequested + ApplicationStopping. 3 unit tests. - docker-dev: restart: unless-stopped on the host anchor (models production supervision) + both redundancy peers in SeedNodes so a restarted node re-forms via either peer. - docs/Redundancy.md: rewrote the split-brain recovery section — younger-loss = in-place fast failover; oldest-loss = exit-and-rejoin under supervision (not in-place); the three requirements (supervisor + watchdog + both-node seeds); flagged HardKillFailoverTests as non-representative (Transport.Shutdown, not a real crash). Instant in-place takeover on ANY single loss needs 3+ members. Cluster.Tests 29/29 (SBR guards), watchdog tests 3/3, full solution builds. Live re-verify of the watchdog image pending (host docker disk full).
ZB.MOM.WW.OtOpcUa.Host.IntegrationTests
Two-node Akka cluster integration tests on top of TwoNodeClusterHarness.
Default mode (no infra required)
dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests
Uses Microsoft.EntityFrameworkCore.InMemory for ConfigDb and a stub ILdapAuthService that
accepts any username when the password is valid-password. Each harness instance creates a
unique in-memory database scoped to its lifetime. This is the mode CI runs by default.
Real-infra mode (SQL Server + OpenLDAP)
When you need to exercise EF behaviors that diverge between providers (index uniqueness,
RowVersion concurrency, JSON columns, migration application) or a real LDAP bind, bring up
the bundled compose stack and set the env-var switches:
docker compose -f tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/docker-compose.yml up -d
export OTOPCUA_HARNESS_USE_SQL=1
export OTOPCUA_HARNESS_USE_LDAP=1
dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests
docker compose -f tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/docker-compose.yml down -v
SQL Server mode (OTOPCUA_HARNESS_USE_SQL=1)
- Container:
mcr.microsoft.com/mssql/server:2022-latestonlocalhost:14331 - Each
TwoNodeClusterHarness.StartAsync()creates a unique databaseOtOpcUa_Harness_{guid}viaDatabase.EnsureCreatedAsync()and drops it onDisposeAsync()(best-effort). - Port
14331chosen to avoid colliding with thedocker-dev/fleet (which uses14330).
LDAP mode (OTOPCUA_HARNESS_USE_LDAP=1)
- Container:
bitnami/openldap:2.6onlocalhost:3894 - Users
alice/alice123andbob/bob123, all underou=FleetAdmin. - Port
3894chosen to avoid colliding with thedocker-dev/fleet (which uses3893).
Local-dev caveat
This dev VM (DESKTOP-6JL3KKO) does not run Docker locally. Real-infra mode runs on the
shared Linux Docker host (10.100.0.35) per docs/v2/dev-environment.md, or in CI on Linux.