Joseph Doherty
cf3bd52f93
Two-node keep-oldest could NEVER survive a crash of the oldest/active node:
Akka.NET 1.5.62 KeepOldest.OldestDecision only lets down-if-alone rescue a
side with >= 2 members, so the 1-vs-1 survivor takes DownReachable and downs
ITSELF — proven live on the rig ('SBR took decision ... including myself')
before this change. static-quorum(1) is worse (IsTooManyMembers -> DownAll);
keep-majority just re-keys the fatal crash to the lowest address.
SplitBrainResolverStrategy gains 'auto-down' (new default): BuildHocon emits
Akka's AutoDowning provider with auto-down-unreachable-after = StableAfter.
The leader among the REACHABLE members downs the unreachable peer, so the
survivor takes over singletons and /health/active in ~25s regardless of which
node died. Accepted trade (explicit owner decision): a real network partition
runs dual-active until an operator restarts one side. keep-oldest remains
supported; DownIfAlone validation is now scoped to it.
Live drill on the rebuilt rig: active-crash TAKEOVER in 28s (victim still
down; all 7 singletons Younger->Oldest), standby-crash removal 27s with 0
routing blips; victims rejoin as standby in 2s. New real-cluster tests pin
both directions (SbrFailoverTests.AutoDown_*); TwoNodeClusterFixture gains a
strategy knob. All 16 appsettings flipped (src, docker, docker-env2, and the
gitignored wonder-app-vd03 overlay on disk — owner must sync to the host).
Docs: decision record docs/plans/2026-07-21-auto-down-availability-decision.md,
Component-ClusterInfrastructure downing section rewritten, drill + README
reworked (active mode now asserts takeover), deferred-work SBR row resolved.
104 lines
4.4 KiB
JSON
104 lines
4.4 KiB
JSON
{
|
|
"ScadaBridge": {
|
|
"Node": {
|
|
"Role": "Site",
|
|
"NodeName": "node-b",
|
|
"NodeHostname": "scadabridge-site-a-b",
|
|
"SiteId": "site-a",
|
|
"RemotingPort": 8082,
|
|
"GrpcPort": 8083,
|
|
"MetricsPort": 8084
|
|
},
|
|
"Cluster": {
|
|
"SeedNodes": [
|
|
"akka.tcp://scadabridge@scadabridge-site-a-a:8082",
|
|
"akka.tcp://scadabridge@scadabridge-site-a-b:8082"
|
|
],
|
|
"SplitBrainResolverStrategy": "auto-down",
|
|
"StableAfter": "00:00:15",
|
|
"HeartbeatInterval": "00:00:02",
|
|
"FailureDetectionThreshold": "00:00:10",
|
|
"MinNrOfMembers": 1
|
|
},
|
|
"Database": {
|
|
// Migration-only as of LocalDb Phase 2. The site config tables now live in the
|
|
// consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain
|
|
// a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has
|
|
// started once.
|
|
"SiteDbPath": "/app/data/scadabridge.db"
|
|
},
|
|
"DataConnection": {
|
|
"ReconnectInterval": "00:00:05",
|
|
"TagResolutionRetryInterval": "00:00:10",
|
|
"WriteTimeout": "00:00:30",
|
|
"SeedReadTimeout": "00:00:30"
|
|
},
|
|
"StoreAndForward": {
|
|
// Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the
|
|
// consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table.
|
|
// SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2
|
|
// file, and is unused after that - keep it until this node has started once.
|
|
"SqliteDbPath": "/app/data/store-and-forward.db"
|
|
},
|
|
"Communication": {
|
|
"CentralContactPoints": [
|
|
"akka.tcp://scadabridge@scadabridge-central-a:8081",
|
|
"akka.tcp://scadabridge@scadabridge-central-b:8081"
|
|
],
|
|
"DeploymentTimeout": "00:02:00",
|
|
"LifecycleTimeout": "00:00:30",
|
|
"QueryTimeout": "00:00:30",
|
|
"TransportHeartbeatInterval": "00:00:05",
|
|
"TransportFailureThreshold": "00:00:15"
|
|
},
|
|
"HealthMonitoring": {
|
|
"ReportInterval": "00:00:30",
|
|
"OfflineTimeout": "00:01:00"
|
|
},
|
|
"SiteEventLog": {
|
|
"RetentionDays": 30,
|
|
"MaxStorageMb": 1024,
|
|
"PurgeScheduleCron": "0 2 * * *"
|
|
},
|
|
"Notification": {},
|
|
"Logging": {
|
|
"MinimumLevel": "Information"
|
|
}
|
|
},
|
|
// Consolidated site database (LocalDb Phase 1): OperationTracking + site_events.
|
|
// On the mounted /app/data volume so it survives container recreate - unlike the
|
|
// legacy site-tracking.db / site_events.db, which defaulted to CWD-relative paths
|
|
// outside the volume and were lost on every recreate.
|
|
// Replication is opt-in and configured separately; absent = local-only.
|
|
"LocalDb": {
|
|
"Path": "/app/data/site-localdb.db",
|
|
// The PASSIVE half of site-a's replicated pair: no PeerAddress, so this node's
|
|
// sync initiator starts and idles while node-a dials in. Its own writes still
|
|
// reach node-a - the stream is bidirectional.
|
|
//
|
|
// The key MUST match node-a's exactly. LocalDbSyncAuthInterceptor is
|
|
// fail-closed, so a typo here does not degrade to unauthenticated replication;
|
|
// it rejects every stream and the pair silently stops converging.
|
|
"Replication": {
|
|
"ApiKey": "dev-site-a-localdb-sync-key",
|
|
// ---- Phase 2 sizing, from the Task 1 rig soak (not from the defaults) ----
|
|
//
|
|
// MaxBatchSize (default 500) is a ROW count, not a byte budget, so the batch
|
|
// size in bytes is set by the widest replicated column. That is
|
|
// deployed_configurations.config_json: ~721 B on this rig, but up to ~60-70 KB
|
|
// in production (measured, Task 1) - and 70 KB x 500 is ~35 MB against gRPC's
|
|
// 4 MB default receive limit. 16 keeps a worst-case batch near 1.1 MB.
|
|
"MaxBatchSize": 16,
|
|
// Backlog caps bound the oplog while the peer is offline. Exceeding them is
|
|
// NOT data loss: the oplog is pruned to the ceiling and needs_snapshot is set,
|
|
// so the peer catches up by snapshot resync instead of incrementally. That
|
|
// makes tighter-than-default correct here - it trades a rare full resync for a
|
|
// bounded file. Sized from the soak's 0.80 sf_messages rows/sec (the only
|
|
// non-zero writer measured): ~69k rows/day, so 2 days is ~138k. 250,000 leaves
|
|
// room for burst without approaching the 1,000,000 default.
|
|
"MaxOplogRows": 250000,
|
|
"MaxOplogAge": "2.00:00:00"
|
|
}
|
|
}
|
|
}
|