Joseph Doherty
139a43fb25
Ran every #116 gate on the local docker rig (main 2d2e9f35), with an auto-down baseline
on the same build:
- active kill: takeover 29s central / 26s site-a; standby kill: downed 29s / 26s, 0 blips;
- boot logs: no deprecation warning, [leader-auto-down] started/unreachable/DOWNING lines;
- partition past the window: accepted dual-active, restart of one side recovers;
- cold start of both pairs together: one cluster each; site-a S&F drains on the survivor.
Extra: docker pause of the active node leaves a permanent second active after unpause on
BOTH strategies (Akka.Cluster ignores ThisActorSystemQuarantinedEvent) — filed as #132.
- docker/docker-compose.leader-auto-down.yml: opt-in overlay (env override) for the central
and site-a pairs; shipped default stays auto-down.
- docker/failover-drill.sh: DRILL_PAIR (central|site-a|site-b|site-c), DRILL_ACTION
(kill|pause), sidecar probes for unpublished site health, strict rejoin + dual-active check.
- docs/plans/2026-11-12-leader-auto-down-rig-drill.md (+ raw logs), docker/README.md.
Refs #116, #106, #132
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
33 lines
3.6 KiB
Plaintext
33 lines
3.6 KiB
Plaintext
pair=central mode=active action=kill active=scadabridge-central-a victim=scadabridge-central-a survivor=scadabridge-central-b
|
|
hit at 2026-11-13T00:48:59Z
|
|
Active kill: waiting for scadabridge-central-b to take over as the active node (victim stays down; budget ~25s + hand-over)...
|
|
survivor logged the Down at +27s
|
|
PASS: scadabridge-central-b took over as active in 27s with the victim still down
|
|
(downed the dead oldest, assumed Oldest, re-hosted the singletons).
|
|
Confirming Traefik routes to the new active node...
|
|
Traefik routing recovered 30s after the hit.
|
|
Recovering scadabridge-central-a (kill -> start)...
|
|
Waiting for the victim to rejoin as a ready standby (scadabridge-central-a /health/ready)...
|
|
Recovered: scadabridge-central-a is ready as standby (rejoined) 3s after recovery.
|
|
Checking for a lingering dual-active (10 samples, 1s apart)...
|
|
Exactly one active node in 10/10 samples.
|
|
Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-central-b):
|
|
[00:49:27 INF] [Central/scadabridge-central-b] Singleton manager started singleton actor [akka://scadabridge/user/site-call-audit-singleton/site-call-audit]
|
|
[00:49:27 INF] [Central/scadabridge-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData
|
|
[00:49:27 WRN] [Central/scadabridge-central-b] ClusterSingletonProxy failed to find an associated singleton named [audit-log-purge] in role [null] after 30 seconds.
|
|
[00:49:27 WRN] [Central/scadabridge-central-b] ClusterSingletonProxy failed to find an associated singleton named [site-call-audit] in role [null] after 30 seconds.
|
|
[00:49:27 WRN] [Central/scadabridge-central-b] ClusterSingletonProxy failed to find an associated singleton named [notification-outbox] in role [null] after 30 seconds.
|
|
[00:49:27 WRN] [Central/scadabridge-central-b] ClusterSingletonProxy failed to find an associated singleton named [pending-deployment-purge] in role [null] after 30 seconds.
|
|
[00:49:27 WRN] [Central/scadabridge-central-b] ClusterSingletonProxy failed to find an associated singleton named [site-audit-reconciliation] in role [null] after 30 seconds.
|
|
[00:49:27 WRN] [Central/scadabridge-central-b] ClusterSingletonProxy failed to find an associated singleton named [audit-log-ingest] in role [null] after 30 seconds.
|
|
[00:49:27 WRN] [Central/scadabridge-central-b] ClusterSingletonProxy failed to find an associated singleton named [kpi-history-recorder] in role [null] after 30 seconds.
|
|
[00:49:27 INF] [Central/scadabridge-central-b] Singleton identified at [akka://scadabridge/user/pending-deployment-purge-singleton/pending-deployment-purge]
|
|
[00:49:27 INF] [Central/scadabridge-central-b] Singleton identified at [akka://scadabridge/user/audit-log-ingest-singleton/audit-log-ingest]
|
|
[00:49:27 INF] [Central/scadabridge-central-b] Singleton identified at [akka://scadabridge/user/audit-log-purge-singleton/audit-log-purge]
|
|
[00:49:27 INF] [Central/scadabridge-central-b] Singleton identified at [akka://scadabridge/user/site-audit-reconciliation-singleton/site-audit-reconciliation]
|
|
[00:49:27 INF] [Central/scadabridge-central-b] Singleton identified at [akka://scadabridge/user/notification-outbox-singleton/notification-outbox]
|
|
[00:49:27 INF] [Central/scadabridge-central-b] Singleton identified at [akka://scadabridge/user/kpi-history-recorder-singleton/kpi-history-recorder]
|
|
[00:49:27 INF] [Central/scadabridge-central-b] Singleton identified at [akka://scadabridge/user/site-call-audit-singleton/site-call-audit]
|
|
SUMMARY pair=central mode=active action=kill down_logged_s=27 rejoin_s=3
|
|
Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary.
|