test(docker): leader-auto-down rig failover drill — #116 rig gate PASS on central + site-a
Ran every #116 gate on the local docker rig (main 2d2e9f35), with an auto-down baseline
on the same build:
- active kill: takeover 29s central / 26s site-a; standby kill: downed 29s / 26s, 0 blips;
- boot logs: no deprecation warning, [leader-auto-down] started/unreachable/DOWNING lines;
- partition past the window: accepted dual-active, restart of one side recovers;
- cold start of both pairs together: one cluster each; site-a S&F drains on the survivor.
Extra: docker pause of the active node leaves a permanent second active after unpause on
BOTH strategies (Akka.Cluster ignores ThisActorSystemQuarantinedEvent) — filed as #132.
- docker/docker-compose.leader-auto-down.yml: opt-in overlay (env override) for the central
and site-a pairs; shipped default stays auto-down.
- docker/failover-drill.sh: DRILL_PAIR (central|site-a|site-b|site-c), DRILL_ACTION
(kill|pause), sidecar probes for unpublished site health, strict rejoin + dual-active check.
- docs/plans/2026-11-12-leader-auto-down-rig-drill.md (+ raw logs), docker/README.md.
Refs #116, #106, #132
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,146 @@
|
||||
# leader-auto-down: docker rig failover drill (ScadaBridge#116)
|
||||
|
||||
**Date:** 2026-11-12 (UTC 00:44–01:14 on 2026-11-13)
|
||||
**Scope:** the local docker rig only (`docker/`, 8 nodes). env2, QA (vq03/vq04) and DEV (vd03) were not touched.
|
||||
**Build:** `scadabridge:latest` rebuilt from `main` @ `2d2e9f35` with `bash docker/deploy.sh`. This build includes Telemetry.Serilog `2d2e9f35`, LocalDb 0.3.7 `12fa7aff` and OPC SDK `e36cd605`.
|
||||
**Strategy under test:** `ScadaBridge:Cluster:SplitBrainResolverStrategy = leader-auto-down` (`LeaderAutoDowningProvider`, #106) on the central pair and the site-a pair. `StableAfter` 15 s, heartbeat 2 s, failure-detection threshold 10 s (rig values, unchanged).
|
||||
**Baseline:** today's default `auto-down` on the same build, run first.
|
||||
**Raw output:** [`2026-11-12-leader-auto-down-rig-drill/`](2026-11-12-leader-auto-down-rig-drill/) (one file per run).
|
||||
|
||||
## Result
|
||||
|
||||
**The #116 rig gate passes on both pairs.** Every gate #116 lists (1–4) passed on central and site-a. The timings match the `auto-down` baseline to within about 2 s.
|
||||
|
||||
One extra scenario failed for both strategies: a node frozen with `docker pause`. The survivor takes over correctly. But when the frozen node is unpaused, it comes back as a **second active node** and stays that way until it is restarted. `auto-down` does exactly the same thing, so this is parity, not a regression. It is filed as **#132** (§5) and does not block this gate.
|
||||
|
||||
| # | Gate | Central | Site-a |
|
||||
|---|---|---|---|
|
||||
| 1/2 | Kill the active (oldest) node: survivor takes over within ~25 s plus hand-over; victim rejoins as standby | **PASS**: Down at +28 s, survivor active at 29 s, Traefik at 30 s, rejoin 2 s | **PASS**: Down at +26 s, survivor active at 26 s, rejoin 11 s |
|
||||
| 1/2 | Kill the standby: it is downed and removed, with no `/health/active` blips | **PASS**: Down at +29 s, 0 blips, rejoin 2 s | **PASS**: Down at +26 s, 0 blips, rejoin 1 s |
|
||||
| 3 | Boot logs: no deprecation warning; `[leader-auto-down] started`, then `is unreachable … after 00:00:15`, then `DOWNING` | **PASS** on both nodes | **PASS** on both nodes |
|
||||
| 4 | Partition held past the window: dual-active happens and both nodes report 200; restarting one side recovers | **PASS**: dual-active at +29 s, still there 60 s after heal, restart recovered in 12 s | **PASS**: dual-active at +27 s, still there 60 s after heal, restart recovered in 32 s |
|
||||
| extra | Cold start of both pairs together (`docker kill` all four, then `docker start` all four) | **PASS**: one 2-member cluster after 6 s, 0 `DOWNING` | **PASS**: one 2-member cluster after 1 s, 0 `DOWNING` |
|
||||
| extra | Site keeps collecting and store-and-forward drains across the failover | n/a | **PASS**: the survivor runs scripts and its S&F depth drops to 0, which includes 3 items the dead node left behind |
|
||||
| extra | Freeze the active node (`docker pause` for about 30 s, then unpause) | Takeover **PASS** (29 s, Traefik 32 s). Victim rejoin **FAIL**: permanent dual-active | Takeover **PASS** (27 s). Victim rejoin **FAIL**: permanent dual-active |
|
||||
|
||||
Each "rejoin" check requires all three: the victim's `/health/ready` is 200, its `/health/active` is 503, and the survivor reports `memberCount 2, unreachableCount 0`. After every rejoin, 10 one-second samples each showed exactly one active node.
|
||||
|
||||
## 1. Method
|
||||
|
||||
- **Config:** a new opt-in overlay, [`docker/docker-compose.leader-auto-down.yml`](../../docker/docker-compose.leader-auto-down.yml). It sets the strategy as an environment override on `central-a/b` and `site-a-a/b`. The mounted appsettings, and the shipped default, still say `auto-down`. Both pairs were switched together and switched back together with `up -d --force-recreate --no-deps central-a central-b site-a-a site-a-b`. There was no rolling restart across the provider change.
|
||||
- **Drill script:** [`docker/failover-drill.sh`](../../docker/failover-drill.sh) was generalised for this run:
|
||||
- `DRILL_PAIR=central|site-a|site-b|site-c`. Site `/health/*` is on container port 8084, which is not published, so the script probes from a throwaway sidecar on `scadabridge-net`.
|
||||
- `DRILL_ACTION=kill|pause`.
|
||||
- A stricter rejoin check (above), followed by a 10-sample dual-active check.
|
||||
- The central behaviour and the Traefik check are unchanged.
|
||||
- **Partition:** iptables `DROP` rules in node B's network namespace, applied from a `NET_ADMIN` sidecar, cutting all traffic between B and A. Nothing else is cut, so both nodes keep SQL, Traefik and the probes. The hold was 50 s on central and 45 s on site-a, longer than the 25 s detection-plus-window. Script: [`partition.sh`](2026-11-12-leader-auto-down-rig-drill/partition.sh).
|
||||
- **Data and S&F:** `/metrics` on 8084 for each site-a node: `scadabridge_site_script_runs_total`, `site_data_connections_{connected,total}`, `store_and_forward_activity_total{Queued,Delivered}` and `store_and_forward_queue_depth_items`.
|
||||
- **Overview:** the local `zb-overview` (`:5320`) page, scraped as text.
|
||||
|
||||
## 2. Baseline: `auto-down` on the same build
|
||||
|
||||
| Run | Result |
|
||||
|---|---|
|
||||
| Boot logs, all 4 nodes | The `auto-down-unreachable-after … keep-majority will be used instead` warning is present on every node. This is the **positive control** for gate 3. |
|
||||
| Central, active kill | Down at +27 s, survivor active at 27 s, Traefik at 30 s, rejoin 3 s, 10/10 single-active ([log](2026-11-12-leader-auto-down-rig-drill/baseline-central-active-kill.log)) |
|
||||
| Site-a, active kill | `Leader is auto-downing` at +26 s, survivor active at 27 s, rejoin 1 s ([log](2026-11-12-leader-auto-down-rig-drill/baseline-site-a-active-kill.log)) |
|
||||
| Central, active pause | Takeover at 29 s, Traefik at 34 s. Then **permanent dual-active**: still both 200 after 120 s. This is the same outcome as `leader-auto-down` ([log](2026-11-12-leader-auto-down-rig-drill/baseline-central-active-pause.log)) |
|
||||
|
||||
The earlier recorded `auto-down` numbers (2026-07-21, `docker/README.md`) were 28 s for active takeover and 27 s for standby removal.
|
||||
|
||||
## 3. `leader-auto-down` runs
|
||||
|
||||
### Gate 3: boot logs
|
||||
|
||||
After the overlay was applied, there were **0** deprecation lines on `central-a`, `central-b`, `site-a-a` and `site-a-b`. `site-b-a`, still on the default, still logged the warning, which serves as a negative control. Each switched node logged:
|
||||
|
||||
```
|
||||
[00:50:59 INF] [Central/scadabridge-central-a] [leader-auto-down] started on akka.tcp://scadabridge@scadabridge-central-a:8081: leader=none (this node is leader: False), unreachable-after=00:00:15, already unreachable=[]
|
||||
[00:51:00 INF] [Central/scadabridge-central-a] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-a:8081 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-central-a:8081); pending unreachable=[]
|
||||
[00:50:55 INF] [Site/scadabridge-site-a-a] [leader-auto-down] started on akka.tcp://scadabridge@scadabridge-site-a-a:8082: leader=none (this node is leader: False), unreachable-after=00:00:15, already unreachable=[]
|
||||
```
|
||||
|
||||
The switch itself was a cold recreate of both pairs at the same moment. Each pair formed one 2-member cluster, with nobody downed.
|
||||
|
||||
### Gate 1: central
|
||||
|
||||
Active kill ([log](2026-11-12-leader-auto-down-rig-drill/lad-central-active-kill.log)). The kill was at 00:51:14:
|
||||
|
||||
```
|
||||
[00:51:27 WRN] [Central/scadabridge-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-a:8081 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-central-b:8081 is leader: False)
|
||||
[00:51:27 INF] [Central/scadabridge-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-b:8081 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-central-b:8081); pending unreachable=[]
|
||||
[00:51:42 WRN] [Central/scadabridge-central-b] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-central-a:8081: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-central-b:8081) is leader
|
||||
[00:51:43 INF] [Central/scadabridge-central-b] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
|
||||
```
|
||||
|
||||
The survivor was active at 29 s and Traefik routed to it at 30 s. The victim rejoined as standby 2 s after `docker start`.
|
||||
|
||||
Afterwards the survivor logged `lost leadership (leader is now central-a)`. This is expected. The Akka *leader* is the lowest Up address, while the *active* node is the oldest Up member, and central-b stayed active.
|
||||
|
||||
Standby kill ([log](2026-11-12-leader-auto-down-rig-drill/lad-central-standby-kill.log)): unreachable at +14 s, `DOWNING` at +29 s, **0** `/health/active` blips (node and Traefik polled every second), rejoin 2 s.
|
||||
|
||||
A second central standby kill was run by mistake with its output discarded, between the two site-a kill runs. It exited without error, and the pair was healthy afterwards. It is not counted.
|
||||
|
||||
### Gate 2: site-a
|
||||
|
||||
Active kill ([log](2026-11-12-leader-auto-down-rig-drill/lad-site-a-active-kill.log), [metrics](2026-11-12-leader-auto-down-rig-drill/lad-site-a-active-kill.metrics)). The kill was at 01:01:56. Unreachable at 01:02:07, `DOWNING` at 01:02:22 (+26 s), `Younger -> Oldest` at 01:02:23, rejoin 11 s.
|
||||
|
||||
Data path across the failover:
|
||||
|
||||
```
|
||||
01:01:55 site-a-a script_runs=525 dc_connected=1/2 sf_queued=393 sf_delivered=390 sf_depth=3 <- active before the kill
|
||||
01:02:46 site-a-b script_runs=17 dc_connected=1/2 sf_queued=12 sf_delivered=15 sf_depth=0 <- survivor, 50 s after the kill
|
||||
01:02:53 site-a-b script_runs=21 dc_connected=1/2 sf_queued=15 sf_delivered=16
|
||||
01:04:14 site-a-b script_runs=85 dc_connected=1/2 sf_queued=63 sf_delivered=65
|
||||
```
|
||||
|
||||
The survivor opened the data connection and kept running scripts. It delivered 3 more S&F items than it queued itself, which are the 3 pending items the dead node left in the replicated buffer, so the inherited queue drained to 0.
|
||||
|
||||
`dc_connected=1/2` is the rig's steady state before and after: one of site-a's two configured connections never connects on this rig. It is not caused by the drill.
|
||||
|
||||
Standby kill ([log](2026-11-12-leader-auto-down-rig-drill/lad-site-a-standby-kill.log)): `DOWNING` at +26 s, **0** active blips, rejoin 1 s. Collection on the active node carried on throughout (script runs 21 → 85).
|
||||
|
||||
### Gate 4: partition
|
||||
|
||||
Central ([log](2026-11-12-leader-auto-down-rig-drill/lad-central-partition.log)). Both sides saw the other unreachable at +12 s, and **both** logged `DOWNING` of the other at +27 s. Both were 200 from +29 s. After the cut was removed at +50 s, both were still 200 at every 10 s sample for 60 s, because each side has quarantined the other.
|
||||
|
||||
During the dual-active, Overview showed `Split brain · …central-a:8081 vs …central-b:8081`, and each node reported `members 1`. Restarting `central-b` brought back one 2-member cluster with central-a active in **12 s**.
|
||||
|
||||
Site-a ([log](2026-11-12-leader-auto-down-rig-drill/lad-site-a-partition.log)): the same sequence, with dual-active from +27 s and Overview showing `Split brain`. Both site nodes had the data connection open and were running scripts, so the data was collected twice. That is the accepted trade. Restarting `site-a-b` recovered in **32 s**.
|
||||
|
||||
This matches plan §6 and the in-process `LeaderAutoDownScenarioTests.PartitionLongerThanWindow_EachSideDownsTheOther_BothStayUp`.
|
||||
|
||||
### Cold start of both pairs together
|
||||
|
||||
[Log](2026-11-12-leader-auto-down-rig-drill/lad-cold-restart-both-pairs.log). All four nodes were killed, then all four started. One 2-member cluster formed in 1 s on site-a and 6 s on central, with exactly one active node per pair and 0 `DOWNING` lines. The same result came from the `--force-recreate` switches at 00:50:42 and 01:01:04.
|
||||
|
||||
## 4. Rig state afterwards
|
||||
|
||||
Both pairs were recreated together on the default (no overlay) at about 01:14. The rig is on `auto-down` again, and the deprecation warning is back on all four nodes, which is expected for the default.
|
||||
|
||||
All 8 nodes are Up. Each pair has one active node (200) and one standby (503): central-a, site-a-a, site-b-a and site-c-a are active ([final status](2026-11-12-leader-auto-down-rig-drill/final-status.txt)).
|
||||
|
||||
## 5. Finding: a frozen node comes back as a permanent second active (#132)
|
||||
|
||||
Runs: [central LAD](2026-11-12-leader-auto-down-rig-drill/lad-central-active-pause.log), [site-a LAD](2026-11-12-leader-auto-down-rig-drill/lad-site-a-active-pause.log), [central auto-down baseline](2026-11-12-leader-auto-down-rig-drill/baseline-central-active-pause.log).
|
||||
|
||||
After `docker unpause`, the frozen node (central-b in the first run) logged:
|
||||
|
||||
```
|
||||
[00:53:22 WRN] … Scheduled sending of heartbeat was delayed. Previous heartbeat was sent [33472.866] ms ago …
|
||||
[00:53:22 WRN] … [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-a:8081 is unreachable; it will be downed if still unreachable after 00:00:15 …
|
||||
[00:53:28 ERR] … AssociationError … Error [The remote system has quarantined this system. No further associations to the remote system are possible until this system is restarted.]
|
||||
[00:53:37 WRN] … [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-central-a:8081: unreachable for 00:00:15 and this node (…central-b:8081) is leader
|
||||
```
|
||||
|
||||
After that, both nodes reported `/health/active` 200 until the zombie was restarted. That took 180 s on central and 90 s on site-a, both times when the drill script gave up.
|
||||
|
||||
The cause is in Akka. Akka.Remote 1.5.62 publishes `ThisActorSystemQuarantinedEvent` (`EndpointManager`), but Akka.Cluster 1.5.62's `ClusterCoreDaemon` subscribes only to the opposite `QuarantinedEvent`. Nothing tells a node that it has been removed. The `auto-down` baseline behaves identically, so `LeaderAutoDowningProvider` is a faithful port in this respect too. It was **not** changed.
|
||||
|
||||
The fix (self-down on `ThisActorSystemQuarantinedEvent`) would also make both sides of a healed partition restart. That changes gate 4's contract, so it is left for an owner decision in #132.
|
||||
|
||||
## 6. Next step
|
||||
|
||||
The next rollout step is **`docker-env2`**, and it **needs owner approval**. Apply the same strategy to its central pair and one site pair, stopping and starting each pair together, and re-run the drill there. After that come the QA pair vq03/vq04, then DEV vd03 and its sites. The default flip and the docs and register row 70 updates wait until the rollout is complete (#116 "Rollout after the gates pass").
|
||||
|
||||
Decide #132 before the default flips.
|
||||
@@ -0,0 +1,32 @@
|
||||
pair=central mode=active action=kill active=scadabridge-central-a victim=scadabridge-central-a survivor=scadabridge-central-b
|
||||
hit at 2026-11-13T00:48:59Z
|
||||
Active kill: waiting for scadabridge-central-b to take over as the active node (victim stays down; budget ~25s + hand-over)...
|
||||
survivor logged the Down at +27s
|
||||
PASS: scadabridge-central-b took over as active in 27s with the victim still down
|
||||
(downed the dead oldest, assumed Oldest, re-hosted the singletons).
|
||||
Confirming Traefik routes to the new active node...
|
||||
Traefik routing recovered 30s after the hit.
|
||||
Recovering scadabridge-central-a (kill -> start)...
|
||||
Waiting for the victim to rejoin as a ready standby (scadabridge-central-a /health/ready)...
|
||||
Recovered: scadabridge-central-a is ready as standby (rejoined) 3s after recovery.
|
||||
Checking for a lingering dual-active (10 samples, 1s apart)...
|
||||
Exactly one active node in 10/10 samples.
|
||||
Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-central-b):
|
||||
[00:49:27 INF] [Central/scadabridge-central-b] Singleton manager started singleton actor [akka://scadabridge/user/site-call-audit-singleton/site-call-audit]
|
||||
[00:49:27 INF] [Central/scadabridge-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData
|
||||
[00:49:27 WRN] [Central/scadabridge-central-b] ClusterSingletonProxy failed to find an associated singleton named [audit-log-purge] in role [null] after 30 seconds.
|
||||
[00:49:27 WRN] [Central/scadabridge-central-b] ClusterSingletonProxy failed to find an associated singleton named [site-call-audit] in role [null] after 30 seconds.
|
||||
[00:49:27 WRN] [Central/scadabridge-central-b] ClusterSingletonProxy failed to find an associated singleton named [notification-outbox] in role [null] after 30 seconds.
|
||||
[00:49:27 WRN] [Central/scadabridge-central-b] ClusterSingletonProxy failed to find an associated singleton named [pending-deployment-purge] in role [null] after 30 seconds.
|
||||
[00:49:27 WRN] [Central/scadabridge-central-b] ClusterSingletonProxy failed to find an associated singleton named [site-audit-reconciliation] in role [null] after 30 seconds.
|
||||
[00:49:27 WRN] [Central/scadabridge-central-b] ClusterSingletonProxy failed to find an associated singleton named [audit-log-ingest] in role [null] after 30 seconds.
|
||||
[00:49:27 WRN] [Central/scadabridge-central-b] ClusterSingletonProxy failed to find an associated singleton named [kpi-history-recorder] in role [null] after 30 seconds.
|
||||
[00:49:27 INF] [Central/scadabridge-central-b] Singleton identified at [akka://scadabridge/user/pending-deployment-purge-singleton/pending-deployment-purge]
|
||||
[00:49:27 INF] [Central/scadabridge-central-b] Singleton identified at [akka://scadabridge/user/audit-log-ingest-singleton/audit-log-ingest]
|
||||
[00:49:27 INF] [Central/scadabridge-central-b] Singleton identified at [akka://scadabridge/user/audit-log-purge-singleton/audit-log-purge]
|
||||
[00:49:27 INF] [Central/scadabridge-central-b] Singleton identified at [akka://scadabridge/user/site-audit-reconciliation-singleton/site-audit-reconciliation]
|
||||
[00:49:27 INF] [Central/scadabridge-central-b] Singleton identified at [akka://scadabridge/user/notification-outbox-singleton/notification-outbox]
|
||||
[00:49:27 INF] [Central/scadabridge-central-b] Singleton identified at [akka://scadabridge/user/kpi-history-recorder-singleton/kpi-history-recorder]
|
||||
[00:49:27 INF] [Central/scadabridge-central-b] Singleton identified at [akka://scadabridge/user/site-call-audit-singleton/site-call-audit]
|
||||
SUMMARY pair=central mode=active action=kill down_logged_s=27 rejoin_s=3
|
||||
Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary.
|
||||
@@ -0,0 +1,11 @@
|
||||
pair=central mode=active action=pause active=scadabridge-central-a victim=scadabridge-central-a survivor=scadabridge-central-b
|
||||
hit at 2026-11-13T00:58:00Z
|
||||
Active pause: waiting for scadabridge-central-b to take over as the active node (victim stays down; budget ~25s + hand-over)...
|
||||
survivor logged the Down at +29s
|
||||
PASS: scadabridge-central-b took over as active in 29s with the victim still down
|
||||
(downed the dead oldest, assumed Oldest, re-hosted the singletons).
|
||||
Confirming Traefik routes to the new active node...
|
||||
Traefik routing recovered 34s after the hit.
|
||||
Recovering scadabridge-central-a (pause -> unpause)...
|
||||
Waiting for the victim to rejoin as a ready standby (scadabridge-central-a /health/ready)...
|
||||
FAIL: scadabridge-central-a not a ready, rejoined standby 120s after recovery
|
||||
@@ -0,0 +1,20 @@
|
||||
pair=site-a mode=active action=kill active=scadabridge-site-a-a victim=scadabridge-site-a-a survivor=scadabridge-site-a-b
|
||||
hit at 2026-11-13T00:49:50Z
|
||||
Active kill: waiting for scadabridge-site-a-b to take over as the active node (victim stays down; budget ~25s + hand-over)...
|
||||
survivor logged the Down at +27s
|
||||
PASS: scadabridge-site-a-b took over as active in 27s with the victim still down
|
||||
(downed the dead oldest, assumed Oldest, re-hosted the singletons).
|
||||
Recovering scadabridge-site-a-a (kill -> start)...
|
||||
Waiting for the victim to rejoin as a ready standby (scadabridge-site-a-a /health/ready)...
|
||||
Recovered: scadabridge-site-a-a is ready as standby (rejoined) 1s after recovery.
|
||||
Checking for a lingering dual-active (10 samples, 1s apart)...
|
||||
Exactly one active node in 10/10 samples.
|
||||
Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-site-a-b):
|
||||
[00:50:16 INF] [Site/scadabridge-site-a-b] Cluster Node [akka.tcp://scadabridge@scadabridge-site-a-b:8082] - Leader is auto-downing unreachable node [akka.tcp://scadabridge@scadabridge-site-a-a:8082]
|
||||
[00:50:16 INF] [Site/scadabridge-site-a-b] Cluster Node [akka.tcp://scadabridge@scadabridge-site-a-b:8082] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-site-a-a:8082] as [Down]
|
||||
[00:50:17 INF] [Site/scadabridge-site-a-b] Member removed [akka.tcp://scadabridge@scadabridge-site-a-a:8082]
|
||||
[00:50:17 INF] [Site/scadabridge-site-a-b] Member removed [akka.tcp://scadabridge@scadabridge-site-a-a:8082]
|
||||
[00:50:17 INF] [Site/scadabridge-site-a-b] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
|
||||
[00:50:17 INF] [Site/scadabridge-site-a-b] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
|
||||
SUMMARY pair=site-a mode=active action=kill down_logged_s=27 rejoin_s=1
|
||||
Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary.
|
||||
@@ -0,0 +1,12 @@
|
||||
central-a ready=200 active=200
|
||||
central-b ready=200 active=503
|
||||
site-a-a ready=200 active=200
|
||||
site-a-b ready=200 active=503
|
||||
site-b-a ready=200 active=200
|
||||
site-b-b ready=200 active=503
|
||||
site-c-a ready=200 active=200
|
||||
site-c-b ready=200 active=503
|
||||
central-a strategy env=0 deprecation=1 lad=0
|
||||
central-b strategy env=0 deprecation=1 lad=0
|
||||
site-a-a strategy env=0 deprecation=1 lad=0
|
||||
site-a-b strategy env=0 deprecation=1 lad=0
|
||||
@@ -0,0 +1,35 @@
|
||||
pair=central mode=active action=kill active=scadabridge-central-a victim=scadabridge-central-a survivor=scadabridge-central-b
|
||||
hit at 2026-11-13T00:51:14Z
|
||||
Active kill: waiting for scadabridge-central-b to take over as the active node (victim stays down; budget ~25s + hand-over)...
|
||||
survivor logged the Down at +29s
|
||||
PASS: scadabridge-central-b took over as active in 29s with the victim still down
|
||||
(downed the dead oldest, assumed Oldest, re-hosted the singletons).
|
||||
Confirming Traefik routes to the new active node...
|
||||
Traefik routing recovered 30s after the hit.
|
||||
Recovering scadabridge-central-a (kill -> start)...
|
||||
Waiting for the victim to rejoin as a ready standby (scadabridge-central-a /health/ready)...
|
||||
Recovered: scadabridge-central-a is ready as standby (rejoined) 2s after recovery.
|
||||
Checking for a lingering dual-active (10 samples, 1s apart)...
|
||||
Exactly one active node in 10/10 samples.
|
||||
Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-central-b):
|
||||
[00:51:27 WRN] [Central/scadabridge-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-a:8081 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-central-b:8081 is leader: False)
|
||||
[00:51:27 INF] [Central/scadabridge-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-b:8081 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-central-b:8081); pending unreachable=[]
|
||||
[00:51:42 WRN] [Central/scadabridge-central-b] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-central-a:8081: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-central-b:8081) is leader
|
||||
[00:51:42 INF] [Central/scadabridge-central-b] Cluster Node [akka.tcp://scadabridge@scadabridge-central-b:8081] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-central-a:8081] as [Down]
|
||||
[00:51:43 INF] [Central/scadabridge-central-b] Member removed [akka.tcp://scadabridge@scadabridge-central-a:8081]
|
||||
[00:51:43 INF] [Central/scadabridge-central-b] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
|
||||
[00:51:43 INF] [Central/scadabridge-central-b] Member removed [akka.tcp://scadabridge@scadabridge-central-a:8081], previous oldest [["UniqueAddress: (akka.tcp://scadabridge@scadabridge-central-a:8081, 1434844774)"]]
|
||||
[00:51:43 INF] [Central/scadabridge-central-b] Member removed [akka.tcp://scadabridge@scadabridge-central-a:8081], previous oldest [["UniqueAddress: (akka.tcp://scadabridge@scadabridge-central-a:8081, 1434844774)"]]
|
||||
[00:51:43 INF] [Central/scadabridge-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData
|
||||
[00:51:43 INF] [Central/scadabridge-central-b] Member removed [akka.tcp://scadabridge@scadabridge-central-a:8081], previous oldest [["UniqueAddress: (akka.tcp://scadabridge@scadabridge-central-a:8081, 1434844774)"]]
|
||||
[00:51:43 INF] [Central/scadabridge-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData
|
||||
[00:51:43 INF] [Central/scadabridge-central-b] Member removed [akka.tcp://scadabridge@scadabridge-central-a:8081], previous oldest [["UniqueAddress: (akka.tcp://scadabridge@scadabridge-central-a:8081, 1434844774)"]]
|
||||
[00:51:43 INF] [Central/scadabridge-central-b] Member removed [akka.tcp://scadabridge@scadabridge-central-a:8081], previous oldest [["UniqueAddress: (akka.tcp://scadabridge@scadabridge-central-a:8081, 1434844774)"]]
|
||||
[00:51:43 INF] [Central/scadabridge-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData
|
||||
[00:51:43 INF] [Central/scadabridge-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData
|
||||
[00:51:43 INF] [Central/scadabridge-central-b] Member removed [akka.tcp://scadabridge@scadabridge-central-a:8081], previous oldest [["UniqueAddress: (akka.tcp://scadabridge@scadabridge-central-a:8081, 1434844774)"]]
|
||||
[00:51:43 INF] [Central/scadabridge-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData
|
||||
[00:51:43 INF] [Central/scadabridge-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData
|
||||
[00:51:46 INF] [Central/scadabridge-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-b:8081 lost leadership (leader is now akka.tcp://scadabridge@scadabridge-central-a:8081); pending unreachable=[]
|
||||
SUMMARY pair=central mode=active action=kill down_logged_s=29 rejoin_s=2
|
||||
Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary.
|
||||
@@ -0,0 +1,11 @@
|
||||
pair=central mode=active action=pause active=scadabridge-central-b victim=scadabridge-central-b survivor=scadabridge-central-a
|
||||
hit at 2026-11-13T00:52:50Z
|
||||
Active pause: waiting for scadabridge-central-a to take over as the active node (victim stays down; budget ~25s + hand-over)...
|
||||
survivor logged the Down at +29s
|
||||
PASS: scadabridge-central-a took over as active in 29s with the victim still down
|
||||
(downed the dead oldest, assumed Oldest, re-hosted the singletons).
|
||||
Confirming Traefik routes to the new active node...
|
||||
Traefik routing recovered 32s after the hit.
|
||||
Recovering scadabridge-central-b (pause -> unpause)...
|
||||
Waiting for the victim to rejoin as a ready standby (scadabridge-central-b /health/ready)...
|
||||
FAIL: scadabridge-central-b not a ready, rejoined standby 180s after recovery
|
||||
@@ -0,0 +1,23 @@
|
||||
01:07:22 before: scadabridge-central-a active=200 scadabridge-central-b active=503 (A ip 192.168.147.6)
|
||||
01:07:23 partition ON (B drops all traffic to/from A)
|
||||
01:07:51 DUAL-ACTIVE reached at +29s
|
||||
01:08:12 end of hold (+50s): scadabridge-central-a active=200 scadabridge-central-b active=200
|
||||
01:08:12 partition HEALED
|
||||
01:08:22 +10s after heal: scadabridge-central-a active=200 scadabridge-central-b active=200
|
||||
01:08:32 +20s after heal: scadabridge-central-a active=200 scadabridge-central-b active=200
|
||||
01:08:43 +30s after heal: scadabridge-central-a active=200 scadabridge-central-b active=200
|
||||
01:08:53 +40s after heal: scadabridge-central-a active=200 scadabridge-central-b active=200
|
||||
01:09:03 +50s after heal: scadabridge-central-a active=200 scadabridge-central-b active=200
|
||||
01:09:13 +60s after heal: scadabridge-central-a active=200 scadabridge-central-b active=200
|
||||
--- scadabridge-central-a downing lines
|
||||
[01:07:35 WRN] [Central/scadabridge-central-a] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-b:8081 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-central-a:8081 is leader: True)
|
||||
[01:07:50 WRN] [Central/scadabridge-central-a] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-central-b:8081: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-central-a:8081) is leader
|
||||
[01:07:50 INF] [Central/scadabridge-central-a] Cluster Node [akka.tcp://scadabridge@scadabridge-central-a:8081] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-central-b:8081] as [Down]
|
||||
--- scadabridge-central-b downing lines
|
||||
[01:07:35 WRN] [Central/scadabridge-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-a:8081 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-central-b:8081 is leader: False)
|
||||
[01:07:35 INF] [Central/scadabridge-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-b:8081 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-central-b:8081); pending unreachable=[]
|
||||
[01:07:50 WRN] [Central/scadabridge-central-b] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-central-a:8081: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-central-b:8081) is leader
|
||||
[01:07:50 INF] [Central/scadabridge-central-b] Cluster Node [akka.tcp://scadabridge@scadabridge-central-b:8081] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-central-a:8081] as [Down]
|
||||
restart-one-side recovered in 12s
|
||||
central-a ready=200 active=200
|
||||
central-b ready=200 active=503
|
||||
@@ -0,0 +1,25 @@
|
||||
pair=central mode=standby action=kill active=scadabridge-central-b victim=scadabridge-central-a survivor=scadabridge-central-b
|
||||
hit at 2026-11-13T00:52:03Z
|
||||
Standby kill: waiting for scadabridge-central-b to DOWN+REMOVE the victim (budget ~25s)...
|
||||
PASS: survivor downed/removed the victim in 29s (budget ~25s: 10s detection + 15s down window).
|
||||
Active-node /health/active blips during the drill: 0 (expected 0 — the active node was never touched).
|
||||
Recovering scadabridge-central-a (kill -> start)...
|
||||
Waiting for the victim to rejoin as a ready standby (scadabridge-central-a /health/ready)...
|
||||
Recovered: scadabridge-central-a is ready as standby (rejoined) 2s after recovery.
|
||||
Checking for a lingering dual-active (10 samples, 1s apart)...
|
||||
Exactly one active node in 10/10 samples.
|
||||
Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-central-b):
|
||||
[00:52:17 WRN] [Central/scadabridge-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-a:8081 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-central-b:8081 is leader: False)
|
||||
[00:52:17 INF] [Central/scadabridge-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-b:8081 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-central-b:8081); pending unreachable=[]
|
||||
[00:52:32 WRN] [Central/scadabridge-central-b] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-central-a:8081: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-central-b:8081) is leader
|
||||
[00:52:32 INF] [Central/scadabridge-central-b] Cluster Node [akka.tcp://scadabridge@scadabridge-central-b:8081] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-central-a:8081] as [Down]
|
||||
[00:52:33 INF] [Central/scadabridge-central-b] Member removed [akka.tcp://scadabridge@scadabridge-central-a:8081]
|
||||
[00:52:33 INF] [Central/scadabridge-central-b] Member removed [akka.tcp://scadabridge@scadabridge-central-a:8081]
|
||||
[00:52:33 INF] [Central/scadabridge-central-b] Member removed [akka.tcp://scadabridge@scadabridge-central-a:8081]
|
||||
[00:52:33 INF] [Central/scadabridge-central-b] Member removed [akka.tcp://scadabridge@scadabridge-central-a:8081]
|
||||
[00:52:33 INF] [Central/scadabridge-central-b] Member removed [akka.tcp://scadabridge@scadabridge-central-a:8081]
|
||||
[00:52:33 INF] [Central/scadabridge-central-b] Member removed [akka.tcp://scadabridge@scadabridge-central-a:8081]
|
||||
[00:52:33 INF] [Central/scadabridge-central-b] Member removed [akka.tcp://scadabridge@scadabridge-central-a:8081]
|
||||
[00:52:34 INF] [Central/scadabridge-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-b:8081 lost leadership (leader is now akka.tcp://scadabridge@scadabridge-central-a:8081); pending unreachable=[]
|
||||
SUMMARY pair=central mode=standby action=kill down_logged_s=29 rejoin_s=2
|
||||
Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary.
|
||||
@@ -0,0 +1,11 @@
|
||||
01:12:12 cold: docker kill + start of central-a/b and site-a-a/b together
|
||||
site-a: one 2-member cluster, both ready, +1s
|
||||
central: one 2-member cluster, both ready, +6s
|
||||
central-a ready=200 active=200
|
||||
central-b ready=200 active=503
|
||||
site-a-a ready=200 active=200
|
||||
site-a-b ready=200 active=503
|
||||
central-a: 0 DOWNING lines, 0 deprecation lines
|
||||
central-b: 0 DOWNING lines, 0 deprecation lines
|
||||
site-a-a: 0 DOWNING lines, 0 deprecation lines
|
||||
site-a-b: 0 DOWNING lines, 0 deprecation lines
|
||||
@@ -0,0 +1,24 @@
|
||||
pair=site-a mode=active action=kill active=scadabridge-site-a-a victim=scadabridge-site-a-a survivor=scadabridge-site-a-b
|
||||
hit at 2026-11-13T01:01:56Z
|
||||
Active kill: waiting for scadabridge-site-a-b to take over as the active node (victim stays down; budget ~25s + hand-over)...
|
||||
survivor logged the Down at +26s
|
||||
PASS: scadabridge-site-a-b took over as active in 26s with the victim still down
|
||||
(downed the dead oldest, assumed Oldest, re-hosted the singletons).
|
||||
Recovering scadabridge-site-a-a (kill -> start)...
|
||||
Waiting for the victim to rejoin as a ready standby (scadabridge-site-a-a /health/ready)...
|
||||
Recovered: scadabridge-site-a-a is ready as standby (rejoined) 11s after recovery.
|
||||
Checking for a lingering dual-active (10 samples, 1s apart)...
|
||||
Exactly one active node in 10/10 samples.
|
||||
Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-site-a-b):
|
||||
[01:02:07 WRN] [Site/scadabridge-site-a-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-site-a-a:8082 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-site-a-b:8082 is leader: False)
|
||||
[01:02:07 INF] [Site/scadabridge-site-a-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-site-a-b:8082 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-site-a-b:8082); pending unreachable=[]
|
||||
[01:02:22 WRN] [Site/scadabridge-site-a-b] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-site-a-a:8082: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-site-a-b:8082) is leader
|
||||
[01:02:22 INF] [Site/scadabridge-site-a-b] Cluster Node [akka.tcp://scadabridge@scadabridge-site-a-b:8082] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-site-a-a:8082] as [Down]
|
||||
[01:02:23 INF] [Site/scadabridge-site-a-b] Cluster Node [akka.tcp://scadabridge@scadabridge-site-a-b:8082] - New incarnation of existing member [UniqueAddress: (akka.tcp://scadabridge@scadabridge-site-a-a:8082, 350892847)] is trying to join. Existing will be removed from the cluster and then new member will be allowed to join.
|
||||
[01:02:23 INF] [Site/scadabridge-site-a-b] Member removed [akka.tcp://scadabridge@scadabridge-site-a-a:8082]
|
||||
[01:02:23 INF] [Site/scadabridge-site-a-b] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
|
||||
[01:02:23 INF] [Site/scadabridge-site-a-b] Member removed [akka.tcp://scadabridge@scadabridge-site-a-a:8082], previous oldest [["UniqueAddress: (akka.tcp://scadabridge@scadabridge-site-a-a:8082, 1416696295)"]]
|
||||
[01:02:23 INF] [Site/scadabridge-site-a-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData
|
||||
[01:02:34 INF] [Site/scadabridge-site-a-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-site-a-b:8082 lost leadership (leader is now akka.tcp://scadabridge@scadabridge-site-a-a:8082); pending unreachable=[]
|
||||
SUMMARY pair=site-a mode=active action=kill down_logged_s=26 rejoin_s=11
|
||||
Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary.
|
||||
@@ -0,0 +1,6 @@
|
||||
01:01:55 site-a-a script_runs=525 dc_connected=1/2 sf_queued=393 sf_delivered=390 sf_depth=3
|
||||
01:01:55 site-a-b script_runs=0 dc_connected=0/0 sf_queued=0 sf_delivered=0 sf_depth=2
|
||||
01:02:46 site-a-a script_runs=0 dc_connected=0/0 sf_queued=0 sf_delivered=0 sf_depth=2
|
||||
01:02:46 site-a-b script_runs=17 dc_connected=1/2 sf_queued=12 sf_delivered=15 sf_depth=0
|
||||
01:02:53 site-a-a script_runs=0 dc_connected=0/0 sf_queued=0 sf_delivered=0 sf_depth=2
|
||||
01:02:53 site-a-b script_runs=21 dc_connected=1/2 sf_queued=15 sf_delivered=16 sf_depth=2
|
||||
@@ -0,0 +1,8 @@
|
||||
pair=site-a mode=active action=pause active=scadabridge-site-a-b victim=scadabridge-site-a-b survivor=scadabridge-site-a-a
|
||||
hit at 2026-11-13T01:04:21Z
|
||||
Active pause: waiting for scadabridge-site-a-a to take over as the active node (victim stays down; budget ~25s + hand-over)...
|
||||
PASS: scadabridge-site-a-a took over as active in 27s with the victim still down
|
||||
(downed the dead oldest, assumed Oldest, re-hosted the singletons).
|
||||
Recovering scadabridge-site-a-b (pause -> unpause)...
|
||||
Waiting for the victim to rejoin as a ready standby (scadabridge-site-a-b /health/ready)...
|
||||
FAIL: scadabridge-site-a-b not a ready, rejoined standby 90s after recovery
|
||||
@@ -0,0 +1,2 @@
|
||||
01:06:21 site-a-a script_runs=73 dc_connected=1/2 sf_queued=54 sf_delivered=75 sf_depth=2
|
||||
01:06:21 site-a-b script_runs=186 dc_connected=1/2 sf_queued=138 sf_delivered=118 sf_depth=5
|
||||
@@ -0,0 +1,21 @@
|
||||
01:09:38 before: scadabridge-site-a-a active=200 scadabridge-site-a-b active=503 (A ip 192.168.147.4)
|
||||
01:09:39 partition ON (B drops all traffic to/from A)
|
||||
01:10:05 DUAL-ACTIVE reached at +27s
|
||||
01:10:24 end of hold (+46s): scadabridge-site-a-a active=200 scadabridge-site-a-b active=200
|
||||
01:10:24 partition HEALED
|
||||
01:10:34 +10s after heal: scadabridge-site-a-a active=200 scadabridge-site-a-b active=200
|
||||
01:11:25 +60s after heal: scadabridge-site-a-a active=200 scadabridge-site-a-b active=200
|
||||
--- scadabridge-site-a-a downing lines
|
||||
[01:09:50 WRN] [Site/scadabridge-site-a-a] [leader-auto-down] akka.tcp://scadabridge@scadabridge-site-a-b:8082 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-site-a-a:8082 is leader: True)
|
||||
[01:10:05 WRN] [Site/scadabridge-site-a-a] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-site-a-b:8082: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-site-a-a:8082) is leader
|
||||
[01:10:05 INF] [Site/scadabridge-site-a-a] Cluster Node [akka.tcp://scadabridge@scadabridge-site-a-a:8082] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-site-a-b:8082] as [Down]
|
||||
--- scadabridge-site-a-b downing lines
|
||||
[01:09:50 WRN] [Site/scadabridge-site-a-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-site-a-a:8082 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-site-a-b:8082 is leader: False)
|
||||
[01:09:50 INF] [Site/scadabridge-site-a-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-site-a-b:8082 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-site-a-b:8082); pending unreachable=[]
|
||||
[01:10:05 WRN] [Site/scadabridge-site-a-b] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-site-a-a:8082: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-site-a-b:8082) is leader
|
||||
[01:10:05 INF] [Site/scadabridge-site-a-b] Cluster Node [akka.tcp://scadabridge@scadabridge-site-a-b:8082] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-site-a-a:8082] as [Down]
|
||||
01:11:25 site-a-a script_runs=313 dc_connected=1/2 sf_queued=234 sf_delivered=259 sf_depth=0
|
||||
01:11:25 site-a-b script_runs=61 dc_connected=1/2 sf_queued=45 sf_delivered=45 sf_depth=2
|
||||
restart-one-side recovered in 32s
|
||||
site-a-a ready=200 active=200
|
||||
site-a-b ready=200 active=503
|
||||
@@ -0,0 +1,20 @@
|
||||
pair=site-a mode=standby action=kill active=scadabridge-site-a-b victim=scadabridge-site-a-a survivor=scadabridge-site-a-b
|
||||
hit at 2026-11-13T01:03:34Z
|
||||
Standby kill: waiting for scadabridge-site-a-b to DOWN+REMOVE the victim (budget ~25s)...
|
||||
PASS: survivor downed/removed the victim in 27s (budget ~25s: 10s detection + 15s down window).
|
||||
Active-node /health/active blips during the drill: 0 (expected 0 — the active node was never touched).
|
||||
Recovering scadabridge-site-a-a (kill -> start)...
|
||||
Waiting for the victim to rejoin as a ready standby (scadabridge-site-a-a /health/ready)...
|
||||
Recovered: scadabridge-site-a-a is ready as standby (rejoined) 1s after recovery.
|
||||
Checking for a lingering dual-active (10 samples, 1s apart)...
|
||||
Exactly one active node in 10/10 samples.
|
||||
Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-site-a-b):
|
||||
[01:03:45 WRN] [Site/scadabridge-site-a-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-site-a-a:8082 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-site-a-b:8082 is leader: False)
|
||||
[01:03:45 INF] [Site/scadabridge-site-a-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-site-a-b:8082 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-site-a-b:8082); pending unreachable=[]
|
||||
[01:04:00 WRN] [Site/scadabridge-site-a-b] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-site-a-a:8082: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-site-a-b:8082) is leader
|
||||
[01:04:00 INF] [Site/scadabridge-site-a-b] Cluster Node [akka.tcp://scadabridge@scadabridge-site-a-b:8082] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-site-a-a:8082] as [Down]
|
||||
[01:04:01 INF] [Site/scadabridge-site-a-b] Member removed [akka.tcp://scadabridge@scadabridge-site-a-a:8082]
|
||||
[01:04:01 INF] [Site/scadabridge-site-a-b] Member removed [akka.tcp://scadabridge@scadabridge-site-a-a:8082]
|
||||
[01:04:02 INF] [Site/scadabridge-site-a-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-site-a-b:8082 lost leadership (leader is now akka.tcp://scadabridge@scadabridge-site-a-a:8082); pending unreachable=[]
|
||||
SUMMARY pair=site-a mode=standby action=kill down_logged_s=27 rejoin_s=1
|
||||
Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary.
|
||||
@@ -0,0 +1,4 @@
|
||||
01:02:53 site-a-a script_runs=0 dc_connected=0/0 sf_queued=0 sf_delivered=0 sf_depth=2
|
||||
01:02:53 site-a-b script_runs=21 dc_connected=1/2 sf_queued=15 sf_delivered=16 sf_depth=2
|
||||
01:04:14 site-a-a script_runs=0 dc_connected=0/0 sf_queued=0 sf_delivered=0 sf_depth=1
|
||||
01:04:14 site-a-b script_runs=85 dc_connected=1/2 sf_queued=63 sf_delivered=65 sf_depth=1
|
||||
+12
@@ -0,0 +1,12 @@
|
||||
8 instances · 4 clusters central Leader · central-a
|
||||
Split brain · akka.tcp://scadabridge@scadabridge-central-a:8081 vs akka.tcp://scadabridge@scadabridge-central-b:8081 2 members · 0 unreachable central-a
|
||||
scadabridge-central-a:5000
|
||||
Leader Active Up
|
||||
All 5 required cluster singletons are reachable. leader akka.tcp://scadabridge@scadabridge-central-a:8081 · members 1 · unreachable 0 · self akka.tcp://scadabridge@scadabridge-central-a:8081
|
||||
Open CentralUI ↗ ready 200 · active 200 central-b
|
||||
scadabridge-central-b:5000
|
||||
All 5 required cluster singletons are reachable. leader akka.tcp://scadabridge@scadabridge-central-b:8081 · members 1 · unreachable 0 · self akka.tcp://scadabridge@scadabridge-central-b:8081
|
||||
Open CentralUI ↗ ready 200 · active 200 site-a Leader · site-a-a 2 members · 0 unreachable site-a-a
|
||||
Leader Active Up
|
||||
Site LocalDb reachable. leader akka.tcp://scadabridge@scadabridge-site-a-a:8082 · members 2 · unreachable 0 · self akka.tcp://scadabridge@scadabridge-site-a-a:8082
|
||||
no management link ready 200 · active 200 site-a-b
|
||||
+1
@@ -0,0 +1 @@
|
||||
Split brain · akka.tcp://scadabridge@scadabridge-site-a-a:8082 vs akka.tcp://scadabridge@scadabridge-site-a-b:8082 2 members · 0 unreachable site-a-a
|
||||
@@ -0,0 +1,30 @@
|
||||
#!/usr/bin/env bash
|
||||
# Evidence copy of the helper used for #116 gate 4 on 2026-11-12 (not a supported rig script).
|
||||
# Prerequisites it assumed: a long-lived probe container on the rig network
|
||||
# docker run -d --name sb-drill-probe --network scadabridge-net sb-drill-net:local sleep infinity
|
||||
# and a local image with iptables + curl:
|
||||
# printf 'FROM alpine:latest\nRUN apk add --no-cache iptables curl\n' | docker build -t sb-drill-net:local -
|
||||
# partition.sh <pair> <hold_s> : cut Akka remoting between the two nodes of a pair (iptables in node B's netns),
|
||||
# hold, heal, then watch. Leaves the dual-active in place for the caller to resolve.
|
||||
set -euo pipefail
|
||||
PAIR=$1; HOLD=$2
|
||||
A=scadabridge-$PAIR-a; B=scadabridge-$PAIR-b
|
||||
[ "$PAIR" = central ] && HP=5000 || HP=8084
|
||||
AIP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $A)
|
||||
c() { docker exec sb-drill-probe curl -s -m 2 -o /dev/null -w %{http_code} "http://$1:$HP/health/active" 2>/dev/null || echo ERR; }
|
||||
echo "$(date -u +%T) before: $A active=$(c $A) $B active=$(c $B) (A ip $AIP)"
|
||||
CUT_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ); T0=$(date +%s)
|
||||
docker run --rm --net container:$B --cap-add NET_ADMIN sb-drill-net:local sh -c "iptables -I INPUT -s $AIP -j DROP && iptables -I OUTPUT -d $AIP -j DROP"
|
||||
echo "$(date -u +%T) partition ON (B drops all traffic to/from A)"
|
||||
DUAL_AT=""
|
||||
while (( $(date +%s) - T0 < HOLD )); do
|
||||
a=$(c $A); b=$(c $B)
|
||||
if [ -z "$DUAL_AT" ] && [ "$a" = 200 ] && [ "$b" = 200 ]; then DUAL_AT=$(( $(date +%s) - T0 )); echo "$(date -u +%T) DUAL-ACTIVE reached at +${DUAL_AT}s"; fi
|
||||
sleep 1
|
||||
done
|
||||
echo "$(date -u +%T) end of hold (+$(( $(date +%s)-T0 ))s): $A active=$(c $A) $B active=$(c $B)"
|
||||
docker run --rm --net container:$B --cap-add NET_ADMIN sb-drill-net:local sh -c "iptables -D INPUT -s $AIP -j DROP && iptables -D OUTPUT -d $AIP -j DROP"
|
||||
echo "$(date -u +%T) partition HEALED"
|
||||
for i in $(seq 1 6); do sleep 10; echo "$(date -u +%T) +$((i*10))s after heal: $A active=$(c $A) $B active=$(c $B)"; done
|
||||
echo "--- $A downing lines"; docker logs --since $CUT_AT $A 2>&1 | grep -E "leader-auto-down|auto-downing|Marking unreachable|quarantined this system" | head -6
|
||||
echo "--- $B downing lines"; docker logs --since $CUT_AT $B 2>&1 | grep -E "leader-auto-down|auto-downing|Marking unreachable|quarantined this system" | head -6
|
||||
Reference in New Issue
Block a user