dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 1 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

docs(docker): record live failover-drill results — standby PASS (27s removal, 0 routing blips), active outage 9s + 4s recovery (R2-01 live drill on 99544985)

Browse Source
This commit is contained in:
Joseph Doherty
2026-07-13 12:01:00 -04:00
parent 9954498548
commit 040cd9a66c
1 changed files with 5 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docker/README.md
+5 -3
View File
@@ -288,12 +288,14 @@ The drill exercises S1 (SBR downing on hard crash), S3 (single active node route
> **Observed results** (plan R2-01 T3):
>
> **PENDING — live drill not yet executed** (requires a dedicated docker cluster rebuild; deferred to avoid disrupting the running environment). The two-mode drill script and its per-direction PASS criteria are in place; the empirical timings below are to be filled by running `DRILL_MODE=standby` then `DRILL_MODE=active` against a freshly-deployed cluster and recording the one-line summaries + the cluster commit SHA.
> **Run 2026-07-13** against a freshly-deployed cluster on `main` @ `99544985` (round-2 merged image; `active=central-a`). Both directions behaved exactly as the design predicts.
>
> | Direction (`DRILL_MODE`) | Outcome | Measured |
> |--------------------------|---------|----------|
> | `standby` (younger-node crash) | _pending live run_ | _pending live run_ |
> | `active` (oldest-node crash) | _pending live run_ | _pending live run_ |
> | `standby` (younger-node crash) | **PASS** — SBR downed+removed the crashed `central-b`; active `central-a` kept all 7 singletons; recovered on restart. | Member removed in **27s** (budget ~25s: 10s detection + 15s stable-after); **0** `/health/active` routing blips (active node never touched); routable **0s** after victim restart. |
> | `active` (oldest-node crash) | **Outage as designed** — killing the oldest/active `central-a` made the younger `central-b` self-down (total central outage — the registered keep-oldest gap); recovered after restarting the victim, `central-b` then assuming Oldest and re-hosting all singletons. | Outage confirmed at **9s**; central routable again **4s** after restarting `central-a`. |
>
> Notes: the `standby` PASS shows the survivable direction is clean end-to-end (SBR `DownUnreachable` decision + per-singleton "Member removed" in the survivor log, zero routing interruption). The `active` result **empirically confirms the deferred keep-oldest topology gap** (master tracker 2026-07-08 / `docs/plans/2026-07-08-deferred-work-register.md`): a hard crash of the active/oldest central node is a total outage until that node (the first seed) is restarted — the remedy remains the pending topology/strategy decision. In-process envelope (`FailoverTimingTests`, plan R2-01 T4) independently measured full failover at **33.7s**.
### Central Failover