dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 5 Pull Requests Actions Packages Projects Releases Wiki Activity

Deploy gate: leader-auto-down rollout. Rig drill PASSED (2026-11-12); next env2 → QA → DEV #116

New Issue
Open
opened 2026-11-10 04:32:09 -05:00 by dohertj2 · 7 comments
dohertj2 commented 2026-11-10 04:32:09 -05:00
Owner
Copy Link
Copy Source

Status (2026-11-12): rig gate PASSED; docker-env2 gate PASSED (env2 now soaking on leader-auto-down). All four gates passed on central and site-a, see the comments below and docs/plans/2026-11-12-leader-auto-down-rig-drill.md. The freeze / healed-partition finding #132 is fixed and closed (952db02a). With it, a frozen node and both sides of a healed partition now converge on one active node with no operator action (rig re-run, drill doc §7).

  • Rig: gates 1–4 on central and site-a (2026-11-12, 139a43fb)
  • Rig: #132 freeze gate (DRILL_ACTION=pause) and split-then-heal gate on central and site-a, plus a kill regression (2026-11-12, 952db02a)
  • docker-env2: all gates on central and site-x incl. freeze and split-then-heal, plus a kill regression (2026-11-12, image scadabridge:env2-116 from a2e9d718 + e8a743de; docs/plans/2026-11-12-leader-auto-down-env2-drill.md). env2 stays on leader-auto-down as the soak
  • QA pair vq03/vq04: next step, needs owner approval
  • DEV vd03 and its sites
  • Flip the shipped default; update CLAUDE.md, Component-ClusterInfrastructure.md and docker/README.md; close register row 70

Deploy gate for the leader-auto-down downing strategy (in-repo LeaderAutoDowningProvider, ScadaBridge#106). The code and in-process tests are done; the strategy is opt-in and not the default. This issue covers the live gates that must pass before the default flips from auto-down, and the rollout after that.

Not run as part of #106: the docker failover drill. It was deliberately skipped because running it would have rebuilt the shared scadabridge:latest image and reused the shared rig's named volumes while another agent was working in the repo.

Gates (rig first)

  1. On the docker rig, set ScadaBridge:Cluster:SplitBrainResolverStrategy = "leader-auto-down" in both docker/central-node-{a,b}/appsettings.Central.json. Stop and start the pair together, then run:
    • DRILL_MODE=active bash docker/failover-drill.sh — kills the oldest node. The survivor must take over within about 25s, and the victim must rejoin as standby.
    • bash docker/failover-drill.sh (standby mode) — the standby is downed and removed, with no /health/active blips.
  2. Do the same on the site-a pair (docker/site-a-node-{a,b}/appsettings.Site.json). failover-drill.sh only targets central today, so a site variant is needed: docker kill the site's oldest node, check the survivor logs [leader-auto-down] DOWNING, the site comes back Up, and the victim rejoins.
  3. Boot logs on every node show no auto-down-unreachable-after … keep-majority will be used instead deprecation warning. They should show the [leader-auto-down] started on … line, and on the drill victim's survivor [leader-auto-down] … is unreachable; it will be downed if still unreachable after 00:00:15 followed by DOWNING.
  4. Partition check (plan §6, contract changed by #132): the accepted dual-active still happens while the partition lasts, and both nodes report /health/active 200. After the heal, exactly one node restarts (the higher address), and the pair converges on one active node with no operator action. Rig: central +76 s and site-a +111 s after the heal. In-process: ShutOutRestartScenarioTests.HealedPartition_BothShutOut_OnlyTheHigherAddressRestarts_AndRejoins. LeaderAutoDownScenarioTests.PartitionLongerThanWindow_EachSideDownsTheOther_BothStayUp covers the during-partition half.

Rollout after the gates pass

Rig → docker-env2 → QA pair vq03/vq04 → DEV vd03 and sites. Each pair stops and starts together; never do a rolling restart across a downing-provider change. The change is an appsettings flip only. Rollback is setting the value back to auto-down. Then switch the shipped default, update CLAUDE.md, Component-ClusterInfrastructure.md and docker/README.md, and close register row 70.

Plan: docs/plans/2026-11-09-autodowning-replacement-plan.md §5–6. Parent: #106, #51.

**Status (2026-11-12):** rig gate PASSED; docker-env2 gate PASSED (env2 now soaking on `leader-auto-down`). All four gates passed on central and site-a, see the comments below and `docs/plans/2026-11-12-leader-auto-down-rig-drill.md`. The freeze / healed-partition finding #132 is **fixed and closed** (`952db02a`). With it, a frozen node and both sides of a healed partition now converge on one active node with no operator action (rig re-run, drill doc §7). - [x] Rig: gates 1–4 on central and site-a (2026-11-12, `139a43fb`) - [x] Rig: #132 freeze gate (`DRILL_ACTION=pause`) and split-then-heal gate on central and site-a, plus a kill regression (2026-11-12, `952db02a`) - [x] `docker-env2`: all gates on central and site-x incl. freeze and split-then-heal, plus a kill regression (2026-11-12, image `scadabridge:env2-116` from `a2e9d718` + `e8a743de`; `docs/plans/2026-11-12-leader-auto-down-env2-drill.md`). env2 stays on `leader-auto-down` as the soak - [ ] QA pair vq03/vq04: **next step, needs owner approval** - [ ] DEV vd03 and its sites - [ ] Flip the shipped default; update CLAUDE.md, `Component-ClusterInfrastructure.md` and `docker/README.md`; close register row 70 --- Deploy gate for the `leader-auto-down` downing strategy (in-repo `LeaderAutoDowningProvider`, ScadaBridge#106). The code and in-process tests are done; the strategy is **opt-in and not the default**. This issue covers the live gates that must pass before the default flips from `auto-down`, and the rollout after that. **Not run as part of #106:** the docker failover drill. It was deliberately skipped because running it would have rebuilt the shared `scadabridge:latest` image and reused the shared rig's named volumes while another agent was working in the repo. ### Gates (rig first) 1. On the docker rig, set `ScadaBridge:Cluster:SplitBrainResolverStrategy = "leader-auto-down"` in **both** `docker/central-node-{a,b}/appsettings.Central.json`. Stop and start the pair together, then run: - `DRILL_MODE=active bash docker/failover-drill.sh` — kills the oldest node. The survivor must take over within about 25s, and the victim must rejoin as standby. - `bash docker/failover-drill.sh` (standby mode) — the standby is downed and removed, with no `/health/active` blips. 2. Do the same on the **site-a** pair (`docker/site-a-node-{a,b}/appsettings.Site.json`). `failover-drill.sh` only targets central today, so a site variant is needed: `docker kill` the site's oldest node, check the survivor logs `[leader-auto-down] DOWNING`, the site comes back Up, and the victim rejoins. 3. Boot logs on every node show **no** `auto-down-unreachable-after … keep-majority will be used instead` deprecation warning. They should show the `[leader-auto-down] started on …` line, and on the drill victim's survivor `[leader-auto-down] … is unreachable; it will be downed if still unreachable after 00:00:15` followed by `DOWNING`. 4. Partition check (plan §6, **contract changed by #132**): the accepted dual-active still happens *while the partition lasts*, and both nodes report `/health/active` 200. After the heal, exactly one node restarts (the higher address), and the pair converges on one active node with no operator action. Rig: central +76 s and site-a +111 s after the heal. In-process: `ShutOutRestartScenarioTests.HealedPartition_BothShutOut_OnlyTheHigherAddressRestarts_AndRejoins`. `LeaderAutoDownScenarioTests.PartitionLongerThanWindow_EachSideDownsTheOther_BothStayUp` covers the during-partition half. ### Rollout after the gates pass Rig → `docker-env2` → QA pair vq03/vq04 → DEV vd03 and sites. Each pair stops and starts together; never do a rolling restart across a downing-provider change. The change is an appsettings flip only. Rollback is setting the value back to `auto-down`. Then switch the shipped default, update CLAUDE.md, `Component-ClusterInfrastructure.md` and `docker/README.md`, and close register row 70. Plan: `docs/plans/2026-11-09-autodowning-replacement-plan.md` §5–6. Parent: #106, #51.
dohertj2 added the type:deploy-gatepriority:p2 labels 2026-11-10 04:32:09 -05:00
dohertj2 added the blocked label 2026-11-10 08:43:35 -05:00
dohertj2 commented 2026-11-12 20:16:28 -05:00
Author
Owner
Copy Link
Copy Source

Rig gate: PASS (local docker rig, 2026-11-12, main @ 2d2e9f35)

Evidence: docs/plans/2026-11-12-leader-auto-down-rig-drill.md and raw logs in docs/plans/2026-11-12-leader-auto-down-rig-drill/. Commit 139a43fb. The run covered the local rig only. env2, QA and DEV were not touched.

How it was run

  • The strategy was set through a new opt-in overlay, docker/docker-compose.leader-auto-down.yml. It is an env override on central-a/b and site-a-a/b, so the shipped appsettings default stays auto-down.
  • Both pairs were switched together, and switched back together afterwards.
  • docker/failover-drill.sh gained DRILL_PAIR=central|site-a|… (the site variant), DRILL_ACTION=kill|pause and a strict rejoin check. The rejoin check requires the victim at ready 200 / active 503, the survivor at memberCount 2, unreachableCount 0, and then 10 samples each with exactly one active node.
  • An auto-down baseline was run on the same build first.
Gate Central Site-a auto-down baseline
1/2 Kill the active node PASS. Unreachable +13 s, DOWNING +28 s, active 29 s, Traefik 30 s, rejoin 2 s PASS. Unreachable +11 s, DOWNING +26 s, active 26 s, rejoin 11 s central 27 s / Traefik 30 s; site-a 27 s
1/2 Kill the standby PASS. DOWNING +29 s, 0 blips, rejoin 2 s PASS. DOWNING +26 s, 0 blips, rejoin 1 s README 2026-07-21: 27 s
3 Boot logs PASS. 0 deprecation lines; [leader-auto-down] started …, then is unreachable … after 00:00:15, then DOWNING PASS, same Warning present on all 4 nodes (positive control)
4 Partition past the window (iptables cut, 50 s / 45 s) PASS. Both sides downed each other at +27 s; dual-active 200/200 from +29 s, still there 60 s after heal; Overview showed Split brain; restarting central-b recovered in 12 s PASS. Dual-active from +27 s, still there after heal; restarting site-a-b recovered in 32 s n/a
Cold start of both pairs together (kill all 4, start all 4) PASS. One 2-member cluster in 6 s, 0 DOWNING PASS. 1 s, 0 DOWNING n/a
Site collects and S&F drains n/a PASS. The survivor opened the data connection and kept running scripts (17 → 85 runs). It delivered 3 more S&F items than it queued, which are the dead node's leftover items, and depth went to 0 n/a

Extra scenario: a freeze (docker pause of the active node, then unpause)

  • The survivor took over correctly: central 29 s (Traefik 32 s), site-a 27 s.
  • After unpause, the frozen node was quarantined. It downed the survivor in its own one-node view and stayed active as well, so both nodes reported 200 until it was restarted.
  • auto-down does exactly the same, so this is parity, not a regression. The cause: Akka.Cluster 1.5.62 never acts on ThisActorSystemQuarantinedEvent.
  • Filed as #132, with a recommended fix: self-down on quarantine, in leader-auto-down only. That fix would change gate 4's contract, because both sides of a healed partition would then restart, so it needs an owner decision.
  • LeaderAutoDowningProvider was not changed. No defect was found in it.

Rig state. Both pairs were recreated together back on the default auto-down. All 8 nodes are Up, with one active and one standby per pair (central-a, site-a-a, site-b-a and site-c-a active).

Next step (needs owner approval): roll out to docker-env2. Apply the same strategy to its central pair and one site pair, stopping and starting each pair together, and re-run the drill there. After that comes the QA pair vq03/vq04, then DEV vd03 and its sites. I suggest deciding #132 before the default flips. This issue stays open for the rollout.

## Rig gate: PASS (local docker rig, 2026-11-12, `main` @ `2d2e9f35`) Evidence: `docs/plans/2026-11-12-leader-auto-down-rig-drill.md` and raw logs in `docs/plans/2026-11-12-leader-auto-down-rig-drill/`. Commit `139a43fb`. The run covered the local rig only. env2, QA and DEV were not touched. **How it was run** - The strategy was set through a new opt-in overlay, `docker/docker-compose.leader-auto-down.yml`. It is an env override on `central-a/b` and `site-a-a/b`, so the shipped appsettings default stays `auto-down`. - Both pairs were switched together, and switched back together afterwards. - `docker/failover-drill.sh` gained `DRILL_PAIR=central|site-a|…` (the site variant), `DRILL_ACTION=kill|pause` and a strict rejoin check. The rejoin check requires the victim at ready 200 / active 503, the survivor at `memberCount 2, unreachableCount 0`, and then 10 samples each with exactly one active node. - An `auto-down` baseline was run on the same build first. | Gate | Central | Site-a | auto-down baseline | |---|---|---|---| | 1/2 Kill the active node | PASS. Unreachable +13 s, `DOWNING` +28 s, active 29 s, Traefik 30 s, rejoin 2 s | PASS. Unreachable +11 s, `DOWNING` +26 s, active 26 s, rejoin 11 s | central 27 s / Traefik 30 s; site-a 27 s | | 1/2 Kill the standby | PASS. `DOWNING` +29 s, **0** blips, rejoin 2 s | PASS. `DOWNING` +26 s, **0** blips, rejoin 1 s | README 2026-07-21: 27 s | | 3 Boot logs | PASS. 0 deprecation lines; `[leader-auto-down] started …`, then `is unreachable … after 00:00:15`, then `DOWNING` | PASS, same | Warning present on all 4 nodes (positive control) | | 4 Partition past the window (iptables cut, 50 s / 45 s) | PASS. Both sides downed each other at +27 s; dual-active 200/200 from +29 s, still there 60 s after heal; Overview showed **Split brain**; restarting `central-b` recovered in 12 s | PASS. Dual-active from +27 s, still there after heal; restarting `site-a-b` recovered in 32 s | n/a | | Cold start of both pairs together (kill all 4, start all 4) | PASS. One 2-member cluster in 6 s, 0 `DOWNING` | PASS. 1 s, 0 `DOWNING` | n/a | | Site collects and S&F drains | n/a | PASS. The survivor opened the data connection and kept running scripts (17 → 85 runs). It delivered 3 more S&F items than it queued, which are the dead node's leftover items, and depth went to 0 | n/a | **Extra scenario: a freeze (`docker pause` of the active node, then unpause)** - The survivor took over correctly: central 29 s (Traefik 32 s), site-a 27 s. - After unpause, the frozen node was quarantined. It downed the survivor in its own one-node view and stayed **active as well**, so both nodes reported 200 until it was restarted. - `auto-down` does **exactly the same**, so this is parity, not a regression. The cause: Akka.Cluster 1.5.62 never acts on `ThisActorSystemQuarantinedEvent`. - Filed as **#132**, with a recommended fix: self-down on quarantine, in `leader-auto-down` only. That fix would change gate 4's contract, because both sides of a healed partition would then restart, so it needs an owner decision. - `LeaderAutoDowningProvider` was **not** changed. No defect was found in it. **Rig state.** Both pairs were recreated together back on the default `auto-down`. All 8 nodes are Up, with one active and one standby per pair (central-a, site-a-a, site-b-a and site-c-a active). **Next step (needs owner approval):** roll out to `docker-env2`. Apply the same strategy to its central pair and one site pair, stopping and starting each pair together, and re-run the drill there. After that comes the QA pair vq03/vq04, then DEV vd03 and its sites. I suggest deciding #132 before the default flips. This issue stays open for the rollout.
dohertj2 changed title from Deploy gate: run the docker failover drill (central + site-a) on leader-auto-down, then roll it out rig → env2 → QA → DEV to Deploy gate: leader-auto-down rollout. Rig drill PASSED (2026-11-12); next env2 → QA → DEV 2026-11-12 20:16:38 -05:00
dohertj2 commented 2026-11-12 21:39:27 -05:00
Author
Owner
Copy Link
Copy Source

Freeze gate: now PASS (#132 fixed, main @ 952db02a + docs 482ac665)

The extra scenario that failed in the first rig run now passes on both pairs. A frozen node, and both sides of a healed partition, now converge to one active node with no operator action. Details and design: #132. Record: docs/plans/2026-11-12-leader-auto-down-rig-drill.md §7. Local rig only; env2, QA and DEV were not touched.

Gate Central Site-a
Freeze (DRILL_MODE=active DRILL_ACTION=pause) PASS. Takeover 27 s. At unpause the victim logged [leader-auto-down] shut out by … RESTARTS, exited 75, and rejoined as standby 34 s after unpause. 10/10 single-active PASS. Takeover 27 s, rejoined 6 s after unpause, 10/10
Split past the window, then heal PASS. Dual-active during the cut, as expected. After the heal only central-b (the higher address) restarted. One active node with 2 members at +76 s after the heal PASS. Only site-a-b restarted. One active node at +111 s after the heal
Kill regression PASS, unchanged: Down +26 s, Traefik 28 s, rejoin 2 s PASS, unchanged: Down +28 s, rejoin 1 s

Gate 4 contract change. The checklist above now reads: dual-active while the partition lasts, then self-healing after it heals (the higher address restarts). The issue body is updated.

The shipped default stays auto-down. The rig is back on it with all 8 nodes healthy: one active node per pair, memberCount 2. The fix applies only under leader-auto-down. The switch is ScadaBridge:Cluster:LeaderAutoDown:RestartWhenShutOut, default on for that strategy, with ShutOutGrace at 60 s. auto-down keeps the old freeze behaviour, which is one more reason to flip the default once the rollout below is done.

For the rollout. The Host now exits 75 after any unplanned ActorSystem termination. Before this it exited 0. Windows service recovery with failureflag 1 restarts only on a non-zero exit, so the self-down restart on the wonder boxes very likely never fired before. This has not been verified on a box. It is worth checking on QA when this build reaches it: stop the peer's network briefly, or check the SCM event log after a drill.

This issue stays open. Next step: docker-env2, which needs owner approval. After that: QA vq03/vq04, then DEV vd03 and its sites, then the default flip.

## Freeze gate: now PASS (#132 fixed, `main` @ `952db02a` + docs `482ac665`) The extra scenario that failed in the first rig run now passes on both pairs. A frozen node, and both sides of a healed partition, now converge to one active node with no operator action. Details and design: #132. Record: `docs/plans/2026-11-12-leader-auto-down-rig-drill.md` §7. Local rig only; env2, QA and DEV were not touched. | Gate | Central | Site-a | |---|---|---| | Freeze (`DRILL_MODE=active DRILL_ACTION=pause`) | **PASS.** Takeover 27 s. At unpause the victim logged `[leader-auto-down] shut out by … RESTARTS`, exited 75, and rejoined as standby 34 s after unpause. 10/10 single-active | **PASS.** Takeover 27 s, rejoined 6 s after unpause, 10/10 | | Split past the window, then heal | **PASS.** Dual-active during the cut, as expected. After the heal only central-b (the higher address) restarted. One active node with 2 members at +76 s after the heal | **PASS.** Only site-a-b restarted. One active node at +111 s after the heal | | Kill regression | **PASS, unchanged**: Down +26 s, Traefik 28 s, rejoin 2 s | **PASS, unchanged**: Down +28 s, rejoin 1 s | **Gate 4 contract change.** The checklist above now reads: dual-active while the partition lasts, then self-healing after it heals (the higher address restarts). The issue body is updated. **The shipped default stays `auto-down`.** The rig is back on it with all 8 nodes healthy: one active node per pair, `memberCount 2`. The fix applies only under `leader-auto-down`. The switch is `ScadaBridge:Cluster:LeaderAutoDown:RestartWhenShutOut`, default on for that strategy, with `ShutOutGrace` at 60 s. `auto-down` keeps the old freeze behaviour, which is one more reason to flip the default once the rollout below is done. **For the rollout.** The Host now exits **75** after any unplanned ActorSystem termination. Before this it exited 0. Windows service recovery with `failureflag 1` restarts only on a non-zero exit, so the self-down restart on the wonder boxes very likely never fired before. This has not been verified on a box. It is worth checking on QA when this build reaches it: stop the peer's network briefly, or check the SCM event log after a drill. This issue stays open. **Next step: `docker-env2`, which needs owner approval.** After that: QA vq03/vq04, then DEV vd03 and its sites, then the default flip.
dohertj2 commented 2026-11-12 23:29:10 -05:00
Author
Owner
Copy Link
Copy Source

docker-env2 gate: PASS (2026-11-12, owner-approved step)

Env2 only. The primary rig, the shared infra, QA and DEV were not touched. The shipped default stays auto-down.

Build. Image scadabridge:env2-116, built from origin/main a2e9d718 plus the env2 tooling commit (e8a743de after rebase, no code change). scadabridge:latest, which the primary rig runs, was not rebuilt or retagged. Record: docs/plans/2026-11-12-leader-auto-down-env2-drill.md, raw logs in docs/plans/2026-11-12-leader-auto-down-env2-drill/. Commits e8a743de, cafdc0ce, 90d20030.

How it was run

  • New committed options, all default-preserving:
    • SCADABRIDGE_IMAGE in docker/build.sh and in docker-env2/docker-compose.yml.
    • docker-env2/docker-compose.leader-auto-down.yml, mirroring the rig overlay.
    • DRILL_ENV=env2 in docker/failover-drill.sh.
  • Bind mounts came from the main checkout (--project-directory), not from the worktree.
  • auto-down baseline first, then both pairs switched together in one recreate.
Gate env2 central env2 site-x
Baseline auto-down kill active Down +27 s, Traefik 29 s, rejoin 2 s; deprecation warning on both nodes (positive control) +28 s, rejoin 12 s; warning on both
Kill active PASS. DOWNING +28 s, active 28 s, Traefik 32 s, rejoin 2 s, 10/10 PASS. +27 s, rejoin 1 s, 10/10
Kill standby PASS. +29 s, 0 blips, rejoin 2 s PASS. +28 s, 0 blips, rejoin 1 s
Boot logs PASS. 0 deprecation lines; [leader-auto-down] started, is unreachable … after 00:00:15, then DOWNING PASS
Freeze (DRILL_ACTION=pause) PASS. Takeover 27 s. Victim logged shut out … RESTARTS, exited 75, rejoined as standby 3 s after unpause, 10/10 PASS. Takeover 27 s, exit 75, rejoined 8 s after unpause, 10/10
Split past the window, then heal (50 s / 45 s) PASS. Dual-active +26 s while cut. After the heal, central-a STAY and central-b RESTART; only central-b restarted; one active node, 2 members at +76 s PASS. Dual-active +26 s. Only site-x-b restarted; one active node at +80 s
Kill regression PASS, unchanged. +28 s, Traefik 30 s, rejoin 2 s, 0 shut-out lines PASS, unchanged. +26 s, rejoin 1 s

Limits

  • env2 is not seeded: 0 sites and 0 instances in ScadaBridgeConfig2. This run proves cluster mechanics only, and the rig's data-path and S&F check was not repeated. seed-sites.sh binds to the windev MxGateway, so it was not run.
  • Transport smoke skipped. The README points only to the RETIRED 2026-05-24 checklist, and the primary rig was in use by another agent (about 300 /management calls at 04:15–04:17 UTC).

Final state: env2 stays on leader-auto-down as the soak. All 4 nodes run scadabridge:env2-116 with the overlay. Each pair has one active node (central-b, site-x-b) and memberCount 2, unreachableCount 0, and Traefik :9100 answers 200. A plain docker-env2/deploy.sh would end the soak (:latest on auto-down). The keep-soak command and the rollback are in docker-env2/README.md.

Still open. Next step: QA pair vq03/vq04, which needs owner approval, so blocked stays. On QA, also verify that the Windows service supervisor restarts the Host on exit 75. Then DEV vd03, then the default flip.

## docker-env2 gate: PASS (2026-11-12, owner-approved step) Env2 only. The primary rig, the shared infra, QA and DEV were not touched. The shipped default stays `auto-down`. **Build.** Image `scadabridge:env2-116`, built from `origin/main` `a2e9d718` plus the env2 tooling commit (`e8a743de` after rebase, no code change). `scadabridge:latest`, which the primary rig runs, was not rebuilt or retagged. Record: `docs/plans/2026-11-12-leader-auto-down-env2-drill.md`, raw logs in `docs/plans/2026-11-12-leader-auto-down-env2-drill/`. Commits `e8a743de`, `cafdc0ce`, `90d20030`. **How it was run** - New committed options, all default-preserving: - `SCADABRIDGE_IMAGE` in `docker/build.sh` and in `docker-env2/docker-compose.yml`. - `docker-env2/docker-compose.leader-auto-down.yml`, mirroring the rig overlay. - `DRILL_ENV=env2` in `docker/failover-drill.sh`. - Bind mounts came from the main checkout (`--project-directory`), not from the worktree. - `auto-down` baseline first, then both pairs switched together in one recreate. | Gate | env2 central | env2 site-x | |---|---|---| | Baseline `auto-down` kill active | Down +27 s, Traefik 29 s, rejoin 2 s; deprecation warning on both nodes (positive control) | +28 s, rejoin 12 s; warning on both | | Kill active | **PASS.** `DOWNING` +28 s, active 28 s, Traefik 32 s, rejoin 2 s, 10/10 | **PASS.** +27 s, rejoin 1 s, 10/10 | | Kill standby | **PASS.** +29 s, **0** blips, rejoin 2 s | **PASS.** +28 s, **0** blips, rejoin 1 s | | Boot logs | **PASS.** 0 deprecation lines; `[leader-auto-down] started`, `is unreachable … after 00:00:15`, then `DOWNING` | **PASS** | | Freeze (`DRILL_ACTION=pause`) | **PASS.** Takeover 27 s. Victim logged `shut out … RESTARTS`, exited 75, rejoined as standby 3 s after unpause, 10/10 | **PASS.** Takeover 27 s, exit 75, rejoined 8 s after unpause, 10/10 | | Split past the window, then heal (50 s / 45 s) | **PASS.** Dual-active +26 s while cut. After the heal, central-a `STAY` and central-b `RESTART`; **only central-b restarted**; one active node, 2 members at +76 s | **PASS.** Dual-active +26 s. Only site-x-b restarted; one active node at +80 s | | Kill regression | **PASS, unchanged.** +28 s, Traefik 30 s, rejoin 2 s, 0 shut-out lines | **PASS, unchanged.** +26 s, rejoin 1 s | **Limits** - env2 is **not seeded**: 0 sites and 0 instances in `ScadaBridgeConfig2`. This run proves cluster mechanics only, and the rig's data-path and S&F check was not repeated. `seed-sites.sh` binds to the windev MxGateway, so it was not run. - **Transport smoke skipped.** The README points only to the RETIRED 2026-05-24 checklist, and the primary rig was in use by another agent (about 300 `/management` calls at 04:15–04:17 UTC). **Final state: env2 stays on `leader-auto-down` as the soak.** All 4 nodes run `scadabridge:env2-116` with the overlay. Each pair has one active node (central-b, site-x-b) and `memberCount 2, unreachableCount 0`, and Traefik `:9100` answers 200. A plain `docker-env2/deploy.sh` would end the soak (`:latest` on `auto-down`). The keep-soak command and the rollback are in `docker-env2/README.md`. **Still open.** **Next step: QA pair vq03/vq04, which needs owner approval**, so `blocked` stays. On QA, also verify that the Windows service supervisor restarts the Host on exit 75. Then DEV vd03, then the default flip.
dohertj2 commented 2026-11-12 23:56:49 -05:00
Author
Owner
Copy Link
Copy Source

env2 seeded data-path soak: done (2026-11-13 UTC 04:32–04:55). Record: docs/plans/2026-11-12-leader-auto-down-env2-drill.md §5 (raw output seeded-* in its directory). Commits 8b6d3268, 6c7e150a, 644f564c on main.

Seeded: docker-env2/seed-sites.sh set up Site-X, the MxGateway connection to windev (used only as a client) and MxAlarmDemo-1. The new docker-env2/seed-data-path.sh adds:

  • env-soak-1: opc-plc SpikeData/RandomSignedInt32, plus a 2 s script that sends a tick-numbered ExternalSystem.CachedCall to an env2-owned HTTP sink, scadabridge-env2-sfsink. Its 503 mode forces store-and-forward to buffer.
  • env-galaxy-1: TestMachine_003.TestChangingInt, read only.

Values are Good and updating through :9100, and both connections are Connected in central's site health report.

Finding 1 — an unreplicated site pair cannot fail over its data. The first seeded active kill passed the cluster gate (DOWNING +28 s, rejoin 1 s). But the survivor loaded no instances, had no connections and ran no scripts, and the victim's 10 buffered messages stayed on the victim. This is by design: site-x had no LocalDb peer, like the rig's site-b and site-c. 6c7e150a makes env2 site-x replicated, mirroring the rig's site-a. After that, the 10 stranded messages were delivered exactly once.

Finding 2 (minor) — boot race when replication is first turned on over diverged stores. The node that boots active loads its stale store about 1 s before the snapshot arrives, and its startup reconcile then sees 0 gaps. The fix was deploy-artifacts plus a redeploy. Steady-state failovers are not affected.

Failovers with data (site-x replicated, sink failing across each hit):

Run Cluster Data gap S&F Duplicate ticks
site-x active kill PASS: DOWNING +29 s, rejoin 1 s, 10/10 30 s 20 delivered (≈11 inherited from the victim + 9 own), depth 0 0
site-x active freeze PASS: takeover 27 s; victim shut out, exit 75, rejoin 38 s 26 s 42 delivered (16 inherited + 26 own), depth 0 4
central active kill PASS: 27 s, Traefik 29 s, rejoin 3 s 0 (43/43 ticks) — 0

No buffered message was lost or delivered twice. The 4 duplicate ticks in the freeze run come from the frozen victim's 6 s zombie window after unpause: its interval timer fired 4 catch-up runs (the in-flight cap, 1 shed) and one run overlapped the survivor's. That is the accepted dual-active trade, and #132's shut-out bounds it. The standby's S&F depth gauge reads stale values (known, 02/C3).

Final env2 state: all 4 nodes on scadabridge:env2-116 + leader-auto-down, each memberCount 2. Active nodes: central-a and site-x-a. Site-x is replicated and the soak load keeps running. The primary rig, scadabridge:latest and shared infra were not touched.

Still open and blocked: the next step is QA vq03/vq04, which needs owner approval.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi

**env2 seeded data-path soak: done (2026-11-13 UTC 04:32–04:55).** Record: `docs/plans/2026-11-12-leader-auto-down-env2-drill.md` §5 (raw output `seeded-*` in its directory). Commits `8b6d3268`, `6c7e150a`, `644f564c` on `main`. **Seeded:** `docker-env2/seed-sites.sh` set up Site-X, the MxGateway connection to windev (used only as a client) and `MxAlarmDemo-1`. The new `docker-env2/seed-data-path.sh` adds: - `env-soak-1`: opc-plc `SpikeData`/`RandomSignedInt32`, plus a 2 s script that sends a tick-numbered `ExternalSystem.CachedCall` to an env2-owned HTTP sink, `scadabridge-env2-sfsink`. Its 503 mode forces store-and-forward to buffer. - `env-galaxy-1`: `TestMachine_003.TestChangingInt`, read only. Values are Good and updating through `:9100`, and both connections are Connected in central's site health report. **Finding 1 — an unreplicated site pair cannot fail over its data.** The first seeded active kill passed the cluster gate (DOWNING +28 s, rejoin 1 s). But the survivor loaded no instances, had no connections and ran no scripts, and the victim's 10 buffered messages stayed on the victim. This is by design: site-x had no LocalDb peer, like the rig's site-b and site-c. **`6c7e150a` makes env2 site-x replicated**, mirroring the rig's site-a. After that, the 10 stranded messages were delivered exactly once. **Finding 2 (minor) — boot race when replication is first turned on over diverged stores.** The node that boots active loads its stale store about 1 s before the snapshot arrives, and its startup reconcile then sees 0 gaps. The fix was `deploy-artifacts` plus a redeploy. Steady-state failovers are not affected. **Failovers with data (site-x replicated, sink failing across each hit):** | Run | Cluster | Data gap | S&F | Duplicate ticks | |---|---|---|---|---| | site-x active kill | PASS: DOWNING +29 s, rejoin 1 s, 10/10 | 30 s | 20 delivered (≈11 inherited from the victim + 9 own), depth 0 | 0 | | site-x active freeze | PASS: takeover 27 s; victim shut out, exit 75, rejoin 38 s | 26 s | 42 delivered (16 inherited + 26 own), depth 0 | 4 | | central active kill | PASS: 27 s, Traefik 29 s, rejoin 3 s | 0 (43/43 ticks) | — | 0 | No buffered message was lost or delivered twice. The 4 duplicate ticks in the freeze run come from the frozen victim's 6 s zombie window after unpause: its interval timer fired 4 catch-up runs (the in-flight cap, 1 shed) and one run overlapped the survivor's. That is the accepted dual-active trade, and #132's shut-out bounds it. The standby's S&F depth gauge reads stale values (known, 02/C3). **Final env2 state:** all 4 nodes on `scadabridge:env2-116` + `leader-auto-down`, each `memberCount 2`. Active nodes: central-a and site-x-a. Site-x is replicated and the soak load keeps running. The primary rig, `scadabridge:latest` and shared infra were not touched. Still open and `blocked`: the next step is QA vq03/vq04, which needs owner approval. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
dohertj2 commented 2026-11-13 02:51:02 -05:00
Author
Owner
Copy Link
Copy Source

QA rollout is bundled into the #107 QA window pack (16209831): docs/deployment/deploy-prep/2026-11-13-qa-vq03-vq04/README.md, sections 4, 5.3, 5.4, gate G14 and section 8. It is not run yet and waits for the owner's go.

  • It is an appsettings flip in the Central and Site files on both boxes. Each pair is started together in the same window as the migrations.
  • The SCM recovery for exit 75 is checked during prep and set if it is missing (failureflag 1).
  • The site-pair kill drill restarts every instance, Z28061 included, so it needs explicit owner OK.
**QA rollout is bundled into the #107 QA window pack** (16209831): `docs/deployment/deploy-prep/2026-11-13-qa-vq03-vq04/README.md`, sections 4, 5.3, 5.4, gate G14 and section 8. It is not run yet and waits for the owner's go. - It is an appsettings flip in the Central and Site files on both boxes. Each pair is started together in the same window as the migrations. - The SCM recovery for exit 75 is checked during prep and set if it is missing (`failureflag 1`). - The site-pair kill drill restarts every instance, Z28061 included, so it needs explicit owner OK.
dohertj2 commented 2026-11-13 07:12:06 -05:00
Author
Owner
Copy Link
Copy Source

QA pack refreshed to main @ 8e32f353: 8ffa6394 (docs/deployment/deploy-prep/2026-11-13-qa-vq03-vq04/, not executed). Per the owner's 2026-11-13 go, the window bundles the leader-auto-down flip in both appsettings files on both boxes plus the exit-code-75 service recovery (sc.exe failure … restart/10000/restart/30000/restart/60000, failureflag 1) on all four services, with no #116 drills (neither central nor site pair). G14 is struck; record "QA on leader-auto-down, not drilled" after the window. Boot-log check G6 ([leader-auto-down] started on all 4) and the new G15 (recovery setting read back) remain.

QA pack refreshed to `main` @ `8e32f353`: `8ffa6394` (`docs/deployment/deploy-prep/2026-11-13-qa-vq03-vq04/`, not executed). Per the owner's 2026-11-13 go, the window bundles the `leader-auto-down` flip in both appsettings files on both boxes plus the exit-code-75 service recovery (`sc.exe failure … restart/10000/restart/30000/restart/60000`, `failureflag 1`) on all four services, with **no #116 drills** (neither central nor site pair). G14 is struck; record "QA on leader-auto-down, not drilled" after the window. Boot-log check G6 (`[leader-auto-down] started` on all 4) and the new G15 (recovery setting read back) remain.
dohertj2 commented 2026-11-13 08:14:51 -05:00
Author
Owner
Copy Link
Copy Source

Deployed to QA vq03/vq04 (2026-11-13). Config is applied, the drills are still pending. Build 8e32f353, record 8ba5d5d7 (docs/deployment/deployment-records/2026-11-13-wonder-app-vq03-vq04-8e32f353.md).

  • ScadaBridge:Cluster:SplitBrainResolverStrategy is now leader-auto-down in appsettings.Central.json and appsettings.Site.json on both boxes. It is the only key that changed: the key-level digest diff shows nothing else, and LeaderAutoDown stays at its defaults.
  • Both pairs were stopped and started together in the #107 window (no rolling across the provider change).
  • G6: [leader-auto-down] started on akka.tcp://scadabridge@10.220.158.{2,3}:{8081,8082} on all four nodes, each with shut-out restart is ON. vq03 gained leadership on both pairs.
  • G5: /health/active sampled 10× gives exactly one 200 per pair every time (vq03).
  • G15: all four services restart 5 s / 5 s / 10 s with reset 1 day and FAILURE_ACTIONS_ON_NONCRASH_FAILURES: TRUE. The flag was already TRUE and was re-asserted; the existing actions were kept, as on DEV.
  • G14 not run (owner's go): QA is on leader-auto-down, not drilled. The central and site-pair kill drills still need an owner slot, because the site drill re-hosts Z28061.
  • Rollback of #116 only: put the two files back from _appsettings-keep-20261113T080106 on both boxes, then restart each pair together.
**Deployed to QA vq03/vq04 (2026-11-13). Config is applied, the drills are still pending.** Build `8e32f353`, record `8ba5d5d7` (`docs/deployment/deployment-records/2026-11-13-wonder-app-vq03-vq04-8e32f353.md`). - `ScadaBridge:Cluster:SplitBrainResolverStrategy` is now `leader-auto-down` in `appsettings.Central.json` and `appsettings.Site.json` on both boxes. It is the only key that changed: the key-level digest diff shows nothing else, and `LeaderAutoDown` stays at its defaults. - Both pairs were stopped and started together in the #107 window (no rolling across the provider change). - **G6:** `[leader-auto-down] started on akka.tcp://scadabridge@10.220.158.{2,3}:{8081,8082}` on **all four** nodes, each with `shut-out restart is ON`. vq03 `gained leadership` on both pairs. - **G5:** `/health/active` sampled 10× gives exactly one 200 per pair every time (vq03). - **G15:** all four services restart 5 s / 5 s / 10 s with reset 1 day and `FAILURE_ACTIONS_ON_NONCRASH_FAILURES: TRUE`. The flag was already TRUE and was re-asserted; the existing actions were kept, as on DEV. - **G14 not run (owner's go):** QA is on leader-auto-down, **not drilled**. The central and site-pair kill drills still need an owner slot, because the site drill re-hosts Z28061. - Rollback of #116 only: put the two files back from `_appsettings-keep-20261113T080106` on both boxes, then restart each pair together.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
blocked
Blocked by dependency or external hold
 cross-repo
Needs coordinated change in another family repo (wire contract etc.)
 migrated
Migrated from a markdown backlog (2026-11 standardization)
 priority:p0
Must-have / blocking
 priority:p1
High priority
 priority:p2
Normal priority
 priority:p3
Low priority / nice to have
 type:bug
Defect in shipped behaviour
 type:debt
Technical debt / hygiene
 type:deploy-gate
Rollout/deploy/migration step not yet done on an environment
 type:docs
Documentation work
 type:feature
Feature implementation
 type:followup
Follow-up deferred from a plan, PR or gate
 type:infra
Infrastructure and scaffolding
 type:review-finding
Unfixed finding from an architecture/code review
 type:test
Test work
No Label blocked priority:p2 type:deploy-gate
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
dohertj2 (dohertj2)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dohertj2/ScadaBridge#116
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?