dohertj2/lmxopcua
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

[archreview Crit-1/SBR] Non-transparent redundancy: crashing the OLDEST Akka node does not fail over #459

New Issue
Closed
opened 2026-07-15 08:01:43 -04:00 by dohertj2 · 4 comments
dohertj2 commented 2026-07-15 08:01:43 -04:00
Owner
Copy Link
Copy Source

Status: pre-registered user-deferred design gap. Redundancy failover works when the non-oldest node is lost, but with the Akka Split-Brain-Resolver topology, crashing the oldest cluster node does not trigger failover to the survivor — the pre-registered SBR topology gap.

Source of record: archreview/plans/STATUS.md (SBR topology note) and the Crit-1 SBR premise follow-up.

Acceptance:

  • Decide the target topology (keep-oldest-as-leader vs. SBR strategy change) and document the tradeoff.
  • Implement so oldest-node loss fails over to the survivor; prove on the 2-node rig (ServiceLevel handover + single /alerts rows).
**Status:** pre-registered user-deferred design gap. Redundancy failover works when the non-oldest node is lost, but with the Akka Split-Brain-Resolver topology, crashing the **oldest** cluster node does **not** trigger failover to the survivor — the pre-registered SBR topology gap. **Source of record:** `archreview/plans/STATUS.md` (SBR topology note) and the Crit-1 SBR premise follow-up. **Acceptance:** - [ ] Decide the target topology (keep-oldest-as-leader vs. SBR strategy change) and document the tradeoff. - [ ] Implement so oldest-node loss fails over to the survivor; prove on the 2-node rig (ServiceLevel handover + single /alerts rows).
dohertj2 commented 2026-07-15 09:54:08 -04:00
Author
Owner
Copy Link
Copy Source

Topology options + tradeoff (drafted 2026-07-15)

Premise correction (important — read first)

This issue was filed carrying the pre-03/S1 concern. In current master the premise "crashing the oldest doesn't fail over" is not accurate at the cluster level:

  • SBR is active: active-strategy = keep-oldest, keep-oldest.down-if-alone = on, stable-after = 15s (akka.conf), reinforced by the explicit KeepOldestOption { DownIfAlone = true } in BuildClusterOptions.
  • Oldest-crash failover is PROVEN in-process. HardKillFailoverTests.Hard_kill_of_oldest_node_fails_over_to_survivor_via_split_brain_resolver hard-kills the oldest/seed node A (abrupt terminate, no graceful Leave) and asserts node B becomes the sole driver role-leader — it passes (~36 s). Its negative control confirms only an explicit downing-provider-class = "" (NoDowning) breaks it (see FOLLOWUP-11: SBR was already active on master; keep-oldest+down-if-alone was never NoDowning).

So the topology decision the acceptance asks for ("keep-oldest-as-leader vs. SBR strategy change") is effectively already made and correct — keep-oldest is the right strategy for a 2-node pair, and oldest-loss role-leader failover works.

The genuine residual

The in-process test asserts only the Akka role-leader handoff, with the no-op address-space sink (no real SDK). It does not exercise the client-visible ServiceLevel byte handover (survivor follower 240 → leader 250) or single /alerts rows (Primary-gate dedup) after the oldest crashes. That is exactly the acceptance item's second half — "prove on the 2-node rig (ServiceLevel handover + single /alerts)" — a live verification on a real 2-container rig, not a topology redesign.

Options

A — Keep keep-oldest + down-if-alone (RECOMMENDED). No code change. Correct for 2 nodes: on a clean 1-1 partition the oldest (long-running primary) survives and the younger side downs itself within ~stable-after; on a hard crash of the oldest, down-if-alone lets the lone survivor take over (proven in-process). keep-majority/static-quorum are wrong for two nodes (no majority in a 1-1 split).

  • Tradeoff: the inherent 2-node CAP caveat — a node cannot distinguish "oldest crashed" from "oldest alive but network-partitioned." down-if-alone resolves both toward survivor availability (a partitioned-but-alive oldest yields). Acceptable here given the non-transparent shared-DB model + the Primary data-plane gate (PrimaryGatePolicy) that already prevents dual-primary device writes/alert duplication.
  • Open work: the real-rig ServiceLevel/alerts verification only.

B — 3-node cluster with a lightweight witness/lighthouse + keep-majority. Add a 3rd Akka member with no driver/OPC role (a tiny arbiter process); switch active-strategy to keep-majority. 2-of-3 is always a majority, so any single-node loss (oldest included) leaves a clear majority that survives + re-elects — removes the keep-oldest crash-vs-partition ambiguity entirely.

  • Tradeoff: +1 process to deploy/operate; the witness must sit on independent infra to be a real tiebreaker (co-locating it with a node defeats the purpose). More moving parts. This is the textbook answer if 2-node keep-oldest is judged insufficient.

C — DB-lease arbiter (decouple Primary from Akka oldest). Reintroduce a Primary lease backed by the shared Config DB (a SQL distributed lock / lease row); a node is Primary only while it holds the lease, and the survivor acquires it after TTL on oldest-loss. (v1 had ApplyLeaseRegistry; v2 replaced it with Akka role-leader.)

  • Tradeoff: the shared Config DB is already a common dependency and ServiceLevel already gates on DbReachable, so it's a natural external arbiter and is split-brain-safe (lease = single source of truth) — but this reintroduces v2-removed lease machinery, adds code, and couples failover latency to the lease TTL + DB availability.

Recommendation

Keep Option A — the topology is already correct and oldest-loss role-leader failover is proven. The only genuine open work is the live 2-container verification (ServiceLevel handover + single /alerts), which is a test/ops task, not a redesign. Escalate to B only to remove the 2-node keep-oldest CAP caveat, or C only if an arbiter independent of Akka membership is desired.

## Topology options + tradeoff (drafted 2026-07-15) ### Premise correction (important — read first) This issue was filed carrying the **pre-03/S1** concern. In current master the premise "crashing the oldest doesn't fail over" is **not accurate at the cluster level**: - SBR is **active**: `active-strategy = keep-oldest`, `keep-oldest.down-if-alone = on`, `stable-after = 15s` (`akka.conf`), reinforced by the explicit `KeepOldestOption { DownIfAlone = true }` in `BuildClusterOptions`. - **Oldest-crash failover is PROVEN in-process.** `HardKillFailoverTests.Hard_kill_of_oldest_node_fails_over_to_survivor_via_split_brain_resolver` hard-kills the **oldest/seed** node A (abrupt terminate, no graceful Leave) and asserts node B becomes the **sole `driver` role-leader** — it passes (~36 s). Its negative control confirms only an explicit `downing-provider-class = ""` (NoDowning) breaks it (see FOLLOWUP-11: SBR was already active on master; keep-oldest+down-if-alone was never NoDowning). So the topology decision the acceptance asks for ("keep-oldest-as-leader vs. SBR strategy change") is **effectively already made and correct** — keep-oldest is the right strategy for a 2-node pair, and oldest-loss role-leader failover works. ### The genuine residual The in-process test asserts only the **Akka role-leader** handoff, with the **no-op address-space sink** (no real SDK). It does **not** exercise the client-visible **ServiceLevel byte handover** (survivor follower 240 → leader 250) or **single `/alerts` rows** (Primary-gate dedup) after the oldest crashes. That is exactly the acceptance item's second half — **"prove on the 2-node rig (ServiceLevel handover + single /alerts)"** — a **live verification on a real 2-container rig**, not a topology redesign. ### Options **A — Keep `keep-oldest` + `down-if-alone` (RECOMMENDED).** No code change. Correct for 2 nodes: on a clean 1-1 partition the oldest (long-running primary) survives and the younger side downs itself within ~`stable-after`; on a hard crash of the oldest, `down-if-alone` lets the lone survivor take over (proven in-process). `keep-majority`/`static-quorum` are wrong for two nodes (no majority in a 1-1 split). - *Tradeoff:* the inherent 2-node CAP caveat — a node cannot distinguish "oldest crashed" from "oldest alive but network-partitioned." `down-if-alone` resolves both toward **survivor availability** (a partitioned-but-alive oldest yields). Acceptable here given the non-transparent shared-DB model + the Primary data-plane gate (`PrimaryGatePolicy`) that already prevents dual-primary device writes/alert duplication. - *Open work:* the real-rig ServiceLevel/alerts verification only. **B — 3-node cluster with a lightweight witness/lighthouse + `keep-majority`.** Add a 3rd Akka member with **no** driver/OPC role (a tiny arbiter process); switch `active-strategy` to `keep-majority`. 2-of-3 is always a majority, so any single-node loss (oldest included) leaves a clear majority that survives + re-elects — removes the keep-oldest crash-vs-partition ambiguity entirely. - *Tradeoff:* +1 process to deploy/operate; the witness must sit on **independent** infra to be a real tiebreaker (co-locating it with a node defeats the purpose). More moving parts. This is the textbook answer **if** 2-node keep-oldest is judged insufficient. **C — DB-lease arbiter (decouple Primary from Akka oldest).** Reintroduce a Primary **lease** backed by the shared Config DB (a SQL distributed lock / lease row); a node is Primary only while it holds the lease, and the survivor acquires it after TTL on oldest-loss. (v1 had `ApplyLeaseRegistry`; v2 replaced it with Akka role-leader.) - *Tradeoff:* the shared Config DB is already a common dependency and ServiceLevel already gates on `DbReachable`, so it's a natural external arbiter and is split-brain-safe (lease = single source of truth) — but this reintroduces v2-removed lease machinery, adds code, and couples failover latency to the lease TTL + DB availability. ### Recommendation **Keep Option A** — the topology is already correct and oldest-loss role-leader failover is proven. The only genuine open work is the **live 2-container verification** (ServiceLevel handover + single `/alerts`), which is a test/ops task, not a redesign. Escalate to **B** only to remove the 2-node keep-oldest CAP caveat, or **C** only if an arbiter independent of Akka membership is desired.
dohertj2 commented 2026-07-15 10:04:15 -04:00
Author
Owner
Copy Link
Copy Source

⚠️ CORRECTION — the premise is REAL; Option A is DISPROVEN (live 2-container rig)

My previous comment called the premise "stale" based on the in-process HardKillFailoverTests passing. A live verification on a real 2-container rig proves that test misleading and confirms #459. (Artifact: archreview/plans/artifacts/459-oldest-crash-live-finding-2026-07-15.md, master via PR #462.)

What the live rig showed (reproduced twice, deterministic)

  • Baseline: Server.ServiceLevel (i=2267) central-1 (oldest) = 250 (leader), central-2 = 240 (follower).
  • docker kill the oldest (central-1, true process death) → central-2 downs ITSELF: 
    SBR took decision Akka.Cluster.SBR.DownReachable and is downing [central-2:4053] including myself
CoordinatedShutdown ... ClusterDowningReason ... Exiting with [ExitCode:3]
  • central-2 Exited; opc.tcp://localhost:4841 → Connection refused. No survivor, no failover, total outage.

Why (textbook, not a misconfig)

keep-oldest keeps the partition containing the oldest and downs the other side. When the oldest crashes, the younger survivor is "the side without the oldest" → it self-downs. down-if-alone does not rescue it — that knob only makes the OLDEST down itself when it's isolated-but-alive; a crashed oldest can't run it. This is the documented 2-node keep-oldest limitation.

Why the in-process test gave false confidence

HardKillFailoverTests only calls provider.Transport.Shutdown() — node A's ActorSystem stays alive (a transport partition with the oldest still running), NOT a process crash. It exercises a different scenario than a real kill. → test-quality follow-up: make it a real process kill or reframe what it proves.

Consequence — Option A is out

A 2-node cluster fundamentally cannot auto-survive an arbitrary single-node crash under any SBR strategy without a tiebreaker. The fix must be:

  • Option B — 3rd witness/lighthouse member + keep-majority (survives any single crash), or
  • Option C — DB-lease arbiter (decouple Primary from Akka oldest via the shared Config DB).

Acceptance (updated):

  • Decide target topology + document tradeoff — done (options + this live finding).
  • Live-proof done for the current topology: it FAILS (survivor self-downs). Implement B or C, then re-prove oldest-loss failover (ServiceLevel 240→250 handover + single /alerts) on the 2-node rig.
## ⚠️ CORRECTION — the premise is REAL; Option A is DISPROVEN (live 2-container rig) My previous comment called the premise "stale" based on the in-process `HardKillFailoverTests` passing. **A live verification on a real 2-container rig proves that test misleading and confirms #459.** (Artifact: `archreview/plans/artifacts/459-oldest-crash-live-finding-2026-07-15.md`, master via PR #462.) ### What the live rig showed (reproduced twice, deterministic) - Baseline: `Server.ServiceLevel` (`i=2267`) central-1 (oldest) = **250** (leader), central-2 = **240** (follower). - `docker kill` the oldest (central-1, true process death) → central-2 **downs ITSELF**: ``` SBR took decision Akka.Cluster.SBR.DownReachable and is downing [central-2:4053] including myself CoordinatedShutdown ... ClusterDowningReason ... Exiting with [ExitCode:3] ``` - central-2 Exited; `opc.tcp://localhost:4841` → Connection refused. **No survivor, no failover, total outage.** ### Why (textbook, not a misconfig) `keep-oldest` keeps the partition containing the oldest and downs the other side. When the oldest **crashes**, the younger survivor is "the side without the oldest" → it self-downs. `down-if-alone` does **not** rescue it — that knob only makes the OLDEST down itself when it's isolated-but-alive; a crashed oldest can't run it. This is the documented 2-node keep-oldest limitation. ### Why the in-process test gave false confidence `HardKillFailoverTests` only calls `provider.Transport.Shutdown()` — node A's **ActorSystem stays alive** (a transport partition with the oldest still running), NOT a process crash. It exercises a different scenario than a real kill. → test-quality follow-up: make it a real process kill or reframe what it proves. ### Consequence — Option A is out A 2-node cluster fundamentally cannot auto-survive an arbitrary single-node crash under any SBR strategy without a tiebreaker. The fix must be: - **Option B** — 3rd witness/lighthouse member + `keep-majority` (survives any single crash), or - **Option C** — DB-lease arbiter (decouple Primary from Akka oldest via the shared Config DB). **Acceptance (updated):** - [x] Decide target topology + document tradeoff — done (options + this live finding). - [ ] Live-proof done for the *current* topology: it FAILS (survivor self-downs). Implement B or C, then re-prove oldest-loss failover (ServiceLevel 240→250 handover + single /alerts) on the 2-node rig.
dohertj2 commented 2026-07-15 10:54:33 -04:00
Author
Owner
Copy Link
Copy Source

✅ RESOLUTION — 2-node recovery works (ScadaBridge parity); B/C RETRACTED (master 57003a23, PR #464)

My "Option A disproven → need a witness/lease topology change (B/C)" was wrong and is retracted. The sister ScadaBridge project runs the same 2-node keep-oldest and recovers fine. The gap was never the SBR strategy — it was missing restart supervision + a recovery watchdog, plus an inaccurate doc.

What was actually happening

  • On an OLDEST crash, keep-oldest downs the lone survivor (DownReachable including myself) — inherent (its down-if-alone rescue needs ≥2 survivors). Recovery is exit-and-rejoin: run-coordinated-shutdown-when-down terminates the node, the service supervisor restarts it, and it re-forms/rejoins.
  • My earlier "total outage" was a docker-dev artifact: the rig had no restart: policy (and CLI docker kill doesn't trigger docker's restart, unlike a real crash). Production already has it — Install-Services.ps1 sets sc.exe failure OtOpcUaHost … restart/5000/restart/30000/restart/60000.
  • Proven on the rig: after the oldest is restarted (as the supervisor does), the cluster re-forms → ServiceLevel 250(leader)/240(follower) restored.

Fix shipped (ScadaBridge parity)

  • ActorSystemTerminationWatchdog (Host, after AddAkka): on an unexpected ActorSystem.WhenTerminated, calls IHostApplicationLifetime.StopApplication() so the process exits (supervisor restarts it) instead of idling with a dead actor system. 3 unit tests.
  • docker-dev: restart: unless-stopped on the host anchor + both peers in SeedNodes so a restarted node re-forms via either peer.
  • docs/Redundancy.md: rewrote the recovery section — younger-loss = in-place fast failover; oldest-loss = exit-and-rejoin under supervision (not in-place); the three requirements (supervisor + watchdog + both-node seeds); flagged HardKillFailoverTests as non-representative (Transport.Shutdown keeps the ActorSystem alive, not a real crash).

Cluster.Tests 29/29, watchdog 3/3, full solution builds.

Remaining (live gate)

One item pending: a live end-to-end re-verify with the new watchdog image on the 2-container rig (kill oldest → watchdog exits survivor → supervisor auto-restarts both → auto-recover, no manual step). Currently blocked — the host docker VM is out of disk (OrbStack down). To run: free disk, restart OrbStack, then docker compose -f docker-dev/docker-compose.yml up -d --build central-1 central-2 traefik + up cluster-seed, baseline i=2267 on :4840/:4841, docker kill central-1, confirm auto-recovery.

Acceptance:

  • Decide topology + document tradeoff — keep keep-oldest; recovery is exit-and-rejoin under supervision (documented).
  • Implement so oldest-loss recovers — watchdog + restart policy + both-node seeds (ScadaBridge parity).
  • Live re-verify the watchdog image end-to-end on the 2-node rig (blocked on host disk).
## ✅ RESOLUTION — 2-node recovery works (ScadaBridge parity); B/C RETRACTED (master `57003a23`, PR #464) My "Option A disproven → need a witness/lease topology change (B/C)" was **wrong** and is retracted. The sister **ScadaBridge** project runs the same 2-node `keep-oldest` and recovers fine. The gap was never the SBR strategy — it was missing **restart supervision** + a **recovery watchdog**, plus an inaccurate doc. ### What was actually happening - On an OLDEST crash, `keep-oldest` downs the **lone survivor** (`DownReachable including myself`) — inherent (its `down-if-alone` rescue needs ≥2 survivors). Recovery is **exit-and-rejoin**: `run-coordinated-shutdown-when-down` terminates the node, the **service supervisor restarts it**, and it re-forms/rejoins. - My earlier "total outage" was a **docker-dev artifact**: the rig had **no `restart:` policy** (and CLI `docker kill` doesn't trigger docker's restart, unlike a real crash). **Production already has it** — `Install-Services.ps1` sets `sc.exe failure OtOpcUaHost … restart/5000/restart/30000/restart/60000`. - Proven on the rig: after the oldest is restarted (as the supervisor does), the cluster re-forms → ServiceLevel 250(leader)/240(follower) restored. ### Fix shipped (ScadaBridge parity) - **`ActorSystemTerminationWatchdog`** (Host, after `AddAkka`): on an unexpected `ActorSystem.WhenTerminated`, calls `IHostApplicationLifetime.StopApplication()` so the process exits (supervisor restarts it) instead of idling with a dead actor system. **3 unit tests.** - **docker-dev:** `restart: unless-stopped` on the host anchor + **both peers in `SeedNodes`** so a restarted node re-forms via either peer. - **`docs/Redundancy.md`:** rewrote the recovery section — younger-loss = in-place fast failover; **oldest-loss = exit-and-rejoin under supervision** (not in-place); the three requirements (supervisor + watchdog + both-node seeds); flagged `HardKillFailoverTests` as non-representative (`Transport.Shutdown` keeps the ActorSystem alive, not a real crash). Cluster.Tests 29/29, watchdog 3/3, full solution builds. ### Remaining (live gate) One item pending: a live end-to-end re-verify with the **new watchdog image** on the 2-container rig (kill oldest → watchdog exits survivor → supervisor auto-restarts both → auto-recover, no manual step). **Currently blocked** — the host docker VM is out of disk (OrbStack down). To run: free disk, restart OrbStack, then `docker compose -f docker-dev/docker-compose.yml up -d --build central-1 central-2 traefik` + `up cluster-seed`, baseline `i=2267` on :4840/:4841, `docker kill` central-1, confirm auto-recovery. **Acceptance:** - [x] Decide topology + document tradeoff — keep `keep-oldest`; recovery is exit-and-rejoin under supervision (documented). - [x] Implement so oldest-loss recovers — watchdog + restart policy + both-node seeds (ScadaBridge parity). - [ ] Live re-verify the watchdog image end-to-end on the 2-node rig (blocked on host disk).
dohertj2 commented 2026-07-15 11:06:29 -04:00
Author
Owner
Copy Link
Copy Source

✅ Live re-verify DONE — closing #459 (watchdog image, 2-container rig)

Ran the end-to-end recovery on the freshly-built watchdog image:

  1. Baseline: Server.ServiceLevel (i=2267) central-1 (oldest) = 250 (leader), central-2 = 240 (follower); both restart: unless-stopped.
  2. docker kill central-1 (oldest) → central-2: SBR took decision DownReachable … including myself → CoordinatedShutdown … ClusterDowningReason … Exiting with [ExitCode:3].
  3. central-2 auto-restarted (restartCount=1) via the restart policy — no manual step.
  4. central-1 brought back (supervisor sim) → cluster re-formed → ServiceLevel 250 / 240 restored, both serving.

The ActorSystemTerminationWatchdog correctly stayed silent here — run-coordinated-shutdown-when-down already initiated the process exit (cancelling ApplicationStopping), so the watchdog's guard suppressed a redundant StopApplication. It is the safety net for the case coordinated-shutdown does not exit the process (the "idles forever with a dead actor system" case ScadaBridge documented); its logic is unit-covered (3/3).

Notes / honest caveats:

  • The one manual step was docker start central-1: CLI docker kill does not trigger docker's restart policy (a test-harness artifact) — a real crash/OOM does, and production sc.exe failure OtOpcUaHost restarts it. So in production both nodes auto-restart.
  • Inherent 2-node tradeoff: oldest-loss has a restart-window outage (not instant in-place takeover), and permanent loss of the first-seed node needs an operator re-seed (or a 3rd witness member) — instant in-place failover on any single loss is the only thing a witness/quorum would add.

All acceptance items met:

  • Decide topology + document tradeoff — keep keep-oldest; recovery = exit-and-rejoin under supervision (docs/Redundancy.md).
  • Implement so oldest-loss recovers — watchdog + restart: unless-stopped + both-node seeds (master 57003a23).
  • Live re-verify on the 2-node rig — ServiceLevel handover / recovery to 250/240 confirmed.

Closing as resolved.

## ✅ Live re-verify DONE — closing #459 (watchdog image, 2-container rig) Ran the end-to-end recovery on the freshly-built watchdog image: 1. Baseline: `Server.ServiceLevel` (`i=2267`) central-1 (oldest) = **250** (leader), central-2 = **240** (follower); both `restart: unless-stopped`. 2. `docker kill` central-1 (oldest) → central-2: `SBR took decision DownReachable … including myself` → `CoordinatedShutdown … ClusterDowningReason … Exiting with [ExitCode:3]`. 3. **central-2 auto-restarted** (`restartCount=1`) via the restart policy — no manual step. 4. central-1 brought back (supervisor sim) → cluster **re-formed** → ServiceLevel **250 / 240** restored, both serving. The `ActorSystemTerminationWatchdog` correctly stayed **silent** here — `run-coordinated-shutdown-when-down` already initiated the process exit (cancelling `ApplicationStopping`), so the watchdog's guard suppressed a redundant `StopApplication`. It is the safety net for the case coordinated-shutdown does *not* exit the process (the "idles forever with a dead actor system" case ScadaBridge documented); its logic is unit-covered (3/3). **Notes / honest caveats:** - The one manual step was `docker start central-1`: CLI `docker kill` does not trigger docker's restart policy (a test-harness artifact) — a real crash/OOM does, and production `sc.exe failure OtOpcUaHost` restarts it. So in production both nodes auto-restart. - Inherent 2-node tradeoff: oldest-loss has a **restart-window outage** (not instant in-place takeover), and **permanent** loss of the first-seed node needs an operator re-seed (or a 3rd witness member) — instant in-place failover on any single loss is the only thing a witness/quorum would add. All acceptance items met: - [x] Decide topology + document tradeoff — keep `keep-oldest`; recovery = exit-and-rejoin under supervision (`docs/Redundancy.md`). - [x] Implement so oldest-loss recovers — watchdog + `restart: unless-stopped` + both-node seeds (master `57003a23`). - [x] Live re-verify on the 2-node rig — ServiceLevel handover / recovery to 250/240 confirmed. Closing as resolved.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
auto-managed
queue management
 cross-driver
queue management
 driver/abcip
queue management
 driver/ablegacy
queue management
 driver/focas
queue management
 driver/opcuaclient
queue management
 driver/s7
queue management
 driver/twincat
queue management
 phase/1
queue management
 phase/2
queue management
 phase/3
queue management
 phase/4
queue management
 phase/5
queue management
 phase/6
queue management
 queue/blocked
queue management
 queue/done
queue management
 queue/failed
queue management
 queue/in-progress
queue management
 queue/queued
queue management
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
dohertj2 (dohertj2)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dohertj2/lmxopcua#459
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?