Deploy to a site pair with one node down: first attempt returns OUTCOME_UNKNOWN (connection failed), retry succeeds #136

New Issue

2026-11-13T00:13:10-05:00

dohertj2 commented

2026-11-13 00:13:10 -05:00

Seen twice while verifying #135 on the primary docker rig (2026-11-13, d7fa31f0; details in the #135 closing comment).

What happened. With site-a-a stopped and site-a-b the sole Up node of site-a, the first instance deploy to site-a returned OUTCOME_UNKNOWN: connection failed…. An immediate retry succeeded against site-a-b. Both times, it was the first deploy after the node went down.

Why it matters. A deploy to a site with one node down is exactly the failover case. Central reports an unknown outcome even though the site is healthy, so an operator or CLI script may treat it as a failure or re-send it.

To do.

Find how central addresses the site for a deploy. It may try the downed node's gRPC address first, or a cached channel to it, without failing over to the other address within the same call.
Reproduce it with a test and/or on the rig.
Fix it so the first deploy fails over to the live node within the call, or returns a definite outcome. Make sure a deploy that did reach the downed node before it died is still reported as unknown and is never silently duplicated.

Seen twice while verifying #135 on the primary docker rig (2026-11-13, `d7fa31f0`; details in the #135 closing comment). **What happened.** With `site-a-a` stopped and `site-a-b` the sole Up node of site-a, the first instance deploy to site-a returned `OUTCOME_UNKNOWN: connection failed…`. An immediate retry succeeded against `site-a-b`. Both times, it was the first deploy after the node went down. **Why it matters.** A deploy to a site with one node down is exactly the failover case. Central reports an unknown outcome even though the site is healthy, so an operator or CLI script may treat it as a failure or re-send it. **To do.** - Find how central addresses the site for a deploy. It may try the downed node's gRPC address first, or a cached channel to it, without failing over to the other address within the same call. - Reproduce it with a test and/or on the rig. - Fix it so the first deploy fails over to the live node within the call, or returns a definite outcome. Make sure a deploy that did reach the downed node before it died is still reported as unknown and is never silently duplicated.

dohertj2 added the type:bug priority:p2 labels 2026-11-13 00:13:10 -05:00

dohertj2 referenced this issue from a commit

2026-11-13 00:26:51 -05:00

fix(communication): fail a site command over within the call when the stopped node never received it (#136)

dohertj2 commented

2026-11-13 00:27:17 -05:00

Fixed in fd80aeab (main).

Root cause. Central keeps one sticky gRPC channel per site node (SitePairChannelProvider), with NodeA preferred. The failback loop only probes while central is off A, so nothing notices that A has stopped until a real call fails. The first deploy after the stop therefore always goes to A. When it failed with Unavailable, SitePairChannelProvider.ExecuteAsync (previously SitePairChannelProvider.cs:326-328) flipped stickiness to B. Because a deploy is not retry-safe and the error had no server-sent scadabridge-dispatch: not-started trailer, it then threw OUTCOME_UNKNOWN instead of trying B. The next call went to B and succeeded, which is the "retry works" symptom.

The rule was too blunt. For a stopped node, the client failed while opening the connection, so the request provably never left central. Grpc.Net.Client reports this as Unavailable / "Error connecting to subchannel." with a SocketException cause (refused, or the stopped container's hostname no longer resolving). Its balancer fails the call at pick time, before any HTTP request exists.

Fix.

New GrpcRetrySafety.WasNeverSent (GrpcRetrySafety.cs:48) returns true only for two shapes:
- Grpc.Net.Client's subchannel connect failure with a SocketException cause.
- An HttpRequestException classed NameResolutionError/ConnectionError whose socket error can only happen before a connection exists (ConnectionRefused, HostNotFound, TryAgain, NoData).
ProvablyNotApplied = WasNotDispatched || WasNeverSent. ExecuteAsync (SitePairChannelProvider.cs:332, :341) now uses it, so a mutation fails over to the peer within the same call.
Everything that may have been delivered stays OUTCOME_UNKNOWN and is not replayed: a reset or premature end on an established connection, GOAWAY, a keepalive timeout, or an Unavailable sent by the server. Deadlines are still never replayed. The request reaches exactly one node, so the site-side dedup (deployment id + query-before-redeploy) is never relied on for this path.
The site→central reconcile path (GrpcCentralTransport) uses the same ProvablyNotApplied rule.

Tests.

GrpcSiteTransportFailoverTests (real Grpc.Net.Client + SocketsHttpHandler, NodeB a TestServer):
- ConnectionRefused_OnNodeA_MutationFailsOverToNodeB_WithinTheCall
- UnresolvableNodeA_MutationFailsOverToNodeB_WithinTheCall (the docker shape)
- ConnectionLostAfterDelivery_StaysOutcomeUnknown_AndIsNotReplayed: A's stub runs the command, then the transport fails with a premature end. Result is Aborted/OUTCOME_UNKNOWN, and B gets 0 calls.
GrpcRetrySafetyNeverSentTests: 9 classifier cases, positive and negative.
Mutation checks:
- Putting back WasNotDispatched in the provider fails both failover tests.
- Making WasNeverSent true for any Unavailable fails ConnectionLostAfterDelivery… and the existing AmbiguousUnavailable_DoesNotReplayLifecycleOnPeer.
Communication.Tests 843/843, Host.Tests 724/724. Release solution build: 0 warnings, 0 errors.

Rig (primary, site-a pair, CLI :9000, instance deploy --id 1492 rig-probe-1).

Before (old image): docker stop scadabridge-site-a-a, then deploy → Deployment failed: Status(StatusCode="Aborted", Detail="OUTCOME_UNKNOWN: connection failed…"). Immediate retry → status 2 (success).
After (scadabridge:latest from fd80aeab; centrals and the site-a pair recreated together): a deploy with both nodes up succeeds. docker stop scadabridge-site-a-a at 05:25:03, then the first deploy → status 2, deployment 48af9dee…. central-a logs 05:25:06 WRN … node A unavailable; failing over to node B. site-a-b logs a single Deploying instance rig-probe-1, deploymentId=48af9dee…, so it was applied once.
Site-a pair recreated together afterwards. All primary-rig nodes are Up and a further deploy succeeds. env2 was not touched.

Side note, not changed here: that warning renders as Site central node A …. The node's enriched SiteId property appears to win over the message-template argument, so the log line names the wrong site.

Fixed in `fd80aeab` (main). **Root cause.** Central keeps one sticky gRPC channel per site node (`SitePairChannelProvider`), with NodeA preferred. The failback loop only probes while central is *off* A, so nothing notices that A has stopped until a real call fails. The first deploy after the stop therefore always goes to A. When it failed with `Unavailable`, `SitePairChannelProvider.ExecuteAsync` (previously `SitePairChannelProvider.cs:326-328`) flipped stickiness to B. Because a deploy is not retry-safe and the error had no server-sent `scadabridge-dispatch: not-started` trailer, it then threw `OUTCOME_UNKNOWN` instead of trying B. The next call went to B and succeeded, which is the "retry works" symptom. The rule was too blunt. For a stopped node, the client failed while *opening* the connection, so the request provably never left central. Grpc.Net.Client reports this as `Unavailable` / `"Error connecting to subchannel."` with a `SocketException` cause (refused, or the stopped container's hostname no longer resolving). Its balancer fails the call at pick time, before any HTTP request exists. **Fix.** - New `GrpcRetrySafety.WasNeverSent` (`GrpcRetrySafety.cs:48`) returns true only for two shapes: - Grpc.Net.Client's subchannel connect failure with a `SocketException` cause. - An `HttpRequestException` classed `NameResolutionError`/`ConnectionError` whose socket error can only happen before a connection exists (`ConnectionRefused`, `HostNotFound`, `TryAgain`, `NoData`). - `ProvablyNotApplied` = `WasNotDispatched || WasNeverSent`. `ExecuteAsync` (`SitePairChannelProvider.cs:332`, `:341`) now uses it, so a mutation fails over to the peer **within the same call**. - Everything that may have been delivered stays `OUTCOME_UNKNOWN` and is not replayed: a reset or premature end on an established connection, GOAWAY, a keepalive timeout, or an `Unavailable` sent by the server. Deadlines are still never replayed. The request reaches exactly one node, so the site-side dedup (deployment id + query-before-redeploy) is never relied on for this path. - The site→central reconcile path (`GrpcCentralTransport`) uses the same `ProvablyNotApplied` rule. **Tests.** - `GrpcSiteTransportFailoverTests` (real Grpc.Net.Client + `SocketsHttpHandler`, NodeB a TestServer): - `ConnectionRefused_OnNodeA_MutationFailsOverToNodeB_WithinTheCall` - `UnresolvableNodeA_MutationFailsOverToNodeB_WithinTheCall` (the docker shape) - `ConnectionLostAfterDelivery_StaysOutcomeUnknown_AndIsNotReplayed`: A's stub runs the command, then the transport fails with a premature end. Result is `Aborted`/`OUTCOME_UNKNOWN`, and B gets 0 calls. - `GrpcRetrySafetyNeverSentTests`: 9 classifier cases, positive and negative. - Mutation checks: - Putting back `WasNotDispatched` in the provider fails both failover tests. - Making `WasNeverSent` true for any `Unavailable` fails `ConnectionLostAfterDelivery…` and the existing `AmbiguousUnavailable_DoesNotReplayLifecycleOnPeer`. - Communication.Tests 843/843, Host.Tests 724/724. Release solution build: 0 warnings, 0 errors. **Rig (primary, site-a pair, CLI :9000, `instance deploy --id 1492` rig-probe-1).** - *Before (old image):* `docker stop scadabridge-site-a-a`, then deploy → `Deployment failed: Status(StatusCode="Aborted", Detail="OUTCOME_UNKNOWN: connection failed…")`. Immediate retry → status 2 (success). - *After (`scadabridge:latest` from `fd80aeab`; centrals and the site-a pair recreated together):* a deploy with both nodes up succeeds. `docker stop scadabridge-site-a-a` at 05:25:03, then the first deploy → **status 2**, deployment `48af9dee…`. central-a logs `05:25:06 WRN … node A unavailable; failing over to node B`. site-a-b logs a single `Deploying instance rig-probe-1, deploymentId=48af9dee…`, so it was applied once. - Site-a pair recreated together afterwards. All primary-rig nodes are Up and a further deploy succeeds. env2 was not touched. Side note, not changed here: that warning renders as `Site central node A …`. The node's enriched `SiteId` property appears to win over the message-template argument, so the log line names the wrong site.

dohertj2 closed this issue

2026-11-13 00:27:17 -05:00

dohertj2 referenced this issue from a commit

2026-11-13 00:28:41 -05:00

fix(communication): site pair log lines name the target site, not the logging node (#136)

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dohertj2/ScadaBridge#136