fix(audit-log): reconciliation pulls fail over to site NodeB when NodeA is unreachable

SiteEntry gains an additive FallbackGrpcEndpoint; SiteEnumerator now picks NodeA
else NodeB as primary (a NodeB-only site is no longer skipped) and carries a
distinct NodeB as the fallback. Both GrpcPull{AuditEvents,SiteCalls} clients dial
the primary first and, on a transport fault (Unavailable/DeadlineExceeded/
Cancelled/HttpRequestException/SocketException/OperationCanceled), fail over to
NodeB once before collapsing to empty — a mapping/unexpected fault does NOT fail
over. Keeps the reconciliation loss-recovery net available during a NodeA outage.
This commit is contained in:
Joseph Doherty
2026-07-09 08:57:06 -04:00
parent 0385942c3f
commit bfd2cc7e6f
8 changed files with 382 additions and 100 deletions
+12
View File
@@ -279,6 +279,18 @@ non-draining (e.g., telemetry actor wedged), central issues a
are flipped to `ForwardState = 'Reconciled'` site-side. Same self-healing
pattern as Site Call Audit's reconciliation of `SiteCalls`.
**Endpoint resolution & NodeB failover.** Each pull dials the site's `NodeA`
gRPC address first; if `NodeA` is blank the site's `NodeB` address becomes the
primary (a NodeB-only site is no longer skipped — only a site with *both*
addresses blank is). When a distinct `NodeB` address also exists it is carried
as a fallback: a transport fault against the primary (site offline, deadline
exceeded, cancelled, connection error) fails over to `NodeB` ONCE before
collapsing to an empty batch. So during a NodeA outage the loss-recovery net
stays available. The failover is scoped to transport faults — a
mapping/unexpected fault collapses to empty without a second dial (the other
node would hit the same fault). The same resolution + failover applies to Site
Call Audit's `PullSiteCalls`.
> **Cursor keyset (tracked follow-up).** The `PullAuditEvents` cursor is still a
> single `sinceUtc` timestamp. Site Call Audit's pull now uses a composite
> `(UpdatedAtUtc, TrackedOperationId)` keyset to avoid a single-timestamp pin