fix(host): wire S&F active-node delivery gate on site nodes (SelfIsPrimary); align S&F doc standby-passive wording

This commit is contained in:
Joseph Doherty
2026-07-08 17:48:31 -04:00
parent 1a53b8082a
commit 41ed57421d
2 changed files with 19 additions and 3 deletions
@@ -75,9 +75,9 @@ There is **no maximum buffer size**. Messages accumulate in the buffer until del
- Buffered messages are persisted to a **local SQLite database** on each site node.
- The active node persists locally and forwards each buffer operation (add, remove, park) to the standby node **asynchronously** via Akka.NET remoting. The active node does not wait for standby acknowledgment — this avoids adding latency to every script that buffers a message.
- The standby node applies the same operations to its own local SQLite database.
- On failover, the new active node has a near-complete copy of the buffer. In rare cases, the most recent operations may not have been replicated (e.g., a message added or removed just before failover). This can result in a few **duplicate deliveries** (message delivered but remove not replicated) or a few **missed retries** (message added but not replicated). Both are acceptable trade-offs for the latency benefit.
- On failover, the new active node resumes delivery from its local copy.
- The standby node applies the same operations to its own local SQLite database but is **passive**: it never runs the delivery sweep. The retry sweep is **gated to the active node** (the oldest Up member / singleton host, re-evaluated every sweep tick), so only one node delivers at a time. The standby applies replicated operations purely to keep its copy warm for a future failover.
- On failover, the new active node has a near-complete copy of the buffer. In rare cases, the most recent operations may not have been replicated (e.g., a message added or removed just before failover). This can result in a few **duplicate deliveries** (message delivered but its `Remove` not yet replicated) or a few **missed retries** (message added but not replicated). Duplicate deliveries are therefore confined to the **failover window** — an in-flight delivery whose `Remove` had not yet replicated — and never occur in steady-state operation (the standby's gate keeps it from delivering the same rows). Both are acceptable trade-offs for the latency benefit.
- On failover, the new active node's gate flips to active within one sweep interval and it resumes delivery from its local copy.
### Operation Tracking Table (lives in Site Runtime, not here)
@@ -859,6 +859,22 @@ akka {{
cancellationToken.ThrowIfCancellationRequested();
await storeAndForwardService.StartAsync();
// Standby must be passive: only the active site node runs the S&F
// delivery sweep. Without this gate BOTH nodes deliver the same
// replicated Pending rows — systematic duplicate external calls and
// DB writes (arch review 02, Stability #2). Re-evaluated per sweep
// tick so failover resumes delivery within one RetryTimerInterval.
// IClusterNodeProvider.SelfIsPrimary is the canonical "this node is the
// oldest Up member (singleton host)" check from the cluster-infrastructure
// fix plan — the shared helper this seam was designed to accept. In a
// non-clustered test host the provider is unregistered, so the gate stays
// unset and the sweep is ungated (legacy behaviour, preserved).
var clusterNodeProvider = _serviceProvider.GetService<ZB.MOM.WW.ScadaBridge.HealthMonitoring.IClusterNodeProvider>();
if (clusterNodeProvider != null)
{
storeAndForwardService.SetDeliveryGate(() => clusterNodeProvider.SelfIsPrimary);
}
// Register the store-and-forward delivery handlers so buffered
// ExternalSystem calls, cached DB writes and notifications are actually
// delivered by the retry sweep. Without this, every buffered message is