diff --git a/docs/requirements/Component-StoreAndForward.md b/docs/requirements/Component-StoreAndForward.md index 85a60c8d..87f6cab6 100644 --- a/docs/requirements/Component-StoreAndForward.md +++ b/docs/requirements/Component-StoreAndForward.md @@ -75,9 +75,9 @@ There is **no maximum buffer size**. Messages accumulate in the buffer until del - Buffered messages are persisted to a **local SQLite database** on each site node. - The active node persists locally and forwards each buffer operation (add, remove, park) to the standby node **asynchronously** via Akka.NET remoting. The active node does not wait for standby acknowledgment — this avoids adding latency to every script that buffers a message. -- The standby node applies the same operations to its own local SQLite database. -- On failover, the new active node has a near-complete copy of the buffer. In rare cases, the most recent operations may not have been replicated (e.g., a message added or removed just before failover). This can result in a few **duplicate deliveries** (message delivered but remove not replicated) or a few **missed retries** (message added but not replicated). Both are acceptable trade-offs for the latency benefit. -- On failover, the new active node resumes delivery from its local copy. +- The standby node applies the same operations to its own local SQLite database but is **passive**: it never runs the delivery sweep. The retry sweep is **gated to the active node** (the oldest Up member / singleton host, re-evaluated every sweep tick), so only one node delivers at a time. The standby applies replicated operations purely to keep its copy warm for a future failover. +- On failover, the new active node has a near-complete copy of the buffer. In rare cases, the most recent operations may not have been replicated (e.g., a message added or removed just before failover). This can result in a few **duplicate deliveries** (message delivered but its `Remove` not yet replicated) or a few **missed retries** (message added but not replicated). Duplicate deliveries are therefore confined to the **failover window** — an in-flight delivery whose `Remove` had not yet replicated — and never occur in steady-state operation (the standby's gate keeps it from delivering the same rows). Both are acceptable trade-offs for the latency benefit. +- On failover, the new active node's gate flips to active within one sweep interval and it resumes delivery from its local copy. ### Operation Tracking Table (lives in Site Runtime, not here) diff --git a/src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs b/src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs index 8a5a61d7..0d5daed6 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs @@ -859,6 +859,22 @@ akka {{ cancellationToken.ThrowIfCancellationRequested(); await storeAndForwardService.StartAsync(); + // Standby must be passive: only the active site node runs the S&F + // delivery sweep. Without this gate BOTH nodes deliver the same + // replicated Pending rows — systematic duplicate external calls and + // DB writes (arch review 02, Stability #2). Re-evaluated per sweep + // tick so failover resumes delivery within one RetryTimerInterval. + // IClusterNodeProvider.SelfIsPrimary is the canonical "this node is the + // oldest Up member (singleton host)" check from the cluster-infrastructure + // fix plan — the shared helper this seam was designed to accept. In a + // non-clustered test host the provider is unregistered, so the gate stays + // unset and the sweep is ungated (legacy behaviour, preserved). + var clusterNodeProvider = _serviceProvider.GetService(); + if (clusterNodeProvider != null) + { + storeAndForwardService.SetDeliveryGate(() => clusterNodeProvider.SelfIsPrimary); + } + // Register the store-and-forward delivery handlers so buffered // ExternalSystem calls, cached DB writes and notifications are actually // delivered by the retry sweep. Without this, every buffered message is