dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

feat(localdb)!: replicate site config + sf_messages via CDC, delete the bespoke replicators

Browse Source 
Tasks 14, 15 and 16, landed as ONE commit.

PLAN DEFECT: these three tasks cannot compile separately. SiteReplicationActor
takes a ReplicationService and calls ReplaceAllAsync (Task 14 deletes both);
DeploymentManagerActor Tells message types declared in ReplicationMessages.cs
(Task 15 deletes it); AkkaHostedService constructs the actor (Task 16). Any
ordering leaves a broken intermediate. Combining them also strengthens the
invariant Task 14 already stated for itself — the two mechanisms never both
run, and never neither.

Registered 8 tables in SiteLocalDbSetup.OnReady: sf_messages plus the 7 site
config tables. notification_lists and smtp_configurations are deliberately NOT
registered — permanently empty by design, so registering them would open a
standing replication channel whose only historical payload was plaintext SMTP
passwords. Migrate stays the LAST call in OnReady, after all registrations, so
migrated rows enter the oplog through live capture triggers.

Deleted: SiteReplicationActor, ReplicationMessages.cs, ReplicationService,
StoreAndForwardStorage.ReplaceAllAsync, and 6 test files. ReplaceAllAsync is
not merely unused but unsafe to keep: a mass DELETE on a now-replicated table
would be captured and shipped to the peer.

Kept ActiveNodeEvaluator (delivery gate + heartbeat still need it) with its doc
corrected, and activeNodeCheck in AkkaHostedService (SiteCommunicationActor).

The positional-argument hazard the plan flagged was real: removing
DeploymentManagerActor's optional IActorRef? replicationActor shifted 6
trailing optionals, and 4 test call sites bound the wrong arguments with no
compile error at some positions. Converted them to named arguments where
possible — Props.Create builds an expression tree, which rejects out-of-position
named args, so the rest are padded positionally with a comment saying why.

The Task 7 'not yet registered' test was INVERTED rather than deleted, and is
exact in both directions: too few means a table silently stops replicating, too
many means the SMTP tables leak. Added a separate security-named test for those
two, and a composite-PK test (LWW keys on the full PK, so a truncated key set
would collapse distinct rows). The convergence suites now get their
registrations from the real OnReady — their temporary harness registration is
deleted, so they prove the cutover rather than agreeing with themselves.

Verified: build 0 warnings; SiteRuntime 512, StoreAndForward 130, Host 330,
AuditLog 355, ExternalSystemGateway 142, HealthMonitoring 97, LocalDb
integration 16 — all pass, 0 failures.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
This commit is contained in:
Joseph Doherty
2026-07-20 04:20:05 -04:00
parent df0c6031ba
commit 037798b367
26 changed files with 160 additions and 2513 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/ZB.MOM.WW.ScadaBridge.StoreAndForward/ReplicationService.cs
-194
View File
@@ -1,194 +0,0 @@
using Microsoft.Extensions.Logging;
using ZB.MOM.WW.ScadaBridge.Commons.Observability;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
namespace ZB.MOM.WW.ScadaBridge.StoreAndForward;
/// <summary>
/// Async replication of buffer operations to standby node.
///
/// - Forwards add/remove/park operations to standby via a replication handler.
/// - No ack wait (fire-and-forget per design).
/// - Standby applies operations to its own SQLite.
/// - On failover, standby resumes delivery from its replicated state.
/// </summary>
public class ReplicationService
{
private readonly StoreAndForwardOptions _options;
private readonly ILogger<ReplicationService> _logger;
private Func<ReplicationOperation, Task>? _replicationHandler;
/// <summary>Initializes a new instance of <see cref="ReplicationService"/>.</summary>
/// <param name="options">Store-and-forward configuration options.</param>
/// <param name="logger">Logger instance.</param>
public ReplicationService(
StoreAndForwardOptions options,
ILogger<ReplicationService> logger)
{
_options = options;
_logger = logger;
}
/// <summary>
/// Sets the handler for forwarding replication operations to the standby node.
/// Typically wraps Akka Tell to the standby's replication actor.
/// </summary>
/// <param name="handler">The async delegate that forwards each replication operation to the standby.</param>
public void SetReplicationHandler(Func<ReplicationOperation, Task> handler)
{
_replicationHandler = handler;
}
/// <summary>
/// Replicates an enqueue operation to standby (fire-and-forget).
/// </summary>
/// <param name="message">The message that was enqueued on the active node.</param>
public void ReplicateEnqueue(StoreAndForwardMessage message)
{
if (!_options.ReplicationEnabled || _replicationHandler == null) return;
FireAndForget(new ReplicationOperation(
ReplicationOperationType.Add,
message.Id,
message));
}
/// <summary>
/// Replicates a remove operation to standby (fire-and-forget).
/// </summary>
/// <param name="messageId">The identifier of the message to remove from the standby buffer.</param>
public void ReplicateRemove(string messageId)
{
if (!_options.ReplicationEnabled || _replicationHandler == null) return;
FireAndForget(new ReplicationOperation(
ReplicationOperationType.Remove,
messageId,
null));
}
/// <summary>
/// Replicates a park operation to standby (fire-and-forget).
/// </summary>
/// <param name="message">The message that was parked on the active node.</param>
public void ReplicatePark(StoreAndForwardMessage message)
{
if (!_options.ReplicationEnabled || _replicationHandler == null) return;
FireAndForget(new ReplicationOperation(
ReplicationOperationType.Park,
message.Id,
message));
}
/// <summary>
/// Replicates an operator-initiated requeue (a parked
/// message moved back to the pending queue) to standby (fire-and-forget). The
/// carried message reflects the active node's post-requeue state (Pending,
/// retry_count = 0) so the standby's copy can be brought into sync.
/// </summary>
/// <param name="message">The message in its post-requeue (Pending, retry_count=0) state.</param>
public void ReplicateRequeue(StoreAndForwardMessage message)
{
if (!_options.ReplicationEnabled || _replicationHandler == null) return;
FireAndForget(new ReplicationOperation(
ReplicationOperationType.Requeue,
message.Id,
message));
}
/// <summary>
/// Applies a replicated operation received from the active node.
/// Used by the standby node to keep its SQLite in sync.
///
/// Add/Park/Requeue are applied as <b>upserts</b> (<see cref="StoreAndForwardStorage.UpsertMessageAsync"/>),
/// not blind INSERT/UPDATE: the full message rides in every one of those operations,
/// so a Park/Requeue whose original Add was lost (fire-and-forget replication is
/// best-effort) self-heals by materialising the row, and a duplicate Add (e.g.
/// re-issued by an anti-entropy resync) applies newest-wins instead of throwing a
/// primary-key violation. Remove is a plain delete.
/// </summary>
/// <param name="operation">The replication operation to apply.</param>
/// <param name="storage">The standby node's store-and-forward storage to update.</param>
/// <returns>A task representing the asynchronous apply operation.</returns>
public async Task ApplyReplicatedOperationAsync(
ReplicationOperation operation,
StoreAndForwardStorage storage)
{
switch (operation.OperationType)
{
case ReplicationOperationType.Add when operation.Message != null:
await storage.UpsertMessageAsync(operation.Message);
break;
case ReplicationOperationType.Remove:
await storage.RemoveMessageAsync(operation.MessageId);
break;
case ReplicationOperationType.Park when operation.Message != null:
operation.Message.Status = StoreAndForwardMessageStatus.Parked;
await storage.UpsertMessageAsync(operation.Message);
break;
case ReplicationOperationType.Requeue when operation.Message != null:
operation.Message.Status = StoreAndForwardMessageStatus.Pending;
operation.Message.RetryCount = 0;
await storage.UpsertMessageAsync(operation.Message);
break;
}
}
private void FireAndForget(ReplicationOperation operation)
{
// Invoked inline, NOT via Task.Run: the handler is a non-blocking Akka
// Tell, and thread-pool hand-off destroyed Add/Remove ordering for the
// same message id (arch review 02). Inline invocation preserves issue
// order; Akka's per-sender/receiver guarantee preserves it on the wire.
try
{
var task = _replicationHandler!.Invoke(operation);
if (!task.IsCompletedSuccessfully)
{
task.ContinueWith(t =>
{
ScadaBridgeTelemetry.RecordReplicationFailure();
_logger.LogWarning(t.Exception,
"Replication of {OpType} for message {MessageId} failed (best-effort); standby buffer may be diverging",
operation.OperationType, operation.MessageId);
},
TaskContinuationOptions.OnlyOnFaulted);
}
}
catch (Exception ex)
{
ScadaBridgeTelemetry.RecordReplicationFailure();
_logger.LogWarning(ex,
"Replication of {OpType} for message {MessageId} failed (best-effort); standby buffer may be diverging",
operation.OperationType, operation.MessageId);
}
}
}
/// <summary>
/// Represents a buffer operation to be replicated to standby.
/// </summary>
public record ReplicationOperation(
ReplicationOperationType OperationType,
string MessageId,
StoreAndForwardMessage? Message);
/// <summary>
/// Types of buffer operations that are replicated.
/// </summary>
public enum ReplicationOperationType
{
Add,
Remove,
Park,
/// <summary>
/// An operator moved a parked message back to the pending
/// queue. The standby resets its matching row to Pending with retry_count = 0.
/// </summary>
Requeue
}
src/ZB.MOM.WW.ScadaBridge.StoreAndForward/ServiceCollectionExtensions.cs
-9
View File
@@ -28,7 +28,6 @@ public static class ServiceCollectionExtensions
var storage = sp.GetRequiredService<StoreAndForwardStorage>();
var options = sp.GetRequiredService<IOptions<StoreAndForwardOptions>>().Value;
var logger = sp.GetRequiredService<ILogger<StoreAndForwardService>>();
var replication = sp.GetRequiredService<ReplicationService>();
// Wire the cached-call lifecycle
// observer + site identity through DI so the S&F retry loop emits
// per-attempt + terminal telemetry under the same TrackedOperationId
@@ -53,19 +52,11 @@ public static class ServiceCollectionExtensions
storage,
options,
logger,
replication,
cachedCallObserver,
siteId,
siteEventLogger);
});
services.AddSingleton<ReplicationService>(sp =>
{
var options = sp.GetRequiredService<IOptions<StoreAndForwardOptions>>().Value;
var logger = sp.GetRequiredService<ILogger<ReplicationService>>();
return new ReplicationService(options, logger);
});
return services;
}
src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardService.cs
+9 -23
View File
@@ -36,7 +36,6 @@ public class StoreAndForwardService
{
private readonly StoreAndForwardStorage _storage;
private readonly StoreAndForwardOptions _options;
private readonly ReplicationService? _replication;
private readonly ILogger<StoreAndForwardService> _logger;
/// <summary>
/// Site-side observer notified
@@ -107,7 +106,7 @@ public class StoreAndForwardService
/// <c>null</c> when no sweep is currently running. Captured when the timer
/// callback starts a sweep so <see cref="StopAsync"/> can wait for it to
/// finish before the host disposes downstream dependencies
/// (<see cref="_storage"/>, <see cref="_replication"/>) that the sweep is
/// (<see cref="_storage"/>) that the sweep is
/// still touching. Written from the timer thread and from
/// <see cref="StopAsync"/>, so reads are synchronised via the
/// <see cref="Volatile"/> APIs.
@@ -227,7 +226,6 @@ public class StoreAndForwardService
/// <param name="storage">The storage backend for buffered messages.</param>
/// <param name="options">Configuration options.</param>
/// <param name="logger">Logger instance.</param>
/// <param name="replication">Optional replication service for standby synchronization.</param>
/// <param name="cachedCallObserver">Optional observer for cached call lifecycle events.</param>
/// <param name="siteId">The site identifier this service belongs to.</param>
/// <param name="siteEventLogger">
@@ -240,7 +238,6 @@ public class StoreAndForwardService
StoreAndForwardStorage storage,
StoreAndForwardOptions options,
ILogger<StoreAndForwardService> logger,
ReplicationService? replication = null,
ICachedCallLifecycleObserver? cachedCallObserver = null,
string siteId = "",
ISiteEventLogger? siteEventLogger = null)
@@ -248,7 +245,6 @@ public class StoreAndForwardService
_storage = storage;
_options = options;
_logger = logger;
_replication = replication;
_cachedCallObserver = cachedCallObserver;
_siteId = string.IsNullOrWhiteSpace(siteId) ? UnknownSiteSentinel : siteId;
_siteEventLogger = siteEventLogger;
@@ -651,7 +647,6 @@ public class StoreAndForwardService
private async Task BufferAsync(StoreAndForwardMessage message)
{
await _storage.EnqueueAsync(message);
_replication?.ReplicateEnqueue(message);
Interlocked.Increment(ref _bufferedCount);
}
@@ -802,7 +797,6 @@ public class StoreAndForwardService
if (success)
{
await _storage.RemoveMessageAsync(message.Id);
_replication?.ReplicateRemove(message.Id);
Interlocked.Decrement(ref _bufferedCount);
RaiseActivity("Delivered", message.Category,
$"Delivered to {message.Target} after {message.RetryCount} retries");
@@ -833,7 +827,6 @@ public class StoreAndForwardService
return RetryOutcome.Skipped;
}
Interlocked.Decrement(ref _bufferedCount);
_replication?.ReplicatePark(message);
RaiseActivity("Parked", message.Category,
$"Permanent failure for {message.Target}: handler returned false");
@@ -869,7 +862,6 @@ public class StoreAndForwardService
return RetryOutcome.Skipped;
}
Interlocked.Decrement(ref _bufferedCount);
_replication?.ReplicatePark(message);
RaiseActivity("Parked", message.Category,
$"Max retries ({message.MaxRetries}) reached for {message.Target}");
_logger.LogWarning(
@@ -1077,17 +1069,12 @@ public class StoreAndForwardService
/// <summary>
/// Retries a parked message (moves back to pending queue).
///
/// An operator requeue is a buffer state change and is
/// replicated to the standby (as a <see cref="ReplicationOperationType.Requeue"/>)
/// so a failover preserves the operator's retry intent.
/// An operator requeue is a buffer state change, and the peer picks it up because
/// <c>sf_messages</c> is a replicated table: the row update is captured and shipped
/// like any other write, so a failover preserves the operator's retry intent. This
/// used to be an explicit Requeue operation sent to the standby.
/// The activity-log entry carries the message's true
/// category rather than a hard-coded one.
/// The parked row is captured <i>before</i> the local
/// requeue write rather than re-read after it, so a concurrent
/// <c>RemoveMessageAsync</c> or <c>DiscardParkedMessageAsync</c> running
/// between the two storage calls cannot leave the standby in <c>Parked</c>
/// while the active node has already requeued — we always have the row in
/// hand for the <c>Requeue</c> replication.
/// </summary>
/// <param name="messageId">The identifier of the message to retry.</param>
/// <returns>True if successfully retried, false otherwise.</returns>
@@ -1117,7 +1104,6 @@ public class StoreAndForwardService
captured.RetryCount = 0;
captured.LastError = null;
captured.LastAttemptAt = null;
_replication?.ReplicateRequeue(captured);
RaiseActivity("Retry", captured.Category,
$"Parked message {messageId} moved back to queue");
@@ -1127,9 +1113,10 @@ public class StoreAndForwardService
/// <summary>
/// Permanently discards a parked message.
///
/// An operator discard is a buffer removal and is replicated
/// to the standby (as a <see cref="ReplicationOperationType.Remove"/>) so the
/// discarded message does not reappear after a failover.
/// An operator discard is a buffer removal. The delete is captured as a tombstone on
/// the replicated <c>sf_messages</c> table and carries the later HLC, so the discarded
/// message cannot reappear after a failover regardless of arrival order. This used to
/// be an explicit Remove operation sent to the standby.
/// The activity-log entry carries the message's true
/// category rather than a hard-coded one.
/// </summary>
@@ -1143,7 +1130,6 @@ public class StoreAndForwardService
var success = await _storage.DiscardParkedMessageAsync(messageId);
if (success)
{
_replication?.ReplicateRemove(messageId);
RaiseActivity("Discard", message?.Category ?? StoreAndForwardCategory.ExternalSystem,
$"Parked message {messageId} discarded");
}
src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardStorage.cs
+1 -34
View File
@@ -65,7 +65,7 @@ public class StoreAndForwardStorage
/// <summary>
/// INSERT statement for a full message row. Shared by <see cref="EnqueueAsync"/>
/// and <see cref="ReplaceAllAsync"/>; bind with <see cref="BindMessageParameters"/>
/// and <see cref="UpsertMessageAsync"/>; bind with <see cref="BindMessageParameters"/>
/// so the column list and the parameters never drift apart.
/// </summary>
private const string InsertMessageSql = @"
@@ -151,39 +151,6 @@ public class StoreAndForwardStorage
return (rows.Take(limit).ToList(), truncated);
}
/// <summary>
/// Atomically replaces the entire buffer with <paramref name="messages"/> in a
/// single transaction (delete-all then insert-all). Standby-side anti-entropy
/// apply: a peer-join resync overwrites the standby's divergent copy with the
/// active node's authoritative snapshot. <b>Never call on an active node</b> —
/// it discards every in-flight row.
/// </summary>
/// <param name="messages">The full buffer snapshot to install.</param>
/// <returns>A task that represents the asynchronous operation.</returns>
public async Task ReplaceAllAsync(IReadOnlyList<StoreAndForwardMessage> messages)
{
await using var connection = OpenConnection();
await using var transaction = (SqliteTransaction)await connection.BeginTransactionAsync();
await using (var deleteCmd = connection.CreateCommand())
{
deleteCmd.Transaction = transaction;
deleteCmd.CommandText = "DELETE FROM sf_messages";
await deleteCmd.ExecuteNonQueryAsync();
}
foreach (var message in messages)
{
await using var insertCmd = connection.CreateCommand();
insertCmd.Transaction = transaction;
insertCmd.CommandText = InsertMessageSql;
BindMessageParameters(insertCmd, message);
await insertCmd.ExecuteNonQueryAsync();
}
await transaction.CommitAsync();
}
/// <summary>
/// Inserts a message, or updates every mutable column in place if a row with the
/// same id already exists (<c>ON CONFLICT(id) DO UPDATE</c>). Used by the standby