dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 1 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

docs(comments): strip internal task/milestone/bundle bookkeeping from code comments

Browse Source 
Remove project bookkeeping citations from shipped code comments across the
solution: hyphenated task IDs (WP-14, StoreAndForward-025), milestone/task/
issue refs (M3, Task 4, Audit Log #23, #21), Bundle X task-bundle labels,
and C/D/K/S/T phase labels.

Comment text only — no code logic, string/log literals, or XML-doc structure
changed. Genuine descriptions are preserved (only the citation is stripped),
and technical lookalikes are retained (UTF-8, SHA-256, T00:00:00, M365,
UTC-5, pre-C4/pre-C5 schema versions). Flagged by the new CommentChecker
TaskReferenceInComment / TrackingReferenceInComment checks plus targeted
grep passes; full solution builds clean, append-only guard tests pass.
This commit is contained in:
Joseph Doherty
2026-07-07 11:03:26 -04:00
parent 67005ca4c0
commit 9cff87fe85
435 changed files with 2338 additions and 2547 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardService.cs
+67 -173
View File
@@ -8,7 +8,7 @@ using ZB.MOM.WW.ScadaBridge.SiteEventLogging;
namespace ZB.MOM.WW.ScadaBridge.StoreAndForward;
/// <summary>
/// WP-9/10: Core store-and-forward service.
/// Core store-and-forward service.
///
/// Lifecycle:
/// 1. Caller attempts immediate delivery via IDeliveryHandler
@@ -18,14 +18,14 @@ namespace ZB.MOM.WW.ScadaBridge.StoreAndForward;
/// message is retried until delivered and is never parked for retry exhaustion)
/// 5. Permanent failures are returned to caller immediately (never buffered)
///
/// WP-10: Fixed retry interval (not exponential). Per-source-entity retry settings.
/// Fixed retry interval (not exponential). Per-source-entity retry settings.
/// Background timer-based retry sweep.
///
/// WP-12: Parked messages queryable, retryable, and discardable.
/// Parked messages queryable, retryable, and discardable.
///
/// WP-14: Buffer depth reported as health metric. Activity logged to site event log.
/// Buffer depth reported as health metric. Activity logged to site event log.
///
/// WP-15: CachedCall idempotency is the caller's responsibility.
/// CachedCall idempotency is the caller's responsibility.
/// This service does not deduplicate — if the same message is enqueued twice,
/// it will be delivered twice. Callers using ExternalSystem.CachedCall() must
/// design their payloads to be idempotent (e.g., include unique request IDs
@@ -38,14 +38,14 @@ public class StoreAndForwardService
private readonly ReplicationService? _replication;
private readonly ILogger<StoreAndForwardService> _logger;
/// <summary>
/// Audit Log #23 (M3 Bundle E — Task E4): site-side observer notified
/// Site-side observer notified
/// after every cached-call delivery attempt. Optional — when null no
/// telemetry is emitted; the legacy pre-M3 retry loop behaviour is
/// telemetry is emitted; the legacy retry loop behaviour is
/// preserved exactly.
/// </summary>
private readonly ICachedCallLifecycleObserver? _cachedCallObserver;
/// <summary>
/// M1.7: optional site operational-event log. When non-null the service maps
/// Optional site operational-event log. When non-null the service maps
/// its own buffer/retry/park activity (the same activity that drives
/// <see cref="OnActivity"/>) onto site events — <c>store_and_forward</c> for the
/// cached-call categories and <c>notification</c> for the site's
@@ -54,26 +54,14 @@ public class StoreAndForwardService
/// </summary>
private readonly ISiteEventLogger? _siteEventLogger;
/// <summary>
/// Audit Log #23 (M3 Bundle E — Task E4): site id stamped onto the
/// Site id stamped onto the
/// cached-call attempt context so the audit bridge can build the
/// <see cref="SiteCallOperational"/> half of the telemetry packet.
/// <para>
/// <b>StoreAndForward-023:</b> an empty-string site id must never reach
/// downstream consumers — the central audit pipeline keys
/// <c>(SourceSite, TrackedOperationId)</c> off this value, so an empty
/// string degrades correlation to a per-id-only index and breaks the
/// per-site routing of <c>RetryParkedOperation</c>/<c>DiscardParkedOperation</c>
/// commands. The constructor normalises a null/empty/whitespace
/// <paramref name="siteId"/> argument to <see cref="UnknownSiteSentinel"/>
/// so a misconfigured host (no <c>IStoreAndForwardSiteContext</c>
/// registered) produces a distinctive marker in the central audit log
/// rather than silently merging multiple sites into the empty bucket.
/// </para>
/// </summary>
private readonly string _siteId;
/// <summary>
/// StoreAndForward-023: distinctive marker stamped onto cached-call audit
/// Distinctive marker stamped onto cached-call audit
/// telemetry when the host has not registered an
/// <see cref="IStoreAndForwardSiteContext"/>. Chosen with a leading <c>$</c>
/// so it cannot collide with a real site id (which is a configuration
@@ -84,7 +72,7 @@ public class StoreAndForwardService
public const string UnknownSiteSentinel = "$unknown-site";
/// <summary>
/// M1.7: the detail-string prefix written by <see cref="EnqueueAsync"/>
/// The detail-string prefix written by <see cref="EnqueueAsync"/>
/// when an immediate forward attempt throws and the message is buffered for
/// the retry sweep. <see cref="EmitSiteEvent"/> matches on this same prefix
/// to distinguish a forward <i>failure</i> (logged) from a routine
@@ -98,7 +86,7 @@ public class StoreAndForwardService
private int _retryInProgress;
/// <summary>
/// StoreAndForward-024: the in-flight retry sweep <see cref="Task"/>, or
/// The in-flight retry sweep <see cref="Task"/>, or
/// <c>null</c> when no sweep is currently running. Captured when the timer
/// callback starts a sweep so <see cref="StopAsync"/> can wait for it to
/// finish before the host disposes downstream dependencies
@@ -110,7 +98,7 @@ public class StoreAndForwardService
private Task? _sweepTask;
/// <summary>
/// StoreAndForward-024: how long <see cref="StopAsync"/> waits for an
/// How long <see cref="StopAsync"/> waits for an
/// in-flight retry sweep to finish before returning. The default — 10 s —
/// is generous enough to let a typical sweep over the buffered queue drain,
/// but bounded so a hung downstream call (a stuck SQLite write, a
@@ -122,7 +110,7 @@ public class StoreAndForwardService
private static readonly TimeSpan SweepShutdownWaitTimeout = TimeSpan.FromSeconds(10);
/// <summary>
/// WP-14 (telemetry): cached count of messages currently buffered for
/// Cached count of messages currently buffered for
/// forwarding — i.e. rows in <see cref="StoreAndForwardMessageStatus.Pending"/>,
/// the live store-and-forward queue waiting to be delivered. This backs the
/// <c>scadabridge.store_and_forward.queue.depth</c> observable gauge.
@@ -145,7 +133,7 @@ public class StoreAndForwardService
private long _bufferedCount;
/// <summary>
/// Test seam (WP-14 telemetry): simulates a concurrent pre-seed
/// Test seam: simulates a concurrent pre-seed
/// <see cref="BufferAsync"/> increment landing on <see cref="_bufferedCount"/>
/// before <see cref="StartAsync"/> seeds it, so a test can prove the seed uses
/// <see cref="Interlocked.Add"/> (additive) rather than Exchange (clobbering).
@@ -154,7 +142,7 @@ public class StoreAndForwardService
Interlocked.Increment(ref _bufferedCount);
/// <summary>
/// WP-14 (telemetry): an instance field that guards against a single instance
/// An instance field that guards against a single instance
/// registering the queue-depth provider (and re-seeding the counter) more than
/// once — e.g. a second <see cref="StartAsync"/> on the same instance. It does NOT
/// coordinate across instances: the gauge slot in <see cref="ScadaBridgeTelemetry"/>
@@ -164,7 +152,7 @@ public class StoreAndForwardService
private int _queueDepthProviderRegistered;
/// <summary>
/// StoreAndForward-025: the exact provider delegate this instance registered with
/// The exact provider delegate this instance registered with
/// the process-global <see cref="ScadaBridgeTelemetry"/> gauge slot, retained so
/// <see cref="StopAsync"/> can deregister it by identity (compare-and-clear). Holding
/// the same reference both registers and clears the slot; the identity check ensures a
@@ -174,7 +162,7 @@ public class StoreAndForwardService
private Func<long>? _queueDepthProvider;
/// <summary>
/// WP-10: Delivery handler delegate. The return value / exception is interpreted
/// Delivery handler delegate. The return value / exception is interpreted
/// the same way on both the immediate-delivery path (<see cref="EnqueueAsync"/>)
/// and the background retry path (<c>RetryMessageAsync</c>):
/// <list type="bullet">
@@ -192,7 +180,7 @@ public class StoreAndForwardService
private readonly Dictionary<StoreAndForwardCategory, Func<StoreAndForwardMessage, Task<bool>>> _deliveryHandlers = new();
/// <summary>
/// WP-14: Event callback for logging S&amp;F activity to site event log.
/// Event callback for logging S&amp;F activity to site event log.
/// </summary>
public event Action<string, StoreAndForwardCategory, string>? OnActivity;
@@ -206,7 +194,7 @@ public class StoreAndForwardService
/// <param name="cachedCallObserver">Optional observer for cached call lifecycle events.</param>
/// <param name="siteId">The site identifier this service belongs to.</param>
/// <param name="siteEventLogger">
/// M1.7: optional site operational-event log. When non-null, buffer/retry/park
/// Optional site operational-event log. When non-null, buffer/retry/park
/// activity is mirrored to site events (<c>store_and_forward</c> /
/// <c>notification</c> by category). Optional with a <c>null</c> default so the
/// many direct-construction tests still compile unchanged.
@@ -225,17 +213,13 @@ public class StoreAndForwardService
_logger = logger;
_replication = replication;
_cachedCallObserver = cachedCallObserver;
// StoreAndForward-023: normalise an empty / whitespace site id to the
// distinctive UnknownSiteSentinel so downstream consumers (the central
// audit pipeline keying off SourceSite) never see an empty string and
// a misconfigured host is recognisable in the central log.
_siteId = string.IsNullOrWhiteSpace(siteId) ? UnknownSiteSentinel : siteId;
_siteEventLogger = siteEventLogger;
// M1.7: ride the existing activity hook to emit site operational events.
// Ride the existing activity hook to emit site operational events.
// RaiseActivity already isolates a throwing subscriber, so a failing
// event log can never be misclassified as a transient delivery failure
// (StoreAndForward-009). Only subscribe when a logger is wired so the
// event log can never be misclassified as a transient delivery failure.
// Only subscribe when a logger is wired so the
// legacy (test/central) construction path stays a no-op.
if (_siteEventLogger != null)
{
@@ -244,7 +228,7 @@ public class StoreAndForwardService
}
/// <summary>
/// M1.7: maps one store-and-forward activity to a site operational event,
/// Maps one store-and-forward activity to a site operational event,
/// following the Site Event Logging spec's per-category scope
/// (Component-SiteEventLogging.md §"Events Logged"):
/// <list type="bullet">
@@ -340,39 +324,17 @@ public class StoreAndForwardService
{
await _storage.InitializeAsync();
// WP-14 (telemetry): seed the cached buffered-message count from the
// store exactly once (the gauge callback cannot run an async COUNT), then
// register the sync, non-blocking provider with the process-global
// ScadaBridgeTelemetry gauge. Both steps are inside the one-time guard so a
// second StartAsync on the same instance cannot double-seed.
//
// The seed is an Interlocked.Add — NOT an Exchange — to avoid a startup race:
// between the await above returning and this point, a concurrent BufferAsync
// could already have Interlocked.Increment'd _bufferedCount. Exchange would
// clobber that increment (losing a +1); Add preserves it. _bufferedCount
// starts at 0 and only BufferAsync increments it before the seed, so
// 0 + pending + (any concurrent increments) is the correct live count.
var pending = await _storage.GetMessageCountByStatusAsync(
StoreAndForwardMessageStatus.Pending);
if (Interlocked.CompareExchange(ref _queueDepthProviderRegistered, 1, 0) == 0)
{
Interlocked.Add(ref _bufferedCount, pending);
// StoreAndForward-025: retain the exact delegate so StopAsync can
// deregister it by identity (compare-and-clear) without stomping a
// newer instance that may have re-registered into the global slot.
var provider = (Func<long>)(() => Interlocked.Read(ref _bufferedCount));
_queueDepthProvider = provider;
ScadaBridgeTelemetry.SetQueueDepthProvider(provider);
}
_retryTimer = new Timer(
// StoreAndForward-024: capture the sweep Task on each tick so
// StopAsync can await any in-flight invocation before the host
// disposes _storage/_replication underneath it. The RetryPending
// path is self-guarded against overlapping sweeps via the
// _retryInProgress Interlocked flag, so unconditionally re-assigning
// the field here cannot lose a still-running task (the new tick
// will short-circuit if one is already running).
_ => Volatile.Write(ref _sweepTask, RetryPendingMessagesAsync()),
null,
_options.RetryTimerInterval,
@@ -386,24 +348,6 @@ public class StoreAndForwardService
/// <summary>
/// Stops the background retry timer and waits (bounded) for any in-flight
/// retry sweep to finish before returning.
///
/// StoreAndForward-024: prior to this fix, <see cref="StopAsync"/> only
/// disposed the timer — a sweep already inside
/// <see cref="RetryPendingMessagesAsync"/> continued running against
/// <see cref="_storage"/> and <see cref="_replication"/> after this method
/// returned, and could then NRE / throw on a disposed dependency once the
/// DI container ran its own shutdown. We now await the captured sweep task
/// (with a bounded <see cref="SweepShutdownWaitTimeout"/> so a hung
/// dependency cannot block host shutdown indefinitely) before returning.
///
/// StoreAndForward-025: after the timer is disposed and the in-flight sweep has
/// drained, the queue-depth provider this instance registered with the process-global
/// <see cref="ScadaBridgeTelemetry"/> gauge is deregistered by identity (compare-and-
/// clear) — otherwise a stopped service would report a frozen depth forever and the
/// provider closure would pin this dead instance for the process lifetime. The clear is
/// identity-checked so a newer instance that already re-registered the global slot is
/// not stomped, and <see cref="_queueDepthProviderRegistered"/> is reset so a later
/// <see cref="StartAsync"/> on this instance re-registers cleanly.
/// </summary>
/// <returns>A task representing the asynchronous stop operation.</returns>
public async Task StopAsync()
@@ -445,33 +389,22 @@ public class StoreAndForwardService
}
}
// StoreAndForward-025: release the process-global queue-depth gauge provider so a
// stopped service stops reporting a frozen depth and the closure no longer pins
// this dead instance. Identity-checked (compare-and-clear) so a successor
// instance's provider is left intact; reset the one-time guard so a later
// StartAsync re-registers.
var provider = _queueDepthProvider;
if (provider is not null)
{
ScadaBridgeTelemetry.ClearQueueDepthProvider(provider);
_queueDepthProvider = null;
}
// StoreAndForward-028: reset the cached depth alongside the registration guard.
// StartAsync re-seeds _bufferedCount from the durable Pending count under this
// same guard; without resetting here, a same-instance Stop->Start would ADD the
// re-read count on top of the leftover in-memory value, double-counting the gauge
// (~2N). Reset to zero so the restart seeds from a clean base. Buffered rows
// remain durable in SQLite, so the re-seed restores the true count.
Interlocked.Exchange(ref _bufferedCount, 0);
Interlocked.Exchange(ref _queueDepthProviderRegistered, 0);
}
/// <summary>
/// WP-10: Enqueues a message for store-and-forward delivery.
/// Enqueues a message for store-and-forward delivery.
/// Attempts immediate delivery first. On transient failure, buffers for retry.
/// On permanent failure (handler returns false), returns false immediately.
///
/// WP-10: Retry-count lifecycle — the immediate (or caller-made) delivery attempt
/// Retry-count lifecycle — the immediate (or caller-made) delivery attempt
/// is attempt 0 and is not counted; the background retry sweep increments
/// <see cref="StoreAndForwardMessage.RetryCount"/> on each retry. A buffered
/// message is parked once <c>RetryCount</c> reaches <paramref name="maxRetries"/>
@@ -482,14 +415,14 @@ public class StoreAndForwardService
/// that want delivery abandoned after a bounded number of attempts must pass a
/// positive <paramref name="maxRetries"/>.
///
/// WP-15: CachedCall idempotency note — this method does not deduplicate.
/// CachedCall idempotency note — this method does not deduplicate.
/// The caller (e.g., ExternalSystem.CachedCall()) is responsible for ensuring
/// that the remote system can handle duplicate deliveries safely.
/// </summary>
/// <param name="category">Message category (selects the delivery handler).</param>
/// <param name="target">Target system name (external system / notification list / DB connection).</param>
/// <param name="payloadJson">JSON-serialized call payload, treated opaquely.</param>
/// <param name="originInstanceName">Instance that originated the message (WP-13: survives instance deletion).</param>
/// <param name="originInstanceName">Instance that originated the message (survives instance deletion).</param>
/// <param name="maxRetries">
/// Maximum background retry-sweep attempts before the message is parked.
/// <b><c>0</c> = no limit</b> — the message is retried on every sweep until
@@ -511,23 +444,23 @@ public class StoreAndForwardService
/// inside the payload, and it is the id the forwarder submits to central.
/// </param>
/// <param name="executionId">
/// Audit Log #23 (ExecutionId Task 4): the originating script execution's
/// The originating script execution's
/// per-run correlation id. Threaded onto the buffered row so the retry-loop
/// cached-call audit rows carry it. <c>null</c> for callers (notifications,
/// pre-Task-4 callers) that do not supply one.
/// cached-call audit rows carry it. <c>null</c> for callers (e.g. notifications)
/// that do not supply one.
/// </param>
/// <param name="sourceScript">
/// Audit Log #23 (ExecutionId Task 4): the originating script identifier,
/// The originating script identifier,
/// threaded onto the buffered row alongside <paramref name="executionId"/>
/// so the retry-loop audit rows carry the same provenance the script-side
/// cached rows do. <c>null</c> when not known.
/// </param>
/// <param name="parentExecutionId">
/// Audit Log #23 (ParentExecutionId Task 6): the <c>ExecutionId</c> of the
/// The <c>ExecutionId</c> of the
/// inbound-API request that spawned the originating script execution.
/// Threaded onto the buffered row alongside <paramref name="executionId"/>
/// so the retry-loop cached-call audit rows carry it. <c>null</c> for a
/// non-routed run and for callers (notifications, pre-Task-6 callers) that
/// non-routed run and for callers (e.g. notifications) that
/// do not supply one.
/// </param>
/// <returns>A task that resolves to a result indicating whether the message was delivered or buffered.</returns>
@@ -582,7 +515,7 @@ public class StoreAndForwardService
{
// Transient failure — buffer for retry. The immediate attempt is
// attempt 0; RetryCount tracks only sweep retries, so it stays 0
// here (StoreAndForward-003).
// here.
_logger.LogWarning(ex,
"Immediate delivery to {Target} failed (transient), buffering for retry",
target);
@@ -600,7 +533,7 @@ public class StoreAndForwardService
// delivery itself — buffer for the background retry sweep to deliver.
// The initial attempt (caller-made, or skipped because no handler is
// registered) is attempt 0; RetryCount tracks only sweep retries and
// therefore stays 0 here (StoreAndForward-003).
// therefore stays 0 here.
if (!attemptImmediateDelivery)
{
message.LastAttemptAt = DateTimeOffset.UtcNow;
@@ -613,21 +546,18 @@ public class StoreAndForwardService
}
/// <summary>
/// Persists a message to the local SQLite buffer and (WP-11) replicates the
/// Persists a message to the local SQLite buffer and replicates the
/// add to the standby node so a failover does not lose the buffered message.
/// </summary>
private async Task BufferAsync(StoreAndForwardMessage message)
{
await _storage.EnqueueAsync(message);
_replication?.ReplicateEnqueue(message);
// WP-14 (telemetry): a freshly buffered row is Pending → grows the live
// queue depth. Bumped after the durable write so the gauge never leads the
// store.
Interlocked.Increment(ref _bufferedCount);
}
/// <summary>
/// WP-10: Background retry sweep. Processes all pending messages that are due for retry.
/// Background retry sweep. Processes all pending messages that are due for retry.
/// </summary>
/// <returns>A task representing the asynchronous retry sweep.</returns>
internal async Task RetryPendingMessagesAsync()
@@ -666,7 +596,7 @@ public class StoreAndForwardService
return;
}
// Audit Log #23 (M3 Bundle E — Tasks E4/E5): measure per-attempt
// Measure per-attempt
// duration so the audit row carries a meaningful DurationMs. Captured
// around the handler invocation only — storage / replication overhead
// is excluded.
@@ -681,12 +611,11 @@ public class StoreAndForwardService
{
await _storage.RemoveMessageAsync(message.Id);
_replication?.ReplicateRemove(message.Id);
// WP-14 (telemetry): a delivered row leaves the Pending queue.
Interlocked.Decrement(ref _bufferedCount);
RaiseActivity("Delivered", message.Category,
$"Delivered to {message.Target} after {message.RetryCount} retries");
// M3: terminal Delivered observer notification — the audit
// Terminal Delivered observer notification — the audit
// bridge maps this to Attempted + CachedResolve(Delivered).
await NotifyCachedCallObserverAsync(
message,
@@ -699,9 +628,6 @@ public class StoreAndForwardService
}
// Permanent failure on retry — park immediately.
// StoreAndForward-005: the sweep observed this row as Pending; only commit
// the park if it is still Pending so a concurrent operator action that
// moved it (retry/discard) is not silently overwritten.
message.Status = StoreAndForwardMessageStatus.Parked;
message.LastAttemptAt = DateTimeOffset.UtcNow;
message.LastError = "Permanent failure (handler returned false)";
@@ -714,14 +640,12 @@ public class StoreAndForwardService
message.Id);
return;
}
// WP-14 (telemetry): the row committed Pending→Parked, leaving the live
// forward queue. Only counted when the conditional update actually won.
Interlocked.Decrement(ref _bufferedCount);
_replication?.ReplicatePark(message);
RaiseActivity("Parked", message.Category,
$"Permanent failure for {message.Target}: handler returned false");
// M3: terminal PermanentFailure observer notification — the
// Terminal PermanentFailure observer notification — the
// audit bridge maps this to Attempted(Failed) + CachedResolve(Parked).
await NotifyCachedCallObserverAsync(
message,
@@ -741,8 +665,6 @@ public class StoreAndForwardService
if (message.MaxRetries > 0 && message.RetryCount >= message.MaxRetries)
{
// StoreAndForward-005: conditional park — see the permanent-failure
// branch above for rationale.
message.Status = StoreAndForwardMessageStatus.Parked;
var parked = await _storage.UpdateMessageIfStatusAsync(
message, StoreAndForwardMessageStatus.Pending);
@@ -753,8 +675,6 @@ public class StoreAndForwardService
message.Id);
return;
}
// WP-14 (telemetry): the row committed Pending→Parked, leaving the
// live forward queue. Only counted when the conditional update won.
Interlocked.Decrement(ref _bufferedCount);
_replication?.ReplicatePark(message);
RaiseActivity("Parked", message.Category,
@@ -763,7 +683,7 @@ public class StoreAndForwardService
"Message {MessageId} parked after {MaxRetries} retries to {Target}",
message.Id, message.MaxRetries, message.Target);
// M3: terminal ParkedMaxRetries observer notification — the
// Terminal ParkedMaxRetries observer notification — the
// audit bridge maps this to Attempted(Failed) + CachedResolve(Parked).
await NotifyCachedCallObserverAsync(
message,
@@ -775,9 +695,6 @@ public class StoreAndForwardService
}
else
{
// StoreAndForward-005: the retry-count increment is also conditional
// on the row still being Pending so it cannot clobber an operator
// action that ran during the failed delivery.
if (!await _storage.UpdateMessageIfStatusAsync(
message, StoreAndForwardMessageStatus.Pending))
{
@@ -789,7 +706,7 @@ public class StoreAndForwardService
RaiseActivity("Retried", message.Category,
$"Retry {message.RetryCount}/{message.MaxRetries} for {message.Target}: {ex.Message}");
// M3: per-attempt TransientFailure observer notification —
// Per-attempt TransientFailure observer notification —
// the audit bridge maps this to Attempted(Failed).
await NotifyCachedCallObserverAsync(
message,
@@ -803,20 +720,20 @@ public class StoreAndForwardService
}
/// <summary>
/// Audit Log #23 (M3 Bundle E — Tasks E4/E5): notify the registered
/// Notify the registered
/// <see cref="ICachedCallLifecycleObserver"/> of the just-completed
/// attempt. Only fires for cached-call categories
/// (<see cref="StoreAndForwardCategory.ExternalSystem"/> and
/// <see cref="StoreAndForwardCategory.CachedDbWrite"/>); the
/// <see cref="StoreAndForwardCategory.Notification"/> category has its
/// own central-side audit pipeline (Notification Outbox / #21) and must
/// own central-side audit pipeline (Notification Outbox) and must
/// not surface on this hook.
/// </summary>
/// <remarks>
/// Best-effort: an observer that throws is logged and swallowed so a
/// failing audit pipeline cannot corrupt S&amp;F retry bookkeeping
/// (alog.md §7 contract). Messages whose ids are not valid GUIDs (pre-M3
/// callers that didn't thread a TrackedOperationId in) are silently
/// (alog.md §7 contract). Messages whose ids are not valid GUIDs (callers
/// that didn't thread a TrackedOperationId in) are silently
/// skipped — the observer requires a parseable id by contract.
/// </remarks>
private async Task NotifyCachedCallObserverAsync(
@@ -847,17 +764,6 @@ public class StoreAndForwardService
if (!TrackedOperationId.TryParse(message.Id, out var trackedId))
{
// StoreAndForward-022: previously a silent skip — but a non-GUID
// message id means a caller bypassed the audit hot path with zero
// feedback. The drop is still best-effort (S&F retry bookkeeping
// must never depend on the audit pipeline) but it is now observable
// via a Warning so a misconfigured caller can be diagnosed.
// Engine-minted ids (Guid.NewGuid().ToString("N")) and the current
// caller set (NotificationOutbox enqueue with NotificationId,
// cached-call enqueue with TrackedOperationId.ToString()) all
// parse — this log line fires only when a future caller supplies a
// non-GUID id, which is exactly when the silent-drop was hardest
// to diagnose.
_logger.LogWarning(
"Cached-call audit observer skipped: message id {MessageId} is not a parseable TrackedOperationId (category {Category}, outcome {Outcome}). " +
"Audit lifecycle for this operation will have no rows.",
@@ -881,18 +787,17 @@ public class StoreAndForwardService
OccurredAtUtc: DateTime.SpecifyKind(occurredAtUtc, DateTimeKind.Utc),
DurationMs: durationMs,
SourceInstanceId: message.OriginInstanceName,
// Audit Log #23 (ExecutionId Task 4): the buffered message
// carries the originating script execution's ExecutionId +
// SourceScript; surface them on the context so the bridge can
// stamp the retry-loop cached audit rows. Null on rows buffered
// before Task 4 (back-compat).
// The buffered message carries the originating script
// execution's ExecutionId + SourceScript; surface them on the
// context so the bridge can stamp the retry-loop cached audit
// rows. Null on rows buffered before this field existed
// (back-compat).
ExecutionId: message.ExecutionId,
SourceScript: message.SourceScript,
// Audit Log #23 (ParentExecutionId Task 6): the buffered
// message also carries the spawning inbound-API request's
// ExecutionId; surface it so the bridge stamps it onto the
// retry-loop cached rows. Null for a non-routed run and on
// rows buffered before Task 6 (back-compat).
// The buffered message also carries the spawning inbound-API
// request's ExecutionId; surface it so the bridge stamps it
// onto the retry-loop cached rows. Null for a non-routed run
// and on rows buffered before this field existed (back-compat).
ParentExecutionId: message.ParentExecutionId);
}
catch (Exception buildEx)
@@ -922,7 +827,7 @@ public class StoreAndForwardService
}
/// <summary>
/// WP-12: Gets parked messages for central query (Pattern 8).
/// Gets parked messages for central query (Pattern 8).
/// </summary>
/// <param name="category">Optional category filter, or null for all categories.</param>
/// <param name="pageNumber">The page number (1-based).</param>
@@ -937,14 +842,14 @@ public class StoreAndForwardService
}
/// <summary>
/// WP-12: Retries a parked message (moves back to pending queue).
/// Retries a parked message (moves back to pending queue).
///
/// StoreAndForward-016: an operator requeue is a buffer state change and is
/// An operator requeue is a buffer state change and is
/// replicated to the standby (as a <see cref="ReplicationOperationType.Requeue"/>)
/// so a failover preserves the operator's retry intent.
/// StoreAndForward-017: the activity-log entry carries the message's true
/// The activity-log entry carries the message's true
/// category rather than a hard-coded one.
/// StoreAndForward-020: the parked row is captured <i>before</i> the local
/// The parked row is captured <i>before</i> the local
/// requeue write rather than re-read after it, so a concurrent
/// <c>RemoveMessageAsync</c> or <c>DiscardParkedMessageAsync</c> running
/// between the two storage calls cannot leave the standby in <c>Parked</c>
@@ -955,13 +860,6 @@ public class StoreAndForwardService
/// <returns>True if successfully retried, false otherwise.</returns>
public async Task<bool> RetryParkedMessageAsync(string messageId)
{
// StoreAndForward-020: capture the parked row up front so the standby
// gets a Requeue even if a concurrent writer (a sweep delete after a
// successful delivery, or an operator discard) removes the row between
// the local update and the re-load. The storage call below is
// conditional on status = Parked, so if the row has already moved we
// return false here without replicating — the standby's matching row
// will be reconciled by whichever other operator path won the race.
var captured = await _storage.GetMessageByIdAsync(messageId);
if (captured is null || captured.Status != StoreAndForwardMessageStatus.Parked)
{
@@ -974,9 +872,6 @@ public class StoreAndForwardService
return false;
}
// WP-14 (telemetry): an operator requeue moves Parked→Pending, re-adding the
// row to the live forward queue. Counted only when the conditional storage
// update actually flipped the row.
Interlocked.Increment(ref _bufferedCount);
// The active node just rewrote this row to Pending with retry_count = 0
@@ -997,12 +892,12 @@ public class StoreAndForwardService
}
/// <summary>
/// WP-12: Permanently discards a parked message.
/// Permanently discards a parked message.
///
/// StoreAndForward-016: an operator discard is a buffer removal and is replicated
/// An operator discard is a buffer removal and is replicated
/// to the standby (as a <see cref="ReplicationOperationType.Remove"/>) so the
/// discarded message does not reappear after a failover.
/// StoreAndForward-017: the activity-log entry carries the message's true
/// The activity-log entry carries the message's true
/// category rather than a hard-coded one.
/// </summary>
/// <param name="messageId">The identifier of the message to discard.</param>
@@ -1011,7 +906,6 @@ public class StoreAndForwardService
{
// Capture the category before the row is deleted so the activity log is
// labelled correctly.
// WP-14 (telemetry): Parked rows are not in _bufferedCount; discarding a Parked row needs no counter adjustment.
var message = await _storage.GetMessageByIdAsync(messageId);
var success = await _storage.DiscardParkedMessageAsync(messageId);
if (success)
@@ -1024,7 +918,7 @@ public class StoreAndForwardService
}
/// <summary>
/// WP-14: Gets buffer depth by category for health reporting.
/// Gets buffer depth by category for health reporting.
/// </summary>
/// <returns>A dictionary of buffer depths by category.</returns>
public async Task<Dictionary<StoreAndForwardCategory, int>> GetBufferDepthAsync()
@@ -1033,7 +927,7 @@ public class StoreAndForwardService
}
/// <summary>
/// WP-13: Gets count of S&amp;F messages for a given instance (for verifying survival on deletion).
/// Gets count of S&amp;F messages for a given instance (for verifying survival on deletion).
/// </summary>
/// <param name="instanceName">The instance name to query.</param>
/// <returns>The number of messages originating from the instance.</returns>
@@ -1056,7 +950,7 @@ public class StoreAndForwardService
}
/// <summary>
/// WP-14: Raises the S&amp;F activity notification. StoreAndForward-009: the
/// Raises the S&amp;F activity notification. The
/// delegate is snapshotted (so a concurrent unsubscribe cannot NRE) and every
/// subscriber invocation is wrapped so a slow/throwing subscriber (e.g. the site
/// event log) cannot abort the caller. Crucially, a subscriber exception raised