fix(auditlog): explicit permanent rejections on the ingest ack (IngestAck.rejected_event_ids, field 2) with a dedicated counter; reconciler ignores transient faults for abandonment; persist the served high-water mark; prune only on a non-empty enumeration (review 2026-10-21 06/PERF-1, 06/IMPL-1 adversarial follow-up)
This commit is contained in:
@@ -311,17 +311,27 @@ A `SiteAuditTelemetryActor` singleton drives the forwarding loop: select up to
|
||||
N `Pending` rows ordered by `OccurredAtUtc`, batch-send them to central via the
|
||||
existing `SiteStream` gRPC channel as `IngestAuditEvents(events)`, and on
|
||||
central-ack flip `ForwardState = 'Forwarded'` for accepted IDs. IDs central
|
||||
answered for but did not accept stay `Pending` for the next sweep, with
|
||||
`AttemptCount` / `LastAttemptUtc` bumped on every such refusal; after
|
||||
`AuditLog:SiteTelemetry:MaxRejectedPushAttempts` (default 10) refusals a row
|
||||
leaves the push queue as `Rejected` and bumps `SiteAuditWriteFailures`
|
||||
(review 2026-10-21 06/IMPL-1). The drain reads oldest-first, so before this one
|
||||
answered for but did not accept stay `Pending` for the next sweep. Central
|
||||
additionally lists, in the ack's additive `rejected_event_ids` (sitestream.proto
|
||||
`IngestAck` field 2), the ids it refused for a **permanent, row-intrinsic** SQL
|
||||
reason (string truncation 8152/2628, NOT NULL 515, conversion/overflow
|
||||
245/241/242/8114/8115/220/232, constraint 547). Only those are refusals: the site
|
||||
bumps `AttemptCount` / `LastAttemptUtc` for them and, after
|
||||
`AuditLog:SiteTelemetry:MaxRejectedPushAttempts` (default 10), moves the row out of
|
||||
the push queue as `Rejected`, counted on its own `scadabridge.site.audit.rejected_rows`
|
||||
counter (not `SiteAuditWriteFailures` — the site's write succeeded). A row merely
|
||||
absent from the accepted list — a SQL maintenance window, a timeout, an exhausted
|
||||
ingest budget — is never counted, and an older central that does not send the field
|
||||
never triggers it (review 2026-10-21 06/IMPL-1). The drain reads oldest-first, so before this one
|
||||
row central can never insert (e.g. an over-width column) sat at the head of every
|
||||
batch forever, failing each set-based insert at central into the per-row fallback.
|
||||
`Rejected` is not a drop: the reconciliation pull still serves the row, so a
|
||||
transient refusal still lands, and a genuinely poison row is recorded centrally as
|
||||
a `ReconciliationAbandoned` row before its cursor retires it. A transport fault (no
|
||||
answer) is never counted as a refusal. A cached-lifecycle row whose tracking
|
||||
answer) is never counted as a refusal. Symmetrically, the reconciliation puller counts
|
||||
only non-transient failures toward its permanent-abandon threshold: a timeout,
|
||||
connection loss, deadlock or throttling error holds the cursor and retries without
|
||||
ever abandoning the row. A cached-lifecycle row whose tracking
|
||||
snapshot is gone past `CachedTrackingGraceSeconds` leaves the cached drain as
|
||||
`PushAbandoned` (it used to be marked `Forwarded`, which only worked because the
|
||||
pull re-served every `Forwarded` row). Cadence is short (default 5 s) when
|
||||
@@ -392,9 +402,23 @@ reports `MoreAvailable` and the cursor advances, mirroring `SiteCallAuditActor`.
|
||||
**Mixed versions:** a pre-PERF-1 site still serves `Forwarded` rows; central
|
||||
tolerates that unchanged (idempotent on `EventId`, merely more pages).
|
||||
|
||||
**Rollback caveat.** Rolling a site back to a pre-PERF-1 binary strands any rows already in
|
||||
`PushAbandoned` / `Rejected` on disk: the old code neither pushes, serves, retires nor purges
|
||||
a state it does not know, so they sit in the audit file (harmless, but never delivered and
|
||||
never purged) until the node is upgraded again — at which point the pull serves them
|
||||
normally. Nothing else in the file changes shape (the `audit_site_state` table is ignored by
|
||||
an old binary).
|
||||
|
||||
**Oversize rows (accepted, documented).** A single site audit row larger than the 32 MB gRPC
|
||||
ceiling (`GrpcMessageLimits.MaxMessageBytes`) cannot be pushed or pulled and would wedge the
|
||||
head of both paths. It is unreachable under the payload caps (8 KB / 64 KB on error by
|
||||
default) unless `DefaultCapBytes`/`ErrorCapBytes` are configured into the tens of megabytes;
|
||||
rows above the 2 MB batch budget but under the ceiling already travel alone.
|
||||
|
||||
**Per-site state follows the site list (review 2026-10-21 06/MEM-2).** Each tick drops
|
||||
the cursor, non-draining counter and stalled latch of any site no longer returned by
|
||||
`ISiteEnumerator` (a latched site publishes `Stalled=false` as it goes), and the
|
||||
`ISiteEnumerator` (a latched site publishes `Stalled=false` as it goes; an EMPTY enumeration
|
||||
prunes nothing, since it is as likely a transient lookup glitch), and the
|
||||
`PullAuditEvents` / `PullSiteCalls` / `PullTraceEvents` invokers evict and dispose any
|
||||
cached `(site, endpoint)` gRPC channel that is not a live site's primary or fallback
|
||||
address on every enumeration. The in-memory abandoned-`EventId` short-circuit set is
|
||||
@@ -476,16 +500,18 @@ carries a second, insertion-order bound:
|
||||
> cursor at all — central ACKED them over the telemetry push path, they are already
|
||||
> purge-eligible, and the pull never served them.
|
||||
|
||||
The bound is per-process, so after a site-node restart the first pull retires nothing
|
||||
and the pull after it resumes normal retirement — conservative in the
|
||||
safe direction (retirement is delayed, never a row lost). The gRPC handler's ordering
|
||||
The bound is persisted in a node-local `audit_site_state` key/value table in the audit
|
||||
file (loaded at start-up, clamped to the live `MAX(rowid)`), so a site-node restart no longer
|
||||
resets it — before that, a healthy site whose pull serves nothing could leave already-served
|
||||
`PushAbandoned`/`Rejected` rows owed (and counted on the backlog) indefinitely. It stays
|
||||
conservative in the safe direction (retirement can be delayed, never a row lost). The gRPC handler's ordering
|
||||
(retire, *then* read) is what keeps the bound from ever vouching for the batch it is
|
||||
about to serve.
|
||||
|
||||
**Liveness note (accepted, documented).** A row that was served but never covered by a
|
||||
later cursor — central reconciliation stopped for good, or the bound reset over a
|
||||
restart before the next cursor arrived — stays `Pending` indefinitely, and the site
|
||||
retention purge never purges `Pending`. That is the hard `ForwardState` invariant
|
||||
later cursor — central reconciliation stopped for good — stays owed (`Pending` /
|
||||
`PushAbandoned` / `Rejected`) indefinitely, and the site retention purge never purges
|
||||
an owed row. That is the hard `ForwardState` invariant
|
||||
working as intended (an unacknowledged row is not droppable), but it means the site
|
||||
store's floor depends on reconciliation actually running, not merely on the retention
|
||||
window elapsing. It is observable rather than silent: `GetBacklogStatsAsync` reports
|
||||
|
||||
@@ -0,0 +1,125 @@
|
||||
using System.Data.Common;
|
||||
using System.Net.Sockets;
|
||||
using Microsoft.Data.SqlClient;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.AuditLog.Central;
|
||||
|
||||
/// <summary>
|
||||
/// Classifies a central audit-insert failure (review 2026-10-21 06/IMPL-1 follow-up) so that a
|
||||
/// DATABASE problem — a maintenance window, a timeout, a dropped connection, an exhausted budget —
|
||||
/// is never mistaken for a verdict about the ROW. Two questions, two predicates:
|
||||
/// <list type="bullet">
|
||||
/// <item><see cref="IsPermanentRowRejection"/> — narrow: the row itself can never be inserted
|
||||
/// (string truncation, NULL into NOT NULL, a conversion / overflow, a constraint). Only these ids
|
||||
/// go back to the site as <c>rejected_event_ids</c>, which is what lets the site give a row up as
|
||||
/// <c>Rejected</c>.</item>
|
||||
/// <item><see cref="IsTransient"/> — broad: anything that may clear on its own. The reconciliation
|
||||
/// puller never counts these toward permanently abandoning a row.</item>
|
||||
/// </list>
|
||||
/// The exception's whole <see cref="Exception.InnerException"/> chain is inspected, because EF Core
|
||||
/// and the execution strategy wrap the provider exception.
|
||||
/// </summary>
|
||||
internal static class AuditIngestErrorClassifier
|
||||
{
|
||||
/// <summary>
|
||||
/// SQL Server errors that describe the ROW, not the server: 8152 / 2628 string or binary data
|
||||
/// would be truncated, 515 NULL into a NOT NULL column, 245 / 241 / 242 / 8114 / 8115 / 220 / 232
|
||||
/// conversion and arithmetic overflow, 547 CHECK / FOREIGN KEY constraint.
|
||||
/// </summary>
|
||||
private static readonly HashSet<int> PermanentRowErrorNumbers = new()
|
||||
{
|
||||
8152, 2628, 515, 245, 241, 242, 8114, 8115, 220, 232, 547,
|
||||
};
|
||||
|
||||
/// <summary>
|
||||
/// SQL Server errors that may clear on their own (timeout, connection loss, deadlock, lock
|
||||
/// timeout, resource / Azure throttling, database unavailable or in a restricted state).
|
||||
/// </summary>
|
||||
private static readonly HashSet<int> TransientErrorNumbers = new()
|
||||
{
|
||||
-2, -1, 0, 2, 20, 53, 64, 121, 233, 1205, 1222, 3960, 4060, 4221, 8645, 8651,
|
||||
10053, 10054, 10060, 10928, 10929, 40143, 40197, 40501, 40540, 40613,
|
||||
49918, 49919, 49920, 942, 952, 983, 976, 978, 3906, 9001, 9002,
|
||||
};
|
||||
|
||||
/// <summary>
|
||||
/// True when the failure is a SQL Server error that says the row itself can never be inserted.
|
||||
/// Everything else — including every transient error and every non-SQL exception — is false,
|
||||
/// so the site keeps such rows Pending and simply retries them.
|
||||
/// </summary>
|
||||
/// <param name="ex">The insert failure.</param>
|
||||
/// <returns>Whether the row should be reported back to the site as permanently rejected.</returns>
|
||||
public static bool IsPermanentRowRejection(Exception ex)
|
||||
{
|
||||
if (IsTransient(ex))
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
for (var e = ex; e is not null; e = e.InnerException)
|
||||
{
|
||||
if (e is SqlException sql)
|
||||
{
|
||||
foreach (SqlError error in sql.Errors)
|
||||
{
|
||||
if (PermanentRowErrorNumbers.Contains(error.Number))
|
||||
{
|
||||
return true;
|
||||
}
|
||||
}
|
||||
return PermanentRowErrorNumbers.Contains(sql.Number);
|
||||
}
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// True when any exception in the chain is a timeout, cancellation, socket / IO / connection
|
||||
/// failure, an EF retry-limit exhaustion, a provider error flagged transient, or a SQL Server
|
||||
/// error number in the transient set.
|
||||
/// </summary>
|
||||
/// <param name="ex">The insert failure.</param>
|
||||
/// <returns>Whether the failure may clear on its own.</returns>
|
||||
public static bool IsTransient(Exception ex)
|
||||
{
|
||||
for (var e = ex; e is not null; e = e.InnerException)
|
||||
{
|
||||
switch (e)
|
||||
{
|
||||
case TimeoutException:
|
||||
case OperationCanceledException:
|
||||
case SocketException:
|
||||
case IOException:
|
||||
case HttpRequestException:
|
||||
return true;
|
||||
case SqlException sql:
|
||||
if (TransientErrorNumbers.Contains(sql.Number))
|
||||
{
|
||||
return true;
|
||||
}
|
||||
foreach (SqlError error in sql.Errors)
|
||||
{
|
||||
if (TransientErrorNumbers.Contains(error.Number))
|
||||
{
|
||||
return true;
|
||||
}
|
||||
}
|
||||
if (sql.IsTransient)
|
||||
{
|
||||
return true;
|
||||
}
|
||||
break;
|
||||
case DbException db when db.IsTransient:
|
||||
return true;
|
||||
}
|
||||
|
||||
// EF Core's RetryLimitExceededException (an InvalidOperationException) wraps a
|
||||
// transient provider error once the execution strategy gives up.
|
||||
if (e.GetType().Name == "RetryLimitExceededException")
|
||||
{
|
||||
return true;
|
||||
}
|
||||
}
|
||||
return false;
|
||||
}
|
||||
}
|
||||
@@ -152,6 +152,7 @@ public class AuditLogIngestActor : ReceiveActor
|
||||
var replyTo = Sender;
|
||||
var nowUtc = DateTime.UtcNow;
|
||||
var accepted = new List<Guid>(cmd.Events.Count);
|
||||
var rejected = new List<Guid>();
|
||||
|
||||
// Resolve the repository for the whole batch — one DbContext per
|
||||
// message, mirroring NotificationOutboxActor.
|
||||
@@ -184,7 +185,7 @@ public class AuditLogIngestActor : ReceiveActor
|
||||
// unregistered (test composition roots) means the per-row catch
|
||||
// simply logs without surfacing on the health dashboard.
|
||||
var failureCounter = scope.ServiceProvider.GetService<ICentralAuditWriteFailureCounter>();
|
||||
await IngestWithRepositoryAsync(repository, redactor, failureCounter, cmd, nowUtc, accepted)
|
||||
await IngestWithRepositoryAsync(repository, redactor, failureCounter, cmd, nowUtc, accepted, rejected)
|
||||
.ConfigureAwait(false);
|
||||
}
|
||||
catch (Exception ex)
|
||||
@@ -201,7 +202,7 @@ public class AuditLogIngestActor : ReceiveActor
|
||||
accepted.Count);
|
||||
}
|
||||
|
||||
replyTo.Tell(new IngestAuditEventsReply(accepted));
|
||||
replyTo.Tell(new IngestAuditEventsReply(accepted) { RejectedEventIds = rejected });
|
||||
}
|
||||
|
||||
private async Task IngestWithRepositoryAsync(
|
||||
@@ -210,7 +211,8 @@ public class AuditLogIngestActor : ReceiveActor
|
||||
ICentralAuditWriteFailureCounter? failureCounter,
|
||||
IngestAuditEventsCommand cmd,
|
||||
DateTime nowUtc,
|
||||
List<Guid> accepted)
|
||||
List<Guid> accepted,
|
||||
List<Guid> rejected)
|
||||
{
|
||||
// Stamp the ingest timestamp here, not at the site. Redact BEFORE the
|
||||
// IngestedAtUtc stamp so the redacted copy carries the central-side
|
||||
@@ -303,6 +305,15 @@ public class AuditLogIngestActor : ReceiveActor
|
||||
catch { /* counter must never throw — defence in depth */ }
|
||||
}
|
||||
|
||||
// Review 2026-10-21 06/IMPL-1: tell the site ONLY about rows the server
|
||||
// refused for a reason intrinsic to the row (truncation, NOT NULL, conversion,
|
||||
// constraint). A timeout / connection loss / maintenance window leaves the id
|
||||
// in neither list, so the site retries it without counting a refusal.
|
||||
if (AuditIngestErrorClassifier.IsPermanentRowRejection(rowEx))
|
||||
{
|
||||
rejected.Add(cmd.Events[i].EventId);
|
||||
}
|
||||
|
||||
_logger.LogError(rowEx,
|
||||
"Failed to persist audit event {EventId} during batch ingest; row will be retried by the site.",
|
||||
cmd.Events[i].EventId);
|
||||
@@ -336,6 +347,7 @@ public class AuditLogIngestActor : ReceiveActor
|
||||
{
|
||||
var replyTo = Sender;
|
||||
var accepted = new List<Guid>(cmd.Entries.Count);
|
||||
var rejected = new List<Guid>();
|
||||
|
||||
try
|
||||
{
|
||||
@@ -370,7 +382,7 @@ public class AuditLogIngestActor : ReceiveActor
|
||||
strategy, dbContext, auditRepo, siteCallRepo, redactor, cmd, accepted, budget.Token)
|
||||
.ConfigureAwait(false))
|
||||
{
|
||||
replyTo.Tell(new IngestCachedTelemetryReply(accepted));
|
||||
replyTo.Tell(new IngestCachedTelemetryReply(accepted) { RejectedEventIds = rejected });
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -445,6 +457,11 @@ public class AuditLogIngestActor : ReceiveActor
|
||||
// sustained dual-write failure surfaces on the dashboard.
|
||||
try { failureCounter?.Increment(); }
|
||||
catch { /* counter must never throw — defence in depth */ }
|
||||
if (AuditIngestErrorClassifier.IsPermanentRowRejection(ex))
|
||||
{
|
||||
rejected.Add(entry.Audit.EventId);
|
||||
}
|
||||
|
||||
_logger.LogError(
|
||||
ex,
|
||||
"Combined telemetry dual-write failed for AuditEvent {EventId} / TrackedOperationId {TrackedOpId}; rolled back.",
|
||||
@@ -463,7 +480,7 @@ public class AuditLogIngestActor : ReceiveActor
|
||||
"Combined telemetry batch ingest failed before per-entry processing.");
|
||||
}
|
||||
|
||||
replyTo.Tell(new IngestCachedTelemetryReply(accepted));
|
||||
replyTo.Tell(new IngestCachedTelemetryReply(accepted) { RejectedEventIds = rejected });
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
|
||||
@@ -302,13 +302,15 @@ public class SiteAuditReconciliationActor : ReceiveActor
|
||||
return;
|
||||
}
|
||||
|
||||
PruneUnknownSites(sites, eventStream);
|
||||
|
||||
if (sites.Count == 0)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
// After the empty guard: an empty enumeration is as likely a transient lookup glitch
|
||||
// as a fleet with no sites, and pruning on it would throw away every cursor.
|
||||
PruneUnknownSites(sites, eventStream);
|
||||
|
||||
// Use CreateAsyncScope + await using so scoped EF Core
|
||||
// services (IAsyncDisposable DbContexts) dispose asynchronously
|
||||
// without blocking on sync Dispose() of pending connection cleanup.
|
||||
@@ -489,34 +491,51 @@ public class SiteAuditReconciliationActor : ReceiveActor
|
||||
// is permanently abandoned (LogCritical + cursor advances past)
|
||||
// to keep a truly broken row from blocking all subsequent
|
||||
// progress for the site.
|
||||
var attempts = _failedInsertAttempts.GetValueOrDefault(evt.EventId) + 1;
|
||||
_failedInsertAttempts[evt.EventId] = attempts;
|
||||
|
||||
if (attempts >= MaxPermanentInsertAttempts)
|
||||
if (AuditIngestErrorClassifier.IsTransient(ex))
|
||||
{
|
||||
_logger.LogCritical(
|
||||
// A database problem (timeout, connection loss, deadlock, throttling, a
|
||||
// maintenance window) is not a verdict about this row: hold the cursor
|
||||
// and retry, but never count it toward permanent abandonment — a long
|
||||
// outage must not turn healthy rows into ReconciliationAbandoned losses
|
||||
// (review 2026-10-21 06/PERF-1 adversarial follow-up).
|
||||
_logger.LogWarning(
|
||||
ex,
|
||||
"Permanently abandoning AuditEvent {EventId} from site {SiteId} after {Attempts} consecutive insert failures; cursor will advance past it.",
|
||||
"Reconciliation ingest of AuditEvent {EventId} from site {SiteId} hit a transient database fault; cursor held back, not counted toward abandonment.",
|
||||
evt.EventId,
|
||||
site.SiteId,
|
||||
attempts);
|
||||
_failedInsertAttempts.Remove(evt.EventId);
|
||||
RememberAbandoned(evt.EventId);
|
||||
advanceForThisRow = true;
|
||||
|
||||
await EmitAbandonmentRecordAsync(repository, evt, site.SiteId, ex, nowUtc)
|
||||
.ConfigureAwait(false);
|
||||
site.SiteId);
|
||||
hasUnresolvedFailure = true;
|
||||
}
|
||||
else
|
||||
{
|
||||
_logger.LogError(
|
||||
ex,
|
||||
"Reconciliation ingest failed for AuditEvent {EventId} from site {SiteId} (attempt {Attempts}/{Max}); cursor held back for retry.",
|
||||
evt.EventId,
|
||||
site.SiteId,
|
||||
attempts,
|
||||
MaxPermanentInsertAttempts);
|
||||
hasUnresolvedFailure = true;
|
||||
var attempts = _failedInsertAttempts.GetValueOrDefault(evt.EventId) + 1;
|
||||
_failedInsertAttempts[evt.EventId] = attempts;
|
||||
|
||||
if (attempts >= MaxPermanentInsertAttempts)
|
||||
{
|
||||
_logger.LogCritical(
|
||||
ex,
|
||||
"Permanently abandoning AuditEvent {EventId} from site {SiteId} after {Attempts} consecutive insert failures; cursor will advance past it.",
|
||||
evt.EventId,
|
||||
site.SiteId,
|
||||
attempts);
|
||||
_failedInsertAttempts.Remove(evt.EventId);
|
||||
RememberAbandoned(evt.EventId);
|
||||
advanceForThisRow = true;
|
||||
|
||||
await EmitAbandonmentRecordAsync(repository, evt, site.SiteId, ex, nowUtc)
|
||||
.ConfigureAwait(false);
|
||||
}
|
||||
else
|
||||
{
|
||||
_logger.LogError(
|
||||
ex,
|
||||
"Reconciliation ingest failed for AuditEvent {EventId} from site {SiteId} (attempt {Attempts}/{Max}); cursor held back for retry.",
|
||||
evt.EventId,
|
||||
site.SiteId,
|
||||
attempts,
|
||||
MaxPermanentInsertAttempts);
|
||||
hasUnresolvedFailure = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -25,7 +25,7 @@ public static class SiteAuditForwardStates
|
||||
public const string PushAbandoned = "PushAbandoned";
|
||||
|
||||
/// <summary>
|
||||
/// A row central answered but refused (not in <c>AcceptedEventIds</c>) on
|
||||
/// A row central explicitly and permanently refused (listed in the ack's <c>rejected_event_ids</c>) on
|
||||
/// <see cref="ZB.MOM.WW.ScadaBridge.AuditLog.Site.Telemetry.SiteAuditTelemetryOptions.MaxRejectedPushAttempts"/>
|
||||
/// consecutive drains. Moved out of the push queue so one poison row cannot wedge every push
|
||||
/// batch behind it; the pull then delivers it, or central's permanent-abandon path records the
|
||||
@@ -53,11 +53,11 @@ public interface ISiteAuditPushLifecycle
|
||||
Task MarkPushAbandonedAsync(IReadOnlyList<Guid> eventIds, CancellationToken ct = default);
|
||||
|
||||
/// <summary>
|
||||
/// Records one refused push attempt for each supplied <see cref="AuditForwardState.Pending"/>
|
||||
/// Records one explicitly-refused push attempt for each supplied <see cref="AuditForwardState.Pending"/>
|
||||
/// row (bumps <c>AttemptCount</c>, stamps <c>LastAttemptUtc</c>) and flips every row that has
|
||||
/// now reached <paramref name="maxAttempts"/> to <see cref="SiteAuditForwardStates.Rejected"/>.
|
||||
/// </summary>
|
||||
/// <param name="eventIds">Event IDs central answered for but did not accept.</param>
|
||||
/// <param name="eventIds">Event IDs central listed as permanently rejected.</param>
|
||||
/// <param name="maxAttempts">Refused attempts after which a row leaves the push queue.</param>
|
||||
/// <param name="ct">Cancellation token.</param>
|
||||
/// <returns>A task that resolves to the number of rows flipped to Rejected by this call.</returns>
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
using System.Diagnostics.Metrics;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Observability;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.AuditLog.Site;
|
||||
|
||||
/// <summary>
|
||||
/// Site audit forwarding instruments. Published on a meter named
|
||||
/// <see cref="ScadaBridgeTelemetry.MeterName"/> — OpenTelemetry matches meters by NAME, so these
|
||||
/// export through the existing <c>ObservedMeters</c> allowlist entry without a Host change.
|
||||
/// </summary>
|
||||
internal static class SiteAuditTelemetryMetrics
|
||||
{
|
||||
private static readonly Meter Meter = new(ScadaBridgeTelemetry.MeterName);
|
||||
|
||||
private static readonly Counter<long> RejectedRows =
|
||||
Meter.CreateCounter<long>("scadabridge.site.audit.rejected_rows", unit: "1",
|
||||
description: "Site audit rows central permanently refused MaxRejectedPushAttempts times, moved out of the push queue as Rejected (still served by the reconciliation pull).");
|
||||
|
||||
/// <summary>Records <paramref name="count"/> rows flipped to Rejected.</summary>
|
||||
/// <param name="count">Rows flipped.</param>
|
||||
public static void RecordRejectedRows(long count)
|
||||
{
|
||||
if (count > 0)
|
||||
{
|
||||
RejectedRows.Add(count);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -78,7 +78,6 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, ISiteAuditPushLi
|
||||
private readonly SqliteAuditWriterOptions _options;
|
||||
private readonly ILogger<SqliteAuditWriter> _logger;
|
||||
private readonly INodeIdentityProvider _nodeIdentity;
|
||||
private readonly IAuditWriteFailureCounter? _failureCounter;
|
||||
private readonly object _writeLock = new();
|
||||
private readonly Channel<PendingAuditEvent> _writeQueue;
|
||||
private readonly Task _writerLoop;
|
||||
@@ -89,17 +88,11 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, ISiteAuditPushLi
|
||||
/// <param name="logger">Logger instance.</param>
|
||||
/// <param name="nodeIdentity">Node identity provider.</param>
|
||||
/// <param name="connectionStringOverride">Optional connection string override.</param>
|
||||
/// <param name="failureCounter">
|
||||
/// Optional <c>SiteAuditWriteFailures</c> sink, bumped once per row the push gives up on as
|
||||
/// <see cref="SiteAuditForwardStates.Rejected"/> (review 2026-10-21 06/IMPL-1). Null on
|
||||
/// composition roots without Site Health Monitoring.
|
||||
/// </param>
|
||||
public SqliteAuditWriter(
|
||||
IOptions<SqliteAuditWriterOptions> options,
|
||||
ILogger<SqliteAuditWriter> logger,
|
||||
INodeIdentityProvider nodeIdentity,
|
||||
string? connectionStringOverride = null,
|
||||
IAuditWriteFailureCounter? failureCounter = null)
|
||||
string? connectionStringOverride = null)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
ArgumentNullException.ThrowIfNull(logger);
|
||||
@@ -108,7 +101,6 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, ISiteAuditPushLi
|
||||
_options = options.Value;
|
||||
_logger = logger;
|
||||
_nodeIdentity = nodeIdentity;
|
||||
_failureCounter = failureCounter;
|
||||
|
||||
var connectionString = connectionStringOverride
|
||||
?? $"Data Source={_options.DatabasePath};Cache=Shared";
|
||||
@@ -116,6 +108,7 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, ISiteAuditPushLi
|
||||
_connection.Open();
|
||||
|
||||
InitializeSchema();
|
||||
LoadMaxServedRowId();
|
||||
|
||||
// Open a second connection for read-only callers
|
||||
// (GetBacklogStatsAsync, ReadPendingAsync, ReadPendingSinceAsync,
|
||||
@@ -275,6 +268,16 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, ISiteAuditPushLi
|
||||
CREATE INDEX IF NOT EXISTS IX_fwd
|
||||
ON audit_forward_state (ForwardState, IsCachedKind, OccurredAtUtc);
|
||||
|
||||
-- Node-local key/value state that must survive a process restart. Holds the
|
||||
-- reconciliation pull's served-rows high-water mark ('max_served_rowid'), so a
|
||||
-- restart no longer resets the bound that lets a covering cursor retire rows the
|
||||
-- pull already delivered (review 2026-10-21 06/PERF-1 adversarial follow-up).
|
||||
-- Additive and idempotent: an older binary simply ignores the table.
|
||||
CREATE TABLE IF NOT EXISTS audit_site_state (
|
||||
Key TEXT NOT NULL PRIMARY KEY,
|
||||
Value INTEGER NOT NULL
|
||||
);
|
||||
|
||||
-- Flow Debugger (design 2026-09-05 §5.4): PullTraceEvents reads audit_event by
|
||||
-- time window + json_extract on ExecutionId/ParentExecutionId; this bounds the scan
|
||||
-- to the window. Additive and idempotent.
|
||||
@@ -345,6 +348,13 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, ISiteAuditPushLi
|
||||
// TryWrite can tell "disposed" from "full" without racing the channel's state.
|
||||
private bool _writerCompleted;
|
||||
|
||||
/// <summary>
|
||||
/// Rows moved to <see cref="SiteAuditForwardStates.Rejected"/> since construction (also exported
|
||||
/// as the <c>scadabridge.site.audit.rejected_rows</c> counter).
|
||||
/// </summary>
|
||||
public long RejectedPushRowCount => Interlocked.Read(ref _rejectedPushRows);
|
||||
private long _rejectedPushRows;
|
||||
|
||||
private async Task ProcessWriteQueueAsync()
|
||||
{
|
||||
var batch = new List<PendingAuditEvent>(_options.BatchSize);
|
||||
@@ -845,6 +855,8 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, ISiteAuditPushLi
|
||||
// pull's 256-rows-per-tick budget was spent on acked rows, so any site above
|
||||
// ~0.85 audit rows/s fell permanently behind and latched the stalled signal.
|
||||
// A Forwarded row is acked by central; the pull has nothing to repair there.
|
||||
IReadOnlyList<AuditEvent> rows;
|
||||
var before = Volatile.Read(ref _maxServedRowId);
|
||||
lock (_readLock)
|
||||
{
|
||||
ObjectDisposedException.ThrowIf(_disposed, this);
|
||||
@@ -884,8 +896,33 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, ISiteAuditPushLi
|
||||
}
|
||||
cmd.Parameters.AddWithValue("$limit", batchSize);
|
||||
|
||||
return Task.FromResult(ReadServedRows(cmd, batchSize));
|
||||
rows = ReadServedRows(cmd, batchSize);
|
||||
}
|
||||
|
||||
// Persist the (possibly advanced) mark OUTSIDE _readLock so the two locks are never
|
||||
// nested in read→write order. Only when it moved — at most one tiny upsert per pull.
|
||||
var mark = Volatile.Read(ref _maxServedRowId);
|
||||
if (mark > before)
|
||||
{
|
||||
try
|
||||
{
|
||||
lock (_writeLock)
|
||||
{
|
||||
if (!_disposed)
|
||||
{
|
||||
PersistMaxServedRowId(mark);
|
||||
}
|
||||
}
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex,
|
||||
"SqliteAuditWriter could not persist the served high-water mark {Mark}; a restart re-learns it from the next pull.",
|
||||
mark);
|
||||
}
|
||||
}
|
||||
|
||||
return Task.FromResult(rows);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
@@ -1005,11 +1042,11 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, ISiteAuditPushLi
|
||||
"of the push queue as Rejected; the reconciliation pull still serves them (central records " +
|
||||
"a ReconciliationAbandoned row if they can never be inserted).",
|
||||
flipped, threshold);
|
||||
for (var i = 0; i < flipped; i++)
|
||||
{
|
||||
try { _failureCounter?.Increment(); }
|
||||
catch { /* counter must never throw — defence in depth */ }
|
||||
}
|
||||
// Its OWN counter, not SiteAuditWriteFailures: the local write succeeded, and
|
||||
// the refusal is central's verdict on the row — blaming the site's writer for it
|
||||
// would be wrong (review 2026-10-21 adversarial follow-up).
|
||||
Interlocked.Add(ref _rejectedPushRows, flipped);
|
||||
SiteAuditTelemetryMetrics.RecordRejectedRows(flipped);
|
||||
}
|
||||
|
||||
return Task.FromResult(flipped);
|
||||
@@ -1154,6 +1191,49 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, ISiteAuditPushLi
|
||||
/// </summary>
|
||||
private long _maxServedRowId;
|
||||
|
||||
private const string MaxServedRowIdKey = "max_served_rowid";
|
||||
|
||||
/// <summary>
|
||||
/// Restores <see cref="_maxServedRowId"/> from <c>audit_site_state</c>, clamped to the live
|
||||
/// <c>MAX(rowid)</c> so a stale value can never vouch for a reused rowid. Before this the
|
||||
/// bound was per-process: after a site-node restart it was 0, and because a healthy site's
|
||||
/// pull now often serves nothing, owed rows central had already ingested (e.g. PushAbandoned)
|
||||
/// could stay unretired — and counted on the backlog — indefinitely.
|
||||
/// </summary>
|
||||
private void LoadMaxServedRowId()
|
||||
{
|
||||
using var cmd = _connection.CreateCommand();
|
||||
cmd.CommandText = """
|
||||
SELECT MIN(
|
||||
IFNULL((SELECT Value FROM audit_site_state WHERE Key = $key), 0),
|
||||
IFNULL((SELECT MAX(rowid) FROM audit_forward_state), 0));
|
||||
""";
|
||||
cmd.Parameters.AddWithValue("$key", MaxServedRowIdKey);
|
||||
var value = Convert.ToInt64(cmd.ExecuteScalar(), System.Globalization.CultureInfo.InvariantCulture);
|
||||
Volatile.Write(ref _maxServedRowId, value);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Durably records the served high-water mark. Runs on the WRITE connection under
|
||||
/// <c>_writeLock</c> (never nested inside <c>_readLock</c>). Best-effort: a failure leaves
|
||||
/// the in-memory bound intact and only means a restart re-learns it from the next pull.
|
||||
/// </summary>
|
||||
private void PersistMaxServedRowId(long value, SqliteTransaction? transaction = null)
|
||||
{
|
||||
using var cmd = _connection.CreateCommand();
|
||||
cmd.Transaction = transaction;
|
||||
cmd.CommandText = """
|
||||
INSERT INTO audit_site_state (Key, Value) VALUES ($key, $value)
|
||||
ON CONFLICT(Key) DO UPDATE SET Value = excluded.Value;
|
||||
""";
|
||||
cmd.Parameters.AddWithValue("$key", MaxServedRowIdKey);
|
||||
cmd.Parameters.AddWithValue("$value", value);
|
||||
cmd.ExecuteNonQuery();
|
||||
}
|
||||
|
||||
/// <summary>Test seam: the current in-memory served high-water mark.</summary>
|
||||
internal long MaxServedRowIdForTests => Volatile.Read(ref _maxServedRowId);
|
||||
|
||||
/// <summary>
|
||||
/// Executes a reconciliation-pull read whose projection carries <c>fs.rowid</c> as an
|
||||
/// 11th column, materialising the canonical rows while advancing
|
||||
@@ -1222,12 +1302,12 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, ISiteAuditPushLi
|
||||
// acked them over the push path, they are already purge-eligible, and flipping
|
||||
// them was one UPDATE per pushed row for no benefit (review 2026-10-21 06/PERF-1).
|
||||
//
|
||||
// Bound state is per-process: after a restart it is 0 until this instance serves
|
||||
// a batch, so the first pull retires nothing and the pull after it
|
||||
// resumes normal retirement. Conservative in the safe direction (delays
|
||||
// retirement, never loses a row); the liveness note in MarkReconciledUpToAsync's
|
||||
// interface doc + Component-AuditLog.md covers what a permanently halted
|
||||
// reconciliation means for such rows.
|
||||
// The bound is persisted in audit_site_state (loaded at construction, clamped to
|
||||
// the live MAX(rowid)), so a site-node restart no longer resets it to 0 and strands
|
||||
// rows the pull had already served. Conservative in the safe direction (it can
|
||||
// only delay retirement, never lose a row); the liveness note in
|
||||
// MarkReconciledUpToAsync's interface doc + Component-AuditLog.md covers what a
|
||||
// permanently halted reconciliation means for such rows.
|
||||
using var cmd = _connection.CreateCommand();
|
||||
cmd.CommandText = $"""
|
||||
UPDATE audit_forward_state AS fs
|
||||
@@ -1506,7 +1586,10 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, ISiteAuditPushLi
|
||||
var liveMax = Convert.ToInt64(maxCmd.ExecuteScalar(),
|
||||
System.Globalization.CultureInfo.InvariantCulture);
|
||||
if (liveMax < Volatile.Read(ref _maxServedRowId))
|
||||
{
|
||||
Volatile.Write(ref _maxServedRowId, liveMax);
|
||||
PersistMaxServedRowId(liveMax, transaction);
|
||||
}
|
||||
}
|
||||
|
||||
transaction.Commit();
|
||||
|
||||
@@ -481,28 +481,30 @@ public class SiteAuditTelemetryActor : ReceiveActor
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Records a refused push attempt for every row that was SENT in an answered batch but is
|
||||
/// missing from the ack's <c>AcceptedEventIds</c>; rows refused
|
||||
/// <see cref="SiteAuditTelemetryOptions.MaxRejectedPushAttempts"/> times leave the push
|
||||
/// queue as <see cref="SiteAuditForwardStates.Rejected"/> (review 2026-10-21 06/IMPL-1).
|
||||
/// Records a refused push attempt for every SENT row central explicitly listed in the ack's
|
||||
/// <c>RejectedEventIds</c> — a permanent, row-intrinsic insert failure. Rows refused
|
||||
/// <see cref="SiteAuditTelemetryOptions.MaxRejectedPushAttempts"/> times leave the push queue as
|
||||
/// <see cref="SiteAuditForwardStates.Rejected"/> (review 2026-10-21 06/IMPL-1).
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Called only after the push RETURNED — a transport fault is not a verdict about any row
|
||||
/// and never counts. Best-effort: a bookkeeping failure leaves the rows Pending, exactly as
|
||||
/// before this existed.
|
||||
/// "Not accepted" is NOT a refusal: central answers with an empty or partial ack during a SQL
|
||||
/// maintenance window, a timeout, or an exhausted ingest budget, and counting those would flip
|
||||
/// healthy rows to Rejected. Only the explicit list counts; an older central never sends it, so
|
||||
/// against it this degrades to the pre-IMPL-1 behaviour (rows stay Pending). Best-effort: a
|
||||
/// bookkeeping failure leaves the rows Pending.
|
||||
/// </remarks>
|
||||
private async Task RecordRefusalsAsync(IReadOnlyList<string> sentIds, IngestAck ack, CancellationToken ct)
|
||||
{
|
||||
if (_queue is not ISiteAuditPushLifecycle lifecycle || sentIds.Count == 0)
|
||||
if (_queue is not ISiteAuditPushLifecycle lifecycle || sentIds.Count == 0 || ack.RejectedEventIds.Count == 0)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
var accepted = new HashSet<string>(ack.AcceptedEventIds, StringComparer.OrdinalIgnoreCase);
|
||||
var sent = new HashSet<string>(sentIds, StringComparer.OrdinalIgnoreCase);
|
||||
var refused = new List<Guid>();
|
||||
foreach (var raw in sentIds)
|
||||
foreach (var raw in ack.RejectedEventIds)
|
||||
{
|
||||
if (!accepted.Contains(raw) && Guid.TryParse(raw, out var id))
|
||||
if (sent.Contains(raw) && Guid.TryParse(raw, out var id))
|
||||
{
|
||||
refused.Add(id);
|
||||
}
|
||||
|
||||
@@ -57,18 +57,22 @@ public sealed class SiteAuditTelemetryOptions
|
||||
public int CachedTrackingGraceSeconds { get; set; } = 300;
|
||||
|
||||
/// <summary>
|
||||
/// Number of drains in which central ANSWERED a push but refused a row (the row is
|
||||
/// missing from <c>AcceptedEventIds</c>) after which the row leaves the push queue as
|
||||
/// <c>Rejected</c>. Default: 10 (≈ 50 s at the busy cadence).
|
||||
/// Number of drains in which central EXPLICITLY refused a row (listed it in the ack's
|
||||
/// additive <c>rejected_event_ids</c> — a permanent, row-intrinsic insert failure such as an
|
||||
/// over-width value) after which the row leaves the push queue as <c>Rejected</c>.
|
||||
/// Default: 10 (≈ 50 s at the busy cadence).
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The push reads oldest-first, so a row central can never insert (e.g. an over-width
|
||||
/// column) used to sit at the head of every batch forever, failing each set-based insert
|
||||
/// at central and dragging the batch into the per-row fallback (review 2026-10-21
|
||||
/// 06/IMPL-1). A transport fault (no answer) never counts. <c>Rejected</c> is NOT a drop:
|
||||
/// 06/IMPL-1). A transport fault, and a row merely ABSENT from the accepted list (a database
|
||||
/// outage, a timeout, an exhausted ingest budget), never counts; an older central that does not
|
||||
/// send the field therefore never triggers it. <c>Rejected</c> is NOT a drop:
|
||||
/// the reconciliation pull still serves the row, so a transient central refusal still
|
||||
/// lands, and a genuinely poison row is recorded centrally as <c>ReconciliationAbandoned</c>
|
||||
/// before it retires. Each flip bumps <c>SiteAuditWriteFailures</c> on the site health report.
|
||||
/// before it retires. Each flip is counted on <c>scadabridge.site.audit.rejected_rows</c> (NOT
|
||||
/// <c>SiteAuditWriteFailures</c> — the site's own write succeeded).
|
||||
/// </remarks>
|
||||
public int MaxRejectedPushAttempts { get; set; } = 10;
|
||||
}
|
||||
|
||||
@@ -82,7 +82,7 @@ public sealed class SiteCommunicationAuditClient : ISiteStreamAuditClient
|
||||
.Ask<IngestAuditEventsReply>(new IngestAuditEventsCommand(events), _askTimeout, ct)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
return ToAck(reply.AcceptedEventIds);
|
||||
return ToAck(reply.AcceptedEventIds, reply.RejectedEventIds);
|
||||
}
|
||||
|
||||
/// <inheritdoc/>
|
||||
@@ -105,16 +105,21 @@ public sealed class SiteCommunicationAuditClient : ISiteStreamAuditClient
|
||||
.Ask<IngestCachedTelemetryReply>(new IngestCachedTelemetryCommand(entries), _askTimeout, ct)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
return ToAck(reply.AcceptedEventIds);
|
||||
return ToAck(reply.AcceptedEventIds, reply.RejectedEventIds);
|
||||
}
|
||||
|
||||
private static IngestAck ToAck(IReadOnlyList<Guid> acceptedEventIds)
|
||||
private static IngestAck ToAck(IReadOnlyList<Guid> acceptedEventIds, IReadOnlyList<Guid> rejectedEventIds)
|
||||
{
|
||||
var ack = new IngestAck();
|
||||
foreach (var id in acceptedEventIds)
|
||||
{
|
||||
ack.AcceptedEventIds.Add(id.ToString());
|
||||
}
|
||||
// Explicit permanent refusals only (review 2026-10-21 06/IMPL-1); empty from an older central.
|
||||
foreach (var id in rejectedEventIds)
|
||||
{
|
||||
ack.RejectedEventIds.Add(id.ToString());
|
||||
}
|
||||
return ack;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -8,4 +8,13 @@ namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.Audit;
|
||||
/// The gRPC handler echoes these ids back over the wire as the <c>IngestAck</c>
|
||||
/// the site uses to flip rows to <c>Forwarded</c>.
|
||||
/// </summary>
|
||||
public sealed record IngestAuditEventsReply(IReadOnlyList<Guid> AcceptedEventIds);
|
||||
public sealed record IngestAuditEventsReply(IReadOnlyList<Guid> AcceptedEventIds)
|
||||
{
|
||||
/// <summary>
|
||||
/// Ids central PERMANENTLY refused — a non-transient per-row insert failure (e.g. an
|
||||
/// over-width value), never a database outage or an exhausted budget. Additive (review
|
||||
/// 2026-10-21 06/IMPL-1): the site counts only these toward giving a row up as
|
||||
/// <c>Rejected</c>; empty from an older central, which therefore never triggers it.
|
||||
/// </summary>
|
||||
public IReadOnlyList<Guid> RejectedEventIds { get; init; } = Array.Empty<Guid>();
|
||||
}
|
||||
|
||||
@@ -7,4 +7,13 @@ namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.Audit;
|
||||
/// UPSERT) committed; entries whose transaction rolled back are absent so the
|
||||
/// site can leave the row Pending and retry on the next drain.
|
||||
/// </summary>
|
||||
public sealed record IngestCachedTelemetryReply(IReadOnlyList<Guid> AcceptedEventIds);
|
||||
public sealed record IngestCachedTelemetryReply(IReadOnlyList<Guid> AcceptedEventIds)
|
||||
{
|
||||
/// <summary>
|
||||
/// Ids central PERMANENTLY refused — a non-transient per-row insert failure (e.g. an
|
||||
/// over-width value), never a database outage or an exhausted budget. Additive (review
|
||||
/// 2026-10-21 06/IMPL-1): the site counts only these toward giving a row up as
|
||||
/// <c>Rejected</c>; empty from an older central, which therefore never triggers it.
|
||||
/// </summary>
|
||||
public IReadOnlyList<Guid> RejectedEventIds { get; init; } = Array.Empty<Guid>();
|
||||
}
|
||||
|
||||
@@ -314,7 +314,9 @@ public static class CentralControlDtoMapper
|
||||
/// </summary>
|
||||
/// <param name="acceptedEventIds">Ids central considers durably persisted.</param>
|
||||
/// <returns>The wire ack carrying the ids in "D" string form, in order.</returns>
|
||||
public static IngestAck ToIngestAck(IReadOnlyList<Guid> acceptedEventIds)
|
||||
/// <param name="rejectedEventIds">Ids central permanently refused (additive field 2); null = none.</param>
|
||||
public static IngestAck ToIngestAck(
|
||||
IReadOnlyList<Guid> acceptedEventIds, IReadOnlyList<Guid>? rejectedEventIds = null)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(acceptedEventIds);
|
||||
|
||||
@@ -324,9 +326,39 @@ public static class CentralControlDtoMapper
|
||||
ack.AcceptedEventIds.Add(id.ToString());
|
||||
}
|
||||
|
||||
if (rejectedEventIds is not null)
|
||||
{
|
||||
foreach (var id in rejectedEventIds)
|
||||
{
|
||||
ack.RejectedEventIds.Add(id.ToString());
|
||||
}
|
||||
}
|
||||
|
||||
return ack;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Reads the permanently-rejected id list (additive field 2) out of an <see cref="IngestAck"/>;
|
||||
/// empty from an older central. Unparseable entries are skipped.
|
||||
/// </summary>
|
||||
/// <param name="ack">The wire ack to read.</param>
|
||||
/// <returns>The rejected event ids, in wire order.</returns>
|
||||
public static IReadOnlyList<Guid> RejectedFromIngestAck(IngestAck ack)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(ack);
|
||||
|
||||
var ids = new List<Guid>(ack.RejectedEventIds.Count);
|
||||
foreach (var id in ack.RejectedEventIds)
|
||||
{
|
||||
if (Guid.TryParse(id, out var g))
|
||||
{
|
||||
ids.Add(g);
|
||||
}
|
||||
}
|
||||
|
||||
return ids;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Reads the accepted-id list back out of an <see cref="IngestAck"/>.
|
||||
/// </summary>
|
||||
|
||||
@@ -198,7 +198,7 @@ public sealed class CentralControlGrpcService : CentralControlService.CentralCon
|
||||
SiteStreamGrpcServer.AuditIngestAskTimeout,
|
||||
context).ConfigureAwait(false);
|
||||
|
||||
return CentralControlDtoMapper.ToIngestAck(reply.AcceptedEventIds);
|
||||
return CentralControlDtoMapper.ToIngestAck(reply.AcceptedEventIds, reply.RejectedEventIds);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
@@ -223,7 +223,7 @@ public sealed class CentralControlGrpcService : CentralControlService.CentralCon
|
||||
SiteStreamGrpcServer.AuditIngestAskTimeout,
|
||||
context).ConfigureAwait(false);
|
||||
|
||||
return CentralControlDtoMapper.ToIngestAck(reply.AcceptedEventIds);
|
||||
return CentralControlDtoMapper.ToIngestAck(reply.AcceptedEventIds, reply.RejectedEventIds);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
|
||||
@@ -88,7 +88,10 @@ public sealed class GrpcCentralTransport : ICentralTransport
|
||||
var batch = CentralControlDtoMapper.ToDto(message);
|
||||
Dispatch(replyTo, SiteStreamGrpcServer.AuditIngestAskTimeout,
|
||||
(c, o) => c.IngestAuditEventsAsync(batch, o),
|
||||
ack => new IngestAuditEventsReply(CentralControlDtoMapper.FromIngestAck(ack)), retrySafe: true);
|
||||
ack => new IngestAuditEventsReply(CentralControlDtoMapper.FromIngestAck(ack))
|
||||
{
|
||||
RejectedEventIds = CentralControlDtoMapper.RejectedFromIngestAck(ack),
|
||||
}, retrySafe: true);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
@@ -97,7 +100,10 @@ public sealed class GrpcCentralTransport : ICentralTransport
|
||||
var batch = CentralControlDtoMapper.ToDto(message);
|
||||
Dispatch(replyTo, SiteStreamGrpcServer.AuditIngestAskTimeout,
|
||||
(c, o) => c.IngestCachedTelemetryAsync(batch, o),
|
||||
ack => new IngestCachedTelemetryReply(CentralControlDtoMapper.FromIngestAck(ack)), retrySafe: true);
|
||||
ack => new IngestCachedTelemetryReply(CentralControlDtoMapper.FromIngestAck(ack))
|
||||
{
|
||||
RejectedEventIds = CentralControlDtoMapper.RejectedFromIngestAck(ack),
|
||||
}, retrySafe: true);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
|
||||
@@ -172,7 +172,15 @@ message AuditEventDto {
|
||||
}
|
||||
|
||||
message AuditEventBatch { repeated AuditEventDto events = 1; }
|
||||
message IngestAck { repeated string accepted_event_ids = 1; }
|
||||
// rejected_event_ids (additive, field 2 — review 2026-10-21 06/IMPL-1): ids central
|
||||
// PERMANENTLY refused (a non-transient per-row SQL error such as an over-width
|
||||
// value). Only these count toward the site's MaxRejectedPushAttempts; ids that are
|
||||
// neither accepted nor rejected (DB outage, budget exhaustion) are simply retried.
|
||||
// An older central never sets it, so the site never counts a refusal.
|
||||
message IngestAck {
|
||||
repeated string accepted_event_ids = 1;
|
||||
repeated string rejected_event_ids = 2;
|
||||
}
|
||||
|
||||
// Audit Log (#23) M3 cached-call combined telemetry: a single packet carries
|
||||
// both the AuditEvent row to insert and the SiteCalls operational-state upsert
|
||||
|
||||
@@ -71,57 +71,58 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
"b2FkX3RydW5jYXRlZBgSIAEoCBINCgVleHRyYRgTIAEoCRIUCgxleGVjdXRp",
|
||||
"b25faWQYFCABKAkSGwoTcGFyZW50X2V4ZWN1dGlvbl9pZBgVIAEoCRITCgtz",
|
||||
"b3VyY2Vfbm9kZRgWIAEoCSI8Cg9BdWRpdEV2ZW50QmF0Y2gSKQoGZXZlbnRz",
|
||||
"GAEgAygLMhkuc2l0ZXN0cmVhbS5BdWRpdEV2ZW50RHRvIicKCUluZ2VzdEFj",
|
||||
"axIaChJhY2NlcHRlZF9ldmVudF9pZHMYASADKAkiiQMKFlNpdGVDYWxsT3Bl",
|
||||
"cmF0aW9uYWxEdG8SHAoUdHJhY2tlZF9vcGVyYXRpb25faWQYASABKAkSDwoH",
|
||||
"Y2hhbm5lbBgCIAEoCRIOCgZ0YXJnZXQYAyABKAkSEwoLc291cmNlX3NpdGUY",
|
||||
"BCABKAkSDgoGc3RhdHVzGAUgASgJEhMKC3JldHJ5X2NvdW50GAYgASgFEhIK",
|
||||
"Cmxhc3RfZXJyb3IYByABKAkSMAoLaHR0cF9zdGF0dXMYCCABKAsyGy5nb29n",
|
||||
"bGUucHJvdG9idWYuSW50MzJWYWx1ZRIyCg5jcmVhdGVkX2F0X3V0YxgJIAEo",
|
||||
"CzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXASMgoOdXBkYXRlZF9hdF91",
|
||||
"dGMYCiABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wEjMKD3Rlcm1p",
|
||||
"bmFsX2F0X3V0YxgLIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXAS",
|
||||
"EwoLc291cmNlX25vZGUYDCABKAkigAEKFUNhY2hlZFRlbGVtZXRyeVBhY2tl",
|
||||
"dBIuCgthdWRpdF9ldmVudBgBIAEoCzIZLnNpdGVzdHJlYW0uQXVkaXRFdmVu",
|
||||
"dER0bxI3CgtvcGVyYXRpb25hbBgCIAEoCzIiLnNpdGVzdHJlYW0uU2l0ZUNh",
|
||||
"bGxPcGVyYXRpb25hbER0byJKChRDYWNoZWRUZWxlbWV0cnlCYXRjaBIyCgdw",
|
||||
"YWNrZXRzGAEgAygLMiEuc2l0ZXN0cmVhbS5DYWNoZWRUZWxlbWV0cnlQYWNr",
|
||||
"ZXQibQoWUHVsbEF1ZGl0RXZlbnRzUmVxdWVzdBItCglzaW5jZV91dGMYASAB",
|
||||
"KAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wEhIKCmJhdGNoX3NpemUY",
|
||||
"AiABKAUSEAoIYWZ0ZXJfaWQYAyABKAkiXAoXUHVsbEF1ZGl0RXZlbnRzUmVz",
|
||||
"cG9uc2USKQoGZXZlbnRzGAEgAygLMhkuc2l0ZXN0cmVhbS5BdWRpdEV2ZW50",
|
||||
"RHRvEhYKDm1vcmVfYXZhaWxhYmxlGAIgASgIIoQBChZQdWxsVHJhY2VFdmVu",
|
||||
"dHNSZXF1ZXN0EhAKCGFmdGVyX2lkGAQgASgJEhUKDWV4ZWN1dGlvbl9pZHMY",
|
||||
"ASADKAkSLQoJc2luY2VfdXRjGAIgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRp",
|
||||
"bWVzdGFtcBISCgptYXhfZXZlbnRzGAMgASgFIkQKF1B1bGxUcmFjZUV2ZW50",
|
||||
"c1Jlc3BvbnNlEikKBmV2ZW50cxgBIAMoCzIZLnNpdGVzdHJlYW0uQXVkaXRF",
|
||||
"dmVudER0byJrChRQdWxsU2l0ZUNhbGxzUmVxdWVzdBItCglzaW5jZV91dGMY",
|
||||
"ASABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wEhIKCmJhdGNoX3Np",
|
||||
"emUYAiABKAUSEAoIYWZ0ZXJfaWQYAyABKAkiaQoVUHVsbFNpdGVDYWxsc1Jl",
|
||||
"c3BvbnNlEjgKDG9wZXJhdGlvbmFscxgBIAMoCzIiLnNpdGVzdHJlYW0uU2l0",
|
||||
"ZUNhbGxPcGVyYXRpb25hbER0bxIWCg5tb3JlX2F2YWlsYWJsZRgCIAEoCCpc",
|
||||
"CgdRdWFsaXR5EhcKE1FVQUxJVFlfVU5TUEVDSUZJRUQQABIQCgxRVUFMSVRZ",
|
||||
"X0dPT0QQARIVChFRVUFMSVRZX1VOQ0VSVEFJThACEg8KC1FVQUxJVFlfQkFE",
|
||||
"EAMqXQoOQWxhcm1TdGF0ZUVudW0SGwoXQUxBUk1fU1RBVEVfVU5TUEVDSUZJ",
|
||||
"RUQQABIWChJBTEFSTV9TVEFURV9OT1JNQUwQARIWChJBTEFSTV9TVEFURV9B",
|
||||
"Q1RJVkUQAiqFAQoOQWxhcm1MZXZlbEVudW0SFAoQQUxBUk1fTEVWRUxfTk9O",
|
||||
"RRAAEhMKD0FMQVJNX0xFVkVMX0xPVxABEhcKE0FMQVJNX0xFVkVMX0xPV19M",
|
||||
"T1cQAhIUChBBTEFSTV9MRVZFTF9ISUdIEAMSGQoVQUxBUk1fTEVWRUxfSElH",
|
||||
"SF9ISUdIEAQy4gQKEVNpdGVTdHJlYW1TZXJ2aWNlElUKEVN1YnNjcmliZUlu",
|
||||
"c3RhbmNlEiEuc2l0ZXN0cmVhbS5JbnN0YW5jZVN0cmVhbVJlcXVlc3QaGy5z",
|
||||
"aXRlc3RyZWFtLlNpdGVTdHJlYW1FdmVudDABEk0KDVN1YnNjcmliZVNpdGUS",
|
||||
"HS5zaXRlc3RyZWFtLlNpdGVTdHJlYW1SZXF1ZXN0Ghsuc2l0ZXN0cmVhbS5T",
|
||||
"aXRlU3RyZWFtRXZlbnQwARJHChFJbmdlc3RBdWRpdEV2ZW50cxIbLnNpdGVz",
|
||||
"dHJlYW0uQXVkaXRFdmVudEJhdGNoGhUuc2l0ZXN0cmVhbS5Jbmdlc3RBY2sS",
|
||||
"UAoVSW5nZXN0Q2FjaGVkVGVsZW1ldHJ5EiAuc2l0ZXN0cmVhbS5DYWNoZWRU",
|
||||
"ZWxlbWV0cnlCYXRjaBoVLnNpdGVzdHJlYW0uSW5nZXN0QWNrEloKD1B1bGxB",
|
||||
"dWRpdEV2ZW50cxIiLnNpdGVzdHJlYW0uUHVsbEF1ZGl0RXZlbnRzUmVxdWVz",
|
||||
"dBojLnNpdGVzdHJlYW0uUHVsbEF1ZGl0RXZlbnRzUmVzcG9uc2USWgoPUHVs",
|
||||
"bFRyYWNlRXZlbnRzEiIuc2l0ZXN0cmVhbS5QdWxsVHJhY2VFdmVudHNSZXF1",
|
||||
"ZXN0GiMuc2l0ZXN0cmVhbS5QdWxsVHJhY2VFdmVudHNSZXNwb25zZRJUCg1Q",
|
||||
"dWxsU2l0ZUNhbGxzEiAuc2l0ZXN0cmVhbS5QdWxsU2l0ZUNhbGxzUmVxdWVz",
|
||||
"dBohLnNpdGVzdHJlYW0uUHVsbFNpdGVDYWxsc1Jlc3BvbnNlQiuqAihaQi5N",
|
||||
"T00uV1cuU2NhZGFCcmlkZ2UuQ29tbXVuaWNhdGlvbi5HcnBjYgZwcm90bzM="));
|
||||
"GAEgAygLMhkuc2l0ZXN0cmVhbS5BdWRpdEV2ZW50RHRvIkMKCUluZ2VzdEFj",
|
||||
"axIaChJhY2NlcHRlZF9ldmVudF9pZHMYASADKAkSGgoScmVqZWN0ZWRfZXZl",
|
||||
"bnRfaWRzGAIgAygJIokDChZTaXRlQ2FsbE9wZXJhdGlvbmFsRHRvEhwKFHRy",
|
||||
"YWNrZWRfb3BlcmF0aW9uX2lkGAEgASgJEg8KB2NoYW5uZWwYAiABKAkSDgoG",
|
||||
"dGFyZ2V0GAMgASgJEhMKC3NvdXJjZV9zaXRlGAQgASgJEg4KBnN0YXR1cxgF",
|
||||
"IAEoCRITCgtyZXRyeV9jb3VudBgGIAEoBRISCgpsYXN0X2Vycm9yGAcgASgJ",
|
||||
"EjAKC2h0dHBfc3RhdHVzGAggASgLMhsuZ29vZ2xlLnByb3RvYnVmLkludDMy",
|
||||
"VmFsdWUSMgoOY3JlYXRlZF9hdF91dGMYCSABKAsyGi5nb29nbGUucHJvdG9i",
|
||||
"dWYuVGltZXN0YW1wEjIKDnVwZGF0ZWRfYXRfdXRjGAogASgLMhouZ29vZ2xl",
|
||||
"LnByb3RvYnVmLlRpbWVzdGFtcBIzCg90ZXJtaW5hbF9hdF91dGMYCyABKAsy",
|
||||
"Gi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wEhMKC3NvdXJjZV9ub2RlGAwg",
|
||||
"ASgJIoABChVDYWNoZWRUZWxlbWV0cnlQYWNrZXQSLgoLYXVkaXRfZXZlbnQY",
|
||||
"ASABKAsyGS5zaXRlc3RyZWFtLkF1ZGl0RXZlbnREdG8SNwoLb3BlcmF0aW9u",
|
||||
"YWwYAiABKAsyIi5zaXRlc3RyZWFtLlNpdGVDYWxsT3BlcmF0aW9uYWxEdG8i",
|
||||
"SgoUQ2FjaGVkVGVsZW1ldHJ5QmF0Y2gSMgoHcGFja2V0cxgBIAMoCzIhLnNp",
|
||||
"dGVzdHJlYW0uQ2FjaGVkVGVsZW1ldHJ5UGFja2V0Im0KFlB1bGxBdWRpdEV2",
|
||||
"ZW50c1JlcXVlc3QSLQoJc2luY2VfdXRjGAEgASgLMhouZ29vZ2xlLnByb3Rv",
|
||||
"YnVmLlRpbWVzdGFtcBISCgpiYXRjaF9zaXplGAIgASgFEhAKCGFmdGVyX2lk",
|
||||
"GAMgASgJIlwKF1B1bGxBdWRpdEV2ZW50c1Jlc3BvbnNlEikKBmV2ZW50cxgB",
|
||||
"IAMoCzIZLnNpdGVzdHJlYW0uQXVkaXRFdmVudER0bxIWCg5tb3JlX2F2YWls",
|
||||
"YWJsZRgCIAEoCCKEAQoWUHVsbFRyYWNlRXZlbnRzUmVxdWVzdBIQCghhZnRl",
|
||||
"cl9pZBgEIAEoCRIVCg1leGVjdXRpb25faWRzGAEgAygJEi0KCXNpbmNlX3V0",
|
||||
"YxgCIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXASEgoKbWF4X2V2",
|
||||
"ZW50cxgDIAEoBSJEChdQdWxsVHJhY2VFdmVudHNSZXNwb25zZRIpCgZldmVu",
|
||||
"dHMYASADKAsyGS5zaXRlc3RyZWFtLkF1ZGl0RXZlbnREdG8iawoUUHVsbFNp",
|
||||
"dGVDYWxsc1JlcXVlc3QSLQoJc2luY2VfdXRjGAEgASgLMhouZ29vZ2xlLnBy",
|
||||
"b3RvYnVmLlRpbWVzdGFtcBISCgpiYXRjaF9zaXplGAIgASgFEhAKCGFmdGVy",
|
||||
"X2lkGAMgASgJImkKFVB1bGxTaXRlQ2FsbHNSZXNwb25zZRI4CgxvcGVyYXRp",
|
||||
"b25hbHMYASADKAsyIi5zaXRlc3RyZWFtLlNpdGVDYWxsT3BlcmF0aW9uYWxE",
|
||||
"dG8SFgoObW9yZV9hdmFpbGFibGUYAiABKAgqXAoHUXVhbGl0eRIXChNRVUFM",
|
||||
"SVRZX1VOU1BFQ0lGSUVEEAASEAoMUVVBTElUWV9HT09EEAESFQoRUVVBTElU",
|
||||
"WV9VTkNFUlRBSU4QAhIPCgtRVUFMSVRZX0JBRBADKl0KDkFsYXJtU3RhdGVF",
|
||||
"bnVtEhsKF0FMQVJNX1NUQVRFX1VOU1BFQ0lGSUVEEAASFgoSQUxBUk1fU1RB",
|
||||
"VEVfTk9STUFMEAESFgoSQUxBUk1fU1RBVEVfQUNUSVZFEAIqhQEKDkFsYXJt",
|
||||
"TGV2ZWxFbnVtEhQKEEFMQVJNX0xFVkVMX05PTkUQABITCg9BTEFSTV9MRVZF",
|
||||
"TF9MT1cQARIXChNBTEFSTV9MRVZFTF9MT1dfTE9XEAISFAoQQUxBUk1fTEVW",
|
||||
"RUxfSElHSBADEhkKFUFMQVJNX0xFVkVMX0hJR0hfSElHSBAEMuIEChFTaXRl",
|
||||
"U3RyZWFtU2VydmljZRJVChFTdWJzY3JpYmVJbnN0YW5jZRIhLnNpdGVzdHJl",
|
||||
"YW0uSW5zdGFuY2VTdHJlYW1SZXF1ZXN0Ghsuc2l0ZXN0cmVhbS5TaXRlU3Ry",
|
||||
"ZWFtRXZlbnQwARJNCg1TdWJzY3JpYmVTaXRlEh0uc2l0ZXN0cmVhbS5TaXRl",
|
||||
"U3RyZWFtUmVxdWVzdBobLnNpdGVzdHJlYW0uU2l0ZVN0cmVhbUV2ZW50MAES",
|
||||
"RwoRSW5nZXN0QXVkaXRFdmVudHMSGy5zaXRlc3RyZWFtLkF1ZGl0RXZlbnRC",
|
||||
"YXRjaBoVLnNpdGVzdHJlYW0uSW5nZXN0QWNrElAKFUluZ2VzdENhY2hlZFRl",
|
||||
"bGVtZXRyeRIgLnNpdGVzdHJlYW0uQ2FjaGVkVGVsZW1ldHJ5QmF0Y2gaFS5z",
|
||||
"aXRlc3RyZWFtLkluZ2VzdEFjaxJaCg9QdWxsQXVkaXRFdmVudHMSIi5zaXRl",
|
||||
"c3RyZWFtLlB1bGxBdWRpdEV2ZW50c1JlcXVlc3QaIy5zaXRlc3RyZWFtLlB1",
|
||||
"bGxBdWRpdEV2ZW50c1Jlc3BvbnNlEloKD1B1bGxUcmFjZUV2ZW50cxIiLnNp",
|
||||
"dGVzdHJlYW0uUHVsbFRyYWNlRXZlbnRzUmVxdWVzdBojLnNpdGVzdHJlYW0u",
|
||||
"UHVsbFRyYWNlRXZlbnRzUmVzcG9uc2USVAoNUHVsbFNpdGVDYWxscxIgLnNp",
|
||||
"dGVzdHJlYW0uUHVsbFNpdGVDYWxsc1JlcXVlc3QaIS5zaXRlc3RyZWFtLlB1",
|
||||
"bGxTaXRlQ2FsbHNSZXNwb25zZUIrqgIoWkIuTU9NLldXLlNjYWRhQnJpZGdl",
|
||||
"LkNvbW11bmljYXRpb24uR3JwY2IGcHJvdG8z"));
|
||||
descriptor = pbr::FileDescriptor.FromGeneratedCode(descriptorData,
|
||||
new pbr::FileDescriptor[] { global::Google.Protobuf.WellKnownTypes.TimestampReflection.Descriptor, global::Google.Protobuf.WellKnownTypes.WrappersReflection.Descriptor, },
|
||||
new pbr::GeneratedClrTypeInfo(new[] {typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.Quality), typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AlarmStateEnum), typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AlarmLevelEnum), }, null, new pbr::GeneratedClrTypeInfo[] {
|
||||
@@ -133,7 +134,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AlarmStateUpdate), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AlarmStateUpdate.Parser, new[]{ "InstanceUniqueName", "AlarmName", "State", "Priority", "Timestamp", "Level", "Message", "Kind", "Active", "Acknowledged", "Confirmed", "ShelveState", "Suppressed", "SourceReference", "AlarmTypeName", "Category", "OperatorUser", "OperatorComment", "OriginalRaiseTime", "CurrentValue", "LimitValue", "NativeSourceCanonicalName", "IsConfiguredPlaceholder", "AckTime" }, null, null, null, null),
|
||||
new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AuditEventDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AuditEventDto.Parser, new[]{ "EventId", "OccurredAtUtc", "Channel", "Kind", "CorrelationId", "SourceSiteId", "SourceInstanceId", "SourceScript", "Actor", "Target", "Status", "HttpStatus", "DurationMs", "ErrorMessage", "ErrorDetail", "RequestSummary", "ResponseSummary", "PayloadTruncated", "Extra", "ExecutionId", "ParentExecutionId", "SourceNode" }, null, null, null, null),
|
||||
new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AuditEventBatch), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AuditEventBatch.Parser, new[]{ "Events" }, null, null, null, null),
|
||||
new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck.Parser, new[]{ "AcceptedEventIds" }, null, null, null, null),
|
||||
new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck.Parser, new[]{ "AcceptedEventIds", "RejectedEventIds" }, null, null, null, null),
|
||||
new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteCallOperationalDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteCallOperationalDto.Parser, new[]{ "TrackedOperationId", "Channel", "Target", "SourceSite", "Status", "RetryCount", "LastError", "HttpStatus", "CreatedAtUtc", "UpdatedAtUtc", "TerminalAtUtc", "SourceNode" }, null, null, null, null),
|
||||
new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CachedTelemetryPacket), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CachedTelemetryPacket.Parser, new[]{ "AuditEvent", "Operational" }, null, null, null, null),
|
||||
new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CachedTelemetryBatch), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CachedTelemetryBatch.Parser, new[]{ "Packets" }, null, null, null, null),
|
||||
@@ -4085,6 +4086,13 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// rejected_event_ids (additive, field 2 — review 2026-10-21 06/IMPL-1): ids central
|
||||
/// PERMANENTLY refused (a non-transient per-row SQL error such as an over-width
|
||||
/// value). Only these count toward the site's MaxRejectedPushAttempts; ids that are
|
||||
/// neither accepted nor rejected (DB outage, budget exhaustion) are simply retried.
|
||||
/// An older central never sets it, so the site never counts a refusal.
|
||||
/// </summary>
|
||||
[global::System.Diagnostics.DebuggerDisplayAttribute("{ToString(),nq}")]
|
||||
public sealed partial class IngestAck : pb::IMessage<IngestAck>
|
||||
#if !GOOGLE_PROTOBUF_REFSTRUCT_COMPATIBILITY_MODE
|
||||
@@ -4121,6 +4129,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
|
||||
public IngestAck(IngestAck other) : this() {
|
||||
acceptedEventIds_ = other.acceptedEventIds_.Clone();
|
||||
rejectedEventIds_ = other.rejectedEventIds_.Clone();
|
||||
_unknownFields = pb::UnknownFieldSet.Clone(other._unknownFields);
|
||||
}
|
||||
|
||||
@@ -4141,6 +4150,17 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
get { return acceptedEventIds_; }
|
||||
}
|
||||
|
||||
/// <summary>Field number for the "rejected_event_ids" field.</summary>
|
||||
public const int RejectedEventIdsFieldNumber = 2;
|
||||
private static readonly pb::FieldCodec<string> _repeated_rejectedEventIds_codec
|
||||
= pb::FieldCodec.ForString(18);
|
||||
private readonly pbc::RepeatedField<string> rejectedEventIds_ = new pbc::RepeatedField<string>();
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
|
||||
public pbc::RepeatedField<string> RejectedEventIds {
|
||||
get { return rejectedEventIds_; }
|
||||
}
|
||||
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
|
||||
public override bool Equals(object other) {
|
||||
@@ -4157,6 +4177,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
return true;
|
||||
}
|
||||
if(!acceptedEventIds_.Equals(other.acceptedEventIds_)) return false;
|
||||
if(!rejectedEventIds_.Equals(other.rejectedEventIds_)) return false;
|
||||
return Equals(_unknownFields, other._unknownFields);
|
||||
}
|
||||
|
||||
@@ -4165,6 +4186,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
public override int GetHashCode() {
|
||||
int hash = 1;
|
||||
hash ^= acceptedEventIds_.GetHashCode();
|
||||
hash ^= rejectedEventIds_.GetHashCode();
|
||||
if (_unknownFields != null) {
|
||||
hash ^= _unknownFields.GetHashCode();
|
||||
}
|
||||
@@ -4184,6 +4206,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
output.WriteRawMessage(this);
|
||||
#else
|
||||
acceptedEventIds_.WriteTo(output, _repeated_acceptedEventIds_codec);
|
||||
rejectedEventIds_.WriteTo(output, _repeated_rejectedEventIds_codec);
|
||||
if (_unknownFields != null) {
|
||||
_unknownFields.WriteTo(output);
|
||||
}
|
||||
@@ -4195,6 +4218,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
|
||||
void pb::IBufferMessage.InternalWriteTo(ref pb::WriteContext output) {
|
||||
acceptedEventIds_.WriteTo(ref output, _repeated_acceptedEventIds_codec);
|
||||
rejectedEventIds_.WriteTo(ref output, _repeated_rejectedEventIds_codec);
|
||||
if (_unknownFields != null) {
|
||||
_unknownFields.WriteTo(ref output);
|
||||
}
|
||||
@@ -4206,6 +4230,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
public int CalculateSize() {
|
||||
int size = 0;
|
||||
size += acceptedEventIds_.CalculateSize(_repeated_acceptedEventIds_codec);
|
||||
size += rejectedEventIds_.CalculateSize(_repeated_rejectedEventIds_codec);
|
||||
if (_unknownFields != null) {
|
||||
size += _unknownFields.CalculateSize();
|
||||
}
|
||||
@@ -4219,6 +4244,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
return;
|
||||
}
|
||||
acceptedEventIds_.Add(other.acceptedEventIds_);
|
||||
rejectedEventIds_.Add(other.rejectedEventIds_);
|
||||
_unknownFields = pb::UnknownFieldSet.MergeFrom(_unknownFields, other._unknownFields);
|
||||
}
|
||||
|
||||
@@ -4242,6 +4268,10 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
acceptedEventIds_.AddEntriesFrom(input, _repeated_acceptedEventIds_codec);
|
||||
break;
|
||||
}
|
||||
case 18: {
|
||||
rejectedEventIds_.AddEntriesFrom(input, _repeated_rejectedEventIds_codec);
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
#endif
|
||||
@@ -4265,6 +4295,10 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
acceptedEventIds_.AddEntriesFrom(ref input, _repeated_acceptedEventIds_codec);
|
||||
break;
|
||||
}
|
||||
case 18: {
|
||||
rejectedEventIds_.AddEntriesFrom(ref input, _repeated_rejectedEventIds_codec);
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,132 @@
|
||||
using Akka.Actor;
|
||||
using Akka.TestKit.Xunit2;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using NSubstitute;
|
||||
using NSubstitute.ExceptionExtensions;
|
||||
using ZB.MOM.WW.Audit;
|
||||
using ZB.MOM.WW.ScadaBridge.AuditLog.Central;
|
||||
using ZB.MOM.WW.ScadaBridge.AuditLog.Tests.TestSupport;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Audit;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Audit;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.AuditLog.Tests.Central;
|
||||
|
||||
/// <summary>
|
||||
/// Review 2026-10-21 06/IMPL-1 adversarial follow-up: central reports a row as REJECTED only for a
|
||||
/// permanent, row-intrinsic SQL failure; a database outage / timeout leaves the id in neither list,
|
||||
/// so the site never mistakes a maintenance window for a poison row. Also pins the classifier the
|
||||
/// reconciliation puller uses to decide what counts toward permanent abandonment (06/COR follow-up 3).
|
||||
/// </summary>
|
||||
public class AuditIngestRejectionTests : TestKit
|
||||
{
|
||||
private static AuditEvent NewEvent() => ScadaBridgeAuditEventFactory.Create(
|
||||
eventId: Guid.NewGuid(),
|
||||
occurredAtUtc: new DateTime(2026, 5, 20, 10, 0, 0, DateTimeKind.Utc),
|
||||
channel: AuditChannel.ApiOutbound,
|
||||
kind: AuditKind.ApiCall,
|
||||
status: AuditStatus.Delivered,
|
||||
sourceSiteId: "site-a");
|
||||
|
||||
/// <summary>The batch reply lists only the over-width row as rejected; the timed-out row is in neither list.</summary>
|
||||
[Fact]
|
||||
public void Ingest_ReportsOnlyPermanentRowFailuresAsRejected()
|
||||
{
|
||||
var ok = NewEvent();
|
||||
var overWidth = NewEvent();
|
||||
var timedOut = NewEvent();
|
||||
var repo = Substitute.For<IAuditLogRepository>();
|
||||
repo.InsertManyIfNotExistsAsync(Arg.Any<IReadOnlyList<AuditEvent>>(), Arg.Any<TimeSpan?>(), Arg.Any<CancellationToken>())
|
||||
.ThrowsAsync(SqlExceptionFactory.Create(2628));
|
||||
repo.InsertIfNotExistsAsync(Arg.Is<AuditEvent>(e => e.EventId == overWidth.EventId), Arg.Any<CancellationToken>())
|
||||
.ThrowsAsync(SqlExceptionFactory.Create(2628));
|
||||
repo.InsertIfNotExistsAsync(Arg.Is<AuditEvent>(e => e.EventId == timedOut.EventId), Arg.Any<CancellationToken>())
|
||||
.ThrowsAsync(SqlExceptionFactory.Create(-2));
|
||||
|
||||
var actor = Sys.ActorOf(Props.Create(() => new AuditLogIngestActor(
|
||||
StubAuditRepositoryProvider.Build(repo), NullLogger<AuditLogIngestActor>.Instance)));
|
||||
actor.Tell(new IngestAuditEventsCommand(new[] { ok, overWidth, timedOut }), TestActor);
|
||||
|
||||
var reply = ExpectMsg<IngestAuditEventsReply>(TimeSpan.FromSeconds(10));
|
||||
Assert.Equal(new[] { ok.EventId }, reply.AcceptedEventIds);
|
||||
Assert.Equal(new[] { overWidth.EventId }, reply.RejectedEventIds);
|
||||
}
|
||||
|
||||
/// <summary>Classifier: row-intrinsic SQL errors are permanent; timeouts, connection loss and non-SQL faults are not.</summary>
|
||||
[Fact]
|
||||
public void Classifier_SeparatesRowVerdictsFromDatabaseProblems()
|
||||
{
|
||||
Assert.True(AuditIngestErrorClassifier.IsPermanentRowRejection(SqlExceptionFactory.Create(2628)));
|
||||
Assert.True(AuditIngestErrorClassifier.IsPermanentRowRejection(SqlExceptionFactory.Create(515)));
|
||||
Assert.True(AuditIngestErrorClassifier.IsPermanentRowRejection(
|
||||
new InvalidOperationException("wrapped", SqlExceptionFactory.Create(8152))));
|
||||
Assert.False(AuditIngestErrorClassifier.IsPermanentRowRejection(SqlExceptionFactory.Create(-2)));
|
||||
Assert.False(AuditIngestErrorClassifier.IsPermanentRowRejection(new TimeoutException()));
|
||||
Assert.False(AuditIngestErrorClassifier.IsPermanentRowRejection(new InvalidOperationException("bug")));
|
||||
|
||||
Assert.True(AuditIngestErrorClassifier.IsTransient(SqlExceptionFactory.Create(-2)));
|
||||
Assert.True(AuditIngestErrorClassifier.IsTransient(SqlExceptionFactory.Create(1205)));
|
||||
Assert.True(AuditIngestErrorClassifier.IsTransient(SqlExceptionFactory.Create(40613)));
|
||||
Assert.True(AuditIngestErrorClassifier.IsTransient(new TimeoutException()));
|
||||
Assert.True(AuditIngestErrorClassifier.IsTransient(new TaskCanceledException()));
|
||||
Assert.True(AuditIngestErrorClassifier.IsTransient(
|
||||
new InvalidOperationException("wrapped", new System.Net.Sockets.SocketException())));
|
||||
Assert.False(AuditIngestErrorClassifier.IsTransient(SqlExceptionFactory.Create(2628)));
|
||||
Assert.False(AuditIngestErrorClassifier.IsTransient(new InvalidOperationException("bug")));
|
||||
}
|
||||
|
||||
private sealed class OneRowClient : IPullAuditEventsClient
|
||||
{
|
||||
private readonly AuditEvent _row;
|
||||
public OneRowClient(AuditEvent row) => _row = row;
|
||||
public System.Collections.Concurrent.ConcurrentQueue<DateTime> Since { get; } = new();
|
||||
public Task<ZB.MOM.WW.ScadaBridge.Commons.Messages.Integration.PullAuditEventsResponse> PullAsync(
|
||||
string siteId, DateTime sinceUtc, string? afterId, int batchSize, CancellationToken ct)
|
||||
{
|
||||
Since.Enqueue(sinceUtc);
|
||||
return Task.FromResult(new ZB.MOM.WW.ScadaBridge.Commons.Messages.Integration.PullAuditEventsResponse(
|
||||
new[] { _row }, MoreAvailable: false));
|
||||
}
|
||||
}
|
||||
|
||||
private sealed class OneSite : ISiteEnumerator
|
||||
{
|
||||
public Task<IReadOnlyList<SiteEntry>> EnumerateAsync(CancellationToken ct = default) =>
|
||||
Task.FromResult<IReadOnlyList<SiteEntry>>(new[] { new SiteEntry("site-a", "http://a:8083") });
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A row that keeps hitting a TRANSIENT database fault is retried with the cursor held back for
|
||||
/// far longer than the permanent-abandon threshold, and is never abandoned.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void Reconciliation_TransientFaults_NeverCountTowardAbandonment()
|
||||
{
|
||||
var row = NewEvent();
|
||||
var repo = Substitute.For<IAuditLogRepository>();
|
||||
repo.InsertManyIfNotExistsAsync(Arg.Any<IReadOnlyList<AuditEvent>>(), Arg.Any<TimeSpan?>(), Arg.Any<CancellationToken>())
|
||||
.ThrowsAsync(SqlExceptionFactory.Create(-2));
|
||||
repo.InsertIfNotExistsAsync(Arg.Any<AuditEvent>(), Arg.Any<CancellationToken>())
|
||||
.ThrowsAsync(SqlExceptionFactory.Create(-2));
|
||||
var client = new OneRowClient(row);
|
||||
|
||||
var actor = Sys.ActorOf(Props.Create(() => new SiteAuditReconciliationActor(
|
||||
new OneSite(), client, StubAuditRepositoryProvider.Build(repo),
|
||||
Microsoft.Extensions.Options.Options.Create(new SiteAuditReconciliationOptions
|
||||
{
|
||||
ReconciliationIntervalOverride = TimeSpan.FromMilliseconds(30),
|
||||
}),
|
||||
NullLogger<SiteAuditReconciliationActor>.Instance)));
|
||||
|
||||
AwaitCondition(() => client.Since.Count >= 12, TimeSpan.FromSeconds(10));
|
||||
Watch(actor);
|
||||
Sys.Stop(actor);
|
||||
ExpectTerminated(actor);
|
||||
|
||||
Assert.All(client.Since, s => Assert.Equal(DateTime.MinValue, s)); // cursor held every tick
|
||||
repo.DidNotReceive().InsertIfNotExistsAsync(
|
||||
Arg.Is<AuditEvent>(e => e.Action.EndsWith(".ReconciliationAbandoned", StringComparison.Ordinal)),
|
||||
Arg.Any<CancellationToken>());
|
||||
}
|
||||
}
|
||||
+4
-2
@@ -116,9 +116,11 @@ public class ReconciliationStatePruningTests : TestKit
|
||||
|
||||
Assert.True(probe.ExpectMsg<SiteAuditTelemetryStalledChanged>(TimeSpan.FromSeconds(5)).Stalled);
|
||||
|
||||
sites.Sites = Array.Empty<SiteEntry>();
|
||||
// Not an empty list: an empty enumeration deliberately prunes nothing.
|
||||
sites.Sites = new[] { new SiteEntry("survivor", "http://s:8083") };
|
||||
|
||||
var cleared = probe.ExpectMsg<SiteAuditTelemetryStalledChanged>(TimeSpan.FromSeconds(5));
|
||||
var cleared = probe.FishForMessage<SiteAuditTelemetryStalledChanged>(
|
||||
m => m.SiteId == "doomed", TimeSpan.FromSeconds(5));
|
||||
Assert.Equal("doomed", cleared.SiteId);
|
||||
Assert.False(cleared.Stalled);
|
||||
|
||||
|
||||
+32
-8
@@ -18,7 +18,6 @@ namespace ZB.MOM.WW.ScadaBridge.AuditLog.Tests.Site;
|
||||
public class SqliteAuditWriterForwardStateTests : IDisposable
|
||||
{
|
||||
private readonly string _dbPath = Path.Combine(Path.GetTempPath(), $"audit-fwdstate-{Guid.NewGuid():N}.db");
|
||||
private readonly CountingFailureCounter _counter = new();
|
||||
|
||||
/// <summary>Deletes the temporary database.</summary>
|
||||
public void Dispose()
|
||||
@@ -32,8 +31,7 @@ public class SqliteAuditWriterForwardStateTests : IDisposable
|
||||
private SqliteAuditWriter CreateWriter() => new(
|
||||
Options.Create(new SqliteAuditWriterOptions { DatabasePath = _dbPath }),
|
||||
NullLogger<SqliteAuditWriter>.Instance,
|
||||
new FakeNodeIdentityProvider(),
|
||||
failureCounter: _counter);
|
||||
new FakeNodeIdentityProvider());
|
||||
|
||||
private static async Task<Guid> WriteAsync(SqliteAuditWriter writer, DateTime occurredAtUtc)
|
||||
{
|
||||
@@ -132,13 +130,39 @@ public class SqliteAuditWriterForwardStateTests : IDisposable
|
||||
Assert.Equal(behind, Assert.Single(await writer.ReadPendingAsync(1)).EventId);
|
||||
Assert.Contains(await writer.ReadPendingSinceAsync(DateTime.MinValue, 100), e => e.EventId == poison);
|
||||
Assert.Equal(0, await writer.PurgeExpiredAsync(DateTime.UtcNow.AddDays(1)));
|
||||
Assert.Equal(1, _counter.Count);
|
||||
Assert.Equal(1, writer.RejectedPushRowCount);
|
||||
}
|
||||
|
||||
private sealed class CountingFailureCounter : IAuditWriteFailureCounter
|
||||
/// <summary>
|
||||
/// The served high-water mark survives a writer restart: rows served before the restart are
|
||||
/// retired by the first cursor that covers them after it, instead of being stranded owed
|
||||
/// (and counted on the backlog) until something happens to re-serve them.
|
||||
/// </summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[Fact]
|
||||
public async Task ServedHighWaterMark_SurvivesRestart_SoTheNextCursorStillRetires()
|
||||
{
|
||||
private int _count;
|
||||
public int Count => Volatile.Read(ref _count);
|
||||
public void Increment() => Interlocked.Increment(ref _count);
|
||||
var t = DateTime.UtcNow.AddMinutes(-5);
|
||||
Guid abandoned;
|
||||
IReadOnlyList<AuditEvent> served;
|
||||
await using (var writer = CreateWriter())
|
||||
{
|
||||
abandoned = await WriteAsync(writer, t);
|
||||
await writer.MarkPushAbandonedAsync(new[] { abandoned });
|
||||
served = await writer.ReadPendingSinceAsync(DateTime.MinValue, 100);
|
||||
Assert.Single(served);
|
||||
}
|
||||
|
||||
await using (var restarted = CreateWriter())
|
||||
{
|
||||
Assert.True(restarted.MaxServedRowIdForTests > 0);
|
||||
|
||||
var flipped = await restarted.MarkReconciledUpToAsync(
|
||||
served[0].OccurredAtUtc.UtcDateTime, served[0].EventId.ToString());
|
||||
|
||||
Assert.Equal(1, flipped);
|
||||
Assert.Equal("Reconciled", restarted.GetForwardStateForTests(abandoned)!.Value.State);
|
||||
Assert.Equal(0, (await restarted.GetBacklogStatsAsync()).PendingCount);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
+18
-13
@@ -785,27 +785,33 @@ public class SiteAuditTelemetryActorTests : TestKit
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Review 2026-10-21 06/IMPL-1: rows central ANSWERED for but did not accept get a refused
|
||||
/// attempt recorded (so a poison head row eventually leaves the push queue); accepted rows
|
||||
/// do not, and a transport fault records nothing.
|
||||
/// Review 2026-10-21 06/IMPL-1 (adversarial follow-up): only rows central EXPLICITLY lists in
|
||||
/// <c>RejectedEventIds</c> get a refused attempt recorded. A row merely missing from the accepted
|
||||
/// list (central DB outage / budget exhaustion) and a transport fault record nothing.
|
||||
/// </summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[Fact]
|
||||
public async Task Drain_RefusedRows_RecordRejectedAttempts_TransportFaultDoesNot()
|
||||
public async Task Drain_OnlyExplicitlyRejectedRows_RecordRefusals()
|
||||
{
|
||||
var queue = Substitute.For<ISiteAuditQueue, ZB.MOM.WW.ScadaBridge.AuditLog.Site.ISiteAuditPushLifecycle>();
|
||||
var lifecycle = (ZB.MOM.WW.ScadaBridge.AuditLog.Site.ISiteAuditPushLifecycle)queue;
|
||||
var accepted = NewEvent();
|
||||
var refused = NewEvent();
|
||||
var poison = NewEvent();
|
||||
var unanswered = NewEvent();
|
||||
var rows = new[] { accepted, poison, unanswered };
|
||||
queue.ReadPendingAsync(Arg.Any<int>(), Arg.Any<CancellationToken>())
|
||||
.Returns(
|
||||
Task.FromResult<IReadOnlyList<AuditEvent>>(new[] { accepted, refused }),
|
||||
Task.FromResult<IReadOnlyList<AuditEvent>>(new[] { accepted, refused }),
|
||||
Task.FromResult<IReadOnlyList<AuditEvent>>(rows),
|
||||
Task.FromResult<IReadOnlyList<AuditEvent>>(rows),
|
||||
Task.FromResult<IReadOnlyList<AuditEvent>>(rows),
|
||||
Task.FromResult<IReadOnlyList<AuditEvent>>(Array.Empty<AuditEvent>()));
|
||||
var explicitAck = AckAll(new[] { accepted });
|
||||
explicitAck.RejectedEventIds.Add(poison.EventId.ToString());
|
||||
_client.IngestAuditEventsAsync(Arg.Any<AuditEventBatch>(), Arg.Any<CancellationToken>())
|
||||
.Returns(
|
||||
_ => Task.FromException<IngestAck>(new InvalidOperationException("transport down")),
|
||||
_ => Task.FromResult(AckAll(new[] { accepted })));
|
||||
_ => Task.FromResult(AckAll(new[] { accepted })), // partial ack, no verdict (old central / DB outage)
|
||||
_ => Task.FromResult(explicitAck));
|
||||
|
||||
Sys.ActorOf(Props.Create(() => new SiteAuditTelemetryActor(
|
||||
queue, _client, Opts(), NullLogger<SiteAuditTelemetryActor>.Instance,
|
||||
@@ -813,13 +819,12 @@ public class SiteAuditTelemetryActorTests : TestKit
|
||||
|
||||
await AwaitAssertAsync(async () =>
|
||||
await lifecycle.Received(1).RecordRejectedPushAttemptsAsync(
|
||||
Arg.Is<IReadOnlyList<Guid>>(g => g.Count == 1 && g[0] == refused.EventId),
|
||||
Arg.Is<IReadOnlyList<Guid>>(g => g.Count == 1 && g[0] == poison.EventId),
|
||||
10,
|
||||
Arg.Any<CancellationToken>()),
|
||||
TimeSpan.FromSeconds(10));
|
||||
await queue.Received(1).MarkForwardedAsync(
|
||||
Arg.Is<IReadOnlyList<Guid>>(g => g.Count == 1 && g[0] == accepted.EventId),
|
||||
Arg.Any<CancellationToken>());
|
||||
TimeSpan.FromSeconds(15));
|
||||
await lifecycle.Received(1).RecordRejectedPushAttemptsAsync(
|
||||
Arg.Any<IReadOnlyList<Guid>>(), Arg.Any<int>(), Arg.Any<CancellationToken>());
|
||||
}
|
||||
|
||||
private sealed class CapturingLogger<T> : Microsoft.Extensions.Logging.ILogger<T>
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
using System.Reflection;
|
||||
using Microsoft.Data.SqlClient;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.AuditLog.Tests.TestSupport;
|
||||
|
||||
/// <summary>
|
||||
/// Fabricates a <see cref="SqlException"/> with a given error number via the driver's internal
|
||||
/// <c>CreateException</c> factory (the type has no public constructor) — same technique as
|
||||
/// ExternalSystemGateway.Tests' <c>FabricateSqlException</c>.
|
||||
/// </summary>
|
||||
internal static class SqlExceptionFactory
|
||||
{
|
||||
/// <summary>Creates a <see cref="SqlException"/> carrying one error with <paramref name="number"/>.</summary>
|
||||
/// <param name="number">The SQL Server error number.</param>
|
||||
/// <param name="message">The error message.</param>
|
||||
/// <returns>The fabricated exception.</returns>
|
||||
public static SqlException Create(int number, string message = "simulated")
|
||||
{
|
||||
var errorCtor = typeof(SqlError).GetConstructors(BindingFlags.Instance | BindingFlags.NonPublic)
|
||||
.First(c => c.GetParameters().Length == 8 && c.GetParameters()[7].ParameterType == typeof(Exception));
|
||||
var sqlError = (SqlError)errorCtor.Invoke(
|
||||
new object?[] { number, (byte)0, (byte)0, "server", message, "procedure", 0, null });
|
||||
|
||||
var collection = (SqlErrorCollection)typeof(SqlErrorCollection)
|
||||
.GetConstructors(BindingFlags.Instance | BindingFlags.NonPublic).First()
|
||||
.Invoke(Array.Empty<object?>());
|
||||
typeof(SqlErrorCollection).GetMethod("Add", BindingFlags.Instance | BindingFlags.NonPublic)!
|
||||
.Invoke(collection, new object?[] { sqlError });
|
||||
|
||||
var create = typeof(SqlException).GetMethod("CreateException",
|
||||
BindingFlags.Static | BindingFlags.NonPublic,
|
||||
new[] { typeof(SqlErrorCollection), typeof(string) })!;
|
||||
return (SqlException)create.Invoke(null, new object?[] { collection, "6.0.0" })!;
|
||||
}
|
||||
}
|
||||
@@ -259,6 +259,23 @@ public class CentralControlDtoMapperTests
|
||||
Assert.Equal(ids, roundTripped);
|
||||
}
|
||||
|
||||
/// <summary>Verifies the additive rejected-id list round-trips and reads empty when absent.</summary>
|
||||
[Fact]
|
||||
public void IngestAck_RejectedIds_RoundTrip_AndAbsentFieldReadsEmpty()
|
||||
{
|
||||
// Review 2026-10-21 06/IMPL-1: additive field 2. An older central never sets it, so the
|
||||
// site must read an EMPTY list (never a refusal) from a pre-field ack.
|
||||
var accepted = new[] { Guid.NewGuid() };
|
||||
var rejected = new[] { Guid.NewGuid(), Guid.NewGuid() };
|
||||
|
||||
var wire = Wire(CentralControlDtoMapper.ToIngestAck(accepted, rejected));
|
||||
|
||||
Assert.Equal(accepted, CentralControlDtoMapper.FromIngestAck(wire));
|
||||
Assert.Equal(rejected, CentralControlDtoMapper.RejectedFromIngestAck(wire));
|
||||
Assert.Empty(CentralControlDtoMapper.RejectedFromIngestAck(
|
||||
Wire(CentralControlDtoMapper.ToIngestAck(accepted))));
|
||||
}
|
||||
|
||||
/// <summary>Verifies that an ingest acknowledgement with nothing accepted stays empty.</summary>
|
||||
[Fact]
|
||||
public void IngestAck_RoundTrip_NothingAccepted_StaysEmpty()
|
||||
|
||||
Reference in New Issue
Block a user