ScadaBridge/code-reviews/SiteCallAudit/findings.md

Code Review — SiteCallAudit

Field	Value
Module	src/ZB.MOM.WW.ScadaBridge.SiteCallAudit
Design doc	docs/requirements/Component-SiteCallAudit.md
Status	Reviewed
Last reviewed	2026-06-20
Reviewer	claude-agent
Commit reviewed	4307c381
Open findings	0

Summary

The module is small (one actor + DI extension + options class). The actor is a central cluster singleton that exposes three responsibility groups: direct UpsertSiteCallCommand ingest, paginated/KPI read handlers, and the central→site Retry/Discard relay. Ingest idempotency is delegated to the repository's monotonic-upsert (the CD-015 check-then-act window is mitigated by the duplicate-key swallow on the insert leg). Findings cluster around two themes: (a) the SupervisorStrategy override is dead-code that contradicts the XML docstring — it governs children, and this actor has none, so the documented "Resume on leaked exception" promise is unenforced; (b) several smaller drifts between the design doc and the code (reconciliation puller + daily purge schedule are still deferred; OnUpsertAsync does not stamp IngestedAtUtc unlike the dual-write path). The relay path is well covered by Akka TestKit unit tests; the ingest + KPI paths are covered by MSSQL-backed integration tests using a shared MsSqlMigrationFixture.

Checklist coverage

#	Category	Examined	Notes
1	Correctness & logic bugs	Yes	OnUpsertAsync does not refresh IngestedAtUtc (Finding 003).
2	Akka.NET conventions	Yes	SupervisorStrategy() override is dead code (Finding 001). Sender correctly captured before first await on every handler. PipeTo used for read replies.
3	Concurrency & thread safety	Yes	_centralCommunication mutated only on actor thread via RegisterCentralCommunication. DI scope-per-message disposed in try/finally. No issues found.
4	Error handling & resilience	Yes	Ingest catches all + replies Accepted=false. Relay distinguishes SiteUnreachable vs OperationFailed. Failover handover does not wait for in-flight async work (Finding 002).
5	Security	Yes	All SQL is parameterised at the repository (FromSqlInterpolated). Relay carries no user-controlled strings beyond SourceSite. No issues found.
6	Performance & resource management	Yes	DI scope-per-message correctly disposed. MaxPageSize=200 clamp present. No issues found.
7	Design-document adherence	Yes	Reconciliation puller and daily terminal-purge scheduler still deferred; design doc reads as if they ship (Finding 004).
8	Code organization & conventions	Yes	RegisterCentralCommunication is a top-level record colocated with the actor — by design (carries IActorRef, cannot live in Commons). No issues found.
9	Testing coverage	Yes	Relay path well covered (6 unit tests). Ingest/KPI well covered by MSSQL fixture. Stuck-only paging boundary edge not directly exercised (Finding 006).
10	Documentation & comments	Yes	XML docstring claims SupervisorStrategy uses Resume — incorrect (Finding 001). AckErrorMessage switch arm for SiteUnreachable falls through instead of throwing (Finding 005).

Re-review 2026-06-20 (commit 4307c381) — full review

Since the 1eb6e97 baseline the module grew the two pieces that were "still deferred" at the prior pass: the periodic per-site reconciliation puller (Piece A — OnReconciliationTickAsync → ReconcileSiteAsync, in-memory per-site cursor + idempotent monotonic upsert + per-site failure isolation) and the daily terminal-row purge scheduler (Piece B — OnPurgeTickAsync → PurgeTerminalAsync, continue-on-error). All of the new code is clean, idiomatic, and well-tested (dedicated SiteCallAuditReconciliationTests, SiteCallAuditPurgeTests, SiteCallRelayTests, KPI-sample and options suites); the M6 SiteCallAuditKpiSampleSource correctly anchors both cutoffs on a single capture instant and emits Global + per-Site + per-Node snapshots, and the per-node KPI handler is a clean additive peer of the per-site one. The three new findings are all forward-compat / cosmetic: a latent purge-gating gap (the purge timer is armed only when the reconciliation collaborators are present — currently masked because Program.cs co-registers them) plus two doc/edge items (the design doc over-claims six charted trend series when only three are charted, and the reconciliation cursor ignores MoreAvailable so a single-timestamp batch saturation re-pulls without progress). No correctness, concurrency, security, or resilience defects in the new code; the inclusive-cursor + idempotent-upsert pairing keeps every edge case safe-by-construction (wasted work, never corruption).

#	Category	Examined	Notes
1	Correctness & logic bugs	Yes	Reconciliation cursor ignores MoreAvailable; a single-UpdatedAtUtc batch saturation re-pulls without progress (Finding 009 — wasted work only, idempotent upsert prevents corruption). Finding 003 (IngestedAtUtc) resolved — now stamped in OnUpsertAsync and ReconcileSiteAsync.
2	Akka.NET conventions	Yes	Sender captured before first await on every async handler; PipeTo used for all read/relay replies; self-ticks scheduled via ScheduleTellRepeatedlyCancelable with Self sender, cancelled in PostStop. No issues found.
3	Concurrency & thread safety	Yes	_reconciliationCursors, _centralCommunication, timers all mutated only on the actor thread. Per-tick/per-message DI scope (CreateAsyncScope for the async tick paths, CreateScope for sync read paths) disposed in finally / await using. No issues found.
4	Error handling & resilience	Yes	Reconciliation has per-site try/catch isolation; purge is continue-on-error; ingest catches all and replies Accepted=false. Deliberate coarse-retry divergence from SiteAuditReconciliationActor (no per-row abandon) is documented and safe given idempotent upsert. No issues found.
5	Security	Yes	All SQL parameterised at the repository. Relay carries no user-controlled strings beyond SourceSite (a site id). No issues found.
6	Performance & resource management	Yes	One DI scope per tick reused across all sites; MaxPageSize=200 clamp; async DbContext disposal off the dispatcher. MoreAvailable-ignoring re-pull is bounded wasted work (Finding 009).
7	Design-document adherence	Yes	Reconciliation puller + purge scheduler now implemented (prior Finding 004 resolved). Design doc over-claims six charted KPI trend series; only three are charted (Finding 008).
8	Code organization & conventions	Yes	Purge timer gated on reconciliation collaborators rather than just the repository — a host registering SiteCallAudit without the reconciliation client silently never purges (Finding 007, latent). KpiSampleSource registered via TryAddEnumerable; options owned by the component.
9	Testing coverage	Yes	Reconciliation/purge/relay/KPI/options all have dedicated suites. No test exercises a saturated reconciliation batch (MoreAvailable: true) or the single-timestamp no-progress edge — every reconciliation test uses MoreAvailable: false (relates to Finding 009).
10	Documentation & comments	Yes	Prior Findings 001/005 doc fixes still in place. Reconciliation ReconcileSiteAsync XML claims parity with SiteAuditReconciliationActor while diverging on MoreAvailable (the sibling reads it for stalled detection; this actor ignores it) — Finding 009 recommendation includes a doc correction.

Findings

SiteCallAudit-001 — SupervisorStrategy override is dead code; XML claims Resume that is not enforced

Severity	Medium
Category	Akka.NET conventions
Status	Resolved
Location	src/ZB.MOM.WW.ScadaBridge.SiteCallAudit/SiteCallAuditActor.cs:32-46, src/ZB.MOM.WW.ScadaBridge.SiteCallAudit/SiteCallAuditActor.cs:147-151

Description

The XML remarks block (lines 32-46) states:

"The SupervisorStrategy uses Resume so an unexpected throw before the catch (defence in depth) does not restart the actor and reset in-flight state."

The override at lines 147-151 returns a OneForOneStrategy with DefaultDecider and maxNrOfRetries: 0. Two problems compound:

1. ActorBase.SupervisorStrategy() governs the actor's children, not the actor itself. SiteCallAuditActor creates no children, so this override is dead code.
2. The returned strategy uses DefaultDecider (Restart for most exceptions), not Directive.Resume. So even if the actor did have children, the strategy would not be Resume — it would be the default Restart-on-most-faults behaviour with maxNrOfRetries: 0 (which forces a Stop after the first failure).

Net effect: the actor's own self-supervision is whatever the parent supplies (SupervisorStrategy.DefaultDecider from the singleton manager / user guardian in tests), which Restarts on most exceptions. If the try/catch in OnUpsertAsync ever leaked (e.g. a synchronous throw constructing replyTo), the actor would Restart, reset _centralCommunication to null, and silently break the relay until RegisterCentralCommunication runs again.

This same pattern (with the same misleading XML doc) exists in AuditLogIngestActor, AuditLogPurgeActor, and SiteAuditReconciliationActor — they were likely cargo-culted; this finding documents the local instance.

Recommendation

Either:

• Remove the SupervisorStrategy() override entirely (it does nothing useful) and revise the XML comment to drop the "Resume" claim. Self-supervision is the parent's concern (the cluster singleton manager); the try/catch in OnUpsertAsync is what actually keeps the actor alive.
• Or, if Resume-on-self-throw is actually desired, that requires wiring a custom supervisor in the parent (ClusterSingletonManager) — not overriding SupervisorStrategy() here. Simpler path: keep the try/catch, drop the override.

The CLAUDE.md "Resume for coordinator actors" decision applies to actors with children (Site Runtime hierarchy) — not to leaf cluster singletons.

Resolution (2026-05-28): Rewrote the class-level XML on SiteCallAuditActor plus the method-level XML on SupervisorStrategy() to accurately describe what the override does — a one-for-one strategy with DefaultDecider (Restart on most exceptions, Stop on ActorInitializationException/ActorKilledException) and maxNrOfRetries: 0, governing the actor's children (the actor has none today, so the override is currently inert). Dropped the misleading "Resume" claim. The new docs make clear that self-supervision of this cluster singleton is the parent ClusterSingletonManager's concern and the actor's own resilience comes from the in-handler try/catch in OnUpsertAsync, not from this override. No behaviour change — pure documentation fix; existing 24 SiteCallAudit tests remain green.

SiteCallAudit-002 — Singleton failover does not wait for in-flight async upserts

Severity	Low
Category	Error handling & resilience
Status	Resolved
Location	src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs:455-462 (singleton wiring), src/ZB.MOM.WW.ScadaBridge.SiteCallAudit/SiteCallAuditActor.cs:153-193

Resolution (2026-05-28): Added a CoordinatedShutdown task in the cluster-leave phase (named drain-site-call-audit-singleton) that issues an explicit GracefulStop(10s) to the SiteCallAudit cluster singleton manager before the cluster-leave proceeds. Akka.NET's singleton handover already waits for the active actor's ReceiveAsync task to complete before signalling HandOverDone, so an in-flight EF UpsertAsync (and its SQL round-trip) drains on the old node before the new singleton starts on the other central node — closing the seam where the new singleton could race a still-running upsert on the old node. The 10-second timeout is bounded so a misbehaving upsert cannot stall coordinated shutdown indefinitely; on timeout the existing PoisonPill termination path takes over and the repository's monotonic-upsert + 2601/2627 duplicate-key swallow remain as the storage-state safety net. Pattern is suitable for the NotificationOutbox singleton too; deferred to keep this change scoped.

Description

The singleton is created with terminationMessage: PoisonPill.Instance. On failover the active node's singleton stops as soon as the mailbox is drained of normal messages and the PoisonPill is processed. An in-flight OnUpsertAsync Task started before the PoisonPill arrived will be allowed to complete (the message-handler runs synchronously from the mailbox's view), but the Akka actor model does NOT cancel the EF Core ExecuteSqlInterpolatedAsync call.

Two consequences:

1. The new singleton on the other node may begin accepting UpsertSiteCallCommand for the same TrackedOperationId while the old singleton's in-flight upsert is still running. The repository's monotonic-upsert and the SQL duplicate-key swallow protect storage state.
2. The original replyTo sender may receive its Accepted=true after the new singleton has already returned a different reply. Idempotency keys protect correctness; wire-level ordering is best-effort by design.

This is consistent with the design ("eventually-consistent mirror, sites are source of truth"), but worth documenting as an explicit invariant. The Notification Outbox sibling has the same pattern.

Recommendation

• Document the failover/handover semantics in the actor's XML remarks: "On cluster singleton handover, in-flight OnUpsertAsync tasks complete on the old node and may produce a late Accepted=true reply; the repository's monotonic upsert ensures storage state is consistent."
• Add an integration test that deliberately races two concurrent upserts on the same TrackedOperationId to verify the duplicate-key swallow + monotonic rank check (the CD-015 race-pattern check the parent task flagged).

SiteCallAudit-003 — OnUpsertAsync does not refresh IngestedAtUtc; direct-write callers must remember to stamp it

Severity	Medium
Category	Correctness & logic bugs
Status	Resolved
Location	src/ZB.MOM.WW.ScadaBridge.SiteCallAudit/SiteCallAuditActor.cs:153-193

Description

The combined-telemetry hot path (AuditLogIngestActor.OnCachedTelemetryAsync) stamps IngestedAtUtc = DateTime.UtcNow on both the AuditLog row and the SiteCall row at central-side persist time (src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogIngestActor.cs:238-239). The design doc treats IngestedAtUtc as "central ingested (or last refreshed) this row" — a central-side timestamp.

SiteCallAuditActor.OnUpsertAsync writes the supplied SiteCall as-is, with whatever IngestedAtUtc the caller stamped. The only current callers are the unit tests (which use DateTime.UtcNow at command-construction time). Once the deferred reconciliation puller lands and starts emitting UpsertSiteCallCommands, the puller (running on central) is responsible for stamping a central timestamp — but if a future direct-write caller forgets, or constructs from a site DTO, the value could drift (e.g. become a site clock value).

This is currently latent because no production caller exists, but it's inconsistent with the dual-write code path and undocumented.

Recommendation

• Either: stamp IngestedAtUtc = DateTime.UtcNow inside OnUpsertAsync before calling UpsertAsync (matching AuditLogIngestActor's behaviour), using cmd.SiteCall with { IngestedAtUtc = DateTime.UtcNow }.
• Or: document in the UpsertSiteCallCommand XML that callers MUST stamp IngestedAtUtc to a central-side DateTime.UtcNow immediately before sending.

Preferred: stamp inside the actor — same as the combined-telemetry path — because callers cannot in general know the actor is colocated on central.

Resolution (2026-05-28): OnUpsertAsync now rewrites the incoming SiteCall via cmd.SiteCall with { IngestedAtUtc = DateTime.UtcNow } immediately before calling repository.UpsertAsync, mirroring AuditLogIngestActor's combined-telemetry hot path. The repository writes IngestedAtUtc on both the insert-if-not-exists and the monotonic UPDATE legs (SiteCallAuditRepository.UpsertAsync), so the column is writable on every upsert. Callers (telemetry, the deferred reconciliation puller, any future direct-write) no longer need to remember to stamp a central-side timestamp — the actor owns it. Existing 24 SiteCallAudit tests remain green (the MSSQL-fixture test constructs rows with DateTime.UtcNow and doesn't assert the exact value, so the actor's re-stamp is backward compatible).

SiteCallAudit-004 — Reconciliation puller and daily terminal-purge scheduler still deferred; design-doc drift

Severity	Low
Category	Design-document adherence
Status	Resolved
Location	src/ZB.MOM.WW.ScadaBridge.SiteCallAudit/SiteCallAuditActor.cs:23-30 (actor XML), src/ZB.MOM.WW.ScadaBridge.SiteCallAudit/ServiceCollectionExtensions.cs:8-13, docs/requirements/Component-SiteCallAudit.md:24-32

Description

The design doc (Component-SiteCallAudit.md lines 24-32) lists five responsibilities, including:

• "Run periodic per-site reconciliation pulls so missed telemetry self-heals."
• "Purge terminal audit rows after a configurable retention window."

The repository exposes PurgeTerminalAsync but nothing in this module schedules a daily call (Notification Outbox owns a MaintenanceService for its equivalent; no SiteCallAuditMaintenanceService exists). The reconciliation puller is acknowledged in the actor XML (only reconciliation remains deferred) but is not surfaced in the design doc as deferred — the doc reads as if it ships.

Recommendation

• Either: implement the deferred pieces (a hosted service that wakes daily and calls repo.PurgeTerminalAsync(now - retentionWindow), plus a per-site reconciliation puller with a cursor + an IPullCachedTelemetryClient).
• Or: add a "Status" / "Deferred" subsection to the design doc explicitly listing what's not yet implemented (matches the pattern Audit Log uses for its tamper-evidence hash chain).

Resolution (2026-05-28):

Updated the class-level XML on SiteCallAuditActor to reflect actual state: telemetry ingest, query/detail/KPI handlers (Task 4), and the central→site Retry/Discard relay (Task 5) are implemented; the periodic reconciliation puller and the daily terminal-row purge scheduler remain deferred. The design doc update is tracked separately.

SiteCallAudit-005 — AckErrorMessage switch arm for SiteUnreachable returns ack message instead of throwing

Severity	Low
Category	Documentation & comments
Status	Resolved
Location	src/ZB.MOM.WW.ScadaBridge.SiteCallAudit/SiteCallAuditActor.cs:548-563

Description

return outcome switch
{
    SiteCallRelayOutcome.Applied => null,
    SiteCallRelayOutcome.NotParked => "The operation is no longer parked at the site (...)",
    SiteCallRelayOutcome.OperationFailed => ack.ErrorMessage,
    // SiteUnreachable is never produced from a ParkedOperationActionAck —
    // unreachable responses are built by UnreachableRetry/UnreachableDiscard
    // before any ack is classified, so this arm is unreachable by construction.
    SiteCallRelayOutcome.SiteUnreachable => ack.ErrorMessage,
    _ => throw new ArgumentOutOfRangeException(...)
};

The comment correctly states the SiteUnreachable arm is unreachable when called from ClassifyAck. The arm therefore exists only to satisfy exhaustiveness, but instead of throwing or returning a sentinel, it falls through to ack.ErrorMessage — indistinguishable from the OperationFailed arm above. If any future caller does feed SiteUnreachable into AckErrorMessage (e.g. via refactor), the result will be a silent wrong-detail-text bug rather than an immediate crash. The default arm correctly throws ArgumentOutOfRangeException, so the SiteUnreachable arm is the inconsistent one.

Recommendation

Replace the SiteUnreachable => ack.ErrorMessage arm with:

SiteCallRelayOutcome.SiteUnreachable =>
    throw new InvalidOperationException(
        "AckErrorMessage cannot be called for SiteUnreachable — those responses "
        + "are built by UnreachableRetry/UnreachableDiscard before classification."),

— fail fast if the invariant is ever violated by a refactor.

Resolution (2026-05-28):

Behaviour kept (return ack.ErrorMessage); AckErrorMessage stays total and side-effect-free. Expanded the inline comment on the SiteUnreachable arm to explain WHY it returns rather than throws: site-unreachable is classified as transient by the upstream relay (which has already built its SiteUnreachable response and detail text via SiteUnreachableMessage), so a defensive fall-through surfaces the ack's message and lets the caller schedule a retry — throwing would turn a benign refactor invariant violation into a relay-path crash.

SiteCallAudit-006 — Stuck-only paging test does not exercise the multi-page boundary with an interleaved non-stuck row at the cursor

Severity	Low
Category	Testing coverage
Status	Resolved
Location	tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests/SiteCallAuditActorTests.cs:335-392

Resolution (2026-05-28): Added SiteCallQueryRequest_StuckOnly_CursorAtNonStuckBoundary_SkipsToNextStuckRow to tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests/SiteCallAuditActorTests.cs — drives six rows interleaved as stuck/non-stuck × 3 (oldest-first), then issues three page-size-1 stuck-only queries. The cursor between each page deliberately lands on a non-stuck row, so the SQL composition of the stuck predicate AND the keyset cursor predicate must skip it. Asserts each page returns exactly one stuck row in DESC-by-CreatedAtUtc order with no overlap and all three stuck rows visited. Locks the invariant that post-filtering does not produce under-filled pages with non-null next cursors.

Description

SiteCallQueryRequest_StuckOnly_PagesAreFull_NoEmptyPagesWithCursor covers the case where stuck rows are interleaved with non-stuck rows (page-1 returns 2 stuck rows, page-2 returns the third). It does not cover the edge where the row at the keyset cursor boundary (AfterCreatedAtUtc + AfterId) is itself a non-stuck row — i.e. the cursor points at a row the next page must SKIP through to find more stuck rows. The repository's SQL composes the cursor predicate (CreatedAtUtc < cursor OR (CreatedAtUtc = cursor AND id < ...)) with the stuck predicate, so it should be honest, but the test only asserts row counts and IsStuck, not that the second-page query specifically skipped non-stuck rows between the cursor and the next stuck row.

Lower priority because the SQL composition is straightforward, but adding a direct test would lock the invariant.

Recommendation

Add a test that (a) inserts 6 rows in interleaved order: stuck, not-stuck, stuck, not-stuck, stuck, not-stuck (oldest first); (b) issues a StuckOnly page-size-1 query; (c) asserts each page returns exactly the stuck row, with no overlap and all 3 stuck rows visited.

SiteCallAudit-007 — Daily purge timer is armed only when the reconciliation collaborators are present; a host without the reconciliation client silently never purges

Severity	Medium
Category	Code organization & conventions
Status	Resolved
Location	src/ZB.MOM.WW.ScadaBridge.SiteCallAudit/SiteCallAuditActor.cs:307-321, src/ZB.MOM.WW.ScadaBridge.SiteCallAudit/SiteCallAuditActor.cs:202-227

Description

StartPurgeTimer (lines 307-321) gates the daily terminal-row purge tick on the reconciliation collaborators being present:

private void StartPurgeTimer()
{
    if (_pullClient is null || _siteEnumerator is null)
    {
        return;
    }
    // ... schedule PurgeTick ...
}

But the purge pass (OnPurgeTickAsync → PurgeWithRepositoryAsync → ISiteCallAuditRepository.PurgeTerminalAsync) needs only the repository — it has no dependency on IPullSiteCallsClient / ISiteEnumerator. The two collaborators are resolved by the production constructor (lines 202-227) via serviceProvider.GetService<IPullSiteCallsClient>() / GetService<ISiteEnumerator>() — both registered by AddAuditLogCentralReconciliationClient (AuditLog/ServiceCollectionExtensions.cs:473,523, registered as ISiteEnumerator and IPullSiteCallsClient). GetService (not GetRequiredService) returns null if that helper was never called, so a host that registers AddSiteCallAudit() without also calling AddAuditLogCentralReconciliationClient(...) constructs the actor with both collaborators null. In PreStart both StartReconciliationTimer and StartPurgeTimer early-return, so the actor runs forever with no purge timer at all → unbounded growth of the central SiteCalls table, with no log line to say the purge was skipped.

This is currently latent, not live: Program.cs co-registers AddAuditLogCentralReconciliationClient(builder.Configuration) (line 107) immediately before AddSiteCallAudit() (line 113), so on the real central node both collaborators resolve and both timers run today. The risk is a future host (or a refactor that splits the reconciliation client out of the central composition root, or a test/embedded host that wants only ingest + purge) silently losing the purge with no diagnostic. The gate exists for a legitimate reason — keeping the repo-only test ctor free of both background timers so the MSSQL read/upsert tests see no scheduled side effects — but it conflates "no reconciliation route" with "no purge", and the actor's own XML (lines 36-38) documents the coupling as deliberate rather than flagging it as a hazard.

Recommendation

Decouple StartPurgeTimer from the reconciliation collaborators — purge needs only a repository, which both the production and the repo-only test ctors always have. Two viable shapes:

• Preferred: gate the purge timer on its own real precondition (a repository is always available, so arm it unconditionally in the production + reconciliation ctors; keep it off only in the repo-only test ctor via an explicit "background timers off" flag rather than by proxy of the reconciliation collaborators). This keeps the MSSQL test isolation while removing the accidental coupling.
• At minimum: log a Warning in StartPurgeTimer (and StartReconciliationTimer) when the timer is not armed on the production path, so a misconfigured host surfaces "SiteCallAudit purge timer not started — PurgeTerminalAsync will never run" instead of growing the table silently.

Severity is a judgment call: Medium because the consequence (unbounded central table growth) is real and silent, but it is latent today (the only production composition root co-registers the reconciliation client), so an argument for Low is defensible. Flagged Medium to weight the silent-failure + data-growth aspect.

Resolution

Resolved 2026-06-20 (commit fd618cf1): decoupled the daily purge timer from the reconciliation collaborators (a new _backgroundTimersEnabled flag) so a central node that omits the reconciliation client still purges — no more silent unbounded SiteCalls growth; a Warning is logged if the reconciliation collaborators are absent. Test added (purge arms without the reconciliation client).

SiteCallAudit-008 — Design doc claims six charted KPI trend series; only three are actually charted

Severity	Low
Category	Design-document adherence
Status	Resolved
Location	docs/requirements/Component-SiteCallAudit.md:149-154, src/ZB.MOM.WW.ScadaBridge.SiteCallAudit/Kpi/SiteCallAuditKpiSampleSource.cs:42-47, src/ZB.MOM.WW.ScadaBridge.Commons/Types/Kpi/KpiMetrics.cs:52-62

Description

The design doc (Component-SiteCallAudit.md lines 149-154, the KPI History interaction) states:

"… the resulting buffered / parked / failedLastInterval / deliveredLastInterval / stuck / oldestPendingAgeSeconds series render as trends on the Site Calls page via KpiTrendChart."

That lists six series as charted. In the code only three are charted:

• The public charted catalog KpiMetrics.SiteCallAudit (KpiMetrics.cs:52-62) exposes exactly Buffered, Parked, and FailedLastInterval — and its own XML says "Charted Site Call Audit (#22) metrics … Rendered by the Central UI Site Calls report trend panel."
• SiteCallAuditKpiSampleSource (lines 42-47) keys those three off the public Commons catalog (KpiMetrics.SiteCallAudit.*) and keeps the other three — deliveredLastInterval, stuck, oldestPendingAgeSeconds — as private string literals, with the comment "Charted metrics share the public Commons catalog … the uncharted internal metrics stay private here (#178)."
• The UI confirms it: SiteCalls/SiteCallsReport.razor.cs calls LoadSeriesAsync for exactly KpiMetrics.SiteCallAudit.Buffered, .Parked, and .FailedLastInterval — three series, no more.

So all six metrics are sampled into the KpiSample history store, but only three are charted. The doc reads as if all six are rendered, which is a small design-doc drift (over-claiming the UI surface). The code is internally consistent and correctly commented; the doc is the stale party.

Recommendation

One-line doc edit on Component-SiteCallAudit.md lines 149-154: state that the three series buffered / parked / failedLastInterval render as trends on the Site Calls page via KpiTrendChart, and that deliveredLastInterval / stuck / oldestPendingAgeSeconds are sampled into the KPI-history store but not charted (available for future trend panels / ad-hoc query). No code change — the code is already the intended state.

Resolution

Resolved 2026-06-20 (commit fd618cf1): the design doc now lists the three actually-charted KPI metrics (buffered/parked/failedLastInterval) and marks the other three as sampled-but-not-yet-charted, matching the KpiMetrics catalog. Doc-only.

SiteCallAudit-009 — Reconciliation cursor advances inclusively and ignores MoreAvailable; a single-timestamp batch saturation re-pulls without progress

Severity	Low
Category	Correctness & logic bugs
Status	Resolved
Location	src/ZB.MOM.WW.ScadaBridge.SiteCallAudit/SiteCallAuditActor.cs:505-535

Description

ReconcileSiteAsync (lines 505-535) pulls rows since the per-site cursor, upserts each, and advances the cursor to the maximum UpdatedAtUtc observed:

var since = _reconciliationCursors.TryGetValue(site.SiteId, out var c) ? c : DateTime.MinValue;
var response = await client.PullAsync(site.SiteId, since, _options.ReconciliationBatchSize, ...);

var maxUpdated = since;
foreach (var row in response.SiteCalls)
{
    await repository.UpsertAsync(row with { IngestedAtUtc = nowUtc });
    if (row.UpdatedAtUtc > maxUpdated) maxUpdated = row.UpdatedAtUtc;
}
_reconciliationCursors[site.SiteId] = maxUpdated;

Two interacting properties:

1. The pull is inclusive — PullAsync(site, since, …) asks for rows with UpdatedAtUtc >= since (documented in the method's XML, lines 497-503), and the cursor advances to the max UpdatedAtUtc seen, which is itself one of the rows just pulled. So the boundary row is re-pulled every tick and deduped by the idempotent monotonic upsert — intended, harmless.
2. response.MoreAvailable is never read at all. The PullSiteCallsResponse carries MoreAvailable (PullSiteCallsResponse.cs:14-17: "True when the site saturated the requested batch size — the caller should advance the cursor and pull again"), but ReconcileSiteAsync ignores it and relies on the natural tick cadence to drain the backlog over successive ticks.

The edge case: if a site has more rows than the batch size all sharing one exact UpdatedAtUtc (e.g. a burst written in the same tick / same clock value), the saturated batch returns rows whose max UpdatedAtUtc equals since. maxUpdated therefore stays at since, the cursor does not advance, and because the pull is inclusive the next tick re-pulls the identical window — and again, and again — making no forward progress on that site's backlog. Because the upsert is idempotent and the SiteCalls table is an eventually-consistent mirror (not the source of truth), this is wasted work, never corruption — but it is an unbounded re-pull loop on a pathological input, and any rows in that backlog beyond the batch ceiling never get reconciled.

The sibling SiteAuditReconciliationActor shares the inclusive-cursor / max-timestamp shape, so the same single-timestamp-saturation no-progress edge applies to it — but that sibling does read MoreAvailable (it feeds its stalled-detection state machine, SiteAuditReconciliationActor.cs:324-325, publishing SiteAuditTelemetryStalledChanged so a non-draining site surfaces a health signal). ReconcileSiteAsync's XML (lines 530-534) claims the no-immediate-re-pull behaviour "match[es] SiteAuditReconciliationActor", which is only half true: it matches the cursor cadence but diverges by dropping MoreAvailable entirely, so this actor has neither a continuation pull nor a stalled signal — a saturated site lags silently with no observability.

Recommendation

• Consume MoreAvailable: either continue draining within the same tick while MoreAvailable is true (bounded by a max-iterations guard), or — matching the sibling — surface a stalled/non-draining signal when a batch comes back saturated so a stuck site is observable rather than silent.
• Defend the single-timestamp no-progress edge with a tiebreaker beyond the timestamp (e.g. advance on (UpdatedAtUtc, TrackedOperationId) as a composite keyset cursor, and ask the pull for rows strictly after that composite), so a burst sharing one UpdatedAtUtc cannot pin the cursor.
• Correct the ReconcileSiteAsync XML (lines 530-534): it claims parity with SiteAuditReconciliationActor while diverging on MoreAvailable (the sibling reads it for stalled detection; this actor ignores it).

Severity Low: idempotent upsert + mirror-not-source-of-truth mean no data corruption, and the saturated-single-timestamp input is pathological; the cost is wasted re-pulls and an un-drained backlog tail on that one input, plus the missing observability.

Resolution

Resolved 2026-06-20 (commit fd618cf1): ReconcileSiteAsync now consumes response.MoreAvailable via a within-tick continuation drain bounded by a max-pages guard, with explicit no-progress (single-timestamp-saturation) detection that breaks and logs a Warning instead of re-pulling forever. The XML claim of parity with the sibling reconciler was corrected. Idempotent upsert retained. Tests added.