Defect: the hourly KPI rollup fold (KpiHistoryRepository.FoldHourlyRollupsAsync)
sums every Rate-classified sample recorded in an hour, correct only if each
sample is a non-overlapping delta covering exactly one recorder tick
(KpiHistoryOptions.SampleInterval). That invariant was an unenforced silent
assumption across three independently-owned config keys — SampleInterval,
NotificationOutboxOptions.DeliveredKpiWindow, SiteCallAuditOptions.KpiInterval —
so widening DeliveredKpiWindow to e.g. 15 min would silently over-count the
rollup ~15x.
Robust fix: NotificationOutboxKpiSampleSource and SiteCallAuditKpiSampleSource
now rescale their Rate-classified samples (deliveredLastInterval,
failedLastInterval) by SampleInterval / (DeliveredKpiWindow or KpiInterval)
before recording, so the rollup's per-hour Sum is correct regardless of how
the other two keys are tuned — the live dashboard tile's own cutoff/window is
untouched. Both sources gain a project reference to KpiHistory to read
SampleInterval; no cycle (KpiHistory references only Commons).
Eager cross-validation (defense-in-depth): KpiRateFoldIntervalCrossValidator
(Host, the one composition root referencing all three component projects)
fails boot naming all three config keys when they diverge, protecting the
separate (still-real) invariant that the recorded/charted KPI-history sample
equals the live tile at the same instant — which the rescale trades away
whenever the scale factor isn't 1. Rides KpiHistoryOptions' existing
ValidateOnStart trigger.
Also: NotificationOutboxOptions had no registered validator at all before this
(the gap this task's "whichever options class owns DeliveredKpiWindow" file
scope pointed at) — added NotificationOutboxOptionsValidator mirroring
SiteCallAuditOptionsValidator's positive-duration/count convention, wired with
ValidateOnStart.
Tests: new rescale tests in both sample-source suites prove the fold is
numerically correct at a non-default SampleInterval (30/40 counted over a
15/20-minute window correctly divides down to a 1-minute-tick delta); new
KpiRateFoldIntervalCrossValidatorTests prove divergent config fails with all
three keys named; new NotificationOutboxOptionsValidatorTests. Every new test
proven to fail against the unfixed computation first (temporarily
short-circuited the scale factor / cross-check to the pre-fix behavior,
confirmed failure, restored) — not via git stash (repo-wide, shared across
concurrent worktrees this session) but via file-copy swap.
Full suites green: Host.Tests (527), ConfigurationDatabase.Tests (401),
KpiHistory.Tests (54), NotificationOutbox.Tests (163),
SiteCallAudit.Tests (67).
Row 65 — trigger FIRED, fixed. BundleUnlockAttempt rows were pruned only by
nodes actively serving unlocks, so a cluster that stopped importing entirely
left its last rows in place forever. The row's own trigger was "Transport gains
a central scheduler a prune tick could ride" — it has one now:
BundleSessionEvictionService, the central-only 1-minute session sweep added by
the transport-session fixes. It now also ticks the new
IBundleUnlockAttemptWindow.PruneAsync. Both drivers share the ONE 15-minute CAS
cadence claim, so ticking on a 1-minute sweep costs at most one bounded DELETE
per quarter hour and the two can never prune concurrently. PruneAsync is
contractually non-throwing (unlike TryRegisterAttemptAsync, which throws and
lets the limiter own the fail-open decision) — a background sweep has no caller
to report to and must survive a database blip. Both Transport docs updated;
three tests in FleetWideUnlockThrottleTests, incl. the retention sweep running
with no unlock traffic anywhere in the test, which is the whole point.
Row 58 — closed with a pin, and the OLD pin was found blind. The existing
Deployments_Dispose_UnsubscribesFromNotifier is behavioural: it is satisfied by
the _disposed guard alone, so deleting the `-=` from Dispose leaves it green
while the singleton notifier keeps a delegate pinning the circuit-scoped
component. Verified by mutation — with the `-=` removed the old test still
passed and only the two new pins failed. Added:
* a structural retention pin asserting the notifier's invocation list actually
shrinks to 0 on Dispose;
* a source scan pinning the row's stated TRIGGER ("a second page subscribes,
or any subscriber lands without a matching Dispose unsubscribe") — every
CentralUI file that subscribes must also unsubscribe and be IDisposable,
with a non-empty guard so the scan cannot pass vacuously.
Row 60 — closed with a pin, and the register's premise corrected.
GlobalArtifactSnapshot does NOT hold EF-materialised entities: it is projected
into pure Commons DTOs before the fan-out. The EF state that genuinely crosses
into the parallel send is the tracked Site entities the lambda closes over, and
that is safe only while lazy loading is off. Pinned in ConfigurationDatabase
.Tests two ways — no proxies extension on the built DbContextOptions, and no
EntityFrameworkCore.Proxies package reference anywhere in the repo (the
structurally stronger half: UseLazyLoadingProxies() is not a compilable call
without it). The real invariant is now spelled out at the fan-out itself.
Row 59 — no product change; added the missing tripwire remark stating the
constraint the safety argument rests on (serial by design, one scoped DbContext,
do not Task.WhenAll; if it gets slow the fix is a scope PER SITE, not a shared
scope under fan-out). Line numbers in the register have drifted to :528-545.
Rows 36, 37, 64: verification only, no change — see the sweep report.
AuditLogIngestActor and SiteCallAuditActor each exposed constructors taking a scoped
repository and caching it on a field for the actor's lifetime. Safe only because no
production call site picked those overloads — i.e. one call site away from re-introducing
the row-45 captive-dependency bug — which is why CaptiveDependencyGuard had to allowlist
both fields.
Preferred fix applied: the test fixtures fold onto the IServiceProvider overload with a
stub provider, and the constructors are gone.
Product deletions:
- AuditLogIngestActor: the (IAuditLogRepository, ILogger) ctor, the _injectedRepository
field and its branch in OnIngestAsync, and OnCachedTelemetryWithoutDualWriteAsync (the
"this ctor cannot dual-write" fallback handler that only that ctor installed).
_serviceProvider is now non-nullable and the null-forgiving operators are gone.
- SiteCallAuditActor: both test ctors (repo-only, and repo + reconciliation
collaborators), the _injectedRepository field and its four branches, and the private
SerializedRepository class. _backgroundTimersEnabled went with them: only the deleted
repo-only ctor ever set it false, so it was a constant and its two guards are removed.
The reconciliation tick is still gated — on the collaborators being resolvable, which
is how production reaches the same state.
Test fixtures (12 call sites): two small TestSupport helpers build the stub providers.
SiteCallAuditActor's production wiring resolves a FRESH repository per scope, so the
serializing decorator existed only to protect fixtures that share one DbContext-backed
instance across every scope — it moves into the test project as
SerializingSiteCallAuditRepository, which is where the hazard is actually created.
Fixtures that relied on it keep it. The relay test's ICentralAuditWriter now arrives via
the provider, exactly as production resolves it.
Behaviour is preserved on the paths under test: the deleted audit ctor passed
redactor:null / failureCounter:null, and a provider registering neither resolves both to
null, so the SafeDefaultAuditRedactor fallback and counter-free logging are unchanged. No
fixture on the folded sites sends IngestCachedTelemetryCommand, so swapping to the real
dual-write handler is inert. The SiteCallAudit purge timer now arms in fixtures that
previously had timers off, but every one of them uses default purge settings whose first
tick is 5 minutes out — nothing fires inside a test.
One reconciliation test was renamed and rewritten rather than deleted:
TestCtor_RepositoryOnly_DoesNotStartReconciliationTick became
NoReconciliationCollaboratorsRegistered_DoesNotStartReconciliationTick — the same
guarantee, expressed the way production actually reaches it.
Guard: the allowlist is now EMPTY. The staleness self-test (every entry must still match a
flagged field) is replaced by two stronger pins — Allowlist_IsEmpty asserts nothing is
exempt at all, and FormerlyAllowlistedActors_NowScanClean scans both actors with the
allowlist DISABLED, so the empty allowlist is evidence the fields are gone rather than
evidence the scan stopped looking.
Negative control: re-adding an ISiteCallAuditRepository field to SiteCallAuditActor fails
both FormerlyAllowlistedActors_NowScanClean and the real
Central_NoActorHoldsCaptiveScopedState scan.
Verification: solution builds clean. AuditLog.Tests 3/3 (383), SiteCallAudit.Tests 3/3
(66), Host.Tests 2/2 (505), IntegrationTests 2/2 (115), Communication.Tests 2/2 (717).
Six adversarial-review findings in the central SQL/ingest layer.
F1 (AuditLogRepository.InsertChunkAsync) — the set-based ingest declared each
string parameter at its COLUMN width (Actor/Target 256, Action 64, Outcome 16,
Category 32, SourceNode 64), so SqlClient truncated an over-long value at bind
time and committed the mutilated row — silent, in an append-only store, with no
PayloadTruncated flag — while the per-row and reconciliation paths sent the same
value in full and let the server reject it with 2628. Bind at the value's own
length instead; explicit SqlDbType is kept (it fixes the VALUES constructor's
derived column types and datetime2 precision). Design: reject everywhere,
truncate nowhere — matching today's per-row behaviour.
F2 (SiteCallAuditRepository.UpsertAsync) — the single-statement upsert ran the
monotonic UPDATE first and INSERTed only if nothing matched. Two writers racing
the first packet of one TrackedOperationId (the cached dual-write and the
reconciliation pull carry DIFFERENT lifecycle states) both matched nothing, and
the loser then skipped its INSERT or swallowed a 2627 — dropping its
Status/RetryCount/HttpStatus/TerminalAtUtc. Legs swapped to
`IF NOT EXISTS … INSERT; UPDATE <monotonic>` — still one round trip, and the
loser's UPDATE now lands on the winner's row. The duplicate-key catch re-runs
the monotonic UPDATE for the same reason. Moved to raw SQL with explicitly-typed
parameters so the intricate rank predicate exists in exactly one place (an
untyped DateTime would bind as `datetime` and round the freshness tiebreaker).
F3 (docs/plans/sql/*.sql) — filtered-index DDL failed with error 1934 under the
documented `docker exec … sqlcmd` path, which defaults QUOTED_IDENTIFIER OFF;
once IX_Notifications_Delivered exists, QI-OFF DML on Notifications fails too.
All four scripts now open with `SET QUOTED_IDENTIFIER ON; SET ANSI_NULLS ON; GO`
(own batch, so it is in force when the next batch parses), and the migration
convention in Component-ConfigurationDatabase.md documents `sqlcmd -I`. Verified
live: the pre-fix script fails 1934 without -I, the fixed one applies.
F4 (SiteCallAuditActor) — the off-mailbox reconciliation/purge passes reuse the
injected repository, so tests drove one DbContext from the pass and a mailbox
handler concurrently. Serialized at the CALL via a private SerializedRepository
wrapper applied only by the test constructors, rather than running the pass
on-mailbox: production keeps its PipeTo shape untouched, and the existing
"a blocked drain does not stall ingest/query/KPI" regression tests stay
meaningful (they would have been invalidated by suspending the mailbox).
F5 (AuditLogIngestActor) — when the batch failed because the 20 s IngestBudget
expired, the per-row fallback reused the same expired token: N instant failures,
N counter bumps, zero accepted. The fallback now gets a fresh 5 s budget (inside
the 30 s outer Ask), and a blown budget bumps the failure counter ONCE for the
batch instead of once per row.
F6 (NotificationOutboxRepository.UpdateAsync) — ExecuteUpdate's row count was
discarded, so an operator Retry/Discard of a notification the retention purge had
already deleted reported success (the pre-ExecuteUpdate code threw
DbUpdateConcurrencyException). UpdateAsync now returns whether a row matched; the
operator one-shots answer "notification not found" and emit no audit row for the
action that did not happen, while the dispatcher logs a warning (its delivery
already happened; nothing to retry). GetByIdAsync switched to AsNoTracking since
the write is out-of-band.
Tests: 5 new SQL-backed regressions (over-long Target rejected on both paths +
boundary round-trip; concurrent first-write and already-created-by-another-writer
upserts; vanished-row UpdateAsync), a token-identity pin on the ingest fallback,
a repository-concurrency detector for the SiteCallAudit passes, and vanished-row
operator-path tests. The F1/F2/F4 regressions were each confirmed failing against
the pre-fix code. Suites: ConfigurationDatabase 369, AuditLog 378, SiteCallAudit
66, NotificationOutbox 152 — all green, solution builds with 0 warnings.
Phase 4 of the ClusterClient -> gRPC migration deleted the Akka transports
but left the naming behind: `ClusterClientSiteAuditClient` was transport-
agnostic and worked unchanged, so it survived the deletion under a name
that now describes a transport the repo no longer has. Same for a scatter
of doc-comments still framing gRPC as "the new transport" beside an Akka
one that is gone.
Renames it to `SiteCommunicationAuditClient` (and its test file) and
rewrites the stale comments to describe the single transport that exists.
Also tightens CLAUDE.md: drops the self-describing directory listing and
the 27-component enumeration in favour of the non-obvious parts only.
Behaviour-neutral: names and prose only. Recorded as the Phase 4
follow-up in docs/plans/2026-07-22-clusterclient-to-grpc-plan.md.
Reconciliation cursor becomes composite (UpdatedAtUtc, TrackedOperationId);
IPullSiteCallsClient/GrpcPullSiteCallsClient forward the Task-15 after_id keyset
(additive param, null preserves the legacy inclusive >= contract). A burst
sharing one exact UpdatedAtUtc now drains via the id tiebreak instead of pinning
forever. A legacy site that ignores after_id is latched + published as
SiteCallReconciliationPinnedChanged on the EventStream (transition-only),
replacing the prior silent log line.
Additive RequestedBy on Retry/DiscardSiteCallRequest, plumbed from the Central UI.
On an Applied relay, SiteCallAuditActor emits one best-effort CachedResolve central
direct-write row (Submitted/Discarded) with the operator as Actor. Gated on an
injected ICentralAuditWriter (null in existing tests → no mirror read, unchanged);
the site remains the source of truth for the state change.
Remove project bookkeeping citations from shipped code comments across the
solution: hyphenated task IDs (WP-14, StoreAndForward-025), milestone/task/
issue refs (M3, Task 4, Audit Log #23, #21), Bundle X task-bundle labels,
and C/D/K/S/T phase labels.
Comment text only — no code logic, string/log literals, or XML-doc structure
changed. Genuine descriptions are preserved (only the citation is stripped),
and technical lookalikes are retained (UTF-8, SHA-256, T00:00:00, M365,
UTC-5, pre-C4/pre-C5 schema versions). Flagged by the new CommentChecker
TaskReferenceInComment / TrackingReferenceInComment checks plus targeted
grep passes; full solution builds clean, append-only guard tests pass.
Remediation from the full per-module code review at 4307c381 (findings recorded
separately in code-reviews/).
Highs fixed:
- DeploymentManager-025/SiteRuntime-031: stop broadcasting notification lists + SMTP
configs (incl. credentials) to sites; site purges already-persisted rows on apply
(enforces the central-only delivery design; clears plaintext SMTP creds at rest).
- DataConnectionLayer-023: guard the native-alarm subscribe path against the
mid-flight-unsubscribe adapter-feed leak (mirrors the DCL-021 tag-path fix).
- SiteEventLogging-024: normalize From/To query bounds to UTC (the -016 fix the
audit trail claimed but never committed).
- KpiHistory-001: add an in-flight guard to the recorder sample tick.
- ScriptAnalysis-001: harden the trust analyzer's TPA-absent fallback (resolve
forbidden anchors in the minimal reference set; warn on degraded mode) — anchors
added to validation references only, never the compile gate.
(InboundAPI-026 left to the feat/ipsen-movein effort per owner decision.)
Medium/Low: DM-026 deterministic deploy-status tiebreaker; SR-027/028/029/030
native-alarm leak/phantom-active/delete-during-redeploy fixes; AL-013/014/016;
TE-024 (folder-mutation audit rows now persisted)/025; SF-025 gauge-provider
clear-on-stop; ESG-025/026; SEC-023/024/025; SCA-007/008/009; plus doc/test
accuracy COM-023/024, HOST-025/026, HM-024/025, NS-027/028.
Full-solution build 0 warnings; ~3560 tests across 18 touched suites green.
- SiteCallAudit/ServiceCollectionExtensions.cs: drop "still deferred" note on relay; point to SiteCallAuditActor where it lives
- Transport/Import/BundleImporter.cs: update "Only LoadAsync implemented" to reflect all three phases shipped
- SiteRuntime/Scripts/AuditingDbCommand.cs: replace two M5-deferred redaction comments with accurate references to AuditLogOptions.PerTargetOverrides
- SiteRuntime/Scripts/ScriptRuntimeContext.cs: replace "M5 will layer redaction" note with accurate description of shipped redactor
- CentralUI/AuditLogPage.razor.cs: replace "Bundle C wires… no-op seam" with accurate description of HandleRowSelected implementation
- docs/plans/2026-05-24-transport-design.md §13: update from "CLI Deferred / not built in v1" to reflect shipped BundleCommands.cs; update Open Questions entry
- docs/plans/2026-05-24-transport.md: convert Out-of-Scope "Do NOT build CLI" reminder to a factual note that it shipped
- docs/plans/2026-05-24-transport.md.tasks.json: flip all 30 tasks from pending → done (entire Transport feature shipped)
Add a daily purge tick to SiteCallAuditActor that drops terminal SiteCalls
rows older than the retention window via ISiteCallAuditRepository.PurgeTerminalAsync.
The threshold is computed each tick as UtcNow - RetentionDays so an operator who
lowers RetentionDays sees it on the next purge without a restart. Mirrors
AuditLogPurgeActor's daily cadence + continue-on-error posture: a purge fault is
logged and swallowed so the central singleton stays alive and retries next tick.
The purge timer is started in PreStart alongside the reconciliation timer and
gates on the same collaborators (pull client + enumerator) being available — the
repo-only test ctor injects neither, so neither background timer runs there.
Options: PurgeInterval (default 24h, clamped >= 1 min so a zero config value
can't spin the scheduler) + RetentionDays (default 365), plus a test-only
override that bypasses the clamp for millisecond cadences.
Tests (all in-memory, no live MSSQL): purge tick calls PurgeTerminalAsync with a
UtcNow - RetentionDays threshold (non-default 30 days); default retention yields
a 365-day threshold; a throwing repo does not kill the singleton (a second tick
still arrives).
Add a periodic reconciliation tick to SiteCallAuditActor that, per site,
pulls changed SiteCall rows since a per-site UpdatedAtUtc cursor and upserts
them idempotently (monotonic UpsertAsync) — the documented self-heal for lost
best-effort gRPC telemetry. Mirrors SiteAuditReconciliationActor's structure
(per-site cursor, per-site try/catch failure isolation, advance cursor by max
observed UpdatedAtUtc) minus the stalled-detection EventStream machinery.
Dependency wiring: add an acyclic SiteCallAudit -> AuditLog project reference
and resolve IPullSiteCallsClient + ISiteEnumerator (central-only singletons
registered by AddAuditLogCentralReconciliationClient) from the IServiceProvider
the production ctor already holds — no Host Props.Create change needed. The
repo-only test ctor injects neither collaborator, so the tick is gated off
there. A new public test ctor injects fake client + enumerator + repo so the
tick is unit-testable in-memory (public, not internal: Akka's ActivatorProducer
uses public-only reflection binding).
Options: ReconciliationInterval (default 5 min, clamped >= 1s so a zero config
value can't spin the scheduler) + ReconciliationBatchSize (default 500), plus a
test-only override that bypasses the clamp for millisecond cadences.
Tests (all in-memory, no live MSSQL): absent row is upserted on a tick; second
tick advances the cursor past already-pulled rows; one failing site does not
sink other sites; repo-only ctor does not start the tick.
Resolve all 622 issues flagged by the enhanced CommentChecker: add missing
<returns> tags (incl. the standard phrasing on non-generic Task methods),
add missing <summary> tags, and replace misused/redundant <inheritdoc/> on
members that override or implement nothing with real documentation.
Documentation-only — no behavior change; solution builds clean.
Joseph Doherty