The racing test encoded two timing assumptions and failed under full-solution
parallel load while passing in isolation. Both are removed; no production code
changed.
1. Assertion shape. failures.ShouldBeEmpty() assumed at most ONE apply-boundary
clear can land inside a single evaluation, so the evaluator's one-shot retry
always suffices. Under contention two can land and the retry legitimately
exhausts. The test now asserts the invariant the retry actually guarantees and
which is timing-independent: no raw disposed-script failure ESCAPES the retry.
Every failure must be the documented exhaustion outcome ("evaluator disposed
during evaluation") and never the pre-fix generic escape ("script threw:
Cannot access a disposed object..."), plus a deliberately loose bound on the
exhaustion rate so the check cannot go vacuous.
2. Workload shape. The fixed 12x400 evaluations against a fixed 45s WaitAsync
budget is CPU-dependent (every clear forces a recompile). Four concurrent
instances of this class reproduced a hard TimeoutException 4/4 - the likely
real shape of the reported load failure. The loops are now duration-bounded
(10s race window or 400 iterations, whichever first); the 45s WaitAsync stays
purely as a hang detector.
Renamed to Evaluate_racing_ClearCompiledScripts_never_leaks_a_raw_disposed_failure
so the name matches what it now asserts; the old name is cited in the doc-comment
for traceability.
Review round on 3e62f161. Two coverage gaps and three wording/comment nits.
IMPORTANT 1 — deregistration is now proven DATA-PRESERVING. The self-heal test
inserted a deployment_pointer row and rebooted unconfigured but never looked at
the row again, so a library regression that touched base-table data during
cleanup would have gone unnoticed — on the very tables that ARE the
boot-from-cache backstop, and on the default-OFF nodes least likely to notice.
The new DisablingReplication_PrunesBookkeepingButNeverTouchesTheCachedDataItself
asserts the row's CONTENT survives (not just its count — a count passes against
a cleanup that blanks columns) while __localdb_row_version and __localdb_oplog,
the bookkeeping deregistration does own, go to zero.
IMPORTANT 2 — mid-loop partial failure is documented and pinned. Both loops flip
four tables with no try/catch, so a throw on the third leaves two flipped and
propagates out of the AddZbLocalDb singleton factory, failing host start. That
is correct — a node that came up anyway would be silently capturing on an
arbitrary subset — and both library calls are idempotent, so the retried boot
completes the flip. Written into OnReady's remarks and pinned by
AFailureMidwayThroughRegistration_FailsBootLoudly_AndTheRetriedBootConverges,
which drives OnReady through a delegating ILocalDb that throws on the Nth
RegisterReplicated, asserts the throw propagates, asserts the wrapper actually
reached that call, then re-runs OnReady unwrapped and asserts all four converge.
MINOR 3 — the Boot helper now records that OnReady's null-logger fallback lands
on Serilog's static Log.Logger, which this assembly never assigns; assertions
about logging pass an explicit logger.
MINOR 5 — log wording: "are both unset" -> "are both absent — unset or blank",
matching IsNullOrWhiteSpace.
MINOR 4 — LocalDbDeploymentArtifactCacheTests said its setup "mirrors
LocalDbSetup.OnReady"; it reproduces the two tables only and registers them
unconditionally without baselining. Softened to say so and to name the
divergence as intentional and inert there.
Falsified both new pins. Wrapper set to failOnCall: 99 (never throws) => the
partial-flip pin is the SOLE failure, so it genuinely exercises the Nth-call
throw path rather than some incidental exception. Predicate forced constant-true
=> five pins red including the new data-preservation one. Both mutations
reverted.
LocalDbSetupTests 15/15. Host.IntegrationTests 226 passed / 0 failed / 5
skipped. Runtime.Tests LocalDbDeploymentArtifactCacheTests 12/12.
LocalDbSetup.OnReady registered all four replicated tables unconditionally on
every driver node, replicating or not. That was not merely wasteful (an
unreplicated node paid two INSERTs plus a json_object of the whole row inside
every write transaction, for an oplog nothing ever drained) — it made
baselineExistingRows unusable, so enabling replication on a long-running
default-OFF node converged on SUBSEQUENT WRITES ONLY and the rows already in the
file stayed invisible to the peer, silently and forever.
Registration now follows one predicate, LocalDbSetup.ReplicationIsConfigured:
LocalDb:Replication:PeerAddress OR :ApiKey non-blank. The OR is load-bearing —
only the dialling half of a pair sets PeerAddress, so keying on it alone would
strip capture from every passive node and the pair would converge in one
direction only. SyncListenPort deliberately does not join it: the passive
endpoint is fail-closed on the key, so a port with no ApiKey syncs with nobody.
configured => RegisterReplicated(table, baselineExistingRows: true), per
table, in the unchanged load-bearing DDL->register order
unconfigured => DeregisterReplicated(table), an idempotent boot self-heal that
drops the capture triggers an earlier build left behind and
prunes that table's oplog/row-version rows. One Information
line, only when it actually cleaned something.
PAIR SYMMETRY: the sync handshake compares registered-table digests fail-closed,
so flipping either key is a both-nodes-together change — set it on both and
restart both. Documented in CLAUDE.md, docs/Configuration.md and the replication
runbook (which claimed "this repo never calls DeregisterReplicated").
Verified against the pinned LocalDb 0.2.1: RegisterReplicated(string, bool) and
DeregisterReplicated(string) -> bool both exist; baselining seeds
__localdb_row_version at HLC 0 and the deregister return value distinguishes
"cleaned" from "nothing to clean". No pin bump needed.
Tests: LocalDbSetupTests gains pins for both branches (configured via
PeerAddress alone and via ApiKey alone; unconfigured => no tables, zero
triggers; late-enable => an HLC-0 baseline ledger entry; the log line fires once
and only when something was cleaned) — four went red against the old
unconditional code, and all four go red again under a constant-true predicate.
LocalDbWiringTests splits its exact-set pin into a replicating graph and a
default-OFF one. LocalDbPairHarness and AlarmSfLegacyMigratorTests built their
database from LocalDb:Path alone while configuring the replication engine from a
second config — production reads one configuration, so the two are now aligned.
Host.IntegrationTests: 224 passed, 0 failed, 5 skipped.
The batched ingest's append-failure catch had zero coverage, and two doc
statements overstated the loss: the Warning said the burst's values were
"dropped", and RecorderStatus.OutboxAppendFailures said "the value was
dropped, not recorded". The outbox contract is prefix-never-orphan, so a
partial-prefix failure leaves entries durably queued that drain normally.
- New test pins the branch: one 3-item burst whose group commit commits a
1-entry prefix then throws => OutboxAppendFailures == 1 (per burst, not
per value), TotalRecorded == 0, actor alive, and the prefix still reaches
the writer via the post-failure DrainTick (drainInterval 30s, so the
steady cadence cannot account for it). Falsified twice: counting per item
and dropping the drain nudge each turn it red.
- Second test pins the OperationCanceledException filter from its
observable side - a cancel that is NOT shutdown is metered, not swallowed.
Removing the 'when (_lifetime.IsCancellationRequested)' turns it red. The
shutdown side has no cheap seam (the counter dies with the actor).
- Docs corrected: OutboxAppendFailures counts FAILED BATCH APPENDS, one per
burst whatever its size (a granularity change from the per-value era), and
is not a count of values lost; TotalRecorded under-counts when a commit
fails after a partial prefix. Log message says "failed to append" and
names the prefix. No functional change beyond the message string.
Also folds in the DrainBatchSize default alignment behind 0298a0b9: the
recorder's two ctor defaults and its non-positive fallback still said 64
while ContinuousHistorizationOptions.DrainBatchSize now defaults to 1024.
No test pinned the old 64.
Joseph Doherty