Final review M2: HandleDeployArtifacts now sends ApplyArtifactDataConnectionsToDcl after the
shipped secrets are written and mirrored (from a finally, so the push still goes out if the
secrets step fails), so a connection created by the same deploy resolves a newly shipped
secret on first connect. Definitions are still stored at the same point; response semantics
unchanged.
Final review I1: a name already ledgered at the shipped revision is now read back every
pass; a missing, tombstoned or edited local row is healed through the mirror-owned re-stamp
path (bump, apply, read back, invalidate) before verify, instead of a local delete being
misreported as a KEK mismatch that blocked every later deploy. Drift logs Warning once per
occurrence. M4: row_json that cannot be parsed becomes a Failed entry for that name via the
new SiteStorageService.GetDeployedSecretsTolerantAsync; an unreadable removed row is still
honoured by name. M10: invalidator ctor doc corrected.
Fix round 2 (review task-9-rereview-1.md), both Minors:
1. Clock skew. ApplyAndConfirmAsync now decides the tombstone-vs-out-of-band split by LEDGER
OWNERSHIP (any mirrored_secrets entry for the name, at any revision), not merely by whether
the local row happens to be a tombstone. A live local row this node already owns but whose
ciphertext no longer matches the newly shipped revision (e.g. an earlier tombstone-recovery
re-stamp landed ahead of a since-shipped central UpdatedUtc, under site/central clock skew)
is now re-stamped and re-applied exactly like a tombstone, instead of being misreported as
someone else's "newer local value" and left permanently un-converged. The out-of-band Failed
message is now only ever produced for a name that is NOT in the ledger. New test:
Reconcile_ClockSkewedRecoveredRow_StillConvergesOnNextCentralRevision (a skewed future
re-stamp followed by a genuinely new central revision converges and re-invalidates the
cache).
2. Real-cipher integration test. Added DeployedSecretsMirrorRealStackTests, running the mirror
against the REAL ZB.MOM.WW.Secrets stack (SqliteSecretStore + AesGcmEnvelopeCipher +
DefaultSecretResolver over a real temp-file SQLite database, plus a hand-written
IMasterKeyProvider test double mirroring the library's own) instead of in-memory fakes:
encrypt -> SecretArtifact -> apply -> delete (tombstone) -> re-add the same artifact -> the
recovered row resolves to the original plaintext. Pins the "UpdatedUtc/Revision are not
AEAD-bound" property against the real cipher rather than by source inspection alone.
Test-only package reference: ZB.MOM.WW.Secrets (SiteRuntime itself still depends only on
ZB.MOM.WW.Secrets.Abstractions).
Verify: dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter FullyQualifiedName~DeployedSecretsMirrorTests (25/25); full SiteRuntime.Tests suite 878/878.
Fix round 1 (review task-9-review.md), all rulings:
1. (Important 1) ApplyReplicatedAsync is no longer trusted blindly. Every apply is
confirmed by a store.GetAsync read-back before ledgering: a tombstoned local row
(routine remove-then-re-add) is safely re-stamped past the tombstone and re-applied
(confirmed neither UpdatedUtc nor Revision is AEAD-bound in AesGcmEnvelopeCipher, so
the bump cannot break decryption); a live local row with different ciphertext (a
strictly-newer out-of-band secret) is left untouched, never ledgered, never verified
against, and reported with a distinct Failed message so a later central tombstone can
never delete it via this mirror.
2. (Important 2) Per-row try/catch (rethrow only on ct.IsCancellationRequested) so one
poisoned row (e.g. an unrecognised ContentType from a newer central) no longer blocks
every row after it.
3. (Important 3) Optional ISecretCacheInvalidator, invalidated after every successful
apply/delete and before verify, so a rotated/removed secret's plaintext cannot be
served stale for the remainder of the resolver's TTL.
4. (Minor 4) Loop and VerifyAsync OperationCanceledException handling now filters on
ct/stoppingToken.IsCancellationRequested specifically, rather than swallowing every OCE.
5. (Minor 5) Class remarks note the mirror assumes a node-local ISecretStore; Task 11
will reject SqlServer secrets replication on the Site role.
6. (Minor 6/7) Added cancellation + concurrent-callers tests, dropped the redundant
FailFor (tests now use ThrowFor/ReturnNullFor to exercise distinct decrypt-failure
paths), and Dispose() now disposes the gate SemaphoreSlim.
Verify: dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter FullyQualifiedName~DeployedSecretsMirrorTests (24/24); full SiteRuntime.Tests suite 876/876.
Script file access (design docs/plans/2026-09-28-script-file-access-design.md): Files.ReadTextAsync /
ReadLinesAsync / ExistsAsync on site ScriptGlobals (+ Instance.Files), backed by the site-singleton
ScriptFileReader. The one host-mediated exception to the System.IO ban - scripts still never name a
System.IO type. OFF by default: ScadaBridge:SiteRuntime:ScriptFileAccess:AllowedRoots (empty = every call
throws Disabled) + MaxFileBytes (1 MiB, validated 1..64 MiB, roots must be absolute). Guard: fully
qualified, no device paths / ADS, canonical path strictly inside a root (directory-boundary match), no
reparse point on the file or any directory below the root, size cap before and during the read.
Errors are one Commons type, ScriptFileAccessException(+ScriptFileAccessFailure); Commons.Types.Scripts
joins the default script imports on every surface so it binds by short name. Mirrored on
ScriptCompileSurface (CompileFiles, parity-pinned) and the Test Run sandbox (binds, refuses Disabled).
Inbound (central) scripts do not get it.
Rig: docker/recipes (sample RecipeValueName,RecipeValue files incl. an invalid one) bind-mounted
read-only at /app/recipes on both site-a nodes with AllowedRoots [/app/recipes]. The TestMesMachine
SimDelmiaRecipe simulator now reads + parses the downloaded file, converts each value to its recipe tag's
type and loads it all-or-nothing (14 new TestDelmiaReceiver attributes); verify-mes-test-machine.sh
group A covers accept / re-download / invalid lines / NotFound / OutsideAllowedRoots, VERIFY_GROUPS
selects groups (rig: AD).
A null resetValue encoded to null, which wrote an empty string and then handed the
waiter a null target (= any change), fast-pathing on the stale cache and reinstating
the race the overload exists to close. The context parameter is now non-nullable and
the accessor throws ArgumentException before any write. Also: additive-timeout note on
all surfaces, batch-failure message no longer names the bool overload, and the
error-message/batch-failure branches are pinned.
Claude-Session: https://claude.ai/code/session_01XPsuy2ZZ5Gbyfpt8PrK5BV
Second slice of the 2026-09-09 whole-system review remediation
(docs/plans/2026-09-09-arch-review.md §10), on top of the High slice 146a1323.
Runtime/devices: SR-2 alarm on-trigger integration scope; SR-3 correlated NativeAlarmActor
subscribe retry; SR-4 immutable audited DB target; SR-5 ordered InstanceActor stop/flush;
SR-12 rehydration fence; SR-13 S&F gate recheck per delivery; DC-4 no OPC UA security
downgrade; DC-5 alarm snapshot for every new subscriber; ORCH-1 sync-dispose joins the pump.
Management/UI: MG-2 site scope on audit tree + Flow Debugger Replay; MG-3 stale-probe memo
removed; MG-4 server-owned bulk-deploy deadline; UI-5 sandbox runs in a killable child
process (new CentralUI.SandboxWorker project); UI-14 bounded sandbox output/IPC.
Edge/audit: EI-1 relay audit ownership; EI-2 SkipBodyCapture suppresses error text; EI-3/EI-11
error-field redaction + caps; EI-4 single relay deadline, pre-execution-only retry; EI-5
authenticated relay provenance; EI-10 body cap before audit read.
Persistence/contracts: DL-1 global Instance.UniqueName unique index + conflict translation;
DL-2 DataConnection protocol config encrypted at rest (migration
20260909203046_EnforceGlobalInstanceNamesAndProtectProtocolConfiguration — coordinated
central upgrade, old binaries cannot read protected rows); DL-3 bounded Replay paging;
DL-6 cross-site Overwrite of a deployed instance rejected; XC-4 proto descriptor contract test.
Config/hygiene: XC-1 central listener isolation by local port; XC-2/XC-3 inert SMTP + cron
keys removed from rig settings; CM-2 env2 bootstrap guard enabled; DC-6 deploy site exit code;
DC-7 notifier retries only explicit standby rejection; XC-5 ClusterClient purged from docs.
Gates recorded in the report: build 0/0, 9 397 passed / 0 failed, rig redeployed on all
eight nodes with the migration applied. 46 Low findings remain open.
Task 5 code-review follow-ups:
- I1: ExternalSystemHelper gates its TraceStepScope on a new isTraced ctor arg
(default false for direct construction), not on `_traceSink is not null` — a
traced run without an inline collector still writes durable ScriptLog rows
that must name their step.
- I2 (design §5): an inline shared script now SHARES its caller's ScriptLogger
via the new `sharedLog` ctor param — one line cap, one anchor, one closing
row. Consequence documented on the Log property: a shared script's lines
carry the PARENT run's ExecutionId and land on the parent's ScriptRun node.
- M3: TraceCapturingLogger swallows a throwing formatter — most of what it
captures is best-effort "audit write failed" warnings logged from inside a
catch block, and a formatting exception must not escape that path.
- Tests: ApiCall step attribution, traced-without-collector, shared-script
logger sharing, nested batch/wait scope attribution + unwind, throwing
formatter; test (c) now also pins the anti-recursion property (EmitLog must
log through the undecorated logger).
CompileSurfaceParityTests stays red until Task 7 mirrors Log on ScriptCompileSurface.
The single-writer queue (d91135f9) made RecordEnqueueAsync return on enqueue, so the
test's verifier-connection UPDATE raced the pump and touched zero rows whenever the pump
lost. Every other verifier-connection test already flushed; this one was missed.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
A traced run refused before launch — by the per-script in-flight cap
(MaxConcurrentRunsPerScript) or by MinTimeBetweenRuns — emitted no ScriptTrace row at
all, so the Flow Debugger reported NO_TRIGGER_WITHIN_WINDOW ("no value-change trigger on
this attribute, or no device echo"), which is wrong in both cases and sends the operator
to check the wrong things.
Both guards now leave a Failed ScriptTrace/ScriptRun row naming the guard that refused
the run, parented to the writing execution and carrying the trigger detail. The new
ScriptRunLauncher.EmitSuppressedRun reuses EmitScriptRun with elapsedMs 0 and no
collector (a shed run has no reply to ride on — the row reaches the debugger via
PullTraceEvents / Replay), resolving the writer and identity providers from a fresh
scope because it runs on the actor thread. Best-effort, never throws, no-op untraced.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
Every emit site hands FlowTraceCollector the same AuditEvent it hands the
writer, but redaction and payload capping lived inside the writers, which
return a NEW filtered record — so the inline copy that rides
ScriptCallResult.TraceEvents / RouteToCallResponse.TraceEvents was the RAW
event: no per-target SQL-parameter redaction, no header scrubbing, no 8 KB /
64 KB truncation. Live and Replay of the same execution could therefore show
different payloads, and the 500-event-only cap left the reply free to breach
the 4 MB gRPC receive limit — which would fail a real call, since the same
reply carries the script's return value.
The collector now applies the host IAuditRedactor itself (outside the lock;
the writers keep their own idempotent Apply) and enforces a byte budget
(DefaultMaxBytes = 1 MB over the redacted DetailsJson) alongside the event
cap, flagging Truncated when either trips. It moves to Commons
(Types/Debugging) so the central FlowDebugRunner can share it.
The redactor is a singleton, so ScriptRunLauncher resolves it from the ROOT
provider and keeps constructing the collector before the try — the traced
ScriptRun row is emitted from the catch blocks too, including the
shed-at-dequeue path that throws before the DI scope exists.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
The Flow Debugger trace arm was recorded when a traced write was ISSUED, not
when the DCL acknowledged it. A write the device rejected (or that timed out)
left the arm standing for the whole TraceArmWindowSeconds (30 s), so the next
unrelated subscription update on that attribute was stamped with the debugger's
execution id and spawned a traced, parented run — the debugger drew a "device
echo" of a write that never landed.
The write's continuation runs off the actor's dispatcher, so it cannot touch
_traceArms directly; it Tells self a new DisarmTrace message instead, handled in
a Receive<>. The disarm is keyed on the writer's execution id, so a late failure
cannot steal an arm a newer traced write has since taken.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
A Flow Debugger live run (execution 1130c01f) spent 108 of a 176 ms site script run
between the cached DB write completing and Notify.Send starting: the immediate-
completion telemetry awaited two autocommit SQLite transactions on the consolidated
LocalDb file (RecordAttemptAsync + RecordTerminalAsync), on the script thread, each a
WAL fsync on a Docker volume; the submit row was a third. The audit half of the same
forwarder was already a channel write, so the tracking half was the whole cost.
OperationTrackingStore is now a single-writer queue: Record*Async enqueue an ordered
work item and return; one background pump drains the bounded channel (8192, wait
backpressure - tracking never drops) and applies each batch (<=256) in ONE transaction,
falling back to per-row apply when a batch fails so a poison row cannot take its
neighbours. Preserved invariants: FIFO across enqueue/attempt/terminal; read-your-
writes (GetStatusAsync and ReadChangedSinceAsync barrier on every write queued before
them, skipped when idle) so Tracking.Status, the telemetry actor's status join and
central's PullSiteCalls never see a stale row; timestamps stamped when recorded;
PurgeTerminalAsync returns after the purge lands; DisposeAsync drains the queue before
closing; sync Dispose keeps its documented in-flight-writes-fail semantics. New public
FlushAsync() is the explicit barrier. Accepted: a queued-but-unapplied row is lost on a
hard crash (ms window); S&F buffer and audit rows are unaffected.
No interface, forwarder or ScriptRuntimeContext change. Tests: write returns while
another connection holds the DB write lock and lands once released (fails on the old
store: 31 s); reads see 100 ordered attempts + terminal; DisposeAsync drains 200
queued rows; write after dispose throws ObjectDisposed. Existing tests that poke the
SQLite file right after a store call now FlushAsync first.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
Review of Task 3: the sink invoke sat before WriteAsync/ForwardAsync inside the
best-effort try, so a throwing sink would have silently dropped the durable
row. GuardTraceSink wraps the delegate in every helper ctor (and Collect
guards EmitTrace); pinned by a throwing-sink test. ScriptRunLauncher no longer
allocates a collector for a traced run with no reply target (Nobody) — the
list would never be read. FlowTraceCollector.Complete pinned at unit level;
Snapshot documented as a non-closing peek.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
ScriptRuntimeContext caches a trace sink (null on an untraced run, so the
production path is byte-identical) and threads it into the ExternalSystem /
Database / Notify helpers and AuditingDbConnection→AuditingDbCommand (onAudit).
All eight outbound emit sites hand the SAME AuditEvent instance to the sink
first, then to the writer/forwarder. FlowTraceCollector gains Completed +
Complete(): ScriptRunLauncher closes the collector where it builds the reply
snapshot, so un-awaited calls finishing after the run returned never mutate a
list the reply was built from. Cap (500) unchanged. Plan 2026-09-06 D5.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
Flow Debugger (design 2026-09-05 §5.3): a traced SetStaticAttributeCommand on a data-sourced
attribute records (attribute → writer, expiry) for TraceArmWindowSeconds (new option,
default 30, validated > 0); the next DCL echo for that attribute is published with the
writer's SourceExecutionId + Trace whatever its value, and the arm is consumed once. Static
writes publish Trace directly. Untraced writes never arm (pinned).
Claude-Session: https://claude.ai/code/session_01GSLkkHKz9JrAtmJjmQK3cP
Flow Debugger (design 2026-09-05 §5.1/§5.4): ScriptRunLauncher emits one ScriptRun row per
traced run on every exit path (OccurredAtUtc = run start), replies with the run's
ExecutionId (always) and the collector snapshot (traced only). ScriptActor forwards
ScriptCallRequest.Trace and, for a value-change trigger fired by a traced stamped change,
parents the run to the writer with trigger detail in Extra; every other trigger stays a root.
Claude-Session: https://claude.ai/code/session_01GSLkkHKz9JrAtmJjmQK3cP
Flow Debugger (design 2026-09-05 §5.1/§5.2): the context is the single funnel for
SetAttribute / WriteBatchAndWait / WaitInternal, so it emits the write/wait rows (collector
first, then the best-effort IAuditWriter whose redactor applies the payload cap), stamps
Trace on SetStaticAttributeCommand and ScriptCallRequest, and hands the flag + sink to the
inline CallShared child context. Untraced contexts emit nothing (pinned).
Claude-Session: https://claude.ai/code/session_01GSLkkHKz9JrAtmJjmQK3cP
Review finding on ab08ecd3: each Good re-armed the watchdog from 'now', so Goods arriving faster than the floor starved the retry. Pinned by FlappingGood_DoesNotDeferTheRetry (12 Goods over 1.2 s; naive re-arm measured 1.43 s, guarded ~0.3 s).
Claude-Session: https://claude.ai/code/session_019ZxXT5sxZs5XrtdwYzJDPV
Task 4 review fix-ups: RepeatedFailures gaps[1] lower bound 600→500 (two compounding jitter draws bottom out at 512 ms); SubscribeToDcl also clears _tagPathToAttributes and its prune is documented as defensive (PreStart is the only caller).
Also from the Task 3 review: FailedSubscribeResponse_IsRetried guards the failure path again (large ceiling); HandleSubscribeTagsResponse reads state instead of get-or-create; SubscribeToDcl prunes stale retry state on re-entry; doc comments refreshed.
Claude-Session: https://claude.ai/code/session_019ZxXT5sxZs5XrtdwYzJDPV
FailedSubscribeResponse_IsRetried and LostSubscribeResponse_IsRetried_AndSuccessStopsRetrying now pin the in-flight ceiling to the retry interval: a lost reply is re-sent only once the outstanding request is presumed lost, by design.
Claude-Session: https://claude.ai/code/session_019ZxXT5sxZs5XrtdwYzJDPV
HandleEnsureInstanceStarted, HandleDeploymentStateQuery, and HandleEnable each
called GetAllDeployedConfigsAsync — materializing every deployed config,
including every ConfigJson blob — just to FirstOrDefault one row. A
volume-restore reconcile healing hundreds of instances turned that into
hundreds of full-table reads.
Add SiteStorageService.GetDeployedConfigAsync(name) (single-row, full
projection incl. ConfigJson, for callers that (re)start an Instance Actor)
and GetDeployedConfigMetadataAsync(name) (single-row, ConfigJson-free
projection, for callers that only need DeploymentId/RevisionHash/IsEnabled).
Switch the three call sites; leave decisions/semantics identical.
HandleEnsureInstanceStarted gets its own _deployedConfigLoader seam,
deliberately separate from the existing _configLoader (still used by the
boot-time load, which legitimately needs every row) so a test overriding one
cannot accidentally gate the other. Generation fencing (StoreGenerationAtRead,
Medium wave T1) is untouched: the generation is still captured on the actor
thread before the read dispatches.
Tests: new SiteStorageServiceTests lock in that both projections return the
same values GetAllDeployedConfigsAsync().FirstOrDefault(...) did. The three
row-101 ensure-started race tests (ReconcileStartsInstanceTests) are ported
to the new seam and still pass, proving generation fencing survives the
projection. Proof of failure: since this is a pure I/O-cost fix with no
observable behavior change, the new/changed tests were confirmed to fail to
even compile against the unfixed code (stash/unstash) — GetDeployedConfigAsync,
GetDeployedConfigMetadataAsync, and the deployedConfigLoader ctor param don't
exist there.
P1-1 — the row-67 ensure-started self-heal re-read the durable row off-thread but
captured no _storeWriteGeneration and never ran IsStoreSnapshotStale, unlike the boot
load and the reconcile sweep. A Disable or Delete committing inside that window left the
result carrying an enabled config that passed every guard (neither command sets any of
the three pipeline markers ShouldAttemptEnsureStart re-checks), so the heal started an
instance the operator had just stopped — PERMANENTLY, since the sweep iterates store rows
only and has no runtime→store stop leg. The generation is now captured at dispatch,
threaded on EnsureInstanceStartedResult and checked on landing, and the comment claiming
the stored row already covered this is corrected. The heal reads through _configLoader,
the same seam the boot load and sweep use (identical production call), so a test can hold
the read open across a committed disable/delete.
P1-2 — the "static overrides reset on redeployment" invariant was racy in both
directions. ApplyDeployment created the successor Instance Actor first and only then
dispatched the clear, queued behind a full config-JSON write, so the successor's PreStart
override load routinely won and applied the PREDECESSOR's overrides — keys absent from
the new config included — to in-memory state for the actor's lifetime while the disk
ended cleared; a later restart then silently flipped the values. The clear is now
dispatched BEFORE CreateInstanceActor and handed to the actor as a gate its override load
chains behind; the persistence task joins the same task, so a failed clear still fails the
deploy exactly as before. Leg 2: the predecessor's PostStop flush was fire-and-forget and
could commit up to one coalescing window AFTER the clear (the Deployment Manager gates the
clear on Terminated), durably resurrecting overrides — PostStop now waits for it, bounded
at 5s with a Warning on overrun.
Tests: new RedeployStaticOverrideResetTests (5) pins both legs plus the gate mechanism and
first-deploy behaviour; three new cases in ReconcileStartsInstanceTests pin the discarded
disable/delete heals and that an unraced heal still starts. InstanceActorTests'
StaticOverride_ResetOnRedeployment comment claimed to simulate an ordering the production
path did not have — corrected.
Alarm on-trigger runs (ScriptRunLauncher.RunAlarmScriptAsync) bypassed the
stuck-script watchdog, the script-error health counter, and
ScriptRunSummaryRecorder entirely, even though they run on the same bounded
ScriptExecutionScheduler pool as instance scripts. A wedged alarm body cost a
pool thread forever with no site event and no detach/replace, and a timed-out
or failed on-trigger run was invisible to health monitoring and the run
summary.
Bring the alarm path to parity with the instance-script path (RunScriptAsync):
capture the worker slot/run stamp in the first synchronous segment and arm
ArmStuckScriptWatchdog with the same detach-and-replace semantics and gauge;
resolve ScriptRunSummaryRecorder/ISiteEventLogger from the same DI provider and
thread RecordStarted/RecordCompleted/RecordFailed/RecordTimedOut; increment
ISiteHealthCollector.IncrementScriptError on both the timeout and general
failure catch blocks. LaunchAlarmScript/RunAlarmScriptAsync now take
healthCollector and serviceProvider parameters, passed from AlarmActor's
existing _healthCollector/_serviceProvider fields. No change to the
enqueue-anchored deadline, the shed cap, or "a stop does NOT cancel an
in-flight run" semantics.
Adds AlarmScriptWatchdogTests (previously zero coverage of LaunchAlarmScript):
wedged-worker detach + replacement, timeout-at-dequeue health counter
increment, and successful-completion summary recording. All three fail
against the pre-fix code (verified via stash) and pass after.
ReadStoreSnapshotAsync's snapshot is read off the actor thread and consumed only
when its piped message lands. Row 101 guarded the DEFINITIONS half of that race;
the CONFIG half was left to machinery whose reach is time-limited, and both legs
of the instance reconcile sweep were reachable through it:
- Row-72 leg: a read dispatched before a deploy's persist, landing after the
deploy completed AND after the pipeline released ownership, carries the
revision the deploy replaced. TryRestartStaleInstance then "repairs" the
instance BACKWARD, re-deploying the snapshot's own superseded ConfigJson.
InstanceIsOwnedByADeployPipeline covers only in-flight deploys, and the
per-revision claim has never seen the old revision, so nothing stopped it.
- Row-78 leg: a read dispatched before a delete/disable commits still shows an
enabled row while the actor is already stopped, so StartStoredInstance
resurrects it — and delete clears _sweepAttemptedRevisions, freeing the claim.
Both verified real by neutralising the freshness check and re-running the new
tests (the fail-before site event literally reads "running revision rev-2 is
stale, applying stored revision rev-1"; the deleted instance came back).
_definitionsWriteGeneration becomes _storeWriteGeneration, covering both tables
(the snapshot is one consistent read and is discarded whole, so one counter is
the right shape); DefinitionsGenerationAtRead -> StoreGenerationAtRead on both
internal records. The generation is now bumped at BOTH ends of every mutation —
when the off-thread write is dispatched and when its completion message lands —
because the config paths advance actor state before the write, so an
initiation-only bump would let an in-window read look fresh. Disable, delete and
the init-failure rollback have no actor-thread completion message of their own,
so a minimal StoreWriteCompleted self-message carries the bump for them.
CDC-replicated writes, snapshot resyncs, restored files and
SiteReconciliationActor's fetch deliberately do NOT bump: discovering exactly
those is the sweep's reason to exist.
A stale sweep result is now dropped WHOLE (no overlay apply, no start, no
restart). A stale boot load cannot be dropped — startup must finish — so it
re-dispatches LoadDeployedConfigs(); a fifth consecutive superseded read is
applied anyway with a Warning, and a late one arriving after the boot completed
is dropped rather than looping against the S5 retry timer.
Tests: three fail-before regressions (backward restart, resurrection, boot
re-read) plus two guard twins pinning that the legitimate row-72 restart and
row-78 start still fire on a current-generation snapshot. The row-101
definitions tests keep passing; FreshEmptySweepSnapshot's hardcoded generation
moves 1 -> 2 now that one artifact apply bumps at both ends.
Docs: Component-SiteRuntime.md's stale-snapshot paragraph now describes the
whole snapshot (both halves, both consumers, the boot re-dispatch, the
CDC-no-bump rule), the components mirror follows, and the arbitration memo gains
§11 extending §10 to the config half.
An off-thread snapshot read racing an artifact apply could land after the
apply populated the overlay; its stale absences then tombstoned and stopped
just-applied LIVE connections, which nothing re-creates until an instance
redeploy (the removal evicts the hash slot the overlay push is gated on).
Each read now captures a definitions-write generation at dispatch; a stale
result is discarded whole (additions too — a stale row would reconfigure a
live connection backward). Deterministic repro fails before the fix; a
fresh-generation guard test pins that row-90 disappearance still fires.
Joseph Doherty