4cfd9337a384abf9460d4110d8a1c748394689d8
1604 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Joseph Doherty
|
25c15dd164 |
ops(docker)+test(host)+docs: close register rows 104 + 105 — offline VACUUM recipe applied to the rig; build-time pin that no replicated table is a cascading-FK parent; record row 106 (LocalDb as_of_seq=0 full-snapshot loop)
Row 104: docker/vacuum-site-localdb.sh stops both nodes of a pair together, sets auto_vacuum=INCREMENTAL + VACUUM on each node volume with integrity_check before/after, then starts both (DRY_RUN=1 reports). Run on all three rig pairs: six files 0 -> 2, ok, ~11% smaller; site-a replication reconnected with backlog 0 and no watermark change — VACUUM is replication-invisible (ledger keys by declared PK, oplog seq is INTEGER PRIMARY KEY AUTOINCREMENT). Topology guide gains the recipe. Row 105: SiteLocalDbReplicatedTableFkGuardTests builds the real site schema via OnReady (unreplicated config so 0.3.3's own guard cannot mask the culprit) and asserts every SiteLocalDbSetup.ReplicatedTables entry (now internal) exists and has no inbound CASCADE/SET NULL/SET DEFAULT foreign key, with a negative control. The constraint is enforced at build instead of at the first replicated boot. Row 106 (found during 104's live verification, NOT caused by it — the identical 127k-row full snapshot also landed at the earlier deploy restart): SnapshotStreamer.SendAsync stamps as_of_seq = MAX(seq), which is 0 once a converged sender's oplog is ack-pruned to empty; a zero-watermark receiver (rebuilt node) is never lifted, so the empty-oplog horizon rule re-streams a full snapshot on every reconnect. Sender-side one-line fix for LocalDb 0.3.4: as_of_seq = max(MAX(seq), last_acked_seq). |
||
|
Joseph Doherty
|
57c8973b0d |
fix(centralui)+test(centralui): Low-wave integration — T12 Topology race fixture gains T13's IStaleInstanceProbe dependency; T11 sandbox-run continuation skips StateHasChanged after Dispose
Cross-task regression caught by the combined sweep: TopologyLiveUpdatesTimerDisposeRaceTests (T12) builds a real DeploymentService, whose constructor grew an IStaleInstanceProbe parameter in T13 (P7-6 up-to-date pre-filter) — 4 DI-resolution failures on the integration branch that neither task saw in isolation. Registered a substitute, matching TopologyPageTests. Bundle review Minor: ApiMethodForm/SharedScriptForm copied TemplateScriptDialog's sandbox pattern exactly, whose finally re-renders unconditionally; the plan asked the continuation to be guarded on disposal, so a _disposed flag now suppresses the post-cancel StateHasChanged (the reference dialog keeps its original shape — out of T11's scope). |
||
|
Joseph Doherty
|
b34ec10f8b |
fix(deploymentmanager): bulk site deploy skips up-to-date instances and un-poisons the shared DbContext (arch-review Low P7-5 + P7-6)
P7-6 (behaviour fix, plan decision 1): DeploySiteAsync redeployed EVERY deployable instance unconditionally. A redeploy is not idempotent at the site — it resets static attribute overrides and clears native alarm state by design — so one site-wide push destroyed live operational state on every unchanged instance in the batch, which is normally most of them. Bulk deploy now pre-filters with IStaleInstanceProbe (the same seam the bundle importer and Deployments page use; its memo timing was fixed by the Medium wave's P7-4): an instance whose freshly computed revision hash still matches its DeployedConfigSnapshot is reported in the existing additive SkippedInstances list with reason "up-to-date" and never flattened, staged or sent. The filter is deliberately conservative — every uncertainty deploys: only an Enabled instance qualifies, a missing snapshot disqualifies, a prior record that ShouldQuerySiteBeforeRedeploy flags goes through prepare so the reconcile that HEALS it still runs, and a null/faulting probe means "indeterminate". New additive MgmtDeploySiteCommand.Force (default false = the new filtered behaviour) plus CLI `deploy site --force` restores the unconditional push for the case central cannot see (restored volume, rebuilt node). Composes with the Medium wave's P7-2: Disabled wins — force suppresses the up-to-date filter only, never the Disabled skip. P7-5 (verified, then fixed): the cascade REPRODUCES. All three bulk phases share one scoped DbContext, and a failed SaveChanges does not untrack the offending entity — EF re-attempts it on the next save, including MarkDeploymentFailedAsync's cleanup write. Measured on a real SQLite-backed ScadaBridgeDbContext: a permanent save failure on instance 1's post-success side effects failed instances 2, 3 and 4 and left all three deployment records stranded InProgress. MarkDeploymentFailedAsync now detaches the pending changes (new IDeploymentManagerRepository.DetachPendingChanges) when the cleanup save fails and retries the Failed write once on the clean tracker, so later instances save cleanly and the record reaches a terminal status. The DeployFailed audit row moved outside the status-write try (best-effort, must not be lost because the status write failed). Tests: 5 new DeploySiteAsyncTests (skip + reason + no site round-trip; --force deploys; Disabled skipped under both force values; the three not-enough-to-skip guards), 2 new IntegrationTests over a real EF context (the poisoned-context cascade, and the narrow EF tracking fact the fix rests on), CLI --force flag parse test, and a MgmtDeploySite wire-default test. Proof of failure: both P7-6 behaviour tests and both P7-5 assertions fail against the unfixed code. |
||
|
Joseph Doherty
|
6d0d8b6258 |
fix(centralui): close register row 103 — take-once session/timer discipline for TransportImport + Topology (P6-6 lesser instances)
Mirrors the Medium wave's DebugView P6-6 fix (_sessionLock + _disposed, publish-then-recheck-disposal under one lock) for the two TTL-bounded/ unbounded lesser instances the Medium wave deferred: - TransportImport.razor.cs: TryLoadAsync awaits IBundleImporter.LoadAsync and used to publish the result into _session unconditionally. A circuit torn down mid-await left Dispose() with nothing to release (session still null), so the freshly-opened server-side BundleSession stayed registered, stranding one of MaxConcurrentImportSessions (register row 43) for the full session TTL. Now publishes and re-checks disposal atomically under _sessionLock, so exactly one side (TryLoadAsync or Dispose) abandons the session. - Topology.razor: OnInitializedAsync awaits LoadDataAsync before calling StartLiveUpdatesTimer. A circuit torn down mid-await left Dispose() with no timer to stop yet, so the timer StartLiveUpdatesTimer creates once the await resumes was never disposed — it ticked against a torn-down component every 15s until process restart. StartLiveUpdatesTimer/StopLiveUpdatesTimer/ Dispose now create/replace/take the timer atomically under _timerLock together with a _disposed check, so a late-arriving (re)start refuses to create a timer once disposed. Tests (bUnit, both proven to fail against the unfixed code): - TransportImportSessionDisposeRaceTests: dispose-during-load abandons the just-opened session immediately; dispose-after-successful-load abandons exactly once. - TopologyLiveUpdatesTimerDisposeRaceTests: a late StartLiveUpdatesTimer call after Dispose leaves no live timer; Dispose stops the timer exactly once (idempotent double-dispose); OnLiveUpdatesToggled(off) + Dispose leaves no live timer. All assert on deterministic field/call-order state, no sleeps. Register row 103 stays open for integration to close per the plan's shared constraints (register edits happen at integration only). |
||
|
Joseph Doherty
|
f1ef95f4da |
fix(centralui): P6-12 — dispose sandbox-run CTS in ApiMethodForm + SharedScriptForm
Neither Test Run form implemented IDisposable, so navigating away mid-run left _runCts uncancelled: the sandbox kept the run alive until its own internal script timeout, and the eventual continuation touched a torn-down component's _running/_runResult fields. TemplateScriptDialog already got this right — both forms now mirror it exactly: @implements IDisposable plus a Dispose() that cancels + disposes _runCts. ScriptAnalysisService.RunInSandboxAsync is marked virtual (test-only seam, mirroring the existing virtual on DebugStreamService.StartStreamAsync/ StopStream) so a TaskCompletionSource-gated stub can hold a run open deterministically. New bUnit tests (ScriptFormSandboxDisposalTests) prove, for both forms: disposing mid-run cancels the token the sandbox observed, and the cancelled continuation resolves with no unhandled exception and no _runResult/ _running mutation. Proof of failure: reverting only the two .razor changes (keeping the virtual seam) reproduces all 4 failures — the token is never cancelled and the no-op assertions time out. |
||
|
Joseph Doherty
|
21d50d4406 |
fix(centralui): P6-11 — request-generation guard on Deployments page
Three fetch initiators (search debounce, paging, push/coalesce reload) could each have a LoadDataAsync flight in flight at once; completion order was not call order, so a slow, stale flight could resolve after a newer one and clobber the grid the operator was already looking at. Add a monotonic _requestGeneration counter, bumped at fetch start and checked at apply — last-write-wins by generation, not by whichever await resolves first. Separately, the filter-debounce timer wrote `_currentPage = 1` directly on its raw System.Threading.Timer callback thread — a Blazor threading violation, since component state may only be mutated on the renderer's sync context — and that write was not covered by the generation guard at all. LoadDataAsync now takes a resetToFirstPage flag, resolves the target page into a local, and applies it only behind the same generation check as every other piece of state, via a new DispatchFilterReloadAsync that marshals through InvokeAsync like the existing push-reload path. Tests (DeploymentsRequestGenerationTests, bUnit): a delayed stale flight arriving after a newer one does not overwrite the grid; a stale filter-debounce flight does not reset the page after a newer navigation. Both driven deterministically via TaskCompletionSource-backed repository stubs. Verified to fail against the unfixed code (swapped via `git show HEAD:<path>`, not `git stash` — repo-wide and unsafe across the parallel worktree wave). |
||
|
Joseph Doherty
|
7ad1872454 |
fix(management): strict enum parsing for DataType/ElementDataType/AlarmTriggerType (P6-10)
ManagementActor's ParseDataType/ParseElementDataType (bare Enum.TryParse) and
the two AlarmTriggerType parse sites in HandleAddAlarm/HandleUpdateAlarm (raw
Enum.Parse) all admitted ordinals ("2"), undefined values ("99" — a
deploy-time bomb that saves fine and only detonates at deploy), and
comma-joined name lists (bitwise-ORed together regardless of [Flags], e.g.
"String,Int32" silently becoming DateType.DateTime). A typo against the raw
Enum.Parse sites also surfaced as a masked "An internal error occurred"
rather than a curated field-naming error.
Adds ParseEnumStrict<TEnum>, mirroring the existing ParseDeploymentStatus
guard (names only, defined values only, comma-token rejection per the
Medium-wave T8 nuance / AuditQueryParamParsers.ParseEnumListStrict pattern),
and routes all four call sites through it.
Tests: 22 new ManagementActorTests cases per parse site (ordinal, undefined
numeric, comma-joined, and typo all rejected with the curated field-naming
error; every defined name still parses through to the service layer).
Proof of failure recorded in each rejection test's doc comment; verified
live by stashing the ManagementActor.cs fix — exactly the 14 new rejection
tests fail pre-fix (13 acceptance tests pass unchanged), confirming the
defect and the fix.
|
||
|
Joseph Doherty
|
88fbc9e351 |
fix(auditlog): P5-6 — deterministic synthetic ReconciliationAbandoned EventId
SiteAuditReconciliationActor minted a fresh random EventId for the synthetic ReconciliationAbandoned row every time a permanently-abandoned row was re-abandoned. Because the reconciliation cursor is held at `since` by ANY co-batch unresolved row (see PullSiteAsync's hasUnresolvedFailure gate), a poisoned row's per-EventId attempt counter restarts every time it reappears in a stuck batch, so it gets re-abandoned roughly every MaxPermanentInsertAttempts ticks — minting an unbounded number of duplicate central audit rows for the SAME lost event. Fix: DeriveAbandonmentEventId(Guid lostEventId) derives the synthetic EventId deterministically (SHA-256 over a fixed namespace GUID + the lost row's bytes, RFC4122-v5-shaped). Re-abandonment of the same lost row now always produces the same synthetic EventId, so IAuditLogRepository.InsertIfNotExistsAsync's existing idempotency collapses every repeat into the first insert. Also adds an in-memory _abandonedEventIds set as a CHEAP SHORT-CIRCUIT ONLY (skips the wasted ~25 min retry-then-abandon cycle for a row already known abandoned this process's lifetime) — it is not the correctness mechanism; it resets on singleton restart/failover, at which point the deterministic id alone still keeps re-abandonment idempotent. Tests (SiteAuditReconciliationActorTests): - ReAbandonment_SameLostEventId_YieldsExactlyOneCentralRow: end-to-end repro of the defect trigger (poison row + a rotating-EventId co-batch row that never itself reaches the threshold, holding the cursor forever) across two full abandon cycles. Proof of failure: run against the unfixed source (git stash of the source file only, pure-function tests temporarily #if false'd to keep the file compiling), the test failed with exactly the predicted defect shape — 2 distinct synthetic rows for one lost EventId. Passes post-fix (1 row). - DeriveAbandonmentEventId_DifferentLostRows_YieldDistinctSyntheticIds and DeriveAbandonmentEventId_IsPureAndStableAcrossIndependentInvocations: pure-function coverage of the derivation directly, including a pinned golden-value regression guard proving it is a function of the input GUID's value alone (no clock/randomness/process state). |
||
|
Joseph Doherty
|
abbbcd1a8d |
fix(kpihistory)+fix(notificationoutbox)+fix(sitecallaudit)+fix(host): P5-5 — rate-fold interval triple made self-consistent + cross-validated
Defect: the hourly KPI rollup fold (KpiHistoryRepository.FoldHourlyRollupsAsync) sums every Rate-classified sample recorded in an hour, correct only if each sample is a non-overlapping delta covering exactly one recorder tick (KpiHistoryOptions.SampleInterval). That invariant was an unenforced silent assumption across three independently-owned config keys — SampleInterval, NotificationOutboxOptions.DeliveredKpiWindow, SiteCallAuditOptions.KpiInterval — so widening DeliveredKpiWindow to e.g. 15 min would silently over-count the rollup ~15x. Robust fix: NotificationOutboxKpiSampleSource and SiteCallAuditKpiSampleSource now rescale their Rate-classified samples (deliveredLastInterval, failedLastInterval) by SampleInterval / (DeliveredKpiWindow or KpiInterval) before recording, so the rollup's per-hour Sum is correct regardless of how the other two keys are tuned — the live dashboard tile's own cutoff/window is untouched. Both sources gain a project reference to KpiHistory to read SampleInterval; no cycle (KpiHistory references only Commons). Eager cross-validation (defense-in-depth): KpiRateFoldIntervalCrossValidator (Host, the one composition root referencing all three component projects) fails boot naming all three config keys when they diverge, protecting the separate (still-real) invariant that the recorded/charted KPI-history sample equals the live tile at the same instant — which the rescale trades away whenever the scale factor isn't 1. Rides KpiHistoryOptions' existing ValidateOnStart trigger. Also: NotificationOutboxOptions had no registered validator at all before this (the gap this task's "whichever options class owns DeliveredKpiWindow" file scope pointed at) — added NotificationOutboxOptionsValidator mirroring SiteCallAuditOptionsValidator's positive-duration/count convention, wired with ValidateOnStart. Tests: new rescale tests in both sample-source suites prove the fold is numerically correct at a non-default SampleInterval (30/40 counted over a 15/20-minute window correctly divides down to a 1-minute-tick delta); new KpiRateFoldIntervalCrossValidatorTests prove divergent config fails with all three keys named; new NotificationOutboxOptionsValidatorTests. Every new test proven to fail against the unfixed computation first (temporarily short-circuited the scale factor / cross-check to the pre-fix behavior, confirmed failure, restored) — not via git stash (repo-wide, shared across concurrent worktrees this session) but via file-copy swap. Full suites green: Host.Tests (527), ConfigurationDatabase.Tests (401), KpiHistory.Tests (54), NotificationOutbox.Tests (163), SiteCallAudit.Tests (67). |
||
|
Joseph Doherty
|
1cea01d924 |
fix(notifications): list sort index + CAS operator Retry/Discard (P5-3, P5-4)
P5-3: add IX_Notifications_Created_NotificationId (CreatedAt DESC, NotificationId DESC) — the Central UI Notification Outbox list page sorts on exactly that pair, but no existing index leads unconditionally on CreatedAt, so an unfiltered/lightly filtered page fell back to a full clustered-index scan + sort on every COUNT + page request. Mirrors the Medium wave's IX_SiteCalls_Created_TrackedOperationId migration + SQL twin pattern faithfully, including the handwritten docs/plans/sql script. P5-4: operator Retry/Discard was read-check-write keyed on NotificationId only — a detached PipeTo task interleaving with the dispatcher's own write (or a second concurrent operator action) could land between the read and the write and get silently clobbered by the unconditional UpdateAsync, stamping a Delivered row Discarded with stale fields. Adds INotificationOutboxRepository.TryUpdateFromParkedAsync, a single-statement compare-and-swap (WHERE Status == Parked) mirroring SecuredWriteRepository.TryMarkApprovedAsync's CAS shape. NotificationOutboxActor's RetryAsync/DiscardAsync now use the CAS write and, on a lost race, re-read to report "notification already progressed to <Status>" instead of silently overwriting whatever is there now. Proof of failure: reverting only the actor's CAS usage while keeping the repository addition reproduces the silent-overwrite bug — 7 tests fail, including two new deterministic (non-sleep) race-proof tests that mock the exact GetByIdAsync -> lost-CAS -> re-read sequence the finding describes. P5-3 is perf-only; verified via migration/SQL-twin parity instead (no runtime proof-of-failure is meaningful for an index). |
||
|
Joseph Doherty
|
11675285af |
fix(siteeventlog)+fix(host): purge startup yield + auto_vacuum reality (arch-review 2026-08-19 P4-5 + P4-6)
P4-5: EventLogPurgeService.ExecuteAsync now yields (Task.Yield()) before running the initial RunPurge(), making "the purge never runs synchronously on the caller's thread" a property of the method itself rather than an artifact of .NET 8+'s BackgroundService scheduling (verified empirically not to reproduce the literal "StartAsync blocks" symptom on this repo's net10.0 target — a 1.5s synchronous Thread.Sleep in ExecuteAsync still lets StartAsync return in ~2ms). Proven instead via a reflection-invoked ExecuteAsync test that bypasses BackgroundService's Task.Run wrapper, which is timing-independent. Also: a throwing isActive predicate now fails CLOSED (logged as inactive, at Warning) instead of silently claiming "active" — the predicate only affects the tick's log line since P4-4, but a throw must not masquerade as a successful check. P4-6: PRAGMA auto_vacuum = INCREMENTAL only takes effect on a table-less database, and LocalDbSchema.EnsureCreated (inside SqliteLocalDb's constructor) always creates LocalDb's own bookkeeping tables before SiteEventLogSchema.Apply ever gets a connection — so the pragma there had never once taken effect, in any real call path. Real fix per plan decision 2: SiteLocalDbSetup.EnsureAutoVacuumOnFreshFile pre-creates a genuinely fresh LocalDb file with the pragma set, called from SiteServiceRegistration.Configure BEFORE AddZbLocalDb's lazy singleton ever constructs SqliteLocalDb — no LocalDb API change needed. Only helps NEW site databases; an existing file (every node on the docker rig today) is left untouched rather than VACUUM'd, per the plan's explicit rejection of an unconditional VACUUM against a live ~1 GB file. The now-dead pragma call in SiteEventLogSchema.Apply is removed and its comment corrected. Tests proven to fail against the unfixed code (git-diff-and-restore, not stash — this repo shares one stash ref across all worktrees): ExecuteAsync_YieldsBeforeRunningTheInitial- PurgeSynchronously and RunPurge_ActiveNodeCheckThrows_TickLogsAsInactive_NotActive fail without the EventLogPurgeService fix; SiteLocalDbAutoVacuumTests fails to compile without EnsureAutoVacuumOnFreshFile. SiteEventLogging.Tests 112/112 and Host.Tests 526/526 green with the fix applied. |
||
|
Joseph Doherty
|
99da32d635 |
fix(communication): P3-4 open-attempt deadline + P3-5 duplicate-subscribe cap order
P3-4 — SiteAlarmAggregatorActor pinned _openInFlight forever on a stream open that produced none of its three peer-driven outcomes. SiteStreamGrpcClient gives up on response headers after ConnectedHeaderTimeout (10s) and then treats the first received EVENT as its fallback connected signal, so an alarm-only site that stays quiet supplies neither — leaving _streamDown=true AND _openInFlight= true, which makes OnReconcileTick's reopen guard (_streamDown && !_openInFlight) skip every tick until the site's 4h max stream lifetime ends the call. Fix: bound one open attempt with a generation-fenced single-shot deadline (GrpcAlarmStreamOpenTimedOut, ctor-injected, production default 30s — above the 10s header timeout, below the 60s reconcile interval). Firing it releases the flag and nothing else: no retry-budget spend, no node flip, no liveness republish, and the abandoned subscription is left running so a merely-slow connect still comes up normally. Self-healing rather than reachability analysis. The deadline is cancelled on connect / error / graceful completion. P3-5 — SiteStreamGrpcServer cancelled the incumbent stream on a duplicate correlation id and only THEN checked the concurrency cap. Checking Count >= max after the removal is right at exactly the cap but wrong above it, and above it is reachable because the cap check is documented check-then-act: two subscribes that both clear it before either registers overshoot by one. In that window a routine reconnect killed a healthy stream AND was refused with ResourceExhausted, leaving the correlation with ZERO streams until its next retry. Fix: claim the removal, consult the cap only for a genuinely NEW correlation id, then cancel. A replacement is net-neutral on the count so it never consults the cap; a refused new subscriber cancelled nothing (TryRemove found nothing), so the incumbent survives. Newest-wins is unchanged — a fresh authenticated subscribe still displaces a dead/wedged incumbent. Tests (both shown failing against the unfixed code): - OpenAttemptThatNeverResolves_ReleasesTheInFlightGuard_SoTheTickCanReopen — fails (reopen never happens) with the deadline arming disabled. - OpenDeadline_DoesNotFire_OnAStreamThatConnects_NorFlipNodeOrBurnBudget — inertness lock on every healthy path. - AtCapOvershoot_DuplicateReconnect_IsNotRefused_AndLeavesExactlyOneLiveStream — fails with ResourceExhausted against the old ordering; the overshoot is driven deterministically by parking both openers on the ServerCallContext token read, which sits between the cap check and the _activeStreams insert. - AtCap_AGenuinelyNewSubscriber_IsStillRefused_AndTheIncumbentSurvives. The P3-4 test's doc comment carries the honesty note: the aggregator-side starvation reproduces exactly, but how often the >10s header delay that gets you there occurs in production is NOT demonstrated — the site flushes headers explicitly as soon as its relay is attached, so the budget has to be consumed by connect + handshake + server scheduling instead. |
||
|
Joseph Doherty
|
6823e7e3fa |
fix(dcl): P2-5 seed probe-resolved tags; P2-6 close the Connecting/Reconnecting AdapterDisconnected gap
P2-5 — the tag-resolution PROBE was the only subscribe path that did not seed. HandleSubscribe carries seeds back on SubscribeCompleted and ReSubscribeAll re-seeds after a reconnect, because a fresh advise only yields a value on the source's next CHANGE. HandleRetryTagResolution did neither, so a tag that resolved LATE (device still booting, item created after deployment) and then never changed had no value path at all. OPC UA masks it with the initial data-change notification on subscription creation; an MxGateway advise can legitimately return VT_EMPTY, so a static tag stayed Uncertain for the life of the connection. New SeedNewlyResolvedTags reads the round's newly resolved tags and delivers them through the normal generation-guarded TagValueReceived path — directly rather than on the completion message (the probe only covers tags already registered in _instancesByTag, so the initial subscribe's ordering hazard does not apply), and as its own fire-and-forget task so the seed reads never delay the BatchSubscribeCompleted that re-arms the retry timer. P2-6 — only the Connected state handled AdapterDisconnected; in Connecting and Reconnecting it fell through to Unhandled. A connect that succeeds and drops again before its piped ConnectResult is applied therefore delivered the disconnect FIRST, where it was discarded, and ConnectResult(true) then declared the connection up on a dead adapter. An adapter raises Disconnected at most once per successful connect (OpcUaDataConnection's once-only _disconnectFired, reset only by the next successful ConnectAsync), so a connection with no subscriptions — nothing to fail at connection level and drive the reconnect machine — sat Connected on a dead adapter until the process restarted. Both states now record the disconnect in _disconnectDuringConnectAttempt (cleared at the top of HandleAttemptConnect, so it only ever refers to the attempt in flight) and both result handlers demote such a "success" to the ordinary connect-FAILURE tail — counting toward the failover budget and re-arming the retry timer. A flag rather than Stash: the stash has no selective drop, so a stashed disconnect would be replayed after the next successful connect and cost a spurious reconnect cycle. AdapterDisconnected is now stamped with the adapter generation (hooked via a per-adapter closure instead of a method group; SwapAdapter hooks AFTER the bump), and IsStaleDisconnect drops a notification from a swapped-out adapter in all three states — the disconnect twin of IsStaleConnectResult. SwapAdapter's unhook cannot stop an event already executing on a background thread, so without the fence an endpoint the operator has just reconfigured AWAY from could knock the healthy replacement into a reconnect. Tests: DataConnectionProbeSeedTests + DataConnectionDisconnectRaceTests (4 new). The race is reproduced by ORDERING, not timing — the connect is held on a gate, Disconnected is raised while the actor sits idle, and a GetHealthReport round trip is the mailbox barrier; no sleeps. All 4 were proved to fail against the pre-fix behaviour via a targeted revert of the four behaviour sites (the stale-generation test cannot be proved by a plain revert — the record carried no generation before this change, so the test would not compile; noted in its doc comment). Full DCL suite 346/346 green. |
||
|
Joseph Doherty
|
65b3c23239 |
perf(siteruntime): projected single-row deployed-config reads (P1-4)
HandleEnsureInstanceStarted, HandleDeploymentStateQuery, and HandleEnable each called GetAllDeployedConfigsAsync — materializing every deployed config, including every ConfigJson blob — just to FirstOrDefault one row. A volume-restore reconcile healing hundreds of instances turned that into hundreds of full-table reads. Add SiteStorageService.GetDeployedConfigAsync(name) (single-row, full projection incl. ConfigJson, for callers that (re)start an Instance Actor) and GetDeployedConfigMetadataAsync(name) (single-row, ConfigJson-free projection, for callers that only need DeploymentId/RevisionHash/IsEnabled). Switch the three call sites; leave decisions/semantics identical. HandleEnsureInstanceStarted gets its own _deployedConfigLoader seam, deliberately separate from the existing _configLoader (still used by the boot-time load, which legitimately needs every row) so a test overriding one cannot accidentally gate the other. Generation fencing (StoreGenerationAtRead, Medium wave T1) is untouched: the generation is still captured on the actor thread before the read dispatches. Tests: new SiteStorageServiceTests lock in that both projections return the same values GetAllDeployedConfigsAsync().FirstOrDefault(...) did. The three row-101 ensure-started race tests (ReconcileStartsInstanceTests) are ported to the new seam and still pass, proving generation fencing survives the projection. Proof of failure: since this is a pure I/O-cost fix with no observable behavior change, the new/changed tests were confirmed to fail to even compile against the unfixed code (stash/unstash) — GetDeployedConfigAsync, GetDeployedConfigMetadataAsync, and the deployedConfigLoader ctor param don't exist there. |
||
|
Joseph Doherty
|
9a3f006721 |
fix(management)+fix(cli): neutralize CSV formula injection + align export columns; deadline + Ctrl-C on audit export streaming (arch-review 2026-08-19 P6-7 + P6-8)
P6-7 (CWE-1236): AuditEndpoints CSV export quoted cells on comma/quote/CR/LF but never neutralized a leading =, +, -, @, TAB, or CR — inbound-API-caller text landing in Target/ErrorMessage/RequestSummary/ResponseSummary/Extra could execute as a formula/DDE trigger when opened in Excel. AppendField now prefixes a literal single quote on free-text columns only (never the GUID/enum/numeric/date/bool columns the exporter formats itself), applied before RFC 4180 quoting. The CSV was also missing ExecutionId/ ParentExecutionId/SourceNode that the JSONL export already carried; both are now 24 columns in the same AuditExportRow order. A second, independently maintained CSV writer (CentralUI's AuditLogExportService, feeding the Audit Log page's bulk-export button) had the identical gap and got the identical fix + column alignment. P6-8: `audit export` was the one CLI HTTP path with no deadline at all — a residual gap from the ManagementHttpClient timeout-honesty fix, which made HttpClient.Timeout infinite and the per-call token the single deadline everywhere except this streamed GET. RunExportAsync now takes a headers- received deadline (CancelAfter, reusing ManagementHttpClient. DefaultConnectTimeout) around the initial GetAsync, an inactivity deadline around the streaming copy (rearmed on every successful read, so large exports are bounded by silence, not total size), and an external CancellationToken wired to Console.CancelKeyPress in AuditCommands so Ctrl-C cancels promptly. A stalled/faulted stream now produces a friendly error noting the partial output file instead of hanging or dumping a stack trace. |
||
|
Joseph Doherty
|
50a77a8170 |
fix(transport): carry ExternalSystem TimeoutSeconds through bundles + diff (arch-review 2026-08-19 P7-1, row-27 shape)
TimeoutSeconds is the per-system HTTP timeout ExternalSystemClient reads at call time and is deployed to sites (proto field 6), but Transport bundles silently dropped it: the DTO never carried it, export never wrote it, the importer's Add path reset it to 0, the Overwrite path left the target's stale value untouched, and ArtifactDiff never compared it (so a bundle whose only change was a tuned timeout previewed as Identical). Adds TimeoutSeconds as an additive-optional trailing field on ExternalSystemDto (schemaVersion stays 1.1, bundleFormatVersion unchanged) and threads it through EntitySerializer (export + import), BundleImporter (Add and Overwrite), and ArtifactDiff.CompareExternalSystem. Old-bundle semantics pinned: an absent field (null) falls back to the entity default on Add and leaves the target's existing value untouched on Overwrite/diff. Beefed up the RoundTripEquivalenceTests external-system fixture with a non-default TimeoutSeconds so the reflection round-trip guard actually exercises the field instead of comparing default against default. Swept the rest of the ExternalSystemDto shape against ExternalSystemDefinition field-by-field; no other operative field is missing. |
||
|
Joseph Doherty
|
013257ec7c |
fix(centralui): per-IP spray ceiling on LoginThrottle (arch-review 2026-08-19 P6-9)
LoginThrottle keyed failures only as {username}|{ip}, so a password spray
from one IP against many distinct usernames never accumulated
MaxLoginFailuresPerWindow failures on any single pair and was never
throttled, despite the class doc claiming spray protection.
Add a secondary per-IP counter (SecurityOptions.PerIpMaxFailures, default
25, validated strictly greater than MaxLoginFailuresPerWindow) that shares
the existing fixed window/lockout machinery but tracks failures by IP
alone. IsLockedOut now refuses a bind when either the per-pair or the
per-IP counter is locked. RecordSuccess clears only the per-pair counter
(never the per-IP one), so a spray that stumbles onto one valid account
cannot use that success to reset its IP budget and keep spraying; window
expiry is the only relief for the per-IP counter. Per-pair behavior for
normal (non-spray) flows is unchanged.
Updates the class doc and Component-Security.md to describe the two-tier
model instead of the single-tier spray claim.
|
||
|
Joseph Doherty
|
561582747e |
fix(centralui): stop debug-stream sessions abandoned mid-start — dispose-vs-connect race (arch-review 2026-08-19 P6-6)
A debug-stream session created after the consumer had already gone away was leaked permanently. Both consumers awaited StartStreamAsync (up to 30s) and only then published the session; a circuit teardown / client disconnect inside that window ran the release path first, found nothing registered and stopped nothing. The session stayed registered in the singleton DebugStreamService, whose keepalive renews registered sessions forever (the 5-minute orphan reaper only collects UNregistered ones), pinning the site gRPC subscription, the DebugStreamBridgeActor and the dead consumer's object graph until process restart. Both call sites now publish the session and re-check liveness under a single monitor, so exactly one side tears it down and neither can miss it: - DebugView.Connect/Dispose/Disconnect share a _sessionLock; Dispose sets _disposed and takes the session in one step, Connect publishes only if not disposed and otherwise stops the session it just created. Dispose can run off the renderer's thread, so the Blazor sync context is not relied on. Disconnect and the onTerminated callback go through the same take-once helper, which also makes a repeated Dispose stop-once instead of stop-again. - DebugStreamHub uses the per-connection Items monitor: OnDisconnectedAsync marks the connection gone before releasing, SubscribeInstance publishes only if neither that flag nor ConnectionAborted is set, and UnsubscribeInstance takes the id atomically. DebugStreamService.StartStreamAsync/StopStream become virtual purely as a test seam (no behaviour change, keepalive/orphan contract untouched) so the race can be driven deterministically by a gated stub. Tests: DebugViewConnectDisposeRaceTests (3) + DebugStreamHubSubscribeRaceTests (3). Verified fail-unfixed: with the two source fixes stashed, both dispose-mid-start tests report zero StopStream calls and the DebugView double-dispose test reports two. |
||
|
Joseph Doherty
|
5cbc143b01 | fix(deploymentmanager): bulk deploy skips Disabled + reconcile circuit-breaker + probe captures pre-flatten versions (arch-review 2026-08-19 P7-2 + P7-3 + P7-4) | ||
|
Joseph Doherty
|
b68f553f1e | fix(host): h2 keepalive on site gRPC listener (arch-review 2026-08-19 P4-3) | ||
|
Joseph Doherty
|
e63ffd90ee |
fix(siteruntime): stamp the ensure-started heal with the store-write generation + order the redeploy override reset ahead of the successor (arch-review 2026-08-19 P1-1 + P1-2)
P1-1 — the row-67 ensure-started self-heal re-read the durable row off-thread but captured no _storeWriteGeneration and never ran IsStoreSnapshotStale, unlike the boot load and the reconcile sweep. A Disable or Delete committing inside that window left the result carrying an enabled config that passed every guard (neither command sets any of the three pipeline markers ShouldAttemptEnsureStart re-checks), so the heal started an instance the operator had just stopped — PERMANENTLY, since the sweep iterates store rows only and has no runtime→store stop leg. The generation is now captured at dispatch, threaded on EnsureInstanceStartedResult and checked on landing, and the comment claiming the stored row already covered this is corrected. The heal reads through _configLoader, the same seam the boot load and sweep use (identical production call), so a test can hold the read open across a committed disable/delete. P1-2 — the "static overrides reset on redeployment" invariant was racy in both directions. ApplyDeployment created the successor Instance Actor first and only then dispatched the clear, queued behind a full config-JSON write, so the successor's PreStart override load routinely won and applied the PREDECESSOR's overrides — keys absent from the new config included — to in-memory state for the actor's lifetime while the disk ended cleared; a later restart then silently flipped the values. The clear is now dispatched BEFORE CreateInstanceActor and handed to the actor as a gate its override load chains behind; the persistence task joins the same task, so a failed clear still fails the deploy exactly as before. Leg 2: the predecessor's PostStop flush was fire-and-forget and could commit up to one coalescing window AFTER the clear (the Deployment Manager gates the clear on Terminated), durably resurrecting overrides — PostStop now waits for it, bounded at 5s with a Warning on overrun. Tests: new RedeployStaticOverrideResetTests (5) pins both legs plus the gate mechanism and first-deploy behaviour; three new cases in ReconcileStartsInstanceTests pin the discarded disable/delete heals and that an unraced heal still starts. InstanceActorTests' StaticOverride_ResetOnRedeployment comment claimed to simulate an ordering the production path did not have — corrected. |
||
|
Joseph Doherty
|
64e6ba70e8 |
fix(management): site-scope ListDataConnections + hoist scope checks above resolve/status oracles (arch-review 2026-08-19 P6-4 + P6-5)
P6-4: HandleListDataConnections took no user at all and returned EVERY site's connections to any authenticated caller, nullifying the row-98 fail-closed gate on HandleGetDataConnection — Get denied exactly what List handed over. DataConnectionPublicShape already scrubs secrets, so the leak was site topology plus a connection-id existence oracle, but it still defeated scoping. List now takes the AuthenticatedUser: an explicit SiteId goes through EnforceSiteScope (the ListAreas shape), and an unfiltered list from a site-scoped non-Administrator caller is narrowed to their permitted sites (the ListSites shape). GetRequiredRoles is untouched — scoping is not a role gate, so the frozen 138-command authz matrix is unaffected. The Central UI Design > Connections page bypasses the actor entirely (ISiteRepository.GetAllDataConnectionsAsync direct) and IS reachable by a scoped user: it is gated RequireDesign, and SiteScopeService explicitly does NOT exempt Designer (register row 86). It now filters sites through SiteScope.FilterSitesAsync and keeps only their connections, mirroring EventLogs/AlarmSummary. P6-5: four handlers resolved their target — and on approve/reject reported its status — BEFORE the scope check, so an out-of-scope caller could tell a real id from a fake one, and read a secured write's lifecycle status, from the error shape alone. The scope check is hoisted above the not-found and status checks in HandleSubmitSecuredWrite, HandleApproveSecuredWrite, HandleRejectSecuredWrite and HandleDebugSnapshot, with the unresolvable case routed through the existing DenyUnresolvedTargetIfScoped so the denial is identical whether or not the target exists. System-wide callers keep their ordinary not-found/status answers, and the P6-1 CAS semantics on approve/reject are unchanged (only the checks above the CAS moved). Tests: 8 new ManagementService tests + 2 CentralUI bUnit tests, all verified to FAIL against the unfixed handlers (src reverted, tests re-run) and pass after — including the unscoped/Administrator control cases proving behaviour for system-wide callers is unchanged. Full ManagementService.Tests 545 green, CentralUI.Tests 1073 green. |
||
|
Joseph Doherty
|
d0a748e632 | fix(communication)+fix(siteruntime): carry DetachedScriptThreads/ScriptRunShedCount/SiteStreamAlarmDropCount across the health wire; restore drained counters on failed send (arch-review 2026-08-19 P8-1) | ||
|
Joseph Doherty
|
eca0ab768d |
fix(auditlog)+fix(configdb): budget the ingest fallback loop; index + varchar-type the SiteCalls list query (arch-review 2026-08-19 P5-1 + P5-2)
P5-1: AuditLogIngestActor.OnCachedTelemetryAsync's per-entry fallback loop (used when the cached-telemetry batch fast path fails) ran with no budget/cancellation, unlike OnIngestAsync's fallback. A slow/wedged DB could freeze the central ingest singleton's mailbox for minutes. The loop now runs under its own fresh IngestFallbackBudget CancellationTokenSource (mirroring OnIngestAsync), passes the token into BeginTransactionAsync/ InsertIfNotExistsAsync/UpsertAsync, and stops early on budget exhaustion, logging how many entries were left for the periodic reconciliation pull to pick up. P5-2: SiteCallAuditRepository.QueryAsync ordered by (CreatedAtUtc DESC, TrackedOperationId DESC) with no index leading unconditionally on CreatedAtUtc, forcing a full scan+sort on an unfiltered page. Added IX_SiteCalls_Created_TrackedOperationId (migration AddSiteCallsListSortIndex + matching hand-written production-parity script under docs/plans/sql/). Also switched the query from FromSqlInterpolated (which binds string filters as nvarchar(4000)) to FromSqlRaw with explicitly SqlDbType.VarChar-typed parameters, matching the SiteCalls columns' IsUnicode(false) declaration and the write path's existing Param(...) pattern, so seeks — including the keyset predicate — are no longer blocked by an implicit conversion. Tests: new AuditLogIngestActorCombinedTelemetryTests case proves the fallback budget bounds the wait (hanging repo stub, ~5s not indefinite), that remaining entries are skipped, and that a non-default/cancellable token reaches the repository call; verified fail-unfixed by reverting the actor change (test times out at its 20s bound). Existing SiteCallAuditRepositoryTests QueryAsync/keyset/stuck-cutoff cases (real MSSQL) confirm the raw-SQL rewrite still returns correct pages. |
||
|
Joseph Doherty
|
6c11362d9f |
fix(auditlog)+fix(siteeventlog): complete audit write tasks only after commit; run event-log retention on standby (arch-review 2026-08-19 P4-2 + P4-4)
P4-2: SqliteAuditWriter.FlushBatch completed each pending row's Task (TrySetResult) inside the per-row insert loop, before transaction.Commit() ran. A later failure in the same batch (another row's insert throwing, or the commit itself failing) rolled the whole transaction back, but the earlier rows' Tasks had already completed successfully -- the caller and FallbackAuditWriter (whose ring-buffer fallback exists precisely for this outage) both saw false success while the rows were actually gone from the database; the outer catch's TrySetException became a silent no-op against an already-completed TaskCompletionSource. Every row's Completion is now deferred and only resolved (TrySetResult for the whole batch, or TrySetException for the whole batch) after transaction.Commit() succeeds or the transaction rolls back, mirroring SiteEventLogger.FlushBatch. P4-4: EventLogPurgeService gated the daily site_events purge on the active-node check, early-exiting on the standby, on the stale premise that "the standby receives no writes". DataConnectionActor runs on both site nodes and writes site_events regardless of active/standby status (register rows 89/90/97), so an unreplicated standby (the rig's site-b/site-c posture) grew its site_events file forever with no retention and no cap. The gate is removed; every node now purges its own file every tick. Safe on a replicated pair because the retention/cap deletes are CDC-captured and converge under LWW regardless of which node issues them; required on an unreplicated pair because each node's file is independent. The active-node check is retained only to annotate the purge tick's log line. Tests: new SqliteAuditWriterCommitFailureTests forces a genuine non-swallowed mid-batch INSERT failure via a real SQLite trigger + custom scalar function on the writer's own connection (reflection seam), proving (a) a row whose own insert succeeded still faults when a later row in the same batch fails, and (b) FallbackAuditWriter rings every faulted row and increments the failure counter for each -- both assertions fail against the pre-fix code (proof-of-failure verified by temporarily reverting the fix). EventLogPurgeServiceTests' standby case is inverted to assert the purge now runs (not skips) on a standby node. |
||
|
Joseph Doherty
|
26ab6f8f09 |
fix(dcl): generation-fence ConnectResult + roll back failed OPC UA subscribe batches (arch-review 2026-08-19 P2-3 + P2-4)
P2-3. ConnectResult was the last piped completion with no adapter generation on
it, so a connect launched against generation N and answered after the adapter had
been swapped (failover, or the row-97 in-place reconfigure) was applied to its
REPLACEMENT. A stale success drove ReSubscribeAll + BecomeConnected against an
adapter that had never connected, and the replacement's own result then landed in
the Connected state where nothing handles it. With a bad new endpoint the batch
subscribes failed at connection level with a log line only, no AdapterDisconnected
ever fired, and the idempotent-reconfigure guard refused a re-send of the same
configuration — only a restart recovered the connection. A stale FAILURE was as
wrong the other way: it counted toward failover and re-armed the reconnect timer
underneath the attempt genuinely in flight.
HandleAttemptConnect now stamps the generation and both handlers open with the
same guard the subscribe leg got in
|
||
|
Joseph Doherty
|
412406e7c5 | fix(communication): cancel-only CTS replacement, deferred channel disposal, single-endpoint debug stream (arch-review 2026-08-19 P3-1 + P3-2 + P3-3) | ||
|
Joseph Doherty
|
fd4cb30273 |
fix(siteruntime): arm stuck-script watchdog + health/summary on alarm runs (arch-review 2026-08-19 P1-3)
Alarm on-trigger runs (ScriptRunLauncher.RunAlarmScriptAsync) bypassed the stuck-script watchdog, the script-error health counter, and ScriptRunSummaryRecorder entirely, even though they run on the same bounded ScriptExecutionScheduler pool as instance scripts. A wedged alarm body cost a pool thread forever with no site event and no detach/replace, and a timed-out or failed on-trigger run was invisible to health monitoring and the run summary. Bring the alarm path to parity with the instance-script path (RunScriptAsync): capture the worker slot/run stamp in the first synchronous segment and arm ArmStuckScriptWatchdog with the same detach-and-replace semantics and gauge; resolve ScriptRunSummaryRecorder/ISiteEventLogger from the same DI provider and thread RecordStarted/RecordCompleted/RecordFailed/RecordTimedOut; increment ISiteHealthCollector.IncrementScriptError on both the timeout and general failure catch blocks. LaunchAlarmScript/RunAlarmScriptAsync now take healthCollector and serviceProvider parameters, passed from AlarmActor's existing _healthCollector/_serviceProvider fields. No change to the enqueue-anchored deadline, the shed cap, or "a stop does NOT cancel an in-flight run" semantics. Adds AlarmScriptWatchdogTests (previously zero coverage of LaunchAlarmScript): wedged-worker detach + replacement, timeout-at-dequeue health counter increment, and successful-completion summary recording. All three fail against the pre-fix code (verified via stash) and pass after. |
||
|
Joseph Doherty
|
91fcb5b951 | fix(commons): reject comma-joined enum tokens in strict audit query parsers (arch-review 2026-08-19 P6-3, register row 99 defect) | ||
|
Joseph Doherty
|
042e2efd1d |
fix(centralui): enforce site scoping on AlarmSummary — filter site list + deny out-of-scope alarm fetches, mirroring DebugView (arch-review 2026-08-19 P6-2 High)
AlarmSummary.razor was gated RequireDeployment but never consulted
SiteScopeService: the site picker listed every site unfiltered and
GetSiteAlarmsAsync ran for whatever siteId the onchange event carried,
so a Deployer scoped to one site could watch any site's live alarms
(including CurrentValue/LimitValue process data). The page was missed
by the
|
||
|
Joseph Doherty
|
61a14bccab |
fix(management): CAS-guard secured-write reject (WHERE Status='Pending') so reject can never falsify an executed two-person write (arch-review 2026-08-19 P6-1 High)
Reject read the row as Pending, then landed a blind full-row UpdateAsync. A concurrent approve could win its own Pending->Approved CAS in that gap and enter the multi-second WriteTagAsync relay; the reject's update then stamped Status/ExecutedAtUtc/ExecutionError to Rejected mid-relay, so the two-person-control record said Rejected for a write that physically executed at the plant. Works cross-node (both centrals share one MS SQL). - ISecuredWriteRepository/SecuredWriteRepository: new TryMarkRejectedAsync, mirroring TryMarkApprovedAsync exactly (conditional UPDATE ... WHERE Status='Pending', writes only the columns the reject transition owns). No schema change, no migration, no rowversion column. - HandleRejectSecuredWrite: uses the CAS; on loss re-reads the row and returns the status that actually won (same UX shape as approve's lost-race error), and emits the SecuredWriteReject audit row ONLY on a won CAS. - Audited the other lifecycle writes: approve/reject/expiry (inline + the opportunistic list sweep) now ALL transition only out of Pending, so winning the approve CAS gives that leg exclusive ownership of the row and its post-relay Executed/Failed update stays safe as an unconditional write. Documented in place. - Tests: repository-level approve-vs-reject races in both orders (exactly one wins, loser false, row reflects the winner only) plus reject-vs-expiry; handler-level lost-race reject (conflict error, no write, no audit row) and approve on an already-rejected row (no CAS, no relay, no audit). |
||
|
Joseph Doherty
|
59468399db |
fix(dcl): reset MxGateway alarm-stream state on reconnect so the native-alarm feed resumes (arch-review 2026-08-19 P2-2 High)
ConnectAsync reset the event loop and client on reconnect but left _alarmCts/_alarmSubCount/_alarmSubSources/_alarmStreamPrefix untouched. The orphaned old RunAlarmStreamAsync loop kept retrying the disposed client every 5s forever, and SubscribeAlarmsAsync's coverage check saw the stale state and short-circuited the actor's post-reconnect transparent re-subscribe (ReSubscribeAllAlarms) without ever opening a stream on the new client — freezing the native-alarm mirror until failover/redeploy. Reset the alarm state in ConnectAsync, mirroring the existing lock-guarded block in DisconnectAsync/DisposeAsync. |
||
|
Joseph Doherty
|
c4d3c8b875 |
fix(dcl): carry adapter generation on SubscribeCompleted/AlarmSubscribeCompleted; drop stale completions and release captured-adapter handles (arch-review 2026-08-19 P2-1 High)
An in-flight subscribe launched against adapter generation N can take up to SeedOverallTimeout (120s) to answer. If the adapter is swapped in that window (failover, or the row-97 in-place reconfigure), the completion was applied unconditionally: it stored a dead session's handle under the tag key, so the LIVE completion for the same tag read as a duplicate and had its working monitored item released (tag resolved forever, no data); it re-stamped the old endpoint's seed values with the CURRENT generation, bypassing the stale-data guard; and a connection-level failure row from the dead adapter knocked the healthy new connection back into Reconnecting. The alarm twin released the new live alarm feed the same way. Both completions now carry the generation AND the adapter instance they ran against, and both handlers drop a completion whose generation no longer matches - no handle store, no seed application, no per-tag or connection-level failure processing, no duplicate-release logic. Handles the dropped completion did create are released best-effort against the CAPTURED adapter (handles are adapter-scoped; MxGateway's are plain ints that collide across sessions), and the existing release sites in both handlers were switched to the message's adapter for the same reason. The in-flight markers are deliberately left alone (ReSubscribeAll/ReSubscribeAllAlarms clear them wholesale and may already have re-registered the tag/source), and the requester gets a failure reply so its retry timer re-sends. Tests: DataConnectionStaleCompletionTests - stale handle not stored / live item not released, release lands on the old adapter only, seed values not delivered, stale connection-level failure does not re-enter Reconnecting, stale alarm completion does not release the live feed. All five fail against the unfixed handlers. |
||
|
Joseph Doherty
|
f02e116688 |
fix(siteruntime): extend the row-101 stale-snapshot guard to the instance-config half
ReadStoreSnapshotAsync's snapshot is read off the actor thread and consumed only when its piped message lands. Row 101 guarded the DEFINITIONS half of that race; the CONFIG half was left to machinery whose reach is time-limited, and both legs of the instance reconcile sweep were reachable through it: - Row-72 leg: a read dispatched before a deploy's persist, landing after the deploy completed AND after the pipeline released ownership, carries the revision the deploy replaced. TryRestartStaleInstance then "repairs" the instance BACKWARD, re-deploying the snapshot's own superseded ConfigJson. InstanceIsOwnedByADeployPipeline covers only in-flight deploys, and the per-revision claim has never seen the old revision, so nothing stopped it. - Row-78 leg: a read dispatched before a delete/disable commits still shows an enabled row while the actor is already stopped, so StartStoredInstance resurrects it — and delete clears _sweepAttemptedRevisions, freeing the claim. Both verified real by neutralising the freshness check and re-running the new tests (the fail-before site event literally reads "running revision rev-2 is stale, applying stored revision rev-1"; the deleted instance came back). _definitionsWriteGeneration becomes _storeWriteGeneration, covering both tables (the snapshot is one consistent read and is discarded whole, so one counter is the right shape); DefinitionsGenerationAtRead -> StoreGenerationAtRead on both internal records. The generation is now bumped at BOTH ends of every mutation — when the off-thread write is dispatched and when its completion message lands — because the config paths advance actor state before the write, so an initiation-only bump would let an in-window read look fresh. Disable, delete and the init-failure rollback have no actor-thread completion message of their own, so a minimal StoreWriteCompleted self-message carries the bump for them. CDC-replicated writes, snapshot resyncs, restored files and SiteReconciliationActor's fetch deliberately do NOT bump: discovering exactly those is the sweep's reason to exist. A stale sweep result is now dropped WHOLE (no overlay apply, no start, no restart). A stale boot load cannot be dropped — startup must finish — so it re-dispatches LoadDeployedConfigs(); a fifth consecutive superseded read is applied anyway with a Warning, and a late one arriving after the boot completed is dropped rather than looping against the S5 retry timer. Tests: three fail-before regressions (backward restart, resurrection, boot re-read) plus two guard twins pinning that the legitimate row-72 restart and row-78 start still fire on a current-generation snapshot. The row-101 definitions tests keep passing; FreshEmptySweepSnapshot's hardcoded generation moves 1 -> 2 now that one artifact apply bumps at both ends. Docs: Component-SiteRuntime.md's stale-snapshot paragraph now describes the whole snapshot (both halves, both consumers, the boot re-dispatch, the CDC-no-bump rule), the components mirror follows, and the arbitration memo gains §11 extending §10 to the config half. |
||
|
Joseph Doherty
|
0d3a90a9fd |
fix(siteruntime): guard the DCL arbitration overlay against stale store snapshots — register row 101
An off-thread snapshot read racing an artifact apply could land after the apply populated the overlay; its stale absences then tombstoned and stopped just-applied LIVE connections, which nothing re-creates until an instance redeploy (the removal evicts the hash slot the overlay push is gated on). Each read now captures a definitions-write generation at dispatch; a stale result is discarded whole (additions too — a stale row would reconfigure a live connection backward). Deterministic repro fails before the fix; a fresh-generation guard test pins that row-90 disappearance still fires. |
||
|
Joseph Doherty
|
16056f8eb6 | fix(management): ParseDeploymentStatus rejects ordinals — Enum.TryParse accepts "2" as InProgress despite the row-88 names-only contract; same LooksNumeric guard as the audit strict parser (found by the Wave 6 rows-98/99 agent survey) | ||
|
Joseph Doherty
|
8a9a8d2951 | merge(agent-c): row 100 — SiteScriptCompileCache becomes an instance-scoped sealed class owned by the ScriptCompilationService DI singleton (zero production Clear() callers existed, so Clear() is deleted outright); row-91 serialized collection relaxed and delta assertions restored to exact counts; new registration test pins the load-bearing singleton lifetime | ||
|
Joseph Doherty
|
152ec85e92 | merge(agent-b): rows 98/99 — site-scope gate fail-closed on unresolvable targets (5 sites, not 2: the inline GetInstance/GetSite/GetDataConnection shapes were an id-enumeration oracle; all route through DenyUnresolvedTargetIfScoped, denial indistinguishable from out-of-scope); CLI audit query/export drop 7 case-sensitive AcceptOnlyFromAmong gates for help-text values + a new strict server-side ParseEnumListStrict (no silent drop, no ordinals) | ||
|
Joseph Doherty
|
7c1268d85c |
feat(dcl+siteruntime): reconfigure a LIVE data connection in place — register row 97
`DataConnectionManagerActor.HandleCreateConnection` is create-only (warns and
returns for a name it already holds), so `EnsureDclConnection`'s documented
"a changed config re-issues the create so the DCL adopts it" was never true on
ANY node: a corrected endpoint/credential/backup/failover-count took effect only
when the connection was next built from scratch (node restart, singleton
failover, remove-then-re-add), while the DM's hash cache recorded it as applied.
Rows 89/90 made that honest with a Warning; this closes it.
New `ReconfigureConnectionCommand` (Commons, same shape as the create):
- `DeploymentManagerActor.EnsureDclConnection` now has three outcomes off its one
per-name hash slot — unchanged -> nothing at all, new -> create, CHANGED ->
reconfigure — and logs "Reconfigured DCL connection {Connection}
(protocol={Protocol}) in place" at Information in place of the old Warning. The
winner-resolution / arbitration / tombstone logic is untouched; only what
happens after a change is detected changed.
- `DataConnectionManagerActor` FORWARDS the command to the owning connection
actor, falling back to a create for a name it does not hold, so the command is
safe whatever the manager's state.
- `DataConnectionActor` handles it in ALL THREE states and never stashes it — a
connection pointed at a wrong or dead endpoint lives in Connecting/Reconnecting,
which is exactly when a correction arrives; stashed it would wait on the broken
configuration forever. Handling replaces protocol/primary/backup/failover-count,
re-arms endpoint selection to Primary (the new config's primary is the intent,
whatever the old config had failed over to), and calls the new shared
`SwapAdapter` helper — now used by both failover sites too, so the swap exists
once: unhook + fire-and-forget dispose -> _factory.Create -> rewire -> bump
_adapterGeneration LAST. `BecomeReconnecting(afterReconfigure: true)` then
reconnects immediately (StartSingleTimer replaces any armed reconnect, so no
racing attempts) with the unstable-disconnect accounting and failover check
suppressed — the link was dropped on purpose, and counting it as an unstable
cycle could trip a failover onto the NEW backup before the new primary had been
tried once. ReSubscribeAll/ReSubscribeAllAlarms then re-issue everything from
the untouched subscription registries. Idempotent: a reconfigure carrying the
configuration already in force is a no-op.
Stop-and-recreate was rejected as destructive: subscriptions live inside
DataConnectionActor and InstanceActor cancels its subscribe-retry timer on first
success, so a rebuilt actor would have no subscribers and nothing would ever
re-subscribe.
Tests: new DataConnectionReconfigureTests (7; 5 fail / 2 pass against a
behavioural revert). DclConnectionArbitrationTests gains two row-97 tests and its
CDC-arrival test now asserts the second push is a reconfigure; its ping-pong
guard also asserts zero reconfigures, since a flip would now show up there.
SiteRuntime-010 renamed to ..._ReissuesReconfigureCommand. 3 fail against a
reverted EnsureDclConnection. DataConnectionLayer.Tests 327/327,
SiteRuntime.Tests 643/643, solution build 0 warnings.
Docs: Component-DataConnectionLayer.md gains an "In-place reconfiguration"
section under Endpoint Redundancy; Component-SiteRuntime.md, and the
docs/components/ mirrors, updated; the arbitration decision memo gains an
appended §9 postscript resolving its residual 1 (residuals 2 and 3 stay open).
|
||
|
Joseph Doherty
|
f33f25a10b |
fix(auth+cli): fail-closed site-scope gate (row 98); case-insensitive audit filter values with a strict server-side check (row 99)
Register row 98 — the site-scope authorization gate was FAIL-OPEN on an
unresolvable target. The resolve-then-check shape was written as
`if (target != null) EnforceSiteScope(...)`, so a null resolve skipped the
check entirely and a site-scoped caller passed the gate for any id in the
system. Authorization now fails CLOSED: a target that cannot be PROVEN to sit
inside the caller's scope is treated as outside it.
The defect turned out to have THREE more instances than the register recorded.
Besides `EnforceSiteScopeForInstance` and `EnforceSiteScopeForIdentifier`, the
same inline shape sat in `HandleGetInstance`, `HandleGetSite` and
`HandleGetDataConnection` — read handlers that resolve their own target and so
never call the helpers. Those were the sharper half of the bug: they answered
success-with-null for a nonexistent id and Unauthorized for an out-of-scope
one, making the pair an id-enumeration oracle over the whole fleet. All five
now route through one shared `DenyUnresolvedTargetIfScoped` arm.
Two properties are load-bearing and pinned by tests:
- System-wide callers (Administrator, or a Deployer with no site grants) are
untouched — they return at the guards above the lookup and keep the
handler's ordinary "not found" / null answer.
- The denial is indistinguishable from an out-of-scope denial: same
ManagementUnauthorized shape, and a message that never states whether the
target exists.
Register row 99 — CLI `audit query --status delivered` failed client-side while
the server would have accepted it: `AcceptOnlyFromAmong` matches ordinally
against the capitalised literals. Same treatment row 88 gave `deploy status` —
the client-side gate is removed, the valid set moves into the option help text,
and the server becomes the single authoritative check.
That required ADDING the server check, not just relying on one: the shared
audit filter parser was LAX, silently dropping an unrecognised value and
leaving the dimension unconstrained — so removing the CLI gate alone would have
turned a loud client-side failure into a plausible, complete-looking page that
did not answer the question asked. New `ParseEnumListStrict` (case-insensitive,
rejects ordinals — `Enum.TryParse` accepts "3", and a filter is a name-based
contract) backs the ManagementService /api/audit/query + /api/audit/export
endpoints, which now return 400 BAD_REQUEST naming the valid set.
Other AcceptOnlyFromAmong sites surveyed in the audit group: `--channel`,
`--kind`, `--status` on BOTH `query` and `export` (6 uses) plus `export
--format` (rejected `--format CSV` though the server lowercases it) — all seven
fixed here. Deliberately left alone: `template folder reorder --direction`
(pure client-side literal, no server-side parse to defer to) and the global
`--format json|table` (CLI-local output selection, never sent to a server).
The CentralUI AuditLogPage drill-in parser and the CentralUI export endpoint it
feeds deliberately KEEP the lax parse — they read a query string a human may
have bookmarked, where a stale fragment must still render the page. Both
contracts now live side by side in AuditQueryParamParsers and are documented as
non-reconcilable.
Docs: Component-ManagementService.md gains a "Site-scope gate: fail-closed on
an unresolvable target" section and a strict-filter note on the audit
endpoints; Component-Security.md records the fail-closed shape under Deployer
site scoping; the CLI README marks the audit filter values case-insensitive and
states the no-silent-drop / no-ordinals contract.
Tests: 5 new ManagementActorTests fail before the row-98 fix (verified by
reverting ManagementActor.cs) and cover scoped+nonexistent -> denied,
system-wide+nonexistent -> unchanged (both Administrator and no-grant Deployer),
scoped+in-scope -> allowed, and the indistinguishability property. 21 CLI tests
fail before the row-99 fix. Four pre-existing tests that pinned the removed
client-side gate were rewritten to pin the new contract.
|
||
|
Joseph Doherty
|
84b405ff5f |
refactor(siteruntime): scope SiteScriptCompileCache to the ScriptCompilationService singleton — close register row 100
The compile cache was a `static` class, so its entry table and its Hits/Count
counters were one process-global blob every test class in the SiteRuntime
assembly wrote into. Row 91 last wave neutered the resulting flakes (per-test
GUID keys + one-sided counter bounds), but the shared mutable state itself
remained: any test calling Clear() evicted entries other parallel classes had
just inserted, and a Clear() from a class outside the serialized collection
would degrade the others to recompiles and re-open the row-91 flake class.
PREMISE CORRECTION: the register speculated production clears "per-instance or
per-deployment script entries" and offered ClearKeys(prefix) as the natural fit.
There are NO production callers of Clear() at all — grep of src/ finds only
GetOrAdd (ScriptCompilationService.CompileCore). Clear() existed solely so
tests could reset the global. A key-scoped API would therefore have been new
production surface built for nobody, so the instance-scoped seam is the fix and
Clear() is deleted outright rather than migrated.
- SiteScriptCompileCache: `internal static class` -> `internal sealed class`;
Cache/EvictionLock/_hits/_accessSequence become instance fields. LRU batch
eviction, the double-checked sweep and the ConcurrentDictionary.ToArray()
snapshot are byte-for-byte the same policy. Clear() removed.
- ScriptCompilationService owns one cache. Public ctor (the DI path) builds its
own; a new internal ctor takes one for tests. Production N4 sharing is
unchanged and now rests on the DI SINGLETON registration — the one service
instance is threaded to DeployCompileValidator and to every Instance Actor —
so that lifetime is load-bearing and is pinned by a new registration test.
- Tests: all three Clear() callers migrated off it. Each test now owns its cache,
which lets the assertions get STRONGER — the row-91 workarounds are retired:
Hits/Count are exact again (no >= baseline bounds), the per-test GUID key
token is gone, and ForeignEntrySlack (512 entries of allowance on the bound)
drops to the racing-thread count in the concurrency test. New test
SeparateCaches_DoNotShareEntriesOrCounters pins the scoping directly: filling
one cache past MaxEntries leaves another's entry, hit count and identity
intact. DeploymentWarmThenGateTests additionally asserts Count == 1 now that
its counter is private.
- The [Collection("SiteScriptCompileCache")] serialization from row 91 is
RELAXED (removed from all three classes) — they no longer touch any shared
state, so there is nothing left to serialize.
- Docs: Component-SiteRuntime.md gains the ownership/singleton note and drops
the Clear() reference; DeploymentManagerActor / DeployCompileValidator /
ServiceCollectionExtensions comments corrected to say where the cache lives.
Build: 0 warnings. SiteRuntime.Tests 643/643, three consecutive full-parallel
runs. AuditLog.Tests correlation classes (other ScriptCompilationService
consumers) green.
|
||
|
Joseph Doherty
|
ad0100f2f9 | merge(agent-a): rows 37/86/87/88 — deployment reads consolidated on QueryDeploymentListPageAsync (QueryDeploymentSummariesAsync deleted); SiteScopeService honors the Administrator system-wide rule (claims no longer scope admins — the documented ManagementActor semantic now holds on both surfaces); site scope rides the query join (whole-table instance read gone), PageSize bounded (default 50, cap 1000); unparseable --status fails loudly naming valid values | ||
|
Joseph Doherty
|
c57e74e660 |
fix(siteruntime): arbitrate DCL connection config — definitions store wins (rows 89 + 90)
Closes deferred-work-register rows 89 and 90, and corrects their premise twice.
THE RULE
The `data_connection_definitions` store wins wherever it has an opinion; an
instance's flattened `Connections` entry is a deploy-time snapshot and is
authoritative only for names the store has never owned (the deploy-only site,
where that table is legitimately empty).
PREMISE CORRECTION 1 — the peer's DCL is EMPTY, not stale
`DeploymentManagerActor` is a `site-{SiteId}`-scoped singleton and the only
sender of Create/RemoveConnectionCommand, to its OWN node's `dcl-manager`
(which exists on every node but is populated only where the singleton runs).
So the standby serves nothing. The real defect is that `_createdConnections`
is in-memory per singleton incarnation: a restart or failover rebuilds every
connection from the flattened snapshots and silently discards a newer
connection-only artifact deploy. Row 89 is therefore create-time, not
message-delivery, and NOT replication-specific — the same loss happens on an
unreplicated single-node restart.
PREMISE CORRECTION 2 — a changed config never reached a LIVE connection either
`DataConnectionManagerActor.HandleCreateConnection` is create-only (warns and
returns on a name it holds), so `EnsureDclConnection`'s documented "re-issues
the create so the DCL adopts the new configuration" has never been true on any
node. Now surfaced with a Warning saying exactly that, once per changed config.
Not fixed here: stop-and-recreate drops every tag subscription and InstanceActor
cancels its subscribe-retry timer on first success, so nothing re-subscribes.
MECHANISM
`data_connection_definitions` gains its first production reader. It is read on
the SAME off-thread task as the deployed configs (`ReadStoreSnapshotAsync`), by
the boot load and every sweep tick, into an actor-thread-confined arbitration
overlay applied BEFORE the boot batch and BEFORE the sweep's repair loop. A
CDC-delivered definition change converges within one tick on whichever node runs
the singleton — no new cross-node message. A failed definitions read yields null
and leaves the overlay untouched (distinct from an empty list, which is a real
answer); clearing it would reintroduce the ping-pong through the error path.
THE PING-PONG, KILLED BY CONSTRUCTION
§7.4 of the sweep memo rejected a store-level pass because two sources would
flip the single per-name hash slot, recreating the actor every tick. That is a
property of WHERE the hash is computed: every ensure path now resolves the
winner BEFORE hashing, so only one configuration can ever occupy the slot. The
artifact-deploy path writes its rows into the overlay in the same pass, closing
the same flip arriving through the other door.
REMOVAL (row 90)
Both removal paths share `RemoveConnectionCentrally` and record an in-memory
tombstone, so a centrally removed connection is not resurrected by the next
instance start — a pre-existing bug. Tombstones are deliberately not persisted;
a definition row reappearing clears one.
TESTS
tests/.../Actors/DclConnectionArbitrationTests.cs — 8 tests. Fail-before
(behavioral revert: overlay inert + tombstone check disabled): 5 fail, 3 pass;
the 3 passers are the two deploy-only regression guards and the ping-pong guard.
DOCS
New decision memo docs/plans/2026-08-17-dcl-connection-arbitration-decision.md
(topology, rule, ping-pong mechanism, tombstones, residuals); Component-SiteRuntime,
Component-DataConnectionLayer + their docs/components mirrors; the §7.4 residual in
the sweep memo marked closed with its premise corrected.
|
||
|
Joseph Doherty
|
9f67147e7b |
refactor(deployments): one paged read of DeploymentRecords + close its three drifts
Deferred-work register rows 37, 86, 87, 88 — the structural duplication and the three behavioral divergences it had accumulated, fixed together. Row 37 (structural). `IDeploymentManagerRepository.QueryDeploymentSummariesAsync` is DELETED. It and `QueryDeploymentListPageAsync` were two paged reads of the same insert-only table with different contracts — raw offset vs page number, no total vs total, no instance join vs joined, instance-id-list scoping vs site-id scoping. The CLI/HTTP `QueryDeployments` handler now goes through `QueryDeploymentListPageAsync` with a `DeploymentListFilter`, exactly like the Central UI page. The `DeploymentRecordSummary` projection went with it (`DeploymentListRow` is the single row shape); its `MaxPageSize` const moved to `DeploymentListPage`, which also gains `DefaultPageSize`. Row 86 (Medium, authorization — the headline). `SiteScopeService.ResolveAsync` was claims-only, so an Administrator carrying `SiteId` claims saw the whole fleet through the CLI (`ManagementActor.EnforceSiteScope` has always exempted Administrator) and only their granted sites on the Deployments page. One user, one dataset, two answers. Administrator = system-wide is the intended semantic on both sides; the UI was the defect. The role claim is read explicitly with an OrdinalIgnoreCase comparison rather than via `IsInRole`, which compares claim VALUES ordinally and only resolves when the identity carries the matching roleType — the two sides must not disagree on a differently-cased role name. An Administrator's site claims are not collected at all, so the invariant "system-wide => empty permitted-id set" holds for every caller. Designer is NOT exempt: the class doc claimed "any Admin/Design user" is system-wide, which was true of neither side, so the doc was corrected rather than the bypass widened. Row 87 (Low). The scoped branch used to resolve the site grant into an instance-id list by loading EVERY instance in the system, non-transactionally, on every scoped query; the grant now rides as `SiteIdScope` and is resolved by the query's own join. An unparseable permitted-site id is DROPPED, never ignored — dropping can only narrow a grant, discarding it would widen it to fleet-wide. And `PageSize <= 0` no longer means UNLIMITED down at the repository (`--page-size 0` materialized the whole retention window): it means "unspecified" and takes the CLI's own default of 50, capped at 1000. Bounding without telling would be silent truncation, so the handler now returns a `DeploymentQueryResult` envelope carrying `totalCount` plus the page coordinates actually applied — with a CLI table projection (mirroring row 57's `AuditConfigTableProjection`) that unwraps it and appends a "showing N of M" note. Row 88 (Low, UX). An unrecognised `--status` fell back to "no filter", so a typo like `--status Sucess` returned every status — a complete-looking answer to a question the caller never asked. It is now a `ManagementError` naming the valid values. Parsing stays case-insensitive; a numeric string is rejected too, since `Enum.TryParse` accepts "7" and yields an undefined value (guarded with `Enum.IsDefined`). Loud failure over silent full results, per row 46. Tests: 7 new `SiteScopeServiceTests` pinning Administrator = system-wide (verified failing 6/7 against the pre-fix service, the Designer pin passing both ways); the ManagementActor deployment suite migrated onto the consolidated query with new pins for the status error, case-insensitive parse, numeric rejection, page-size default/ cap/floor, empty-scope-on-garbage-grant, and "reads no instances at all"; a repository test for site-scope+status composing over a bounded page; and 6 CLI tests for the new table projection. |
||
|
Joseph Doherty
|
5162408a9f | merge(agent-b): rows 81+84 — storage cap now measures site_events' own footprint (sampled page-layout model; dbstat probed unavailable and pinned); backlog note once per tick | ||
|
Joseph Doherty
|
ddb3c32e06 |
fix(siteeventlog): scope the storage cap to site_events' own pages (register rows 81, 84)
Row 81 — the cap could not be enforced on a replicated node. MaxStorageMb was compared against page_count-freelist_count over the WHOLE consolidated LocalDb file, which since Phase 1/2 holds ten tables plus __localdb_oplog / __localdb_row_version / __localdb_snapshot_state. Every 1000-row trim batch freed ~390 KB of site_events pages and added ~1000 oplog rows + ~1000 tombstones to the same file, so the measured size did not fall, the "size did not decrease" guard fired, and the trim stopped 2.5x over a 64 MB cap with 226 k rows still present. A node whose peer was down could not enforce its cap at all, and the event log's budget was in practice shared with bookkeeping it does not own. The cap now measures site_events AND ITS FOUR INDEXES only, so replication bookkeeping can neither consume nor block the budget — and the cap stops being a function of the peer's health, which was the wrong direction of coupling: an inflated oplog is a replication fault, and letting it drive event deletion would destroy operational history in response to one. dbstat — the exact per-table measurement the register proposed — is NOT usable here, verified before implementing. It is a compile-time option and the bundled SQLitePCLRaw.lib.e_sqlite3 (SQLite 3.53.3) does not enable it: PRAGMA compile_options lists no ENABLE_DBSTAT_VTAB, pragma_module_list returns ten modules with neither dbstat nor sqlite_dbpage, and selecting from dbstat fails with "no such table". Switching SQLite builds was rejected — the provider is shared with ZB.MOM.WW.LocalDb and every other consumer, so it trades cap accuracy for fleet-wide version skew. The footprint is therefore modelled from sampled row width (500 rows each end, octet_length so multi-byte UTF-8 is not under-counted) through a page-layout model; whole pages rather than a flat rate, because page fill collapses once rows approach half a page and a linear model underestimates that by ~40%. Clamped by whole-file logical size, a true upper bound that can only correct an over-estimate and never binds on a replicated node. Index pages count against the cap deliberately: measured at ~34% of the table's footprint, and the 1 GB default was sized against a 393 B/row figure that already included them. Guard semantics restated — two backstops replace the single on-disk check: (1) row width unsampleable => refuse to trim (checked before the cap comparison, since a zero width makes "over cap" unanswerable rather than false; the old loop's answer to an unanswerable cap was to keep deleting); (2) estimated size not falling across a batch => stop. (2) is the old guard re-based onto the table-scoped metric, so it can no longer be tripped by replication bookkeeping and now means exactly one thing: events are arriving as fast as the trim removes them. For that to be observable the loop re-reads the row count each batch, while bytes-per-row is sampled once per tick and held fixed — re-sampling per batch would jitter the estimate as the trim eats the head of the table and could stall the loop through guard (2), the same self-defeating measurement as row 81 itself. Row 84 — LogReplicationBacklogNote moved from per-phase to once per tick, driven by the shared PurgeBudget. A tick where retention AND the cap trim both deleted used to emit two identical Information notes; the tick total is also the more useful number beside an oplog spike. Measured model error: -0.8% / +8.4% / +12.8% across representative workloads, biased high on purpose (an early cap forfeits budget; a low one fails at its only job). Independently cross-checks row 34's harness at 397.2 vs 393 B/row. Fail-before: reverting the estimator to whole-file semantics fails 5 of the new tests and reproduces the original signature exactly (remaining=2000 estimate=21725184 cap=11534336 — rows destroyed, cap still unmet); restoring per-phase notes fails the row-84 test with "collection contained 2 matching items". No config keys added or renamed, so no rig/deploy changes. Decision record: docs/plans/2026-08-17-site-event-log-cap-semantics-decision.md Tests: SiteEventLogging 104/104 (was 93), Host 520/520, solution build clean. |
||
|
Joseph Doherty
|
2cae8773bf | merge(agent-a): rows 85+83 — EnsureDclConnections moved into the CreateInstanceActor funnel (self-healed/enabled instances get their DCL connections); ReconcileStartsInstanceTests gated on first instance-count report | ||
|
Joseph Doherty
|
8f2c2813ea |
fix(siteruntime): self-healed and newly-enabled instances now get their DCL connections (register row 85)
An Instance Actor started by a self-heal came up with NO DCL connection actors, so every tag it subscribed was answered "Unknown connection: <name>" by the DCL manager indefinitely. Proven live 2026-08-16 by the row-78 close-out probe: the sweep correctly started the replication-seeded instances, then every subscribe failed for 6+ min until a second pair restart. Root cause is narrower than the register assumed. A DCL connection actor is never created from data_connection_definitions -- that table has no production reader at all (GetAllDataConnectionDefinitionsAsync is called only by a test). It exists only as a side effect of starting an instance, from the Connections map inside that instance's own flattened config, via EnsureDclConnections. That ensure was the CALLER's job and only two of the four callers made it: CreateInstanceActorBatch (boot) ensured OK ApplyDeployment (deploy) ensured OK StartStoredInstance MISSING <- row-67 ensure-started + row-78 sweep HandleEnableResult MISSING <- enabling a disabled instance The two misses are the two paths added after the original pair -- the signature of a convention living at call sites instead of in the funnel. It was never replication-specific: the boot load reads only ENABLED configs, so a node booting with an instance disabled never creates its connections either, and enabling it afterwards hit the identical dead end on an ordinary unreplicated site. Fix: EnsureDclConnections moves into CreateInstanceActor, already documented as the single Instance Actor creation funnel (it is where the running revision is stamped for the same reason), and the two caller-side calls are removed. Placed before the already-exists guard so the two paths that already called it keep their exact previous semantics. No new message, option or DCL change -- it reuses EnsureDclConnection's existing per-name hash guard, so steady state is silent. Structural rather than another guard: a fifth start path cannot forget. Deliberately NOT done, with rationale in the memo: a sweep pass over data_connection_definitions. It would not have fixed row 85 (that table is empty on a site that has only ever received instance deploys), and re-hashing from it would fight the deploy path over the same per-name hash slot, recreating connection actors and dropping live subscriptions on a loop. Two residual sibling gaps (connection-only artifact deploy replicating to a peer that never adopts it, and the mirror-image removal case) are recorded in the memo and the component docs as open, not silently skipped. Tests: SelfHealCreatesDclConnectionTests, 7 tests asserting on the CreateConnectionCommand messages sent to a recording DCL stand-in. Verified fail-before against a behavioral revert (call removed from the funnel, restored at the two original sites): 4 fail / 3 pass, the 3 being steady-state silence plus two refactor guards that must pass on both sides. Docs: sweep design memo gains section 7; Component-SiteRuntime, Component-DataConnectionLayer, docs/components/SiteRuntime and docs/components/DataConnectionLayer updated (the deploy-flow step ordering in docs/components/SiteRuntime was stale as written). |
||
|
Joseph Doherty
|
2807b5a3ea |
fix(transport)+test: close deferred-work register rows 58, 60, 65 (Low-tail sweep)
Row 65 — trigger FIRED, fixed. BundleUnlockAttempt rows were pruned only by
nodes actively serving unlocks, so a cluster that stopped importing entirely
left its last rows in place forever. The row's own trigger was "Transport gains
a central scheduler a prune tick could ride" — it has one now:
BundleSessionEvictionService, the central-only 1-minute session sweep added by
the transport-session fixes. It now also ticks the new
IBundleUnlockAttemptWindow.PruneAsync. Both drivers share the ONE 15-minute CAS
cadence claim, so ticking on a 1-minute sweep costs at most one bounded DELETE
per quarter hour and the two can never prune concurrently. PruneAsync is
contractually non-throwing (unlike TryRegisterAttemptAsync, which throws and
lets the limiter own the fail-open decision) — a background sweep has no caller
to report to and must survive a database blip. Both Transport docs updated;
three tests in FleetWideUnlockThrottleTests, incl. the retention sweep running
with no unlock traffic anywhere in the test, which is the whole point.
Row 58 — closed with a pin, and the OLD pin was found blind. The existing
Deployments_Dispose_UnsubscribesFromNotifier is behavioural: it is satisfied by
the _disposed guard alone, so deleting the `-=` from Dispose leaves it green
while the singleton notifier keeps a delegate pinning the circuit-scoped
component. Verified by mutation — with the `-=` removed the old test still
passed and only the two new pins failed. Added:
* a structural retention pin asserting the notifier's invocation list actually
shrinks to 0 on Dispose;
* a source scan pinning the row's stated TRIGGER ("a second page subscribes,
or any subscriber lands without a matching Dispose unsubscribe") — every
CentralUI file that subscribes must also unsubscribe and be IDisposable,
with a non-empty guard so the scan cannot pass vacuously.
Row 60 — closed with a pin, and the register's premise corrected.
GlobalArtifactSnapshot does NOT hold EF-materialised entities: it is projected
into pure Commons DTOs before the fan-out. The EF state that genuinely crosses
into the parallel send is the tracked Site entities the lambda closes over, and
that is safe only while lazy loading is off. Pinned in ConfigurationDatabase
.Tests two ways — no proxies extension on the built DbContextOptions, and no
EntityFrameworkCore.Proxies package reference anywhere in the repo (the
structurally stronger half: UseLazyLoadingProxies() is not a compilable call
without it). The real invariant is now spelled out at the fan-out itself.
Row 59 — no product change; added the missing tripwire remark stating the
constraint the safety argument rests on (serial by design, one scoped DbContext,
do not Task.WhenAll; if it gets slow the fix is a scope PER SITE, not a shared
scope under fan-out). Line numbers in the register have drifted to :528-545.
Rows 36, 37, 64: verification only, no change — see the sweep report.
|
||
|
Joseph Doherty
|
f46cc19736 | Merge rows 78+72: DeploymentManager singleton instance sweep — starts replication-seeded enabled instances and converges stale revisions through the real HandleDeploy path; one repair attempt per (instance, revision) |