P4-5: EventLogPurgeService.ExecuteAsync now yields (Task.Yield()) before running the
initial RunPurge(), making "the purge never runs synchronously on the caller's thread"
a property of the method itself rather than an artifact of .NET 8+'s BackgroundService
scheduling (verified empirically not to reproduce the literal "StartAsync blocks" symptom
on this repo's net10.0 target — a 1.5s synchronous Thread.Sleep in ExecuteAsync still lets
StartAsync return in ~2ms). Proven instead via a reflection-invoked ExecuteAsync test that
bypasses BackgroundService's Task.Run wrapper, which is timing-independent. Also: a
throwing isActive predicate now fails CLOSED (logged as inactive, at Warning) instead of
silently claiming "active" — the predicate only affects the tick's log line since P4-4,
but a throw must not masquerade as a successful check.
P4-6: PRAGMA auto_vacuum = INCREMENTAL only takes effect on a table-less database, and
LocalDbSchema.EnsureCreated (inside SqliteLocalDb's constructor) always creates LocalDb's
own bookkeeping tables before SiteEventLogSchema.Apply ever gets a connection — so the
pragma there had never once taken effect, in any real call path. Real fix per plan
decision 2: SiteLocalDbSetup.EnsureAutoVacuumOnFreshFile pre-creates a genuinely fresh
LocalDb file with the pragma set, called from SiteServiceRegistration.Configure BEFORE
AddZbLocalDb's lazy singleton ever constructs SqliteLocalDb — no LocalDb API change
needed. Only helps NEW site databases; an existing file (every node on the docker rig
today) is left untouched rather than VACUUM'd, per the plan's explicit rejection of an
unconditional VACUUM against a live ~1 GB file. The now-dead pragma call in
SiteEventLogSchema.Apply is removed and its comment corrected.
Tests proven to fail against the unfixed code (git-diff-and-restore, not stash — this repo
shares one stash ref across all worktrees): ExecuteAsync_YieldsBeforeRunningTheInitial-
PurgeSynchronously and RunPurge_ActiveNodeCheckThrows_TickLogsAsInactive_NotActive fail
without the EventLogPurgeService fix; SiteLocalDbAutoVacuumTests fails to compile without
EnsureAutoVacuumOnFreshFile. SiteEventLogging.Tests 112/112 and Host.Tests 526/526 green
with the fix applied.
P3-4 — SiteAlarmAggregatorActor pinned _openInFlight forever on a stream open
that produced none of its three peer-driven outcomes. SiteStreamGrpcClient gives
up on response headers after ConnectedHeaderTimeout (10s) and then treats the
first received EVENT as its fallback connected signal, so an alarm-only site
that stays quiet supplies neither — leaving _streamDown=true AND _openInFlight=
true, which makes OnReconcileTick's reopen guard (_streamDown && !_openInFlight)
skip every tick until the site's 4h max stream lifetime ends the call.
Fix: bound one open attempt with a generation-fenced single-shot deadline
(GrpcAlarmStreamOpenTimedOut, ctor-injected, production default 30s — above the
10s header timeout, below the 60s reconcile interval). Firing it releases the
flag and nothing else: no retry-budget spend, no node flip, no liveness
republish, and the abandoned subscription is left running so a merely-slow
connect still comes up normally. Self-healing rather than reachability analysis.
The deadline is cancelled on connect / error / graceful completion.
P3-5 — SiteStreamGrpcServer cancelled the incumbent stream on a duplicate
correlation id and only THEN checked the concurrency cap. Checking Count >= max
after the removal is right at exactly the cap but wrong above it, and above it
is reachable because the cap check is documented check-then-act: two subscribes
that both clear it before either registers overshoot by one. In that window a
routine reconnect killed a healthy stream AND was refused with
ResourceExhausted, leaving the correlation with ZERO streams until its next
retry.
Fix: claim the removal, consult the cap only for a genuinely NEW correlation id,
then cancel. A replacement is net-neutral on the count so it never consults the
cap; a refused new subscriber cancelled nothing (TryRemove found nothing), so
the incumbent survives. Newest-wins is unchanged — a fresh authenticated
subscribe still displaces a dead/wedged incumbent.
Tests (both shown failing against the unfixed code):
- OpenAttemptThatNeverResolves_ReleasesTheInFlightGuard_SoTheTickCanReopen —
fails (reopen never happens) with the deadline arming disabled.
- OpenDeadline_DoesNotFire_OnAStreamThatConnects_NorFlipNodeOrBurnBudget —
inertness lock on every healthy path.
- AtCapOvershoot_DuplicateReconnect_IsNotRefused_AndLeavesExactlyOneLiveStream —
fails with ResourceExhausted against the old ordering; the overshoot is driven
deterministically by parking both openers on the ServerCallContext token read,
which sits between the cap check and the _activeStreams insert.
- AtCap_AGenuinelyNewSubscriber_IsStillRefused_AndTheIncumbentSurvives.
The P3-4 test's doc comment carries the honesty note: the aggregator-side
starvation reproduces exactly, but how often the >10s header delay that gets you
there occurs in production is NOT demonstrated — the site flushes headers
explicitly as soon as its relay is attached, so the budget has to be consumed by
connect + handshake + server scheduling instead.
P2-5 — the tag-resolution PROBE was the only subscribe path that did not seed.
HandleSubscribe carries seeds back on SubscribeCompleted and ReSubscribeAll re-seeds
after a reconnect, because a fresh advise only yields a value on the source's next
CHANGE. HandleRetryTagResolution did neither, so a tag that resolved LATE (device still
booting, item created after deployment) and then never changed had no value path at all.
OPC UA masks it with the initial data-change notification on subscription creation; an
MxGateway advise can legitimately return VT_EMPTY, so a static tag stayed Uncertain for
the life of the connection. New SeedNewlyResolvedTags reads the round's newly resolved
tags and delivers them through the normal generation-guarded TagValueReceived path —
directly rather than on the completion message (the probe only covers tags already
registered in _instancesByTag, so the initial subscribe's ordering hazard does not
apply), and as its own fire-and-forget task so the seed reads never delay the
BatchSubscribeCompleted that re-arms the retry timer.
P2-6 — only the Connected state handled AdapterDisconnected; in Connecting and
Reconnecting it fell through to Unhandled. A connect that succeeds and drops again before
its piped ConnectResult is applied therefore delivered the disconnect FIRST, where it was
discarded, and ConnectResult(true) then declared the connection up on a dead adapter. An
adapter raises Disconnected at most once per successful connect (OpcUaDataConnection's
once-only _disconnectFired, reset only by the next successful ConnectAsync), so a
connection with no subscriptions — nothing to fail at connection level and drive the
reconnect machine — sat Connected on a dead adapter until the process restarted. Both
states now record the disconnect in _disconnectDuringConnectAttempt (cleared at the top of
HandleAttemptConnect, so it only ever refers to the attempt in flight) and both result
handlers demote such a "success" to the ordinary connect-FAILURE tail — counting toward
the failover budget and re-arming the retry timer. A flag rather than Stash: the stash has
no selective drop, so a stashed disconnect would be replayed after the next successful
connect and cost a spurious reconnect cycle.
AdapterDisconnected is now stamped with the adapter generation (hooked via a
per-adapter closure instead of a method group; SwapAdapter hooks AFTER the bump), and
IsStaleDisconnect drops a notification from a swapped-out adapter in all three states —
the disconnect twin of IsStaleConnectResult. SwapAdapter's unhook cannot stop an event
already executing on a background thread, so without the fence an endpoint the operator
has just reconfigured AWAY from could knock the healthy replacement into a reconnect.
Tests: DataConnectionProbeSeedTests + DataConnectionDisconnectRaceTests (4 new). The
race is reproduced by ORDERING, not timing — the connect is held on a gate, Disconnected
is raised while the actor sits idle, and a GetHealthReport round trip is the mailbox
barrier; no sleeps. All 4 were proved to fail against the pre-fix behaviour via a
targeted revert of the four behaviour sites (the stale-generation test cannot be proved
by a plain revert — the record carried no generation before this change, so the test
would not compile; noted in its doc comment). Full DCL suite 346/346 green.
HandleEnsureInstanceStarted, HandleDeploymentStateQuery, and HandleEnable each
called GetAllDeployedConfigsAsync — materializing every deployed config,
including every ConfigJson blob — just to FirstOrDefault one row. A
volume-restore reconcile healing hundreds of instances turned that into
hundreds of full-table reads.
Add SiteStorageService.GetDeployedConfigAsync(name) (single-row, full
projection incl. ConfigJson, for callers that (re)start an Instance Actor)
and GetDeployedConfigMetadataAsync(name) (single-row, ConfigJson-free
projection, for callers that only need DeploymentId/RevisionHash/IsEnabled).
Switch the three call sites; leave decisions/semantics identical.
HandleEnsureInstanceStarted gets its own _deployedConfigLoader seam,
deliberately separate from the existing _configLoader (still used by the
boot-time load, which legitimately needs every row) so a test overriding one
cannot accidentally gate the other. Generation fencing (StoreGenerationAtRead,
Medium wave T1) is untouched: the generation is still captured on the actor
thread before the read dispatches.
Tests: new SiteStorageServiceTests lock in that both projections return the
same values GetAllDeployedConfigsAsync().FirstOrDefault(...) did. The three
row-101 ensure-started race tests (ReconcileStartsInstanceTests) are ported
to the new seam and still pass, proving generation fencing survives the
projection. Proof of failure: since this is a pure I/O-cost fix with no
observable behavior change, the new/changed tests were confirmed to fail to
even compile against the unfixed code (stash/unstash) — GetDeployedConfigAsync,
GetDeployedConfigMetadataAsync, and the deployedConfigLoader ctor param don't
exist there.
P6-7 (CWE-1236): AuditEndpoints CSV export quoted cells on comma/quote/CR/LF
but never neutralized a leading =, +, -, @, TAB, or CR — inbound-API-caller
text landing in Target/ErrorMessage/RequestSummary/ResponseSummary/Extra
could execute as a formula/DDE trigger when opened in Excel. AppendField now
prefixes a literal single quote on free-text columns only (never the
GUID/enum/numeric/date/bool columns the exporter formats itself), applied
before RFC 4180 quoting. The CSV was also missing ExecutionId/
ParentExecutionId/SourceNode that the JSONL export already carried; both are
now 24 columns in the same AuditExportRow order. A second, independently
maintained CSV writer (CentralUI's AuditLogExportService, feeding the Audit
Log page's bulk-export button) had the identical gap and got the identical
fix + column alignment.
P6-8: `audit export` was the one CLI HTTP path with no deadline at all — a
residual gap from the ManagementHttpClient timeout-honesty fix, which made
HttpClient.Timeout infinite and the per-call token the single deadline
everywhere except this streamed GET. RunExportAsync now takes a headers-
received deadline (CancelAfter, reusing ManagementHttpClient.
DefaultConnectTimeout) around the initial GetAsync, an inactivity deadline
around the streaming copy (rearmed on every successful read, so large exports
are bounded by silence, not total size), and an external CancellationToken
wired to Console.CancelKeyPress in AuditCommands so Ctrl-C cancels promptly.
A stalled/faulted stream now produces a friendly error noting the partial
output file instead of hanging or dumping a stack trace.
TimeoutSeconds is the per-system HTTP timeout ExternalSystemClient reads at
call time and is deployed to sites (proto field 6), but Transport bundles
silently dropped it: the DTO never carried it, export never wrote it, the
importer's Add path reset it to 0, the Overwrite path left the target's
stale value untouched, and ArtifactDiff never compared it (so a bundle
whose only change was a tuned timeout previewed as Identical).
Adds TimeoutSeconds as an additive-optional trailing field on
ExternalSystemDto (schemaVersion stays 1.1, bundleFormatVersion unchanged)
and threads it through EntitySerializer (export + import), BundleImporter
(Add and Overwrite), and ArtifactDiff.CompareExternalSystem. Old-bundle
semantics pinned: an absent field (null) falls back to the entity default
on Add and leaves the target's existing value untouched on Overwrite/diff.
Beefed up the RoundTripEquivalenceTests external-system fixture with a
non-default TimeoutSeconds so the reflection round-trip guard actually
exercises the field instead of comparing default against default.
Swept the rest of the ExternalSystemDto shape against
ExternalSystemDefinition field-by-field; no other operative field is
missing.
LoginThrottle keyed failures only as {username}|{ip}, so a password spray
from one IP against many distinct usernames never accumulated
MaxLoginFailuresPerWindow failures on any single pair and was never
throttled, despite the class doc claiming spray protection.
Add a secondary per-IP counter (SecurityOptions.PerIpMaxFailures, default
25, validated strictly greater than MaxLoginFailuresPerWindow) that shares
the existing fixed window/lockout machinery but tracks failures by IP
alone. IsLockedOut now refuses a bind when either the per-pair or the
per-IP counter is locked. RecordSuccess clears only the per-pair counter
(never the per-IP one), so a spray that stumbles onto one valid account
cannot use that success to reset its IP budget and keep spraying; window
expiry is the only relief for the per-IP counter. Per-pair behavior for
normal (non-spray) flows is unchanged.
Updates the class doc and Component-Security.md to describe the two-tier
model instead of the single-tier spray claim.
A debug-stream session created after the consumer had already gone away was
leaked permanently. Both consumers awaited StartStreamAsync (up to 30s) and only
then published the session; a circuit teardown / client disconnect inside that
window ran the release path first, found nothing registered and stopped nothing.
The session stayed registered in the singleton DebugStreamService, whose
keepalive renews registered sessions forever (the 5-minute orphan reaper only
collects UNregistered ones), pinning the site gRPC subscription, the
DebugStreamBridgeActor and the dead consumer's object graph until process
restart.
Both call sites now publish the session and re-check liveness under a single
monitor, so exactly one side tears it down and neither can miss it:
- DebugView.Connect/Dispose/Disconnect share a _sessionLock; Dispose sets
_disposed and takes the session in one step, Connect publishes only if not
disposed and otherwise stops the session it just created. Dispose can run off
the renderer's thread, so the Blazor sync context is not relied on. Disconnect
and the onTerminated callback go through the same take-once helper, which also
makes a repeated Dispose stop-once instead of stop-again.
- DebugStreamHub uses the per-connection Items monitor: OnDisconnectedAsync marks
the connection gone before releasing, SubscribeInstance publishes only if
neither that flag nor ConnectionAborted is set, and UnsubscribeInstance takes
the id atomically.
DebugStreamService.StartStreamAsync/StopStream become virtual purely as a test
seam (no behaviour change, keepalive/orphan contract untouched) so the race can
be driven deterministically by a gated stub.
Tests: DebugViewConnectDisposeRaceTests (3) + DebugStreamHubSubscribeRaceTests
(3). Verified fail-unfixed: with the two source fixes stashed, both
dispose-mid-start tests report zero StopStream calls and the DebugView
double-dispose test reports two.
P1-1 — the row-67 ensure-started self-heal re-read the durable row off-thread but
captured no _storeWriteGeneration and never ran IsStoreSnapshotStale, unlike the boot
load and the reconcile sweep. A Disable or Delete committing inside that window left the
result carrying an enabled config that passed every guard (neither command sets any of
the three pipeline markers ShouldAttemptEnsureStart re-checks), so the heal started an
instance the operator had just stopped — PERMANENTLY, since the sweep iterates store rows
only and has no runtime→store stop leg. The generation is now captured at dispatch,
threaded on EnsureInstanceStartedResult and checked on landing, and the comment claiming
the stored row already covered this is corrected. The heal reads through _configLoader,
the same seam the boot load and sweep use (identical production call), so a test can hold
the read open across a committed disable/delete.
P1-2 — the "static overrides reset on redeployment" invariant was racy in both
directions. ApplyDeployment created the successor Instance Actor first and only then
dispatched the clear, queued behind a full config-JSON write, so the successor's PreStart
override load routinely won and applied the PREDECESSOR's overrides — keys absent from
the new config included — to in-memory state for the actor's lifetime while the disk
ended cleared; a later restart then silently flipped the values. The clear is now
dispatched BEFORE CreateInstanceActor and handed to the actor as a gate its override load
chains behind; the persistence task joins the same task, so a failed clear still fails the
deploy exactly as before. Leg 2: the predecessor's PostStop flush was fire-and-forget and
could commit up to one coalescing window AFTER the clear (the Deployment Manager gates the
clear on Terminated), durably resurrecting overrides — PostStop now waits for it, bounded
at 5s with a Warning on overrun.
Tests: new RedeployStaticOverrideResetTests (5) pins both legs plus the gate mechanism and
first-deploy behaviour; three new cases in ReconcileStartsInstanceTests pin the discarded
disable/delete heals and that an unraced heal still starts. InstanceActorTests'
StaticOverride_ResetOnRedeployment comment claimed to simulate an ordering the production
path did not have — corrected.
P6-4: HandleListDataConnections took no user at all and returned EVERY
site's connections to any authenticated caller, nullifying the row-98
fail-closed gate on HandleGetDataConnection — Get denied exactly what List
handed over. DataConnectionPublicShape already scrubs secrets, so the leak
was site topology plus a connection-id existence oracle, but it still
defeated scoping. List now takes the AuthenticatedUser: an explicit SiteId
goes through EnforceSiteScope (the ListAreas shape), and an unfiltered list
from a site-scoped non-Administrator caller is narrowed to their permitted
sites (the ListSites shape). GetRequiredRoles is untouched — scoping is not
a role gate, so the frozen 138-command authz matrix is unaffected.
The Central UI Design > Connections page bypasses the actor entirely
(ISiteRepository.GetAllDataConnectionsAsync direct) and IS reachable by a
scoped user: it is gated RequireDesign, and SiteScopeService explicitly does
NOT exempt Designer (register row 86). It now filters sites through
SiteScope.FilterSitesAsync and keeps only their connections, mirroring
EventLogs/AlarmSummary.
P6-5: four handlers resolved their target — and on approve/reject reported
its status — BEFORE the scope check, so an out-of-scope caller could tell a
real id from a fake one, and read a secured write's lifecycle status, from
the error shape alone. The scope check is hoisted above the not-found and
status checks in HandleSubmitSecuredWrite, HandleApproveSecuredWrite,
HandleRejectSecuredWrite and HandleDebugSnapshot, with the unresolvable case
routed through the existing DenyUnresolvedTargetIfScoped so the denial is
identical whether or not the target exists. System-wide callers keep their
ordinary not-found/status answers, and the P6-1 CAS semantics on
approve/reject are unchanged (only the checks above the CAS moved).
Tests: 8 new ManagementService tests + 2 CentralUI bUnit tests, all verified
to FAIL against the unfixed handlers (src reverted, tests re-run) and pass
after — including the unscoped/Administrator control cases proving behaviour
for system-wide callers is unchanged. Full ManagementService.Tests 545 green,
CentralUI.Tests 1073 green.
P5-1: AuditLogIngestActor.OnCachedTelemetryAsync's per-entry fallback loop
(used when the cached-telemetry batch fast path fails) ran with no
budget/cancellation, unlike OnIngestAsync's fallback. A slow/wedged DB could
freeze the central ingest singleton's mailbox for minutes. The loop now runs
under its own fresh IngestFallbackBudget CancellationTokenSource (mirroring
OnIngestAsync), passes the token into BeginTransactionAsync/
InsertIfNotExistsAsync/UpsertAsync, and stops early on budget exhaustion,
logging how many entries were left for the periodic reconciliation pull to
pick up.
P5-2: SiteCallAuditRepository.QueryAsync ordered by
(CreatedAtUtc DESC, TrackedOperationId DESC) with no index leading
unconditionally on CreatedAtUtc, forcing a full scan+sort on an unfiltered
page. Added IX_SiteCalls_Created_TrackedOperationId (migration
AddSiteCallsListSortIndex + matching hand-written production-parity script
under docs/plans/sql/). Also switched the query from FromSqlInterpolated
(which binds string filters as nvarchar(4000)) to FromSqlRaw with explicitly
SqlDbType.VarChar-typed parameters, matching the SiteCalls columns'
IsUnicode(false) declaration and the write path's existing Param(...)
pattern, so seeks — including the keyset predicate — are no longer blocked
by an implicit conversion.
Tests: new AuditLogIngestActorCombinedTelemetryTests case proves the
fallback budget bounds the wait (hanging repo stub, ~5s not indefinite),
that remaining entries are skipped, and that a non-default/cancellable
token reaches the repository call; verified fail-unfixed by reverting the
actor change (test times out at its 20s bound). Existing
SiteCallAuditRepositoryTests QueryAsync/keyset/stuck-cutoff cases (real
MSSQL) confirm the raw-SQL rewrite still returns correct pages.
P4-2: SqliteAuditWriter.FlushBatch completed each pending row's Task
(TrySetResult) inside the per-row insert loop, before transaction.Commit()
ran. A later failure in the same batch (another row's insert throwing, or
the commit itself failing) rolled the whole transaction back, but the
earlier rows' Tasks had already completed successfully -- the caller and
FallbackAuditWriter (whose ring-buffer fallback exists precisely for this
outage) both saw false success while the rows were actually gone from the
database; the outer catch's TrySetException became a silent no-op against
an already-completed TaskCompletionSource. Every row's Completion is now
deferred and only resolved (TrySetResult for the whole batch, or
TrySetException for the whole batch) after transaction.Commit() succeeds
or the transaction rolls back, mirroring SiteEventLogger.FlushBatch.
P4-4: EventLogPurgeService gated the daily site_events purge on the
active-node check, early-exiting on the standby, on the stale premise that
"the standby receives no writes". DataConnectionActor runs on both site
nodes and writes site_events regardless of active/standby status (register
rows 89/90/97), so an unreplicated standby (the rig's site-b/site-c
posture) grew its site_events file forever with no retention and no cap.
The gate is removed; every node now purges its own file every tick. Safe
on a replicated pair because the retention/cap deletes are CDC-captured
and converge under LWW regardless of which node issues them; required on
an unreplicated pair because each node's file is independent. The
active-node check is retained only to annotate the purge tick's log line.
Tests: new SqliteAuditWriterCommitFailureTests forces a genuine
non-swallowed mid-batch INSERT failure via a real SQLite trigger + custom
scalar function on the writer's own connection (reflection seam), proving
(a) a row whose own insert succeeded still faults when a later row in the
same batch fails, and (b) FallbackAuditWriter rings every faulted row and
increments the failure counter for each -- both assertions fail against
the pre-fix code (proof-of-failure verified by temporarily reverting the
fix). EventLogPurgeServiceTests' standby case is inverted to assert the
purge now runs (not skips) on a standby node.
P2-3. ConnectResult was the last piped completion with no adapter generation on
it, so a connect launched against generation N and answered after the adapter had
been swapped (failover, or the row-97 in-place reconfigure) was applied to its
REPLACEMENT. A stale success drove ReSubscribeAll + BecomeConnected against an
adapter that had never connected, and the replacement's own result then landed in
the Connected state where nothing handles it. With a bad new endpoint the batch
subscribes failed at connection level with a log line only, no AdapterDisconnected
ever fired, and the idempotent-reconfigure guard refused a re-send of the same
configuration — only a restart recovered the connection. A stale FAILURE was as
wrong the other way: it counted toward failover and re-armed the reconnect timer
underneath the attempt genuinely in flight.
HandleAttemptConnect now stamps the generation and both handlers open with the
same guard the subscribe leg got in c4d3c8b8, dropping a mismatched result whole.
The adapter instance is deliberately NOT carried: unlike a subscribe completion a
connect result holds no adapter-scoped handles to release, and SwapAdapter has
already disposed the outgoing adapter. Dropping cannot wedge the machine — every
generation bump (CountFailureAndMaybeFailover, BecomeReconnecting's failover
branch, HandleReconfigure through it) arms an AttemptConnect for the new
generation, and the state stays Connecting/Reconnecting until that answers.
P2-4. CreateSubscriptionsCoreAsync registers each monitored item as it places it
(_callbacks, _itemShard, shard budget, the SDK subscription's item list) and only
then issues ApplyChanges, so a throw there propagated with every registration
still in place and nothing holding a handle to release them. The actor reads the
throw as a whole-chunk fault and re-issues the SAME batch on its backoff timer, so
each probe grew the bookkeeping again. Placement + apply moved to
PlaceAndApplyAsync behind a try/catch that unwinds via the existing
ReleaseFailedItem (per-item try, so one failure neither stops the rest nor
replaces the original fault); no ApplyChanges is re-issued, and items a partial
apply did reach the server with are reclaimed when the shard and session are torn
down on reconnect.
That reconnect is the other half: IsConnectionLevelFailure knew nothing about
ServiceResultException, which is how the SDK reports a dead session — so a dead
session classified as a per-tag resolution failure for the whole chunk. It now
also matches a narrow set of session/channel status codes (BadSessionIdInvalid,
BadSessionClosed, BadSessionNotActivated, BadSecureChannelClosed,
BadSecureChannelIdInvalid, BadServerNotConnected, BadConnectionClosed,
BadCommunicationError, BadNotConnected). Per-node codes such as BadNodeIdUnknown
stay per-tag — that is exactly the device-still-booting case the resolution timer
exists for.
Tests: DataConnectionStaleConnectResultTests (stale success neither connects nor
re-subscribes against the new adapter; stale failure does not disturb the current
attempt; the current generation's result is still processed after a stale drop) —
all 3 fail with the guards removed. DataConnectionOpcUaFaultClassificationTests
(bad-session drives reconnect — fails unfixed; bad-node-id stays per-tag).
RealOpcUaClientBatchRollbackTests (no residual registrations after an ApplyChanges
throw; repeated failed batches do not grow the bookkeeping) — both fail with the
rollback removed. Full DCL suite 342/342 green.
Alarm on-trigger runs (ScriptRunLauncher.RunAlarmScriptAsync) bypassed the
stuck-script watchdog, the script-error health counter, and
ScriptRunSummaryRecorder entirely, even though they run on the same bounded
ScriptExecutionScheduler pool as instance scripts. A wedged alarm body cost a
pool thread forever with no site event and no detach/replace, and a timed-out
or failed on-trigger run was invisible to health monitoring and the run
summary.
Bring the alarm path to parity with the instance-script path (RunScriptAsync):
capture the worker slot/run stamp in the first synchronous segment and arm
ArmStuckScriptWatchdog with the same detach-and-replace semantics and gauge;
resolve ScriptRunSummaryRecorder/ISiteEventLogger from the same DI provider and
thread RecordStarted/RecordCompleted/RecordFailed/RecordTimedOut; increment
ISiteHealthCollector.IncrementScriptError on both the timeout and general
failure catch blocks. LaunchAlarmScript/RunAlarmScriptAsync now take
healthCollector and serviceProvider parameters, passed from AlarmActor's
existing _healthCollector/_serviceProvider fields. No change to the
enqueue-anchored deadline, the shed cap, or "a stop does NOT cancel an
in-flight run" semantics.
Adds AlarmScriptWatchdogTests (previously zero coverage of LaunchAlarmScript):
wedged-worker detach + replacement, timeout-at-dequeue health counter
increment, and successful-completion summary recording. All three fail
against the pre-fix code (verified via stash) and pass after.
AlarmSummary.razor was gated RequireDeployment but never consulted
SiteScopeService: the site picker listed every site unfiltered and
GetSiteAlarmsAsync ran for whatever siteId the onchange event carried,
so a Deployer scoped to one site could watch any site's live alarms
(including CurrentValue/LimitValue process data). The page was missed
by the ad0100f2 scoping survey.
Fix mirrors EventLogs.razor/DebugView.razor: OnInitializedAsync filters
_sites through SiteScope.FilterSitesAsync, and both the site-change
handler and RefreshAsync (the shared fetch path used by the manual
refresh button and the 15s poll timer) independently re-check the
selected site against the filtered list before ever calling
AlarmSummaryService, denying/resetting an out-of-scope id rather than
merely hiding it from the dropdown.
Tests: bUnit render tests added to AlarmSummaryRenderTests.cs proving
(1) a scoped Deployer's picker omits the out-of-scope site and (2) a
forged out-of-scope onchange value is denied — no row renders and
GetSiteAlarmsAsync is never called for that site. AlarmSummaryVirtualizeTests.cs
updated to register the now-required SiteScopeService.
Reject read the row as Pending, then landed a blind full-row UpdateAsync. A
concurrent approve could win its own Pending->Approved CAS in that gap and enter
the multi-second WriteTagAsync relay; the reject's update then stamped
Status/ExecutedAtUtc/ExecutionError to Rejected mid-relay, so the two-person-control
record said Rejected for a write that physically executed at the plant. Works
cross-node (both centrals share one MS SQL).
- ISecuredWriteRepository/SecuredWriteRepository: new TryMarkRejectedAsync,
mirroring TryMarkApprovedAsync exactly (conditional UPDATE ... WHERE
Status='Pending', writes only the columns the reject transition owns). No schema
change, no migration, no rowversion column.
- HandleRejectSecuredWrite: uses the CAS; on loss re-reads the row and returns the
status that actually won (same UX shape as approve's lost-race error), and emits
the SecuredWriteReject audit row ONLY on a won CAS.
- Audited the other lifecycle writes: approve/reject/expiry (inline + the
opportunistic list sweep) now ALL transition only out of Pending, so winning the
approve CAS gives that leg exclusive ownership of the row and its post-relay
Executed/Failed update stays safe as an unconditional write. Documented in place.
- Tests: repository-level approve-vs-reject races in both orders (exactly one wins,
loser false, row reflects the winner only) plus reject-vs-expiry; handler-level
lost-race reject (conflict error, no write, no audit row) and approve on an
already-rejected row (no CAS, no relay, no audit).
ConnectAsync reset the event loop and client on reconnect but left
_alarmCts/_alarmSubCount/_alarmSubSources/_alarmStreamPrefix untouched.
The orphaned old RunAlarmStreamAsync loop kept retrying the disposed
client every 5s forever, and SubscribeAlarmsAsync's coverage check saw
the stale state and short-circuited the actor's post-reconnect
transparent re-subscribe (ReSubscribeAllAlarms) without ever opening a
stream on the new client — freezing the native-alarm mirror until
failover/redeploy. Reset the alarm state in ConnectAsync, mirroring the
existing lock-guarded block in DisconnectAsync/DisposeAsync.
An in-flight subscribe launched against adapter generation N can take up to
SeedOverallTimeout (120s) to answer. If the adapter is swapped in that window
(failover, or the row-97 in-place reconfigure), the completion was applied
unconditionally: it stored a dead session's handle under the tag key, so the
LIVE completion for the same tag read as a duplicate and had its working
monitored item released (tag resolved forever, no data); it re-stamped the old
endpoint's seed values with the CURRENT generation, bypassing the stale-data
guard; and a connection-level failure row from the dead adapter knocked the
healthy new connection back into Reconnecting. The alarm twin released the new
live alarm feed the same way.
Both completions now carry the generation AND the adapter instance they ran
against, and both handlers drop a completion whose generation no longer
matches - no handle store, no seed application, no per-tag or connection-level
failure processing, no duplicate-release logic. Handles the dropped completion
did create are released best-effort against the CAPTURED adapter (handles are
adapter-scoped; MxGateway's are plain ints that collide across sessions), and
the existing release sites in both handlers were switched to the message's
adapter for the same reason. The in-flight markers are deliberately left alone
(ReSubscribeAll/ReSubscribeAllAlarms clear them wholesale and may already have
re-registered the tag/source), and the requester gets a failure reply so its
retry timer re-sends.
Tests: DataConnectionStaleCompletionTests - stale handle not stored / live item
not released, release lands on the old adapter only, seed values not delivered,
stale connection-level failure does not re-enter Reconnecting, stale alarm
completion does not release the live feed. All five fail against the unfixed
handlers.
The WP-4 runs measured the stream path only up to the bounded send channel;
the probe deliberately substituted the socket hop with a controllable reader,
which is exactly where the R2 SiteStreamEventBatcher lives — so batching had
never been in any benchmark's hot path. The harness now also attaches per-site
wire probes: a real Kestrel h2c listener hosting the production
SiteStreamGrpcServer, dialed by the production SiteStreamGrpcClient with
batching negotiated, batch sizes read from the production batch_size histogram
via a MeterListener (--wire-probes-per-site, default 10; 0 restores the old
shape). Full-scale run (10x500x75, 20-min window, results doc §8 + raw JSON):
899,807 events in 12,462 frames (mean 72.2 ev/frame, p50=p99=75), wire P99
43.4 ms vs the 250 ms budget — every batch pays the full 25 ms linger since a
75-event burst never hits the 100 cap — zero drops, zero faults, offered load
100.0% of nominal with fan-out doubled. CI smoke Fact now asserts the wire
path carries traffic and observes frames.
The gauge test polls baseline+1, but the process-global gauge's baseline is
captured once: a parallel class's streams closing after capture lowers the
gauge below baseline+1 forever. Survey confirmed only SiteStreamGrpcServerTests
and CleanupVerificationTests drive the real SubscribeInstance handler that
moves the counter; both now share the SiteStreamSubscriberGauge collection
(row-96 pattern). 4/4 full-suite runs green, durations unchanged.
Row 79 executed its own prescribed remedy without waiting for a second
corruption: all ten rig data dirs (8 docker/ nodes + 2 docker-env2/ site
nodes) move from virtiofs bind mounts to named docker volumes
(scadabridge-<node>-data), seeded in place with both rigs stopped and
live-verified (site-a localDbReplicationConnected=true / backlog 0 over
the migrated files; env2 stable, 0 restarts). The feared inspectability
loss is moot — OrbStack exposes volumes at ~/OrbStack/docker/volumes/.
The rebuild surfaced latent env2 drift (pre-Secrets-0.6.2 image): env2
now carries the x-secrets-store-env posture + central data volumes.
Row 93's silent maintenance trap is now loud: SiteEventsSchemaCalibrationTests
pins the site_events columns/indexes the EventLogPurgeService size model is
calibrated against, failing with a pointer to the memo §4 recalibration on
any schema drift. Rows 92 (accepted by design) and 94 (standing tripwire)
annotated accordingly.
ReadStoreSnapshotAsync's snapshot is read off the actor thread and consumed only
when its piped message lands. Row 101 guarded the DEFINITIONS half of that race;
the CONFIG half was left to machinery whose reach is time-limited, and both legs
of the instance reconcile sweep were reachable through it:
- Row-72 leg: a read dispatched before a deploy's persist, landing after the
deploy completed AND after the pipeline released ownership, carries the
revision the deploy replaced. TryRestartStaleInstance then "repairs" the
instance BACKWARD, re-deploying the snapshot's own superseded ConfigJson.
InstanceIsOwnedByADeployPipeline covers only in-flight deploys, and the
per-revision claim has never seen the old revision, so nothing stopped it.
- Row-78 leg: a read dispatched before a delete/disable commits still shows an
enabled row while the actor is already stopped, so StartStoredInstance
resurrects it — and delete clears _sweepAttemptedRevisions, freeing the claim.
Both verified real by neutralising the freshness check and re-running the new
tests (the fail-before site event literally reads "running revision rev-2 is
stale, applying stored revision rev-1"; the deleted instance came back).
_definitionsWriteGeneration becomes _storeWriteGeneration, covering both tables
(the snapshot is one consistent read and is discarded whole, so one counter is
the right shape); DefinitionsGenerationAtRead -> StoreGenerationAtRead on both
internal records. The generation is now bumped at BOTH ends of every mutation —
when the off-thread write is dispatched and when its completion message lands —
because the config paths advance actor state before the write, so an
initiation-only bump would let an in-window read look fresh. Disable, delete and
the init-failure rollback have no actor-thread completion message of their own,
so a minimal StoreWriteCompleted self-message carries the bump for them.
CDC-replicated writes, snapshot resyncs, restored files and
SiteReconciliationActor's fetch deliberately do NOT bump: discovering exactly
those is the sweep's reason to exist.
A stale sweep result is now dropped WHOLE (no overlay apply, no start, no
restart). A stale boot load cannot be dropped — startup must finish — so it
re-dispatches LoadDeployedConfigs(); a fifth consecutive superseded read is
applied anyway with a Warning, and a late one arriving after the boot completed
is dropped rather than looping against the S5 retry timer.
Tests: three fail-before regressions (backward restart, resurrection, boot
re-read) plus two guard twins pinning that the legitimate row-72 restart and
row-78 start still fire on a current-generation snapshot. The row-101
definitions tests keep passing; FreshEmptySweepSnapshot's hardcoded generation
moves 1 -> 2 now that one artifact apply bumps at both ends.
Docs: Component-SiteRuntime.md's stale-snapshot paragraph now describes the
whole snapshot (both halves, both consumers, the boot re-dispatch, the
CDC-no-bump rule), the components mirror follows, and the arbitration memo gains
§11 extending §10 to the config half.
An off-thread snapshot read racing an artifact apply could land after the
apply populated the overlay; its stale absences then tombstoned and stopped
just-applied LIVE connections, which nothing re-creates until an instance
redeploy (the removal evicts the hash slot the overlay push is gated on).
Each read now captures a definitions-write generation at dispatch; a stale
result is discarded whole (additions too — a stale row would reconfigure a
live connection backward). Deterministic repro fails before the fix; a
fresh-generation guard test pins that row-90 disappearance still fires.
`DataConnectionManagerActor.HandleCreateConnection` is create-only (warns and
returns for a name it already holds), so `EnsureDclConnection`'s documented
"a changed config re-issues the create so the DCL adopts it" was never true on
ANY node: a corrected endpoint/credential/backup/failover-count took effect only
when the connection was next built from scratch (node restart, singleton
failover, remove-then-re-add), while the DM's hash cache recorded it as applied.
Rows 89/90 made that honest with a Warning; this closes it.
New `ReconfigureConnectionCommand` (Commons, same shape as the create):
- `DeploymentManagerActor.EnsureDclConnection` now has three outcomes off its one
per-name hash slot — unchanged -> nothing at all, new -> create, CHANGED ->
reconfigure — and logs "Reconfigured DCL connection {Connection}
(protocol={Protocol}) in place" at Information in place of the old Warning. The
winner-resolution / arbitration / tombstone logic is untouched; only what
happens after a change is detected changed.
- `DataConnectionManagerActor` FORWARDS the command to the owning connection
actor, falling back to a create for a name it does not hold, so the command is
safe whatever the manager's state.
- `DataConnectionActor` handles it in ALL THREE states and never stashes it — a
connection pointed at a wrong or dead endpoint lives in Connecting/Reconnecting,
which is exactly when a correction arrives; stashed it would wait on the broken
configuration forever. Handling replaces protocol/primary/backup/failover-count,
re-arms endpoint selection to Primary (the new config's primary is the intent,
whatever the old config had failed over to), and calls the new shared
`SwapAdapter` helper — now used by both failover sites too, so the swap exists
once: unhook + fire-and-forget dispose -> _factory.Create -> rewire -> bump
_adapterGeneration LAST. `BecomeReconnecting(afterReconfigure: true)` then
reconnects immediately (StartSingleTimer replaces any armed reconnect, so no
racing attempts) with the unstable-disconnect accounting and failover check
suppressed — the link was dropped on purpose, and counting it as an unstable
cycle could trip a failover onto the NEW backup before the new primary had been
tried once. ReSubscribeAll/ReSubscribeAllAlarms then re-issue everything from
the untouched subscription registries. Idempotent: a reconfigure carrying the
configuration already in force is a no-op.
Stop-and-recreate was rejected as destructive: subscriptions live inside
DataConnectionActor and InstanceActor cancels its subscribe-retry timer on first
success, so a rebuilt actor would have no subscribers and nothing would ever
re-subscribe.
Tests: new DataConnectionReconfigureTests (7; 5 fail / 2 pass against a
behavioural revert). DclConnectionArbitrationTests gains two row-97 tests and its
CDC-arrival test now asserts the second push is a reconfigure; its ping-pong
guard also asserts zero reconfigures, since a flip would now show up there.
SiteRuntime-010 renamed to ..._ReissuesReconfigureCommand. 3 fail against a
reverted EnsureDclConnection. DataConnectionLayer.Tests 327/327,
SiteRuntime.Tests 643/643, solution build 0 warnings.
Docs: Component-DataConnectionLayer.md gains an "In-place reconfiguration"
section under Endpoint Redundancy; Component-SiteRuntime.md, and the
docs/components/ mirrors, updated; the arbitration decision memo gains an
appended §9 postscript resolving its residual 1 (residuals 2 and 3 stay open).
Register row 98 — the site-scope authorization gate was FAIL-OPEN on an
unresolvable target. The resolve-then-check shape was written as
`if (target != null) EnforceSiteScope(...)`, so a null resolve skipped the
check entirely and a site-scoped caller passed the gate for any id in the
system. Authorization now fails CLOSED: a target that cannot be PROVEN to sit
inside the caller's scope is treated as outside it.
The defect turned out to have THREE more instances than the register recorded.
Besides `EnforceSiteScopeForInstance` and `EnforceSiteScopeForIdentifier`, the
same inline shape sat in `HandleGetInstance`, `HandleGetSite` and
`HandleGetDataConnection` — read handlers that resolve their own target and so
never call the helpers. Those were the sharper half of the bug: they answered
success-with-null for a nonexistent id and Unauthorized for an out-of-scope
one, making the pair an id-enumeration oracle over the whole fleet. All five
now route through one shared `DenyUnresolvedTargetIfScoped` arm.
Two properties are load-bearing and pinned by tests:
- System-wide callers (Administrator, or a Deployer with no site grants) are
untouched — they return at the guards above the lookup and keep the
handler's ordinary "not found" / null answer.
- The denial is indistinguishable from an out-of-scope denial: same
ManagementUnauthorized shape, and a message that never states whether the
target exists.
Register row 99 — CLI `audit query --status delivered` failed client-side while
the server would have accepted it: `AcceptOnlyFromAmong` matches ordinally
against the capitalised literals. Same treatment row 88 gave `deploy status` —
the client-side gate is removed, the valid set moves into the option help text,
and the server becomes the single authoritative check.
That required ADDING the server check, not just relying on one: the shared
audit filter parser was LAX, silently dropping an unrecognised value and
leaving the dimension unconstrained — so removing the CLI gate alone would have
turned a loud client-side failure into a plausible, complete-looking page that
did not answer the question asked. New `ParseEnumListStrict` (case-insensitive,
rejects ordinals — `Enum.TryParse` accepts "3", and a filter is a name-based
contract) backs the ManagementService /api/audit/query + /api/audit/export
endpoints, which now return 400 BAD_REQUEST naming the valid set.
Other AcceptOnlyFromAmong sites surveyed in the audit group: `--channel`,
`--kind`, `--status` on BOTH `query` and `export` (6 uses) plus `export
--format` (rejected `--format CSV` though the server lowercases it) — all seven
fixed here. Deliberately left alone: `template folder reorder --direction`
(pure client-side literal, no server-side parse to defer to) and the global
`--format json|table` (CLI-local output selection, never sent to a server).
The CentralUI AuditLogPage drill-in parser and the CentralUI export endpoint it
feeds deliberately KEEP the lax parse — they read a query string a human may
have bookmarked, where a stale fragment must still render the page. Both
contracts now live side by side in AuditQueryParamParsers and are documented as
non-reconcilable.
Docs: Component-ManagementService.md gains a "Site-scope gate: fail-closed on
an unresolvable target" section and a strict-filter note on the audit
endpoints; Component-Security.md records the fail-closed shape under Deployer
site scoping; the CLI README marks the audit filter values case-insensitive and
states the no-silent-drop / no-ordinals contract.
Tests: 5 new ManagementActorTests fail before the row-98 fix (verified by
reverting ManagementActor.cs) and cover scoped+nonexistent -> denied,
system-wide+nonexistent -> unchanged (both Administrator and no-grant Deployer),
scoped+in-scope -> allowed, and the indistinguishability property. 21 CLI tests
fail before the row-99 fix. Four pre-existing tests that pinned the removed
client-side gate were rewritten to pin the new contract.
The compile cache was a `static` class, so its entry table and its Hits/Count
counters were one process-global blob every test class in the SiteRuntime
assembly wrote into. Row 91 last wave neutered the resulting flakes (per-test
GUID keys + one-sided counter bounds), but the shared mutable state itself
remained: any test calling Clear() evicted entries other parallel classes had
just inserted, and a Clear() from a class outside the serialized collection
would degrade the others to recompiles and re-open the row-91 flake class.
PREMISE CORRECTION: the register speculated production clears "per-instance or
per-deployment script entries" and offered ClearKeys(prefix) as the natural fit.
There are NO production callers of Clear() at all — grep of src/ finds only
GetOrAdd (ScriptCompilationService.CompileCore). Clear() existed solely so
tests could reset the global. A key-scoped API would therefore have been new
production surface built for nobody, so the instance-scoped seam is the fix and
Clear() is deleted outright rather than migrated.
- SiteScriptCompileCache: `internal static class` -> `internal sealed class`;
Cache/EvictionLock/_hits/_accessSequence become instance fields. LRU batch
eviction, the double-checked sweep and the ConcurrentDictionary.ToArray()
snapshot are byte-for-byte the same policy. Clear() removed.
- ScriptCompilationService owns one cache. Public ctor (the DI path) builds its
own; a new internal ctor takes one for tests. Production N4 sharing is
unchanged and now rests on the DI SINGLETON registration — the one service
instance is threaded to DeployCompileValidator and to every Instance Actor —
so that lifetime is load-bearing and is pinned by a new registration test.
- Tests: all three Clear() callers migrated off it. Each test now owns its cache,
which lets the assertions get STRONGER — the row-91 workarounds are retired:
Hits/Count are exact again (no >= baseline bounds), the per-test GUID key
token is gone, and ForeignEntrySlack (512 entries of allowance on the bound)
drops to the racing-thread count in the concurrency test. New test
SeparateCaches_DoNotShareEntriesOrCounters pins the scoping directly: filling
one cache past MaxEntries leaves another's entry, hit count and identity
intact. DeploymentWarmThenGateTests additionally asserts Count == 1 now that
its counter is private.
- The [Collection("SiteScriptCompileCache")] serialization from row 91 is
RELAXED (removed from all three classes) — they no longer touch any shared
state, so there is nothing left to serialize.
- Docs: Component-SiteRuntime.md gains the ownership/singleton note and drops
the Clear() reference; DeploymentManagerActor / DeployCompileValidator /
ServiceCollectionExtensions comments corrected to say where the cache lives.
Build: 0 warnings. SiteRuntime.Tests 643/643, three consecutive full-parallel
runs. AuditLog.Tests correlation classes (other ScriptCompilationService
consumers) green.
Joseph Doherty