Cross-task regression caught by the combined sweep: TopologyLiveUpdatesTimerDisposeRaceTests
(T12) builds a real DeploymentService, whose constructor grew an IStaleInstanceProbe
parameter in T13 (P7-6 up-to-date pre-filter) — 4 DI-resolution failures on the
integration branch that neither task saw in isolation. Registered a substitute, matching
TopologyPageTests.
Bundle review Minor: ApiMethodForm/SharedScriptForm copied TemplateScriptDialog's sandbox
pattern exactly, whose finally re-renders unconditionally; the plan asked the continuation
to be guarded on disposal, so a _disposed flag now suppresses the post-cancel
StateHasChanged (the reference dialog keeps its original shape — out of T11's scope).
Mirrors the Medium wave's DebugView P6-6 fix (_sessionLock + _disposed,
publish-then-recheck-disposal under one lock) for the two TTL-bounded/
unbounded lesser instances the Medium wave deferred:
- TransportImport.razor.cs: TryLoadAsync awaits IBundleImporter.LoadAsync and
used to publish the result into _session unconditionally. A circuit torn
down mid-await left Dispose() with nothing to release (session still null),
so the freshly-opened server-side BundleSession stayed registered, stranding
one of MaxConcurrentImportSessions (register row 43) for the full session
TTL. Now publishes and re-checks disposal atomically under _sessionLock, so
exactly one side (TryLoadAsync or Dispose) abandons the session.
- Topology.razor: OnInitializedAsync awaits LoadDataAsync before calling
StartLiveUpdatesTimer. A circuit torn down mid-await left Dispose() with no
timer to stop yet, so the timer StartLiveUpdatesTimer creates once the await
resumes was never disposed — it ticked against a torn-down component every
15s until process restart. StartLiveUpdatesTimer/StopLiveUpdatesTimer/
Dispose now create/replace/take the timer atomically under _timerLock
together with a _disposed check, so a late-arriving (re)start refuses to
create a timer once disposed.
Tests (bUnit, both proven to fail against the unfixed code):
- TransportImportSessionDisposeRaceTests: dispose-during-load abandons the
just-opened session immediately; dispose-after-successful-load abandons
exactly once.
- TopologyLiveUpdatesTimerDisposeRaceTests: a late StartLiveUpdatesTimer call
after Dispose leaves no live timer; Dispose stops the timer exactly once
(idempotent double-dispose); OnLiveUpdatesToggled(off) + Dispose leaves no
live timer. All assert on deterministic field/call-order state, no sleeps.
Register row 103 stays open for integration to close per the plan's shared
constraints (register edits happen at integration only).
Neither Test Run form implemented IDisposable, so navigating away mid-run left
_runCts uncancelled: the sandbox kept the run alive until its own internal
script timeout, and the eventual continuation touched a torn-down component's
_running/_runResult fields. TemplateScriptDialog already got this right — both
forms now mirror it exactly: @implements IDisposable plus a Dispose() that
cancels + disposes _runCts.
ScriptAnalysisService.RunInSandboxAsync is marked virtual (test-only seam,
mirroring the existing virtual on DebugStreamService.StartStreamAsync/
StopStream) so a TaskCompletionSource-gated stub can hold a run open
deterministically. New bUnit tests (ScriptFormSandboxDisposalTests) prove, for
both forms: disposing mid-run cancels the token the sandbox observed, and the
cancelled continuation resolves with no unhandled exception and no _runResult/
_running mutation. Proof of failure: reverting only the two .razor changes
(keeping the virtual seam) reproduces all 4 failures — the token is never
cancelled and the no-op assertions time out.
Three fetch initiators (search debounce, paging, push/coalesce reload) could
each have a LoadDataAsync flight in flight at once; completion order was not
call order, so a slow, stale flight could resolve after a newer one and
clobber the grid the operator was already looking at. Add a monotonic
_requestGeneration counter, bumped at fetch start and checked at apply —
last-write-wins by generation, not by whichever await resolves first.
Separately, the filter-debounce timer wrote `_currentPage = 1` directly on
its raw System.Threading.Timer callback thread — a Blazor threading
violation, since component state may only be mutated on the renderer's
sync context — and that write was not covered by the generation guard at
all. LoadDataAsync now takes a resetToFirstPage flag, resolves the target
page into a local, and applies it only behind the same generation check as
every other piece of state, via a new DispatchFilterReloadAsync that
marshals through InvokeAsync like the existing push-reload path.
Tests (DeploymentsRequestGenerationTests, bUnit): a delayed stale flight
arriving after a newer one does not overwrite the grid; a stale
filter-debounce flight does not reset the page after a newer navigation.
Both driven deterministically via TaskCompletionSource-backed repository
stubs. Verified to fail against the unfixed code (swapped via
`git show HEAD:<path>`, not `git stash` — repo-wide and unsafe across the
parallel worktree wave).
P6-7 (CWE-1236): AuditEndpoints CSV export quoted cells on comma/quote/CR/LF
but never neutralized a leading =, +, -, @, TAB, or CR — inbound-API-caller
text landing in Target/ErrorMessage/RequestSummary/ResponseSummary/Extra
could execute as a formula/DDE trigger when opened in Excel. AppendField now
prefixes a literal single quote on free-text columns only (never the
GUID/enum/numeric/date/bool columns the exporter formats itself), applied
before RFC 4180 quoting. The CSV was also missing ExecutionId/
ParentExecutionId/SourceNode that the JSONL export already carried; both are
now 24 columns in the same AuditExportRow order. A second, independently
maintained CSV writer (CentralUI's AuditLogExportService, feeding the Audit
Log page's bulk-export button) had the identical gap and got the identical
fix + column alignment.
P6-8: `audit export` was the one CLI HTTP path with no deadline at all — a
residual gap from the ManagementHttpClient timeout-honesty fix, which made
HttpClient.Timeout infinite and the per-call token the single deadline
everywhere except this streamed GET. RunExportAsync now takes a headers-
received deadline (CancelAfter, reusing ManagementHttpClient.
DefaultConnectTimeout) around the initial GetAsync, an inactivity deadline
around the streaming copy (rearmed on every successful read, so large exports
are bounded by silence, not total size), and an external CancellationToken
wired to Console.CancelKeyPress in AuditCommands so Ctrl-C cancels promptly.
A stalled/faulted stream now produces a friendly error noting the partial
output file instead of hanging or dumping a stack trace.
A debug-stream session created after the consumer had already gone away was
leaked permanently. Both consumers awaited StartStreamAsync (up to 30s) and only
then published the session; a circuit teardown / client disconnect inside that
window ran the release path first, found nothing registered and stopped nothing.
The session stayed registered in the singleton DebugStreamService, whose
keepalive renews registered sessions forever (the 5-minute orphan reaper only
collects UNregistered ones), pinning the site gRPC subscription, the
DebugStreamBridgeActor and the dead consumer's object graph until process
restart.
Both call sites now publish the session and re-check liveness under a single
monitor, so exactly one side tears it down and neither can miss it:
- DebugView.Connect/Dispose/Disconnect share a _sessionLock; Dispose sets
_disposed and takes the session in one step, Connect publishes only if not
disposed and otherwise stops the session it just created. Dispose can run off
the renderer's thread, so the Blazor sync context is not relied on. Disconnect
and the onTerminated callback go through the same take-once helper, which also
makes a repeated Dispose stop-once instead of stop-again.
- DebugStreamHub uses the per-connection Items monitor: OnDisconnectedAsync marks
the connection gone before releasing, SubscribeInstance publishes only if
neither that flag nor ConnectionAborted is set, and UnsubscribeInstance takes
the id atomically.
DebugStreamService.StartStreamAsync/StopStream become virtual purely as a test
seam (no behaviour change, keepalive/orphan contract untouched) so the race can
be driven deterministically by a gated stub.
Tests: DebugViewConnectDisposeRaceTests (3) + DebugStreamHubSubscribeRaceTests
(3). Verified fail-unfixed: with the two source fixes stashed, both
dispose-mid-start tests report zero StopStream calls and the DebugView
double-dispose test reports two.
P6-4: HandleListDataConnections took no user at all and returned EVERY
site's connections to any authenticated caller, nullifying the row-98
fail-closed gate on HandleGetDataConnection — Get denied exactly what List
handed over. DataConnectionPublicShape already scrubs secrets, so the leak
was site topology plus a connection-id existence oracle, but it still
defeated scoping. List now takes the AuthenticatedUser: an explicit SiteId
goes through EnforceSiteScope (the ListAreas shape), and an unfiltered list
from a site-scoped non-Administrator caller is narrowed to their permitted
sites (the ListSites shape). GetRequiredRoles is untouched — scoping is not
a role gate, so the frozen 138-command authz matrix is unaffected.
The Central UI Design > Connections page bypasses the actor entirely
(ISiteRepository.GetAllDataConnectionsAsync direct) and IS reachable by a
scoped user: it is gated RequireDesign, and SiteScopeService explicitly does
NOT exempt Designer (register row 86). It now filters sites through
SiteScope.FilterSitesAsync and keeps only their connections, mirroring
EventLogs/AlarmSummary.
P6-5: four handlers resolved their target — and on approve/reject reported
its status — BEFORE the scope check, so an out-of-scope caller could tell a
real id from a fake one, and read a secured write's lifecycle status, from
the error shape alone. The scope check is hoisted above the not-found and
status checks in HandleSubmitSecuredWrite, HandleApproveSecuredWrite,
HandleRejectSecuredWrite and HandleDebugSnapshot, with the unresolvable case
routed through the existing DenyUnresolvedTargetIfScoped so the denial is
identical whether or not the target exists. System-wide callers keep their
ordinary not-found/status answers, and the P6-1 CAS semantics on
approve/reject are unchanged (only the checks above the CAS moved).
Tests: 8 new ManagementService tests + 2 CentralUI bUnit tests, all verified
to FAIL against the unfixed handlers (src reverted, tests re-run) and pass
after — including the unscoped/Administrator control cases proving behaviour
for system-wide callers is unchanged. Full ManagementService.Tests 545 green,
CentralUI.Tests 1073 green.
AlarmSummary.razor was gated RequireDeployment but never consulted
SiteScopeService: the site picker listed every site unfiltered and
GetSiteAlarmsAsync ran for whatever siteId the onchange event carried,
so a Deployer scoped to one site could watch any site's live alarms
(including CurrentValue/LimitValue process data). The page was missed
by the ad0100f2 scoping survey.
Fix mirrors EventLogs.razor/DebugView.razor: OnInitializedAsync filters
_sites through SiteScope.FilterSitesAsync, and both the site-change
handler and RefreshAsync (the shared fetch path used by the manual
refresh button and the 15s poll timer) independently re-check the
selected site against the filtered list before ever calling
AlarmSummaryService, denying/resetting an out-of-scope id rather than
merely hiding it from the dropdown.
Tests: bUnit render tests added to AlarmSummaryRenderTests.cs proving
(1) a scoped Deployer's picker omits the out-of-scope site and (2) a
forged out-of-scope onchange value is denied — no row renders and
GetSiteAlarmsAsync is never called for that site. AlarmSummaryVirtualizeTests.cs
updated to register the now-required SiteScopeService.
Deferred-work register rows 37, 86, 87, 88 — the structural duplication and the
three behavioral divergences it had accumulated, fixed together.
Row 37 (structural). `IDeploymentManagerRepository.QueryDeploymentSummariesAsync`
is DELETED. It and `QueryDeploymentListPageAsync` were two paged reads of the same
insert-only table with different contracts — raw offset vs page number, no total vs
total, no instance join vs joined, instance-id-list scoping vs site-id scoping. The
CLI/HTTP `QueryDeployments` handler now goes through `QueryDeploymentListPageAsync`
with a `DeploymentListFilter`, exactly like the Central UI page. The
`DeploymentRecordSummary` projection went with it (`DeploymentListRow` is the single
row shape); its `MaxPageSize` const moved to `DeploymentListPage`, which also gains
`DefaultPageSize`.
Row 86 (Medium, authorization — the headline). `SiteScopeService.ResolveAsync` was
claims-only, so an Administrator carrying `SiteId` claims saw the whole fleet through
the CLI (`ManagementActor.EnforceSiteScope` has always exempted Administrator) and
only their granted sites on the Deployments page. One user, one dataset, two answers.
Administrator = system-wide is the intended semantic on both sides; the UI was the
defect. The role claim is read explicitly with an OrdinalIgnoreCase comparison rather
than via `IsInRole`, which compares claim VALUES ordinally and only resolves when the
identity carries the matching roleType — the two sides must not disagree on a
differently-cased role name. An Administrator's site claims are not collected at all,
so the invariant "system-wide => empty permitted-id set" holds for every caller.
Designer is NOT exempt: the class doc claimed "any Admin/Design user" is system-wide,
which was true of neither side, so the doc was corrected rather than the bypass
widened.
Row 87 (Low). The scoped branch used to resolve the site grant into an instance-id
list by loading EVERY instance in the system, non-transactionally, on every scoped
query; the grant now rides as `SiteIdScope` and is resolved by the query's own join.
An unparseable permitted-site id is DROPPED, never ignored — dropping can only narrow
a grant, discarding it would widen it to fleet-wide. And `PageSize <= 0` no longer
means UNLIMITED down at the repository (`--page-size 0` materialized the whole
retention window): it means "unspecified" and takes the CLI's own default of 50,
capped at 1000. Bounding without telling would be silent truncation, so the handler
now returns a `DeploymentQueryResult` envelope carrying `totalCount` plus the page
coordinates actually applied — with a CLI table projection (mirroring row 57's
`AuditConfigTableProjection`) that unwraps it and appends a "showing N of M" note.
Row 88 (Low, UX). An unrecognised `--status` fell back to "no filter", so a typo like
`--status Sucess` returned every status — a complete-looking answer to a question the
caller never asked. It is now a `ManagementError` naming the valid values. Parsing
stays case-insensitive; a numeric string is rejected too, since `Enum.TryParse`
accepts "7" and yields an undefined value (guarded with `Enum.IsDefined`). Loud
failure over silent full results, per row 46.
Tests: 7 new `SiteScopeServiceTests` pinning Administrator = system-wide (verified
failing 6/7 against the pre-fix service, the Designer pin passing both ways); the
ManagementActor deployment suite migrated onto the consolidated query with new pins
for the status error, case-insensitive parse, numeric rejection, page-size default/
cap/floor, empty-scope-on-garbage-grant, and "reads no instances at all"; a repository
test for site-scope+status composing over a bounded page; and 6 CLI tests for the new
table projection.
ROW 80 (observability correctness). An unreplicated site published
LocalDbReplicationConnected=false / LocalDbOplogBacklog=0, contradicting the
nullable semantics every doc and the proto already state ("null means no
reading, NOT disconnected with an empty backlog").
Root cause: the replication engine is registered unconditionally and idles
without a peer, so ISyncStatus answers false/0 on a node that was never meant to
replicate, and LocalDbReplicationStatusReporter passed that straight through.
Two consequences: an operator read connected:false on a healthy site as a fault,
and central's null branches — row 33's conditional KPI emission and
sustained-badge anchor exclusion — never received the null they exist for, so
rig site-b/site-c emitted a permanently misleading
localDbReplicationConnected=0 series.
Fix mirrors the SiteEventLogReplicationCheck precedent: the Host evaluates
SiteLocalDbSetup.ReplicationIsConfigured ONCE and passes it to
AddLocalDbReplicationHealthBridge as a third provider; unconfigured
short-circuits the probe to (null, null). HealthMonitoring never reads LocalDb
config itself, and the same captured value now also feeds
SiteEventLogReplicationCheck so the two surfaces cannot disagree.
ISiteHealthCollector.SetLocalDbReplicationStatus widens to bool? — the
non-nullable bool was the type-level root cause (null was inexpressible).
false now means exactly one thing: configured, peer session down = a real fault.
Wire/central/UI needed no change: the proto already carries BoolValue/Int64Value
wrappers and the aggregator + KPI source already branch on null. This makes code
match doc, so no doc claim was weakened.
Watch the trap that caused this: configured-ness is PeerAddress OR ApiKey (only
the dialling half sets a peer address). The pre-existing SiteLocalDbWiringTests
fixture sets ApiKey and IS a replicating node, so its false/0 assertion is
correct and is kept, with a comment saying so; the new
UnreplicatedSiteLocalDbHealthTests fixture is the genuinely-unreplicated one.
ROW 73 (silent loss of self-heal). AkkaHostedService skips
SiteReconciliationActor when no IDeploymentConfigFetcher is resolvable, and that
skip was a lone boot Warning — a node with no self-heal path at all went on
reporting itself perfectly healthy forever.
Adds tri-state SiteHealthReport.StartupReconcilerActive (null = not reported,
i.e. the central self-report or a pre-row-73 site build; true = running;
false = booted without it), latched once at boot by the Host in both branches.
Additive proto field 29 (BoolValue, vendored C# regenerated via
docker/regen-proto.sh centralcontrol; csproj restored, verified empty diff).
The skip branch also raises its log to Error — no configuration makes the
fetcher legitimately absent on a site node — and writes a best-effort
deployment/Error site event so the evidence outlives the boot log.
Central treatment: a "No self-heal" WARNING badge on the site tile, not a
failure. Config still reaches the node via LocalDb replication while it is up;
what it lost is recovery from a deploy that landed while it was down — degraded,
not offline. Only the explicit false is badged (badging null would light up the
whole fleet the moment central is upgraded ahead of the sites), and offline
sites are skipped, matching the replication-backlog badge.
Fail-before evidence: with the row-80 gate behaviourally reverted,
UnreplicatedSite_HealthReport_ReportsNullForBothLocalDbFields fails
"Expected: null / Actual: False" and 3 of the 8 new reporter unit tests fail.
Tests: HealthMonitoring 128, Host 518, CentralUI 1051, Communication 717,
Commons 715, SiteEventLogging 81 — all green.
The Secured Writes page rendered Approve and Reject as IDENTICAL red buttons.
Row 28 fixed the labels but left the styling, and the styling was the worse
half: at the moment of a two-person-control decision the colour carried no
discriminating information at all, and the affirmative half of the control wore
the delete affordance.
Root cause is that `danger` was a two-state flag for a three-state concept, so a
consequential-but-not-destructive action had to choose between looking like a
delete and looking like the safe default. Adds the missing middle rung:
DialogSeverity { Normal -> btn-primary, Caution -> btn-warning,
Destructive -> btn-danger }
DECISION. Approve -> Caution (amber): approving executes a real write to live
plant equipment, so it is not a benign primary action, but it is not destroying
anything either. Reject KEEPS Destructive (red): it is terminal and irreversible
for the request — the Operator must resubmit — and holding it red is what keeps
the two opposite decisions visually distinct now that Approve is amber. Dropping
Reject to neutral was considered and rejected: it would invert the bias, making
declining look like the safe default. Colour is never the only signal — both
buttons carry their verb.
Only Destructive inherits the row-28 "Delete" label fallback; an unlabelled
Caution reads "Confirm", since a caution action is by definition not a delete —
otherwise row 28 would come back through the new door.
ADDITIVE OVERLOAD, deliberately. The boolean form stays the signature for all
~34 existing call sites, which are genuinely two-state, so nothing in src/
outside SecuredWrites.razor changed. Adding a fifth optional parameter to the
boolean form instead would have silently un-matched every existing four-argument
NSubstitute stub — the exact row-28 gotcha (null Task, caller NREs, no compile
error). The overload cannot: the only breakage was six hand-written IDialogService
fakes failing to compile, which is the loud failure mode. That gotcha did land
once, in SecuredWritesTests' own stub, and is now stubbed on both overloads with
a comment saying why.
DialogState's `bool Danger` became `DialogSeverity Severity` rather than gaining
a second field — it is read only by DialogHost, and one field cannot disagree
with itself.
Fail-before, both halves:
- Approve call site reverted to `danger: true` -> both new Approve pins fail.
- Caution remapped to btn-danger in DialogHost -> the amber styling pin fails.
Tests: DialogSeverityStylingTests renders the REAL DialogService behind a real
DialogHost (FailoverConfirmLabelTests' harness), so it pins what the user sees,
not what an argument was — all three severities, both label fallbacks, and the
boolean adapter. SecuredWritesTests pins that the page asks for Caution, with a
negative control that it did NOT also go through the boolean form (without which
the assertion would pass on a call site left at danger:true). Reject's red is
pinned too, so a later "make this page less alarming" pass cannot flatten both
to one colour.
No Playwright selectors needed updating — the suite has no Secured Writes
coverage, and no `.btn-danger` selector elsewhere targets this page.
Docs: Component-CentralUI.md gains the severity contract under Technology (next
to the row-28 label contract) plus the decision on the Secured Writes section.
CentralUI.Tests: 1055/1055 green.
The Health dashboard's "Trigger failover" confirmation offered a red button
labelled *Delete*. The copy was right; the button was not. `DialogHost` derived
the confirm label solely from `danger`, so `danger: true` meant the verb was
literally "Delete" for every caller — and no caller could say otherwise, because
`IDialogService.ConfirmAsync` had no label parameter.
`danger` now selects styling only; a new optional `confirmLabel` names the action
and wins over the fallback. The kind/danger fallback is deliberately RETAINED
("Save" for a prompt, "Delete" for a bare danger confirm, "Confirm" otherwise):
around 20 call sites are genuine deletes where it is correct, and the Playwright
suites click the literal "Delete". Requiring the label everywhere would have
churned those 20 sites for no behaviour change, with the miss-detection resting
on Playwright runs that need a live rig.
Audited all 34 confirm call sites; 9 read "Delete" for something that is not a
delete and are relabelled:
Fail over CentralFailoverControl (central pair, site pair)
Approve SecuredWrites — approve (this one WRITES TO A DEVICE)
Reject SecuredWrites — reject
Disable Topology — disable instance
Discard SiteCallsReport, NotificationReport, ParkedMessages (bulk + single)
The remaining 25 are correct as they stand: 18 genuine deletes plus 7 non-danger
confirms ("Revert to base", the three Retry paths) whose button reads "Confirm".
Tests: FailoverConfirmLabelTests drives the real DialogService behind a real
DialogHost — a substitute would pin the argument but not what the operator reads
— covering both branches, that the button keeps btn-danger, and that an
unlabelled danger confirm still falls back to "Delete". Note its click is
deliberately not awaited: with a real dialog the handler parks on ConfirmAsync
until answered, so awaiting deadlocks. Existing IDialogService fakes and
NSubstitute matchers gain the 4th parameter (an optional param is baked into the
call site, so a 3-arg Arg.Any stub silently stops matching a labelled call).
LocalDbOplogBacklog was reported and Prometheus-exported with nothing consuming
it: not sampled into KPI history, so a purge-burst spike could not be reviewed
after the fact, and no threshold treatment anywhere. Both halves shipped.
KPI SAMPLING. SiteHealthKpiSampleSource now emits localDbOplogBacklog and
localDbReplicationConnected (0/1), and the backlog is charted on the Health
dashboard's per-site trend panel. Charting is the point: a point-in-time tile
can never answer 'did that spike drain?', which is the only thing separating an
expected purge burst from a stalled peer.
Both are CONDITIONALLY emitted rather than coerced with ?? 0 like the
auditBacklogPending metric beside them. Both health-report fields are nullable
and null means 'no reading' — replication not configured (rig site-b/site-c) or
a failed poll — never zero. A fabricated 0 would render a pair that cannot read
its own oplog, or one deliberately unreplicated, as perfectly healthy once a
minute forever, burying the real series. So a site emits 12 metrics when
unreplicated and 14 when replicated, and a GAP in these two series truthfully
means 'not replicated / not reporting'. Note false and null differ for the
connected flag: a configured link that is DOWN samples as 0; no link samples as
nothing. Both fold as Gauge (summing an oplog depth, or a 0/1 flag, per hour is
meaningless).
THRESHOLD SEMANTICS — sustained, not instantaneous. This is the design, not a
refinement. The site_events retention purge is CDC-captured like any other
write, so the backlog jumps by the deleted batch and drains as the peer acks
it; a bare 'backlog > threshold' badge would fire on every replicated site
every day and be ignored within a week. Healthy is defined by the backlog
RETURNING to ~0, which only a dwell can express. CentralHealthAggregator holds
LocalDbBacklogAboveSince — the instant the backlog first rose above
LocalDbBacklogThreshold (default 1000) — and the badge appears only after
LocalDbBacklogSustainedFor (default 15 min). A report at or below the threshold
clears the anchor; one above leaves the ORIGINAL instant intact, so the window
is sustained rather than sliding.
The null case is deliberately a third branch: a null backlog leaves the anchor
UNCHANGED rather than clearing it. Null is evidence neither of a drain nor of a
backlog, so a node going blind mid-climb must not silently reset its own dwell,
and an unreplicated site (null forever) never accumulates an anchor and can
never be badged.
Validation rejects a non-positive threshold (0 would badge every replicated site
permanently, since an idle backlog already sits at it) and a dwell shorter than
one ReportInterval (satisfiable by a single sample, which collapses the check
back into the bare threshold it replaces).
DOC CORRECTION worth flagging: this row's premise was 'mirror the SiteAuditBacklog
configurable threshold badge'. That badge does not exist — Component-HealthMonitoring.md
described it in two places but no option, no comparison and no markup were ever
implemented. The two stale claims are corrected here rather than left to send the
next reader looking for it; the LocalDb badge is built fresh on the Metrics-stale
badge's pattern, which is a real configurable-threshold badge.
Tests: HealthMonitoring 115 (9 new aggregator tests incl. purge-burst-drains,
unreplicated-never-anchors, null-mid-climb, exact-threshold, stale-report; 5 new
validator tests; 4 new sampler tests incl. the null-semantics and false-vs-null
cases); CentralUI 1042 (5 new badge/chart tests incl. the purge-burst negative
control); Commons Kpi 93; KpiHistory 54.
BundleUnlockRateLimiter was registered (ServiceCollectionExtensions), injected
into BundleImporter, unit-tested, documented on
TransportOptions.MaxUnlockAttemptsPerIpPerHour = 10 and pinned in
appsettings.Central.json — but _unlockRateLimiter was never invoked. A
brute-force passphrase attacker against a stolen bundle was bounded only by the
per-bundle 3-strike lockout, which a re-export side-steps (new ContentHash).
LoadAsync now registers every unlock attempt against the limiter. Ordering is
load-bearing: AFTER the per-bundle lockout check (hammering a locked bundle
spends no hourly budget) and BEFORE the decrypt (so a denial burns no per-bundle
strike — otherwise the throttle would be a way to lock a bundle someone is
legitimately importing). Over budget throws the new
BundleUnlockThrottledException, fail-closed with a clear message on both
surfaces; it lives in Commons because the management surface (Commons-only
references) must catch it to return a curated error rather than MapFault's
generic internal-error text.
Client key plumbing:
- Central UI: new scoped CircuitClientAddressProvider snapshots
Connection.RemoteIpAddress at circuit establishment — the same constraint and
the same pattern as CookieAuthenticationStateProvider, since HttpContext is
null by the time a circuit submits a passphrase.
- CLI/management: additive ManagementEnvelope.ClientIp, stamped by
ManagementEndpoints and threaded through DispatchCommand to the preview/import
bundle handlers.
- No attributable address => a single SHARED "<unattributed>" bucket, not a
bypass: unknown clients get a stricter budget, never none. Per-client
isolation is only real behind a ForwardedHeaders-trusted proxy, the same
caveat the login throttle carries (Component-Security.md).
Docs: Component-Transport.md + docs/components/Transport.md record the two-level
ordering, the client-key sourcing, the shared-bucket fallback and a
troubleshooting entry for the new error.
Tests: 6 importer tests (cap enforced; denial != passphrase failure, no strike
burned; per-client isolation; shared unattributed bucket; unencrypted bundles
consume no budget; a locked bundle spends no hourly budget) — 4 of the 6 fail
with the limiter call disabled, i.e. against the dead-code state. Plus
ManagementActor tests pinning the ClientIp forwarding and the curated
throttle error.
Only BundleImporter.ApplyAsync released an import session, so every path that
opened one and did NOT apply held one of MaxConcurrentImportSessions (8) — and
the DECRYPTED bundle content that T-007 deliberately zeroes on the apply paths —
for the full BundleSessionTtlMinutes (30). Eight abandonments locked out all
imports on the active node; observed live (R6: 3 back-to-back suite runs held 6
of 8 slots until central-b was restarted).
New IBundleImporter.Abandon(sessionId): the same zero-then-remove release
ApplyAsync performs, idempotent (double release / unknown id / expired id are
no-ops) and only ever given the random-GUID id its caller opened, so it can
never evict another circuit's session.
Wired into every abandonment path:
- Central UI wizard — ReleaseSession() from new upload, TryLoadAsync (before it
overwrites _session), Back-to-Upload / ResetSessionState, the 3-strike lockout
reset, and Dispose (navigation away / circuit teardown).
- CLI transport preview — a preview never applies, so nothing released it at
all; now released in a finally.
- CLI transport import — covers the pre-apply aborts (blocker rows, unresolved
site/connection mappings).
Class doc's "an abandoned wizard does not leak" claim is true again (it only
ever covered the staged plaintext temp file). Component-Transport.md +
docs/components/Transport.md now record the explicit-release contract and that
BundleSessionEvictionService is a TTL backstop, not an abandonment path.
Tests: 4 bUnit tests over a REAL BundleSessionStore (dispose / back-to-upload /
apply-then-dispose idempotency / concurrent-circuit isolation) — 3 of the 4 fail
against the pre-fix page; 4 importer tests (release + zeroing, idempotency,
cross-session isolation, and the slot-cap regression).
The Deployment Status page client-materialized the whole deployment list. It read
EVERY DeploymentRecord — an insert-only table, one row per deploy attempt for the
retention window — plus EVERY Instance, then site-scoped, sorted, counted the four
status tiles and sliced a 25-row page in the Blazor circuit's memory. That ran on
first render AND on every IDeploymentStatusNotifier push, so the cost scaled with
the age of the system rather than the size of the page.
All four jobs move into SQL:
- `IDeploymentManagerRepository.QueryDeploymentListPageAsync(filter, page, size)`
returns one page of `DeploymentListRow` — DeploymentRecord INNER JOINed to
Instance, so the instance display name and site travel with the rows that need
them — plus the total count of the filtered set. The join is exact: the FK is
Restrict and DeleteInstanceAsync removes the records first, so no orphan exists.
- `GetDeploymentStatusCountsAsync(filter)` returns the tile counts from ONE grouped
aggregation, deliberately ignoring the filter's Status: the tiles are the status
BREAKDOWN of the filtered set, so honouring it would zero three of four tiles the
moment an operator clicked one.
- Site scoping runs in the query as `SiteIdScope` resolved through the record's
instance (DeploymentRecord has no SiteId of its own). An EMPTY grant stays a real
filter matching nothing, never "unconstrained".
- The now-callerless whole-table `GetAllDeploymentRecordsAsync` is deleted.
OFFSET paging, not the Audit Log's keyset cursor, and deliberately so: this page's
pager is numbered and jump-to-any-page, so it needs a page count, which only a total
can give it — a keyset cursor can express neither, and the total is required for the
tiles regardless. The deep-offset cost that pushes high-volume tables to keyset is
bounded here by the terminal-record retention purge, unlike the 365-day AuditLog.
This mirrors the Notification Outbox, offset-paged for the same reason. Ordering is
DeployedAt DESC, Id DESC — the Id tie-break is load-bearing, because DeployedAt ties
on rapid redeploys and an unstable sort key makes offset paging repeat or drop rows.
UI: the four status tiles become the status filter (click to apply, click again to
clear, aria-pressed, phrasing-only content so a <button> stays valid), plus a
free-text search matched DB-side against instance name, deployment id, revision hash
and initiating user. Search is TRAILING-edge debounced at 500ms — the same Timer +
lock + _disposed idiom as the existing leading-edge push coalescer, minus the leading
edge, because a search box must not query on the first keystroke. A filter change
resets to page 1; a page past the end falls back to the last real page. Bootstrap
only, existing PagerWindow pager retained.
The WP2.4 push coalescing is unchanged and still earns its keep: server paging shrank
what a reload costs, not how many arrive — it now bounds database round-trips rather
than table scans.
Tests: 19 new SQLite repository tests (paging slice + total, tie-break stability
across pages, page/size clamping, past-the-end, the joined projection, every filter
dimension incl. the empty-scope security case, and the grouped counts' status-blind
contract); 15 new bUnit page tests (page-1 request, server total drives the pager,
Next re-queries, tiles show server counts not page counts, tile filter + toggle,
system-wide vs site-scoped scope push, debounce collapses a keystroke burst to one
query, clear-filters, dispose with an armed timer). The two existing Deployments
suites re-point their reload assertions at the new query.
Doc: Component-CentralUI.md Deployment section — the "no server-side paging" known
residual is replaced by the shipped design.
ScriptOptions.WithReferences(Assembly[]) resolves each assembly through
MetadataReference.CreateFromFile, which does not cache: every call mints a
fresh AssemblyMetadata -> PEReader -> NativeHeapMemoryBlock holding an
unmanaged copy of the assembly metadata that nothing disposes. Building the
options per compile therefore leaked native memory permanently — invisible to
the GC, to gcdump and to the managed allocation counters, so the working set
grew while the GC heap did not.
Diagnosed from a live dump of a wonder-app-vd03 Site node: 2,885 MB working
set 78 min after a cold start, only 150 MB live GC heap, ~2,469 MB on the
default process heap across ~6,700 undisposed AssemblyMetadata instances
against 473 DLLs on disk.
Three sites, all hoisted to static readonly:
- SiteRuntime ScriptCompilationService (the dumped one)
- InboundAPI InboundScriptExecutor — same defect on the central node; method
compiles recur on every re-registration and revision change
- CentralUI ScriptAnalysisService — CreateFromFile per sandbox run
ScriptAnalysis RoslynScriptCompiler also builds options per call but draws
from the static ScriptTrustPolicy.DefaultReferences, so it mints no metadata
and is left alone.
Guarded by reference-equality on the artifact rather than by watching memory:
a bytes-watching test would be flaky, and the leak is native so the managed
counters cannot see it at all. The test is proven to fail before the fix.
This does NOT close the AddTemplateScript OOM — that path was shown twice not
to compile scripts. It explains how a long-running node reaches a native
memory state where a large allocation fails with gigabytes free, which is a
lead worth re-testing, not a closure.
Gating the detail modal/drawer on a held id rather than on the row resolving
(d14e0ee4) made it the surface's own job to clear that id when navigation
invalidates page-scoped state. ParkedMessages and ConfigurationAuditLog did not,
so paging away from an open row left the surface mounted on a notice it could
never recover from — reachable only by paging back.
The criterion is per-surface and comes down to whether the modal has content of
its own:
ParkedMessages, ConfigurationAuditLog — no keyed detail fetch; content resolves
from the loaded page alone. An entry paged out of view can never resolve again,
so these must clear on paging. They already cleared it on Search/OnSiteChanged
for the same reason, and clear _selectedIds on paging for the same reason
again; paging was simply missed.
NotificationReport, SiteCallsReport — fetch detail by id, so the modal still
shows real content after its row leaves the page. These deliberately do NOT
clear on paging and are unchanged.
Clearing on an explicit navigation action is user intent, not a resolve-driven
unmount, so this cannot reopen the handler-disposal race that d14e0ee4 closed.
PageScopedDetailStateTests covers all three paging entry points and records the
criterion so the next reader can tell why two surfaces clear and two do not. Run
against both clears reverted, all three fail; restored, all three pass.
CentralUI.Tests 994/994, solution build 0/0.
Also corrects two comments in ParkedMessages left stale by d14e0ee4 — they still
described the drawer as self-closing when a row stops resolving, which is the
behaviour that change deliberately removed.
The sweep's modal re-key (holding the row's id and re-resolving it, rather than
holding the record) also used that resolve as the modal's visibility gate. That
makes the modal's existence a function of list contents: any render where the
row is momentarily unresolvable unmounts the whole subtree and disposes every
event-handler id inside it, Close's included. A click already in flight against
a disposed handler makes the renderer throw GetRequiredEventBindingEntry during
DispatchEventAsync — which is how this surfaced, as an intermittent failure of
CloseButton_DismissesModal (989/990 on one run, green on re-run).
The record-held form made that structurally impossible: the modal existed
because the user opened it, and no list mutation could retract that. This
restores the property while keeping the re-key's actual benefit. Visibility now
gates on the held id; the resolve drives only content. An unresolvable row
degrades to an explicit notice and hides the row-scoped actions, while the frame
and Close stay mounted. Detail fetched by id still renders, so the user does not
lose the body they opened.
Applied to all four surfaces that shared the construction: NotificationReport,
ConfigurationAuditLog, ParkedMessages (offcanvas drawer) and SiteCallsReport.
Modal_StaysOpen_WhenItsRowLeavesThePage drops the opened row from the next query
and asserts the modal survives, keeps its fetched body, hides Retry/Discard, and
that Close still works. It was run against a deliberately restored defective
gate and failed there before passing here — a regression test that passes both
ways would be worthless against a race. 20 consecutive runs of the previously
flaky class: no failures. CentralUI.Tests 991/991, solution build 0/0.
The plan doc gains a section recording that the sweep was reported as
behaviour-preserving when it was not, and why the merge review missed it.
Applies the family-wide admin-UI cleanup playbook to the Central UI so the
Blazor surfaces stop diverging from the shared kit: buttons are grouped rather
than individually sized, long cell values are contained instead of widening
tables, and hard-coded colours give way to theme tokens.
The headline fix is that MainLayout passed Accent="#2f5fd0" to ThemeShell,
which the kit emits as an inline style on the shell root. Being a descendant of
<html>, it beat the [data-bs-theme="dark"] override for the entire app, so the
dark accent had never rendered. Declaring --accent in site.css :root instead
lets both schemes resolve; light is unchanged because the value already matched
the kit's light default.
Theme pins to 0.4.1, which upstreams the local .btn sizing block verbatim, so
that block is deleted here rather than duplicated. Verified byte-identical
before removal; the repo now declares no --bs-btn-* anywhere.
NOT purely cosmetic, contrary to the sweep's stated scope: four detail-modal
surfaces (NotificationReport, ConfigurationAuditLog, ParkedMessages,
SiteCallsReport) were additionally refactored from holding the selected record
to holding its id and re-resolving from the current page each render, with the
resolve doubling as the visibility gate. A background refresh that drops the
row now closes the modal instead of showing a stale snapshot. This is a
behaviour change and is called out rather than buried: a full-suite run turned
up one intermittent CentralUI failure, CloseButton_DismissesModal, whose stack
(GetRequiredEventBindingEntry during DispatchEventAsync) indicates the handler
was disposed between render and click — a window the previous field-held record
made structurally impossible. Treat the modal lifecycle here as unreviewed.
Build 0/0; suite green apart from that one intermittent failure.
MES alarm-status API §5.2 (docs/plans/2026-06-30-mes-alarm-status-api.md,
Phase 1 tasks 2-4). Site `Call` scripts had NO way to read alarm condition
state: the `Alarm` global exists only inside an on-trigger handler and
describes the one alarm that fired, and native mirrored conditions were
reachable only from the Debug View. That gap blocked the CvdReactor
SimpleAlarmStatus/AlarmStatus scripts entirely -- they cannot be written
without it. `Alarms.CurrentAsync()` closes it.
The data was already local: the script runs inside its own Instance Actor's
context, so this is a LOCAL Ask -- the same mechanism attribute reads use, no
cross-cluster hop. A dedicated GetAlarmSnapshotRequest/Response is used rather
than reusing DebugSnapshotRequest, which would materialise every attribute
value on every alarm poll; both are served from the same
BuildAlarmStatesSnapshot(), so the script view and the operator's Debug View
can never disagree.
Deliberate shape decisions:
- NOT scope-prefixed, unlike Attributes. Alarm identity is not a
scope-relative attribute name (computed alarms are keyed by configured
name, native conditions by a source-supplied reference), so prefixing
would hand a composed script a silently truncated list.
- Read-only. Native alarms are a read-only mirror of the source (no
ack-back), so no acknowledge/shelve operation is exposed.
- Placeholder rows are NOT pre-filtered: a caller must be able to tell
"binding configured and quiet" from "binding unknown". The documented
filter is `Active && !IsConfiguredPlaceholder`.
- ScriptAlarm lives in Commons so the runtime accessor and the compile-only
surface project to the SAME type -- a script binding at the design-time
gate binds identically at the site. Condition is the authority for
active/acked/severity, so one filter expression works across computed and
native alarms.
Mirrored on BOTH design-time surfaces. ScriptCompileSurface is covered by the
reflection parity guard (AlarmsAccessor added to its mirror pairs). The Central
UI Test-Run SandboxScriptHost is the third, hand-maintained mirror that the
parity test cannot reach (Central UI does not reference Site Runtime); without
it the design page would false-flag CS1061 on scripts the deploy gate accepts.
It throws a labelled ScriptSandboxException at run time -- there is no central
route to per-instance alarm state, and returning an empty list would read as
"nothing is in alarm", which is worse than an error.
ScriptTrustPolicy needs NO change, and the reason is structural rather than
incidental: the trust boundary is a deny-list over API roots, not an allow-list
of context members. A test pins that no ForbiddenScopes entry prefixes the
Commons script-surface namespace, so a future deny-list entry cannot silently
make ScriptAlarm untouchable.
Tests: 6 accessor cases (Ask contract, full native projection incl. AckTime,
unacked, computed-alarm derivation, placeholder visibility, scope-independence),
2 InstanceActor snapshot cases incl. equality with the Debug View row set, the
full MES script shape compiling against ScriptCompileSurface, 2 trust cases,
and a sandbox diagnose-clean case reading every projected ScriptAlarm field.
T34c only fixed the bounded modal-surface offenders. Bootstrap 5.3's bg-light
and bg-white are fixed light values that do NOT flip under [data-bs-theme=dark],
so every remaining use was a dark-mode contrast break. 35 swaps across 19 files:
surface / <pre> / <code> bg-light -> bg-body-secondary
panel bg-white -> bg-body
neutral badge bg-light text-dark -> bg-secondary-subtle text-secondary-emphasis
muted badge / input group bg-light text-muted -> bg-body-secondary text-body-secondary
DELIBERATELY LEFT (7 sites): the neutral member of a status-badge switch or
ternary whose siblings are all solid, non-theme-aware colours (bg-success,
bg-danger, bg-warning) — swapping only the neutral one to a subtle token breaks
the visual weight of the set, so these stay until the whole family is retoned:
Topology.razor:513 (Current badge) and :588 (InstanceState.NotDeployed)
InstanceConfigure.razor:1520 (same InstanceState switch)
NotificationReport.razor StatusBadgeClass fallback
TransportImport.razor ConflictKind badge fallback
SecuredWrites.razor text-bg-light fallback (a text-bg-* family with no
theme-aware member at all)
Health.razor:377 depth ternary
Also untouched by design: SchemaBuilder.razor:88 (already bg-light-subtle,
theme-aware), bg-dark text-light console panels, and site.css / #reconnect-modal.
InstanceConfigure.razor is edited concurrently elsewhere; its change here is 5
pure class-string swaps on existing lines, no reflow. No test asserted any of
these classes. Also marks all four M10 residuals done in the plan doc.
DECISION: a pager whose only two controls are permanently disabled is noise, so
the conventional behaviour ships — the bar is hidden when the result set is a
single page. NotificationReport, which raised the finding, opts in.
RATIONALE for opt-in rather than default-on. OffsetPager is shared, and its
summary span ("Page N of Y · T total") doubles as the ONLY total-count readout
on ConfigurationAuditLog — auto-hiding there would silently delete the result
count for every query returning under a page, and the existing Playwright
fixture asserts "3 total" on exactly such a query. So the behaviour lives in the
shared component (reusable for the next consumer) behind HideWhenSinglePage,
default false, which keeps every current consumer byte-identical.
The guard fires only when single-page-ness is POSITIVELY established:
PageCount <= 1 AND Page <= 1 AND !HasNextPage. A null TotalCount means the host
cannot count, so the bar renders — the controls never vanish while a further
page still exists.
NotificationReport's own Playwright pagination test seeds 51 rows (2 pages) and
is unaffected. 5 new bUnit tests cover hide-on-single-page, hide-on-empty,
show-on-multi-page, show-when-TotalCount-unknown, and the default-off path.
TemplateEdit was the last page still hand-rolling its own modal chrome after
M10 — T34c only tokenized its backdrops. All four member-authoring forms
(Attribute, Alarm, Native Alarm Source, Script) now open through
IDialogService.ShowAsync, so the single DialogHost in MainLayout owns the
backdrop, focus trap, Escape and focus restoration.
Pattern copied from the already-migrated pages (MoveDataConnectionDialog +
Templates' Move/Rename dialogs): each body is its own component beside the page
taking a DialogContext<bool>, owning its form state, rendering validation and
server errors INLINE while staying open, and closing with Close(true) only once
the save succeeded — at which point the page reloads. An inline RenderFragment
would NOT have worked: DialogHost renders the captured fragment in its own tree,
so the page's StateHasChanged could never refresh it.
Persistence deliberately stayed on the page (it owns TemplateService, the
inherited-member rules, and the repository-direct native-source path) and is
reached through an OnSaveAsync delegate returning null on success or the message
to display. Behaviour preserved verbatim, including the List-attribute encode +
Decode round-trip check, the name/trigger-type read-only-on-edit rules, the
duplicate-native-source-name guard, and NormalizeExecutionTimeout.
TemplateScriptDialog keeps all four tab panels mounted (Monaco and the JSONJoy
island must not tear down on tab switch) and hosts the Test Run panel, which
needs the live unsaved editor buffer, so it injects ScriptAnalysisService
directly and cancels an in-flight run on dispose.
Extraction moved markup that three structural source-scanning tests pinned;
all three were repointed at the new files rather than weakened:
- TemplateNativeAlarmSourceEditorTests (+ a new test asserting the form is
host-mounted and the body renders no chrome of its own)
- AttributeListEditorTests (list-editor reveal now in the dialog body)
- TestRunWarningTests (Real I/O warning travelled with the script panel)
Build 0/0; CentralUI.Tests 973/973 green.
TreeView<TItem> handled only Enter/Space on the chevron; the tree itself was
unreachable by keyboard. Implements the WAI-ARIA tree pattern:
- Roving tabindex on li[role=treeitem] — exactly one node is tabbable, so the
whole tree is a single Tab stop. Target resolved per render: last-focused node
while still visible -> SelectedKey -> first visible node. Browser focus follows
via a per-node ElementReference + FocusAsync in OnAfterRenderAsync, guarded by
the same JSException/JSDisconnectedException/InvalidOperationException triple
the context-menu focus already used (a no-op under bUnit).
- ArrowDown/ArrowUp move between VISIBLE nodes; ArrowRight expands a collapsed
branch else moves to first child; ArrowLeft collapses an expanded branch else
moves to parent; Home/End jump to first/last; Enter/Space activate through the
SAME path a click takes (OnContentClick / OnCheckboxToggle) so selection
semantics never diverge between input modes.
- ARIA: aria-level, aria-posinset, aria-setsize added; aria-selected now renders
true/false on selectable+checkbox trees and stays absent on non-selectable ones.
- Event scoping: @onkeydown:stopPropagation on the li, chevron, checkbox and
content slot, so nested nodes do not double-handle and consumer controls inside
NodeContent keep their own key handling. Browser scroll-on-Space/Arrow is
suppressed by a minimal NATIVE inline onkeydown on the root ul, targeted at
treeitems only — Blazor's preventDefault directive is all-or-nothing per
element, so on the li it would trap Tab and cancel Enter/Space on consumer
buttons. No CSP is configured, so the inline handler runs.
BuildVisibleNodes() mirrors RenderNode's visibility rules and must stay in step.
Tests: 31 new bUnit tests in TreeViewKeyboardNavigationTests; the 47 existing
TreeView tests are unchanged and still green. Docs: new keyboard/a11y section in
docs/components/TreeView.md.
The Native Alarm Source Overrides card had no bulk affordance — the CSV path
shipped CLI-only (instance native-alarm-source import --file), so the UI could
only retarget one source at a time inline.
Adds a second InputFile on that card, mirroring the attribute importer's UX
(hidden input behind a button-styled label, 512 KB cap, success/error alert
with the per-line error list, toast). Parsing reuses the SHARED
NativeAlarmSourceOverrideCsvParser — the exact parser the CLI uses — and the
new pure InstanceConfigure.BuildNativeAlarmSourceCsvImport applies the same
batch rules the server enforces in
ManagementActor.HandleSetInstanceNativeAlarmSourceOverrides: the source must
resolve, must not be template-locked, and may appear at most once; any error
rejects the whole file and applies nothing.
Semantics match the CLI: merge, not full replace — sources absent from the
file keep their existing override, a blank field keeps the inherited value,
and an all-blank row clears that source's override (equivalent to the CLI's
all-null override row, without leaving a dead row behind).
Persistence reuses the inline editor's path: SaveNativeOverride's upsert body
is extracted to UpsertNativeOverrideCore (no SaveChangesAsync inside), so the
import commits the validated batch in one SaveChangesAsync — no new server
method, no duplicated parsing.
Tests: InstanceConfigureNativeAlarmCsvImportTests (9) — happy path, blank-field
inheritance, all-blank clear, merge semantics, unknown/locked/duplicate source
and parser-error rejection, plus structural pins on the InputFile wiring.
Docs: Component-CentralUI.md native-alarm-source card gains the import bullet.
Central and each site are SEPARATE Akka clusters, so central cannot act on a
site's membership -- it asks. New TriggerSiteFailover/SiteFailoverAck contract
travels the existing ClusterClient command/control channel (mirroring the
RetryParkedOperation relay); the site's own SiteCommunicationActor performs the
graceful Leave and acks the outcome.
- ClusterFailoverCoordinator moved out of Host into Communication/ClusterState,
beside ActiveNodeEvaluator. Both paths now share ONE oldest-Up implementation;
SiteCommunicationActor cannot reference Host, and the two definitions must not
drift or the node asked to leave stops being the singleton host.
- Site scope is the SITE-SPECIFIC role (site-{SiteId}), not the base Site role --
site singletons are placed on the former, so the base role would move the wrong
node. Pinned by a unit test asserting the role string and by a real-cluster test.
- Site-side guards: refuses a command addressed to another site (a misroute must
never fail over a site the operator did not select), refuses when there is no
peer, and reports a fault as an ack rather than throwing into supervision --
a restart there would drop central's Ask into a bare timeout and lose the reason.
- Ack is sent before the Leave takes effect so it still reaches central.
- UI: the same control now serves both scopes via a SiteId parameter. The site
confirmation deliberately does NOT claim the admin's page will disconnect --
it won't, and crying wolf there devalues the central warning that is real. A
site refusal and an unreachable site surface distinctly.
- Rolling upgrade: a site on an older binary has no handler, so the message
dead-letters and the Ask times out, reported as "site did not respond". That
is the honest outcome; documented on the contract.
Fallout fixed: HealthPageTests now renders the page inside
CascadingAuthenticationState with the real policy set and IAuthorizationService,
because the cards embed an AuthorizeView. That mirrors production, where the
layout supplies the cascading value.
CentralFailoverControl lives in Components/Health/ (matching AuditKpiTiles /
SiteCallKpiTiles) rather than inline in Health.razor, so it is testable without
standing up the whole dashboard's DI graph.
- Admin-gated via AuthorizeView + RequireAdmin. The Health page itself is
intentionally all-roles, so the gate belongs on the control, not the page.
- Disabled with an explanatory title when the pair has no online standby;
the authoritative guard remains server-side against live cluster membership.
- Confirmation dialog (IDialogService, the page idiom) warns that singletons
hand over, in-flight work on the active node is interrupted, and THIS PAGE
will disconnect and reconnect against the new active node -- Traefik routes
the UI to the node being restarted, so a working failover otherwise reads as
a crash the admin caused.
- A refused failover (service returns null) surfaces the refusal; the UI never
reports a failover that did not happen.
7 bUnit tests. Two harness requirements that bit first: AuthorizeView needs a
cascading AuthenticationState (the app supplies it from the layout), and
BunitContext pre-registers a placeholder IAuthorizationService that throws on
policy evaluation -- both handled the same way NavMenuTests documents.
Runbook paragraphs added to Component-ClusterInfrastructure.md (new Manual
Failover section) and docker/README.md.
Admin-triggered failover of the central pair. IManualFailoverService is declared in
CentralUI (plain strings, so that project stays Akka-free); AkkaManualFailoverService
implements it in the Host and is registered only in the Central branch.
- Leave, never Down: singletons hand over instead of being killed.
- Target = oldest Up member with the Central role, mirroring ActiveNodeEvaluator, so
the node acted on is exactly the one hosting the singletons (never the leader,
whose address-ordered definition diverges from singleton placement after a restart).
- Peer guard: returns null when fewer than 2 Up Central members — failing over a lone
node is an outage, not a failover.
- Audited BEFORE the Leave is issued via ICentralAuditWriter: the acting node can be
the one that goes away, and an audit written after could be lost to the shutdown it
describes. Best-effort — audit failure never blocks the failover.
New audit taxonomy: AuditChannel.Cluster + AuditKind.ManualFailover (operator-initiated
topology actions are not script trust-boundary crossings, but are exactly what an audit
log exists to attribute). Lock-in tests updated 5->6 channels, 16->17 kinds.
alog.md §4 updated per the lock-in tests' contract. Both tables were already stale --
the Channel row omitted SecuredWrite and the Kind table claimed 10 while the code had
16 -- so they are completed here, not merely appended to.
ManualFailoverTests: 3 real-cluster tests (oldest leaves + survivor takes over, peer
guard refuses with a positive still-running assert, dry-run probe does not perturb).
Tasks 2 + 5a-writer + 5b of the LocalDb Phase 1 plan, landed together because
they are one indivisible change: the schema, the writer that fills it, and every
consumer that assumed the old id semantics.
WHY the id changes. Site pairs will replicate site_events with last-writer-wins on
the primary key. With INTEGER PRIMARY KEY AUTOINCREMENT both nodes independently
mint 1, 2, 3... for unrelated events, so sync would treat them as the same row and
silently overwrite. A GUID makes the event log a pure union across the pair.
Task 2 - schema extracted so the Host's AddZbLocalDb onReady can create the tables
before RegisterReplicated installs capture triggers (pre-registration rows are
never captured):
- new OperationTrackingSchema.Apply / SiteEventLogSchema.Apply, plain
Microsoft.Data.Sqlite, no LocalDb dependency
- both stores delegate their InitializeSchema to them (idempotent, so a
directly-constructed store still works)
- OperationTracking is unchanged - it already had a TEXT PK and replicates as-is
Task 5a - SiteEventLogger mints Guid.NewGuid("N") per event and inserts it.
Task 5b - three consumers assumed a monotonic integer id. All three move to
timestamp ordering; leaving any one behind would be a live bug:
- EventLogQueryService: "id > $afterId" would return an ARBITRARY subset of a
GUID-keyed table and SILENTLY DROP ROWS from page-through. Now a composite
(timestamp, id) keyset cursor with an opaque string token; timestamps are not
unique, so id is the tie-break that guarantees exactly-once paging.
- EventLogPurgeService: "ORDER BY id ASC LIMIT 1000" would delete a RANDOM batch
instead of the oldest. Now orders by timestamp.
- EventLogEntry.Id and both ContinuationTokens: long -> string.
WIRE COMPATIBILITY. Those DTOs cross the site<->central Akka boundary
(SiteCommunicationActor -> CommunicationService -> ManagementActor / CentralUI).
No rolling-upgrade shim is needed because both sides ship in the same deployable
and the rig redeploys as a unit. Checked: no Akka serializer binding pins these
types by name. A stale numeric token degrades to "start from the beginning"
(a visible repeat) rather than throwing or losing rows.
Tests: the two that encoded the old semantics were rewritten to guard the new
invariant rather than deleted - uniqueness instead of monotonicity, and
oldest-purged-first keyed on timestamp. That second test also exposed a latent
weakness: the bulk seed stamped every row with the same UtcNow, so "oldest" was
never actually well-defined; rows now get distinct increasing timestamps, kept
inside the retention window so the retention purge does not eat them first.
Verified:
dotnet build ZB.MOM.WW.ScadaBridge.slnx -> 0 Error(s), 0 Warning(s)
SiteEventLogging.Tests -> 70 passed (12 new schema tests, red-first)
CentralUI.Tests -> 925 passed
Commons.Tests -> 684 passed
SiteRuntime.Tests -> 529 passed, 1 pre-existing flaky failure
(InstanceActorChildAttributeRaceTests - passes 3/3 in
isolation with and without this change)
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Joseph Doherty